_compute_user_env_var_status keyed coverage off global_values presence, so a
per-user var with a same-named empty-valued global was reported not-required
while _resolve_static_headers_with_env_vars (which now filters empty globals)
still 412s on it. Apply the same empty-global filter so the status endpoint and
the tool-call path agree on what the user must supply.
A leading or trailing space in a static header value made httpx/h11 reject it
as an illegal header value, which aborted the MCP transport task group,
cancelled session.initialize(), and surfaced only as an opaque "Failed to
connect to MCP server". Strip surrounding whitespace from header names and
values in _get_auth_headers so every path (test connection, runtime, config,
DB) is protected, and mirror the trim in the UI static-header reducers so
stored values stay clean.
When the transport task group does fail, unwrap the ExceptionGroup raised on
transport context exit and re-raise the real cause (httpx ConnectError,
LocalProtocolError, ...) instead of the cancellation, so the proxy logs and
the admin test endpoint explain why the connection failed. The test endpoint
maps known causes to an actionable, secret-safe message and stops swallowing
genuine cancellation
An empty scope=global value was keyed by presence, so it masked a referenced
per-user var (suppressing the 412 prompt) and won the merge over a value the
user did supply, interpolating an empty string into the header. Drop empty
globals so they neither cover a required user var nor override a user value;
the unresolved ${NAME} is then left untouched like any undefined reference.
* fix(anthropic): route Claude Opus 4.8 through adaptive thinking
Opus 4.8 uses the same adaptive thinking contract as 4.6/4.7
(thinking.type=adaptive plus output_config.effort), but
_is_adaptive_thinking_model only recognized 4.6/4.7 by name and otherwise
leaned on the supports_adaptive_thinking cost-map flag. The Bedrock,
Vertex, and Azure 4.8 entries don't carry that flag, so a
bedrock/us.anthropic.claude-opus-4-8 request fell back to the legacy
thinking.type=enabled shape and Bedrock rejected it with "thinking.type.enabled
is not supported for this model".
Add _is_claude_4_8_model and wire it in next to the existing 4.6/4.7
matchers in the adaptive-thinking detection, the effort=max gate, and the
supported-params check, so every provider path treats 4.8 as adaptive
regardless of whether its cost-map entry advertises the flag.
* refactor(anthropic): drive Opus 4.8 adaptive thinking from the cost map
Replace the _is_claude_4_8_model name matcher with cost-map data. Add
supports_adaptive_thinking to every Opus 4.8 provider variant (Bedrock
regional/global, Vertex, Azure) in both the root and bundled cost maps, and
move the prefix-resolving capability lookup (_supports_model_capability) down
to AnthropicModelInfo so _is_adaptive_thinking_model reads the flag through the
bedrock/invoke/, bedrock/, and vertex_ai/ prefixes. The 4.6/4.7 name checks
stay as a fallback since their provider entries don't carry the flag yet.
A pure data fix is not enough on its own: _supports_factory doesn't strip the
us.anthropic./invoke/ prefixes, so bedrock/invoke/us.anthropic.claude-opus-4-8
would still miss the flag without the resolver change.
Add a cost-map guardrail test asserting every claude-opus-4-8 variant carries
the flag, so a future variant added without it fails CI instead of silently
sending the legacy thinking.type=enabled shape that the provider rejects.
pg_advisory_xact_lock() returns void; routing it through query_raw made
the Prisma engine try to deserialize that column and raise RawQueryError,
500ing POST /v1/mcp/server/{id}/user-env-vars. execute_raw doesn't
deserialize a result set, so the lock is taken without the void error.
Add a regression test whose fake tx.query_raw raises the real void
RawQueryError while execute_raw succeeds, and switch the concurrency
test's fake to execute_raw so it can no longer pass against the bug.
* Add OAuth M2M support for A2A agents targeting Databricks Apps
Databricks App endpoints reject static bearer tokens and require a
short-lived OAuth token minted via the workspace OIDC token endpoint.
A2A agents could previously only authenticate outbound with static_headers
or client header passthrough, so Databricks App agents could not be
registered.
Agents configured with a databricks_oauth block in litellm_params now mint
and cache a client_credentials token and attach it as the outbound
Authorization header on both message/send and message/stream calls,
overriding any statically configured Authorization.
* Add tests covering Databricks App OAuth token error paths
Cover the HTTP status error, transport error, non-object JSON body, and
invalid expires_in fallback branches in the token cache so the failure
handling is locked in by regression tests.
* Harden Databricks App OAuth token cache
Cap the cache TTL at the token's own lifetime so a token whose validity is
shorter than the refresh buffer is never cached and served stale; include a
digest of client_secret in the cache key so a rotated secret mints a fresh
token instead of reusing the old one; and prune the per-key lock when its
cached token is evicted so the lock map stays bounded by the live key set.
* Clear per-key locks on Databricks OAuth cache flush
* fix(a2a/databricks): mint OAuth token via Basic auth header, not unsupported auth= kwarg
litellm's AsyncHTTPHandler.post (what get_async_httpx_client returns) has no
auth parameter, so minting a Databricks App OAuth token raised
"AsyncHTTPHandler.post() got an unexpected keyword argument 'auth'" before any
network call ever left the proxy, breaking the feature end to end. The handler
also calls raise_for_status() internally and re-raises a MaskedHTTPStatusError
(a subclass of httpx.HTTPStatusError), so the explicit raise_for_status() after
post() was dead code.
Build the HTTP Basic Authorization header by hand and pass it via headers, which
is what the Databricks workspace OIDC token endpoint documents for client
authentication. The token-cache tests now model the real handler contract with
create_autospec so the rejected auth= signature is enforced; the previous mocks
accepted any kwargs and silently hid the bug.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Prune Databricks OAuth lock on the short-lived-token path
When expires_in is below the refresh buffer the token is intentionally
not cached, so _remove_key never runs for that key and the per-key lock
created by _get_lock leaked permanently. Drop the lock in that branch so
_locks stays bounded by the live key set, and assert the cleanup in the
short-lived-token test
* Gate A2A Databricks OAuth on the databricks_oauth block at the call site
Make the gating explicit where the header is applied so it is clear that only
agents configured with a databricks_oauth block enter the OAuth path; every
other agent is left untouched. Add a regression test asserting a non-Databricks
agent never invokes the token resolver.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver
Introduces a small helper layer used by every proxy-side rate-limit
hook so that the 429 they raise carries a populated llm_provider /
model — instead of an empty exception.llm_provider that downstream
loggers (Prometheus failure metric, observability callbacks) read as
'no provider attribution'.
ProxyHTTPRateLimitError inherits from both fastapi.HTTPException
(so the proxy server still renders it as a 429) and
litellm.exceptions.RateLimitError (so isinstance checks and
PrometheusLogger._get_exception_class_name pick up llm_provider).
We deliberately don't call RateLimitError.__init__ — it constructs
an httpx.Response we don't need and would just add failure surface;
attribute parity is what downstream consumers care about.
resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider
defensively. Internal limiter hooks fire from async_pre_call_hook —
well before get_llm_provider runs anywhere else in the request
lifecycle — so we have to call it ourselves at raise time. If the
model is missing or unparseable (alias, router-only model) we fall
back to llm_provider='litellm_proxy' rather than letting a second
exception leak out and break the request path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on parallel-request 429s
Both v1 and v3 parallel-request limiters fired bare HTTPException(429)
from inside async_pre_call_hook. The downstream Prometheus failure
metric reads exception.llm_provider via _get_exception_class_name —
the empty value showed up as exception_class='HTTPException' and
left model_id='None' on the time series.
Threads requested_model through every raise site in:
* parallel_request_limiter.py:
- check_key_in_limits (the per-key/per-model/per-user/per-team/
per-customer over-limit path)
- raise_rate_limit_error (zero-limit + global_max_parallel_requests
paths) — now takes an optional requested_model kwarg
* parallel_request_limiter_v3.py:
- _handle_rate_limit_error (the OVER_LIMIT translator), called
from both the should_rate_limit pre-check and the TPM
reservation path
Resolved via resolve_llm_provider_for_rate_limit so unknown / missing
models silently fall back to llm_provider='litellm_proxy' instead of
breaking the request path with a second exception.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s
Same plumbing change as the parallel limiters, applied to both
dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3:
* v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve
data['model'] -> (model, llm_provider) once and pass it into both
raises.
* v3: All three raise sites in _check_rate_limits — the
model_saturation_check enforced raise, the priority_model
enforced raise, and the fail-closed unknown-descriptor branch —
now attribute the 429 to the actual provider.
Falls back to llm_provider='litellm_proxy' when the model can't be
resolved.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s
batch_rate_limiter._raise_rate_limit_error now takes a
requested_model kwarg threaded from data['model'] in
_check_and_increment_batch_counters. The batch-creation 429 is what
gets raised when the input file's tokens/requests count would push
the per-key TPM/RPM window over its limit.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on budget/iterations 429s
Final batch of internal raise sites — the user/session-budget and
max-iterations hooks. Same pattern: resolve data['model'] once at
raise time, attach to ProxyHTTPRateLimitError so Prometheus and
observability callbacks can attribute the 429.
Hooks updated:
* max_budget_limiter (per-user max_budget exceeded)
* max_iterations_limiter (per-session agent iteration cap)
* max_budget_per_session_limiter (per-session dollar cap)
All three fall back to llm_provider='litellm_proxy' when data['model']
is missing or unparseable. Drops the now-unused HTTPException import
from each module.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy/hooks): pin provider field on internal rate-limit 429s
Regression coverage for the 'provider field missing' bug across every
proxy-side rate-limit hook + the helper layer:
* ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError,
dict-detail stringification, None-provider normalization).
* resolve_llm_provider_for_rate_limit happy paths
(gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback
branches (None, '', unknown name) plus a 'get_llm_provider raises'
case that asserts we swallow the secondary exception.
* For each limiter (parallel v1/v3, dynamic v1/v3, batch,
max_budget, max_iterations, max_budget_per_session): assert the
raised exception is a RateLimitError carrying the resolved
model + llm_provider, and a sibling test that asserts the
fallback path returns 'litellm_proxy' without leaking a second
exception.
* Two PrometheusLogger._get_exception_class_name pins so the
Prometheus failure metric label flips from 'HTTPException' to
'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on
fallback) — that's what dashboards consume.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* perf(proxy/hooks): defer provider resolution to over-limit branches
* fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail
* Consolidate rate_limiter_utils imports in dynamic_rate_limiter
* fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError
ProxyHTTPRateLimitError inherits from RateLimitError but did not call
RateLimitError.__init__, so num_retries/max_retries were never set.
When Starlette's HTTPException lacks __str__, MRO falls through to
RateLimitError.__str__, which unconditionally reads these attributes
and raises AttributeError during logging/traceback formatting.
Initialize them to None defensively.
* fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError
HTTPException declares 'status_code: int' while openai.RateLimitError
(via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy
flags the multi-base override as [misc] in CI lint. The runtime semantics
are fine (we set self.status_code in __init__), so silence the
class-level annotation conflict with a targeted ignore.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The merge of litellm_internal_staging brought in the location-pinned fetch()
lint rule from #29723, which bans raw fetch() outside src/lib/http/. The three
MCP per-user env-var helpers (getMCPUserEnvVars, storeMCPUserEnvVars,
listMCPUserEnvVarStatus) still called fetch() directly, pushing networking.tsx
to 244 no-restricted-syntax violations against a grandfathered baseline of 241;
because bulk suppressions are count-based, the overflow invalidated the whole
file's suppression and turned the lint gate red.
They now go through the shared apiClient, matching the rest of the file, which
keeps the count at the 241 baseline with no new grandfathered suppressions.
listMCPUserEnvVarStatus keeps its best-effort empty-list fallback so a failed
status fetch still can't break the page
Master-key rotation is a rare, high-stakes batch operation that previously
ran silently on success; only per-row decrypt failures were logged. Each of
the three MCP rotation steps (server credentials/global env vars, per-user
credentials, per-user env vars) now emits one info line summarising how many
rows were rotated and skipped, so an operator can confirm the step ran and
sanity-check the counts after rotating the key.
Regression test asserts the per-user env-var summary reports rotated and
skipped counts tied to real work: a decryptable row is re-encrypted and
counted as rotated while an undecryptable row is left untouched and counted
as skipped
* feat(guardrails): add sensitive data routing to on-premise models
When a guardrail detects sensitive data, route to an on-premise model
instead of blocking or redacting. All subsequent requests in that
session continue routing to the same model (sticky routing).
New config options for guardrails:
- on_sensitive_data: 'block' (default) or 'route'
- sensitive_data_route_to_model: target model for rerouting
- sticky_session_routing: persist routing for session (default: true)
New exception SensitiveDataRouteException triggers rerouting when raised
by guardrails. The proxy catches it, stores the routing decision in
cache, and modifies the request's model field.
New hook _PROXY_SensitiveDataRoutingHandler checks incoming requests
against cached routing decisions and applies sticky routing.
https://claude.ai/code/session_01SQd4isBa3UyouRoGVou9dK
* fix: black formatting for custom_guardrail.py
https://claude.ai/code/session_01SQd4isBa3UyouRoGVou9dK
* test: improve test coverage for sensitive data routing feature
Add additional tests for:
- Cache key format and TTL constants
- Session ID extraction from multiple locations
- Custom guardrail initialization with routing config
- Exception string representation and custom messages
- Redis cache paths including fallback behavior
- Edge cases in pre-call hook
https://claude.ai/code/session_01SQd4isBa3UyouRoGVou9dK
* fix: use correct GuardrailRaisedException parameters
Replace invalid 'source' parameter with 'guardrail_name' to match
the exception's actual signature.
https://claude.ai/code/session_01SQd4isBa3UyouRoGVou9dK
* test: move sensitive data routing tests to hooks directory
Move test file to align with source code structure.
https://claude.ai/code/session_01SQd4isBa3UyouRoGVou9dK
* fix(guardrails): honor sticky_session_routing flag and scope session routing per API key
Propagate sticky_session_routing through SensitiveDataRouteException so a
guardrail configured with sticky_session_routing=False reroutes only the
triggering request without persisting a session override. Scope the routing
cache key to the requesting API key so sessions from different tenants cannot
collide, and warn when sticky routing is requested but the hook is not
registered.
* refactor(guardrails): dedupe session-id extraction and drop redundant import
Extract the shared session-id lookup into get_session_id_from_request_data
so the sensitive-data routing hook and CustomGuardrail no longer keep two
identical copies of the logic. Remove the redundant local import of
GuardrailRaisedException in handle_sensitive_data_detection, and document
that detection_info is surfaced in request metadata and logs so it must not
carry raw sensitive values.
* fix(guardrails): guard None user_api_key_dict in sensitive data route handler
* fix(responses): send application/json Content-Type on responses DELETE
OpenAI's responses DELETE endpoint now rejects requests that arrive without
a Content-Type header, defaulting them to application/octet-stream and
returning 'Unsupported content type: application/octet-stream'. The delete
handler sent no body and therefore no Content-Type, so the request failed.
Declare application/json on the delete request, matching the OpenAI SDK.
* fix(guardrails): backfill in-memory cache after redis hit in sensitive data routing
When _get_routed_model resolves a routing override from Redis it now also
populates the local in-memory cache. Without the write-back, a non-writing
instance that only ever reads from Redis would lose the sticky routing
decision the moment Redis became unavailable, silently reverting sensitive
sessions to the default model.
* fix(guardrails): scope sticky sensitive-data routing to JWT principal
Keyless auth (JWT and similar) has no api_key, so every such caller shared
the "default" cache namespace. One authenticated user could reuse another
user's session_id, trip the guardrail, and silently force the other user's
subsequent requests onto the cached on-prem model for the TTL.
Resolve the routing tenant from the api_key when present, otherwise from a
stable principal built from the user/team/org identity, before reading or
writing the session route.
* fix(guardrails): require route target model when on_sensitive_data='route'
* fix(guardrails): mark user_api_key_dict Optional in sensitive-data route handler
* fix(guardrails): use remaining redis ttl for local backfill and str env default
* fix(guardrails): graceful block when routing configured but no session_id
handle_sensitive_data_detection promised to raise only SensitiveDataRouteException
or GuardrailRaisedException, but when routing was configured and the request had no
session_id it let a ValueError from raise_sensitive_data_route_exception propagate,
surfacing as an HTTP 500 instead of a block. Fall back to a graceful block in that
case so the documented contract holds.
* fix(guardrails): run remaining guardrails after sensitive-data reroute
Defer the SensitiveDataRouteException until every guardrail in the
pre-call loop has run, so downstream security guardrails are no longer
skipped when an earlier guardrail triggers routing. The first reroute
wins and a later guardrail that blocks still propagates.
Also normalize on_sensitive_data to lowercase like sibling on_* config
fields so case-insensitive values are accepted.
* fix(guardrails): classify sensitive-data reroute as guardrail intervention
* fix(guardrails): record sensitive-data reroute as prometheus intervention not error
* fix(guardrails): record service span for routing guardrail and move case-normalizer to base params
Drop the early continue so a guardrail that signals sensitive-data routing still
emits its PROXY_PRE_CALL service span like every other callback.
Move the lowercase normalizer onto BaseLitellmParams so on_sensitive_data is
normalized consistently when BaseLitellmParams is constructed directly, matching
the cross-field route->model validator that already lives on the base.
* fix(proxy): stop team model name corruption on edit (#28382) (#29001)
Team-scoped ("Team-BYOK") models store an internal routing key
model_name_{team_id}_{uuid} in the model_name column and the user-facing
name in model_info.team_public_model_name. The internal name leaked into
/v1, /v2, and /model/info responses; the dashboard bound its edit form to
it, so any non-rename save (e.g. a TPM tweak) PATCHed the internal name
back. The update path then treated it as a rename, overwriting
team_public_model_name and rewriting the team's models[] ACL with the
mangled string -- breaking team key calls with team_model_access_denied.
Two-layer fix:
- Read path (root cause): add _translate_model_name_for_response and apply
it in model_info_v2 and _get_proxy_model_info so /v1, /v2, and
/model/info surface the public name for team-scoped rows. The DB column
and router index keep the internal name as the routing key; this is a
presentation-layer swap on a shallow copy (never mutates input).
- Write path (defense in depth): harden _get_public_model_name so a value
matching the internal shape, or a no-op against the current DB column,
is never treated as a rename -- for both the top-level model_name and an
explicit model_info.team_public_model_name.
Tests: regression for the reported scenario, full branch coverage of
_get_public_model_name, two internal-shape guard cases, an end-to-end
PATCH through _update_team_model_in_db (asserts the team ACL is untouched),
and four response-translation cases. 60 passed (model management),
181 passed (proxy server).
* fix(ui): key Agent Builder agent selection on model_info.id (#29729)
* fix(ui): key Agent Builder agent selection on model_info.id
Once team-scoped BYOK models can share a public name (the backend now
returns the public name on /model/info instead of the internal routing
key), selecting agents by model_name collides. Key selection, create,
update and delete on the stable model_info.id instead, falling back to
model_name only for config-defined agents that have no id.
* fix(ui): add name-match fallback to post-create agent selection
If the just-created agent's id is not yet present in the re-fetched
list, try matching by name before falling back to the first agent.
Addresses greptile review on #29729.
---------
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
* refactor(ui): add shared HTTP client and pin raw fetch() to one file
Introduce src/lib/http/client.ts, a single typed wrapper that owns the only
fetch() in the dashboard. It centralizes the base URL, the auth header, error
parsing (deriveErrorMessage), non-2xx -> thrown ApiError, and JSON parsing, and
is framework-agnostic (no React) so it can run from client and, later, server
components. The base URL, auth header name and the logout side effect are injected
through createApiClient.
networking.tsx builds one configured apiClient and the 29 functions whose
boilerplate maps exactly to the client's default behavior (canonical
deriveErrorMessage + handleError + res.json() template) now call it instead of
hand-rolling fetch. Names, signatures, return types and error behavior are
unchanged; this is a pure refactor that drops ~440 lines.
The no-restricted-syntax fetch rule now points at the client and a
files: ["src/lib/http/**"] override makes that the only place fetch() is allowed.
Re-baselined eslint-suppressions.json: networking.tsx fetch suppressions drop
270 -> 241; no other rule's counts change.
The remaining networking.tsx fetches and the ~61 scattered component/hook fetches
diverge from the default client behavior (text() error bodies, no res.ok check,
no handleError side effect) and stay grandfathered for a follow-up burndown.
* fix(ui): make the HTTP client tolerate non-JSON error bodies
The non-2xx branch parsed the error body with response.json(), so a gateway
returning HTML (502/503 from a reverse proxy) threw a SyntaxError before onError
fired or ApiError was built, dropping the user-facing notification. This matched
the old per-function behavior, but the client is now the single error path so it
is the right place to harden. Read the body as text once, try JSON.parse for the
existing deriveErrorMessage path, and fall back to the raw text (or the HTTP
status) otherwise. The success path stays strict json() so return types are
unchanged.
* fix(ui): await the returned apiClient promise in 6 migrated functions
The codemod rendered the `return response.json()` tail as `return apiClient.x()`
without `await`. Inside the surrounding try/catch that returns an unawaited
promise, so the catch never runs and its console.error log is dropped on failure;
4 of the 6 were `return await response.json()` originally, so this restores their
exact behavior. Use `return await apiClient.x()` in all six.
* refactor(ui): widen onError type and handle empty success bodies
Address review notes on the shared client. Type onError as
(message: string) => void | Promise<void> so the fire-and-forget async contract
(networking passes the async handleError) is explicit rather than silently
discarded by void. On the success path, read the body as text and return
undefined for an empty body (e.g. a 204 No Content) instead of throwing a
SyntaxError, while still parsing non-empty bodies strictly so a malformed JSON
response surfaces rather than being masked. Add tests for the 204 case.
* restore an explicit no-match policy
* fix(jwt): fix AUTO_REGISTER sentinel bypass, race condition, and inline import comment
- AUTO_REGISTER now evicts stale __NO_MAPPING__ sentinel instead of silently
returning None when cached under a prior fallback_team_mapping config
- Race condition in _auto_register_jwt_mapping: catch P2002 unique-constraint
violation on concurrent creates, fetch the winning mapping, proceed cleanly
- Added comment on inline generate_key_helper_fn import explaining the circular
dependency (key_management_endpoints imports user_api_key_auth at line 51)
- 3 new tests: stale sentinel eviction, race condition winner fallback, and the
existing auto_register happy path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(jwt): cache __NO_MAPPING__ sentinel before raising 403 in REJECT mode
REJECT mode was raising HTTPException immediately on a DB miss without writing
the __NO_MAPPING__ sentinel, causing every subsequent rejected request to
re-query the DB. Write the sentinel first so repeated rejections are served
from cache within virtual_key_mapping_cache_ttl.
Adds test asserting DB is not hit on the second reject after a cache-warm miss.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(jwt): enforce no-match policy when prisma_client is None
The early `if prisma_client is None: return None` guard ran before the
no-match policy check, silently bypassing REJECT and AUTO_REGISTER — every
JWT client fell through to team auth regardless of configuration.
Fix: treat prisma_client=None as a definitive DB miss and fall through to the
same policy block as a real miss. REJECT now raises 403, AUTO_REGISTER raises
500 with a clear message (can't create keys without a DB), FALLBACK_TEAM_MAPPING
returns None unchanged.
Adds three tests: REJECT/403 with no DB, FALLBACK returns None with no DB,
AUTO_REGISTER/500 with no DB.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(jwt): consistent AUTO_REGISTER on cached sentinel; clean up race orphans
Addresses Greptile review on PR #25570 cherry-pick.
1. Inconsistent AUTO_REGISTER when __NO_MAPPING__ sentinel is cached:
The cached-sentinel branch silently returned None when prisma_client was
None, while the fresh path raised HTTP 500 under the same config. Same
request, different access-control outcome depending on cache state. Both
paths now raise the same 500.
2. Orphaned virtual keys from race-condition losers:
On unique-constraint conflict, generate_key_helper_fn had already persisted
an unrestricted virtual key in LiteLLM_VerificationToken with the cleartext
in request memory. Under sustained concurrency these accumulated
indefinitely. The loser now deletes its orphan before falling back to the
winner's mapping; failure to delete is logged but does not fail the request.
Also corrects a latent FK bug surfaced while fixing #2: the mapping row was
storing the plaintext key in LiteLLM_JWTKeyMapping.token, but that column FKs
to the hashed LiteLLM_VerificationToken.token — now hashed at the call site.
Tests:
- updated test_auto_register_creates_key_and_mapping to assert the hashed
token is stored, not the plaintext
- updated test_auto_register_race_condition_unique_conflict to assert the
orphan is deleted with the correct hashed token
- added test_auto_register_raises_500_when_sentinel_cached_and_no_db
- added test_auto_register_race_conflict_tolerates_delete_failure
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(jwt): close REJECT bypass when JWT omits the configured claim field
A JWT presented without the configured `virtual_key_claim_field` previously
returned None at the `claim_value is None` guard before the
`unregistered_jwt_client_behavior` check ran. A caller who knows the configured
claim-field name could bypass REJECT by simply omitting that field and falling
through to team-based JWT auth.
Apply the no-match policy on a missing claim:
- REJECT → 403
- AUTO_REGISTER → 403 (no stable identity to map; refuse rather than
create a sentinel-keyed record)
- FALLBACK_TEAM_MAPPING → return None (unchanged, backward-compatible)
Adds three tests covering each branch of the missing-claim path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(jwt): AUTO_REGISTER inherits team_id so keys are bounded by team limits
Auto-registered virtual keys were created with no team, model, route, rate, or
budget constraints — broader access than the standard team-based JWT auth path
the same client would have taken. Under AUTO_REGISTER, resolve the team_id
from the JWT (via the operator-configured team_id_jwt_field / team_id_default)
and stamp it on the new key. Downstream auth then applies the team's
budget/models/tpm/rpm/allowed_routes via the existing virtual-key flow.
Policy when team_id_jwt_field is configured:
- JWT carries team claim → stamp resolved team_id
- JWT lacks claim + team_id_default set → stamp default
- JWT lacks claim + no default → 403 (refuse to create an unbounded key)
When neither team_id_jwt_field nor team_id_default is configured, the
operator has explicitly opted out of team-based limits — the auto-created
key has no team_id (matches what team-auth would do in the same config).
Adds 4 tests covering each branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(jwt): make AUTO_REGISTER functional in prod; raise on missing winner
Two correctness fixes flagged by Greptile on the AUTO_REGISTER path:
1. generate_key_helper_fn was called without table_name="key". Without that,
the helper falls into the user-upsert branch (table_name in (None, "user"))
and tries to insert into LiteLLM_UserTable with user_id=None, which hits
the NOT NULL @id constraint. AUTO_REGISTER would never have succeeded in
production. Now passes table_name="key" explicitly, matching the
/key/generate caller.
2. When the race loser refetches the winner's mapping and gets None (winner
row concurrently deleted), the previous code returned None — and the
caller in _resolve_jwt_to_virtual_key then fell through to less-
restrictive team-based JWT auth, silently bypassing the configured
AUTO_REGISTER policy. Now raises HTTP 503 so the caller retries against
a stable state rather than getting unintended fallback access.
Adds one test for the 503 winner-vanishes path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(jwt): defer AUTO_REGISTER until JWT policy is enforced by auth_builder
Closes the JWT policy bypass on the AUTO_REGISTER path flagged by veria-ai.
Before: when unregistered_jwt_client_behavior=auto_register and the JWT's
claim was unmapped, _resolve_jwt_to_virtual_key validated the JWT signature
and then immediately created a virtual key + mapping. JWTAuthManager.auth_builder
never ran for the first request (the new key short-circuited the team-auth
path), and every subsequent request hit the cached mapping — so custom_validate,
RBAC, scope_mappings, and user_allowed_email_domain were never enforced for
auto-registered clients.
After: _resolve_jwt_to_virtual_key returns a _PendingAutoRegister signal
instead of creating the key. The caller in _user_api_key_auth_builder runs
JWTAuthManager.auth_builder, then — only on a validated, policy-passing
result — calls _auto_register_jwt_mapping with the team_id / user_id from
that result. The created key inherits team + user limits from the validated
identity, and future cache hits load that already-policy-checked key.
Also drops the interim _resolve_inherited_team_id helper that pulled team_id
from raw JWT claims — same bypass risk; team_id now comes exclusively from
auth_builder.
Tests:
- Rewrote two existing tests to assert _resolve_jwt_to_virtual_key returns
_PendingAutoRegister (no key created yet) for both the fresh-DB-miss
and stale-sentinel branches
- Added a contract test that _auto_register_jwt_mapping stamps the
validated team_id/user_id onto generate_key_helper_fn
- Removed four stale team-binding tests that exercised the prior
raw-claim helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update user_api_key_auth.py
* fix(jwt): cache proxy-admin AUTO_REGISTER path to avoid repeated DB lookups
Cache-miss regression introduced by the deferred-auto-register refactor:
when a JWT under AUTO_REGISTER resolved to a proxy admin, the is_proxy_admin
early-return in _user_api_key_auth_builder ran *before* the pending
auto-register cache-write block. Result: no cache entry, so every
subsequent proxy-admin request re-queried get_jwt_key_mapping_object
indefinitely.
Fix: write a __JWT_PROXY_ADMIN__ sentinel to user_api_key_cache before the
early return when a pending auto-register existed. _resolve_jwt_to_virtual_key
treats that sentinel as "skip mapping, fall through to auth_builder", so
future requests from the same JWT identity hit the cache instead of the DB.
auth_builder still runs full JWT policy on every request — only the
mapping DB lookup is short-circuited.
Adds one test asserting the sentinel cache-hit returns None without
hitting prisma_client.db.litellm_jwtkeymapping.find_first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(proxy): stamp org context on JWT auto-registered keys
AUTO_REGISTER keys were created with team_id and user_id only, so org budget checks were skipped after switching to the key-scoped path.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): only flag bare fetch() outside React Query queryFn/mutationFn
The frontend lint rule banned every fetch() call by static AST name match,
so a fetch wrapped in a React Query queryFn/mutationFn tripped it just like
a loose fetch in a component. esquery (no-restricted-syntax) can't express
"has ancestor", so this replaces that selector with a small custom rule
(local/no-bare-fetch) that exempts a fetch lexically inside a queryFn or
mutationFn and reports everything else.
Re-baselined eslint-suppressions.json under the new rule id (same 44 files /
331 violations) so existing code keeps its grandfathered suppressions.
Adds a RuleTester suite covering wrapped (valid) vs unwrapped, the standalone
*Api.ts function pattern, queryKey, and computed-key cases.
* chore(ui): remove the bare-fetch lint rule
Drop the fetch lint gate (and its 331 grandfathered suppressions) ahead of
the networking refactor. The plan is to centralize all fetching in a single
shared http client and enforce that with a location-based rule, so keeping a
fetch rule in place now would only block CI while functions are routed
through the new client. Removing it unblocks that work; the location-based
rule lands with the client in a follow-up.
store_user_env_vars was superseded by the advisory-locked
merge_user_env_vars and had no production caller; it survived only as a
test seeding helper. Remove it (and the test-only _captured_values_blob),
seed encrypted blobs through a small _encrypted_user_env_blob helper, and
move the no-plaintext-at-rest assertion onto the live merge_user_env_vars
path so that security regression guards real code rather than a dead
function. Also drop an unused import flagged in the same file
management_endpoint_wrapper serializes management responses into OTEL
spans, and _redact_env_var_values only scrubbed env_vars at the top
level. GET /v1/mcp/server/submissions returns MCPSubmissionsSummary with
items[].env_vars carrying decrypted scope="global" values for full
admins, so those upstream credentials were stringified into the
response.items span attribute and readable by observability users on the
legacy OTEL path.
Walk the items list too and blank each record's env_vars values, copying
the record (model_copy / dict spread) so the HTTP response the admin gets
back keeps the decrypted values for the edit form. Regression test
asserts the nested secret never reaches the span while names and scopes
survive; reverting the items walk makes it fail
* test(proxy): stop running real-DB tests in GitHub Actions unit jobs
GitHub Actions unit jobs were spinning up a Postgres service container, but
the only active tests that touched it either used the DB incidentally (a
cargo-culted prisma_client.connect()) or were genuine integration tests
mislabeled as unit. Mock the incidental ones so the proxy-db job needs no
container, and move the tests that genuinely need a database (proxy
management behavior, master-key-not-persisted, schema-migration sync) to
CircleCI, which is already the real-infrastructure lane.
* test(proxy): restore no-unexpected-startup-writes canary in master-key test
Greptile noted the hash-match assertion no longer catches other unexpected
startup writes (a default key, a rotation artifact). The CircleCI job gives
each run a fresh DB, so a clean startup must leave the table empty; add that
canary back alongside the precise master-key assertion.
Prisma may hand back env_vars as a raw JSON string on write paths instead
of a parsed list. The previous decrypt_global_env_var_values call iterated
the string character-by-character (silent no-op), leaving global secrets
encrypted on the returned row. add_server then seeded the registry with
ciphertext, and the immediately-following reload_servers_from_database
reused that broken entry because updated_at matched, so headers forwarded
ciphertext upstream until a later edit changed updated_at.
Wrap the create/update decrypt in _decrypt_env_vars_on_returned_row, which
parses a string payload back into a list before the in-place decrypt runs
and writes it back onto the row. Also harden _reencrypt_global_env_var_values
against the same string shape so master-key rotation doesn't crash on
dict(v) over a string.
test_custom_tokenizer_bug.py loaded Xenova/llama-3-tokenizer from
HuggingFace Hub at test time, so it flaked on shared CI runners whenever
HF returned 429 Too Many Requests; the surfaced LocalEntryNotFoundError
made it look like a connectivity bug.
Rewrite the suite to mock the one network boundary
(litellm.utils.Tokenizer.from_pretrained) while running the proxy's real
extraction-and-selection path. The regression test now asserts the
configured identifier from model_info.custom_tokenizer actually reaches
from_pretrained and that the response reports the huggingface tokenizer,
which the previous llama-3-named test could not distinguish from the
default path. A control test pins the no-custom-tokenizer case to the
OpenAI tokenizer with from_pretrained asserted unused.
Verified by reintroducing the original bug (model_info left unpopulated
from the deployment): the regression test fails (from_pretrained called 0
times) while the control stays green.
pg_advisory_xact_lock keyed via hashtextextended requires PostgreSQL 11+;
older clusters fail on every merge_user_env_vars call. Derive a stable
64-bit signed key from (user_id, server_id) with blake2b in Python and pass
it straight to pg_advisory_xact_lock, so the lock works on every supported
PostgreSQL version and no longer depends on an internal hash function
In Pydantic v2, Optional[T] without a default is a required field. Any
row with budget_id=null triggered a validation error and returned 401.
Co-authored-by: Florent Chenebault <florent.chenebault@lifen.fr>
* change deployment configs to include a litellm.cache for litellm-backend pod mirroring litellm-gateway pod
* omit backend annotations block when config and podAnnotations are both empty
* reuse gateway config/configmap for backend instead of separate backend config
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Tin Chi Lo <tin@Tins-MBP.localdomain>
Co-authored-by: Tin Chi Lo <tin@Tins-MacBook-Pro.local>
store_mcp_user_env_vars read existing values, merged the submitted updates,
and wrote the result in three separate steps. Two simultaneous POSTs from the
same user to the same server could both read the same snapshot and the second
write would silently clobber the first. Move the read-modify-write into a new
merge_user_env_vars DB helper that runs inside a transaction guarded by a
(user_id, server_id) advisory lock, so concurrent writes are serialized and no
update is lost.
Add the squash-merged SHA of #29622 (style(ui): run prettier --write
across the dashboard) to .git-blame-ignore-revs so the bulk reformat
stops masking the real authors of those lines in git blame and the
GitHub blame UI
* fix(azure): apply api_version fallback chain to image edit URL
`AzureImageEditConfig.get_complete_url` only read `api_version` from
`litellm_params`. When callers configured it via `litellm.api_version`
or `AZURE_API_VERSION`, the constructed URL had no `?api-version=` and
Azure responded `404 Resource not found`.
Apply the same fallback chain the Azure chat path already uses in
`common_utils.py`:
litellm_params > litellm.api_version > AZURE_API_VERSION env >
litellm.AZURE_DEFAULT_API_VERSION
Adds 5 unit tests pinning each layer of the chain plus a regression
guard for `api_base` that already carries `?api-version=`.
* feat(mcp): core sampling and elicitation flow with security hardening
- Add sampling_handler.py: full MCP sampling/createMessage flow with
model selection (hint-based + priority-based), auth enforcement,
budget checks, route restriction gates, and tag policy pre-auth
- Add elicitation_handler.py: MCP elicitation/create relay with
downstream client capability detection
- Wire sampling/elicitation callbacks in mcp_server_manager.py
gated behind allow_sampling/allow_elicitation config flags
- Add allow_sampling/allow_elicitation fields to MCPServer type
- Fix session lock deadlock: skip lock for JSON-RPC response POSTs
(elicitation/sampling replies) with truncated-body heuristic
- Extend client.py with sampling_callback and elicitation_callback
- Security: RouteChecks gate, tag-budget bypass fix, x-forwarded-for
spoofing fix, Latin-1 header encoding guard
- Add 4 new test modules (model access, priority selection, request
builder, tool conversion) + update existing MCP tests
* fix(security): run pre-call guardrails before MCP sampling acompletion
Without this, an upstream MCP server with allow_sampling enabled could
send prompts that bypass every guardrail (content filtering, PII
redaction, prompt-injection detection) configured on /chat/completions.
- Call proxy_logging_obj.pre_call_hook(call_type='acompletion') before
llm_router.acompletion so guardrails fire for sampling sub-calls
- Add HTTPException to the re-raise list so guardrail rejections
propagate correctly instead of being swallowed as generic errors
* feat(bedrock_mantle): add Responses API support (/openai/v1/responses) (#29490)
* feat(bedrock_mantle): add Responses API transformation config
* test(bedrock_mantle): cover trailing-slash api_base normalization
* feat(bedrock_mantle): export BedrockMantleResponsesAPIConfig
* feat(bedrock_mantle): register gpt-5.x Responses config (gpt-oss unchanged)
* feat(bedrock_mantle): add gpt-5.5/gpt-5.4 Responses price-map entries
* refactor(bedrock_mantle): exclude gpt-oss instead of allow-listing gpt-5 for Responses routing
Frontier OpenAI models on Bedrock Mantle are Responses-only on /openai/v1/responses;
gpt-oss is the legacy family that also speaks chat-completions. Gate by excluding
gpt-oss (which keeps its chat-completions emulation) and defaulting everything else
to the native Responses config, so future frontier models (gpt-6, etc.) route
correctly without a code change. Verified against the live us-east-2 Mantle endpoint:
gpt-oss 400s on /openai/v1/responses while gpt-5.5 400s on both standard paths.
* test(bedrock_mantle): cover supports_native_websocket opt-out
Closes the one uncovered line flagged by codecov on the Responses config.
The assertion documents that Mantle Responses has no realtime/websocket
transport, so realtime routing must not attempt a socket it cannot serve.
* fix(bedrock_mantle): route file_search through emulation instead of forwarding to Mantle
BedrockMantleResponsesAPIConfig inherited supports_native_file_search()
-> True from OpenAIResponsesAPIConfig but never overrode it. Mantle has no
OpenAI vector stores, so a forwarded file_search tool is rejected with a
400 (verified upstream: Tool type 'file_search' is not supported). Opting
out, like the existing supports_native_websocket override, routes the tool
through LiteLLM's file_search emulation instead.
* fix(bedrock_mantle): only route openai.gpt frontier models to Responses
The previous gate excluded gpt-oss and routed every other model to the
native Responses config. But on Mantle only the OpenAI gpt frontier models
(gpt-5.x) are served on /openai/v1/responses; gpt-oss and the non-OpenAI
families (nvidia, mistral, google, zai, ...) are chat-completions only and
400 on that path. Allow-list the openai.gpt- family (excluding gpt-oss)
instead, so chat-only models fall through to the chat-completions emulation.
Verified against the live us-east-2 endpoint: nvidia.nemotron-nano-9b-v2
returns 400 on /openai/v1/responses and 200 on /v1/chat/completions.
* feat(custom_llm): allow streaming/astreaming to yield ModelResponseStream (#27580)
* fix(custom_llm): allow streaming/astreaming to yield ModelResponseStream directly
* fix(streaming): enhance ModelResponseStream handling for custom LLM providers
* fix(streaming): strip finish_reason from content chunks and ensure tool_calls are preserved
* fix(streaming): add type ignore for finish_reason assignment in CustomStreamWrapper
* fix(proxy): strip stack trace from HTTP 503 responses (CWE-209) (#28330)
* fix(proxy/cwe-209): strip Python traceback from HTTP 503 error responses
The /cache/ping endpoint included a full Python traceback in its 503 error
response body (inside the ProxyException message), leaking internal file
paths, line numbers, and call stacks to any caller. Two MCP route handlers
in proxy_server.py similarly interpolated str(e) into "Internal server
error" detail strings.
Fix: log the traceback server-side via verbose_proxy_logger.exception()
and omit it from the ProxyException payload / HTTPException detail returned
to clients. Tests updated to assert no "traceback" keyword or frame paths
appear in the 503 body, with a new dedicated regression test.
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy/cwe-209): apply Greptile P2 fixes and add MCP exception-path tests
Greptile 4/5 review identified two remaining gaps and Codecov reported
0% coverage on the two MCP handler exception branches:
1. caching_routes.py — str(e) in "Service Unhealthy ({str(e)})" could
still leak Redis hostnames/IPs; replaced with static "Service Unhealthy".
HTTPException is now re-raised before the generic handler so the
"cache not initialized" 503 still reaches callers with its detail.
Removed the redundant str(e) arg from verbose_proxy_logger.exception()
(exception() already appends the traceback automatically).
2. tests — two new unit tests cover the exception paths in
dynamic_mcp_route and toolset_mcp_route that were previously at 0%:
- test_dynamic_mcp_route_unexpected_exception_returns_500_without_traceback
- test_toolset_mcp_route_unexpected_exception_returns_500_without_traceback
All 25 tests pass (9 caching + 16 MCP).
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion in test_cache_ping_no_cache_initialized
The assertion was weakened to `"Cache not initialized" in str(data)`, which
matches the raw string of the entire response dict and would pass even if the
error moved to an unexpected field or changed structure.
Restore a targeted check on the parsed response: assert the exact string in
the correct field `data["detail"]`, matching FastAPI's HTTPException
serialisation format {"detail": "<message>"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion and add CWE-209 no-cache path test
The assertion in test_cache_ping_no_cache_initialized was weakened to
`"Cache not initialized" in str(data)`, which matched against the raw string
representation of the entire response dict. This would pass silently even if
the error message moved to an unexpected field or the structure changed.
Restore a targeted assertion on the parsed field:
assert data["detail"] == "Cache not initialized. litellm.cache is None"
matching FastAPI's HTTPException serialisation format exactly.
Add test_cache_ping_no_cache_does_not_expose_internals to show the code path
is still working correctly after the CWE-209 fix: verifies that the HTTPException
is re-raised as-is (no traceback, no source paths), and asserts the complete
response structure is exactly {"detail": "Cache not initialized. litellm.cache is None"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(caching_routes): restore ProxyException envelope for null-cache 503
The except HTTPException: raise guard (added in the CWE-209 fix) caused
the null-cache HTTPException to escape as FastAPI's {"detail": "..."} shape
instead of the {"error": {...}} ProxyException envelope that callers expect.
Move the null-cache guard before the try block and raise ProxyException
directly so the response structure is consistent with all other /cache/ping
503s, and the except HTTPException: raise guard is only reachable by
unexpected downstream HTTPExceptions.
Update the two no-cache tests to assert the correct ProxyException envelope.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update utils.py (#26609)
* feat(pricing): add Snowflake Cortex REST API model pricing (#26612)
* feat(pricing): add Snowflake Cortex REST API model pricing
## Summary
Adds pricing and context window information for 20+ Snowflake Cortex REST API models to `model_prices_and_context_window.json`.
## What's included
- **7 Claude models** (sonnet-4-5, sonnet-4-6, 4-sonnet, 4-opus, haiku-4-5, 3-7-sonnet, 3-5-sonnet) — with prompt caching rates
- **4 OpenAI models** (gpt-4.1, gpt-5, gpt-5-mini, gpt-5-nano) — with prompt caching rates
- **5 Llama models** (3.1-8b, 3.1-70b, 3.1-405b, 3.3-70b, 4-maverick)
- **1 DeepSeek model** (deepseek-r1)
- **1 Mistral model** (mistral-large2)
- **1 Snowflake model** (snowflake-llama-3.3-70b)
- **2 Embedding models** (arctic-embed-l-v2.0, arctic-embed-m-v2.0)
Each entry includes `input_cost_per_token`, `output_cost_per_token`, `cache_read_input_token_cost` (where applicable), `max_input_tokens`, `max_output_tokens`, and capability flags (`supports_function_calling`, `supports_vision`, `supports_prompt_caching`, `supports_reasoning`).
## Pricing source
All prices are in USD per token, sourced from the official [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf) — Tables 6(b) (REST API with Prompt Caching) and 6(c) (REST API).
## Context
The existing `snowflake/` provider has zero model entries in the pricing JSON, which means LiteLLM cannot track costs for Snowflake Cortex calls. This PR fills that gap.
## Related
- Existing provider: `litellm/llms/snowflake/`
- Cortex REST API docs: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-rest-api
* Update model_prices_and_context_window.json
Fix the JSON parsing error
* Update model_prices_and_context_window.json
Removed the duplicate entry
* fix(utils): copy extra_body before adding unknown params to prevent model config mutation (#29620)
Fixes#29615. In add_provider_specific_params_to_optional_params, the line:
extra_body = passed_params.pop("extra_body", None) or {}
returns the original dict reference when extra_body is non-empty (truthy).
Subsequent writes like extra_body[k] = passed_params[k] then mutate the
shared model config object held by the router, poisoning /model/info and
all subsequent requests for that deployment.
The or {} short-circuit creates a new dict only when extra_body is falsy
(None or {}), which is why the bug does not reproduce with extra_body: {}.
Fix: wrap in dict() so we always work on a fresh shallow copy.
* fix(vertex_ai): Bake tool_choice into Gemini CachedContent body to prevent silent drop (#29097)
* fix(vertex_ai): bake tool_choice into Gemini CachedContent body to prevent silent drop
* address greptile feedback on tool_choice cache test
* adds test that uses ToolConfig(functionCallingConfig=FunctionCallingConfig(mode=ANY)) instead of a dict literal, mirroring what map_tool_choice_values actually produce
* fix(gemini/veo): move image from parameters into instances[0] (#29501)
* fix(gemini/veo): move image from parameters into instances[0]
Veo's predictLongRunning schema puts image (and prompt) on the
instances element; parameters is for aspectRatio/durationSeconds/etc.
The Gemini path was leaving image in params_copy, so it ended up
nested under parameters and the API silently ignored it.
The Vertex path already builds the instance dict explicitly, so this
just aligns the Gemini path with it.
Fixes#29498
* address greptile: unconditional pop + BytesIO test
- Pop `image` from params_copy unconditionally so it never reaches
GeminiVideoGenerationParameters even when None, removing implicit
reliance on Pydantic's extra-field-ignore.
- Add test_transform_video_create_request_image_filelike_goes_to_instance
covering the BytesIO path (_convert_image_to_gemini_format) — round-trips
the base64 to confirm encoding.
- Add test_transform_video_create_request_image_none_is_dropped covering
the new None branch.
* fix(huggingface): handle special token text in embedding usage (#29660)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params (#29655)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params
ToolPermissionGuardrail builds self.rules and the compiled target/pattern
maps only in __init__. The base update_in_memory_litellm_params re-sets raw
attributes via setattr but never rebuilds those maps, so a guardrail updated
in place (PUT /guardrails, or the immediate in-memory sync) keeps enforcing
the construction-time rules until it is reinitialized (PATCH path, periodic
DB poll, or restart).
Extract the compile step into _load_rules and override
update_in_memory_litellm_params to rebuild from it (dict- and model-safe),
re-normalizing default_action / on_disallowed_action. Mirrors the existing
PresidioGuardrail override of the same method. Adds regression tests.
Fixes#29592.
* fix(guardrails): handle dict params in ToolPermissionGuardrail in-memory update
Delegate to super() only for LitellmParams input (the base setattr loop is
model-only); apply the raw-dict case inline. Fixes the mypy arg-type error
and makes the recompile work when the proxy passes the raw DB dict.
* fix(guardrails): preserve tool-permission rules on a partial in-memory update
A partial update (e.g. a LitellmParams whose rules field is None) ran through
the generic setattr, which set self.rules to None, and the recompile was
skipped, leaving the guardrail with no rules. Snapshot the previous rules and
restore them when the update carries no rules; an explicit empty list still
clears them. Adds a regression test for the rules-absent case.
Addresses the Greptile review note on #29655.
* fix(bedrock): stop base_model label from stripping tools/tool_choice (#29621)
* fix(bedrock): stop base_model label from stripping tools/tool_choice
A Router/proxy Bedrock deployment whose model_info.base_model is a friendly
label (e.g. claude-haiku-4-5) silently lost tools/tool_choice: the outgoing
Converse request was built without toolConfig, so the model behaved as if no
tools were provided. Worked in v1.84.0, regressed in v1.85.0, and with
drop_params=true it failed silently.
Two changes compound into the bug. completion() passed model_info.base_model
as the model argument to get_optional_params, so the real Bedrock model id
never reached supported-param resolution; and get_supported_openai_params
resolved the provider config's params from base_model or model, letting the
label fully replace the real model. For Bedrock the label resolves to no tool
support, so tools/tool_choice were dropped before transformation.
completion() now keeps model as the real deployment model and threads the
resolved base_model (kwarg or model_info) through separately, and
get_supported_openai_params treats base_model as additive: it returns the
union of the params supported by model and by base_model. A hint can only add
capabilities, never strip ones the real model already exposes, which also
preserves the original base_model behavior from #27717 and Azure's base_model
driven model-type detection.
Fixes#29618
* test(main): make base_model param test robust to new parametrize cases
Restore an explicit per-case expected_model_param literal instead of
hardcoding the gemini id, so a future case with a different model can't
produce a misleading assertion failure.
* fix(fireworks_ai): pass response_format json_schema through unchanged (#29606)
FireworksAIConfig.map_openai_params was rewriting the OpenAI strict
`{type: json_schema, json_schema: {name, strict, schema}}` shape into
`{type: json_object, schema: ...}` before sending to Fireworks, dropping
`strict` and `name` and changing the `type`. Per Fireworks' docs json_object
means "force any valid JSON output (no specific schema)", so the schema
constraint was effectively dropped and grammar-guided decoding never ran;
model output silently violated the schema.
The rewrite landed in #7085 (Dec 2024) when Fireworks did not yet accept
native json_schema. Fireworks accepts the OpenAI strict shape natively now,
so the rewrite has become a regression.
Removes the rewrite. Passes response_format through unchanged. Updates the
existing test_map_response_format to assert pass-through. Adds focused
regression tests in tests/test_litellm/ covering preservation of type,
strict, name, and schema body, plus that json_object alone still works.
* fix(types): import Required from typing_extensions in gemini types
* style: reformat sampling_handler.py for py312 black compat
* refactor(mcp-sampling): extract helpers to fix PLR0915 too-many-statements in handle_sampling_create_message
* fix(proxy-server): add explicit ProxyLogging type annotation to proxy_logging_obj to fix mypy inference
* fix(mcp-sampling): suppress mypy assignment error on ImportError fallback for proxy_logging_obj
* fix(test): use .value when comparing LlmProviders enum against string in test_default_api_base
* fix(test): iterate LlmProviders enum in test_default_api_base to avoid str pollution from custom provider registration
litellm.provider_list is a mutable global initialized to list(LlmProviders) but custom_llm_setup() appends plain provider strings to it. When a test_custom_llm.py test runs first in the same xdist worker, provider_list contains a str and calling .value on it raises AttributeError. Iterate the immutable LlmProviders enum instead, which is deterministic and what the check intends.
* fix(mcp): depth-aware JSON-RPC response detection and neutral speed-priority fallback
Replace the flat substring check in the truncated-body routing path with a
top-level-key scan so a JSON-RPC response whose result payload nests a
"method" field is still detected as a response and skips the session lock,
removing a deadlock against the in-flight tool call awaiting it.
Drop the inverse max_output_tokens speed proxy when no model exposes
output_tokens_per_second; context-window size does not track latency, so a
neutral score avoids biasing speedPriority toward the smallest-context model.
* fix(guardrails): make ToolPermission rule reload atomic on invalid regex
_load_rules appended each rule to self.rules before compiling its regex, so an
invalid pattern raised mid-loop after the bad rule was already live but without
a _compiled_rule_targets entry. _matches_regex reads a missing compiled target
as a None pattern and returns True, turning the bad rule into a match-all that
silently applies its decision to every tool. Via update_in_memory_litellm_params
(PUT /guardrails) this corrupted the live guardrail.
Build the parsed rules and compiled maps into locals and swap them in only after
every regex compiles, and restore the previous ruleset if a live update is
rejected, so an invalid regex now fails the update without leaving the guardrail
enforcing a broken policy.
* test(mcp): cover sampling conversion, model resolution, and elicitation relay paths
The MCP sampling and elicitation handlers shipped with partial test
coverage, leaving the response-to-MCP conversion, the model resolution
fallback chain, completion-kwargs assembly, guardrail routing, and the
entire elicitation relay untested. That pulled the PR's diff (patch)
coverage below the codecov threshold even though overall project
coverage rose.
Add focused unit tests for _convert_openai_response_to_mcp_result,
_convert_mcp_tools_to_openai, _convert_mcp_tool_choice_to_openai, image
and audio content conversion, the hint-matching and fallback branches of
_resolve_model_from_preferences, _build_completion_kwargs, the router and
guardrail-rejection paths of _run_guardrails_and_call_llm, the
handle_sampling_create_message success and error-propagation flows, the
marker-hoisting fallback for tool content on unexpected roles, and the
elicitation form/url/generic relay together with its decline paths
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: lengkejun <lengkejun@xd.com>
Co-authored-by: Yug <yugborana000@gmail.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: tanmay958 <53569547+tanmay958@users.noreply.github.com>
Co-authored-by: DrishnaTrivedi <142084770+DrishnaTrivedi@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Navnit Shukla <Navnit.shukla25@gmail.com>
Co-authored-by: PRABHU KIRAN VANDRANKI <72809214+VANDRANKI@users.noreply.github.com>
Co-authored-by: Adrian Lopez <109683617+adriangomez24@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: JooHo Lee <96564470+BWAAEEEK@users.noreply.github.com>
Co-authored-by: Dinesh Girbide <85330597+Dinesh-Girbide@users.noreply.github.com>
Co-authored-by: cloudwiz <22098246+andrey-dubnik@users.noreply.github.com>
Co-authored-by: Ahmad Khan <ahmadkhan2508@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The non-admin sanitizer only blanked global env var values, leaving the
names visible. Those names (DB_PASSWORD, GITHUB_API_KEY, ...) reveal which
secrets the admin configured, so a non-admin could enumerate them via the
list and detail endpoints. Drop env_vars wholesale like the virtual-key
sanitizer does; non-admins still get the per-user vars they must fill in
from the dedicated /user-env-vars/status endpoint.
A variable declared with both global and user scope is covered by the
global value (globals win in the merge), so the tool-call path must
resolve it from the global rather than raising a 412 when the user has
not filled in the per-user value. Restrict the missing-var check to
referenced user vars that lack a global fallback.
test_tokenizers downloads Xenova/llama-3-tokenizer from the HuggingFace
Hub via create_pretrained_tokenizer. On the CI runners the Hub keeps
returning 429 Too Many Requests, which propagated into the blanket
except and turned a third-party rate-limit into a hard pytest.fail. The
same test already skips its llama2 differentiation assertion when the
Hub is unreachable; this extends that exact handling to the custom
tokenizer download so a HuggingFace outage/rate-limit no longer fails
the suite while still failing on real assertion or logic errors.
The server-row delete is the commit point; a transient failure cleaning the
FK-less per-user env var rows now logs a warning instead of propagating, so a
successful delete is no longer turned into a caller error that triggers a retry
and a 404 for an already-gone server. Also mark the health-check env-var
round-trip test as asyncio so it runs explicitly like its siblings.
The frontend-lint baseline this branch had grown carried three
react-hooks/set-state-in-effect entries and four raw-fetch entries that were
added rather than fixed. Load the per-user env-var data in UserEnvVarsModal and
mcp_servers through React Query (useQuery/useMutation) so the setState-in-effect
findings go away instead of being baselined, and capture the ?fill_env_vars deep
link in lazy initial state so the modal target is derived during render rather
than set from an effect
Delete the unused clearMCPUserEnvVars wrapper, the one new networking fetch with
no caller. The remaining three wrappers (GET status, GET server vars, POST save)
still need a raw fetch because networking.tsx is the only HTTP layer and React
Query consumes it, so they stay grandfathered; the baseline only ratchets down:
UserEnvVarsModal 1->0, mcp_servers set-state 4->2, networking fetch 274->273
When prisma_client is None, _load_user_env_vars returned an empty dict,
which on the tool-call path was indistinguishable from "user has no
stored values" and produced a misleading 412 directing the user to set
up credentials they can never store without a database. Raise instead so
the tool-call path fails with a clear error and the listing path stays
best-effort via its existing catch.
The frontend-lint job (added on the base branch after this branch diverged)
runs prettier and eslint on the UI files a PR touches, measuring eslint errors
against the committed eslint-suppressions.json baseline. Pulling the base in
brings that gate, its config, and the baseline.
Format the touched MCP env var components and networking.tsx so they are
prettier-clean, and extend the suppressions baseline to cover the findings this
branch adds in files that already carry grandfathered entries: the four extra
raw fetch wrappers in networking.tsx (the API layer, where 270 raw fetches are
already grandfathered and there is no React Query alternative) and the
setState-in-effect findings in mcp_servers.tsx and UserEnvVarsModal.tsx, matching
the same rule already baselined in the sibling MCP components.
* fix(gemini-realtime): use GA event names for Pipecat 1.3.x compatibility
Pipecat v1.3.0 adopted the OpenAI Realtime API GA event naming:
response.audio.delta -> response.output_audio.delta
response.text.delta -> response.output_text.delta
response.audio.done -> response.output_audio.done
response.text.done -> response.output_text.done
The proxy was still emitting the old beta names; Pipecat's
`parse_server_event` raises "Unimplemented server event type" for any
unknown type, which killed the receive task handler and broke audio
playback and tool-call delivery.
Also:
- conversation.item.created -> conversation.item.added (already handled)
- client audio is buffered until backend setupComplete in deferred mode
- call_id fallback UUID when Gemini returns empty id
- status_details / token detail fields added to Pydantic-strict events
The _GA_TO_BETA_EVENT_TYPES map in RealTimeStreaming already translates
GA names back to beta for clients that opt in with the openai-beta
header, so legacy clients are unaffected.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): address greptile review comments
- emit outputTranscription as response.output_audio_transcript.delta
instead of suppressing it; GA_TO_BETA map handles translation for
legacy clients
- cap pre-setup audio buffer at 200 frames to prevent memory exhaustion;
log a warning when the limit is hit and additional frames are dropped
- log remaining dropped message count on flush error
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): address veria review comments
- remove unused OpenAIRealtimeConversationItemCreated import
- fix guardrail bypass: semantic_vad early-return now preserves
create_response when set so a guardrail-injected create_response:false
is not silently dropped
- add per-connection 10 MB byte cap alongside the 200-frame count cap
for the pre-setup audio buffer to prevent memory exhaustion
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): fix mypy arg-type on _finalize_gemini_live_setup
setup parameter typed as BidiGenerateContentSetup to match the TypedDict
passed at both call sites; was dict which mypy rejected.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): widen _finalize_gemini_live_setup to Dict[str, Any]
BidiGenerateContentSetup (TypedDict) is a subtype of Dict[str,Any] so
both call sites (one passing a plain dict, one passing the TypedDict)
satisfy mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): cast BidiGenerateContentSetup to Dict at _finalize call site
mypy rejects TypedDict as dict[str, Any] argument; cast at the call site
where follow_up_setup is BidiGenerateContentSetup to satisfy the checker.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Gemini realtime beta compatibility
* Fix deferred Gemini setup audio ordering
* fix: preserve Gemini audio transcript ids
* fix(realtime): cap pre-setup client buffer on all append paths
Route every append to the deferred-setup pending buffer through the
per-connection message/byte caps. Previously only the audio-buffer
fast path enforced the caps; once one frame was buffered, a client
that withheld session.update could stream arbitrary frames into
_pending_messages_until_setup unbounded and exhaust proxy memory.
* style(gemini-realtime): apply black formatting to transformation.py
* fix(gemini-realtime): log beta-translation fallback and name native-audio marker
Surface the previously swallowed exception in _send_event_to_client so a
failed GA->beta translation is observable instead of silently forwarding the
untranslated event. Extract the native-audio model substring used by
_finalize_gemini_live_setup into a named constant documenting why speechConfig
is dropped on those setups.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): match passthrough registry routes bare-to-bare with SERVER_ROOT_PATH
After #28547, get_request_route strips the deployment prefix while registry
lookup still re-inflated stored paths via SERVER_ROOT_PATH, causing 404s
under paths like /llmproxy/ml. Compare normalized bare routes in both
is_registered_pass_through_route and get_registered_pass_through_route.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): patch utils.get_server_root_path in passthrough auth tests
After removing get_server_root_path from pass_through_endpoints, route
and JWT tests must mock litellm.proxy.utils where normalization reads it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini): keep googleSearch with server-side tools and googleMaps JSON schema
Wire include_server_side_tool_invocations through completion() so mixed
google_search and function tools are not dropped on Gemini 3+. Rewrite
generationConfig to responseFormat when googleMaps is used with JSON schema.
Fixes#27479Fixes#29451
Co-authored-by: Cursor <cursoragent@cursor.com>
* address greptile review feedback (greploop iteration 1)
* style: fix black formatting in main.py for py312 compat
* Fix Gemini Google Maps extra_body JSON rewrite
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
