A slow or unreachable counter store made a pre-relay rejection wait behind
the reservation release before the client saw the error event and the close.
Close first and release in finally, mirroring the relay's own failure path,
so a client that already hung up still gets its reservation released.
The three pre-relay exits of realtime_websocket_endpoint (missing model,
key/model access denied, pre-call rejection such as a rate limit or a
guardrail) returned before the finally that releases the auth-time budget
reservation, so a rejected session pinned the key at the reserved amount
until the counter TTL expired and its next requests got budget_exceeded
while /key/info showed spend 0. A single _reject_realtime_session helper
now releases the reservation before sending the error event and closing,
and release_or_invalidate_budget_reservation shields the release from a
second cancellation and logs, rather than raises, a failing invalidate
fallback so it can never mask the session's own outcome.
The blanket finally release from the previous commit also zeroed the reservation
of successful sessions. Success settlement is enqueued on the logging worker, not
awaited, so the endpoint's finally ran first and released the reservation the cost
callback still had to reconcile, dropping the real spend from the key/team/user
counters.
The relay now stamps a synchronous marker (REALTIME_SESSION_SUCCESS_LOGGED_KEY) on
the shared logging object at the single success-dispatch site, and the endpoint
releases the reservation only when that marker is absent. Refused or failed
sessions, which never log success, still release; successful sessions leave the
reservation for the cost callback to settle to actual spend. Exactly one settler
touches each reservation, so the idempotent reconcile never double-adjusts.
A refused or failed /v1/realtime session never ran the success cost callback
or a failure hook, so its pre-call budget reservation stayed open and kept the
key/team/user spend counters pinned above real spend, 429ing later requests on
the same key until the counter's TTL expired. The endpoint now reconciles the
reservation in a finally, reusing a shared release_or_invalidate_budget_reservation
helper that mirrors the success/failure paths (release to zero, else invalidate
the reserved counters and finalize).
The relayed upstream close message and reason also go through the proxy's
client-facing redaction, so a credential, internal hostname, private IP, or
server path echoed by the upstream never reaches the client verbatim.
* fix(proxy): mark session/SSO/SAML cookies Secure behind a TLS-terminating reverse proxy
litellm only sees a plain-HTTP hop when TLS terminates at a reverse proxy,
so cookie Secure attributes previously derived from (or defaulted without
regard to) the literal request scheme could be dropped in production. The
token session cookie set by every login path never carried Secure/HttpOnly/
SameSite at all.
Adds IPAddressUtils.is_request_https, a single trust-aware resolver used by
every cookie-setting call site: PROXY_BASE_URL, then X-Forwarded-Proto only
from a configured trusted proxy (general_settings.use_x_forwarded_for +
mcp_trusted_proxy_ranges), then the literal scheme. An unconfigured or
untrusted caller cannot spoof the header to force Secure on.
Resolves LIT-6748
* fix(proxy): make the shared session-cookie helper public, type new test helpers
set_session_token_cookie is imported across modules (ui_sso.py ->
proxy_server.py), so the leading underscore was misleading and breached
basedpyright's reportPrivateUsage budget with zero headroom. Also adds
missing parameter/return type annotations to the new test helper functions
per repo convention.
* test(proxy): verify NO_DOCS/NO_REDOC/NO_OPENAPI gate every doc surface
Confirms /docs, /redoc and /openapi.json are reachable by default and
each 404 with no schema in the body when NO_DOCS/NO_REDOC/NO_OPENAPI are
set, and that disabling them does not affect other routes.
* test(proxy): pin the production docs_url construction to the real helpers
Addresses Greptile 4/5 feedback: the disabled-path tests built a fresh
FastAPI() app, which could miss drift in the actual
FastAPI(docs_url=..., ...) call in proxy_server.py. This pins that
exact call site against _get_docs_url/_get_redoc_url/_get_openapi_url.
Coordination Redis (spend counters, budget-window enforcement, and the
reset_spend cache-eviction broadcast) previously only attached when a
deployment set general_settings.coordination_redis or
litellm_settings.cache. Bare REDIS_HOST/REDIS_PORT env vars alone did
nothing, so a multi-replica proxy with no cache block got no cross-pod
coordination at all: a key reset on one pod never cleared another
pod's stale budget enforcement.
The inferred Redis is pinged before being adopted, and a malformed
REDIS_CLUSTER_NODES/REDIS_SENTINEL_NODES value is tolerated too: env
vars can be set for an unrelated reason with nothing reachable there,
and guessing wrong must not turn a previously harmless in-memory-only
proxy into one that fails to boot or raises on its next cache write.
Adds a configurable password-strength policy (default: min 12 chars,
upper/lower/number/special, all individually toggleable, floored at 8
so a misconfigured minimum cannot disable the length check, and
unicode-aware so an accented letter cannot satisfy the special-
character requirement) enforced on every path that sets a local
user's password: /user/update, /user/bulk_update, and the invitation
onboarding claim flow.
Adds general_settings.disable_password_login_when_sso_enabled, which
rejects username/password login on /login, /v2/login and /v3/login
(including the UI_USERNAME/UI_PASSWORD admin fallback) once ANY
configured SSO provider is FULLY ready: every companion secret/
endpoint an OAuth provider needs, checked independently per provider
so a stray leftover client id for an unused provider can't mask a
different, fully configured one; and for SAML, the optional
python3-saml runtime being importable, checked without letting a
fully-missing package's ModuleNotFoundError take down password login
itself. SSO becomes the enforced boundary for interactive UI access
without an incomplete, mixed, or half-installed SSO setup locking
every admin out or breaking login outright. Master-key API access is
untouched, and unsetting the setting plus a restart restores password
login as the documented recovery path.
* feat(alerting): slack alerts for per-user daily/monthly spend thresholds and spend anomaly detection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(alerting): use specific ValidationError matches in config rejection test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): tolerate mocked slack alerting args when scheduling user spend scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): reject non-finite values in user spend alert settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The one-time seed for a budget window row subtracted the batch's own
LiteLLM_SpendLogs rows by request_id, and request_id is the client's
x-litellm-call-id whenever the response carries no id of its own. Carrying
that set through the queue meant an unbounded, client-controlled aggregate
that the commit-failure requeue kept alive across retries.
Every log row at or after a batch's earliest start is owed by an increment
that still reaches the row, so summing only rows before it needs nothing
from the request. That drops request_ids end to end and closes the
cross-pod double count the id list could not see.
request_id can be chosen by the client through x-litellm-call-id, so an
unbounded NOT (request_id = ANY(batch)) let a replayed old id drop that id's
historical LiteLLM_SpendLogs row from the one-time seed while its increment
still landed. The increment now carries the request start, the batch keeps
the earliest one, and the seed only excludes ids whose startTime is at or
after it.
The where clause now uses the exact name string directly and skips the DB query when the typed search cannot occur in that name, so no new mutable literals are added (LIT002 gate).
With model=<group>&search=<term>, the router list was narrowed to the group but the DB query only matched the substring, so other groups' rows leaked into the page and total_count.
The router reload triggered by /model/new read the model table through
the read replica, so a lagging replica made the reload miss the just
committed row and fail the request with a 500 even though the write was
durable. Fixes#38556
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Router fallbacks configured in router_settings were attempted without
re-checking whether the calling key could use the fallback model, so a key
limited to one access group was served by any model listed as a fallback
for something it could call. Auth only validated the requested model and
fallbacks sent in the request body.
Add a fallback_access_check predicate to Router, consulted before every
cross-model-group fallback attempt; rejected targets are skipped and the
primary's own error is raised when none remain. The proxy injects a check
that runs the same key, team and project model access checks the requested
model goes through.
LiteLLM_PromptTable is unique on (prompt_id, version, environment) and
version numbering restarts at 1 per environment, but the in-memory
registry keyed prompts as {prompt_id}.v{version} with no environment, so
environments sharing a prompt id shadowed each other and only one
environment's template ever served.
Registry entries are now keyed {versioned_id}::{environment}, and serve
time resolution goes through resolve_prompt_spec(base_id, version,
environment): production > staging > development when no environment is
requested, latest version within the chosen environment when no version
is requested. Chat requests can pin an environment with a new optional
prompt_environment body param, filtered from provider-bound params like
prompt_id and prompt_version. The newest-updated_at dedupe in
_init_prompts_in_db is dropped since registry keys can no longer
collide, and the key-parsing serve helpers plus dead registry getters
are removed
Reverts #37725. The field existed so SDK callers that cannot read
`x-litellm-model-id` could tell which tier an auto-router picked, and the
framework that motivated it was LangChain. `@langchain/openai` builds
`additional_kwargs` and `response_metadata` from fixed key allowlists and drops
unknown fields at both the chunk top level and inside `delta`, so no
proxy-side placement of a namespaced key can reach a LangChain caller.
The complexity router's existing `return_raw_model_name` already covers that
case: it puts the resolved model in the standard `model` field, which
LangChain does propagate (`model_name` is on its metadata allowlist), and the
proxy honors it on both the streaming and non-streaming paths.
Keeps the unrelated cleanup from #37725 that dropped the redundant
function-local `ProxyBaseLLMRequestProcessing` import shadowing the
module-level one in `async_data_generator`.
`TestModelGroupAliasReachesPreRoutingStrategies` asserted on the marker as a
proof of strategy dispatch; the surviving `response.model == "gemini-flash"`
assertion already proves it.
Keys minted by /key/generate get no LiteLLM_UserTable row, so /v2/model/info?user_models_only=true for such a user hit the new None guard and returned 400 where the merge base returned the user's own models. Skip the team-model merge for a missing row instead of raising, since a user with no row belongs to no team
