* fix(proxy): reset a stuck team member's budget
A per-team-member budget check reads a cross-pod spend counter that
nothing ever invalidates. Once a member exceeds their per-member
budget, resetting the key's spend, raising the user's or the team's
own budget, or issuing a new key all leave the member stuck, because
none of them touch this counter or its cached membership object.
Add POST /team/{team_id}/member/{user_id}/reset_spend to reset a
member's tracked spend, and invalidate the same cached state from
/team/member_update when it raises a member's own budget, so that
path also takes effect immediately instead of waiting on the
membership cache's TTL. Name the entity in the check's error message
so a stuck member is diagnosable from the 429 body alone.
* fix(proxy): close reset-vs-floor-read race and surface double Redis write failure on member spend reset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast spend reset as a SET so the handler's self-delivered message cannot erase the reset guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit null fields from the invalidation message so plain evictions keep the old wire format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
The Admin UI mints every session key against the UI_TEAM_ID sentinel,
which by design never has a team row, so every UI request hit the new
refusal with no override. Exempt UI_TEAM_ID explicitly so it keeps
reconstructing from the token unconditionally, matching how the MCP
handler and agent_permission_handler already special-case it.
Resolves LIT-5522
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
team_allowed_routes and admin_allowed_routes only matched exact strings or named route groups, so a whole prefix of pass-through endpoints had to be listed route by route in config. Match trailing-wildcard patterns with the same helper the key-level allowed_routes check uses, so "/prefix/*" covers endpoints registered later.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A request can name more than one model, through a comma-separated model or target_model_names on
the batch and fine-tuning routes, and the gate only looked at the string case, so an unpriced model
riding alongside a priced one went through and billed. Check every candidate and name the unpriced
ones in the 403
Aliases had the same problem on the other side: a group that prices itself through its model_info
block lands in the cost map under its deployment id, and the explicit-cost check walked the raw
model list by group name, so an alias pointing at that group read as unpriced. Resolve the group
through the router the way the pricing check already does
Also correct the 403 copy. Providers that return their own usage cost still bill for these models,
so the accurate claim is that litellm has no pricing of its own for them
A deployment that overrides any cost_per field, including at zero, now counts as priced so it is not blocked as unpriced
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.
Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:
- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
on the failure path, so the NameError, not the assertion, is what ran.
test_llm_guard_error_raising is the worst: it passes today with content
safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
supports 3.10. Guarded behind the exceptiongroup backport that anyio already
pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
a helper that catches HTTPException by a name it never imported, so the
challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
test_claude_agent_sdk, and an unused helper in test_end_users calling a
function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
FileNotFoundError behind a NameError.
Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
* fix(proxy): cache tag-name registry so unregistered request tags skip Postgres
Request tags are free-form attribution labels, so most have no LiteLLM_TagTable
row. get_tag_objects_batch never cached that absence: every tagged request ran
a find_many that came back empty, and under Prisma pool contention those
per-request queries queued for minutes inside user_api_key_auth.
Cache the bounded set of registered tag names under one aggregate key with the
management-object TTL. Uncached request tags are filtered against it before any
per-tag DB fetch, so unregistered tags cost zero DB reads on a warm path. An
empty registry is cached as a valid answer; DB errors are not cached and fall
back to the per-tag lookup; tables past TAG_REGISTRY_MAX_SIZE cache an overflow
sentinel that disables filtering. Tag create/update/delete endpoints now evict
the registry and per-tag keys and publish cross-worker invalidation (they
previously evicted nothing). The per-tag write-back also gains the management
TTL it was missing, and the hand-built tag:{name} key strings are replaced with
a shared builder.
* fix(proxy): skip per-request end-user DB reads via restricted-id registry
Every request carrying a user id ran get_end_user_object, and with high-cardinality
auto-created end-user rows (hundreds of thousands of ids, all restriction fields
NULL) the per-pod cache missed on nearly every request, so each one paid a Postgres
find_unique that queued behind the Prisma pool during background-job bursts. True
misses were never cached, and unknown ids paid the read twice per request.
Cache the bounded set of end-user ids that carry any restriction (blocked, budget,
region, default model, or object permission) under one aggregate key with the
management-object TTL. When an id misses the per-id cache and is absent from a
usable registry, get_end_user_object returns None with zero DB reads; restricted
ids keep today's fetch-and-cache path. The skip is bypassed whenever
litellm.max_end_user_budget_id is set (default budgets make unrestricted rows
behaviorally distinct from missing rows), validate_end_user_id_in_db is on
(existence checks need the row), or the token carries end_user_max_budget from
custom auth (the row's recorded spend seeds the budget counter). Empty registries
cache as a valid answer, DB errors are never cached, and oversized tables cache an
overflow sentinel that disables filtering. Customer create/update/block/delete now
evict the registry and per-id keys and publish cross-worker invalidation (they
previously evicted nothing), and the per-id write-back gains the management TTL it
was missing so Redis entries no longer live forever.
* refactor(proxy): single generic registry loader with error sentinel and single-flight
Code review follow-ups on the two registry caches. Registry DB errors now cache
the overflow sentinel for a short REGISTRY_ERROR_NEGATIVE_CACHE_TTL window and
log at warning, so a degraded Postgres stops paying the failing registry scan on
every request on top of the per-id fallback. Cold registry loads are single-flight
per worker behind per-registry locks with a recheck after acquire, so a TTL expiry
no longer fans out one full-table scan per in-flight request. The tag and end-user
loaders collapse into one _load_bounded_registry with per-entity fetch closures,
and the triplicated evict-then-broadcast protocol becomes one evict_and_broadcast
helper beside publish_auth_cache_invalidation, shared by the tag, customer, and
project eviction paths.
* chore(lint): suppress fail-safe registry excepts and ratchet BLE001 budget
* docs(proxy): trim registry cache commentary to single-line why docstrings
* fix(lint): move tag fetch return to else block to satisfy TRY300 budget
This reverts commit ab2333b6c4.
Every Admin UI login mints its session key against the sentinel team_id
`litellm-dashboard`, and no LiteLLM_TeamTable row is ever created for it.
That lookup is therefore a provably-absent row on every UI request, which
#36837 turned into a hard refusal with no override, so the whole dashboard
404s.
Reverting restores the token-derived fallback. The model-access widening
#36837 closed is reopened and needs a re-land that exempts the UI sentinel
team.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
CLI session tokens minted by /sso/cli/poll set team_id and team_alias but
never team_models or team_model_aliases, so the token carried a team with
none of that team's grants. /v1/models bails out to "unrestricted" when both
key_models and team_models are empty and listed the whole proxy, and team
model aliases never resolved because both can_team_access_model and the
pre-call rewrite read team_model_aliases off the token.
The team data was not close at hand: _fetch_cli_sso_team_details projected
full team rows down to team_id and team_alias before they reached the mint.
Widen that projection to include the team's models and its joined alias
table, and populate both fields at mint time.
Also stop writing the user's personal allowlist into the key models slot
when a team is bound, matching virtual-key semantics where a team-bound
credential is governed by the team grant.
Because an empty team grant is itself a real value meaning unrestricted, a
team whose grants cannot be resolved must not be minted as empty: that is
the same "unrestricted" bail-out this fix exists to close. The poll now
refuses to mint when the selected team has no complete cached detail.
That refusal is only safe because a login can no longer be pinned to a team
whose grants will never resolve. Deleting an organization drops its team
rows but leaves the memberships behind, so the login now offers only teams
whose rows still exist, and a lookup that fails outright fails the login
rather than caching a session that silently drops every team.
* feat(proxy): add apply_user_budget_to_team_keys opt-in
PR #32005 made a user's personal max_budget apply to their team-scoped keys
too, and PR #35271 reverted the whole thing (behavior plus the
skip_user_budget_on_team_key opt-out) because that flipped the default for
everyone. This brings the behavior back the other way round: default is
unchanged, and general_settings.apply_user_budget_to_team_keys opts a
deployment into charging the key owner's personal budget on team keys.
The flag reaches all three personal-budget gates so an opted-in deployment
enforces consistently: the read-time check in common_checks, the optimistic
reservation counter in _get_budget_counters, and the _PROXY_MaxBudgetLimiter
pre-call hook. It is also in the /config/list allowed args and, unlike the
reverted flag, in the _update_general_settings propagation allowlist, so the
Admin UI General Settings toggle actually takes effect at runtime; an explicit
YAML value still wins over the DB value on reload.
get_config_list's allowed_args moves to a module-level frozen mapping of
field name to type string, dropping 18 LIT002 violations and rebuilding one
less dict per request.
* style(proxy): drop explanatory comments from the budget flag paths
* fix(proxy): invalidate cached project object on /project/update and /project/delete
The auth path reads projects cache-first via get_project_object with a 60s
TTL and no freshness check, but no project write endpoint ever evicted the
project_id:{id} cache entry. A project cached before /project/update added a
model allowlist kept an empty models list in cache, so _run_project_checks
skipped can_project_access_model and project-bound keys could call team
models outside the project allowlist until the TTL expired. The same
staleness applied to blocked status and budget fields, and /project/delete
left the deleted project enforceable from cache.
Evict the cache entry after the DB write in update_project and
delete_project via a shared delete_cached_project_object helper, with the
cache key derivation shared with get_project_object.
* fix(proxy): broadcast project cache invalidation to all workers and make eviction best-effort
Single-worker eviction leaves every other worker serving its in-memory copy
of the mutated project until the 60s TTL expires, so a project allowlist
change was still bypassable on multi-worker deployments. Add a coordination
Redis pub/sub channel (litellm_proxy.auth_cache_invalidation): project
eviction publishes the cache key and a per-worker subscriber deletes the
local in-memory entry, with the next auth read refetching from the DB.
Subscriber starts on any deployment with a coordination Redis and falls back
to the TTL when none is configured.
Also wrap the eviction in a best-effort catch: the DB write has already
committed when eviction runs, so a cache backend error must not turn a
successful update into a 500 or abort the remaining ids in /project/delete.
* fix(lint): sort auth cache invalidation import and suppress best-effort shutdown catch
The strict-budget gate flagged the new import block as un-sorted (I001) and
the broad except in stop_auth_cache_invalidation_subscriber (BLE001); the
catch is intentional since a failing stop must not break proxy shutdown, so
it carries a named suppression instead of counting against the budget.
* fix(proxy): give proxy_admin_viewer read parity with proxy_admin
Route-level checks already default-allow management GETs for the viewer
role, but ~15 handlers compared user_role to PROXY_ADMIN only, dropping
viewers into regular-user scoping (/key/list, /user/info, /model/info,
guardrails, prompts, agents, memory, workflows, MCP catalog, coordination
redis settings, credential migration check, enterprise projects). Swap
those read paths to user_api_key_has_admin_view; write gates unchanged.
The dashboard now presents the viewer session as Admin for all gating
(effectiveSessionRole) so every page fetches with admin visibility, with
userRoleLabel/isViewOnly preserving the account-menu label and the
playground cost guard. The server remains the write authority.
* refactor(agents): remove side-effectful health_check param from GET /v1/agents
Addresses a security review finding on the admin viewer read parity change:
listing agents with health_check=true made the proxy issue a server-side GET
to every agent URL, so a read-scoped caller could trigger request fan-out
beyond their object permissions. The list endpoint is now a pure read for
every role.
Removes the query param, the URL probing helper and its timeouts, the
AgentHealthCheck httpx provider tag, and the dashboard's Health Check
toggle. Requests still passing health_check=true get the full list back
with the param ignored.
* fix(proxy): keep credential encryption check proxy_admin only
The residual scan behind GET /credentials/migrate-encryption/check loads
every model, credential, MCP, team, and verification-token row and runs a
decryption attempt on each stored value. Extending it to proxy_admin_viewer
let a read-only account repeatedly trigger deployment-wide scans, so the
route keeps its original full-admin gate.
* fix(agents): restore health_check, keep list fast path proxy_admin only
Restores the agent health_check feature exactly as before this PR: the
query param, the URL probing helper, the httpx provider tag, and the
dashboard toggle all return, so existing callers keep the filtering
contract. The viewer expansion is instead reverted at its source: the
GET /v1/agents admin fast path stays PROXY_ADMIN only, so a
proxy_admin_viewer goes through the object-permission scoped branch as
before and cannot fan out health checks beyond their allowlist. The
viewer read of a single agent stays viewer-inclusive since it has no
side effects.
* fix(proxy): backfill null user_email on existing users during JWT auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): guard mapped-key email backfill and make null update atomic
Resolve Greptile review on the JWT user_email backfill:
- only backfill when the mapped virtual-key owner is the JWT principal, so a
mismatched admin-created mapping cannot write one user's email onto another
- make the best-effort mapped-key enrichment non-fatal so a database outage on
a cached-key request no longer fails otherwise-valid authentication
- persist the backfill with an atomic null-guarded update_many so concurrent
writers cannot overwrite an already-populated email
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep cache coherent when a concurrent backfill wins the null-email update
* fix(proxy): cache DB-persisted email after JWT backfill, not the proposed value
Resolve the Greptile finding that a successful null-guarded backfill could
cache this request's proposed email even if a concurrent ordinary user update
wrote a different email first. The helper now always re-reads the row after the
atomic update and refreshes the cache from the value the database holds, so
cache-hit auth and attribution stay consistent with the persisted record.
Annotate the Prisma and model_copy dict literals to keep the LIT002 budget within its ceiling.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(proxy): only enforce budgets on routes that can spend
Budget checks ran inside common_checks with no route filter, so an
over-budget user, team, organization or tag got a 429 on every
authenticated route, including the management calls the Admin UI makes
on load. An internal user who exhausted their budget could not open the
dashboard to see why, and a max_budget of 0 locked them out from the
moment the account existed.
Gate the scope budget checks on RouteChecks.is_llm_api_route, matching
the virtual key budget check, the reservation path and the global proxy
budget check, which already scope themselves this way. /health/services
keeps enforcing because it fires Slack, email and webhook sends.
The Admin UI is affected because a UI login mints a virtual key scoped
to the litellm-dashboard pseudo-team. That token was shielded from
personal budgets by the team-key exemption until #32005 removed it.
* fix(proxy): keep budget enforcement on provider-calling health routes
/health and /health/test_connection are not LLM API routes but both run
litellm.ahealth_check against real deployments, so exempting them let an
exhausted budget keep incurring provider spend.
Add them alongside /health/services in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES
and cover all three with a regression test.
* chore(ui): drop env-dependent schema.d.ts regeneration from this PR
The regenerated diff was union-member reordering only, with no change to
the represented types, and the ordering differs between a local run and
CI. Keeping the committed file as-is lets the drift check pass and keeps
this PR to the auth change.
* chore(ui): restore schema.d.ts to the branch base
The previous commit restored it from the staging tip, which pulled in
unrelated merged changes. This PR changes no backend models, so the file
should be untouched.
Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
Add regression tests for the db-fetch paths whose converted construction
lines were uncovered: the auth_checks getters (default end user budget, end
user, team membership, access group, team by alias, org by alias, object
permission, managed vector stores, project), get_all_team_memberships and
list_available_teams in team_endpoints, and the proxy admin user info
helper. Each test feeds a mocked prisma row through the real function and
asserts the validated model's fields, so a bad model_validate conversion on
any of these paths now fails a test instead of only dropping coverage.
* fix(auth): route JWT default-team into memberships instead of the create payload
JWT auto-provisioning (get_user_object with user_id_upsert) merged
litellm.default_internal_user_params verbatim into the Prisma user create,
including a teams key. When a default team is configured through the Admin
UI it is stored as a list of NewUserRequestTeam objects, but the user
table's teams column is String[], so the create raised a Prisma type error
and every JWT-authenticated request 401'd with the user never created.
Mirror the /user/new path: strip teams (and available_teams) out of the
create payload, then route the configured default team through
check_if_default_team_set / add_new_user_to_default_team so provisioned
users get real membership rows. Reuse the synthetic PROXY_ADMIN
UserAPIKeyAuth pattern already used by the team-upsert path to satisfy the
membership permission gate, and import the helpers lazily to avoid the
auth_checks <-> internal_user_endpoints import cycle.
* fix(auth): propagate max_budget_in_team when adding users to default teams
* fix: use pipe union instead of Optional for UP045 budget
Greptile review: _cache_team_object runs after a successful DB fetch in
get_team_object and after every team mutation's DB write, but
DualCache.async_delete_cache propagates backend errors, so a Redis blip
during invalidation would turn a healthy team lookup into a 404 and a
committed /team/update into a 500. Both the internal usage cache delete
and the alias-key invalidation now log a warning and continue on failure,
matching how DualCache.async_set_cache already swallows write errors.
Worst case on failure is worker-local staleness bounded by the internal
cache's in-memory TTL, the same bound other workers already have
get_team_object consults proxy_logging_obj.internal_usage_cache before
user_api_key_cache, but _cache_team_object (the refresh every team
mutation goes through) only wrote user_api_key_cache. With
enable_redis_auth_cache both caches share one Redis, so any request
backfills the internal cache's in-memory tier with the team object and
that copy keeps shadowing the freshly written team until its TTL expires.
The auth builder then wrote the team object it had just read back into
the cache after check 6, clobbering the fresh Redis value with the stale
one, which made the staleness self-sustaining under traffic: keys with
models=["all-team-models"] kept getting 403 team_model_access_denied
for models added via /team/update, and kept access to removed ones.
_cache_team_object now deletes the internal usage cache entry before
writing the refreshed team, and the auth-time write-back is removed so
only authoritative writers (DB reads and team mutations) populate the
team cache, mirroring how key objects already handle this (see
test_auth_does_not_rewrite_cached_key_object_back_into_cache).
The LIT-4000 test pinning the removed write-back is deleted; its
concern (team object cached under the canonical key) is handled by
_cache_team_object inside get_team_object's DB path and pinned by
test_cache_team_object_writes_team_id_and_invalidates_team_alias
Resolves LIT-4391
The JWT first-login upsert in get_user_object creates the user row by
merging default_internal_user_params straight into table.create, so a
configured budget_duration landed with budget_reset_at NULL. The reset
sweep now heals such rows (PR #33623), but until the next sweep the row
shows a null reset time and its first window starts at the sweep instead
of one full duration after creation. Compute budget_reset_at at creation
like every other write path (/user/new, UI SSO, /key/generate, /team/new)
already does
* fix(proxy): stop treating upstream model body field as a LiteLLM model on auth-enforced pass-through routes
An auth: true user-defined pass-through endpoint runs full virtual-key auth, and get_model_from_request unconditionally extracted the request body model field, so key/team/user/project model allowlist checks rejected requests whose model only exists upstream (key_model_access_denied), even when the key was explicitly granted the route via allowed_passthrough_routes.
The pass-through route registry moves to a leaf module (route_registry.py) that the auth layer can import without re-entering the pass_through_endpoints -> user_api_key_auth -> auth_utils import cycle. get_model_from_request now returns None for routes registered as user-defined pass-through endpoints (exact and subpath), which skips model allowlist and per-model budget enforcement on those routes while key auth, allowed_passthrough_routes, and spend/budget checks stay intact. Built-in provider passthrough routes (/vertex_ai, /gemini, ...) keep model enforcement.
Resolves LIT-4299
* fix(proxy): key pass-through model-access skip on the dispatched endpoint, not the request path
Addresses a model-authorization bypass: the first version decided whether to skip
model-allowlist extraction by matching the request path against the pass-through
route registry. That ignored the HTTP method and, more importantly, whether the
request was actually dispatched to a pass-through handler. A custom pass-through
whose path collides with a built-in route (e.g. /v1/chat/completions, or an
include_subpath prefix of one) still writes a registry entry even though FastAPI
serves the built-in handler, so a normal request to that route had its model checks
skipped and could reach a model outside the key/team/user/project allowlist.
The skip is now keyed off the FastAPI-resolved endpoint. create_pass_through_route
tags its handler with LITELLM_PASS_THROUGH_ENDPOINT_MARKER, and get_model_from_request
returns None only when request.scope["endpoint"] carries that marker. Because routing
runs before auth dependencies, this reflects the handler that actually serves the
request: on a collision the built-in handler is dispatched and carries no marker, so
model enforcement stays on. This also removes the need for the separate route_registry
module, so that extraction is reverted.
Regression tests cover a pass-through-dispatched request (model suppressed), a
built-in-dispatched request on the same path (model still enforced), and the no-request
budget path.
Resolves LIT-4299
With enable_redis_auth_cache and multiple replicas, /key/update and
/key/delete delete the Redis auth blob and the handling pod's in-memory
entry, but two read-path writers re-published the stale blob from any other
replica's per-pod memory back to Redis with a fresh 60s TTL on every
request: the post-auth re-cache in user_api_key_auth and the spend writeback
in update_cache. Replicas whose in-memory entries expired then re-primed
themselves from the poisoned Redis entry, so key limit and access changes
never took effect fleet-wide while traffic continued, and a deleted key kept
authenticating.
The auth object is now written only by the DB-load paths
(IdentityStore._resolve_key, get_key_object): the post-auth re-cache is
removed outright (even a local-only write could race an invalidation and
resurrect a revoked key on this worker) and spend tracking no longer writes
the auth object back at all; spend is tracked through the spend🔑*
counters. The remaining spend writebacks for user, team, end-user, and tag
objects become local-only so they cannot republish stale management objects
either, with one deliberate exception: the proxy-wide
{litellm_proxy_admin_name}:spend scalar keeps its shared Redis write because
the global max_budget check reads it between authoritative DB reloads, and
it carries no limits or permissions so sharing it cannot resurrect an
invalidated auth blob.
DualCache's redis-to-memory read backfill also ignored default_in_memory_ttl,
pinning backfilled entries for InMemoryCache's 600s default instead of the
configured 60s auth TTL; the backfill now injects the configured default like
every write path already does, so a replica primed from Redis converges
within the auth cache TTL as well.
Consolidates the sibling stale-auth-recache branch; the delete-propagation
case is the duplicate ticket LIT-4350.
Resolves LIT-4219
* fix: enforce user budget on team keys
User budget was skipped when the key belonged to a team, letting
users exceed their personal budget by going through a team key.
Remove the team_object guard in _user_max_budget_check so user
budgets are always enforced. Add skip_user_budget_on_team_key
general_settings flag to opt back into the old behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: update test to expect user budget enforcement on team keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI
Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert budget_exceeded ProxyException in personal budget test
Tighten the broad pytest.raises(Exception) so the test only passes when
the auth flow rejects with a budget_exceeded ProxyException, and switch
the new ConfigGeneralSettings field to Optional[bool] to match the
surrounding annotation style
* fix: revert to bool | None to stay under UP045 strict budget
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction
Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them
Add an opt-in mode so a key that exceeds its own max_budget is throttled to a
globally configured percentage of its TPM/RPM instead of being blocked entirely.
A new litellm_settings global, budget_exceeded_throttle_percentage, sets the
fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in
key metadata via the existing management-endpoint metadata routing) opts the key
in. When both are set and the key is over budget, the budget check records the
percentage on a request-scoped budget_throttle_pct instead of raising, and the
rate limiter scales the key's configured TPM/RPM by it. Keys without the flag
keep hard-blocking; team/user/org budgets are unaffected.
The throttle is recomputed from the key's original limits on every request and
the decision is cleared before the auth object is cached, so it never compounds
across requests. Both the budget read-time check and the budget reservation path
honor the opt-in, and both the v3 and legacy rate limiters apply the scaling.
Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an
admin-imposed hard budget block into a soft throttle that keeps spending past
max_budget, so a non-admin must not be able to self-opt-in and bypass their own
spend cap. Both /key/generate and /key/update reject a non-admin setting it to
true (update only gates the transition to enabled, so a non-admin can still edit
other fields and turn the flag off). This matches the feature being wholly
proxy-admin operated: the global percentage is admin-only too.
A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays
hard-blocked rather than serving unlimited requests past its budget (fail-safe).
The global budget_exceeded_throttle_percentage is configurable from the admin UI
(Settings -> General Settings), persisted through litellm_settings so it survives
a restart, not only from config.yaml.
Resolves LIT-3894. Scope for LIT-3893.
* Revert "fix(auth): deny model access for teamless keys with all-team-models (#32022)"
This reverts commit dfbbda4f19.
* revert: undo teamless all-team-models denial from PR #29746
Reverts the team_id guard in _resolve_key_models_for_auth_check and
get_key_models so teamless keys with all-team-models resolve to []
(unrestricted = all proxy models) rather than being denied.
Adds hardened regression tests across listing (get_key_models), inference
(_enforce_key_and_fallback_model_access, can_key_call_model,
can_key_call_resolved_model), and batch (_enforce_batch_file_model_access)
paths that enforce teamless all-team-models == all-proxy-models and will
fail if anyone re-introduces a team_id guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
_enforce_key_and_fallback_model_access and can_key_call_resolved_model
both unconditionally skipped key-level model checks whenever
all-team-models was present, without verifying the key actually belongs
to a team. PR #29746 fixed the listing path (get_key_models) but these
two call-path checks were left untouched, letting teamless keys call
any model via /chat/completions while seeing an empty model list.
Add team_id is not None guard to both bypass conditions so teamless
keys fall through to can_key_call_model, which already correctly
rejects unresolvable all-team-models sentinels
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
increment_spend_counters was parallelized in #31578, but the dominant
per-request cost under high concurrency is the pre-call budget enforcement
in common_checks, which still ran a Redis-first get_current_spend per scope
(team, team windows, key windows, org, tag, user, team member, end user)
one sequential await after another inside the auth span.
The per-scope reads target distinct counter keys with no cross-scope
ordering dependency, so they now run concurrently under asyncio.gather.
Key metadata.tags injection still runs before the gather so the tag budget
check sees it, and every scope settles before the first error in
scope-priority order propagates, preserving the previous rejection semantics.
Resolves LIT-4090
general_settings.user_api_key_cache_ttl was ignored for every management-object
write into user_api_key_cache. The configured value is propagated to the cache's
default_in_memory_ttl at startup, but DualCache only applies that default when no
explicit ttl kwarg is passed, and every management-object writer passed
ttl=DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL (60s), which always won. So keys,
teams, users, budgets, object permissions, vector stores, JWT user syncs and MCP
caches all expired after 60s regardless of the setting.
Adds get_management_object_ttl(cache) in user_api_key_cache.py, which returns the
configured default_in_memory_ttl and falls back to the 60s constant only when no
default is set, and routes every management-object writer through it. The helper
takes a DualCache so it works at the many call sites that are typed UserApiKeyCache
but exercised with a bare DualCache.
Also covers the spend-update writeback in update_cache (async_set_cache_pipeline),
which hardcoded ttl=60 on the same key/user/team objects and reset an active key's
cache entry back to 60s on every priced request, so the configured TTL was never
observed for keys receiving traffic.
Resolves LIT-3338
* fix(cli): mint per-session agent credential on lite login
The `lite login` command was producing a shared UI session token that broke agent use in three ways: a $0.25 budget cap (from max_ui_session_budget) that killed agent sessions in minutes, a fixed identity "cli-jwt-token" shared across every user preventing per-session spend attribution, and auth gated behind EXPERIMENTAL_UI_LOGIN so the token was rejected on default deployments.
This fixes all three. Each login now generates a unique cli-session-{uuid} token with no per-key budget cap (enforced via shared team/user counters instead), and the decrypt path activates for any non-sk- token without requiring EXPERIMENTAL_UI_LOGIN.
* fix(cli): address review feedback on EXPERIMENTAL_UI_LOGIN gate and e2e test
Restore EXPERIMENTAL_UI_LOGIN=false as an explicit opt-out: operators who set it to false keep the old boundary; unset (new default) and true both attempt NaCl decryption, which fails closed for non-blob tokens.
In the e2e test: replace the silent Redis fallback with pytest.skip so a missing Redis instance is explicit rather than silently degrading to a directly-minted token. Write the seeded flow back as JSON (proxy reads it via json.loads on cache fetch) instead of Python repr, and build the updated flow immutably.
* fix(key-management): cap CLI session token delegation budget to team ceiling
A CLI session token intentionally carries max_budget=None to avoid a per-session LLM spend cap. The key-generation delegation check (GHSA-q775-qw9r-2r4g) previously skipped non-admin callers with max_budget=None, treating them as having unlimited delegation authority. This allowed any internal user with a lite login session to mint virtual keys with arbitrary budgets.
Adds is_session_token=True to UserAPIKeyAuth for CLI session tokens and uses the caller's team budget as the delegation ceiling in that case, so the effective limit is min(requested_budget, team.max_budget) rather than unbounded.
* chore: regenerate dashboard OpenAPI types
The is_session_token field added to UserAPIKeyAuth cascades to the
dashboard schema. Regenerate types from the updated OpenAPI spec.
* fix(key-management): block personal key budget delegation from CLI session tokens
When team_table is None (personal key, no team_id in request), the personal key
has no team-budget enforcement at request time. A session token therefore cannot
delegate any explicit max_budget for a personal key -- that would open a budget
bypass path. Block the request with a clear 400 directing the caller to use a
team_id instead.
* test(auth): add unit coverage for non-admin CLI session token production path
* fix(type-check): use model_validate in _return_user_api_key_auth_obj to fix reportArgumentType gate
UserAPIKeyAuth(**user_api_key_kwargs) spread triggers a basedpyright
reportArgumentType error for each named field in UserAPIKeyAuth because
the dict's inferred value type (str | Span | LitellmUserRoles | Unknown)
is not assignable to each field's specific type. Adding is_session_token:
bool introduced +2 more such errors, breaching the gate cap.
model_validate accepts an untyped dict without per-field argument checking,
which eliminates the +2 new errors and also ratchets down the pre-existing
333 errors at those call sites. basedpyright-code-budget.json is updated
to reflect the new lower baseline (1814, down from 1934).
* fix(type-check): ratchet down reportArgumentType baseline only
The previous lint-budget-update captured all baselines from the local
environment, raising many ceilings vs the merge-base and failing the
non-gating budget_ratchet_check. Restore staging's values for every
rule and only lower reportArgumentType (1934 -> 1814) to reflect the
reduction from switching to model_validate in _return_user_api_key_auth_obj.
* fix(auth): set max_budget on CLI session token to enforce max_ui_session_budget
CLI session tokens were missing max_budget, so _virtual_key_max_budget_check
had no per-session ceiling to enforce. Operators relying on max_ui_session_budget
could be bypassed for the full token lifetime. Mirrors the existing UI token path.
* revert(auth): remove max_ui_session_budget from CLI session token
max_ui_session_budget defaults to $0.25 and is sized for the UI chat
pane (10-min sessions). CLI sessions are 24-hour tokens for real work;
capping them at that ceiling would throttle users under their actual
user/team budget. Budget enforcement for CLI sessions is via the shared
user and team counters as originally intended.
* fix(auth): cap CLI session at max_ui_session_budget only when user and team have no budget
When neither the user nor their team has a budget configured, CLI sessions
were fully uncapped. The poll endpoint now looks up the real user and team
objects from DB; if both have no max_budget, it passes litellm.max_ui_session_budget
as the token's per-key ceiling. Users or teams that already have a budget
configured are unaffected and continue to rely on the shared counters.
* fix(auth): fix black formatting and update test mock for cli_poll_key budget lookup
The get_user_object and get_team_object async calls in cli_poll_key were
not mocked in the existing test, causing MagicMock await errors. Patch
both functions at the auth_checks module level. Also apply black formatting
to ui_sso.py which CI rejected.
* fix(auth): skip fallback budget cap when team lookup fails for cli session token
* test(auth): pin cli session budget cap to user/team budget presence
The session_max_budget fallback in cli_poll_key only applied
max_ui_session_budget when neither the user nor the resolved team had a
budget. The existing coverage exercised only the team-lookup-failure
branch. Add two regression tests: a user with a configured budget must
not receive the fallback cap, and a session with no user and no team
budget must fall back to max_ui_session_budget. Mutating either guard
out of the branch now fails these tests.
* fix: remove CLI poll session budget cap
* revert(auth): restore CLI session fallback budget cap
Bugbot autofix (60b81fb8) removed the user/team budget lookup in
cli_poll_key and stopped passing max_budget to the session token,
making CLI sessions fully uncapped whenever neither the user nor the
team has an explicit budget.
That reintroduces the unbounded-spend bypass veria flagged as High
("CLI session budget bypass"): on deployments that rely on
max_ui_session_budget rather than per-user/team budgets, a completed
lite login could run LLM calls with no ceiling for the whole token
lifetime. The fallback only applies when no other budget bounds the
session, so users and teams with a configured budget are unaffected and
keep relying on their shared counters.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes#30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.
* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos
test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.
* fix(interactions): drop role from Interaction response to match Google spec
Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.
* fix: align all-team-models sentinel access
* fix(router): forward include_fallback_errors through multi-hop fallbacks
run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.
---------
Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu>
Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com>
Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com>
Co-authored-by: dav nguyxn <hoangson091104@gmail.com>
Co-authored-by: Tal Marian <tal.marian@island.io>
Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
