Commit graph

186 commits

Author SHA1 Message Date
mateo-berri
f9010a1c45 fix(auth): type the 403 by the key's grant and check auto-router previews as their team 2026-11-17 03:41:42 -08:00
mateo-berri
671ebeac68 fix(auth): keep empty access groups so a fully foreign group grants nothing
A group whose members all belong to other teams was dropped from the
team-filtered map, so its name leaked into /v1/models as a plain model.
Keep the group with no members, and treat a grant that expands to
nothing as no-default-models instead of unrestricted. All-team-models
team keys get team_model_access_denied again, as on main.
2026-11-16 23:52:29 -08:00
mateo-berri
305fe443b5 Merge remote-tracking branch 'origin/main' into litellm_access_group_team_scope
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts:
#	litellm/proxy/auth/model_checks.py
#	litellm/proxy/auth/user_api_key_auth.py
#	litellm/proxy/proxy_server.py
#	litellm/proxy/utils.py
#	tests/unit/llms/anthropic/messages/test_safeguards.py
#	tests/unit/proxy/auth/test_model_checks.py
#	tests/unit/proxy/proxy_server/test_team_model_name_translation.py
#	tests/unit/proxy/utils/helpers/test_model_access.py
2026-11-16 22:41:06 -08:00
moe-berri
1efd8b38d3
feat(proxy): support Cursor CLI subscriptions (#52205)
* feat(proxy): relay Cursor CLI subscription sessions

Adapt dm3ch's native Cursor relay to current main, preserving separate gateway authentication, opaque streams, and policy admission checks.

* fix(proxy): enforce Cursor relay policies and preserve stream accounting

* fix(proxy): allow logging callbacks on Cursor relay

---------

Co-authored-by: Dmitry Chepurovskiy <me@dm3ch.net>
2026-11-16 21:49:12 -08:00
yujonglee
c25c669167
feat(auth): decide key and JWT authentication in a Rust core (#51374)
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.

The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.

Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.

A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.

The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
2026-11-17 14:07:47 +09:00
yuneng-jiang
7f7bb13707
fix(auth): apply current team limits in the pre-body rate limit check (#52169)
* fix(auth): apply current team limits in the pre-body rate limit check

The check that refuses an exhausted key before reading the request body
used the team limits copied onto the cached key. Raising a team's rpm or
tpm limit with /team/update did not reach that copy, so a throttled key
stayed throttled until the key cache expired, and the full auth path that
does refresh team fields never ran.

The check now overlays the cached team's newer fields the same way full
auth does, through one shared helper. When the team is not in the cache,
as on a peer right after an invalidation, it defers to full auth instead
of judging the key on stale limits.

* fix(auth): keep key limits in the early check when the team is not cached

On a team cache miss the early check skipped every pre-body limit, so an
exhausted key read its body before being refused. Apply the key's own
limits and defer to full auth only when the cached team limit refuses.
2026-11-16 18:24:43 -08:00
Kerry Lu
fe3e6bbb01
fix(mcp): let zero-budget users call MCP tools that cost nothing (#52130)
* fix(mcp): let zero-budget users call MCP tools that cost nothing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): price a tool call made by its display name as the tool it runs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep budget checks for bare display names a paid server can run and for cut non-tool-call bodies

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): budget check reserved tool names and display names that resolve on to a paid tool

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(mcp): follow chained display names without recursion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): price a scoped tool call on the servers it is scoped to and follow display names as execution does

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep the budget check for MCP servers that use display names

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 17:44:44 -08:00
joshua-berri
f94c79290c
feat(ui): show an optional currency estimate on the Usage spend card (#51979)
* feat(ui): show the usage spend total in another currency at ECB reference rates

Adds GET /get/reference_exchange_rates, which returns the current ECB
reference rates as USD per unit for every feed currency. It refreshes at
most once a day under a lock, keeps fresh rates when a refresh fails and
answers 503 instead of serving rates older than seven days.

The usage overview's spend card gets a Spend display selector. It asks for
rates only once opened and then shows an estimated amount with the rate date
under the unchanged USD total. Every other figure stays in USD.

* fix(ui): fall back to USD when spend display rates stop being usable

The selector now shows USD whenever no estimate can be shown, so a failed
refetch never leaves EUR selected beside a USD-only notice, and an opened
selector refreshes its rates hourly. The endpoint gets its HTTP client
inside the async fetch so the client belongs to the serving event loop.

* fix(proxy): import the ECB rates endpoint HTTP client helper at module level

Binds get_async_httpx_client at import so its type is known. The client is
still built inside the awaited fetch on the serving event loop.

* fix(proxy): fetch display rates with a short-lived handler closed after each refresh

Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, and drops the descriptive comment on the spend display component.

* chore(ui): regenerate API types for the reference exchange rates endpoint

* fix(proxy): use the cached async client factory for display rate refreshes

Direct AsyncHTTPHandler construction fails the async client check, so the
refresh fetch goes back to get_async_httpx_client, called inside the awaited
fetch, with a named type suppression for the factory's untyped params.

* fix(ui): tidy spend display helper, hook options and fallback test

Groups the rate snapshot into one estimatedSpend argument, names the rates
query options, names the missing-estimate condition, and waits for the
unavailable notice before asserting the USD fallback.

* chore(ui): drop a restating comment on the spend estimate helper

* test(proxy): prove concurrent first rate requests share one ECB fetch

* fix(proxy): wait a minute before retrying a failed display rate refresh

A failed ECB refresh left rates due, so every request after it waited on
the lock and fetched again, and N queued callers paid N timeouts during an
outage. The endpoint now skips refreshing for 60 seconds after a failed
attempt, serving held rates or a 503, and retries on the first request
after that. Concurrent waiters share the failed attempt. Also drops the
usd_rates docstring.

* fix(ui): retry unavailable currency estimates on reopen

* refactor(ui): name the currency selector open handler

* style(ui): format currency selector props

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 16:00:20 -08:00
joshua-berri
1540acd7e4
feat(proxy): accept request token prices in a declared currency (#51971)
* feat(proxy): convert per-request token prices from pricing_currency to USD

A trusted chat completion request can now send pricing_currency with
input_cost_per_token and/or output_cost_per_token. The proxy converts the
supplied prices to USD once at ECB reference rates in pre-call, right after
the existing pricing strip, and removes the field before routing. Requests
without the field, and USD, behave as before and make no network call.

The field sits behind the same two gates as request prices. Other routes,
invalid codes or prices, other pricing fields alongside it, and rates that
are unavailable, stale or out of range are rejected instead of being priced
as USD. Cold lookups share one bounded refresh and fresh rates never wait.

* test(proxy): type pricing currency tests and cover failed rate refreshes

Types the new pre-call tests, waits for the shared refresh to finish even
when the due-refresh test fails, and covers stale rates with a failed
refresh (503), a failed background refresh keeping fresh rates, and a
refresh that raises.

* test(proxy): restore an unrelated import line reformatted by mistake

* test(proxy): use the current chat model in request pricing fixtures

* fix(proxy): reject pricing_currency on generic pass-through requests

An authorized key keeps request prices on a generic pass-through, which
bills them as USD, while pricing_currency was forwarded upstream untouched.
The pass-through body boundary now rejects a non-null pricing_currency with
a 400 and drops a null one, matching the chat-only conversion.

* fix(proxy): type the request pricing conversion call and ECB client import

Validates the request body to a typed mapping only when pricing_currency is
present, converts the supplied prices from that view, and writes them back
onto the original request. Imports the async HTTP client helper at module
level so its binding is typed. The client is still built inside the fetch.

* test(proxy): cover a null pricing_currency through the pre-call pipeline

* fix(proxy): fetch ECB rates with a short-lived handler closed after each refresh

Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, instead of binding the untyped cached client factory.

* fix(proxy): wait a minute before retrying a failed ECB rate refresh

A failed refresh left rates due, so every later non-USD request started a new
ECB fetch during an outage. The shared lookup now reuses a failed refresh
for 60 seconds from its start, so requests in that window get the held
rates or a 503 without fetching again, and the next request after it
retries. It also goes back to the cached async client factory, with a named
type suppression for the factory's untyped params argument.

* fix(proxy): convert pricing_currency on every chat completions alias

The /engines/{model}/chat/completions and /openai/deployments/{model}/chat/completions
aliases reach the same chat handler but were rejected, because the check
compared the literal request path. It now compares the registered route
template, falling back to the path when no template is available. Also
covers a cancelled refresh not blocking the next lookup.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 15:16:24 -08:00
Kerry Lu
c7e5f78724
fix(proxy): register new customer org aliases one at a time per worker so a burst of first requests is served (#52080)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 14:00:04 -08:00
yuneng-jiang
47a0d50c35
test: remove dead skip-marked tests and rewrite the Redis buffer drain test (#52063)
Fourteen tests carried an unconditional skip and could never run. The unit
conftest deletes DATABASE_URL and REDIS_* before every test, so the prisma
and Redis ones would fail in tests/unit even without the marker. Each
behavior they claimed is already covered by mocked unit tests or by the
integration suite against real Postgres and Redis, and the gpt-4o token
counter test patched a function removed in #10409.

The Redis buffer drain test was the one with unique value, so it is
rewritten in test_redis_update_buffer.py against the fakeredis-backed
RedisCache fixture. It fills every queue a spend flush writes, drains the
buffer, and asserts no Redis key is left behind.
2026-11-16 10:50:52 -08:00
Yassin Kortam
723ef77441
feat(control_plane): dataplane registry, credential strategies and client (#49655)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 17:36:23 +00:00
yujonglee
76e45df333
refactor(auth): plan the authentication path in Rust (#51605)
The auth flags read through python-compat's IsTrue adapter and public_routes
through RouteList on both hosts, and python-bridge reads those keys through
GeneralSettings' own serde rules. A flag is on only for a boolean true, and a
public_routes value that is not a list of strings adds no public route, so
the Rust gateway and the Python proxy read the same settings the same way.
2026-11-16 19:36:57 +09:00
yujonglee
4aa00352bb
refactor(auth): decide route rules in Rust (#51531)
Whether a route may be called was decided by Python helpers spread over
RouteChecks and the managed agent module: wildcard, prefix and deny matching,
pass-through grants, and the managed agent route table. The Rust auth core
needs the same rules, and a second copy would drift.

The rules move into a new gateway-auth-core crate under routes/, and Python
calls them through _native. RouteChecks and managed_authorization keep their
function names and now delegate, so every existing caller is unchanged. The
LiteLLMRoutes groups stay Python's and reach Rust through
litellm/rust_bridge/host/route_groups.py.

Route templates parse in a new litellm-starlette crate, which follows
starlette's compile_path and never fails. A pass-through path with an int,
float or uuid parameter used to panic the route group load and now matches
one segment, as the Python matcher did. Both matchers share one walk over
the template that keeps the set of offsets reached so far, so a template
with many path parameters answers in polynomial time where trying every
split, as Python's re does, never finishes. Route lists from key and team
metadata, and JWT team_allowed_routes, read as litellm-config's RouteList, so
a value that is not a list of strings answers with a 403 instead of failing
the request: a malformed allow list grants nothing and a malformed deny list
denies every pass-through route. The Rust route groups are rebuilt whenever
LiteLLMRoutes changes, since an auth-enforced pass-through endpoint adds its
path to openai_routes while the proxy runs. Reading config
files and their include moves out of litellm-config into a new
litellm-config-load crate, which leaves litellm-config the typed data model.

How Python reads a config value it never validates now lives in one place,
python-compat's lenient module: `value is True`, `bool(value)`, and a value of
the expected shape or Malformed. Config types use them as serde_with adapters,
so the gateway's YAML load and python-bridge's from_py read a value the same
way. general_settings.pass_through_endpoints is typed with them, its path as a
starlette RouteTemplate. The config types derive Debug, and veil's Redact where
a field can carry a credential, in place of their hand-written impls.

No change to which well-formed routes are allowed or denied is intended.
2026-11-16 18:55:32 +09:00
shrey
58d86c2148
fix(proxy)!: match the sign-in email fallback to the right user (#51897) 2026-11-15 19:13:38 -08:00
Kerry Lu
60716572a4
feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705)
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares

Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap

One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn

A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 12:05:58 -08:00
Kerry Lu
063b63de6c
fix(proxy): let an exhausted key keep listing models and counting tokens (#51675)
* fix(proxy): keep an exhausted key on model listing and token counting routes

The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.

* fix(proxy): still refuse a GET that declares a body before reading it

A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.

* test(proxy): fail the preflight test on any outcome other than admitted or 429

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:26 +08:00
Nate Armstrong
09ce4f50ed
perf(auth): read token model fields from the class instead of deprecated __fields__ (#51901)
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* perf(auth): read token model fields from the class instead of deprecated __fields__

* test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps
2026-11-15 07:07:49 +00:00
yuneng-jiang
da55893656
ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864)
* test(unit): reset module state only when a test changed it

The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.

The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.

* test(unit): remove six flaky tests

They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.

* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests

proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.

windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.

* test: move 109 tests that start a process or open a socket into tests/integration

tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.

The tests themselves are unchanged apart from one repository-root path in the
CLI test.

* test(unit): copy back only the dict entries a test changed

A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.

* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate

A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.
2026-11-14 15:41:30 -08:00
tavren27
9063ba6929
fix(proxy): stop wildcard aliases over unknown orgs listing every catalog id (#51805)
* fix(proxy): stop listing catalog ids under an org the catalog lacks for every provider whose path is its model id

* fix(proxy): keep listing Fireworks router and Azure model_router wildcards

The catalog cannot price the names these routes call, so the strict org check dropped callable ids. They now list as before, and every exempt provider has a test row that fails if it leaves the list

* fix(proxy): check unknown orgs beneath router routes and keep firerouter pools

* fix(proxy): keep listing Fireworks routers/firerouter pools and Azure AI speech wildcards

* fix(proxy): check unknown orgs under OpenAI and Cohere and drop unreachable Fireworks router ids

* fix(proxy): keep the full wildcard listing for deployments behind a custom api_base

* fix(proxy): count a provider base URL set in the env or litellm.api_base as a custom endpoint

* fix(proxy): match a gateway env var on any part of the provider name

* fix(proxy): count a provider's off-pattern base URL env vars as a gateway

* test(proxy): cover every off-pattern base URL env var in the gateway table

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 16:34:59 -05:00
giladk1221
9d827093c5
feat(github_copilot): support GitHub App authentication with PEM upload (#51427)
* feat(github_copilot): support GitHub App installation authentication

* Add GitHub App authentication to the Copilot credential form

* Explain secure private-key setup beside Copilot App fields

* Store uploaded Copilot App keys through encrypted named credentials

* fix(github_copilot): handle App validation errors and credential replacement

* fix(ui): make GitHub App field help accessible

* fix(github_copilot): preserve saved App key references on edit

* fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health

* fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses

Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers.

App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares.

A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change.

App deployments are deployment-owned, so they share the response cache like shared device-login deployments.

App credentials are validated even when credential_info names no provider.

* fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled

The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future

* fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs

A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again

* fix(github_copilot): report a deployment naming an unloaded credential as unhealthy

The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login

* fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base

A Copilot deployment that names an unloaded credential uses the shared device login again, as on base.
A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call.
get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores.

* fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure

* fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403

Only an App deployment reports its App host as api_base before a configured api_base.
Per-user OAuth sessions report the same api_base they did on base.
The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication.
The token-dir fixture in the App api_base test is applied with usefixtures.

* fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete

* test(github_copilot): type the incomplete App settings test parameter

---------

Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 10:33:20 -05:00
norkelbyte
ac1cc5111c
fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions (#51753)
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions

* fix(proxy): check the key's max_budget on /queue/chat/completions

* fix(proxy): run budget, tag, and request policy checks on queued chat calls

* fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls

A queued stream now hands its logging object to the stream generator, so a
client that leaves mid-stream is billed for the reply it already got, and a
stream left before the first model chunk releases its budget reservation.
A team's require_trace_id and the failure logging for rejected calls now
use the same route check as the other inference checks.

* fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses

* fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls

* fix(proxy): send the call ID on a queued call the provider rejects before streaming

* fix(proxy): release a dropped stream's budget hold only when nothing bills it

A guardrail that buffers the stream can pull model chunks the client never
saw. The early-drop release settled the hold at the input cost first, then
the partial bill for those chunks added its full cost on top. The release
now runs after partial billing and only when no bill fired

* fix(proxy): fold the caller's litellm_metadata into a queued call's metadata

A queued call that sent a litellm_metadata bucket, even an empty one, made tag
routing and logging read that bucket, so the key and team tags the handler
merged into metadata no longer narrowed deployments. The queue handler now
folds the bucket into metadata and drops it, as chat does, after stripping the
proxy-owned user_api_key_* and internal control fields from both buckets, so a
caller can no longer set the queued spend row's organization ID either

* fix(proxy): release a queued call's budget hold when the call is cancelled before it answers

* fix(proxy): refund a cancelled queued call's hold in full until the model call claims it

* fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 08:54:54 -05:00
quietjules
007d1c1b5e
feat(proxy): support budget_limits windows on internal users (#51763)
* feat(proxy): support internal user budget windows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): make user budget migration repeatable and sync API types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): bound lock wait for user budget migration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): omit unset JSON windows when creating users

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): retain personal budget windows for proxy admins

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): give user budget windows their own counters and keep them on the user

User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path

* fix(proxy): keep windows on team auto keys and write all users' windows in one transaction

* fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock

An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets.

JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do

* fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock

Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses

The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits

* test(proxy): check default window resets against the reset helper instead of the wall clock

* fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes

* fix(proxy): fail an all-users bulk update whole when its transaction fails

The per-user fallback after a failed all-users transaction wrote the newest
user and then crashed on a missing user_email attribute, so the call answered
500 with one user changed. Racing window bulk updates past the 5s transaction
limit reached it. Answer 500 with no user changed instead

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 07:44:35 -05:00
joshua-berri
d11917e89c
fix(proxy): preserve caller metadata on auth failures (#51745)
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs

Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.

* fix(proxy): type the spend logs metadata header parser honestly

The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.

* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket

A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.

* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler

Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.

* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer

The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.

* test(proxy): read auth failure spend rows from a real unconnected PrismaClient

Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 16:13:50 -08:00
moe-berri
0d422a1b6c
fix(auth): let dashboard sessions use the user's team models (#51663)
* fix: inherit team models for dashboard sessions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: authorize dashboard models from owner grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve dashboard model listing filters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): attribute dashboard team model spend to granting teams

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Keep dashboard fallbacks on the original billing team

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Preserve admitted dashboard billing through primary model rewrites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reject dashboard batches that need team model grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Respect wildcard exclusions in dashboard model listings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 17:13:06 -05:00
moe-berri
1a682f9dbf
feat(customer-portal): add free tier limits with Enterprise upgrade path (#51687)
* feat: add customer portal free tier limits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: narrow customer portal free tier enforcement

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Show customer portal free tier limits in dashboard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: check customer key scope before organization registration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(customer-portal): preserve overflow requests and refresh limits after deletion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 14:43:15 -05:00
Kerry Lu
367890f958
fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias (#51674)
* fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias

* test(proxy): a pinned key naming an unknown org alias is refused with 403 even when registration would fail

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:03:43 -08:00
Kerry Lu
48f23ea1cb
fix(proxy): register a customer member in a customer org with no default budget (#51660)
* fix(proxy): register a customer member in a customer org with no default budget

A request that named a customer org and a customer member the proxy had never
seen created the customer org and charged it, then dropped the customer member
because a new customer org has no default budget. The customer member is now
registered in the default customer team with no budget, so its spend is tracked
from the first request and nothing caps it until an admin assigns one. Admin
creates still require a budget.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): write only the budget fields a customer member patch names

An alias edit wrote back the budget fields it read, so a budget assigned
between its read and its write was erased. The update now leaves the budget
columns alone unless the patch names them.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 18:43:51 +00:00
ishaan-j-berri
df8b8d1b51
feat(customer-portal): complete hierarchy management (#51521)
* feat(customer-portal): complete hierarchy management

* fix(ci): update Soup Sieve across supported Python versions

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 17:58:34 +00:00
ishaan-j-berri
a60e16b87d
fix(proxy): display customer identity in spend logs (#51596)
* fix(proxy): display customer identity in spend logs

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): omit absent customer identity from spend metadata

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): retain only authorized customer log identities

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* feat(ui): clarify customer identities on spend logs

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 17:39:50 +00:00
ishaan-j-berri
6d15d68bc8
feat(proxy): register an unknown customer organization alias as a new organization charged for the request (#51637) 2026-11-13 17:12:45 +00:00
ishaan-j-berri
9980f9f4d0
feat(proxy): pick the customer organization's default budget template by name on create and by id on patch (#51636) 2026-11-13 17:10:04 +00:00
ishaan-j-berri
616471d400
feat(proxy): run requests with unknown customer ids untracked instead of refusing them (#51548) 2026-11-13 16:56:21 +00:00
ishaan-j-berri
ca4c6c0a6d
feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582)
* fix(proxy): reset customer org, team and member spend on their budget row's window

The customer tables carried their own budget_reset_at next to a budget_id whose
row already has one. Drop the column from the three tables and let the budget
table cascade zero the customer rows it governs, with a member following its
unexpired override when that override has a duration and its template
otherwise. Customer budget writes stamp budget_reset_at from budget_duration the
way /budget/new does, and the management responses report the governing budget's
window under the same budget_reset_at field.

* feat(budgets): support concurrent customer budget windows

* fix(budgets): resolve compatibility and validation checks

* feat(proxy): name budget rows, give customer teams an override and members a parent org

* fix(budgets): preserve admitted windows during accounting

* chore(schema): allow dropping the unreleased customer budget_reset_at columns

* fix(budgets): apply standard migration lock timeout

* feat(proxy): customer members belong to many teams, charged to the one the request names

* test(migrations): allow the unreleased customer budget_reset_at drop

* style(ui): prettier the customer dashboard files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(reset-job): name the batch bound on the customer budget IN filters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(budgets): preserve mixed limits and resolve CI regressions

* fix(budgets): reuse shared spend verification for customer windows

* refactor(ui): isolate customer budget group creation

Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild.

* fix(proxy): restore Field import and bound the member team lookup after the merge

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* test(proxy): send parent_team_ids in the budget window customer tests

Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): keep rejecting another organization's template after the budget_template_ids rename

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* test(proxy): cover customer budget increase requests and the cap_policy field

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* chore(ui): regenerate API types for cap_policy and customer budget increase requests

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): hang the increase request back-reference on the customer member model

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): declare increase request list filters with Annotated Query

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 16:05:35 +00:00
ishaan-j-berri
6eda17aadb
fix(proxy): enforce settled customer budgets across workers (#51568)
* fix(proxy): enforce settled customer budgets across workers

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): sync customer budget reset API types

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reseed customer spend through shared counters

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 07:33:24 -08:00
joshua-berri
ac5fd9e0d0
fix(interactions): route Vertex lifecycle calls through the selected deployment (#51511)
* fix(interactions): route Vertex interaction follow-ups to the selected deployment

GET, cancel and DELETE by interaction id accept a model group or a deployment
id through the model query parameter or the x-litellm-model header, authorize
it against the key and team like any model, and run with that deployment's
project and credentials. Without a selector they still go to Google AI Studio.
Vertex cancel now uses the documented /cancel path.

* test(bedrock): pin the SigV4 clock in the embedding signature test

SigV4Auth.add_auth stamps the request with botocore.auth.get_current_datetime(), overwriting the timestamp the test set on the replay. When the proxy call and the replay landed in different seconds, the signatures differed. Fix the module clock for both signings and drop the ineffective manual timestamp.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 00:02:43 -08:00
ishaan-j-berri
27e9a04d62
feat(proxy): register unknown customer members in the aliased organization's default team (#51546)
* feat(proxy): name the customer organization by alias with x-litellm-customer-org-alias

A request may carry x-litellm-customer-org-alias next to, or instead of,
x-litellm-customer-member-id. Alone, the alias charges and enforces the
organization level only. Next to the member header it must name the
member's organization, else 403. Unknown alias is 401, an alias shared
by several organizations is 400. Organization alias changes evict the
cached lookup.

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* chore(ui): regenerate schema.d.ts for optional customer team and member ids

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* feat(proxy): register unknown customer members in the aliased organization's default team

A request carrying x-litellm-customer-org-alias and an
x-litellm-customer-member-id the proxy has never seen creates the member
under the organization's default team, on the organization's default
budget, and serves the request. Without an alias, or when the
organization has no default budget, the member stays unknown and the
request is still refused.

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): declare the member registrar callable as a TypeAlias

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 05:15:45 +00:00
Kerry Lu
d30b2b9b26
feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489)
* feat(proxy): enforce customer org, team and member budgets and limits on the request path

Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): customer hierarchy survives virtual key auth without a legacy end user

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(auth): keys pinned to a customer organization may only act for that organization's members

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): keep the customer hierarchy spend maps within the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): check the key's customer org scope after the key is resolved

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the hierarchy cache key family and keep type counts flat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): resolve the customer hierarchy in the shared authorization gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop a member's cached hierarchy through the declared cache target

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(auth): keep the hierarchy reset within the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): type the requested models as a sequence

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(spend): add daily customer usage and budget resets (#51493)

* feat(spend): add daily customer usage and budget resets

* chore(spend): document customer reset query bounds

* fix(spend): chart customer queue metrics and update recovery test

* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "feat(spend): add daily customer usage and budget resets (#51493)"

This reverts commit ccac47608f.

* fix(proxy): evict cached member hierarchies when team, org or template budgets change

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the auth cache dependency alias as a TypeAlias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"

This reverts commit 7542471e17.

* fix(spend): reconcile the daily customer spend table with main

Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(pagerduty): tolerate failure metadata without customer hierarchy ids

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(accounting): key daily customer spend rows on main's hashed identity

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(pagerduty): read customer ids from failure metadata through a typed lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
2026-11-12 20:43:44 -08:00
Mateo Wang
b60246c352
fix(proxy): keep call id, model and key logging on keys refused before the body is read (#51525)
* Fix failure logging context for cached key preflight rejections

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Centralize preflight failure route classification and restore precedence

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Bound preflight logging reads and tolerate model lookup failures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 19:39:51 -08:00
yujonglee
2be96aed13
refactor(redis): delete the Python RedisBatch now that every request op is native (#51383)
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call
deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss
tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left
2026-11-13 03:22:28 +00:00
ishaan-j-berri
b251f125a7
feat(proxy): customer org, team and member hierarchy with budget templates (#51459)
* feat: add customer hierarchy budget management

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* refactor: clarify customer hierarchy management and enforce ownership

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix: return 201 for customer hierarchy creates

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* Refine customer budget templates and expiring member overrides

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix: guard customer hierarchy migration and register tables for span redaction

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix: chunk template budget lookup and lock the organization row while rewriting templates

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix: allow management_routes keys on customer routes and block /budget/delete on org default budgets

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix: clearing a member override drops its expiry and a null budget clears org and team budgets

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-12 15:11:51 -08:00
shrey
6685525fa2
feat(access-groups): grant vector stores through access groups (#50290) 2026-11-12 14:59:17 -08:00
Kerry Lu
98dbe80ada
fix(proxy): refuse blocked and over-limit keys before reading the request body (#51428)
* fix(proxy): refuse blocked and over-limit keys before reading the request body

A client retrying 429s with multi-megabyte payloads made every worker buffer each payload in full before the refusal went out, so the rejections themselves drove memory. Auth now checks the cached key for a block or an exhausted key-level rate limit from the headers alone and answers before the body is read; admitted requests and cache misses take the unchanged path.

* fix(proxy): pick the effective key header for the early refusal and keep its failure log

The headers-only check now reads x-litellm-api-key when present, skips a missing Authorization header instead of raising, stamps the route on the cached identity so the failure is logged, and honors an exception a callback returns from the failure hook.

* fix(proxy): honor litellm_key_header_name in the early refusal

The headers-only check now reads the key from the header general_settings.litellm_key_header_name names when the request carries it, the same precedence the full auth path applies.

* fix(proxy): keep the early-rejection path inside the type gate ceilings

Read the configured key header through general_settings_view() instead of importing the untyped
general_settings global, and drop the isinstance on an exception post_call_failure_hook already types as
HTTPException.

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 14:05:05 -08:00
Mateo Wang
00bd56760d
fix(auth): enforce JWT end-user budget rate limits (#51405)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 08:51:27 -08:00
Yassin Kortam
db6dfb6bb6
chore: move workspace packages under litellm (#51000)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 04:19:57 +00:00
Yassin Kortam
f6bee81c73
chore: move ui and vscode-extension under litellm and delete docs (#50992)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 04:19:57 +00:00
joshua-berri
a5d04ec9ac
fix(auth): allow video retrieval after budget exhaustion (#51227)
* fix(auth): allow paid video retrieval after budget exhaustion

* test(auth): check video retrieval boundary on registered route

* fix(auth): exempt video retrieval in post-custom-auth checks

* style(auth): format custom video budget exemption

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-11 17:10:01 -08:00
shrey
8ee4ce8418
fix(proxy): stop request body fields from conflicting with admin-set deployment fields (#51254) 2026-11-11 23:39:17 +00:00
joshua-berri
deb7697a8e
fix(auth): enforce model access on Google and image URL routes (#51226)
* fix(auth): authorize URL-selected Google and image models

* fix(auth): preserve server model selection and budget fallbacks

* test(auth): isolate URL model authorization settings

* fix(auth): type handler model selection boundaries

* fix(auth): validate request model selection inputs

* fix(auth): tolerate absent request path metadata

* fix(auth): support requests without path metadata

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-11 12:22:12 -08:00
quietjules
ac6e78cb1f
fix(proxy): list callable ids for partial aliases over a catch-all deployment (#50963)
* fix(proxy): list ids that route back for a prefixed partial alias over a catch-all deployment

* test(proxy): call the router registry in the catch-all alias listing test

* fix(proxy): keep main's listing for whole-name aliases over a catch-all

Only partial aliases like team1/gpt-* over a bare * run the per-id route check. A whole-name alias like team1/* keeps the string-only listing it had, so it lists the same ids at the same cost. Adds a test that a more specific sibling alias's ids are left out of the partial alias's list.

* fix(proxy): list catch-all aliases fast and skip ids a sibling alias takes

* fix(proxy): check team wildcard aliases without a router call per id and skip suffix ids that name another catalog model

Team keys now read the team's own wildcard patterns the way the router does (any team pattern first, then the provider form when its prefix can be a provider), so ids no rival can take skip the router call. A suffix alias such as team1/*-mini no longer lists team1/gpt-5.4-mini for gpt-5.4.

* fix(proxy): keep team key listings fast when aliases overlap and honor a pinned provider on partial aliases

A team key whose team defines the same alias as the global config, or a team
pattern that can take a global alias's ids, now checks only the team's own
patterns before calling the router, so the listing no longer pays a router
call per catalog id. The router also tries the provider-prefixed form of an id
only when the bare form matches nothing.

A partial alias over `*` with `custom_llm_provider` set now lists only that
provider's catalog, so `team1/gpt-*` pinned to OpenAI no longer lists
`team1/gpt-claude-*` ids that answer 404.

* fix(proxy): ask a pinned catch-all's provider for its models once per listing

A partial alias over a `*` deployment pinned with custom_llm_provider fetched the whole `*` catalog before taking the pinned branch, so with check_provider_endpoint on each listing called the provider's /v1/models one extra time. The pinned branch now runs before that fetch. Tests also cover a provider-less team alias that takes ids under their provider and a wildcard model_group_alias

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 22:20:06 -08:00