A group whose members all belong to other teams was dropped from the
team-filtered map, so its name leaked into /v1/models as a plain model.
Keep the group with no members, and treat a grant that expands to
nothing as no-default-models instead of unrestricted. All-team-models
team keys get team_model_access_denied again, as on main.
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.
The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.
Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.
A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.
The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
* fix(auth): apply current team limits in the pre-body rate limit check
The check that refuses an exhausted key before reading the request body
used the team limits copied onto the cached key. Raising a team's rpm or
tpm limit with /team/update did not reach that copy, so a throttled key
stayed throttled until the key cache expired, and the full auth path that
does refresh team fields never ran.
The check now overlays the cached team's newer fields the same way full
auth does, through one shared helper. When the team is not in the cache,
as on a peer right after an invalidation, it defers to full auth instead
of judging the key on stale limits.
* fix(auth): keep key limits in the early check when the team is not cached
On a team cache miss the early check skipped every pre-body limit, so an
exhausted key read its body before being refused. Apply the key's own
limits and defer to full auth only when the cached team limit refuses.
* fix(mcp): let zero-budget users call MCP tools that cost nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): price a tool call made by its display name as the tool it runs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep budget checks for bare display names a paid server can run and for cut non-tool-call bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): budget check reserved tool names and display names that resolve on to a paid tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): follow chained display names without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): price a scoped tool call on the servers it is scoped to and follow display names as execution does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep the budget check for MCP servers that use display names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): show the usage spend total in another currency at ECB reference rates
Adds GET /get/reference_exchange_rates, which returns the current ECB
reference rates as USD per unit for every feed currency. It refreshes at
most once a day under a lock, keeps fresh rates when a refresh fails and
answers 503 instead of serving rates older than seven days.
The usage overview's spend card gets a Spend display selector. It asks for
rates only once opened and then shows an estimated amount with the rate date
under the unchanged USD total. Every other figure stays in USD.
* fix(ui): fall back to USD when spend display rates stop being usable
The selector now shows USD whenever no estimate can be shown, so a failed
refetch never leaves EUR selected beside a USD-only notice, and an opened
selector refreshes its rates hourly. The endpoint gets its HTTP client
inside the async fetch so the client belongs to the serving event loop.
* fix(proxy): import the ECB rates endpoint HTTP client helper at module level
Binds get_async_httpx_client at import so its type is known. The client is
still built inside the awaited fetch on the serving event loop.
* fix(proxy): fetch display rates with a short-lived handler closed after each refresh
Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, and drops the descriptive comment on the spend display component.
* chore(ui): regenerate API types for the reference exchange rates endpoint
* fix(proxy): use the cached async client factory for display rate refreshes
Direct AsyncHTTPHandler construction fails the async client check, so the
refresh fetch goes back to get_async_httpx_client, called inside the awaited
fetch, with a named type suppression for the factory's untyped params.
* fix(ui): tidy spend display helper, hook options and fallback test
Groups the rate snapshot into one estimatedSpend argument, names the rates
query options, names the missing-estimate condition, and waits for the
unavailable notice before asserting the USD fallback.
* chore(ui): drop a restating comment on the spend estimate helper
* test(proxy): prove concurrent first rate requests share one ECB fetch
* fix(proxy): wait a minute before retrying a failed display rate refresh
A failed ECB refresh left rates due, so every request after it waited on
the lock and fetched again, and N queued callers paid N timeouts during an
outage. The endpoint now skips refreshing for 60 seconds after a failed
attempt, serving held rates or a 503, and retries on the first request
after that. Concurrent waiters share the failed attempt. Also drops the
usd_rates docstring.
* fix(ui): retry unavailable currency estimates on reopen
* refactor(ui): name the currency selector open handler
* style(ui): format currency selector props
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(proxy): convert per-request token prices from pricing_currency to USD
A trusted chat completion request can now send pricing_currency with
input_cost_per_token and/or output_cost_per_token. The proxy converts the
supplied prices to USD once at ECB reference rates in pre-call, right after
the existing pricing strip, and removes the field before routing. Requests
without the field, and USD, behave as before and make no network call.
The field sits behind the same two gates as request prices. Other routes,
invalid codes or prices, other pricing fields alongside it, and rates that
are unavailable, stale or out of range are rejected instead of being priced
as USD. Cold lookups share one bounded refresh and fresh rates never wait.
* test(proxy): type pricing currency tests and cover failed rate refreshes
Types the new pre-call tests, waits for the shared refresh to finish even
when the due-refresh test fails, and covers stale rates with a failed
refresh (503), a failed background refresh keeping fresh rates, and a
refresh that raises.
* test(proxy): restore an unrelated import line reformatted by mistake
* test(proxy): use the current chat model in request pricing fixtures
* fix(proxy): reject pricing_currency on generic pass-through requests
An authorized key keeps request prices on a generic pass-through, which
bills them as USD, while pricing_currency was forwarded upstream untouched.
The pass-through body boundary now rejects a non-null pricing_currency with
a 400 and drops a null one, matching the chat-only conversion.
* fix(proxy): type the request pricing conversion call and ECB client import
Validates the request body to a typed mapping only when pricing_currency is
present, converts the supplied prices from that view, and writes them back
onto the original request. Imports the async HTTP client helper at module
level so its binding is typed. The client is still built inside the fetch.
* test(proxy): cover a null pricing_currency through the pre-call pipeline
* fix(proxy): fetch ECB rates with a short-lived handler closed after each refresh
Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, instead of binding the untyped cached client factory.
* fix(proxy): wait a minute before retrying a failed ECB rate refresh
A failed refresh left rates due, so every later non-USD request started a new
ECB fetch during an outage. The shared lookup now reuses a failed refresh
for 60 seconds from its start, so requests in that window get the held
rates or a 503 without fetching again, and the next request after it
retries. It also goes back to the cached async client factory, with a named
type suppression for the factory's untyped params argument.
* fix(proxy): convert pricing_currency on every chat completions alias
The /engines/{model}/chat/completions and /openai/deployments/{model}/chat/completions
aliases reach the same chat handler but were rejected, because the check
compared the literal request path. It now compares the registered route
template, falling back to the path when no template is available. Also
covers a cancelled refresh not blocking the next lookup.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Fourteen tests carried an unconditional skip and could never run. The unit
conftest deletes DATABASE_URL and REDIS_* before every test, so the prisma
and Redis ones would fail in tests/unit even without the marker. Each
behavior they claimed is already covered by mocked unit tests or by the
integration suite against real Postgres and Redis, and the gpt-4o token
counter test patched a function removed in #10409.
The Redis buffer drain test was the one with unique value, so it is
rewritten in test_redis_update_buffer.py against the fakeredis-backed
RedisCache fixture. It fills every queue a spend flush writes, drains the
buffer, and asserts no Redis key is left behind.
The auth flags read through python-compat's IsTrue adapter and public_routes
through RouteList on both hosts, and python-bridge reads those keys through
GeneralSettings' own serde rules. A flag is on only for a boolean true, and a
public_routes value that is not a list of strings adds no public route, so
the Rust gateway and the Python proxy read the same settings the same way.
Whether a route may be called was decided by Python helpers spread over
RouteChecks and the managed agent module: wildcard, prefix and deny matching,
pass-through grants, and the managed agent route table. The Rust auth core
needs the same rules, and a second copy would drift.
The rules move into a new gateway-auth-core crate under routes/, and Python
calls them through _native. RouteChecks and managed_authorization keep their
function names and now delegate, so every existing caller is unchanged. The
LiteLLMRoutes groups stay Python's and reach Rust through
litellm/rust_bridge/host/route_groups.py.
Route templates parse in a new litellm-starlette crate, which follows
starlette's compile_path and never fails. A pass-through path with an int,
float or uuid parameter used to panic the route group load and now matches
one segment, as the Python matcher did. Both matchers share one walk over
the template that keeps the set of offsets reached so far, so a template
with many path parameters answers in polynomial time where trying every
split, as Python's re does, never finishes. Route lists from key and team
metadata, and JWT team_allowed_routes, read as litellm-config's RouteList, so
a value that is not a list of strings answers with a 403 instead of failing
the request: a malformed allow list grants nothing and a malformed deny list
denies every pass-through route. The Rust route groups are rebuilt whenever
LiteLLMRoutes changes, since an auth-enforced pass-through endpoint adds its
path to openai_routes while the proxy runs. Reading config
files and their include moves out of litellm-config into a new
litellm-config-load crate, which leaves litellm-config the typed data model.
How Python reads a config value it never validates now lives in one place,
python-compat's lenient module: `value is True`, `bool(value)`, and a value of
the expected shape or Malformed. Config types use them as serde_with adapters,
so the gateway's YAML load and python-bridge's from_py read a value the same
way. general_settings.pass_through_endpoints is typed with them, its path as a
starlette RouteTemplate. The config types derive Debug, and veil's Redact where
a field can carry a credential, in place of their hand-written impls.
No change to which well-formed routes are allowed or denied is intended.
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares
Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap
One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn
A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep an exhausted key on model listing and token counting routes
The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.
* fix(proxy): still refuse a GET that declares a body before reading it
A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.
* test(proxy): fail the preflight test on any outcome other than admitted or 429
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): read token model fields from the class instead of deprecated __fields__
* test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps
* test(unit): reset module state only when a test changed it
The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.
The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.
* test(unit): remove six flaky tests
They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.
* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests
proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.
windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.
* test: move 109 tests that start a process or open a socket into tests/integration
tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.
The tests themselves are unchanged apart from one repository-root path in the
CLI test.
* test(unit): copy back only the dict entries a test changed
A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.
* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate
A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.
* fix(proxy): stop listing catalog ids under an org the catalog lacks for every provider whose path is its model id
* fix(proxy): keep listing Fireworks router and Azure model_router wildcards
The catalog cannot price the names these routes call, so the strict org check dropped callable ids. They now list as before, and every exempt provider has a test row that fails if it leaves the list
* fix(proxy): check unknown orgs beneath router routes and keep firerouter pools
* fix(proxy): keep listing Fireworks routers/firerouter pools and Azure AI speech wildcards
* fix(proxy): check unknown orgs under OpenAI and Cohere and drop unreachable Fireworks router ids
* fix(proxy): keep the full wildcard listing for deployments behind a custom api_base
* fix(proxy): count a provider base URL set in the env or litellm.api_base as a custom endpoint
* fix(proxy): match a gateway env var on any part of the provider name
* fix(proxy): count a provider's off-pattern base URL env vars as a gateway
* test(proxy): cover every off-pattern base URL env var in the gateway table
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(github_copilot): support GitHub App installation authentication
* Add GitHub App authentication to the Copilot credential form
* Explain secure private-key setup beside Copilot App fields
* Store uploaded Copilot App keys through encrypted named credentials
* fix(github_copilot): handle App validation errors and credential replacement
* fix(ui): make GitHub App field help accessible
* fix(github_copilot): preserve saved App key references on edit
* fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health
* fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses
Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers.
App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares.
A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change.
App deployments are deployment-owned, so they share the response cache like shared device-login deployments.
App credentials are validated even when credential_info names no provider.
* fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled
The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future
* fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs
A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again
* fix(github_copilot): report a deployment naming an unloaded credential as unhealthy
The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login
* fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base
A Copilot deployment that names an unloaded credential uses the shared device login again, as on base.
A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call.
get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores.
* fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure
* fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403
Only an App deployment reports its App host as api_base before a configured api_base.
Per-user OAuth sessions report the same api_base they did on base.
The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication.
The token-dir fixture in the App api_base test is applied with usefixtures.
* fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete
* test(github_copilot): type the incomplete App settings test parameter
---------
Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions
* fix(proxy): check the key's max_budget on /queue/chat/completions
* fix(proxy): run budget, tag, and request policy checks on queued chat calls
* fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls
A queued stream now hands its logging object to the stream generator, so a
client that leaves mid-stream is billed for the reply it already got, and a
stream left before the first model chunk releases its budget reservation.
A team's require_trace_id and the failure logging for rejected calls now
use the same route check as the other inference checks.
* fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses
* fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls
* fix(proxy): send the call ID on a queued call the provider rejects before streaming
* fix(proxy): release a dropped stream's budget hold only when nothing bills it
A guardrail that buffers the stream can pull model chunks the client never
saw. The early-drop release settled the hold at the input cost first, then
the partial bill for those chunks added its full cost on top. The release
now runs after partial billing and only when no bill fired
* fix(proxy): fold the caller's litellm_metadata into a queued call's metadata
A queued call that sent a litellm_metadata bucket, even an empty one, made tag
routing and logging read that bucket, so the key and team tags the handler
merged into metadata no longer narrowed deployments. The queue handler now
folds the bucket into metadata and drops it, as chat does, after stripping the
proxy-owned user_api_key_* and internal control fields from both buckets, so a
caller can no longer set the queued spend row's organization ID either
* fix(proxy): release a queued call's budget hold when the call is cancelled before it answers
* fix(proxy): refund a cancelled queued call's hold in full until the model call claims it
* fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): support internal user budget windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make user budget migration repeatable and sync API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound lock wait for user budget migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit unset JSON windows when creating users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): retain personal budget windows for proxy admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): give user budget windows their own counters and keep them on the user
User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path
* fix(proxy): keep windows on team auto keys and write all users' windows in one transaction
* fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock
An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets.
JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do
* fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock
Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses
The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits
* test(proxy): check default window resets against the reset helper instead of the wall clock
* fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes
* fix(proxy): fail an all-users bulk update whole when its transaction fails
The per-user fallback after a failed all-users transaction wrote the newest
user and then crashed on a missing user_email attribute, so the call answered
500 with one user changed. Racing window bulk updates past the 5s transaction
limit reached it. Answer 500 with no user changed instead
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs
Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.
* fix(proxy): type the spend logs metadata header parser honestly
The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.
* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket
A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.
* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler
Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.
* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer
The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.
* test(proxy): read auth failure spend rows from a real unconnected PrismaClient
Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix: inherit team models for dashboard sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: authorize dashboard models from owner grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve dashboard model listing filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): attribute dashboard team model spend to granting teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Keep dashboard fallbacks on the original billing team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve admitted dashboard billing through primary model rewrites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reject dashboard batches that need team model grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Respect wildcard exclusions in dashboard model listings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias
* test(proxy): a pinned key naming an unknown org alias is refused with 403 even when registration would fail
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): register a customer member in a customer org with no default budget
A request that named a customer org and a customer member the proxy had never
seen created the customer org and charged it, then dropped the customer member
because a new customer org has no default budget. The customer member is now
registered in the default customer team with no budget, so its spend is tracked
from the first request and nothing caps it until an admin assigns one. Admin
creates still require a budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): write only the budget fields a customer member patch names
An alias edit wrote back the budget fields it read, so a budget assigned
between its read and its write was erased. The update now leaves the budget
columns alone unless the patch names them.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset customer org, team and member spend on their budget row's window
The customer tables carried their own budget_reset_at next to a budget_id whose
row already has one. Drop the column from the three tables and let the budget
table cascade zero the customer rows it governs, with a member following its
unexpired override when that override has a duration and its template
otherwise. Customer budget writes stamp budget_reset_at from budget_duration the
way /budget/new does, and the management responses report the governing budget's
window under the same budget_reset_at field.
* feat(budgets): support concurrent customer budget windows
* fix(budgets): resolve compatibility and validation checks
* feat(proxy): name budget rows, give customer teams an override and members a parent org
* fix(budgets): preserve admitted windows during accounting
* chore(schema): allow dropping the unreleased customer budget_reset_at columns
* fix(budgets): apply standard migration lock timeout
* feat(proxy): customer members belong to many teams, charged to the one the request names
* test(migrations): allow the unreleased customer budget_reset_at drop
* style(ui): prettier the customer dashboard files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(reset-job): name the batch bound on the customer budget IN filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(budgets): preserve mixed limits and resolve CI regressions
* fix(budgets): reuse shared spend verification for customer windows
* refactor(ui): isolate customer budget group creation
Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild.
* fix(proxy): restore Field import and bound the member team lookup after the merge
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): send parent_team_ids in the budget window customer tests
Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): keep rejecting another organization's template after the budget_template_ids rename
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): cover customer budget increase requests and the cap_policy field
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(ui): regenerate API types for cap_policy and customer budget increase requests
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): hang the increase request back-reference on the customer member model
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): declare increase request list filters with Annotated Query
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(interactions): route Vertex interaction follow-ups to the selected deployment
GET, cancel and DELETE by interaction id accept a model group or a deployment
id through the model query parameter or the x-litellm-model header, authorize
it against the key and team like any model, and run with that deployment's
project and credentials. Without a selector they still go to Google AI Studio.
Vertex cancel now uses the documented /cancel path.
* test(bedrock): pin the SigV4 clock in the embedding signature test
SigV4Auth.add_auth stamps the request with botocore.auth.get_current_datetime(), overwriting the timestamp the test set on the replay. When the proxy call and the replay landed in different seconds, the signatures differed. Fix the module clock for both signings and drop the ineffective manual timestamp.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(proxy): name the customer organization by alias with x-litellm-customer-org-alias
A request may carry x-litellm-customer-org-alias next to, or instead of,
x-litellm-customer-member-id. Alone, the alias charges and enforces the
organization level only. Next to the member header it must name the
member's organization, else 403. Unknown alias is 401, an alias shared
by several organizations is 400. Organization alias changes evict the
cached lookup.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for optional customer team and member ids
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): register unknown customer members in the aliased organization's default team
A request carrying x-litellm-customer-org-alias and an
x-litellm-customer-member-id the proxy has never seen creates the member
under the organization's default team, on the organization's default
budget, and serves the request. Without an alias, or when the
organization has no default budget, the member stays unknown and the
request is still refused.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): declare the member registrar callable as a TypeAlias
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): enforce customer org, team and member budgets and limits on the request path
Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): customer hierarchy survives virtual key auth without a legacy end user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auth): keys pinned to a customer organization may only act for that organization's members
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): keep the customer hierarchy spend maps within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): check the key's customer org scope after the key is resolved
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the hierarchy cache key family and keep type counts flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): resolve the customer hierarchy in the shared authorization gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop a member's cached hierarchy through the declared cache target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(auth): keep the hierarchy reset within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): type the requested models as a sequence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend): add daily customer usage and budget resets (#51493)
* feat(spend): add daily customer usage and budget resets
* chore(spend): document customer reset query bounds
* fix(spend): chart customer queue metrics and update recovery test
* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit ccac47608f.
* fix(proxy): evict cached member hierarchies when team, org or template budgets change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the auth cache dependency alias as a TypeAlias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit 7542471e17.
* fix(spend): reconcile the daily customer spend table with main
Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): tolerate failure metadata without customer hierarchy ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(accounting): key daily customer spend rows on main's hashed identity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): read customer ids from failure metadata through a typed lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call
deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss
tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left
* fix(proxy): refuse blocked and over-limit keys before reading the request body
A client retrying 429s with multi-megabyte payloads made every worker buffer each payload in full before the refusal went out, so the rejections themselves drove memory. Auth now checks the cached key for a block or an exhausted key-level rate limit from the headers alone and answers before the body is read; admitted requests and cache misses take the unchanged path.
* fix(proxy): pick the effective key header for the early refusal and keep its failure log
The headers-only check now reads x-litellm-api-key when present, skips a missing Authorization header instead of raising, stamps the route on the cached identity so the failure is logged, and honors an exception a callback returns from the failure hook.
* fix(proxy): honor litellm_key_header_name in the early refusal
The headers-only check now reads the key from the header general_settings.litellm_key_header_name names when the request carries it, the same precedence the full auth path applies.
* fix(proxy): keep the early-rejection path inside the type gate ceilings
Read the configured key header through general_settings_view() instead of importing the untyped
general_settings global, and drop the isinstance on an exception post_call_failure_hook already types as
HTTPException.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): allow paid video retrieval after budget exhaustion
* test(auth): check video retrieval boundary on registered route
* fix(auth): exempt video retrieval in post-custom-auth checks
* style(auth): format custom video budget exemption
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): list ids that route back for a prefixed partial alias over a catch-all deployment
* test(proxy): call the router registry in the catch-all alias listing test
* fix(proxy): keep main's listing for whole-name aliases over a catch-all
Only partial aliases like team1/gpt-* over a bare * run the per-id route check. A whole-name alias like team1/* keeps the string-only listing it had, so it lists the same ids at the same cost. Adds a test that a more specific sibling alias's ids are left out of the partial alias's list.
* fix(proxy): list catch-all aliases fast and skip ids a sibling alias takes
* fix(proxy): check team wildcard aliases without a router call per id and skip suffix ids that name another catalog model
Team keys now read the team's own wildcard patterns the way the router does (any team pattern first, then the provider form when its prefix can be a provider), so ids no rival can take skip the router call. A suffix alias such as team1/*-mini no longer lists team1/gpt-5.4-mini for gpt-5.4.
* fix(proxy): keep team key listings fast when aliases overlap and honor a pinned provider on partial aliases
A team key whose team defines the same alias as the global config, or a team
pattern that can take a global alias's ids, now checks only the team's own
patterns before calling the router, so the listing no longer pays a router
call per catalog id. The router also tries the provider-prefixed form of an id
only when the bare form matches nothing.
A partial alias over `*` with `custom_llm_provider` set now lists only that
provider's catalog, so `team1/gpt-*` pinned to OpenAI no longer lists
`team1/gpt-claude-*` ids that answer 404.
* fix(proxy): ask a pinned catch-all's provider for its models once per listing
A partial alias over a `*` deployment pinned with custom_llm_provider fetched the whole `*` catalog before taking the pinned branch, so with check_provider_endpoint on each listing called the provider's /v1/models one extra time. The pinned branch now runs before that fetch. Tests also cover a provider-less team alias that takes ids under their provider and a wildcard model_group_alias
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>