* fix(proxy): stop listing catalog ids under an org the catalog lacks for every provider whose path is its model id
* fix(proxy): keep listing Fireworks router and Azure model_router wildcards
The catalog cannot price the names these routes call, so the strict org check dropped callable ids. They now list as before, and every exempt provider has a test row that fails if it leaves the list
* fix(proxy): check unknown orgs beneath router routes and keep firerouter pools
* fix(proxy): keep listing Fireworks routers/firerouter pools and Azure AI speech wildcards
* fix(proxy): check unknown orgs under OpenAI and Cohere and drop unreachable Fireworks router ids
* fix(proxy): keep the full wildcard listing for deployments behind a custom api_base
* fix(proxy): count a provider base URL set in the env or litellm.api_base as a custom endpoint
* fix(proxy): match a gateway env var on any part of the provider name
* fix(proxy): count a provider's off-pattern base URL env vars as a gateway
* test(proxy): cover every off-pattern base URL env var in the gateway table
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(github_copilot): support GitHub App installation authentication
* Add GitHub App authentication to the Copilot credential form
* Explain secure private-key setup beside Copilot App fields
* Store uploaded Copilot App keys through encrypted named credentials
* fix(github_copilot): handle App validation errors and credential replacement
* fix(ui): make GitHub App field help accessible
* fix(github_copilot): preserve saved App key references on edit
* fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health
* fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses
Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers.
App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares.
A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change.
App deployments are deployment-owned, so they share the response cache like shared device-login deployments.
App credentials are validated even when credential_info names no provider.
* fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled
The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future
* fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs
A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again
* fix(github_copilot): report a deployment naming an unloaded credential as unhealthy
The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login
* fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base
A Copilot deployment that names an unloaded credential uses the shared device login again, as on base.
A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call.
get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores.
* fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure
* fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403
Only an App deployment reports its App host as api_base before a configured api_base.
Per-user OAuth sessions report the same api_base they did on base.
The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication.
The token-dir fixture in the App api_base test is applied with usefixtures.
* fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete
* test(github_copilot): type the incomplete App settings test parameter
---------
Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions
* fix(proxy): check the key's max_budget on /queue/chat/completions
* fix(proxy): run budget, tag, and request policy checks on queued chat calls
* fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls
A queued stream now hands its logging object to the stream generator, so a
client that leaves mid-stream is billed for the reply it already got, and a
stream left before the first model chunk releases its budget reservation.
A team's require_trace_id and the failure logging for rejected calls now
use the same route check as the other inference checks.
* fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses
* fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls
* fix(proxy): send the call ID on a queued call the provider rejects before streaming
* fix(proxy): release a dropped stream's budget hold only when nothing bills it
A guardrail that buffers the stream can pull model chunks the client never
saw. The early-drop release settled the hold at the input cost first, then
the partial bill for those chunks added its full cost on top. The release
now runs after partial billing and only when no bill fired
* fix(proxy): fold the caller's litellm_metadata into a queued call's metadata
A queued call that sent a litellm_metadata bucket, even an empty one, made tag
routing and logging read that bucket, so the key and team tags the handler
merged into metadata no longer narrowed deployments. The queue handler now
folds the bucket into metadata and drops it, as chat does, after stripping the
proxy-owned user_api_key_* and internal control fields from both buckets, so a
caller can no longer set the queued spend row's organization ID either
* fix(proxy): release a queued call's budget hold when the call is cancelled before it answers
* fix(proxy): refund a cancelled queued call's hold in full until the model call claims it
* fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): support internal user budget windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make user budget migration repeatable and sync API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound lock wait for user budget migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit unset JSON windows when creating users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): retain personal budget windows for proxy admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): give user budget windows their own counters and keep them on the user
User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path
* fix(proxy): keep windows on team auto keys and write all users' windows in one transaction
* fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock
An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets.
JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do
* fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock
Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses
The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits
* test(proxy): check default window resets against the reset helper instead of the wall clock
* fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes
* fix(proxy): fail an all-users bulk update whole when its transaction fails
The per-user fallback after a failed all-users transaction wrote the newest
user and then crashed on a missing user_email attribute, so the call answered
500 with one user changed. Racing window bulk updates past the 5s transaction
limit reached it. Answer 500 with no user changed instead
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs
Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.
* fix(proxy): type the spend logs metadata header parser honestly
The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.
* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket
A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.
* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler
Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.
* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer
The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.
* test(proxy): read auth failure spend rows from a real unconnected PrismaClient
Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix: inherit team models for dashboard sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: authorize dashboard models from owner grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve dashboard model listing filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): attribute dashboard team model spend to granting teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Keep dashboard fallbacks on the original billing team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve admitted dashboard billing through primary model rewrites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reject dashboard batches that need team model grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Respect wildcard exclusions in dashboard model listings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias
* test(proxy): a pinned key naming an unknown org alias is refused with 403 even when registration would fail
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): register a customer member in a customer org with no default budget
A request that named a customer org and a customer member the proxy had never
seen created the customer org and charged it, then dropped the customer member
because a new customer org has no default budget. The customer member is now
registered in the default customer team with no budget, so its spend is tracked
from the first request and nothing caps it until an admin assigns one. Admin
creates still require a budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): write only the budget fields a customer member patch names
An alias edit wrote back the budget fields it read, so a budget assigned
between its read and its write was erased. The update now leaves the budget
columns alone unless the patch names them.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset customer org, team and member spend on their budget row's window
The customer tables carried their own budget_reset_at next to a budget_id whose
row already has one. Drop the column from the three tables and let the budget
table cascade zero the customer rows it governs, with a member following its
unexpired override when that override has a duration and its template
otherwise. Customer budget writes stamp budget_reset_at from budget_duration the
way /budget/new does, and the management responses report the governing budget's
window under the same budget_reset_at field.
* feat(budgets): support concurrent customer budget windows
* fix(budgets): resolve compatibility and validation checks
* feat(proxy): name budget rows, give customer teams an override and members a parent org
* fix(budgets): preserve admitted windows during accounting
* chore(schema): allow dropping the unreleased customer budget_reset_at columns
* fix(budgets): apply standard migration lock timeout
* feat(proxy): customer members belong to many teams, charged to the one the request names
* test(migrations): allow the unreleased customer budget_reset_at drop
* style(ui): prettier the customer dashboard files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(reset-job): name the batch bound on the customer budget IN filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(budgets): preserve mixed limits and resolve CI regressions
* fix(budgets): reuse shared spend verification for customer windows
* refactor(ui): isolate customer budget group creation
Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild.
* fix(proxy): restore Field import and bound the member team lookup after the merge
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): send parent_team_ids in the budget window customer tests
Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): keep rejecting another organization's template after the budget_template_ids rename
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): cover customer budget increase requests and the cap_policy field
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(ui): regenerate API types for cap_policy and customer budget increase requests
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): hang the increase request back-reference on the customer member model
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): declare increase request list filters with Annotated Query
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(interactions): route Vertex interaction follow-ups to the selected deployment
GET, cancel and DELETE by interaction id accept a model group or a deployment
id through the model query parameter or the x-litellm-model header, authorize
it against the key and team like any model, and run with that deployment's
project and credentials. Without a selector they still go to Google AI Studio.
Vertex cancel now uses the documented /cancel path.
* test(bedrock): pin the SigV4 clock in the embedding signature test
SigV4Auth.add_auth stamps the request with botocore.auth.get_current_datetime(), overwriting the timestamp the test set on the replay. When the proxy call and the replay landed in different seconds, the signatures differed. Fix the module clock for both signings and drop the ineffective manual timestamp.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(proxy): name the customer organization by alias with x-litellm-customer-org-alias
A request may carry x-litellm-customer-org-alias next to, or instead of,
x-litellm-customer-member-id. Alone, the alias charges and enforces the
organization level only. Next to the member header it must name the
member's organization, else 403. Unknown alias is 401, an alias shared
by several organizations is 400. Organization alias changes evict the
cached lookup.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for optional customer team and member ids
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): register unknown customer members in the aliased organization's default team
A request carrying x-litellm-customer-org-alias and an
x-litellm-customer-member-id the proxy has never seen creates the member
under the organization's default team, on the organization's default
budget, and serves the request. Without an alias, or when the
organization has no default budget, the member stays unknown and the
request is still refused.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): declare the member registrar callable as a TypeAlias
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): enforce customer org, team and member budgets and limits on the request path
Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): customer hierarchy survives virtual key auth without a legacy end user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auth): keys pinned to a customer organization may only act for that organization's members
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): keep the customer hierarchy spend maps within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): check the key's customer org scope after the key is resolved
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the hierarchy cache key family and keep type counts flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): resolve the customer hierarchy in the shared authorization gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop a member's cached hierarchy through the declared cache target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(auth): keep the hierarchy reset within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): type the requested models as a sequence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend): add daily customer usage and budget resets (#51493)
* feat(spend): add daily customer usage and budget resets
* chore(spend): document customer reset query bounds
* fix(spend): chart customer queue metrics and update recovery test
* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit ccac47608f.
* fix(proxy): evict cached member hierarchies when team, org or template budgets change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the auth cache dependency alias as a TypeAlias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit 7542471e17.
* fix(spend): reconcile the daily customer spend table with main
Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): tolerate failure metadata without customer hierarchy ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(accounting): key daily customer spend rows on main's hashed identity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): read customer ids from failure metadata through a typed lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call
deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss
tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left
* fix(proxy): refuse blocked and over-limit keys before reading the request body
A client retrying 429s with multi-megabyte payloads made every worker buffer each payload in full before the refusal went out, so the rejections themselves drove memory. Auth now checks the cached key for a block or an exhausted key-level rate limit from the headers alone and answers before the body is read; admitted requests and cache misses take the unchanged path.
* fix(proxy): pick the effective key header for the early refusal and keep its failure log
The headers-only check now reads x-litellm-api-key when present, skips a missing Authorization header instead of raising, stamps the route on the cached identity so the failure is logged, and honors an exception a callback returns from the failure hook.
* fix(proxy): honor litellm_key_header_name in the early refusal
The headers-only check now reads the key from the header general_settings.litellm_key_header_name names when the request carries it, the same precedence the full auth path applies.
* fix(proxy): keep the early-rejection path inside the type gate ceilings
Read the configured key header through general_settings_view() instead of importing the untyped
general_settings global, and drop the isinstance on an exception post_call_failure_hook already types as
HTTPException.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): allow paid video retrieval after budget exhaustion
* test(auth): check video retrieval boundary on registered route
* fix(auth): exempt video retrieval in post-custom-auth checks
* style(auth): format custom video budget exemption
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): list ids that route back for a prefixed partial alias over a catch-all deployment
* test(proxy): call the router registry in the catch-all alias listing test
* fix(proxy): keep main's listing for whole-name aliases over a catch-all
Only partial aliases like team1/gpt-* over a bare * run the per-id route check. A whole-name alias like team1/* keeps the string-only listing it had, so it lists the same ids at the same cost. Adds a test that a more specific sibling alias's ids are left out of the partial alias's list.
* fix(proxy): list catch-all aliases fast and skip ids a sibling alias takes
* fix(proxy): check team wildcard aliases without a router call per id and skip suffix ids that name another catalog model
Team keys now read the team's own wildcard patterns the way the router does (any team pattern first, then the provider form when its prefix can be a provider), so ids no rival can take skip the router call. A suffix alias such as team1/*-mini no longer lists team1/gpt-5.4-mini for gpt-5.4.
* fix(proxy): keep team key listings fast when aliases overlap and honor a pinned provider on partial aliases
A team key whose team defines the same alias as the global config, or a team
pattern that can take a global alias's ids, now checks only the team's own
patterns before calling the router, so the listing no longer pays a router
call per catalog id. The router also tries the provider-prefixed form of an id
only when the bare form matches nothing.
A partial alias over `*` with `custom_llm_provider` set now lists only that
provider's catalog, so `team1/gpt-*` pinned to OpenAI no longer lists
`team1/gpt-claude-*` ids that answer 404.
* fix(proxy): ask a pinned catch-all's provider for its models once per listing
A partial alias over a `*` deployment pinned with custom_llm_provider fetched the whole `*` catalog before taking the pinned branch, so with check_provider_endpoint on each listing called the provider's /v1/models one extra time. The pinned branch now runs before that fetch. Tests also cover a provider-less team alias that takes ids under their provider and a wildcard model_group_alias
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): judge the zero-cost budget waiver by the model a key or team alias serves
The zero-cost budget waiver judged the name the client sent, but key `aliases` and team
`model_aliases` are applied later during request preparation. An over-budget key got a 422
for an alias of a $0 model, and an alias named like a $0 group that points at a paid model
was waived and served past the budget.
For a string body `model` on chat, completions, responses and messages, the key-path,
centralized and custom-auth waivers now judge the model request preparation serves. That
model comes from the same team, key, global and router alias functions that request
preparation uses, so the two cannot drift. Other routes and the JWT path keep judging the
requested model. common_checks no longer overrides a caller's explicit decision not to skip
budgets, and proxy-made sub-calls are judged by their own model.
Resolves LIT-10283
* fix(proxy): waive only once auth has applied the router aliases, and keep empty team alias targets
The served-model waiver now applies only after auth's router-alias rewrite has run on the
request body. Custom auth without common checks runs its waiver before that rewrite, so it
keeps judging the requested model instead of a key or team alias of a name the router
rewrites elsewhere. The waiver also keeps an empty-string team alias target, matching
request preparation.
Resolves LIT-10283
* fix(proxy): waive by the served model only for a client's own body-model request
The waiver resolved aliases for any request on a chat, completions, responses or messages
path, including the HTTP request the Responses WebSocket builds for first-frame admission.
That session dispatches the model as an endpoint model, which skips the team alias, so a
team alias of a paid model to a free one waived the paid model's budget.
The waiver now resolves aliases only when FastAPI matched the request to one of the body
model routes, which requests built inside the proxy and path model routes never are. The
waiver tests are folded into one table, share a priced router fixture, and cover the
stale team alias flag cache.
Resolves LIT-10283
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(router): re-check and reserve per-model rate limits on fallback targets
A provider error fallback handed the caller the fallback model group without
its per-model TPM/RPM limits ever being checked. The router gets a
fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires
it to the v3 limiter, which reserves the target model's per-model descriptors
before each fallback hop and skips targets that are over their limit.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): dedupe a repeated fallback target before the reserving rate limit check
Both walks now consult the attempted set before the rate limit check, so a
chain like ["B", "B"] reserves B once. The deprecated settings parity test
also learns the new setting.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): a bidirectional fallback chain never re-reserves the requested model group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): match the surfaced failure in the fallback reservation tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): keep the admission stash untouched while building fallback descriptors
Building the fallback target's descriptors could rewrite the request stash's
PTU ceiling, which post-call settlement reads for the model the caller named.
The fallback check now hides the stash while it builds, like the frame check.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The PATCH model update endpoint was missing from the management_routes group, so virtual keys restricted to management_routes got a 403 on it while the legacy POST /model/update worked. Fixes#50918.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): reject client-supplied Vertex location and project unless the admin opts in
A request body could carry vertex_location, vertex_project, or their
vertex_ai_* aliases and override the region and project pinned on a
Vertex AI deployment, sending the admin's credentials to a caller-chosen
project or region. Add the four fields to the banned request params so
they need allow_client_side_credentials or configurable_clientside_auth_params.
/vertex_ai/live and /rag/ingest name their Vertex project and location
by design and have no deployment pin to override, so they keep those two
fields while every other banned param is still rejected there
* fix(proxy): close the remaining Vertex target overrides and keep inert fields working
/vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in
A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL
Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target
* test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads
* fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest
* fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests
* test(proxy): give the batch tests' fake router the alias and routed-group lookups
* fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review
A Vertex opt-in now counts only when every group the request can reach opts in, router,
context window, and content policy fallbacks and one alias hop included, while a group
with no Google deployment never needs the fields. A model named in the route path is
checked next to the body model.
RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a
vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin
saved before the upgrade, and reads a corpus named by project number on the proxy's
own project.
/vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves
a setup frame sent as a binary message onto the session's target, and resolves the
region from default_vertex_config before the model path.
* fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one
With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project,
including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those
projects now connects, and any other project still closes with 1008 before a token is minted.
* fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest
A non-admin ingest into a store an admin created named by project number had its corpus moved onto
the proxy's own project before the admin-store exemption ran, so the upload went to a different
corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the
project-number rewrite and the target check.
* fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request
* fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list
* fix(proxy): check team public models before allowing target overrides
* fix(proxy): include team wildcard targets in opt-in checks
* fix(proxy): validate overrides against authenticated team routes
* fix(proxy): retain URL model during deferred credential checks
* feat(proxy): let teams send allowed S3 buckets for batch files and read them back
A team's metadata can now list allowed_s3_buckets. Keys on that team may send one
of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on
POST /v1/batches, and the upload and batch-create handlers check the bucket against
the team list plus general_settings.allowed_client_side_s3_buckets. The early auth
body check still rejects bucket fields on every other route and method
Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri
through an internal stored_s3_bucket_name param, so content, retrieve, delete and the
batch cost job can read files written to a request-chosen bucket. Callers can never
send that param, and files in the model bucket resolve exactly as before
* test(proxy): assert outcomes in team bucket tests and type the managed content read
Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count
* fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back
A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix
* fix(proxy): keep the request's approved S3 bucket when merging deployment credentials
Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none
* fix(proxy): keep the request bucket only on upload and batch create, not on reads
The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for
* fix(proxy): only proxy admins can set a team's allowed_s3_buckets
Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes
* feat(budgets): add per-model tpm and rpm limit maps to budgets and customers
Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more).
/budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth
path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and
end_user_model_rpm_limit so the limiter can enforce them in a follow-up.
* fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes
Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata,
which the limiter reads, so the shared budget allowlist leaves those two fields to customers.
Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy
the type discipline gate. Customer info tests expect the two new budget keys.
* fix(organizations): keep stripping budget_id and flattening nested per-model maps on update
The update path pops every budget field from the organization row again, so a caller's budget_id
never changes the attached budget, and a nested litellm_budget_table still flattens the two maps
into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS.
* fix(types): keep basedpyright and ruff strict counts flat for the new budget fields
Reads the per-model maps off the end-user params with membership checks instead of .get on an
untyped dict, writes the inline customer budget through the client's typed jsonify_object, and
builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never
receive object-typed kwargs.
* chore: drop unrelated reformatting from project endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): log keys and teams still on the deprecated empty models defaults at boot
* fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup
* fix(proxy): say which empty-models rows keep access in 2.0.0
Teamless keys listing all-team-models get no models in 2.0.0, so the
boot warning no longer says every counted row keeps its access. It
names each upgrade outcome only when its kind is in use, says the counts
include blocked and expired rows, and words each kind by what the
setting means rather than what it grants
* fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning
The warning no longer says the upgrade keeps access for empty models
lists, since no shipped code or public page backs that yet. It says
2.0.0 removes the defaults and how to keep today's access, and keeps
the advice for teamless all-team-models keys. The integration test
drops an empty-database assertion that ran before the check finished
* fix(proxy): leave rows with access groups out of the empty models boot warning
* fix(proxy): count access group rows, skip management and read-only keys in the empty models warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): read an absent team membership row as zero spend under fail-closed budget enforcement
A team user under a default member budget has no LiteLLM_TeamMembership row until spend is persisted. SpendCounterReseed.from_db returned None for that absent row, the same value as a failed DB read, so fail_closed_budget_enforcement rejected every request from that user with 503. The DB answering that the row is absent is a verified zero, so from_db now returns 0.0 for a team-member counter without a row and still returns None when the read fails.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only read an absent membership row as zero when the team member key parses unambiguously
A team_id or user_id containing ":" makes the counter key split onto the wrong membership row, so that miss keeps returning None instead of a verified zero. Type the new unit tests.
* fix(proxy): refuse a fail-closed team member with no membership row with a 403 that names /team/member_add
Replaces the earlier reading of an absent LiteLLM_TeamMembership row as verified zero spend. A key minted with team_id and user_id but no /team/member_add has no row and the spend writer never creates one, so admitting it at zero would leave its member spend tracked only in Redis. With fail_closed_budget_enforcement on, _check_team_member_budget now rejects that key with a 403 that names the user, the team and the member_add call, before any spend read. With the flag off nothing changes. The membership load raises on a database failure and returns None only for a genuinely absent row, so the 403 never fires during an outage. spend_counter_reseed.py and the counter prefix constant go back to main.
* test(proxy): drive the no-membership 403 tests through the real membership and spend readers
Fake Postgres rows and the FakeSpendCounters double stand in for the external stores instead of patching get_team_membership and get_current_spend. Strip CR/LF from the ids in the fail-closed warning log.
* fix(proxy): read fail_closed_budget_enforcement from general_settings in the membership check
Importing proxy_server's private helper grew reportPrivateUsage past the type gate ceiling. Type the integration helper's return as JsonValue.
* fix(proxy): expose fail_closed_budget_enforcement_enabled for the membership check
Reading general_settings directly from auth_checks grew the unknown-type counts past the type gate ceiling; a small public wrapper in proxy_server keeps the gate flat.
* test(proxy): drop docstrings that restate the no-membership tests
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auth): grant access groups through principal attribute rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): satisfy cache tracing and recursion CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): close access group rule authorization bypasses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): reject self-editable identity rule attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Centralize JWT model authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Cover centralized key checks and name denied JWT models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Test centralized authorization for custom auth and JWT mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve model normalization before budget validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use a controlled clock for WebSocket session deadline tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Unify JWT credential admission and simplify team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Simplify request-time access group authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use real requests in custom auth budget fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL
* fix(prometheus): carry live spend for the user, org and customer budget gauges too
The remaining budget gauges for users, orgs and customers read the same cached rows as the key and
team gauges, so they read high for about a minute after a charge. Auth now carries the live counter
for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded.
The x-litellm-key-spend header and logging spend fields keep their current values.
* fix(prometheus): show committed spend in the remaining budget gauges
Reads twin counters that take only charged cost, so a worst-case budget hold
never shows as spent and a request that finishes last cannot set a gauge back.
The twins exist only while the Prometheus logger is registered, and every path
that resets a spend counter drops them. Failed requests now set the org gauge
from the carried org row and committed spend, not the cached org row.
* fix(prometheus): count a charge the cached customer row already holds once
* fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets
Auth now reads committed spend through the admission MGET and names the counters a charge
may move, so the master key and unbudgeted entities get no twin. Organization spend edits
and the global spend reset drop their twins, and a long request no longer reads as charged.
* fix(prometheus): read the committed org spend on failed requests without the auth object
* test(prometheus): cover a failed request with no carried org
* fix(prometheus): move the committed counters for charges with no auth on the row
A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them.
* fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes
Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a
charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed
counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys
and teams in bounded pages.
* fix(prometheus): count a charged failure's own cost in the org gauge
A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request
* fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog
* fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete
Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins
* fix(prometheus): keep committed spend values out of logger dumps of pass-through rows
CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values
* fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster
The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files
* fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): discover upstream LiteLLM Gateway wildcard models
* fix(proxy): use the typed shared client for gateway discovery
* fix(ui): show the selected LiteLLM Gateway logo
* feat(proxy): hash UI passwords with PBKDF2 in FIPS mode
A FIPS OpenSSL refuses scrypt, so every password write fails there and
nobody can sign in with a password. When OpenSSL enforces FIPS, passwords
now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's
DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is
refused with one error log and left as it is, and the boot migration does
nothing, because FIPS mode is for new deployments.
PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the
five request paths run it on one thread instead of the event loop. A single
thread keeps a login burst from using up a CPU quota, which is what stalls
the loop with asyncio.to_thread. A login that queued for that thread asks the
throttle again before it answers, so a burst of guesses can't get past a block
that the failures ahead of it just set.
The breached password check is off in FIPS mode, because it runs the
password through SHA-1.
A standard OpenSSL reads False at every new branch, so standard mode keeps
scrypt, inline, on the same rows.
Resolves LIT-8425
* fix(proxy): take FIPS logins one at a time and recheck the throttle first
A login that waited for the KDF thread asked the throttle again only after
deriving, so a burst for one user kept that thread busy long after the user
was blocked. Logins now wait for their turn on a lock and ask the throttle
when it comes, so a blocked login costs no PBKDF2 and only one login at a time
holds a place in the KDF queue. Password changes and invite claims no longer
wait behind a whole burst of logins.
* fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check
A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the
real password stopped working and the old row text signed in. The migration now
leaves a row of that exact shape alone. Any other pbkdf2: text still migrates.
FIPS mode now logs a warning when it skips a breach check that is turned on.
The one-thread test no longer depends on timing, and the new test variables are
annotated Final.
* fix(proxy): keep the breached password check on in FIPS mode
* fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL
* test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt
* feat(redis): route the router, rate limiters and auth through the native store families
* style(redis): satisfy the python lint gates
Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): give the registry's default usage store a memory-only handle
RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): build stores from native handles and wait for router tasks
Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise
Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the
value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way.
s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls
* refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map
* fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request
