136 commits

Author	SHA1	Message	Date
mateo-berri	f296c26d4e	Merge remote-tracking branch 'origin/main' into litellm_chatgpt_team_subscriptions ... # Conflicts: # litellm/router_python_core.py # tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py # tests/unit/proxy/credential_endpoints/test_endpoints.py # tests/unit/test_router/test_router_helper_utils.py # tests/unit/test_utils.py # ui/litellm-dashboard/src/components/model_add/CredentialModal.tsx # ui/litellm-dashboard/src/components/model_add/CredentialsPanel.tsx	2026-11-10 21:08:00 -08:00
joshua-berri	8092edc7a1	fix(proxy): judge the zero-cost budget waiver by the model a key or team alias serves (#51058) ... * fix(proxy): judge the zero-cost budget waiver by the model a key or team alias serves The zero-cost budget waiver judged the name the client sent, but key `aliases` and team `model_aliases` are applied later during request preparation. An over-budget key got a 422 for an alias of a $0 model, and an alias named like a $0 group that points at a paid model was waived and served past the budget. For a string body `model` on chat, completions, responses and messages, the key-path, centralized and custom-auth waivers now judge the model request preparation serves. That model comes from the same team, key, global and router alias functions that request preparation uses, so the two cannot drift. Other routes and the JWT path keep judging the requested model. common_checks no longer overrides a caller's explicit decision not to skip budgets, and proxy-made sub-calls are judged by their own model. Resolves LIT-10283 * fix(proxy): waive only once auth has applied the router aliases, and keep empty team alias targets The served-model waiver now applies only after auth's router-alias rewrite has run on the request body. Custom auth without common checks runs its waiver before that rewrite, so it keeps judging the requested model instead of a key or team alias of a name the router rewrites elsewhere. The waiver also keeps an empty-string team alias target, matching request preparation. Resolves LIT-10283 * fix(proxy): waive by the served model only for a client's own body-model request The waiver resolved aliases for any request on a chat, completions, responses or messages path, including the HTTP request the Responses WebSocket builds for first-frame admission. That session dispatches the model as an endpoint model, which skips the team alias, so a team alias of a paid model to a free one waived the paid model's budget. The waiver now resolves aliases only when FastAPI matched the request to one of the body model routes, which requests built inside the proxy and path model routes never are. The waiver tests are folded into one table, share a priced router fixture, and cover the stale team alias flag cache. Resolves LIT-10283 --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-11-10 18:28:49 -08:00
Kerry Lu	e2625d0509	fix(router): re-check and reserve per-model rate limits on fallback targets (#50994) ... * fix(router): re-check and reserve per-model rate limits on fallback targets A provider error fallback handed the caller the fallback model group without its per-model TPM/RPM limits ever being checked. The router gets a fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires it to the v3 limiter, which reserves the target model's per-model descriptors before each fallback hop and skips targets that are over their limit. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): dedupe a repeated fallback target before the reserving rate limit check Both walks now consult the attempted set before the rate limit check, so a chain like ["B", "B"] reserves B once. The deprecated settings parity test also learns the new setting. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): a bidirectional fallback chain never re-reserves the requested model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): match the surfaced failure in the fallback reservation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the admission stash untouched while building fallback descriptors Building the fallback target's descriptors could rewrite the request stash's PTU ceiling, which post-call settlement reads for the model the caller named. The fallback check now hides the stash while it builds, like the frame check. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 16:36:10 -08:00
Nurlan Mammadli	285ddf82e1	fix(proxy): allow management_routes keys to call PATCH /model/{model_id}/update (#50979) ... The PATCH model update endpoint was missing from the management_routes group, so virtual keys restricted to management_routes got a 403 on it while the legacy POST /model/update worked. Fixes #50918. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 16:32:49 -08:00
devin-ai-integration[bot]	0fd6dd4168	fix(proxy): recheck missing users when resolving session grants (#51030) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 21:55:53 +00:00
Mateo Wang	4867a56e9a	fix(proxy)!: reject client-supplied Vertex location and project unless the admin opts in (#47028) ... * fix(proxy): reject client-supplied Vertex location and project unless the admin opts in A request body could carry vertex_location, vertex_project, or their vertex_ai_* aliases and override the region and project pinned on a Vertex AI deployment, sending the admin's credentials to a caller-chosen project or region. Add the four fields to the banned request params so they need allow_client_side_credentials or configurable_clientside_auth_params. /vertex_ai/live and /rag/ingest name their Vertex project and location by design and have no deployment pin to override, so they keep those two fields while every other banned param is still rejected there * fix(proxy): close the remaining Vertex target overrides and keep inert fields working /vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target * test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads * fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest * fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests * test(proxy): give the batch tests' fake router the alias and routed-group lookups * fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review A Vertex opt-in now counts only when every group the request can reach opts in, router, context window, and content policy fallbacks and one alias hop included, while a group with no Google deployment never needs the fields. A model named in the route path is checked next to the body model. RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin saved before the upgrade, and reads a corpus named by project number on the proxy's own project. /vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves a setup frame sent as a binary message onto the session's target, and resolves the region from default_vertex_config before the model path. * fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project, including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those projects now connects, and any other project still closes with 1008 before a token is minted. * fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest A non-admin ingest into a store an admin created named by project number had its corpus moved onto the proxy's own project before the admin-store exemption ran, so the upload went to a different corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the project-number rewrite and the target check. * fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request * fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list * fix(proxy): check team public models before allowing target overrides * fix(proxy): include team wildcard targets in opt-in checks * fix(proxy): validate overrides against authenticated team routes * fix(proxy): retain URL model during deferred credential checks	2026-11-10 20:50:17 +00:00
mrinal-berri	9ba551b871	feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571) ... * feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes	2026-11-10 11:06:17 -08:00
Kerry Lu	64d84e557e	feat(budgets): add per-model tpm and rpm limit maps to budgets and customers (#50797) ... * feat(budgets): add per-model tpm and rpm limit maps to budgets and customers Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more). /budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and end_user_model_rpm_limit so the limiter can enforce them in a follow-up. * fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata, which the limiter reads, so the shared budget allowlist leaves those two fields to customers. Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy the type discipline gate. Customer info tests expect the two new budget keys. * fix(organizations): keep stripping budget_id and flattening nested per-model maps on update The update path pops every budget field from the organization row again, so a caller's budget_id never changes the attached budget, and a nested litellm_budget_table still flattens the two maps into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS. * fix(types): keep basedpyright and ruff strict counts flat for the new budget fields Reads the per-model maps off the end-user params with membership checks instead of .get on an untyped dict, writes the inline customer budget through the client's typed jsonify_object, and builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never receive object-typed kwargs. * chore: drop unrelated reformatting from project endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 22:15:18 -08:00
paveln83	69b0f1deac	feat(proxy): log keys and teams still on the deprecated empty models defaults at boot (#50720) ... * feat(proxy): log keys and teams still on the deprecated empty models defaults at boot * fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup * fix(proxy): say which empty-models rows keep access in 2.0.0 Teamless keys listing all-team-models get no models in 2.0.0, so the boot warning no longer says every counted row keeps its access. It names each upgrade outcome only when its kind is in use, says the counts include blocked and expired rows, and words each kind by what the setting means rather than what it grants * fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning The warning no longer says the upgrade keeps access for empty models lists, since no shipped code or public page backs that yet. It says 2.0.0 removes the defaults and how to keep today's access, and keeps the advice for teamless all-team-models keys. The integration test drops an empty-database assertion that ran before the check finished * fix(proxy): leave rows with access groups out of the empty models boot warning * fix(proxy): count access group rows, skip management and read-only keys in the empty models warning --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 04:05:25 +00:00
Kerry Lu	92c2e69ac5	fix(proxy): fail-closed budget enforcement returns 403 naming /team/member_add for a team user with no membership row (#50603) ... * fix(proxy): read an absent team membership row as zero spend under fail-closed budget enforcement A team user under a default member budget has no LiteLLM_TeamMembership row until spend is persisted. SpendCounterReseed.from_db returned None for that absent row, the same value as a failed DB read, so fail_closed_budget_enforcement rejected every request from that user with 503. The DB answering that the row is absent is a verified zero, so from_db now returns 0.0 for a team-member counter without a row and still returns None when the read fails. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only read an absent membership row as zero when the team member key parses unambiguously A team_id or user_id containing ":" makes the counter key split onto the wrong membership row, so that miss keeps returning None instead of a verified zero. Type the new unit tests. * fix(proxy): refuse a fail-closed team member with no membership row with a 403 that names /team/member_add Replaces the earlier reading of an absent LiteLLM_TeamMembership row as verified zero spend. A key minted with team_id and user_id but no /team/member_add has no row and the spend writer never creates one, so admitting it at zero would leave its member spend tracked only in Redis. With fail_closed_budget_enforcement on, _check_team_member_budget now rejects that key with a 403 that names the user, the team and the member_add call, before any spend read. With the flag off nothing changes. The membership load raises on a database failure and returns None only for a genuinely absent row, so the 403 never fires during an outage. spend_counter_reseed.py and the counter prefix constant go back to main. * test(proxy): drive the no-membership 403 tests through the real membership and spend readers Fake Postgres rows and the FakeSpendCounters double stand in for the external stores instead of patching get_team_membership and get_current_spend. Strip CR/LF from the ids in the fail-closed warning log. * fix(proxy): read fail_closed_budget_enforcement from general_settings in the membership check Importing proxy_server's private helper grew reportPrivateUsage past the type gate ceiling. Type the integration helper's return as JsonValue. * fix(proxy): expose fail_closed_budget_enforcement_enabled for the membership check Reading general_settings directly from auth_checks grew the unknown-type counts past the type gate ceiling; a small public wrapper in proxy_server keeps the gate flat. * test(proxy): drop docstrings that restate the no-membership tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 15:38:59 -08:00
Krrish Dholakia	70f399b8c5	feat(auth): grant access groups through principal attribute rules (#50455) ... * feat(auth): grant access groups through principal attribute rules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): satisfy cache tracing and recursion CI checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): close access group rule authorization bypasses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): reject self-editable identity rule attributes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Centralize JWT model authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Cover centralized key checks and name denied JWT models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Test centralized authorization for custom auth and JWT mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Preserve model normalization before budget validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Use a controlled clock for WebSocket session deadline tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Unify JWT credential admission and simplify team selection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Simplify request-time access group authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Use real requests in custom auth budget fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 13:59:40 -08:00
zoveki	abe3b11f7a	fix(proxy): show committed spend in the remaining budget gauges (#49199) ... * fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 20:05:37 +00:00
simonp9	1f15088fc9	feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408) ... * feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes The native Codex catalog now lists the key's other chat and responses models after ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list refreshes on Codex's timer. A request for a model configured by exact name on another provider goes through /v1/responses without the ChatGPT bearer or account header, while ChatGPT names and wildcard-only matches keep the subscription passthrough. Reasoning items another provider produced are dropped before forwarding to ChatGPT. * fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while `codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native /backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no default, so the provider applies its own default exactly as `codex -m` does * fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may call /v1/responses, drop unpersisted reasoning ids on the passthrough when store is not true, keep any slug the caller's ChatGPT account lists on the subscription via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard precedence over a catch-all deployment. * fix(proxy): keep Codex's native route on the subscription when the account list is unknown A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the account's list is known and leaves it out. A failed catalog read is remembered for 60 s and the Codex version comes only from a User-Agent whose product matches the originator header. The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows, and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so Codex stops refetching or dropping its catalog cache. LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge decodes wrapped thinking blocks with their signature. The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the native route checks the chatgpt/ form of a name the key holds only through that group. A none reasoning effort is dropped only under drop_params. The guide covers app tools, every routing case, and the ETag, and the catalog logs how many models the byte limit left out. * fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API * fix(proxy): keep Codex native turns working when the key's model listing fails A listing callback that raises no longer turns chatgpt.com's catalog or responses answer into a 500. The catalog keeps upstream's body and ETag, and responses keep upstream's X-Models-Etag. Under drop_params, a `none` reasoning effort is now dropped only when a candidate deployment does not explicitly support it, so an alias to a model like openai/gpt-5.4 keeps it. The account slug list and its unavailable marker are keyed on the account id plus the caller's ChatGPT Authorization header, so a failed fetch with one bearer cannot turn off LiteLLM routing for another caller on the same account. The auth step checks the Codex route before importing the Codex module. * fix(proxy): keep Codex model switches working across LiteLLM providers On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400 Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows * fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the native route now hoists them without flattening, like OpenAI and Azure and the chat completions bridge. Other native Responses providers still get flat names, now built with the bridge's namespace_tool_name_map, and each function_call they return gets its tool name and namespace back in the non-streaming body and in streamed output_item.added, output_item.done, and completed/incomplete/failed events. Every other event passes through unchanged * fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools Codex sends its freeform exec tool as a custom tool on every turn, and xAI's Responses API answers 422 for custom, namespace and additional_tools shapes. On the Codex native route, a model whose deployments include a native Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now asks for the chat completions bridge with use_chat_completions_api, which already turns custom and namespace tools into functions, hoists additional_tools, and puts the namespace back on each returned call OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock Mantle still get it on their own Responses API with additional_tools hoisted. The route's namespace flattening and the response-side namespace restore are removed, since no path sends flattened names anymore * fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve A reasoning item another deployment produced, replayed with store false and no blob the target can decrypt, now reaches the target as its readable summary without id or status, and is dropped when nothing readable is left. OpenAI answered 400 on the status field and 404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment replays and stored conversations keep the item whole * refactor(codex): reach encrypted reasoning stripping through the router registry The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise every target must decrypt it * fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog * fix(proxy): drop Codex's none effort for an alias whose native entry names no default * fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop * test(proxy): map the pinned reasoning efforts field in the router registry stub * fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases Codex native routes now send a key to LiteLLM only when admin_only_routes allows it, a native alias with configured service tiers starts with no tier picked, and the account slug cache declares its Redis key family. * fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items * fix(proxy): check vector store grants on file_search tools inside additional_tools items A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input. * fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes * fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply * fix(guardrails): check additional_tools items next to top-level function tools in tool_policy ToolPolicyGuardrail read tools from the request body only when no top-level function tool existed, so a blocked tool defined in an additional_tools item reached Fireworks once the Responses transform moved the item into tools after the pre-call hooks. Check every list on each request, including namespace members inside items. * fix(router): serve decryptable_reasoning from the Rust core's optional checks The merge with main removed the Python router core, so the registry's host has no optional_callbacks and no routing context, and native Codex requests with encrypted reasoning raised RustCoreUnsupported. The filter now lives with the optional checks, which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core Also give the static litellm.Router a declared constructor. Since the Python core was deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...) is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change * fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures. * fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias * fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases Auth no longer asks ChatGPT for the account's model list when the bearer is missing or is the LiteLLM key, so a key sent as the bearer never reaches it. tool_policy now reads the members of a top-level namespace, which is where the native route's hoist puts them. A global alias now reads encrypted content affinity from its target group. A non-list top-level tools value and a non-string function name now reach the provider, which refuses them, instead of raising a 500. The catalog ETag no longer depends on the order a process iterates priced service tiers in, so workers and pods agree on it. * fix(proxy): list a team's models in the order the team set them get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker * fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models and /backend-api/codex/responses only when the JWT auth flow would admit a direct /v1/responses call for that model, so team_allowed_routes, admin_allowed_routes and RBAC route limits hold on the handoff too. The Fireworks Responses transform also leaves additional_tools input items in place when the tools param is not a list, instead of raising while hoisting * fix(fireworks): hoist additional_tools when tools is a tuple litellm.responses passes a tuple tools argument through to the Fireworks transform unchanged, so the non-list guard skipped hoisting and left the additional_tools item in the input. Treat tuples like lists and cover both shapes in the drop_params merge test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 11:14:47 -08:00
ishaan-j-berri	1bcd11706b	fix(proxy): discover models for LiteLLM Gateway wildcard providers (#50374) ... * fix(proxy): discover upstream LiteLLM Gateway wildcard models * fix(proxy): use the typed shared client for gateway discovery * fix(ui): show the selected LiteLLM Gateway logo	2026-11-06 09:51:45 +00:00
yucheng-berri	4086d5ccaa	feat(proxy): hash UI passwords with PBKDF2 in FIPS mode (#49874) ... * feat(proxy): hash UI passwords with PBKDF2 in FIPS mode A FIPS OpenSSL refuses scrypt, so every password write fails there and nobody can sign in with a password. When OpenSSL enforces FIPS, passwords now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is refused with one error log and left as it is, and the boot migration does nothing, because FIPS mode is for new deployments. PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the five request paths run it on one thread instead of the event loop. A single thread keeps a login burst from using up a CPU quota, which is what stalls the loop with asyncio.to_thread. A login that queued for that thread asks the throttle again before it answers, so a burst of guesses can't get past a block that the failures ahead of it just set. The breached password check is off in FIPS mode, because it runs the password through SHA-1. A standard OpenSSL reads False at every new branch, so standard mode keeps scrypt, inline, on the same rows. Resolves LIT-8425 * fix(proxy): take FIPS logins one at a time and recheck the throttle first A login that waited for the KDF thread asked the throttle again only after deriving, so a burst for one user kept that thread busy long after the user was blocked. Logins now wait for their turn on a lock and ask the throttle when it comes, so a blocked login costs no PBKDF2 and only one login at a time holds a place in the KDF queue. Password changes and invite claims no longer wait behind a whole burst of logins. * fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the real password stopped working and the old row text signed in. The migration now leaves a row of that exact shape alone. Any other pbkdf2: text still migrates. FIPS mode now logs a warning when it skips a breach check that is turned on. The one-thread test no longer depends on timing, and the new test variables are annotated Final. * fix(proxy): keep the breached password check on in FIPS mode * fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL * test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt	2026-11-06 01:31:59 -08:00
yujonglee	b1b236826a	feat(redis): serve the proxy's common Redis call with Rust stores (#50170) ... * feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 06:01:11 +00:00
ryan-crabbe-berri	48ff826cbc	feat(proxy)!: only let clients pick S3 buckets the admin allowed (#47496) ... * feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way. s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls * refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map * fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request	2026-11-06 02:50:15 +00:00
Nate Armstrong	067f93340f	refactor(router): delete the python router core, the rust core is the only core (#50178)	2026-11-05 10:40:46 -08:00
tin-berri	aa7917a3d2	feat(skills): show download counts and downloader history (#50165)	2026-11-05 05:25:08 -08:00
tin-berri	0ad5d750d4	feat(skills)!: version published Skills Hub updates (#50152)	2026-11-05 00:19:38 -08:00
Nate Armstrong	9752869d84	feat(router): add the Rust router core with the host as its only boundary (#49513) ... Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/), served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in the host-side litellm/router_rust/ package (steps/, invoke/, surface/) driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state in litellm/router_rust/surface/ (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. The crate records each attempt's outcome itself rather than through litellm callbacks. A streamed attempt is deferred: it counts its request when the stream opens, and the host's stream wrappers report the success with the final usage, or a failure after the first chunk, when the stream ends, as the Python core's callbacks do. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores, with nothing pending; only the names of the Python core's own outcome callbacks are asserted on Python alone. litellm.Router is now final: subclassing it raises TypeError at the class statement, and the machinery that kept subclass overrides working is gone. The Rust core never honoured subclass overrides, so they would have silently stopped running there. Custom routing goes through register_plugin(), settings through apply() and outcomes through observe(). This is a breaking change for anyone subclassing Router. Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries, max_fallbacks, allowed_fails, retry_after, timeout, the tag settings, routing_strategy_args and the fallback lists as given (the crate reads them as numbers, update_settings casts as Python does); a deprecated `model_list = [...]` write reaches the Rust core; a failed set_model_list is all or nothing on both cores (the deployments served before the call are put back and the error raised); config_deployments() is a generator and `del` of a plain attribute works on the Rust core. Least-busy pick reservations from main now work on the Rust core: its selectors share one reservation store, routed calls reserve and free their unstarted picks, and select() and batch cancel do not reserve. On the Python core a dropped upsert takes its name back off deployment_names, and get_deployment_order reads every numeric spelling as one int so mixed orders no longer raise TypeError. Rulings on the remaining differences, the Python core changed where it was the odd one out: sync picks and picks by deployment id apply the team filter as async picks do, so another team's deployment is refused; the fallback lists and default_litellm_params in RouterSettings are copies, so an in-place edit is ignored and settings change only through apply(); one health record with an unreadable timestamp drops out alone; an empty fallback entry is skipped; a bool cooldown_time is refused at construction, a bool or string cooldown_time on a deployment or a call falls back to the router default, and a failure status that no HTTP status can be is recorded as 500; an infinite or NaN cost and a whole number wider than 64 bits (ptu_shares) are refused on both cores with the same error (litellm/router_utils/deployment_numbers.py); and a /v1/messages stream whose provider iterator raises cools the deployment and retries like a non-stream call, with the Rust core waiting for that cooldown before its retry picks. Differences the proxy parity fuzz and its regression tests found on the Rust core: a file upload or a batch on an uploaded file falls back only within its own model group, as on the Python core, since the file lives with one provider (the host sends same_group_fallbacks_only and the walk skips targets in other groups); Decisions calls drop deployments whose provider has no Decisions config, and deployments that refuse the audio or videos a call carries, unless that would leave none, as the Python core does, while the retry count still reads the whole group; a file upload that fails on one order level is created on the next level's deployments rather than the first level again; and a response that cannot be converted (a text_completion body without a message) reports only its own traceback, so a rate limit handled by the attempt before it no longer turns the error into a 429, and both cores return 500.	2026-11-04 21:47:58 -08:00
Mateo Wang	80b18d165b	chore(proxy): use neutral example names for baggage tag mappings (#50011)	2026-11-04 14:49:55 +00:00
tessellark	1cc7da6274	fix(proxy): list slash-less wildcard groups like claude-* on /v1/models and /model/info (#49832) ... * fix(proxy): list slash-less wildcard groups like claude-* on /v1/models and /model/info * fix(proxy): expand team-owned wildcard rows on /model/info through their public name * fix(proxy): keep listing wildcard names wider than their deployment and skip the Azure placeholder A group like *gemini-3* over gemini/gemini-3* lists the catalog ids the name matches and routes back, as main did on /model/info, now on /v1/models too. Wildcard listings no longer list the Azure-LLM placeholder. Team-owned wildcard rows on /model/info go back to main's behavior. * fix(proxy): keep Azure listings on slashed wildcards and list bare deployments through custom_llm_provider get_provider_models returns main's list again, and only slash-less names skip the Azure-LLM catalog placeholder. A slash-less name over a bare wildcard deployment now lists the catalog of its custom_llm_provider. * test(proxy): pin custom_llm_provider guards and slashed wider-name listing * fix(proxy): list bare ids for wider names over a custom_llm_provider deployment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 16:14:04 +00:00
Emerson Gomes	ecb8ef5ca8	feat(realtime)!: support latest OpenAI audio models (#35600) ... * feat(realtime): support latest OpenAI audio models * fix(transcription): type SDK stream duration metadata explicitly * fix(realtime): preserve transport settings and translation duration * refactor(realtime): reuse SDK client initialization * fix(audio): bill finalized streams and authorize transcription models * refactor(audio): keep transcription pricing selection together * fix(realtime): reconcile model metadata and test contracts * fix(realtime): preserve provider duration and model capabilities * fix(realtime): keep transcription validation compatible with cost map guard * fix(realtime): address transcription and billing review findings * fix(realtime): preserve input usage and provider routing * fix(realtime): bill retained output and route Azure aliases * fix(realtime): reject malformed provider durations * fix(realtime): preserve batch cache rates and Azure v1 routes * fix(transcription): reject nonfinite provider durations * refactor(realtime): align annotations with current lint rules * fix(realtime): preserve translation deployment pricing and input usage * test(transcription): verify synchronous stream finalization * fix(realtime): retain missing translation input usage at close * fix(transcription): reject streams that bypass output guardrails * fix(realtime): reconcile audio protocols billing and current CI * fix(realtime): trust confirmed formats and authorize default models * fix(realtime): preserve absent models and reject unsupported guardrails * fix(realtime): reject translation policy pipelines * chore(ci): rerun transient infrastructure timing failure * fix(realtime): reconcile audio protocols with current upstream * fix(audio): validate provider responses at SDK boundaries * fix(realtime): preserve minimum OpenAI SDK compatibility * fix(realtime): preserve deferred Live session authorization * fix(realtime): preserve upstream authentication and transcription behavior * fix(audio): scope provider duration precedence to configured models * fix(typing): avoid Final declarations inside transcription cost loop * refactor(audio): reuse transcription mapping and duration resolution * fix(realtime): enforce model budgets and preserve translation routing * fix(audio): log streamed transcripts whose done event reports detected languages * fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers Providers without a streaming transcription handler got UnsupportedParamsError for stream=True, languages and keywords, where the merge base forwarded them as provider-specific params. Only openai and azure validate them now. * fix(audio): bill streamed gpt-transcribe from the provider's reported duration The stream path computed cost after logging had replaced the transcription's duration usage, so it fell back to the measured file length while the unstreamed call billed the provider's seconds. * fix(transcription): log streamed transcription usage from the provider response The stream logging path built the spend payload from the response after its duration usage was transformed away, so whisper-1 with stream=true and streamed gpt-transcribe logged an empty usage object. Bill and log from the provider's TranscriptionResponse in both the sync and async paths. * fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes - Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape - Use the input transcription model as the session model only for transcription sessions - Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main - Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set - Return x-litellm headers on streamed transcriptions and mark the request done - Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure - Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing - Bill translation sessions on output audio seconds only, at the single published rate - Document the 201 SDP answer on the translation calls routes and the transcription 200 response * fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the default dated version on the deployments route for every model. The SDK client secret keeps session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the gpt-transcribe format check only reads the request provider's cost map entry. * fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it Transcription reaches every provider as on main again, sends stream only when asked, forwards language and languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from one source and spend rows carry the billed seconds. Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA get beta-shaped session events, and transcription sessions send the deployment model in both shapes. * fix(realtime): address review findings on transcription billing, streaming and realtime auth Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata * fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only * fix(realtime): preserve prior merge resolutions after rebase * fix(realtime): enforce translation policies and normalize Azure transcription URLs * fix(cookbook): stop translation once output idles after the upload ends * fix(realtime): bill translation on input audio and drop pre-read stream cost headers Translation spend now uses the input audio seconds the session sent, priced through input_cost_per_second, instead of the translated output duration. A streamed transcription no longer returns cost headers before the stream is read, since the cost is only known after the last event. Also removes an unused realtime guardrail helper. * fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop * test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription * fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA Bill translation sessions from the deployment's own per-second rates, add per-second prices for Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models * fix(realtime): bill a translation rate the deployment omits from the session model * test(e2e): check translated speech is not English instead of exact French wording * fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending * fix(realtime): keep the proxy's realtime session gate off translation sessions A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects * fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params. * fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree * fix(model-catalog): accept the realtime translation endpoints in the cost map * fix(cookbook): close the translation session and read until session.closed * fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard An empty `stream` form value runs the transcription unstreamed again, as OpenAI does, instead of answering 400. The four translate cost-map entries drop the translation routes from supported_endpoints until the main branch's catalog parser accepts them, and the nested transcription model lookup uses the router registry entry point. * test(proxy): drop the mid-stream error frame pin that main's error shape change replaced * fix(transcription): bill uploaded seconds when a stream fails mid-transcript * fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream * fix(audio): keep transcription's positional arguments and redact streamed error details - Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning - Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do - Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite * fix(audio): stream the guarded transcript to SDK callers that asked for a stream * fix(logging): keep a streamed transcript out of callbacks that turned off message logging A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does * fix(proxy): stream a transcription the deployment streamed when the client sent no stream field --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:27:52 +00:00
devin-ai-integration[bot]	08b951a814	fix(vllm): list hosted_vllm wildcard models when the server has no api key (#40066) ... * fix(proxy): expand wildcard models for providers with model info but no cost map entry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): stop requiring an api key to list vllm models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve model ids during wildcard expansion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): forward api key for model discovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): support bearer auth for model discovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): list hosted_vllm models from the server chat uses, with a bounded wait The hosted_vllm listing now resolves HOSTED_VLLM_API_BASE and HOSTED_VLLM_API_KEY like hosted_vllm chat, instead of the /vllm pass-through's VLLM_API_BASE, and waits at most NON_LLM_CONNECTION_TIMEOUT rather than the completion timeout. vllm/* keeps listing nothing, since its chat runs in process. The router pass-through now forwards the deployment's api_key. * fix(vllm): skip the placeholder key, honor deployment TLS and headers, and retry failed listings after a minute * fix(discovery): cache only timed-out model listings, so a fast failure is tried again on the next call * fix(proxy): list wildcard models off the event loop, so a hung provider no longer stalls other requests * fix(utils): keep a concurrent call's model listing when another call times out * fix(proxy): run model discovery on its own thread pool and keep credential lookups off the event loop * fix(proxy): stop the file credential lookup at the first match before listing wildcard models The fallback lookup now walks the allowed model names lazily, in the same order as before, so a hung vLLM server is listed only when no configured name serves the requested provider. The vLLM auth headers no longer mutate the caller's headers dict. * fix(proxy): try every configured wildcard route before listing any wildcard's models in the file credential lookup * fix(proxy): answer model lists from cached listings first and share one listing per server * test(vllm): give the self-signed cert fixture fixed validity dates * fix(proxy): skip wildcard servers a key cannot reach on /v1/model/info /v1/model/info listed every wildcard deployment and filtered by the key afterwards, so a key limited to one model waited on every wildcard server and sent listing calls to servers it cannot use. The listing now skips a wildcard whose patterns match none of the key's models before it asks the server. A key or team granted hosted_vllm/* with no deployment of that name now lists no models, like openrouter/*, since chat answers 400 for every id the environment server would list. * fix(proxy): match a wildcard's own row pattern before asking its server The /v1/model/info pre-filter matched a key's models against both the deployment name and its litellm_params.model, so a key granted hosted_vllm/* still asked a vllm-secure/* -> hosted_vllm/* server whose rows it can never see. It now matches only the pattern expansion names the rows with, picked by one shared helper. * test(vllm): cover the vLLM listings and no-slash wildcard aliases the patch left unrun Adds tests for hosted_vllm get_models with and without a server, vllm/* never asking a configured server, and /model/info pre-filtering of wildcard aliases whose name has no slash. Drops a test comment that only repeated a method signature. --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 04:13:28 +00:00
mateo-berri	994f2caf8f	Merge remote-tracking branch 'origin/main' into litellm_chatgpt_team_subscriptions ... # Conflicts: # litellm/main.py # litellm/router.py	2026-11-02 17:58:08 -08:00
zoveki	33457d3ac9	fix(proxy): check the model in every OpenAI websocket passthrough frame (#48979) ... * test(proxy): cover per-frame model checks and session pricing on the OpenAI websocket passthrough * fix(proxy): check the model in every OpenAI websocket passthrough frame and price the session * fix(proxy): bill each OpenAI websocket turn on its own row and run full auth per frame Each finished turn (completed, incomplete, or failed) now logs its own spend row under the model and access groups that allowed it, so group budgets see spend as turns end instead of at close. Every model a frame names goes through the key's full auth, and a stored prompt reference with no model is refused for a key held to some models. A frame still in flight when the client leaves is recorded before the relay gives up on the send * fix(proxy): keep the stored prompt refusal's first sentence inside the websocket close reason The close reason carries the refusal's first sentence, and the stored prompt sentence ran past the 123 bytes a close frame allows, so it was cut mid-word. The sentence is shorter now and the rest of the explanation stays in the error event. * fix(proxy): bill pipelined websocket turns to their own frame and close the review gaps * fix(proxy): log no extra websocket spend row for frames after the last turn * fix(proxy): price websocket Responses turns by served tier and hosted tools, and bill turns a client walked out of A websocket Responses turn is now priced through the same Responses cost path the HTTP passthrough uses, so a priority-tier answer and a hosted web search cost what they cost over HTTP. When the client leaves mid-answer, the gateway keeps reading the provider's socket until that turn ends, up to 10 minutes, so the turn the provider still bills reaches spend * fix(proxy): bill websocket turns by lane, skip turns OpenAI refuses, and stop frame checks counting the session's own reservation An error event refuses a queued response.create, so the next turn is no longer charged to the refused frame's model. Turns on separate stream_id lanes are charged to the frame that started each one. The frame model gate settles the connect reservation before it re-runs auth, so a key with less budget left than one turn's max cost is no longer refused on its first frame. * fix(proxy): bill each websocket lane cut off at session end on its own row When a client leaves while turns on several stream_id lanes are still running, each unfinished turn now gets its own spend row under the groups and model its own frame asked for, instead of one merged row. Also pins the PR's new single-assigned test variables as Final and swaps datetime.now() in the logging-handler and success-handler fixtures for fixed timestamps. * test(proxy): mark the remaining new websocket test variables Final Builds the logging mocks with model_call_details in the constructor instead of setting it afterwards. * fix(proxy): check every model the realtime URL names before the OpenAI websocket relay connects * fix(proxy): keep reading running answers after a refused websocket frame so they are billed * fix(proxy): bill realtime transcription sessions on the OpenAI websocket passthrough A transcription session names its model only in session.updated under audio.input.transcription, and reports usage on each completed transcription instead of response.done, so its row logged unknown at $0 and a budgeted key never spent. * test(proxy): mark the turn-taking client's delivered list Final * fix(proxy): hold a websocket session's parallel slot until the socket closes Each finished turn logged its spend row under the session's own call id, so the first turn's success callback freed the key's max_parallel_requests slot while the socket was still open, and a key limited to one session could open several at once. The session now claims its slot under its own call id, every turn row logs under a call id of its own, and the slot is released when the session ends, as before Priced turn rows also take their row's call id as their id, so two turns that start in the same instant no longer share a spend log id and drop one row * fix(proxy): bill realtime transcription clips that finish outside an answer A conversation.item.input_audio_transcription.completed frame that arrived after a session's last response.done was never billed, because the session end only owes a row for a turn still running. A clip that completes while no answer is running now logs its own row, and a clip inside a running answer stays in that answer's row Each row also carries the session's latest session.updated frame, since later rows lost the transcription model it names and priced clips against the realtime model. A newer session.updated replaces the older one, so the stale model is never read first It also drops the docstrings this PR had added, since the names already say what each helper does * fix(cost): price a realtime transcription clip at the transcription model the session set A realtime conversation session opens with session.created naming the realtime model and no transcription model, and the client sets the transcription model later in session.update. The cost code took the first session frame and fell back to its session model, so a clip was priced at the realtime model. That model has no per-second rate, so a gpt-transcribe clip cost $0, and a token-billed clip used the realtime model's audio rate instead of the transcription model's. Now the newest session frame that names a transcription model sets the price, and the first session model is only the fallback when no frame names one, as before * fix(cost): price each realtime transcription clip at the model set when it ran A session that changes its transcription model mid-way kept every logged clip on the newest model. Each clip now takes the transcription model from the last session event before it, and falls back to the session's own model * fix(passthrough): keep the session update a buffered clip ran under until its answer is billed A transcription that completes while an answer streams waits in the buffer for that answer's response.done. A session.updated in between used to drop the earlier update, so the clip was priced at the opening configuration. An earlier update now stays while an unbilled clip follows it, and drops at the next update once that clip is billed. * fix(passthrough): keep reading a websocket turn the client left before OpenAI started it A client that sends response.create and hangs up before OpenAI answers with response.created made the walkout checks see no turn in flight, so the gateway closed the provider socket and the turn OpenAI still ran and billed never reached spend. The relay now counts every forwarded create until the provider starts or refuses it, and keeps draining while one is owed * fix(passthrough): settle a websocket turn that ends without its start frame A turn the provider ended without sending response.created first left its create counted as not started, so a client leaving later held the provider socket for the whole walkout drain with nothing left to bill. A turn's end now settles one waiting create when its lane saw no start frame * fix(proxy): refuse unparseable and unroutable websocket frames and hold legacy slots per socket A client frame too deep or too long for the JSON parser closed the provider socket mid-turn, so the turn in flight was never billed. The gate now refuses that frame with an error event and keeps draining the open turn. A stream_id OpenAI rejects comes back as an error that names no lane, which dropped another turn's authorization. The gate now refuses those frames before they reach OpenAI. With the legacy parallel request limiter, each per-turn spend row freed the key's slot while the socket stayed open. Websocket passthrough rows now free nothing, and the socket frees its slot once when it closes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:48:59 -08:00
moe-berri	bba63c1934	fix(proxy): serve /a2a/{agent_id}/message/stream routes (#49471) ... * test(a2a): cover message stream route aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): serve /a2a/{agent_id}/message/stream routes Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): document A2A stream route schemas * fix(proxy): use immutable A2A schema mappings * fix(proxy): map v1 a2a stream route to call types and keep stream alias unbilled * test(a2a): use real guardrail registry and typed locals in stream route tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com>	2026-10-31 16:11:51 -07:00
ishaan-j-berri	941f52f26f	fix(proxy): preserve discovery after budget exhaustion (#48809) (#49463) ... * fix(proxy): preserve discovery after budget exhaustion (#48809) * refactor(proxy): decide budget exemption once (#48809) * refactor(proxy): centralize request budget policy (#48809) * fix(proxy): keep custom pass-through budget checks (#48809)	2026-10-31 19:00:51 +00:00
mateo-berri	252337f60c	Merge remote-tracking branch 'origin/main' into litellm_chatgpt_team_subscriptions ... # Conflicts: # tests/unit/proxy/credential_endpoints/test_endpoints.py	2026-10-31 03:10:47 -07:00
Nate Armstrong	87d91d78b7	refactor(router): route the router family through RoutingContext and close the outside surface (#49118) ... Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality sub-routers now take a RoutingContext instead of the Router, and read the router only through registry(), settings(), health() and the context's forwarding methods. Router-free helpers move from router_utils to litellm_core_utils, outside callers of router state switch to entry points, and the surface allowlist shrinks to the endpoint methods plus the entry points	2026-10-30 19:18:54 -07:00
tavren27	7e37c20748	fix(responses): run the guardrails a Responses WebSocket frame selects (#48725) ... * fix(responses): run the guardrails a Responses WebSocket frame selects * fix(responses): name the field a malformed frame guardrail selection got wrong A frame whose guardrails or guardrail_config has the wrong shape now gets an error event naming the field, instead of pydantic's raw validation dump * fix(responses): accept every guardrail selection shape a Responses WebSocket frame sends, and run during_call guardrails A frame can pick guardrails as a list of names or name-to-config dicts, inside metadata or litellm_metadata, with policies validated as a list of strings. during_call guardrails now run before the turn goes upstream, and selection fields sent next to a nested response are rejected instead of dropped * fix(proxy): keep a key's configured guardrails when policies add guardrails A policy's guardrails were merged with dict.fromkeys, so a configured guardrail entry ({name: {extra_body}}) in key metadata raised unhashable type: 'dict' and every request naming policies failed with a 500 on HTTP and on the Responses WebSocket. The merge now keeps existing entries and skips policy guardrails already selected by name. Also types the WebSocket test guardrail's hooks. * fix(responses): run streaming-scoped guardrails on socket turns, keep guardrail fields on blocks, merge guardrail lists by name * fix(responses): check frame guardrail picks against the team permission, honor picks beside a nested response, prefer a guardrail entry with settings * style(responses): format the frame guard and give the pass-through guardrail test an enterprise license * fix(proxy): keep pass-through guardrail settings, gate frame policy picks, skip moderation-only hooks on socket turns * fix(responses): keep string metadata guardrail values as client metadata on socket turns * fix(responses): ask no team permission for string metadata named like a guardrail pick on socket turns * fix(proxy): socket guardrail picks read the key's team metadata when the proxy has no database --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 14:12:51 -07:00
norkelbyte	5898a42f5a	fix(openrouter)!: list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on (#48820) ... * fix(openrouter): list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on * fix(openrouter): list models off the event loop and keep only chat-callable catalog ids The model-list builders are sync, and with check_provider_endpoint on they call OpenRouter's catalog with a 10 second timeout and no caching of failures. They ran on the event loop inside /v1/models, /model/info, /model_group/info and the per-user listing, so one slow catalog froze every request on the worker. Those calls now run in a worker thread with asyncio.to_thread File, batch and RAG credential lookups only need each name's deployment, and a wildcard that is itself a deployment sits in the list ahead of its expansions and resolves its own credentials. get_complete_model_list takes expand_deployed_wildcard_routes, which that lookup sets to False, so a deployed wildcard never fetches its provider catalog there. A key wildcard wider than any deployment, like anthropic/* over a deployed anthropic/claude-*, still expands so it finds the matching deployment's credentials. Every other caller keeps the default The pattern router's model lookup tolerates a pattern removed between its two reads of the pattern table, which a deployment delete on the event loop can now do while a listing runs in a thread Listing now reads only OpenRouter's models/user catalog. OpenRouter's own /models list carries no embedding models, the OpenAI-format /v1/models has no mode field so chat pickers cannot tell embeddings apart, /model_group/info labels them mode chat, and one failing catalog emptied the whole list. Ids ending in :batch are dropped because chat completions answers 404 for them, while :free and every other variant stay. An api_base that already ends in /chat/completions, which chat accepts as is, has that suffix stripped before the catalog route is joined * fix(proxy): keep model listing off the shared thread pool and out of credential lookups Listings run on their own thread pool so many slow catalogs cannot starve the default executor that chat and Responses calls share. File and batch credential lookups match deployed names against a key's wildcard grants instead of listing any provider catalog. The token counter keeps OpenRouter deployments on their previous path, and a catch-all "*" over provider/* lists bare ids that route back through it * test(proxy): give the off-loop listing tests a generous deadlock guard and typed ids * fix(proxy): keep routed models in wildcard model info, skip key-only OpenRouter catalogs, cache failed listings briefly * fix(utils): cache failed model listings under their own key so a late failure never hides a good listing * fix(proxy): remember only timed-out provider listings so a provider that failed fast is listed again * fix(proxy): ask each provider catalog once per worker, keep deployed names out of catch-all ids, and match file credentials by name Listings that start together for one deployment now wait for the first catalog request. A catch-all wildcard no longer lists a name a deployment serves by name. A key wildcard only finds file and batch credentials on deployments named under it, so a bare name keeps the default credentials as before. * fix(proxy): keep the listing failure log line the same for timed-out listings * refactor(proxy): fold the cached listing lookup back into the provider listing function * fix(proxy): list each model id once and keep a named deployment's row over a wildcard's * fix(proxy): drop a wildcard's model info row when named deployments serve every name it lists --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 12:19:51 -07:00
Etienne Chabert	372c096388	fix(proxy): evict a deleted key from every worker's auth cache (#47609) ... * fix(proxy): evict a deleted key from every worker's auth cache /key/delete dropped the key object only from the in-memory cache of the worker that served the call. Auth serves key objects cache-first, so every other worker, and the Redis copy when enable_redis_auth_cache is on, kept authenticating the deleted key until user_api_key_cache_ttl expired. /key/update already evicts and broadcasts through evict_and_broadcast, delete now does the same for the tokens it deletes * fix(proxy): evict only the deleted rows and broadcast them in one batch Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker * refactor(proxy): drop the comment above the delete eviction call * fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait * test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests * test(proxy): wait for the peer to apply the broadcast instead of polling its cache * test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer * fix(key_management): evict every requested key on delete, including keys whose row is already gone A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone * test(key_management): split the gone-row eviction assertion into one comprehension per worker --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 10:20:00 -07:00
zoveki	4786dd861b	fix(proxy)!: scope /health by wildcard keys and team model allowlists the way auth does (#49155) ... * test(proxy): cover /health scoping for wildcard keys and team model allowlists * fix(proxy): scope /health with the model access check auth runs on a request * fix(proxy): scope /health to the model group an allowed name routes to A key alias, team alias, or router model_group_alias lets auth admit a requested name by checking its target, so /health also kept the deployment under the requested name, one the key's requests never reach. Keep only the served model group each admitted name dispatches to, through the same alias layers /v1/models mirrors, and let alias_map read its own read-only output. * test(proxy): cover /health skipping a deployment that has no model name for a restricted key * fix(health): show deployments reached through wildcard and alias allowlists, and by id when an alias shadows the name * chore(router): drop should_include_deployment from the outside-use allowlist now that /health no longer calls it * fix(health): never list a default-fallback deployment for an unserved grant, and keep access-group keys to their group's deployments * fix(health): resolve ?model= through the caller's aliases the way chat does * fix(proxy): apply user and agent model limits to /health and yield between checks /health now runs the same user and agent model checks chat runs, so a personal key whose owner may call only some models, or an agent key limited by its access group, no longer sees or probes other deployments. Each per-name check yields to the event loop, and ?model= reaches the wildcard deployment serving an allowed name * fix(proxy): check owner and agent grants for wildcard routes in /health and fail on non-access auth errors * fix(proxy): answer /health auth lookup failures the way chat does and record ?model= checks under the requested name * test(proxy): pass recorded_model_name in the direct health save tests * fix(proxy): count alias names as /health candidates so aliased names a key may call get probed * fix(health): narrow wildcard-served deployments to the key's access groups * fix(health): narrow every deployment serving a name to the key's access groups, team-owned ones included * fix(health): narrow a team deployment's internal name to the key's access groups like its public name * fix(health): keep a team deployment's internal name narrow when a limit holds only that name An agent or key limit that holds one team deployment's internal name no longer reaches every deployment behind the team's public name. The internal name maps to its public name only when the public name is also admitted. * test(health): cover all-proxy-models next to * for a key that also holds an access group * refactor(auth): keep the _as_proxy_exception name so main's tests still find it * fix(health): keep an internal team name a wildcard grant admits as its own target --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 07:36:01 -07:00
zoveki	87fec5467f	fix(sso): keep CLI SSO and Claude Code gateway team lists in the user record's order (#49033) ... * fix(sso): bind CLI SSO and Claude Code gateway logins to the first team on the user record * fix(proxy): keep SCIM and JWT team sync in record order, and look up SSO groups once SCIM PATCH and JWT team sync saved a user's teams from a set, so the first team on the record, the one a Claude Code gateway sign-in binds, could change on any sync. Both now keep the record's order and append new teams in the order the IdP sent them. SSO group sync now drops claim groups that are not LiteLLM teams with one lookup before adding the missing teams one at a time, so a claim full of non-team groups no longer costs one failed add per group on every sign-in. * test(sso): expect the gateway team dropdown in the user record's order * fix(sso): read team existence from the writer, keep new-user and SCIM replace team order * fix(scim): keep the teams a user already has in record order on PATCH and PUT * fix(scim): keep record team order when a SCIM POST matches an existing email * fix(scim): join new teams in request order so a failed roster write keeps it * fix(scim): let a groups replace win over later group ops in a PATCH again Applying group adds and removes after a replace made a PATCH that deactivates a user and adds a team that does not exist answer 404 and leave the user active, where it answered 200 before. Keep the record's team order, but ignore group ops after the last replace as the base did. * test(sso): type the CLI team detail row helper --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: rj-cairn <349020452+rj-cairn@users.noreply.github.com>	2026-10-30 05:07:00 -07:00
mika-lt	cfb77ecc3a	fix(proxy): list only routable models for a wildcard alias over a provider org (#49152) ... * fix(proxy): list only routable models for a wildcard alias over a provider org A wildcard alias over an org-scoped deployment (together_ai/* over together_ai/nosuchorg/*) listed the provider's whole catalog on /v1/models and /model/info, and every listed id routed to a model the provider 404s. A skipped deployment path segment now counts as a routing prefix only when the routed model resolves to a catalog id through the get_model_info lookup order, so responses/, Bedrock converse/, and region paths list as before. Together's model list now also carries the cost map's together_ai rows, skipping the together-ai-* pricing tiers, so an alias over a real org lists the models Together serves today. * fix(proxy): keep flat catalog ids under routes the cost map can't parse, drop retired Together rows * fix(proxy): check skipped path segments only on Together so Hugging Face provider routes keep listing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 00:14:49 -07:00
Mateo Wang	636e707089	fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) ... * fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes The error objects these routes return defaulted `type` and `param` to the four-character string "None", so a client's error handler matched no known OpenAI type and fell into its generic branch. Route them through the shared openai_error_payload helpers instead, so `type` comes from the carried type or the HTTP status and `param` serializes as JSON null. * fix(proxy): stop answering with the literal string "None" on the spend and management routes The error objects these routes return defaulted `type` and `param` to the four-character string "None", so a client's error handler matched no known OpenAI type and fell into its generic branch. Route them through the shared openai_error_payload helpers instead, so `type` comes from the carried type or the HTTP status and `param` serializes as JSON null. * test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param * fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None". * fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400 * fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap * fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param * fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error. * test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor	2026-10-29 18:20:52 -07:00
Juan V.	60601ca995	feat(pass-through): add eu/jev decision API pass-through with spend tracking (#46875) ... * feat(pass-through): add eu/jev decision API pass-through with spend tracking Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted decision API that speaks the same POST /v1/systemone contract as TypeSafe. It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to https://jev.bevel.software), books spend through the shared decision logging handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the cost map * fix(pass-through): default eu_jev to its canonical eu-jev.com origin * fix(pass-through): reject custom_body on the eu_jev route A key limited to other models could send an eu/jev request through custom_body, because auth checks the outer body while the shared pass-through forwards custom_body upstream. The route now answers 400 before anything is forwarded, the same way validate_oss_request guards the laya and bespoke routes * fix(pass-through): give the eu_jev route request and 200 response schemas The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and OA003. The route now documents the eu/jev System One request body and its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI document. The lazy OpenAPI snapshot and the dashboard API types are regenerated with python -m litellm.proxy._lazy_openapi_snapshot and npm run gen:api * fix(pass-through): reject streaming on the eu_jev route A body with stream=true switched the route to streaming, which skips the decision spend tracking. Reject it with a 400, as validate_oss_request does for laya and bespoke. * fix(pass-through): type the eu_jev request body like the other decision routes * fix(pass-through): check eu_jev decisions against the model eu/jev runs Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted model and the legacy names to eu-jeff-latest for key model checks and pricing, check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when EU_JEV_API_KEY is unset * fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and upstream still gets the client's exact body. eu_jev is a registered decisions provider, so access groups, config deployments, /v1/systemone and /v1/decisions accept it, and the Admin UI can add it. A release without its own cost map row is priced as eu-jeff-latest instead of $0 * fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models * fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it * fix(eu_jev): take pass-through credentials only from proxy-wide deployments The eu/jev route now ignores team-scoped deployments when it picks the upstream key and base, so one team's deployment can no longer redirect another team's calls, and a base ending in /v1 is not doubled * fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it Scope the mapped pass-through and gateway allowlist entries to the decision route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep their auth, metering, and gateway behavior from before this PR * fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it Drop the decision path from the mapped pass-through list and classify it with one check that asks the pass-through registry, so an operator's own pass-through at that path keeps its auth, metering, LLM-route status and litellm_user_api_key handling from before this PR. The built-in route now yields on a per-request root path and on any configured method, and the gateway keeps only the exact decision path * fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding * test(eu_jev): annotate new test variables with Final * fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches Swap an updated pass-through's routes with no await between removal and re-registration, so the built-in eu/jev route never sees the path as unconfigured mid-update. The built-in eu/jev handler no longer takes a configured route's target, and the built-in route also yields on a method-only match, so GET and PUT on the no-slash form of a slash route redirect again. * test(pass-through): annotate the route-swap test's locals as Final * fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker * fix(eu_jev): leave a null model out of the upstream decision request A JSON model of null is limited and counted as eu-jeff-latest, so send the request upstream without a model field, the way an omitted model goes, since eu/jev answers a null model with 400 Unknown model * fix(pass-through): keep route checks on a path another auth endpoint still serves after an update Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one * test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it * refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 17:52:43 -07:00
Nate Armstrong	ead221995e	refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) ... Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None.	2026-10-29 17:26:33 +00:00
mateo-berri	2132c83dd1	feat(chatgpt): bind a ChatGPT subscription login to one team or key ... A chatgpt/ deployment whose litellm_credential_name names a subscription login now runs on the access token the caller's own team or key owns, and answers 401 for any other caller instead of falling back to the host's auth file. A request body cannot override that credential name, the login values never merge into call kwargs, and their keys are server owned so a body or non-admin credential write cannot set them. acompletion also passes the call's api_key into its first provider lookup, so a bound login no longer triggers the host device-code login	2026-10-29 10:01:50 -07:00
devin-ai-integration[bot]	e85cb4052e	test: fix flaky spend-log cleanup budget, login lockout, and wedged Redis tests (#47108) ... * test: stabilize cleanup, lockout, and post-kill timing tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bound the wedged Redis burst p95 by the socket timeout the test sets The burst p95 under a fixed 0.75s cap tracked how fast the box ran the 30 concurrent writes, so a loaded runner failed it at unchanged code. A handler that waits on the wedged publish takes at least the coordination Redis socket timeout, so the test now sets that timeout and asserts the p95 stays under it. * test: pin the lockout clock and bound the wedged Redis burst by the proxy's 2s wait cap The lockout tests now set the login throttle's clock instead of reading the real one, so Retry-After is pinned at 77 and a round-down fails on every run The wedged Redis burst p95 is now bounded by the 2s cap the proxy puts on its own Redis waits, so a burst wait capped under the 5s socket timeout still fails the test The fresh post-kill clients keep the harness's 2s keep-alive expiry, here and in the two tests #47080 changed * test: tell ceil from round in the lockout tests and bound the burst by the imported eviction cap The refused attempt now runs with 76.4s left on the 77s block, where ceil gives 77 but round and int give 76, so both lockout tests fail if Retry-After stops rounding up. The burst bound imports _BULK_EVICTION_WAIT_SECONDS instead of copying its 2s value, so lowering the cap moves the bound with it. * test: bound the wedged Redis burst by the shorter of the eviction cap and the socket timeout Any wait the proxy puts on the stopped Redis lasts at least the shorter of the two, so the bound now follows whichever one a later change lowers. * test: drive the tool index cleanup test with the manual clock and keep the post-kill block as on main The post-kill reset is already fixed on main by making kill() wait for the worker's last thread, so the fresh-client and keep-alive edits go back to main's version. The tool index cleanup test now advances the injected clock instead of timing the real one, and the lockout clock helper gets a type hint. * test: advance the spend log cleanup tests' manual clock through its own method * test: freeze the throttle clock in the login_utils lockout tests and drive the run budget tests with the manual clock --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:46:32 -07:00
devin-ai-integration[bot]	3c1eb73a06	fix(proxy)!: list every priced Vertex family under a vertex_ai/* grant (#41959) ... * fix(proxy): list every priced Vertex family under a vertex_ai/* grant A vertex_ai/* key listed none of the 68 cost map rows priced under the Mistral, Llama, Qwen, AI21, OpenAI, image, video and embedding labels because their sets were never part of the vertex_ai union, and Qwen rows had no ingestion branch at all. Wildcard expansion also dropped a meta/ or openai/ publisher segment as if it were a provider prefix, and the wildcard health check would have picked $0-per-token rows first once the union grew, so it now sorts paid rows ahead of unpriced ones. The VERTEX_FAMILIES_A_VERTEX_WILDCARD_GRANT_DOES_NOT_LIST allowlist is gone so the cost map guard enforces every label from now on. * fix(health): route every wildcard health-check pick through the listed provider * fix(health-check): keep per-character-billed rows ahead of rows priced $0 everywhere in the wildcard pick * fix(health-check): break $0-per-token ties toward per-character-billed rows in the wildcard pick * fix(health): only health-check ids the router sends to the provider The wildcard health check picker now skips ids that get_llm_provider routes to a different provider or to none, so the longer vertex_ai union cannot pull partner rows into the check. Drops the forced custom_llm_provider and the per-character tie-break from earlier revisions, and pins the replicate and wandb wildcard listing to each row's full owner id. * fix(health): match wildcard health picks against the provider's own route The previous check kept a bare id only when get_llm_provider routed it to the exact provider name, which dropped every ai21 row because bare ai21 ids route to the ai21_chat alias, so ai21/* health checks sent no request. Compare the bare id's route with the route of the provider-prefixed id instead, so aliases match. Resolve routes with inferred_provider, which returns the declared provider for github_copilot and chatgpt without starting their device-code login. * fix(health): keep wildcard health picks that only route through a saved custom_llm_provider Rank picks routed to the checked provider first and ids routed to no provider second, each by cost, and skip only ids routed to a different provider, so a wildcard saved with custom_llm_provider still sends its health request * refactor(vertex_ai): mark the new Vertex registry sets Final * chore: re-run CI against current main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:31:04 -07:00
paveln83	3f5e42d032	fix(proxy): enforce team and key Allowed Vector Stores when no vector store registry is loaded (#48915) ... * test(proxy): cover team and key vector store allowlists when no vector store registry is loaded * fix(proxy): enforce team and key Allowed Vector Stores when no vector store registry is loaded * fix(proxy): skip non-string vector_store_ids entries instead of failing auth * fix(proxy): answer 400 on a malformed vector_store_ids in legacy mode too Legacy vector store access checks now parse the request the same way vector_store_deny_by_default does, so a vector_store_ids that is not a list of strings is a 400 with or without a registry. * fix(proxy): honor drop_params and a tool's string vector_store_ids in the legacy vector store check Under drop_params the check reads the stores registry retrieval uses instead of answering 400, and retrieval reads only the string entries of a top-level list. Without it, a tool's single string id is checked as one store, the way the registry reads it. * fix(proxy): honor deployment and router drop settings for a malformed vector_store_ids * test(proxy): drop vector store registry mocks the access check no longer reads * fix(auth): answer 400 for a malformed vector_store_ids only under a key or team allowlist * fix(auth): match managed vector store ids by their provider store and read a string id as one store A LiteLLM-managed store id now matches an allowlist entry for the provider store it decodes to, in both directions, so an allowlist naming store A no longer refuses A's managed id. A top-level vector_store_ids string, which is how ?vector_store_ids= arrives, is checked as one store id instead of answering 400. Registry retrieval reads an SDK tuple of ids again --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 23:25:14 -07:00
mika-lt	de42b67252	fix(proxy)!: enforce an end user's per-model budget from their linked budget (#48815) ... * fix(proxy): enforce an end user's per-model budget from their linked budget The shared auth checks now refuse a request once the end user's spend on that model reaches the model_max_budget of their budget (a linked budget or the default applied to them), for virtual keys and the master key alike. The cap is read from the end-user row, since the master key's cached auth object keeps whichever end user's limits it was cached with. The per-model end-user check inside virtual-key validation is removed: it read a token field that is never set there, so it never ran. * fix(auth): enforce the end user's per-model budget on JWT auth and skip it on budget-exempt routes JWT tokens now carry the end user's per-model caps so spend is charged against them and over-cap requests are refused. Routes that skip budget checks, like /spend/calculate, skip this check too. The proxy default end-user budget is covered by a test. * fix(auth): write the end user's per-model cap onto every token and check it after the member fallback The shared auth checks now resolve the end user's per-model cap, falling back to the key's or the proxy's default end-user budget when the end user has no row yet, and write it onto the token so the spend hook charges the cap the check enforces on every auth path. The check runs after the team-member budget fallback, on the model it picked, and the JWT builder's duplicate check is removed * fix(auth): keep only a per-model end-user cap that custom auth itself set A key-auth token reached through custom auth's auto hand-back now takes the current end user's cap, so a cached master-key or virtual-key object no longer carries an earlier end user's cap into another end user's request. * test(auth): build the custom-auth token with model_copy instead of setting the flag in place * test(auth): hold an end user to their own or their key's budget cap when the proxy has no default budget * fix(proxy): give each request its own copy of a custom-auth token so end-user caps and ids stay per request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:24:11 -07:00
Mateo Wang	f46adf80e4	fix(proxy): let typesafe/* allowlists cover /typesafe Jev calls and list Jev in the model picker (#47697) ... * fix(proxy): list TypeSafe Jev models in the admin model picker * fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones. * fix(typesafe): match only published Jev ids and keep picker loads off the secret manager Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file. * test(typesafe): type the picker test fixture and tests * fix(typesafe): match only published Jev versions and cover non-version suffixes in tests * fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker * fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers The dashboard sends include_model_access_groups on every /models call, so gate the TypeSafe names on wildcard routes too and skip access-group-only listings. Also say the typesafe-jev-ids rule matches case-insensitively. * fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main A typesafe/* router deployment registers typesafe/* in the cost map, so the allowlist pickers offered a bare *, which allows every model. The pickers now list only priced names the rule sends to TypeSafe. The non-expand list also stops de-duplicating every entry and only skips TypeSafe names already listed.	2026-10-28 20:19:58 -07:00
Mateo Wang	567383c503	feat(proxy)!: model family and access group entries in model_max_budget (#47722) ... * feat(ui): let the per-model budget field take a model pattern or access group * feat(proxy): let model_max_budget entries name a model family or access group, shared across replicas * fix(proxy): open each shared model budget window once for every replica, and match patterns in linear time Without Redis, the first replica to charge an entry in a new window now opens it in the window spend table and the others join it, so no flush replaces spend another replica already added. Budget patterns match by splitting on `*` instead of a regular expression, a refused request names the entry that refused it, and the key form no longer offers a typed pattern another row already budgets. * fix(proxy): read the shared budget window from the writer, and keep concurrent first charges on one replica Opening a shared window now reads the row from the writer, since a lagging read replica could hand a replica its own start and let a flush replace the others' spend. A charge that waited on the open no longer resets a local window another charge on the same replica opened meanwhile. Window tests use fixed timestamps. * fix(proxy): stop a non proxy admin from lifting a team model cap with a looser group or pattern entry Group and pattern entries now take precedence over other entries, so a team update by anyone but a proxy admin must keep each new or changed entry within every existing cap it can take precedence over. A pattern can match a provider-prefixed or wildcard-routed name, so any two patterns count as overlapping, and a model or group entry counts as outranking every pattern. * fix(proxy): narrow model budget pattern overlap, keep team caps under key patterns, and accept only typed patterns in the UI * fix(proxy): keep model family budgets through database blips, team models and key entries A failed window open no longer restarts the window at this replica's clock, so it cannot wipe shared spend. Spend charged meanwhile stays local and joins the shared window once the database answers A key entry named after a model replaces only the team entry of that name, so team pattern and group caps still apply. Team-scoped models are gated and charged under their public name, and group lookups see team deployments The team update guard compares caps by window length and only ranks patterns that can match the same name, including Bedrock vendor patterns behind a bedrock wildcard route * fix(proxy): refuse team entries that take precedence over a capped model_max_budget entry A non-proxy-admin could add a carve-out (an exact model, a narrower pattern or a group) at or under the family cap, and its own counter then multiplied the team's spend on that family. Any new entry that takes precedence over a capped entry now needs a proxy admin, and group members include the team's own public model names so team deployments cannot hide a group overlap. * test(router): cover team_public_model_names_of * docs(proxy): name pattern and access group keys in model_max_budget docstrings * fix(proxy): name the entry holding the cap in the team model budget raise refusal * fix(proxy): check precedence when a team edit makes an unenforced model_max_budget entry take effect * fix(proxy): let an org admin give a legacy model_max_budget row a budget_duration at the same or a lower cap * fix(proxy): keep a legacy model_max_budget row's stored cap when an org admin gives it a budget_duration	2026-10-28 15:40:57 -07:00
moe-berri	ed5fe048ff	fix(proxy/auth): apply max_end_user_budget_id to end users without a row on cached keys (#48704) ... * test(proxy/auth): cover default budgets on cached virtual keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy/auth): use in-memory cache in cached budget tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy/auth): apply max_end_user_budget_id to end users without a row on cached keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy/auth): share cached end-user parameter update Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 09:19:36 -07:00
ryan-crabbe-berri	34134d7ae5	fix(auth): log routine 401 rejections as one warning without a traceback (#48613) ... A missing or malformed Authorization header, a blocked key, a non-admin key on an admin-only or master-key-only route and a SCIM-deactivated user were raised as bare exceptions, so the auth failure handler could not see the 401 and logged each one at ERROR with a full traceback, which flooded the log under scanner traffic They now raise a typed 401 carrying the quiet-log marker that the non-sk bearer rejection already used, and the unknown sk- key rejection joins the same path, so each lands as one WARNING line on stdout with no traceback. The client response is unchanged	2026-10-27 12:46:22 -07:00
Mateo Wang	0a1dbc0fb4	refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) ... * refactor(mcp): drop the experimental label from the MCP server and client paths Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 17:47:50 +00:00