948 commits

Author	SHA1	Message	Date
joshua-berri	894552205b	feat(spend): report guardrail spend separately in daily activity (#51010) ... * feat(spend): report guardrail spend separately in daily activity Model and key spend in /user/daily/activity/aggregated include the cost of guardrails like Bedrock guardrails, and nothing in the response says how much. A customer that bills tenants per key and per model could not get model cost alone, so tenants that pay AWS for their own guardrails were charged for them twice Daily spend rows now carry a guardrail_spend column next to spend, and every breakdown in the daily activity responses exposes it, so model cost is spend - guardrail_spend. spend keeps its meaning, so budgets still count guardrail cost as before The value comes from the request's cost_breakdown.guardrail_cost, capped at the row's spend because an allowance discount or a repeat job poll can charge less than the guardrail cost the breakdown records. The delivery ledger freezes it into the accounting payload next to accounting_savings, since the stored payload does not keep cost_breakdown, and the legacy daily queue writes the same value The migration only adds the column with a 0 default, so rows written before it read 0 * fix(spend): count guardrail spend on rows billed without a cost breakdown A request blocked before the LLM call is billed for its guardrails straight from guardrail_information, so it carries no cost_breakdown and its guardrail_spend read 0. Both writers now price guardrail_spend from guardrail_information with guardrail_information_cost, the same function success and failure rows use to add guardrail cost to spend Also fills guardrail_spend in the client usage view built from request logs, qualifies the field description for rows recorded before it existed, and adds the lock timeout the migration checks require * fix(ui): avoid a nested ternary when reading guardrail entries from request logs --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-11-10 13:02:26 -08:00
Mateo Wang	4867a56e9a	fix(proxy)!: reject client-supplied Vertex location and project unless the admin opts in (#47028) ... * fix(proxy): reject client-supplied Vertex location and project unless the admin opts in A request body could carry vertex_location, vertex_project, or their vertex_ai_* aliases and override the region and project pinned on a Vertex AI deployment, sending the admin's credentials to a caller-chosen project or region. Add the four fields to the banned request params so they need allow_client_side_credentials or configurable_clientside_auth_params. /vertex_ai/live and /rag/ingest name their Vertex project and location by design and have no deployment pin to override, so they keep those two fields while every other banned param is still rejected there * fix(proxy): close the remaining Vertex target overrides and keep inert fields working /vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target * test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads * fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest * fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests * test(proxy): give the batch tests' fake router the alias and routed-group lookups * fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review A Vertex opt-in now counts only when every group the request can reach opts in, router, context window, and content policy fallbacks and one alias hop included, while a group with no Google deployment never needs the fields. A model named in the route path is checked next to the body model. RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin saved before the upgrade, and reads a corpus named by project number on the proxy's own project. /vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves a setup frame sent as a binary message onto the session's target, and resolves the region from default_vertex_config before the model path. * fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project, including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those projects now connects, and any other project still closes with 1008 before a token is minted. * fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest A non-admin ingest into a store an admin created named by project number had its corpus moved onto the proxy's own project before the admin-store exemption ran, so the upload went to a different corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the project-number rewrite and the target check. * fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request * fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list * fix(proxy): check team public models before allowing target overrides * fix(proxy): include team wildcard targets in opt-in checks * fix(proxy): validate overrides against authenticated team routes * fix(proxy): retain URL model during deferred credential checks	2026-11-10 20:50:17 +00:00
mrinal-berri	9ba551b871	feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571) ... * feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes	2026-11-10 11:06:17 -08:00
devin-ai-integration[bot]	d47c03175b	fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions (#50968) ... * fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): explicitly route bridge fixtures to responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kenneth <kenneth@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 18:33:27 +00:00
shrey	e6db6206d1	fix(mcp): accept comma-separated vendor scope grants on the connect check (#50717)	2026-11-10 10:23:50 -08:00
paveln83	90f37e113c	fix(proxy): send a prompt-only call_endpoint token count to every provider count API (#50783) ... * fix(proxy): count a prompt-only call_endpoint request with the provider on every counter POST /utils/token_counter?call_endpoint=true with only a prompt sent the prompt to the provider count API on Gemini and Vertex only. Anthropic, OpenAI, Bedrock and Azure AI (Claude) deployments answered a local estimate instead. Every counter now gets the prompt as one user message, so a provider error surfaces the same as the messages form, and the framed stand-in after a failed provider count frames the prompt as one user turn, matching the messages form * fix(proxy): count a call_endpoint prompt like a user message on providers without a count API * fix(proxy): count a call_endpoint prompt as a turn only on chat deployments A prompt-only /utils/token_counter request with call_endpoint=true now goes to the count API as one user message only when the deployment serves chat (mode chat, responses, or unknown). Embedding rows and models with no deployment keep the raw local count. The Anthropic, Azure AI, and OpenAI count calls now honor the deployment's timeout, and a count error with an empty body names its status code. * fix(proxy): fall back to router_settings.timeout for count calls and answer 408 on a count timeout Count calls to Anthropic, Azure AI, and OpenAI now pick their timeout the way a chat call to the same deployment does: the deployment's timeout, else router_settings.timeout, else request_timeout. A count call that times out answers 408 instead of a 500 processing error. * fix(proxy): count a prompt as a turn only where a count API exists and give count calls chat's timeout order * fix(proxy): read the count call timeout through the router settings snapshot The token counter now asks llm_router.settings().timeout_for() for the chat timeout order instead of a new Router member, so the router surface does not grow * fix(router): answer timeout_for on every router settings snapshot The proxy reads the count call timeout through llm_router.settings().timeout_for(), but only the Rust core's snapshot had that method, so a plain RouterSettings (the unit tests' router stub) raised AttributeError. timeout_for now lives on RouterSettings and reads the core's _core_deployment_call_timeout, like the snapshot's other members --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 18:09:19 +00:00
Kerry Lu	c1abd490b2	feat(customers): reject unknown fields on /customer/new and /customer/update instead of dropping them (#50965) ... * feat(customers): reject unknown fields on customer create and update instead of dropping them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(customers): derive the budget hint from the budget request fields and fix the create example Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 18:08:21 +00:00
Kerry Lu	4714aac42b	feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter (#50857) ... * feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(rate-limit): build model scope settlement targets from one owner tuple Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 09:22:18 -08:00
zoveki	901f914ba2	fix(responses): replay Claude tool search on follow-up turns (#50821) ... * fix(responses): replay Claude tool search on follow-up turns Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400 The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors * fix(responses): emit Claude's tool search output in stream order The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives. * fix(anthropic): keep the caller's tool names in tool search results A tool name Anthropic cannot take gets rewritten on the request, so Claude's tool_search_tool_result named the rewritten tool. The response now maps those tool references back to the caller's names, streamed and not, and a replayed search result maps them forward again on the next turn. * fix(responses): replay Claude's tool search only on routes that run it A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call * fix(responses): keep a replayed search when the caller sent Anthropic's own search tool A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it * fix(responses): replay a BM25 tool search under its own name * fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its search arguments, a stale search drops only the tool references the follow-up no longer sends, and the stored previous_response_id path keeps the base gate for routes the cost map does not flag * fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had * fix(responses): replay a stored Claude search only when the follow-up still sends a search tool A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level. handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors. * fix(responses): leave out a stored Claude search kept in server_tool_calls Main stores Claude's server-side tool search on the assistant message in provider_specific_fields.server_tool_calls and tool_results, not in tool_calls. The previous_response_id path only looked at tool_calls, so a follow-up that sent no search tool still replayed the stored search. The session handler now finds the search in either place and drops it from both server_tool_calls and tool_results when the follow-up cannot run a search. * test(responses): give the replayed search guardrail fixture its own call id The guardrail test reused toolu_01Weather, an id that streaming and bridge tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran first in the same worker, the guardrail write-back took its rebuild path and the test failed for a reason unrelated to the replayed search. * fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups A search-only assistant turn now gets its search result renamed to the sanitized tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on Claude routes replay the search as base did, trimmed to the tools still sent, and Bedrock invoke sends the tool search beta whenever the history holds a search * test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool * fix(anthropic): keep Anthropic's tool names in chat tool search results A chat history whose search result used the request's tool names (weather.get) got Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a rollback. Chat completions now returns Claude's search result as main does, and the Responses bridge maps a rewritten name back to the request's tool when it builds tool_search_output. That also fills tool_search_output on Bedrock invoke streams, which carry no name map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 17:13:40 +00:00
tavren27	d70855ca82	fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id (#50789) ... * fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id The detail lookup matched request_id OR litellm_call_id, which Postgres runs as a full table scan when the litellm_call_id index is missing or invalid. Read by request_id, the primary key, first and by litellm_call_id only on a miss, both under the spend log statement timeout. * test(proxy): type the call ID index test helpers * fix(proxy): limit the Logs drawer call ID lookup to start_date and answer 400 for a malformed date * fix(proxy): let spend log reads wait for a busy connection like a plain query * test(proxy): let the session pages read double take max_wait --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 17:03:21 +00:00
tavren27	1c68ec429a	fix(proxy): bill an OpenAI pass-through background response once across repeated reads (#50841) ... * fix(proxy): bill an OpenAI pass-through background response once across repeated reads A finished background response now bills only on the first pass-through call that sees it finished with usage. The claim is an insert on the new LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds across workers and replicas. Every later retrieve, cancel, streamed create, or resumed stream logs like an unbilled read. With no database or a failed insert the call still bills. * fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse * fix(proxy): claim background responses on failed streams and bound the claim A resumed stream that fails after response.completed billed its usage through the partial-usage path without claiming the response id, so a later read billed it again. That path now claims the id and logs $0 when an earlier read paid. The claim uses create_many(skip_duplicates=True) so a repeat read no longer raises a unique violation in Postgres, and it waits at most 2 s before billing, since the claim now runs before the client gets its response. * fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids * test(pass_through): cover a failed stream whose failure logging cannot be scheduled * fix(proxy): start the background response settlement migration with a lock timeout * fix(passthrough): leave a failed background stream's charge to its next read A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 07:33:25 -08:00
devin-ai-integration[bot]	52cd9ef0e4	fix(proxy): let the OS assign Prisma engine ports (#50850) ... * fix(proxy): let the OS assign Prisma engine ports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): clear Prisma connection state after failed startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 07:28:55 +00:00
ryan-crabbe-berri	87be092af7	fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer (#50736) ... * fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525, first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown Codex versions still get the catalog cut to 1 MiB, since they drop a larger one. * fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those clients counted as too old for the uncut catalog, their custom reasoning levels and stock entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does. * test(proxy): type the Codex catalog byte-limit test * refactor(proxy): gate the uncut Codex catalog on the 0.166 release line Read only the major and minor numbers of client_version instead of a shared semver comparison, and leave the reasoning level and stock entry version checks as they are on main. * refactor(proxy): read every Codex client_version with one helper Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version parser rejects. The catalog byte limit, the per-model minimum version and the custom reasoning level checks now share one reader for Codex's format, which also bounds digit counts so an absurd version can't fail the listing.	2026-11-10 07:10:30 +00:00
Mateo Wang	ac08eb5663	fix: make scheduled key, user, and team spend resets idempotent (#50298) ... * fix: make scheduled spend resets idempotent Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover reset roundoff with shared Redis workers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: isolate reset workers and type raw adapters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: keep updated_at moving on zero-spend budget resets * fix: never leave spend below zero after a budget reset * fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time * fix(proxy): keep a resent budget reset time only once it is due A save that resends the same budget_duration keeps the stored reset time only when that time has already passed, so the scheduled sweep still resets the period it is due for. A row that is not due yet realigns its reset time on save, the same as before. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 22:21:43 -08:00
Kerry Lu	3daeb8daf2	fix(transcription): stop billing a FLAC with no sample count as 576 trillion seconds (#50842) ... * fix(transcription): treat libsndfile's unknown frame count as no duration * refactor(transcription): inline the unknown frame count check * fix(transcription): read the frame count once to keep the type gate flat * test(transcription): annotate the new passthrough test's locals as Final --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-10 06:19:33 +00:00
Kerry Lu	64d84e557e	feat(budgets): add per-model tpm and rpm limit maps to budgets and customers (#50797) ... * feat(budgets): add per-model tpm and rpm limit maps to budgets and customers Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more). /budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and end_user_model_rpm_limit so the limiter can enforce them in a follow-up. * fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata, which the limiter reads, so the shared budget allowlist leaves those two fields to customers. Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy the type discipline gate. Customer info tests expect the two new budget keys. * fix(organizations): keep stripping budget_id and flattening nested per-model maps on update The update path pops every budget field from the organization row again, so a caller's budget_id never changes the attached budget, and a nested litellm_budget_table still flattens the two maps into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS. * fix(types): keep basedpyright and ruff strict counts flat for the new budget fields Reads the per-model maps off the end-user params with membership checks instead of .get on an untyped dict, writes the inline customer budget through the client's typed jsonify_object, and builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never receive object-typed kwargs. * chore: drop unrelated reformatting from project endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 22:15:18 -08:00
tavren27	96b916028a	fix(proxy)!: return managed file ids from rag ingest and use team credentials on vector store file routes (#50459) ... * fix(proxy): give rag ingest managed file ids and team credentials on vector store file routes RAG ingest through a team's OpenAI deployment now registers the uploaded file as a LiteLLM managed file and returns its managed id, so GET /v1/files lists it for the caller. Re-ingesting a managed id attaches the provider file and returns the caller's managed id Model-less vector store file calls (create, list, retrieve, content, update, delete) from a team key now use the team's OpenAI deployment that ingest uploads through, via the router, and fall back to the direct path when the key may not use one. Explicit api_key, api_base, litellm_credential_name, and forwarded provider keys still win, and blank values count as absent * fix(proxy): route managed vector store file ids to their team deployment and keep missing-file 404s off the router Model-less team vector store file calls now go straight to the provider with the team deployment's credentials, so 404s on missing files no longer put the team deployment in cooldown. A managed file id on the vector store file routes now uses the deployment named by its llm_output_file_model_id when that deployment is proxy-wide or belongs to the caller's team, and falls back to target_model_names otherwise. * fix(rag): keep managed ids and team checks on every file and search path Ingest no longer registers a raw file id when file_url is empty, a managed file id on a proxy without the managed files hook answers 400, and vector store search and /v1/rag/query hits show the managed id ingest returned. Model-less vector store file calls use the team deployment only after the key, team, and member model checks pass, fall through to the router when the team has no OpenAI deployment, and log the deployment they used. * fix(rag): keep provider ids on lookup failures and mint managed ids only for keys that can use them A database error while mapping search hits to managed ids keeps OpenAI's ids instead of failing a finished search. Ingest registers a managed id only when the caller passes the team deployment's model checks, so a model-limited key keeps the raw id it can still use, and the file routes share that check. * fix(rag): attach the managed file copy that matches the ingest credentials * fix(vector_store_files): route a managed file id to the caller team's copy * fix(vector_store_files): keep the upload copy for a store registered with its own credentials * fix(vector_store_files): use a managed file's team copy only when the store holds it * fix(vector_store_files): attach a managed file's team copy and keep named credential options out of the store check * fix(vector_store_files): attach the team copy only to a store that routes to the team deployment * fix(vector_store_files): keep a managed store's routing and check the upload copy through its own deployment * fix(vector_store_files): route a managed file to its upload deployment only when the key may call it A managed file id names its upload deployment, and with require_managed_files off that id is not checked against an ownership record. The routes now use that deployment only when the caller may call its public model, else they keep target_model_names routing as on main. * fix(vector_store_files): log a pinned managed file call under its deployment's public model name * fix(vector_store_files): send a pinned managed file call's deployment key only to that deployment's base * fix(vector_store_files): keep stored credential headers intact and fail the store check on errors other than 404 * fix(vector_store_files): answer a failed store check with the provider status and copy credentials read by name * fix(vector_store_files): send a call to the upload copy when the store check fails without a 404 A store check error other than a 404 now leaves the call on the upload copy, the way main always routed it, so the real call keeps the router's retries, the provider's status, and its failure log instead of failing on one transient error * fix(vector_store_files): keep a credential's no-log key out of the store check A named credential carrying a no-log key made the store check call aretrieve with no-log twice, so every managed id call through that deployment answered 500 where main answered 200 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 05:44:28 +00:00
norkelbyte	68e78bc53a	fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted (#50730) ... * fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted * fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full. * fix(responses): finish a turn's early token count when the client leaves A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors * fix(router): count one router request per Responses WebSocket turn A socket whose finished turns each counted their own request in router usage no longer adds one more request when it closes, so N turns count N requests, as N HTTP calls do. A socket with no counted turn still counts 1 * fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 04:13:51 +00:00
paveln83	69b0f1deac	feat(proxy): log keys and teams still on the deprecated empty models defaults at boot (#50720) ... * feat(proxy): log keys and teams still on the deprecated empty models defaults at boot * fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup * fix(proxy): say which empty-models rows keep access in 2.0.0 Teamless keys listing all-team-models get no models in 2.0.0, so the boot warning no longer says every counted row keeps its access. It names each upgrade outcome only when its kind is in use, says the counts include blocked and expired rows, and words each kind by what the setting means rather than what it grants * fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning The warning no longer says the upgrade keeps access for empty models lists, since no shipped code or public page backs that yet. It says 2.0.0 removes the defaults and how to keep today's access, and keeps the advice for teamless all-team-models keys. The integration test drops an empty-database assertion that ran before the check finished * fix(proxy): leave rows with access groups out of the empty models boot warning * fix(proxy): count access group rows, skip management and read-only keys in the empty models warning --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 04:05:25 +00:00
ryan-crabbe-berri	e099925923	fix(proxy): list images in the Codex catalog only for models known to read them (#50738) ... * fix(proxy): list images in the Codex catalog only for models known to read them A translated Codex entry used to offer image input whenever no deployment declared supports_vision, so Codex let users attach images to models that may not read them. Image input now goes out only when every deployment behind the name says it reads images. * test(proxy): type the vision combination test parameters * refactor(proxy): rename the vision helper and drop the extra vision docstring sentence	2026-11-10 03:46:37 +00:00
kenneth-berri	7ccc0e0fff	feat(usage): split customer spend by the key's current team (#50764) ... * feat: add customer spend breakdown and filtering by current team Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: recover legacy customer team spend and tolerate lookup outages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf: resolve customer team keys once per daily activity query Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: paginate customer team activity in SQL Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 19:35:31 -08:00
ryan-crabbe-berri	16fad64e95	fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments (#50743) ... * fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments Codex builds separate catalogs for Bedrock's runtime and Mantle endpoints that turn off what Bedrock does not support yet. A model whose deployments run on Bedrock now takes its entry from those catalogs, vendored from Codex 0.166.0, instead of from Codex's OpenAI catalog. * fix(proxy): keep Bedrock catalog entries from Codex releases that cannot decode them codex debug models leaves out fields such as supports_parallel_tool_calls that Codex releases before 0.148.0 require, so a Bedrock entry now needs at least 0.148.0 as well as its source entry's minimum.	2026-11-09 19:00:22 -08:00
Mateo Wang	71a9fcb56a	fix(sso): read the SSO client secret before the refresh-token DB check (#50773) ... * fix(sso): read the SSO client secret before the refresh-token DB check so a concurrent env sync cannot split the client id from its secret Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(sso): type the secret-race test and read the provider from the IdP boundary mock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 18:54:15 -08:00
Kerry Lu	a56a27deb6	fix(prometheus): record litellm_overhead_latency_metric for streams, stamp plus the proxy's first-chunk work (#49920) ... * fix(prometheus): record litellm_overhead_latency_metric for streams and count the proxy's per-chunk work in it * fix(proxy): sum the per-chunk stream work through Logging.add_streaming_overhead so the lint gates hold * fix(proxy): count the per-chunk work of chat completions streams too, and drop a comment that repeated the code * fix(prometheus): keep the handoff overhead stamp when a /v1/responses stream finishes The Responses streaming iterator calls update_response_metadata again at stream end with include_overhead=False, which replaced the timing stamped at handoff with one that had no overhead, so litellm_overhead_latency_metric still had no sample for streamed /v1/responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type the parameters of the per-chunk overhead tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prometheus): add only the proxy's first-chunk work to a stream's overhead, not a sum over every chunk A 1 ms hook on each of 500 chunks delays the client by 1 ms per chunk, not 500 ms, so the overhead metric keeps the stamp plus the work on the first chunk and stops summing later ones Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive the first-chunk overhead tests with a fake clock instead of real sleeps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): a keepalive ping is not a stream's first chunk, and test the overhead with a real ProxyLogging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 18:45:00 -08:00
yujonglee	523b3efea2	refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) ... * refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:37:35 +00:00
devin-ai-integration[bot]	13df479397	fix(proxy): write SpendLogs failure rows for config-defined pass-through endpoints (#41831) ... * fix(proxy): write SpendLogs failure rows for config-defined pass-through endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): simplify pass-through failure api_base lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): exempt pass-through spend log regression from quality gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse failure payload builder in pass-through exception path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover config-defined pass-through failure spend rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover failure spend route eligibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check configured routes for pass-through failure logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: kenneth <kenneth@berri.ai>	2026-11-09 17:05:10 -08:00
Nate Armstrong	1ddc913002	fix(messages): log native messages success once, after the proxy releases it (#50420) ... * fix(messages): log native messages success once, after the proxy releases it The proxy defers non-stream success logging past post-call guardrails, but the native host parked its log in its own slot and the release only ran for OCR, so a unary /v1/messages call served by Rust logged no success event, no SpendLogs row and no key, team, user or end-user spend. A deferred native success now parks in enqueue_deferred_logging with deferred_success_result, the same slots Python uses, so the proxy releases it like any other call and its guardrail step pruning and cost preparation see it too Python runs sync success callbacks inside the deferred logging closure, so a post-call hook that blocks the response drops them. A parked native success now carries its sync callbacks and runs them on release A refused first attempt and its fallback share one logger. Python logs the async success once, for the first attempt, so the native adapter now claims the async success log once across fallback attempts. Like Python, an attempt that loses the claim runs its sync callbacks only when logging is not deferred, and an internal call logs nothing * test(messages): cover the deferred native success park and release	2026-11-09 16:06:53 -08:00
jesus-berri	c76a95fc7e	feat(proxy): let a key's logging entry reference a proxy credential (#50572) ... * feat(proxy): let a key's logging entry reference a proxy credential A virtual key's metadata.logging entry can name a proxy credential with litellm_credential_name. At request time the credential's values, which config credential_list resolves through the secret manager at boot, are merged under the entry's callback_vars, with the entry's own values winning. An entry whose credential is not defined is dropped, so it never falls back to the proxy's own callback environment. Only proxy admins may change the logging entries of a key when the stored or requested list holds a reference (403), and a reference to an undefined credential is a 400. The check runs on key generate, update, team bulk update, regenerate and /user/new, and compares decrypted entries so re-saving metadata read back from /key/info is not a change. Resolves LIT-10061 * fix(proxy): drop a key logging entry whose referenced credential holds no logging values * fix(proxy): only require newly added credential references to exist on key writes * fix(proxy): log a dropped credential reference as an error so /key/health reports it * fix(proxy): gate credential references on bulk-created user keys * fix(proxy): keep failing requests on keys with non-object logging metadata A key whose stored metadata.logging is not a list of objects failed every request with a 500 before this branch. The credential reference rewrite turned those values into no entries, so the request quietly succeeded without the key's logging. Raise a TypeError for those shapes again so the request path behaves as it did on main, while empty values and objects with unknown fields still go through the existing skip. * test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging The malformed logging cell expects the base behavior that 7ac4ab1356 restored, a 500 for a key whose stored metadata.logging is not a list of objects and a 200 for an empty list * fix(proxy): keep a 500 for teams with non-object logging metadata On main, a team whose stored metadata.logging held non-object items failed chat requests with a 500, because AddTeamCallback(**item) raised a TypeError. This branch switched to AddTeamCallback.model_validate, whose before-validator raises AttributeError on those items, and the proxy maps AttributeError to a 400 invalid_request_error. Raise a TypeError for non-mapping items before validation so the status matches main again, as the key path already does.	2026-11-09 16:02:46 -08:00
joshua-berri	35386e3f4a	fix(complexity_router): keep Claude Code subagent and side-request traffic off other agents' tier pins (#50741) ... * fix(complexity_router): keep Claude Code subagent and side-request traffic off other agents' tier pins Claude Code sends one x-claude-code-session-id for the main agent, every subagent and their side requests. In user_turn mode a subagent's new ask rewrote the parent's pin (#50099), and the agent progress summaries Claude Code fires during a subagent run rewrote the subagent's pin to the cheapest tier (#50444). Subagent requests now get their own tier pin keyed on session id plus x-claude-code-agent-id. Requests marked x-claude-code-request-class: auxiliary route as housekeeping, and only main and subagent turns may write a pin. Without the hint header, the summary and prompt-suggestion prompts are matched as housekeeping sentinels. * chore(ui): regenerate schema.d.ts for the housekeeping_patterns description * fix(complexity_router): let the request-class hint only mark housekeeping, not suppress pin writes A client-sent request class must not decide whether a decision is saved. Tagging a real ask as workflow or auxiliary dropped its routing decision, so later continuations replayed an older pin with the new history. The auxiliary hint now only routes the request through the housekeeping path, which is already never pinned and defers to custom classifiers and keyword rules. * test(complexity_router): cover the auxiliary hint on a prompt no sentinel matches --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-11-09 15:52:38 -08:00
Kerry Lu	92c2e69ac5	fix(proxy): fail-closed budget enforcement returns 403 naming /team/member_add for a team user with no membership row (#50603) ... * fix(proxy): read an absent team membership row as zero spend under fail-closed budget enforcement A team user under a default member budget has no LiteLLM_TeamMembership row until spend is persisted. SpendCounterReseed.from_db returned None for that absent row, the same value as a failed DB read, so fail_closed_budget_enforcement rejected every request from that user with 503. The DB answering that the row is absent is a verified zero, so from_db now returns 0.0 for a team-member counter without a row and still returns None when the read fails. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only read an absent membership row as zero when the team member key parses unambiguously A team_id or user_id containing ":" makes the counter key split onto the wrong membership row, so that miss keeps returning None instead of a verified zero. Type the new unit tests. * fix(proxy): refuse a fail-closed team member with no membership row with a 403 that names /team/member_add Replaces the earlier reading of an absent LiteLLM_TeamMembership row as verified zero spend. A key minted with team_id and user_id but no /team/member_add has no row and the spend writer never creates one, so admitting it at zero would leave its member spend tracked only in Redis. With fail_closed_budget_enforcement on, _check_team_member_budget now rejects that key with a 403 that names the user, the team and the member_add call, before any spend read. With the flag off nothing changes. The membership load raises on a database failure and returns None only for a genuinely absent row, so the 403 never fires during an outage. spend_counter_reseed.py and the counter prefix constant go back to main. * test(proxy): drive the no-membership 403 tests through the real membership and spend readers Fake Postgres rows and the FakeSpendCounters double stand in for the external stores instead of patching get_team_membership and get_current_spend. Strip CR/LF from the ids in the fail-closed warning log. * fix(proxy): read fail_closed_budget_enforcement from general_settings in the membership check Importing proxy_server's private helper grew reportPrivateUsage past the type gate ceiling. Type the integration helper's return as JsonValue. * fix(proxy): expose fail_closed_budget_enforcement_enabled for the membership check Reading general_settings directly from auth_checks grew the unknown-type counts past the type gate ceiling; a small public wrapper in proxy_server keeps the gate flat. * test(proxy): drop docstrings that restate the no-membership tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 15:38:59 -08:00
paveln83	0021d5493c	fix(bedrock): answer 400, not 500, for malformed count-tokens requests (#50708) ... * fix(bedrock): answer 400, not 500, for malformed native count-tokens requests The Bedrock count-tokens handler now checks the body before any AWS call and raises a 400 BedrockError for a missing or empty messages list, a message without role or content, or a missing model. The token counter marks that rejection as local, so it skips the Mantle fallback and the framing estimate, and /v1/messages/count_tokens answers invalid_request_error get_bedrock_base_model returns a bare cross-region prefix such as us or global unchanged instead of raising IndexError, so Bedrock judges the model id and its own 400 reaches the caller * fix(bedrock): answer 400 for a malformed message on Mantle token counts * fix(bedrock): limit the region-only model id fix to token counting * fix(bedrock): count a routing-prefixed region-only id and leave other providers' count errors as they were --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 23:04:36 +00:00
Krrish Dholakia	70f399b8c5	feat(auth): grant access groups through principal attribute rules (#50455) ... * feat(auth): grant access groups through principal attribute rules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): satisfy cache tracing and recursion CI checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): close access group rule authorization bypasses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): reject self-editable identity rule attributes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Centralize JWT model authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Cover centralized key checks and name denied JWT models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Test centralized authorization for custom auth and JWT mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Preserve model normalization before budget validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Use a controlled clock for WebSocket session deadline tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Unify JWT credential admission and simplify team selection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Simplify request-time access group authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Use real requests in custom auth budget fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 13:59:40 -08:00
zoveki	abe3b11f7a	fix(proxy): show committed spend in the remaining budget gauges (#49199) ... * fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 20:05:37 +00:00
yucheng-berri	457365f627	fix(proxy): build plugin session claims from FIPS provider services in fips_mode (#50544) ... * fix(proxy): build plugin session claims from FIPS provider services in fips_mode With general_settings.fips_mode on, the per-plugin key comes from the salt key KBKDF and the Fernet token is built from ssl.RAND_bytes, AES-128-CBC and hmac.digest. Standard mode is unchanged. * test(proxy): freeze the clock in the fips_mode plugin claim test	2026-11-09 11:07:57 -08:00
quietjules	001126cd1c	feat(proxy): add OpenSearch index pass-through with per-key read/write grants (#50573) ... * feat(proxy): add OpenSearch index pass-through with per-key read/write grants Adds /opensearch/<index>/<api> for LiteLLM vector store indexes on OpenSearch stores. A key or team needs read or write on the index in allowed_vector_store_indexes, and the proxy signs each call with the vector store's AWS identity, so callers never hold AWS credentials The route maps the LiteLLM index name to the store's provider index, refuses any path, query parameter, or body field that could reach another index, and sends the exact signed bytes upstream with only the signed headers * fix(proxy): let OpenSearch _bulk and _msearch NDJSON bodies through auth opensearch-py sends _bulk and _msearch NDJSON with content-type application/json, and the auth-time body read parsed it as one JSON document and answered 400 before the route ran. Auth now skips parsing OpenSearch NDJSON bodies the same way it skips Azure Speech audio. * feat(proxy): accept a _bulk _index naming the path index and allow named search pipelines A _bulk action line may name the index in the URL path, and LiteLLM rewrites it to the real index, which is what opensearch-py's helpers.bulk sends. The search_pipeline query param now goes through so hybrid search works, while an inline pipeline stays refused. Also types the type-checking Router stub with _router_init's parameters again, which make lint's tests/e2e check needs since the Rust core change. * fix(opensearch): answer 401 when the AWS credential source cannot read its token file * refactor(proxy): check OpenSearch request bodies without recursive helpers * fix(opensearch): answer 401 when AWS credentials fail to refresh at signing time * fix(opensearch): keep auth off OpenSearch bodies and widen the read list to index checks Auth no longer parses any OpenSearch request body, so a document field named model, api_base or base_url is written instead of answering 401 or 403. GET and HEAD on the index and HEAD on _doc and _source are reads, each operation checks the exact grant the list names, an index whose store name two stores share answers 400, and an _msearch header that names the path index is sent to the mapped index * fix(opensearch): need read for writes that return documents and keep document fields out of LiteLLM params * fix(opensearch): keep documents out of LiteLLM hooks, keep main's credential statuses, make _msearch admin only, and bill only searches * fix(proxy): bill only an exact /opensearch/<index>/_search path A document read such as /opensearch/docs/_doc/_search ended in /_search and was counted as a billable search. The check now matches the index segment and _search exactly * fix(proxy): only proxy admins set vector store index grants, and bill no OpenSearch document call An index grant now opens document writes through /opensearch, so key generate, update, bulk update, regenerate, team new and update, /user/new, and bulk user creation answer 403 when a non-admin changes allowed_vector_store_indexes, top-level or under metadata. Resending the stored grants is not a change An /opensearch path is matched before the LLM route suffixes, so a document id such as search or embeddings never makes a document call count as a billable request * fix(proxy): treat an empty index grant list from a non-admin as a change An explicit empty top-level allowed_vector_store_indexes passed the admin-only check as unchanged and then replaced the stored grants. It now counts as a change, so a non-admin gets 403. The new grant tests also carry parameter types --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 18:54:20 +00:00
tavren27	2f430873ae	fix(bedrock): derive the web identity session policy per AWS caller (#50592) ... * fix(bedrock): derive the web identity session policy per AWS caller Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session. * fix(bedrock): let async invoke write its S3 output under the web identity session policy Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com). * fix(opensearch): give OpenSearch vector store search its own web identity session policy * fix(bedrock): give every remaining AWS caller its own web identity session policy Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes * test(batches): expect config export to skip every underscore attribute The session policy class attribute is private, so batches config export no longer returns single-underscore names * fix(bedrock): scope knowledge base ingestion role grants to the roles it creates Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included * fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies * fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through, or realtime The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions, document content, and ingested document ACLs, and a test derives that list from botocore Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and embeddings never open a bidirectional stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 19:13:53 +00:00
paveln83	141a580951	fix(proxy)!: price expanded wildcard /model/info rows as the model their calls route to (#49128) ... * fix(proxy): price expanded wildcard /model/info rows as the model their calls route to * fix(proxy): read wildcard routing through the router registry entry points * fix(proxy): price a wildcard row only when the router sends its name there Use the router's own route candidates for the listed name, so an alias, a routing group, a more specific pattern, or a team's own deployment keeps a wildcard row unpriced * fix(proxy): show no price on a wildcard row whose calls go to another deployment * fix(proxy): keep the listed price on catch-all wildcard rows and match the edit path to the read path * fix(proxy): price a wildcard row aliased to a sibling name as the alias target * fix(proxy): price wildcard rows through the caller's alias renames and show nothing for rows another deployment takes * fix(proxy): keep same-model wildcard row prices, skip deleted team aliases, use the caller's team * fix(proxy): keep live team aliases when skipping deleted team alias targets * fix(proxy): show no wildcard row price when the same-model deployment serving it sets custom rates * fix(model_info): recognize a replayed wildcard row on a worker that has not loaded the wildcard * test(proxy): feed wildcard discovery real provider data in model info tests * fix(proxy): price a wildcard row another deployment serves as the model its calls bill A row whose calls reach another deployment now shows that deployment's model and limits, and goes blank only when those deployments bill more than one model or declare their own rate. A name no route serves keeps its own price, and a replayed row stores none of the metadata it showed. Drop the router's per-call scan of every deployment for a team key, since the team and name indexes already cover those names. * fix(proxy): price a wildcard row aliased to an aliased group as the group its calls reach An alias target that is itself aliased onward listed the onward group's deployments too, so the row showed no price while calls billed the first target's model. The billed model now comes only from the route candidates * fix(proxy): price a wildcard row billed by another deployment without the wildcard's base_model A row aliased to another deployment priced as that deployment's model, but the lookup still swapped in the source wildcard's base_model, so the row showed a rate its calls never bill --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 19:08:22 +00:00
zoveki	cfddaafe18	fix(cli): end the local login when lite logout cannot check the OS keychain (#50098) ... * fix(cli): end the local login when logout cannot check the OS keychain With no usable keyring, lite logout kept a secret-free token.json as a note that the keychain went unchecked. Every reader took that note for a live keychain-backed login, so lite whoami, lite auth print-token, and the bare lite shell all reported a login the user had just ended, and a keychain that answered again could pair the note with an earlier login's entry and sign the machine back in. Logout now removes token.json whatever the keychain answers. When the keychain could not be checked and there was a login to end, it records that in a separate owner-only keychain_unchecked file beside it, written after the removal so a full disk still has room for it. A repeat logout still warns while that file is there, and a logout that confirms the keychain is clean removes it. * fix(cli): keep warning after a logout the keychain refused to clear A keychain that will not give up its entry gets the same note as one that could not be checked, so a later logout made while the keychain is out of reach does not report a clean logout while the credential is still there. * fix(cli): revoke the login a logout left in an unreachable keychain once it answers A logout made while the keychain was locked ended the local login and dropped the metadata that pairs with the refresh token still in the keychain, so the next logout had nothing to revoke. The not-cleared note now keeps that record without its secret, and a later logout pairs it with the keychain entry and revokes it before erasing. * fix(cli): revoke a locked-keychain login on the next login, and only once after a refused erase A login made after a logout that could not reach the keychain now revokes the login it replaces, read from the keychain_not_cleared note, as the merge base did through the kept token.json. A keychain that refused the erase had already handed that logout the refresh token, so its note no longer pairs with the entry and the next logout does not revoke the same token again. * fix(cli): keep the first login revocable after a second locked-keychain logout A logout whose login kept its secret in token.json never reached the keychain, so it now leaves the note an earlier logout wrote in place, or writes an empty one, instead of overwriting it. The first login's refresh token stays revocable once the keychain answers, and a keychain-less machine no longer keeps user and team ids in the note. Also documents the limits that remain in the CLI README and drops the docstrings from the tests this PR added. * fix(cli): revoke the keychain's login after a login made while the keychain was locked * fix(cli): stop lite from reading the stored login before login and logout The cli group resolved the stored API key before every command, and that read moved a locked-keychain login's secret over the keychain entry the logout note names, so logout and login never saw the first login to revoke it. * fix(cli): keep the login a locked logout left in the keychain revocable While the keychain still holds the login an earlier logout could not end, reading the stored login now serves the secret token.json kept and leaves that keychain entry alone, whatever the login times say. A clock that stepped back no longer serves the ended login, and commands other than logout and login no longer move the newer secret over it. A silent renewal that writes over the entry revokes that login on the proxy. * fix(cli): keep a login revocable when its note cannot be written, and never block logout on an earlier login's proxy A locked logout whose note cannot take the login's record now puts token.json back, so the next logout that reaches the keychain still revokes that login, as the base did. A logout that ends both a current login and an earlier one left in the keychain warns when the earlier login's proxy cannot record the revocation, as lite login does, instead of refusing to log out of the current proxy. * fix(cli): move a locked keychain login's token file into its note instead of putting it back * fix(cli): never tell a logout to delete the record of a keychain login it could not move aside A read-only ~/.litellm made the locked-keychain logout say to delete token.json, which would leave the keychain's login unrevoked. It now says the logout did not happen and asks to make the directory writable. The README also says an older lite cannot finish a logout this release started. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 11:17:59 +00:00
joshua-berri	43112547fb	feat(proxy): total_timeout bounds a whole non-streaming request (#50496) ... * feat(proxy): total_timeout bounds a whole non-streaming request A `total_timeout` body field or `x-litellm-total-timeout` header (the header wins) gives a non-streaming /v1/chat/completions, /v1/messages or /v1/responses request one budget for preparation, retries, moderation and post-call processing. Past it the proxy answers 408 and cancels the upstream call, and a retry whose backoff cannot fit ends early with the error it would have retried. A body value on a streaming or background request is refused with a 400, a header value leaves it unbounded. A timed out request is billed like a client disconnect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): ignore any total_timeout header on requests it cannot bound, and never interrupt pre-call admission A globally configured x-litellm-total-timeout that is malformed no longer fails the streaming and background requests it is ignored on. The deadline holds while pre-call admission runs, since admission reserves shared quota before it records it, and an expired deadline then ends the request before dispatch, where the failure path hands the reservation back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): a total_timeout cuts slow pre-call hooks short, holding only the rate limit reservation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 00:43:50 -08:00
Mateo Wang	7ea4028625	fix(proxy): align ownership and scope handling (#50600)	2026-11-08 07:03:36 +00:00
dingdangmao	c61ec51108	fix(key/info): return 400 instead of 500 when no key is provided (#43577) ... * fix(key/info): return 400 instead of 500 when no key is provided When GET /key/info is called without a ?key= query parameter and the authenticated principal has no virtual key (e.g. an admin-UI session token whose UserAPIKeyAuth.api_key is None), info_key_fn passed hashed_key=None to Prisma find_unique(where={"token": None}), which raised MissingRequiredValueError and surfaced as an unhandled 500. Add an explicit None/empty guard that raises a clean 400 ProxyException before the DB lookup, and add a regression test asserting that prisma find_unique is never called with token=None. Fixes #43571 Signed-off-by: apex-mochen <2756823972@qq.com> * test: remove redundant assertion comment * fix(key/info): remove redundant None check to satisfy basedpyright delta gate After if not key: raise, key is narrowed to str. The subsequent if key is not None: guard is always true and triggers basedpyright reportUnnecessaryComparison, which fails the delta gate. Collapse to a single hash_token_if_needed(token=key) call. Signed-off-by: apex-mochen <2756823972@qq.com> --------- Signed-off-by: apex-mochen <2756823972@qq.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-11-08 02:49:34 +00:00
joshua-berri	82e43cd55a	fix(accounting): stop delivery metrics from scanning retained receipts on every flush (#50565) ... * fix(accounting): stop delivery metrics from scanning retained receipts on every flush The backlog query now reads only pending and parked deliveries, so its cost follows the backlog instead of retained receipt history. Batch count and table size are measured once per check interval per pod, and the storage and delayed-delivery alerts repeat at most once per interval instead of every flush Fixes #50475 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove the storage query reruns only when its interval is due Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read the check interval on each flush so YAML environment_variables apply The writer is built at proxy import, before the config's environment_variables load, so reading the interval there ignored values set in config.yaml Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(router): wait for router accounting instead of sleeping in user-config limit tests Each request ended with a 0.3s sleep and hoped the router's usage writes had landed, so under CI load a concurrent request could read counters mid-update. The tests now flush the logging worker and wait for every task the requests scheduled, and a retired router's counter flush lands before the next router reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:21:32 -08:00
Kerry Lu	07c7229895	feat(proxy): per-request header opts a stream into usage.cost without the global include_cost_in_streaming_usage flag (#50574) ... * feat(proxy): let one streaming request opt into usage.cost with x-litellm-include-cost-in-streaming-usage The global include_cost_in_streaming_usage flag moved every stream off the fast path. A request can now send x-litellm-include-cost-in-streaming-usage: true to get usage.cost on its usage-carrying stream event (/v1/messages and the Anthropic and OpenAI pass-throughs) while requests without the header keep the fast path. The global flag keeps working unchanged. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: type the new stream cost header tests * fix(types): keep reportUnknownArgumentType at its merge-base count * fix(proxy): the stream cost header only accepts true --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 16:57:20 -08:00
yuneng-jiang	c024a4d2e3	test(proxy): delete 28 dead skipped tests in test_key_generate_prisma.py (#50550) ... 57 of the 62 tests in this file are hard-skipped because they need a live Prisma DB, so they never run. This removes the 20 whose behavior a running integration or e2e test already asserts, and the 8 that assert nothing meaningful or can never pass. The imports and the one helper only they used go with them. The other 29 skipped tests stay as a list of behavior still to port into tests/integration.	2026-11-07 16:54:23 -08:00
Kerry Lu	51b49973c9	fix(proxy): hold DB_RECONCILE_LOCK around credential create, update and delete (#50551) ... * fix(proxy): hold MODEL_RECONCILE_LOCK around credential create, update and delete The config reload snapshots the credential table, awaits config rows, then prunes and upserts litellm.credential_list from that snapshot. A credential write landing inside that window was undone in memory until the next reload. The three endpoints now take the same lock the reload holds. Fixes #50273 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): take MODEL_RECONCILE_LOCK before the credential update reads its row Two PATCHes queued behind a reload read the same old row and the second overwrote the first. The relay now keeps enough bytes to spot a split after marker, and the new unit tests are fully typed. * test(credentials): annotate the reconcile lock test locals with Final * refactor(proxy): rename MODEL_RECONCILE_LOCK to DB_RECONCILE_LOCK The lock gates every db-to-memory reconcile and now the credential writes too, so the model-only name undersold what it protects. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 16:02:44 -08:00
yucheng-berri	b603e84c02	fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF (#50032) ... * fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF With OpenSSL enforcing FIPS, hashlib.scrypt raises, so every MCP session token and bridge envelope request failed with a 500. Under FIPS the keys now come from LITELLM_SALT_KEY through SP 800-108r1 KBKDF run by the provider's EVP_KDF. Without FIPS the scrypt keys are unchanged * fix(mcp): keep master key rotation invalidating MCP tokens under FIPS Pass the master key as the KBKDF context, so rotating it changes the three FIPS-mode keys as it changes the scrypt keys * test(mcp): annotate new FIPS key test locals with Final * fix(mcp): derive the FIPS MCP keys only when general_settings.fips_mode is on On a FIPS OpenSSL 1.1.1 host (RHEL 8) scrypt still works, so keying the switch on openssl_enforces_fips() sent those hosts to the OpenSSL 3 only KBKDF and broke MCP sessions and bridge envelopes that worked on main. fips_mode_active() is the same reader B1 adds, set by enforce_fips_mode at startup. * refactor(fips): move the salt key KBKDF into encrypt_decrypt_utils The KBKDF helper derives keys from LITELLM_SALT_KEY, so it now sits next to the other salt key helpers, and fips.py goes back to main's version. * Pin the MCP key KBKDF fetch to the FIPS provider Fetch KBKDF with the property query fips=yes, so a change to the process's default properties after boot can't move the derivation to a non-FIPS provider. The shared fips_openssl test fixture drops the pin, since the test OpenSSL has no FIPS provider.	2026-11-07 13:33:27 -08:00
Yogeshwaran Ravichandran	8d9ddb349e	feat(guardrails): add prebuilt credential patterns for OpenAI, Anthropic, and Gemini (#50268) ... * feat(guardrails): add prebuilt credential patterns for JWT, and OpenAI, Anthropic, Gemini API keys * fix(guardrails): prevent quadratic backtracking in jwt_token pattern * test(guardrails): move credential pattern tests into test_patterns.py * feat(guardrails): remove jwt_token to keep credential patterns strictly bounded * fix(guardrails): mask whole provider keys and keep openai_api_key to OpenAI's key shapes End each pattern so a key ending in "-" is masked in full, require OpenAI's T3BlbkFJ marker on bare sk- keys so other vendors' sk- keys and LiteLLM virtual keys are left alone, add sk-None- user keys, and replace the registration-only tests with whole-match, length-boundary, and negative cases * fix(guardrails): require the T3BlbkFJ marker on every OpenAI key form and match keys after escaped separators openai_api_key now needs the marker on the sk-proj-, sk-svcacct-, sk-admin- and sk-None- forms too, as gitleaks and TruffleHog do. anthropic_api_key and gemini_api_key also start a match right after a literal \n or \t or a percent-encoded byte, so keys in pasted JSON strings and URL-encoded logs are masked. * fix(guardrails): bound the OpenAI key regex, match Gemini prefixes case-sensitively, and hold back long keys in streamed replies * fix(guardrails): size the streaming holdback per pattern, stop OpenAI key tails before the next key, and allow non-ASCII text after a key * fix(guardrails): mask content filter pattern matches found in the original text so overlapping patterns cannot split a key * fix(guardrails): let a block pattern see the text after earlier masking patterns, as before Masking patterns still match the original text and merge overlapping matches into one tag, but a BLOCK pattern checks the text with the masking patterns listed before it applied, so a request main masked and passed is not blocked now. Detections name only the patterns whose tag was written. * fix(guardrails): keep content filter masks in order and hold back the longest OpenAI key Masking patterns run in configured order on the masked text as on main, so a contextual pattern still matches next to an earlier tag, and each also matches the original text so an earlier tag cannot split a key. Detections follow the configured pattern, not its name, and the OpenAI streaming holdback covers the pattern's 419-character longest match. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-07 12:29:12 -08:00
yujonglee	574fcea691	fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) ... * fix(redis): share one native circuit breaker between RedisCache and the Rust stores RedisCache and the native Redis client each kept their own breaker, so once the cache learned the coordination Redis was wedged, a spend-counter write through the native store still paid a full socket timeout. Every Redis target now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected into both the RedisCache guard and the native client. The Python FSM is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(rust): cargo fmt the shared breaker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(redis): close the breaker when the recovery probe gets a Redis reply error A command error such as WRONGTYPE means Redis answered, but the probe was left unsettled and its drop reopened the breaker for another recovery timeout. Both the RedisCache guard and the native client now settle such a probe as a success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw. * fix(redis): key the shared breaker by server and close the three audit gaps The breaker registry now keys on the Redis server (host and port, unix socket, cluster node set or sentinel service) instead of the full connection target, so caches that differ only in db, credentials or url spelling share one breaker. A READONLY reply from a replica mid-failover no longer counts as Redis answering, so it can't reset a run of failures. A new RedisCache refreshes the breaker state gauge, which the deleted Python breaker did at construction. A shared native client is only reused by a store that passes the same breaker. * refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations lib.rs now only declares modules and re-exports, with resolution, target, tls, auth and endpoint each in their own file. Derived FromPyObject already reads a struct field as the attribute of the same name, so #[pyo3(attribute)] on the breaker and store settings did nothing. * refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot Derived FromPyObject already reads each field as the attribute of the same name. The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in the same order. * refactor(redis): return a HashMap from the three memory-tier snapshots The snapshots read cache-memory's HashMap, so there is no order to keep. PyO3 builds the same dict from a std HashMap, which replaces the hand-written PyDict loops in the spend counter and router usage stores and the IndexMap in the cooldown store. The rate-limit slot registry keeps IndexMap because it hands back a Python dict in the order Python wrote it. * refactor(redis): share Redis breakers and clients through one RedisHosts per host redis-connect's RedisHosts now owns the per-server breaker and the shared clients that python-bridge kept in two process-wide statics. A client takes breaker settings and admits through its server's breaker, so no caller can pair a client with a breaker of its own, and clients are keyed by Resolution alone. This drops RedisClient::guarded_by and Breaker::same. python-bridge keeps one RedisHosts per process and calls it with the GIL released. The gateway builds one per connect, so its coordination and response cache on the same server now share a breaker. RedisStoreSettings.breaker is back to breaker settings, as on main. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 17:50:06 +09:00
quietjules	210e2a075a	fix(vertex_ai): authenticate Vertex AI pass-through requests once (#48942) ... * test(vertex_ai): run auth once per Vertex pass-through request Drive /vertex_ai, /vertex-ai and /vertex_ai/discovery requests through the real proxy app with a counting custom_auth, and assert each request authenticates once and is served as the caller that auth returned. The generate-content routes fail today because the route authenticates a second time. * fix(vertex_ai): authenticate Vertex pass-through requests once vertex_proxy_route already resolves the caller through Depends(user_api_key_auth), but _base_vertex_proxy_route ignored it and authenticated the request again. A custom_auth function ran twice per request, and a key with max_budget reserved its estimated cost twice, so the unreleased first reservation exhausted the budget after one or two calls. The base route now uses the caller it is given and authenticates once itself only when none is passed, which is the discovery route's case. * fix(vertex): read the caller's key only from x-litellm-api-key or Authorization on Vertex routes The single auth check on /vertex_ai and /vertex-ai now reads the same two headers main's effective check did, so a key sent only in another key header gets 401 again instead of authenticating. * fix(vertex): authenticate the discovery route through the same Vertex key check Both Vertex pass-through routes now resolve the caller once through _authenticate_vertex_caller and hand it to _base_vertex_proxy_route, which no longer re-runs auth. x-litellm-api-key accepts the same scheme prefixes on /vertex_ai/discovery as on /vertex_ai and every other proxy route. * fix(proxy): size a Gemini request's budget hold from its prompt and maxOutputTokens A Gemini generateContent body's budget hold now counts its contents and systemInstruction and reads generationConfig.maxOutputTokens. Before, it held the model's whole context window, about $0.85 on gemini-3.8-flash, so a key with a small budget got 422s on back-to-back calls it could afford Also removes get_litellm_virtual_key, which nothing calls since the Vertex routes stopped using it * fix(proxy): count every Gemini system part and candidateCount in the budget hold A Gemini generateContent body with several systemInstruction text parts lost every part after the first when LiteLLM translated it to a chat request. That dropped the later system text for non-Gemini models behind /v1beta/models/<model>:generateContent, and made the budget hold count only the first part. The adapter now keeps every text part, as a plain string when there is one part and as text blocks when there are several. The output hold now also multiplies by generationConfig.candidateCount, the same way chat bodies use n, so a request asking for several candidates holds budget for all of them. * fix(proxy): keep the context-window hold for Gemini bodies the prompt count cannot see A Gemini generateContent body with media, cachedContent, other roles, or non-function tools keeps the full context-window hold, since the chat translation drops or undercounts those parts. The hold also reads the snake_case generation_config output fields, and several systemInstruction text parts reach non-Gemini models joined into one system string * fix(proxy): read only camelCase Gemini output fields in the budget hold /v1beta forwards only generationConfig.maxOutputTokens to non-Gemini models, so a snake_case cap the model never receives must not shrink the hold. Snake_case bodies now get the default output hold. * fix(proxy): count snake_case candidate_count in the Gemini budget hold * fix(vertex_ai): skip malformed system parts and read Gemini output fields only on Gemini bodies A systemInstruction part whose text is not a string is skipped, as main did, instead of failing the request. The budget hold reads generationConfig output and candidate fields, and counts contents, only when the body has no messages, prompt, input, query or documents field. An output cap of 0 or below counts as no cap. Test recorders keep tuples instead of mutating lists. * fix(proxy): take the budget hold's request shape from the route On generateContent routes the hold counts only contents and reads its output cap and multiplier from generationConfig, so stray fields of other APIs can no longer hide the prompt. A max_tokens of 0 holds no output again, while negative values and a Gemini maxOutputTokens of 0 or below hold the default output * fix(proxy): hold one candidate for a count too large to price and read system parts like content parts * fix(proxy): keep the spend already counted when a budget hold is far over the budget * fix(proxy): count candidateCount in the budget hold only for models that generate the candidates A non-Gemini model reached through /v1beta generateContent never receives candidateCount, so a huge count held the whole budget for the length of the call while the model produced one answer * fix(proxy): hold a candidate count only when every deployment generates it, and drop n when every deployment drops it * fix(proxy): count candidateCount when any deployment generates it and always count n The stricter rules from the previous commit let a key spend past max_budget through a Gemini name no deployment serves, an allowed_openai_params n, and a router fallback. Restore the earlier rules and add regression tests for those three shapes. * fix(proxy): cap the candidateCount hold at main's default output hold A candidateCount like 1e300 sent to a mixed Gemini and Claude group, or through a key alias from a Gemini name to Claude, held a whole key while Claude served the call and answered a strict key 422. On generateContent routes the counted candidates now raise the output hold at most to the default output hold main used for the model, so no count makes a call hold more than on main. Chat n is unchanged. * fix(proxy): hold every Gemini candidate up to the merge base's whole hold The previous cap stopped the candidateCount output hold at the default 16384 tokens. Because the hold now counts the Gemini prompt instead of the whole input window, a strict key admitted a call like 4 candidates of 8000 tokens and then spent past max_budget, where the merge base refused it. Every candidate is now held at its output cap, like chat n, and the whole hold is clamped at what the merge base held for the same call (the model's input window plus the default output), never below one candidate's hold. A huge count still holds no more than the merge base did. Chat n is unchanged * fix(proxy): let a Gemini output cap that /v1beta drops only raise the hold The Vertex and Gemini pass-throughs forward every spelling of the output cap, but /v1beta passes only generationConfig.maxOutputTokens to other models. The hold read only that spelling, so a snake_case cap above the default output held less than the pass-through could spend. A cap in any other spelling now raises the hold to it and never lowers it * fix(proxy): hold Gemini candidates of a model with no context window up to the default output A Gemini model the cost map has no context window for, priced in its deployment, held one candidate for any candidateCount, so parallel calls could pass their budget holds. Its candidates are held again up to its counted prompt plus the default output hold * fix(proxy): keep main's Gemini hold unless the request caps its output An uncapped generateContent call, or one capped only by a snake_case spelling, now holds what main holds (the context window plus the default output). A call with generationConfig.maxOutputTokens holds its counted prompt plus cap times candidateCount, never clamped to the deployment's max_output_tokens and never more than main's hold. A model with no context window keeps main's no-hold, and candidateCount always counts on generateContent routes. Also keeps get_litellm_virtual_key in pass_through_endpoints/common_utils.py, so an out-of-tree importer does not break * fix(proxy): keep main's budget hold on generateContent routes The PR now leaves the generateContent budget estimate as main has it. A live risk check showed the counted Gemini hold under-reserves (the prompt counter undercounts Gemini bodies, tiered and image output rates are not read, and request fields the handler merges later are not counted), so that estimate moves to a follow-up. The single auth pass on the Vertex pass-through and the systemInstruction adapter fix stay * fix(vertex_ai): keep main's Gemini adapter and pass a missing Authorization as no key The Gemini adapter change for later systemInstruction parts goes back to main, since it is not the double-auth fix this PR is for. The Vertex auth dependency now hands a missing Authorization header to the auth check as no key, like main's first auth pass, so custom auth and the 401 text see what they see on main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-07 00:25:51 -08:00
HootRock	8f19607259	fix(proxy)!: preserve Responses SSE event framing (#47980) ... * fix(proxy): preserve Responses SSE event framing * style(proxy): format Responses stream terminator condition * fix(proxy): validate Responses guardrail event names * fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing * test(proxy): use controlled clocks for Responses keepalive failures * fix(proxy): end late Responses failures with an error event the SDK raises on Errors that arrive after keepalive pings, and guardrail error strings yielded mid-stream, now end /v1/responses with a named `error` event carrying the same error object chat sends, so the OpenAI SDK still raises APIError with a clean message and provider_specific_fields. Background polling passes the same framing and records an `error` event as a failed response. The first data line of a frame is found by splitting on newlines only, so a U+2028 in an error message no longer hides a first-event error. * fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses * fix(proxy): keep the reason when a background Responses request fails A background poll of a request a post-call guardrail blocked saved a null code and message, because the guardrail's error event nests them. A request whose first chunk was an error polled as completed with no error. Both now poll failed with the same code and message a late failure gets. * test(proxy): type the first-chunk poll test parameters * fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors * fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE]. * test(proxy): type the first-chunk failure test's parameters * fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors * fix(proxy): encode non-dict Mapping Responses chunks as JSON objects * fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects * test(proxy): freeze the hook-output fixture * fix(proxy): keep masked mappings and odd keys safe in Responses stream frames Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before. * fix(proxy): never fail a Responses stream frame on a value JSON cannot encode Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection. * fix(proxy): keep what a nested value's repr hides out of Responses stream frames * fix(proxy): print values in Responses stream frames the way each base path did * fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames * fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did * fix(proxy): read hook container storage the way repr does and send error-event models as str() * fix(proxy): frame streamed Responses reads and send unreadable hook values as base did Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings * fix(proxy): match base on Responses stream edge cases Read a decimal-string or integral-float status_code as its integer again, so "429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr once and reuse it, so a repr that changes between calls cannot reach the client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge one cannot break the failure event. Build nested JSON with one frame per level again, which restores the nesting depth sent as JSON. Count a pre-framed frame as terminal from its data type, and a named error frame only when its data carries an error, the way the OpenAI SDKs read them. * fix(proxy): print hook proxies once and keep base's chunk tracking Read-only mappings in hook chunks now emit the text from their first print, hook chunks are read without their error field, only a non-empty error object makes a framed error event terminal, and framed sequence numbers stay below 2**63-1 while hook chunks keep base's unbounded tracking * fix(proxy): print each read-only mapping item once and poll framed error events as failed A MappingProxyType in a hook chunk or error detail now expands from the mapping it wraps, so each item is printed once, the way base printed it. A value whose repr changes between calls stays masked, a value whose repr works once is kept, and deep nesting keeps one stack frame per level The background reader takes each frame's type from the same terminal rule the stream uses, so an event: error frame with an error object polls failed even when its data names another type or none * fix(proxy): print self-holding hook values once and poll how the stream ended A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it * fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status A Responses stream's first chunk answers as a JSON error only for codes 400 and up * fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE] A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {} * fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE] A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did * fix(proxy): JSON-encode only plain hook chunks on Responses streams A hook chunk whose whole tree is builtin JSON types streams as its named event. Any other chunk keeps the base data frame, so nothing a custom repr hides reaches the wire. Error event details encode the way the chat error frame does. * fix(proxy): skip containers already visited when checking hook chunks for JSON * fix(responses): keep streaming past an event with no type The Responses stream iterator read event.type directly when it noted a yielded event, so an event whose JSON had no type, like a custom hook's dict that the proxy now sends as JSON, raised AttributeError and ended the stream. Read it with getattr, as the iterator's other checks do. * fix(proxy): keep hook dicts with no type on the base Responses frame A hook dict with no type is not a Responses event, so it gets no event name either way. Sending it as JSON only made litellm SDKs that read type on every event stop the stream. Keep such dicts on the frame the base sends, so every SDK version reads them as before. * fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 20:17:57 -08:00