505 commits

Author	SHA1	Message	Date
Kerry Lu	fb555b00dc	fix(proxy): price a billed failure row at the deployment's own rate like the success path (#47437) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 19:55:21 -07:00
Mrunmayee Daware	2196943195	fix(proxy): match partial wildcards against prefix-stripped model ids (#46390) ... * fix(proxy): match partial wildcards against prefix-stripped model ids (issue #46384) Refactor model checks to improve filtering logic and type hints. * test(proxy): partial wildcard listing regression coverage (issue #46384) * fix(proxy): drop unneeded annotation on suffix list (LIT001) * fix(proxy): annotate suffix list with reasoned mutable-ok (basedpyright) Change suffix_appended_wildcard_models to a typed list. * Add lock timeout before creating telemetry report table Set local lock timeout to 10 seconds before creating the telemetry report table. * fix(proxy): list alias wildcards as routable ids, not raw catalog ids An alias whose suffix collides with a catalog sub-prefix (e.g. public "cloudflare/@cf/*" over deployment "cloudflare/*") took the partial-filter branch and listed raw catalog ids. Routing substitutes the captured suffix into the deployment pattern, so those listed ids routed to targets that do not exist. Only use filter semantics when the deployment repeats the wildcard suffix (or no deployment is available); aliases prepend their prefix to provider-stripped catalog ids so every listed id routes back to a real catalog model. Addresses Greptile review on #46390. * fix(proxy): derive partial wildcard listings from the deployment's routing pattern List an id for a partial wildcard only when routing it through the deployment's pattern lands on a catalog model, so aliases, catch-alls, and key filters over a wildcard deployment list ids that answer when called. /model/info expands a bare * row the same way /v1/models does. * fix(proxy): drop fragment ids for a catch-all over a partial wildcard and route bare aliases through the deployment pattern A * catch-all over a partial deployment now lists no catalog ids, a bare alias like my-cf/* over cloudflare/@cf/* names ids by the deployment pattern, an alias whose wildcard count differs from its deployment lists nothing, and /model/info rows show the model each listed id routes to * fix(proxy): keep listed ids on /model/info alias rows and clean up test-registered models * fix(proxy): list catalog models through routing-prefix wildcard deployments * fix(proxy): keep namespace and bare catch-all wildcards on routable listings A leading deployment segment the model name repeats, or the provider name itself, is no longer read as a routing segment, so same-name namespace wildcards and a key filter over `*` -> `openai/*` stop listing ids that route to no model. A bare `*` deployment under a prefixed name keeps the legacy listing. `*` over a partial deployment now lists the names that route through it, dropping empty ids. * fix(proxy): list nothing for a catch-all over a mid-word wildcard deployment * fix(proxy): list only wildcard ids that route back to their deployment and catalog model A listed id now has to round-trip through the alias pattern to the catalog model it names, and the router's wildcard match for it has to pick this deployment, so an id a more specific sibling wildcard serves is no longer listed under the catch-all on /v1/models or /model/info * fix(proxy): list wildcard aliases that route at a word boundary and ids a sibling serves with the same model --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 02:49:42 +00:00
Kerry Lu	0e7c4908bb	fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row (#47436) ... * fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row * fix(proxy): read the provider handoff before the proxy-only error handler stamps it so a pre-handoff 499 is not priced * fix(proxy): drop the pre-logging handoff read now that pre_call no longer stamps proxy-only failures --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 19:08:30 -07:00
devin-ai-integration[bot]	1660c8ba1f	feat(key_management)!: apply default_key_generate_params.allowed_routes on key generation (#41236) ... * feat(key_management): apply default_key_generate_params.allowed_routes on key generation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(key_management): avoid mutating default routes payload Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): apply default allowed_routes to key_type presets it covers * fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy * fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes * fix(proxy): keep owners from changing routes a key got from default_key_generate_params * docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off * test(proxy): type the default_key_generate_params allowed_routes tests * fix(proxy): let key owners send unchanged allowed_routes back on /key/update * fix(proxy): ignore a route default with a non-string entry and match resent routes exactly A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths. * test(proxy): cover a non-string allowed_routes default in the key integration tests * test(proxy): read key rows from the owned proxy's database in the default routes integration tests * fix(proxy): leave allowed_routes out of a key update that resends them unchanged --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 18:53:37 -07:00
Kerry Lu	c6c303a618	fix(batches): apply the Vertex regional endpoint uplift to batch cost (#47466) ... * fix(batches): apply the Vertex regional endpoint uplift to batch cost batch_cost_calculator takes vertex_location and multiplies by the model's regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift, and every batch cost path (aretrieve_batch pricing, completed-batch logging, native Vertex batch rows, the proxy batch cost poll and its per-line items) resolves the deployment's location and passes it down Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com> * test(logging): point the resolver tests at resolve_vertex_location_for_cost --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>	2026-10-20 01:51:51 +00:00
Mateo Wang	6c4c134f23	fix(secret_managers)!: resolve oidc/file and oidc/env refs in admin-configured api_key per request (#47341) ... * fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper now reads the referenced file or env var on each call and never writes the token back, so a rotated token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400 * fix(secret_managers): strip only the leading oidc/ from a secret reference A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed * fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them The advisor tool's own api_key now answers a 400 when it holds an oidc/ reference, since its sub-call runs through the client wrapper that resolves one. Stored fallback targets, auto-router tiers and user_config refuse only an oidc/ api_key, inline or through the credential they name, so their workload identity federation handling stays as it was. Provider model discovery for a wildcard deployment resolves the reference before calling the provider. * fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments * fix(utils): resolve oidc api_key refs only from admin-configured sources Mark an oidc/ api_key read from a proxy-level deployment or a stored credential, and resolve only that marked value. Team deployments, prompt templates, nested configs and a direct SDK api_key send the text as written. An unsupported reference kind raises a 401 so the router fails over to the group's next deployment. * fix(utils): send a credential's oidc token only to its own api_base host A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the credential's own api_base or base_url, so a team model or vector store that names the credential with another api_base gets a 401 and that host gets nothing. Saving a team model with an inline oidc/ api_key answers 400, since team rows never resolve one. * fix(proxy): send an admin oidc api_key only to the hosts configured beside it Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths. * fix(proxy): redact the token path from token count auth errors * test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs * fix(proxy): redact the token path from secret reference errors where they are raised * fix(proxy): keep the secret reference's path out of the error a caller sees The 401 for an unreadable oidc api_key or pass-through credential now names only the reference kind and a fixed reason, so a token path under /run/secrets or a custom credential dir never reaches the caller, while the server log keeps the full reference for the operator * fix(secret_managers): strip line breaks from the logged secret reference A reference holding a newline can no longer forge a server log line * fix(secret_managers): label an unknown oidc reference kind instead of echoing it An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in full in the caller's 401 and in the spend log, since the kind was whatever text followed oidc/. A kind get_secret does not know now reads oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>, which the client-message redaction on chat routes leaves whole. * test(health): run the real model auth check in the oidc test connection test * test(secret_managers): read an oidc/file token stored under a directory named oidc * fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references A huggingface/* wildcard call could name its own endpoint URL as the model and receive the resolved token. The origin check now treats a URL in the model as a host the reference must allow. The integration contract also reads the proxy's own database instead of the pytest worker's. * fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443, though both reach the same server. The origin check now compares scheme, host and effective port, as assert_same_origin does, and an unparseable api_base is refused instead of raising. The advisor test sets the real proxy flag instead of replacing our own helpers. * test: type the oidc request-body helpers and drive the tier test through mock_response * fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host	2026-10-20 01:43:59 +00:00
Kerry Lu	3fa1587f43	fix(logging): do not stamp a provider handoff when pre_call runs for a proxy-only failure (#47456) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 00:59:53 +00:00
Nate Armstrong	497f31d43a	perf(savings): read deployment pricing from the router's cached model info (#47400) ... Savings and the savings baseline priced deployments through Router.get_deployment_model_info, which deep-copies the cost map entries on every call. The router already keeps an lru_cache of the same lookup that is cleared whenever the model list or cost map changes, and budget reservation reads from it, so the savings path now reads from it too	2026-10-19 17:07:00 -07:00
Kerry Lu	7b76b2b15f	fix(proxy): price the estimated input tokens on a read-timed-out request's spend row (#47435) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 16:55:13 -07:00
Yassin Kortam	6f66b7a201	feat(proxy)!: personal keys inherit MCP server, MCP tool and agent access from the owner's teams (#46293) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 16:01:40 -07:00
Yassin Kortam	ad1f3c5e91	fix(mcp): retry upstream 429 on tools/list and error when every server fails (#46649) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 16:01:23 -07:00
Yassin Kortam	48763ccb50	fix(scim)!: store PATCH enterprise extension attributes in metadata.scim_enterprise (#46738) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 15:58:29 -07:00
hamzashah-dev	9b4f44135d	fix(proxy): refresh cached team on /team/block and /team/unblock (#35643) ... * fix(proxy): refresh cached team on /team/block and /team/unblock Both endpoints wrote `blocked` to Postgres and returned without touching the management-object cache. Auth resolves teams through `team_id:{team_id}` before reading the DB and `common_checks` enforces `blocked` off that cached object, so a team whose traffic had already warmed the cache kept serving requests after being blocked, and kept getting rejected after being unblocked, until the entry expired. `_refresh_cached_team` already exists for exactly this and its docstring says every endpoint mutating `litellm_teamtable` must call it; these two were the only mutating endpoints that didn't. The update now also asks for `object_permission` so the refreshed cache entry doesn't drop the relation, matching `team_model_add` and `team_model_delete`. * fix(proxy): keep a team block from being undone by in-flight spend writes update_cache read the cached team when a request finished and wrote that whole copy back in a background task, so a team blocked or invalidated while requests were in flight came back unblocked on every busy worker until its 60 s cache entry expired. The team spend now goes onto the team held in this worker's memory at write time, and a missing entry is left missing for the next auth read to load. Block and unblock responses keep litellm_model_table and object_permission null, as before this PR, while the cache still gets both relations. * test(proxy): type the team spend cache test parameters * test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock --------- Co-authored-by: Hamza Shah <hamzasepal@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 22:50:33 +00:00
Yassin Kortam	1525e4bce5	feat(cli): keep the claude.ai login with lite claude --subscription and save pkce in config (#47385) ... `lite claude --subscription` sends the LiteLLM key in the x-litellm-api-key header through ANTHROPIC_CUSTOM_HEADERS and leaves ANTHROPIC_AUTH_TOKEN and ANTHROPIC_API_KEY unset, so Claude Code stays on its claude.ai login and its connectors keep working. The pre-launch key check sends the key the same way. `claude_subscription` and `pkce` are now `lite config` keys that default the new `--subscription/--no-subscription` flag on `lite claude` and the `--pkce/--no-pkce` flag on `lite login`. The flags still win for a single run. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 15:33:29 -07:00
Mateo Wang	cf9ebeda07	fix(proxy): stop billing every OpenAI pass-through /responses sub-route as a Responses call (#47171) ... * test(proxy): cover OpenAI pass-through billing for each /responses sub-route * fix(proxy): bill only the OpenAI pass-through Responses calls OpenAI bills * fix(proxy): fail closed on unknown /responses sub-routes and label unbilled spend rows Bill every OpenAI pass-through /responses call except input_tokens, input_items, delete, and a non-background retrieve or cancel, so an encoded or new sub-route is never free. Unbilled calls keep the response's model on their spend row, and a resumed background stream takes its model from the stream's final event. * fix(proxy): give billed background Responses reads their own spend log row A billed retrieve or cancel of a finished background response reused the create's resp_ id, so the spend log insert dropped the row that carried the cost. It now takes the native path's _background_cost suffix. Adds Azure path coverage and types the test recorder. * fix(proxy): bill a background Responses read only once it carries usage An in-progress poll took the _background_cost spend log id with $0, so the finished read's row that carried the cost was dropped again. Polls with no usage are now unbilled like any other read. * fix(proxy): keep OpenAI's response id for callbacks on a billed background read * fix(openai): give a streamed background create the settlement spend row id * fix(proxy): keep a request body from giving its spend row the background settlement id * test(proxy): give the forged settlement marker test a fixed time * fix(passthrough): move the background settlement marker into the pass-through types module A test that reloads spend_tracking_utils gave it a new enum class, so the handler's marker no longer matched by identity and the settlement row lost its _background_cost id. The types module is never reloaded.	2026-10-19 22:04:25 +00:00
yucheng-berri	6162184b1c	fix(managed_files): send provider file ids when creating a fine-tuning job from managed files (#46552) ... * fix(managed_files): send provider file ids when creating a fine-tuning job from managed files Fine-tuning job creation with a managed training file sent the gateway's managed id and the provider-prefixed deployment model to OpenAI, which rejected the job. The pre-call hook now builds the file id mapping for training_file and validation_file, the deployment hook swaps both, the SDK strips the provider prefix from the model, and the create response returns the caller's own file ids * fix(managed_files): route managed fine-tuning creates only to deployments holding every file Fine-tuning creates now narrow routing the same way managed batches do, so a deployment added after the upload is never picked. Adds a two-worker chaos integration test and runs tests/unit/fine_tuning in the endpoints shard. * fix(managed_files): keep fine-tuning creates on the model group when no deployment holds the files A managed file used on a model group it was never uploaded for kept failing with the provider's 400 before the routing filter. The filter turned that into a 429 no-deployments error, so fine-tuning now falls back to the whole group in that case while batch input files keep routing only to deployments that hold them. * test(fine_tuning): cover the provider prefix strip for OpenAI and Azure creates through the proxy * test(fine_tuning): start each multi-deployment case from an empty upstream log * test(fine_tuning): boot both proxies of the late deployment cell on one scratch database --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-19 14:40:15 -07:00
Kerry Lu	62ce515d15	feat(router): let a deployment set a timeout per service_tier (#47406) ... * feat(router): let a deployment set a timeout per service_tier Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's litellm_params. A request whose service_tier matches a key uses that value in place of the deployment's timeout or stream_timeout. A timeout the client sends still wins and requests without a matching tier are unchanged Co-authored-by: adc77 <amandogra2016@gmail.com> * style(router): sort the request_timeout_resolver import * fix(router): read the service tier maps without the untyped params get * fix(router): look the service tier timeout up through a typed get --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: adc77 <amandogra2016@gmail.com>	2026-10-19 14:27:29 -07:00
Mateo Wang	b51b9b7d43	fix(anthropic)!: answer 400 for a malformed tool_result text on /v1/messages and count_tokens (#47347) ... * fix(anthropic): answer 400 for a malformed tool_result text on the Responses bridge and count_tokens A tool_result text block whose text is null, a number, or missing made /v1/messages and /v1/messages/count_tokens answer 500 on OpenAI and Azure models. Both routes now answer 400 naming the block, and drop the block when drop_params is on. * fix(anthropic): keep a missing tool_result text as empty text, bound the error, honor body drop_params on count_tokens * fix(anthropic): apply the tool_result text check on the chat/completions bridge and every count route The chat/completions bridge used for Azure, Gemini and other providers now answers the same 400 for a null or numeric tool_result text and drops the block under drop_params, so a model group that mixes OpenAI with another provider answers the same way on every deployment. Gemini and Vertex count_tokens apply the same check, count_tokens turns on drop_params for Claude Code and Codex the way /v1/messages does, and a tool_result without a tool_use_id is checked too * fix(proxy): answer 400 for a malformed tool_result text on count_tokens models without a count API Providers with no token counting API fall back to the local tokenizer, which counted a null or numeric tool_result text as fine while /v1/messages on the same model answers 400. The fallback now runs the same check, unless drop_params applies to the deployment or the request * fix(proxy): honor request drop_params on the local count fallback when no deployment resolves * test(integration): cover non-string tool_result text on both /v1/messages bridges and count routes * fix(anthropic): keep a dropped tool_result on Mistral and reject it before a guardrail rewrite Under drop_params a tool_result whose only text block was null or a number became an empty tool message, and Mistral's empty-message filter removed it, so Mistral answered 400 for a missing tool reply. The filter now only removes empty assistant messages, which is what Mistral rejects. A guardrail that rewrites structured_messages wrote the dropped text back as "", so a null or numeric text was silently blanked even without drop_params. The rewrite path now answers the same 400 as the bridges unless the request or global drop_params is set. * test(integration): cover the Mistral empty tool reply and the guardrail rewrite tool_result text check * fix(anthropic): keep a malformed tool_result text through a guardrail rewrite and keep Mistral's null-message drop A guardrail that rewrites structured_messages now writes the caller's null or numeric tool_result text back in place, so the route's bridge and the deployment's drop_params decide after routing, the same as with no guardrail. The rewrite path no longer answers its own 400 before routing. Mistral drops a null-content system or user message again, as main does, and only a tool reply is exempt from that filter so a dropped tool_result keeps its tool call. * test(mistral): expect empty system and user turns dropped and the empty tool reply kept * perf(anthropic): restore kept tool_result text parts in one linear pass * test(integration): cover the tool_result text restore across tool_results, the chat bridge, and a batch upload behind a rewriting guardrail * fix(anthropic): let guardrails scan malformed tool_result text and honor drop_params on native Anthropic A rewriting guardrail now sees a non-string tool_result text as its JSON text and the write-back restores the caller's value by the sanitized tool_use_id when the guardrail left it unchanged. Native Anthropic drops the bad block under drop_params on /v1/messages and count_tokens, and Mistral keeps an empty tool reply only when it answers a tool call, sent as empty text. * fix(anthropic): compare a kept tool_result text to its scanned form once and align the wire tests with drop_params * test(integration): cover native Anthropic drop_params resolution, guardrail scanning of numeric tool_result text, and the Mistral orphan tool message * fix(count_tokens): honor the router's default drop_params when counting a malformed tool_result text * test(count_tokens): send the tool_result count tests through a real request body instead of a mocked parser * test(count_tokens): send the native Claude count test through a real request body * fix(anthropic): keep each non-string tool_result text when tool_use_ids sanitize to the same id * fix(anthropic): keep non-ASCII characters in the guardrail's view of a non-string tool_result text * test(gemini): mock the count_tokens upstream with respx instead of patching the client factory * fix(mistral): keep an empty tool reply only for an open call, and forward non-object message items on the native drop An empty tool reply re-answering a call the conversation already moved past made Mistral reject the request, and a tool call id that is not a string crashed the filter. Native Anthropic under drop_params assumed every message item was an object. * fix(mistral): find answering tool replies in one pass instead of copying the open call set per reply * fix(mistral): ignore dropped empty placeholders when tracking open tool calls * fix(mistral): drop an empty tool reply when another reply to the same call carries the result * refactor(anthropic): share tool_result text helpers to shrink the diff * fix(anthropic): answer 400 for a non-string tool_result text on the OpenAI Responses bridge	2026-10-19 21:13:46 +00:00
Justin Israel	40e4ade6a6	feat(proxy): opt-in self-serve budget edits for key owners on /key/update (#44763) ... * feat(proxy): opt-in self-serve budget edits for key owners on /key/update Adds general_settings.self_serve_budget_policy (disabled by default) so a non-admin can change max_budget or budget_limits on a key they created and still own. lower_only allows adding or lowering limits. ceiling also allows raising or removing them when the owner could create that key with the new budget via /key/generate (same member permissions and delegation ceiling). soft_budget and spend stay admin-only * fix(proxy): sync dashboard API types and cover the UI session cap on update * test(proxy): cover team role restriction for owner budget raises Drop two unreachable branches from the non-raising key generation check. * fix(proxy): close self-serve budget bypasses for linked budgets, null caps and retained limits Keys with a linked budget_id need an admin to change max_budget, owners cannot leave a key uncapped when an upperbound max_budget is configured, and the ceiling check now runs on the budget the key would carry after the update. * fix(proxy): keep open key budget windows and the admin path on self-serve budget edits A resubmitted budget_limits window now keeps its stored reset_at while that period is still running, so adding or editing one window no longer restarts the others. Team and org admins answer False on a ceiling loosen so they fall through to _check_key_admin_access as before. The self-serve tests feed the team through the cache and prisma mocks instead of patching get_team_object, and every helper is typed * perf(proxy): group key budget windows by duration in one pass The per-duration minimum used to rescan the whole window list once per distinct duration, so a request with thousands of windows pinned the event loop before any permission check ran. Sort once and take each group's minimum instead * refactor(proxy): take budget windows as a Sequence in the delegation ceiling checks * test(proxy): assert allowed self-serve budget edits return no denial Also drop the unreachable assert_never arm from the exhaustive budget change match. --------- Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-10-19 11:40:38 -07:00
devin-ai-integration[bot]	f8f6a01cae	feat(ui): show kubernetes pods on the model page (#44666) ... Show discovered Kubernetes pods on the model page for admins Resolves LIT-9168 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 09:58:20 -07:00
devin-ai-integration[bot]	f85926a652	feat(router): keep a session on one kubernetes pod (#44393) ... Requests with an explicit session id stick to one discovered pod via rendezvous hashing Requests without a session id keep round robin Pod routing records are included in logs, spend logs, and traces Resolves LIT-9168 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-19 09:58:19 -07:00
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
alexcastgonz	6762d65eb8	fix(vertex_ai): price streamed pass-through calls with the regional uplift (#44795) ... * fix(vertex_ai): price streamed pass-through calls with the regional uplift The streamed Vertex pass-through handler builds the response with stream_chunk_builder, which stamps a cost into _hidden_params without the vertex_location. The handler then prices the response again with the location from the URL, but only into kwargs, and the logger prefers the stamped cost. Streamed calls on a regional endpoint were billed at the global rate. Stamp the location-aware cost on the response in the shared generateContent payload builder so the logger records it. Fixes #44755 * style(vertex_ai): keep the response_cost assignment within the 120-char limit * refactor(vertex_ai): set the passthrough response cost with set_hidden_param Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this. * test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed * fix(proxy): price streamed Vertex pass-through spend like the non-streamed call Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls * fix(proxy): price custom-priced Vertex pass-through deployments at their own rates The stamped response_cost from the generateContent logging helper is returned before the logger checks custom pricing, so a deployment with custom input/output rates was billed at the cost-map price. The helper now passes custom_pricing and the deployment id into completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls, and deployments without custom pricing keep the regional uplift. * fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price * fix(proxy): give the new pyright ignore a reason for the LIT004 gate * test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included * fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same * fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini A top-level base_model in a pass-through body now reaches the logging params only for a key or team with allow_client_pricing_override, so a plain key cannot pick a cheaper model's price once the logger prices these calls. The test that pinned every litellm-owned body name into the logging params now leaves base_model to its own key-gated test. The Vertex handler now classifies a non-Google upstream host as Vertex only for Vertex resource paths, so a /gemini stream through a GEMINI_API_BASE gateway is priced as Gemini without the regional uplift. Adds the audit's integration spend test for Claude on Vertex, /v1beta, the named custom-priced /gemini deployment, and a cut OpenAI stream. * test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path. * fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model The Vertex pass-through handler picked Gemini pricing from the URL path, so a configured pass-through to a Vertex gateway with a short path (no /projects/ segment) was billed as Gemini, and its Claude streams at $0. The provider now comes from the route: a call through /gemini or to generativelanguage.googleapis.com prices as Gemini, and everything else as Vertex, the same rule main used. Matching the GEMINI_API_BASE host was rejected because one gateway can serve both APIs, and the path rule was rejected because short gateway paths look like Gemini paths. A non-string base_model (a number, list or object) made the cost calculation fail and bill $0 once the stamped stream costs were gone. get_base_model_from_metadata now ignores it and falls back to the deployment's base_model, so the call bills the list price. A 400 was the other option, but the logger runs after the upstream answered. * test(integration): bill a non-string body base_model at the list price on every pass-through route * fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info A key allowed client pricing can send metadata.model_info in a pass-through body. A non-string model_info.base_model made the cost lookup fail, and once the stamped stream and Gemini costs were gone those calls billed $0 on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores a base_model that is not a string, the same rule the root base_model got, so the call bills the list price. A model_info that is not an object (a string or a list) broke several logging steps that read it as a mapping, so a non-streamed /gemini call fired no success callback and no pass-through call wrote a spend row. The pass-through route now leaves a non-object model_info out of the logging metadata, so the call bills the list price. A 400 was the other option, but it would turn calls that answer 200 today into errors, and guarding every reader of model_info would spread the same check over many files. * fix: reject a non-object model_info before the provider call again The previous commit made the base_model lookup return None for a model_info that is not an object. That lookup also runs during call setup on every router route and SDK entrypoint, where such a value used to fail before any provider request. With the early return the call went upstream, the provider billed it, and the request then failed in a later reader that still expects an object. On /v1/messages and /v1/responses the key was also charged for a response the client never got. The lookup again reads base_model from any truthy model_info, so a non-object value fails during setup the way it did before this PR. A non-string base_model inside an object still bills the list price, and the pass-through route still drops a non-object client model_info at intake, so pass-through calls keep billing the list price. * fix: match the Gemini API host exactly when picking the pass-through cost provider The provider check accepted any host ending in generativelanguage.googleapis.com, so a Vertex host such as vertex-generativelanguage.googleapis.com was priced as Gemini and lost the regional uplift. It now accepts that host or a subdomain of it. * test: pin that a subdomain of the Gemini API host keeps the Gemini price --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 08:10:52 +00:00
vanraj Solanki	8b90649fb7	fix(proxy): sanitize unexpected /key/info database failures (#46190)	2026-10-18 19:25:14 -07:00
joshua-berri	9c958aff34	fix(logging): deliver completed batch success callbacks consistently (#47192) ... * fix(batches): dispatch configured success callbacks during reconciliation * fix(batches): retain billed job claim when sync dispatch shuts down * fix(batches): finish reconciliation when logging executor is stopped * fix(logging): avoid duplicate custom hooks during async batch dispatch * fix: preserve sync-only custom callback delivery * fix: scope callback deduplication to per-call async recipients * refactor: type per-call async success callbacks * refactor: retain logging constructor callback compatibility * test: support current batch reconciliation metadata * fix(logging): honor per-logger redaction during sync success delivery * fix(logging): centralize completed success callback dispatch * fix(guardrails): avoid duplicate success callback dispatch --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-18 09:19:04 -07:00
Mateo Wang	5012083b72	feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes (#47185) ... * feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes * fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting The drop list now runs on the final upstream headers, after x-pass- expansion, through an upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a dropped value to Vertex. A malformed setting now answers 500 only for requests that carry anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working. * test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests * fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces * fix(vertex_ai): split drop-betas list entries on commas like the header	2026-10-18 12:51:27 +00:00
Mrunmayee Daware	e979bd0ecf	fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel (#47067) ... * fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel A unified batch id resolves to the deployment's internal model_name (model_name_{team_id}_{uuid}), which is in no key or team allowlist, so authorize_model_for_key always denied and team keys got 403 retrieving or cancelling their own batches. Authorize against model_info team_public_model_name when the resolved deployment is team-scoped, falling back to the resolved name otherwise. Fixes #46958 * fix(proxy): authorize team-scoped batch models by their public name on both auth paths * fix(proxy): accept the owning team's public name for its own team deployment on every auth path Managed batch, file, fine-tuning, vector store, character and video ids resolve to the team deployment's internal name again, so teams that granted the internal name keep access. The model access check now also accepts the owning team's public name and its legacy team aliases for that internal name, and only when the caller's team owns the deployment. * fix(proxy): match a team deployment's public and alias names against the caller's own grants --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-18 07:18:11 +00:00
moyai-berriai[bot]	f370235bb4	fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) ... * fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans * Preserve main's per-block streaming fixes and regression coverage * fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails * fix(bedrock): accept blank content on tool-call-only Converse turns A vendor that returns "" or [] where an assistant row carried only tool calls no longer fails the request closed. Also type the leaf lookup without Any and restore the original layout of the existing unit tests. * fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back. The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names. * fix(bedrock): keep new Converse helpers inside the type discipline rules * test(bedrock): pin Converse row edits that change shape, role or placement * fix(bedrock): type the Converse body once so its edits keep their reference * fix(bedrock): list request-field texts where their Converse row sits * fix(bedrock): send a reshaped Converse tool result as one text block A vendor that compresses or summarises a tool result, such as Headroom collapsing a JSON list into a table string, returned a 500 because the row could not be split back into the original text and JSON parts. The tool result now carries the vendor's string as one text block where its parts were, keeping images and documents in place. It still fails closed when the flat texts masked that tool result or the row changed its tool call id. * fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it A guardrail that masks a Converse tool description or schema now has that mask written into toolConfig, a tool it removes is dropped, and a tool it adds, renames or reorders is refused before Bedrock. Headroom, Compresr and TypeSafe get only the flat texts on Converse again, as on main, because the pass-through route has no loop to answer their retrieve tool * test(bedrock): refuse a guardrail that changes a Converse tool's kind * fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS and HiddenLayer scan only part of the rows or rebuild their returned texts from them, so on Converse they get the flat texts only, as on main. When a vendor returns changed rows, its texts are applied only if they keep the flat layout and change only rows the vendor also changed, so row-shaped texts of the same length no longer overwrite unrelated Converse fields. * test(bedrock): pin how flat texts sit next to rewritten Converse rows * fix(bedrock): keep the Bedrock guardrail on flat Converse texts With experimental_use_latest_role_message_only, the Bedrock guardrail places its mask by counting text parts per row, and Converse rows do not line up with the flat texts, so the mask could land in tool call arguments. Give it the flat texts only, as on main. * fix(bedrock): keep Akto on flat Converse texts Akto maps its masked payload back onto the flat texts by value, and a Converse row joins or serialises tool call arguments and JSON tool results, so a masked card there could not be placed and the request was blocked. Give Akto the flat texts only, as on main, where the mask is applied. * fix(bedrock): list request-field texts last again, as on main Flat texts listed additionalModelRequestFields strings right after the system prompt, so flat-text guardrails got a different order than on main. Akto's live verdict on a benign request depends on that order and started blocking it. Rows already keep their own order and texts that do not follow rows are dropped, so the flat texts go back to main's order. * fix(bedrock): list request-field texts next to their row for row readers Row readers that rebuild texts from rows could swap a masked request field with a masked user turn when the field strings came last. Row readers now get those strings right after the system prompt, where their row sits. Flat-text guardrails keep the end-of-list order. * fix(bedrock): list nested request-field texts in their row's order for row readers Row readers that rebuild texts from rows could write a masked request field onto a sibling key when the field held nested objects, because the texts walked the fields depth first and the row breadth first. Row readers now get those strings in the row's own order. * fix(bedrock): walk request fields breadth first instead of resolving each path Resolving every request-field string by its path revalidated the parent object per string, so a wide additionalModelRequestFields took seconds. The string walk now goes breadth first for row readers, the same order the row lists them, in one pass. A unit test pins that texts and rows map the same way when the vendor masks them differently. * fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry When a vendor returns changed rows whose texts do not line up with them, the rows win. A text the vendor masked on a field its rows left untouched (a tool call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead, unless the edited body holds the changed text elsewhere, as texts rebuilt from the rows do. * fix(bedrock): read same-length texts by position when rows drop a mask Matching a dropped text against any string of the edited body let a row mask on another field with the same token cancel it, so the raw tool call argument still went out. Texts of the length that was sent are now checked position by position, and only texts of another length, which cannot be placed, are matched by value. * test(bedrock): count each rebuilt text against the rows' edits * fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed PromptGuard rebuilds its returned texts from the user rows, and a vendor that does so while masking only row content let a card in a tool call argument reach Bedrock, which main masked. PromptGuard now gets the flat texts as on main, and a reply whose rows changed next to texts of another length fails closed, since what those texts mask cannot be checked. * test(bedrock): let the structured checks decide the unsafe tool call and missing row cases The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for texts that cannot be placed answered first and the tool call and row count checks went unexercised. * fix(bedrock): send request field strings to row readers after the conversation, where main puts them Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last, as main sends those strings in texts * test(bedrock): pin nested request field order for flat readers and the texts view Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader getting the row order or a texts view written in main's order fails * refactor(bedrock): ask each guardrail whether Converse should hand it rows Converse picked the guardrails that get only the flat texts from a list of provider names. CustomGuardrail now has supports_structured_messages, True by default, and the ten classes on that list override it to return False, so the handler asks the guardrail instead of the list. Every guardrail class gets the same answer as before * fix(bedrock): keep Singulr on flat Converse texts as on main Singulr wraps each flat text as a user message and its backend judges only the latest user message, so on main a Converse tool result was the turn it judged. Handed rows, the tool result arrives with role tool and the latest user row is the original question, so an injection in a tool result reached Bedrock. Singulr now returns False from supports_structured_messages * fix(bedrock): keep Qualifire on flat Converse texts as on main Qualifire builds its evaluation input from the last user row only when that row's content is a string, so Converse rows (content-block lists) gave an evaluation_id policy an empty input. Opt it out of structured messages so it gets main's flat texts again. * fix(bedrock): keep Conduct on flat Converse texts as on main The conduct-litellm-guard plugin scans only user rows, so with Converse rows a system prompt, assistant text or tool result left its scan. Opt it out of structured messages so it gets main's flat texts again, all sent as user turns. * fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects A guardrail that removed every tool, or the tool toolChoice names, produced an empty tools list or a dangling toolChoice that Bedrock answers with an opaque ValidationException. Refuse those rewrites before Bedrock, like other tool changes this route cannot carry. * fix(bedrock): list request-field strings in main's order for every guardrail The request-field row walked additionalModelRequestFields breadth first, and guardrails that get rows had their texts follow that order. With two or more strings the last text changed, so Ovalix, which blocks only when the last text is flagged, rewrote an injection main blocked. The row now walks the fields in the order main lists them in the texts, and the texts use main's order for every guardrail. * test(bedrock): pin a container's own request-field strings before its nested ones * test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools * fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock, converted the way chat completions converts tools, with cache points kept behind their tool. When every tool is removed, toolConfig is dropped and earlier tool calls and results become text, as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed. The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer hits the recursion limit. * test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly * fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply * fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail on_error: allow sent the original request on when the endpoint refused a guardrail rewrite it could not apply, so a Converse request the guardrail masked reached Bedrock raw. The refusal now counts as a fail on every endpoint. * fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say A rewrite the endpoint cannot apply is neither a content verdict nor a technical error, so on_fail next or allow and on_error allow must not send the original request on * test(bedrock): trim the Converse guardrail tests Drops the chaos and per-vendor integration cells and the duplicate unit cases, and leaves the existing handler tests as they are on main * fix(bedrock): put Converse request-field strings in the latest user message for row readers A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their own only when no user message has text. * fix(bedrock): create toolConfig when a guardrail adds the first Converse tool A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main. * test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows * fix(guardrails): keep Converse tool matching and write-back linear in request width Matching each returned tool rescanned the whole tool list, and every write-back step revalidated the list or object it passed through, so a request with thousands of tools or request fields held the event loop for seconds. Returned tools are now looked up by name in one index, and path steps read the child directly. * fix(guardrails): let later Converse guardrails scan guardContent text When a guardrail removes every tool on a Converse request that carries guardrailConfig, tool results are rewritten as guardContent blocks. The flat text extractor read only top-level text, so a later masker in the same request never saw them and a card in a tool result reached Bedrock unmasked. Chat completions and /v1/messages keep tool results visible to later guardrails after the tools are removed. The extractor now reads guardContent text the same way the structured rows already do, so flat-text guardrails also scan guardContent text a caller sends. * fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer, Qualifire) check what the caller wrote When a guardrail removes every tool, earlier tool calls and results are now written as plain text blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own guardrail keeps checking every message, since a single guardContent block makes it skip all plain text * fix(guardrails): split a newline-joined Converse message back into its parts A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline. * fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped and the history turned into text only when the request is signed, as chat completions does at request build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400 Request-field strings join the latest user message that has text when a tool-result-only turn follows it, so a guardrail judging only that message reads the caller's text with them A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500 Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 20:38:14 -07:00
devin-ai-integration[bot]	4bebcdf757	fix(databricks): use the native Anthropic route for Claude /v1/messages (#46091) ... * fix(databricks): use the native Anthropic route for Claude /v1/messages Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge * fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again * fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent * fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials Unity Catalog model-service names now reach the AI Gateway's native Anthropic route instead of a serving-endpoints path that answers 404 for them, and the native route resolves the key and base through the same order the chat route uses, so environment-only deployments keep working. The native route is limited to Databricks foundation-model Claude names, an unlisted provider forwards safeguards and the beta as sent, and the ten betas the workspace rejects are marked unsupported for Databricks in the beta map and its Rust mirror. * test(rust): expect Databricks to drop the advanced-tool-use beta * fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback Only the OpenAI-compatible JSON providers drop safeguards now, so a deployment opted into /v1/messages passthrough forwards the field and the beta as sent. When a route drops safeguards and the proxy has a classifier model, the proxy answers the verdict itself, as the chat completions bridge did. The Databricks Messages route also reads the databricks_user_agent param like the chat route. * fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL * fix(databricks): drop container on the native Messages route, as the chat completions bridge did * fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name * test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire * test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas * fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth Databricks chat completions, called directly or through the Messages bridge, no longer receive the client's anthropic-beta and anthropic-version, as at the merge base. The native Messages route still forwards them, filtered by the beta map, without counting them as caller credentials, so a deployment with no api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to the native route only for Unity Catalog names, matching the chat route. * test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route * fix(databricks): keep client anthropic headers off an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and that opts into /v1/messages with supported_endpoints falls back to the OpenAI-like passthrough. That config does not filter betas through the beta map, yet it received the native-only Databricks header entry, so Claude Code's prompt-caching-scope beta reached the upstream raw and got a 400 where the merge base answered 200. Each Messages config now says whether it is the provider's native route, the OpenAI-like passthrough says no, and the handler passes that answer when it reads provider-specific headers, so native-only entries reach only native configs. * test(databricks): cover client anthropic betas on an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and whose model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m betas through the proxy and asserts the caller gets the upstream's message and the gateway receives no anthropic-beta, as on the merge base. The scripted gateway answers 400 on the prompt-caching-scope beta, as the real one does * fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do. * test(databricks): cover a loaded beta map that marks the safeguards beta unsupported A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict. * test(anthropic): annotate the MCP beta map recordings as Final * fix(anthropic): drop safeguards on routes without the classifier only under drop_params A route whose upstream does not answer safeguards (a JSON provider, or a map that marks the dangerous-tool-use beta unsupported) now forwards the field and the beta as the merge base did unless drop_params is set or the proxy runs its own classifier, matching how litellm treats any unsupported param. Databricks keeps dropping both, since its merge base bridge never forwarded the field. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-18 02:44:38 +00:00
Mateo Wang	68a8b12ab3	fix(proxy)!: end a chat completions stream with an error event when a guardrail blocks mid-stream (#46953) ... * fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out. * fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE] A guardrail block before the first chunk is raised again, so its response stays the 400 event-stream body it is on main. A mid-stream error event now carries the failure hook's replacement error when one is returned, is followed by data: [DONE] on OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details JSON cannot encode are written as strings instead of breaking the event. * fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400 * fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band * fix(proxy): encode non-JSON guardrail details in early stream error frames An early guardrail block whose detail holds a value JSON cannot encode sent 400 headers and then cut the body. The error frame now writes such a value as its string form, the same as the mid-stream event. The keepalive test now runs on a virtual clock instead of real sleeps. * fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload. * fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client * fix(proxy): print CLI stream errors without Rich highlighting or emoji codes * test(proxy): cover the CLI's plain-text 400 error print * test(proxy): annotate the new client stream test locals as Final * fix(proxy): send the failure hook replacement on Responses streams and before the first chunk * fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception * fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it A non-HTTP failure whose hook returns an HTTP error was raised before the first chunk, which create_response turned into the hook's status but which a bare StreamingResponse route such as /queue/chat/completions answered as a 200 with zero bytes. Only the original HTTP errors raise now. The replacement goes out as the stream's error event, which create_response still answers with the hook's status as a JSON error. Adds audit round 2's integration cells for failure hook replacements on chat and Responses streams, built-in exceptions on Responses streams, and the Python client. * test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent	2026-10-17 18:00:12 -07:00
Mateo Wang	75faaeab64	feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) ... * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop	2026-10-17 17:11:27 -07:00
joshua-berri	3fb95c8fe3	fix(batches): preserve spend across malformed output and metadata failures (#47184) ... * fix(batches): preserve recoverable spend and terminal reconciliation state * fix(batches): narrow validated envelope types * fix(batches): serialize stored metadata when releasing claims * fix(batches): sanitize metadata before claiming completed batches * refactor(batches): expose existing metadata serializer for shared use --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-17 17:10:22 -07:00
Shivam Rawat	c58fc0c996	fix(rag)!: use team provider credentials and allow viewer and JWT team access (#45497) ... * fix(rag): use team provider credentials and allow viewer and JWT team access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rag): assert persisted managed store data excludes team credentials Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(jwt): run real team model checks in the route denial test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rag): scope team provider credentials to openai vector stores Bedrock and other providers kept their pre-PR credential and region resolution: a team bedrock deployment's region no longer overrides the caller's region on RAG query. Query tests now fake the provider at the HTTP boundary with a real vector store registry. * fix(rag): use only the team's own OpenAI deployment for RAG credentials * fix(rag): ignore caller credential fields the query search never uses when picking team credentials * fix(rag): treat blank credential fields as absent when picking team credentials * test(rag): delete the stores the integration ingests create before the owned proxy exits * fix(proxy): let a restricted key use the team deployment its wildcard or model covers A key restricted to gpt-4o on a team whose own OpenAI deployment is openai/* never resolved that deployment's credentials, because the key allowlist was matched against the deployment's public name literally. The allowlist now matches a wildcard on either side the way request authorization does, through the auth layer's own helper, so /rag, /files and /batches resolve the team deployment for every key that may call a model on it. * fix(batches): read a restricted key's batch input through its team deployment The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment * fix(rag): load deployments oldest first, reject file_id on stores that cannot take one, and show ingest errors in the dashboard * fix(rag): pick the team deployment with the lowest id, return 400 for caller mistakes, keep the ingest id in errors, and show partial success in the dashboard * fix(rag): pick the team's oldest OpenAI deployment so adding one never moves the team * fix(rag): keep config file team deployments first and try JWT teams in sorted order A team's config file OpenAI deployments now win in file order, as on the base, before its DB ones oldest first, so upgrading never moves a team that has both. A JWT in several teams tries them in sorted order, so every worker resolves the same team. * fix(rag): return 400 for Bedrock, S3 Vectors and Gemini store config mistakes on ingest * fix(rag): keep forwarded provider keys on /rag/query and mixed JWT team claims working A forwarded x-api-key now counts as a caller credential, so the /rag/query search uses it over the team deployment as at the merge base. Sorting the JWT team claim compares entries as text, so a stray number or null no longer turns into a 401. A Gemini store with no proxy key is a 500 again, and a non-string vector_store_id keeps the merge base's 400 message. * fix(rag): let a forwarded provider key win over team credentials on files, batches and ingest * fix(rag): keep team credentials when a request carries its own key or api_base A forwarded x-api-key or a body api_key now replaces only the key of the team deployment, keeping its api_base, as chat completions does. A body litellm_credential_name no longer turns team credentials off on /v1/batches. * fix(batches): keep the team key on the team host when a request sends only api_base A request api_base now replaces the team deployment's api_base only when the request also brings its own api_key, and a non-string api_key or api_base is ignored, as on the merge base. * fix(files): ignore a blank request api_key or api_base next to team credentials * fix(files): answer 400 for a request api_key that cannot be sent as an HTTP header * fix(rag): honor a body drop_params for a forwarded key that cannot be an HTTP header * fix(rag): read the request field for drop_params only on multipart ingest bodies * fix(rag): type the ingest body read for drop_params * fix(batches): count a model-routed batch's input with the deployment key it is created with * fix(batches): reject a batch create whose x-litellm-model headers name different models * fix(rag): treat a null provider and whitespace store credentials as unset when picking team credentials * refactor(files): type the team deployment the credential resolver ranks --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 23:51:48 +00:00
Misbah Syed	8b442f00fc	fix(proxy): count the whole minor line in the upgrade notice for patch releases (#46942) ... * fix(proxy): count the whole minor line in the upgrade notice for patch releases A patch release's notes only list its backports, so v1.105.1 showed "0 new features, 1 fix, and 0 other updates" even though upgrading from v1.104.x brings everything in v1.105.0 too. When the latest release is a patch, also fetch the recent release list and sum the stable releases on the same major.minor line (skipping rc/dev prereleases and drafts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): use tuples for release lists and sync the dashboard API types Satisfies the LIT001 type-discipline gate and regenerates the latest_release_info description in schema.d.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): cover the bad release-list payload and non-semver latest tag Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): type the release-line Counters so basedpyright sees no unknowns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): fetch earlier patches of the minor line by tag Reading the first page of /releases could miss the line's .0 release when prereleases push it off the page, and one unrelated release with a null body failed validation for the whole list. Fetch each earlier patch tag directly instead (v1.105.0 .. v1.105.N-1), skip tags with no stable release, and treat a null release body as empty notes. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 15:49:57 -07:00
yuneng-jiang	4bafebbcdc	fix(ui): take the dashboard's user role and identity from the server (#47025) ... * fix(ui): take the dashboard's user role and identity from the server Adds an internal GET /management/v1/users/me that returns the user id, role and email the proxy resolved for the calling key. The dashboard now reads identity from it instead of the session cookie's claims, so its navigation and role gating always match what the backend authorizes. The layout waits on it behind the existing loading screen, and a failed lookup clears the session and returns to login. Sessions that must change their password can also read it * fix(ui): sign out when the current user lookup fails and wait for it on chat The dashboard layout now holds the loading screen and signs out when /management/v1/users/me fails, instead of mounting the shell and looping back to loading as each consumer refetched the failed query. The chat layout also waits for the identity so its panels never query without a user id. The lookup goes through the typed client with the regenerated schema.d.ts * fix(ui): send the session's auth header on the first user lookup and keep chat closed until authorized The first lookup runs before AuthProvider registers the custom header name, so it now reads auth_header_name from the session cookie. Chat also waits for isAuthorized, so a failed lookup no longer mounts the shell with an empty user id * fix(ui): type the dashboard callers of the now typed user id useAuthorized used to hand back the cookie's user id untyped, which hid six strict type errors in its callers. The id is now string | null, so the production build failed. Callers handle the null case, and query key filters accept the boolean blocked filter * fix(ui): look up the current user only after the proxy URL is discovered Pages outside the dashboard shell, like chat and connect, could start the lookup before UI config discovery set the proxy base URL. On a split-origin deployment both attempts then hit the UI origin and a valid session was signed out * fix(ui): keep a loaded identity while UI config discovery reruns Pausing the lookup by passing a null token switched the query key, so on the dashboard the identity the layout had already loaded vanished until the shell's own UI config request finished. The key now stays the same and only the fetch waits for discovery	2026-10-17 14:39:33 -07:00
Mateo Wang	897dda7920	fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263) ... * fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row A deployment with an explicit custom_llm_provider and model_info.mode: chat used to overwrite the shared litellm.model_cost row, moving every other deployment and plain SDK calls onto Chat Completions until restart, because the downgrade guard read a double-prefixed key and never fired. The guard now reads the real key, and a deployment's own mode decides its routing in responses_api_bridge_check, except on chatgpt and github_copilot where Chat Completions is dead for responses-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): keep the bridge-check mode tests in the sharded bridge flag file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the downgrade guard's mode from the row registration writes A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit responses/ prefix now also outranks a deployment's chat pin. * fix(router): let an explicit responses/ prefix outrank a registered chat row The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the lookup found mode chat and the request went to Chat Completions as responses/<model>. * fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions. * fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them. * fix(router): skip a non-string model_info.mode when keeping the registered mode * fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API A Mantle row that also serves the Responses API now follows the GPT-5 tool and reasoning summary bridge rules, since Mantle Chat Completions rejects function tools on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat gets a dict reasoning_effort as its effort level. The opt-in auto summary asks each Responses config for its level, auto on Mantle's OpenAI path, so a client's own detailed summary follows drop_params as at base. No mode replaces a registered responses mode, and chatgpt and github_copilot rows take any pin until they are responses, which restores the Copilot chat opt-out. * fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort * test(integration): run the Mantle region-prefixed v1 base case again This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip * fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request * fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request * fix(router): keep a non-string deployment mode off the shared routing row A deployment whose model_info.mode was an int, a list, or an object still reached the shared cost-map row, so an unpinned sibling on the same backend stopped routing to the Responses API. The shared registration now drops a mode that is not a string, and the deployment keeps its own raw value. Adds integration cells for per-deployment routing across the OpenAI and Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream outage, config changes under traffic, and a worker kill plus restart. * test(integration): find each routing cell's spend row by the response it logged A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts * fix(bedrock_mantle): run extra_body fields through the chat checks A chat-pinned Mantle request that nests n or reasoning_effort in extra_body now gets the same 400, drop, and effort-level handling as the top-level fields. The bridge test injects a MockTransport client instead of patching HTTPHandler.post. * test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels * fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort * fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request * fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode * fix(router): drop the debug log that wrote a request's model name from keep_registered_mode * test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort * fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down * test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks * test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them * fix(router): route a fine-tune the same whichever deployment registers first Main's register_model now seeds an unmapped fine-tune's row from its base model's built-in row, mode included, on the first registration. The mode guard read only the live row, so a responses pin marked the shared row only when it registered before its siblings. The guard now reads the row the registration merges onto, the live row or else the row it would seed. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 21:39:14 +00:00
Mateo Wang	ef1d3959a0	fix(count_tokens)!: answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens (#47163) ... * fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens * fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests Vertex has no count-tokens route for Mistral or codestral models, so they now get the same 400 as other partner models without a count API instead of a 404 that points operators at IAM. * fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens * fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model * fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors * fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer * fix(count_tokens): pass through an OpenAI provider error that omits param or code * fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route * test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy * test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes	2026-10-17 21:15:38 +00:00
Kerry Lu	1552b5b69f	fix(proxy): warn at boot when a config environment_variables key replaces a value already set in the process environment (#47165) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 20:52:01 +00:00
devin-ai-integration[bot]	4273996985	fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs (#41150) ... * fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): update payload expectations for user email metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(gcs_pubsub): add user_api_key_user_email to the spend log fixture Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): drop untyped metadata key args from the debug log to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): pass metadata keys to the debug log as a lazy %-style argument Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(spend_logs): store user email in spend logs only behind store_user_email_in_spend_logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(spend_logs): store user email in spend logs by default, drop store_user_email_in_spend_logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): store an empty user email as null, narrow the expected payload test diff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 13:26:09 -07:00
alberto	bbd446b476	fix(proxy): report requested model in streamed Responses response.model (#46662) ... * chore: Add failing test case Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Restamp nested response.model in streamed Responses events Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Add tests Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Run format and lint, and fix Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Type nested Responses restamp to satisfy basedpyright gate Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(proxy): restamp the nested model on streamed response.queued events Background streams send response.queued as a GenericEvent, so its nested response.model still carried the backend name. Also drop the new comments * fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events * test(proxy): drive typed events through the nested restamp no-op test * fix(proxy): restamp the nested model on guardrail block frames in streamed responses * test(proxy): assert a cache-polled background response reports the requested model * fix(proxy): restamp nested model on dict events and on every SSE data line --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 19:52:10 +00:00
yuneng-jiang	4846cfabc7	fix(vector-stores): keep proxy context out of provider vector store metadata (#47021) ... * fix(vector-stores): keep proxy context out of provider vector store metadata The proxy merged its request context (user_api_key_* fields, endpoint, user_agent, requester_ip_address, litellm_api_version) and the router's deployment info into the vector store request's metadata. For vector stores that field is also OpenAI's own metadata, which is capped at 16 keys, so a store created with a team key was full on creation and every later metadata update returned 400. Vector store routes now keep that context in litellm_metadata, like batches, files and responses, and the router does the same for avector_store_create. Provider-side metadata holds only what the caller sent. Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in script that removes the proxy-written keys from stores created before this change. It is a dry run unless --apply is passed. * fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values. * fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata Vector store routes now keep proxy context in litellm_metadata, but the v3 limiter only read agent_id from metadata, and logging let provider metadata tags or spend_logs_metadata replace the proxy's values. A string tags value also crashed get_request_tags. The limiter now prefers the proxy-resolved agent_id in litellm_metadata. Logging combines list tags from both buckets (metadata first, deduplicated) and merges dict spend_logs_metadata with litellm_metadata winning, ignoring values of the wrong type. * fix(vector-stores): narrow the logging fallback to mistyped caller values The tag union and spend_logs_metadata merge changed logged tags and spend_logs_metadata on every route that has both metadata buckets, deduped tags quadratically, and stopped background interaction polling from keeping string tags as sent. Caller values now win exactly as before, and the proxy's litellm_metadata tags or spend_logs_metadata only replace a caller value of the wrong type. The limiter reads agent ids through a pydantic model instead of a cast and ignores non-string ids. * fix(vector-stores): log the proxy's full tag list on litellm_metadata routes On routes that keep proxy context in litellm_metadata, the proxy builds litellm_metadata.tags from the caller's tags plus key, team and header tags, but get_request_tags preferred the caller's metadata.tags. A caller sending metadata.tags on a vector store call dropped the key's tags from request_tags, which tag spend tracking reads. get_request_tags now prefers a non-empty litellm_metadata tag list, and falls back to metadata.tags as before. * fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets Moving vector store proxy context into litellm_metadata broke shared readers (agent limits, request tags, merged logging metadata), and fixing them widened the change to every litellm_metadata route. This reverts the bucket move, the router change and the reader fixes, so proxy context stays in metadata exactly as on main. Vector store create and update now drop the keys LiteLLM itself writes (user_api_key*, requester and deployment keys, agent ids) from the metadata sent to the provider, in the shared optional-param helper both paths use. The cleanup script reuses the same key list. * fix(vector-stores): pass non-string metadata keys through the provider metadata filter * fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls * fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy * fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies * fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed	2026-10-17 12:51:47 -07:00
joshua-berri	4143e0869e	fix(batches): handle empty models when logging batch cost (#47155) ... Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-17 12:49:31 -07:00
mrinal-berri	a79dae764f	fix(proxy): route team key vector store calls to a team model of the store's provider (#46894) ... * test(vector-stores): reproduce team key search routed to a team model of the wrong provider A team key's model-less vector store search picks the team's first deployment, so a langgraph team model turns a Bedrock store search into "Vector store search is not supported for langgraph". Cover map_team_model picking by provider and a registered Bedrock store search reaching Bedrock with the registry's credentials. * fix(router): match model-less team deployment to the requested provider map_team_model now takes an optional custom_llm_provider. With no model given, it returns the first team deployment serving that provider, or None when none does, instead of the team's first deployment. Without a provider it behaves as before, so BYOK teams calling raw vector store IDs keep their team key. * fix(proxy): pass the vector store provider when mapping team models Vector store search, retrieve, update and delete pass the store's custom_llm_provider to map_team_model, so a team key's model-less call lands on a team deployment of that provider, or on the direct path with the store's own credentials when the team has none. Other model-less routes keep today's behavior. * fix(proxy): type the vector store provider passed to map_team_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vector-stores): cover team key vector store search provider routing end to end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): ruff format route_llm_request.py Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type vector store routing tests and mock Bedrock transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin model-less vector store calls when a public name mixes providers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): keep team provider routing in map_team_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 14:32:03 -05:00
ryan-crabbe-berri	f6ae25939e	feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) ... * feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries * fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5 A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow * refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass * chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py	2026-10-17 12:22:41 -07:00
Yassin Kortam	4b635f18e4	fix(mcp): bound paginated tools/list by LITELLM_MCP_TOOL_LISTING_TIMEOUT (#46513) ... Preserve explicit per-server timeouts while bounding paginated listing Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 12:16:46 -07:00
Yassin Kortam	7049a5ff4d	fix(mcp)!: register the gateway's own DCR client for bridge servers instead of relaying the caller's loopback redirect (#46940) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 12:14:28 -07:00
Yassin Kortam	03e3e71efa	fix(proxy): let proxy_admin_viewer store its own MCP credentials (#46994) ... Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 11:31:37 -07:00
devin-ai-integration[bot]	8226b2be9a	fix(otel): report cache hits at zero cost with a cache-hit flag in V1 and V2 spans (#43604) ... * fix(litellm): report zero cost and cache-hit attrs on cache-served spans * fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects gauntlet-mini on ace57d25f verified two live gaps. TextCompletionResponse and ResponsesAPIResponse carry _hidden_params as a HiddenParams object, so the dict-gated marker write skipped them and the served hit still stamped the full price. The marker write is now container-agnostic, which it already was for the partial-embedding stamp, and a regression test exercises the carrier end to end. _uncached_response_cost also restores billed_token_rates, zero_cost_warned, and the response_cost_failure_debug_information key alongside cost_breakdown, so the probe leaves no pricing state on the call it reads. * test(caching): cover cache-hit marker on HiddenParams carriers through the caching handler * fix(logging): keep the uncached price probe's restores to the state it can leak * fix(otel): emit V2 LLM-call span for cache hits with explicit cache attributes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): read cache payload fields defensively without casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(otel): clarify pyright suppression reason on cache payload reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): build the cache-hit view from typing_extensions.TypedDict so import works on Python 3.10 and 3.11 * fix(otel): report cache hits at zero cost everywhere and mark misses Cache hits now carry a zero cost breakdown and a zero hidden response cost, the /v1/messages hit cost header follows chat and responses hits, every span carries litellm.cache_hit, V2 hit spans drop server.address, an async hit keeps the provider-prefixed model name, and the V2 attribute budget counts preset keys * fix(otel): send a 0 cost header on cache hits and keep hits out of deployment TPM Cache hits now send x-litellm-response-cost: 0.0 on chat, responses and /v1/messages, while a fresh call whose cost is 0 still sends no header. The async hit path stamps the same model name as main again. Stamping the provider-prefixed name made a hit's tokens count toward the deployment's TPM limit, so repeated cached requests hit 429s under usage-based routing. Drop comments that restated fields and annotate new test values as Final. * fix(otel): keep the served pricing basis on a cache hit's cost breakdown A hit's breakdown now keeps service_tier, data_residency, vertex_location, and the discount and margin rates the served call recorded, with every amount at 0, so savings reports still read the basis. The hit header test now drives the live non-streaming header path. * fix(otel): send zero breakdown headers on a cache hit and give hit spans a fresh call's cost keys A hit's x-litellm-response-cost-original, -input, and -output headers now read 0.0 instead of going missing or carrying the full price. A hit's cost breakdown now comes from pricing the response as fresh, so every route exports the same keys, pricing basis, and rates with every amount at 0. * test(integration): assert zero cost and the cache-hit flag on cache-served OTel V1 and V2 spans * test: wait for the cache entry before the cache-hit request and build cost headers without a route_request double * test: annotate the new cache-hit unit test locals with Final --------- Co-authored-by: shrey kharbanda <shreshth@berri.ai> Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 18:25:32 +00:00
yucheng-berri	223800f916	fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) ... * fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130) Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main with the unit test moved to tests/unit after the test tree relocation * fix(presidio): label an overlapping run by its outermost detection, add integration coverage A run of overlapping detections now takes its entity type from the highest-scoring detection that no other detection in the run contains, matching Presidio's own anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER. The resolver returns typed detections instead of Any. * fix(presidio): drop the redundant comment in the numbered token loop * fix(presidio): bump a numbered token another part of the request already holds Each message and content part numbers its tokens from 1 into one request-wide pii_tokens map. Collapsing overlapping detections made an overlapping value's token well formed, so it collided with the same type in another part and both restored as the later value. A token that maps to a different value now moves to the next free number, and a token for the same value is reused * fix(presidio): give a clashing token a spare number instead of scanning for one Scanning upward from the clashing number probed every token an earlier part had claimed, so two parts with thousands of values of one type took seconds of synchronous work. No token in the map is numbered above the map's size, so numbers above both that size and the part's run count are always free * test(presidio): make the anonymizer double reject detections the analyzer never returned Pick the token in one conditional expression instead of reassigning it, and have the scripted anonymizer answer 500 when the proxy forwards spans that differ from the analyzer's, so the default-scope cells check what was sent --------- Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 11:21:03 -07:00
Kerry Lu	571ef3e2ae	fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost (#47013) ... * fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost * test(responses): keep the websocket unit test file on main's formatting * fix(responses): type the websocket frame guard for the lint gates * fix(responses): pass the websocket frame guard as a named parameter and mark the guardrail dict edits * fix(responses): bill a guardrail-rejected Responses WebSocket frame as a failed request with its guardrail cost * fix(responses): type the recorded guardrail records as Mapping for the lint gate * fix(responses): bill a rejected websocket frame through the spend logger only so the connection keeps its rate-limit slot * fix(responses): guard nested response.create frames, strip forged user_api_key_* frame metadata, copy guarded tools back and give each rejected frame its own spend row id * fix(responses): mark the spend logger's bare-dict failure hook for the type gate * fix(responses): reject response.create frames that carry input at both levels and keep billing of a rejected frame non-blocking * fix(responses): name the non-blocking billing catch for the strict lint gate * fix(responses): keep the admitted guardrail list on every websocket frame, a frame cannot replace it with its own guardrails list * test(responses): wait for the key spend total before asserting it in the websocket guardrail cost test * fix(responses): copy the admitted metadata shallowly per websocket frame, the OTEL parent span cannot be deep-copied * fix(responses): reject a WebSocket turn a pre_call guardrail rerouted to another model * fix(responses): bill a rerouted WS turn and carry Presidio pii_tokens from the frame guard to the socket unmasker * fix(responses): default pii_tokens through default_factory, Python 3.13 rejects a mappingproxy dataclass default * fix(responses): run the per-frame guardrail walk on managed Responses WebSockets too * fix(responses): unmask managed Responses WebSocket replies with the frame guard's Presidio pii_tokens * fix(responses): strip client guardrail cost records from WS frames so a blocked frame bills the real scan * fix(responses): forward a WS frame without the fields a guardrail deleted --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 11:00:51 -07:00