811 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
alexcastgonz	6762d65eb8	fix(vertex_ai): price streamed pass-through calls with the regional uplift (#44795) ... * fix(vertex_ai): price streamed pass-through calls with the regional uplift The streamed Vertex pass-through handler builds the response with stream_chunk_builder, which stamps a cost into _hidden_params without the vertex_location. The handler then prices the response again with the location from the URL, but only into kwargs, and the logger prefers the stamped cost. Streamed calls on a regional endpoint were billed at the global rate. Stamp the location-aware cost on the response in the shared generateContent payload builder so the logger records it. Fixes #44755 * style(vertex_ai): keep the response_cost assignment within the 120-char limit * refactor(vertex_ai): set the passthrough response cost with set_hidden_param Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this. * test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed * fix(proxy): price streamed Vertex pass-through spend like the non-streamed call Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls * fix(proxy): price custom-priced Vertex pass-through deployments at their own rates The stamped response_cost from the generateContent logging helper is returned before the logger checks custom pricing, so a deployment with custom input/output rates was billed at the cost-map price. The helper now passes custom_pricing and the deployment id into completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls, and deployments without custom pricing keep the regional uplift. * fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price * fix(proxy): give the new pyright ignore a reason for the LIT004 gate * test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included * fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same * fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini A top-level base_model in a pass-through body now reaches the logging params only for a key or team with allow_client_pricing_override, so a plain key cannot pick a cheaper model's price once the logger prices these calls. The test that pinned every litellm-owned body name into the logging params now leaves base_model to its own key-gated test. The Vertex handler now classifies a non-Google upstream host as Vertex only for Vertex resource paths, so a /gemini stream through a GEMINI_API_BASE gateway is priced as Gemini without the regional uplift. Adds the audit's integration spend test for Claude on Vertex, /v1beta, the named custom-priced /gemini deployment, and a cut OpenAI stream. * test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path. * fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model The Vertex pass-through handler picked Gemini pricing from the URL path, so a configured pass-through to a Vertex gateway with a short path (no /projects/ segment) was billed as Gemini, and its Claude streams at $0. The provider now comes from the route: a call through /gemini or to generativelanguage.googleapis.com prices as Gemini, and everything else as Vertex, the same rule main used. Matching the GEMINI_API_BASE host was rejected because one gateway can serve both APIs, and the path rule was rejected because short gateway paths look like Gemini paths. A non-string base_model (a number, list or object) made the cost calculation fail and bill $0 once the stamped stream costs were gone. get_base_model_from_metadata now ignores it and falls back to the deployment's base_model, so the call bills the list price. A 400 was the other option, but the logger runs after the upstream answered. * test(integration): bill a non-string body base_model at the list price on every pass-through route * fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info A key allowed client pricing can send metadata.model_info in a pass-through body. A non-string model_info.base_model made the cost lookup fail, and once the stamped stream and Gemini costs were gone those calls billed $0 on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores a base_model that is not a string, the same rule the root base_model got, so the call bills the list price. A model_info that is not an object (a string or a list) broke several logging steps that read it as a mapping, so a non-streamed /gemini call fired no success callback and no pass-through call wrote a spend row. The pass-through route now leaves a non-object model_info out of the logging metadata, so the call bills the list price. A 400 was the other option, but it would turn calls that answer 200 today into errors, and guarding every reader of model_info would spread the same check over many files. * fix: reject a non-object model_info before the provider call again The previous commit made the base_model lookup return None for a model_info that is not an object. That lookup also runs during call setup on every router route and SDK entrypoint, where such a value used to fail before any provider request. With the early return the call went upstream, the provider billed it, and the request then failed in a later reader that still expects an object. On /v1/messages and /v1/responses the key was also charged for a response the client never got. The lookup again reads base_model from any truthy model_info, so a non-object value fails during setup the way it did before this PR. A non-string base_model inside an object still bills the list price, and the pass-through route still drops a non-object client model_info at intake, so pass-through calls keep billing the list price. * fix: match the Gemini API host exactly when picking the pass-through cost provider The provider check accepted any host ending in generativelanguage.googleapis.com, so a Vertex host such as vertex-generativelanguage.googleapis.com was priced as Gemini and lost the regional uplift. It now accepts that host or a subdomain of it. * test: pin that a subdomain of the Gemini API host keeps the Gemini price --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 08:10:52 +00:00
yuneng-jiang	19f69ed6b6	test(integration): wait for every worker in Scenario.model (#47277)	2026-10-18 14:49:25 -07:00
joshua-berri	9c958aff34	fix(logging): deliver completed batch success callbacks consistently (#47192) ... * fix(batches): dispatch configured success callbacks during reconciliation * fix(batches): retain billed job claim when sync dispatch shuts down * fix(batches): finish reconciliation when logging executor is stopped * fix(logging): avoid duplicate custom hooks during async batch dispatch * fix: preserve sync-only custom callback delivery * fix: scope callback deduplication to per-call async recipients * refactor: type per-call async success callbacks * refactor: retain logging constructor callback compatibility * test: support current batch reconciliation metadata * fix(logging): honor per-logger redaction during sync success delivery * fix(logging): centralize completed success callback dispatch * fix(guardrails): avoid duplicate success callback dispatch --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-18 09:19:04 -07:00
Mateo Wang	7e6fb09da3	fix(streaming)!: map stream read timeouts to litellm.Timeout on chat, messages, and responses (#47198) ... * fix(streaming): map stream read timeouts to litellm.Timeout * revert(streaming): move the timeout mapping to a later commit * fix(streaming): map stream read timeouts to litellm.Timeout on chat, messages, and responses * fix(streaming): keep stream fallback and cooldown unchanged for transport timeouts A chat stream served through the Responses bridge now keeps mid-stream fallback on a stall, because the fallback exemption also covers a litellm.Timeout whose cause is an httpx transport timeout. The Responses stream hands the raw transport error to failure callbacks again, so router cooldown sees a stall as it did before. The 408 deployment info now lists stream_timeout. * test(integration): cover a stalled hosted_vllm stream on chat, Responses, and Messages with its spend log row * fix(router): keep a fallback deployment whose stream stalled out of cooldown * test(integration): cover bridged chat, fallback and cooldown for stalled hosted_vllm streams	2026-10-18 13:58:34 +00:00
Mateo Wang	5012083b72	feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes (#47185) ... * feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes * fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting The drop list now runs on the final upstream headers, after x-pass- expansion, through an upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a dropped value to Vertex. A malformed setting now answers 500 only for requests that carry anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working. * test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests * fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces * fix(vertex_ai): split drop-betas list entries on commas like the header	2026-10-18 12:51:27 +00:00
yuneng-jiang	eb114e638d	test(integration): look up owned cache keys and clean up sweep models (#47255) ... Three integration tests diffed the whole shared Redis keyspace (scan_iter or dbsize before and after), so a sibling test's cache write could satisfy or break them. Each now finds its own entry by a value only it wrote: the Gemini reasoning test by a per-run Gemini responseId, the max_messages test by a per-test answer text, and the S3 cache-hit test by its request marker The sweep auth tests created models through raw /model/new on the shared gateway and never deleted them. They now create them through Scenario.model, which deletes each model on exit, including when setup fails	2026-10-18 10:27:53 +00:00
yuneng-jiang	44d99475a5	test(integration): stop GET /__observations from draining the upstream log (#47256) ... * test(integration): stop GET /__observations from draining the upstream log GET /__observations used to empty the queue, so a second read in a test saw nothing and a "never reached upstream" check could pass whether or not the request was forwarded. GET now returns every request since the last reset and DELETE /__observations is the reset. An autouse fixture resets the worker upstream before each test, which also covers tests that read without the gateway fixture. Discarded reads used as resets become DELETEs, helpers that accumulated drained reads now read one snapshot, and tests with several phases reset between them. The three vacuous negative checks (rag allowlist, guardrail opt-out chaos, xAI batch output) now assert the absence on the same read that shows a request which did reach the upstream. * style(integration): space the observations unit test like its neighbours	2026-10-18 09:58:05 +00:00
moyai-berriai[bot]	f370235bb4	fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) ... * fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans * Preserve main's per-block streaming fixes and regression coverage * fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails * fix(bedrock): accept blank content on tool-call-only Converse turns A vendor that returns "" or [] where an assistant row carried only tool calls no longer fails the request closed. Also type the leaf lookup without Any and restore the original layout of the existing unit tests. * fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back. The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names. * fix(bedrock): keep new Converse helpers inside the type discipline rules * test(bedrock): pin Converse row edits that change shape, role or placement * fix(bedrock): type the Converse body once so its edits keep their reference * fix(bedrock): list request-field texts where their Converse row sits * fix(bedrock): send a reshaped Converse tool result as one text block A vendor that compresses or summarises a tool result, such as Headroom collapsing a JSON list into a table string, returned a 500 because the row could not be split back into the original text and JSON parts. The tool result now carries the vendor's string as one text block where its parts were, keeping images and documents in place. It still fails closed when the flat texts masked that tool result or the row changed its tool call id. * fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it A guardrail that masks a Converse tool description or schema now has that mask written into toolConfig, a tool it removes is dropped, and a tool it adds, renames or reorders is refused before Bedrock. Headroom, Compresr and TypeSafe get only the flat texts on Converse again, as on main, because the pass-through route has no loop to answer their retrieve tool * test(bedrock): refuse a guardrail that changes a Converse tool's kind * fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS and HiddenLayer scan only part of the rows or rebuild their returned texts from them, so on Converse they get the flat texts only, as on main. When a vendor returns changed rows, its texts are applied only if they keep the flat layout and change only rows the vendor also changed, so row-shaped texts of the same length no longer overwrite unrelated Converse fields. * test(bedrock): pin how flat texts sit next to rewritten Converse rows * fix(bedrock): keep the Bedrock guardrail on flat Converse texts With experimental_use_latest_role_message_only, the Bedrock guardrail places its mask by counting text parts per row, and Converse rows do not line up with the flat texts, so the mask could land in tool call arguments. Give it the flat texts only, as on main. * fix(bedrock): keep Akto on flat Converse texts Akto maps its masked payload back onto the flat texts by value, and a Converse row joins or serialises tool call arguments and JSON tool results, so a masked card there could not be placed and the request was blocked. Give Akto the flat texts only, as on main, where the mask is applied. * fix(bedrock): list request-field texts last again, as on main Flat texts listed additionalModelRequestFields strings right after the system prompt, so flat-text guardrails got a different order than on main. Akto's live verdict on a benign request depends on that order and started blocking it. Rows already keep their own order and texts that do not follow rows are dropped, so the flat texts go back to main's order. * fix(bedrock): list request-field texts next to their row for row readers Row readers that rebuild texts from rows could swap a masked request field with a masked user turn when the field strings came last. Row readers now get those strings right after the system prompt, where their row sits. Flat-text guardrails keep the end-of-list order. * fix(bedrock): list nested request-field texts in their row's order for row readers Row readers that rebuild texts from rows could write a masked request field onto a sibling key when the field held nested objects, because the texts walked the fields depth first and the row breadth first. Row readers now get those strings in the row's own order. * fix(bedrock): walk request fields breadth first instead of resolving each path Resolving every request-field string by its path revalidated the parent object per string, so a wide additionalModelRequestFields took seconds. The string walk now goes breadth first for row readers, the same order the row lists them, in one pass. A unit test pins that texts and rows map the same way when the vendor masks them differently. * fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry When a vendor returns changed rows whose texts do not line up with them, the rows win. A text the vendor masked on a field its rows left untouched (a tool call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead, unless the edited body holds the changed text elsewhere, as texts rebuilt from the rows do. * fix(bedrock): read same-length texts by position when rows drop a mask Matching a dropped text against any string of the edited body let a row mask on another field with the same token cancel it, so the raw tool call argument still went out. Texts of the length that was sent are now checked position by position, and only texts of another length, which cannot be placed, are matched by value. * test(bedrock): count each rebuilt text against the rows' edits * fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed PromptGuard rebuilds its returned texts from the user rows, and a vendor that does so while masking only row content let a card in a tool call argument reach Bedrock, which main masked. PromptGuard now gets the flat texts as on main, and a reply whose rows changed next to texts of another length fails closed, since what those texts mask cannot be checked. * test(bedrock): let the structured checks decide the unsafe tool call and missing row cases The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for texts that cannot be placed answered first and the tool call and row count checks went unexercised. * fix(bedrock): send request field strings to row readers after the conversation, where main puts them Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last, as main sends those strings in texts * test(bedrock): pin nested request field order for flat readers and the texts view Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader getting the row order or a texts view written in main's order fails * refactor(bedrock): ask each guardrail whether Converse should hand it rows Converse picked the guardrails that get only the flat texts from a list of provider names. CustomGuardrail now has supports_structured_messages, True by default, and the ten classes on that list override it to return False, so the handler asks the guardrail instead of the list. Every guardrail class gets the same answer as before * fix(bedrock): keep Singulr on flat Converse texts as on main Singulr wraps each flat text as a user message and its backend judges only the latest user message, so on main a Converse tool result was the turn it judged. Handed rows, the tool result arrives with role tool and the latest user row is the original question, so an injection in a tool result reached Bedrock. Singulr now returns False from supports_structured_messages * fix(bedrock): keep Qualifire on flat Converse texts as on main Qualifire builds its evaluation input from the last user row only when that row's content is a string, so Converse rows (content-block lists) gave an evaluation_id policy an empty input. Opt it out of structured messages so it gets main's flat texts again. * fix(bedrock): keep Conduct on flat Converse texts as on main The conduct-litellm-guard plugin scans only user rows, so with Converse rows a system prompt, assistant text or tool result left its scan. Opt it out of structured messages so it gets main's flat texts again, all sent as user turns. * fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects A guardrail that removed every tool, or the tool toolChoice names, produced an empty tools list or a dangling toolChoice that Bedrock answers with an opaque ValidationException. Refuse those rewrites before Bedrock, like other tool changes this route cannot carry. * fix(bedrock): list request-field strings in main's order for every guardrail The request-field row walked additionalModelRequestFields breadth first, and guardrails that get rows had their texts follow that order. With two or more strings the last text changed, so Ovalix, which blocks only when the last text is flagged, rewrote an injection main blocked. The row now walks the fields in the order main lists them in the texts, and the texts use main's order for every guardrail. * test(bedrock): pin a container's own request-field strings before its nested ones * test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools * fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock, converted the way chat completions converts tools, with cache points kept behind their tool. When every tool is removed, toolConfig is dropped and earlier tool calls and results become text, as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed. The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer hits the recursion limit. * test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly * fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply * fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail on_error: allow sent the original request on when the endpoint refused a guardrail rewrite it could not apply, so a Converse request the guardrail masked reached Bedrock raw. The refusal now counts as a fail on every endpoint. * fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say A rewrite the endpoint cannot apply is neither a content verdict nor a technical error, so on_fail next or allow and on_error allow must not send the original request on * test(bedrock): trim the Converse guardrail tests Drops the chaos and per-vendor integration cells and the duplicate unit cases, and leaves the existing handler tests as they are on main * fix(bedrock): put Converse request-field strings in the latest user message for row readers A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their own only when no user message has text. * fix(bedrock): create toolConfig when a guardrail adds the first Converse tool A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main. * test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows * fix(guardrails): keep Converse tool matching and write-back linear in request width Matching each returned tool rescanned the whole tool list, and every write-back step revalidated the list or object it passed through, so a request with thousands of tools or request fields held the event loop for seconds. Returned tools are now looked up by name in one index, and path steps read the child directly. * fix(guardrails): let later Converse guardrails scan guardContent text When a guardrail removes every tool on a Converse request that carries guardrailConfig, tool results are rewritten as guardContent blocks. The flat text extractor read only top-level text, so a later masker in the same request never saw them and a card in a tool result reached Bedrock unmasked. Chat completions and /v1/messages keep tool results visible to later guardrails after the tools are removed. The extractor now reads guardContent text the same way the structured rows already do, so flat-text guardrails also scan guardContent text a caller sends. * fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer, Qualifire) check what the caller wrote When a guardrail removes every tool, earlier tool calls and results are now written as plain text blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own guardrail keeps checking every message, since a single guardContent block makes it skip all plain text * fix(guardrails): split a newline-joined Converse message back into its parts A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline. * fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped and the history turned into text only when the request is signed, as chat completions does at request build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400 Request-field strings join the latest user message that has text when a tool-result-only turn follows it, so a guardrail judging only that message reads the caller's text with them A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500 Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 20:38:14 -07:00
yuneng-jiang	fc27dfdc7d	test(integration): give each owned proxy a private database and Redis by default (#47213) ... * test(integration): give each owned proxy a private database and Redis by default owned_proxy and owned_proxy_process now boot on a scratch database and a per-proxy redis-server unless the caller opts into the worker's state with state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips DATABASE_URL. Under the replica profile the private database gets the writer and reader roles so the reader path stays covered. Gateway carries the database_url its proxy writes to, and the Scenario cleanup checks and delete_key_if_present read from it, so they keep checking something on a private database. The 189 test files with owned proxy sites that depend on the worker's database or Redis opt in with a shares_worker_state module marker, which a module-scoped conftest fixture turns into the shared state. The 7 helper modules with such sites opt in per call, since conftest fixtures and other test files reach them. * test(integration): reach session fixtures with the shared marker and keep the worker retry default The module-scoped fixture that applied shares_worker_state ran after session-scoped fixtures, so proxies those booted went private in marked files. The marker is now applied around each marked item's setup, call and teardown, which covers fixtures of every scope. Owned proxies on the worker database inherited the worker gateway's router_settings.num_retries of 0 from it. A private proxy now gets the same default in its config copy unless the config sets num_retries itself. Five providers files read the owned proxy's spend rows or response cache through the worker's database or Redis and opt into sharing.	2026-10-18 03:37:41 +00:00
devin-ai-integration[bot]	4bebcdf757	fix(databricks): use the native Anthropic route for Claude /v1/messages (#46091) ... * fix(databricks): use the native Anthropic route for Claude /v1/messages Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge * fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again * fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent * fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials Unity Catalog model-service names now reach the AI Gateway's native Anthropic route instead of a serving-endpoints path that answers 404 for them, and the native route resolves the key and base through the same order the chat route uses, so environment-only deployments keep working. The native route is limited to Databricks foundation-model Claude names, an unlisted provider forwards safeguards and the beta as sent, and the ten betas the workspace rejects are marked unsupported for Databricks in the beta map and its Rust mirror. * test(rust): expect Databricks to drop the advanced-tool-use beta * fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback Only the OpenAI-compatible JSON providers drop safeguards now, so a deployment opted into /v1/messages passthrough forwards the field and the beta as sent. When a route drops safeguards and the proxy has a classifier model, the proxy answers the verdict itself, as the chat completions bridge did. The Databricks Messages route also reads the databricks_user_agent param like the chat route. * fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL * fix(databricks): drop container on the native Messages route, as the chat completions bridge did * fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name * test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire * test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas * fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth Databricks chat completions, called directly or through the Messages bridge, no longer receive the client's anthropic-beta and anthropic-version, as at the merge base. The native Messages route still forwards them, filtered by the beta map, without counting them as caller credentials, so a deployment with no api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to the native route only for Unity Catalog names, matching the chat route. * test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route * fix(databricks): keep client anthropic headers off an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and that opts into /v1/messages with supported_endpoints falls back to the OpenAI-like passthrough. That config does not filter betas through the beta map, yet it received the native-only Databricks header entry, so Claude Code's prompt-caching-scope beta reached the upstream raw and got a 400 where the merge base answered 200. Each Messages config now says whether it is the provider's native route, the OpenAI-like passthrough says no, and the handler passes that answer when it reads provider-specific headers, so native-only entries reach only native configs. * test(databricks): cover client anthropic betas on an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and whose model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m betas through the proxy and asserts the caller gets the upstream's message and the gateway receives no anthropic-beta, as on the merge base. The scripted gateway answers 400 on the prompt-caching-scope beta, as the real one does * fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do. * test(databricks): cover a loaded beta map that marks the safeguards beta unsupported A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict. * test(anthropic): annotate the MCP beta map recordings as Final * fix(anthropic): drop safeguards on routes without the classifier only under drop_params A route whose upstream does not answer safeguards (a JSON provider, or a map that marks the dangerous-tool-use beta unsupported) now forwards the field and the beta as the merge base did unless drop_params is set or the proxy runs its own classifier, matching how litellm treats any unsupported param. Databricks keeps dropping both, since its merge base bridge never forwarded the field. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-18 02:44:38 +00:00
Mateo Wang	68a8b12ab3	fix(proxy)!: end a chat completions stream with an error event when a guardrail blocks mid-stream (#46953) ... * fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out. * fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE] A guardrail block before the first chunk is raised again, so its response stays the 400 event-stream body it is on main. A mid-stream error event now carries the failure hook's replacement error when one is returned, is followed by data: [DONE] on OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details JSON cannot encode are written as strings instead of breaking the event. * fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400 * fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band * fix(proxy): encode non-JSON guardrail details in early stream error frames An early guardrail block whose detail holds a value JSON cannot encode sent 400 headers and then cut the body. The error frame now writes such a value as its string form, the same as the mid-stream event. The keepalive test now runs on a virtual clock instead of real sleeps. * fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload. * fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client * fix(proxy): print CLI stream errors without Rich highlighting or emoji codes * test(proxy): cover the CLI's plain-text 400 error print * test(proxy): annotate the new client stream test locals as Final * fix(proxy): send the failure hook replacement on Responses streams and before the first chunk * fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception * fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it A non-HTTP failure whose hook returns an HTTP error was raised before the first chunk, which create_response turned into the hook's status but which a bare StreamingResponse route such as /queue/chat/completions answered as a 200 with zero bytes. Only the original HTTP errors raise now. The replacement goes out as the stream's error event, which create_response still answers with the hook's status as a JSON error. Adds audit round 2's integration cells for failure hook replacements on chat and Responses streams, built-in exceptions on Responses streams, and the Python client. * test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent	2026-10-17 18:00:12 -07:00
Mateo Wang	75faaeab64	feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) ... * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop	2026-10-17 17:11:27 -07:00
Mateo Wang	52f5a28679	fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) ... * fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it * fix(bedrock): retry with the json tool when Bedrock refuses the native output schema Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller * fix(bedrock): make the native output retry send what an opted-out deployment sends The retry after a native output rejection now reruns the same optional param mapping completion() ran, with the deployment model_info opted out of supports_bedrock_native_structured_output, so tool_choice, effort and the json tool schema match an opted-out deployment instead of a hand-built tool. Each config keeps only the predicate that decides when native output came from the opt-in key. Bedrock batch records for Anthropic models map with the opt-in turned off, since a batch job has no retry. Models on supports_native_structured_output keep native output in batch * fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged * fix(bedrock): give sync converse stream errors the upstream body as text * fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising A Converse json_schema request whose caller outputConfig failed litellm's shape check raised a pydantic error before the request was sent, so the caller got a 500 where the merge base passed Bedrock's 400 through. The opt-in check now treats that outputConfig as carrying no native output. Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of native output, and extends the Bedrock native output integration tests. * fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported A caller can send its own outputConfig on Converse. When Bedrock refused that schema on an opt-in deployment, the retry still carried the caller's textFormat, so Bedrock got the same request twice and the warning told the operator to turn off an opt-in that was not the cause. The retry is now skipped when the opted-out request would still carry a native textFormat, so Bedrock's 400 reaches the caller after one call as on the merge base. The opt-out advice is logged only for the account-level "Extra inputs are not permitted" rejection. * fix(bedrock): keep the caller's continuation message on the native output retry The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt. * test(bedrock): pin additional_drop_params on the /v1/messages native output retry * fix(bedrock): give the Converse native output retry the fake stream timeout	2026-10-18 00:06:07 +00:00
Shivam Rawat	c58fc0c996	fix(rag)!: use team provider credentials and allow viewer and JWT team access (#45497) ... * fix(rag): use team provider credentials and allow viewer and JWT team access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rag): assert persisted managed store data excludes team credentials Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(jwt): run real team model checks in the route denial test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rag): scope team provider credentials to openai vector stores Bedrock and other providers kept their pre-PR credential and region resolution: a team bedrock deployment's region no longer overrides the caller's region on RAG query. Query tests now fake the provider at the HTTP boundary with a real vector store registry. * fix(rag): use only the team's own OpenAI deployment for RAG credentials * fix(rag): ignore caller credential fields the query search never uses when picking team credentials * fix(rag): treat blank credential fields as absent when picking team credentials * test(rag): delete the stores the integration ingests create before the owned proxy exits * fix(proxy): let a restricted key use the team deployment its wildcard or model covers A key restricted to gpt-4o on a team whose own OpenAI deployment is openai/* never resolved that deployment's credentials, because the key allowlist was matched against the deployment's public name literally. The allowlist now matches a wildcard on either side the way request authorization does, through the auth layer's own helper, so /rag, /files and /batches resolve the team deployment for every key that may call a model on it. * fix(batches): read a restricted key's batch input through its team deployment The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment * fix(rag): load deployments oldest first, reject file_id on stores that cannot take one, and show ingest errors in the dashboard * fix(rag): pick the team deployment with the lowest id, return 400 for caller mistakes, keep the ingest id in errors, and show partial success in the dashboard * fix(rag): pick the team's oldest OpenAI deployment so adding one never moves the team * fix(rag): keep config file team deployments first and try JWT teams in sorted order A team's config file OpenAI deployments now win in file order, as on the base, before its DB ones oldest first, so upgrading never moves a team that has both. A JWT in several teams tries them in sorted order, so every worker resolves the same team. * fix(rag): return 400 for Bedrock, S3 Vectors and Gemini store config mistakes on ingest * fix(rag): keep forwarded provider keys on /rag/query and mixed JWT team claims working A forwarded x-api-key now counts as a caller credential, so the /rag/query search uses it over the team deployment as at the merge base. Sorting the JWT team claim compares entries as text, so a stray number or null no longer turns into a 401. A Gemini store with no proxy key is a 500 again, and a non-string vector_store_id keeps the merge base's 400 message. * fix(rag): let a forwarded provider key win over team credentials on files, batches and ingest * fix(rag): keep team credentials when a request carries its own key or api_base A forwarded x-api-key or a body api_key now replaces only the key of the team deployment, keeping its api_base, as chat completions does. A body litellm_credential_name no longer turns team credentials off on /v1/batches. * fix(batches): keep the team key on the team host when a request sends only api_base A request api_base now replaces the team deployment's api_base only when the request also brings its own api_key, and a non-string api_key or api_base is ignored, as on the merge base. * fix(files): ignore a blank request api_key or api_base next to team credentials * fix(files): answer 400 for a request api_key that cannot be sent as an HTTP header * fix(rag): honor a body drop_params for a forwarded key that cannot be an HTTP header * fix(rag): read the request field for drop_params only on multipart ingest bodies * fix(rag): type the ingest body read for drop_params * fix(batches): count a model-routed batch's input with the deployment key it is created with * fix(batches): reject a batch create whose x-litellm-model headers name different models * fix(rag): treat a null provider and whitespace store credentials as unset when picking team credentials * refactor(files): type the team deployment the credential resolver ranks --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 23:51:48 +00:00
Mateo Wang	16645cc963	fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) ... * fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts	2026-10-17 22:08:02 +00:00
Mateo Wang	897dda7920	fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263) ... * fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row A deployment with an explicit custom_llm_provider and model_info.mode: chat used to overwrite the shared litellm.model_cost row, moving every other deployment and plain SDK calls onto Chat Completions until restart, because the downgrade guard read a double-prefixed key and never fired. The guard now reads the real key, and a deployment's own mode decides its routing in responses_api_bridge_check, except on chatgpt and github_copilot where Chat Completions is dead for responses-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): keep the bridge-check mode tests in the sharded bridge flag file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the downgrade guard's mode from the row registration writes A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit responses/ prefix now also outranks a deployment's chat pin. * fix(router): let an explicit responses/ prefix outrank a registered chat row The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the lookup found mode chat and the request went to Chat Completions as responses/<model>. * fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions. * fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them. * fix(router): skip a non-string model_info.mode when keeping the registered mode * fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API A Mantle row that also serves the Responses API now follows the GPT-5 tool and reasoning summary bridge rules, since Mantle Chat Completions rejects function tools on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat gets a dict reasoning_effort as its effort level. The opt-in auto summary asks each Responses config for its level, auto on Mantle's OpenAI path, so a client's own detailed summary follows drop_params as at base. No mode replaces a registered responses mode, and chatgpt and github_copilot rows take any pin until they are responses, which restores the Copilot chat opt-out. * fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort * test(integration): run the Mantle region-prefixed v1 base case again This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip * fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request * fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request * fix(router): keep a non-string deployment mode off the shared routing row A deployment whose model_info.mode was an int, a list, or an object still reached the shared cost-map row, so an unpinned sibling on the same backend stopped routing to the Responses API. The shared registration now drops a mode that is not a string, and the deployment keeps its own raw value. Adds integration cells for per-deployment routing across the OpenAI and Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream outage, config changes under traffic, and a worker kill plus restart. * test(integration): find each routing cell's spend row by the response it logged A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts * fix(bedrock_mantle): run extra_body fields through the chat checks A chat-pinned Mantle request that nests n or reasoning_effort in extra_body now gets the same 400, drop, and effort-level handling as the top-level fields. The bridge test injects a MockTransport client instead of patching HTTPHandler.post. * test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels * fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort * fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request * fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode * fix(router): drop the debug log that wrote a request's model name from keep_registered_mode * test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort * fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down * test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks * test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them * fix(router): route a fine-tune the same whichever deployment registers first Main's register_model now seeds an unmapped fine-tune's row from its base model's built-in row, mode included, on the first registration. The mode guard read only the live row, so a responses pin marked the shared row only when it registered before its siblings. The guard now reads the row the registration merges onto, the live row or else the row it would seed. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 21:39:14 +00:00
yuneng-jiang	44fcbc30b0	test(mcp): replace the live Linear MCP OAuth e2e tests with integration coverage (#47154) ... * test(mcp): move the MCP OAuth happy path from a live Linear e2e to an integration test GitHub Actions now runs unit tests only, and the MCP OAuth happy path needed a captured Linear browser login to click through a real consent page. The OAuth flows it exercised are already integration tests against an owned authorization-server double, except one behavior: the stored per-user token must be served by a proxy process that never saw the consent, with no shared Redis to fall back on. That case is now in test_mcp_oauth_flows.py The e2e test, its owned-gateway helper, its workflow, the mcp_oauth_live marker and the JUnit gate script only that workflow used are removed * test(mcp): chat completions run an OAuth server tool with the caller's own stored token * test(e2e): drop the Linear chat-completion MCP OAuth e2e and its helpers The behavior it checked, a chat completion running an OAuth server's tool with the caller's stored per-user token, is now an integration test in test_mcp_llm_endpoints.py that needs no saved Linear browser login * test(mcp): decode the stored per-user credential as urlsafe base64 The plaintext check used b64decode, which drops the - and _ of a urlsafe value and raised on roughly one ciphertext in five * test(mcp): drop the chat OAuth integration test that duplicates test_mcp_slots F2-chat security/test_mcp_slots.py already sends a per-user OAuth token through /v1/chat/completions and asserts the peer receives it. Every mutant the new test killed was also killed by the existing suite, so it added no signal	2026-10-17 14:29:59 -07:00
yucheng-berri	b3c0f3457d	fix: handle missing choice indices in streamed output recording (#46947) ... * fix: handle missing choice indices in streamed output recording * test: verify recorded partial output on client disconnect * test: correlate streaming regressions with persisted request IDs * test: build progress-stream configs without mutation --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 14:17:46 -07:00
Mateo Wang	ef1d3959a0	fix(count_tokens)!: answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens (#47163) ... * fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens * fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests Vertex has no count-tokens route for Mistral or codestral models, so they now get the same 400 as other partner models without a count API instead of a 404 that points operators at IAM. * fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens * fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model * fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors * fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer * fix(count_tokens): pass through an OpenAI provider error that omits param or code * fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route * test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy * test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes	2026-10-17 21:15:38 +00:00
yuneng-jiang	159f185ff4	test(integration): report uncollected shard files as collection errors instead of crashing xdist (#47075) ... The integration conftest raised UsageError from pytest_collection_modifyitems when a shard-selected file collected nothing. Under xdist that ran on the workers after the unfiltered item list was already published, so the worker ended its session with tests still queued and the controller hit xdist's crashitem assertion, naming an unrelated queued test and hiding the real ImportError. A selected file that collects nothing now produces a failed collection report instead. Files whose own collection already failed keep pytest's real error. Serial runs stop with "Interrupted: N errors during collection", xdist runs report the error and exit 1, and both still fail the shard.	2026-10-17 14:00:48 -07:00
yuneng-jiang	0431e75716	fix(tests): wait for every thread of a killed worker before treating it as gone (#47148) ... WorkerKill returned once the victim's main thread was a zombie. On Linux the process keeps its sockets open until its last thread exits, so a pooled keep-alive connection to the killed worker still looked idle and the next request through it got ECONNRESET. A zombie now counts as gone only once num_threads() is at most 1. Where psutil cannot read threads of a zombie (macOS) it raises ZombieProcess, a NoSuchProcess, so the old behavior holds	2026-10-17 13:51:53 -07:00
yuneng-jiang	603ba6fe00	test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy (#47152) ... * test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy These two files register deployments with their own model settings on the gateway. They now boot a module-scoped proxy on a scratch database, so their setup cannot affect other tests that share the worker's proxy * test(integration): seed the isolated gateway with the same router settings as the worker gateway	2026-10-17 13:51:31 -07:00
alberto	bbd446b476	fix(proxy): report requested model in streamed Responses response.model (#46662) ... * chore: Add failing test case Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Restamp nested response.model in streamed Responses events Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Add tests Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Run format and lint, and fix Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Type nested Responses restamp to satisfy basedpyright gate Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(proxy): restamp the nested model on streamed response.queued events Background streams send response.queued as a GenericEvent, so its nested response.model still carried the backend name. Also drop the new comments * fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events * test(proxy): drive typed events through the nested restamp no-op test * fix(proxy): restamp the nested model on guardrail block frames in streamed responses * test(proxy): assert a cache-polled background response reports the requested model * fix(proxy): restamp nested model on dict events and on every SSE data line --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 19:52:10 +00:00
mrinal-berri	a79dae764f	fix(proxy): route team key vector store calls to a team model of the store's provider (#46894) ... * test(vector-stores): reproduce team key search routed to a team model of the wrong provider A team key's model-less vector store search picks the team's first deployment, so a langgraph team model turns a Bedrock store search into "Vector store search is not supported for langgraph". Cover map_team_model picking by provider and a registered Bedrock store search reaching Bedrock with the registry's credentials. * fix(router): match model-less team deployment to the requested provider map_team_model now takes an optional custom_llm_provider. With no model given, it returns the first team deployment serving that provider, or None when none does, instead of the team's first deployment. Without a provider it behaves as before, so BYOK teams calling raw vector store IDs keep their team key. * fix(proxy): pass the vector store provider when mapping team models Vector store search, retrieve, update and delete pass the store's custom_llm_provider to map_team_model, so a team key's model-less call lands on a team deployment of that provider, or on the direct path with the store's own credentials when the team has none. Other model-less routes keep today's behavior. * fix(proxy): type the vector store provider passed to map_team_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vector-stores): cover team key vector store search provider routing end to end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): ruff format route_llm_request.py Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type vector store routing tests and mock Bedrock transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin model-less vector store calls when a public name mixes providers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): keep team provider routing in map_team_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 14:32:03 -05:00
ryan-crabbe-berri	f6ae25939e	feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) ... * feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries * fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5 A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow * refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass * chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py	2026-10-17 12:22:41 -07:00
devin-ai-integration[bot]	84d91bbe87	fix(vertex): match standalone 403 status tokens instead of any substring (#47134) ... * fix(vertex): match standalone 403 status tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex): keep text 403 fallback for wrapped 500 errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex): keep BadRequestError for explicit 403 text matches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex): let structured body codes override the 403 text fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 19:17:18 +00:00
devin-ai-integration[bot]	9e9dccb9d4	fix(register_model)!: keep bedrock geo deployments priced across price data reloads (#41523) ... * fix(register_model): keep bedrock geo deployments priced across price data reloads A request-time pricing registration inserting into litellm.model_cost while a price data reload walked it could abort the reload (dictionary changed size during iteration in add_known_models), or trip the lookup map rebuild inside get_model_info so the deployment replay fell through to the unmapped path and wrote a raw bedrock/au.anthropic.<model> entry with no token pricing, a $0 shadow re-rolled on every later reload. Walk snapshots of the map, bind the case-insensitive lookup map locally, and let the unmapped path inherit the whole built-in entry of a provider get_model_info would accept. * fix(register_model): keep synthesized zero costs out of inherited built-in pricing An unmapped alias that inherits its built-in entry now takes only the token prices the catalogue really carries, so a synthesized zero never lands next to the caller's custom price. The router regression test reports the writer's failures instead of swallowing them, and the local names carry Final. * fix(register_model): match inherited built-ins by the key provider prefix when the value names none * fix(utils): resolve a tripped lookup from the key's own catalog row before its region sibling * fix(register_model): keep custom flat prices above inherited tier rates * fix(register_model): keep the other direction's tier rates when a custom flat price covers one * fix(register_model): keep a catalog row's tiers when its lookup trips, and snapshot request-path cost map walks A tripped get_model_info on a key that is itself a catalog row now inherits that row whole, so a custom flat price bills a long prompt at the row's above-threshold rate on both lookup paths instead of flipping until the next reload. The fal_ai, fireworks, ollama and gpt-5 request-path walks over litellm.model_cost iterate a snapshot so a concurrent reload cannot raise mid-request. * fix(register_model): price route-prefixed geo keys on a tripped lookup and replace cost map entries atomically * fix(register_model): move every alias of a cost map entry to its registered price * perf(register_model): read alias names from the shared entry instead of scanning the cost map Each registration scanned every cost map entry to find the names sharing it, about 1 ms on the 4,189-entry map, and a proxy request to a custom-priced deployment registers twice. Aliased entries now carry the names they are served under, so a registration reads them directly, and an alias the caller pointed at its own entry keeps that entry. The tests this PR added lose their docstrings * fix(utils): keep capability flags and write through on a tripped lookup in register_model A key the lookup cannot map now seeds its registered entry from the capability rules its unregistered lookup resolves to, so registering a region alias no longer turns off supports_system_messages and similar flags. A lookup that trips mid-walk on a key naming a catalog row writes into that row instead of leaving a shadow key that goes stale. Adds regression tests for both and for the request-path snapshot walks. * fix(utils): retry a tripped registration lookup and rebuild a registration a reload races A lookup a concurrent write trips is retried instead of writing into the catalog row its key names, so a per-request price never reprices a shared row. The inherited entry leaves the matched row's provider behind, so a provider-less azure_ai key stays priced, and a registration whose map a reload swapped mid-build is rebuilt from the new map. * fix(utils): write each registration into the map it was built from and rebuild it when a reload replaced that map * test(integration): price data reload keeps every registered deployment priced --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 19:10:04 +00:00
devin-ai-integration[bot]	8226b2be9a	fix(otel): report cache hits at zero cost with a cache-hit flag in V1 and V2 spans (#43604) ... * fix(litellm): report zero cost and cache-hit attrs on cache-served spans * fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects gauntlet-mini on ace57d25f verified two live gaps. TextCompletionResponse and ResponsesAPIResponse carry _hidden_params as a HiddenParams object, so the dict-gated marker write skipped them and the served hit still stamped the full price. The marker write is now container-agnostic, which it already was for the partial-embedding stamp, and a regression test exercises the carrier end to end. _uncached_response_cost also restores billed_token_rates, zero_cost_warned, and the response_cost_failure_debug_information key alongside cost_breakdown, so the probe leaves no pricing state on the call it reads. * test(caching): cover cache-hit marker on HiddenParams carriers through the caching handler * fix(logging): keep the uncached price probe's restores to the state it can leak * fix(otel): emit V2 LLM-call span for cache hits with explicit cache attributes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): read cache payload fields defensively without casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(otel): clarify pyright suppression reason on cache payload reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): build the cache-hit view from typing_extensions.TypedDict so import works on Python 3.10 and 3.11 * fix(otel): report cache hits at zero cost everywhere and mark misses Cache hits now carry a zero cost breakdown and a zero hidden response cost, the /v1/messages hit cost header follows chat and responses hits, every span carries litellm.cache_hit, V2 hit spans drop server.address, an async hit keeps the provider-prefixed model name, and the V2 attribute budget counts preset keys * fix(otel): send a 0 cost header on cache hits and keep hits out of deployment TPM Cache hits now send x-litellm-response-cost: 0.0 on chat, responses and /v1/messages, while a fresh call whose cost is 0 still sends no header. The async hit path stamps the same model name as main again. Stamping the provider-prefixed name made a hit's tokens count toward the deployment's TPM limit, so repeated cached requests hit 429s under usage-based routing. Drop comments that restated fields and annotate new test values as Final. * fix(otel): keep the served pricing basis on a cache hit's cost breakdown A hit's breakdown now keeps service_tier, data_residency, vertex_location, and the discount and margin rates the served call recorded, with every amount at 0, so savings reports still read the basis. The hit header test now drives the live non-streaming header path. * fix(otel): send zero breakdown headers on a cache hit and give hit spans a fresh call's cost keys A hit's x-litellm-response-cost-original, -input, and -output headers now read 0.0 instead of going missing or carrying the full price. A hit's cost breakdown now comes from pricing the response as fresh, so every route exports the same keys, pricing basis, and rates with every amount at 0. * test(integration): assert zero cost and the cache-hit flag on cache-served OTel V1 and V2 spans * test: wait for the cache entry before the cache-hit request and build cost headers without a route_request double * test: annotate the new cache-hit unit test locals with Final --------- Co-authored-by: shrey kharbanda <shreshth@berri.ai> Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 18:25:32 +00:00
yucheng-berri	223800f916	fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) ... * fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130) Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main with the unit test moved to tests/unit after the test tree relocation * fix(presidio): label an overlapping run by its outermost detection, add integration coverage A run of overlapping detections now takes its entity type from the highest-scoring detection that no other detection in the run contains, matching Presidio's own anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER. The resolver returns typed detections instead of Any. * fix(presidio): drop the redundant comment in the numbered token loop * fix(presidio): bump a numbered token another part of the request already holds Each message and content part numbers its tokens from 1 into one request-wide pii_tokens map. Collapsing overlapping detections made an overlapping value's token well formed, so it collided with the same type in another part and both restored as the later value. A token that maps to a different value now moves to the next free number, and a token for the same value is reused * fix(presidio): give a clashing token a spare number instead of scanning for one Scanning upward from the clashing number probed every token an earlier part had claimed, so two parts with thousands of values of one type took seconds of synchronous work. No token in the map is numbered above the map's size, so numbers above both that size and the part's run count are always free * test(presidio): make the anonymizer double reject detections the analyzer never returned Pick the token in one conditional expression instead of reassigning it, and have the scripted anonymizer answer 500 when the proxy forwards spans that differ from the analyzer's, so the default-scope cells check what was sent --------- Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 11:21:03 -07:00
devin-ai-integration[bot]	cd20ecace1	test(cost): add tier 1 cost tracking cells and sync usage shapes for 2026-10-17 (#47141) ... * test(cost): add tier 1 cost tracking cells and sync usage shapes for 2026-10-17 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): restore tiered cost cases and revert out of scope usage edits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): add remaining tier 1 endpoint cells for 2026-10-17 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 11:11:46 -07:00
devin-ai-integration[bot]	836f326611	test(integration): remove BUG skips for 2 tests that pass on main (#47142) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 11:11:16 -07:00
Kerry Lu	571ef3e2ae	fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost (#47013) ... * fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost * test(responses): keep the websocket unit test file on main's formatting * fix(responses): type the websocket frame guard for the lint gates * fix(responses): pass the websocket frame guard as a named parameter and mark the guardrail dict edits * fix(responses): bill a guardrail-rejected Responses WebSocket frame as a failed request with its guardrail cost * fix(responses): type the recorded guardrail records as Mapping for the lint gate * fix(responses): bill a rejected websocket frame through the spend logger only so the connection keeps its rate-limit slot * fix(responses): guard nested response.create frames, strip forged user_api_key_* frame metadata, copy guarded tools back and give each rejected frame its own spend row id * fix(responses): mark the spend logger's bare-dict failure hook for the type gate * fix(responses): reject response.create frames that carry input at both levels and keep billing of a rejected frame non-blocking * fix(responses): name the non-blocking billing catch for the strict lint gate * fix(responses): keep the admitted guardrail list on every websocket frame, a frame cannot replace it with its own guardrails list * test(responses): wait for the key spend total before asserting it in the websocket guardrail cost test * fix(responses): copy the admitted metadata shallowly per websocket frame, the OTEL parent span cannot be deep-copied * fix(responses): reject a WebSocket turn a pre_call guardrail rerouted to another model * fix(responses): bill a rerouted WS turn and carry Presidio pii_tokens from the frame guard to the socket unmasker * fix(responses): default pii_tokens through default_factory, Python 3.13 rejects a mappingproxy dataclass default * fix(responses): run the per-frame guardrail walk on managed Responses WebSockets too * fix(responses): unmask managed Responses WebSocket replies with the frame guard's Presidio pii_tokens * fix(responses): strip client guardrail cost records from WS frames so a blocked frame bills the real scan * fix(responses): forward a WS frame without the fields a guardrail deleted --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 11:00:51 -07:00
devin-ai-integration[bot]	656c63c0b4	test(translation): add claude-haiku-5-5 vertex ai cases (#47136) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 17:53:59 +00:00
devin-ai-integration[bot]	fa716ec7c6	test(translation): allow anthropic usage speed in cross-model contract (#47130) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 17:15:05 +00:00
devin-ai-integration[bot]	46ae68021e	fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) ... * fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events When the proxy renders a complete Anthropic Messages response as an SSE stream for a client that asked to stream, the message_delta event carried only stop_reason and stop_sequence. A real Anthropic stream puts safeguard_results and stop_details inside message_delta.delta, so a client such as Claude Code auto mode lost the server-side classifier verdict and a refusal lost its stop_details. Both stream builders now copy those keys from the source response through one shared helper when they are present. * fix(guardrails): keep safeguard_results and stop_details when re-emitting a rewritten Anthropic stream A post-call guardrail that rewrites an Anthropic stream (presidio, bedrock, tool permission, model armor) now carries the upstream message_delta safeguard_results and stop_details into the re-emitted stream. The router fallback message_start no longer leaks those final fields early * fix(guardrails): keep stop_details and safeguard_results in rewritten Anthropic streams without explanations A guardrail that rewrites a /v1/messages stream now carries stop_details (type and category) and the dangerous_tool_use verdicts for the tool calls it still delivers, with every explanation dropped so a masked stream never echoes the unmasked text through those fields. * fix(guardrails): copy only upstream stop_details and forward-compatible safeguard_results into rewritten Anthropic streams A rewritten stream now takes stop_details and safeguard_results from the upstream message_delta alone, so a refusal the upstream sent without stop_details is never given one by the OpenAI round trip. Unknown safeguard types, statuses, reasons and extra fields pass through, a verdict follows a tool call only when its input is unchanged, and tool_permission keeps the explanations since it rewrites no text. * fix(guardrails): apply the safeguard_results and stop_details rules to policy pipeline stream rewrites * fix(router): deliver a refused stream built from a finished agentic turn instead of falling back Websearch, MCP and advisor streams replay a complete reply whose tools already ran, so a refusal fallback would run every tool again. Also give the recovered response's message_start a null stop_details as the API sends it, drop safeguard_results a masking rewrite cannot read, and add the audit cells for synthesized streams. * test(router): annotate the finished-turn refusal stream test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 16:48:37 +00:00
devin-ai-integration[bot]	5b042c0db6	fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards (#46102) ... * fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards * test(rust): check the dangerous-tool-use beta reaches the wire with safeguards * fix(rust): derive the dangerous-tool-use beta only for hosts with a beta map column DeepSeek and the other hosts without a column in anthropic_beta_headers_config.json keep forwarding safeguards and client betas as sent. Adds the wire-level audit tests. * fix(rust): add the safeguards beta on Azure AI and Claude Platform hosts Azure AI and Claude Platform Messages transforms now add the dangerous-tool-use beta for a non-null safeguards field, through the Azure AI and Anthropic beta-map columns. The safeguards wire and chaos tests get unique file names so they do not collide with the Python fix's tests, the Python-route test asserts the fixed behavior behind a BUG skip, and the DeepSeek test checks that no beta is added * fix(rust): keep Claude Platform out of the safeguards beta until a route can prove it Claude Platform has no beta-map column and no QA account, so it goes back to sending safeguards with no derived beta, like DeepSeek. Also release held chaos-test requests before the thread pool waits on them, and mark the no-column test as rstest. * fix(rust): derive the safeguards beta for Claude Platform and keep the null-cell test off Databricks Claude Platform on AWS answers 400 to safeguards without dangerous-tool-use-2026-09-03 and 200 with it, so its Messages transform derives the beta through the Anthropic column again. The null-cell case now uses only Bedrock Converse, since Databricks support for the beta is changing in a sibling PR. * fix(rust): drop a null safeguards key instead of sending it without its beta Anthropic rejects any safeguards key, null included, when the dangerous-tool-use beta is missing, so an explicit "safeguards": null on the Rust gateway still answered 400. The shared Messages transform now drops a null safeguards key on every host, as the proxy already drops None params before calling Rust, so it answers 200 with no beta. The scripted peer in the integration wire tests now rejects the key on presence, matching the real API. * test: cover Bedrock Mantle in the null safeguards case and match the vendor's non-list wording The Rust null-safeguards regression test now also runs for bedrock_mantle, whose beta is keyed on the field being present. The scripted Anthropic peer answers a non-list safeguards value with the real API's "valid array" wording. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 16:48:05 +00:00
devin-ai-integration[bot]	eb5295b20c	fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards (#43955) ... * fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards * test(azure_ai): pin the outgoing Foundry request keeps the dangerous-tool-use beta next to safeguards * test(azure_ai): inject the Foundry client and scope the beta cache reset to the test * test(azure_ai): own and close the mock transport client, drop the docstring entry * test(integration): audit the safeguards beta derivation across anthropic-compatible providers * fix(github_copilot): keep safeguards from deriving the dangerous-tool-use beta * test(integration): kill the chaos victim worker through the harness helper * fix(anthropic): derive the dangerous-tool-use beta only where the beta map supports it and honor additional_drop_params The safeguards beta is now added only for providers whose beta map column accepts it, so github_copilot, Eden AI and the OpenAI-compatible JSON providers keep their base wire. additional_drop_params now runs before the provider headers are built, so a dropped param no longer drives its derived beta. * fix(anthropic): load the beta map off the event loop and pin a dropped context_management The first /v1/messages request that needs the remote beta map now loads it in a worker thread, so a slow fetch no longer stalls every other request on the worker. An integration cell pins that a deployment dropping context_management sends neither the field nor its derived beta. * fix(anthropic): load the beta map only for safeguards requests, once per worker * fix(anthropic): gate safeguards on the bundled beta map and load the remote map on its own thread --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 16:46:25 +00:00
devin-ai-integration[bot]	5dcf2c2062	test(translation): add sonnet 5.5 and gemini vertex rows to cross-model mid tier (#46896) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 09:36:40 -07:00
devin-ai-integration[bot]	0c3087eb20	test(translation): recapture azure ai claude bases after usage speed field (#47117) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 09:30:34 -07:00
devin-ai-integration[bot]	b578dcf278	test(translation): recapture anthropic claude bases after usage speed field (#47116) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 09:29:56 -07:00
Mateo Wang	c143954493	fix(streaming): report the deployment's api_base when a stream fails mid-way (#47045) ... * fix(streaming): report the deployment's api_base when a stream fails mid-way * fix(responses): report the deployment's api_base when a native responses stream fails mid-way * fix(streaming): name the deployment's configured api_base in mid-stream errors and keep prompt text out Mid-stream errors on chat and native /v1/responses streams now prefer the router's configured api_base over the logged request URL, websocket turns keep the connection's api_base, and the mid-stream mapping no longer adds the prompt to the debug info. * fix(streaming): keep api_base credentials out of error debug info and leave websocket turn logging unchanged * fix(streaming): name the websocket connection's api_base on turn failures and strip scheme-less userinfo * fix(streaming): strip api_base userinfo before cutting the query so a raw ? or # in a password stays out Adds the audit's integration tests for the mid-stream alert API Base line and the SDK debug info. * fix(streaming): leave the API Base line out when an @ follows the host A raw / ? or # in a password and an @ in a query or path read the same in a malformed api_base, so either guess could show a secret or a wrong host. * fix(streaming): keep the API Base line when an @ sits in a provider path Vertex partner model ids like claude-opus-5-5@default put an @ in the URL path, so the fail-closed rule left out the line main printed. The line is now left out only when the text before the first / ? or # is not a host, and a :// counts as the scheme only after a valid scheme name. * test(streaming): cover a numeric password with a slash in the API Base line The audit found that a password like 1234/function-secret, which looks like a port, had no case. The line is left out on this branch and printed the raw secret on main	2026-10-17 15:41:24 +00:00
Mateo Wang	82ca10cc2d	feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes (#47040) ... * feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes * fix(bedrock): move Invoke body requestMetadata into the signed header and match Bedrock's ASCII whitespace rule * fix(bedrock): keep Invoke body requestMetadata in the body and answer bad metadata as AWS does Invoke requests whose body already carries requestMetadata keep the resolved pairs in the body instead of the 8,500-character signed header. Malformed caller metadata now answers 400 with the ValidationException error type the AWS SDK expects, and drop_params drops bad pairs on both the pass-through routes and /v1/chat/completions. The caller-pair dedupe is linear, and the chat validator treats \s as ASCII whitespace, as Bedrock does * fix(bedrock): match Bedrock's 400s and header limit on pass-through request metadata Invoke metadata too long for the 8,500-character header now goes in the body, Invoke requestMetadata null answers 400 like Bedrock, Converse type errors carry SerializationException, violation messages name the failing key, and the reserved user_api_key_ prefix is matched case-insensitively after trimming whitespace * fix(bedrock): trim the Invoke metadata header to Bedrock's limit and drop bad requestMetadata under drop_params without the setting * fix(bedrock): leave /bedrock pass-through bodies as sent when request metadata forwarding is off Global drop_params no longer rewrites requestMetadata on the pass-through routes unless bedrock_request_metadata_fields is set. Chat keeps dropping bad pairs under drop_params. Adds a tests/integration wire test for the four pass-through actions and removes an unused parameter from the Invoke header helper. * test(bedrock): drive the pass-through metadata wire test with boto3's bearer key env var * fix(bedrock): trim the pass-through invoke metadata header and stop Nova invoke from checking metadata it drops * test(bedrock): cover pass-through metadata header trimming, malformed caller metadata, and chat drop_params on the wire * fix(bedrock): match requestMetadata keys and values against the whole string so a trailing newline cannot pass the 256-character limit	2026-10-17 14:01:50 +00:00
jesus-berri	caad4d6204	feat(azure-ai): support Entra ID auth for Microsoft-Decision-1 on /v1/decisions and /v1/systemone (#46539) ... * fix(azure-ai): support Entra auth for Decisions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(azure-ai): select Decisions provider by name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(azure-ai): prioritize deployment credentials over env key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(azure-ai): restore api key precedence for Decisions auth Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 09:18:34 +00:00
yucheng-berri	9f115be88d	fix(bedrock)!: scan Claude invoke-with-response-stream passthrough output with post_call guardrails (#46524) ... * fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618) Invoke streams of Claude models now go through the same frame-by-frame relay as converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages stream translation, a block ends the stream with an AWS exception frame, and a held stream gets its masks written back into re-encoded chunk frames. Claude is told by its model id or, behind an application inference profile, by the anthropic_version its request body carries. Streams of other model families stay as they were * test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618) * fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618) A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered path, with the rewrites written back into the chunk frames that carried each text. Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does, so a vendor can tie the scans of one streamed reply together. * refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618) * fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618) An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned. The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream. * test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618) * test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618) * test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests * test(bedrock): type the remaining parameters of the Claude invoke stream tests * refactor(bedrock): bind the buffered invoke rewrite in its condition --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 01:17:19 -07:00
yucheng-berri	ef1cb19351	fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307) ... * test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 00:53:53 -07:00
Mateo Wang	4a58784fa2	fix(cost): price exact cost-map names before price-free capability rules, and fix SCIM and MCP integration tests on CircleCI (#47037) ... * fix(test-harness): preserve sibling auth modules in isolated worker configs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect servers in MCP connect-flow descriptions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect ptu_hours in daily activity metrics after #43043 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): kill every serving worker once the SCIM push blocks on the row lock * fix(cost): price exact cost-map names before price-free capability rule matches * test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): price an explicit base_model under its own provider after candidate reordering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): retain priced generalized candidates and skip explicit-pricing lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): assert custom pricing makes no remote model-info requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): keep the per-candidate pricing provider final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments * fix(cost): sort cost candidates from the static cost map and rules, without provider lookups * test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-17 06:46:52 +00:00
yuneng-jiang	c32eea634a	test(integration): send post-kill requests on a fresh connection in two worker-kill tests (#47080) ... kill() returns once the killed worker's main thread is a zombie, but its other threads can still be exiting and holding its sockets open. A keep-alive connection pooled to that worker still looks healthy, so the next request goes out on it and the closing socket answers with a reset. The old reap wait used to hide this. A new connection can only be accepted by a live worker, so the surviving-worker assertions now run on a fresh client	2026-10-16 23:27:51 -07:00
yuneng-jiang	e754857de9	test(integration): write worker configs beside test-supplied configs (#47066) ... Under worker isolation the harness rewrote every proxy config into the owned proxy's directory, so modules a test placed next to its own config (custom_auth) no longer resolved and the proxy died at boot. The copy now lands next to a test-supplied config, while configs inside the checkout still go to the worker directory so nothing is written into the repo tree	2026-10-16 23:21:02 -07:00
Mateo Wang	e960d96ae9	fix(prometheus): set the remaining rate-limit gauges from decimal headers again (#47039) ... * test(integration): wait for both workers to load each metered model before sending * fix(prometheus): read remaining rate-limit headers as numbers and let the router fill unreadable ones * fix(prometheus): keep deployment metrics on non-string header keys and cover decimal and router-filled gauges Accept any mapping key when reading additional headers, so a non-string key no longer drops every success metric. The router fallback test now uses a real Router and covers inf, nan and unreadable requests values, the decimal test uses a fractional requests value, and the integration cell sends 12.5 and an unreadable tokens value to deployments with tpm set * fix(prometheus): leave the router fallback unchanged and keep only the decimal header fix * fix(prometheus): read remaining rate-limit headers with float() like every release did * test(integration): check the remaining rate-limit gauges on /v1/responses and /v1/messages too	2026-10-17 05:20:00 +00:00