57 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
Mateo Wang	16645cc963	fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) ... * fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts	2026-10-17 22:08:02 +00:00
yuneng-jiang	4846cfabc7	fix(vector-stores): keep proxy context out of provider vector store metadata (#47021) ... * fix(vector-stores): keep proxy context out of provider vector store metadata The proxy merged its request context (user_api_key_* fields, endpoint, user_agent, requester_ip_address, litellm_api_version) and the router's deployment info into the vector store request's metadata. For vector stores that field is also OpenAI's own metadata, which is capped at 16 keys, so a store created with a team key was full on creation and every later metadata update returned 400. Vector store routes now keep that context in litellm_metadata, like batches, files and responses, and the router does the same for avector_store_create. Provider-side metadata holds only what the caller sent. Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in script that removes the proxy-written keys from stores created before this change. It is a dry run unless --apply is passed. * fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values. * fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata Vector store routes now keep proxy context in litellm_metadata, but the v3 limiter only read agent_id from metadata, and logging let provider metadata tags or spend_logs_metadata replace the proxy's values. A string tags value also crashed get_request_tags. The limiter now prefers the proxy-resolved agent_id in litellm_metadata. Logging combines list tags from both buckets (metadata first, deduplicated) and merges dict spend_logs_metadata with litellm_metadata winning, ignoring values of the wrong type. * fix(vector-stores): narrow the logging fallback to mistyped caller values The tag union and spend_logs_metadata merge changed logged tags and spend_logs_metadata on every route that has both metadata buckets, deduped tags quadratically, and stopped background interaction polling from keeping string tags as sent. Caller values now win exactly as before, and the proxy's litellm_metadata tags or spend_logs_metadata only replace a caller value of the wrong type. The limiter reads agent ids through a pydantic model instead of a cast and ignores non-string ids. * fix(vector-stores): log the proxy's full tag list on litellm_metadata routes On routes that keep proxy context in litellm_metadata, the proxy builds litellm_metadata.tags from the caller's tags plus key, team and header tags, but get_request_tags preferred the caller's metadata.tags. A caller sending metadata.tags on a vector store call dropped the key's tags from request_tags, which tag spend tracking reads. get_request_tags now prefers a non-empty litellm_metadata tag list, and falls back to metadata.tags as before. * fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets Moving vector store proxy context into litellm_metadata broke shared readers (agent limits, request tags, merged logging metadata), and fixing them widened the change to every litellm_metadata route. This reverts the bucket move, the router change and the reader fixes, so proxy context stays in metadata exactly as on main. Vector store create and update now drop the keys LiteLLM itself writes (user_api_key*, requester and deployment keys, agent ids) from the metadata sent to the provider, in the shared optional-param helper both paths use. The cleanup script reuses the same key list. * fix(vector-stores): pass non-string metadata keys through the provider metadata filter * fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls * fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy * fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies * fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed	2026-10-17 12:51:47 -07:00
mrinal-berri	a79dae764f	fix(proxy): route team key vector store calls to a team model of the store's provider (#46894) ... * test(vector-stores): reproduce team key search routed to a team model of the wrong provider A team key's model-less vector store search picks the team's first deployment, so a langgraph team model turns a Bedrock store search into "Vector store search is not supported for langgraph". Cover map_team_model picking by provider and a registered Bedrock store search reaching Bedrock with the registry's credentials. * fix(router): match model-less team deployment to the requested provider map_team_model now takes an optional custom_llm_provider. With no model given, it returns the first team deployment serving that provider, or None when none does, instead of the team's first deployment. Without a provider it behaves as before, so BYOK teams calling raw vector store IDs keep their team key. * fix(proxy): pass the vector store provider when mapping team models Vector store search, retrieve, update and delete pass the store's custom_llm_provider to map_team_model, so a team key's model-less call lands on a team deployment of that provider, or on the direct path with the store's own credentials when the team has none. Other model-less routes keep today's behavior. * fix(proxy): type the vector store provider passed to map_team_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vector-stores): cover team key vector store search provider routing end to end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): ruff format route_llm_request.py Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type vector store routing tests and mock Bedrock transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin model-less vector store calls when a public name mixes providers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): keep team provider routing in map_team_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 14:32:03 -05:00
Mateo Wang	d76ec3a47e	fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583) ... * fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing A non-list tools value crashed the vector store lookup during auth and the built-in tool cost lookup in the failure logging path, so the proxy answered an unhandled 500 and closed the keep-alive connection. count_tokens now answers a 400 invalid_request_error for a malformed body, drops malformed tools when drop_params is on, and both lookups skip values of the wrong shape. * fix(proxy): skip non-list tools in the tool allowlist check Also validate the count_tokens body without recursion, which the recursion detector rejects. * test(integration): restore the int tools case on count_tokens The proxy now answers a non-list tools value with a 400, so the case is deterministic again. * fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids * fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids Check tools after request setup so the Claude Code default and the logged call type apply, read the request, global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off * fix(proxy): keep the request's call type when logging a gate rejection after request setup The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses rejection as aembedding, so it now only applies when the logging object carries no known call type * fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry * fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket * refactor(proxy): leave the responses websocket out of the request-level tool check * fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections * fix(proxy): read router default drop lists and deployment ids in the malformed tools check * fix(proxy): leave malformed tools to the router when it can pick another deployment Fallbacks, auto-routers, model strings, comma batches and team model names let the router pick a deployment after the proxy checks the request, so the proxy now forwards a malformed tools field there as main does, and only rejects it when the deployment it checks is the one that serves. Token counting never reroutes, so count_tokens and input_tokens keep reading the group they count with. additional_drop_params names are read the way core reads them, so a list with a non-string entry still names its strings. * test(integration): cover malformed tools on reroutable models, router defaults and deployment ids * fix(proxy): resolve malformed tools at the deployment the router picks * fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells. * test(integration): expect a 400 for an empty-string tools field on /v1/messages Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test * fix(router): keep web search routing for tools passed as a tuple * fix(router): keep a deployment's own tools when drop_params drops a malformed request tools * test(router): list _merged_tools as covered through the deployment tools merge tests * fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them * fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError * fix(prometheus): keep deployment health metrics clean on a malformed tools rejection * test: type the new malformed tools tests * test: mark new test locals Final and type kwargs dicts	2026-10-16 14:57:53 -07:00
devin-ai-integration[bot]	8bb8ac7f18	fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure (#45136) ... * fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * chore: drop duplicate litellm import and mark the stream_ended rebind * fix(proxy): log only router-superseded attempts as error-frame failures A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log * test: type the _settle helper's recorded events --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 21:19:58 +00:00
Mateo Wang	f8217d2a03	fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count (#46843) ... * fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count /v1/messages runs clear_tool_uses context edits before it sends an Anthropic body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted the body unedited, so a request with older tool results cleared counted about 2.8x what inference bills. The count now runs the same history and context-edit step inference runs, honors additional_drop_params on the deployment, and skips compact edits so a count never makes a paid summary call. * fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs Run compact edits in the count without the summary call so trigger checks and history slicing match /v1/messages, honor a request-level additional_drop_params, keep the full body when any deployment in the model group drops context_management, and answer 400 naming a block Gemini cannot count when an edit trigger fails on it * fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check * fix(count_tokens): log a failed edit in the count at debug, since the send already logs it * fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does * fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits. * fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name * fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to * fix(messages): drop an invalid context_management spec under drop_params so the send matches the count * fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes. * test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code	2026-10-16 17:53:06 +00:00
Ayush Shekhar	7930c0e434	feat(router): mid-stream fallback continuation for chat completions (#41127) ... * feat(router): continue chat streams on fallback after a mid-stream break A chat-completions stream that breaks after content has been delivered was re-raised and the fallback deployment never ran (#40404), while the Responses-API path already continues via a prefilled assistant turn. Add the same for chat completions behind `enable_mid_stream_fallback_continuation` (opt-in, async-only), so a post-content break re-enters the fallback chain with the partial text as an assistant prefill instead of surfacing the error. Continuation is only attempted when it is safe: the stream emitted plain assistant text (no tool/function calls, thinking blocks, reasoning items, audio or images), the request is not constrained output (response_format or a forced tool_choice) and not merge-reasoning mode, and the fallback target's model supports assistant prefill. The target check runs as a deployment pre-call filter, so a chain with no prefill-capable deployment empties and the original error is surfaced rather than a duplicated or rejected request being sent. Plain reasoning_content is treated as out-of-band and does not block a continuation, matching the Responses-API path; Anthropic thinking is excluded because its signed thinking blocks cannot ride a text-only prefill. CustomStreamWrapper now latches whether a disqualifying delta was streamed and carries it on MidStreamFallbackError so the router can decide without re-scanning chunks. Default behavior is unchanged. * fix(router): harden mid-stream continuation eligibility and prefill folding Address review of the mid-stream continuation: - keep response_format={"type": "text"} eligible; only json_object / json_schema and other structured formats decline, since "text" is the unconstrained default - fold a new partial into an existing trailing assistant prefill instead of appending a second one, so a nested mid-stream break stays a single prefill turn even on providers that do not merge consecutive assistant messages - split the deployment prefill-capability lookup into two readable steps - cover the merge_reasoning decline branch, the text response_format pass, and the prefill-folding path with tests * fix(router): address review of mid-stream continuation - register the continuation deployment filter as a process-global singleton instead of tracking it per router, so discarding one router no longer removes the type-deduplicated filter that other live routers still depend on - disqualify reasoning_content: it reaches the caller as visible reasoning that a text-only prefill cannot carry, so a reasoning-capable fallback would re-derive it and produce an incoherent response - move MID_STREAM_CONTINUATION_KWARG to constants and lazy-import the filter class in the router, breaking the module-level import cycle CodeQL flagged - trim the added docstrings to the repository comment policy * fix(router): keep mid-stream continuation within the type-discipline budget The LIT* type-discipline gate counts mutable-collection use the ruff pass does not see. Annotate the reads with read-only Sequence/Mapping views, compare response_format against a frozen MappingProxyType sentinel, and mark the genuinely mutable litellm message and deployment-list boundaries with mutable-ok reasons, so LIT001 and LIT002 stay under their limits. * fix(router): clear basedpyright operator error and the import-cycle alert - narrow the accumulated stream delta content to str before appending, so the shared helper no longer trips reportOperatorIssue (str += object) - keep the continuation marker in the pre-call-check module and lazy-import it at the two router call sites instead of routing it through constants, so the router has no module-level dependency that closes an import cycle * fix(router): make the mid-stream continuation marker unforgeable The deployment filter is process-global and read the continuation marker as a truthy value from request_kwargs. Since the proxy can forward arbitrary request-body fields into the router, a client could send `"_mid_stream_continuation": true` on a normal request to filter a mixed model group down to prefill-capable deployments and steer their prompt onto that provider. Set a private sentinel object internally and check it by type instead of truthiness: a JSON request body cannot construct one, so a forged flag is ignored. Adds a regression test that forged client values (true, "true", 1, a dict) leave the deployment list untouched. * feat(router): honor per-deployment model_info for continuation prefill capability The deployment filter only read supports_assistant_prefill from the cost map, so a model configured with `model_info: {supports_assistant_prefill: true}` on the deployment (the usual way to declare a capability for a model that is not in the cost map, or to override it) was wrongly skipped. Check the deployment's model_info first (True or False both authoritative), then fall back to the cost map. Adds tests for the override in both directions. * fix(router): bound mid-stream continuation to the caller's output allowance A continuation handed the fallback the full max_tokens / max_completion_tokens even though part of the answer had already streamed, so the total could exceed the caller's limit and repeated interruptions could re-grant the allowance on every hop, amplifying spend. Reduce each configured ceiling by the tokens already emitted before re-entry, and decline the continuation (surface the original error) when the allowance is exhausted. The reductions compound across hops because the trimmed ceiling is what the next hop sees. Also add the missing explanatory comment on the model_info fall-through except. * refactor(router): extract mid-stream decline to stay within the complexity budget The output-ceiling branch pushed stream_with_fallbacks past the C901 limit. Both decline paths (ineligible after content, and exhausted allowance) re-raised the original provider error the same way, so pull that into _raise_original_mid_stream_error. It removes the duplicated branching, drops the function back under the complexity ceiling, and reads clearer. * fix(cost-map): mark Claude Sonnet 4.6 as not supporting assistant prefill The Anthropic API and Bedrock both reject an assistant-last message on claude-sonnet-4-6 with "This model does not support assistant message prefill", so the continuation filter must not route a mid-stream continuation to it. Flip the flag on every claude-sonnet-4-6 entry and on databricks' opus-4-6 entry, whose anthropic sibling already says false. The prefill check tests now register synthetic cost-map entries instead of pinning real models, which broke when claude-3-opus left the map. * refactor(router): type the streaming fallback kwargs and drop narration comments Removes an unused mutable-ok marker (LIT013) and the explanatory comments Greptile flagged, and types initial_kwargs on both streaming fallback iterators as dict[str, Any] like their Responses siblings so the new continuation helpers take a typed mapping and the basedpyright budget stays at or below main. * fix(router): run post_call_rules over the joined text of a continued stream Each provider stream wrapper validates only the fragment it produced, so a rule that trips on the primary's prefix plus the continuation's text never fired. The router now feeds litellm.post_call_rules the emitted text plus every continuation chunk before yielding it. Also drops a narrating test-helper docstring. * refactor(router): read a deployment's declared prefill support without an empty except * test(router): cover _raise_original_mid_stream_error directly for the router coverage gate * fix(router): fold the continuation prefill into a trailing assistant turn A request that already ends in an assistant message took the partial output as a second assistant turn, which prefill providers reject. The partial now extends that turn in place, with prefix set, so the continuation request never ends in two assistant messages. * fix(router): keep content blocks on a trailing assistant turn when extending it with the partial output A trailing assistant message whose content is a list of blocks was stringified before the partial output was appended, so the fallback received the Python repr of the blocks as its prefill. The partial now lands as one more text block, and a content shape that cannot be extended surfaces the original stream error instead of a garbled continuation. * fix(router): continue the trailing text block in place when extending a prefill with content blocks Anthropic strips trailing whitespace per content block, so appending the partial output as a new text block after an assistant turn that ends in "Sure, " would send the prefill as "Sure,here". The continuation now extends the last text block itself and only opens a new text block when the trailing block is not text (an image, say), so the prefill reads exactly as the text the caller already received * test(router): type the prefill check's deployment helper * fix(router): keep a continued stream on the primary's identity and widen what can continue Continuation chunks now keep the primary stream's id and created time and drop the fallback's second role delta. Requests with n > 1 no longer continue. Prefill support reads model_info.base_model and litellm_params.base_model before the deployment model. enable_mid_stream_fallback_continuation is runtime-updatable through update_settings and registers the prefill filter. The bridge rebuild of MidStreamFallbackError keeps the inner stream's disqualifier. A fallback that repeats the prefill is not doubled in the continuation input. * fix(router): decline mid-stream continuation after a content filter, merged reasoning, or a plain trailing assistant turn Also count the partial stream's completion tokens in the continued stream's usage, accept string values for enable_mid_stream_fallback_continuation, and list the new router helpers in the router coverage check * fix(router): drop re-emitted whitespace at the continuation join and sum both legs' usage consistently * fix(router): keep the full prefill so a nested break keeps the separator * fix(router): bill both calls of a continued stream and fix Bedrock Converse, Databricks, and Vertex DeepSeek prefill routes * test(router): integration cells for mid-stream fallback continuation Covers the continued stream the caller gets, the prefill turn and reduced max_tokens the fallback receives, the prefill-capability filter, the runtime toggle, repeated requests through the response cache, and the spend row of a continued stream * fix(router): log a continued stream under the caller's id, price each cut on its own prompt, and keep prefill flags per deployment * fix(router): price and log each cut leg before its continuation reuses the logging object Price the cut stream's partial and let its failure log finish (bounded) before the fallback overwrites a shared proxy logging object, and skip continuation when an output guardrail is registered or a request fallback carries its own messages. * test(router): integration cells for the output-guardrail and request-fallback declines and the cut call's failure log Cover a registered post_call guardrail and a request fallback carrying its own messages keeping the original error, an input-only guardrail leaving continuation on, and the cut call's failure event naming its own deployment and price. * fix(router): skip mid-stream continuation when a request fallback overrides request params A request-body fallback entry that sets anything besides model (stream, messages, max_tokens) changes the request the prefill continuation relies on, so a stream: false entry ended the answer with no error. * fix(router): skip mid-stream continuation for a fallback group target that overrides request params The override check read only top-level fallback entries, so a target inside a group's list, such as {"primary": [{"model": "backup", "stream": false}]} in the request, the config, or a key's router settings, still ended the answer with no error. * fix(router): check only the request's own fallback chain for request param overrides Scanning every group's chain turned continuation off for all requests when one unrelated group's target overrode a param. Resolve the chain the router will use for this request, the same way the fallback path does. * fix(router): keep the original error when request fallbacks are malformed The request-param override check resolved a standard-shape chain outside the router's own error handling, so an empty entry or a non-list chain raised StopIteration or TypeError and replaced the original mid-stream error. Skip empty entries and treat a non-list chain as no chain. * fix(router): read deployment-configured constraints, decline deployment-id fallbacks without prefill, strip only the last Bedrock text block * fix(router): read a deployment's tool_choice ahead of router defaults in the continuation check, and restore the renamed prefill test names * fix(router): filter continuation deployments on every routing strategy, read extra_body and fallback deployments' own params The deployment filter now also runs on the sync selection path that usage-based-routing v1 and custom strategies use, one predicate reads n, response_format, tool_choice and merge_reasoning_content_in_choices with extra_body laid over them, fallback deployments whose own config constrains the output are skipped, and continuation declines when the cut deployment's id is not in the router (per-request credential deployments) * fix(router): decline a cut mid-echo, reduce extra_body output limits, and check post_call_rules once on nested continuations --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 13:51:32 +00:00
武本龍門	43be1c2ae7	fix(router)!: keep RPM, region, and param checks when token counting fails (#44544) ... * fix(router): stop skipping rate-limit checks when token counting fails Router._pre_call_checks() caught any token_counter() exception and returned the full, unfiltered deployment list, bypassing the RPM, region, and invalid-params checks for every deployment - not just the context-window check that actually depends on the count. A content block token_counter() cannot parse (e.g. an invalid image_url detail value) silently disabled rate-limit enforcement for that request, with only a log line and no error surfaced to the caller. Now a counting failure only disables the context-window check; the other checks still run for every deployment as before. Input-token resolution is pulled into its own method and computed once before the loop, so nothing needs to track "did counting fail" with a mutable flag across iterations. * test(router): cover the async region check when token counting fails Replace the unit test of the private input-token resolver with a regression test through async_get_healthy_deployments, the path the proxy uses: an EU-pinned request whose content cannot be token-counted must not get the US deployment * test(router): cover the param filter when token counting fails and satisfy router coverage * fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params, drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away instead of a retried 429. Deployment context windows are resolved once per call. * fix(router): keep the retried 429 when a cooling-down deployment takes the param * fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks * fix(router): steer by deployment capability, apply drop settings after region and RPM checks A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing * test(router): make the github_copilot pre-call filter test hermetic in CI * test(router): cover pre-call region and response_format filters through the proxy * fix(router): judge the 400 against the request's own deployments, not a name lookup * fix(router): judge the 400 against the passed deployments when the group name lists none * fix(router): let a request's drop and allow settings replace the deployment's in the param check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 09:20:34 +00:00
devin-ai-integration[bot]	e344dbd8db	feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) ... * feat(proxy): split Azure PTU deployments across teams by share A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on, routing hides the deployment from teams holding no share, the v3 limiter enforces each share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team Usage page report PTU-hours next to tokens. * fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name The PTU flag is read from the process environment on every check instead of through the secret-manager path, so a proxy with a hosted secret manager does not pay a round trip per request. A team's ceiling now carries the model's cached-input ratio and settlement charges cache reads at that ratio, matching how Azure sizes a PTU. A deployment registered for one team is found by its team_public_model_name for PTU-hours and the ceiling, since its model_name is rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and parsed_ptu_shares refuses booleans, strings, and non-string team ids, a fractional ptu_count reserves nothing, and models_processed counts distinct deployments rather than holdings. * fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations * fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id * fix(ptu): refuse ptu_shares naming a team that does not exist * refactor(ptu): keep Azure detection under llms/azure and check share teams with one query * fix(proxy): drop the reverted TeamIdSearchFilter import after merging main Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally * fix(proxy): chunk the ptu_shares team lookup under the IN-list bound * fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling * fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling * fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour * fix(ui): seed total_ptu_hours in the empty daily activity metadata * fix(ptu): size usage rows by the name the request used * fix(ptu): size a team's usage rows by the deployment it is served from * fix(ptu): never size a team's usage row by another team's reservation * fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length * fix(ptu): read a share map only through the terms registration requires A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and ptu_effective_from, or one that does not add up to ptu_count, is refused at registration but was enforced once loaded: every other team got the 400 and the share holder a ceiling while no flat cost accrued. _deployment_shares now reads the shares through ptu_terms, so such a row reserves nothing, the same as before the map was read. * fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off 6fb24d1d read ptu_shares through the pricing terms so a termless map reserved nothing, but with the flag off the router skips ptu_config_error for config.yaml entries, so such an entry loaded and served every team, against the rule that a declared split restricts access whether or not cost attribution is on. _deployment_shares reads the declared map again, and the router now raises ptu_config_error for a config-sourced entry declaring ptu_shares regardless of the flag, so a malformed split is refused at boot instead of being honoured or ignored silently. * fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling A name served by a wildcard route now resolves to the shared deployment behind it, so the limiter's ceiling and the team daily activity sizing both find it. Pass-through responses settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a reservation taken at admission is settled even when the success metadata lost the team. * fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids. * test(router): pin the wildcard route precedence that PTU sizing and the router share * fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was refused with a 422 and dropped at load. Fractional, infinite, boolean and string shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix, zero-token Responses usage, fractional-count and sizing-warning paths. * fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move * fix(proxy): write each PTU deployment day's team charges in one transaction The backfill counts a deployment day as priced once any team's row exists for it, so a rollup that wrote team A's share and lost team C's left team C unbilled for good. Each deployment day's team rows now go through one Prisma batch with the retry around the whole batch, so a day is fully written or not written at all and the next backfill prices it for every team. * fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes * test(router): list the cached PTU ceiling helpers in the router coverage check * fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force * fix(proxy): bill a rerun PTU day to the current share holders alone * fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list * fix(proxy): read the PTU flag from the secret manager first, like get_secret * fix(router): scope retry health lookups to the caller only for PTU share holders * fix(router): move the share-holder retry check into ptu_shares Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them * test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 18:44:30 +00:00
devin-ai-integration[bot]	fbc2e358bc	fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703) ... * fix(responses): bill a background response once, from the poll job Reading a completed background response with GET /v1/responses/{id} wrote a full-cost spend row on every read, on top of the row the background cost poll writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and the poll job is the single biller. The poll job now bills with the creating request's attribution (key hash, team, org, user, aliases, tags) read off the managed object row, through a helper it shares with the batch cost poll. The background create persists the creating key and request tags so that attribution survives until the poll runs. * test(proxy): a background response read reports zero cost * fix(responses): bill a background response once, on the first read or poll that sees it finished A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll * fix(responses): keep the caller's response id out of the billing failure warning * fix(responses): keep the creating client's User-Agent tags on the one background charge * fix(responses): keep managed background reads free and bill each response once The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404. * fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts * fix(responses): give the background billing event its own spend log row and bill responses the create left untracked * fix(responses): narrow the managed files hook with a protocol instead of an enterprise import * fix(responses): leave a read the caller retargets with client-side credentials to the cost poll * fix(responses): bill background responses on the creator's end user and report the charge on the settling read The read that settles the bill returns the charge in x-litellm-response-cost, the billed event carries the creator's end user and the key, team, and user budget fields, and its spend row is named after the id the client got. Override reads, cancels, and deletes settle from the configured deployment, a streamed create marks its row billed, stored rows keep no model output, and model insights counts the billed row's spend without a request. * test(responses): cover background response billing across proxies, reads and chaos * fix(responses): settle background billing only for callers allowed to address the response A delete now settles the bill only after the response id access check passes, a streamed create is marked billed when it sets background inside extra_body or its completed event arrives after the last chunk, the billing event keeps the deployment's litellm_params base_model, a key with no user and no team bills no user instead of its owner scope, and the enterprise cost check import is optional * fix(responses): record the bound on the pending status IN filters * fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was The read that settles a background response now sends the whole cost header family from the billed event, so the breakdown headers add up to the charge. Batch cost events go back to main's attribution (creator id as the user, no budget fields); only background responses get the end user and budgets. Integration tests expect the charge on the settling read (or none when the poll billed first), one charge for a streamed create, and SDK flows that poll with the id each read returns. * fix(responses): bill each background response once across upgrades, streams, and failed settles Rows stored before this billing model are found by the issued id and rekeyed by the provider id, and the cost poll retires one a read already billed. A streamed background create is free on its own and bills the finished response through the same managed row before the client sees response.completed, so a racing read cannot bill it again. A settle that fails retries in the background, a retargeted read with no row tracks and bills from the configured deployment, and a delete keeps its billed cost header. Reverts the unrelated json.dumps narrowing in the completion transformation. * fix(responses): bill cache-polled background streams and match poll rows by the issued id * test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing Adds integration cells for the background response billing paths the audit inventory lists: a DELETE that settles the bill and refuses a stranger, a settle the database refuses that retries once it recovers, the Prometheus remaining budget, base_model pricing from model_info and litellm_params, the daily model usage rollup, the end user, the settling read's cost split, stored rows without model output, a forged managed marker on the create, streamed creates with background in extra_body, and the cache-polled create --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 11:00:14 -07:00
devin-ai-integration[bot]	c00706218b	fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097) ... * fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 03:21:48 -07:00
devin-ai-integration[bot]	bf56719f09	fix: cap cold MCP schema workers at usable CPUs and deflake wall-clock unit tests (#45791) ... * test: inject clocks and wait on conditions in three wall-clock-flaky unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: keep generic API logger tests on 3.10 and cover automatic sending Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: pin router usage minute in shared rate-limit header test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: stabilize three wall-clock-sensitive unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): cap cold schema validation workers at the CPUs the proxy can use Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: inject the cache clock and wait on lock contention instead of 1s deadlines Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): wait on listing events instead of 1s deadlines in the discovery drain test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai>	2026-10-15 00:47:59 -07:00
devin-ai-integration[bot]	e136e91273	fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) ... * fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise. * fix(router): give prompt cache pins their own tier and honour any declared ttl Pins now live in a DualCache of their own instead of the router's 200-slot in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows. The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not only the literal "1h". * fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache Compute the cache_control ttl scan through offload_token_count beside the prefix hash, clear the dedicated pin cache in Router.flush_cache, and point the order-fallback and priority-group pin tests at the pin cache they read * fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots * fix(router): match the 1h pin gate to what Bedrock forwards Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock, the same check that decides whether the 1h ttl reaches Bedrock, so an application inference profile ARN keeps a 5 minute pin whatever its base_model prices * test(router): give the anthropic_messages pin fixture its provider * fix(router): let each pin write set its own lifetime and require a positive 1h price In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support. * test(router): integration cells for the one-hour prompt cache pin lifetime Cover every surface the pin lifetime touches: chat and /v1/messages through both SDKs, the responses bridge, response-cache hits, each provider family's 1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed and cross-group routing, concurrency, every Redis attach path, the in-memory slot pressure without Redis reads, a Redis outage and a killed worker * fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes * test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap * fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry * fix(caching): keep the in-memory cache bounded when entries have no usable expiry * test(caching): cover the response cache bound when cached entries have no usable ttl --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 21:10:09 -07:00
Kerry Lu	63c3b7a34e	fix(router): read the first stream chunk through __aiter__ so MCP auto-run streaming works again (#46578) ... * fix(router): read the first stream chunk through __aiter__ so wrappers that hand iteration off (MCPStreamWrapper) stream again * fix(router): mark the _RestT TypeVar Final for the type discipline gate --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 01:51:07 +00:00
Yassin Kortam	6f5f9ce665	fix(router): report fallbacks skipped for access or budget instead of calling them failed (#46291) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 15:08:47 -07:00
Kerry Lu	83980c9d63	fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493) ... * fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): keep only the raising call inside pytest.raises for the pre-content stream tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): yield the primed first chunk from a helper generator so the stream iterators keep their complexity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): expect the empty invoke stream to be reported from router.completion now that the first chunk is read there Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: give router stream mocks a first chunk or an ending so the eager first read neither hangs nor retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): name the first-chunk helpers after what they do (_read_first_chunk, _FirstChunk, _chunks_with_first) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): drop the string annotation on _FirstChunk.chunk, ModelResponseStream is imported at runtime --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 14:19:25 -07:00
Kerry Lu	4f34856a65	refactor(router): name the pre-content stream retry helpers for both stream paths and let the retry loop take its attempt function (#46474) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 18:25:15 +00:00
Rick	4f3cb48dc1	feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706) ... * feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model Studio's async task API: create hits /api/v1/services/aigc/video-generation/video-synthesis and status and content poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the qwencloud and qwen_ai_platform brand aliases, following the same get_dashscope_family_*_config pattern image generation uses. Unlike the Sora-shaped providers the request body is nested, so prompt and media go under input and everything else goes under parameters. X-DashScope-Async is always sent, because DashScope rejects the call outright without it OpenAI params translate to DashScope's contract. seconds becomes duration, and size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus 720P), since sending only the ratio leaves the request on the default 1080P and silently bills at that rate. input_reference becomes a single media entry, base64 encoded into a data URI for file inputs. Which media type a model takes (first_frame or, on the reference-to-video models, reference_image) is read from provider_specific_entry.dashscope_video_reference_input via get_model_info rather than hardcoded; models without it predate the media array and get the flat img_url field. A caller-supplied media array always wins, as that is the only way to express multi-role combinations The create call is the only billed call, so it bills what DashScope will deliver: the requested duration and tier, or the documented 5 second and 1080P defaults when they are omitted. Smart duration (duration -1) lets the service pick the length after that call, so it is rejected rather than recorded at zero, with a 400. Task creation and task lookup both report failures as a 200 carrying top-level code and message with no output, so that shape is raised instead of surfacing as a queued video with an empty id. Status responses re-encode the deployment model from the polled id, so follow-up polls and downloads route to the same deployment. Timestamps parse as UTC+8. DashScope publishes no list, delete or remix endpoint, so those raise rather than guess at a URL The end-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the JSON body httpx actually encoded. That is what catches a mapper that drops input_reference before the request is built, or a body fragment json cannot serialize, neither of which a test calling the transform directly would see Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v, wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models, for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output second by resolution tier from Model Studio's USD rate cards. dashscope and qwen_ai_platform default to the Beijing host and use its rate card; qwencloud defaults to the international host and uses Singapore's. The three *_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920 Co-authored-by: AaronHowell <237895480@qq.com> * fix(dashscope): fail loudly on video responses that cannot be read Addresses the review on the video generation PR. A body that is not the documented task shape, an output block missing task_id or task_status, an undocumented task_status and a missing output all raise DashScopeVideoError now, instead of defaulting into a queued video with an empty id that callers poll forever. A JSONDecodeError from a gateway response no longer escapes as a generic connection error. task_id and task_status are required on _TaskOutput, and a missing output raises even when the body carries no code or message. Download failures move into the 4xx range, since a 5xx has the proxy retry them: 409 while the task is still running, 400 for a task that failed, 404 for an id that expired. The status a provider reported is kept when it reported one, so an in-body failure that arrives on a 200 is billed as the 400 it is rather than as a retryable connection error. characters and image are only dropped when drop_params is set; the params that are rebuilt into the body or consumed by the caller (model, prompt, user, extra_*) stay silently skipped, since the proxy stamps user from the caller's end-user header. Co-authored-by: AaronHowell <237895480@qq.com> * fix(dashscope): clear the lint gates after merging main * fix(dashscope): accept file tuples and paths as video input_reference video_generation() types input_reference as FileTypes, which includes (filename, content[, content_type[, headers]]) tuples and pathlib paths, but the DashScope encoder only read bytes and file objects, so those image-to-video calls raised before reaching DashScope. Bare strings stay URLs and are never opened as paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dashscope): snap size to a supported ratio and read file tuples and paths * test(dashscope): assert whole mapped values and bill against hand-written rates The tier pricing test read its expected rate from the same map the cost calculator uses, so a wrong rate passed. Billing is now checked against fixed rates driven through the status response usage, while the map test only asserts every supported tier has its own rate. Size mapping tests compare the full mapped dict, and the single-test wiring class is a plain function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(videos): download content through the caller's client and stop retrying dead tasks Two Cursor Bugbot findings on the DashScope video PR. The DashScope artifact lives on OSS, so transform_video_content_response makes a second request. It built its own cached client, discarding an injected client and its ssl_verify. Thread the client through the content response the way transform_video_status_retrieve_response already does, and use it in the three providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The remaining providers accept it and read the response body, matching how they already accept an unused client on status retrieve. A FAILED or CANCELED task reported without a code or message fell past the in-flight check into the 502 branch, so the proxy retried a task that can never produce a video. Raise 400 for a terminal status instead. * fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes A download that lands while the task is still PENDING or RUNNING answered 409. That reads as a client error, which was the intent, but litellm's retry predicate counts 409 as retryable, so the proxy polled DashScope twice more before giving up and then counted the caller's own early download against the model's error rate 404 is what the OpenAI videos API documents for content that is not ready, and it is the one status in that family the retry predicate leaves alone. The rest of the mapping is unchanged: 400 for a task the provider reported as failed, 404 for an expired task id, 502 when a succeeded task carries no artifact The download tests now drive avideo_content and assert the caller sees NotFoundError rather than an internal server error * fix(dashscope): route a video poll back to the model group that created the task The create response encoded the upstream model name into the returned id. Both the dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts, so a proxy with a deployment per region had deployments that only the model group told apart. Encoding the bare model name resolved a poll to whichever deployment the router lists first, which sent it to the other region's host with that region's key and a task id that region never issued. That is the reroute the earlier proof records as the dashscope and qwen_ai_platform poll ids coming back tagged qwencloud The id now carries the model group the create was routed to, which is what the proxy resolves a poll back to, and falls back to the model name for a direct SDK call that has no router The regression test drives a real router with one deployment per region, under a different prefix and under the same one, and asserts the poll carries the key of the region that created the task * fix(dashscope): read the routed model group from litellm_metadata The router stamps the model group it picked into litellm_metadata. metadata only carries a copy when the caller sent no metadata of its own, and every proxy request sends some, so a proxied create still encoded the bare model name and the poll resolved to whichever region's deployment the router lists first The regression test now also creates the video with caller metadata, which fails without this change * fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients * fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body * fix(dashscope): check video request values before the billed create and honor per-call drop_params Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task, pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and refuse to map size onto models litellm has no price for. * fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll A key scoped to an alias or to a name a wildcard serves could create a video and then got a 403 on the poll and the download, because the id carried the group that served the call (wan3, qwencloud/*) and the key was never given that name. The id now carries the name the router was asked for on that hop, and the router resolves an alias, routing group or wildcard name in an id back to itself when no deployment name or provider model matches it first * fix(dashscope): keep a routing group's video polls on the member that created the task A routing group spans several model groups, so an id carrying the group's name load-balanced its polls, and some left for another region's host and key with a task id that region never issued. The id now carries the member that served the task when the request named a routing group, and keeps the requested name for a model group, an alias, or a wildcard match. * fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region * fix(proxy): pin a video's served group only after key and team aliases resolve * test(integration): audit DashScope video create, poll, download, routing and chaos paths * fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group * test(routing): expect a forged served group to be dropped from the polled video id * test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 10:10:07 -07:00
devin-ai-integration[bot]	e0398455d0	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) ... * test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 07:58:53 +00:00
devin-ai-integration[bot]	1dbde0845e	test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) ... * test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-11 00:17:21 -07:00
devin-ai-integration[bot]	31e2ce1427	feat: forward-port dd user_email span tag and model hub description from internal staging (#42749) ... * feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit b4f9e14a44) * refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 819faa01c3) * feat(model_hub): surface model_info.description in model group info and Model Hub UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit b224b15b9d) * refactor(router): aggregate model group description without in-place mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 342b93fe0a) * test(dd_span_tagger): assert the user_email tag on the active span instead of on the patched setter The two ported tests only inspected the mock that replaced the tag setter, which the test quality gate counts as two new mock-echo tests over main. They now read the tags that land on the active span * refactor(router): read the model group description through the typed model_info reader The ported lookup read model_info off the untyped deployment dict, which added three basedpyright errors over main. It now goes through the existing validated model_info reader --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-10-10 16:48:49 -07:00
devin-ai-integration[bot]	8a103e26ab	test: move routing, caching and callback tests out of local_testing (#45761) ... * test: move routing, caching and callback tests out of local_testing Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion * test: cover least-busy routing and tighten migrated router, cache and pass-through tests Add router-level least-busy tests for the three legacy rows that had no destination, move completion fallback and model alias tests to their mirror files, move router batch tests next to the router, and strengthen tests whose mutants survived (specific deployment, batch pair grouping, custom cache key, router cache ttl). Add a unit test for configured pass-through Authorization forwarding. * test: tighten migrated callback, proxy config and cache ttl tests Assert the full StandardLoggingPayload key set and the wire user on callback tests, add an async embedding failure callback test, drive the prometheus redis failure through a real RedisCache, use the real salt key in proxy config tests, pin the redis ttl value and rename fallback test helpers * test: fix semantic cache assertions and order-dependent router and callback tests Count semantic cache embeddings on the route, drain async cache writes before the second lookup, echo the requested model from the callback provider mock so response_cost does not depend on model_cost entries other tests register, and move the two-key authorization assertion to the auth failure test it belongs to * test: cover model_list fan-out and the usage-based routing tpm limit lost in the move * test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard * test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file * ci: run the remaining router-keyword legacy tests on one node * test: refuse the redis write in-process instead of resolving an invalid host --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-10 09:33:09 -07:00
yuneng-jiang	3eeca75ca0	test: move offline router, retry and latency tests from local_testing to tests/unit (#45551) ... * test: move offline router, latency-routing, retry and exception tests from local_testing into tests/unit Moves 29 offline nodes into the tests/unit files that mirror the code they exercise. Hosted-provider calls are mocked at the HTTP boundary with respx, and sleeps that only built a start/end gap are replaced by explicit timestamps. Legacy files left empty are deleted. * ci: record the live prompt caching cases left in tests/local_testing as unrun Moving the router prompt caching test to tests/unit leaves only live Anthropic and Vertex cases in tests/local_testing/test_anthropic_prompt_caching.py, which every CircleCI -k already deselects * test: freeze the clock in the moved lowest-latency routing tests The latency logger buckets usage by the current minute, so a run that crosses a minute boundary could grow the cache and fail the memory check. The moved tests now use fixed timestamps and a frozen clock in the logger module * ci: describe the mixed prompt caching file accurately in the coverage allowlist	2026-10-09 09:47:04 -07:00
devin-ai-integration[bot]	8a176c0f0a	fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt (#45393) ... * fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt An order-based or configured fallback hop replayed the failed provider's encrypted reasoning items to the next deployment, which answered 400 (Bedrock Mantle: invalid encrypted reasoning; OpenAI: invalid_encrypted_content), so every multi-turn Responses fallback for Codex-style clients failed. The hop now drops the encrypted reasoning its target cannot decrypt and keeps each item's readable summary. With encrypted_content_affinity on, the pin narrows to the hop's target order instead of emptying it, so the hop reaches the next order instead of failing with no deployments available. * fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt Unmarked encrypted reasoning on a hop is attributed to the deployment that just failed, read from the retry breadcrumb, so a hop to a deployment on the same api_base and api_key keeps it and a cross-provider hop still drops it. The hop tests script the upstream at the httpx boundary instead of doubling the handler, and the router coverage script lists the two hop helpers with their tests * fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot * test(router): use a real Router without the origin deployment in the hop strip test * test(router): cover the hop strip when no failed deployment is known * fix(proxy): drop the router's fallback hop state keys from the client body * fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys * test(integration): audit cells for the fallback hop encrypted reasoning strip Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 17:30:01 -07:00
devin-ai-integration[bot]	f09408dd37	test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) ... * test: replace 61 live logging and otel tests with offline unit and integration coverage * test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts. * test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request * test: isolate the prometheus latency test on its own deployment so counts do not depend on order * test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only * test: drain the redis test's logging and scope otel span checks to the test's own trace * test: drive the periodic slack flush without a wall-clock interval * test: scope router hook events to the test, script the db clock, split a nested comprehension --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 15:55:57 -07:00
devin-ai-integration[bot]	d5173e3d70	test: move 76 live top-level tests to offline unit and integration coverage (#45352) ... * test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 10:33:35 -07:00
devin-ai-integration[bot]	04d97abffb	test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs (#45298) ... * test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs * test: restore lost coverage in litellm_utils offline replacements Bound live health-check tasks to the concurrency limit so eager task creation behind a semaphore fails, route the Langfuse trace-id check through completion -> Logging.get_trace_id with all four metadata combinations, run the callback dedup checks through acompletion, pass the dynamic key via litellm_params, cover the env-inferred default model list, and add the per-provider audio transcription config lookup * test: restore router coverage lost in the offline move Add the missing non-stream LIT-3058 header/count-once test, pin the UTC minute on every usage-counter test, assert router-level client reuse for transcription, and replace the private selector-attribute checks with routing outcomes per strategy. Assert outbound bodies for speech, rerank, image, assistants and moderation, add timeouts to event waits, and drop the router_unit_tests husk files that no longer hold tests * test: restore sync stream, router sync stream, error event and field-type coverage for migrated responses tests Add the sync streaming logging and sync Router.responses streaming cases the offline replacements dropped, port the legacy per-event and response field-type validation, assert raw headers, the float created_at conversion, MCP tool headers, the search_context_size input, and add an offline replacement for the in-stream context-window error event. Remove helpers left dead in the legacy file. * test: run numpydoc-backed unit tests in GHA and drop a test that pinned a crash * test: restore sequence number, item id and content part checks in the responses stream validator * test: match router logging events to the test's own deployment so late events from other tests are ignored * test: restore the s3 cold storage history test to its original assertions --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 02:37:55 -07:00
Mateo Wang	328f5a720c	fix(router): honor a per-request fallbacks list on a mid-stream fallback (#45227) ... * fix(router): honor a per-request fallbacks list on a mid-stream fallback The completion entrypoints record the per-request fallbacks, context_window_fallbacks, and content_policy_fallbacks in the mid-stream controls carrier the Responses and Messages paths already use, and each completion attempt restores them into the kwargs its stream re-enters the fallback chain with, so a streaming request that dies before its first chunk fails over to the list the request named instead of the router-level list only. The sync vs async parity cell now asserts the backup's text on both twins. * test(router): annotate the new fallback test locals as Final and flatten the leak check * fix(proxy): skip null key and team router settings when merging per-request overrides * test(proxy): annotate the null router settings test locals and drop a redundant comment	2026-10-07 22:25:22 -07:00
Mateo Wang	a7ee038592	fix(router): keep include_fallback_errors off the provider call on the sync Router path (#45218) ... * fix(router): keep include_fallback_errors off the provider call on the sync Router path Router.completion spread include_fallback_errors into litellm.completion, so OpenAI answered 400 Unknown parameter and a fallback walk lost the backup's reply. Only Router._acompletion popped the router-only flags. Both paths now build the provider call through one shared helper that drops them, and a wire-level respx test covers each path. * test(router): pin include_fallback_errors off the wire across the sync, async, and proxy paths	2026-10-07 22:25:12 -07:00
RachelHuangZW	bf9f59d76b	fix(scheduler): remove a request's queue entry once it stops waiting (#43061) ... * fix(scheduler): remove a request's queue entry once it stops waiting Requests admitted while a healthy deployment existed never left the priority queue, and neither did requests cancelled while waiting. The stale entries blocked later requests during cooldown and, with Redis, made add_request raise a TypeError on queues read back as JSON lists Both scheduling paths now share one polling helper that removes the entry in a finally block, whether the request was admitted, timed out or cancelled Related to #43059 * test(router): allowlist _wait_for_scheduler_turn in the router coverage check The coverage script only counts direct calls in test files. The helper is exercised through prioritized acompletion and atext_completion in test_router.py, like the other allowlisted entries * fix(scheduler): admit healthy requests before reading the queue, and clean up cancelled enqueues poll() raised on an empty queue before checking for healthy deployments. With the cleanup now rewriting the queue after every admission, a concurrent write from another replica can erase a waiting request's entry, and that request then failed while a deployment was healthy. poll() now admits as soon as a deployment is healthy and only reads the queue during cooldown add_request also moved inside the try block, so a request cancelled while its queue write is in flight still has its entry removed * refactor(scheduler): move the wait loop into Scheduler.wait_for_turn The router passes a healthy-deployments callable into the scheduler, so tests inject a Scheduler directly instead of replacing the router's scheduler attribute. The cancelled-mid-enqueue test moves to test_scheduler.py with the other scheduler tests, and poll() takes the deployments as a Sequence since it only checks whether any are healthy * test(scheduler): move scheduler tests into tests/unit * fix(scheduler): finish the queue removal when cancellation is delivered again during cleanup A second cancel, or an anyio cancel scope that re-cancels on every await, interrupted remove_request mid-write and left the entry in Redis. * fix(scheduler): read queue entries back from redis as tuples * fix(scheduler): admit a request whose queue entry vanished during cooldown * fix(scheduler): re-enqueue a request whose queue entry vanished during cooldown * test(integration): cover priority scheduler queue cleanup across instances Adds tests/integration/routing/test_priority_scheduler_queue_cleanup.py: 27 cells against the real proxy (two workers, Postgres, Redis, a scripted upstream) for every prioritized surface (/v1/chat/completions, /v1/completions, /queue/chat/completions, streaming and not, OpenAI SDK sync and async, raw httpx), the non-integer priority pass-through, the in-memory queue on one proxy, two proxies sharing a Redis queue (served requests leave no entry, a dead replica's entry is skipped or expires, a waiter that times out or disconnects removes only itself), a Redis outage mid burst and a SIGKILLed worker. Each cell asserts the caller's response, the upstream's requests by marker and the Redis queue contents. On the merge base the cross-instance cells fail with the list-of-lists TypeError and the in-memory cell with 408s behind the leaked entry; on this branch every cell passes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 17:34:38 -07:00
amarrtech	88f15e7572	fix(router): resume sync streaming fallbacks without retrying primary (#43959) ... * fix(router): resume sync streaming fallback chain Signed-off-by: amarrtech <272048731+amarrtech@users.noreply.github.com> * test(router): cover the sync mid-stream fallback walking every configured target * test(router): cover the sync stream fallback walk against a wire upstream --------- Signed-off-by: amarrtech <272048731+amarrtech@users.noreply.github.com> Co-authored-by: amarrtech <272048731+amarrtech@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 15:50:24 -07:00
devin-ai-integration[bot]	fa2c8984ba	test: move the unit half of 126 mixed legacy files into tests/unit (#45090) ... * test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-07 14:07:43 -07:00
devin-ai-integration[bot]	dfe4df8c8a	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) ... * test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-07 05:02:28 +00:00
devin-ai-integration[bot]	c3a23fe499	refactor: expose core private helpers under public names (#44871) ... * refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 00:05:27 +00:00
tin-berri	a193c67347	feat(otel): trace auto-router configuration and classifier failures (#44926)	2026-10-06 15:40:34 -07:00
devin-ai-integration[bot]	837c6a7481	fix(security): remove the publicly known master key from the repo (#44718) ... * fix(security): hash the publicly known master key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: replace weak master key examples and regenerate artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: replace weak key fixtures with generated test keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: generate master keys for proxy startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve lens dev key entropy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore proxy key compatibility in scrub examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: scrub merged SSO fixture and refresh dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): stabilize test keys and metadata collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 10:55:24 -07:00
devin-ai-integration[bot]	10444df3a0	test: deflake the JEV classifier select and two router tests that inherited leaked state (#44840) ... * test(ui): wait for the classifier model popup before picking its option Base UI exposes its select option asynchronously, and the helper waits for the option and its positioner to become clickable before selection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: isolate two router tests from work leaked by earlier tests Collect earlier tests' garbage before warning capture so their unawaited coroutines cannot be attributed to the target's warning assertion Filter success-event callbacks by the request's litellm_call_id so queued logging work cannot replace the current request's captured messages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 10:27:49 +00:00
devin-ai-integration[bot]	c13746c951	fix(router): price a model group from the deployments that serve it (#44732) ... * fix(router): price a model group from the deployments that serve it A model group's info read composed the group's own model_group_alias entry into its deployments, a hop the router never takes: an alias is resolved exactly once at request time, so a group reached as an alias target is served by its own deployments. For the chain X -> T -> U the price read for X included U's deployments too, and the free-model budget waiver refused a free request to X on an over-budget key, while GET /model_group/info reported U's providers and price for X. The group info read now prices a group from the deployments routing serves it with: the ones named after it, the routing group of that name, or the wildcard route matching it when neither exists. The budget waiver, GET /model_group/info, the rate limiters, and the response headers all read the same set as routing. get_model_list keeps its behavior for every other caller. * test(router): give the paid fixtures explicit per-token prices * test(router): call the routed-group read by name so the router coverage gate sees it * test(integration): audit the alias chain budget waiver on every route, shape, and outage Thirty-four cells under the management group prove an over-budget key is served through an alias chain entry at the price of the deployment that serves it, on chat, responses, and messages, sync and streamed, through the OpenAI and Anthropic SDKs and raw httpx on both replicas, with the chain middle, the reverse chain, a cost-map priced middle, a ghost middle, wildcard and routing-group targets, malformed and hostile inputs, a cached reply, a repointed alias, a provider failure, and two chaos bursts (a killed worker, a scripted outage) --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-06 05:31:09 +00:00
devin-ai-integration[bot]	68d9b8bbb8	fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) ... * fix(router): retry a /v1/messages stream the provider drops before the first content chunk A /v1/messages stream that the upstream closed before any content reached the client answered an error event after a single attempt, so the router's num_retries never applied to that drop. The pre-content failure is now retried within the model group before the fallback chain runs, with the budget resolved the way a failure raised before the stream opened resolves it: a retry policy that names the error class, then the request's num_retries, then the deployment's, then the router's. A drop after content reached the client keeps surfacing the provider's error after one attempt. Fixes #44238 * fix(router): hand a retry's non-retriable error to the fallback chain and type the retry helpers A retry that failed before its stream opened with an error no retry covers raised straight to the client, skipping a fallback the first attempt would have used. assert_never now comes from typing_extensions so the router imports on Python 3.10, and the retry helpers read their kwargs through typed narrowing instead of Mapping[str, Any] * fix(router): cast the untyped router fallback defaults the stream retry gate reads The retry gate passed the router's fallback attributes, declared without element types, to the typed request override helper, which basedpyright counted as new unknown-argument errors * fix(router): consult context_window_fallbacks when a retried /v1/messages stream overflows A retry attempt raising ContextWindowExceededError reached the fallback chain inside its mid-stream envelope, so only the regular fallbacks list matched. The fallback attempt now unwraps it the way it unwraps a content policy error. The new router helpers are covered for the router code coverage check with two direct-call tests and named covering tests * fix(router): retry a 408 raised by a /v1/messages retry and honor deployment num_retries before the stream opens * fix(router): attribute a retried /v1/messages stream to the deployment that served it and bound the retry-policy hold * fix(router): retry /v1/messages error frames under their retry-policy class and keep the first drop's committed budget An `event: error` frame that arrives before the first content delta now raises the exception class the pre-stream mapping gives an HTTP answer with the same status (429 RateLimitError, 500 and 529 InternalServerError, 503 ServiceUnavailableError, 504 Timeout), so a retry policy's per-class budget governs it the way it governs the error before the stream opened. The status the client sees is unchanged A retry that lands on a sibling deployment keeps the budget the first drop committed to, read back from the request's attempted_retries and max_retries, instead of recomputing it from the new deployment's num_retries, matching the pre-stream retry loop * refactor(anthropic): keep the error-frame exception mapping under llms and type the retry test helper The status-to-exception mapping an `event: error` frame gets before the retry policy is consulted now lives next to the Anthropic error status map in llms/anthropic/common_utils.py, with its own unit test, and the two-deployment retry test helper takes explicit typed parameters instead of a bare dict and untyped kwargs * refactor(anthropic): map an error frame's status with explicit returns on every path * fix(router): map stream error frames through the pre-stream exception mapping An overloaded `event: error` frame on a /v1/messages stream now raises the InternalServerError a 529 answer maps to, built by exception_type from the frame's own body, so one retry policy class governs the error before and after the first byte; a failed fallback after such a frame answers 500 like every other litellm path instead of the frame map's 503 A model_group_retry_policy that does not parse (a non-integer budget, an entry that is not a mapping) no longer fails every healthy stream of that group before its first attempt: the stream runs with no policy and the plain num_retries budget, with a warning naming the group * fix(router): forward an error frame nothing can take over for as the provider sent it A pre-content error frame whose class the retry policy grants no retry, with no fallback configured, raised an HTTP error only on the first attempt while the same frame after exhausted retries reached the client verbatim. Both now pass through as sent, the way the merge base forwarded every frame. * test(integration): audit /v1/messages pre-content retry across routes and budgets Adds the /audit cells for the pre-content stream retry: the native Anthropic route (drops and error frames before content, HTTP rejections before the stream opens, SDK sync and async, after-content and non-retriable controls, budget exhaustion, cache twin, spend row and headers), the chat and responses bridges, the generic routes (responses, chat, vllm pass-through, Gemini generateContent, fine-tuning jobs list), owned two-worker proxies for router-level budgets, retry policies and fallbacks, and two chaos cells (a worker killed mid burst, an outage on every first attempt). Shared helpers for scripted Anthropic SSE upstreams and OpenAI-compatible wire replies live in tests/integration/_support --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-05 22:53:39 +00:00
devin-ai-integration[bot]	0ed1c08f02	feat(anthropic): workload identity federation and pluggable identity sources (#44448) ... * feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * fix(proxy): let a deployment write name a federated credential reject_federated_credential_reference runs from is_request_body_safe, which pre_db_read_auth_checks calls on every route, so it also fired on POST /model/new, /model/update, /model/{id}/update and /health/test_connection. A proxy admin could no longer attach a federated credential to a deployment over the API or the Admin UI, leaving a static config.yaml entry as the only way to configure the feature the rejection told the caller to go configure, and _reject_non_admin_wif_write never got to make the call it exists to make. is_request_body_safe now takes the route and skips only the credential-reference check on the routes that reach can_user_make_model_call. Federation fields typed inline into a body stay refused everywhere, and a call naming a federated credential still cannot pick the identity it mints as. * refactor(proxy): derive health display policy from the federation key sets The health check module hand-copied the five workload identity fields whose value is a credential, so a shared proxy surface named provider-specific parameters and a newly added secret-bearing field would have gone on being displayed until someone remembered both places WIF_SECRET_BEARING_KEYS now sits beside the key sets it splits out of, types/utils derives secret_bearing_wif_litellm_params from it, and the health layer splats that tuple the same way it already splats the admin-only one * fix(anthropic_wif): treat blank identity-source fields as unset * test(proxy): classify the federation params in the credential slot registry main's registry test (#43298) now fails the build for any credential-named deployment param without a classification. The five federation fields that carry a token, a token file path, or a signing or client secret reference are Unplanted, matching WIF_SECRET_BEARING_KEYS; the four remaining Keycloak settings name a URL, a client id, an auth method, or a scope and are NotSecret * fix(anthropic_wif): declare federation params as owned connection leaves and chart their metrics Register the 18 Anthropic and 3 OpenAI federation params as frozen ConnectionSettings leaves so the owned-kwarg registry, the kwargs funnel and the request-body ban list read one declaration. Pass the deployment api_base through to the count-tokens handler instead of a pre-suffixed URL, which doubled the /count_tokens path on main's prompt-cache predictor. Add the five litellm_anthropic_wif_* families to the all-metrics Grafana dashboard. * fix(credentials): gate PATCH on WIF fields resolved from model_id The credential PATCH handler checked server-owned workload identity federation fields only on the values the caller sent, while a body that named a deployment through model_id had its credential values resolved after that check. A non-admin could therefore copy a federated deployment's WIF fields onto an ordinary credential. Resolve the incoming values first and run the non-admin gate on them, matching the POST path * fix(anthropic): count tokens with ANTHROPIC_AUTH_TOKEN through the shared auth header Count-tokens walked its own credential ladder: a static key, else skip minting when ANTHROPIC_AUTH_TOKEN is set, else mint a federated token. With only the auth token set it forwarded nothing and the proxy silently fell back to its local tokenizer while chat on the same deployment authenticated with that token. The handler now takes the auth header that AnthropicModelInfo.aget_auth_header resolves, the same ladder chat, files, batches and skills use, and merges the oauth beta a minted or consumer token carries with the token-counting beta --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> Co-authored-by: mateo-berri <happymvw@gmail.com>	2026-10-03 17:08:30 -07:00
devin-ai-integration[bot]	1ef0fe9790	fix(auth): resolve hidden model_group_alias entries in the zero-cost budget check (#43741) ... * fix(auth): resolve hidden model_group_alias entries in the zero-cost budget check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): key the zero-cost cache by the resolved model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): keep the zero-cost verdict per requested name and include hidden aliases * test(integration): audit the zero-cost bypass through hidden model_group_alias names * test(router): cover the extracted routing strategy switch * test(integration): record a pre-flip burst before the alias flip --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-03 16:35:11 -07:00
devin-ai-integration[bot]	fe683ea139	feat(proxy): serve a Codex-native model catalog with per-model service_tiers from /v1/models (#44136) ... * feat(proxy): serve a Codex-native model catalog with per-model service_tiers from /v1/models GET /v1/models and /models answer Codex CLI's catalog fetch (the request carrying its client_version query parameter) with Codex's own {"models": [...]} shape: a model Codex knows keeps the metadata of its bundled 0.159.3 catalog (vendored), any other model gets Codex's fallback entry, and model_info.service_tiers becomes each entry's service tiers so Codex offers them as slash commands that send service_tier upstream. Without the parameter the OpenAI list shape is unchanged. The CLI's litellm agents codex catalog shares the same builder. * fix(proxy): offer a Codex service tier only when every deployment of the model lists it * fix(codex-catalog): an invalid service_tiers value offers no tier for the model * fix(codex-catalog): read service tiers off the deployments the key's team can route to A tier is offered to Codex only when every deployment of the model name a request from the key's team can route to lists it, so another team's deployment of the name and a deployment an admin paused via model_info.blocked no longer withhold or add tiers for requests that never reach them The catalog's always-null fields are annotated NoneType so the module imports under pydantic 2.12.0 on Python 3.14, the lowest pin the MCP resolve job installs, which rejects a None annotation with a None default * test(codex-catalog): drop the redundant module docstring and sort the imports * test(integration): add the Codex catalog audit cells and the multi-worker convergence note * test(integration): clean up every catalog test model and answer the refresh GET * fix(proxy): keep tiered models under Codex's catalog cut and resolve alias tiers Under Codex's 1 MiB catalog limit the entries offering a service tier are kept ahead of those offering none, each group in model_list order, with every kept entry at its listing position, so the model an operator configured tiers for survives a wide key's long listing. A model_group_alias row reads its target's deployments, so it carries the target's tiers and stock metadata under the alias name. * fix(proxy): pick Codex catalog metadata per team and skip entries too large for the cut The upstream model that selects Codex's stock entry was read off the first deployment of a name without checking the key's team, so a team whose requests route to a different deployment could be handed another team's prompt, reasoning levels, and tiers. The upstream model and the tiers now come from the same team-aware selection routing uses, and a caller with no team reads the deployments no team owns The byte cut kept a prefix of the tier-first order, so one entry larger than the whole limit emptied the catalog. An entry too large for the bytes left is now passed over and the smaller ones after it are still kept --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-03 20:42:55 +00:00
devin-ai-integration[bot]	4c6c84afb7	perf(proxy): stop prompt-cache eligibility from tokenizing the whole conversation (#44221) ... is_prompt_caching_valid_prompt ran the full Python token_counter over every message to compare against the deployment's prompt cache minimum, 500 to 1000 ms at 440k to 740k tokens on every request that reaches the prompt_caching pre-call check, Rust on or off. messages_reach_token_count does the same arithmetic as token_counter(...) >= threshold and stops at the first message that reaches the threshold. Groups with one healthy deployment skip the prefix hash and pin lookup, which cannot change the result for them Four fixed name span events make the pre-LLM phases measurable with OTel v2: litellm.request.body_received (with body_bytes) once per body read before parsing, on the JSON, binary and form branches, body_parsed, pre_call_completed, and deployment_selected emitted once per pick inside Router.async_get_available_deployment and get_available_deployment with attempt, reason and model group, so every router surface, retry and fallback is covered. Measured locally on /v1/chat/completions, /v1/messages and /v1/responses at 440k tokens with Rust on and off against a fake upstream Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 09:20:28 -07:00
devin-ai-integration[bot]	564d236985	fix(otel): nest cache spans under their operation and name service spans by purpose (#44150) ... Response cache reads and writes open cache.get llm_response and cache.set llm_response phase spans with their Redis spans nested underneath, on the Python path and on the native Rust path, and deployment selection runs inside a route {model_group} phase so the cooldown, usage and model-id reads the router issues nest under it before chat {model}. The autorouter classifier call nests under that route phase as well and carries its typed internal origin on litellm.request.purpose, so it is told apart from the provider attempt. Service spans are named {service}.{verb} {target} from a low-cardinality key family the producer declares (llm_response, auth_objects, spend_counters, router_cooldowns, claude_code_session_router_binding, rate_limits, pod_lock, budget_reset, ...) instead of the raw method or a per-request pipeline length; a pipeline flush is targeted by the one family its ops share or by mixed with the sorted families on litellm.redis.families, a batch op keeps the family it was declared under whichever pipeline or standalone read settles it, and the ambient family labels Redis spans only, never the DB write-back a task spawned inside that context performs later. The raw method stays on litellm.service.call_type and on the Prometheus and Datadog labels. Caller attribution is carried across asyncio task boundaries on a ContextVar so forwarder-only chains no longer surface, the raw cache key is dropped from Redis span metadata, pipeline op counts land as an integer attribute, every call_type the Redis cache layer emits maps to a verb, and a scan over litellm/ and enterprise/ fails when a Redis producer, batch reservation included, declares no key family. A V2 logger built for a key or team logging entry while the operator's V2 logger is already registered keeps only the exporters its own preset contributed, whether or not the operator holds credentials for that backend, so every chat span no longer reaches the operator's collector twice. A span the success callback has to open itself, with no pre-call carrier, starts at the provider handoff (api_call_start_time) instead of the logging object's creation. Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 09:20:27 -07:00
devin-ai-integration[bot]	314ff111e5	fix(router): carry per-request routing reads on context variables instead of public method kwargs (#43814) ... Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-30 07:34:06 +00:00
devin-ai-integration[bot]	ffb15f946f	perf(proxy): one request-scoped Redis pipeline for auth, spend, rate-limit and routing reads (#43407) ... RedisBatch: one pipeline per Redis backend for independently declared operations (MGET, GET, Lua scripts, INCRBYFLOAT, SET, DEL), a future per operation so each owner keeps its own fallback, Redis Cluster hash-slot fallback. A request-scoped batch middleware shares that pipeline across the auth identity reads and write-back, the spend counter MGET, the rate limiter Lua groups and the routing read. A rate-limit denial stands when another pipelined group fails; every pipelined group is refunded on rejection; local cooldowns win over the prefetch. The routing prefetch failure log line strips request line breaks (CodeQL py/log-injection) Resolves LIT-8882 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-29 16:42:13 -07:00
tin-berri	92c0d6f5c8	fix(router): bind Claude Code background sessions to their auto-router (#43767) ... Claude Code background sessions (claude --bg) stamp x-app: cli-bg on every request, including main-loop turns. The session router binding only accepted x-app: cli, so a background session never bound and its subagents' concrete-model calls bypassed the router. The binding write already requires the requested model to resolve to a pre-routing strategy, so background side calls naming plain models still never bind. Since f6eff1bde0 removed the clear path, the x-app check guarded nothing else. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-29 15:17:14 -07:00
devin-ai-integration[bot]	d46304900f	fix(router): stream /v1/messages lifecycle frames live when no fallback can take over (#43600) ... * fix(router): stream anthropic messages lifecycle frames live when no fallback can take over The /v1/messages streaming wrapper buffered message_start and content_block_start until the first content_block_delta and dropped pings behind buffered frames unconditionally, even for requests no fallback could ever recover. With adaptive thinking on Bedrock or Vertex the client saw no bytes for the whole thinking pass and hit read timeouts. Buffering now applies only while a fallback can still take over (generic or refusal chain resolving), and a ping is always forwarded live since it carries no lifecycle and keeps the connection alive. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): mirror every dispatcher fallback path in the anthropic stream gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): skip already-tried order levels in the anthropic stream gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): keep a transport-split ping behind buffered lifecycle frames instead of forwarding its head live Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(router): credit the #39566 branch this fix supersedes Co-authored-by: Radu Swigler <radu.porumba@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Radu Swigler <radu.porumba@gmail.com>	2026-09-29 09:22:05 -07:00
devin-ai-integration[bot]	26bf575f15	feat(guardrails): scan retrieved vector store chunks with the request's pre-call guardrails (#43271) ... * feat(guardrails): scan retrieved vector store chunks with the request's pre-call guardrails Vector store retrieval runs inside acompletion after the proxy's pre-call guardrails have already seen the request, so a retrieved chunk carrying an injection reached the prompt unscanned. Each retrieved context message now goes through every pre-call guardrail the request is subject to before it is injected: a block raises the same 400 the guardrail gives for user text, a masking guardrail rewrites the context, and a guardrail that fails while scanning fails the request instead of injecting the chunk unscanned * fix(guardrails): return a guardrail block unmapped from exception_type so the Responses API surfaces the guardrail's own 400 * fix(guardrails): build the deployment hooks' identity from stamped metadata only Top-level user_api_key_* fields in a request body are client controlled, so the pre-call, chunk scan, and post-call deployment hooks now take UserAPIKeyAuth from the metadata the proxy stamped, and the chunk scanner returns or raises on every branch. * fix(guardrails): block route verdicts on retrieved chunks, keep guardrail verdicts out of router retries and fallbacks, and scan chunks against the client's request * fix(guardrails): keep the merged guardrail list when scanning chunks against the client's request The scan request laid the client's kept body over the deployment kwargs, so a client that sent its own top-level guardrails list shadowed the merged metadata.guardrails list and a key or team guardrail skipped the chunk scan. The kwargs now win and the keys the proxy relocates into metadata are dropped from the body's contribution. * fix(guardrails): strip the deployment's guardrail keys from the chunk scan request so merged team guardrails still run * test(guardrails): type the vector store scan test doubles * test(guardrails): type the scan double's request data --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-09-26 14:18:38 -07:00