261 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	a44e44b980	fix(cache_control): pick the chat-path dialect from the deployment's provider prefix (#45295) ... * fix(cache_control): resolve the chat-path dialect with the deployment api_base * test(cache_control): type the chat-path resolver helpers * fix(cache_control): pick the chat-path dialect from the deployment's provider prefix Resolving with the deployment api_base followed the dispatch's azure_ai to azure flip, so chat on a non-Foundry Azure host sent Anthropic cache_control while /v1/messages, told azure_ai, sent the OpenAI breakpoint. Use the model's provider prefix when the caller does not name a provider, so chat matches the told path whatever AZURE_AI_API_BASE holds, and keep name-only resolution everywhere else. * fix(cache_control): keep litellm_proxy routing when the chat path reads the provider prefix With USE_LITELLM_PROXY or litellm.use_litellm_proxy on, every call goes to the upstream proxy, so the provider prefix no longer decides the cache format there and resolution matches main. The plain OpenAI check reads the same provider value as the hosted check, so one function no longer mixes two lookups * fix(cache_control): resolve the chat-path dialect with the deployment api_base, as the call is routed The hosted cache-format check now resolves the provider with the deployment's own api_base when the caller names none, the same provider the call dispatches to and the Responses path passes, so AZURE_AI_API_BASE no longer decides the chat format. The plain OpenAI check keeps main's name-only lookup. * fix(cache_control): send acompletion fallbacks the caller's request, not the primary's hooked kwargs acompletion handed async_completion_with_fallbacks the kwargs the cache seed and prompt hook had already rewritten for the first model, so a Claude fallback behind a Foundry GPT model received prompt_cache_options and got a 400. Fallbacks now get a copy of the caller's model, messages, tools, and kwargs taken before the hooks, as completion already does with its args. Both cache seeds also read base_url before api_base, since completion sends the call to base_url when both are set. * fix(cache_control): keep the caller's tools list intact for acompletion fallbacks The vector-store hook pops recognized file_search tools from the request's tools list in place, so the fallback request built from the pre-hook snapshot lost the retrieved context. Snapshot a copy of the list instead. * fix(cache_control): skip acompletion's prompt hooks when fallbacks will run them Each fallback attempt runs the prompt hooks for its own model, so running them first in the outer call only cost a vector-store search whose result the hand-off threw away. * fix(cache_control): run acompletion's prompt hooks for fallbacks and leave only the cache hook to each attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 05:11:42 -07:00
devin-ai-integration[bot]	02ef44c388	fix(streaming): convert upstream logprobs so streams with logprobs end with their finish chunk (#33549) ... * fix(streaming): normalize provider logprobs to prevent MockValSer serialization crash * fix(streaming): convert upstream chunk logprobs to litellm ChoiceLogprobs so the finish chunk serializes A streaming chat completion with logprobs on the OpenAI SDK path attached the SDK's own ChoiceLogprobs object to the outgoing chunk when the chunk had no content to rebuild from, which is the provider's finish chunk (empty delta, logprobs for the end token, finish_reason). The SDK parses stream chunks without building that nested class, so its pydantic serializer is still a MockValSer and model_dump_json fails. The proxy then sent an error frame in place of the finish chunk, and when the same chunk also carried usage the SDK wrapper itself raised MidStreamFallbackError. The attach site now validates upstream logprobs through a TypeAdapter that dumps any non-litellm pydantic model before validating into litellm's ChoiceLogprobs, and passes litellm instances through unchanged. * fix(streaming): keep upstream logprobs that don't fit ChoiceLogprobs as plain JSON so the stream still finishes * fix(streaming): build ChoiceLogprobs' serializer up front and keep nonconforming logprobs as JSON on chunk rebuilds ChoiceLogprobs inherits the openai SDK's deferred build, so a fresh worker whose first logprobs arrive on the finish chunk still had no serializer. Foreign logprobs models now dump in full, so a missing top_logprobs becomes [] and SDK callers keep attribute access, and chunk rebuilds for content and usage keep logprobs that don't fit as plain JSON instead of raising. * fix(streaming): keep only the logprobs keys upstream sent and type refusal logprobs * fix(types): pass nonconforming non-streaming logprobs through instead of failing the response --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 04:41:02 -07:00
Mateo Wang	373f1752d3	fix(anthropic): send PDF image_url content to Claude as document blocks (#46984) ... * fix(anthropic): convert PDF image_url content to documents Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt_templates): strip content-type parameters from fetched media Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(prompt_templates): format MIME normalization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 02:21:39 -07:00
Mateo Wang	4c8b90945f	fix(realtime)!: send the Gemini Live setup at the first content so session.update settings apply (#39864) ... * fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its default audio setup at connect, so a client's session.update (GA output_modalities or beta modalities) was dropped and every answer came back as audio. The setup now goes out on the client's first message: a session.update becomes that setup, anything else gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a provider hook. Text setups label deltas response.output_text.delta, and downgrading TEXT to AUDIO on an audio-only model logs a warning * fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup * fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup Gemini and Vertex Live accept one setup per connection, so the bridge used to turn only the first session.update into that setup and drop the rest without telling the client. Now every session.update that arrives before the first content frame is held, deep-merged (later values win per field), acknowledged with a session.updated built from the setup it would produce, and the merged session becomes the one setup sent with the first conversation item, audio frame, or tool result. The session events also surface the systemInstruction text as instructions, which the old string check never matched because the setup carries it as a Content object. * fix(realtime): cap the session.update merge depth so the recursive detector accepts it _merge_session_fields recurses once per nested mapping level, so CI's recursive_detector flagged it. Bound the descent at eight levels (the OpenAI session object nests four deep at most), replace anything deeper wholesale with the later session.update's value, and list the helper with the other depth-capped functions the detector allows. * fix(gemini-live): accept a string output modality and drop values Live cannot produce A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once. * fix(gemini realtime): answer an error for session updates Gemini Live cannot apply An output modality the model cannot produce, or a session.update sent after the first content, now gets an invalid_request_error event instead of a silent swap or drop, unless drop_params is set. The synthetic session.created reports the model's default modality, setupComplete no longer adds an extra session.updated, and merging nested session updates is linear in their size * fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns A pre_call guardrail blocking the first typed message now goes out after the setup, so Gemini and Vertex no longer close the session with 1007. Push-to-talk sessions (turn_detection null or create_response false) wrap each audio turn in empty activityStart and activityEnd messages, which Gemini needs when automatic activity detection is off. A session with no content after 60 seconds sends its setup, since Vertex refuses a setup 300 seconds after connect. Held session.update frames past 10 MB answer session_update_too_large, and deferred mode now errors on a second session.update unless drop_params is set. * fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection An update that leaves the applied setup unchanged now gets session.updated instead of session_update_not_applied, so clients that resend their session with the first response keep working. Deferred setup mode merges held updates like the default mode, turn_detection null turns Gemini's activity detection off, drop_params drops a voice outside the documented Live voices, and the lazy setup deadline moves to 240 seconds * fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content * fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name. * fix(realtime): hold first content so a following session.update joins the Gemini setup Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too * fix(realtime): check AI Studio Live voices like Vertex * fix(realtime): forward any voice to AI Studio Live models that accept it * fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup. * fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock The deadline and grace tests waited on real 50 ms timers, and the merge cost check compared wall-clock runs. The session now reads an injected clock, time.monotonic by default, the two tests step it, and the merge cost check moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes the backend through wrap_backend, which main added after the test was written	2026-10-30 01:31:39 -07:00
devin-ai-integration[bot]	56388fd13d	fix(count_tokens): count a history that carries a compaction block (#44784) ... * fix(count_tokens): count a compaction block instead of answering 500 A message history that replays Anthropic's signed compaction block made /v1/messages/count_tokens answer HTTP 500: neither the Anthropic nor the Vertex AI partner count-tokens request carried the compact-2026-09-04 beta, the provider rejected the block with a 400, the proxy fell back to the local tokenizer, and that raised on the unknown block type. Both provider count paths now add the compaction beta whenever the history carries a compaction block, and the local tokenizer counts the block's summary text like a text block, skipping the signature. * fix(token_counter): skip the tokenizer for empty thinking and compaction text * fix(count_tokens): forward context_management on Anthropic counts and count chat compaction blocks locally Vertex AI compaction support for inference and counting moves to its own PR, so the Vertex count handler is back to main here * fix(anthropic): map list-form context_management to edits for count_tokens and inference betas Count requests forwarded the OpenAI list form of context_management unchanged, while inference maps it to Anthropic edits, and the inference beta helper read the list as no edits. Both now map it once. A replayed compaction block with an empty summary no longer reaches the tokenizer. * fix(anthropic): forward count context_management only with edits, map Vertex list-form betas, and return provider context_management * fix(anthropic): drop an unmappable context_management list under drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 00:11:55 -07:00
zoveki	dbcacc7ab7	fix(health): probe only models a partial wildcard serves (#49060) ... * fix(health): probe only models a partial wildcard serves A partial wildcard health check now probes the cheapest cost-map chat models the wildcard matches, keeps provider route prefixes such as Bedrock converse/ on the probe, and falls back only within the wildcard. A wildcard with no matching cost-map chat model reports an error naming model_info.health_check_model * fix(health): keep the provider-wide probe for an unmatched wildcard, skip retired models, and ignore Bedrock ARNs A partial wildcard with no served cost-map chat model falls back to the provider's cheapest chat models instead of failing, and fails with a message naming model_info.health_check_model only when the provider has none. The cheapest-model picker skips models past their cost-map deprecation_date, and a Bedrock ARN segment is never read as a route prefix. * fix(health): fall back to the provider's cheapest probe when a wildcard's own models fail A partial wildcard now probes up to three of its own cheapest chat models first and then the provider-wide cheapest ones, all from one cost-map scan, so stale or unpriced rows the wildcard matches cannot turn a healthy deployment red. Retired models are read by their resolved cost-map key, both deprecation checks use the UTC date, Bedrock only treats converse, invoke, and region names as route segments, and equal-price ties break by model id * fix(health): cap wildcard health probes at three and rank retired cost-map rows last A partial wildcard now probes two of its own cheapest cost-map chat models and then one provider-wide model, so a failing deployment makes no more calls than before. Rows past their cost-map deprecation date go last instead of being dropped, since some still serve, and the shared cheapest-model picker returns what it returned before. * fix(health): probe a wildcard's own models first, then the same three models main probes A partial wildcard's health check tries its two cheapest own cost-map models, then the provider's three cheapest models main already probes, so a row main shows healthy stays healthy. The deprecation check uses the server's local date, like the Baseten listing * fix(health): keep main's provider-wide probe order so equal-price ties match main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 23:34:57 -07:00
mika-lt	f32ac94ad1	fix(openai): keep the upstream error body on e.response for OpenAI and Azure errors (#49079) ... * fix(openai): keep the upstream error response on mapped exceptions so e.response.text has the wire body * fix: keep the upstream error body on 429 and 5xx errors RateLimitError, InternalServerError, BadGatewayError, and ServiceUnavailableError rebuilt their response and dropped the upstream body. They now keep the body and exactly the upstream headers, and an unread streaming response no longer raises. * fix: keep a gzip-encoded 429 or 5xx error readable The kept body is already decoded, so the rebuilt response drops content-encoding instead of decoding the body a second time and raising DecodingError. * fix: keep the upstream headers and drop credentials on every mapped provider error The copied error response now keeps every upstream header and carries a request with only the method and the URL, so no provider key, query secret, or prompt reaches e.request. Timeout and APIError get the response too, and azure/ chat, streaming, embedding, and image errors attach it * fix: keep the upstream error body on gateway timeouts and re-raised gateway errors * test(azure): cover the 404, 500, 502, and 504 error bodies and the missing-key placeholder * fix: keep no e.response on timeouts and errors litellm raises itself * fix: keep the upstream elapsed time on error responses and the upstream body on azure_text errors * fix: keep the upstream body without credentials on Azure content filter errors and message-matched OpenAI errors --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 23:30:02 -07:00
paveln83	b91fc92b6d	fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054) ... * fix(router): cap the default OTPM reservation for requests without max_tokens * fix(router): reconcile ITPM and OTPM reservations after every successful call * fix(router): read the default OTPM reservation from the served model and make its fraction configurable The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check. * fix(router): refund a failed attempt's io token reservation in make_call The proxy passes one logging object to every retry, and it runs failure callbacks only for the first attempt, so the last attempt's ITPM and OTPM reservation stayed held until the minute ended. Each attempt that raises now gives its reservation back before the retry or fallback * fix(router): count Anthropic cache writes once in ITPM reconcile Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens. * fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction * fix(router): reserve with the deployment cap only through the param the request's API sends * fix(router): never let a deployment's output cap raise the reservation above the default * test(router): move the reservation fraction env var test to the integration suite The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation. * test(router): retry the reservation fraction scenario when the minute rolls over The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute. * test(router): give the rollover retry its own deployment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 20:32:04 -07:00
Kerry Lu	733cfeba93	fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced (#48953) ... * fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced * test(integration): a reported cost is attributed to output in the gateway parity cases * fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects * fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge * fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost * fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill * fix(cost): read the stored breakdown total through the logging object's typed field * refactor(cost): name the providers billed from their own usage fields for what they are * fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 20:30:42 -07:00
Kerry Lu	23595f7fe5	fix(bedrock): backfill published regional service tier prices (#49113) ... * fix(bedrock): backfill published regional service tier prices * fix(bedrock): remove the eu.anthropic.claude-haiku-4-5-20251001-v1:0 row --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 19:46:57 -07:00
norkelbyte	44cd858697	fix(cost): bill the requested model's rates when the response names an unpriced model (#48900) ... * fix(cost): bill the requested model's rates when the response names an unpriced model Fireworks answers chat and /v1/messages calls with a display name such as "Qwen 3.8 Max" in the response's model field. The cost path priced that name, the Fireworks calculator fell back to its free default bucket without raising, and the call billed $0 even though the requested model has a row. When the response model resolves to no cost-map row or rule that declares rates and the requested model does, price from the requested model. A priced response model still wins, so dated snapshots keep their own rates, and an unpriced requested model such as a router alias leaves the result unchanged. * perf(cost): check the exact cost-map row before generalization rules when picking the priced model * fix(cost): apply the priced fallback to the provider response model streaming records * perf(cost): skip the served model's rule lookup when the requested model has no rates * fix(cost): price the requested model without a model arg and keep Fireworks size buckets behind the proxy When completion_cost runs with no model, the requested model now comes from the call's hidden litellm_model_name, so a display-name response still bills at the requested model's rates. A Fireworks model whose name carries a parameter size counts as priced through its size bucket. The Fireworks per-token lookup now treats a cost-map row that prices nothing, like the empty row the Router registers for an unlisted deployment, as missing, so an unlisted Fireworks model bills at its size bucket instead of $0 even when Fireworks echoes its slug. The "row prices anything" helpers move to llm_cost_calc/utils so the Fireworks lookup can reuse them without a circular import. * fix(fireworks): keep prices from a matching fallback rule for an unlisted model An unlisted Fireworks model that only a fallback generalization rule prices has no raw cost map row, so the empty-row check sent it to the size bucket and dropped the rule's rates. The Fireworks lookup now keeps the model info when the row or a matching rule declares a rate. * fix(cost): price a bare response model under the requested provider and drop the unlisted Fireworks bucket change The logging payload picks the pricing model without a provider, so a bare echo like mistral-large-latest failed its price check and the requested alias took over model_map_key. The served name is now priced under the requested model's provider, and an echo that is the requested name minus its provider prefix skips the checks. Treating the Router's empty price row as missing let unlisted Fireworks deployments bill from a name guess, which misprices dedicated deployment ids and routes some short names to another provider's rates. That change and its tests go back to the merge base, so unlisted Fireworks models keep billing $0 behind the proxy as on main. * perf(cost): read the response model's price row without building full model info An unpriced display name like Qwen 3.8 Max made the price check build full model info, and Fireworks' supported params lookup scans the whole cost map for an unknown name. The row check now resolves the cost map key through the cached model info helper, so that pick takes about 0.45 ms instead of several. A regional test covers a Bedrock call whose response names an unpriced display name with no model passed, which bills the requested model's regional row. The duplicate get_model_info wrapper goes, since its twin already does the same thing. * fix(cost): keep the served model's filled-in and size-bucket rates, and cache misses A served row priced only by a fill_missing_for_providers rule bills those rates on main, but the price check read only the raw row and billed the requested alias instead. It now merges the rule's fill-in keys through the helper model info lookup uses. A served Fireworks or Together AI id that names its parameter count keeps its size-bucket price, as on main, instead of switching to the requested row. The served model is checked first, cost map misses are cached and cleared with the model info cache, and the rules are read only for a name that resolved through one. A priced snapshot behind an Azure deployment name takes about 19 us of CPU time against 7 us on the merge base, down from about 300 us. * fix(cost): bill the requested model's listed rates over a served size-bucket guess A served Fireworks or Together AI name that is not in the cost map but names its parameter count, like a display name with a size in it, now bills at the requested model's listed rates instead of the provider's size-bucket price. The bucket price still applies when the requested model has no rates, as on main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 19:19:36 -07:00
Mateo Wang	326f17594e	fix(batches): return real batch token counts next to OpenAI's batch usage fields (#47740) ... * fix(batches): read OpenAI batch usage into the batch token counts * fix(batches): serve OpenAI's batch usage shape from retrieve and the cost poller * fix(batches): keep the chat usage keys next to OpenAI's batch usage fields Batch usage now carries every chat key it had plus OpenAI's five batch fields with the same counts, so poller-billed batches keep prompt_tokens and cache-write tokens, and older proxies still read the new rows. Usage with missing or null detail counts reads its real input and output counts. * test(batches): check the batch usage counts in the e2e terminal-state test * fix(types): apply nested include and exclude to both batch usage shapes * fix(batches): log no cost for a batch retrieve that is not final yet A retrieve now reads OpenAI's batch usage into the chat counts, so a retrieve that saw the batch completed before its output file arrived priced those counts for the logging callbacks, and the final retrieve priced the whole batch again. A batch that is not final now logs no cost, as it did before. Batch usage dumps also pass exclude_unset and exclude_defaults through to the usage keys. * fix(batches): pass through every usage detail field the provider sends Batch dumps rebuilt input_tokens_details and output_tokens_details from the chat keys, which dropped any detail field past cached_tokens and reasoning_tokens that the merge base passed through. The dump now starts from the provider's stored details and only sets the two counts derived from the chat keys * fix(batches): skip sync loggers on the cost poller's own retrieve and honor exclude_none on provider details * fix(batches): keep sync logger events at $0 on poller-owned retrieves, pass unparsed provider counts through, log batch usage in chat shape * fix(batches): store only chat usage extras on batch spend rows Spend rows for batch retrieves copied OpenAI's batch fields (input_tokens, output_tokens and their details) into metadata.additional_usage_values next to the chat counts they duplicate. Build the row from the batch's chat usage, shared with the logging payload through batch_chat_usage.	2026-10-29 18:29:30 -07:00
simonp9	594f862bf3	fix(streaming): rebuild each choice of an n>1 chat stream on its own (#49002) ... * fix(streaming): rebuild each choice of an n>1 chat stream on its own stream_chunk_builder folded every choice of an n>1 chat stream into choice 0, so post-call guardrails such as tool_permission re-emitted one choice holding every choice's tool calls, and spend logs and logging callbacks kept only the first choice's text. Each choice is now assembled from its own chunks, while usage, cost, and hidden params still read the whole stream. Cisco AI Defense compares each choice's streamed text on its own so safe n>1 streams still pass through unchanged. * fix(streaming): keep bridged reasoning summaries on choice 0 and infer an unreported choice finish reason * fix(streaming): match MCP follow-ups and bridged summary text to the rebuilt choices MCP auto-execute now answers only the first choice's tool calls, since its follow-up turn continues that choice alone. The Responses bridge streams a space before each reasoning summary part after the first, matching the non-streamed reply. build_base_response now takes the per-choice streams instead of having its choices replaced afterwards * fix(streaming): space every later reasoning summary part, deny per judged call, and rebuild in one text pass * fix(guardrails): key tool_permission denials for custom tool calls too, and type the stream rebuild's messages * fix(mcp): continue the first choice that called a tool when an n>1 reply auto-executes MCP tools * fix(noma): scan each choice of a multi-choice reply on its own * fix(noma): keep masked choices when a later choice's scan fails open on a stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 18:29:14 -07:00
Mateo Wang	23ad8870b4	fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill (#47554) ... * test(token_counter): price PDF pages for OpenAI and Azure models the way they bill * fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill * fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding * fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing Detect raster images from the page content stream, including inline images on pages with empty resources, and walk each form once. Strip only openai/, azure/ and Azure data-zone prefixes, bound the model family scan, read Azure-only chat models from the cost map, and keep cl100k_base for listed non-chat OpenAI models. * fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources Prices a PDF by the model name's last path segment, the same name the proxy counts with, keeps listed non-chat models on Claude pricing, walks each form once per inherited resources dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing only when a message carries a PDF * fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes count like their base model * fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models * fix(token_counter): count reservation and project ITPM with the serving deployment's model Budget reservation and project ITPM counted with the requested group name, so a GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count with each deployment's model minus its provider prefix, the name count_tokens uses, and take the largest. A custom OpenAI tokenizer with no model counts with cl100k again instead of raising, and listed Azure gpt- chat entries price as OpenAI. * perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator pypdf parsed every operator of a page's content stream to find the images it draws, so a compressed page of a million operators took about 30 seconds to price for an OpenAI model. A linear scan of the decoded stream for name Do and inline images finds the same draws. * fix(token-counter): count each team deployment by its PDF pricing and honor base_model Budget reservation and the TPM limiter now look up the deployments that serve the caller's team, by base_model when set. A deployment that reads PDF pages and prices them unlike the group name counts with its own model, any other with the group name. A comment between an XObject name and Do now reads as whitespace, and the timing tests moved to CodSpeed benchmarks. * fix(benchmarks): install pypdf in the benchmark environment The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path. * fix(proxy): count PDF pages for deployments served under multi-segment model names * fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge * fix(proxy): count routing-group and alias members with the caller's team, as routing serves them	2026-10-29 18:24:04 -07:00
Alon Sokolovski	2b4cb92c8b	fix(context_management)!: run clear_tool_uses in the gateway on the Responses API path (#47861) ... * fix(context_management): run clear_tool_uses in the gateway on the Responses API path /v1/messages to openai/ models goes through the Responses API, which translated only compact_20260112 (to its native compaction) and dropped every other edit without a warning, so clear_tool_uses_20250919 never cleared anything. Edits without a Responses API form now run in the same polyfill as the chat completions path, before the call, and their applied_edits come back on the response and the final message_delta. compact stays native. * fix(context_management): check the sync polyfill result instead of casting it The strict ruff gate (TID251) bans typing.cast. * test(context_management): cover the Responses path's polyfill error handling A polyfill crash is a 500 before the call, a rejected edit under drop_params is skipped, and the sync stream reports its edits. normalize_spec never raises for a dict spec, so its guard is gone. * fix(context_management): count unreadable blocks and check tool_result text before clearing clear_tool_uses now maps blocks the token counter cannot read (server tool, compaction) to JSON text, so an input_tokens trigger no longer 500s on that history. The Responses API path checks malformed tool_result text on the original messages before clearing, as the chat completions path does. Also removes comments that repeated the code and makes the test fixtures immutable. * fix(context_management): count only blocks the model receives and keep list specs unsent clear_tool_uses now leaves blocks token_counter cannot read, like server tool results and compaction, out of its count instead of counting them as JSON text. No adapter sends them to the model, so an input_tokens trigger no longer fires on tokens the model never sees. On the Responses API path, an OpenAI-style list context_management stays unsent, as it was before, instead of reaching OpenAI's native compaction, where a threshold under 1000 is a 400. On the chat completions path, a clear edit listed before compact_20260112 now runs on the turns after the latest compaction block, as Anthropic does, instead of counting and clearing turns the compaction block drops. The crash test that replaced apply_context_management with a mock is removed. * fix(context_management): apply clear edits in OpenAI count_tokens and skip nested off-spec blocks when counting * fix(context_management): run edits on the history drop_params sends, so count_tokens and /v1/messages clear alike * fix(context_management): count clear_at_least freed tokens over sent blocks only * fix(context_management): count tools with JSON Schema shorthands and reject a non-object trigger or keep A tool whose schema has an array without items, a boolean subschema, or a search_result block without a title made the token counter raise, so a clear edit answered 500. A string trigger or keep also answered 500, even under drop_params. Both now count or get the API's 400, and drop_params drops the malformed field. * fix(token_counter): count a malformed tool schema keyword as absent and move shorthand framed-estimate cases * fix(count_tokens): pass the system prompt into clearing so count_tokens matches inference * fix(context_management): keep a malformed system prompt from failing clear_tool_uses on the Responses API A number, an object, or a list holding non-object items as system made the polyfill answer 500. The polyfill now keeps a list's object items and treats any other shape as no system, on both /v1/messages and count_tokens, so the two paths still count the same text --------- Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 17:42:11 -07:00
devin-ai-integration[bot]	a0a8ca97d9	fix(cost): finalize synchronous response cost once for header, spend log and rollups (#48993) ... * fix(cost): finalize synchronous response cost before accounting callbacks * fix(cost): keep serving and billing when cost finalization fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): read the finalized cost only off real logging objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): follow the renamed success dispatch and deferred sync callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): keep cache hits free and their headers finalized when logging is deferred Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): preserve completed synchronous charges after hook failures * fix(cost): retain completed usage across cancellation and log suppression * fix(cost): reconcile completed failures and retain cached usage metadata * fix(cost): retain finalized request tags in failure budget updates * fix(cost): honor disabled spend tracking for completed failures * fix(cost): preserve pass-through accounting across cancellation --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 11:46:49 -07:00
jesco	3057c47d6d	fix(caching): keep reasoning, citations, and audio on streamed cache hits (#32096) ... * fix(proxy): preserve reasoning fields on streaming cache hits * fix(proxy): validate cached streaming delta fields Signed-off-by: jesco-absolut <team@srswti.com> * fix(caching): replay cached audio, images, reasoning and citations in stream order A cached answer replayed as a stream now ends with its own empty finish chunk, so a chunk that carries only audio or images keeps its payload and role. Reasoning fields lead on a chunk with no text, and annotations arrive once on the last text slice, matching a live stream's order. * fix(caching): replay a cached tool call after the text, like a live stream Only the role, reasoning fields and provider fields lead a replayed answer. Tool calls now arrive on their own chunk after the last text slice, and logprobs ride the first text chunk, so a cached answer with reasoning and a tool call keeps a live stream's reasoning, text, tool call order. * fix(caching): replay the reasoning before a cached tool call that has no text A cached answer that thinks and then only calls a tool now replays its reasoning on its own chunk before the tool call, the order a live stream sends, instead of one chunk carrying both. * fix(caching): keep blank-only text when replaying a cached answer with reasoning A cached answer that thinks, sends only whitespace as text, and then calls a tool now replays that whitespace between the reasoning and the tool call, instead of dropping it. --------- Signed-off-by: jesco-absolut <team@srswti.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 10:08:12 -07:00
devin-ai-integration[bot]	c45b264024	fix(proxy): persist tag spend and drain success callbacks within the shutdown budget (#48988) ... * fix: drain shutdown callbacks and persist pending tag spend * fix: catch shutdown timeouts on Python 3.10 * fix: share shutdown callback budget with HTTP draining * fix: type shutdown cancellation tasks and document its clock * test: retain callbacks after bounded worker cancellation * test(proxy): inspect the lifespan body after the SIGTERM wrapper split Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): keep the success callback enqueue on its own task The synchronous enqueue spawned the logging worker loop inside the awaited call and ran callbacks that per-call loops used to drop, which hung the SDK callback tests on the autouse worker flush. The shutdown worker flush covers the queued callback on its own, so the one-tick task is restored Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback The aggressive clear ran the rejected callback and marked it done against the queue it was never put in, so flush() could return while a real callback was still running and the shutdown drain cancelled it early. The rejected callback is now re-enqueued once the clear made room, and only runs uncounted when the queue is still full Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging_worker): build the aggressive-clear run set immutably Final tuple instead of an appended list, with the re-enqueue in a _requeue helper, and Final annotations on the overflow regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): track overflow callbacks through shutdown * chore(logging): explain shared callback ownership * test(logging): isolate deferred callback completion --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 09:32:18 -07:00
paveln83	06cdee91da	fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails (#49035) ... * feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines Gives OpenAITextCompletionHandler a process_output_streaming_response that assembles the buffered TextCompletionResponse chunks per choice index at end of stream, runs the guardrail over the assembled text, and writes text rewrites back into the first text-carrying chunk of each choice. Tests and live proof still pending. * fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs. * test(guardrails): cover streamed /v1/completions under post_call guardrails * fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through * fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter * fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent The content filter's streaming hook held back the end of a choice until its finish_reason, which litellm sends only for the first choice of an n>1 stream, so choice 1 lost its last characters on text and chat streams. A closing chunk now releases every unfinished choice when the stream ends. A blocked text stream's closing chunk reported 0/0 usage when the upstream sent none; it now carries usage only when the upstream reported it. * fix(guardrails): send no usage on the content filter's end-of-stream chunk * fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions * fix(guardrails): stream Responses API events through the content filter as they arrive * fix(guardrails): treat a null text completion choice index as choice zero when assembling streams --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 08:46:49 -07:00
Mateo Wang	be6d082bdc	fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) ... * fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts * refactor(anthropic): hold the chat completions document change for a later commit * fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai * fix(snowflake): turn file parts into document blocks for Cortex Claude * fix(prompt_security): sanitize chat file parts that carry file_data * fix: answer 400 or drop PDFs for models that cannot read them Models whose model_info or cost map entry sets supports_pdf_input to false, and Ollama, now get the PDF dropped under drop_params and a 400 from litellm without it. GigaChat uploads file parts, SAP keeps sending them as image_url data URLs, azure_ai passes dict content through, and annotations other than url_citation move to provider_specific_fields instead of failing the response. * fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider annotations to provider_specific_fields, and split them the same way when a stream is rebuilt. * fix(vertex_ai): send PDF file parts to Vertex partner models as image_url Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL. * fix(cost-map): leave native DeepSeek PDF support unset /v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there and would only change chat completions, which this PR does not otherwise touch. * fix(main): drop a user turn that held only a PDF under drop_params Dropping the PDF from a PDF-only user turn left an empty content list, which Foundry and OpenAI-shaped providers reject with a 400, so drop_params still failed for that request. The emptied user turn is now removed. * fix: read OCI PDFs, gate deployment base models, and stop empty drops OCI generic chat sends file parts as image data URLs again, so OCI Gemini keeps reading PDFs from /v1/messages and chat file parts. The PDF gate now reads model_info.base_model, drops PDFs under drop_params for azure_ai deployments that never declared PDF support (the old flattening behavior), and answers 400 when dropping the PDF leaves no user message. The cost map marks the Vertex partner text models that a live probe showed answering without reading the PDF. * test(integration): cover the PDF input gate, messages PDF file parts and provider annotations Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed * test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells * fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload * fix(main): PDF check reads the deployment's top-level base_model * fix(main): keep roles alternating after drop_params removes a PDF-only turn When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other * test: type the PDF and annotation tests the PR added * test: type the GigaChat, SAP and Snowflake PDF tests	2026-10-29 08:06:12 -07:00
Otis Cui	c13ecb3085	fix(exceptions)!: honor status code when mapping OpenAI invalid_request_error exceptions (#32537) ... * fix: openai exception mapper ignored status code for invalid_request_error _map_openai_exception coerced anything whose body carried type invalid_request_error into BadRequestError, guarded only by excluding the single phrase Incorrect API key provided, and this ran before the status-code checks. OpenAI and openai-compatible providers return 401 and 404 bodies with type invalid_request_error, so an auth failure like Invalid API key mapped to a 400 instead of AuthenticationError, and a 404 mapped to BadRequestError instead of NotFoundError. That changes the status returned to proxy clients and router retry, cooldown, and fallback handling. Mirror the Azure mapper by only coercing to BadRequestError when status_code is None or 400, so 401 and 404 fall through to their existing handlers. * fix(exceptions): map OpenAI-family invalid_request_error 404s to NotFoundError Move the contributor's test to the mapped unit test file, assert the raised class, status code and message, and cover the 400 and status-less cases that must still raise BadRequestError Add a regression test for the flow the bug was found on. Retrieving a missing OpenAI vector store, sync or async, now raises NotFoundError with status 404 The router test that pinned an OpenAI 404 typed invalid_request_error as BadRequestError now expects NotFoundError * fix(a2a): answer a provider NotFoundError as JSON-RPC invalid params * fix(exceptions)!: limit the invalid_request_error status guard to 401 and 404 Every other status keeps raising BadRequestError as on main, so 402-415, 429 and 5xx bodies typed invalid_request_error keep their class and router retry behavior. A2A goes back to main's mapping, which answers NotFoundError and AuthenticationError as JSON-RPC internal errors. * fix(exceptions): keep the provider error fields on mapped 401 and 404 invalid_request_errors Raise AuthenticationError and NotFoundError straight from the invalid_request_error branch with the provider body, so code, param and type survive and the proxy's error.type matches main. The Rust mapper copy skips 401 and 404 the same way. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 08:05:47 -07:00
Dan Lemon	cd58014315	feat(soniox): add realtime speech-to-text on /v1/realtime (#44822) ... * feat(soniox): add realtime speech-to-text on /v1/realtime Maps OpenAI realtime transcription sessions onto the Soniox websocket API (stt-rt-v5): server VAD uses Soniox endpoint detection, commit sends a manual finalize, deployment litellm_params pass Soniox options through, translation sessions transcribe the translated tokens, idle streams get keepalives, and cost is tracked from processed audio duration. * fix(soniox): bill realtime usage from integer milliseconds * fix(soniox): bound session-start queue and cover realtime backend edge paths * fix(soniox): bill realtime stream time, start sessions on connect and surface client protocol errors Soniox charges for the whole open stream, so session close now bills the connection's open time. The backend announces the session on connect, which flushes audio sent before any session.update, and a single upstream pump makes recv safe to cancel. Invalid client frames now get an OpenAI error event instead of a silent disconnect * fix(soniox): skip deferred setup so client frames reach Soniox in order Drops the connect marker frame, which counted as an upstream frame and hid refusals, and could start a buffered stream at the default rate before the client's session.update. Soniox now opts out of deferred setup, the handler announces a synthetic session.created whenever no setup went upstream, a session.update that would change a started stream gets an error event, and stream time is billed only once a stream started * fix(soniox): judge a follow-up session.update only on the settings it sends A partial follow-up session.update was rebuilt with defaults for the settings it left out, so restating part of a 16 kHz or VAD-off stream was rejected as a change * fix(soniox): treat an omitted rate in a follow-up session.update as unchanged * chore(soniox): drop comments that restate the code * fix(soniox): return speaker labels, log refused sessions as failures, and probe realtime health Deltas carry speaker and language when diarization or language identification is on, and the completed event lists segments per speaker and language. A session Soniox refuses before any audio is processed is logged as a failure with no spend. Billing starts at the start request. Deferred-setup providers (Meta, Vertex Chirp) send the error event and still end the session. Soniox realtime models get a websocket /health probe and cost rows for the stt-rt-v5 aliases. * fix(soniox): bill failed streams, probe health with deployment options, and keep token accumulation linear Only request rejections (400, 401, 402, 403, 429) before any processed audio count as a refused session, so a 408 or 5xx still bills the open stream. The /health probe starts Soniox with the deployment's options, sends 100 ms of silence, and passes only on Soniox's finished frame. A turn's final tokens append in constant time instead of copying the tuple per token. * fix(soniox): log every Soniox error before any processed audio as a failed session * fix(soniox): keep untranslated speech, support input_audio_buffer.clear, ignore an empty language * fix(soniox): reject frames without a type, keep strict language hints, queue finalizes in a deque * fix(soniox): answer a commit with no speech with an empty completed turn * fix(soniox): let a server VAD turn answer a commit that is still in flight * fix(soniox): let a server VAD turn answer only the oldest pending commit --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 07:58:38 -07:00
mika-lt	c21e576a07	refactor(types): replace Any with proven types in 14 files (#48928) ... * refactor(types): replace Any with proven types in 57 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): hide input values in validation errors and drop secret-bearing seams Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert the Phoenix prompt seam that rejects string content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert files that break mocked tests and Python 3.10 imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert pegasus response validation to main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert azure search, milvus and onyx response validation to main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep accepting numeric keys in /cache/delete * fix(caching): answer 400 for a non-list /cache/delete keys and accept null Stability lists Return pydantic errors without the caller's input on a bad /cache/delete body, let the Bedrock Stability image edit schema take null finish_reasons and images like the old reads did, declare user_id and user_role on the plugin session claim, wrap the retyped Vertex fields in ReadOnly, and drop unused Guardrails AI types * refactor(types): leave provider reply parsers for a follow-up with live provider calls Restores the Bedrock Stability, Bedrock KB, AgentCore, Replicate, Recraft, Aleph Alpha and Guardrails AI parsers to main, since no live provider call could prove the stricter schemas. Also types the cache delete route tests. * refactor(types): keep /cache/delete on main's behavior Restores the cache delete route and its test to main, so this types PR changes no response. Rejecting a non-list keys is left for its own change, since an object keys deletes its keys on main today. * fix(types): keep text completion, batch ids and guardrail callback on main's code * fix(types): keep policy suggestion and auto router parsing on main's code --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 07:26:07 -07:00
devin-ai-integration[bot]	444423a5ca	fix(cost): bill a deployment's per-second rates on native Anthropic /v1/messages (#42417) ... * fix(cost): honor per-second custom pricing on chat completions for every provider * test(cost): assert a per-second priced deployment bills instead of staying at $0 The zero-cost diagnostic tests from #42345 used a per-second-only entry as their misconfigured fixture, which this branch now bills. Switch that fixture to a per-query-only entry, which is still selected as the deployment's own pricing and still prices chat usage at $0, and add a per-second test asserting the call duration is billed with no diagnostic Also let a caller's explicit total_time outrank the logging window in completion_cost, so the SDK precedence stays stamped response, caller, logging * test(response_metadata): move the per-second pricing regression into the mapped tests/unit file * fix(cost): bill a deployment's per-second rates on /v1/responses and /v1/messages The proxy builds the request's logging state before the router picks a deployment, and only the chat path re-read the deployment's input_cost_per_second / output_cost_per_second afterwards. The shared update step every other route runs (Logging.update_from_kwargs) now seeds those custom pricing keys from the router-merged kwargs too, so custom_pricing flips on and the cost calculator prices the call against the router-registered deployment id on every surface. * fix(cost): price a /v1/messages call off the response time the proxy reports A /v1/messages result is a plain dict, so the proxy keeps its response time on the logging object. The cost step now reads it there for a non-streamed call, so the cost header and the spend log bill the same duration. * fix(anthropic): bill a deployment's per-second rate only on the native /v1/messages route Move the custom pricing copy from the shared logging refresh into the native Messages handler, reusing the helper the Responses handler already had, so generateContent and other routes keep their current billing * fix(cost): pass only per-second rates to native /v1/messages cost tracking Other deployment overrides keep billing as on main, matching chat's per-second gate * test(anthropic): pin per-second-only pricing on native /v1/messages with a long-context-rate deployment * fix(cost): bill native /v1/messages from logging setup, as main did, to the end the proxy reports --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 05:29:03 -07:00
tavren27	e7d3b24d7c	fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs (#48888) ... * fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs and logging payloads * fix(router): keep the mirror flag when litellm_metadata is set and strip a client-sent is_silent_experiment Spend rows read router metadata from litellm_metadata when a call carries it, so the router now sets the flag there too. The proxy drops a client-sent is_silent_experiment with the other router-owned keys, so real traffic cannot be recorded as mirrored. * fix(logging): mark mirrored calls with a router-only key so clients cannot set is_silent_experiment Spend rows and the standard logging payload now derive is_silent_experiment from _litellm_silent_experiment, which only the router writes and the proxy and pass-through routes strip from client metadata. A caller's own is_silent_experiment key stays untouched, so it keeps its current effect on mirroring and on provider metadata, and unmirrored calls log false in both places. * test(langfuse): expect is_silent_experiment false on every logged generation * fix(logging): mark mirrored calls with a router-only enum value so no client route can set is_silent_experiment The router stamps SilentExperimentMarker.MIRRORED, which client JSON can never produce, so batches, Responses WebSocket frames and pass-through need no strip. Langfuse layers the flag after its requester-metadata dedupe. * fix(logging): set is_silent_experiment only on mirrored calls so real calls keep their payloads Real calls no longer gain an is_silent_experiment key in spend rows or the standard logging payload, so OTEL, S3, GCS and every other logger receive what they did before. The Langfuse regression test now runs the real prompt helper. * fix(logging): keep the mirror marker out of callback metadata and read the spend-row flag from the logging payload Logging now strips the router's mirror marker from the request metadata buckets before any callback or vendor sees it, without touching the router's own dicts, so retries keep it. Spend rows read is_silent_experiment from the standard logging payload, which survives the collector's JSON round trip, so collector-mode rows keep the flag too. * fix(logging): mark mirrored calls through a context variable so router usage counts them once * fix(router): leave callback-started calls unmarked and stop client flags from turning mirroring off Logging handlers now run outside the mirrored-call context, so an LLM call a success or failure callback starts while logging a mirrored call is not flagged as mirrored. The guard against mirroring a mirror reads the same context variable instead of request metadata, so a client-sent is_silent_experiment no longer skips the mirrors. The unused module-level get_standard_logging_metadata stops copying resolved keys from raw metadata. * fix(router): keep main's client flag mirroring skip and the other logging metadata keys A client's is_silent_experiment: true skips mirroring again as on main, the module-level get_standard_logging_metadata drops only the flag, and the silent experiment tests wait on a logged-call signal instead of polling a deadline * fix(logging): take the mirror flag in Prometheus labels and Datadog cost tags only from the router * test: type the new parametrized tests and keep the recorder from changing its argument * fix(datadog): keep client request tags from setting the silent experiment cost tag A request tag is_silent_experiment:true set the Datadog cost tag on a direct call. Only the router flag sets it now. Also covers the sync failure handler's mirrored-call reset with a failure function callback test. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 05:12:50 -07:00
Priyansh Nandwana	67ae3fb527	fix(websearch)!: keep the provider prefix on the agentic follow-up model (#38885) ... * fix(websearch): keep the provider prefix on the agentic follow-up model The agentic follow-up rebuilt its model string with "a slash means it is already qualified". Handlers are handed the provider-stripped model, and for a provider that routes through a sub-path the remainder still holds a slash, so bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5 and stayed that way. The follow-up acompletion could not resolve a provider from it and raised "LLM Provider NOT provided". Prefix unless the model is already qualified with that same provider, in one shared helper used by both the handler and the websearch patch builder, which carried copies of the same heuristic. This also fixes openrouter, whose stripped model keeps a slash for the same reason, and the old startswith test that read openai as a prefix of openai_like. Fixes #38829 * test(websearch): cover the agentic follow-up call site The helper had unit tests but the call site did not, so the two changed lines in _execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with mock_response instead of patching litellm, which keeps the test on the behaviour: against the old code the sub-path case raises "LLM Provider NOT provided", and the ordinary-model case passes either way as a guard. * refactor(websearch): trim the comments on the provider-qualify helper * refactor(websearch): cut the helper and test docstrings to one line * fix(websearch): leave a hook-qualified follow-up model on its own provider A hook that patches in its own model owns that choice, so re-qualifying it sent a cross-provider follow-up to the original request's provider instead. Only a bare patched model and the request's provider-stripped model take the prefix now, and the sibling agentic-loop path resolves the model the same way. * fix(websearch): fall back to the request model when a patch carries no model An empty patched model dispatched a provider-only name instead of falling back, and a provider-stripped sub-path copied into the patch lost its prefix. A patched model now keeps its own provider only when its leading segment names one. * fix(websearch): keep the provider prefix on the responses follow-up model too The responses patch builder still used the old "a slash means already qualified" test, so a provider-stripped sub-path model reached the follow-up unqualified on that surface. It now resolves the model the same way the chat completion builder does. * fix(websearch): read custom_llm_provider without a member call on untyped kwargs * fix(websearch): type the chat patch builder kwargs instead of working around them * test(websearch): run the real search path and flatten the qualify helper tests * fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model * fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider * fix(websearch): send the agentic follow-up to the model the caller requested completion() and responses() record the caller's model before routing strips it, and the chat and Responses follow-up builders prefer that model. This keeps bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/* deployments on their own route for the follow-up. * test(websearch): cover the recorded caller model in Headroom's retrieval follow-up * test(websearch): cover the recorded caller model in Compresr's retrieval follow-up Also merges the repeated provider-prefix check in core_helpers into one helper and drops a test docstring * test(websearch): drive the real search path through a mocked Perplexity HTTP route * refactor(websearch): read the provider from kwargs inside _caller_model * test(http-handler): type the agentic follow-up provider tests * refactor(websearch): record agentic loop params through a Logging method Moving the recording out of completion() keeps it under basedpyright's code-flow complexity limit after the merge with main. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 04:47:18 -07:00
quietjules	d037d86a94	fix(token_counter)!: keep a loaded Hugging Face tokenizer when a slower download fails (#48916) ... * fix(utils): keep a loaded Hugging Face tokenizer over a racing failure and retry once per window A download that fails after another count loaded the same tokenizer no longer hides it, the first count after the retry window claims the retry while the others keep using tiktoken, and HUGGINGFACE_TOKENIZER_LOAD_RETRY_SECONDS defaults to 60s to match the proxy's tokenizer reuse window * fix(utils): keep Hugging Face tokenizer outcomes until their retry time instead of a TTL The outcome store no longer expires entries after twice the retry window, so after a long idle stretch the first count still claims the one retry and the other counts keep using tiktoken instead of each downloading again. The store is a small lock-guarded holder of an immutable mapping keyed by tokenizer kind and backend * fix(token_counter): keep the loaded tokenizer in its outcome and run one retry at a time The loaded outcome now holds the tokenizer, so an LRU eviction can no longer send every count back to an unguarded download. A retry blocks further retries until it ends, even when it runs past the window, and a download that fails after another count loaded the tokenizer counts with that tokenizer * fix(token_counter): retry a failed Hugging Face tokenizer download in the background After the retry window, the count that finds the stored failure starts one background download and answers with tiktoken, so no live count waits on a retry. A failure from a download that started before the first failure no longer resets a running retry, and a failed retry never replaces a tokenizer another download loaded. The outcome store takes its clock and retry starter as dependencies, so the tests inject them instead of patching time.monotonic. * fix(token_counter): keep retrying a failed Hugging Face tokenizer download when its retry thread cannot start If starting the background retry thread raises, the count answers with tiktoken and the failure is stored with a fresh retry time, so the next window retries again instead of the tokenizer staying on tiktoken until restart. * fix(token_counter): let a forked process or a crashed retry thread retry a failed Hugging Face tokenizer download again A failed download's in-flight retry mark is now the pid of the process running it, so a process forked during a retry starts its own instead of waiting on a thread it does not have. The retry thread clears its mark in a finally, so a native panic or fork guard error escaping it leaves the next window free to retry. The fork guard error is no longer swallowed in the retry thread and reaches the thread's exception hook, as on the first load * fix(token_counter): let a process that exits during a background tokenizer retry exit cleanly instead of aborting --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:50:58 -07:00
devin-ai-integration[bot]	b93d26bc85	fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path (#45330) ... * fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path * refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge * fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper * fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints Also drop a marker set on the message item itself, sanitize before the cache_control conversion so a client typo no longer erases that breakpoint, and skip items that carry no marker so unmarked history costs no extra walk * fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a litellm_proxy alias) kept its valid markers before this PR and must keep them under drop_params. base_model is read from litellm_params first, then model_info, as chat completions does. * fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged * fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers * fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied * fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one Under drop_params the marker step now hands back the caller's own objects when every marker is already wire-safe, so a request with valid markers is no longer copied and walked a second time. A deployment that turns cache_control into markers keeps the caller's own markers too, whatever its base_model says, so one request never gets two answers about breakpoint support. On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion, so the block gets the cache hint cache_control alone would give, as on /v1/responses. * test(responses): keep a null marker and the original key order on a model that reads breakpoints --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:37:54 -07:00
zoveki	310c96162b	fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) ... * fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:36:20 -07:00
paveln83	032342d2b7	fix(responses): resume background response streams with stream and starting_after (#48819) ... * fix(responses): resume background response streams with stream and starting_after Retrieving a response with stream=true and starting_after=N now returns the replayed event stream from OpenAI and Azure instead of one JSON object, on the SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id} Providers that cannot replay a stored stream drop the params under drop_params and raise UnsupportedParamsError without it. Malformed values answer 400 with the param named. A replay that reaches its terminal event settles the background response's bill once and passes the provider's usage through without estimating one. An empty replay (cursor at the end) ends normally * fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams A replayed background stream now gets the same response id encryption as a created one, so the key that made the response can reuse the id. Azure AI Foundry deployments resume streams like Azure ones, and the chat-bridged Responses iterator closes its wrapped stream instead of raising. * fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query. * fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params * fix(responses): keep deployment stream off retrieves and reject oversized cursors * fix(responses): keep router default stream off plain retrieves * test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing * test(responses): replay a real stream in the billing test and type the fake wrapper chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 02:31:22 -07:00
simonp9	dff474e450	fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912) ... * fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn * fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards * fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set * fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit * fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum * fix(responses): leave a turn without a phase when its content message cannot be read * test(responses): add black box tests for chat turn history replay * test(responses): expect the replay record on bridged chat replies * test(responses): expect the replay record in bridged chat translation cases * fix(streaming): keep the served service tier on the finish chunk the stream adds * test(responses): read the expected chat message without JSON validation so the replay matcher stays --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:56:02 -07:00
devin-ai-integration[bot]	3c1eb73a06	fix(proxy)!: list every priced Vertex family under a vertex_ai/* grant (#41959) ... * fix(proxy): list every priced Vertex family under a vertex_ai/* grant A vertex_ai/* key listed none of the 68 cost map rows priced under the Mistral, Llama, Qwen, AI21, OpenAI, image, video and embedding labels because their sets were never part of the vertex_ai union, and Qwen rows had no ingestion branch at all. Wildcard expansion also dropped a meta/ or openai/ publisher segment as if it were a provider prefix, and the wildcard health check would have picked $0-per-token rows first once the union grew, so it now sorts paid rows ahead of unpriced ones. The VERTEX_FAMILIES_A_VERTEX_WILDCARD_GRANT_DOES_NOT_LIST allowlist is gone so the cost map guard enforces every label from now on. * fix(health): route every wildcard health-check pick through the listed provider * fix(health-check): keep per-character-billed rows ahead of rows priced $0 everywhere in the wildcard pick * fix(health-check): break $0-per-token ties toward per-character-billed rows in the wildcard pick * fix(health): only health-check ids the router sends to the provider The wildcard health check picker now skips ids that get_llm_provider routes to a different provider or to none, so the longer vertex_ai union cannot pull partner rows into the check. Drops the forced custom_llm_provider and the per-character tie-break from earlier revisions, and pins the replicate and wandb wildcard listing to each row's full owner id. * fix(health): match wildcard health picks against the provider's own route The previous check kept a bare id only when get_llm_provider routed it to the exact provider name, which dropped every ai21 row because bare ai21 ids route to the ai21_chat alias, so ai21/* health checks sent no request. Compare the bare id's route with the route of the provider-prefixed id instead, so aliases match. Resolve routes with inferred_provider, which returns the declared provider for github_copilot and chatgpt without starting their device-code login. * fix(health): keep wildcard health picks that only route through a saved custom_llm_provider Rank picks routed to the checked provider first and ids routed to no provider second, each by cost, and skip only ids routed to a different provider, so a wildcard saved with custom_llm_provider still sends its health request * refactor(vertex_ai): mark the new Vertex registry sets Final * chore: re-run CI against current main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:31:04 -07:00
rj-cairn	5eb1042bbf	fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728) ... * fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): move the o200k provenance note into the assert message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust_bridge): azure/gpt-4o now resolves to o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know * fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer * fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model. * fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model * fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model * fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted. * test: type the new token-counting tests' parameters * fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it * fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test * perf(token-counter): classify each distinct served model once when walking a model group Budgeted requests walk every deployment in their model group, since the router's cached copy is outside the router surface the proxy may use. Classifying each distinct served model once instead of once per deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms with the same result. --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:08:12 -07:00
devin-ai-integration[bot]	a66dc50efb	test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio (#43370) ... * test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio The drain regression guard compared the wall-clock time of draining 80,000 values against 40,000 and tripped its 3x bound whenever the CI runner was busy. A cursor drain allocates only per-value decode scratch while every re-slice of the remainder copies it, so the guard now bounds peak allocation during the drain, which does not move with runner load and still fails on every quadratic variant * test(json_fragment_accumulator): widen the drain's peak bound to a quarter of the payload * test(json_fragment_accumulator): guard copying drains by peak memory and rescanning drains by CPU-time scaling * test(json_fragment_accumulator): move the drain's timing guard from the unit suite into a CodSpeed benchmark * test(json_fragment_accumulator): measure only the pops after the buffer is joined, so a drain that copies once passes * test(json_fragment_accumulator): separate every other streamed value with a space, so a copy after whitespace fails the drain test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 22:04:47 -07:00
Mateo Wang	f46adf80e4	fix(proxy): let typesafe/* allowlists cover /typesafe Jev calls and list Jev in the model picker (#47697) ... * fix(proxy): list TypeSafe Jev models in the admin model picker * fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones. * fix(typesafe): match only published Jev ids and keep picker loads off the secret manager Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file. * test(typesafe): type the picker test fixture and tests * fix(typesafe): match only published Jev versions and cover non-version suffixes in tests * fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker * fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers The dashboard sends include_model_access_groups on every /models call, so gate the TypeSafe names on wildcard routes too and skip access-group-only listings. Also say the typesafe-jev-ids rule matches case-insensitively. * fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main A typesafe/* router deployment registers typesafe/* in the cost map, so the allowlist pickers offered a bare *, which allows every model. The pickers now list only priced names the rule sends to TypeSafe. The non-expand list also stops de-duplicating every entry and only skips TypeSafe names already listed.	2026-10-28 20:19:58 -07:00
Kerry Lu	4fc2bba1f2	fix(stream): the last usage frame carries litellm's bill in usage.cost (#48955) ... * fix(proxy): copy the stream's hidden response_cost into the client's streamed usage.cost The stream is priced once into hidden response_cost (stream builder for chat and text completions, the Responses iterator for the terminal event) and the proxy egress copies that bill into the streamed usage.cost on a copy of the event. Inside the pipeline usage.cost stays what the upstream reported. Spend rows do not change; the billing fix for upstream-reported usage.cost follows in the next PR cut from #48287. * fix(proxy): type the bridge bill carry and the builder's reported-cost guard for the lint gates * style: format the chat-over-Responses bridge helper signature * refactor(bridge): build the billed Responses event with model_copy instead of a TypeAdapter chain * fix(stream): keep writing the computed cost into usage.cost as on main, the hidden copy is only for the proxy egress * fix(stream): write litellm's bill into the usage frame's usage.cost instead of a hidden field the proxy copies * test(stream): keep main's formatting in test_streaming_handler.py, only the new test changes * refactor(stream): drop the docstring on _stamp_usage_cost, the body says it --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 19:32:44 -07:00
Nate Armstrong	7650f766b0	refactor(router): fence the router surface and document the router-core boundary (#48839) ... Add rustdoc overviews for a new litellm-router-core crate and for litellm-model-catalog, a module docstring on litellm/router.py, and a shrink-only allowlist of the Router members and router_utils modules used outside the router, checked in CI by router_surface_check.py. Move the router-free helpers out of litellm/router_utils: batch_utils to litellm_core_utils, and the auto-router naming contract, tuning baseline and baseline request next to the auto-router plugin. No behaviour change.	2026-10-29 00:01:52 +00:00
rj-cairn	d06b889d41	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) ... * fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>	2026-10-28 17:00:29 -07:00
Kerry Lu	e21de0e418	fix(logging): bill each SDK call its own native Bedrock guardrail trace when calls share one metadata dict (LIT-10239) (#48923) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 15:11:26 -07:00
moe-berri	cf7c35a117	fix(anthropic): map model_context_window_exceeded stop reason to length (#48874) ... * test(anthropic): cover model_context_window_exceeded finish reason on chat and responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): map model_context_window_exceeded stop reason to length Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rust): mirror model_context_window_exceeded finish reason mapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 13:48:37 -07:00
Mateo Wang	dfd9ad4103	fix(responses)!: bill partial usage on mid-stream failures and report bridged /v1/messages failures to the proxy (#47537) ... A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them	2026-10-28 13:32:09 -07:00
Simon	aa769fbc69	fix(streaming): surface upstream abort as server errors (LIT-10159) (#48328) ... * fix(streaming): surface upstream abort finish reasons as errors * fix(streaming): keep SDK chunks with null choices streaming past the abort check * fix(streaming): treat null choices as empty in the raw SSE abort check * refactor(streaming): bind raw SSE choices as Final instead of rebinding --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 13:18:05 -07:00
yujonglee	1f68cc5649	refactor(rust-bridge): require the native extension (#48852) ... * refactor(rust-bridge): require the native extension _native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): build Rust bridge in Windows smoke tests --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 13:16:56 -07:00
tin-berri	ca6c830221	fix(proxy): account for media in combined TPM reservations (#48865)	2026-10-28 12:39:41 -07:00
moe-berri	365d3d3729	feat(decisions): support Ollama decision models (#48871) ... * test: cover local Ollama Decisions wire contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): support ollama providers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(decisions): account for Ollama model-info requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 12:35:26 -07:00
Onur C. Cakmak	d9aae590e3	fix(chatgpt): derive the upstream session id from prompt_cache_key (#42014) ... * fix(chatgpt): derive Responses session id from prompt_cache_key ChatGPT partitions the Codex prompt cache by the Responses session-id header (the Codex CLI sends a stable per-conversation key derived from prompt_cache_key). The provider only consulted explicit session ids and otherwise stamped a per-request uuid4, so no turn ever reused the prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude quota/cost multiplier on subscription accounts. Resolution order after this change: explicit session config (session_id / litellm_session_id / metadata.session_id) > prompt_cache_key (sanitized with _safe_header_value) > litellm-internal ids > uuid4. Two guards, mirroring get_fireworks_session_id: - proxy-generated session ids (missing_session_id: "generate") are per-request and are skipped, checked in both "metadata" and "litellm_metadata" (LITELLM_METADATA_ROUTES, responses included) - internal per-request ids (litellm_trace_id / litellm_call_id) lose to a stable conversation key Supersedes the body-key forwarding of #37630: a random session header overrides the body key on the provider side, so the header must carry the caller's key. Related: #29978, #29993. Verified against a real ChatGPT Codex account: stable session id -> 97-100% cached on repeat turns; random per-request -> 0%. * style: ruff format * fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments - hash prompt_cache_keys containing header-unsafe chars instead of mapping them through _safe_header_value, whose replacement char could collapse distinct keys onto one cache shard and serialize unrelated requests (greptile review) - strip the explanatory comment blocks per AGENTS.md comment policy - move the tests into the existing mapped responses transformation test file per AGENTS.md test-placement rule * fix(chatgpt): keep session typing and test diff scoped * fix(chatgpt): hash surrogate cache keys, type session params, flatten tests * test(chatgpt): record upstream requests with a mock instead of appending to a list * fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys * fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid * test(chatgpt): type the session id test parameters * fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key * fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence * fix(chatgpt): keep the trace-id session marker out of provider params The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 10:57:51 -07:00
devin-ai-integration[bot]	3afe779125	fix(cost-map): rolling provider pricing, capability and retirement audit (#48188) ... * fix(cost-map): registry audit 2026-10-24, add provider retirement dates and fix mistral-large-4 and azure_ai sonnet 5.5 prices Co-authored-by: Hannes Brantner <desktop.stilted-9t@icloud.com> Co-authored-by: Aasif-Simpplr <aasif.multani@simpplr.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mistral): derive mistral-large-4 cost expectations from the registry row instead of pinned prices Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): add fireworks spot routers and reprice together Qwen3.7-Max Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): absorb open registry PRs and update azure_ai mistral-small-2503 retirement date Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Aryan <aryan.m@kelpglobal.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(cost-map): credit absorbed registry PR authors Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Aryan <aryan.m@kelpglobal.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): add azure gpt-6.1-terra rows and fireworks models/rune path Co-authored-by: ceng-p <christianpetersen98@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): add azure_ai gpt-6.1-terra global standard row Co-authored-by: Ahmet Selim FEDAKAR <122402435+selimfedakar@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): price mistral-large-4 cache reads at the published $0.14 Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(cost-map): leave mistral-large-4 rows to #48825 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): refresh provider pricing and retirement metadata Absorb the verified GPT-5.6 Terra and Luna price fields from #48709. Add official Groq caching, xAI batch pricing, Moonshot metadata, and retirement dates without replacing existing model entries. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Hannes Brantner <desktop.stilted-9t@icloud.com> Co-authored-by: Aasif-Simpplr <aasif.multani@simpplr.com> Co-authored-by: Aryan <aryan.m@kelpglobal.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: ceng-p <christianpetersen98@gmail.com> Co-authored-by: Ahmet Selim FEDAKAR <122402435+selimfedakar@users.noreply.github.com> Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com>	2026-10-28 16:07:22 +00:00
norkelbyte	940b865424	refactor(fallback-generalizations)!: remove the legacy extends and mixed-rule shim (#48737) ... * refactor(fallback-generalizations): remove the legacy extends and mixed-rule shim Main's remote cost map, the bundled backup, and the root map all ship the split routing/capability rule format, so the shim for the old format is dead code. A rule mixing litellm_provider with other model_info keys is now warned about and skipped. * test(fallback-generalizations): fail when a shipped cost map rule would be skipped at install * test(fallback-generalizations): cover ignored extends and annotate the shipped-rules test * fix(fallback-generalizations): warn about old-format rules with a pointer to a current cost map and skip extends rules * fix(fallback-generalizations): tell local cost map users to replace the bundled backup in the old-format warning --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 05:17:28 -07:00
Kerry Lu	7acdf9f00d	fix(anthropic): count search results and reject unsupported token-count blocks (#48699) ... * fix(anthropic): count search results and reject unsupported token-count blocks Co-authored-by: Prasanna Jain <prassu017@gmail.com> * fix(anthropic): avoid a Final assignment inside the token-count loop Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Prasanna Jain <prassu017@gmail.com>	2026-10-28 02:33:38 +00:00
tin-berri	b9e4298506	fix: preserve media across Responses and Gemini conversions (#48484)	2026-10-27 18:07:10 -07:00