* fix(proxy): persist sent model_info fields on POST /model/update
POST /model/update rebuilt litellm_params from the request but dropped the
request's model_info, so changing base_model (or mode, tier, etc.) on an
existing DB model was silently ignored. Clients that only use this endpoint,
like the Terraform provider, could never fix a deployment's base_model
without deleting and recreating it.
Persist the model_info fields the caller actually sent, on top of the stored
ones. ModelInfo defaults never overwrite stored values, server-owned, team,
PTU and pricing fields stay on their dedicated write paths, and values echoed
from a /model/info response are not pinned.
* fix(proxy): keep echoed and blank model_info values out of POST /model/update writes
Blank strings from typed clients no longer overwrite stored values. Values echoed from
/model/info (the stored base model's cost map entry and discovered limits) are not
pinned, and an echoed catalog value resets a differing stored override like PATCH does.
direct_access and access_via_team_ids stay display-only. The tests stop patching the
auth check class and give the team lookup a real row instead.
* fix(proxy): lock the model row and skip unchanged model_info on POST /model/update
* fix(proxy): treat only keyed /model/update bodies as echoes and write the stored model_info as saved
A typed model_info value that matched the cost map, or the previous base model's entry,
was dropped and could erase a stored override. Echo detection now follows PATCH, only a
body carrying the /model/info key counts, compared against the base model the row has
after the write. The locked write patches the stored JSON instead of dumping the hydrated
Deployment, so litellm_params pricing is no longer copied into model_info. Tests run the
real Router with discovery over a mock upstream
* docs(terraform): note that model_info updates now take effect and overwrite values set outside Terraform
* test: expect POST /model/update to save the cache point flag, and wrap long test lines
test_x06 pinned the old contract, where POST /model/update answered 200 and ignored model_info.
This PR makes POST save model_info the way PATCH does, so x06 now asserts what its PATCH sibling
x01 asserts, the stored flag flips to true and cache points come back on.
* fix(proxy): answer 400 when POST /model/update gets an access_groups that is not a list of strings
* fix(proxy): read discovered model info through the router registry entry point
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): route unmapped converse-only family model ids to converse
#42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from
falling to the invoke route. The same gap exists for the other families that
have no invoke transformation at all: minimax, nvidia, writer, zai, google and
xai. An id from one of them that is newer than the cost map hits
get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None ->
BedrockError, when Converse would have served it
Add a second family regex for those six and default them to converse, the same
way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because
the prompt-caching rule is openai-specific
Every shipped id of these families already resolves to a converse entry via its
base model (checked against the cost map), so this only changes behaviour for
ids that are not in the map yet
Tests: six unmapped ids route to converse, and an unmapped meta.* id still
routes to invoke since meta has an invoke handler
* fix(bedrock): route a model id with no Invoke provider to Converse
A bedrock/<model> id missing from the cost map fell to the Invoke route,
where an id that names no Invoke provider (a new vendor's model, a bare
provisioned-model, imported-model, or custom-model-deployment ARN) fails
locally with "Unknown provider=None" before any request is sent. Such ids
now go to Converse, which replaces the per-family regex with the general
rule. Ids that name an Invoke provider and explicit route prefixes are
unchanged.
* test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests
* fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id
* fix(bedrock): answer 404 for an invalid model id instead of AWS's 400
AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown
* fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list
* fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about
A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there
* fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows
The regex check now strips route and region prefixes from the model and its
base_model before asking whether LiteLLM knows the model, so
bedrock/converse/<id> gets the same answer in a direct transform call as on the
completion path. A cost-map row counts only when it names a provider, so a
price-only register_model row leaves an unlisted id unknown, as on the proxy.
AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping
an identical copy.
* fix(bedrock): treat a region path like the bare id when the cost map does not list it
A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools,
reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works
on it and a mistyped one still answers 404.
* fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local
* fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local
* fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider
An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse.
atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do.
* fix(text completion): keep /v1/completions error mapping as it is on main
Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep a deployment's own model_id over the request body's
A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one.
* fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat
Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win
over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists
model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a
client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component
* fix(router): keep a SageMaker deployment's component when it pins it through extra_headers
* fix(router): read the SageMaker component header as a pin only on SageMaker deployments
* fix(sagemaker): send the inference component only as a header, never as a body model_id
SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat
container body and legacy parameters no longer carry model_id. Nova endpoints accept a
fixed body field list, so a pinned Nova deployment could reject the extra field.
* fix(health): probe header-pinned SageMaker deployments at their component
Health checks bypass the router, so a legacy sagemaker/ deployment pinned only
through its inference component header sent no component on /health while routed
requests now send it. Also update the two SageMaker integration tests that still
asserted the old precedence and the dropped body field.
* fix(health): apply the router's model_id pin rule to health probes
Health probes now share the router's pin function, so a deployment that lets
clients set model_id gets no header pin and its own model_id wins without
reading the header. Header keys that are not strings are skipped when
reading the inference component header instead of failing the call.
* fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks
The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply
Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its
default audio setup at connect, so a client's session.update (GA output_modalities or
beta modalities) was dropped and every answer came back as audio. The setup now goes
out on the client's first message: a session.update becomes that setup, anything else
gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a
provider hook. Text setups label deltas response.output_text.delta, and downgrading
TEXT to AUDIO on an audio-only model logs a warning
* fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup
* fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup
Gemini and Vertex Live accept one setup per connection, so the bridge used to
turn only the first session.update into that setup and drop the rest without
telling the client. Now every session.update that arrives before the first
content frame is held, deep-merged (later values win per field), acknowledged
with a session.updated built from the setup it would produce, and the merged
session becomes the one setup sent with the first conversation item, audio
frame, or tool result. The session events also surface the systemInstruction
text as instructions, which the old string check never matched because the
setup carries it as a Content object.
* fix(realtime): cap the session.update merge depth so the recursive detector accepts it
_merge_session_fields recurses once per nested mapping level, so CI's
recursive_detector flagged it. Bound the descent at eight levels (the OpenAI
session object nests four deep at most), replace anything deeper wholesale
with the later session.update's value, and list the helper with the other
depth-capped functions the detector allows.
* fix(gemini-live): accept a string output modality and drop values Live cannot produce
A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once.
* fix(gemini realtime): answer an error for session updates Gemini Live cannot apply
An output modality the model cannot produce, or a session.update sent after the
first content, now gets an invalid_request_error event instead of a silent swap
or drop, unless drop_params is set. The synthetic session.created reports the
model's default modality, setupComplete no longer adds an extra session.updated,
and merging nested session updates is linear in their size
* fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns
A pre_call guardrail blocking the first typed message now goes out after the
setup, so Gemini and Vertex no longer close the session with 1007.
Push-to-talk sessions (turn_detection null or create_response false) wrap each
audio turn in empty activityStart and activityEnd messages, which Gemini needs
when automatic activity detection is off.
A session with no content after 60 seconds sends its setup, since Vertex
refuses a setup 300 seconds after connect. Held session.update frames past
10 MB answer session_update_too_large, and deferred mode now errors on a
second session.update unless drop_params is set.
* fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection
An update that leaves the applied setup unchanged now gets session.updated
instead of session_update_not_applied, so clients that resend their session
with the first response keep working. Deferred setup mode merges held updates
like the default mode, turn_detection null turns Gemini's activity detection
off, drop_params drops a voice outside the documented Live voices, and the
lazy setup deadline moves to 240 seconds
* fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content
* fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices
A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name.
* fix(realtime): hold first content so a following session.update joins the Gemini setup
Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too
* fix(realtime): check AI Studio Live voices like Vertex
* fix(realtime): forward any voice to AI Studio Live models that accept it
* fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config
The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup.
* fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock
The deadline and grace tests waited on real 50 ms timers, and the merge cost
check compared wall-clock runs. The session now reads an injected clock,
time.monotonic by default, the two tests step it, and the merge cost check
moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes
the backend through wrap_backend, which main added after the test was written
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126)
The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests.
* test(proxy): pin provider metadata and key billing on /cursor/chat/completions
A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land
* fix(proxy): pick the Cursor route's metadata field from the body shape
A Responses-style body on /cursor/chat/completions keeps proxy metadata in
litellm_metadata, the same as /v1/responses, while a Chat Completions body
keeps metadata, the same as /v1/chat/completions. Adding the whole route to
the internal-metadata list let a client's chat-body tags replace the key's
tags in spend logs and skip tag budgets.
* fix(proxy): keep get_metadata_variable_name callable with only the request
* fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies
A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field
* fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies
* fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string
* fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure_ai): keep OpenAI prompt cache breakpoints on the Foundry chat completions path
* fix(azure_ai): keep only OpenAI content part types when preserving prompt cache breakpoints
* fix(azure_ai): share the breakpoint check with the cache-mark step and keep only text parts
A deployment's model_info opt-in is registered under its provider-prefixed cost-map key without a litellm_provider, so that prefix now names the provider the flag speaks for.
* refactor(azure_ai): read text parts through a walrus and mark new test values Final
* fix(azure_ai): read a router-registered breakpoint opt-in only for azure_ai and keep a message mark off dropped file parts
* fix(azure_ai): read the breakpoint opt-in from each deployment's own model_info and base_model
A router deployment without its own supports_prompt_cache_breakpoint flag now
reads the cost map as loaded, so a sibling's opt-in on the same model string no
longer turns its requests into 400s. The shared hook no longer reads router
registered keys for azure_ai.
* fix(azure_ai): send breakpoints in Foundry's shape under drop_params and fall back to the model name
* test(integration): expect the injected breakpoint on the Foundry GPT-6 chat wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(dashscope): add image edits and text-to-speech
* fix(dashscope): bill reported speech characters and input images, follow the chat region for edits, keep the call timeout on the audio download
* fix(dashscope): bill deployment image prices as main does and route edits to the host speech uses
A deployment that sets its own input_cost_per_image, output_cost_per_image or input_cost_per_pixel bills through the default image calculator again, so an override is never read as a per-input-image price and pixel pricing on qwen-image-3.0-pro no longer raises.
Edits now resolve the caller's api_base or DASHSCOPE_API_BASE through the same API-root helper speech uses, on any host, with DASHSCOPE_API_BASE_IMAGE still winning. An image response whose usage is null or not an object keeps its images instead of raising.
* fix(images): bill a deployment's own prices on image edits through the proxy
The proxy builds the logging object before the router picks a deployment, so image_edit never handed the deployment's price fields to its cost calculation and billed the cost-map price instead. Pass the custom pricing params with the pre-call update, as the Responses API does.
* fix(images): bill per-pixel image prices at a DashScope W*H size
DashScope's mapped size reads 1328*1328, which the cost size reader rejected, so a deployment's input_cost_per_pixel billed every DashScope generation or edit at the 1024x1024 default.
* fix(dashscope): bill image prices by their meaning, report the rendered size, prefer the deployment edit base, fetch audio over https
* fix(dashscope): read the rendered image size from output_width and output_height
* test(images): bill a fal edit at its deployment's own image price
* fix(dashscope): keep the qwen-image-3.0 input-image fee out of the shared cost map row
Released proxies read the cost map from main and bill input_cost_per_image as the
flat price of each DashScope image, so the row's $0.003 input fee billed every
qwen-image-3.0 image there at a tenth of its $0.03 price. A deployment can still
set input_cost_per_image to bill DashScope's input images
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Without drop_params, every copy of a tool from additional_tools input items
is forwarded as sent, so a provider that rejects duplicate tools returns its
own 400. Codex keeps the merge because the proxy turns drop_params on for it.
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts
* refactor(anthropic): hold the chat completions document change for a later commit
* fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai
* fix(snowflake): turn file parts into document blocks for Cortex Claude
* fix(prompt_security): sanitize chat file parts that carry file_data
* fix: answer 400 or drop PDFs for models that cannot read them
Models whose model_info or cost map entry sets supports_pdf_input to
false, and Ollama, now get the PDF dropped under drop_params and a 400
from litellm without it. GigaChat uploads file parts, SAP keeps sending
them as image_url data URLs, azure_ai passes dict content through, and
annotations other than url_citation move to provider_specific_fields
instead of failing the response.
* fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys
Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and
litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider
annotations to provider_specific_fields, and split them the same way when a stream is rebuilt.
* fix(vertex_ai): send PDF file parts to Vertex partner models as image_url
Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages
reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as
unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL.
* fix(cost-map): leave native DeepSeek PDF support unset
/v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there
and would only change chat completions, which this PR does not otherwise touch.
* fix(main): drop a user turn that held only a PDF under drop_params
Dropping the PDF from a PDF-only user turn left an empty content list, which
Foundry and OpenAI-shaped providers reject with a 400, so drop_params still
failed for that request. The emptied user turn is now removed.
* fix: read OCI PDFs, gate deployment base models, and stop empty drops
OCI generic chat sends file parts as image data URLs again, so OCI Gemini
keeps reading PDFs from /v1/messages and chat file parts.
The PDF gate now reads model_info.base_model, drops PDFs under drop_params
for azure_ai deployments that never declared PDF support (the old flattening
behavior), and answers 400 when dropping the PDF leaves no user message.
The cost map marks the Vertex partner text models that a live probe showed
answering without reading the PDF.
* test(integration): cover the PDF input gate, messages PDF file parts and provider annotations
Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed
* test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells
* fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload
* fix(main): PDF check reads the deployment's top-level base_model
* fix(main): keep roles alternating after drop_params removes a PDF-only turn
When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other
* test: type the PDF and annotation tests the PR added
* test: type the GigaChat, SAP and Snowflake PDF tests
* fix(github_copilot): treat a malformed api-key.json like a missing one
* fix(github_copilot): read the key and the endpoints of api-key.json as separate strict views
* fix(github_copilot): ignore a non-UTF-8 api-key.json like any other malformed file
Read the cached key file as bytes so a UTF-16 file, which Windows
PowerShell 5.1 writes with >, fails JSON validation instead of raising
the decoder's error, and the request gets the refresh or the login hint
* fix(github_copilot): decode UTF-16 key files and refresh over a malformed endpoints field
* fix(github_copilot): decode a UTF-16 access-token file and refuse an undecodable one like a missing one
* fix(github_copilot): use the refreshed key when api-key.json cannot be written
* fix(github_copilot): keep a refreshed key that cannot be saved in memory
* fix(github_copilot): serve a kept key and its endpoint together after the key file is repaired
A key held in memory is now checked before api-key.json, so the token and the endpoint come from the same source, and the map holds one entry per key file with the access token beside it
* test(github_copilot): type the authenticator test fixture and the tests this branch adds
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): use collection endpoint for custom context cache
* fix(vertex_ai): build the cachedContents collection URL under a custom api_base
* fix(vertex_ai): keep a gateway prefix that contains /publishers/
* test(integration): expect the cachedContents collection URL on a custom Gemini api_base
* fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix
* fix(vertex_ai): cache under a publisher model api_base in that model's location
A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so
no cache is created that the chat call cannot use. The cache body takes its project and
location from the api_base model, matching the collection URL.
* fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base
A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field.
* fix(vertex_ai): check the cache token minimum against the api_base's model
The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create.
* fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached
---------
Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path
* refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge
* fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper
* fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints
Also drop a marker set on the message item itself, sanitize before the
cache_control conversion so a client typo no longer erases that breakpoint,
and skip items that carry no marker so unmarked history costs no extra walk
* fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it
A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a
litellm_proxy alias) kept its valid markers before this PR and must keep them under
drop_params. base_model is read from litellm_params first, then model_info, as chat
completions does.
* fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged
* fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers
* fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied
* fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one
Under drop_params the marker step now hands back the caller's own objects when every marker is
already wire-safe, so a request with valid markers is no longer copied and walked a second time.
A deployment that turns cache_control into markers keeps the caller's own markers too, whatever
its base_model says, so one request never gets two answers about breakpoint support.
On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion,
so the block gets the cache hint cache_control alone would give, as on /v1/responses.
* test(responses): keep a null marker and the original key order on a model that reads breakpoints
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): resume background response streams with stream and starting_after
Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}
Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally
* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams
A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.
* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve
A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.
* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params
* fix(responses): keep deployment stream off retrieves and reject oversized cursors
* fix(responses): keep router default stream off plain retrieves
* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing
* test(responses): replay a real stream in the billing test and type the fake wrapper chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn
* fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards
* fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set
* fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit
* fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum
* fix(responses): leave a turn without a phase when its content message cannot be read
* test(responses): add black box tests for chat turn history replay
* test(responses): expect the replay record on bridged chat replies
* test(responses): expect the replay record in bridged chat translation cases
* fix(streaming): keep the served service tier on the finish chunk the stream adds
* test(responses): read the expected chat message without JSON validation so the replay matcher stays
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover Codex tool lists repeated across turns on the Bedrock chat bridge and Mantle
* fix(responses): keep each Codex tool once when its tool list repeats across turns
* test(responses): type the new tool list dedupe unit tests
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed
* fix(responses): drop a null created_by from replayed input items too
A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null.
* fix(responses): drop a null created_by from counted input items too
* test(responses): cover saved reasoning nulls and re-compaction of a saved conversation
* fix(responses): drop a set created_by from input under drop_params
* fix(types): type the Responses output serializer's parameters
* fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): preserve every grounding web source
* test(gemini): cover all grounding sources per passage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Zabir Raihan <zabirraihan@icloud.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): send Google's current Managed Agents fields on create and forward agent_config on interactions
* fix(gemini): accept id on managed agent get, delete, and list_versions
* fix(gemini): drop unsupported agent fields under drop_params and keep agent_config keyword-only
A model interaction drops agent_config under drop_params, agent create drops wrongly typed optional fields under drop_params and otherwise answers 400 naming the field the caller sent, and agent_config on interactions create is keyword-only so positional response settings stay in place
* fix(gemini): drop a non-object agent_config under drop_params and name the id alias the caller sent
* fix(interactions): reject agent_config on bridged providers and drop a typeless one under drop_params
Bridged providers answer 400 for agent_config without drop_params and drop it with drop_params. A Gemini agent run drops an agent_config without a string type under drop_params. The Gemini agents wire test expects Google's current upstream field names.
* fix(interactions): honor additional_drop_params for agent_config on bridged providers
* fix(agents): drop a clashing instructions alias under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(azure): audit #47348
Add proxy integration tests for #47348's cached Entra token providers and
off-loop token refresh, and fix the token herd the audit found: once the
fetch moved to a worker thread, concurrent cold requests each fetched their
own token. Every cached bearer token provider now serializes its calls so
the first fetch fills the cache and the rest reuse it
* fix(azure): build each cached Entra token provider once under concurrent first calls
lru_cache does not hold back concurrent misses, so worker threads that miss together each built their own credential and token provider, and each fetched its own token. A lock around each provider cache makes the first caller build it and the rest reuse it
* fix(azure): share Entra token fetches only while no valid token is held
Concurrent callers now share one result (or one error) from the cold
fetch, while callers inside the refresh window keep the still-valid
token. The oidc/azure provider build and the client-assertion exchange
are single-flighted per audience, and the shared helpers no longer call
subscripted generic aliases, which crashed on Python 3.10.
* fix(azure): refresh Entra tokens once while other requests keep the valid token
MSAL treats a token with less than five minutes left as expired, so every
request in that window fetched a new one. Now one request refreshes while
the others keep using the still-valid token.
Requests that waited on a fetch that failed retry once through a fresh
fetch instead of all failing with the first error.
Credentials that only implement get_token keep working, and OIDC token
exchanges are cached per scope.
* fix(azure): bound Entra token waits during an outage
Requests waiting on a failed token fetch now retry only when the fetch
before it succeeded, so during an Entra outage later bursts fail after one
fetch instead of two.
Requests during a refresh only get the held token while it has more than
60 seconds left, and a failed refresh falls back to that token instead of
failing its request.
* fix(azure): share one retry per failed Entra token fetch and raise on a failed refresh
Waiters of a failed fetch now retry the flight they joined, so a fast-failing Entra makes one retry instead of one per waiter. A failed refresh raises again instead of returning the held token, which Postgres token auth read as a fresh one. The DefaultAzureCredential fallback reuses one provider per scope.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): audit #47363
Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses.
* fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call
A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name.
* fix(responses): count a lone Anthropic toolset and name the uncountable tool
The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type.
* fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400
The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint.
* fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through
* fix(responses): count tool_search and namespace tools the way Claude chat inference sends them
The count refused a lone tool_search with LiteLLM's 400 and sent a lone
namespace raw or squashed it into an empty tool, while Claude chat
inference maps tool_search to Anthropic's tool search tool and unpacks
a namespace into its function tools. Both now go through the
deployment's own map_openai_params, so the count sends what inference
sends, and tool_search leaves the list of uncountable types.
* fix(bedrock): count a Claude request with its deployment's own chat translation
Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's
tool search tool, but the Bedrock counter always translated with the Anthropic
config, so the count answered 400 for a request inference serves.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): keep the framed estimate when a region-less token_count_model cannot count (audit #47441)
* fix(count_tokens): let framing stand in for a shipped token_count_model, keep a configured one's error
The framed local estimate now replaces a 400/404 from any count model the cost map ships, Mantle-pinned rows included, and an admin-set token_count_model keeps its error as the docs say.
* fix(count_tokens): warn with the region when framing stands in for a pinned count model
Test a Mantle row #47441 pinned in its own region, and keep the region hint check on an admin-set count model.
* fix(count_tokens): name the deployment in the framing stand-in warning
* fix(count_tokens): let framing stand in when a cost map count model's region denies the account
A count model the cost map pins to a region can answer 401 or 403 when the
account has not enabled that region or lacks Mantle count access there. With
framing set, the estimate now stands in, as it does for 400 and 404. An admin
set count model and a count model with no pinned region keep the error.
The warning now names the model group, the count model and its region, and
says the proxy falls back to the estimate.
* test(count_tokens): cover framed 401 and 403 from a shipped count model region on the wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(images): cover SDK retry budgets and router ownership
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): forward max_retries to the OpenAI image generation client
Per-call, deployment and default_litellm_params max_retries were filtered
out before reaching the OpenAI-compatible image handler, which then always
used 2 SDK retries. Forward the caller's value, and pin it to 0 for routed
calls so the Router stays the only retry owner, matching completion()
Co-authored-by: Ericz <yonglzh@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): read image max_retries through a typed kwargs view
Co-authored-by: Ericz <yonglzh@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): assert the public invalid retry error mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): preserve the invalid retry status code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): group the proxy test imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): discard proxy retry test routers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ericz <yonglzh@gmail.com>
* test(huggingface): cover TEI rerank truncate passthrough and server default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): stop forcing truncate=false on TEI rerank and pass truncate params through
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): keep TEI truncate options provider-specific
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(huggingface): use MockTransport for rerank truncation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
* test(nvidia_riva): audit #41021
Adds an owned Riva ASR gRPC server and provider integration cells for the riva_offline request-body ban, the offline and streaming paths, the per-deployment caller opt-in, and the 413 and 408 error mappings
* test(nvidia_riva): check word timing requests and audio samples in the Riva double
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Covers the json_schema envelope edge cases #47478 fixed on the Anthropic json tool path: a key listed only in
`required`, a tool-named wrapper on a stream, non-ASCII and lone-surrogate answers, a stream cut before the tool
block closes (proxy and sync SDK, Anthropic and Bedrock Invoke), and a string `output_format` with and without
`drop_params` on Anthropic and Vertex chat and /v1/messages. The scripted Anthropic peer now refuses a non-object
`output_format` the way Anthropic does
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(anthropic, bedrock): audit #47489 thinking replay across Claude routes and Bedrock Converse
* test(bedrock): compare the whole replayed Converse turn in the merged-message and cache point cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46815
Add integration tests for every count route #46815 changed, and count a Claude
subscription caller's tokens with their forwarded token when the deployment has
no configured key, which #46815 turned into a 401
* test(proxy): accept the forwarded subscription token in the count endpoint mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(gemini): audit #47542
Proxy integration cells for the Gemini and Vertex default output budget: the env budget reaches Google on chat and responses, turns on long decode, yields to the caller's own limit, caps per deployment, skips resumes, and a camelCase two-candidate request is never continued
* test(gemini): annotate the budget wire module's pytestmark as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Checks that a SageMaker stream failing after its first chunks reaches /v1/chat/completions, /v1/messages and /v1/responses callers as an error with no success end event, and that each writes a failed spend log row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search
Tavily search dropped every field except title, url and content, so
include_raw_content, include_answer, include_images and include_favicon
reached Tavily but their output never reached the caller. Each result now
carries raw_content, score and favicon when Tavily returned them, and the
top-level fields (answer, images, request_id, usage, ...) come back as
provider_metadata, same as the Exa fix in #47423
* test(tavily): move the Tavily search wire test's sample data to module constants, like the Exa test
* test(integration): update stale expectations to intended provider behavior
Each expectation moved because a merged PR changed behavior on purpose.
- chatgpt string input is wrapped as one user message (#41522)
- Gemini 2.5 tool media sits beside function_response (#48428)
- sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645)
- a body base_model is stripped without allow_client_pricing_override (#47639)
- ai_decide answers without usage return usage null (#48261)
- azure_ai forwards mapped client anthropic-beta values (#48033)
- the OpenCode scripted upstream mints a fresh id per request
* test(e2e-ui): type the guardrail name before deleting it
GuardrailDeleteModal requires the exact name before Delete enables since
#45893. All three guardrail delete flows now go through one helper.
* test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body
OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces,
so the gateway's traceresponse now ends in -03. The producer copies the server
span's own trace flags, so the tests now accept any two-digit flags with the
sampled bit set instead of pinning -01
#48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an
object" on purpose, so the [1] body case is renamed non_object_body and
expects that 400
* test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test
The outage test waited for "Exception while exporting Span batch" in the proxy
log, which is OpenTelemetry's wording and no longer appears with 1.45. The test
keeps asserting what the gateway guarantees, callers unaffected during the
outage, a live gateway, and every span after recovery delivered once with no
outage span duplicated
* test(guardrails): count the Responses disconnect spend rows #47611 added
Since #47611 a streamed /v1/responses disconnect writes its own spend row, so
the concurrent disconnect test saw 20 rows where it counted 10 by model group.
It now reads each row's call type and expects ten acompletion rows with the
outage verdict and ten aresponses rows with success. /v1/messages disconnects
still write no row on main, so they are left out of the count rather than
pinned as absent
The responses-httpx case of the single disconnect test no longer needs its
LIT-8603 skip and passes
* test(observability): size K4's timeout from the harness boot budget
K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to
58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of
30 runs hit pytest-timeout. The cap and the restart gate wait now come from
readiness_seconds and graceful_stop_seconds, like other multi-boot tests
* feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): validate the guardrail trace with one pydantic model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): record the native guardrail entry when litellm_params metadata is None
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): import the renamed messages invoke transformation module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): walk the guardrail trace iteratively when redacting it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type the native guardrail trace fixtures and locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): round-trip Anthropic compaction blocks as compaction items
/v1/responses on Anthropic routes now returns Claude's compaction block as a
compaction output item (and streams it as output_item.added/done), and a
compaction input item is replayed to Claude as its native compaction block,
so the next turn reuses the summary instead of compacting again. Other routes
receive the summary as assistant text. The Anthropic stream handler now keeps
the summary text that arrives in compaction_delta.
Fixes#41456
* fix(responses): replay compaction blocks natively only when the request compacts
* fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails
Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem
objects on replay, strip the encrypted_content affinity wrap before decoding,
scan and rewrite compaction summaries in Responses guardrails, drop the history
a summary replay replaces, send the summary as a user message so thinking
ordering holds, and send the streamed compaction .done with the newest block.
* fix(responses): count, route, guard, and redact compaction items consistently
A compaction item now reads as its summary everywhere the proxy counts or routes
a Responses request: pre-call context checks, the TPM limiter, the input token
endpoint, the stream usage estimate, and the previous_response_id session rebuild
all drop the history the item replaces. A Claude compaction item sent to a
non-Anthropic model (including a fallback) becomes a user summary message instead
of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform
takes the summary path. Output guardrails see the summary of a returned
compaction item, and logging redaction hides it.
* fix(responses): start previous_response_id replay at a stored compaction and count reservations from it
* fix(responses): keep a stored compaction summary whole in spend logs
* fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning
A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a
client-built item no longer reach the database whole. Message redaction also nulls the streamed
compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning
item that follows it. Adds the audit's integration tests for compaction items on the wire.
* test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order
Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order.
* fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes
A failed /v1/responses or /v1/messages call keeps its proxy request headers in
litellm_metadata, but the failure row read redaction headers from metadata only,
so x-litellm-enable-message-redaction never redacted those rows.
* fix(spend_logs): lift only the redaction opt-in header into failure rows
The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main
* test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes
* fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary
A previous_response_id rebuild that starts at a compaction now keeps the system and developer
messages of every earlier turn. Token estimates for raw Responses input count a compaction item
as its summary text instead of the serialized item.
* fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id
* fix(responses): count only message items with a string role as compaction instructions
* test(responses): flatten the last two-for comprehension in the compaction wire test
* fix(responses): replay from a compaction in stored Responses output and SDK compaction items
A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths.
* fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history
* fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary
A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a
replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from
rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and
MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound.
* fix(responses): replay a compacted session only through the referenced response
A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item
* fix(spend-logs): keep main's stored value for a prompt length limit under 2
At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where
main stores the marker followed by the whole value
* fix(spend-logs): keep main's stored tail for a negative prompt length limit
At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start
* fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages
* test(responses): run the compaction wire tests on the worker database under isolated workers
* fix(router): keep the encrypted content affinity pin out of provider requests
* fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction
* fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message
* fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses
* fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call
* test(proxy): run the dispatched-call-type failure test through the real request setup
* fix(proxy): count a failed Cursor stream's input from the newest compaction item
The Cursor route maps to no call type, so the stream generator's failure hook fell
back to the route and counted the whole history. Pass the processor's route_type.
* fix(proxy): count a broken Cursor stream's prompt from the newest compaction item
* fix(responses): keep a tool call a later output answers when trimming before a compaction item
* fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes
* perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time
* test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests
* fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item
* fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item
* fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content
* test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks
* fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once
* fix(responses): count input_tokens after a compaction item the way inference sends it
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* fix(llms): join api_base and route with one shared helper
A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base
* refactor(llms): move per-provider api_base strips onto join_api_base
Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join
* ci: run the trailing-slash URL builder test in the llms shard
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
* fix(llms): send every provider route through join_api_base
Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it
Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call
The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash
* fix(llms): keep the route skip only where a builder already had it
join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path
already ends in it, and it is used only where the merge base or the trailing-slash test PR head
already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI
model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the
path so a host named anthropic-* builds /anthropic/v1/messages
* fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL
* test(llms): run the Gemini RAG trailing-slash case through aingest
The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears
* test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base
* test(integration): cover api_base joins across providers, query strings and named hosts
* feat(url_utils): add join_api_base and ensure_api_route helpers
* fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host
join_api_base and ensure_api_route now keep the // of a base left with only a scheme,
so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client
base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it
* fix(azure_ai): build the count_tokens URL from the messages URL
* fix(azure_ai): match anthropic as a whole path segment in the messages URL
* fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints
Azure AI chat and messages go back to cutting the base at the first
/anthropic, as on main, and count_tokens keeps its path check, so a
gateway path like /anthropic-proxy builds the same URLs as main. A base
that already ends in /v1/messages still gets count_tokens next to it.
A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is
used as its text again in AgentCore, Bedrock responses and the RAGFlow
base, instead of falling back to the default host.
get_volcengine_base_url and GigaChat's get_api_base strip a trailing
slash again for outside callers. The cache-hit wire test is skipped as a
known proxy bug, since one cache hit writes two cache_hit spend rows.
* fix(llms): drop a slash before the api_base query and tighten the route test checks
* test(llms): annotate new test locals as Final and flatten the builder generator
* fix(url_utils): keep a scheme-only api_base hostless so it fails before sending
---------
Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>
* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes
The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so
converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock
unguarded while /v1/chat/completions on the same deployment was blocked. The
signer now sends it the way Bedrock reads it, in the body for Converse and as the
X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent
* test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400
The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply
* test(integration): stabilize auth and team delete checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make loop-blocking and tampered-signature checks deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): isolate model-cost state and synchronize loop progress
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): fail loop-progress handshakes when the blocking call never starts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
