* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes
The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so
converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock
unguarded while /v1/chat/completions on the same deployment was blocked. The
signer now sends it the way Bedrock reads it, in the body for Converse and as the
X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent
* test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400
The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply
* test(integration): stabilize auth and team delete checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make loop-blocking and tampered-signature checks deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): isolate model-cost state and synchronize loop progress
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): fail loop-progress handshakes when the blocking call never starts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): read the stream stall cooldown spend rows from the owned proxy's database
The cooldown cells boot an owned proxy, which writes its spend logs to its
own scratch database since owned proxies went private, while the cells polled
the worker's database and never saw a row. Read each row through the gateway
that served the request, so the cells check the proxy they called.
* fix(tests): read passthrough spend rows from the owned proxy's database
The vertex regional uplift and passthrough stream pricing spend files boot
private owned proxies, then polled the worker database for spend rows that
only reach the proxy's own database. Read them through the owned proxy's
gateway instead.
Also hold the stream stall file's hold_headers peer past the longest row
wait, so a slow spend row cannot fail the peer teardown.
* fix(tests): check agent deletion on the owned proxy's database
The agent least privilege test deletes its agent through an owned proxy,
then checked the worker database for the row, which is always empty
there. Read it through the owned proxy's gateway so a delete that leaves
the row fails the test.
* fix(tests): read turn admission spend rows from the owned proxy's database
#47525 renamed _deployment to _fallback_deployment; #47363 was written against the old name and merged after it, leaving three F821 undefined-name errors that fail the python lint job on main
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count-tokens): translate chat tool use before Claude token counts
Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent.
* test(count-tokens): compare Claude count bodies with inference for Responses tool use
* fix(responses): forward the caller's Responses body to OpenAI token counts
OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it
* fix(responses)!: count Claude Responses bodies the way inference sends them
Non-native Responses counts now run the body through the inference bridge, so
text.format, tool_choice, reasoning, built-in tools, and file_search count the
way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The
OpenAI native count body uses the drop_params-resolved tool fields.
* fix(responses): honor drop_params and replay previous turns in Claude counts
Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation
* fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts
* fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params
* fix(responses): leave tool_choice out of counts when drop_params drops every tool
* fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send
The Bedrock Mantle fallback now counts thinking, tool_choice and output_config
like the runtime count. Claude Responses counts leave out cache_control marks
again, so more than four marks count instead of failing. Counts answer 400
before any provider call for user blocks chat inference refuses and for
Responses tools with no Chat Completions equivalent, instead of dropping the
block or counting a placeholder tool. Adds the wire tests for these paths
* fix(count-tokens): count Responses tools the way the inference bridge sends them
A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request
* fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id
* test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler
* fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent
* test(integration): follow #47441 metadata-routed Mantle counting in count_tokens wire tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): build the runtime count body without mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sagemaker): parse every streamed record however SageMaker cuts the payload parts
SageMaker cuts a container's response stream into PayloadParts at arbitrary
byte offsets, so one part can hold several SSE events or JSON lines, or only
part of one. The decoder treated each part as one record, so vLLM's [DONE]
was logged at ERROR on every stream and a split at an unlucky byte dropped
the rest of the answer.
Peel records off a running buffer with JSONDecoder.raw_decode, skipping SSE
framing and [DONE] between them.
* test(sagemaker): cover streamed records cut or merged across payload parts
* fix(sagemaker): decode UTF-8 across parts, skip SSE comment lines, and log container error records
A UTF-8 character split across two PayloadParts now decodes whole through one incremental decoder per stream.
SSE comment lines (TGI's keep-alive) and event, id and retry field lines are skipped as framing.
A container error record is logged at ERROR with its message and skipped, and any other record that is not a
chunk is logged and skipped, the same way on the sync and async paths.
* fix(sagemaker): raise a container's error record and ModelStreamError on both stream paths
A record with a non-null error now raises a 500 SagemakerError carrying the record after the chunks before it, on sync and async. Async no longer swallows a non-200 event, so a ModelStreamError raises as it does on sync. A chunk with a null error field is delivered, and any other record that is not a chunk is logged and skipped on both paths.
* fix(sagemaker): answer a streamed error with the container's status and fail a stream cut mid-record
A container's error record now raises with the status the container gave it (vLLM's error.code, the older flat vLLM shape, TGI's http_status_code or error_type) instead of a fixed 500 that the router retried as a 503. A ModelStreamError raises a SagemakerError naming its ErrorCode and Message, a stream that ends inside a record raises instead of ending as a stop, and a skipped record is logged by its keys only
* fix(sagemaker): trust only 422 and 429 from a streamed error record and fail LMI generation errors
* test(sagemaker): drive an LMI failed-generation stream through the proxy on a sagemaker/ deployment
* test(sagemaker): fall back to the next model when a stream fails before any content
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path
* fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format
* fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params
A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did.
A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper.
Adds integration cases for the json_schema envelope and the cut stream.
* fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates
Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded
* fix(anthropic): join a split answer when the stream ends in its last block
A cut during the second json tool block now runs the held block through
the same flush as a block stop, so it joins the streamed keys or the
earlier split blocks instead of sending two objects back to back. The
Vertex output_format fold drops its unreachable drop_params branch,
since both parent configs drop a non-object value first.
* test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages
Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing
* fix(databricks): stream Claude json_schema answers as text on every streaming path
Async and proxy streams on Databricks Claude json_schema returned the
internal json_tool_call tool call with empty text, because the async
stream factory dropped json_mode. Databricks now passes it through, the
json tool's argument fragments stream verbatim as content (a lone {}
fragment was blanked before, corrupting nested empty objects), and the
stream ends with finish_reason stop like the non-stream answer.
* test(databricks): keep the json_schema stream regression tests ahead of the fix
Holds the Databricks transformation at the merge base so the first approval covers the regression tests; the fix commit follows.
* fix(databricks): stream Claude json_schema answers as text on every streaming path
* fix(databricks): keep finish_reason tool_calls when a json_schema stream also calls a user tool
* fix(databricks): drop text beside the json tool and renumber user tools in Claude json_schema streams
With reasoning_effort set, Claude can write prose before json_tool_call or call it next to a user tool.
Streams now hold text until the end and drop it when the json tool was called, renumber user tool
deltas past the dropped json tool so indexes start at 0, and non-stream returns the json tool's
arguments as content next to the user tools.
* fix(databricks): stream JSON-looking text at once and keep prose ahead of user tools in Claude json_schema streams
* fix(databricks): hold a user tool call behind held prose so a later json tool answer does not glue to it
A json_schema stream with thinking on that wrote prose, called a user tool, then called the json tool sent the prose at the user tool call and appended the JSON answer to it, so content did not parse. User tool deltas behind held prose are now held with it and sent merged per index at the json tool or the finish, and a json tool after text already sent is dropped so the answer is not repeated.
* test(databricks): cover json_schema answers on every endpoint in the integration wire cells
* fix(friendliai): drop litellm-internal assistant fields before sending
Claude Code replays thinking blocks on every turn after the first. The
Anthropic adapter turns them into thinking_blocks plus reasoning_content
on the chat assistant message, and FriendliAI rejects any message key it
does not know, so multi-turn Claude Code sessions on friendliai/ models
failed with a 500 after the first tool call
FriendliAI accepts reasoning_content and GLM reads it back, so only
thinking_blocks and provider_specific_fields are dropped. together_ai
already had this exact helper as a private function, so it moves to
common_utils next to strip_litellm_internal_message_fields and both
providers share it
Related to #32188
* test(friendliai): cover a replayed tool call turn
The regression fixture is now the shape Claude Code replays after a tool
call: an assistant turn with thinking_blocks, tool_calls and no content,
followed by the matching tool result. The test asserts both survive the
sync and async request paths unchanged
* ci: retrigger checks
* ci: retrigger benchmarks after runner timeout
* chore(friendliai): mark inherited list signatures as mutable-ok
The type-discipline gate on main now caps LIT001 at the merge-base count,
so the _transform_messages overloads need the same inherited-contract
marker together_ai uses for its identical override
* ci(test-unit): add tests/unit/llms/friendliai to the llms-providers shard
* test(friendliai): check the outbound FriendliAI body through the proxy for chat and messages replays
Both replay shapes go through a real proxy to a scripted FriendliAI upstream, and the test reads the
request body FriendliAI received. It fails at the merge base, where thinking_blocks and
provider_specific_fields reach the upstream, and passes with the field stripping in place.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
The override test slept a fixed 10 s after writing responses_websocket_session_limit_seconds and then
opened sockets on both workers. A worker whose config resync ran later than that kept the old 7200 s
cap, so its socket never closed and the test failed after 75 s
A test-only /health/integration-general-setting route returns the value a worker holds along with its
pid. The test polls it over fresh connections until every worker reports the new cap, and after the
delete until every worker reports it gone
The cooldown cases in test_hosted_vllm_stream_stall_wire.py boot an owned proxy, which writes to a
private database since #47213. The file was written on a base without #47213 and read its rows from
the worker database through read_rows, so every cooldown case that waits for a failure row timed out
on main with State did not converge: []
Every row read now goes through Gateway.rows on the gateway that served the request
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* refactor(count_tokens): hold the tool counting wiring for a follow-up commit
* refactor(count_tokens): hold the chat template tool sections for a follow-up commit
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* fix(count_tokens): skip tool_choice and non-function tools for chat template families, leave out Fireworks V4.1 spelling
* fix(count_tokens): count Anthropic tools in the shape the messages bridge sends them
A missing description is no longer rendered as null and strict is counted when set, matching what DeepSeek V4 and Qwen3.5 deployments receive. Adds tests that pin each template's schema rendering and its system turn.
* test(count_tokens): check the prompt caching threshold agrees with token_counter for DeepSeek V4 and Qwen3.5 tools
* fix(count_tokens): skip Anthropic web search tools and count unmapped Anthropic tool keys for chat template families
* test(token_counter): cover chat-template tool counts across count routes, budgets, ITPM and context edits
Integration cells check that DeepSeek V4 and Qwen3.5 tools are counted the way their chat templates render them on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens, strict budget reservation, project ITPM on chat, messages and responses, and the clear_tool_uses trigger, with grok controls keeping OpenAI framing
* fix(streaming): map stream read timeouts to litellm.Timeout
* revert(streaming): move the timeout mapping to a later commit
* fix(streaming): map stream read timeouts to litellm.Timeout on chat, messages, and responses
* fix(streaming): keep stream fallback and cooldown unchanged for transport timeouts
A chat stream served through the Responses bridge now keeps mid-stream
fallback on a stall, because the fallback exemption also covers a
litellm.Timeout whose cause is an httpx transport timeout. The Responses
stream hands the raw transport error to failure callbacks again, so
router cooldown sees a stall as it did before. The 408 deployment info
now lists stream_timeout.
* test(integration): cover a stalled hosted_vllm stream on chat, Responses, and Messages with its spend log row
* fix(router): keep a fallback deployment whose stream stalled out of cooldown
* test(integration): cover bridged chat, fallback and cooldown for stalled hosted_vllm streams
* feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes
* fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting
The drop list now runs on the final upstream headers, after x-pass- expansion, through an
upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a
dropped value to Vertex. A malformed setting now answers 500 only for requests that carry
anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working.
* test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests
* fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces
* fix(vertex_ai): split drop-betas list entries on commas like the header
* test(integration): stop GET /__observations from draining the upstream log
GET /__observations used to empty the queue, so a second read in a test
saw nothing and a "never reached upstream" check could pass whether or not
the request was forwarded. GET now returns every request since the last
reset and DELETE /__observations is the reset. An autouse fixture resets
the worker upstream before each test, which also covers tests that read
without the gateway fixture.
Discarded reads used as resets become DELETEs, helpers that accumulated
drained reads now read one snapshot, and tests with several phases reset
between them. The three vacuous negative checks (rag allowlist, guardrail
opt-out chaos, xAI batch output) now assert the absence on the same read
that shows a request which did reach the upstream.
* style(integration): space the observations unit test like its neighbours
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response
Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.
* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests
* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing
* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown
Moves the cost map lookup off the event loop
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing
A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.
* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates
* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment
A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.
* test(count_tokens): use the shared integration port allocator in the fallback wire test
* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today
* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings
Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it
* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body
A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.
* fix(count_tokens): answer the provider's error whenever a framed estimate raises
A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.
* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate
A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.
* fix(count_tokens): look up framing under the deployment's custom_llm_provider
* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error
A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.
* fix(count_tokens): keep one tokenizer download per key running and load the local route per request
* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible
* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error
Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500
* feat(proxy): count tokens with token_count_model in its pinned region
token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable
* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys
* fix(proxy): keep the upstream error envelope when adding the region hint
* test(proxy): annotate the token count setting echo test
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(count_tokens): answer the token_count_model's error even when framing is set
* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate
* fix(count_tokens): estimate Google contents the route translated to messages
* fix(count_tokens): keep the provider error for Google contents with non-text parts
* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock
* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags
* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer
* test(bedrock): keep the Kimi K3 image check in-process
* fix(count_tokens): join the region hint to an upstream message that already ends with a period
Also pins that a blank token_count_model in config falls through to the cost map default
* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts
The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts
The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool
A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning
* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them
A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.
* test(count_tokens): annotate new token counter test locals with Final
* fix(count_tokens): keep shipped count regions inside the deployment's geography
A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added
* fix(count_tokens): resolve count settings off the event loop
* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout
A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream.
* test(timeout): cover stream_timeout on live streams only and the router's timeout split
* fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management
* fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface
A faked stream now takes the longer of timeout and stream_timeout, so a caller
who raised stream_timeout to cover a slow fake stream keeps that headroom.
Azure o-series dispatch is mirrored when deciding whether a stream is faked.
text_completion, /v1/messages and /v1/responses through the Router now get
timeout and stream_timeout separately, like chat calls.
* fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages
* fix(router): report and log the timeout each stream was bounded by
Native /v1/messages and /v1/responses now record the timeout they applied, so
x-litellm-timeout and the cooldown guard read the real limit instead of the raw
request value. A prompt-managed model also passes on only the caller's or its
own timeout, so the target deployment's timeout still wins.
* fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit
* fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses
* fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit
* test(integration): cover faked, live and converted stream timeout limits per endpoint
* test(router): regression tests for faked-stream timeouts
* fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row
A deployment with an explicit custom_llm_provider and model_info.mode: chat
used to overwrite the shared litellm.model_cost row, moving every other
deployment and plain SDK calls onto Chat Completions until restart, because
the downgrade guard read a double-prefixed key and never fired. The guard now
reads the real key, and a deployment's own mode decides its routing in
responses_api_bridge_check, except on chatgpt and github_copilot where Chat
Completions is dead for responses-mode models.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): keep the bridge-check mode tests in the sharded bridge flag file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read the downgrade guard's mode from the row registration writes
A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read
openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit
responses/ prefix now also outranks a deployment's chat pin.
* fix(router): let an explicit responses/ prefix outrank a registered chat row
The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the
lookup found mode chat and the request went to Chat Completions as responses/<model>.
* fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages
A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions.
* fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto
A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them.
* fix(router): skip a non-string model_info.mode when keeping the registered mode
* fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API
A Mantle row that also serves the Responses API now follows the GPT-5 tool and
reasoning summary bridge rules, since Mantle Chat Completions rejects function tools
on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat
gets a dict reasoning_effort as its effort level.
The opt-in auto summary asks each Responses config for its level, auto on Mantle's
OpenAI path, so a client's own detailed summary follows drop_params as at base.
No mode replaces a registered responses mode, and chatgpt and github_copilot rows
take any pin until they are responses, which restores the Copilot chat opt-out.
* fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort
* test(integration): run the Mantle region-prefixed v1 base case again
This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip
* fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin
A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request
* fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request
* fix(router): keep a non-string deployment mode off the shared routing row
A deployment whose model_info.mode was an int, a list, or an object still
reached the shared cost-map row, so an unpinned sibling on the same backend
stopped routing to the Responses API. The shared registration now drops a
mode that is not a string, and the deployment keeps its own raw value.
Adds integration cells for per-deployment routing across the OpenAI and
Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream
outage, config changes under traffic, and a worker kill plus restart.
* test(integration): find each routing cell's spend row by the response it logged
A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts
* fix(bedrock_mantle): run extra_body fields through the chat checks
A chat-pinned Mantle request that nests n or reasoning_effort in
extra_body now gets the same 400, drop, and effort-level handling as
the top-level fields. The bridge test injects a MockTransport client
instead of patching HTTPHandler.post.
* test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels
* fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort
* fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request
* fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode
* fix(router): drop the debug log that wrote a request's model name from keep_registered_mode
* test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort
* fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down
* test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks
* test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them
* fix(router): route a fine-tune the same whichever deployment registers first
Main's register_model now seeds an unmapped fine-tune's row from its base
model's built-in row, mode included, on the first registration. The mode
guard read only the live row, so a responses pin marked the shared row only
when it registered before its siblings. The guard now reads the row the
registration merges onto, the live row or else the row it would seed.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens
* fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests
Vertex has no count-tokens route for Mistral or codestral models, so they now
get the same 400 as other partner models without a count API instead of a 404
that points operators at IAM.
* fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens
* fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model
* fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors
* fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer
* fix(count_tokens): pass through an OpenAI provider error that omits param or code
* fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route
* test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy
* test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes
* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy
These two files register deployments with their own model settings on the
gateway. They now boot a module-scoped proxy on a scratch database, so their
setup cannot affect other tests that share the worker's proxy
* test(integration): seed the isolated gateway with the same router settings as the worker gateway
* test(vector-stores): reproduce team key search routed to a team model of the wrong provider
A team key's model-less vector store search picks the team's first deployment, so a langgraph team model turns a Bedrock store search into "Vector store search is not supported for langgraph". Cover map_team_model picking by provider and a registered Bedrock store search reaching Bedrock with the registry's credentials.
* fix(router): match model-less team deployment to the requested provider
map_team_model now takes an optional custom_llm_provider. With no model given, it returns the first team deployment serving that provider, or None when none does, instead of the team's first deployment. Without a provider it behaves as before, so BYOK teams calling raw vector store IDs keep their team key.
* fix(proxy): pass the vector store provider when mapping team models
Vector store search, retrieve, update and delete pass the store's custom_llm_provider to map_team_model, so a team key's model-less call lands on a team deployment of that provider, or on the direct path with the store's own credentials when the team has none. Other model-less routes keep today's behavior.
* fix(proxy): type the vector store provider passed to map_team_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector-stores): cover team key vector store search provider routing end to end
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format route_llm_request.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type vector store routing tests and mock Bedrock transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin model-less vector store calls when a public name mixes providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep team provider routing in map_team_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries
* fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5
A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow
* refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping
Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass
* chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client
The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py
* fix(vertex): match standalone 403 status tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): keep text 403 fallback for wrapped 500 errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): keep BadRequestError for explicit 403 text matches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): let structured body codes override the 403 text fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards
* test(rust): check the dangerous-tool-use beta reaches the wire with safeguards
* fix(rust): derive the dangerous-tool-use beta only for hosts with a beta map column
DeepSeek and the other hosts without a column in anthropic_beta_headers_config.json
keep forwarding safeguards and client betas as sent. Adds the wire-level audit tests.
* fix(rust): add the safeguards beta on Azure AI and Claude Platform hosts
Azure AI and Claude Platform Messages transforms now add the
dangerous-tool-use beta for a non-null safeguards field, through the
Azure AI and Anthropic beta-map columns. The safeguards wire and chaos
tests get unique file names so they do not collide with the Python
fix's tests, the Python-route test asserts the fixed behavior behind a
BUG skip, and the DeepSeek test checks that no beta is added
* fix(rust): keep Claude Platform out of the safeguards beta until a route can prove it
Claude Platform has no beta-map column and no QA account, so it goes back to sending
safeguards with no derived beta, like DeepSeek. Also release held chaos-test requests
before the thread pool waits on them, and mark the no-column test as rstest.
* fix(rust): derive the safeguards beta for Claude Platform and keep the null-cell test off Databricks
Claude Platform on AWS answers 400 to safeguards without dangerous-tool-use-2026-09-03 and 200 with it, so its Messages transform derives the beta through the Anthropic column again. The null-cell case now uses only Bedrock Converse, since Databricks support for the beta is changing in a sibling PR.
* fix(rust): drop a null safeguards key instead of sending it without its beta
Anthropic rejects any safeguards key, null included, when the
dangerous-tool-use beta is missing, so an explicit "safeguards": null on the
Rust gateway still answered 400. The shared Messages transform now drops a
null safeguards key on every host, as the proxy already drops None params
before calling Rust, so it answers 200 with no beta.
The scripted peer in the integration wire tests now rejects the key on
presence, matching the real API.
* test: cover Bedrock Mantle in the null safeguards case and match the vendor's non-list wording
The Rust null-safeguards regression test now also runs for bedrock_mantle, whose beta is keyed on the field being present. The scripted Anthropic peer answers a non-list safeguards value with the real API's "valid array" wording.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes
* fix(bedrock): move Invoke body requestMetadata into the signed header and match Bedrock's ASCII whitespace rule
* fix(bedrock): keep Invoke body requestMetadata in the body and answer bad metadata as AWS does
Invoke requests whose body already carries requestMetadata keep the resolved pairs in the body
instead of the 8,500-character signed header. Malformed caller metadata now answers 400 with the
ValidationException error type the AWS SDK expects, and drop_params drops bad pairs on both the
pass-through routes and /v1/chat/completions. The caller-pair dedupe is linear, and the chat
validator treats \s as ASCII whitespace, as Bedrock does
* fix(bedrock): match Bedrock's 400s and header limit on pass-through request metadata
Invoke metadata too long for the 8,500-character header now goes in the body,
Invoke requestMetadata null answers 400 like Bedrock, Converse type errors carry
SerializationException, violation messages name the failing key, and the reserved
user_api_key_ prefix is matched case-insensitively after trimming whitespace
* fix(bedrock): trim the Invoke metadata header to Bedrock's limit and drop bad requestMetadata under drop_params without the setting
* fix(bedrock): leave /bedrock pass-through bodies as sent when request metadata forwarding is off
Global drop_params no longer rewrites requestMetadata on the pass-through routes unless
bedrock_request_metadata_fields is set. Chat keeps dropping bad pairs under drop_params.
Adds a tests/integration wire test for the four pass-through actions and removes an unused
parameter from the Invoke header helper.
* test(bedrock): drive the pass-through metadata wire test with boto3's bearer key env var
* fix(bedrock): trim the pass-through invoke metadata header and stop Nova invoke from checking metadata it drops
* test(bedrock): cover pass-through metadata header trimming, malformed caller metadata, and chat drop_params on the wire
* fix(bedrock): match requestMetadata keys and values against the whole string so a trailing newline cannot pass the 256-character limit
* fix(azure-ai): support Entra auth for Decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure-ai): select Decisions provider by name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): prioritize deployment credentials over env key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): restore api key precedence for Decisions auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
kill() returns once the killed worker's main thread is a zombie, but its other threads can
still be exiting and holding its sockets open. A keep-alive connection pooled to that worker
still looks healthy, so the next request goes out on it and the closing socket answers with
a reset. The old reap wait used to hide this. A new connection can only be accepted by a
live worker, so the surviving-worker assertions now run on a fresh client
* test(integration): wait for both workers to load each metered model before sending
* fix(prometheus): read remaining rate-limit headers as numbers and let the router fill unreadable ones
* fix(prometheus): keep deployment metrics on non-string header keys and cover decimal and router-filled gauges
Accept any mapping key when reading additional headers, so a non-string key no longer drops every success metric. The router fallback test now uses a real Router and covers inf, nan and unreadable requests values, the decimal test uses a fractional requests value, and the integration cell sends 12.5 and an unreadable tokens value to deployments with tpm set
* fix(prometheus): leave the router fallback unchanged and keep only the decimal header fix
* fix(prometheus): read remaining rate-limit headers with float() like every release did
* test(integration): check the remaining rate-limit gauges on /v1/responses and /v1/messages too
* fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it
Bedrock accepts outputConfig structured output on every reachable profile of
Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so
json_schema now goes out natively there instead of as a forced tool, which kept
Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its
keys go back to the tool path. GovCloud keys keep their flags, and the drift
tests let GovCloud lag its commercial base but never lead it.
* fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path
Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf,
additionalProperties true, and recursive or unresolved $ref, all of which the
forced tool accepted. Claude schemas on the Bedrock native paths now go through
the cleaner the Anthropic native path already uses, and a schema with a
recursive or unresolved $ref keeps the forced tool, inline, or drop path it had.
* fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key
Older installs fetch the model map from main, so flipping supports_native_structured_output
moved them onto native output with their old schema handling. The new
supports_bedrock_native_structured_output key is ignored by older code. Schemas native output
cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new
key, models on the old key keep their current behavior for those shapes, and the Anthropic
API, Vertex AI, and Azure keep their current behavior.
* chore(cost-map): regenerate the model prices schema for the Bedrock native output key
* fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool
Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since
Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters.
Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects
typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1.
* fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking
* fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model
Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1
* fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas
The Bedrock native output key now applies only when the call asks for thinking, so calls without
thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts,
or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open
objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned
them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request.
* fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke
* test(bedrock): cover native structured output and json tool routing through the proxy
Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking
* fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode
Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas.
* fix(anthropic): keep native output for dict fields unless the json tool can be forced
* fix(anthropic): keep native output for dict fields when the caller sends tools
* test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects
* test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas
Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool
* fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output
* test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects
* fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice
* test(bedrock): name the malformed realtime setting test for its string cases
* fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit
Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params.
* fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats
* fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices
* test(bedrock): cover realtime replay, refusal and committed failure through a real proxy
Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again
* fix(openai_like): surface provider response headers on non-streaming chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(openai_like): mark the header test locals Final
* fix(proxy): leave out response header values HTTP cannot carry
Non-streaming OpenAI-like chat now forwards every provider header, so on the
httpx transport a provider header value outside latin-1 made the proxy answer
500 after the provider had served the call. get_custom_headers now leaves such
a value out and keeps the other headers, which also ends the same 500 on
streaming
Adds integration cells for the provider request id on non-streaming chat and
the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend
parity under an upstream cost header, the latin-1 case, and the logging
callback payload
* fix(cost): ignore an upstream response cost header that is negative or not a number
* test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint
* fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number
* test(prometheus): type the rate-limit header test and drop its clock reads
* test(integration): cover deployment metrics under an unparseable provider rate-limit header
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets
* test(integration): wait for the team spend flush before killing a worker
* test(integration): poll for the S3 attempt the batch sinks write concurrently
* test(integration): queue the terminal object and its sibling into the same s3_v2 flush
* test(integration): time the long Bedrock version against an equally long unversioned name
* test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it
* test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing
A non-list tools value crashed the vector store lookup during auth and the
built-in tool cost lookup in the failure logging path, so the proxy answered
an unhandled 500 and closed the keep-alive connection. count_tokens now
answers a 400 invalid_request_error for a malformed body, drops malformed
tools when drop_params is on, and both lookups skip values of the wrong shape.
* fix(proxy): skip non-list tools in the tool allowlist check
Also validate the count_tokens body without recursion, which the recursion detector rejects.
* test(integration): restore the int tools case on count_tokens
The proxy now answers a non-list tools value with a 400, so the case is deterministic again.
* fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids
* fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids
Check tools after request setup so the Claude Code default and the logged call type apply, read the request,
global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini
generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off
* fix(proxy): keep the request's call type when logging a gate rejection after request setup
The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses
rejection as aembedding, so it now only applies when the logging object carries no known call type
* fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry
* fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket
* refactor(proxy): leave the responses websocket out of the request-level tool check
* fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections
* fix(proxy): read router default drop lists and deployment ids in the malformed tools check
* fix(proxy): leave malformed tools to the router when it can pick another deployment
Fallbacks, auto-routers, model strings, comma batches and team model names
let the router pick a deployment after the proxy checks the request, so the
proxy now forwards a malformed tools field there as main does, and only
rejects it when the deployment it checks is the one that serves. Token
counting never reroutes, so count_tokens and input_tokens keep reading the
group they count with. additional_drop_params names are read the way core
reads them, so a list with a non-string entry still names its strings.
* test(integration): cover malformed tools on reroutable models, router defaults and deployment ids
* fix(proxy): resolve malformed tools at the deployment the router picks
* fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools
Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells.
* test(integration): expect a 400 for an empty-string tools field on /v1/messages
Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test
* fix(router): keep web search routing for tools passed as a tuple
* fix(router): keep a deployment's own tools when drop_params drops a malformed request tools
* test(router): list _merged_tools as covered through the deployment tools merge tests
* fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them
* fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError
* fix(prometheus): keep deployment health metrics clean on a malformed tools rejection
* test: type the new malformed tools tests
* test: mark new test locals Final and type kwargs dicts
* fix(bedrock): drop dangerous beta without safeguards
* fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle
Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards.
* fix(anthropic): keep the AnthropicThinkingParam import that main now uses
* fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent
Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version.
* fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards
* test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses
* test(bedrock): type the new dangerous-tool-use beta tests
* test(bedrock_mantle): type the dangerous-tool-use beta Mantle test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count
/v1/messages runs clear_tool_uses context edits before it sends an Anthropic
body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted
the body unedited, so a request with older tool results cleared counted about
2.8x what inference bills.
The count now runs the same history and context-edit step inference runs,
honors additional_drop_params on the deployment, and skips compact edits so a
count never makes a paid summary call.
* fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs
Run compact edits in the count without the summary call so trigger checks and
history slicing match /v1/messages, honor a request-level additional_drop_params,
keep the full body when any deployment in the model group drops context_management,
and answer 400 naming a block Gemini cannot count when an edit trigger fails on it
* fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check
* fix(count_tokens): log a failed edit in the count at debug, since the send already logs it
* fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does
* fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check
The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits.
* fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name
* fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to
* fix(messages): drop an invalid context_management spec under drop_params so the send matches the count
* fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them
The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes.
* test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code
