A tools entry of only {"type": "function"} has nothing for the Responses
bridge to convert, and the bridge raised a 500 for it where the chat
route returns the provider's own 400. The gate now counts a tool as a
function tool only when it carries a function body or a top-level name,
on every provider the gate serves
Foundry's OpenAI v1 chat surface rejects function tools with an explicit
reasoning_effort from gpt-5.6 on and with reasoning left on from gpt-6 on,
while gpt-5.4, gpt-5.5 and unset-effort gpt-5.6 serve them. Key the
azure_ai bridge on those measured boundaries instead of the azure
provider's gpt-5.4+ rule so working chat traffic keeps its n, logprobs,
seed and chatcmpl ids.
Forwarded client headers on bridged /v1/responses calls were serialized into the
OpenAI JSON body as extra_headers when EXPERIMENTAL_OPENAI_BASE_LLM_HTTP_HANDLER
was set, and OpenAI rejected the request with unknown_parameter. The headers are
already merged into the outgoing HTTP headers, so only set the SDK-style
optional param on the SDK client path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The rebuild's tool-call selection and its text-only fast path only looked at
choice 0 of each chunk, so a chunk that packs several choices (Gemini with
candidateCount above 1) lost a tool call carried by a later candidate, and a
chunk whose later choice had no tool calls at all made the rebuild raise.
Both now consider every choice in the chunk.
* fix(streaming): keep usage-only chunks from crashing streams with empty stream_options
The usage-only chunk branch in CustomStreamWrapper.chunk_creator indexed stream_options["include_usage"] directly, so a caller passing stream_options={} hit a KeyError that surfaced as MidStreamFallbackError. Streaming mock_response with an admission input_tokens count (#40637) now always emits such a chunk, which made the crash reachable. Reuse the send_stream_usage policy computed at init instead. Also annotate the #40637 test bindings with Final.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): report admitted zero prompt tokens instead of recounting in mock streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(mock): emit admission-time usage chunk on streaming mock_response
Streaming mock_response chunks carried no usage, so the chunk builder re-tokenized the whole prompt in Python after the stream ended even when budget reservation had already counted it at admission. The mock streaming generators now yield a final usage-only chunk carrying the admission prompt count (same completion count as the non-streaming path). Without an admission count the old tokenizer fallback stays.
* fix(mock): type the mock stream generators and keep the usage chunk on the content stream id
Review follow-up: the usage-only chunk was built with a fresh id, so CustomStreamWrapper switched response_id for the finish-reason and usage chunks. It now copies the content stream id. The generators also get full parameter and return annotations.
---------
Co-authored-by: yassin <yassin@berri.ai>
* feat(mock): report admission-time input token count in mock_response usage
Mock completions always reported prompt_tokens=10, so spend tracking, TPM metrics, budgets and the tokens-per-minute autoscaling signal saw 10 tokens for a 100k-token request. Budget reservation now carries the admission-time input token count in the reservation record, and mock_completion reads it back so mock traffic exercises the same spend and TPM paths as real traffic without any extra tokenizer work.
* fix(mock): keep a zero admission input token count instead of falling back to 10
---------
Co-authored-by: yassin <yassin@berri.ai>
Narrows the no-choices guard so a dict, string, or None still raises the APIError while an empty list passes through,
guards the non-stream Anthropic bridge against indexing an empty choices list, and repairs test_completion_missing_role,
whose raw-response mock was patched in as the create() callable itself so the handler only ever saw a MagicMock
The auto-bridge that moves gpt-5.4+ requests carrying function tools and no
reasoning_effort onto /v1/responses only fired when the resolved api_base was
the literal https://api.openai.com/v1, so a deployment pointed at an OpenAI
PrivateLink hostname (<region>.privatelink.api.openai.com) or a port-qualified
or trailing-slash default stayed on Chat Completions and got OpenAI's 400 back.
Gate on the resolved URL's hostname instead: api.openai.com or any subdomain of
it bridges, every other custom base still stays on chat
Review turned up two real problems in the TTS path.
Router.aspeech forwarded voice=None whenever the caller omitted it, which overwrote a
voice set in the deployment's litellm_params, so a configured fallback voice was
ignored on voice-less requests. It now leaves the key alone when no voice is passed.
get_complete_url also fell back to MISTRAL_API_BASE, but speech() always receives a
non-null api_base from get_llm_provider, whose mistral branch only reads
MISTRAL_AZURE_API_BASE and otherwise hardcodes the public host. That branch could
never run, and its unit test asserted a behavior the real path does not have. The
working override is api_base on the deployment, now pinned by an end-to-end test
Resolves two conflicts:
- tests/test_litellm/vector_stores/test_main.py: staging moved search() to a
RouterVectorStoreEmbeddingExecutor while this branch parametrized the same
test over query; keep both the executor assertions and the parametrize.
- tests/logging_callback_tests/test_bedrock_knowledgebase_hook.py: staging
carries a duplicate embedding_executor kwarg that makes the file a
SyntaxError; drop the trailing duplicate.
Resolves the tests/test_litellm/test_main.py collision, where both sides appended a
new test at the end of the file, by keeping both.
Also carries the one-line fix from #39502: staging arrived with a duplicate
embedding_executor kwarg in the Bedrock KB fake handler, which ruff rejects as a
syntax error, so every commit here would otherwise fail lint. The change is byte
identical to #39502, so that PR merges cleanly once it lands.
Restate xAI's usage.cost_in_usd_ticks as usage.cost on chat and responses
replies, streamed ones included, then let the cost calculator own the
figure: a deployment with its own input_cost_per_token and
output_cost_per_token keeps that price, cost margins apply on chat streams
as they already did on non-streamed calls, and only OpenRouter's usage
cost becomes the llm_provider-x-litellm-response-cost header, so xAI
streams no longer skip the calculator through the header or the
stream_chunk_builder hidden response_cost.
Streamed responses through the proxy previously exposed no usable cost:
the x-litellm-response-cost header is unreadable mid-stream and the final
usage chunk carried only tokens, priced against an alias model name the
client cannot resolve. The include_cost_in_streaming_usage flag existed
but was off by default and only fixed the wire, not SDK clients.
Stamp usage.cost into the joined streaming response by default wherever a
final usage object is built: the chat-completions stream_chunk_builder,
the native /v1/responses RESPONSE_COMPLETED event, and synthetic response
events. Provider-reported cost always wins over the computed value, and
only positive computed costs are stamped so unpriceable alias responses
keep deferring to the logging object's own calculation. Per-chunk SSE
cost injection (/v1/messages, generateContent, passthrough) stays behind
the flag.
Also normalize non-litellm usage objects in stream_chunk_builder: openai
CompletionUsage lacks Usage.__contains__, so membership probes silently
returned False and client-side rebuilds dropped the wire cost and
recounted token usage locally. Wire token counts and cost now survive.
Resolves LIT-6427
Router._add_deployment called get_llm_provider without the deployment's api_base, so a config entry with a bare model plus a known OpenAI-compatible endpoint failed startup validation with LLM Provider NOT provided and the proxy returned 400 no healthy deployments for that model group. acompletion had the same gap at request time: it forwarded only base_url into its get_llm_provider call, dropping the api_base kwarg the router passes. Both now forward api_base so endpoint matching resolves the provider the same way sync completion already does
