OSV started flagging claude-agent-sdk 0.1.44 today, which turns OSV Scan red on every PR. 0.2.165 is the newest release outside the 3-day uv cooldown. The package sits only in the ci dependency group and is only imported by the cookbook
* feat(ui): add clean-slate playground chat prototype
* feat(ui): replace the playground chat with the redesigned composer and inspector
* fix(ui): close the playground chat gaps found in self-review
Resend only complete text exchanges, so empty replies, MCP direct calls and
transcriptions no longer leak into chat history. A stopped request can no longer
append output, attach its gateway trace to a newer reply or clear the newer
request's loading state. MCP selections are only pruned once both the server and
toolset lists load. The MCP tool picker lists toolsets again and calls the server
that backs the chosen tool, and a saved model the current key cannot use is
cleared when the model list loads, matching the old chat
Also vendors the scroll-fade-x utility the attachment primitive uses
* fix(ui): resolve playground toolset tools per server and drop late session ids
A toolset now lists only its own tools, each resolved on the server the toolset
names, so the args form and the call always target the same tool. A saved
toolset no longer asks for tools with its own id while the toolset list loads,
and a failed tool list says so instead of claiming the server has no tools
A stopped request can no longer overwrite the Responses session id, starting a
new chat drops a request that is still running, embeddings replies are no longer
resent as chat history, and a saved model missing from an already cached model
list is cleared too. A failing toolset list falls back right away instead of
retrying while the MCP pickers show loading
* fix(ui): guard playground sends and label inspector stats
Pressing Enter twice no longer sends the same message twice, an attachment the
new endpoint cannot take is dropped when the endpoint changes, a stopped
Responses request can no longer show its code output on a newer reply, and a
reply that broke mid-stream is no longer resent as history
The inspector's speed, cost and routing stats get the same icons as the reply
footer, and the footer's metrics now line up with its copy and Inspect buttons
* fix(ui): write playground key settings only through the encoded storage path
Key source, virtual key and custom headers now go through their own write loop, separate from the plain settings, so a key value can never take the plain sessionStorage branch
* fix(ui): restore playground BYOK connect and reconnect
Picking a BYOK server for a direct tool call now shows a key button next to the
server picker, so the key can be connected without leaving the MCP endpoint. A
connected BYOK server gets a Replace key action in the MCP dialog, also shown
when its tools fail to load, and saving a key reloads that server's tool list
* revert: restore the single playground settings storage loop
Reverts ca4e742115. The CodeQL finding covers the same encoded sessionStorage handling the old chat used, so it is left as is
* test(ui): type the BYOK credential mock without deep instantiation
* fix(ui): let users back out of replacing a BYOK key
The replace-key form in the MCP dialog now has Cancel, and it resets when another server is picked or the dialog closes, so a working key is never forced to be overwritten
* refactor(ui): name the MCP dialog row focus handler
* fix(ui): accept exact custom models, tags and decimal temperatures in the playground
A typed model or tag that is a prefix of a listed one could not be used,
because Enter picked the partial match and the custom action only showed
when nothing matched. The custom value is now a list option whenever no
exact match exists. Temperature gets an editable input beside the slider
so values finer than the 0.1 step can be sent again.
* fix(ui): stop claiming no guardrails ran when the gateway sent no headers
Direct MCP calls, A2A agents and any request that fails before headers
arrive built their trace from an empty header set, so the inspector said
no guardrails or policies ran. The gateway fields are now left unset in
that case and the inspector says the endpoint does not report them.
* fix(ui): leave policies unknown when the gateway headers are hidden
The proxy exposes x-litellm-applied-guardrails to other origins but not
x-litellm-applied-policies, so a dashboard on another origin read every
request as having run no policies. Policies are now unknown unless the
call id header is readable, and the inspector says so.
* fix(bedrock): drop dangerous beta without safeguards
* fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle
Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards.
* fix(anthropic): keep the AnthropicThinkingParam import that main now uses
* fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent
Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version.
* fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards
* test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses
* test(bedrock): type the new dangerous-tool-use beta tests
* test(bedrock_mantle): type the dangerous-tool-use beta Mantle test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): add the 1 hour cache write cost to the add model and model edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover both cache write tiers in the add model and edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): hoist the expected pricing payload out of the inline assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): share the per-token conversion between both cache write tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): derive the touched pricing field type from the pricing field list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): collapse the duplicated pricing source keys into one map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): convert add model pricing once for all selected models and describe the 1 hour fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): type the cache write tier deployment read back and fill the 1 hour field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(e2e): keep the cache write tier spec on the dashboard prettier config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openai): route gpt-transcribe to GPT audio transcription config
* test(openai): move gpt-transcribe tests under tests/unit
* fix(openai): bill gpt-transcribe from provider usage seconds when duration is unknown
Fall back to the provider-reported usage.seconds when neither a measured
file duration nor a response duration is available, so json responses
for undecodable uploads are no longer billed at zero. Move the routing
test next to the other provider config tests and replace the structure
test with a behavior test.
* fix(openai): bill provider usage seconds only for gpt transcription models
* fix(cost): decide provider-billed transcription seconds from the deployment model, not its pricing id
* test(integration): cover gpt-transcribe default request, provider-billed spend, and health probe
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): remove price fields the real gemini and azure rows do not carry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): key vertex gemini cases on the vertex_ai prefixed rows and add gemini-4-argon
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add current gemini and vertex usage keys to stored responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add bedrock mantle claude-opus-5-5 cost tracking cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add bedrock mantle gpt cost tracking cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_two_identical_cache_hit_rejects_write_matching_spend_rows passed every run with the skip removed, so the dropped spend row it was guarding against no longer reproduces
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): streaming responses hand the event loop back every 50 ms instead of holding it for the whole buffered stream
* fix(proxy): yield to the event loop on every streamed chunk, no hold budget or clock
* docs(proxy): say why the streaming loop yields on every chunk
* fix(proxy): streaming loop yields once it has held the event loop 50 ms, clock read inline
* docs(proxy): streaming loop comment names the 50 ms hold
* Revert "docs(proxy): streaming loop comment names the 50 ms hold"
This reverts commit cbf1371873.
* refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50
* Revert "refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50"
This reverts commit 356454def6.
* docs(proxy): streaming loop comment notes the yield is a floor, not a count of releases
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count
/v1/messages runs clear_tool_uses context edits before it sends an Anthropic
body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted
the body unedited, so a request with older tool results cleared counted about
2.8x what inference bills.
The count now runs the same history and context-edit step inference runs,
honors additional_drop_params on the deployment, and skips compact edits so a
count never makes a paid summary call.
* fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs
Run compact edits in the count without the summary call so trigger checks and
history slicing match /v1/messages, honor a request-level additional_drop_params,
keep the full body when any deployment in the model group drops context_management,
and answer 400 naming a block Gemini cannot count when an edit trigger fails on it
* fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check
* fix(count_tokens): log a failed edit in the count at debug, since the send already logs it
* fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does
* fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check
The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits.
* fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name
* fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to
* fix(messages): drop an invalid context_management spec under drop_params so the send matches the count
* fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them
The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes.
* test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code
* fix(responses): log and bill every round of non-stream MCP auto-execute
Both model rounds run as internal sub-calls on the request's logging object, and the outer call logs the response the client received with usage, cost, and cost breakdown summed across the rounds. The internal-call gate now also covers the sync success callbacks, so an inner round no longer takes their first-wins slot
* fix(caching): let the parent call log and bill an internal sub-call's cache hit
A round of non-stream MCP auto-execute that the response cache served fired the
cache-hit callbacks on the shared logging object, so the spend log kept round 1's
function_call row and dropped the billed row for the answer. Internal sub-call cache
hits now fire no callbacks and carry a zero cost, and the summed cost breakdown leaves
out rounds served from the cache
* fix(logging): keep a parent row unique when its response came from the response cache
A RAG query or an MCP turn whose final round is served from the cache hands back the cached response's id, which already belongs to the row that cached it, so SpendLogs dropped the new row as a duplicate. Such a row now gets the cache-hit id suffix, and it is logged as a cache hit when it cost nothing.
* fix(logging): skip the zero-cost warning for a response served from the cache at no charge
A non-stream MCP auto-execute request whose rounds the response cache served is priced
at $0 before the parent marks it a cache hit, so it logged a false 'priced at $0'
warning and counted in litellm_zero_cost_requests_total. The diagnostic now treats a
response the cache served at no charge the same as a marked cache hit
* fix(responses): bill the paid first MCP round when a later step of the request fails
* fix(responses): log each MCP tool call under its own id so a failed follow-up keeps its spend row
The non-stream auto-execute path passed the request's call id to the tool call, so the tool's spend row and the request's failure row shared one request_id and the failure row was dropped on insert, leaving the paid first round unbilled.
* fix(responses): bill round 1 on a client disconnect, keep MCP tool rows in the request's session, and charge a fixed margin once per request
* fix(responses): keep bridged MCP auto-execute in one session and trace and continue its conversation
The bridged follow-up no longer chains on round 1, so a thinking model keeps its reasoning and the request stays in one session. The tool call log continues the request trace without renaming it, the tools are fetched once, and a later turn finds the MCP turn under the id the client got, skipping the tool rows.
* fix(responses): log the non-stream MCP tool listing in the request's session and trace
* fix(responses): find a bridged MCP turn by the id the client holds and log its listing in the bridged trace
* fix(responses): derive the MCP tool row id from the request's call id
The tool call's spend row now uses <call id>_<tool call id>, so it stays
linked to the request without colliding with the request's failure row.
Also tighten the internal cache-hit test to a fixed clock and typed params.
* fix(responses): bill a retried MCP attempt for its own rounds
A router retry or fallback reuses the request's logging object, so a failed attempt's stashed round 1 usage made the next attempt's success row log stale tokens. Each non-stream MCP attempt now starts without that stash.
* fix(responses): bill every failed MCP attempt's paid first round on the request failure
A retry or fallback now sets a failed attempt's stashed round 1 aside instead of dropping it, and the proxy's failure row adds it to the last attempt's own, so a request whose later attempt fails on its first call still bills the round the provider charged. A cached final round also keeps the paid first round's cost breakdown.
* fix(responses): keep MCP tool catalog and results out of the bridged next-turn replay
* fix(responses): keep MCP tool spend, client body keys, and Langfuse traces working on auto-execute
Count the MCP tools an auto-executed Responses API call ran in the tool spend rollup, accept a client body key named previous_response_client_ids without a 500, and point MCP listing and tool spans at the trace Langfuse gives the parent request, including x-litellm-session-id requests
* fix(responses): join MCP spans to the Langfuse trace when the request sends body metadata
* test(integration): cover non-stream MCP auto-execute billing, failure rows, and bridged follow-ups
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): add tier 1 cost tracking cases for gpt-5.6-terra, gemini-4-argon and gemini-3.1-pro-preview
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add audit 3 shape cases for vertex_ai and azure_ai claude
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): mark azure gpt-5.6-terra priority case as known bug
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep reasoning, thinking blocks and the json_mode finish_reason on fake streams
A Bedrock Converse model without native structured output fakes the stream
when response_format carries a json_schema. The fake stream transformed the
response without json_mode and rebuilt a chunk from text, tool_use and token
counts only, so it dropped reasoning_content, the signed thinking_blocks and
the reasoning token count, and ended on finish_reason tool_calls.
The three fake-stream sites now pass json_mode to _transform_response and
stream the resulting ModelResponse through the shared
convert_model_response_to_streaming converter, which now numbers each
streamed thinking block with its index the way real streams do
* fix(bedrock): parse text tool calls on Converse fake streams like the non-stream answer
* fix(responses): stream the answer text when one chat chunk carries reasoning and the answer
* fix(streaming): keep every field on the OpenAI and openai_like fake streams and split multi-block thinking
Azure o-series and openai_like fake streams now use the shared MockResponseIterator, so extra tool calls,
reasoning and cached and reasoning tokens survive; the Bedrock-local iterator is deleted.
The /v1/responses bridge also splits a chunk whose thinking block is signed or redacted with no reasoning
text, and the sync iterator closes the reasoning item before the answer opens, like the async one.
The /v1/messages bridge gives each thinking block in a chunk its own content block, starts a new block when
the thinking block index changes, and streams redacted thinking as redacted_thinking.
Snowflake replays drop the stream index from thinking blocks.
* fix(anthropic): stream each thinking block's text once on /v1/messages
An unsigned thinking chunk now opens its content block with empty text, since its thinking_delta carries the text. A multi-block chunk keeps each piece's thinking block, so an unsigned block followed by a signed one keeps its own text and index.
* docs(anthropic): describe which thinking block start carries its text
* fix(streaming): close merged reasoning before the answer when one chunk carries both
* test(streaming): cover fake-stream reasoning across the proxy bridges and type the Snowflake replay test
* fix(proxy): delete callbacks configured under callbacks or failure_callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert on configured callback rows and use a callback absent from CI configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): follow logging suite conventions for callback delete tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): flatten remaining callbacks with a helper generator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type callback delete helpers to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): save callback config without mutating the loaded dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): delete callbacks from every litellm_settings list GET shows
POST /config/callback/delete only searched success_callback while
GET /get/config/callbacks also lists failure_callback and callbacks, so
the Admin UI Delete returned 404 for those rows
Remove the name from every list that holds it, matching entries
case-insensitively, and skip lists the config file owns when the
database owns another copy so the save is not refused
Replace the e2e test with an integration test: the shared e2e stack
config owns litellm_settings.callbacks, so it cannot exercise that list
* test(proxy): cover config-owned success_callback on callback delete and wait on runtime state
Add the mirror mixed-ownership case where the config file sets
success_callback and the database owns failure_callback, which main
refused with a 400, plus a string-valued list case
Poll GET listings and /active/callbacks instead of reading them once, so
a request served by a worker that has not synced yet cannot fail the
test, and reuse the existing callback list key map in the route
* test(proxy): annotate callback delete test helper locals with Final
* fix(proxy): save callback delete through the config it read so concurrent settings edits are kept
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): stop queueing budget increments for Redis when no Redis is configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert Redis-less model budget limiter tracks spend locally without queueing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): keep turn_detection.type when a transcript guardrail disables auto-response
* test(realtime): type the GA session.update helper and mark its locals Final
* test(realtime): expect typed turn_detection on the voice guardrail session updates
* test(realtime): accept either gate path for the client's first voice session.update
* fix(gemini): drop VAD tuning when the guardrail turns auto-response off
* fix(gemini): check create_response without a redundant isinstance
* fix(vertex_ai): keep the client's VAD tuning on Vertex when auto-response is off
* fix(realtime): keep the transcript-guardrail gate typeless on xAI so it does not auto-answer
* fix(realtime): replace any client turn_detection with the untyped gate on xAI
xAI answers on its own under a typed server_vad even with create_response off,
so a client's typed turn_detection must not reach xAI while the transcript
guardrail gates the turn.
* fix(realtime): keep a client's early turn_detection and hide the guardrail cancel's error
A session.update sent before session.created reached OpenAI ahead of the proxy's own gate, so the gate replaced the client's turn_detection. The proxy now re-sends the client's gated turn_detection after the gate when the two differ.
The guardrail's response.cancel now carries a litellm_guardrail_ event_id, and the error OpenAI returns for it (response_cancel_not_active when nothing is answering) no longer reaches the client.
* fix(realtime): send the guardrail gate before a client's early session.update and let Muse and Chirp fall back to server_vad under drop_params
* fix(realtime): hide only the cancel errors that echo the guardrail's own response.cancel ids
Track the event ids of the guardrail cancels this session sent and drop a backend error only when its
error.event_id is one of them, so a client's own event id with the same prefix still gets its error.
Cover the scripted response.cancel reply in the integration upstream, update the stale list-mode test
to the typed gate, and mark the xAI cancel error cell as an expected failure, since xAI replies with
its own event id.
* test(integration support): deliver the disconnect a real server sends after the client goes away
The scripted upstream now keeps reading frames after a reply hits a closed socket, so the unit test's
fake ASGI receive has to end with the websocket.disconnect message a real server delivers.
* fix(realtime): delete the blocked transcript item before the guardrail's refusal turn
OpenAI GA answered the blocked question in some refusal turns because the
blocked user item stayed in the conversation. The raw OpenAI-protocol path
now sends conversation.item.delete for it right after the cancel. Provider
config backends have no such event, so that path no longer passes an item id.
Also bound the guardrail integration file by the shared graceful-stop
timeout and make the xAI cancel-error xfail require an AssertionError.
* fix(realtime): keep the blocked item on xAI, which cannot delete an audio input item
* test(realtime): assert the xAI voice block sends no conversation.item.delete upstream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(volcengine): add Doubao-Seedance video generation via the OpenAI /v1/videos surface
New VolcEngineVideoConfig maps litellm's OpenAI Videos API onto Volcengine
Ark's contents/generations task API: create hits
/api/v3/contents/generations/tasks and returns a cgt-... task id, status and
content poll /api/v3/contents/generations/tasks/{id}, list maps the same
collection (limit -> page_size, Ark's filter.* through extra_query), delete
maps DELETE on the task URL, and edit/extension reuse create with a
reference_video content item since Ark models both as ordinary generation
tasks. Ark's own edit and extension require ratio adaptive (edit also
duration -1), so those are pinned unless the caller overrides them. Remix has
no Ark equivalent and raises.
The body is flat apart from a typed content array: the prompt is a text item
and every image/video/audio input is a url item carrying a role, which is the
only way to express reference-to-video, edit and extension jobs, so a
caller-supplied content array wins outright. input_reference becomes a
first_frame item, base64 encoded into a data URI for file inputs.
OpenAI params translate per Ark's contract: seconds becomes duration and size
yields both a ratio and a named resolution tier, with the ratio snapped onto
the set Ark accepts (854x480 reduces to 427:240, which Ark rejects) and the
tier taken from the shortest side. Emitting only the ratio leaves the request
on the model default and bills at that default's rate.
Billing reads the delivered duration and resolution rather than the requested
ones, because duration -1 and ratio adaptive both let Ark deliver something
else. Ark bills per output token (tokens = width * height * 24 * seconds /
1024), so the cost map carries per-second rates by resolution tier, derived
from the console's CNY per million token list rates (2.5 70, 2.0 46, 2.0-fast
37, 2.0-mini 23 without a video input) at the repo's 7.0 CNY/USD convention.
Requests that do carry a video input bill at the lower rate instead, which
litellm's per-second tiers cannot express, so those are overestimated.
usage.total_tokens rides along for invoice reconciliation.
Ark echoes the delivered geometry only as a resolution tier plus a ratio, and
the 2.0 family's rendered frame does not follow the tier name: its 16:9 "480p"
canvas is 864x496, not the 854x480 the tier implies, and its other ratios
differ from the tier's nominal shortest side too. A size reconstructed from
the tier alone therefore reports pixels the model never produces and scales
the 480p rate against the wrong pixel count, so both the reported size and
the 2.0 480p rates come from the measured frame (a per-family table, sourced
from the Seedance 2.0 Mini model card and re-verified against a live gateway).
The create response reports the requested duration and resolution, since only
the create, edit and remix calls are costed and Ark's create response carries
no usage block, so without it every video logs zero spend. A caller that
leaves seconds off is not free either: Ark runs its own default length, which
is 5s for the 2.0 family, so that is what the create response bills, while
the 2.5 family's default is -1 (Ark picks) and stays unbilled until the
status response. The create response also echoes the requested seconds and
the frame the job will be rendered at.
Editing and extending need the source video as a URL: Ark's task query is a
GET while litellm's edit prefetch hook only POSTs, so a task id cannot be
resolved into one. The model comes from an explicit extra_body.model, else
the encoded video id, else the model the router already resolved, with
litellm's provider prefix dropped.
The credential's api_base is the Anthropic-compat surface
(/api/compatible), so the base is cut at the first /api/ marker rather than
by a /api/v3 suffix before the video paths append their own version.
The cost-map schema test's exception list gains the 2.0 4K tier, whose
$1.28/second rate exceeds its $1 sanity bound, alongside the existing
runwayml/seedance2 entry.
* fix(volcengine): answer 4xx when a Seedance video is downloaded before it finishes
Downloading a task that is still queued or running raised a bare ValueError, which the
video content handler synthesizes into a 500. The proxy then retried the same poll twice
and reported the caller's own early download as a gateway fault, and the router went
looking for a model group fallback for a job that was simply unfinished
The download path now raises VolcEngineVideoError carrying the status the OpenAI videos
API documents for content that is not ready (404), 400 for a task the provider reported
as failed, 404 for an expired task record, and 502 for a succeeded task with no artifact.
None of them are 5xx, so the router stops retrying them
Regression tests drive avideo_content and assert the caller sees NotFoundError or
BadRequestError instead of an internal server error
* fix(volcengine): clear the type-discipline and override findings on the Seedance video path
The module annotated four mutable collections and carried five `# mutable-ok` markers on
lines that suppressed nothing, so the type-discipline gate counted LIT001 x4 and LIT013
x5 against the PR. The content-item and task-dump helpers hand back read-only views of
what they build, so they now say Mapping. VideoObject.error is declared as a dict by the
type it feeds, so that one marker moves onto the annotation it actually suppresses
transform_video_status_retrieve_response also dropped the client keyword the base class
declares, which basedpyright reported as an incompatible override
* fix(volcengine): bill every Seedance video at create and keep its polls on the deployment that made it
Ark bills a task when it finishes while litellm bills it at create, so a request
whose cost is not known then is rejected: Ark-picked length (duration -1), frames,
a video input, and with it edit and extension. An omitted length is sent as 5s,
since the 2.5 family otherwise lets Ark pick it and logged $0.
Video ids now carry the requested and the served model group, read from
litellm_metadata, so polls and downloads go back to the deployment that created
the task. The routing and input_reference helpers move out of the DashScope
config into a shared module both providers use.
A download before the render finishes answers 400, unreadable or unmapped task
bodies answer 502, malformed seconds, size, parameters and Ark fields raise
unless drop_params is set, an unpriced resolution tier is refused, and the
rendered frame table moves into the cost map.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(volcengine): reject a text item in content so the guarded prompt is the only text sent
Input guardrails scan data["prompt"], so a caller could pass a harmless prompt
for scanning and a different, unscanned text item in content, which this branch
kept and the guarded prompt was dropped. The prompt now always leads the content
array and a second text item answers 400.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(volcengine): route video to VOLCENGINE_API_BASE when the deployment sets no api_base
* fix(volcengine): let a model with no tier prices keep its resolution instead of rejecting every tier
* fix(volcengine): answer 400 for malformed or unknown create fields, send a 720p default tier, and stop the list cursor
* fix(volcengine): map size to the closest tier by pixel count, reject draft, remix and list cursors, map user to safety_identifier, read ARK_API_BASE
* fix(volcengine): drop unsupported create values under drop_params, take OpenAI's object input_reference, default to 4 seconds, price endpoint ids by base_model
* fix(videos): unwrap nested file tuples without recursion
* fix(volcengine): price Seedance tiers at Ark's own per-resolution token rates
Ark prices 1080p and 4k per token above and below the 480p/720p rate, so the
single 46 CNY per million tokens the rows were derived from billed 2.0 1080p
and 2.5 1080p about 10% low and 2.0 4k 77% high. Ark's per-resolution table
and its own 16:9 examples agree: 1080p 2.48 CNY/s, 4k 5.05 CNY/s for 2.0 and
3.74 CNY/s for 2.5 1080p, which is 51, 26 and 77 CNY per million tokens.
Rewrites the tier test to check each rate against the rendered pixels at that
tier's own price instead of assuming every tier scales from the base rate,
and drops the 4k cost exception that only existed for the old rate.
Co-Authored-By: Bytechoreographer <noreply@bytechoreographer.com>
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(volcengine): unpack Ark parameters sent through extra_body
* fix(volcengine): ignore the extra_body keys litellm maps itself
The shared request builder merges the caller's raw extra_body back over the
mapped params, so a key this adapter maps itself comes back a second time as a
field of its own, and the unknown field check turned the supported request
into a 400. 2aefc18b0a covered parameters; size, seconds and user still failed
that way.
All three are already mapped from the top-level copy the same builder writes,
so the leftover is ignored rather than rejected. characters and image stay
rejected, since Ark has no equivalent and accepting them would hide a
parameter that never took effect.
Co-Authored-By: Bytechoreographer <noreply@bytechoreographer.com>
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(volcengine): reject litellm request fields inside parameters and accept litellm's own controls
A seconds, size, user, prompt or nested parameters inside an Ark parameters object, and an extra_body copy of
prompt, model or extra_headers, answer 400 again unless drop_params is on, instead of being ignored. A user that
is not a string answers 400 the same way. drop_params, additional_drop_params and a forwarded organization are
no longer rejected as unknown Ark fields, and additional_drop_params removes the fields it names.
* fix(volcengine): apply additional_drop_params before mapping request fields
A drop list naming user, seconds, size, parameters or an unsupported field such as image had no effect, because
the fields were already mapped to Ark names before the list was applied. The list now removes them first, so the
request sends what it would send if those fields were never set
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: zijie.zhou <zijie.zhou@momenta.ai>
Co-authored-by: Bytechoreographer <noreply@bytechoreographer.com>
* feat(router): continue chat streams on fallback after a mid-stream break
A chat-completions stream that breaks after content has been delivered was
re-raised and the fallback deployment never ran (#40404), while the
Responses-API path already continues via a prefilled assistant turn. Add the
same for chat completions behind `enable_mid_stream_fallback_continuation`
(opt-in, async-only), so a post-content break re-enters the fallback chain with
the partial text as an assistant prefill instead of surfacing the error.
Continuation is only attempted when it is safe: the stream emitted plain
assistant text (no tool/function calls, thinking blocks, reasoning items,
audio or images), the request is not constrained output (response_format or a
forced tool_choice) and not merge-reasoning mode, and the fallback target's
model supports assistant prefill. The target check runs as a deployment
pre-call filter, so a chain with no prefill-capable deployment empties and the
original error is surfaced rather than a duplicated or rejected request being
sent. Plain reasoning_content is treated as out-of-band and does not block a
continuation, matching the Responses-API path; Anthropic thinking is excluded
because its signed thinking blocks cannot ride a text-only prefill.
CustomStreamWrapper now latches whether a disqualifying delta was streamed and
carries it on MidStreamFallbackError so the router can decide without
re-scanning chunks. Default behavior is unchanged.
* fix(router): harden mid-stream continuation eligibility and prefill folding
Address review of the mid-stream continuation:
- keep response_format={"type": "text"} eligible; only json_object / json_schema
and other structured formats decline, since "text" is the unconstrained default
- fold a new partial into an existing trailing assistant prefill instead of
appending a second one, so a nested mid-stream break stays a single prefill
turn even on providers that do not merge consecutive assistant messages
- split the deployment prefill-capability lookup into two readable steps
- cover the merge_reasoning decline branch, the text response_format pass, and
the prefill-folding path with tests
* fix(router): address review of mid-stream continuation
- register the continuation deployment filter as a process-global singleton
instead of tracking it per router, so discarding one router no longer removes
the type-deduplicated filter that other live routers still depend on
- disqualify reasoning_content: it reaches the caller as visible reasoning that
a text-only prefill cannot carry, so a reasoning-capable fallback would
re-derive it and produce an incoherent response
- move MID_STREAM_CONTINUATION_KWARG to constants and lazy-import the filter
class in the router, breaking the module-level import cycle CodeQL flagged
- trim the added docstrings to the repository comment policy
* fix(router): keep mid-stream continuation within the type-discipline budget
The LIT* type-discipline gate counts mutable-collection use the ruff pass does
not see. Annotate the reads with read-only Sequence/Mapping views, compare
response_format against a frozen MappingProxyType sentinel, and mark the
genuinely mutable litellm message and deployment-list boundaries with
mutable-ok reasons, so LIT001 and LIT002 stay under their limits.
* fix(router): clear basedpyright operator error and the import-cycle alert
- narrow the accumulated stream delta content to str before appending, so the
shared helper no longer trips reportOperatorIssue (str += object)
- keep the continuation marker in the pre-call-check module and lazy-import it
at the two router call sites instead of routing it through constants, so the
router has no module-level dependency that closes an import cycle
* fix(router): make the mid-stream continuation marker unforgeable
The deployment filter is process-global and read the continuation marker as a
truthy value from request_kwargs. Since the proxy can forward arbitrary
request-body fields into the router, a client could send
`"_mid_stream_continuation": true` on a normal request to filter a mixed model
group down to prefill-capable deployments and steer their prompt onto that
provider.
Set a private sentinel object internally and check it by type instead of
truthiness: a JSON request body cannot construct one, so a forged flag is
ignored. Adds a regression test that forged client values (true, "true", 1, a
dict) leave the deployment list untouched.
* feat(router): honor per-deployment model_info for continuation prefill capability
The deployment filter only read supports_assistant_prefill from the cost map, so
a model configured with `model_info: {supports_assistant_prefill: true}` on the
deployment (the usual way to declare a capability for a model that is not in the
cost map, or to override it) was wrongly skipped. Check the deployment's
model_info first (True or False both authoritative), then fall back to the cost
map. Adds tests for the override in both directions.
* fix(router): bound mid-stream continuation to the caller's output allowance
A continuation handed the fallback the full max_tokens / max_completion_tokens
even though part of the answer had already streamed, so the total could exceed
the caller's limit and repeated interruptions could re-grant the allowance on
every hop, amplifying spend. Reduce each configured ceiling by the tokens already
emitted before re-entry, and decline the continuation (surface the original
error) when the allowance is exhausted. The reductions compound across hops
because the trimmed ceiling is what the next hop sees.
Also add the missing explanatory comment on the model_info fall-through except.
* refactor(router): extract mid-stream decline to stay within the complexity budget
The output-ceiling branch pushed stream_with_fallbacks past the C901 limit. Both
decline paths (ineligible after content, and exhausted allowance) re-raised the
original provider error the same way, so pull that into
_raise_original_mid_stream_error. It removes the duplicated branching, drops the
function back under the complexity ceiling, and reads clearer.
* fix(cost-map): mark Claude Sonnet 4.6 as not supporting assistant prefill
The Anthropic API and Bedrock both reject an assistant-last message on
claude-sonnet-4-6 with "This model does not support assistant message
prefill", so the continuation filter must not route a mid-stream
continuation to it. Flip the flag on every claude-sonnet-4-6 entry and on
databricks' opus-4-6 entry, whose anthropic sibling already says false.
The prefill check tests now register synthetic cost-map entries instead
of pinning real models, which broke when claude-3-opus left the map.
* refactor(router): type the streaming fallback kwargs and drop narration comments
Removes an unused mutable-ok marker (LIT013) and the explanatory comments
Greptile flagged, and types initial_kwargs on both streaming fallback
iterators as dict[str, Any] like their Responses siblings so the new
continuation helpers take a typed mapping and the basedpyright budget
stays at or below main.
* fix(router): run post_call_rules over the joined text of a continued stream
Each provider stream wrapper validates only the fragment it produced, so a
rule that trips on the primary's prefix plus the continuation's text never
fired. The router now feeds litellm.post_call_rules the emitted text plus
every continuation chunk before yielding it. Also drops a narrating
test-helper docstring.
* refactor(router): read a deployment's declared prefill support without an empty except
* test(router): cover _raise_original_mid_stream_error directly for the router coverage gate
* fix(router): fold the continuation prefill into a trailing assistant turn
A request that already ends in an assistant message took the partial output as a second assistant turn, which prefill providers reject. The partial now extends that turn in place, with prefix set, so the continuation request never ends in two assistant messages.
* fix(router): keep content blocks on a trailing assistant turn when extending it with the partial output
A trailing assistant message whose content is a list of blocks was stringified before the partial output was appended, so the fallback received the Python repr of the blocks as its prefill. The partial now lands as one more text block, and a content shape that cannot be extended surfaces the original stream error instead of a garbled continuation.
* fix(router): continue the trailing text block in place when extending a prefill with content blocks
Anthropic strips trailing whitespace per content block, so appending the partial output as a new text block after an assistant turn that ends in "Sure, " would send the prefill as "Sure,here". The continuation now extends the last text block itself and only opens a new text block when the trailing block is not text (an image, say), so the prefill reads exactly as the text the caller already received
* test(router): type the prefill check's deployment helper
* fix(router): keep a continued stream on the primary's identity and widen what can continue
Continuation chunks now keep the primary stream's id and created time and
drop the fallback's second role delta. Requests with n > 1 no longer continue.
Prefill support reads model_info.base_model and litellm_params.base_model
before the deployment model. enable_mid_stream_fallback_continuation is
runtime-updatable through update_settings and registers the prefill filter.
The bridge rebuild of MidStreamFallbackError keeps the inner stream's
disqualifier. A fallback that repeats the prefill is not doubled in the
continuation input.
* fix(router): decline mid-stream continuation after a content filter, merged reasoning, or a plain trailing assistant turn
Also count the partial stream's completion tokens in the continued stream's usage, accept string
values for enable_mid_stream_fallback_continuation, and list the new router helpers in the
router coverage check
* fix(router): drop re-emitted whitespace at the continuation join and sum both legs' usage consistently
* fix(router): keep the full prefill so a nested break keeps the separator
* fix(router): bill both calls of a continued stream and fix Bedrock Converse, Databricks, and Vertex DeepSeek prefill routes
* test(router): integration cells for mid-stream fallback continuation
Covers the continued stream the caller gets, the prefill turn and reduced
max_tokens the fallback receives, the prefill-capability filter, the runtime
toggle, repeated requests through the response cache, and the spend row of a
continued stream
* fix(router): log a continued stream under the caller's id, price each cut on its own prompt, and keep prefill flags per deployment
* fix(router): price and log each cut leg before its continuation reuses the logging object
Price the cut stream's partial and let its failure log finish (bounded) before the fallback
overwrites a shared proxy logging object, and skip continuation when an output guardrail is
registered or a request fallback carries its own messages.
* test(router): integration cells for the output-guardrail and request-fallback declines and the cut call's failure log
Cover a registered post_call guardrail and a request fallback carrying its own messages keeping the original error, an input-only guardrail leaving continuation on, and the cut call's failure event naming its own deployment and price.
* fix(router): skip mid-stream continuation when a request fallback overrides request params
A request-body fallback entry that sets anything besides model (stream, messages, max_tokens) changes the
request the prefill continuation relies on, so a stream: false entry ended the answer with no error.
* fix(router): skip mid-stream continuation for a fallback group target that overrides request params
The override check read only top-level fallback entries, so a target inside a group's list, such as {"primary": [{"model": "backup", "stream": false}]} in the request, the config, or a key's router settings, still ended the answer with no error.
* fix(router): check only the request's own fallback chain for request param overrides
Scanning every group's chain turned continuation off for all requests when one unrelated group's target overrode a param. Resolve the chain the router will use for this request, the same way the fallback path does.
* fix(router): keep the original error when request fallbacks are malformed
The request-param override check resolved a standard-shape chain outside the router's own error handling, so an empty entry or a non-list chain raised StopIteration or TypeError and replaced the original mid-stream error. Skip empty entries and treat a non-list chain as no chain.
* fix(router): read deployment-configured constraints, decline deployment-id fallbacks without prefill, strip only the last Bedrock text block
* fix(router): read a deployment's tool_choice ahead of router defaults in the continuation check, and restore the renamed prefill test names
* fix(router): filter continuation deployments on every routing strategy, read extra_body and fallback deployments' own params
The deployment filter now also runs on the sync selection path that usage-based-routing v1 and custom strategies use,
one predicate reads n, response_format, tool_choice and merge_reasoning_content_in_choices with extra_body laid over
them, fallback deployments whose own config constrains the output are skipped, and continuation declines when the cut
deployment's id is not in the router (per-request credential deployments)
* fix(router): decline a cut mid-echo, reduce extra_body output limits, and check post_call_rules once on nested continuations
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): scope session history lookup to the referenced key
* Implement SQLite-backed DB for session tests
Added a SQLite-backed database class for testing and a new test to ensure session isolation between different API keys.
* fix(proxy): honor caller litellm_session_id in spend log session_id
Updated docstring to clarify session ID resolution logic for spend log rows.
* test(proxy): caller litellm_session_id wins over the trace id
* style(proxy): fix blank lines around new tests
* fix(proxy): read the caller session from the logging payload for spend logs
* fix(responses): match user and team too when replaying session history
* fix(proxy): keep bridged Responses chains whole and scope session history by owner
* fix(proxy): keep a metadata session next to a trace header and let session headers win
* fix(proxy): send the trace-header session only to spend logs and let session headers win beside a trace header
* fix(proxy): drop a client-sent caller session and keep it out of requester metadata
* fix(proxy): keep the trace id as the spend log session when the caller session contains a NUL byte
* fix(responses): keep a continued bridged chain's session under request_correlation_in_logs
With the flag on, the caller session outranked the chain session the bridge sets for a continued Responses chain, so a chain whose session changed or first appeared on a later turn split. The continuation now logs under the chain's session and drops the turn's caller session key, as on main. Adds the /audit integration cells for the spend log caller session.
* fix(responses): log a continued bridged chain under its session through the spend-log key only
The previous commit set the chain session as the turn's litellm_session_id, which also moved the OTel conversation id of a continued turn away from its trace header. The bridge now writes the chain session into the internal spend-log session key of the metadata the spend log reads, so only the spend log session changes, as on main. Adds an OTel cell for a continued chain.
* test(spend): cover bridged chains that share one caller session
* test(spend): cover the caller session on configured pass-through routes
* test(spend): type the SQLite fake and split stacked comprehensions in the caller session tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys
A settlement claim is now a lease, so a row whose settler died between claim and
outcome is claimable again after the lease and boot resumes it. A resumed poll
gets the full poll window and fetches with the serving deployment's credentials
from the live router. The stored create context keeps the request tags and the
request metadata that the creating worker's own spend row used
* fix(interactions): bill a lapsed claim's takeover only when no spend row exists
A settler that billed and then died, or failed to record the outcome, left the row unsettled, so the replica that took over its lapsed claim billed the interaction a second time. A takeover now looks up the spend row keyed by the interaction id and records the row as billed when it finds one.
* fix(interactions): settle Vertex AI and edited deployments, never double bill with spend logs off
A takeover with spend logs off records the interaction unsettled instead of
billing it again, and a release records its outcome before freeing the budget
reservation. A resumed or live fetch carries the deployment's Vertex AI project,
location, and credentials, and a stale deployment id falls back to the model
group only when one same-provider deployment serves it. A poll stops at the
first 400, 401, 403, or 404, string request tags no longer fail the create,
and settled_at gets an index built concurrently.
* fix(interactions): leave a live replica's interaction to it when a resumed poll cannot read it
A replica that boots with credentials unable to read an interaction got a 404 on its resumed poll and settled the row as unsettled at once, so the replica that created it, still polling, found it settled and never billed it. A resumed poll that gets a 400, 401, 403, or 404 now stops fetching and keeps checking the row until its poll window ends, and settles it as unsettled only when no other replica did.
* fix(interactions): keep the creator's metadata on a resumed bill and retry outcome writes inside the claim lease
A bill another replica writes now carries the metadata the creating worker's own
spend row is built from, so a client's metadata.model_group no longer relabels it.
A failed outcome write is retried for half the claim lease before the budget
reservation is left to a takeover, and the creating replica retries every fetch
error for its whole window again, as main does. The resume integration case now
checks that a resumed fetch uses the deployment's own key.
* fix(interactions): resume background interaction polls only after database models load
Boot resume ran before the proxy loaded the models stored in the database,
so a resumed poll for a model added through /model/new or the Admin UI
fetched with env credentials. A 4xx there stops a resumed poll, so the bill
was lost. The settlement now installs after the database models load.
The integration test resumes a database model with an eager first fetch
and fails with the old startup order.
* fix(interactions): renew a settling claim so a slow settle is never taken over and billed twice
A settle runs every success callback in turn, and one slow callback could
hold it past the 5 minute claim lease. A booting replica or a DELETE then
took the live claim over, found no spend row yet, and billed the
interaction a second time. The settler now renews its claim every third
of the lease until it records the outcome, so only a settler that stopped
renewing is taken over.
A takeover also read the spend log after writing its claim, so a failed
read left the row under the taker's own fresh claim, which its poll then
took for another settler's and left unbilled until a reboot. The spend
row is now read before the claim is written, so a failed read takes
nothing and the claim is retried on the next poll.
* test(spend): cover lapsed claim takeovers, rotated deployment keys, late restarts, and resumed bill tags
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set
* fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails
* fix(proxy): log a prompt-only token count fallback as a one-line warning
* fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI
* fix(proxy): count a token_counter prompt sent next to an empty messages or contents list
* fix(proxy): count a prompt next to messages on the local token count instead of answering 500
* fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set
With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form.
* test(proxy): count tools with a prompt in the token counter fallback test
* test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests
* test(proxy): name the token counter fallback test by what it checks
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model_hub): surface model_info.description in model group info and Model Hub UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): aggregate model group description without in-place mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional
completion() only stores a custom prompt template key when its value is
truthy, and custom_prompt() already defaults both prompt values to "".
Seven call sites read them back with direct indexing, so a template that
sets only roles raises KeyError instead of rendering, on ollama, petals,
vllm, predibase, codestral, anthropic text completion, and
response_schema_prompt. Five other providers already read defensively, so
the same config works on sagemaker and 500s on ollama.
Read both keys with .get(key, ""), matching those five providers and the
defaults custom_prompt() already declares. Templates that set all three
keys render exactly as before.
* test(prompt-templates): drop the explanatory docstrings from the optional-keys regression test
* fix(health): skip background health check DB writes when the latest-row read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt-templates): read every custom prompt template field through one typed helper
Templates that omit roles, initial_prompt_value, or final_prompt_value now render with empty defaults across every provider that reads them, instead of failing with KeyError or TypeError. A field with the wrong type fails with a validation error naming it.
* fix(prompt-templates): keep templates that rendered on main working and fail bad fields with a 400
Only the roles a request uses are checked, null fields mean unset, and extra role keys are ignored, so every template main rendered renders the same. A wrong-type field now raises a BadRequestError naming it. initial_prompt_value is declared litellm-owned so it stays out of provider request bodies, like final_prompt_value. The template tests move into the mapped factory test file, and the predibase test asserts the real rendered prompt
* fix(prompt-templates): keep bad-template failover and render non-mapping role entries like main
A wrong-type template field now raises a 500 InternalServerError naming the field, so the router
still retries and cools down the bad deployment. A roles value or role entry that is not a mapping
renders unwrapped as it did on main, and role keys are only checked for roles a request uses.
* fix(prompt-templates): warn when a custom prompt template roles value is not a mapping
* fix(prompt-templates): keep deployments in rotation on a bad template and reject a template with nothing to render
* test(integration): audit custom prompt templates across endpoints, providers, and chaos
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): stop skipping rate-limit checks when token counting fails
Router._pre_call_checks() caught any token_counter() exception and
returned the full, unfiltered deployment list, bypassing the RPM,
region, and invalid-params checks for every deployment - not just the
context-window check that actually depends on the count. A content
block token_counter() cannot parse (e.g. an invalid image_url detail
value) silently disabled rate-limit enforcement for that request, with
only a log line and no error surfaced to the caller.
Now a counting failure only disables the context-window check; the
other checks still run for every deployment as before. Input-token
resolution is pulled into its own method and computed once before the
loop, so nothing needs to track "did counting fail" with a mutable
flag across iterations.
* test(router): cover the async region check when token counting fails
Replace the unit test of the private input-token resolver with a regression
test through async_get_healthy_deployments, the path the proxy uses: an
EU-pinned request whose content cannot be token-counted must not get the US
deployment
* test(router): cover the param filter when token counting fails and satisfy router coverage
* fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params
The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params,
drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every
available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away
instead of a retried 429. Deployment context windows are resolved once per call.
* fix(router): keep the retried 429 when a cooling-down deployment takes the param
* fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks
* fix(router): steer by deployment capability, apply drop settings after region and RPM checks
A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing
* test(router): make the github_copilot pre-call filter test hermetic in CI
* test(router): cover pre-call region and response_format filters through the proxy
* fix(router): judge the 400 against the request's own deployments, not a name lookup
* fix(router): judge the 400 against the passed deployments when the group name lists none
* fix(router): let a request's drop and allow settings replace the deployment's in the param check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(gemini): allow disabling thought summaries while keeping thinking on
Both reasoning-effort mappers hardcoded includeThoughts to True whenever
thinking was on, so thinking ON with summaries OFF was unreachable even though
that is the google SDK's own default. Summaries are the narration rather than
the reasoning, and for callers that discard them they are pure added latency.
Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing
changes for anyone who does not set it. It gates the includeThoughts=True
branches in the thinkingBudget mapper, the thinkingLevel mapper, and the
anthropic-style thinking param mapper.
The disable and none paths still force includeThoughts=False: with thinking
off there is nothing to summarise.
* test(gemini): assert thought-summary flag through the outbound Gemini request
* feat(gemini): let a request's thought summary ask override the gateway default
thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing
* feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default
* fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag
Normalize gemini_include_thoughts_default through normalize_drop_params so a
YAML or env value written as "false" or "0" turns thought text off and a
null or unknown value keeps the default on. thinking.display "omitted" no
longer counts as a per-request ask, so the gateway flag decides, matching
/v1/messages. Adds proxy integration cells for chat, /v1/messages and
/v1/responses that prove the flag, every per-request ask, streaming, and a
burst across workers.
* test(gemini): match the streamed Responses usage, which omits null token details
* feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks
* fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias
A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply
* fix(caching): key the response cache on the summary alias wherever the request carries it
The cache key now reads reasoning_summary and reasoningSummary through the same
reader as the summary fold, so an alias inside extra_body gets its own key too
* test: type the new summary-ask unit tests and mark their locals Final
* test: annotate the remaining summary-ask unit tests the PR touches
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): track and drain management audit log writes on shutdown
* fix(proxy): track hook spawns and defer registration in audit log drain
* fix: satisfy LIT006 on audit log task registry by documenting the required cast
* fix(proxy): satisfy LIT006 and LIT010 in audit task registry
* test(proxy): drive the audit drain tests with events instead of wall-clock waits
* fix(proxy): drain audit writes beside the shutdown flushes and track only premium writes
Run the audit drain concurrently with the gateway and request-error flushes so
the two waits overlap inside the teardown budget, register the running audit
task only after the premium check, drop the four duplicate task-failure
loggers, and cover the shutdown order and a real hook spawn with tests
* fix(proxy): start hook audit writes before their email and secret manager calls
Key and user hooks now spawn their tracked audit write first, so a slow
invite email or secret manager call no longer spends the drain budget
before the write starts. The endpoints stop tracking the hook tasks, so
shutdown never waits on hook I/O when audit logging is off.
Adds regression tests for the hook order, the failed block or unblock
update, and the shutdown drain without a database.
* fix(proxy): track only the audit insert, not the code that awaits it
The shutdown drain now waits on each audit row insert itself, so it never waits on a caller's later work and covers every caller without per-endpoint wrappers. The user/new hook still tracks its own task because it reads the user row before the insert
* test(proxy): drain the audit registry in the config audit tests
The audit insert now runs in its own tracked task, one hop after the caller's task, so a single yield no longer reaches it
* fix(proxy): name dropped audit rows in the drain timeout warning
Each tracked audit task is named after its action, table, object id and destination, so a shutdown that drops a row says which one. Adds tests that the drain waits for slow audit callbacks and that the warning names the dropped row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(ui): look up a team member's budget row once per render
TeamMemberTab resolved each row's membership with Array.find() over
team_memberships inside every extra column's render and sortValue, so a
render walked the full membership list once per cell. Build the user_id ->
membership map once per team_memberships change and read from it instead.
* perf(ui): page the member tables instead of rendering every row
MemberTable rendered one row per member, so a team or an org with a couple of
thousand members paid for every row on each render. Its DataTable now runs
client-side pagination, like the other list tables in the dashboard.
* fix(ui): keep the member table on its page when the roster is refetched
* test(ui): cover a team member's spend and budget refreshing after a refetch
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: preserve non-null anyOf branches when stripping null types for Vertex
* refactor: build anyOf branches immutably and drop explanatory comments
* style: apply ruff format
* fix: annotate anyOf list construction for the LIT002 budget
* fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising
* test(vertex_ai): pin empty anyOf passthrough
* fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold
* fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing
* fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): keep user messages with None or empty list content so requests do not end on an assistant turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): avoid mutable empty-list construction in sanitizer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): avoid rebinding sanitizer input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep a trailing user turn whose content key was dropped
litellm.completion strips None fields from every message before the provider
transform, so a user message sent with content null reaches the Anthropic
converter with no content key and raised KeyError. Read the content with get
so that turn gets the same placeholder as an empty list.
* fix(anthropic): only fill user turns with empty content and keep their cache_control
* fix(anthropic): fill each empty user message in place like an empty string
* fix(anthropic): add the placeholder only for an empty turn or a cache breakpoint, in linear time
* test(anthropic): cover the tool-result guard with cache_control and widen the linear-time bound
* test(anthropic): cover empty user turns through the proxy on chat, responses and messages
* test(anthropic): type the empty user turn tests and move the linear-time check to a benchmark
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test
* fix(guardrails): record every content filter match in logging_only and offer the mode in the UI
* style(tests): format the content filter logging_only integration test
* refactor(guardrails): keep the logging_only predicate and scan override within the type gates
* fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning
* test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test
* test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects
* refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach
* test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter
* test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up
* test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): preserve provider request IDs in streaming failure logs
* fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams
Bedrock models that run through the chat-completions bridge (Claude,
converse, invoke) raise in-stream failures from the event-stream decoder
as a BedrockError without headers, so their failure logs kept
error_provider_request_id null. The decoder now passes the stream's
request-id headers to those errors.
Both the native Responses iterator and the decoder pass only the
PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs
pick up the request id and none of the 200 response's other headers.
MidStreamFallbackError gets its response at construction instead of
mutating it afterwards.
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
* fix(proxy): count endpoints return missing-key and request errors in their source API shape
* fix(proxy): drop imports the count error change left unused
* fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope
* fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures
* fix(proxy): count with the global OpenAI and Anthropic keys chat already uses
* fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway
* fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response
* fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads
* fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses
* fix(proxy): mint the OpenAI workload identity bearer inside the count error handling
* fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(bedrock): grant sts:GetWebIdentityToken in web identity session policy for claude_platform
Claude Platform on AWS calls STS GetWebIdentityToken (outbound identity
federation) with the caller's session before serving inference. The
session policy passed to AssumeRoleWithWebIdentity is a permission
ceiling and never granted it, so every bedrock/claude_platform request
using OIDC web identity auth failed with 403 "no session policy allows
the sts:GetWebIdentityToken action", even when the role's identity
policy allowed it.
Add a ClaudePlatformOutboundIdentityLiteLLM statement granting only
sts:GetWebIdentityToken, plus tests that it is granted, scoped to its
own statement, and that no sts:* wildcard is introduced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): also grant sts:TagGetWebIdentityToken for claude_platform web identity
Claude Platform on AWS tags the outbound web identity token it requests, so
the caller also needs sts:TagGetWebIdentityToken. Found end-to-end on a
deployment: with only GetWebIdentityToken granted, calls failed with
"not authorized to perform: sts:TagGetWebIdentityToken on resource:
arn:aws:sts::<account>:self"; granting both made /v1/chat/completions and
/v1/messages (streamed and not) succeed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): only let Claude Platform use the outbound identity STS grant
Scope the ClaudePlatformOutboundIdentityLiteLLM session policy statement with aws:CalledViaLast = aws-external-anthropic.amazonaws.com, so a web identity session can no longer call sts:GetWebIdentityToken or sts:TagGetWebIdentityToken directly. Drop the code comment and fix the stale test messages.
* test(bedrock): fail when the Claude Platform caller condition reaches other statements
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): honor explicit bedrock base model capacity
* fix(router): honor an exact base_model's capacity for every non-Azure provider
Extends the Bedrock-only check to every non-Azure provider and only uses the
base_model when it has its own exact cost-map entry, so an unmapped or wildcard
base_model falls back to the route id instead of raising in the router.
* test(router): call _maps_to_exact_model_info directly in its cost-map tests
* fix(router): let an exact base_model only widen a deployment's context window
A base_model used as a pricing stand-in with a smaller window no longer narrows the route's limit, and a route with no window never takes one from base_model
* test(router): count the base_model window helpers as covered through get_router_model_info
* fix(router): widen only from a model_info base_model naming the same model
A stand-in base_model for a different, larger model widened a smaller
deployment's window, so pre-call checks sent it prompts the provider
rejects with 400. The window now widens only when model_info.base_model
names the same provider model id as the route, which is also the field
/model/info reads.
* fix(router): match a base_model's model id across Bedrock region prefixes and @ qualifiers
* fix(router): read litellm_params.base_model too and look the base row up as written
* test(router): cover a same-model base_model window on chat, messages, responses, and complexity escalation
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(soniox): decode proxy form fields for context, translation and flags
Multipart form fields reach the proxy as strings, so Soniox received
context and translation as JSON strings and booleans as "true".
Decode them at the handler entry point and add typed shapes for the
context and translation request fields.
* fix(soniox): keep bracket-prefixed free-form context and move types under llms
Free-form context such as "[inaudible] consultation" is valid Soniox
input, so a string that fails JSON parsing is now forwarded unchanged
instead of returning a 400. The Soniox request types live next to the
provider code.
* test(soniox): exercise SDK-typed context and translation shapes through the decoder
* refactor(soniox): derive handler options and provider params without mutating a local dict
* chore(soniox): drop explanatory comment on the non-forwarded param set
* refactor(soniox): return invalid form bool as a value and raise once in the handler
* fix(soniox): validate decoded form fields and keep language with hints
Validate structured context before sending it as an object, answer 400 for a
bad polling interval or flag unless drop_params is on, nest SDK bracketed
extra_body fields, keep the OpenAI language at the head of language_hints, and
return the translated text for one-way translation without tags.
* fix(soniox): keep the source transcript, rebuild SDK list fields, and reject malformed JSON values
* fix(soniox): keep cleanup strings whole and drop the unused translation type
* fix(soniox): let explicit language_hints win over language and test the proxy form path
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The operator trace wait stopped at the root and redis spans, but the gen-AI
span is closed later by the async success callback. When it had not landed
yet, the tenant mirror was built without it and the gen-AI assertion failed.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(bedrock-passthrough): return AWS-shaped errors so AWS SDKs read the code and message
A guardrail block on a /bedrock route now answers with x-amzn-errortype ValidationException and a top-level message, and an error Bedrock returned (such as a 429 ThrottlingException) is passed through with its status, x-amzn-errortype and body
* fix(bedrock-passthrough): keep 5xx guardrail failures and failure-hook replacements unchanged
* test(bedrock-passthrough): guardrail vendor failures and unknown models keep the litellm error
* fix(bedrock-passthrough): only answer guardrail exceptions that blocked content as ValidationException
* fix(bedrock-passthrough): pass Bedrock's content-type through with its error body
* refactor(bedrock-passthrough): wrap the guardrail ProxyException instead of copying its fields
Unit tests drive the error response with the real request processor and ProxyLogging
* test(bedrock-passthrough): cover the direct model route, invoke, and auth, budget and rate-limit errors
* fix(bedrock-passthrough): pass through only errors a Bedrock model operation returned
A guardrail service's HTTP error that carries x-amzn-errortype keeps the litellm error
* refactor(bedrock-passthrough): mark Bedrock's errors where the passthrough raises them instead of matching the URL
A guardrail service's HTTP error never goes through the passthrough call, so it keeps the litellm error whatever its URL
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(ui): gate Add Model's Test Connect on the form's own validation
Test Connect stays disabled until the Add Model form would pass validation,
with a tooltip saying why. Provider password, text and textarea inputs now
render empty after a successful add resets the form, and an empty prepared
request shows a plain message instead of a destructuring crash
* fix(ui): validate the Add Model form when Test Connect is clicked
Test Connect stays enabled and, on click, runs the same validation Add Model
runs. An incomplete form now shows the inline required errors instead of
opening a crashing connection test, and a valid form tests as before. This
replaces the disabled-button gate, which read a validity flag that went stale
after programmatic form updates
* test(ui): cover a blank required Azure API Base on Test Connect
* test(e2e): mixed_size_requests_2rps perf case (coding-agent bodies, JWT and key callers, two guardrails)
* test(e2e_harness): test_observe builds DbCounters with statements and expects the background Postgres line
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared disconnect release dispatched to the legacy limiter, which decrements unconditionally, so a rejected realtime attempt freed a slot it never took. The generic release is v3-only again and the Responses WebSocket close releases through its own method
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
