* test(cost): run the azure_ai claude bridge cost parity pairs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add anthropic usage speed and vertex gemini audio priority price
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update update_cache pins for synchronous write-back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): wrap long assert in update_cache tag test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover refill race and pin TTL with injected clock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count global proxy spend for every request with a user, cached on this worker or not
* fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter
The global proxy spend bump read the total, awaited, and wrote old + cost back
from a background task, so a budget reset that deleted the key in between got
the pre-reset total written back, and workers overwrote each other's totals.
RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key
exists, keeping its TTL, and this worker's copy is then set from the shared
total with no await between its read and write. The bump now runs whenever a
proxy max_budget is set, matching the DB writer, and the armed post-call read
and the batch-read declaration it used are removed.
* fix(proxy): report the global spend Redis increment to service metrics and request traces
The global proxy spend increment now emits Redis service success and
failure events under the spend_counters target and passes the request
span through, so it shows as redis.incr spend_counters like the write
it replaced.
The unit tests run the real RedisCache increment against fakeredis
instead of mocks, and a new test checks the span lands under the
request span on success and on a Redis failure.
* test(proxy): keep the spend counter Redis fake off the network
* test(proxy): count increment_if_present as a shared-cache call in the key-family guard
* fix(proxy): keep the newer global spend total when Redis replies arrive out of order
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): read the stream stall cooldown spend rows from the owned proxy's database
The cooldown cells boot an owned proxy, which writes its spend logs to its
own scratch database since owned proxies went private, while the cells polled
the worker's database and never saw a row. Read each row through the gateway
that served the request, so the cells check the proxy they called.
* fix(tests): read passthrough spend rows from the owned proxy's database
The vertex regional uplift and passthrough stream pricing spend files boot
private owned proxies, then polled the worker database for spend rows that
only reach the proxy's own database. Read them through the owned proxy's
gateway instead.
Also hold the stream stall file's hold_headers peer past the longest row
wait, so a slow spend row cannot fail the peer teardown.
* fix(tests): check agent deletion on the owned proxy's database
The agent least privilege test deletes its agent through an owned proxy,
then checked the worker database for the row, which is always empty
there. Read it through the owned proxy's gateway so a delete that leaves
the row fails the test.
* fix(tests): read turn admission spend rows from the owned proxy's database
#47525 renamed _deployment to _fallback_deployment; #47363 was written against the old name and merged after it, leaving three F821 undefined-name errors that fail the python lint job on main
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count-tokens): translate chat tool use before Claude token counts
Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent.
* test(count-tokens): compare Claude count bodies with inference for Responses tool use
* fix(responses): forward the caller's Responses body to OpenAI token counts
OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it
* fix(responses)!: count Claude Responses bodies the way inference sends them
Non-native Responses counts now run the body through the inference bridge, so
text.format, tool_choice, reasoning, built-in tools, and file_search count the
way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The
OpenAI native count body uses the drop_params-resolved tool fields.
* fix(responses): honor drop_params and replay previous turns in Claude counts
Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation
* fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts
* fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params
* fix(responses): leave tool_choice out of counts when drop_params drops every tool
* fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send
The Bedrock Mantle fallback now counts thinking, tool_choice and output_config
like the runtime count. Claude Responses counts leave out cache_control marks
again, so more than four marks count instead of failing. Counts answer 400
before any provider call for user blocks chat inference refuses and for
Responses tools with no Chat Completions equivalent, instead of dropping the
block or counting a placeholder tool. Adds the wire tests for these paths
* fix(count-tokens): count Responses tools the way the inference bridge sends them
A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request
* fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id
* test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler
* fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent
* fix(langfuse_otel): send the completion start time
langfuse_otel never set langfuse.observation.completion_start_time, so
Langfuse showed no time to first token for any call, streamed or not.
The legacy langfuse callback already sends kwargs["completion_start_time"].
Send the same value as an ISO timestamp with an explicit offset.
* fix(langfuse_otel): type kwargs and test against a real span
Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any],
the declared type of model_call_details, so the new completion start time
read adds no unknown-type errors. The tests now check a real SDK span
instead of patching safe_set_attribute, and the parameter is typed.
* fix(otel): send the completion start time from the v2 Langfuse mapper
With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper,
which had no completion start time either. Carry the payload's
completionStartTime on LLMCallSpanData for streamed calls, the same calls
that get time_to_first_chunk_seconds, and map it to
langfuse.observation.completion_start_time.
* fix(langfuse_otel): take the logging kwargs as a read-only Mapping
The new kwargs annotation tripped LIT001. Both helpers only read the
logging kwargs, so type them Mapping[str, Any] instead of dict.
* fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter
* fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON
The native Gemini, Interactions and A2A streams never recorded a first chunk, so with
completion_start_time now sent both emitters showed TTFT equal to latency on them. They
now record it, and Gemini and Interactions take the call's start time like /v1/messages.
Both emitters send the completion start time as a JSON string, the form the Langfuse SDK
and the langfuse callback send, through one serializer.
* fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event
A2A streams took their start time when the iterator was built, so their latency left out
the work before it. They now use the call's start time like every other route.
Cached /v1/messages stream replays recorded no first event, so their time to first token
fell back to the end time. The replay now records it on the first event it yields.
The tests use the real Interactions config and typed logger hooks.
* fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free
* test(a2a): type the card fetch test and stop rebinding the seed helper's recorder
* test(otel): expect completion_start_time and cache_hit in the B3 span keys
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): billed background responses count toward the creator's project and model budgets
* fix(responses): carry the creator's project and per-model budgets on the billed background response
* fix(responses): count a key's budget tier toward background per-model budgets
* fix(responses): bill a background response's project only while the key stays on its team
* revert(responses): keep billing the creating key's project after a team move
A key can only keep or detach its project, so the billed project always pairs with the team it was created under. This reverts commit ded4f084c5.
* fix(responses): bill a background response under the model name the client sent, with its project alias
* fix(responses): name the team's public model as a billed background response's model group
The model group lookup now passes the creating key's team, so a team model or team wildcard bills under the public name a normal call logs and per-model budgets on that name move
* test(integration): follow #47441 metadata-routed Mantle counting in count_tokens wire tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): build the runtime count body without mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sagemaker): parse every streamed record however SageMaker cuts the payload parts
SageMaker cuts a container's response stream into PayloadParts at arbitrary
byte offsets, so one part can hold several SSE events or JSON lines, or only
part of one. The decoder treated each part as one record, so vLLM's [DONE]
was logged at ERROR on every stream and a split at an unlucky byte dropped
the rest of the answer.
Peel records off a running buffer with JSONDecoder.raw_decode, skipping SSE
framing and [DONE] between them.
* test(sagemaker): cover streamed records cut or merged across payload parts
* fix(sagemaker): decode UTF-8 across parts, skip SSE comment lines, and log container error records
A UTF-8 character split across two PayloadParts now decodes whole through one incremental decoder per stream.
SSE comment lines (TGI's keep-alive) and event, id and retry field lines are skipped as framing.
A container error record is logged at ERROR with its message and skipped, and any other record that is not a
chunk is logged and skipped, the same way on the sync and async paths.
* fix(sagemaker): raise a container's error record and ModelStreamError on both stream paths
A record with a non-null error now raises a 500 SagemakerError carrying the record after the chunks before it, on sync and async. Async no longer swallows a non-200 event, so a ModelStreamError raises as it does on sync. A chunk with a null error field is delivered, and any other record that is not a chunk is logged and skipped on both paths.
* fix(sagemaker): answer a streamed error with the container's status and fail a stream cut mid-record
A container's error record now raises with the status the container gave it (vLLM's error.code, the older flat vLLM shape, TGI's http_status_code or error_type) instead of a fixed 500 that the router retried as a 503. A ModelStreamError raises a SagemakerError naming its ErrorCode and Message, a stream that ends inside a record raises instead of ending as a stop, and a skipped record is logged by its keys only
* fix(sagemaker): trust only 422 and 429 from a streamed error record and fail LMI generation errors
* test(sagemaker): drive an LMI failed-generation stream through the proxy on a sagemaker/ deployment
* test(sagemaker): fall back to the next model when a stream fails before any content
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path
* fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format
* fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params
A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did.
A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper.
Adds integration cases for the json_schema envelope and the cut stream.
* fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates
Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded
* fix(anthropic): join a split answer when the stream ends in its last block
A cut during the second json tool block now runs the held block through
the same flush as a block stop, so it joins the streamed keys or the
earlier split blocks instead of sending two objects back to back. The
Vertex output_format fold drops its unreachable drop_params branch,
since both parent configs drop a non-object value first.
* test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages
* fix(router): schedule default_priority requests once and reject a non-integer priority
A router configured with default_priority failed every request that carried no
priority with a 500 naming a missing 'priority' argument, and an explicit
priority re-entered acompletion and hit the same error. acompletion now resolves
the priority once (the request's integer, else the default, with 0 kept as a
real priority) and schedules through _schedule_factory directly, while
schedule_acompletion delegates to acompletion so the two paths share one queue
pass. A non-integer priority answers 400 naming the param, or is dropped under
drop_params, instead of leaking to the provider or crashing the scheduler.
* fix(router): honor router-level drop_params and schedule prompt-management models under a priority
* fix(router): drop a non-integer priority when the model's deployments set drop_params
* fix(router): resolve the drop_params deployment scope for team routes and deployment ids
* fix(router): read the request and router drop_params flags without a loop-assigned Final
* refactor(router): type the prompt-management wrapper's return and mock the provider in its test
* perf(router): consult drop_params for the priority only when it is not an integer
* test(integration): assert the new priority contract for non-integer and zero priorities
* docs(router): describe default_priority as the priority every request without one takes
* fix(router): parse default_priority once and apply it to text completions and the queue route
A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig.
* fix(router): type the priority dispatch so the lint budgets hold
* fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order
* fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses
* fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority
A request whose model is a deployment id now counts that deployment as healthy in the
scheduler, since routing by id already ignores cooldown. Before, the scheduler found no
healthy deployment for an id, so the request waited behind every entry ahead of it.
A boolean default_priority (true, or YAML off) is ignored at startup with a warning,
like any other value that is not a whole number.
The scheduler integration tests now share worker state for the defaulted pair, cool a
group through /v1/responses so no stale in-memory queue is left behind, and use "high"
instead of "1" for the non-integer cases. The concurrent burst test no longer asserts
that the queue drains, matching the explicit-priority burst test, since the scheduler's
read-modify-write queue can race across workers.
* test(router): keep the queue drain check as a BUG-skipped integration test
* fix(router): queue default_priority chat requests once instead of failing them
* test(router): open with the default_priority regression tests alone
* fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name
* fix(router): scope tag routing to prefixed tags once tag_routing_prefix is set
With router_settings.tag_routing_prefix configured, an unprefixed caller tag
(for example a spend tag such as user_id:234) no longer drives tag routing, so
it can no longer refuse a model group whose deployments are untagged.
Key/team/project tags (metadata.inherited_tags) keep applying with or without
the prefix. Rows of batches the proxy runs itself now carry those tags as
inherited_tags too, so they keep routing on them once the prefix is set.
Takes over #39903.
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* test(integration): pin that proxy-run batch rows keep key/team tag protection
* refactor(router): filter routing tags from the raw request tags; deterministic tests and key-tag cells
* test(integration): audit cells for tag_routing_prefix: project tags, forged inherited_tags, odd tag values, outage, fallback, user-agent routing, concurrent burst
* test(integration): proxy-run batch rows keep key and project tags too
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* fix(search): price Tavily and Linkup searches at their per-mode cost map entries
* test(search): read the reported cost through the public hidden_params property
* fix(search): keep the type gate flat on the cost map membership check
* test(search): drop vendor price guards and a redundant case, type helper params as mappings
* fix(search): keep a registered search model's own price, cover every interception endpoint, hostile modes and a failing burst
* fix(search): keep a runtime-registered base search price for modes that have no runtime price of their own
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(videos): keep model_id in proxy video ids so per-model keys resolve on status/content
Provider transforms encode returned video ids with an empty model_id, so
GET /v1/videos/{id} and /content cannot resolve the deployment (and its
per-model api_key) and fail with invalid_api_key.
Re-stamp the returned VideoObject id with the resolved model_id in
video_generation, video_status, video_remix, video_edit and
video_extension.
Fixes#33423
* fix(videos): return a copy instead of mutating the response, read hidden params via helper
* test(videos): fully type the new video id re-stamping tests
Annotate every parameter and return added by this PR (hidden_params,
harness, response, -> None), drop Any from the test helper, mark locals
Final, and narrow the endpoint/helper results with isinstance(VideoObject)
instead of reading .id off an untyped value. Route the mocked processor
response through a typed Harness.respond_with, read NotRequired
DecodedVideoId keys with .get, and name the rule and reason on the
_hidden_params suppressions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): fill only an empty model in returned video ids
Keep an id that already names a model as is, so the asked-for name,
routing group and served group in it still drive access checks and
routing. Fill an empty model from the model the caller asked for
before the routed deployment id, and read the request model through a
typed view so the type gate does not grow
* fix(videos): route video calls to the deployment that created the video
Video ids now carry the id of the deployment that created them. Status,
content, remix, edit and extension calls go to that deployment when it
belongs to the requested group, so a group spread over two provider
accounts no longer polls the wrong one. A provider model shared by
several groups resolves to the creating deployment's group, and the
asked-for model is written into an id only when the router can route it.
* fix(videos): pin video calls only to deployments the caller may use
Pin a video id's creating deployment at deployment selection, and only when the caller may use that
deployment in the requested group, so aliases, routing groups and access groups still apply and the
spend label and fallbacks keep the requested group.
A status poll keeps the deployment its id names and takes the deployment that answered when the id
names none. An alias that routes to the creating deployment's group is kept on status and content.
* test(router): reset module fallback globals between unit tests
* fix(videos): check and route polls by the name the router serves
A poll on a video created through a provider wildcard is checked against the name the wildcard serves, a team model is checked by its public name, a poll stamps the deployment that answered, and the sync Router pins video calls the same way the async path does
* fix(videos): pin polls only among the deployments routing allows
Video calls route by group again and the creating deployment is picked only from the deployments left after access, team, health, cooldown, budget and tag filters, so a pinned poll no longer skips them. Wildcard groups now pin too. Adds tests for cooldown, wildcard pinning and character lookups through a provider wildcard, and return types on the auth tests
* fix(videos): scope the video pin to video calls, keep fallback-served videos reachable, skip cooldown on video 404s
* fix(videos): stamp the requested group only when one of its router fallbacks served the create
* fix(videos): follow chained router fallbacks for fallback-served videos, test with a real Router
* test(router): list the video pin helper with its sibling in the router coverage exemptions
* refactor(auth): walk chained router fallbacks without recursion
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions
stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt
Read chunk fields as typed attributes instead of untyped subscripts, which
clears the reportUnknownArgumentType breach the lint gate flagged
* fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): rebuild dict-form text completion chunks again
stream_chunk_builder routes dict chunks holding TextChoices to the text
completion builder, which now reads attributes. Convert them to
TextCompletionResponse at the routing point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path
Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage
* fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild
* fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions
* fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null
* fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path
Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`.
The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text`
next to `delta`, so streamed /v1/responses calls to Together ended without response.completed.
Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about
the same as rebuilding an n=1 stream of the same size.
* fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs
A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising
Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals
* fix(streaming): rebuild text completion choices without an index at index 0
Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base
* fix(streaming): rebuild text completion choices with a string index under that int
A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string
* test(streaming): drop a docstring that repeats its test
* fix(streaming): fall back to choice 0 for an infinite text completion index
A float infinity index made int() raise OverflowError, which the index
coercion did not catch, so the rebuild failed where the merge base
returned choice 0.
---------
Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(mcp): cover one spend row billing both rounds of a chat completions MCP auto-execute
* fix(mcp): bill both model calls of a chat completions MCP auto-execute on one spend row
* fix(mcp): keep provider usage fields when summing both chat rounds of an MCP auto-execute
* fix(mcp): log each auto-executed MCP tool call under its own spend-log id
A chat MCP auto-execute logged its tool calls under the request's call id, which is also the id of the request's failure row. When the follow-up call failed, the database kept the tool row and dropped the failure row that bills the paid first call
* test(mcp): cover the margin, cache, failure row and parallel tool rows of MCP auto-execute billing
* feat(router): apply each deployment's reasoning_effort_levels on every provider and route
A deployment's declared reasoning_effort_levels now decide the effort sent upstream on /v1/chat/completions, /v1/messages and /v1/responses for every provider. Requests round up by default, litellm.reasoning_effort_rounding and model_info.reasoning_effort_rounding switch to down, a declaration lets providers like azure_ai accept reasoning_effort, and each remap logs one INFO line with structured fields.
* fix(router): round reasoning effort only on a deployment's own declared levels
Cost map levels no longer round requests or force-allow reasoning_effort, so undeclared sibling deployments and catalog-only models behave as on main. A declaring deployment lets azure_ai accept any level the resolver sends, and a hosted_vllm chat thinking budget is rounded too.
* fix(router): drop undeclared reasoning effort under drop_params and round native /v1/messages effort
Under drop_params a level outside the deployment's reasoning_effort_levels is
dropped instead of sent. Providers that take /v1/messages natively now round
output_config.effort onto the deployment's declaration. The rounding setting
ignores case and surrounding spaces.
* fix(router): strip line breaks from every field in the reasoning effort remap log
* fix(router): round thinking-budget efforts onto a declaring deployment on every provider
* fix(router): keep the caller's thinking display when rounding a thinking budget on adaptive Claude
Warn on a reasoning_effort_rounding value other than up or down instead of silently rounding up
* test(integration): cover deployment reasoning_effort_levels rounding on chat, messages and responses
Wire cells assert the exact body each provider receives when a deployment
declares reasoning_effort_levels, plus undeclared controls, malformed input,
drop_params, rounding settings, fallbacks, model groups, live updates and
the response cache. Chaos cells cover a killed worker, a proxy restart, an
upstream outage, a slow stream and the INFO remap log.
* test(integration): cut held streams when the upstream stops in the reasoning effort chaos cell
The upstream-stop cell waited for the held streams before stopping the upstream, so they only ended on a client or proxy timeout and the result depended on which fired first. The upstream now stops while the streams are held, and each cut stream must carry an error and no answer
* fix(router): round thinking budgets only on declaring deployments and round Converse output_config
An undeclared hosted_vllm deployment serving a Claude model name now maps a thinking budget the same as any other undeclared hosted_vllm deployment. A model_info that echoes a cost map key no longer borrows that entry's levels. A non-numeric thinking.budget_tokens answers 400, or is dropped under drop_params, where main answered 500. A declaring Bedrock Converse Claude deployment now rounds output_config.effort, which /v1/messages sends for Claude Code
* test(integration): cover output_config rounding and non-numeric thinking budgets in the reasoning effort cells
* test(integration): keep an explicit reasoning_effort winning over a malformed thinking budget on hosted_vllm
* test(integration): cover output_config rounding on openai, azure and azure_ai chat
* fix(hosted_vllm): let an explicit effort win over a malformed budget and reject one on /v1/messages
An explicit reasoning_effort beside a non-numeric thinking.budget_tokens keeps main's
behavior again. A declared hosted_vllm deployment with a Claude model name maps a
budget through the generic table instead of the adaptive Claude one. /v1/messages to a
non-Claude target hands a malformed budget to completion(), which answers 400 or drops
it under drop_params, where it answered 500
* fix(router): drop thinking beside an explicit effort when a declaring deployment's provider cannot read it
* test(integration): cover malformed thinking budgets on /v1/messages to non-Claude targets
* fix(anthropic): answer 400 for a malformed thinking budget on /v1/messages to any non-Claude model, or drop it under drop_params
* test(integration): cover an explicit effort beside thinking on azure_ai and drop_params on the /v1/messages bridges
* fix(anthropic): answer 400 for a malformed thinking budget through adapter_completion without drop_params
* fix(anthropic): type the adapter_completion request so the drop_params read adds no unknown argument
* fix(router): a default effort beside a thinking budget sends no effort on chat
Chat now leaves the budget-derived effort out when the caller's effort resolves to none or asks output_config for one, matching /v1/messages. The chat thinking decision moves into resolve_chat_reasoning so completion() stays under basedpyright's complexity limit.
* test: type respx_mock and mark the adapter result Final in the reasoning effort tests
* fix(router): keep the caller's thinking beside a kept effort and the budget effort beside a dropped one
A non-default output_config.effort beside a thinking budget now leaves the thinking as sent, so Bedrock
Invoke keeps display updates. A level drop_params drops counts as never sent, so the budget still sets the
effort. output_config.effort replaces a thinking budget the provider cannot read, and a non-numeric string
budget on adaptive Claude answers 400 in place of 500, or drops under drop_params.
* fix(router): send a kept output_config effort as reasoning_effort where the provider cannot read thinking
On a declaring hosted_vllm or azure_ai deployment, a thinking budget goes out as reasoning_effort. An
output_config.effort beside it now takes that slot, rounded onto the declared levels, instead of leaving
only output_config, which those backends ignore.
* fix(reasoning-effort): honor global drop_params for a non-numeric budget and keep the budget effort beside a dropped /v1/messages level
* fix(reasoning-effort): send a level a gpt-5 deployment declares even where the cost map does not list it
* fix(reasoning): apply declared levels on azure_ai gpt-5 and reject a bool thinking budget on adaptive Claude
* fix(router): drop a thinking budget's effort that a deployment declaring no ranked levels excludes
A deployment declaring [] or ["none"] skipped the budget check, so adaptive Claude
still sent the budget-derived output_config.effort under drop_params on chat,
Bedrock Converse, and native /v1/messages. The adaptive thinking now goes out
without that effort
* fix(anthropic): reject a false, null or empty thinking budget on adaptive Claude
Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing
* perf(streaming): skip full-chunk dump and delta rebuild on openai-compatible chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(streaming): reuse the built choice and delta on plain openai-compatible chunks
Co-authored-by: yassin <yassin@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): copy upstream chunk attributes in upstream key order
The scalar fast path iterated a frozenset of keys, so the SSE key order of chunk extras like service_tier and obfuscation changed with each worker's hash seed
* test(streaming): count chunk_creator calls with a profiler instead of patching classes
* test(streaming): check service_tier still passes through chunk_creator
* fix(streaming): reuse the built choice for chunks with upstream choice extras
* test(streaming): cover ModelResponseStream chunks on the choice reuse path
* fix(streaming): keep main's delta rebuild for deltas with nested values
Clearing role in place on a delta that holds tool calls left its nested
fields unset, so chunk.model_dump() dropped keys like a custom tool call's
name on later chunks. Deltas holding only scalars keep the in-place path.
* perf(streaming): build each plain OpenAI chunk's delta once and skip pydantic attribute misses
Later plain ChatCompletionChunks now build one Delta and one StreamingChoices, with no full-chunk model_dump, and read stored values instead of attribute probes that raise and catch AttributeError. Nested-value chunks no longer pay the dict(model) misses that made them slower than the merge base.
* fix(streaming): read hidden params a chunk carries as a pydantic extra
* test(streaming): pin logprobs, Azure filter extras, and audio and image deltas on later OpenAI chunks
* test(streaming): cover role, reasoning, choice extras and billing on plain OpenAI SDK chunks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): stream Claude json_schema answers as text on every streaming path
Async and proxy streams on Databricks Claude json_schema returned the
internal json_tool_call tool call with empty text, because the async
stream factory dropped json_mode. Databricks now passes it through, the
json tool's argument fragments stream verbatim as content (a lone {}
fragment was blanked before, corrupting nested empty objects), and the
stream ends with finish_reason stop like the non-stream answer.
* test(databricks): keep the json_schema stream regression tests ahead of the fix
Holds the Databricks transformation at the merge base so the first approval covers the regression tests; the fix commit follows.
* fix(databricks): stream Claude json_schema answers as text on every streaming path
* fix(databricks): keep finish_reason tool_calls when a json_schema stream also calls a user tool
* fix(databricks): drop text beside the json tool and renumber user tools in Claude json_schema streams
With reasoning_effort set, Claude can write prose before json_tool_call or call it next to a user tool.
Streams now hold text until the end and drop it when the json tool was called, renumber user tool
deltas past the dropped json tool so indexes start at 0, and non-stream returns the json tool's
arguments as content next to the user tools.
* fix(databricks): stream JSON-looking text at once and keep prose ahead of user tools in Claude json_schema streams
* fix(databricks): hold a user tool call behind held prose so a later json tool answer does not glue to it
A json_schema stream with thinking on that wrote prose, called a user tool, then called the json tool sent the prose at the user tool call and appended the JSON answer to it, so content did not parse. User tool deltas behind held prose are now held with it and sent merged per index at the json tool or the finish, and a json tool after text already sent is dropped so the answer is not repeated.
* test(databricks): cover json_schema answers on every endpoint in the integration wire cells
* feat(key_management): apply default_key_generate_params.allowed_routes on key generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(key_management): avoid mutating default routes payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): apply default allowed_routes to key_type presets it covers
* fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy
* fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes
* fix(proxy): keep owners from changing routes a key got from default_key_generate_params
* docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off
* test(proxy): type the default_key_generate_params allowed_routes tests
* fix(proxy): let key owners send unchanged allowed_routes back on /key/update
* fix(proxy): ignore a route default with a non-string entry and match resent routes exactly
A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths.
* test(proxy): cover a non-string allowed_routes default in the key integration tests
* test(proxy): read key rows from the owned proxy's database in the default routes integration tests
* fix(proxy): leave allowed_routes out of a key update that resends them unchanged
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): keep the gateway handler off the SDK path when no tool is an MCP tool
completion(), responses() and anthropic messages imported the MCP gateway handler
whenever tools were passed, and responses() did so on every call. Since the handler
began importing the MCP catalog at module level in #42568, that import needs the
optional mcp package, so a base SDK install failed with ModuleNotFoundError on any
function-tool call and on every responses() call
A new import-safe pre-check skips the handler unless some tool is a dict of type mcp,
which is the only shape the handler's gateway decision can accept. MCP gateway tools
still reach the handler as before
Adds in-process regression tests for the three entry points and a function-tool
completion to the base SDK install check
* fix(mcp): leave one-shot tool iterables unread in the gateway pre-check
responses() declares tools as an Iterable and the handler scans the same
object a second time, so a pre-check that read a generator would hand the
handler one tool fewer. The pre-check now scans only collections and reports
anything else as possibly naming the gateway, which keeps the previous path
The Anthropic messages regression test now mocks the httpx layer with respx
and turns off the aiohttp transport, instead of replacing AsyncHTTPHandler
* fix(mcp): scan only lists and tuples in the gateway pre-check
The pre-check read any Collection, but a Collection that yields its items only
once would still be exhausted by the handler's second scan. Only the list and
tuple builtins are guaranteed to iterate the same way twice, so they are the
only inputs read now. Everything else keeps the previous path and is reported
as possibly naming the gateway
The unread-input test now covers a generator, an iterator, a one-shot
collection and a deque
* fix(mcp): scan only exact list and tuple types in the gateway pre-check
isinstance accepted list subclasses, and a subclass that iterates only once
would still be consumed by the handler's second scan. The fast path now reads
only objects whose type is exactly list or tuple. Every other input, subclasses
included, keeps the previous path and is reported as possibly naming the gateway
The unread-input test gains a one-shot list subclass and a tuple subclass
* fix(sdk): decide MCP gateway routing without importing the proxy handler
The pre-check now holds the handler's exact gateway test, so remote MCP tools, mcp 1.x installs and any tools iterable stay off the handler, and the base SDK check calls responses() with a remote MCP tool.
* fix(sdk): send /mcp/<name> remote MCP URLs to the provider when the gateway registry cannot load
A path-only /mcp/<name> match now counts as a gateway tool only when the MCP server registry imports, so SDK installs without the proxy extras pass remote MCP servers like Zapier to the provider. Also drops a dead test patch.
* test(sdk): send a /mcp/<name> remote MCP tool in the base SDK install check
* style(sdk): drop docstrings from the MCP gateway pre-check and its tests
* test(mcp): cover /mcp/<name> gateway routing on the proxy and Messages tools on a base SDK install
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request
An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's
litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper
now reads the referenced file or env var on each call and never writes the token back, so a rotated
token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as
server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback
targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400
* fix(secret_managers): strip only the leading oidc/ from a secret reference
A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed
* fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them
The advisor tool's own api_key now answers a 400 when it holds an oidc/
reference, since its sub-call runs through the client wrapper that resolves
one. Stored fallback targets, auto-router tiers and user_config refuse only an
oidc/ api_key, inline or through the credential they name, so their workload
identity federation handling stays as it was. Provider model discovery for a
wildcard deployment resolves the reference before calling the provider.
* fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments
* fix(utils): resolve oidc api_key refs only from admin-configured sources
Mark an oidc/ api_key read from a proxy-level deployment or a stored
credential, and resolve only that marked value. Team deployments, prompt
templates, nested configs and a direct SDK api_key send the text as
written. An unsupported reference kind raises a 401 so the router fails
over to the group's next deployment.
* fix(utils): send a credential's oidc token only to its own api_base host
A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the
credential's own api_base or base_url, so a team model or vector store that names the
credential with another api_base gets a 401 and that host gets nothing. Saving a team model
with an inline oidc/ api_key answers 400, since team rows never resolve one.
* fix(proxy): send an admin oidc api_key only to the hosts configured beside it
Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths.
* fix(proxy): redact the token path from token count auth errors
* test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs
* fix(proxy): redact the token path from secret reference errors where they are raised
* fix(proxy): keep the secret reference's path out of the error a caller sees
The 401 for an unreadable oidc api_key or pass-through credential now names only
the reference kind and a fixed reason, so a token path under /run/secrets or a
custom credential dir never reaches the caller, while the server log keeps the
full reference for the operator
* fix(secret_managers): strip line breaks from the logged secret reference
A reference holding a newline can no longer forge a server log line
* fix(secret_managers): label an unknown oidc reference kind instead of echoing it
An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in
full in the caller's 401 and in the spend log, since the kind was whatever
text followed oidc/. A kind get_secret does not know now reads
oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>,
which the client-message redaction on chat routes leaves whole.
* test(health): run the real model auth check in the oidc test connection test
* test(secret_managers): read an oidc/file token stored under a directory named oidc
* fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references
A huggingface/* wildcard call could name its own endpoint URL as the model and
receive the resolved token. The origin check now treats a URL in the model as
a host the reference must allow. The integration contract also reads the
proxy's own database instead of the pytest worker's.
* fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin
A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443,
though both reach the same server. The origin check now compares scheme, host
and effective port, as assert_same_origin does, and an unparseable api_base is
refused instead of raising. The advisor test sets the real proxy flag instead of
replacing our own helpers.
* test: type the oidc request-body helpers and drive the tier test through mock_response
* fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host
test_router_cooldown_failure_row.py boots an owned proxy, which writes to a private database since #47213. The file was written on a base without #47213 and read its rows from the worker database through read_rows, so all 13 cases timed out with State did not converge: []
Every row read now goes through Gateway.rows on the owned proxy's gateway, the same fix#47410 made for the stream stall cooldown cases
* fix(friendliai): drop litellm-internal assistant fields before sending
Claude Code replays thinking blocks on every turn after the first. The
Anthropic adapter turns them into thinking_blocks plus reasoning_content
on the chat assistant message, and FriendliAI rejects any message key it
does not know, so multi-turn Claude Code sessions on friendliai/ models
failed with a 500 after the first tool call
FriendliAI accepts reasoning_content and GLM reads it back, so only
thinking_blocks and provider_specific_fields are dropped. together_ai
already had this exact helper as a private function, so it moves to
common_utils next to strip_litellm_internal_message_fields and both
providers share it
Related to #32188
* test(friendliai): cover a replayed tool call turn
The regression fixture is now the shape Claude Code replays after a tool
call: an assistant turn with thinking_blocks, tool_calls and no content,
followed by the matching tool result. The test asserts both survive the
sync and async request paths unchanged
* ci: retrigger checks
* ci: retrigger benchmarks after runner timeout
* chore(friendliai): mark inherited list signatures as mutable-ok
The type-discipline gate on main now caps LIT001 at the merge-base count,
so the _transform_messages overloads need the same inherited-contract
marker together_ai uses for its identical override
* ci(test-unit): add tests/unit/llms/friendliai to the llms-providers shard
* test(friendliai): check the outbound FriendliAI body through the proxy for chat and messages replays
Both replay shapes go through a real proxy to a scripted FriendliAI upstream, and the test reads the
request body FriendliAI received. It fails at the merge base, where thinking_blocks and
provider_specific_fields reach the upstream, and passes with the field stripping in place.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* fix(speech): correctly propagate spend logging and usage for Gemini TTS bridge
* fix(speech): call Gemini once per TTS request and log the bridge's token usage
aspeech re-ran speech whenever the first result was not a coroutine, so the
speech-to-completion bridge sent every Gemini TTS request upstream twice. It
now returns the sync result.
The bridge's inner completion runs as an internal sub call, so only the
speech call fires success callbacks, and the completion's usage rides on the
audio response so the spend log and usage_object carry real token counts.
Replaces the separate inner Logging object and the cost_calculator fallback,
which never reached the spend row.
* fix(speech): flag a cached Gemini TTS reply as a cache hit so it logs at no charge
* fix(speech): price a cached Gemini TTS reply's savings from its token usage and keep its cache key
* test(integration): cover Gemini TTS speech billing, logging, cache hits and deployment TPM
* test(integration): start the speech TPM cell in a minute's first 10 s so the two requests never cross a minute
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): refresh cached team on /team/block and /team/unblock
Both endpoints wrote `blocked` to Postgres and returned without touching the
management-object cache. Auth resolves teams through `team_id:{team_id}`
before reading the DB and `common_checks` enforces `blocked` off that cached
object, so a team whose traffic had already warmed the cache kept serving
requests after being blocked, and kept getting rejected after being
unblocked, until the entry expired.
`_refresh_cached_team` already exists for exactly this and its docstring says
every endpoint mutating `litellm_teamtable` must call it; these two were the
only mutating endpoints that didn't. The update now also asks for
`object_permission` so the refreshed cache entry doesn't drop the relation,
matching `team_model_add` and `team_model_delete`.
* fix(proxy): keep a team block from being undone by in-flight spend writes
update_cache read the cached team when a request finished and wrote that
whole copy back in a background task, so a team blocked or invalidated
while requests were in flight came back unblocked on every busy worker
until its 60 s cache entry expired. The team spend now goes onto the team
held in this worker's memory at write time, and a missing entry is left
missing for the next auth read to load.
Block and unblock responses keep litellm_model_table and object_permission
null, as before this PR, while the cache still gets both relations.
* test(proxy): type the team spend cache test parameters
* test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock
---------
Co-authored-by: Hamza Shah <hamzasepal@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The override test slept a fixed 10 s after writing responses_websocket_session_limit_seconds and then
opened sockets on both workers. A worker whose config resync ran later than that kept the old 7200 s
cap, so its socket never closed and the test failed after 75 s
A test-only /health/integration-general-setting route returns the value a worker holds along with its
pid. The test polls it over fresh connections until every worker reports the new cap, and after the
delete until every worker reports it gone
The cooldown cases in test_hosted_vllm_stream_stall_wire.py boot an owned proxy, which writes to a
private database since #47213. The file was written on a base without #47213 and read its rows from
the worker database through read_rows, so every cooldown case that waits for a failure row timed out
on main with State did not converge: []
Every row read now goes through Gateway.rows on the gateway that served the request
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
#46662 made streamed /v1/responses lifecycle events report the requested model in the nested response.model, matching the non-streamed body. The arize v2 caller-stream helper still expected the upstream deployment name there, so f1 and a6 failed on every main run since bbd446b476
The SSO removal test configured Google sign-in, which fetches its discovery
document from accounts.google.com before redirecting. CircleCI blocks that
egress, so the test failed on every main run since #46709 added it. Microsoft
sign-in builds its endpoints locally and goes through the same SSO environment
variable handling, so the test now uses it
* fix(managed_files): send provider file ids when creating a fine-tuning job from managed files
Fine-tuning job creation with a managed training file sent the gateway's managed id and the provider-prefixed deployment model to OpenAI, which rejected the job. The pre-call hook now builds the file id mapping for training_file and validation_file, the deployment hook swaps both, the SDK strips the provider prefix from the model, and the create response returns the caller's own file ids
* fix(managed_files): route managed fine-tuning creates only to deployments holding every file
Fine-tuning creates now narrow routing the same way managed batches do, so a
deployment added after the upload is never picked. Adds a two-worker chaos
integration test and runs tests/unit/fine_tuning in the endpoints shard.
* fix(managed_files): keep fine-tuning creates on the model group when no deployment holds the files
A managed file used on a model group it was never uploaded for kept failing
with the provider's 400 before the routing filter. The filter turned that into
a 429 no-deployments error, so fine-tuning now falls back to the whole group in
that case while batch input files keep routing only to deployments that hold
them.
* test(fine_tuning): cover the provider prefix strip for OpenAI and Azure creates through the proxy
* test(fine_tuning): start each multi-deployment case from an empty upstream log
* test(fine_tuning): boot both proxies of the late deployment cell on one scratch database
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(anthropic): answer 400 for a malformed tool_result text on the Responses bridge and count_tokens
A tool_result text block whose text is null, a number, or missing made
/v1/messages and /v1/messages/count_tokens answer 500 on OpenAI and Azure
models. Both routes now answer 400 naming the block, and drop the block
when drop_params is on.
* fix(anthropic): keep a missing tool_result text as empty text, bound the error, honor body drop_params on count_tokens
* fix(anthropic): apply the tool_result text check on the chat/completions bridge and every count route
The chat/completions bridge used for Azure, Gemini and other providers now answers the same 400 for a
null or numeric tool_result text and drops the block under drop_params, so a model group that mixes
OpenAI with another provider answers the same way on every deployment. Gemini and Vertex count_tokens
apply the same check, count_tokens turns on drop_params for Claude Code and Codex the way /v1/messages
does, and a tool_result without a tool_use_id is checked too
* fix(proxy): answer 400 for a malformed tool_result text on count_tokens models without a count API
Providers with no token counting API fall back to the local tokenizer, which counted a null or numeric tool_result text as fine while /v1/messages on the same model answers 400. The fallback now runs the same check, unless drop_params applies to the deployment or the request
* fix(proxy): honor request drop_params on the local count fallback when no deployment resolves
* test(integration): cover non-string tool_result text on both /v1/messages bridges and count routes
* fix(anthropic): keep a dropped tool_result on Mistral and reject it before a guardrail rewrite
Under drop_params a tool_result whose only text block was null or a number became an
empty tool message, and Mistral's empty-message filter removed it, so Mistral answered
400 for a missing tool reply. The filter now only removes empty assistant messages,
which is what Mistral rejects.
A guardrail that rewrites structured_messages wrote the dropped text back as "", so a
null or numeric text was silently blanked even without drop_params. The rewrite path now
answers the same 400 as the bridges unless the request or global drop_params is set.
* test(integration): cover the Mistral empty tool reply and the guardrail rewrite tool_result text check
* fix(anthropic): keep a malformed tool_result text through a guardrail rewrite and keep Mistral's null-message drop
A guardrail that rewrites structured_messages now writes the caller's null or numeric
tool_result text back in place, so the route's bridge and the deployment's drop_params
decide after routing, the same as with no guardrail. The rewrite path no longer answers
its own 400 before routing.
Mistral drops a null-content system or user message again, as main does, and only a tool
reply is exempt from that filter so a dropped tool_result keeps its tool call.
* test(mistral): expect empty system and user turns dropped and the empty tool reply kept
* perf(anthropic): restore kept tool_result text parts in one linear pass
* test(integration): cover the tool_result text restore across tool_results, the chat bridge, and a batch upload behind a rewriting guardrail
* fix(anthropic): let guardrails scan malformed tool_result text and honor drop_params on native Anthropic
A rewriting guardrail now sees a non-string tool_result text as its JSON text and the write-back
restores the caller's value by the sanitized tool_use_id when the guardrail left it unchanged.
Native Anthropic drops the bad block under drop_params on /v1/messages and count_tokens, and
Mistral keeps an empty tool reply only when it answers a tool call, sent as empty text.
* fix(anthropic): compare a kept tool_result text to its scanned form once and align the wire tests with drop_params
* test(integration): cover native Anthropic drop_params resolution, guardrail scanning of numeric tool_result text, and the Mistral orphan tool message
* fix(count_tokens): honor the router's default drop_params when counting a malformed tool_result text
* test(count_tokens): send the tool_result count tests through a real request body instead of a mocked parser
* test(count_tokens): send the native Claude count test through a real request body
* fix(anthropic): keep each non-string tool_result text when tool_use_ids sanitize to the same id
* fix(anthropic): keep non-ASCII characters in the guardrail's view of a non-string tool_result text
* test(gemini): mock the count_tokens upstream with respx instead of patching the client factory
* fix(mistral): keep an empty tool reply only for an open call, and forward non-object message items on the native drop
An empty tool reply re-answering a call the conversation already moved past made Mistral reject the request, and a tool call id that is not a string crashed the filter. Native Anthropic under drop_params assumed every message item was an object.
* fix(mistral): find answering tool replies in one pass instead of copying the open call set per reply
* fix(mistral): ignore dropped empty placeholders when tracking open tool calls
* fix(mistral): drop an empty tool reply when another reply to the same call carries the result
* refactor(anthropic): share tool_result text helpers to shrink the diff
* fix(anthropic): answer 400 for a non-string tool_result text on the OpenAI Responses bridge
* fix(prometheus): label deployment failures with the deployment's provider instead of None
* test(prometheus): drive every deployment failure surface through the provider label
Streaming chat, responses and messages, dropped and stalled upstreams,
JSON-configured, custom-provider-map, prompt-management and unprefixed
deployments, forged request metadata, an unauthenticated request, the
OpenAI and Anthropic SDKs sync and async, and a concurrent burst counted
exactly once per deployment
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
68db022d57: `AssertionError: (('success',),)`).
The test now reads the spend row first, which is written once the proxy has settled the disconnect, and
releases the upstream after it. The proxy's behavior is unchanged; only the test's ordering is.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): record cooled deployments on the all-cooled failure row
A RouterRateLimitError now carries the cooled deployments of the route the
request resolves to for the caller's team (model group, alias, wildcard or
team route), each with its masked api_base. get_error_information copies them
into error_information.error_cooldown_deployments so the SpendLogs failure row
and every logging callback can tell which deployments were cooling
* fix(router): name cooled deployments from the router's own resolved route
Use the deployments the router resolved for the request, before health and
cooldown filtering, instead of re-deriving the route. Alias chains, the
provider-model fallback, wildcards and team filtering now match routing by
construction. A routing-strategy refusal records null.
* fix(router): name only the cooled deployments tag and region routing allow
The refusal's cooled-deployment list now leaves out deployments the
request's tags or the end user's allowed region keep it from, so a
tenant's refusal row no longer shows another tenant's api_base
* fix(router): keep failure logging working for refusal subclasses that skip init
A RouterRateLimitError subclass that never calls its parent init now logs a null cooled-deployment list instead of breaking the failure payload
* test(router): call the refusal routing helpers directly
The router coverage check needs each router function called by name from a unit test
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* refactor(count_tokens): hold the tool counting wiring for a follow-up commit
* refactor(count_tokens): hold the chat template tool sections for a follow-up commit
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* fix(count_tokens): skip tool_choice and non-function tools for chat template families, leave out Fireworks V4.1 spelling
* fix(count_tokens): count Anthropic tools in the shape the messages bridge sends them
A missing description is no longer rendered as null and strict is counted when set, matching what DeepSeek V4 and Qwen3.5 deployments receive. Adds tests that pin each template's schema rendering and its system turn.
* test(count_tokens): check the prompt caching threshold agrees with token_counter for DeepSeek V4 and Qwen3.5 tools
* fix(count_tokens): skip Anthropic web search tools and count unmapped Anthropic tool keys for chat template families
* test(token_counter): cover chat-template tool counts across count routes, budgets, ITPM and context edits
Integration cells check that DeepSeek V4 and Qwen3.5 tools are counted the way their chat templates render them on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens, strict budget reservation, project ITPM on chat, messages and responses, and the clear_tool_uses trigger, with grok controls keeping OpenAI framing
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): price streamed pass-through calls with the regional uplift
The streamed Vertex pass-through handler builds the response with
stream_chunk_builder, which stamps a cost into _hidden_params without the
vertex_location. The handler then prices the response again with the
location from the URL, but only into kwargs, and the logger prefers the
stamped cost. Streamed calls on a regional endpoint were billed at the
global rate.
Stamp the location-aware cost on the response in the shared
generateContent payload builder so the logger records it.
Fixes#44755
* style(vertex_ai): keep the response_cost assignment within the 120-char limit
* refactor(vertex_ai): set the passthrough response cost with set_hidden_param
Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this.
* test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed
* fix(proxy): price streamed Vertex pass-through spend like the non-streamed call
Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls
* fix(proxy): price custom-priced Vertex pass-through deployments at their own rates
The stamped response_cost from the generateContent logging helper is returned before the
logger checks custom pricing, so a deployment with custom input/output rates was billed at
the cost-map price. The helper now passes custom_pricing and the deployment id into
completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls,
and deployments without custom pricing keep the regional uplift.
* fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price
* fix(proxy): give the new pyright ignore a reason for the LIT004 gate
* test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included
* fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same
* fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini
A top-level base_model in a pass-through body now reaches the logging
params only for a key or team with allow_client_pricing_override, so a
plain key cannot pick a cheaper model's price once the logger prices
these calls. The test that pinned every litellm-owned body name into the
logging params now leaves base_model to its own key-gated test.
The Vertex handler now classifies a non-Google upstream host as Vertex
only for Vertex resource paths, so a /gemini stream through a
GEMINI_API_BASE gateway is priced as Gemini without the regional uplift.
Adds the audit's integration spend test for Claude on Vertex, /v1beta,
the named custom-priced /gemini deployment, and a cut OpenAI stream.
* test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing
The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path.
* fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model
The Vertex pass-through handler picked Gemini pricing from the URL path,
so a configured pass-through to a Vertex gateway with a short path (no
/projects/ segment) was billed as Gemini, and its Claude streams at $0.
The provider now comes from the route: a call through /gemini or to
generativelanguage.googleapis.com prices as Gemini, and everything else
as Vertex, the same rule main used. Matching the GEMINI_API_BASE host
was rejected because one gateway can serve both APIs, and the path rule
was rejected because short gateway paths look like Gemini paths.
A non-string base_model (a number, list or object) made the cost
calculation fail and bill $0 once the stamped stream costs were gone.
get_base_model_from_metadata now ignores it and falls back to the
deployment's base_model, so the call bills the list price. A 400 was the
other option, but the logger runs after the upstream answered.
* test(integration): bill a non-string body base_model at the list price on every pass-through route
* fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info
A key allowed client pricing can send metadata.model_info in a pass-through
body. A non-string model_info.base_model made the cost lookup fail, and
once the stamped stream and Gemini costs were gone those calls billed $0
on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores
a base_model that is not a string, the same rule the root base_model got,
so the call bills the list price.
A model_info that is not an object (a string or a list) broke several
logging steps that read it as a mapping, so a non-streamed /gemini call
fired no success callback and no pass-through call wrote a spend row. The
pass-through route now leaves a non-object model_info out of the logging
metadata, so the call bills the list price. A 400 was the other option,
but it would turn calls that answer 200 today into errors, and guarding
every reader of model_info would spread the same check over many files.
* fix: reject a non-object model_info before the provider call again
The previous commit made the base_model lookup return None for a
model_info that is not an object. That lookup also runs during call
setup on every router route and SDK entrypoint, where such a value used
to fail before any provider request. With the early return the call went
upstream, the provider billed it, and the request then failed in a later
reader that still expects an object. On /v1/messages and /v1/responses
the key was also charged for a response the client never got.
The lookup again reads base_model from any truthy model_info, so a
non-object value fails during setup the way it did before this PR. A
non-string base_model inside an object still bills the list price, and
the pass-through route still drops a non-object client model_info at
intake, so pass-through calls keep billing the list price.
* fix: match the Gemini API host exactly when picking the pass-through cost provider
The provider check accepted any host ending in
generativelanguage.googleapis.com, so a Vertex host such as
vertex-generativelanguage.googleapis.com was priced as Gemini and lost
the regional uplift. It now accepts that host or a subdomain of it.
* test: pin that a subdomain of the Gemini API host keeps the Gemini price
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
