The Responses stream that wraps the planned chat stream now logs the request,
so the spend row carries the Responses object the client received instead of
the chat completion. The chat stream hands its assembled response to the
bridge instead of dispatching success logging itself, and deferred stream
logging is parked through Logging.hold_stream_completion.
The chat stream that wraps the native Responses stream now logs the request,
so the spend row carries the chat completion the client received instead of
the Responses object. The Responses iterator hands its completed response to
the bridge instead of dispatching success logging itself.
Decide the chat-to-Responses bridge from the provider's Responses config, so a
responses-mode model on a provider without one stays on chat instead of bouncing
through responses() and back. That bounce was the only loop
_skip_responses_api_bridge guarded, so the flag goes away.
The chat-to-Responses bridge called litellm.responses() from inside the chat
call, so the @client pass and its deployment hooks ran twice per request. It now
hands the translated request to the native Responses executor, shared with the
/v1/messages bridge in litellm/responses/bridge_executor.py, and keeps
responses() for requests that need what only it adds
/v1/responses on a provider without a Responses config now plans the chat
call once and runs execute_chat directly instead of re-entering
litellm.acompletion, so deployment hooks run once per request.
The Responses surface now logs the non-streaming row, so it carries the
Responses shape. The previous_response_id session handler reads both row
shapes, since streamed turns are still logged in chat shape
The /v1/messages bridge for chat-only models now plans the chat call once
and runs execute_chat directly instead of re-entering litellm.acompletion,
so deployment hooks run once per request. Requests that need the chat
surface (fallbacks, model_list, MCP gateway tools, prompt management,
litellm_system_prompt) still go through litellm.acompletion
completion() keeps the surface steps (MCP gateway, prompt management,
fallbacks, model_list) and hands the provider work to a new execute_chat(),
so bridges can call the chat provider path without a second @client pass
The /v1/messages bridge for Responses-native models re-entered
litellm.aresponses, which ran a second @client pass and needed the
logging object's call_type rewritten so the spend row stayed on
/v1/messages. It now resolves the deployment and calls the native
Responses executor directly, so the outer anthropic_messages call owns
logging. Calls without a logging object (the sync SDK entry point) and
calls that need prompt management hooks keep the old path
Messages, chat and Responses each decided inline whether to bridge to
another API format. The decisions now live in litellm/native/route_plan.py
as a tagged union so the upcoming executor split has one place to read
* fix(harness): set PWD to the working directory of local sandbox commands
LocalSandbox started CLIs in the sandbox workdir but passed the parent's PWD through. OpenCode reads PWD as its project directory, so a litellm.agent() turn worked in the Python process cwd instead of the sandbox.
* fix(harness): re-home resumed OpenCode sessions into the sandbox folder
OpenCode files each session under the folder it ran in and only streams a resumed session's events from that folder, so a session saved before PWD pointed at the sandbox, or resumed in another sandbox, hung. Resume now runs opencode export and import in the sandbox first. Also types the CLI handler's config so its member accesses are known.
* fix(harness): fail an OpenCode resume that cannot move its session, with a timeout
The export and import step now has a 30 s bound, and a failure or timeout raises HarnessError instead of leaving the resumed turn waiting forever. A restart of a session the handler already ran skips the step.
* fix(harness): resume an OpenCode session elsewhere as a copy and fail the turn when it cannot
Importing the exported session moved it, so the original stopped working
where it was saved. Resume now imports a copy under fresh ids, filed under
the sandbox folder, and continues the copy. A failed or timed-out export or
import ends the next turn with runtime_error, as base does for a missing
session, instead of raising from agent_resume.
* fix(harness): retry a failed OpenCode resume on the next turn and keep resumed copies stable
Resume now only records the saved id, and the first turn runs the OpenCode export and import under the caller's timeout. A failed export fails that turn and the next turn tries again. The local sandbox kills a command when its run is cancelled, and the copy names the sandbox folder so OpenCode 1.15.5 and older keep it there.
* test(harness): keep subprocess tests out of tests/unit
The PWD regression test moves to tests/integration/sdk with an isolated child interpreter, and the cancel test checks collect_output with an in-process process instead of a real one. collect_output takes the sandbox Process protocol.
* fix(harness): rewrite only id fields when copying an OpenCode session for resume
* fix(bedrock): send batch jobs to a configured control-plane endpoint
Bedrock batch create, retrieve and cancel always built the regional
bedrock.{region} host, so a VPC endpoint or compatible gateway could not
be used. Add aws_bedrock_control_plane_endpoint as a deployment setting
read by the signed HTTP path and the boto3 path, carry it through the
proxy credential filter, and refuse it in request bodies like the other
endpoint overrides. api_base stays out of it because on a Bedrock
deployment it points at the runtime host.
Fixes#45056
* fix(bedrock): read the control-plane endpoint from typed kwargs in job status
The named parameter added one unknown-argument error in litellm/batches/main.py,
which unpacks untyped kwargs into this function
* docs(bedrock): explain batch control-plane endpoint configuration
* fix(bedrock): keep the batch control-plane endpoint out of chat request bodies
A deployment that sets aws_bedrock_control_plane_endpoint also serves chat,
and the unregistered key was swept into the Bedrock request body, so every
chat call on it failed with "Extra inputs are not permitted". Register it
with the other Bedrock batch connection fields so it stays a LiteLLM-owned
kwarg on every route while batches still read it from litellm_params.
* test(bedrock): verify endpoint signatures without internal mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill partial Responses API streams when the client disconnects
Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.
* fix(proxy): satisfy lint gates for partial Responses API disconnect billing
Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.
* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect
The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.
* Potential fix for pull request finding 'CodeQL / Log Injection'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Every /v1/responses spend log asked its worker to flush the queue right away so a
chained previous_response_id could find the row. Only the chat-completions bridge
reads that row. Native Responses providers hold the conversation upstream, so their
rows now wait for the batch write like chat rows do
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): replace Any with proven types in 9 shared files
* refactor(types): replace Any with proven types in the remaining sweep files
* refactor(types): keep unproven integrations on the merge base and pin malformed-body handling
* refactor(types): leave five request paths on the merge base so non-object bodies keep their 400
* refactor(types): drop a docstring that repeats the helper name
* refactor(types): leave six files with no live malformed-payload proof on the merge base
* refactor(types): leave Anthropic pass-through SSE parsing on the merge base
* refactor(types): leave Responses output walking, the retry ceiling and concatenated tool JSON on the merge base
* fix(types): keep main's parsing in the budget window reset, utils loaders and Langfuse attributes
The new strict checks rejected inputs main skips or keeps, such as a JSONB
string in budget_limits, so these three files go back to main's version
* refactor(types): return the request tag parse failure as a value
* fix(bedrock): return a 400 for malformed batch JSONL lines
A malformed line in a Bedrock batch input file escaped as a JSONDecodeError
from the upload and file content conversions, which the proxy reported as a
500. Parse those files line by line and raise a 400 that names the bad line.
Object name inference keeps its fallback for files that are not JSONL.
Fixes#45060
* test(bedrock): cover blank lines in malformed batch JSONL
* fix(bedrock): reject a malformed batch line whatever the upload's MIME type
The upload only converts, and so re-checks, batch files whose MIME type is on the
allowlist. Falling back to the generic object name let a malformed file sent as
application/x-jsonl skip both checks and land in S3 unchanged, so the object-name
step now runs the same strict per-line parse and raises the 400.
* fix(bedrock): reject every malformed batch input line with a 400 naming the line
Non-object lines, a missing or non-object body, a non-string body.model,
non-UTF-8 bytes, nesting too deep to parse, and rows the Bedrock mapping
cannot convert escaped as AttributeError, TypeError, UnicodeDecodeError or
RecursionError and surfaced as 500s. They now raise a 400 BedrockError in
the same "Batch input file line N ..." wording the proxy pre-check and the
Vertex adapter use. Object naming reads only the first line and never
raises, and a UTF-8 byte order mark is accepted.
* docs(bedrock): point the batch file name comment at the helper that builds it
* test(bedrock): mark new batch test locals Final and give the line builders explicit params
* test(bedrock): use single quotes inside the batch-line f-strings so the file parses on Python 3.10 and 3.11
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* test: bill each completed request once and price unpriced returned decision models at the requested model
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(logging,decisions): bill each request once and price returned decision models only when they carry rates
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(decisions): leave the double-billing fix to #47510 and keep the decisions pricing and missing-answers fixes
* fix(decisions): name malformed answer entries and pick the Arize cache-hit span by attribute
* test(decisions): cover a returned-model row with metadata but no rates
* fix(decisions): keep upstream reply text out of the parse error so it stays a 500
* fix(decisions): answer every malformed upstream reply with a 500, whatever text it carries
A reply that fails the decisions response schema now raises
DecisionsResponseValidationError, which the decisions call maps straight to
InternalServerError instead of passing it through exception_type. Its text
matching read an answer key like rate_limit, or a value like "Request timed
out", as a 429 or a 408, so the router cooled down a deployment for a reply
that fails the same way on every retry.
The message also drops the empty location for a reply that is not an
object, the Arize family test loses the cache_hit entries main already
added, and the Bedrock Mantle count-tokens wire test calls
_metadata_deployment, since main renamed the _deployment helper under it.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(proxy): size pass drops the oldest spend log partitions first
On a range-partitioned LiteLLM_SpendLogs the size pass drops fully
elapsed partitions oldest first, never the current or DEFAULT one,
remeasures after each drop and falls back to oldest-first row deletes
only while still above the target
* fix(proxy): a partition drop rebases the size state instead of reading as a statistics reset
Bugbot on #47449: dropping a partition removes that child's n_tup_ins and
n_tup_del from the summed probe, which advance_size_state took for a reset
and reseeded from n_dead_tup. The pass now saves the counters of the last
post-drop probe as the state's base and scales the reclaimable count to
the surviving files
* style(proxy): keep the mutable-ok reason on the line it suppresses
* fix(proxy): size pass stops when a post-drop re-measure fails and keeps day partitions ensured
A failed storage probe after a partition drop no longer reads as under target. The pass logs a warning, skips the row step and saves no count, so neither live rows get deleted on the pre-drop numbers nor the next check reads the drop as a statistics reset. The partitioned size path also calls ensure_partitions before dropping, so a size-only setup keeps getting day partitions instead of writing into DEFAULT
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stabilize auth and team delete checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make loop-blocking and tampered-signature checks deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): isolate model-cost state and synchronize loop progress
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): fail loop-progress handshakes when the blocking call never starts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in free-tier allowance for Google Search grounding spend
* fix(proxy): declare the search free tier counter's service target
* test(cost_map): allow search_free_requests in the cost map schema
* fix(proxy): skip cache hits and never free more than the billed cost in the search free tier
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds general_settings.maximum_spend_logs_size,
maximum_spend_logs_cleanup_target_size and
maximum_spend_logs_size_check_interval, the spend_log_size_check job,
maximum_spend_logs_cleanup_cron precedence, oldest-first bounded deletes
down to the target, storage fields in the run summary and the
litellm_spend_log_storage_bytes gauge, with a Postgres integration test
covering the post-VACUUM no-refire case
Resolves LIT-9831
Fixes#44589
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
_failure_spend_rows yields the final row plus one re-keyed row per lifted billed_failed_attempts entry (<request_id>-attempt-<position>, the attempt's usage, cost, retry count and deployment id), each written through DBSpendUpdateWriter.update_database so it also deducts from key, team and user spend
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update update_cache pins for synchronous write-back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): wrap long assert in update_cache tag test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover refill race and pin TTL with injected clock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count global proxy spend for every request with a user, cached on this worker or not
* fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter
The global proxy spend bump read the total, awaited, and wrote old + cost back
from a background task, so a budget reset that deleted the key in between got
the pre-reset total written back, and workers overwrote each other's totals.
RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key
exists, keeping its TTL, and this worker's copy is then set from the shared
total with no await between its read and write. The bump now runs whenever a
proxy max_budget is set, matching the DB writer, and the armed post-call read
and the batch-read declaration it used are removed.
* fix(proxy): report the global spend Redis increment to service metrics and request traces
The global proxy spend increment now emits Redis service success and
failure events under the spend_counters target and passes the request
span through, so it shows as redis.incr spend_counters like the write
it replaced.
The unit tests run the real RedisCache increment against fakeredis
instead of mocks, and a new test checks the span lands under the
request span on success and on a Redis failure.
* test(proxy): keep the spend counter Redis fake off the network
* test(proxy): count increment_if_present as a shared-cache call in the key-family guard
* fix(proxy): keep the newer global spend total when Redis replies arrive out of order
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): resolve AWS region and secret lookups off the event loop
Bedrock request paths that resolve their region, endpoint or bearer token
inside the async handler now do it in the AWS signing executor, so a slow
secret manager read no longer stalls every other request on the worker
* fix(bedrock): type the STS client kwargs and claude platform environment override
Keeps the basedpyright counts at or below the merge base after the off-loop wraps, by typing the STS client kwargs as a TypedDict, the claude platform environment override with concrete dicts, and the realtime region call through the typed getter
* test(bedrock): detect event loop stalls without wall-clock timing
The Vault fake now answers a read only after the test loop runs a callback, so a read made on the loop thread is caught deterministically instead of by a sleep and a heartbeat threshold
* fix(completion): send prompt_cache_retention to OpenAI instead of silently dropping it
litellm.completion(), Router.completion() and the proxy's /v1/chat/completions
accepted prompt_cache_retention and listed it as a supported OpenAI param, but
completion() never handed it to get_optional_params, and the leftover-kwargs
paths skip known OpenAI params, so it never reached the request body. Make it a
named argument of completion() and acompletion() like prompt_cache_key and pass
it on.
A unit guard sends every OpenAI body param through completion() and fails when a
param added to OPENAI_CHAT_COMPLETION_PARAMS has no case proving it reaches the
wire, the same hole store and prompt_cache_key fell into in #33184. Live e2e
tests capture the body OpenAI receives for the proxy, the SDK and the Router on
both openai/gpt-5.5 and openai/responses/gpt-5.5.
* chore(completion): type the param-forwarding unit test and keep the basedpyright count flat
Every new completion() kwarg adds one unknown-argument error at each call
site that splats an untyped dict into it; typing print_verbose's argument
and Chat's params removes more than prompt_cache_retention added.
* test: freeze the outbound-body model and read captured requests from the mock
Addresses review: the e2e body model is frozen, the backend params are a
tuple, and the unit test reads the request from the respx route instead of
appending to a list.
* fix(cost): price base_model deployments past rule-only cost map entries and warn when base_model has no rates
* fix(router): warn on unpriced base_model from add_deployment too and honour model_info prices
* refactor(cost): drop the unpriced base_model startup warning, keep the one-line ladder fix
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: isolate model cost maps in runtime registration tests
* test: restore cost map entries and provider model lists after unit tests
* test: restore provider model sets after unit tests and cover the catalog restore
* test: snapshot litellm globals before scanning provider model sets
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): lift every earlier billed timed-out retry attempt onto the failed request
_billed_earlier_attempts reads the router's previous_models records, keeps the Timeout ones other than the final attempt (matched on deployment_id plus attempted_retries, since the count restarts after a fallback hop and the fallback handler re-logs the exhausting attempt) and prices each at the deployment that served it. post_call_failure_hook lifts the tuple as billed_failed_attempts; nothing writes it yet. previous_models joins the router-reserved metadata fields so a client-planted record never reaches it
* fix(proxy): price earlier timed-out attempts from the prompt estimate only
The final row may lift a mid-stream recovered usage (prompt plus delivered chunks) or a failed_attempts_usage sum; earlier attempts never produced output, so they are priced from _estimated_prompt_usage, and when failed_attempts_usage already bills them into the final row none is lifted again
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count-tokens): translate chat tool use before Claude token counts
Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent.
* test(count-tokens): compare Claude count bodies with inference for Responses tool use
* fix(responses): forward the caller's Responses body to OpenAI token counts
OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it
* fix(responses)!: count Claude Responses bodies the way inference sends them
Non-native Responses counts now run the body through the inference bridge, so
text.format, tool_choice, reasoning, built-in tools, and file_search count the
way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The
OpenAI native count body uses the drop_params-resolved tool fields.
* fix(responses): honor drop_params and replay previous turns in Claude counts
Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation
* fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts
* fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params
* fix(responses): leave tool_choice out of counts when drop_params drops every tool
* fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send
The Bedrock Mantle fallback now counts thinking, tool_choice and output_config
like the runtime count. Claude Responses counts leave out cache_control marks
again, so more than four marks count instead of failing. Counts answer 400
before any provider call for user blocks chat inference refuses and for
Responses tools with no Chat Completions equivalent, instead of dropping the
block or counting a placeholder tool. Adds the wire tests for these paths
* fix(count-tokens): count Responses tools the way the inference bridge sends them
A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request
* fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id
* test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler
* fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent
PR #47394 withheld cache_control_injection_points from the OpenAI SDK extra_body for
every provider on that path, which also covers litellm_proxy. The downstream LiteLLM
proxy applies the point itself, so litellm_proxy keeps forwarding it while OpenAI and the
other OpenAI-compatible providers still never receive it
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
get_standard_logging_object_payload decided custom_pricing from
litellm_params alone, so on an Azure PTU spillover response
model_map_information kept the deployment's zeroed PTU entry while
response_cost already used the standard model rate. Reuse the billing
side LitellmLogging._custom_pricing_for so both read the same decision
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic, bedrock): cover a redacted_thinking content item on every Claude route
* fix(anthropic, bedrock): keep a redacted_thinking content item on every Claude route
* fix(anthropic, bedrock): send each replayed thinking block once, without cache_control or blank data
* fix(anthropic, bedrock): keep thinking order and gate replay key drops on drop_params
A thinking block that only one of content and thinking_blocks carries now goes
back in its original place instead of being prepended. cache_control on a
thinking block and blank redacted data are dropped only under drop_params and
otherwise reach Claude as sent. Converse drops a cachePoint after reasoning
under drop_params, skips a thinking_blocks entry that is not a block, and the
cache control hook no longer marks a thinking part.
* fix(anthropic, snowflake): match a signed thinking block on its signature and send each block once on Snowflake Cortex
A signed thinking block in both content and thinking_blocks is the same block when the signatures match, even if
the thinking text differs, so it is sent once from content instead of twice. Snowflake Cortex now merges
thinking_blocks into content the same way the Anthropic routes do. A redacted_thinking item with blank data no
longer counts as sent for system message placement.
* fix(anthropic, bedrock): send a thinking block once when consecutive assistant messages both carry it
Consecutive assistant messages merge into one Claude turn, so a block one message holds in content and the next
holds in thinking_blocks went out twice and Claude answered 400. The merged turn now keeps the first copy of each
redacted or signed block on the Anthropic routes and both Bedrock Converse builders.
Snowflake Cortex goes back to its previous message building, since its double send cannot be proven live without a
Cortex account and is tracked on its own.
* test(anthropic, bedrock): cover a redacted block with null data and a non-block part beside a cache breakpoint
* fix(anthropic, bedrock): match only text-valued thinking items when merging replayed blocks
* fix(langfuse_otel): send the completion start time
langfuse_otel never set langfuse.observation.completion_start_time, so
Langfuse showed no time to first token for any call, streamed or not.
The legacy langfuse callback already sends kwargs["completion_start_time"].
Send the same value as an ISO timestamp with an explicit offset.
* fix(langfuse_otel): type kwargs and test against a real span
Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any],
the declared type of model_call_details, so the new completion start time
read adds no unknown-type errors. The tests now check a real SDK span
instead of patching safe_set_attribute, and the parameter is typed.
* fix(otel): send the completion start time from the v2 Langfuse mapper
With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper,
which had no completion start time either. Carry the payload's
completionStartTime on LLMCallSpanData for streamed calls, the same calls
that get time_to_first_chunk_seconds, and map it to
langfuse.observation.completion_start_time.
* fix(langfuse_otel): take the logging kwargs as a read-only Mapping
The new kwargs annotation tripped LIT001. Both helpers only read the
logging kwargs, so type them Mapping[str, Any] instead of dict.
* fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter
* fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON
The native Gemini, Interactions and A2A streams never recorded a first chunk, so with
completion_start_time now sent both emitters showed TTFT equal to latency on them. They
now record it, and Gemini and Interactions take the call's start time like /v1/messages.
Both emitters send the completion start time as a JSON string, the form the Langfuse SDK
and the langfuse callback send, through one serializer.
* fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event
A2A streams took their start time when the iterator was built, so their latency left out
the work before it. They now use the call's start time like every other route.
Cached /v1/messages stream replays recorded no first event, so their time to first token
fell back to the end time. The replay now records it on the first event it yields.
The tests use the real Interactions config and typed logger hooks.
* fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free
* test(a2a): type the card fetch test and stop rebinding the seed helper's recorder
* test(otel): expect completion_start_time and cache_hit in the B3 span keys
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost-map): add volcengine doubao-seed-2-1-pro-260915 and doubao-seed-2-1-lite-260915
Pro row absorbed from #47372, lite row added from the same Volcengine Ark pricing and model list docs
Co-authored-by: donghongfei <1247543758@qq.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-map): cover volcengine doubao-seed-2-1-pro-260915 row
Co-authored-by: donghongfei <1247543758@qq.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): add undated azure/eu and azure/us rows for o1, o3, o3-mini, o4-mini, gpt-4o, gpt-4o-mini
Absorbed from #47499
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: donghongfei <1247543758@qq.com>
* fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): use lane AWS credentials for the Kimi K3 messages test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bypass the proxy cache and declare test metadata for the Kimi K3 probe
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): billed background responses count toward the creator's project and model budgets
* fix(responses): carry the creator's project and per-model budgets on the billed background response
* fix(responses): count a key's budget tier toward background per-model budgets
* fix(responses): bill a background response's project only while the key stays on its team
* revert(responses): keep billing the creating key's project after a team move
A key can only keep or detach its project, so the billed project always pairs with the team it was created under. This reverts commit ded4f084c5.
* fix(responses): bill a background response under the model name the client sent, with its project alias
* fix(responses): name the team's public model as a billed background response's model group
The model group lookup now passes the creating key's team, so a team model or team wildcard bills under the public name a normal call logs and per-model budgets on that name move
Vertex AI and Gemini context caching now remember each cache they create or find until 10s before its expireTime, so a repeat cache-marked request goes straight to generateContent instead of listing every cachedContents first. Concurrent async requests for one prefix share one lookup or create
When generation rejects a remembered cache with a 400, 403 or 404 that names the cached content, the request rebuilds the cache and sends once more. Any other 400, 403 or 404 drops the record without a retry
Fixes#36395
* feat(embeddings): add max_batch_inputs pre-call validation
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* add unit tests for max embedding batch size
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* feat(embeddings): enforce batch limit on extra_body input override
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* chore(dashboard): update schema.d.ts for embedding batch size model field
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): close Bedrock batch bypass and satisfy ruff PT012
Count max(top-level input, extra_body.input) for batch limits so providers
that ignore extra_body cannot bypass max_embedding_batch_size. Add router
tests and refactor pytest.raises blocks for ruff-tests.
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): drop unchecked casts in router batch-size check
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): route oversized batches to a group deployment that admits them
* fix(embeddings): count extra_body input for any top-level input and prefer admitting deployments before cooldown
* fix(embeddings): log cap rejections to failure callbacks and scope batch routing to embedding calls
* fix(embeddings): apply the batch cap after tag and pre-call filters and fall back on cap rejections
* fix(router): keep embedding cap rejections out of deployment health and hold only for routable cooling deployments
* fix(router): clear the embedding cap rejection mark before a fallback's provider call
A fallback or retry shares the request's logging object, so the mark a rejected attempt set made telemetry skip the fallback deployment's real call. Unit tests now pass a fixed logging start time instead of reading the clock.
* fix(router): log embedding batch rejections once, before a fallback runs
* refactor(router): drop docstrings that restate the code
---------
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): let proxy admin keys read raw cloud storage file ids on GET /files/{id}/content
Since #30595 the endpoint answered 400 for every raw gs:// or s3:// id, which broke the
documented unmanaged Vertex batch flow at its last step, while a double-encoded id slipped
past the guard entirely. Raw ids are now readable by proxy admin keys only, every other key
gets 403, and the guard decodes the id fully before deciding, so encoding depth no longer matters
* fix(proxy): decode nested percent-encoded file ids iteratively
An id percent-encoded a thousand or more times sent the recursive decoder past the
interpreter's recursion limit, so a proxy admin key got a 500 instead of a verdict on
the id. The decoder now walks the encodings in a loop, so any depth resolves
* fix(proxy): cap the decode passes the raw cloud storage file id guard makes
Any authenticated key could send a file id wrapped in tens of thousands of percent-encoding layers and the guard would unwrap every one of them, each pass walking the whole id, so a 120 KB request line held a worker for about five seconds and a 2 MB one for minutes. The guard now unwraps at most eight layers, and an id still changing after that is treated as a raw cloud storage id, so it stays admin-only instead of being decoded any further
* refactor(proxy): type the decode step the raw cloud storage file id guard folds over
* test(proxy): return a streaming result from the admin raw id test mock now that vertex_ai file content streams
* test(proxy): send raw cloud ids to the content route the way uvicorn delivers them
Starlette's TestClient decodes the path twice, so the twice-encoded cases reached the route fully decoded and never hit the decode loop. The cases now name the id the route receives, and the helper tests take a typed quote helper and Final constants.
* test(proxy): mark the raw cloud id test locals Final
* fix(otel): leave out span cost when the price lookup failed
A model missing from the price map gets response_cost 0.0 in the standard logging payload, with response_cost_failure_debug_info set. The OpenInference writer (arize, phoenix, langfuse_otel) and the OTel v2 span data copied that 0.0 into llm.cost.total, litellm.cost.total and langfuse.observation.cost_details, so backends showed the call as free. Skip the cost when the lookup failed, except on cache hits, which are free.
* test(otel): keep the guardrail charge on the guardrail span for unpriced calls
* refactor(otel): type the Arize cost writer input and reuse the cache_hit flag
* fix(otel): keep a billed guardrail fee on an unpriced call's Arize span cost
Only the placeholder 0 is left out when the price lookup failed, so a
guardrail fee LiteLLM billed still reaches Arize, Phoenix, Langfuse OTel
and W&B Weave OTel. Adds unit cases for that fee and a malformed cost.
* fix(arize): leave out the span cost on any failed price lookup, matching OTel v2
* fix(logging): clear a stale cost failure flag when a later pricing succeeds
Stream-start pricing on the request model name can fail and set the cost
failure flag even when the final pricing succeeds (OpenRouter usage.cost,
an Azure deployment name priced from the response model). The new span gate
then dropped a real cost. Each pricing now resets the flag, and
uncached_price restores whatever flag it found.
* test(logging): pin the start time in the stale cost flag tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
