* feat(vertex-ai): support gemini 3.1 flash tts
* fix(vertex-ai): keep TTS metadata compatible with the cost map guard
* fix(vertex-ai): preserve existing Gemini TTS models and voice mappings
* fix(vertex-ai): recognize provider-stripped TTS model names
* test(vertex-ai): match current speech HTTP error response
* fix(vertex-ai): preserve nested speech locale and voice
* fix(vertex-ai): use provider speech media type in proxy
* fix(gemini): preserve TTS dispatch and nested speech settings
* test(gemini): run TTS regressions in CI shards
* fix(vertex-ai): account for Gemini Cloud TTS audio cost
* fix(vertex-ai): bill Gemini TTS without optional audio decoder
* test(gemini): cover TTS mapping and billing edge cases
* test(vertex-ai): cover Gemini voice normalization in provider shard
* fix(vertex-ai): keep Gemini TTS routing with generic provider metadata
* chore(vertex-ai): remove obsolete lint suppressions
* fix(gemini): strip audio format from speech configuration
* chore(vertex-ai): remove obsolete mutation suppressions
* fix(vertex-ai): preserve FLAC billing and wrapped audio headers
* fix(vertex-ai): reject unmeasurable FLAC before billing
* fix(vertex-ai): reconcile speech support with current provider mappings
* fix(vertex-ai): validate speech mappings and usage types
* fix(vertex-ai): preserve speech mapper override forwarding
* fix(gemini): map voice aliases across speech request forms
* fix(vertex-ai): send only formats the speech bridge cannot serve to Cloud TTS
Gemini TTS detection now goes by model name, so newer Vertex Gemini TTS
models keep working, and only mp3, opus, alaw, mulaw and other container
formats move to Cloud Text-to-Speech. wav and pcm requests stay on
generateContent as on main.
A Cloud TTS response whose duration cannot be measured now returns the
audio unbilled with a warning instead of raising after a paid call. FLAC
handling is removed because Cloud TTS has no FLAC encoding.
The cost map, the google_genai mappers and the proxy error path go back
to main, since those changes were outside this feature.
* fix(vertex_ai): keep Gemini TTS formats, region and instructions on routes that serve them
Only mp3, opus, alaw and mulaw go to Cloud Text-to-Speech. Every other
format gets a local 400 that lists the supported ones, before Google is
called.
Cloud TTS uses the deployment's us or eu regional host. Other regions
and a custom api_base get a 400 that says to request wav or pcm.
instructions is sent as the Cloud TTS style prompt, and a request with
no voice keeps the Gemini model.
get_usage_object is back to main's code, so dict usages still go
through Usage.__init__, and the proxy no longer labels pcm16 or
linear16 output as WAV.
* fix(vertex_ai): send Gemini TTS to every regional Cloud TTS host and keep speak tags as text
Cloud Text-to-Speech serves Gemini TTS on a host named after each Vertex
location, so a non-global vertex_location now uses
{location}-texttospeech.googleapis.com instead of only the us and eu hosts.
Google answers for regions it does not serve.
Gemini TTS voices reject SSML, so input that contains <speak> is sent as
text for Gemini TTS models. Chirp keeps the SSML auto-detect.
Removes the proxy alaw and mulaw media-type entries. Cloud TTS returns
those encodings with a WAV header, so the response is labeled audio/wav
and the entries never applied.
Adds a regression test that custom per-token pricing bills the usage a
speech response reports.
* fix(vertex_ai): map Gemini TTS regions to Google's Cloud TTS locations and keep a bad voice from cooling down the deployment
US and EU Vertex regions go to the us and eu Cloud TTS hosts, northamerica-northeast1 to its own, global or unset to global, and any other region gets a local 400. A Cloud TTS 404 on the Gemini path, such as a misspelled voice, comes back as a 400 so the router keeps the deployment serving wav. Usage reads the sent request from the response instead of the logging object.
* fix(vertex_ai): default Gemini TTS Cloud formats to a Gemini voice and validate speakers
Parse only usage fields from the sent Cloud TTS request, so a numeric speakingRate no longer fails after Google bills it.
Send the Gemini voice Kore when no voice is named, and reject Cloud TTS voice names unless drop_params is set.
Reject malformed speaker entries unless drop_params is set, and translate Cloud TTS speaker fields on wav and chat audio.
Use the Vertex default region when no location is set, so mp3 and wav share a region.
* test(vertex_ai): cover two-speaker Gemini TTS mp3 through aspeech with fake credentials
Replaces two tests that patched the Vertex TTS config's own token and URL methods.
* fix(vertex_ai): reject malformed Gemini TTS voices and keep beta Chirp and Lyria params unmapped
* fix(vertex_ai): drop unknown Gemini TTS voice keys on wav and pcm instead of rejecting them
* fix(vertex_ai): reject unknown Gemini TTS voice keys on wav, pcm, and chat audio unless drop_params is set
* fix(vertex_ai): honor every drop_params setting in the Gemini TTS voice key check
Global drop_params, batch row and deployment drop_params on Vertex batch
file uploads now drop unknown voice keys. A voice object with both
speechConfig and speech_config gets a 400 for the ignored speech_config.
* fix(gemini): drop unknown nested voice fields under drop_params and detect TTS by served model
* fix(gemini): filter unknown speech config fields with fixed-depth helpers and pass audio params through unchanged
* fix(gemini): camel-case snake replicated voice keys so drop_params keeps them
* fix(vertex-ai): reject voice fields Cloud TTS cannot send unless drop_params is set
* fix(vertex-ai): reject wrong-typed Cloud TTS voice values and check speakers as sent
Gemini TTS on mp3, opus, alaw, or mulaw now answers 400 for a voice field whose value Cloud TTS cannot send, such as a list languageCode or a string voiceConfig, and for nested voice fields under a speaker that also names speakerId. drop_params still drops them.
* fix(vertex-ai): reject wrong-typed speech config wrappers and speakerId overriding a prebuilt voice on Cloud TTS
* fix(vertex-ai): keep speakers and voice_config beside a speech_config wrapper on Cloud TTS
* fix(vertex-ai): pick Cloud TTS voices from the same speech config the WAV path sends to Gemini
* fix(files): carry the deployment's drop_params on model= file uploads so Vertex batch rows drop unknown voice keys
* fix(gemini): keep a structured speech voice when the model is a model_alias_map alias for a Gemini TTS model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): send pi's session id from lite pi so sessions group and stick
* docs(cli): drop the provider_block docstring lines the README already covers
* docs(cli): note the session id format the proxy reads for lite pi
* test(integration): keep autovacuum off the size cleanup test's scratch tables
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(circleci): keep the MCP SDK1 peer venv on mcp 1.x despite the project override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover billing every round of a streamed MCP auto-execute
* fix(responses): bill every round of a streamed MCP auto-execute on one spend row
* fix(responses): bill every round of a streamed MCP auto-execute on bridged providers
A streamed /v1/responses MCP auto-execute on a provider bridged through chat
completions (for example Anthropic) billed only round 1, because each round's
chat stream dispatched on the shared logging object and the first dispatch won.
Bridged rounds now go through the same one-row billing as native rounds. A
round is billed as soon as its chat stream ends, before its closing events
reach the client. The row carries the final response, usage and cost summed
over every round, and the fixed margin once. A round served from the response
cache is billed at zero. A follow-up chat stream that breaks bills the paid
rounds on its failure row. A single-round stream logs exactly as before.
* fix(responses): charge one fixed margin and keep bridged usage fields on streamed MCP rounds
A follow-up round that fails after earlier rounds were billed now charges the
per-request fixed margin once, and its failure row carries no cost breakdown
that would disagree with the summed cost.
A bridged multi-round stream logs a plain cache_hit flag like a single-round
stream does, and its usage keeps the summed provider usage fields of every
round.
A new test covers a stream dropped after its event loop closed, which must
release its parked log without an unawaited coroutine warning.
* fix(responses): price each streamed MCP round on its own and bill every round when a guardrail blocks
Each round of a streamed MCP auto-execute is now priced from its own
uncached cost, so a provider that states its cost (xAI) charges the stated
cost of every round plus the fixed margin once, and the cost breakdown
always agrees with the billed cost.
Summed usage goes through a JSON-safe dump, so a nested provider field such
as xAI's context_details no longer breaks a failed stream's response.failed
event.
A post-call guardrail that blocks a multi-round bridged stream now drops the
parked success log and bills every round on the failure row, and a failed
follow-up round's failure row carries the summed breakdown of both rounds.
* fix(responses): keep earlier MCP rounds on their success row when a later round fails
When a later round of a streamed MCP auto-execute fails, the rounds before
it are now logged on their success row, and the failure row bills only the
failed round with no second fixed margin. That failure log runs on its own
copy of the request's logging state, so its usage and cost never leak into
the earlier rounds' row.
A post-call guardrail that blocks a native multi-round stream keeps the
summed success row beside the $0 failure row, as at the merge base, while a
bridged stream still bills every round on its failure row.
* fix(responses): keep a failed later MCP round on its own failure row
A streamed MCP auto-execute whose later round failed wrote only the earlier
rounds' success row. The failure was logged on a copy of the logging state,
while the proxy builds its failure row from the request's own state, which by
then held the success payload, so the failure row took the success row's id
and was dropped.
Now the failure is logged on the request's own state as at the merge base,
and once a later round starts, the success row for the rounds before it is
logged from a copy of the state taken at that moment, so the two rows never
mix.
* fix(proxy): let an MCP tool call reuse its request's max_parallel_requests slot
* test(proxy): create the other-request task in a way Python 3.10 supports and mark its bindings Final
* fix(proxy): keep a reused max_parallel slot until the request's last round
An MCP tool call that reuses its request's max_parallel_requests slot no
longer loses it to the first round's late success callback, so the key
stays counted for the rounds still to run. The reused call also stops
replacing the request's rate limit headers.
* fix(responses): keep an unpriced streamed MCP round unpriced instead of billing it as free
* fix(responses): log a bridged MCP stream as a chat completion with every round's usage
* feat(gemini): LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS sets the long decode budget for requests that send no limit
* test(gemini): cover a default long decode budget set by LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS
* fix(gemini): apply the default output budget only to requests asking for text output
* refactor(gemini): apply the default output budget once in the async request transform
* fix(gemini): leave the default output budget off non-chat models and a snake_case generation_config
* fix(gemini): skip the default output budget for audio-only models and continuation resumes
Lyria and other Gemini chat models whose cost map row lists no text output answer 400 when sent a maxOutputTokens, and a request that resumes from a continuation token answers 400 when a limit is added beside it. Both now go out without the default budget.
* fix(vertex_ai): skip the Gemini default budget for a model a deployment's model_info overrides, and pass odd generationConfig shapes through
* fix(vertex_ai): cap the Gemini default budget per deployment at the loaded cost map and its own model_info limit
* test(vertex_ai): wrap a long test signature under the 120 character limit
* fix(vertex_ai): keep a direct call's default output budget under the loaded cost map limit
A litellm.Router deployment's model_info writes onto the shared cost map row, so a later direct call with no model_info read that raised value and Vertex answered 400. The direct call now caps at the lower of the loaded row and the live row, so litellm.register_model can still lower the cap.
* fix(vertex_ai): ignore a deployment output limit that is not positive
A model_info.max_output_tokens of 0 or -1 capped the default budget at that value, and Google answers 400 to a limit that is not positive. It now counts as unreadable, so the cost map limit applies.
* fix(gemini): never continue a request asking for more than one candidate in camelCase
* fix(gemini): keep continuing a request whose candidate count is null or 0
* fix(router): unregister strategy selectors from global callbacks on discard
Router.discard only removed its own bound methods from litellm's callback
lists, so the strategy selectors it registered (least-busy, usage-based,
latency-based, cost-based and routing group selectors) stayed behind and
kept receiving events after the router was gone. Reuse the identity based
unregister that update_settings already uses so a discarded router takes
only its own selectors with it
* fix(router): purge discarded selectors from every event callback list
function_setup copies litellm.callbacks into the success and failure
lists on every request, so a router that served traffic still leaked its
selectors there after discard. Route the cleanup through
remove_callback_from_all_lists, which now also covers input_callback, and
fold the selector gathering into discard so the router coverage check
sees no untested helper
* fix(router): keep live routers, in-flight hooks and user_config limits intact on discard
discard() now rebuilds every global callback list without the router's selectors, optional pre-call checks and own usage callbacks instead of removing them in place, so a callback loop already running never skips a hook. It then re-registers what the remaining live routers need, so a same-strategy survivor keeps its deduped handler. The proxy's per-request user_config router is discarded only after its call returns.
* fix(router): bind user_config handlers to the request and tighten discard
When a user_config request carries a logging object, its router is
discarded before the request runs and its handlers ride on that
logging object, so v1 rpm and v2 tpm limits count every request and
concurrent requests count once. Without one, the router is discarded
after the request as before. The router's pre-call loops also read
the request-scoped handlers.
discard() now refills only the handler keys it vacated, guards the
callback lists and the live-router set with a lock, covers routers
built without a model_list, and drops its Slack alerting, adaptive
router hook and budget limiter sync task
* fix(proxy): reuse one user_config router per config and bind its handlers to each request
Every user_config request now goes through a cached router built once per
config. Its strategy handlers leave the global callback lists and are bound
to the request's logging object instead, so limits count each request once
and nothing is left behind after the request.
* fix(router): keep the proxy budget limiter on user_config requests
A user_config router now swaps in only its strategy selectors, so the proxy's
RouterBudgetLimiting stays on the request and keeps counting its spend. Usage
logged after a strategy is retired, such as a request still in flight when its
user_config router is evicted, now schedules one more push to Redis.
* fix(proxy): init the retire fields on the key model budget limiter
The key model budget limiter builds its own state instead of calling the base
init, so with the proxy's Redis attached every spend increment raised on the
missing retire flag and the key's model budget never counted spend.
* fix(router): bind the deployment callbacks to user_config requests
A cached user_config router serves requests through each request's logging
object but bound only its selectors and policies there, so a failing
deployment was never cooled down and its usage counters missed requests.
* fix(router): retry an evicted router's budget writes and keep the adaptive header on user_config requests
A retired budget limiter now keeps retrying its queued Redis increments every
sync interval until Redis accepts them, instead of giving up after one failed
write.
The x-litellm-adaptive-router-model header now comes from one stateless hook in
litellm.callbacks that holds no router, so user_config adaptive routers, whose
per-router hooks are request-scoped, still answer with the header.
* fix(router): lock request callback setup against router cleanup and drain in-flight budget writes on retire
function_setup now edits the global callback lists under the same lock that
router cleanup uses, so a router dropping its callbacks on another thread can
no longer shift the indexes function_setup pops or put a dropped handler back.
A budget limiter retired while a Redis write is still in flight now keeps
retrying that batch if the write fails, instead of leaving the spend queued
with no task left to send it.
* fix(utils): set up loggers after releasing the callback lists lock
function_setup held the callback lists lock while set_callbacks ran its one-time
logger setup, which can block on a Slack package install, so other threads and
Router.discard waited on it. The lock now covers only the list reads and writes.
* test(router): annotate the retire test's locals as Final
* fix(logging): skip replaced global callbacks on the input path too
replace_global_callback already kept a same-key global logger out of
success and failure dispatch. pre_call and post_call still sent it the
input events, so a logger with an input hook could count a request start
it never saw finish. Both now share one filter with the success path
* test(logging): type the input-path regression test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The OpenAI fine-tuning handler built its SDK client from every local it
received, so an api_version or litellm_params set on an openai
finetune_settings entry reached OpenAI.__init__ and every fine-tuning
call on that provider failed with "unexpected keyword argument
'api_version'". Build the client from its named params instead and
cover it with a request-level regression test
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tokenizer): isolate HuggingFace tokenizer caches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): clear the traceback before re-raising a cached tokenizer load error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): retry failed HuggingFace tokenizer loads after the cache window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): start tokenizer retry TTL after a failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): lock every HuggingFace tokenizer load and keep load failures as values
Concurrent llama-2, llama-3 and cohere counts each started their own failed
download because only the bundled tokenizers took a load lock. Every tokenizer
kind now has one, so a burst during an outage makes a single attempt and two
counts can no longer read the same expiring entry at once.
A failed load is now remembered as a small value with the error message
instead of the error object, so re-raising it no longer keeps the first
failing request's frames alive. Adds tests for the concurrent outage and for
fork-guard errors staying raised and unremembered.
* fix(utils): keep Hugging Face tokenizer downloads unlocked and warn once per failed download
Reverts the per-kind load lock so concurrent first downloads still run in parallel, as #47353 intends. A short lock around the failure cache read and write stops two counts racing on an expiring entry, and the tiktoken fallback warning now fires only when a download fails, not on every count that reuses the remembered failure.
* fix(caching): read an entry another reader just expired as a cache miss
Two threads reading the same expiring key in InMemoryCache could both pass the membership check, and the second then raised KeyError after the first removed it. get_cache now treats that key as a miss. This replaces the extra lock the tokenizer failure cache took, and the regression test forces the interleaving without timing.
* fix(caching): treat a TTL another reader just removed as not expired
InMemoryCache's expiry check read the TTL after a separate membership check, so a reader racing another reader's eviction raised KeyError. It now reads the TTL once and treats a missing one as not expired, and get_cache already reads the vanished entry as a miss.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): delete a group through the team delete's bulk path instead of one member at a time
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(scim): demote a deleted admin group's members before the team delete so a failed delete stays retryable
* fix(scim): recompute member roles again after the team delete so a concurrent group write cannot regrant admin
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(scim): write recomputed member roles only where the membership still backs them
* fix(team): report the users a team delete detaches so SCIM demotes late-added members
* test(scim): audit cells for the bulk group delete
Move the SCIM group helpers into tests/integration/_support/scim.py and add the delete cells: a member's other
group survives, the admin group's members are demoted and the audit row names the SCIM caller, a member a
concurrent PUT adds while the delete waits on the team lock is demoted too, unknown, repeated, non-admin and
hostile ids, an empty group, the peer proxy, a 20-delete burst across both proxies, and a delete after a worker
was SIGKILLed
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(team): sweep deleted team references only under the lock so a late-added member is demoted
The unlocked sweep that ran before the team's advisory lock stripped a member a
concurrent group write added after the roster was read, so neither the roster
snapshot nor the locked sweep's RETURNING named them and the SCIM recompute
left them proxy_admin after the admin group was gone. The locked sweep does the
same strip and reports whom it stripped, so it is the only sweep now.
The integration cell holds the delete on a member's user row so a PATCH lands
between the roster read and the lock, then checks the late member is demoted.
Both waiter queries in the SCIM support module return their count as text so
the first poll parses.
* chore(team): carry the unbounded-IN baseline entries to delete_validated_teams
The delete pass moved from delete_team and the unlocked sweep into delete_validated_teams, so the checker's baseline names the function the two team_id IN lists now live in
* test(scim): expect the master key alias in the audit row and send post-kill requests on a fresh connection
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* test(scim): cover deleting a SCIM group through the team delete's bulk path
* fix(scim): evict a deleted group's member caches in bulk and drop cached roles after a recompute
* fix(scim): answer 404 to a group delete that lost a race to another delete
* fix(scim): chunk the locked team lookup and delete in the team delete transaction
* fix(scim): keep service-account keys and team models on a group delete and write its records once under the lock
* fix(scim): demote a deleted group's members inside the delete transaction
The admin-group role write now runs on the team delete's transaction, over the
roster read under the team lock, so a failed delete leaves roles untouched and
a member added while the delete waited is demoted with the rest. The delete
transaction also waits for a free connection as long as it may run, so a burst
of deletes queues instead of failing after Prisma's two-second default. Every
member in the 5-vs-500 statement test now holds a key.
* fix(audit): write deleted keys' audit rows in one insert
A delete that removes many keys wrote one audit insert task per key, and those tasks held pool connections the delete's follow-up writes waited on. The race test now waits on any lock wait behind the delete.
* fix(ui): show the budget spend limit and rate limits without a collapsed section
The create and edit budget dialogs hid max budget and reset period behind a collapsed Optional Settings section that sat 80px below the last field, and dropped both values from the request unless the section was open on submit
Both dialogs now share BudgetFormFields. Budget ID comes first, then the spend limit as one row ("$500 per month"), then the three rate limits as short fields beside their labels. A secondary Optional badge marks the spend limit and the rate limits group, so people can still tell those fields are optional. Every field is always submitted, and the edit dialog sends the existing spend limit and reset period back unchanged
* fix(ui): send only changed fields when editing a budget
Saving the edit budget dialog resent every field, so an unrelated rate
limit edit rounded a stored sub-cent spend limit to 2 decimals (0.001
became 0) and resent the reset period, which makes the proxy recompute
the next reset for custom durations. The dialog now sends the budget ID
plus the fields the user changed, and the proxy keeps everything else.
Also updates the budgets Playwright spec for the new field labels and
the removed Optional Settings section.
* fix(ui): track budget edit changes from render so a reverted field cannot hide a later one
react-hook-form's formState is a render-time snapshot. Reading dirtyFields
only inside the submit handler meant that after a field was changed and
changed back, a later edit to another field was missing from the first
save. Reading dirtyFields during render keeps the snapshot current.
* refactor(ui): rename the budget components to PascalCase
Renames budget_modal, edit_budget_modal and budget_panel (and their tests) to BudgetModal, EditBudgetModal and BudgetPanel, and drops their filename-pascal-case suppressions.
* feat(model hub): show admin-defined custom tags and metadata from model_info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): move model hub custom field merging into router_utils
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): share the custom tag badge list between the column and details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(model hub): cover public hub, patch, null, hostile, forwarding, and killed-worker paths for custom fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): drop custom_metadata entries with non-string keys or non-finite numbers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): use isinstance narrowing instead of TypeGuard for custom_metadata values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): build custom tags and metadata in the proxy instead of the router
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): serve custom tags and metadata on model_group_alias rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remove docstring from model_group_custom_fields.py
* style(model hub): fix lint and format in custom metadata tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(model hub): drop leading blank line in custom fields helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later
Google ignores temperature, top_p and top_k on Gemini 3.6 Flash, Gemini 3.5
Flash-Lite and every later Gemini release, and says a later generation will
return 400 for them. The model map now carries supports_sampling_params: false
on those rows, and the Gemini and Vertex transformation reads the flag the way
the Anthropic and Bedrock paths already do, dropping the params under
drop_params and otherwise raising the client-side 400 that names drop_params
BREAKING CHANGE: a request that sends temperature, top_p or top_k to a flagged
Gemini model without drop_params now gets a 400 instead of a 200 whose
sampling params Google silently ignored
Fixes#47015
* fix(gemini): gate the passed-through top_k where the request body is built and keep allowed_openai_params working
top_k is a provider-specific kwarg that never reaches the OpenAI param mapping,
so the flag has to be applied while the Gemini request body is built, the same
boundary Anthropic uses. The mapping now skips a flagged sampling param instead
of raising, because the shared unsupported-param check already returns the 400
that names drop_params, and a param the caller lists in allowed_openai_params
is forwarded verbatim as that 400 promises
* fix(gemini): filter top_k out of the generation config instead of mutating optional_params
* test(gemini): type the sampling-flag twin test and keep its locals out of the loop body
* fix(vertex_ai): reject or drop sampling params on batch rows for flagged Gemini models
The batch JSONL wrapper calls map_openai_params directly and skips get_optional_params,
so a flagged model's temperature, top_p and top_k vanished from the uploaded row with no
400. Batch rows now get the same contract as a real-time call: a 400 naming drop_params,
or the knobs dropped when litellm.drop_params is on
* fix(vertex_ai): ignore null sampling knobs on batch rows for flagged Gemini models
A batch row carrying temperature: null, top_p: null or top_k: null means no
override, the same as on a real-time call, so the row is no longer refused
with a 400 and the null knobs are simply not forwarded
* fix(proxy): attribute retry attempt spend rows to their own deployment and keep them out of daily request counts
Each <request_id>-attempt-<n> row now carries the model, provider, api_base and model group of the deployment that billed it (looked up by model_info.id through the proxy router, injected as deployment_lookup) instead of copying the final row's, and is stamped internal_call_origin=billed_retry_attempt so the daily spend tables keep its spend and tokens while counting zero api_requests and failed_requests for it
* fix(proxy): declare DeploymentLookup as a TypeAlias and leave the test file's untouched lines as they were
* fix(proxy): mark attempt rows with billed_retry_attempt instead of internal_call_origin so model usage keeps their spend
Bugbot: an internal_call_origin stamp made build_model_usage_transaction drop the attempt rows from LiteLLM_DailyModelUsage. The rows now carry their own SpendLogsMetadata flag, which only the daily request counters read
* fix(proxy): read the billed_retry_attempt flag through a typed helper so the basedpyright gate holds
* fix(proxy): keep the typed litellm_params local in _deployment_provider so the basedpyright gate holds
* fix(proxy): annotate the validated litellm_params as LiteLLM_Params in _deployment_provider
* fix(proxy): keep billed retry attempts out of daily request errors and strip the flag from caller metadata
Bugbot: _record_request_error counted every attempt row as a caller failure in LiteLLM_DailyRequestErrors. Veria: a caller could send metadata.billed_retry_attempt to zero its own request counts, so the key joins the untrusted metadata fields the proxy strips on the way in
* fix(proxy): stamp billed_retry_attempt explicitly in the spend log metadata so the basedpyright gate holds
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): log Gemini 3 sampling deprecation once per process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): bound the Gemini 3 sampling warn-once cache and test it by behavior
* fix(vertex_ai): log the Gemini 3 sampling deprecation once even when first calls race
Pin the warned-model cap to its own constant so the shared LRU size knob cannot turn the dedupe off, and serialize the cached call so concurrent first requests for one model log it once
* fix(vertex_ai): stop logging the Gemini 3 sampling deprecation for new models once 64 are remembered
* fix(vertex_ai): give each mapped Gemini 3 model its own sampling notice and share one for unmapped names
* fix(vertex_ai): key the Gemini 3 sampling notice on the loaded catalog so runtime-priced names share one notice
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): the billed background response cost reaches the creating key's and team's loggers
* fix(responses): send the billed background response cost to the creating key's and team's loggers
* fix(responses): bill an unreadable or rejected creator through proxy loggers, redact and honor disabled callbacks on the billed event
* fix(responses): follow the creator's message logging setting on the billed event and read team logging for non-key creators
The billed event no longer forces redaction. It follows the creating key's or team's turn_off_message_logging, else the proxy's, and redacts when the creator cannot be read or its settings are rejected. A creator without a virtual key hash bills through its team's settings. function_setup keeps its original callback split, and the billed path uses a separate non-mutating helper.
* fix(responses): bill rotated, deleted, and custom auth creators through their own logging and retry failed creator lookups
* fix(responses): follow a key's whole grace-period rotation chain to the live key for its billed charge's logging
* fix(responses): bill a background response whose creator cannot be read, without the creator's settings
* docs(enterprise): say the background response row is claimed before its creator is read
* fix(responses): keep the loaded creator rows' budgets and aliases when one creator lookup fails
* fix(proxy): flush tool discovery queue when use_redis_transaction_buffer is enabled
With the Redis transaction buffer on, discovered tools were never written to
LiteLLM_ToolTable: the only call to _flush_tool_discovery_queue was on the
non-Redis commit path. Each pod's tool queue is never buffered in Redis, so
every pod now flushes its own queue on each spend tick, before the leader
lock check.
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* test(proxy): exercise real leader election in the tool discovery flush test
The writer's PodLockManager had no Redis in the test, so neither case was ever
the leader and the lock-held parameter changed nothing. The fake Redis also had
no async_rpush_pipeline, so every tick took the push-failure branch. Wire the
fake into the lock manager the way ProxyLogging does in production, back it
with real lists, and assert only the lock holder drains the spend buffer.
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* feat(nvidia_riva): add riva_offline mode for offline-only ASR models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(nvidia_riva): assert offline vs streaming via fake service call tracking
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after user soft_budget docstring removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(nvidia_riva): parse riva_offline strings, map oversize offline audio to 413
A multipart riva_offline=false no longer turns offline mode on, a malformed value drops under drop_params and answers 400 without it, and a Riva RESOURCE_EXHAUSTED for a message over the server's max now returns a non-retryable 413 instead of a retried 429
* fix(nvidia_riva): apply the request timeout to offline recognition and keep riva_offline on an api_base override
Offline Recognize now runs as a gRPC future waited on with the request
timeout, so a stalled Riva server returns a 408 and the call is cancelled.
riva_offline is a mode flag, not a credential, so an api_base override no
longer clears it. The 413 hint for audio over the gRPC message limit is
reworded, and the new tests carry type annotations.
* fix(nvidia_riva): wait the request timeout in seconds whatever its form and cap the riva client to versions that import
A timeout from os.environ, a multipart form field, or an httpx.Timeout reached the gRPC future as a string or object,
so result() raised TypeError and every offline call came back as a false 408. The handler now turns the timeout into
float seconds with CompletionTimeout.as_limit, taking the read limit of an httpx.Timeout, for both the offline future
and the streaming deadline
The stt-nvidia-riva extra caps nvidia-riva-client below 2.17, since every later release either pins an older grpcio
or ships stubs that need a newer one than the grpcio==1.78.0 pin, so a fresh install failed at import
* fix(nvidia_riva): parse the timeout only where the call uses it and drop the riva client cap
A malformed timeout no longer fails streaming calls on clients that take no
timeout. On offline calls it is a 400, or the default under drop_params, and
an httpx.Timeout with no read limit waits the default instead of forever.
The nvidia-riva-client cap made installs next to a newer client unresolvable,
so the extra goes back to main's specifier.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap
MCP SDK 2.x reads Streamable HTTP responses through httpx2's SSE parser,
which rejects any single event over 1 MiB, so a large tools/call result
failed with 'SSE stream ended without a response'. mcp 2.3.0 exposes
max_sse_event_size, and MCPClient now passes a 64 MiB default,
configurable with LITELLM_MCP_MAX_SSE_EVENT_SIZE_BYTES
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fall back for invalid SSE event limits
* test(mcp): verify configured SSE limits and recovery
* chore(mcp): remove redundant dependency comment
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(extraction): add Clairic extraction types and Cloudflare provider config
* feat(extraction): add the /v1/extraction route, router wiring, and Clairic billing
* fix(extraction): refuse or drop stream, pass unknown keys, run guardrails, keep the requested model name
Clairic does not stream, so stream=true answers 400 and is dropped and billed under drop_params. Keys LiteLLM does not know pass through to Cloudflare. Guardrails scan state, field instructions and text answers. Frozen decision and extraction responses now carry the requested model name. List fields must name their item type. /health/test_connection and the Add Model page accept the extraction mode.
* fix(extraction): keep upstream model names on decisions, scan images, keep typed values, refuse OpenAI params
Scope the frozen-response model stamp to extraction so /v1/decisions and /v1/systemone return the upstream name as on main. Pass request images to guardrails as data URLs and drop answer spans when a response guardrail rewrites state_text. Field spec values keep their JSON type, so a wrong type answers 400 instead of being coerced. OpenAI generation params answer 400 unless drop_params. The Model Hub shows a /v1/extraction snippet for extraction models.
* fix(extraction): send only Clairic's request fields, count extraction tokens, keep typed answers typed, drop spans a request guardrail made stale, and honor --max_tokens
* fix(extraction): scan number answers in response guardrails, rewrite guardrail fields in linear time, and leave --max_tokens as on main
* fix(extraction): scan float answers as whole numbers so fraction digits never trip a response guardrail
* fix(extraction): round only mean and median for response guardrails so fractional values keep their decimal text
* fix(extraction): count state and fields for budget reservations and scan estimate quantiles in response guardrails
* fix(extraction): send each repeated text to guardrails once and keep the context-window budget reservation
* test(extraction): type request_data in the guardrail test helpers
* fix(logging): report pay-as-you-go rates for spilled over PTU requests without base_model
* fix(logging): keep the provider response model so spilled over PTU payloads price the served model
* fix(logging): keep a provider_response_model the response already carries
* fix(logging): read the carried provider_response_model through a typed adapter
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate
* fix(router): read the retry record's base_model through the typed metadata helper
* fix(router): keep the four-record retry breadcrumb cap
* fix(router): carry a deployment's litellm_params base_model on its retry record
* chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): price cost-based routing through model_info.base_model
* fix(router): keep cost-based routing pricing helpers fully typed
* fix(router): drop typing.cast from cost-based routing pricing helpers
* fix(router): read the cost map through a typed accessor in cost-based routing
* fix(router): avoid a Final rebind inside the cost-based routing loop
* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing
* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing
* refactor(router): inline the deployment pricing-model wrapper in cost-based routing
* refactor(router): read deployment pricing models through one shared helper
deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names
* refactor(router): share the str-keyed mapping validator with cost-based routing
* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): fire one CustomLogger hook per async call outside the async flag list
aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are
missing from is_sync_litellm_request's flag list, so the sync success and
failure handlers called a CustomLogger's sync hook on top of the async hook the
async handler had already called for the same request.
The sync handlers now skip a CustomLogger's log_success_event and
log_failure_event when that run is paired with an async handler run
(for_async_request) and the async handler gives the logger its own async hook.
The async client wrapper, dispatch_failure_handlers, the proxy-only failure
dispatch and the native bridge's sync failure step for an async call mark their
sync failure run as paired.
Resolves LIT-9486
* fix(logging): keep the sync CustomLogger hook when no async success runs
The async client wrapper skips async success logging for calls carrying a
fallbacks kwarg, and the native bridge skips it for internal and fallbacks
calls, yet the paired sync success run still dropped a both-hooks logger's
sync hook, so that logger got no success event at all.
The sync success run now suppresses the sync hook only when the async success
handler was actually scheduled for the call: the client wrapper passes
async_handler_runs=not is_completion_with_fallbacks, and the native host
records per call whether it enqueued or deferred the async success.
Tests that pinned the doubled failure hook on native aocr, messages and
responses calls and on proxy-rejected requests now expect the async hook
alone. The integration test adds /v1/messages and /v1/responses cells and
runs the key-auth rejection on every route.
* fix(logging): type dispatch_success_handlers kwargs
The new async_handler_runs keyword on success_handler made the untyped
**kwargs that dispatch_success_handlers forwards an unknown argument for it,
which grew the reportUnknownArgumentType count past its merge-base ceiling.
Typing the forwarded kwargs as Any, as _success_handler_body already does,
removes those diagnostics without changing what is forwarded.
* test(integration): cover CustomLogger hook counts across modes, chaos and router strategies
Moves the hook count rig into tests/integration/_support/hook_counts.py and
extends the cells it drives:
- streamed chat, Messages and Responses requests, success and failure
- loggers the async handler never reaches (sync-only overrides, sync event
lists only) keep one sync hook per request on every surface outside the
async flag list, key-auth rejections included
- a concurrent burst while the logger's destination stalls and then fails,
and a burst while a worker is killed, never give a request two hooks
- least-busy keeps counting a request in flight while async speech requests
finish around it, and usage-based routing v2 counts an async image
request's tokens once
* test(integration): Datadog and LangSmith deliver one record per async request
Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local
sink and checks every async surface outside the flag list: one Datadog log
entry and one LangSmith run per request (two on the merge base), and never two
OpenTelemetry spans, which its own emit-once guard already ensured.
* fix(logging): give a both-hooks logger its sync hook when the async run never reaches it
The sync hook was skipped whenever the logger sat in the async callback list, so a
logger the async run never reached got no hook at all: a sibling stalled past the
logging worker timeout, a batch poll that skips default logging, or an async
failure run cut off. The sync run now holds the hook and the async run hands it
back unless it reached the logger
* fix(logging): run the held sync success hooks when the proxy drops a blocked response's success
A post-call guardrail block drops the parked async success, which left the sync
success hook held for it unrun. Base gave a both-hooks logger that hook on these
call types, so the drop now ends the pairing and the held hook runs as before
* fix(logging): run held sync hooks only after every sync run of the request has closed
A held hook released by the async run could run on another thread while the sync
run was still walking its other callbacks. The pairing now waits until the async
run has ended and every sync run has closed, and the last sync run to close runs
the held hooks itself
* fix(logging): end the async run when its coroutine is dropped before it starts
A logging queue that drops the async success coroutine, or a deferred success
that fails to schedule, left the sync hooks held for it unrun. The enqueued async
run now ends when its coroutine is discarded, and the proxy ends it when firing
the deferred success raises
* fix(logging): leave a still-queued async run to the exit flush
The discard finalizer ran at interpreter exit before the logging worker's exit
flush, so a queued request could get its held sync hook and then its async hook
* test: keep the router and file-stream tests in step with main
Main gives each owned proxy a private Redis by default, so the router cells
share the worker's Redis to read its counters, and the file-stream stub takes
the hook pairing the stream now passes
* fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up
A during_call guardrail that raised after the LLM call parked its deferred
success skipped the proxy's flush, and async runs the exit flush never started
were never ended, so a both-hooks logger lost its sync hook in both cases
* fix(logging): skip the sync failure hook when the async failure run logs to the logger
An async call outside is_sync_litellm_request's flag list ran both failure
handlers, so a CustomLogger with both failure hooks got both. The sync run
of an async request now skips log_failure_event for a logger that overrides
async_log_failure_event and is registered for async failures. The four sync
runs of async failures (dispatch_failure_handlers, the @client async wrapper,
the proxy-only failure thread and the Rust host) pass for_async_request=True.
Drops the HookPairing fallback. Successes are left to #47192, which already
gives such a logger only async_log_success_event.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed
* fix(proxy): gate the zero-cost 499 row on the success log being enqueued, not merely claimed
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines
* refactor(rust-bridge): split and document route-specific detectors
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates
* refactor(rust-bridge): simplify catalog compatibility checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): scope the search free-tier counter to the request's provider
* fix(proxy): read the provider with get so partial spend payloads still get the free tier
* test(proxy): give the grounded spend payload fixture its provider
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes
The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so
converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock
unguarded while /v1/chat/completions on the same deployment was blocked. The
signer now sends it the way Bedrock reads it, in the body for Converse and as the
X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent
* test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400
The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata
* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep tool names over 64 chars when a guardrail rewrites a /v1/messages request
The guardrail translation shortens long tool names for the OpenAI-shaped guardrail input
and wrote the shortened names back into tools and tool_use blocks, while tool_reference
blocks and tool_choice kept the original names, so Bedrock rejected the request
* fix(anthropic): restore declared tool names only from the request's own tool list
A history tool_use name whose shortened form matched a declared short tool renamed that
declared tool, so a forced tool_choice no longer matched and a tool allowlist saw a
different name than the one forwarded
* fix(anthropic): keep a tool_use name the caller already used as written
A history tool_use named exactly like the shortened form of another long name was
rewritten to that long name; such an ambiguous alias is now left alone
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
