* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The OpenAI fine-tuning handler built its SDK client from every local it
received, so an api_version or litellm_params set on an openai
finetune_settings entry reached OpenAI.__init__ and every fine-tuning
call on that provider failed with "unexpected keyword argument
'api_version'". Build the client from its named params instead and
cover it with a request-level regression test
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tokenizer): isolate HuggingFace tokenizer caches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): clear the traceback before re-raising a cached tokenizer load error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): retry failed HuggingFace tokenizer loads after the cache window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): start tokenizer retry TTL after a failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): lock every HuggingFace tokenizer load and keep load failures as values
Concurrent llama-2, llama-3 and cohere counts each started their own failed
download because only the bundled tokenizers took a load lock. Every tokenizer
kind now has one, so a burst during an outage makes a single attempt and two
counts can no longer read the same expiring entry at once.
A failed load is now remembered as a small value with the error message
instead of the error object, so re-raising it no longer keeps the first
failing request's frames alive. Adds tests for the concurrent outage and for
fork-guard errors staying raised and unremembered.
* fix(utils): keep Hugging Face tokenizer downloads unlocked and warn once per failed download
Reverts the per-kind load lock so concurrent first downloads still run in parallel, as #47353 intends. A short lock around the failure cache read and write stops two counts racing on an expiring entry, and the tiktoken fallback warning now fires only when a download fails, not on every count that reuses the remembered failure.
* fix(caching): read an entry another reader just expired as a cache miss
Two threads reading the same expiring key in InMemoryCache could both pass the membership check, and the second then raised KeyError after the first removed it. get_cache now treats that key as a miss. This replaces the extra lock the tokenizer failure cache took, and the regression test forces the interleaving without timing.
* fix(caching): treat a TTL another reader just removed as not expired
InMemoryCache's expiry check read the TTL after a separate membership check, so a reader racing another reader's eviction raised KeyError. It now reads the TTL once and treats a missing one as not expired, and get_cache already reads the vanished entry as a miss.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): delete a group through the team delete's bulk path instead of one member at a time
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(scim): demote a deleted admin group's members before the team delete so a failed delete stays retryable
* fix(scim): recompute member roles again after the team delete so a concurrent group write cannot regrant admin
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(scim): write recomputed member roles only where the membership still backs them
* fix(team): report the users a team delete detaches so SCIM demotes late-added members
* test(scim): audit cells for the bulk group delete
Move the SCIM group helpers into tests/integration/_support/scim.py and add the delete cells: a member's other
group survives, the admin group's members are demoted and the audit row names the SCIM caller, a member a
concurrent PUT adds while the delete waits on the team lock is demoted too, unknown, repeated, non-admin and
hostile ids, an empty group, the peer proxy, a 20-delete burst across both proxies, and a delete after a worker
was SIGKILLed
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(team): sweep deleted team references only under the lock so a late-added member is demoted
The unlocked sweep that ran before the team's advisory lock stripped a member a
concurrent group write added after the roster was read, so neither the roster
snapshot nor the locked sweep's RETURNING named them and the SCIM recompute
left them proxy_admin after the admin group was gone. The locked sweep does the
same strip and reports whom it stripped, so it is the only sweep now.
The integration cell holds the delete on a member's user row so a PATCH lands
between the roster read and the lock, then checks the late member is demoted.
Both waiter queries in the SCIM support module return their count as text so
the first poll parses.
* chore(team): carry the unbounded-IN baseline entries to delete_validated_teams
The delete pass moved from delete_team and the unlocked sweep into delete_validated_teams, so the checker's baseline names the function the two team_id IN lists now live in
* test(scim): expect the master key alias in the audit row and send post-kill requests on a fresh connection
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* test(scim): cover deleting a SCIM group through the team delete's bulk path
* fix(scim): evict a deleted group's member caches in bulk and drop cached roles after a recompute
* fix(scim): answer 404 to a group delete that lost a race to another delete
* fix(scim): chunk the locked team lookup and delete in the team delete transaction
* fix(scim): keep service-account keys and team models on a group delete and write its records once under the lock
* fix(scim): demote a deleted group's members inside the delete transaction
The admin-group role write now runs on the team delete's transaction, over the
roster read under the team lock, so a failed delete leaves roles untouched and
a member added while the delete waited is demoted with the rest. The delete
transaction also waits for a free connection as long as it may run, so a burst
of deletes queues instead of failing after Prisma's two-second default. Every
member in the 5-vs-500 statement test now holds a key.
* fix(audit): write deleted keys' audit rows in one insert
A delete that removes many keys wrote one audit insert task per key, and those tasks held pool connections the delete's follow-up writes waited on. The race test now waits on any lock wait behind the delete.
* fix(ui): show the budget spend limit and rate limits without a collapsed section
The create and edit budget dialogs hid max budget and reset period behind a collapsed Optional Settings section that sat 80px below the last field, and dropped both values from the request unless the section was open on submit
Both dialogs now share BudgetFormFields. Budget ID comes first, then the spend limit as one row ("$500 per month"), then the three rate limits as short fields beside their labels. A secondary Optional badge marks the spend limit and the rate limits group, so people can still tell those fields are optional. Every field is always submitted, and the edit dialog sends the existing spend limit and reset period back unchanged
* fix(ui): send only changed fields when editing a budget
Saving the edit budget dialog resent every field, so an unrelated rate
limit edit rounded a stored sub-cent spend limit to 2 decimals (0.001
became 0) and resent the reset period, which makes the proxy recompute
the next reset for custom durations. The dialog now sends the budget ID
plus the fields the user changed, and the proxy keeps everything else.
Also updates the budgets Playwright spec for the new field labels and
the removed Optional Settings section.
* fix(ui): track budget edit changes from render so a reverted field cannot hide a later one
react-hook-form's formState is a render-time snapshot. Reading dirtyFields
only inside the submit handler meant that after a field was changed and
changed back, a later edit to another field was missing from the first
save. Reading dirtyFields during render keeps the snapshot current.
* refactor(ui): rename the budget components to PascalCase
Renames budget_modal, edit_budget_modal and budget_panel (and their tests) to BudgetModal, EditBudgetModal and BudgetPanel, and drops their filename-pascal-case suppressions.
* feat(model hub): show admin-defined custom tags and metadata from model_info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): move model hub custom field merging into router_utils
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): share the custom tag badge list between the column and details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(model hub): cover public hub, patch, null, hostile, forwarding, and killed-worker paths for custom fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): drop custom_metadata entries with non-string keys or non-finite numbers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): use isinstance narrowing instead of TypeGuard for custom_metadata values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): build custom tags and metadata in the proxy instead of the router
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): serve custom tags and metadata on model_group_alias rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remove docstring from model_group_custom_fields.py
* style(model hub): fix lint and format in custom metadata tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(model hub): drop leading blank line in custom fields helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show a failed state with Retry in DataTable when the fetch errors instead of an empty table
* fix(ui): show the shared failed state with Retry on the Usage spend totals and the Caching activity chart when their fetch errors
* refactor(ui): name the settled-fetch objects so the inline object arg lint budget holds
* test(ui): find the caching chart card through the existing cardTitled helper
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later
Google ignores temperature, top_p and top_k on Gemini 3.6 Flash, Gemini 3.5
Flash-Lite and every later Gemini release, and says a later generation will
return 400 for them. The model map now carries supports_sampling_params: false
on those rows, and the Gemini and Vertex transformation reads the flag the way
the Anthropic and Bedrock paths already do, dropping the params under
drop_params and otherwise raising the client-side 400 that names drop_params
BREAKING CHANGE: a request that sends temperature, top_p or top_k to a flagged
Gemini model without drop_params now gets a 400 instead of a 200 whose
sampling params Google silently ignored
Fixes#47015
* fix(gemini): gate the passed-through top_k where the request body is built and keep allowed_openai_params working
top_k is a provider-specific kwarg that never reaches the OpenAI param mapping,
so the flag has to be applied while the Gemini request body is built, the same
boundary Anthropic uses. The mapping now skips a flagged sampling param instead
of raising, because the shared unsupported-param check already returns the 400
that names drop_params, and a param the caller lists in allowed_openai_params
is forwarded verbatim as that 400 promises
* fix(gemini): filter top_k out of the generation config instead of mutating optional_params
* test(gemini): type the sampling-flag twin test and keep its locals out of the loop body
* fix(vertex_ai): reject or drop sampling params on batch rows for flagged Gemini models
The batch JSONL wrapper calls map_openai_params directly and skips get_optional_params,
so a flagged model's temperature, top_p and top_k vanished from the uploaded row with no
400. Batch rows now get the same contract as a real-time call: a 400 naming drop_params,
or the knobs dropped when litellm.drop_params is on
* fix(vertex_ai): ignore null sampling knobs on batch rows for flagged Gemini models
A batch row carrying temperature: null, top_p: null or top_k: null means no
override, the same as on a real-time call, so the row is no longer refused
with a 400 and the null knobs are simply not forwarded
* fix(proxy): attribute retry attempt spend rows to their own deployment and keep them out of daily request counts
Each <request_id>-attempt-<n> row now carries the model, provider, api_base and model group of the deployment that billed it (looked up by model_info.id through the proxy router, injected as deployment_lookup) instead of copying the final row's, and is stamped internal_call_origin=billed_retry_attempt so the daily spend tables keep its spend and tokens while counting zero api_requests and failed_requests for it
* fix(proxy): declare DeploymentLookup as a TypeAlias and leave the test file's untouched lines as they were
* fix(proxy): mark attempt rows with billed_retry_attempt instead of internal_call_origin so model usage keeps their spend
Bugbot: an internal_call_origin stamp made build_model_usage_transaction drop the attempt rows from LiteLLM_DailyModelUsage. The rows now carry their own SpendLogsMetadata flag, which only the daily request counters read
* fix(proxy): read the billed_retry_attempt flag through a typed helper so the basedpyright gate holds
* fix(proxy): keep the typed litellm_params local in _deployment_provider so the basedpyright gate holds
* fix(proxy): annotate the validated litellm_params as LiteLLM_Params in _deployment_provider
* fix(proxy): keep billed retry attempts out of daily request errors and strip the flag from caller metadata
Bugbot: _record_request_error counted every attempt row as a caller failure in LiteLLM_DailyRequestErrors. Veria: a caller could send metadata.billed_retry_attempt to zero its own request counts, so the key joins the untrusted metadata fields the proxy strips on the way in
* fix(proxy): stamp billed_retry_attempt explicitly in the spend log metadata so the basedpyright gate holds
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): log Gemini 3 sampling deprecation once per process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): bound the Gemini 3 sampling warn-once cache and test it by behavior
* fix(vertex_ai): log the Gemini 3 sampling deprecation once even when first calls race
Pin the warned-model cap to its own constant so the shared LRU size knob cannot turn the dedupe off, and serialize the cached call so concurrent first requests for one model log it once
* fix(vertex_ai): stop logging the Gemini 3 sampling deprecation for new models once 64 are remembered
* fix(vertex_ai): give each mapped Gemini 3 model its own sampling notice and share one for unmapped names
* fix(vertex_ai): key the Gemini 3 sampling notice on the loaded catalog so runtime-priced names share one notice
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): the billed background response cost reaches the creating key's and team's loggers
* fix(responses): send the billed background response cost to the creating key's and team's loggers
* fix(responses): bill an unreadable or rejected creator through proxy loggers, redact and honor disabled callbacks on the billed event
* fix(responses): follow the creator's message logging setting on the billed event and read team logging for non-key creators
The billed event no longer forces redaction. It follows the creating key's or team's turn_off_message_logging, else the proxy's, and redacts when the creator cannot be read or its settings are rejected. A creator without a virtual key hash bills through its team's settings. function_setup keeps its original callback split, and the billed path uses a separate non-mutating helper.
* fix(responses): bill rotated, deleted, and custom auth creators through their own logging and retry failed creator lookups
* fix(responses): follow a key's whole grace-period rotation chain to the live key for its billed charge's logging
* fix(responses): bill a background response whose creator cannot be read, without the creator's settings
* docs(enterprise): say the background response row is claimed before its creator is read
* fix(responses): keep the loaded creator rows' budgets and aliases when one creator lookup fails
Pillow 13.0.0 requires Python >= 3.11, so the ci group pin is split by
Python version and the 3.10 resolution stays on 12.3.0 with a 7-day
osv-scanner override. Dashboard vite moves from 7.3.5 to 7.3.7. The
certifi fix (2026.10.19) is still inside the 3-day uv exclude-newer
window, so its advisory is ignored until 2026-10-23
* fix(proxy): flush tool discovery queue when use_redis_transaction_buffer is enabled
With the Redis transaction buffer on, discovered tools were never written to
LiteLLM_ToolTable: the only call to _flush_tool_discovery_queue was on the
non-Redis commit path. Each pod's tool queue is never buffered in Redis, so
every pod now flushes its own queue on each spend tick, before the leader
lock check.
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* test(proxy): exercise real leader election in the tool discovery flush test
The writer's PodLockManager had no Redis in the test, so neither case was ever
the leader and the lock-held parameter changed nothing. The fake Redis also had
no async_rpush_pipeline, so every tick took the push-failure branch. Wire the
fake into the lock manager the way ProxyLogging does in production, back it
with real lists, and assert only the lock holder drains the spend buffer.
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* feat(nvidia_riva): add riva_offline mode for offline-only ASR models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(nvidia_riva): assert offline vs streaming via fake service call tracking
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after user soft_budget docstring removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(nvidia_riva): parse riva_offline strings, map oversize offline audio to 413
A multipart riva_offline=false no longer turns offline mode on, a malformed value drops under drop_params and answers 400 without it, and a Riva RESOURCE_EXHAUSTED for a message over the server's max now returns a non-retryable 413 instead of a retried 429
* fix(nvidia_riva): apply the request timeout to offline recognition and keep riva_offline on an api_base override
Offline Recognize now runs as a gRPC future waited on with the request
timeout, so a stalled Riva server returns a 408 and the call is cancelled.
riva_offline is a mode flag, not a credential, so an api_base override no
longer clears it. The 413 hint for audio over the gRPC message limit is
reworded, and the new tests carry type annotations.
* fix(nvidia_riva): wait the request timeout in seconds whatever its form and cap the riva client to versions that import
A timeout from os.environ, a multipart form field, or an httpx.Timeout reached the gRPC future as a string or object,
so result() raised TypeError and every offline call came back as a false 408. The handler now turns the timeout into
float seconds with CompletionTimeout.as_limit, taking the read limit of an httpx.Timeout, for both the offline future
and the streaming deadline
The stt-nvidia-riva extra caps nvidia-riva-client below 2.17, since every later release either pins an older grpcio
or ships stubs that need a newer one than the grpcio==1.78.0 pin, so a fresh install failed at import
* fix(nvidia_riva): parse the timeout only where the call uses it and drop the riva client cap
A malformed timeout no longer fails streaming calls on clients that take no
timeout. On offline calls it is a 400, or the default under drop_params, and
an httpx.Timeout with no read limit waits the default instead of forever.
The nvidia-riva-client cap made installs next to a newer client unresolvable,
so the extra goes back to main's specifier.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap
MCP SDK 2.x reads Streamable HTTP responses through httpx2's SSE parser,
which rejects any single event over 1 MiB, so a large tools/call result
failed with 'SSE stream ended without a response'. mcp 2.3.0 exposes
max_sse_event_size, and MCPClient now passes a 64 MiB default,
configurable with LITELLM_MCP_MAX_SSE_EVENT_SIZE_BYTES
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fall back for invalid SSE event limits
* test(mcp): verify configured SSE limits and recovery
* chore(mcp): remove redundant dependency comment
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(extraction): add Clairic extraction types and Cloudflare provider config
* feat(extraction): add the /v1/extraction route, router wiring, and Clairic billing
* fix(extraction): refuse or drop stream, pass unknown keys, run guardrails, keep the requested model name
Clairic does not stream, so stream=true answers 400 and is dropped and billed under drop_params. Keys LiteLLM does not know pass through to Cloudflare. Guardrails scan state, field instructions and text answers. Frozen decision and extraction responses now carry the requested model name. List fields must name their item type. /health/test_connection and the Add Model page accept the extraction mode.
* fix(extraction): keep upstream model names on decisions, scan images, keep typed values, refuse OpenAI params
Scope the frozen-response model stamp to extraction so /v1/decisions and /v1/systemone return the upstream name as on main. Pass request images to guardrails as data URLs and drop answer spans when a response guardrail rewrites state_text. Field spec values keep their JSON type, so a wrong type answers 400 instead of being coerced. OpenAI generation params answer 400 unless drop_params. The Model Hub shows a /v1/extraction snippet for extraction models.
* fix(extraction): send only Clairic's request fields, count extraction tokens, keep typed answers typed, drop spans a request guardrail made stale, and honor --max_tokens
* fix(extraction): scan number answers in response guardrails, rewrite guardrail fields in linear time, and leave --max_tokens as on main
* fix(extraction): scan float answers as whole numbers so fraction digits never trip a response guardrail
* fix(extraction): round only mean and median for response guardrails so fractional values keep their decimal text
* fix(extraction): count state and fields for budget reservations and scan estimate quantiles in response guardrails
* fix(extraction): send each repeated text to guardrails once and keep the context-window budget reservation
* test(extraction): type request_data in the guardrail test helpers
* fix(logging): report pay-as-you-go rates for spilled over PTU requests without base_model
* fix(logging): keep the provider response model so spilled over PTU payloads price the served model
* fix(logging): keep a provider_response_model the response already carries
* fix(logging): read the carried provider_response_model through a typed adapter
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate
* fix(router): read the retry record's base_model through the typed metadata helper
* fix(router): keep the four-record retry breadcrumb cap
* fix(router): carry a deployment's litellm_params base_model on its retry record
* chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): price cost-based routing through model_info.base_model
* fix(router): keep cost-based routing pricing helpers fully typed
* fix(router): drop typing.cast from cost-based routing pricing helpers
* fix(router): read the cost map through a typed accessor in cost-based routing
* fix(router): avoid a Final rebind inside the cost-based routing loop
* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing
* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing
* refactor(router): inline the deployment pricing-model wrapper in cost-based routing
* refactor(router): read deployment pricing models through one shared helper
deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names
* refactor(router): share the str-keyed mapping validator with cost-based routing
* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): fire one CustomLogger hook per async call outside the async flag list
aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are
missing from is_sync_litellm_request's flag list, so the sync success and
failure handlers called a CustomLogger's sync hook on top of the async hook the
async handler had already called for the same request.
The sync handlers now skip a CustomLogger's log_success_event and
log_failure_event when that run is paired with an async handler run
(for_async_request) and the async handler gives the logger its own async hook.
The async client wrapper, dispatch_failure_handlers, the proxy-only failure
dispatch and the native bridge's sync failure step for an async call mark their
sync failure run as paired.
Resolves LIT-9486
* fix(logging): keep the sync CustomLogger hook when no async success runs
The async client wrapper skips async success logging for calls carrying a
fallbacks kwarg, and the native bridge skips it for internal and fallbacks
calls, yet the paired sync success run still dropped a both-hooks logger's
sync hook, so that logger got no success event at all.
The sync success run now suppresses the sync hook only when the async success
handler was actually scheduled for the call: the client wrapper passes
async_handler_runs=not is_completion_with_fallbacks, and the native host
records per call whether it enqueued or deferred the async success.
Tests that pinned the doubled failure hook on native aocr, messages and
responses calls and on proxy-rejected requests now expect the async hook
alone. The integration test adds /v1/messages and /v1/responses cells and
runs the key-auth rejection on every route.
* fix(logging): type dispatch_success_handlers kwargs
The new async_handler_runs keyword on success_handler made the untyped
**kwargs that dispatch_success_handlers forwards an unknown argument for it,
which grew the reportUnknownArgumentType count past its merge-base ceiling.
Typing the forwarded kwargs as Any, as _success_handler_body already does,
removes those diagnostics without changing what is forwarded.
* test(integration): cover CustomLogger hook counts across modes, chaos and router strategies
Moves the hook count rig into tests/integration/_support/hook_counts.py and
extends the cells it drives:
- streamed chat, Messages and Responses requests, success and failure
- loggers the async handler never reaches (sync-only overrides, sync event
lists only) keep one sync hook per request on every surface outside the
async flag list, key-auth rejections included
- a concurrent burst while the logger's destination stalls and then fails,
and a burst while a worker is killed, never give a request two hooks
- least-busy keeps counting a request in flight while async speech requests
finish around it, and usage-based routing v2 counts an async image
request's tokens once
* test(integration): Datadog and LangSmith deliver one record per async request
Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local
sink and checks every async surface outside the flag list: one Datadog log
entry and one LangSmith run per request (two on the merge base), and never two
OpenTelemetry spans, which its own emit-once guard already ensured.
* fix(logging): give a both-hooks logger its sync hook when the async run never reaches it
The sync hook was skipped whenever the logger sat in the async callback list, so a
logger the async run never reached got no hook at all: a sibling stalled past the
logging worker timeout, a batch poll that skips default logging, or an async
failure run cut off. The sync run now holds the hook and the async run hands it
back unless it reached the logger
* fix(logging): run the held sync success hooks when the proxy drops a blocked response's success
A post-call guardrail block drops the parked async success, which left the sync
success hook held for it unrun. Base gave a both-hooks logger that hook on these
call types, so the drop now ends the pairing and the held hook runs as before
* fix(logging): run held sync hooks only after every sync run of the request has closed
A held hook released by the async run could run on another thread while the sync
run was still walking its other callbacks. The pairing now waits until the async
run has ended and every sync run has closed, and the last sync run to close runs
the held hooks itself
* fix(logging): end the async run when its coroutine is dropped before it starts
A logging queue that drops the async success coroutine, or a deferred success
that fails to schedule, left the sync hooks held for it unrun. The enqueued async
run now ends when its coroutine is discarded, and the proxy ends it when firing
the deferred success raises
* fix(logging): leave a still-queued async run to the exit flush
The discard finalizer ran at interpreter exit before the logging worker's exit
flush, so a queued request could get its held sync hook and then its async hook
* test: keep the router and file-stream tests in step with main
Main gives each owned proxy a private Redis by default, so the router cells
share the worker's Redis to read its counters, and the file-stream stub takes
the hook pairing the stream now passes
* fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up
A during_call guardrail that raised after the LLM call parked its deferred
success skipped the proxy's flush, and async runs the exit flush never started
were never ended, so a both-hooks logger lost its sync hook in both cases
* fix(logging): skip the sync failure hook when the async failure run logs to the logger
An async call outside is_sync_litellm_request's flag list ran both failure
handlers, so a CustomLogger with both failure hooks got both. The sync run
of an async request now skips log_failure_event for a logger that overrides
async_log_failure_event and is registered for async failures. The four sync
runs of async failures (dispatch_failure_handlers, the @client async wrapper,
the proxy-only failure thread and the Rust host) pass for_async_request=True.
Drops the HookPairing fallback. Successes are left to #47192, which already
gives such a logger only async_log_success_event.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed
* fix(proxy): gate the zero-cost 499 row on the success log being enqueued, not merely claimed
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines
* refactor(rust-bridge): split and document route-specific detectors
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates
* refactor(rust-bridge): simplify catalog compatibility checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The team Model-Specific Rate Limits picker built its options from the
team's direct models only. A team that gets its models through access
groups has models ["no-default-models"], so the sentinel was the only
option, and a saved {"no-default-models": N} limit never enforces since
the limiter matches the requested model name.
Build one option list for every team per-model field from direct plus
access-group models, expand wildcards, treat an empty list as all
models, and drop sentinels and wildcard patterns, which no limit or
budget lookup ever matches. The team per-model budget field shared the
broken list and is fixed with it.
LIT-9490
* fix(proxy): scope the search free-tier counter to the request's provider
* fix(proxy): read the provider with get so partial spend payloads still get the free tier
* test(proxy): give the grounded spend payload fixture its provider
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes
The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so
converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock
unguarded while /v1/chat/completions on the same deployment was blocked. The
signer now sends it the way Bedrock reads it, in the body for Converse and as the
X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent
* test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400
The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata
* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep tool names over 64 chars when a guardrail rewrites a /v1/messages request
The guardrail translation shortens long tool names for the OpenAI-shaped guardrail input
and wrote the shortened names back into tools and tool_use blocks, while tool_reference
blocks and tool_choice kept the original names, so Bedrock rejected the request
* fix(anthropic): restore declared tool names only from the request's own tool list
A history tool_use name whose shortened form matched a declared short tool renamed that
declared tool, so a forced tool_choice no longer matched and a tool allowlist saw a
different name than the one forwarded
* fix(anthropic): keep a tool_use name the caller already used as written
A history tool_use named exactly like the shortened form of another long name was
rewritten to that long name; such an ambiguous alias is now left alone
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
