Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing
* fix(router): key prompt caching affinity on tools and the /v1/messages system prompt
Co-authored-by: nuernber <benjamin.nuernberger@jpl.nasa.gov>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read a 1h tool cache_control nested under function for the pin ttl
---------
Co-authored-by: nuernber <benjamin.nuernberger@jpl.nasa.gov>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): match partial wildcards against prefix-stripped model ids (issue #46384)
Refactor model checks to improve filtering logic and type hints.
* test(proxy): partial wildcard listing regression coverage (issue #46384)
* fix(proxy): drop unneeded annotation on suffix list (LIT001)
* fix(proxy): annotate suffix list with reasoned mutable-ok (basedpyright)
Change suffix_appended_wildcard_models to a typed list.
* Add lock timeout before creating telemetry report table
Set local lock timeout to 10 seconds before creating the telemetry report table.
* fix(proxy): list alias wildcards as routable ids, not raw catalog ids
An alias whose suffix collides with a catalog sub-prefix (e.g. public
"cloudflare/@cf/*" over deployment "cloudflare/*") took the partial-filter
branch and listed raw catalog ids. Routing substitutes the captured suffix
into the deployment pattern, so those listed ids routed to targets that do
not exist. Only use filter semantics when the deployment repeats the
wildcard suffix (or no deployment is available); aliases prepend their
prefix to provider-stripped catalog ids so every listed id routes back to
a real catalog model. Addresses Greptile review on #46390.
* fix(proxy): derive partial wildcard listings from the deployment's routing pattern
List an id for a partial wildcard only when routing it through the deployment's
pattern lands on a catalog model, so aliases, catch-alls, and key filters over a
wildcard deployment list ids that answer when called. /model/info expands a bare
* row the same way /v1/models does.
* fix(proxy): drop fragment ids for a catch-all over a partial wildcard and route bare aliases through the deployment pattern
A * catch-all over a partial deployment now lists no catalog ids, a bare alias like my-cf/* over cloudflare/@cf/* names ids by the deployment pattern, an alias whose wildcard count differs from its deployment lists nothing, and /model/info rows show the model each listed id routes to
* fix(proxy): keep listed ids on /model/info alias rows and clean up test-registered models
* fix(proxy): list catalog models through routing-prefix wildcard deployments
* fix(proxy): keep namespace and bare catch-all wildcards on routable listings
A leading deployment segment the model name repeats, or the provider name
itself, is no longer read as a routing segment, so same-name namespace
wildcards and a key filter over `*` -> `openai/*` stop listing ids that
route to no model. A bare `*` deployment under a prefixed name keeps the
legacy listing. `*` over a partial deployment now lists the names that
route through it, dropping empty ids.
* fix(proxy): list nothing for a catch-all over a mid-word wildcard deployment
* fix(proxy): list only wildcard ids that route back to their deployment and catalog model
A listed id now has to round-trip through the alias pattern to the catalog model it names, and the router's wildcard match for it has to pick this deployment, so an id a more specific sibling wildcard serves is no longer listed under the catch-all on /v1/models or /model/info
* fix(proxy): list wildcard aliases that route at a word boundary and ids a sibling serves with the same model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(streaming): skip full-chunk dump and delta rebuild on openai-compatible chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(streaming): reuse the built choice and delta on plain openai-compatible chunks
Co-authored-by: yassin <yassin@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): copy upstream chunk attributes in upstream key order
The scalar fast path iterated a frozenset of keys, so the SSE key order of chunk extras like service_tier and obfuscation changed with each worker's hash seed
* test(streaming): count chunk_creator calls with a profiler instead of patching classes
* test(streaming): check service_tier still passes through chunk_creator
* fix(streaming): reuse the built choice for chunks with upstream choice extras
* test(streaming): cover ModelResponseStream chunks on the choice reuse path
* fix(streaming): keep main's delta rebuild for deltas with nested values
Clearing role in place on a delta that holds tool calls left its nested
fields unset, so chunk.model_dump() dropped keys like a custom tool call's
name on later chunks. Deltas holding only scalars keep the in-place path.
* perf(streaming): build each plain OpenAI chunk's delta once and skip pydantic attribute misses
Later plain ChatCompletionChunks now build one Delta and one StreamingChoices, with no full-chunk model_dump, and read stored values instead of attribute probes that raise and catch AttributeError. Nested-value chunks no longer pay the dict(model) misses that made them slower than the merge base.
* fix(streaming): read hidden params a chunk carries as a pydantic extra
* test(streaming): pin logprobs, Azure filter extras, and audio and image deltas on later OpenAI chunks
* test(streaming): cover role, reasoning, choice extras and billing on plain OpenAI SDK chunks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): stream Claude json_schema answers as text on every streaming path
Async and proxy streams on Databricks Claude json_schema returned the
internal json_tool_call tool call with empty text, because the async
stream factory dropped json_mode. Databricks now passes it through, the
json tool's argument fragments stream verbatim as content (a lone {}
fragment was blanked before, corrupting nested empty objects), and the
stream ends with finish_reason stop like the non-stream answer.
* test(databricks): keep the json_schema stream regression tests ahead of the fix
Holds the Databricks transformation at the merge base so the first approval covers the regression tests; the fix commit follows.
* fix(databricks): stream Claude json_schema answers as text on every streaming path
* fix(databricks): keep finish_reason tool_calls when a json_schema stream also calls a user tool
* fix(databricks): drop text beside the json tool and renumber user tools in Claude json_schema streams
With reasoning_effort set, Claude can write prose before json_tool_call or call it next to a user tool.
Streams now hold text until the end and drop it when the json tool was called, renumber user tool
deltas past the dropped json tool so indexes start at 0, and non-stream returns the json tool's
arguments as content next to the user tools.
* fix(databricks): stream JSON-looking text at once and keep prose ahead of user tools in Claude json_schema streams
* fix(databricks): hold a user tool call behind held prose so a later json tool answer does not glue to it
A json_schema stream with thinking on that wrote prose, called a user tool, then called the json tool sent the prose at the user tool call and appended the JSON answer to it, so content did not parse. User tool deltas behind held prose are now held with it and sent merged per index at the json tool or the finish, and a json tool after text already sent is dropped so the answer is not repeated.
* test(databricks): cover json_schema answers on every endpoint in the integration wire cells
* test(perf): gate the gateway's own overhead header, redis latency and postgres load in the perf cases
* test(perf): keep the gateway_added_seconds limits next to the header ones
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row
* fix(proxy): read the provider handoff before the proxy-only error handler stamps it so a pre-handoff 499 is not priced
* fix(proxy): drop the pre-logging handoff read now that pre_call no longer stamps proxy-only failures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(key_management): apply default_key_generate_params.allowed_routes on key generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(key_management): avoid mutating default routes payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): apply default allowed_routes to key_type presets it covers
* fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy
* fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes
* fix(proxy): keep owners from changing routes a key got from default_key_generate_params
* docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off
* test(proxy): type the default_key_generate_params allowed_routes tests
* fix(proxy): let key owners send unchanged allowed_routes back on /key/update
* fix(proxy): ignore a route default with a non-string entry and match resent routes exactly
A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths.
* test(proxy): cover a non-string allowed_routes default in the key integration tests
* test(proxy): read key rows from the owned proxy's database in the default routes integration tests
* fix(proxy): leave allowed_routes out of a key update that resends them unchanged
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(batches): apply the Vertex regional endpoint uplift to batch cost
batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* test(logging): point the resolver tests at resolve_vertex_location_for_cost
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* fix(mcp): keep the gateway handler off the SDK path when no tool is an MCP tool
completion(), responses() and anthropic messages imported the MCP gateway handler
whenever tools were passed, and responses() did so on every call. Since the handler
began importing the MCP catalog at module level in #42568, that import needs the
optional mcp package, so a base SDK install failed with ModuleNotFoundError on any
function-tool call and on every responses() call
A new import-safe pre-check skips the handler unless some tool is a dict of type mcp,
which is the only shape the handler's gateway decision can accept. MCP gateway tools
still reach the handler as before
Adds in-process regression tests for the three entry points and a function-tool
completion to the base SDK install check
* fix(mcp): leave one-shot tool iterables unread in the gateway pre-check
responses() declares tools as an Iterable and the handler scans the same
object a second time, so a pre-check that read a generator would hand the
handler one tool fewer. The pre-check now scans only collections and reports
anything else as possibly naming the gateway, which keeps the previous path
The Anthropic messages regression test now mocks the httpx layer with respx
and turns off the aiohttp transport, instead of replacing AsyncHTTPHandler
* fix(mcp): scan only lists and tuples in the gateway pre-check
The pre-check read any Collection, but a Collection that yields its items only
once would still be exhausted by the handler's second scan. Only the list and
tuple builtins are guaranteed to iterate the same way twice, so they are the
only inputs read now. Everything else keeps the previous path and is reported
as possibly naming the gateway
The unread-input test now covers a generator, an iterator, a one-shot
collection and a deque
* fix(mcp): scan only exact list and tuple types in the gateway pre-check
isinstance accepted list subclasses, and a subclass that iterates only once
would still be consumed by the handler's second scan. The fast path now reads
only objects whose type is exactly list or tuple. Every other input, subclasses
included, keeps the previous path and is reported as possibly naming the gateway
The unread-input test gains a one-shot list subclass and a tuple subclass
* fix(sdk): decide MCP gateway routing without importing the proxy handler
The pre-check now holds the handler's exact gateway test, so remote MCP tools, mcp 1.x installs and any tools iterable stay off the handler, and the base SDK check calls responses() with a remote MCP tool.
* fix(sdk): send /mcp/<name> remote MCP URLs to the provider when the gateway registry cannot load
A path-only /mcp/<name> match now counts as a gateway tool only when the MCP server registry imports, so SDK installs without the proxy extras pass remote MCP servers like Zapier to the provider. Also drops a dead test patch.
* test(sdk): send a /mcp/<name> remote MCP tool in the base SDK install check
* style(sdk): drop docstrings from the MCP gateway pre-check and its tests
* test(mcp): cover /mcp/<name> gateway routing on the proxy and Messages tools on a base SDK install
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request
An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's
litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper
now reads the referenced file or env var on each call and never writes the token back, so a rotated
token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as
server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback
targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400
* fix(secret_managers): strip only the leading oidc/ from a secret reference
A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed
* fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them
The advisor tool's own api_key now answers a 400 when it holds an oidc/
reference, since its sub-call runs through the client wrapper that resolves
one. Stored fallback targets, auto-router tiers and user_config refuse only an
oidc/ api_key, inline or through the credential they name, so their workload
identity federation handling stays as it was. Provider model discovery for a
wildcard deployment resolves the reference before calling the provider.
* fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments
* fix(utils): resolve oidc api_key refs only from admin-configured sources
Mark an oidc/ api_key read from a proxy-level deployment or a stored
credential, and resolve only that marked value. Team deployments, prompt
templates, nested configs and a direct SDK api_key send the text as
written. An unsupported reference kind raises a 401 so the router fails
over to the group's next deployment.
* fix(utils): send a credential's oidc token only to its own api_base host
A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the
credential's own api_base or base_url, so a team model or vector store that names the
credential with another api_base gets a 401 and that host gets nothing. Saving a team model
with an inline oidc/ api_key answers 400, since team rows never resolve one.
* fix(proxy): send an admin oidc api_key only to the hosts configured beside it
Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths.
* fix(proxy): redact the token path from token count auth errors
* test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs
* fix(proxy): redact the token path from secret reference errors where they are raised
* fix(proxy): keep the secret reference's path out of the error a caller sees
The 401 for an unreadable oidc api_key or pass-through credential now names only
the reference kind and a fixed reason, so a token path under /run/secrets or a
custom credential dir never reaches the caller, while the server log keeps the
full reference for the operator
* fix(secret_managers): strip line breaks from the logged secret reference
A reference holding a newline can no longer forge a server log line
* fix(secret_managers): label an unknown oidc reference kind instead of echoing it
An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in
full in the caller's 401 and in the spend log, since the kind was whatever
text followed oidc/. A kind get_secret does not know now reads
oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>,
which the client-message redaction on chat routes leaves whole.
* test(health): run the real model auth check in the oidc test connection test
* test(secret_managers): read an oidc/file token stored under a directory named oidc
* fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references
A huggingface/* wildcard call could name its own endpoint URL as the model and
receive the resolved token. The origin check now treats a URL in the model as
a host the reference must allow. The integration contract also reads the
proxy's own database instead of the pytest worker's.
* fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin
A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443,
though both reach the same server. The origin check now compares scheme, host
and effective port, as assert_same_origin does, and an unparseable api_base is
refused instead of raising. The advisor test sets the real proxy flag instead of
replacing our own helpers.
* test: type the oidc request-body helpers and drive the tier test through mock_response
* fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host
test_router_cooldown_failure_row.py boots an owned proxy, which writes to a private database since #47213. The file was written on a base without #47213 and read its rows from the worker database through read_rows, so all 13 cases timed out with State did not converge: []
Every row read now goes through Gateway.rows on the owned proxy's gateway, the same fix#47410 made for the stream stall cooldown cases
* fix(friendliai): drop litellm-internal assistant fields before sending
Claude Code replays thinking blocks on every turn after the first. The
Anthropic adapter turns them into thinking_blocks plus reasoning_content
on the chat assistant message, and FriendliAI rejects any message key it
does not know, so multi-turn Claude Code sessions on friendliai/ models
failed with a 500 after the first tool call
FriendliAI accepts reasoning_content and GLM reads it back, so only
thinking_blocks and provider_specific_fields are dropped. together_ai
already had this exact helper as a private function, so it moves to
common_utils next to strip_litellm_internal_message_fields and both
providers share it
Related to #32188
* test(friendliai): cover a replayed tool call turn
The regression fixture is now the shape Claude Code replays after a tool
call: an assistant turn with thinking_blocks, tool_calls and no content,
followed by the matching tool result. The test asserts both survive the
sync and async request paths unchanged
* ci: retrigger checks
* ci: retrigger benchmarks after runner timeout
* chore(friendliai): mark inherited list signatures as mutable-ok
The type-discipline gate on main now caps LIT001 at the merge-base count,
so the _transform_messages overloads need the same inherited-contract
marker together_ai uses for its identical override
* ci(test-unit): add tests/unit/llms/friendliai to the llms-providers shard
* test(friendliai): check the outbound FriendliAI body through the proxy for chat and messages replays
Both replay shapes go through a real proxy to a scripted FriendliAI upstream, and the test reads the
request body FriendliAI received. It fails at the merge base, where thinking_blocks and
provider_specific_fields reach the upstream, and passes with the field stripping in place.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: pin the validation lines #44798 kept with failing cases
* fix(containers): accept endpoint table entries that omit their param lists
* test: keep the endpoint table and xAI adapter as on main
The endpoint table ships inside the package, so optional param lists fixed no
reachable state and turned a loud failure into silently dropped params. Drops
the Gemini case that main's tests already cover.
* test: cover the cost map backup and Gemini image config validation lines
The cost map backup adapter's hide flag, and the validation itself, could be
removed with every test still green. The Gemini imageConfig adapter could be
narrowed with only the integration suite noticing, which does not run on PRs.
The endpoint table cases now read a real file through the module's location
instead of replacing open, so reading the file another way keeps them green.
* test: pin the container handler's loud table failure and loosen two pinned drops
A container endpoint table entry without path_params or query_params fails
the call on main, and no test held that. The new case trims each key from the
packaged table and expects the validation error.
The Gemini imageConfig case now accepts a drop or UnsupportedParamsError for a
non-object value, and the Bedrock beta case checks only that numbers come back
as strings, so neither pins behavior the drop_params convention may change.
* test: check a numeric beta array is never split as text and use the latest Gemini image model
The Bedrock case now also fails when a JSON array of numbers falls through to
the comma split, which returns bracketed fragments, while a drop or a string
coercion both stay allowed.
memory_growth_megabytes compared end-of-run RSS with RSS read right after boot, before a worker had freed its startup allocations, so a light case (small_bodies_5rps) read negative growth. The case now replays its first minute of traffic before the baseline read, so the metric is the memory the measured traffic added
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_increment_deployment_usage looked up the deployment, resolved its model info and model_dump'ed it on every call just to decide whether the deployment has any tpm/rpm/itpm/otpm limit. That answer only changes when the model list or discovered limits change, so it is now memoized per deployment and cleared in _invalidate_model_group_info_cache
* chore(hooks): drop Conventional Branches pre-push hook in favor of litellm_ prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(makefile): describe install-hooks as Conventional Commits only
* docs(contributing): scope the litellm_ branch prefix to internal contributors
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Savings and the savings baseline priced deployments through Router.get_deployment_model_info, which deep-copies the cost map entries on every call. The router already keeps an lru_cache of the same lookup that is cleared whenever the model list or cost map changes, and budget reservation reads from it, so the savings path now reads from it too
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* fix(speech): correctly propagate spend logging and usage for Gemini TTS bridge
* fix(speech): call Gemini once per TTS request and log the bridge's token usage
aspeech re-ran speech whenever the first result was not a coroutine, so the
speech-to-completion bridge sent every Gemini TTS request upstream twice. It
now returns the sync result.
The bridge's inner completion runs as an internal sub call, so only the
speech call fires success callbacks, and the completion's usage rides on the
audio response so the spend log and usage_object carry real token counts.
Replaces the separate inner Logging object and the cost_calculator fallback,
which never reached the spend row.
* fix(speech): flag a cached Gemini TTS reply as a cache hit so it logs at no charge
* fix(speech): price a cached Gemini TTS reply's savings from its token usage and keep its cache key
* test(integration): cover Gemini TTS speech billing, logging, cache hits and deployment TPM
* test(integration): start the speech TPM cell in a minute's first 10 s so the two requests never cross a minute
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): refresh cached team on /team/block and /team/unblock
Both endpoints wrote `blocked` to Postgres and returned without touching the
management-object cache. Auth resolves teams through `team_id:{team_id}`
before reading the DB and `common_checks` enforces `blocked` off that cached
object, so a team whose traffic had already warmed the cache kept serving
requests after being blocked, and kept getting rejected after being
unblocked, until the entry expired.
`_refresh_cached_team` already exists for exactly this and its docstring says
every endpoint mutating `litellm_teamtable` must call it; these two were the
only mutating endpoints that didn't. The update now also asks for
`object_permission` so the refreshed cache entry doesn't drop the relation,
matching `team_model_add` and `team_model_delete`.
* fix(proxy): keep a team block from being undone by in-flight spend writes
update_cache read the cached team when a request finished and wrote that
whole copy back in a background task, so a team blocked or invalidated
while requests were in flight came back unblocked on every busy worker
until its 60 s cache entry expired. The team spend now goes onto the team
held in this worker's memory at write time, and a missing entry is left
missing for the next auth read to load.
Block and unblock responses keep litellm_model_table and object_permission
null, as before this PR, while the cache still gets both relations.
* test(proxy): type the team spend cache test parameters
* test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock
---------
Co-authored-by: Hamza Shah <hamzasepal@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The override test slept a fixed 10 s after writing responses_websocket_session_limit_seconds and then
opened sockets on both workers. A worker whose config resync ran later than that kept the old 7200 s
cap, so its socket never closed and the test failed after 75 s
A test-only /health/integration-general-setting route returns the value a worker holds along with its
pid. The test polls it over fresh connections until every worker reports the new cap, and after the
delete until every worker reports it gone
`lite claude --subscription` sends the LiteLLM key in the x-litellm-api-key header through
ANTHROPIC_CUSTOM_HEADERS and leaves ANTHROPIC_AUTH_TOKEN and ANTHROPIC_API_KEY unset, so Claude Code
stays on its claude.ai login and its connectors keep working. The pre-launch key check sends the key
the same way.
`claude_subscription` and `pkce` are now `lite config` keys that default the new
`--subscription/--no-subscription` flag on `lite claude` and the `--pkce/--no-pkce` flag on
`lite login`. The flags still win for a single run.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cooldown cases in test_hosted_vllm_stream_stall_wire.py boot an owned proxy, which writes to a
private database since #47213. The file was written on a base without #47213 and read its rows from
the worker database through read_rows, so every cooldown case that waits for a failure row timed out
on main with State did not converge: []
Every row read now goes through Gateway.rows on the gateway that served the request
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
#46662 made streamed /v1/responses lifecycle events report the requested model in the nested response.model, matching the non-streamed body. The arize v2 caller-stream helper still expected the upstream deployment name there, so f1 and a6 failed on every main run since bbd446b476
The SSO removal test configured Google sign-in, which fetches its discovery
document from accounts.google.com before redirecting. CircleCI blocks that
egress, so the test failed on every main run since #46709 added it. Microsoft
sign-in builds its endpoints locally and goes through the same SSO environment
variable handling, so the test now uses it
* test(proxy): cover OpenAI pass-through billing for each /responses sub-route
* fix(proxy): bill only the OpenAI pass-through Responses calls OpenAI bills
* fix(proxy): fail closed on unknown /responses sub-routes and label unbilled spend rows
Bill every OpenAI pass-through /responses call except input_tokens, input_items,
delete, and a non-background retrieve or cancel, so an encoded or new sub-route
is never free. Unbilled calls keep the response's model on their spend row, and
a resumed background stream takes its model from the stream's final event.
* fix(proxy): give billed background Responses reads their own spend log row
A billed retrieve or cancel of a finished background response reused the
create's resp_ id, so the spend log insert dropped the row that carried the
cost. It now takes the native path's _background_cost suffix. Adds Azure
path coverage and types the test recorder.
* fix(proxy): bill a background Responses read only once it carries usage
An in-progress poll took the _background_cost spend log id with $0, so the
finished read's row that carried the cost was dropped again. Polls with no
usage are now unbilled like any other read.
* fix(proxy): keep OpenAI's response id for callbacks on a billed background read
* fix(openai): give a streamed background create the settlement spend row id
* fix(proxy): keep a request body from giving its spend row the background settlement id
* test(proxy): give the forged settlement marker test a fixed time
* fix(passthrough): move the background settlement marker into the pass-through types module
A test that reloads spend_tracking_utils gave it a new enum class, so the
handler's marker no longer matched by identity and the settlement row lost
its _background_cost id. The types module is never reloaded.
* fix(managed_files): send provider file ids when creating a fine-tuning job from managed files
Fine-tuning job creation with a managed training file sent the gateway's managed id and the provider-prefixed deployment model to OpenAI, which rejected the job. The pre-call hook now builds the file id mapping for training_file and validation_file, the deployment hook swaps both, the SDK strips the provider prefix from the model, and the create response returns the caller's own file ids
* fix(managed_files): route managed fine-tuning creates only to deployments holding every file
Fine-tuning creates now narrow routing the same way managed batches do, so a
deployment added after the upload is never picked. Adds a two-worker chaos
integration test and runs tests/unit/fine_tuning in the endpoints shard.
* fix(managed_files): keep fine-tuning creates on the model group when no deployment holds the files
A managed file used on a model group it was never uploaded for kept failing
with the provider's 400 before the routing filter. The filter turned that into
a 429 no-deployments error, so fine-tuning now falls back to the whole group in
that case while batch input files keep routing only to deployments that hold
them.
* test(fine_tuning): cover the provider prefix strip for OpenAI and Azure creates through the proxy
* test(fine_tuning): start each multi-deployment case from an empty upstream log
* test(fine_tuning): boot both proxies of the late deployment cell on one scratch database
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer 400 for a malformed tool_result text on the Responses bridge and count_tokens
A tool_result text block whose text is null, a number, or missing made
/v1/messages and /v1/messages/count_tokens answer 500 on OpenAI and Azure
models. Both routes now answer 400 naming the block, and drop the block
when drop_params is on.
* fix(anthropic): keep a missing tool_result text as empty text, bound the error, honor body drop_params on count_tokens
* fix(anthropic): apply the tool_result text check on the chat/completions bridge and every count route
The chat/completions bridge used for Azure, Gemini and other providers now answers the same 400 for a
null or numeric tool_result text and drops the block under drop_params, so a model group that mixes
OpenAI with another provider answers the same way on every deployment. Gemini and Vertex count_tokens
apply the same check, count_tokens turns on drop_params for Claude Code and Codex the way /v1/messages
does, and a tool_result without a tool_use_id is checked too
* fix(proxy): answer 400 for a malformed tool_result text on count_tokens models without a count API
Providers with no token counting API fall back to the local tokenizer, which counted a null or numeric tool_result text as fine while /v1/messages on the same model answers 400. The fallback now runs the same check, unless drop_params applies to the deployment or the request
* fix(proxy): honor request drop_params on the local count fallback when no deployment resolves
* test(integration): cover non-string tool_result text on both /v1/messages bridges and count routes
* fix(anthropic): keep a dropped tool_result on Mistral and reject it before a guardrail rewrite
Under drop_params a tool_result whose only text block was null or a number became an
empty tool message, and Mistral's empty-message filter removed it, so Mistral answered
400 for a missing tool reply. The filter now only removes empty assistant messages,
which is what Mistral rejects.
A guardrail that rewrites structured_messages wrote the dropped text back as "", so a
null or numeric text was silently blanked even without drop_params. The rewrite path now
answers the same 400 as the bridges unless the request or global drop_params is set.
* test(integration): cover the Mistral empty tool reply and the guardrail rewrite tool_result text check
* fix(anthropic): keep a malformed tool_result text through a guardrail rewrite and keep Mistral's null-message drop
A guardrail that rewrites structured_messages now writes the caller's null or numeric
tool_result text back in place, so the route's bridge and the deployment's drop_params
decide after routing, the same as with no guardrail. The rewrite path no longer answers
its own 400 before routing.
Mistral drops a null-content system or user message again, as main does, and only a tool
reply is exempt from that filter so a dropped tool_result keeps its tool call.
* test(mistral): expect empty system and user turns dropped and the empty tool reply kept
* perf(anthropic): restore kept tool_result text parts in one linear pass
* test(integration): cover the tool_result text restore across tool_results, the chat bridge, and a batch upload behind a rewriting guardrail
* fix(anthropic): let guardrails scan malformed tool_result text and honor drop_params on native Anthropic
A rewriting guardrail now sees a non-string tool_result text as its JSON text and the write-back
restores the caller's value by the sanitized tool_use_id when the guardrail left it unchanged.
Native Anthropic drops the bad block under drop_params on /v1/messages and count_tokens, and
Mistral keeps an empty tool reply only when it answers a tool call, sent as empty text.
* fix(anthropic): compare a kept tool_result text to its scanned form once and align the wire tests with drop_params
* test(integration): cover native Anthropic drop_params resolution, guardrail scanning of numeric tool_result text, and the Mistral orphan tool message
* fix(count_tokens): honor the router's default drop_params when counting a malformed tool_result text
* test(count_tokens): send the tool_result count tests through a real request body instead of a mocked parser
* test(count_tokens): send the native Claude count test through a real request body
* fix(anthropic): keep each non-string tool_result text when tool_use_ids sanitize to the same id
* fix(anthropic): keep non-ASCII characters in the guardrail's view of a non-string tool_result text
* test(gemini): mock the count_tokens upstream with respx instead of patching the client factory
* fix(mistral): keep an empty tool reply only for an open call, and forward non-object message items on the native drop
An empty tool reply re-answering a call the conversation already moved past made Mistral reject the request, and a tool call id that is not a string crashed the filter. Native Anthropic under drop_params assumed every message item was an object.
* fix(mistral): find answering tool replies in one pass instead of copying the open call set per reply
* fix(mistral): ignore dropped empty placeholders when tracking open tool calls
* fix(mistral): drop an empty tool reply when another reply to the same call carries the result
* refactor(anthropic): share tool_result text helpers to shrink the diff
* fix(anthropic): answer 400 for a non-string tool_result text on the OpenAI Responses bridge
* fix(prometheus): label deployment failures with the deployment's provider instead of None
* test(prometheus): drive every deployment failure surface through the provider label
Streaming chat, responses and messages, dropped and stalled upstreams,
JSON-configured, custom-provider-map, prompt-management and unprefixed
deployments, forged request metadata, an unauthenticated request, the
OpenAI and Anthropic SDKs sync and async, and a concurrent burst counted
exactly once per deployment
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(logging): report the cost-map entry billing used in model_map_information
Billing resolved the deployment's own cost-map entry and logging ran a second,
name-only lookup, so callbacks and spend logs showed the catalog rates for a
custom-priced deployment. completion_cost now stores the entry it billed from on
the logging object and the standard logging payload reads it, falling back to the
old lookup only when billing never ran.
* fix(logging): pass the deployment id into the logging cost-map lookup
Drop the logging-object plumbing and keep the lookup where it was:
get_standard_logging_object_payload now hands the deployment id to
get_model_cost_information as router_model_id, the same id billing uses, so
the deployment's own cost-map entry is reported instead of the catalog entry.
* test(logging): mirror the proxy metadata in the deployment-price regression test
* fix(logging): keep the catalog model_map_key, report only the deployment's own rates
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure): cache the remaining per-request Entra credentials and keep token refresh off the event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): await the now async azure health-check auth header helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure): await async azure_ad_token_provider results in the threaded wrapper
* fix(azure): keep transport params when validating async Azure client params on pydantic 2.11
* test(azure): assert the token and validation thread hops by thread identity instead of sleeps
* fix(azure): give the OIDC token exchange its own short-lived HTTP client
The exchange now runs from worker threads, so it no longer shares litellm.module_level_client's connection pool with other threads
* test(azure): check the OIDC exchange closes its client through an in-process transport
* refactor(responses): merge caller extra_headers into a new dict instead of updating validated headers in place
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
