Commit graph

56552 commits

Author SHA1 Message Date
mateo-berri
9a388a58c9 Merge remote-tracking branch 'origin/main' into HEAD
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts:
#	litellm-rust/crates/host-python/src/marshal.rs
#	litellm/rust/crates/host-python/src/depythonize.rs
2026-11-14 04:22:21 -05:00
Mateo Wang
1a8ed27237
refactor: clear fresh debt in customer hierarchy and v2 callback modules (#51804)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 01:14:06 -08:00
jesus-berri
3988ce2865
fix(proxy): honor cancel_on_disconnect on the rerank endpoints (#51252)
Some checks are pending
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / mcp-elicitation (push) Blocked by required conditions
unit / mcp-server (push) Blocked by required conditions
unit / proxy-auth (push) Blocked by required conditions
unit / proxy-endpoints (push) Blocked by required conditions
unit / proxy-extras (push) Blocked by required conditions
unit / proxy-feature-endpoints (push) Blocked by required conditions
unit / proxy-hooks-client (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(proxy): honor cancel_on_disconnect on the rerank endpoints

* test(proxy): cover a connected rerank succeeding with cancel_on_disconnect on

* test(proxy): cover a connected rerank answer through the cancel watcher in the integration rig

* fix(proxy): read cancel_on_disconnect on rerank the same way the chat path does

* fix(proxy): set up the rerank request logger before routing so a hang-up after dispatch logs the query's tokens like chat

* fix(proxy): keep flag-off rerank logging unchanged and price a cut-off rerank from its query and documents

The rerank endpoint now builds its request logger only under cancel_on_disconnect, so a flag-off rerank logs one unbilled 408 row as on main.

The failure estimate for rerank counts the query and every document sent, prices the deployment the router picked, and bills one query on a model priced per query.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:59:22 -05:00
Yogeshwaran Ravichandran
35b9df1f40
feat(mistral): add native upstream token counting support (#50652) (#50653)
* feat(mistral): add native upstream token counting support

* test(mistral): standardize on mistral-large-4 and add tool calls tests

* fix(mistral): unify credential resolution and guard Azure endpoint boundary

* fix(mistral): resolve api base from provider env vars and route responses count natively

* fix(test): use groq for bridged proxy tests and increase mistral coverage

Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov.

* fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter

* fix(mistral): suppress basedpyright warnings on untyped httpx handler

* fix(mistral): keep chat credential resolution and count with the key chat resolves

The token counter now resolves its Mistral key and base through the same helper chat uses, so
a count goes to the endpoint chat would call. A deployment that would route to Azure, or an
Azure-only env key, falls back to the local estimate instead of sending that key to Mistral.
Chat key and base precedence is unchanged from main.

* fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route

* fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them

Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge,
which restores previous_response_id history. The count now takes the same bridge instead of forwarding
previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone.

The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an
input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them.

Tests now go through the real handler with respx, and the integration Mistral peer serves the count route.

* fix(mistral): leave empty assistant turns out of the count, as Mistral chat does

Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count
endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400
through previous_response_id, so the count now skips the same turns chat skips.

* fix(mistral): count what chat sends and keep the bundled Mistral model list

A user turn that mixed file_id files with an image or an inline file lost the image
or inline file in the count, since the converted content arrives as a tuple and only
lists were read. Both content readers now take either shape

A /v1/responses count with a json_schema text.format now sends that format to
Mistral, so the count matches what the bridged chat call bills

Counts for Mistral models that are not chat models (embeddings, OCR, transcription)
now fall back to the local estimate instead of returning Mistral's 400

Count calls now use the deployment's timeout when one is set, and a count response
without input_tokens falls back to the local estimate

MistralConfig.get_models goes back to the bundled model list, so turning on
check_provider_endpoint does not change what /v1/models shows for mistral/*

The previous_response_id count test now fakes the spend logs table instead of the
history lookup, so the real lookup and its query run

* fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral

* fix(mistral): answer 408 on a count timeout and use the shared deployment timeout

Main's native counters answer a count timeout with 408 and a connection
error with 500. The Mistral counter fell back to a local estimate on both,
so a hung Mistral endpoint looked like a successful count. It now raises the
same count errors, and it reads the deployment timeout with the shared
deployment_timeout helper instead of its own copy

* fix(mistral): join count text blocks the way Mistral chat does

* fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat

* fix(mistral): count the thinking chunks Mistral chat replays in assistant history

The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content
carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is
skipped the way chat skips it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:19:57 -05:00
quietjules
9ba216d521
fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239)
* fix(vertex_ai): reuse one Google token across vector store searches

* fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst

* test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy

* test(vertex_ai): type the fixtures and token callback in the concurrent search test

* fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects

A shared RAG Engine config made one store's slow token fetch hold up every
other store, and kept a rotated key file's deleted key until a restart.
First loads and expired refreshes now take a lock per credentials and project,
refreshes take a lock per credentials object, and any RefreshError reloads the
credentials once.

* fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls

* test(vertex_ai): check async callers sharing an expired credential refresh it once

* fix(vertex_ai): reload Google credentials only when a refresh is rejected for good

A retryable refresh error such as a 503 now raises without reloading the key, so an outage
does not double google-auth's retries. Credentials that cannot be weak-referenced share the
config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase
and holds the token response until the other callers have started

* fix(vertex_ai): reload Google credentials only when Google answers invalid_grant

google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads

The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did

* fix(vertex_ai): pick up a swapped Google key file on the next call

The credential cache key now carries the key file's inode, mtime and size, for a
vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced
or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of
after Google rejects the old key or the proxy restarts

* fix(vertex_ai): keep the earlier key while a replaced key file does not load

A key file caught halfway through a non-atomic write, or replaced by one that does not load,
now keeps the credentials loaded from its earlier version in use and logs a warning, as the
cache did before key files were re-read. The next complete file is loaded on the next call.
The pub/sub test no longer writes a key file it only reaches through the seeded cache

* fix(vertex_ai): try a replaced key file again when its first token fetch fails

The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one

* fix(vertex): raise when an async reload keeps an earlier key Google rejected

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:12:09 -05:00
tavren27
c4c16c7b3f
fix(azure): drop the traceback from stream-cut errors sent to clients and alerts (#51760)
* fix(azure): drop the traceback from stream-cut errors sent to clients and alerts

An Azure chat or text completion stream cut mid-response raised an APIConnectionError whose message carried the full Python traceback, so the in-stream error frame, the /v1/completions 500 body, and every llm_exceptions alert showed server file paths. The Azure no-status branch now keeps the error line alone, and route_failure_proxy_exception redacts the message it answers clients with, the way the chat route already does

* fix(proxy): keep route error messages as raised and test the Azure stream cut through the proxy routes

Drop the route-level client message redaction, which rewrote valid guardrail and hook 400 messages, and cover the Azure chat and text stream cut end to end through respx

* fix(azure): name the error type when a connection error has no message

An httpx reset raises ReadError with an empty message, which left the client line with no cause once the traceback was gone

* test(proxy): drop the docstring from the Azure stream cut fixture

* test(proxy): stop pinning the MidStreamFallbackError prefix in the Azure text stream cut test

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 08:00:58 +00:00
quietjules
93f8d86c89
fix(gateway): return Claude thinking blocks on chat completions instead of a 500 (#51774)
* fix(gateway): return Claude thinking blocks on chat completions instead of a 500

The gateway's Anthropic chat transform answered 500 on any thinking or redacted_thinking block. It now maps them into reasoning_content and thinking_blocks the way the Python proxy does.

* fix(rust): replay Claude thinking blocks a chat client sends back on its next turn

A client that sends a thinking reply back now gets it replayed to Anthropic the way Python does it, dropping blocks Anthropic could not have signed and unwrapping Bedrock-wrapped redacted data, where the gateway used to decline with 501

* fix(rust): replay a cut-off Claude thinking reply and send a repeated thinking block once

A reply that runs out of max_tokens mid-thought comes back with content null and thinking_blocks. Sending it back now replays its thinking blocks alone, the way Python does, instead of answering 501

A thinking block that several merged assistant messages carry goes to Anthropic once, where it first appears, like Python's _without_repeated_thinking

New and changed tests use rstest, as litellm/rust/AGENTS.md asks

* perf(rust): drop repeated thinking blocks in one pass

* fix(rust): send a thinking reply on to OpenAI-compatible models like Python

Python passes chat messages to OpenAI-compatible endpoints verbatim, so a Claude reply carrying thinking_blocks and reasoning_content reaches the model as sent. The gateway declined it with 501 instead

* fix(rust): drop null message fields before an OpenAI-compatible call like Python

Python's cleanup_none_field_in_message removes None fields before the call, since some endpoints reject them

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:44:23 -05:00
paveln83
dcba0cf6f0
fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content (#51747)
* fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content

Foundry's Fireworks-hosted Kimi K3 takes reasoning_effort as a level string, but azure_ai refused it (400 on /v1/messages with thinking) or dropped it. Models whose cost-map row declares reasoning_effort_levels now get the param, rounded up to a declared level, with deployment-declared levels taking precedence. A dict effort sends its level, and one without a level answers 400 or drops under drop_params. FW- models also keep reasoning_content on replayed assistant turns, which Kimi reads back across tool calls

* fix(azure_ai): read declared reasoning levels from the loaded catalog

The router copies a deployment's model_info, reasoning_effort_levels included, onto the shared azure_ai/<model> key in litellm.model_cost, so reading that live map let one deployment's levels open reasoning_effort for an undeclared sibling. Reading the catalog as loaded keeps the gate on what the cost map itself declares, while deployment levels still round the request through the deployment path

* fix(azure_ai): pass Foundry reasoning levels through and match the catalog row regardless of case

Foundry Kimi K3 accepts every reasoning_effort level, so the level is sent as the caller wrote it instead of rounded onto the catalog row. "default" leaves the level to the model. A value outside the level names answers 400, or is dropped with a warning under drop_params. An effort object sends its level and warns that its other fields are dropped. The catalog row lookup now also matches a key that differs only in case

* fix(azure_ai): read base_model for Foundry reasoning effort and replay, accept auto

* test(azure_ai): cover global drop_params for a Foundry effort that is not a level

* fix(azure_ai): keep replayed reasoning_content for Foundry kimi- models

Foundry's MoonshotAI-format Kimi deployments (kimi-k2.6, Kimi-K3.1-Pro) read
replayed reasoning_content back, so it is no longer stripped for them. The
effort-object warning now names only the fields it drops.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:08:11 -05:00
tavren27
d76cdcbb8a
fix(proxy): list key and team aliases on /model_group/info and apply listing callbacks to them (#51754)
* fix(proxy): list key and team model aliases on /model_group/info

* fix(proxy): keep listing-callback and pause filters on model alias rows

A name a legacy or V2 listing callback hides stays hidden on /v1/models,
/v1/models/{model_id} and /model_group/info even when it is also an alias.
V2 Hook.list_models subscribers now limit which target an alias row
describes, /model_group/info describes no alias by a paused model, and
?model_group=<name> offers listing callbacks only the names that row reads.

* test(proxy): annotate the new alias test locals as Final

* fix(proxy): keep a caller alias listed when a router alias of the same name points at a hidden model

A key or team alias always wins over a router model_group_alias of the same name, so alias rows are now dropped only for names a listing callback hid itself, not for router aliases whose target it hid

* fix(proxy): describe an alias by an unlisted target only when listing callbacks keep it

Listing callbacks are now also offered the unlisted model each listed key or team alias
sends requests to, and the model a router alias of that name resolves to. The alias row
is described by that model only when the callback keeps every one of those names, and a
name the callback hides there hides no listed name. Before, any installed listing
callback, V2 included, left such an alias with no limits and a 404 lookup

* fix(proxy): describe an alias to a team model's public name when the key lists its internal name

/model_group/info lists team models by their internal routing name, so an alias to the
public name found no listed target and got no row while /v1/models listed it. Alias
resolution now reads the listed names by their public names. The listing callback
docstring also names litellm.model_alias_map among the alias sources.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:10:32 -05:00
zoveki
8973e2dc4f
feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project (#50732)
* feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project

Adds the opt-in litellm.vertex_request_label_fields setting, the Vertex twin of bedrock_request_metadata_fields. When set, Gemini, embeddings, Imagen and Gemini image generation, rerank, and Claude on Vertex (chat and /v1/messages) send the listed proxy identity fields and spend_logs_metadata as request labels, identity first, with the user_api_key_ prefix reserved so a caller cannot forge or evict them. Identity text is rewritten to fit Google's label rules. Gemini image generation now forwards labels at all.

* feat(vertex_ai): send the opt-in identity labels on multimodal embeddings, the Google GenAI route, and batches

With litellm_settings.vertex_request_label_fields on, the multimodal embedding
predict body, the native generateContent and streamGenerateContent body, and the
BatchPredictionJob now carry the key's identity labels the same way chat does

Caller labels on GenAI are resolved after the extra_body merge, so neither a
top-level labels field nor extra_body.labels can write the reserved prefix. Batch
labels come only from the proxy's litellm_metadata, never the caller's OpenAI
batch metadata. With the setting off every route sends the same body as before

* fix(vertex_ai): keep identity labels when a Gemini caller sends extra_body labels

* fix(vertex_ai): label Interactions, image edits, Mistral, and the /vertex_ai pass-through, and gate Gemini image labels on the setting

* fix(vertex_ai): move body labels into the labels header on Vertex rawPredict pass-through calls

* fix(vertex_ai): send Mistral extra_body labels in the labels header instead of the body

Labels a caller nests in extra_body now merge over top-level labels before the
X-Vertex-AI-Labels header is built, the way the Gemini body merges them, and
leave the outbound body. The token-fetch stubs in the Mistral and multimodal
embedding tests are scoped to their instance so later tests see the real method

* fix(vertex_ai): limit pass-through labels to POST and to publishers that take the labels header

rawPredict to custom endpoints and publishers other than Anthropic and Mistral, and PATCH or PUT bodies, now reach Vertex as the caller sent them. The pass-through drops a caller's labels header where it writes body labels, a single field written as text turns the setting on for that field, and an undecodable Claude labels header falls back to the caller's labels

* fix(vertex): label Veo video, Gemini transcription, and pass-through predictLongRunning requests

* fix(vertex): read the pass-through publisher from the end of the upstream path

A project id named publishers shifted the first-match lookup, so a forged X-Vertex-AI-Labels header reached Claude or Mistral untouched

* fix(vertex): drop a caller labels header on body-labelled Vertex routes while identity labels are on

* test(vertex): script the v1beta1 GenAI target in the labels header integration test

* test(vertex): type the transcription test's access token override

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 00:39:01 -05:00
Mateo Wang
c6aa2a3b2a
test(logging): scope bridged call recorder to its request (#51339)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:34:56 -08:00
Mateo Wang
e87d7bd714
fix(anthropic): restore sync messages streams for OpenAI and Azure (#51305)
* fix(anthropic): return a sync stream from sync messages.create on openai and azure models

* fix(anthropic): keep the sync messages stream readable with async for and end it on close

* fix(responses): close the chat completions stream when a sync messages stream closes

* fix(anthropic): release sync streams on async close

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:34:21 -08:00
Mateo Wang
ccb7f753c9
fix(mistral): send Magistral reasoning as supported native levels (#51329)
* fix(mistral): send reasoning_effort and thinking to magistral models as a level Mistral accepts

magistral-small-latest and magistral-medium-latest now take reasoning_effort
(none or high) as their reasoning switch, but LiteLLM dropped reasoning_effort
and thinking for any magistral model and injected Magistral's deprecated
reasoning system prompt instead, so streamed calls returned no reasoning and
"none" still reasoned.

Forward reasoning_effort rounded to the cost map's declared levels, map
thinking onto it (disabled becomes none), drop the system prompt, and declare
none and high on both magistral rows.

* fix(mistral): send adaptive thinking to magistral as a reasoning level

* fix(mistral): reject reasoning_effort strings that are not a level, and leave "default" to Mistral

A string such as "HIGH" or "auto" went to Mistral as sent and Mistral answered 422, even with drop_params. It now takes the same path as a number: dropped with a warning under drop_params, a 400 without it. "default", which litellm.completion lists as a value, sends no reasoning_effort so Mistral applies its own default.

* fix(mistral): preserve thinking when unsupported effort is dropped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(mistral): raise reasoning failures at the public boundary

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mistral): share fixed capabilities with warning regressions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:33:54 -08:00
norkelbyte
8decbe4801
fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources (#50577)
* fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources

An s3:// URL in an image_url or file part now becomes a Converse
s3Location source instead of failing with "Unsupported image type".
The format comes from the part's format or the key's extension, an
optional bucket_owner is sent as bucketOwner, and an unknown or
unsupported format answers 400. An image in an assistant turn no
longer crashes the async path.

* fix(proxy): limit non-admin s3:// media to allowed buckets and refuse managed prefixes

A non-admin key may only send an s3:// media URL whose bucket is listed in
general_settings.allowed_client_side_s3_buckets (or when client-side
credentials are allowed), and never one under litellm's managed prefixes.
Proxy admins are not limited. A bucket_owner on an s3:// URL that is not a
non-empty string is a 400, the Responses bridge forwards the owner only for
s3:// URLs, audio on S3 is a 400, and video in a tool result is sent to Bedrock.

* fix(proxy): check s3:// media in every request field and refuse it in Bedrock batch files

The s3:// media check now reads the whole request body instead of a fixed
list of fields, and it also reads Gemini file_uri and fileUri, so a
/v1beta generateContent request with an s3:// file part gets the same
bucket rules as chat, Responses, and Anthropic requests. A Bedrock batch
input line with an s3:// media URL is a 400 before upload, since Bedrock
batch files do not take s3:// media URLs yet.

* fix(bedrock): say that LiteLLM refuses s3:// media in Bedrock batch files

The 400 for a Bedrock batch input line with an s3:// media URL now says
"LiteLLM does not accept s3:// media URLs in Bedrock batch files", since
Bedrock batch takes S3 locations and it is LiteLLM that refuses the line.

* fix(proxy): check s3:// URLs only inside media parts and refuse them only on Converse batch lines

The s3:// bucket check now reads URL fields only inside the content parts that a Bedrock transform turns into an S3 source, and it uses the same s3:// test as the prompt factory. Values in metadata, annotations, tool arguments and other fields pass as before. Each URL is named once in the 403 and 400 messages

Bedrock batch files refuse s3:// media only on the lines that go through the Converse transform, so passthrough-provider lines upload unchanged

* test(proxy): check that a media part's non-URI fields never count as s3 media

* fix(proxy): check s3 URIs inside encoded file ids and on queued chat completions

The s3 media check now decodes model-embedded and unified file ids, since
completion() turns them back into the raw s3 URI they carry. The
/queue/chat/completions endpoint skips common request processing, so it now
runs the same check. The media walk goes level by level instead of
recursing, which the recursive_detector code-quality step requires.

* fix(proxy): let a key send managed file ids it may read and refuse s3 sources hidden in model_file_id_mapping

The s3 media gate refused every LiteLLM managed file id whose upload sits under the
managed prefix, even when the key's own user or team created it. It now asks the
managed files hook which unified ids the key may read and skips only the objects
those ids embed. A forged or unowned id, or a proxy without the hook, still gets a 403.

The gate also reads the provider file ids in a client-supplied model_file_id_mapping,
so an alias can no longer point a non-admin request at an unlisted or managed object.

Markdown documents (.md, .markdown) now map to Bedrock's md document format, so an
s3 Markdown source without an explicit format no longer fails with an unknown media type.

* fix(proxy): check s3 media on every Responses WebSocket frame and keep inline tool-result video dropped

Every response.create frame on WS /v1/responses now runs the same s3 media check as
HTTP, so a non-admin key can no longer send an unlisted or managed s3 object over a
WebSocket

A tool result forwards a video to Bedrock only when it is an s3Location. An inline
base64 video is dropped with a warning, as it was before this PR, since Converse
rejects video in a toolResult block

A non-admin key that sends an AWS identity its deployment admits through
configurable_clientside_auth_params (aws_role_name, aws_web_identity_token or
aws_profile_name) skips the bucket list, because Bedrock then reads the object with
that identity. The managed-prefix 403 still applies. A WebSocket frame's own AWS
identity fields never open the bucket list, since the managed WebSocket handler
does not forward them and Bedrock reads with the deployment's identity

The check now skips the body walk when no string starts with s3:// and no file_id or
model_file_id_mapping key is present, which keeps large image bodies cheap

* fix(proxy): let a client AWS identity skip the s3 bucket list only when Bedrock signs with it

An admitted aws_role_name, aws_web_identity_token or aws_profile_name now skips the
bucket list only when it decides the credentials get_credentials picks, on every
deployment of the model group. A web identity token without a role and a session
name does not, since get_credentials then ignores it. A profile does not when the
deployment or the environment sets a role, since the role branch runs first. In
those cases Bedrock still reads with the proxy's credentials, so the bucket list
applies

BaseAWSLLM.identity_choosing_params mirrors the branch order of get_credentials, and
both read the same AWS_* environment fallbacks through one helper

* fix(proxy): keep the s3 bucket list for non-admin keys that send a client AWS identity

A client AWS identity field does not prove that the caller can read the bucket. A
role or profile name picks credentials the proxy holds, and Bedrock reads the object
with them. So the bucket list now applies to every non-admin s3 media part, whatever
AWS identity fields the request sends. allow_client_side_credentials: true still
turns it off

This removes BaseAWSLLM.identity_choosing_params and the router lookup that only the
skip needed

* fix(proxy): look up managed file ids only when they can change the s3 refusal

* fix(bedrock): check every file id field, skip MCP tool calls, accept registered S3 media types

* fix(proxy): refuse unlisted s3 media on the rag query and adapter routes

* style(proxy): format the merged collections.abc import

* fix(proxy): walk tuples when checking s3 media and skip plain number lists

* fix(proxy): skip numbers when walking a request body for s3 media

A body with large token-id lists and any s3:// string or file_id key made
the full walk build a node per number, taking seconds per request. The walk
and its prefilter now skip int, float, bool and None values.

* test(proxy): check the s3 refusal through the real request setup

* test(proxy): exercise real setup for allowed s3 media

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(model-armor): fail closed on remote s3 documents

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): suppress cyclic fallback payload inference

Limit the suppression to reportGeneralTypeIssues on the legacy payload assignment. The fallback payload is constructed as a dict and the runtime AST is unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:33:21 -08:00
Nate Armstrong
90122c07a2
ci(integration): point the rust gateway install step at litellm/packaging (#51743) 2026-11-14 13:36:57 +09:00
mika-lt
7ab37a1529
fix(count_tokens): answer 400, not 500, for a message without role or content on Anthropic and Azure AI (#51284)
* fix(count_tokens): answer 400, not 500, for a message without role or content

Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.

* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count

* refactor(bedrock): leave message validation to the Bedrock count handlers

Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy

* fix(count_tokens): name the chat message chat translation cannot send

A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back

* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content

Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it

* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse

The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate

* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse

A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again

* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given

* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks

* test(integration): expect the 400 for a count whose leading system message has no content

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:21:29 +00:00
norkelbyte
dc378de207
fix(cli): let Claude Code renew a --pkce login's key through lite's apiKeyHelper (#51319)
* fix(cli): let Claude Code renew a --pkce login's key through lite's apiKeyHelper

* fix(cli): keep an explicit --api-key over lite's apiKeyHelper and persist its absolute path

An explicit --api-key or LITELLM_PROXY_API_KEY is now exported to Claude Code
even when lite's renewing helper is configured, since the helper prints the
stored login's key instead. The helper command also uses the absolute path of
the lite found on PATH, so a relative PATH entry cannot make Claude Code run
another repository's bin/lite

* fix(cli): keep an explicit key and --pkce when lite's renewing helper is set

A --settings that blanks apiKeyHelper now keeps the exported key, and a key passed with --api-key turns lite's helper off for that launch, so Claude Code no longer warns about two credentials or sends the stored login's key beside it

The helper names the lite installed beside the Python that ran the login before one found on PATH, so a pyenv shim no longer fails in projects pinned to another Python

lite claude signs in again with --pkce when the stored login used it, and print-token's hints name the proxy and --pkce

* fix(cli): keep a shell token from beating the renewing helper and reuse --pkce only for the same proxy

The --pkce --config-claude login now writes an empty env.ANTHROPIC_AUTH_TOKEN beside the helper, since a token exported in the shell wins over apiKeyHelper

lite claude sends any other key through a private settings file that blanks the helper and merges the user's own --settings

A re-login keeps --pkce only when the stored login is for the same proxy, else the saved default decides

* fix(cli): let any lite install's helper renew for its proxy and name the renewing login in print-token hints

lite claude now treats a lite auth print-token helper for this proxy as renewing whichever lite path it names, since every lite reads the same token store

Every print-token hint to sign in again, including an expired key and an unreadable keychain, names lite login --pkce with the proxy

Shorter docstrings

* fix(cli): send a launch key to the proxy it was issued for

The private settings file lite claude writes for a key that lite's helper would not renew now also sets ANTHROPIC_BASE_URL. Without it, lite --base-url B --api-key <B key> claude after a --pkce --config-claude login to A sent the B key to A, since the login's settings point Claude Code at A

* fix(proxy): preserve caller metadata on auth failures (#51745)

* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs

Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.

* fix(proxy): type the spend logs metadata header parser honestly

The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.

* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket

A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.

* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler

Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.

* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer

The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.

* test(proxy): read auth failure spend rows from a real unconnected PrismaClient

Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>

* fix(logging): keep a content-block system prompt in logged messages (#51699)

* fix(logging): keep a content-block system prompt in logged messages

A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.

Fixes #36402

* fix(logging): type the logged system message

* fix(logging): type the system message without a mutable annotation, close the test client

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter (#50483)

* test(mcp): check the tool dict conversion without patching the MCP handler

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter

* refactor(mcp): type the semantic filter's default tool loader

Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.

* fix(mcp): give the MCP client exactly the servers left on in the connect page (#50973)

* fix(cli): send the launch key under a project apiKeyHelper and give each key its own settings file

A `lite login --pkce --config-claude` leaves an empty ANTHROPIC_AUTH_TOKEN in Claude Code's user settings. When a repo's project settings set their own apiKeyHelper, `lite claude` skipped its private settings file, so the empty token hid the exported key and Claude Code sent the project helper's key. `lite claude` now also writes that file whenever the settings files leave ANTHROPIC_AUTH_TOKEN empty

Every key launch also wrote the one file `--subscription` uses, so two launches started together could read each other's key and proxy. A key launch's file is now named after a hash of its contents, and every overlay is written to a temp file and renamed so a starting Claude Code never reads half of one

* fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)

* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

* test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cli): skip the working directory when the login looks up lite for its helper

On Windows, shutil.which searches the current directory before the directories it is given, so a lite.exe planted in the repo where someone ran `lite login --pkce --config-claude` could become their user-level apiKeyHelper

The login now checks each directory by full path, this Python's scripts directory first and then each PATH entry, skipping empty entries

* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)

* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved

Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.

* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows

* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs

* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params

* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params

* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase

* fix(bedrock): mark the service tier block field read-only

* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params

* test(prometheus): sort the service tier label test imports

* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400

* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova

* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk

The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one

* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name

The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it

* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added

The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

* fix(cli): delete the key files lite claude leaves on logout and unconfigure

lite claude hands its process to Claude Code, so the --settings files holding
its key stayed in the Claude config dir. lite logout and lite unconfigure claude
now delete them, and Claude Code reads one only at startup, so open sessions
keep working

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: joshua-berri <joshua@berri.ai>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: shrey <shrey@berri.ai>
Co-authored-by: Ankit Jha <45779665+PyExtreme@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: quietjules <349048734+quietjules@users.noreply.github.com>
2026-11-14 04:04:38 +00:00
zoveki
01de1a686d
feat(opensearch): map vector store search filters to k-NN filters and add hybrid search (#51131)
* feat(opensearch): map vector store search filters to k-NN filters and add hybrid search

* test(code-quality): allow the bounded OpenSearch filter recursion

* fix(opensearch): retry an unfilterable engine without filters under drop_params, match strings on keyword subfields, keep per-request field keys

Under drop_params, a 400 'does not support filters' from an nmslib or older
engine re-runs the search once without filters, as filtered searches behaved
before. String eq, ne, in and nin match the key's .keyword subfield, falling
back to the key where no subfield is mapped. Only the index name, search
pipeline and hybrid flag are registration-only, so opensearch_text_field and
opensearch_embedding_field pass through per request again

* fix(opensearch): match string filters on the exact keyword field and embed once on the filter retry

String filters now read the index mapping and compare the exact keyword field, refusing a key the index cannot match exactly. The drop_params retry reuses the first embedding and also covers OpenSearch before 2.4. An index that refuses filters without drop_params answers litellm's own 400, and a filters value that is not an object stays a 400 under drop_params. The Admin UI store form gains the search pipeline and hybrid search fields

* test(ui): expect the OpenSearch search pipeline and hybrid search fields in the provider field list

* fix(opensearch): drop string filters under drop_params when the mapping read fails, and skip normalized keyword fields

* fix(opensearch): name the permissions a refused index mapping read needs instead of returning OpenSearch's body

* fix(opensearch): check the index mapping for every filter, resolve aliases, and keep failed mapping reads from dropping filters

* fix(opensearch): keep an alias filter's own name and refuse rewrites that reach an index without the alias

* fix(vector_stores): sign the caller's headers as given and reject filters on keys only some OpenSearch indexes map

A null or non-str extra_headers value no longer fails Bedrock Knowledge Base, S3 Vectors, or other provider searches before signing. The mapping read gets only the string-valued headers.

A filter on a key that some of a store's indexes map and others do not answers 400 (drops under drop_params), since OpenSearch returns its k-NN hits on some calls and none on others.

* fix(opensearch): send the caller's headers on the index mapping read as given and name the fix for keys only some indexes map

A header httpx cannot send now fails the filtered search on the mapping read, the same way it fails the search, before the query is embedded. Bytes headers now reach the mapping read too.

* test(opensearch): pin that a timeout on the index mapping read answers 408 without embedding the query

* fix(opensearch): answer 400 for a filter key inside a nested field or naming an object

A root-level term query on a field under a nested mapping, or on an object field, matches nothing, so the search answered 200 with no hits. These keys now answer 400 naming the mapping, or drop the filters under drop_params

* fix(opensearch): filter on a nested field whose values OpenSearch copies to the root

A nested field mapped with include_in_root, or include_in_parent on it and
every nested field above it, has its values in the root document, so a root
term query matches it. Those filters now search instead of answering 400.
A key that names an object or nested field answers that it holds fields
rather than values before the nested check, so its remedy no longer leads
into a second 400

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:30:55 +00:00
moe-berri
752eb7c5b7
fix(opencode): stream Go Messages-protocol models through chat completions (#51715)
* test(opencode): cover Go Messages chat streaming

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): delegate Go Messages chat streaming to Anthropic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): preserve Messages stream usage and cleanup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 22:21:52 -05:00
Hoya
8ef451a187
fix: raise on failed OpenRouter non-streaming completions (#50880)
* fix: propagate OpenRouter errors and preserve failure usage

* test: cover OpenRouter errors and failure accounting

* fix: bill a recovered OpenRouter failure on the success log and parse the body once

A retry or fallback that recovers from an OpenRouter finish_reason error now
carries the failed attempt's usage and cost into the success log, so the key
pays for both calls. The response body is parsed once and handed to the error
check, and the failure-log test no longer waits on a 10-second deadline

* fix: carry a failed attempt's usage before each SDK retry

* test: cover OpenRouter's documented failure body without usage

Pins the recovered-failure test's start time to a fixed date and adds a regression test for the documented non-streaming error body, which carries no usage

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:02:00 +00:00
Nate Armstrong
cd2e4b6b60
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0)

* feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope

* fix(custom-logger-v2): add field projections and keep the request scope across copies

* fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line

* fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists

_safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones.

* fix(custom-logger-v2): contain a failing v2 callback on logging hooks

A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged.

* fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines

run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path.

* fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point

Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes.

* fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope

The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected.

* fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object

Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged.

* refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests

Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow.

* fix(custom_logger_v2): await a single-entry v2 stage in the caller's context

A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather.

* fix(custom-logger-v2): raise a native callback panic as a runtime error

* fix(custom-logger-v2): run a synchronous v2 line on a private event loop

* fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler

* fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure

* fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope

* fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once

* fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure

* fix(callbacks): reject a selection name or alias that is not lowercase

* fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks

* fix(router): type the fallback event kwargs handed to the v2 dispatch

* feat(custom_logger_v2): type the contract fields as records and mark derived-only fields

Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy

* feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts

A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers

* feat(callbacks-python): hand the logging payload to native callbacks by reference

NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy

* feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity

Typed members keep the bool, int, float or str python holds through a Scalar
type on both sides, so ids, limits and priorities no longer lose values that
lax pydantic refused. A member the contract cannot carry is named in rejected
so a port can reproduce what python does with it.

Logging hooks get logged_key_identity, built from the payload metadata the
python loggers read on every route, while key_identity stays on request hooks
and keeps reading the auth object.

* fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways

budget_reservation read the dict budget_reservation.py builds with a KeyError
on callback_bound. Undeclared keys are now ignored, callback_bound is a member,
entries keep a non-mapping item as null so capacity's model access group check
stays exact, and costs read as the float budgets computes with float(x or 0.0).

Float members cross as the raw json marker when they are not finite, and a
native verdict returns through json_to_py, so an infinite or NaN cost survives
a read and a write instead of turning into null.

* feat(callbacks): carry a python value as its str() text and truthiness in scalar members

Scalar gains a Stringified variant with the str() text and bool() of a value
that is not a bool, int within i128, float or str, so a port can reproduce
the label, path or log text python formats and the falsiness of an empty
container without confusing it with a real string. Budget reservations say
whether python held the entries as a list.

* feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary

A Scalar member now carries a list, dict, set, bytes, object or an int past
i128 as its str() text and truthiness instead of naming it in rejected, which
stays for values whose str() or bool() raises. request_tags carries any
iterable as the items prometheus iterates, and budget reservation entries
carry what budgets iterates with entries_is_list for capacity's list check,
never written back to the shared dict.

* feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int

A Stringified scalar now carries hashable, from trying hash() at the
boundary, so a port can raise the TypeError capacity's priority lookup and
the limiter's descriptor identities raise for a list, dict or set and not
for a tuple or object. It also carries integer, so an int past i128 is told
apart from any other value whose text is digits.

* fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does

Prometheus treats any guardrail_mode that is not a list as a single mode, so a
tuple mode holds no str mode and a missing mode is one None mode. The
derivation unpacked tuples and rejected other non-str modes instead.

* feat(callbacks): let a native handler await typed host calls the python caller serves

* test(callbacks-python): drive a native handler's host calls through the python dispatcher

* test(custom_logger_v2): route a native logger with a host through the execution driver

* fix(custom_logger_v2): type the wrapped guardrail modes for pyright

* feat(custom_logger_v2): carry the python type name and hash error text of a stringified value

* feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it

* feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none

* feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs

* feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one

* fix(callbacks-python): render null keys like a python dict merge, adding a missing key last

* feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar

* feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload

* feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end

* feat(custom_logger_v2): carry the start and end times prometheus reads from the call details

* feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call

* fix(callbacks): export the limiter request record

* test(custom_logger_v2): give the type name cases stable ids and sort the record exports

* fix(custom_logger_v2): type the new derivations and tests for pyright

* fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads

* style(callbacks): rustfmt the record exports

* refactor(custom_logger_v2): rename native callbacks to rust callbacks

native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle,
NativeState and HostedNative renamed to their rust forms. The pyo3 classes
NativeCallback and NativeRequestState become RustCallback and
RustRequestState, and the tests, fixtures and messages follow.

* refactor(custom_logger_v2): rename slots to field sources

slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is
SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A
table that only derives is named DERIVED_*_FIELDS, and a table that can
write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The
include_v2 comment moves down to the functions that take it.

* refactor(litellm_logging): place the v2 members of Logging beside what they serve

v2_scope, v2_auth and v2_failure_delivered join the other per-request
class defaults, v2_logging_scope and its opt-out helper follow
should_run_callback, v2_call_facts sits by the standard logging payload
builder, and claim_v2_failure sits above failure_handler.
bind_v2_request_scope checks for a Logging object directly instead of two
runtime protocols.

* refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate

* refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field

A logged payload that is present but not a mapping now looks the same to a
rust callback as a missing one, a parity gap accepted for a case that does
not occur. metadata_labels, input_sequence_length and limiter_request move
down to bits 44-46, on both sides of the contract.

* feat(callbacks): type a rust callback's request state

Callback gains an associated State type: request_state returns it and
handle and handle_hosted receive Option<&State> beside the input, which no
longer carries the state. ErasedCallback is the object-safe form the python
wrapper stores; it carries the state type-erased and downcasts it once,
returning CallbackError::StateType rather than panicking on a mismatch.
handle_hosted's default names its unused host _host. The record sample
count follows the dropped logged_payload_not_a_mapping field.

* docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are

* refactor(callbacks-python): render payloads with python-compat's json writers

pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr.
python-compat's json module now exposes write_string, generic over a String
or byte sink, and float_text, and the payload renderer uses them.

* docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet

* docs(custom_logger_v2): add a README for the package

* fix(custom_logger_v2): import the field sources tests from field_sources

* test(callbacks-python): stub the litellm package in each payload test

* fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check
2026-11-14 02:33:27 +00:00
norkelbyte
5849906ddb
fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079)
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route

/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.

DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.

* perf(proxy): delete count drop paths from one copy of the options

additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear

* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing

Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills

Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400

additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500

The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models

* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's

A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference

A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does

The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any

* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude

A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status

/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none

* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking

A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts

A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it

* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking

A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count

* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle

* fix(bedrock): send the thinking beta headers on Mantle counts

A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends

* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test

* fix(deepseek): count only the tools DeepSeek chat inference sends

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:29:16 +00:00
mika-lt
b77a1295a5
fix(gemini): stream TTS audio in delta.audio on chat completions (#51750)
* fix(gemini): stream TTS audio in delta.audio on chat completions

* fix(logging): drop audio from logged responses when message logging is off

turn_off_message_logging blanked the text of a logged response object
but left message.audio and delta.audio in place, so callbacks reading
response_obj or the assembled stream still got the whole clip. The dict
path already cleared audio. Streamed Gemini TTS now carries audio, which
made this reachable on a second path.

Also pin the TTS stream test to gemini-3.8-flash-tts.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:27:55 +00:00
quietjules
fa79311dfc
feat(model-catalog): read min_billed_input_tokens (#51128)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:16:53 +00:00
quietjules
f5ef14308b
fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved

Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.

* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows

* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs

* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params

* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params

* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase

* fix(bedrock): mark the service tier block field read-only

* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params

* test(prometheus): sort the service tier label test imports

* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400

* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova

* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk

The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one

* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name

The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it

* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added

The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:33:57 +00:00
devin-ai-integration[bot]
03f14051b9
test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 01:05:05 +00:00
Ankit Jha
5f381ad472
fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)
* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:03:19 +00:00
shrey
68d6db45ef
fix(mcp): give the MCP client exactly the servers left on in the connect page (#50973) 2026-11-13 16:36:49 -08:00
Mateo Wang
c4a72b8fac
refactor(mcp): let the semantic filter take its MCP tool loader as a parameter (#50483)
* test(mcp): check the tool dict conversion without patching the MCP handler

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter

* refactor(mcp): type the semantic filter's default tool loader

Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.
2026-11-13 16:34:00 -08:00
ryan-crabbe-berri
0f97850c50
fix(logging): keep a content-block system prompt in logged messages (#51699)
* fix(logging): keep a content-block system prompt in logged messages

A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.

Fixes #36402

* fix(logging): type the logged system message

* fix(logging): type the system message without a mutable annotation, close the test client
2026-11-13 16:23:29 -08:00
joshua-berri
d11917e89c
fix(proxy): preserve caller metadata on auth failures (#51745)
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs

Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.

* fix(proxy): type the spend logs metadata header parser honestly

The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.

* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket

A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.

* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler

Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.

* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer

The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.

* test(proxy): read auth failure spend rows from a real unconnected PrismaClient

Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 16:13:50 -08:00
joshua-berri
2df4f9a78f
fix: preserve routed prompt generation settings (#51702)
* fix: preserve async prompt generation settings

* fix: honor ignored settings on routed prompts

* fix: type prompt option handoff boundaries

* fix: bound prompt hook types without duplicate inference

* fix: preserve legacy prompt initializer parameters

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 16:00:18 -08:00
yucheng-berri
8cf7376df6
test(s3_v2): give the slow-sink ramp test a callback limit that outlasts its burst (#51744) 2026-11-13 23:23:55 +00:00
devin-ai-integration[bot]
d2bae98d97
fix(model-prices): drop Flex prices from Vertex AI Gemini 4 Argon and Hydrogen (#51735)
* fix(model-prices): drop Flex prices from Vertex AI Gemini 4 Argon and Hydrogen

* test(cost-calc): stop comparing Flex prices between Vertex AI and Gemini API Gemini 4 Argon

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
2026-11-13 23:18:03 +00:00
yujonglee
6523ab920a
fix(cost-map): restore backward compatibility after the Rust loader move (#51545)
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.

Also restores behavior the Rust cost map move changed for SDK callers:

- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
  RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
  utils._runtime_registered_model_cost, which downstream test code replaces

litellm_core_utils/AGENTS.md now says which removed names come back.
2026-11-14 08:12:17 +09:00
mrinal-berri
cd6ef91115
fix(helm): route every gateway allowlist path to the gateway in the componentized ingress (#50760)
* fix(helm): route every gateway allowlist path to the gateway in the componentized ingress

The ingress gateway prefix list had drifted from gateway/routes/allowlist.py, so requests for decisions, systemone, extraction, traces, logs, live sessions and several provider passthroughs fell into the backend catch-all, and /eu_jev/v1/systemone returned 404. The missing paths are added, the exact gateway paths move into a list, and a unit test now fails whenever the chart and the allowlist disagree in either direction

* fix(helm): keep extraPaths that repeat a built-in gateway path rendering on upgrade

Values written for an older chart add the newly built-in gateway prefixes through ingress.extraPaths with service gateway, which the duplicate check now rejected, so helm upgrade failed to render. Such an entry is skipped, and a built-in path pointed at another Service or port is still rejected. Also asserts that the new exact path and a new prefix render to the gateway Service

* revert(helm): drop the extraPaths skip for built-in gateway paths

Keeps this PR to the missing paths and the drift test. An extraPaths entry that repeats a now built-in gateway path is rejected at render time as before, and operators remove it on upgrade. The checks that the new paths render to the gateway Service stay

* fix(helm): skip an ingress.extraPaths entry identical to a built-in route

Values that added a path through extraPaths before the chart routed it stopped rendering once the path became built in, since the duplicate check rejected every repeat. A repeat with the same path, pathType, Service and port (by number or the http port name) now renders nothing, and a repeat that routes elsewhere is still rejected

* refactor(helm): limit the ingress fix to gateway-only routes and the drift test

Drops the extraPaths identical-route skip, so the duplicate check is the same as on main. /v1/traces and /v1/logs leave the gateway list and keep going to the backend, since the backend also serves them and their gateway entries date from the retired OTLP ingest. The drift test now requires only paths that the gateway alone keeps

* test(helm): require every gateway exact path the backend does not list exactly

A backend prefix such as /debug/ made /debug/memory/summary count as shared, so the drift test stopped requiring its Exact gateway rule. Exact paths now count as shared only when the backend lists the same exact path

* chore(helm): keep the gateway prefix comment as it is on main

* fix(helm): fit the componentized ingress under the default ALB rule quota

The AWS Load Balancer Controller makes one listener rule per Ingress path and an ALB allows 100 rules by default, so the 113 single-host paths would fail with TooManyRules. Gateway prefixes are now listed in pairs, and on alb each pair is one rule, its first prefix as the Ingress path with a use-annotation forward to the gateway and the second added through a conditions annotation. The default alb render drops to 62 rules for the same path patterns, and nginx, monolith and component-host renders keep one path per prefix

* fix(helm): order grouped ALB rules by their shorter prefix and keep gateway conditions

Each gateway prefix pair now lists its shorter prefix first and uses it as the Ingress path, so a longer extraPaths child of either prefix still sorts ahead of the grouped rule. Conditions an operator set for the gateway Service are carried onto every grouped rule, which uses a generated backend name. The grouped annotations and paths are written inline from the pair list
2026-11-13 15:06:44 -08:00
Mateo Wang
ac3277dce4
fix(dynamic_rate_limiter_v3): publish a team's share gauges for requests served while the model is idle (#51709) 2026-11-13 14:58:54 -08:00
moe-berri
62cf355d9f
fix(spend-logs): make token redaction opt-in (#51729) 2026-11-13 14:41:12 -08:00
Kerry Lu
db75214bdc
fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry (#51694)
* fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry

The reset job now evicts the customer_org_alias entries of the organizations it resets, and a
hierarchy load that began before an eviction can no longer cache what it read. The cache hands
a lease out on a miss and refuses the write when the key was evicted since, on this worker and,
with enable_redis_auth_cache, on any worker through a generation key checked in Redis.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction

delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on
the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for
a cascade with no customer organizations.

* fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too

The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation
write still let the DEL run and an in-flight load could re-cache what it read.

* fix(proxy): delete the auth cache entry before the generation write inside the script

A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL
never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it.

* test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test

* fix(proxy): refuse a leased auth cache write for a key no generation key can guard

* test(proxy): undo the unrelated reformat of the spend ledger parity tests

* test(redis-store): mark the updated memory-only auth cache test with rstest

* chore(proxy): drop two pointer comments from the cache lease code

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 14:21:20 -08:00
moe-berri
0d422a1b6c
fix(auth): let dashboard sessions use the user's team models (#51663)
* fix: inherit team models for dashboard sessions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: authorize dashboard models from owner grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve dashboard model listing filters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): attribute dashboard team model spend to granting teams

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Keep dashboard fallbacks on the original billing team

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Preserve admitted dashboard billing through primary model rewrites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reject dashboard batches that need team model grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Respect wildcard exclusions in dashboard model listings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 17:13:06 -05:00
Nate Armstrong
20b6e855bd
feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) 2026-11-13 22:08:05 +00:00
tin-berri
7b0a37ba74
feat(ui): add overall customer usage and sortable budgets (#51713)
* feat(ui): add overall customer usage and sortable budgets

* test(ui): complete customer usage metric fixtures

* fix(ui): preserve exact customer organization IDs in overview
2026-11-13 22:03:22 +00:00
devin-ai-integration[bot]
219c54ee1d
chore(cost-map): move azure_ai claude-haiku-4-5 and claude-opus-4-5 retirement to 2027-03-16 (#51716)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 22:00:12 +00:00
Mateo Wang
c7747002e5
test(integration): measure dynamic limiter shares after saturation (#51714)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:46:06 +00:00
Kerry Lu
10a9da3116
fix(deps): bump opentelemetry, dnspython and tornado past the Nov 13 advisories so osv-scan passes (#51712)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:42:44 +00:00
kenneth-berri
31067d0f9f
fix(bedrock): log and bill Titan embedding replies on the invoke passthrough (#51701)
* fix(bedrock): log Titan embedding replies on the invoke passthrough as embeddings

A native /bedrock/model/<titan-embed>/invoke call returned its vector, but success logging read the reply as a chat completion and crashed, so the call was never priced or written to spend logs. Fixes #51455

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): log binary Titan embeddings and bill input images on /invoke passthrough

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): charge text tokens alongside the image on Titan multimodal /invoke passthrough

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:41:53 +00:00
devin-ai-integration[bot]
f4a17048c2
chore(cost-map): add the 4K output rate to runwayml/seedance2_5 (#51697)
* chore(cost-map): add the 4K output rate to runwayml/seedance2_5

* test(cost-map): cap per-second, per-image and per-request prices at 1000 and drop the seedance exceptions

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:40:54 +00:00
ishaan-j-berri
8e9db8ab67
fix(docker): refresh Wolfi base to patched glibc (#51550) 2026-11-13 13:40:36 -08:00
joshua-berri
88873a70d4
fix(proxy): retain unwritten rollups after database disconnects (#51706)
* fix(proxy): retain unwritten rollups after database disconnects

* fix(proxy): initialize auto-router turns per client

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 13:36:12 -08:00
devin-ai-integration[bot]
49ea54ee7a
test(streaming): make unbuffered Noma secret exposure deterministic (#51707)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:25:53 +00:00