* fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed
* fix(proxy): gate the zero-cost 499 row on the success log being enqueued, not merely claimed
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ffb7133d32)
* fix(proxy): scope the search free-tier counter to the request's provider
* fix(proxy): read the provider with get so partial spend payloads still get the free tier
* test(proxy): give the grounded spend payload fixture its provider
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 640f2d865b)
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata
* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 2a268951c4)
* fix(proxy): bill partial Responses API streams when the client disconnects
Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.
* fix(proxy): satisfy lint gates for partial Responses API disconnect billing
Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.
* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect
The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.
* Potential fix for pull request finding 'CodeQL / Log Injection'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
(cherry picked from commit f0ae445cc4)
Every /v1/responses spend log asked its worker to flush the queue right away so a
chained previous_response_id could find the row. Only the chat-completions bridge
reads that row. Native Responses providers hold the conversation upstream, so their
rows now wait for the batch write like chat rows do
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 423362bab2)
* feat(proxy): opt-in free-tier allowance for Google Search grounding spend
* fix(proxy): declare the search free tier counter's service target
* test(cost_map): allow search_free_requests in the cost map schema
* fix(proxy): skip cache hits and never free more than the billed cost in the search free tier
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit e63aec029a)
_failure_spend_rows yields the final row plus one re-keyed row per lifted billed_failed_attempts entry (<request_id>-attempt-<position>, the attempt's usage, cost, retry count and deployment id), each written through DBSpendUpdateWriter.update_database so it also deducts from key, team and user spend
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ad8ca533ca)
* feat(proxy): lift every earlier billed timed-out retry attempt onto the failed request
_billed_earlier_attempts reads the router's previous_models records, keeps the Timeout ones other than the final attempt (matched on deployment_id plus attempted_retries, since the count restarts after a fallback hop and the fallback handler re-logs the exhausting attempt) and prices each at the deployment that served it. post_call_failure_hook lifts the tuple as billed_failed_attempts; nothing writes it yet. previous_models joins the router-reserved metadata fields so a client-planted record never reaches it
* fix(proxy): price earlier timed-out attempts from the prompt estimate only
The final row may lift a mid-stream recovered usage (prompt plus delivered chunks) or a failed_attempts_usage sum; earlier attempts never produced output, so they are priced from _estimated_prompt_usage, and when failed_attempts_usage already bills them into the final row none is lifted again
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 7a4a255d1d)
* fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row
* fix(proxy): read the provider handoff before the proxy-only error handler stamps it so a pre-handoff 499 is not priced
* fix(proxy): drop the pre-logging handoff read now that pre_call no longer stamps proxy-only failures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 0e7c4908bb)
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
(cherry picked from commit 62ce515d15)
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit e979bd0ecf)
Applies #45874's diff at 1091b9054c onto rc/1.107.0. The only conflict was BedrockPassthroughConfig.sign_request,
where rc/1.107.0 has no bedrock_request_metadata_fields handling yet, so the Converse body conversion keeps
#45874's pre-merge form
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(ui): take the dashboard's user role and identity from the server
Adds an internal GET /management/v1/users/me that returns the user id, role
and email the proxy resolved for the calling key. The dashboard now reads
identity from it instead of the session cookie's claims, so its navigation and
role gating always match what the backend authorizes. The layout waits on it
behind the existing loading screen, and a failed lookup clears the session and
returns to login. Sessions that must change their password can also read it
* fix(ui): sign out when the current user lookup fails and wait for it on chat
The dashboard layout now holds the loading screen and signs out when
/management/v1/users/me fails, instead of mounting the shell and looping back
to loading as each consumer refetched the failed query. The chat layout also
waits for the identity so its panels never query without a user id. The
lookup goes through the typed client with the regenerated schema.d.ts
* fix(ui): send the session's auth header on the first user lookup and keep chat closed until authorized
The first lookup runs before AuthProvider registers the custom header name, so it now reads auth_header_name from the session cookie. Chat also waits for isAuthorized, so a failed lookup no longer mounts the shell with an empty user id
* fix(ui): type the dashboard callers of the now typed user id
useAuthorized used to hand back the cookie's user id untyped, which hid six strict type errors in its callers. The id is now string | null, so the production build failed. Callers handle the null case, and query key filters accept the boolean blocked filter
* fix(ui): look up the current user only after the proxy URL is discovered
Pages outside the dashboard shell, like chat and connect, could start the lookup before UI config discovery set the proxy base URL. On a split-origin deployment both attempts then hit the UI origin and a valid session was signed out
* fix(ui): keep a loaded identity while UI config discovery reruns
Pausing the lookup by passing a null token switched the query key, so on the dashboard the identity the layout had already loaded vanished until the shell's own UI config request finished. The key now stays the same and only the fetch waits for discovery
(cherry picked from commit 4bafebbcdc)
* fix(proxy)!: count prompt-only token_counter requests with the provider when call_endpoint is set (#46839)
* fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set
* fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails
* fix(proxy): log a prompt-only token count fallback as a one-line warning
* fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI
* fix(proxy): count a token_counter prompt sent next to an empty messages or contents list
* fix(proxy): count a prompt next to messages on the local token count instead of answering 500
* fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set
With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form.
* test(proxy): count tools with a prompt in the token counter fallback test
* test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests
* test(proxy): name the token counter fallback test by what it checks
(cherry picked from commit 2dc6aafa6b)
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count (#46843)
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count
/v1/messages runs clear_tool_uses context edits before it sends an Anthropic
body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted
the body unedited, so a request with older tool results cleared counted about
2.8x what inference bills.
The count now runs the same history and context-edit step inference runs,
honors additional_drop_params on the deployment, and skips compact edits so a
count never makes a paid summary call.
* fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs
Run compact edits in the count without the summary call so trigger checks and
history slicing match /v1/messages, honor a request-level additional_drop_params,
keep the full body when any deployment in the model group drops context_management,
and answer 400 naming a block Gemini cannot count when an edit trigger fails on it
* fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check
* fix(count_tokens): log a failed edit in the count at debug, since the send already logs it
* fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does
* fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check
The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits.
* fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name
* fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to
* fix(messages): drop an invalid context_management spec under drop_params so the send matches the count
* fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them
The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes.
* test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code
(cherry picked from commit f8217d2a03)
* fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583)
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing
A non-list tools value crashed the vector store lookup during auth and the
built-in tool cost lookup in the failure logging path, so the proxy answered
an unhandled 500 and closed the keep-alive connection. count_tokens now
answers a 400 invalid_request_error for a malformed body, drops malformed
tools when drop_params is on, and both lookups skip values of the wrong shape.
* fix(proxy): skip non-list tools in the tool allowlist check
Also validate the count_tokens body without recursion, which the recursion detector rejects.
* test(integration): restore the int tools case on count_tokens
The proxy now answers a non-list tools value with a 400, so the case is deterministic again.
* fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids
* fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids
Check tools after request setup so the Claude Code default and the logged call type apply, read the request,
global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini
generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off
* fix(proxy): keep the request's call type when logging a gate rejection after request setup
The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses
rejection as aembedding, so it now only applies when the logging object carries no known call type
* fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry
* fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket
* refactor(proxy): leave the responses websocket out of the request-level tool check
* fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections
* fix(proxy): read router default drop lists and deployment ids in the malformed tools check
* fix(proxy): leave malformed tools to the router when it can pick another deployment
Fallbacks, auto-routers, model strings, comma batches and team model names
let the router pick a deployment after the proxy checks the request, so the
proxy now forwards a malformed tools field there as main does, and only
rejects it when the deployment it checks is the one that serves. Token
counting never reroutes, so count_tokens and input_tokens keep reading the
group they count with. additional_drop_params names are read the way core
reads them, so a list with a non-string entry still names its strings.
* test(integration): cover malformed tools on reroutable models, router defaults and deployment ids
* fix(proxy): resolve malformed tools at the deployment the router picks
* fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools
Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells.
* test(integration): expect a 400 for an empty-string tools field on /v1/messages
Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test
* fix(router): keep web search routing for tools passed as a tuple
* fix(router): keep a deployment's own tools when drop_params drops a malformed request tools
* test(router): list _merged_tools as covered through the deployment tools merge tests
* fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them
* fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError
* fix(prometheus): keep deployment health metrics clean on a malformed tools rejection
* test: type the new malformed tools tests
* test: mark new test locals Final and type kwargs dicts
(cherry picked from commit d76ec3a47e)
* ci(cost map): classify token_count_model as an object key in the schema generator (#47057)
* ci(cost map): classify token_count_model as an object key in the schema generator
* ci(cost map): require non-blank token_count_model strings and test the region rules
(cherry picked from commit 6e191a58f1)
* ci(cost map): remove the generated model prices JSON schema (#47064)
* ci(cost map): remove the generated model prices JSON schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost map): keep schema-independent cost map tests after dropping the schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(cost map): keep the schema file until main's guard stops requiring it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(cost map): let bot sync PRs keep touching the schema file until it is deleted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ae6f7f6613)
* fix(count_tokens)!: answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens (#47163)
* fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens
* fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests
Vertex has no count-tokens route for Mistral or codestral models, so they now
get the same 400 as other partner models without a count API instead of a 404
that points operators at IAM.
* fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens
* fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model
* fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors
* fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer
* fix(count_tokens): pass through an OpenAI provider error that omits param or code
* fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route
* test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy
* test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes
(cherry picked from commit ef1d3959a0)
* feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777)
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response
Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.
* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests
* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing
* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown
Moves the cost map lookup off the event loop
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing
A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.
* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates
* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment
A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.
* test(count_tokens): use the shared integration port allocator in the fallback wire test
* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today
* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings
Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it
* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body
A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.
* fix(count_tokens): answer the provider's error whenever a framed estimate raises
A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.
* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate
A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.
* fix(count_tokens): look up framing under the deployment's custom_llm_provider
* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error
A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.
* fix(count_tokens): keep one tokenizer download per key running and load the local route per request
* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible
* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error
Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500
* feat(proxy): count tokens with token_count_model in its pinned region
token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable
* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys
* fix(proxy): keep the upstream error envelope when adding the region hint
* test(proxy): annotate the token count setting echo test
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(count_tokens): answer the token_count_model's error even when framing is set
* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate
* fix(count_tokens): estimate Google contents the route translated to messages
* fix(count_tokens): keep the provider error for Google contents with non-text parts
* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock
* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags
* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer
* test(bedrock): keep the Kimi K3 image check in-process
* fix(count_tokens): join the region hint to an upstream message that already ends with a period
Also pins that a blank token_count_model in config falls through to the cost map default
* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts
The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts
The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool
A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning
* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them
A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.
* test(count_tokens): annotate new token counter test locals with Final
* fix(count_tokens): keep shipped count regions inside the deployment's geography
A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added
* fix(count_tokens): resolve count settings off the event loop
(cherry picked from commit 75faaeab64)
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307)
* test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515)
* fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515)
A post_call guardrail that scans through apply_guardrail now runs on
/bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the
caller as the upstream sends them, every fifth frame and the end of the stream
are scanned, and a block after the first frame ends the stream with an AWS
exception frame. streaming_buffer_until_moderated still holds the stream and
applies masks per content block. Guardrails that need the whole response
(own async_post_call_success_hook, mask_response_content, native hooks, policy
pipelines) keep the buffered path.
* fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515)
* fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515)
* fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515)
- The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the
upstream ends with messageStop, an exception or error frame, or a dropped connection
- A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the
failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no
failure row
- A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does
- Closing the response before its body starts closes the guardrail layers and the upstream
- Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never
reaches frames a later layer still holds
- Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser
* fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515)
* test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515)
* fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515)
Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client
disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them
from the request the way chat completions streams do
* fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515)
* fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515)
* fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515)
A held post_call guardrail with streaming_buffer_release_on_scan releases each
window as received once its scan passes, so the relay delivered the original
text instead of the guardrail's masks. Such a stream keeps the buffered path,
which applies masks per content block as on main.
* fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515)
The relay ran every guardrail in config order, so a run_in_parallel guardrail
listed before a masking guardrail scanned the unmasked text. Main's buffered
path runs parallel guardrails after the sequential ones and discards their
rewrites, which the relay cannot reproduce, so these streams stay buffered
* fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves
- Noma v2 scans on the converse-stream relay carry stream, and the last one
carries the finished Converse response with its stopReason, as chat sends
them, so Noma keeps one record per reply. Other guardrails get the request
unchanged. When the last sampled scan already covered the whole answer, the
end-of-stream scan still runs for Noma so it gets the finished answer
- A held relay stream whose client leaves before the first frame stops
reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet,
the budget reservation and the parallel-request slot are released
- A cancelled wait for the first streamed chunk now stops the upstream read
instead of leaving it running
- A Noma v2 scan cancelled before Noma answers records not_run instead of
success with an empty response
* fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks
* fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input
* test(bedrock): expect the ValidationException main now returns for a held block before the first frame
Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400.
* fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message
A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw
event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the
messageStop, skipped its write-back and released the held text without its masks. The Converse handler now
reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text
* fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame
The relay skipped releasing the slot and budget reservation whenever a spend
flush was scheduled, but a flush of a partial first frame builds no response
and logs nothing, so the slot stayed held until it expired. The flush now says
whether it logged the stream.
---------
Backport of #46307 to rc/1.107.0
Cherry-picked from ef1cb19351 (main)
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(bedrock)!: scan Claude invoke-with-response-stream passthrough output with post_call guardrails (#46524)
* fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618)
Invoke streams of Claude models now go through the same frame-by-frame relay as
converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages
stream translation, a block ends the stream with an AWS exception frame, and a held
stream gets its masks written back into re-encoded chunk frames. Claude is told by
its model id or, behind an application inference profile, by the anthropic_version
its request body carries. Streams of other model families stay as they were
* test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618)
* fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618)
A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while
converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered
path, with the rewrites written back into the chunk frames that carried each text.
Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does,
so a vendor can tie the scans of one streamed reply together.
* refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618)
* fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618)
An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned
as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned.
The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream.
* test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618)
* test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618)
* test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests
* test(bedrock): type the remaining parameters of the Claude invoke stream tests
* refactor(bedrock): bind the buffered invoke rewrite in its condition
---------
Backport of #46524 to rc/1.107.0
Cherry-picked from 9f115be88d (main)
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(vector-stores): keep proxy context out of provider vector store metadata
The proxy merged its request context (user_api_key_* fields, endpoint,
user_agent, requester_ip_address, litellm_api_version) and the router's
deployment info into the vector store request's metadata. For vector
stores that field is also OpenAI's own metadata, which is capped at 16
keys, so a store created with a team key was full on creation and every
later metadata update returned 400.
Vector store routes now keep that context in litellm_metadata, like
batches, files and responses, and the router does the same for
avector_store_create. Provider-side metadata holds only what the caller
sent.
Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in
script that removes the proxy-written keys from stores created before
this change. It is a dry run unless --apply is passed.
* fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values
Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values.
* fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata
Vector store routes now keep proxy context in litellm_metadata, but the v3
limiter only read agent_id from metadata, and logging let provider metadata
tags or spend_logs_metadata replace the proxy's values. A string tags value
also crashed get_request_tags.
The limiter now prefers the proxy-resolved agent_id in litellm_metadata.
Logging combines list tags from both buckets (metadata first, deduplicated)
and merges dict spend_logs_metadata with litellm_metadata winning, ignoring
values of the wrong type.
* fix(vector-stores): narrow the logging fallback to mistyped caller values
The tag union and spend_logs_metadata merge changed logged tags and
spend_logs_metadata on every route that has both metadata buckets, deduped
tags quadratically, and stopped background interaction polling from keeping
string tags as sent. Caller values now win exactly as before, and the
proxy's litellm_metadata tags or spend_logs_metadata only replace a caller
value of the wrong type.
The limiter reads agent ids through a pydantic model instead of a cast and
ignores non-string ids.
* fix(vector-stores): log the proxy's full tag list on litellm_metadata routes
On routes that keep proxy context in litellm_metadata, the proxy builds
litellm_metadata.tags from the caller's tags plus key, team and header
tags, but get_request_tags preferred the caller's metadata.tags. A caller
sending metadata.tags on a vector store call dropped the key's tags from
request_tags, which tag spend tracking reads. get_request_tags now prefers
a non-empty litellm_metadata tag list, and falls back to metadata.tags as
before.
* fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets
Moving vector store proxy context into litellm_metadata broke shared readers
(agent limits, request tags, merged logging metadata), and fixing them
widened the change to every litellm_metadata route. This reverts the bucket
move, the router change and the reader fixes, so proxy context stays in
metadata exactly as on main.
Vector store create and update now drop the keys LiteLLM itself writes
(user_api_key*, requester and deployment keys, agent ids) from the metadata
sent to the provider, in the shared optional-param helper both paths use.
The cleanup script reuses the same key list.
* fix(vector-stores): pass non-string metadata keys through the provider metadata filter
* fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls
* fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy
* fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies
* fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed
(cherry picked from commit 4846cfabc7)
* feat(ui): backport the OAuth Delegate endpoint and scope fields to rc/1.107.0 (#46941)
Backport of #46941 to rc/1.107.0. Cherry-picked from 3637dd711f
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp)!: backport the gateway-owned DCR client for bridge servers to rc/1.107.0 (#46940)
Backport of #46940 to rc/1.107.0. Cherry-picked from 7049a5ff4d
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47094 to rc/1.107.0.
Cherry-picked from 8b1859c50d (main).
(cherry picked from commit 8b1859c50d)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): accept an ID-JAG as an RFC 7523 jwt-bearer grant on the gateway token endpoint (#46748)
Backport of #46748 to rc/1.107.0.
Cherry-picked from bca7650e8f (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sso): capture the IdP id_token on Microsoft and Google logins for oauth2_id_jag (#46747)
Backport of #46747 to rc/1.107.0.
Cherry-picked from 3501e36fa9 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sso): retain the SAML assertion at login as the saml2 subject token for oauth2_id_jag (#46745)
Backport of #46745 to rc/1.107.0.
Cherry-picked from f02449ad38 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the OIDC login refresh token as the ID-JAG subject when the SSO client is the ID-JAG client (#47004)
Backport of #47004 to rc/1.107.0.
Cherry-picked from fc484be77a (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): give /team/delete transactions a 30s timeout (#46979)
Both interactive transactions in /team/delete used Prisma's 5s default.
A lock wait or a short database stall past 5s expired the transaction
after the keys were already deleted, so the call returned 503 and left
the team row, memberships and team models behind
(cherry picked from commit 6530b329f4)
* fix(sso): concurrent first sign-ins land on the winner's account, worker-kill test stops waiting on the reap (#46855)
* fix(sso): concurrent first sign-ins land on the account the winner created, worker-kill test pins half its streams to each worker
* test(integration): a worker kill returns once the worker is dead, the reap waits at block exit
uvicorn can take a full worker healthcheck to reap a worker that died mid-ping, so kill() held
the caller that long and peers held on the surviving worker hit their client timeout
* fix(sso): a sign-in that loses the first-insert race joins the default teams
The losing sign-in could read the winner's row before new_user added the default teams, so its
CLI key was minted with no team. The race tests now run the real new_user against a fake user table
(cherry picked from commit 29cd11bec6)
* test(e2e): fix MCP tool search hit matching and the retired Bedrock image model (#46998)
* test(e2e): compare parsed MCP tool search hit names instead of raw text
* test(e2e): move Bedrock image generation off the retired Nova Canvas model
(cherry picked from commit e9a8581a02)
* test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001)
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets
* test(integration): wait for the team spend flush before killing a worker
* test(integration): poll for the S3 attempt the batch sinks write concurrently
* test(integration): queue the terminal object and its sibling into the same s3_v2 flush
* test(integration): time the long Bedrock version against an equally long unversioned name
* test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it
* test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries
(cherry picked from commit 910c30381b)
* test(ui): wait for the logout navigation before checking the session is gone (#47008)
(cherry picked from commit 53ef63145b)
* fix(ui): preserve key settings drafts during background refreshes (#41350)
* fix(ui): preserve key settings drafts during background refreshes
* fix(ui): refresh untouched key settings while preserving edits
* fix(ui): keep team-change MCP clears across key refreshes
clearMcpGrants now marks the MCP fields dirty, so a background refresh with
keepDirtyValues no longer restores the grants a team change just cleared.
Also adapts the draft tests to the onSubmit(values, dirtyFields) signature
on main and asserts a refresh does not mark untouched fields dirty, since the
team admin member-key path rejects saves with non-budget dirty fields.
KeyEditView becomes a one-line arrow to stay under the max-lines limit.
* fix(ui): keep MCP tool defaults out of a key's edited fields
MCPToolPermissions writes non-delete tool defaults for unrestricted servers
on mount. Marking that write dirty made a team admin's budget-only save on a
member key get rejected for editing mcp_tool_permissions. onChange now says
whether a write is a default or a user edit, and the key editor dirties only
edits.
The edit session is now keyed by token or token_id, matching the form's own
fallback, so switching between keys that only carry token_id starts fresh.
* fix(ui): keep a new MCP server's default tool limits across refreshes
Default tool writes for a server the user just added in this edit session
are part of that edit, so they are marked dirty and survive a background
refresh. Defaults written for an already saved server stay clean, so a team
admin's budget-only save on a member key is still accepted.
(cherry picked from commit 37b3a697c7)
* fix(logging): stop the spend metadata merge from walking a dict the logging thread writes to (#46044)
(cherry picked from commit aa32e4cc88)
* test(integration): expect a pre-token invalid prompt on a bridged stream to fall back (#47020)
#46493 reads the first stream chunk inside the router's retry and fallback
chain, so an invalid prompt rejected before any token now falls back to the
configured fallback deployment instead of answering 400. The author
confirmed this is intended
The fallback test now asserts the spare serves the stream. The old 400
without a fallback attempt still holds when fallback_on_unsupported_params
is false, so that contract moves to a second proxy with that setting
(cherry picked from commit 856a3f62f3)
* test(e2e): repair fallback management test for config-owned fallbacks (#47023)
* test(e2e): repair fallback round trip for config-owned fallbacks
The e2e stack config sets router_settings.fallbacks, and since #45860 a
/fallback write to a key the config file owns is refused with a 400. The
round trip now runs on context_window fallbacks, which the stack config
leaves to the database, and also deletes the rule and checks it is gone.
A second test locks the new contract: a /fallback write to the
config-owned fallbacks key returns 400 naming router_settings.fallbacks,
and the config rule keeps being served unchanged.
* test(e2e): assert the refused fallback rule is never served
(cherry picked from commit 59fa785de0)
* test(e2e): assert the Vertex cache-creation billing contract in the prompt caching test (#47027)
The cache-creating call now reports cache_creation_input_tokens on top of
Vertex's own prompt count and bills the write at the input rate, as Google
does. The old assertions assumed neither, so the test went red on every run.
Check the cache write, the discounted read, and that a second identical call
reuses the cache instead.
(cherry picked from commit 81a3802a92)
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): stop proxy-started legacy listeners from consuming server rpm
The listener the proxy opens on notifications/initialized for legacy MCP
sessions ran the server rate-limit admission once per target, so every
session spent an rpm slot before the client sent anything. Admission is
now injected into the listener and applied only to client-initiated
subscriptions (subscriptions/listen and resources/subscribe). Authorization
and its periodic recheck still run on every path.
* test(mcp): key the fake admission window by caller and server
* fix(mcp): stop subscription catalog refreshes from consuming server rpm
A subscription listener lists the caller's catalog to diff list_changed
events, at setup and on every upstream event. Those listings went through
the normal per-server admission, so against an upstream that declares
listChanged each legacy session spent several rpm slots before the client
asked for anything. The listener now marks its targets as already admitted
for its own lifetime, so its refreshes reuse the subscription's admission.
Client requests on the session are still metered.
* test(mcp): exercise subscription admission through the proxy rate limiter
(cherry picked from commit ac6300f380)
* fix(proxy): track and drain management audit log writes on shutdown
* fix(proxy): track hook spawns and defer registration in audit log drain
* fix: satisfy LIT006 on audit log task registry by documenting the required cast
* fix(proxy): satisfy LIT006 and LIT010 in audit task registry
* test(proxy): drive the audit drain tests with events instead of wall-clock waits
* fix(proxy): drain audit writes beside the shutdown flushes and track only premium writes
Run the audit drain concurrently with the gateway and request-error flushes so
the two waits overlap inside the teardown budget, register the running audit
task only after the premium check, drop the four duplicate task-failure
loggers, and cover the shutdown order and a real hook spawn with tests
* fix(proxy): start hook audit writes before their email and secret manager calls
Key and user hooks now spawn their tracked audit write first, so a slow
invite email or secret manager call no longer spends the drain budget
before the write starts. The endpoints stop tracking the hook tasks, so
shutdown never waits on hook I/O when audit logging is off.
Adds regression tests for the hook order, the failed block or unblock
update, and the shutdown drain without a database.
* fix(proxy): track only the audit insert, not the code that awaits it
The shutdown drain now waits on each audit row insert itself, so it never waits on a caller's later work and covers every caller without per-endpoint wrappers. The user/new hook still tracks its own task because it reads the user row before the insert
* test(proxy): drain the audit registry in the config audit tests
The audit insert now runs in its own tracked task, one hop after the caller's task, so a single yield no longer reaches it
* fix(proxy): name dropped audit rows in the drain timeout warning
Each tracked audit task is named after its action, table, object id and destination, so a shutdown that drops a row says which one. Adds tests that the drain waits for slow audit callbacks and that the warning names the dropped row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test
* fix(guardrails): record every content filter match in logging_only and offer the mode in the UI
* style(tests): format the content filter logging_only integration test
* refactor(guardrails): keep the logging_only predicate and scan override within the type gates
* fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning
* test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test
* test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects
* refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach
* test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter
* test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up
* test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
* fix(proxy): count endpoints return missing-key and request errors in their source API shape
* fix(proxy): drop imports the count error change left unused
* fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope
* fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures
* fix(proxy): count with the global OpenAI and Anthropic keys chat already uses
* fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway
* fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response
* fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads
* fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses
* fix(proxy): mint the OpenAI workload identity bearer inside the count error handling
* fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(bedrock-passthrough): return AWS-shaped errors so AWS SDKs read the code and message
A guardrail block on a /bedrock route now answers with x-amzn-errortype ValidationException and a top-level message, and an error Bedrock returned (such as a 429 ThrottlingException) is passed through with its status, x-amzn-errortype and body
* fix(bedrock-passthrough): keep 5xx guardrail failures and failure-hook replacements unchanged
* test(bedrock-passthrough): guardrail vendor failures and unknown models keep the litellm error
* fix(bedrock-passthrough): only answer guardrail exceptions that blocked content as ValidationException
* fix(bedrock-passthrough): pass Bedrock's content-type through with its error body
* refactor(bedrock-passthrough): wrap the guardrail ProxyException instead of copying its fields
Unit tests drive the error response with the real request processor and ProxyLogging
* test(bedrock-passthrough): cover the direct model route, invoke, and auth, budget and rate-limit errors
* fix(bedrock-passthrough): pass through only errors a Bedrock model operation returned
A guardrail service's HTTP error that carries x-amzn-errortype keeps the litellm error
* refactor(bedrock-passthrough): mark Bedrock's errors where the passthrough raises them instead of matching the URL
A guardrail service's HTTP error never goes through the passthrough call, so it keeps the litellm error whatever its URL
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
The shared disconnect release dispatched to the legacy limiter, which decrements unconditionally, so a rejected realtime attempt freed a slot it never took. The generic release is v3-only again and the Responses WebSocket close releases through its own method
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(proxy): reset end-user spend in committed pages so the budget cascade survives large LiteLLM_EndUserTable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): collapse _enduser_ids_by_tier return to one line for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the end-user page IN list as bounded for the unbounded-IN check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn
* test(responses): mock the websocket slot release and mark the per-turn start time naive
* style(responses): keep the per-turn start time line within ruff format
* refactor(responses): type websocket turn events as read-only mappings
* fix(responses): settle the websocket budget reservation on the first turn only
Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly
* fix(proxy): keep the legacy limiter's websocket slot until disconnect
A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache
SAML login tickets, the single-use assertion mark and cached IdP metadata
lived in user_api_key_cache, which only gets Redis when
enable_redis_auth_cache is on. With several replicas or workers the IdP
callback often landed on a process that never saw the ticket and got a
401, and IdP-initiated sign-in was refused even with Redis configured.
SAML state now has its own DualCache that _attach_redis_usage_cache
always wires to Redis, like the CLI SSO session cache, and the SAML cache
calls are async.
* test(sso): drop docstring from the SAML proxy pair fixture
* test(sso): cover the SAML login and metadata endpoints
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* test(count_tokens): expect the converted Responses input in the budgeted-key count integration test
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(guardrails): honor streaming settings on the Noma v2 guardrail
The Noma v2 initializer now passes streaming_buffer_until_moderated,
streaming_end_of_stream_only, streaming_sampling_rate and
streaming_buffer_release_on_scan from the guardrail config to
NomaV2Guardrail, so the unified streaming hook applies them. The
constructor validates them, so a custom guardrail class built on
NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults,
and an invalid value is ignored with a warning naming the guardrail, so
the guardrail still loads.
Resolves LIT-9633
* test(guardrails): cover Noma v2 streaming settings across clients, failures and workers
Adds integration cells for the custom-subclass loader, string and invalid
values, unset and false keys, a Noma outage with and without block_failures,
an upstream rejection, n=2 choices, pre_call and non-streaming guardrails,
the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a
PATCH while traffic flows, a burst while Noma is down, a slow Noma, and
replaced workers.
Resolves LIT-9633
* test(guardrails): pin the PATCH probes to one worker at a time
Each probe suspends the other worker and opens a fresh connection, so the
test checks that every worker applies the patched setting.
Resolves LIT-9633
* test(guardrails): check sampled scans by count, not window position
/v1/messages hands the streaming hook raw network reads, so where the
default sampling windows fall depends on timing; the base leg caught one
shifted window. Check the scan count and the final full-text scan instead.
Resolves LIT-9633
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): carry a 429 status on cooldown and no-deployment errors
RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the
proxy answered an all-deployments-cooled-down request as a generic 500 class
error and logged the bug report notice for every cooldown. Both now carry
status_code 429, which the proxy maps to a rate_limit_error and which the
retry policy already treats as retryable
* fix(proxy): start Sentry at config load and never attach request bodies
Sentry only came up inside set_callbacks on the first LLM call, so boot and
background errors were never reported, and general_settings.alerting: [sentry]
always raised Missing SENTRY_DSN because the alerting path read a dead
litellm.utils pin instead of the live SDK. The proxy now initializes Sentry
once while loading the config whenever sentry is a configured callback or
alerting target, the alerting and failure handlers read the live instance, and
max_request_body_size is pinned to never so prompts and completions stay out
of Sentry events
* fix(tracing): log Lens failures at ERROR and export row counters to Prometheus
A tracing store that fails to start, or a Lens service that rejects a batch,
was only logged at WARNING, so operators running at the default ERROR level
never saw tracing silently go dark. Both now log at ERROR and name the lens
store. The exporter's rows_written and rows_dropped tallies were never
exported, so the two counts are now Prometheus counters
litellm_lens_export_rows_written_total and
litellm_lens_export_rows_dropped_total, registered lazily on the default
registry like the other proxy-level counters and charted on the all-metrics
Grafana dashboard
* test(proxy): drop the litellm import the alerting tests no longer use
* fix(proxy): start Sentry once a DSN is set, from any config source
Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it
* fix(sentry): keep stack frame locals out of events unless PII is opted in
Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII
* test(router): drop the test that only pinned the 429 class attribute
* fix(proxy): start Sentry at boot when it is only an input callback
* fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
