response.output_item.done for a reasoning item only carried the summary
text. The signed and redacted thinking blocks were encoded into
encrypted_content on response.completed alone, so a client that persists
the streamed items replays the next turn without the signed thinking.
Encode the accumulated thinking_blocks with the shared helper when the
done event is built, so the item carries the same encrypted_content as
the completed response. Nothing changes when no snapshot exists yet.
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream
A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out.
* fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE]
A guardrail block before the first chunk is raised again, so its response stays the
400 event-stream body it is on main. A mid-stream error event now carries the failure
hook's replacement error when one is returned, is followed by data: [DONE] on
OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details
JSON cannot encode are written as strings instead of breaking the event.
* fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400
* fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band
* fix(proxy): encode non-JSON guardrail details in early stream error frames
An early guardrail block whose detail holds a value JSON cannot encode sent
400 headers and then cut the body. The error frame now writes such a value as
its string form, the same as the mid-stream event. The keepalive test now runs
on a virtual clock instead of real sleeps.
* fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds
ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload.
* fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground
The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client
* fix(proxy): print CLI stream errors without Rich highlighting or emoji codes
* test(proxy): cover the CLI's plain-text 400 error print
* test(proxy): annotate the new client stream test locals as Final
* fix(proxy): send the failure hook replacement on Responses streams and before the first chunk
* fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception
* fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it
A non-HTTP failure whose hook returns an HTTP error was raised before the first
chunk, which create_response turned into the hook's status but which a bare
StreamingResponse route such as /queue/chat/completions answered as a 200 with
zero bytes. Only the original HTTP errors raise now. The replacement goes out as
the stream's error event, which create_response still answers with the hook's
status as a JSON error.
Adds audit round 2's integration cells for failure hook replacements on chat and
Responses streams, built-in exceptions on Responses streams, and the Python client.
* test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk
The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent
Follow-up to #40152. Greptile flagged the two new reasoning_effort tests for
assigning config and result without Final. Annotates every local in the file
so it passes the LIT010 check cleanly
* feat(ui): show gateway URL and agent setup instructions after creating a key
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): make the selected agent button visibly distinct in dark mode
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): hide agent instructions by default and use a generic title
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): use one shared agent instructions card with Claude Code and Codex logos
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(baseten): forward reasoning_effort to Baseten Model API
Baseten's Model API accepts a top-level reasoning_effort parameter and
validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning),
but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM
silently dropped it when drop_params=True and raised UnsupportedParamsError
otherwise.
Add reasoning_effort to the supported params so map_openai_params forwards
it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten
returns a clear 400 for values outside a model's supported set and ignores
the parameter on non-reasoning models, so verbatim passthrough is safe for
the whole catalog.
[Claude_Code]
* fix(baseten): forward reasoning_effort to Baseten Model API
Baseten's Model API accepts a top-level reasoning_effort parameter and
validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning),
but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM
silently dropped it when drop_params=True and raised UnsupportedParamsError
otherwise.
Add reasoning_effort to the supported params so map_openai_params forwards
it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten
returns a clear 400 for values outside a model's supported set and ignores
the parameter on non-reasoning models, so verbatim passthrough is safe for
the whole catalog.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response
Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.
* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests
* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing
* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown
Moves the cost map lookup off the event loop
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing
A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.
* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates
* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment
A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.
* test(count_tokens): use the shared integration port allocator in the fallback wire test
* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today
* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings
Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it
* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body
A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.
* fix(count_tokens): answer the provider's error whenever a framed estimate raises
A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.
* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate
A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.
* fix(count_tokens): look up framing under the deployment's custom_llm_provider
* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error
A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.
* fix(count_tokens): keep one tokenizer download per key running and load the local route per request
* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible
* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error
Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500
* feat(proxy): count tokens with token_count_model in its pinned region
token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable
* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys
* fix(proxy): keep the upstream error envelope when adding the region hint
* test(proxy): annotate the token count setting echo test
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(count_tokens): answer the token_count_model's error even when framing is set
* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate
* fix(count_tokens): estimate Google contents the route translated to messages
* fix(count_tokens): keep the provider error for Google contents with non-text parts
* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock
* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags
* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer
* test(bedrock): keep the Kimi K3 image check in-process
* fix(count_tokens): join the region hint to an upstream message that already ends with a period
Also pins that a blank token_count_model in config falls through to the cost map default
* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts
The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts
The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool
A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning
* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them
A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.
* test(count_tokens): annotate new token counter test locals with Final
* fix(count_tokens): keep shipped count regions inside the deployment's geography
A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added
* fix(count_tokens): resolve count settings off the event loop
* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
* fix(rag): use team provider credentials and allow viewer and JWT team access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rag): assert persisted managed store data excludes team credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(jwt): run real team model checks in the route denial test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rag): scope team provider credentials to openai vector stores
Bedrock and other providers kept their pre-PR credential and region
resolution: a team bedrock deployment's region no longer overrides the
caller's region on RAG query. Query tests now fake the provider at the
HTTP boundary with a real vector store registry.
* fix(rag): use only the team's own OpenAI deployment for RAG credentials
* fix(rag): ignore caller credential fields the query search never uses when picking team credentials
* fix(rag): treat blank credential fields as absent when picking team credentials
* test(rag): delete the stores the integration ingests create before the owned proxy exits
* fix(proxy): let a restricted key use the team deployment its wildcard or model covers
A key restricted to gpt-4o on a team whose own OpenAI deployment is openai/*
never resolved that deployment's credentials, because the key allowlist was
matched against the deployment's public name literally. The allowlist now
matches a wildcard on either side the way request authorization does, through
the auth layer's own helper, so /rag, /files and /batches resolve the team
deployment for every key that may call a model on it.
* fix(batches): read a restricted key's batch input through its team deployment
The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it
Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment
* fix(rag): load deployments oldest first, reject file_id on stores that cannot take one, and show ingest errors in the dashboard
* fix(rag): pick the team deployment with the lowest id, return 400 for caller mistakes, keep the ingest id in errors, and show partial success in the dashboard
* fix(rag): pick the team's oldest OpenAI deployment so adding one never moves the team
* fix(rag): keep config file team deployments first and try JWT teams in sorted order
A team's config file OpenAI deployments now win in file order, as on the
base, before its DB ones oldest first, so upgrading never moves a team that
has both. A JWT in several teams tries them in sorted order, so every worker
resolves the same team.
* fix(rag): return 400 for Bedrock, S3 Vectors and Gemini store config mistakes on ingest
* fix(rag): keep forwarded provider keys on /rag/query and mixed JWT team claims working
A forwarded x-api-key now counts as a caller credential, so the /rag/query
search uses it over the team deployment as at the merge base. Sorting the
JWT team claim compares entries as text, so a stray number or null no
longer turns into a 401. A Gemini store with no proxy key is a 500 again,
and a non-string vector_store_id keeps the merge base's 400 message.
* fix(rag): let a forwarded provider key win over team credentials on files, batches and ingest
* fix(rag): keep team credentials when a request carries its own key or api_base
A forwarded x-api-key or a body api_key now replaces only the key of the team
deployment, keeping its api_base, as chat completions does. A body
litellm_credential_name no longer turns team credentials off on /v1/batches.
* fix(batches): keep the team key on the team host when a request sends only api_base
A request api_base now replaces the team deployment's api_base only when the
request also brings its own api_key, and a non-string api_key or api_base is
ignored, as on the merge base.
* fix(files): ignore a blank request api_key or api_base next to team credentials
* fix(files): answer 400 for a request api_key that cannot be sent as an HTTP header
* fix(rag): honor a body drop_params for a forwarded key that cannot be an HTTP header
* fix(rag): read the request field for drop_params only on multipart ingest bodies
* fix(rag): type the ingest body read for drop_params
* fix(batches): count a model-routed batch's input with the deployment key it is created with
* fix(batches): reject a batch create whose x-litellm-model headers name different models
* fix(rag): treat a null provider and whitespace store credentials as unset when picking team credentials
* refactor(files): type the team deployment the credential resolver ranks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(openinfer): add OpenInfer as OpenAI-compatible provider
Register OpenInfer in the JSON provider registry with catalog models,
proxy create fields, and unit coverage so requests route via openinfer/
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openinfer): assert catalog invariants instead of vendor rates
Pinning per-token prices made CI fail when the vendor list changed. Keep
the backup agreement and positivity checks our code owns.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openinfer): add openinfer to the Rust OpenAI-compatible provider mirror
* fix(openinfer): align catalog with OpenInfer's live /v1/models
Replace the four catalog rows with the two models OpenInfer serves and
publishes a price for (@oi/GLM-5.3-Flash, @oi/Qwen3.5-27B), using its
listed per-token rates and context windows, and price-check them in tests
* fix(openinfer): use OpenInfer's native Responses and Messages APIs, allow tools on unpriced models, price Qwen3.5-9B
Declare /v1/responses and /v1/messages for openinfer, since OpenInfer serves both natively. Let JSON providers declare provider-wide capabilities in providers.json, read through get_provider_info, so tool calls reach OpenInfer for models without a cost map row. Send max_completion_tokens as is, set output caps to each model's context window, mark the three rows as reasoning models, add Qwen3.5-9B at its listed rate, and wire openinfer into models_by_provider so the openinfer/* wildcard lists its models.
* fix(openinfer): let a model's own capability row win over provider defaults in get_model_info
Provider defaults now only fill capability fields the cost-map row leaves unset, so a row or deployment that turns function calling off stays off. Also marks openinfer_models as Final.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count the whole minor line in the upgrade notice for patch releases
A patch release's notes only list its backports, so v1.105.1 showed
"0 new features, 1 fix, and 0 other updates" even though upgrading from
v1.104.x brings everything in v1.105.0 too. When the latest release is a
patch, also fetch the recent release list and sum the stable releases on
the same major.minor line (skipping rc/dev prereleases and drafts).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): use tuples for release lists and sync the dashboard API types
Satisfies the LIT001 type-discipline gate and regenerates the
latest_release_info description in schema.d.ts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): cover the bad release-list payload and non-semver latest tag
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): type the release-line Counters so basedpyright sees no unknowns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): fetch earlier patches of the minor line by tag
Reading the first page of /releases could miss the line's .0 release when
prereleases push it off the page, and one unrelated release with a null
body failed validation for the whole list. Fetch each earlier patch tag
directly instead (v1.105.0 .. v1.105.N-1), skip tags with no stable
release, and treat a null release body as empty notes.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout
A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream.
* test(timeout): cover stream_timeout on live streams only and the router's timeout split
* fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management
* fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface
A faked stream now takes the longer of timeout and stream_timeout, so a caller
who raised stream_timeout to cover a slow fake stream keeps that headroom.
Azure o-series dispatch is mirrored when deciding whether a stream is faked.
text_completion, /v1/messages and /v1/responses through the Router now get
timeout and stream_timeout separately, like chat calls.
* fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages
* fix(router): report and log the timeout each stream was bounded by
Native /v1/messages and /v1/responses now record the timeout they applied, so
x-litellm-timeout and the cooldown guard read the real limit instead of the raw
request value. A prompt-managed model also passes on only the caller's or its
own timeout, so the target deployment's timeout still wins.
* fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit
* fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses
* fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit
* test(integration): cover faked, live and converted stream timeout limits per endpoint
* test(router): regression tests for faked-stream timeouts
* fix(ui): take the dashboard's user role and identity from the server
Adds an internal GET /management/v1/users/me that returns the user id, role
and email the proxy resolved for the calling key. The dashboard now reads
identity from it instead of the session cookie's claims, so its navigation and
role gating always match what the backend authorizes. The layout waits on it
behind the existing loading screen, and a failed lookup clears the session and
returns to login. Sessions that must change their password can also read it
* fix(ui): sign out when the current user lookup fails and wait for it on chat
The dashboard layout now holds the loading screen and signs out when
/management/v1/users/me fails, instead of mounting the shell and looping back
to loading as each consumer refetched the failed query. The chat layout also
waits for the identity so its panels never query without a user id. The
lookup goes through the typed client with the regenerated schema.d.ts
* fix(ui): send the session's auth header on the first user lookup and keep chat closed until authorized
The first lookup runs before AuthProvider registers the custom header name, so it now reads auth_header_name from the session cookie. Chat also waits for isAuthorized, so a failed lookup no longer mounts the shell with an empty user id
* fix(ui): type the dashboard callers of the now typed user id
useAuthorized used to hand back the cookie's user id untyped, which hid six strict type errors in its callers. The id is now string | null, so the production build failed. Callers handle the null case, and query key filters accept the boolean blocked filter
* fix(ui): look up the current user only after the proxy URL is discovered
Pages outside the dashboard shell, like chat and connect, could start the lookup before UI config discovery set the proxy base URL. On a split-origin deployment both attempts then hit the UI origin and a valid session was signed out
* fix(ui): keep a loaded identity while UI config discovery reruns
Pausing the lookup by passing a null token switched the query key, so on the dashboard the identity the layout had already loaded vanished until the shell's own UI config request finished. The key now stays the same and only the fetch waits for discovery
* fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row
A deployment with an explicit custom_llm_provider and model_info.mode: chat
used to overwrite the shared litellm.model_cost row, moving every other
deployment and plain SDK calls onto Chat Completions until restart, because
the downgrade guard read a double-prefixed key and never fired. The guard now
reads the real key, and a deployment's own mode decides its routing in
responses_api_bridge_check, except on chatgpt and github_copilot where Chat
Completions is dead for responses-mode models.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): keep the bridge-check mode tests in the sharded bridge flag file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read the downgrade guard's mode from the row registration writes
A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read
openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit
responses/ prefix now also outranks a deployment's chat pin.
* fix(router): let an explicit responses/ prefix outrank a registered chat row
The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the
lookup found mode chat and the request went to Chat Completions as responses/<model>.
* fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages
A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions.
* fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto
A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them.
* fix(router): skip a non-string model_info.mode when keeping the registered mode
* fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API
A Mantle row that also serves the Responses API now follows the GPT-5 tool and
reasoning summary bridge rules, since Mantle Chat Completions rejects function tools
on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat
gets a dict reasoning_effort as its effort level.
The opt-in auto summary asks each Responses config for its level, auto on Mantle's
OpenAI path, so a client's own detailed summary follows drop_params as at base.
No mode replaces a registered responses mode, and chatgpt and github_copilot rows
take any pin until they are responses, which restores the Copilot chat opt-out.
* fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort
* test(integration): run the Mantle region-prefixed v1 base case again
This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip
* fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin
A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request
* fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request
* fix(router): keep a non-string deployment mode off the shared routing row
A deployment whose model_info.mode was an int, a list, or an object still
reached the shared cost-map row, so an unpinned sibling on the same backend
stopped routing to the Responses API. The shared registration now drops a
mode that is not a string, and the deployment keeps its own raw value.
Adds integration cells for per-deployment routing across the OpenAI and
Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream
outage, config changes under traffic, and a worker kill plus restart.
* test(integration): find each routing cell's spend row by the response it logged
A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts
* fix(bedrock_mantle): run extra_body fields through the chat checks
A chat-pinned Mantle request that nests n or reasoning_effort in
extra_body now gets the same 400, drop, and effort-level handling as
the top-level fields. The bridge test injects a MockTransport client
instead of patching HTTPHandler.post.
* test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels
* fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort
* fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request
* fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode
* fix(router): drop the debug log that wrote a request's model name from keep_registered_mode
* test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort
* fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down
* test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks
* test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them
* fix(router): route a fine-tune the same whichever deployment registers first
Main's register_model now seeds an unmapped fine-tune's row from its base
model's built-in row, mode included, on the first registration. The mode
guard read only the live row, so a responses pin marked the shared row only
when it registered before its siblings. The guard now reads the row the
registration merges onto, the live row or else the row it would seed.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): move the MCP OAuth happy path from a live Linear e2e to an integration test
GitHub Actions now runs unit tests only, and the MCP OAuth happy path needed a
captured Linear browser login to click through a real consent page. The OAuth
flows it exercised are already integration tests against an owned
authorization-server double, except one behavior: the stored per-user token
must be served by a proxy process that never saw the consent, with no shared
Redis to fall back on. That case is now in test_mcp_oauth_flows.py
The e2e test, its owned-gateway helper, its workflow, the mcp_oauth_live marker
and the JUnit gate script only that workflow used are removed
* test(mcp): chat completions run an OAuth server tool with the caller's own stored token
* test(e2e): drop the Linear chat-completion MCP OAuth e2e and its helpers
The behavior it checked, a chat completion running an OAuth server's tool with
the caller's stored per-user token, is now an integration test in
test_mcp_llm_endpoints.py that needs no saved Linear browser login
* test(mcp): decode the stored per-user credential as urlsafe base64
The plaintext check used b64decode, which drops the - and _ of a urlsafe value
and raised on roughly one ciphertext in five
* test(mcp): drop the chat OAuth integration test that duplicates test_mcp_slots F2-chat
security/test_mcp_slots.py already sends a per-user OAuth token through
/v1/chat/completions and asserts the peer receives it. Every mutant the new test
killed was also killed by the existing suite, so it added no signal
* fix(router): read an application profile's base model from metadata on a cache hit
The 1h prompt-cache pin check read base_model only from the top level of
litellm_params. On a response-cache hit the params it sees carry the base
model only under metadata.model_info, so an application-profile ARN
resolved to itself and the pin was written for 5 minutes while the wire
carried a 1h TTL. Read it through get_base_model_from_metadata, which
also checks metadata.model_info
* test(router): pin top-level base_model precedence over metadata model_info
* fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens
* fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests
Vertex has no count-tokens route for Mistral or codestral models, so they now
get the same 400 as other partner models without a count API instead of a 404
that points operators at IAM.
* fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens
* fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model
* fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors
* fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer
* fix(count_tokens): pass through an OpenAI provider error that omits param or code
* fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route
* test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy
* test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes
The 48 MB budget on the third batch kept failing on the x86 release gate with no
leak behind it. Isolated rc/1.107.0 and v1.105.1 stacks both grew about 60 MB on
the third batch with gc-tracked objects flat, and both levelled off to a couple of
MB per batch after twenty batches. Measure batches 3-4 against a 200 MB per-worker
ceiling instead, which a native leak of 400 KB per failed request still trips. The
stored-request size check stays the deterministic catch for the v1.100.0 leak
The integration conftest raised UsageError from pytest_collection_modifyitems when a
shard-selected file collected nothing. Under xdist that ran on the workers after the
unfiltered item list was already published, so the worker ended its session with tests
still queued and the controller hit xdist's crashitem assertion, naming an unrelated
queued test and hiding the real ImportError.
A selected file that collects nothing now produces a failed collection report instead.
Files whose own collection already failed keep pytest's real error. Serial runs stop with
"Interrupted: N errors during collection", xdist runs report the error and exit 1, and
both still fail the shard.
WorkerKill returned once the victim's main thread was a zombie. On Linux the
process keeps its sockets open until its last thread exits, so a pooled
keep-alive connection to the killed worker still looked idle and the next
request through it got ECONNRESET. A zombie now counts as gone only once
num_threads() is at most 1. Where psutil cannot read threads of a zombie
(macOS) it raises ZombieProcess, a NoSuchProcess, so the old behavior holds
#46484 moved the litellm_redis_circuit_breaker_* metrics behind service_callback prometheus_system, so the chaos proxy, which only set callbacks prometheus, stopped exposing them and the breaker assertion read 0
* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy
These two files register deployments with their own model settings on the
gateway. They now boot a module-scoped proxy on a scratch database, so their
setup cannot affect other tests that share the worker's proxy
* test(integration): seed the isolated gateway with the same router settings as the worker gateway
* fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): update payload expectations for user email metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gcs_pubsub): add user_api_key_user_email to the spend log fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): drop untyped metadata key args from the debug log to keep the type gate flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): pass metadata keys to the debug log as a lazy %-style argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend_logs): store user email in spend logs only behind store_user_email_in_spend_logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(spend_logs): store user email in spend logs by default, drop store_user_email_in_spend_logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): store an empty user email as null, narrow the expected payload test diff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: Add failing test case
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Restamp nested response.model in streamed Responses events
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Add tests
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Run format and lint, and fix
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Type nested Responses restamp to satisfy basedpyright gate
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): restamp the nested model on streamed response.queued events
Background streams send response.queued as a GenericEvent, so its nested
response.model still carried the backend name. Also drop the new comments
* fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events
* test(proxy): drive typed events through the nested restamp no-op test
* fix(proxy): restamp the nested model on guardrail block frames in streamed responses
* test(proxy): assert a cache-polled background response reports the requested model
* fix(proxy): restamp nested model on dict events and on every SSE data line
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vector-stores): keep proxy context out of provider vector store metadata
The proxy merged its request context (user_api_key_* fields, endpoint,
user_agent, requester_ip_address, litellm_api_version) and the router's
deployment info into the vector store request's metadata. For vector
stores that field is also OpenAI's own metadata, which is capped at 16
keys, so a store created with a team key was full on creation and every
later metadata update returned 400.
Vector store routes now keep that context in litellm_metadata, like
batches, files and responses, and the router does the same for
avector_store_create. Provider-side metadata holds only what the caller
sent.
Adds db_scripts/strip_vector_store_internal_metadata.py, an opt-in
script that removes the proxy-written keys from stores created before
this change. It is a dry run unless --apply is passed.
* fix(vector-stores): page the metadata cleanup script without recursion and tolerate non-string values
Listing every store recursed once per page, which overflows the interpreter stack on large projects. Pages are now walked iteratively. Metadata values are no longer required to be strings, and a malformed response exits 2 with a fixed message instead of a traceback that could echo metadata values.
* fix(vector-stores): read proxy agent id, tags and spend_logs_metadata from litellm_metadata
Vector store routes now keep proxy context in litellm_metadata, but the v3
limiter only read agent_id from metadata, and logging let provider metadata
tags or spend_logs_metadata replace the proxy's values. A string tags value
also crashed get_request_tags.
The limiter now prefers the proxy-resolved agent_id in litellm_metadata.
Logging combines list tags from both buckets (metadata first, deduplicated)
and merges dict spend_logs_metadata with litellm_metadata winning, ignoring
values of the wrong type.
* fix(vector-stores): narrow the logging fallback to mistyped caller values
The tag union and spend_logs_metadata merge changed logged tags and
spend_logs_metadata on every route that has both metadata buckets, deduped
tags quadratically, and stopped background interaction polling from keeping
string tags as sent. Caller values now win exactly as before, and the
proxy's litellm_metadata tags or spend_logs_metadata only replace a caller
value of the wrong type.
The limiter reads agent ids through a pydantic model instead of a cast and
ignores non-string ids.
* fix(vector-stores): log the proxy's full tag list on litellm_metadata routes
On routes that keep proxy context in litellm_metadata, the proxy builds
litellm_metadata.tags from the caller's tags plus key, team and header
tags, but get_request_tags preferred the caller's metadata.tags. A caller
sending metadata.tags on a vector store call dropped the key's tags from
request_tags, which tag spend tracking reads. get_request_tags now prefers
a non-empty litellm_metadata tag list, and falls back to metadata.tags as
before.
* fix(vector-stores): strip LiteLLM-written keys at the provider boundary instead of moving metadata buckets
Moving vector store proxy context into litellm_metadata broke shared readers
(agent limits, request tags, merged logging metadata), and fixing them
widened the change to every litellm_metadata route. This reverts the bucket
move, the router change and the reader fixes, so proxy context stays in
metadata exactly as on main.
Vector store create and update now drop the keys LiteLLM itself writes
(user_api_key*, requester and deployment keys, agent ids) from the metadata
sent to the provider, in the shared optional-param helper both paths use.
The cleanup script reuses the same key list.
* fix(vector-stores): pass non-string metadata keys through the provider metadata filter
* fix(vector-stores): strip LiteLLM-written metadata keys only on proxy and router calls
* fix(vector-stores): send the provider exactly the metadata the caller sent through the proxy
* fix(vector-stores): snapshot caller metadata from the raw request body and keep it out of logged request bodies
* fix(vector-stores): copy the parsed-body metadata when the raw request body cannot be reparsed
* test(vector-stores): reproduce team key search routed to a team model of the wrong provider
A team key's model-less vector store search picks the team's first deployment, so a langgraph team model turns a Bedrock store search into "Vector store search is not supported for langgraph". Cover map_team_model picking by provider and a registered Bedrock store search reaching Bedrock with the registry's credentials.
* fix(router): match model-less team deployment to the requested provider
map_team_model now takes an optional custom_llm_provider. With no model given, it returns the first team deployment serving that provider, or None when none does, instead of the team's first deployment. Without a provider it behaves as before, so BYOK teams calling raw vector store IDs keep their team key.
* fix(proxy): pass the vector store provider when mapping team models
Vector store search, retrieve, update and delete pass the store's custom_llm_provider to map_team_model, so a team key's model-less call lands on a team deployment of that provider, or on the direct path with the store's own credentials when the team has none. Other model-less routes keep today's behavior.
* fix(proxy): type the vector store provider passed to map_team_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector-stores): cover team key vector store search provider routing end to end
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format route_llm_request.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type vector store routing tests and mock Bedrock transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin model-less vector store calls when a public name mixes providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep team provider routing in map_team_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries
* fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5
A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow
* refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping
Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass
* chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client
The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py
* fix(vertex): match standalone 403 status tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): keep text 403 fallback for wrapped 500 errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): keep BadRequestError for explicit 403 text matches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): let structured body codes override the 403 text fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(register_model): keep bedrock geo deployments priced across price data reloads
A request-time pricing registration inserting into litellm.model_cost while a
price data reload walked it could abort the reload (dictionary changed size
during iteration in add_known_models), or trip the lookup map rebuild inside
get_model_info so the deployment replay fell through to the unmapped path and
wrote a raw bedrock/au.anthropic.<model> entry with no token pricing, a $0
shadow re-rolled on every later reload. Walk snapshots of the map, bind the
case-insensitive lookup map locally, and let the unmapped path inherit the
whole built-in entry of a provider get_model_info would accept.
* fix(register_model): keep synthesized zero costs out of inherited built-in pricing
An unmapped alias that inherits its built-in entry now takes only the token
prices the catalogue really carries, so a synthesized zero never lands next to
the caller's custom price. The router regression test reports the writer's
failures instead of swallowing them, and the local names carry Final.
* fix(register_model): match inherited built-ins by the key provider prefix when the value names none
* fix(utils): resolve a tripped lookup from the key's own catalog row before its region sibling
* fix(register_model): keep custom flat prices above inherited tier rates
* fix(register_model): keep the other direction's tier rates when a custom flat price covers one
* fix(register_model): keep a catalog row's tiers when its lookup trips, and snapshot request-path cost map walks
A tripped get_model_info on a key that is itself a catalog row now inherits that row whole, so a custom flat price bills a long prompt at the row's above-threshold rate on both lookup paths instead of flipping until the next reload. The fal_ai, fireworks, ollama and gpt-5 request-path walks over litellm.model_cost iterate a snapshot so a concurrent reload cannot raise mid-request.
* fix(register_model): price route-prefixed geo keys on a tripped lookup and replace cost map entries atomically
* fix(register_model): move every alias of a cost map entry to its registered price
* perf(register_model): read alias names from the shared entry instead of scanning the cost map
Each registration scanned every cost map entry to find the names sharing it,
about 1 ms on the 4,189-entry map, and a proxy request to a custom-priced
deployment registers twice. Aliased entries now carry the names they are
served under, so a registration reads them directly, and an alias the caller
pointed at its own entry keeps that entry. The tests this PR added lose their
docstrings
* fix(utils): keep capability flags and write through on a tripped lookup in register_model
A key the lookup cannot map now seeds its registered entry from the capability
rules its unregistered lookup resolves to, so registering a region alias no
longer turns off supports_system_messages and similar flags. A lookup that
trips mid-walk on a key naming a catalog row writes into that row instead of
leaving a shadow key that goes stale. Adds regression tests for both and for
the request-path snapshot walks.
* fix(utils): retry a tripped registration lookup and rebuild a registration a reload races
A lookup a concurrent write trips is retried instead of writing into the
catalog row its key names, so a per-request price never reprices a shared
row. The inherited entry leaves the matched row's provider behind, so a
provider-less azure_ai key stays priced, and a registration whose map a
reload swapped mid-build is rebuilt from the new map.
* fix(utils): write each registration into the map it was built from and rebuild it when a reload replaced that map
* test(integration): price data reload keeps every registered deployment priced
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(litellm): report zero cost and cache-hit attrs on cache-served spans
* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects
gauntlet-mini on ace57d25f verified two live gaps. TextCompletionResponse and
ResponsesAPIResponse carry _hidden_params as a HiddenParams object, so the
dict-gated marker write skipped them and the served hit still stamped the full
price. The marker write is now container-agnostic, which it already was for the
partial-embedding stamp, and a regression test exercises the carrier end to end.
_uncached_response_cost also restores billed_token_rates, zero_cost_warned, and
the response_cost_failure_debug_information key alongside cost_breakdown, so the
probe leaves no pricing state on the call it reads.
* test(caching): cover cache-hit marker on HiddenParams carriers through the caching handler
* fix(logging): keep the uncached price probe's restores to the state it can leak
* fix(otel): emit V2 LLM-call span for cache hits with explicit cache attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read cache payload fields defensively without casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel): clarify pyright suppression reason on cache payload reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): build the cache-hit view from typing_extensions.TypedDict so import works on Python 3.10 and 3.11
* fix(otel): report cache hits at zero cost everywhere and mark misses
Cache hits now carry a zero cost breakdown and a zero hidden response cost,
the /v1/messages hit cost header follows chat and responses hits, every span
carries litellm.cache_hit, V2 hit spans drop server.address, an async hit keeps
the provider-prefixed model name, and the V2 attribute budget counts preset keys
* fix(otel): send a 0 cost header on cache hits and keep hits out of deployment TPM
Cache hits now send x-litellm-response-cost: 0.0 on chat, responses and
/v1/messages, while a fresh call whose cost is 0 still sends no header.
The async hit path stamps the same model name as main again. Stamping the
provider-prefixed name made a hit's tokens count toward the deployment's
TPM limit, so repeated cached requests hit 429s under usage-based routing.
Drop comments that restated fields and annotate new test values as Final.
* fix(otel): keep the served pricing basis on a cache hit's cost breakdown
A hit's breakdown now keeps service_tier, data_residency, vertex_location, and the discount and margin rates the served call recorded, with every amount at 0, so savings reports still read the basis. The hit header test now drives the live non-streaming header path.
* fix(otel): send zero breakdown headers on a cache hit and give hit spans a fresh call's cost keys
A hit's x-litellm-response-cost-original, -input, and -output headers now read 0.0 instead of going missing or carrying the full price. A hit's cost breakdown now comes from pricing the response as fresh, so every route exports the same keys, pricing basis, and rates with every amount at 0.
* test(integration): assert zero cost and the cache-hit flag on cache-served OTel V1 and V2 spans
* test: wait for the cache entry before the cache-hit request and build cost headers without a route_request double
* test: annotate the new cache-hit unit test locals with Final
---------
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130)
Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main
with the unit test moved to tests/unit after the test tree relocation
* fix(presidio): label an overlapping run by its outermost detection, add integration coverage
A run of overlapping detections now takes its entity type from the highest-scoring
detection that no other detection in the run contains, matching Presidio's own
anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER.
The resolver returns typed detections instead of Any.
* fix(presidio): drop the redundant comment in the numbered token loop
* fix(presidio): bump a numbered token another part of the request already holds
Each message and content part numbers its tokens from 1 into one request-wide
pii_tokens map. Collapsing overlapping detections made an overlapping value's
token well formed, so it collided with the same type in another part and both
restored as the later value. A token that maps to a different value now moves
to the next free number, and a token for the same value is reused
* fix(presidio): give a clashing token a spare number instead of scanning for one
Scanning upward from the clashing number probed every token an earlier part
had claimed, so two parts with thousands of values of one type took seconds
of synchronous work. No token in the map is numbered above the map's size,
so numbers above both that size and the part's run count are always free
* test(presidio): make the anonymizer double reject detections the analyzer never returned
Pick the token in one conditional expression instead of reassigning it, and
have the scripted anonymizer answer 500 when the proxy forwards spans that
differ from the analyzer's, so the default-scope cells check what was sent
---------
Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost
* test(responses): keep the websocket unit test file on main's formatting
* fix(responses): type the websocket frame guard for the lint gates
* fix(responses): pass the websocket frame guard as a named parameter and mark the guardrail dict edits
* fix(responses): bill a guardrail-rejected Responses WebSocket frame as a failed request with its guardrail cost
* fix(responses): type the recorded guardrail records as Mapping for the lint gate
* fix(responses): bill a rejected websocket frame through the spend logger only so the connection keeps its rate-limit slot
* fix(responses): guard nested response.create frames, strip forged user_api_key_* frame metadata, copy guarded tools back and give each rejected frame its own spend row id
* fix(responses): mark the spend logger's bare-dict failure hook for the type gate
* fix(responses): reject response.create frames that carry input at both levels and keep billing of a rejected frame non-blocking
* fix(responses): name the non-blocking billing catch for the strict lint gate
* fix(responses): keep the admitted guardrail list on every websocket frame, a frame cannot replace it with its own guardrails list
* test(responses): wait for the key spend total before asserting it in the websocket guardrail cost test
* fix(responses): copy the admitted metadata shallowly per websocket frame, the OTEL parent span cannot be deep-copied
* fix(responses): reject a WebSocket turn a pre_call guardrail rerouted to another model
* fix(responses): bill a rerouted WS turn and carry Presidio pii_tokens from the frame guard to the socket unmasker
* fix(responses): default pii_tokens through default_factory, Python 3.13 rejects a mappingproxy dataclass default
* fix(responses): run the per-frame guardrail walk on managed Responses WebSockets too
* fix(responses): unmask managed Responses WebSocket replies with the frame guard's Presidio pii_tokens
* fix(responses): strip client guardrail cost records from WS frames so a blocked frame bills the real scan
* fix(responses): forward a WS frame without the fields a guardrail deleted
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
