* test(e2e): mixed_size_requests_2rps perf case (coding-agent bodies, JWT and key callers, two guardrails)
* test(e2e_harness): test_observe builds DbCounters with statements and expects the background Postgres line
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared disconnect release dispatched to the legacy limiter, which decrements unconditionally, so a rejected realtime attempt freed a slot it never took. The generic release is v3-only again and the Responses WebSocket close releases through its own method
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(cost): bill Responses API image_generation_call tool usage
Price completed image_generation_call output items from tool_usage.image_gen token counts when present, otherwise per image from the quality and size on the item, with the image model taken from tools[].model (default gpt-image-1). Add the result into tool_usage_cost alongside web search and file search, and read tool_usage from the top level or usage.tool_usage. Squashed and rebased onto main
* style(cost): annotate read-only views and mark casts for the type discipline gate
* fix(cost): price Responses image tool from tools[].model only, no default model or provider
* refactor(cost): drop redundant status check, unread usage fields and unused output_type literal
* test(cost): integration case for Responses image_generation tool token billing
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf datastore limits count what a request waits on, background Postgres and Redis work reported on its own
postgres_calls_per_request and redis_calls_per_request were whole-server totals (pg_stat_database commits, Redis INFO
commands), so spend flushes and config reloads moved them and a new per-request lookup could hide in the noise. They
now count the statements and commands a request waits on before it is forwarded (auth and routing reads, the
rate-limiter script), from pg_stat_statements read before and after the run and a Redis MONITOR connection held
during it. Spend writes, daily aggregates, config reloads and cache invalidations are postgres_background_calls_per_request
and redis_background_calls_per_request. The run output lists the top statements and commands per bucket, and the
counters are read right after the run so the test's own spend sample is not counted
* test(e2e): perf datastore reads only the runtime user's pg_stat_statements rows, other users' rows have a NULL queryid
* test(e2e): perf Redis classifier strips the litellm.caching: namespace the deployed gateway puts on every key
* test(e2e): small_bodies_5rps limits from ten split-metric runs (postgres awaited 1.8, redis awaited 18, postgres background 12.5), spend buffers and cron locks classified as background
* test(e2e): small_bodies_5rps asserts only the awaited Postgres and Redis counts, background lines stay printed
* test(e2e): perf Redis classifier puts the cron lock script's GET and DEL in background, not awaited
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset end-user spend in committed pages so the budget cascade survives large LiteLLM_EndUserTable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): collapse _enduser_ids_by_tier return to one line for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the end-user page IN list as bounded for the unbounded-IN check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* test(e2e): attach perf case metrics and limits as JUnit properties
The perf case only printed its metrics table, so a green run left no numbers
behind. Each observed metric and its limit now land as <property> entries on
the test's JUnit testcase, named perf_<metric>[_p<percentile>] and
perf_<metric>[_p<percentile>]_limit, so project-releaser can collect them
* test(e2e): compare perf properties exactly instead of through pytest.approx
basedpyright flags pytest.approx as partially unknown in the e2e lint step,
so the harness test now uses metric values that are exact in binary floating
point
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn
* test(responses): mock the websocket slot release and mark the per-turn start time naive
* style(responses): keep the per-turn start time line within ruff format
* refactor(responses): type websocket turn events as read-only mappings
* fix(responses): settle the websocket budget reservation on the first turn only
Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly
* fix(proxy): keep the legacy limiter's websocket slot until disconnect
A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache
SAML login tickets, the single-use assertion mark and cached IdP metadata
lived in user_api_key_cache, which only gets Redis when
enable_redis_auth_cache is on. With several replicas or workers the IdP
callback often landed on a process that never saw the ticket and got a
401, and IdP-initiated sign-in was refused even with Redis configured.
SAML state now has its own DualCache that _attach_redis_usage_cache
always wires to Redis, like the CLI SSO session cache, and the SAML cache
calls are async.
* test(sso): drop docstring from the SAML proxy pair fixture
* test(sso): cover the SAML login and metadata endpoints
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* test(count_tokens): expect the converted Responses input in the budgeted-key count integration test
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(guardrails): honor streaming settings on the Noma v2 guardrail
The Noma v2 initializer now passes streaming_buffer_until_moderated,
streaming_end_of_stream_only, streaming_sampling_rate and
streaming_buffer_release_on_scan from the guardrail config to
NomaV2Guardrail, so the unified streaming hook applies them. The
constructor validates them, so a custom guardrail class built on
NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults,
and an invalid value is ignored with a warning naming the guardrail, so
the guardrail still loads.
Resolves LIT-9633
* test(guardrails): cover Noma v2 streaming settings across clients, failures and workers
Adds integration cells for the custom-subclass loader, string and invalid
values, unset and false keys, a Noma outage with and without block_failures,
an upstream rejection, n=2 choices, pre_call and non-streaming guardrails,
the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a
PATCH while traffic flows, a burst while Noma is down, a slow Noma, and
replaced workers.
Resolves LIT-9633
* test(guardrails): pin the PATCH probes to one worker at a time
Each probe suspends the other worker and opens a fresh connection, so the
test checks that every worker applies the patched setting.
Resolves LIT-9633
* test(guardrails): check sampled scans by count, not window position
/v1/messages hands the streaming hook raw network reads, so where the
default sampling windows fall depends on timing; the base leg caught one
shifted window. Check the scan count and the final full-text scan instead.
Resolves LIT-9633
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
K2 in test_logging_only_scope_chaos.py flaked on CircleCI with
AttributeError: type object 'BaseModel' has no attribute '__pydantic_core_schema__'
raised inside the OpenAI client while its 30-thread control burst parsed responses
The SDKs define their response models with deferred schemas, so the first parse of each
model rebuilds it, and pydantic's model_rebuild is not safe when several threads hit it
at once. Every integration worker now builds those schemas in pytest_configure, before
any test can start a thread pool
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): preserve explicit websocket API keys
Refresh the one-line fix from BerriAI/litellm#34253 against the current default branch
Co-authored-by: puffer <389053104@qq.com>
* test(responses): verify websocket authorization
Cover explicit deployment keys, precedence over global keys, and the existing global fallback at the upstream WebSocket transport boundary
* test(responses): type the authorization capture
Replace the mutable header list with a typed, final Future carrying the upstream authorization value, addressing the review on #40853
* Refactor test_openai_websocket_authorization method
* fix(responses): resolve websocket keys with the deployment's provider
Pass the deployment custom_llm_provider to get_llm_provider so an api_base
host never swaps in another provider's env key, and drop the OpenAI-only
global key fallbacks so non-OpenAI websocket deployments resolve their own
env keys the way the HTTP route does
* fix(responses): let a deployment's own websocket key beat env keys
A deployment with an api_base on a host litellm recognizes, and no declared
provider, got that host's env key ahead of its own api_key. The explicit key
now always wins, matching the HTTP Responses route. Drop the litellm_params
api_key fallback, which could never be set because api_key is a named argument
* fix(responses): keep openai deployments on other providers' known hosts on the HTTP stream
An openai deployment whose api_base is a known OpenAI-compatible provider
host (DeepSeek, Groq, ...) opened a native Responses WebSocket there and got
a 405, because those hosts serve /responses over HTTP only. Route it through
the managed HTTP stream with the deployment's own key instead.
* fix(responses): resolve openai deployments on known provider hosts as that provider
An openai deployment whose api_base is a known provider host (DeepSeek,
Mistral, ...) now resolves to that host's provider on the Responses
WebSocket again, as before this branch, so it keeps that provider's
chat completions bridge, pricing and env key fallback. Its own api_key
still wins over the env key. The known-host test now mocks the upstream
HTTP route and checks the route, the bearer and response.completed.
* test(responses): prove websocket deployment keys and declared providers reach the provider socket
* test(responses): use the reported bare model shape for the websocket deployment key cell
---------
Co-authored-by: Ysh <734645956@qq.com>
Co-authored-by: puffer <389053104@qq.com>
* refactor(rust-bridge): let each route host pick the body params
The native routes projected request body params from a name list, hard-coded in
Rust for Messages and read from `PARAMETERS` for Chat Completions and Responses.
Each route host now answers `body_params(request)` over the keywords the call would
send, so the Python side owns which params reach the provider. No behavior change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust-host): use read-only body parameter types
Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): move the route host modules under host/routes
Each native route's Python half lived in its own `<route>/route_host.py`, apart
from the shared host modules Rust also imports by name. They now sit together in
`litellm/rust_bridge/host/routes/<route>.py`, with their unit tests moved to the
mirrored path. No behavior change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-host): reference messages route alias
Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): close each federation spec API connection so the proxy cannot drop an idle keep-alive socket
* test(ui): send Connection: close on every integration browser spec API call
* test(ui): send Connection: close on every integration browser request from the Playwright config
* fix(router): carry a 429 status on cooldown and no-deployment errors
RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the
proxy answered an all-deployments-cooled-down request as a generic 500 class
error and logged the bug report notice for every cooldown. Both now carry
status_code 429, which the proxy maps to a rate_limit_error and which the
retry policy already treats as retryable
* fix(proxy): start Sentry at config load and never attach request bodies
Sentry only came up inside set_callbacks on the first LLM call, so boot and
background errors were never reported, and general_settings.alerting: [sentry]
always raised Missing SENTRY_DSN because the alerting path read a dead
litellm.utils pin instead of the live SDK. The proxy now initializes Sentry
once while loading the config whenever sentry is a configured callback or
alerting target, the alerting and failure handlers read the live instance, and
max_request_body_size is pinned to never so prompts and completions stay out
of Sentry events
* fix(tracing): log Lens failures at ERROR and export row counters to Prometheus
A tracing store that fails to start, or a Lens service that rejects a batch,
was only logged at WARNING, so operators running at the default ERROR level
never saw tracing silently go dark. Both now log at ERROR and name the lens
store. The exporter's rows_written and rows_dropped tallies were never
exported, so the two counts are now Prometheus counters
litellm_lens_export_rows_written_total and
litellm_lens_export_rows_dropped_total, registered lazily on the default
registry like the other proxy-level counters and charted on the all-metrics
Grafana dashboard
* test(proxy): drop the litellm import the alerting tests no longer use
* fix(proxy): start Sentry once a DSN is set, from any config source
Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it
* fix(sentry): keep stack frame locals out of events unless PII is opted in
Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII
* test(router): drop the test that only pinned the 429 class attribute
* fix(proxy): start Sentry at boot when it is only an input callback
* fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* ci(cost map): classify token_count_framing and token_count_fallback_model in the schema generator
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* test(cost map): reject a fractional token_count_framing in the generated schema
* test(cost map): pin that a token_count_framing of 0 passes the generated schema
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): narrow the provider response id with isinstance instead of cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): validate callback fields for current type gates
* test(gcs_pubsub): assert provider response id in spend metadata
* fix(ollama): normalize reused response ids automatically
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf streams take a reasoning or tool-call delta as the first token, tool-call replies had no time_to_first_token
* test(e2e): perf users start at a random offset and jitter each interval by up to 25%, no more lockstep bursts at the planned rate
* test(e2e): small_bodies_5rps sends from the mixed case's callers (two teams, JWT users, keys in and out of teams) through the pre and post guardrail
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking
* fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds
Bedrock Invoke copied every caller thinking key when adding display updates,
so an unknown key still reached the provider with drop_params on. Filter that
copy the same way, and cover each chat call site plus the global
litellm.drop_params path in tests
* fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params
* test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions
* fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel
* feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI
Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing
early-access row, carrying the same limits and capabilities. The AI Studio row
mirrors the early-access entry; the Vertex row differs only in provider and
pricing source. Prices follow once Google publishes them.
Extend the mapped metadata test to cover all three ids, that each routes to its
provider, that its backup copy matches, and that the public rows carry the same
limits as the early-access row.
* feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops
* chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap
* fix(gemini): follow continuation hops in a loop instead of recursion
The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls
* fix(gemini): bill Google Search grounding from every continuation request
Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain
* fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI
A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream
On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued
On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports
continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch
* chore(gemini): name the type errors the dual-mode stream iterator trips
The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors
This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up
* fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens
A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request
A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate
Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk
The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment
* fix(gemini): keep one response id across a continued answer
A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream
A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key
* test(gemini): add integration cells for long decode continuation
Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs
Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once
* fix(vertex_ai): fetch the access token again before each continuation request
A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length
Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged
* fix(vertex_ai): start each continuation stream with an empty parser buffer
When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error
Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async
* fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up
A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token
Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way
* fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer
A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer
The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes
* fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata
A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex
A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not
* fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer
A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0
A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer
Adds the audit's wire tests for both, which fail 24 cells before this change
* test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index
Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed
* fix(vertex_ai): log a continued answer's raw response as one object
A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them
It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted
Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch
* test(integration): check a continued answer logs one merged raw response
A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch
* test(vertex_ai): cover a follow-up tool call and its thought signature in long decode
* test(gemini): pin that a one-request answer logs Google's body unchanged
* test(integration): assert the streamed usage.cost matches the spend row on every cost case
* test(integration): type the streamed usage frames as JsonValue
* test(integration): only require streamed usage.cost when the client received a usage frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses
* fix(ui): send detail on playground Responses image parts and code snippets
* test(messages): cover image detail on the Responses bridge wire
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it
Fixes#31449.
Two OCI GENERIC request-building defects that made Grok-on-OCI unusable
for tool-using coding-agent clients (Codex over the Responses API, Claude
Code over the Anthropic Messages API):
1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only
supports function tools")` for any tool whose type is not "function",
aborting the whole request when a client mixed a built-in/server tool
in with its function tools. Non-function tools are now skipped with a
warning; if none remain, `tools`/`toolChoice` are omitted from the
request instead of sending an empty array.
2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere
model, but OCI rejects it for all served xAI Grok models
(`Model grok-4.20 does not support parameter reasoningEffort`) and for
OpenAI's non-reasoning models (`Unrecognized request argument supplied:
reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-*
and Meta/Google models accept it. `_model_supports_reasoning_effort`
encodes that, `get_supported_openai_params` stops advertising the param
for those models, and `map_openai_params` follows the standard
unsupported-param contract (dropped under drop_params, clear error
otherwise) so `drop_params` / `additional_drop_params` now work as
documented instead of the 400 surfacing from the service.
Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model.
The two existing reasoning_effort normalisation tests move from a Grok
model to openai.gpt-5, which is the case they exercise. 21 new tests; the
OCI suite is 406 passed.
* fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools
Review follow-ups on #40732:
- `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag
of the model's `oci/` catalog entry through `get_model_info` instead of
hardcoding xAI / gpt-oss branches. Catalogued models without the flag
(xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent
from the catalog keep the historical pass-through.
- Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning
models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b`
(pricing from Oracle's public price list, 128K context per OCI docs).
- `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only.
OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output
cap silently disappears) and Llama rejects it with 400, so the catalog's
reasoning flag must not drive that switch.
- After tool adaptation, a `FUNCTION` tool_choice naming a tool that was
skipped (built-in / server tools) is dropped with a warning instead of
being sent to OCI, which rejects the unmatched choice.
Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a
forced built-in now completes and still calls the function tool; Gemini and
gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under
drop_params; gpt-5 keeps maxCompletionTokens.
* style(oci): use an immutable set for the tool-name lookup
Keeps the type-discipline gate (LIT002, mutable collections) within its
budget; the lookup is read-only.
* fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive
Review follow-ups on #40732:
- When tool_choice forces a function that is not among the adapted tools
(a skipped built-in or a misspelled name), send `toolChoice: NONE` instead
of removing the choice. Falling back to the service default would let the
model call any other supplied function even though the caller forced a
specific one, which an agent processing untrusted content must not do.
- Move the #31449 regression tests into the mapped test module
(`test_oci_chat_transformation.py`) instead of a separate file.
Verified live against OCI GenAI: Grok and Llama with a forced skipped
built-in now complete with no tool call; a forced kept function is still
honoured.
* test(oci): point reasoning_effort comments at the in-module gating tests
The referenced module was folded into this file.
* fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels
Non-function tools now follow the unsupported-param contract: a 400 naming
drop_params without it, and dropped with a warning under it, with a forced
dropped tool turning tool calls off. reasoning_effort is gated on the model's
reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the
Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop
sending a param the service rejects
* fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it
* fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name
* fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name
* fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap
* fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR
Under drop_params, non-function tools still drop when function tools remain and
tool_choice does not force a dropped tool. A request whose every tool is dropped,
or whose tool_choice forces one, now answers 400 instead of a tool-less reply the
model can fill with invented results, or a swap to another tool.
The maxTokens routing change for OCI Gemini goes back to base, since it is
outside this fix and an open PR carries it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
GET /v1/mcp/server takes an optional user_id (admin, or the user themselves) and returns the union of the servers each of that user's teams reaches plus allow_all_keys servers. A user in no team gets the unscoped list, a deleted team still listed on the user is skipped, an unknown user is a 404 and a restricted virtual key is refused like for team_id. team_id keeps precedence when both are passed.
The dashboard MCPServerSelector passes the team and user it edits (key edit sends keyData.team_id and user_id, key create sends the picked team and owner, user edit sends the user) and narrows the server options to members of the access groups selected in the same control, keeping allow_all_keys servers and servers already granted so they can still be removed.
Resolves LIT-7784
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fastapi_sso opens a bare httpx.AsyncClient for the non-PKCE token exchange and
userinfo call, so litellm_settings.ssl_verify and SSL_VERIFY never applied and
an IdP on an internal CA failed with an opaque 503. A LiteLLMHttpClientSSO mixin
replays that exchange on litellm's shared SSO client for the generic, Google and
Microsoft providers and surfaces IdP connection and TLS failures as a 401
ProxyException naming the endpoint.
Resolves LIT-9445
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): name the successor model for deprecating deployments
* chore(ui): add successor_model to the generated deprecation api types
* fix(proxy): keep the cost map successor when a deployment overrides only the date
* fix(proxy): read the cost map successor from metadata.successor
The cost map schema rejects a top-level successor_model key, and curated
entries already carry the successor under metadata.successor. Read that key,
and look it up even when the cost map entry has no deprecation date, so a
deployment that sets only its own date still names the successor.
* fix(proxy): keep the first deployment when deduping, let a blank successor hide the cost map one, and name a served successor by its model group
* fix(proxy): name only the deployment's own successor_model, since cost map successors are LiteLLM ids rather than served model names
* fix(proxy): leave out a successor_model the deployment's callers cannot call, with a warning
* fix(proxy): report successor_model as the admin set it and only warn when no model matches it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs
* fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off
* fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off
* test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it
With modify_params on, a trailing tool search turn whose call has a kept result
got a placeholder answer from the factory and went out as a marked tool_use,
but the cache control hook counted marks before that placeholder existed and
skipped the mark, so a request at the cap reached Anthropic with 5 marks and
failed with a 400. The hook now finds client answers on the sanitized messages,
the same list the factory reads.
Also make the replay checks cheaper (prefix check before validation, early
return without provider_specific_fields, and look up client answers only when
a tool call carries cache_control), drop the two audit cells that cover old
gaps outside the tool search fix, and derive the pass-through test timeout
from the graceful stop budget.
* fix(anthropic): keep the duplicate tool result warning out of the cache control hook
The hook finds client answers with only the step of the modify_params sanitize
pass that pairs tool calls with their results, so a request with a duplicate
tool result logs the warning once again instead of once per mark count.
Also restore the Snowflake non-stream and cached-stream tool search cells as
known bugs with pytest.skip instead of deleting them, and put the Anthropic
pass-through test timeout back to 240 seconds.
* fix(anthropic): count cache marks next to a message the pairing step cannot read
With modify_params on and a marked tool call, the cache control hook paired tool
calls with their results before counting. A message without a get method (a
stray string or null in extra_body.messages, or an openai SDK
ChatCompletionMessage) made that pairing raise AttributeError before any
provider call, where main counted the marks and let the request through.
The hook now pairs only when every message can be read the way the pairing
step reads it, and otherwise counts on the messages as sent, like main did.
* fix(anthropic): count cache marks after placing a later system message the way the request builder does
* test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census
* test(anthropic): align the tool search replay cells with main and cover a system message after the search turn
The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882.
The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool
reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell
expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its
mark, and a new cell covers a system message after the trailing search turn in both stream modes
* fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request
* test(anthropic): cover marked tool calls next to a malformed tool id and a later system message
* perf(anthropic): look up client tool answers only when a marked tool call is a server tool call
Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked
tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count
marks
* test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell
* perf(anthropic): place mid-conversation system messages in linear time for models that keep them
* fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time
* test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn
* fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag
* test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router
* fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure
* test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement
* fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests
Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route.
* test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse
* fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow
* test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt
* refactor(test): avoid local list mutation and satisfy type discipline gate
* style(prompt_templates): keep line length under 120 chars for ruff format
* style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401
* style(prompt_templates): eliminate basedpyright errors in function_call_prompt
* fix(prompt_templates): keep tuple and None system content when adding the function prompt
Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged
* fix(prompt_templates): reject unreadable system content without echoing it
Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt
* fix(prompt_templates): leave non-list system content failing as before
None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated
* fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params
* test(prompt_templates): pin the function prompt on every system message
* test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
