Two NVIDIA-Riva-specific fields consumed by the audio-transcription
handler via the provider's `optional_params` passthrough were not
covered by the proxy's existing banned-request-body list or the
admin-config clearing list applied on `api_base` BYOK override:
* `nvcf_function_id`
* `use_ssl`
Add both to `_BANNED_REQUEST_BODY_PARAMS` in
`litellm/proxy/auth/auth_utils.py` and to the kwargs-only list in
`_admin_config_fields_to_clear_on_base_override()` in
`litellm/router_utils/clientside_credential_handler.py`, next to the
analogous provider-specific entries already there (`aws_bedrock_*`,
OCI provider fields, etc.). Same admin opt-ins as every other entry
on those lists (`general_settings.allow_client_side_credentials`
proxy-wide, or `configurable_clientside_auth_params` per deployment).
Regression tests in `tests/test_litellm/proxy/auth/test_auth_utils.py`
cover root-level rejection, the historical `api_key` bypass, both
admin opt-in paths (proxy-wide and per-deployment), nested-container
smuggling via the existing recursive walk, and clearing on
`api_base` override. Mutation check verified.
Resolves VERIA-493
* test(pass-through): de-flake vertex spend-log assertion by re-billing
The vertex pass-through spend-log test asserted that a single billed
generateContent call moved the global spend aggregate within a fixed
wait. CI failures show the call returning a valid response with real
usage, yet spend never increasing over a 240s poll.
Pass-through spend logging is best-effort: the success handler is
enqueued on a background worker that can drop or time out an individual
event under load and never retries it, so one billed call occasionally
never reaches LiteLLM_SpendLogs. Waiting longer cannot recover a dropped
event; only re-issuing the call can.
Re-bill the call up to a few times and require at least one to be
tracked, mirroring the sibling jest test that already retries. The test
still fails hard if cost tracking is actually broken, since then every
call records nothing. Also sum spend across all returned days instead of
matching the runner's local 'today', removing a separate UTC-rollover
flake.
* test(pass-through): route vertex spend-log test through proxy via direct HTTP
The vertexai SDK, configured with location="global" and an http api_endpoint
override, intermittently sends generateContent to the public Vertex endpoint
instead of the proxy. Proxy logs from a failing run show all 46 of the test's
own spend-log polls reaching the proxy while zero generateContent calls did, so
LiteLLM never saw the billed call and no spend was ever recorded; re-billing
through the SDK could not help because every retry bypassed the proxy too.
Issue the pass-through request directly over HTTP so it always hits the proxy,
minting a Google token from the same service-account credentials, then assert
that the specific call's own spend log lands with spend > 0, a gemini model, and
custom_llm_provider vertex_ai. A small best-effort retry covers the rare case
where the background logging worker drops a single event; failing every attempt
still fails hard so the test keeps its teeth if cost tracking breaks.
* test(pass-through): reuse LITE_LLM_ENDPOINT and drop needless async in get_tracked_spend
* fix(token_counter): count legacy function_call.arguments (VERIA-492)
token_counter handled the modern assistant tool_calls field but had no
branch for the legacy OpenAI function_call payload. The value is a dict,
so it skipped every special-cased branch in _count_messages and fell
through to the unsupported-key continue, letting arbitrary text in
function_call.arguments slip past the count.
Resolves VERIA-492
* refactor(token_counter): raise on unexpected key in _count_function_call_tokens
Address Greptile P2: the helper's fallback branch previously applied
function_call logic to any key that wasn't tool_calls. Make the contract
explicit so a future caller can't silently miscount.
The Presidio streaming post-call hooks (_stream_apply_output_masking for
apply_to_output and _stream_pii_unmasking for output_parse_pii) collected every
upstream chunk, reassembled the full completion with stream_chunk_builder at
end-of-stream, ran Presidio over it, then emitted one reconstructed SSE chunk.
Time-to-first-token collapsed to the total generation time and token-by-token
streaming was lost whenever Presidio output handling was enabled. With the
default presidio_filter_scope both, an apply_to_output masking instance is always
created, so even the unmask configuration buffered the stream.
Both paths now transform and forward chunks as they arrive. The unmask path
replaces placeholder tokens per chunk, holding back only the trailing run that
could still grow into a token so a placeholder split across SSE chunks
(<PER + SON_1>) is still rewritten atomically. The mask path emits a prefix only
when masking it in isolation matches the corresponding prefix of masking the
whole buffer, with a lookahead margin still buffered past the cut, so an entity
straddling the cut is detected and held until complete; past
_PRESIDIO_STREAM_MAX_BUFFER the run is bounded without splitting an entity.
Tool-call and legacy function-call argument fragments are accumulated per choice
and transformed once the choice closes, content is buffered independently per
choice index for correct n>1 streaming, raw Anthropic SSE bytes and /v1/responses
events pass through with any held content flushed first so events never reorder,
and a masking error redacts only the affected chunk (fail closed, keeping
finish_reason) while the stream continues.
Resolves LIT-3222
RealTimeStreaming.log_messages dispatched the success handler with a bare
asyncio.create_task, bypassing GLOBAL_LOGGING_WORKER (which gives a per-coroutine
timeout and a concurrency cap). On a long-lived realtime websocket a slow logging
callback left one suspended task per logged turn, each pinning that turn's
assembled response, accumulating without bound (~12-15k in-flight under load in a
repro) until OOM. Route realtime success logging through the bounded worker so
in-flight logging is capped and a hung callback is cancelled at the worker
timeout.
The chat and responses streaming success-logging paths are intentionally left
unchanged: their success callbacks must complete within the call's event-loop run
(the non-streaming path pairs the worker with a synchronous callback; the
streaming path has no such companion), so deferring them through the worker would
drop logs for one-shot SDK calls and breaks test_async_custom_handler_stream.
Bounding those paths needs a load-shedding approach and is left to a follow-up.
When redis_startup_nodes is set the async cluster client was built with no health check and no TCP keepalive, so a connection silently dropped by a cluster restart (e.g. ElastiCache Serverless maintenance) stayed in the pool and got reused while dead; the first command after the restart stalled in re-initialization until the LoggingWorker timeout cancelled it, surfacing as CancelledError then TimeoutError on the spend-counter path
Build the async cluster client with a 25s health_check_interval and socket_keepalive so an idle connection is PING-validated and reconnected before reuse, and expose both through the cluster kwarg allow-list so an explicit value from config still wins
Resolves LIT-4083
update_spend_logs flushes the queue with a single create_many per batch, so one
row carrying bytes Postgres refuses (a residual NUL byte is the canonical case)
fails the entire insert and drops every good spend log alongside it. PR #29515
strips NUL bytes from the JSON columns, but the scalar string columns (end_user,
model, session_id, ...) still flow through unsanitized, so a poisoned row can
still reach the write and take a batch of up to 1000 good rows down with it.
On a genuine data-layer rejection the batch is now bisected so the good rows
still persist and only the offending row is dropped and logged with its
request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error
(matched by exact type so systemic subclasses like a missing table are not
mistaken for a single poison row), which keeps prisma an in-function import and
litellm.proxy.utils importable without the proxy extra. Transport failures,
including the "can't reach database server" outage that prisma mislabels as a
DataError, are re-raised unchanged so the existing connection-retry path still
runs and a transient outage never turns into silent per-row data loss.
The bisection carries a per-batch isolation budget so an authenticated caller
flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed
inserts and N log lines; once the budget is spent the still-failing remainder
is dropped wholesale under a single log line.
Resolves LIT-4103
* feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints
The unified /v1/messages proxy endpoint always translated inbound Anthropic
requests down to /v1/chat/completions (or the Responses API for openai) when the
deployment's provider lacked a native Anthropic-messages config, dropping
Anthropic-only features like cache_control and thinking. Some customers run
OpenAI-compatible servers (self-hosted vLLM, DeepSeek's Anthropic endpoint, etc.)
that also natively expose /v1/messages and want the raw Anthropic payload
forwarded untranslated, while keeping provider openai so /v1/chat/completions to
the same deployment stays native.
Opt in per deployment via model_info.supported_endpoints containing
/v1/messages. When present, the gate routes to a generic, provider-agnostic
OpenAILikeAnthropicMessagesConfig that POSTs the Anthropic payload to
{api_base}/v1/messages with Bearer auth, instead of translating. Default
behavior is unchanged. Generalizes and supersedes the hosted_vllm-only,
env-var-toggled PR #28745.
* fix(messages): preserve standard-cased caller headers in native passthrough
The OpenAI-like Anthropic passthrough config only checked for lowercase header
names before injecting Bearer auth, anthropic-version, and content-type
defaults. A caller sending standard-cased Authorization, Anthropic-Version, or
Content-Type was treated as missing those headers, so LiteLLM added duplicate
lowercase variants and overwrote the caller's credential/version at the HTTP
layer. Header presence is now checked case-insensitively and the merge no longer
mutates the caller dict.
Also moves the feature docs out of the main repo (docs live in litellm-docs).
* fix(openai_like/messages): delegate to parent transform and inject anthropic-beta headers
The passthrough config bypassed the parent transform and skipped header beta injection. Both gaps cause native /v1/messages features (context management, advisor tool, fast mode, structured outputs, reasoning_effort, advisor stripping) to silently degrade on opted-in deployments. Reuse the parent's pipeline and call _update_headers_with_anthropic_beta after merging defaults
* fix: normalize anthropic-beta header key case before beta injection
* style: collapse anthropic-beta header normalization to single line
ruff format --check requires the comprehension on one line (it fits within
the 120 char limit); fixes the lint job failure on the bugbot autofix commit
* fix(messages): forward anthropic-beta to native passthrough upstream
The shared anthropic_messages HTTP handler ran update_headers_with_filtered_beta
with the deployment's custom_llm_provider after validate. For the native
/v1/messages passthrough that provider is openai, which has no beta-header
mapping, so every anthropic-beta value (caller-supplied or feature-derived for
speed/context_management/etc.) was stripped to empty before the upstream
request, breaking beta passthrough to the Anthropic-compatible endpoint.
Beta filtering only makes sense on cross-provider translation paths where the
upstream cannot understand Anthropic betas. Gate it on a new
should_filter_anthropic_beta_headers() that defaults to True (bedrock, vertex_ai,
native anthropic unchanged) and is overridden to False by
OpenAILikeAnthropicMessagesConfig, whose upstream is a native Anthropic endpoint,
so betas pass through verbatim.
* chore: remove accidentally committed local QA logs and config
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(mcp): stop one unauthenticated server from emptying the aggregate tools/list
On the aggregate MCP route (/mcp), the gateway fans out to every server the caller can access and
flattens their tools. _fetch_and_filter_server_tools re-raises MCPUpstreamAuthError unconditionally
(added with the OAuth passthrough feature in #28356) so it surfaces a 401 on single-server routes,
but on the aggregate route that exception propagates through the asyncio.gather fan-out and the
outer handler turns it into an empty list. The result: a single delegate/passthrough OAuth server
the user has not authenticated (e.g. a delegate-auth server) zeroes the tools of every other server,
including the ones that resolve fine, so the client connects and sees no tools.
Surface the upstream auth error only when a single server was explicitly targeted (so that route
still drives the upstream OAuth flow); across the aggregate, absorb it to [] for that one server so
the rest still list their tools. This restores the graceful per-server degradation that predated
#28356.
Adds regression tests: the aggregate keeps a healthy server's tools when a sibling raises
MCPUpstreamAuthError, and a single-server listing still surfaces it.
* fix(mcp): decide aggregate vs single-server listing by route scope, not server count
Addresses review: keying the surface-vs-absorb decision off the server count (len(allowed_mcp_servers),
and even len(mcp_servers)) misclassifies an aggregate /mcp request from a key that can access exactly
one server as a targeted single-server listing, so that one server's MCPUpstreamAuthError re-raises and
empties the aggregate again for one-server permission sets.
Use the path-derived single-server scope instead: _mcp_gateway_server_name, set by
_gateway_initialize_instructions_request_scope only when the request path names exactly one upstream
server (/<server>/mcp) and never from client headers, is None on the aggregate route (/mcp) regardless
of how many servers the key can access. Single-server routes still surface the upstream-auth challenge;
the aggregate absorbs it per server.
Adds a regression test that an aggregate request with a single accessible server still absorbs, plus
renames the single-server test to drive the route scope explicitly. The new test fails on the
count-based logic.
* fixing aggregation error
* style(mcp): collapse single-line debug log to satisfy ruff format
Register claude-sonnet-5 across the Anthropic, Bedrock (base + global/us/eu/au/jp
cross-region inference profiles), Vertex AI, and Azure AI cost-map entries in both
the root and bundled-backup model maps, plus BEDROCK_CONVERSE_MODELS and the
setup-wizard provider list.
Sonnet 5 ships with the gen-5 adaptive-thinking profile (adaptive thinking always
on, no extended thinking, effort defaults to high), so the entries mirror the
Fable 5 / Opus 4.8 sampling-param and prefill restrictions rather than the older
Sonnet 4.6 behavior: supports_sampling_params and supports_assistant_prefill are
false while supports_adaptive_thinking, supports_xhigh_reasoning_effort, and
supports_max_reasoning_effort are true. Pricing follows standard Sonnet rates
($3 / $15 per MTok) with the 10% regional premium on the us/eu/au/jp profiles.
Add a reasoning-effort grid entry for the Anthropic direct route and a regression
test pinning pricing, capabilities, regional premiums, backup parity, and bare-name
provider resolution.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds `!` prefix negation to tag-based routing so callers can exclude deployments by exact tag value without enumerating every allowed alternative. `!provider:anthropic` removes all deployments tagged exactly `provider:anthropic` before routing, and positive and negation tags compose. Matching is exact literal membership (frozenset intersection), so there is no regex or ReDoS surface for client-supplied tags. Ban-only requests that carry only negation tags stay within the default pool, mirroring untagged-request semantics so callers can't use negation to escape it. Fallback chains keep working because get_deployments_for_tag runs on each routing hop
Copy of #31680; implementation credit to @deepanshululla
Co-authored-by: deepanshululla <15312873+deepanshululla@users.noreply.github.com>
* feat(guardrails): expose streaming knobs on generic_guardrail_api
Wire streaming_end_of_stream_only and streaming_sampling_rate through
optional params, initialize_guardrail, and get_config_model so the
generic guardrail API participates in UnifiedLLMGuardrails streaming
checks with configurable cadence and end-of-stream-only mode.
* fix(guardrails): use builtin type[] in get_config_model return
Avoids a new UP006 violation that tripped the ruff strict-rule budget
gate on the PR lint job.
* fix(guardrails): default optional streaming knobs to None
Non-None Pydantic defaults on GenericGuardrailAPIOptionalParams made
_get_config_value treat unset nested fields as explicit values, which
shadowed top-level litellm_params streaming flags whenever any other
optional_params key was present. Real defaults stay in the constructor.
* fix(guardrails): address review nits on generic_guardrail_api streaming
Validate streaming_sampling_rate >= 1 in the constructor and Pydantic
optional_params (ge=1), and add /v1/responses streaming coverage through
the unified post-call hook so Responses API usage is exercised alongside
chat completions.
* fix(guardrails): read nested streaming config from dict optional_params
Guardrail API/UI delivers optional_params as a plain dict, so getattr was
silently ignoring streaming_sampling_rate and streaming_end_of_stream_only.
Handle both dict and model shapes in _get_config_value with regression tests.
* fix(guardrails): clear ruff findings in generic_guardrail_api tests/types
* style(guardrails): ruff format generic_guardrail_api modules
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
Guard the per-request CPU cost of the chat completion, MCP tool and A2A
message transforms against regressions on every commit. All benchmarks are
pure in-process work with no network I/O so they stay deterministic under
CodSpeed's simulation mode, and they import under the base dependency set the
benchmark job installs.
Inference covers the full SDK overhead via mock_response (simple, multi-turn,
tools, streaming) plus convert_to_model_response_object as a deterministic
anchor. MCP covers the client-side tool translation and the proxy server-side
tool-name prefix round-trip. A2A covers the client request/response transforms
and the proxy server-ingress message conversion.
Adds the mcp and a2a-sdk packages to the benchmark run since those transform
modules need them, and broadens the workflow triggers to litellm_internal_staging
so the internal branch flow is benchmarked too.
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
The proxy auth path calls phase_span() and seed_request_identity() in
litellm/integrations/otel/runtime.py on every request, each doing a
try/except lazy import of litellm.integrations.otel.logger. When the
OpenTelemetry SDK is not installed (the default), that import raises, and
CPython never caches a failed import, so every request re-scanned sys.path
and contended on the import lock. At 750 concurrent users this cost about
12% throughput versus v1.85.0.
Resolve the hooks once and cache the outcome, absence included, with
functools.cache, so the import is attempted a single time instead of per
request. Throughput returns to the v1.85.0 baseline.
* feat(proxy): type Customer Management response_model for OpenAPI coverage
Add response_model to the five remaining untyped /customer operations
(block, unblock, new, update, delete) so the generated OpenAPI schema
documents a concrete response body. new/update reuse the canonical
LiteLLM_EndUserTable (matching info/list); block, unblock, and delete
get small dedicated models in
litellm/types/proxy/management_endpoints/customer_endpoints.py.
Together with the already-typed info/list/daily-activity routes this
brings the Customer Management group to full response_model coverage.
Regression tests assert each public /customer/* route declares the
expected response_model and that /customer/new surfaces a typed schema
in app.openapi(), so dropping a response_model fails CI.
* fix(proxy): keep budget_id in typed customer responses
Address review feedback on the Customer Management response_model typing.
Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new
and /customer/update silently drops fields the raw Prisma model_dump()
echoed. Checking the schema, budget_id is the only such scalar column that
was missing from the Pydantic model (created_at/updated_at/tpm_limit do not
exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable.
This restores budget_id on new/update and also fixes the pre-existing gap
where /customer/info and /customer/list (already typed) dropped it, which
the UI Customer type expects. A regression test pins budget_id surviving the
response_model filter on /customer/update.
Also document UnblockUsersResponse.blocked_users via a Field description: it
holds the users that remain blocked after the call. The key name predates
this PR and is kept to avoid a backwards-incompatible rename on a beta route.
* fix(proxy): keep nested budget fields in customer responses
response_model=LiteLLM_EndUserTable nests the budget as the narrow write
allowlist LiteLLM_BudgetTable, which silently drops the server-managed
fields the customer endpoints used to return (budget_reset_at, created_at).
Introduce CustomerResponse, a thin response model that nests
LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on
/customer/new, /customer/update, /customer/info and /customer/list. list
also builds CustomerResponse so its budget isn't narrowed at construction
time. created_by/updated_at/updated_by remain omitted, matching how budgets
are returned elsewhere.
The shared LiteLLM_EndUserTable is left untouched: it's constructed in many
places that pass narrow budget instances, and pydantic v2 won't coerce a
budget instance into a wider nested model. Typing only at the response
boundary (where the handler hands FastAPI a dict) sidesteps that. A
regression test pins budget_reset_at + created_at through the filter and
asserts the internal audit fields stay out.
* test(proxy): add golden-master characterization tests for customer responses
Lock the exact JSON body each customer-object endpoint (info/list/new/update)
and delete return today, so the upcoming type-safety refactor of the handlers
is only allowed to land if it reproduces these byte for byte. Pins null-field
inclusion, the nested budget shape (server fields kept, audit fields dropped),
and object_permission reverse-relation stripping. Green against current code.
* refactor(proxy): make the customer response flow type-safe
Replace the untyped dict + bolt-on response_model pattern on the customer
object endpoints with explicit typed construction. A single mapper,
_to_customer_response, validates a DB row into CustomerResponse at one
Any -> typed seam; new/update/info/list now return it (or a list of it) and
carry real -> CustomerResponse / -> List[CustomerResponse] return
annotations, and delete returns DeleteCustomersResponse. basedpyright now
verifies the handlers' return shapes instead of a runtime filter doing it
silently.
This also deletes the four copy-pasted object_permission reverse-relation
cleanup loops: pydantic's extra=ignore drops those undeclared fields during
validation, so the loops were dead code (proven by the golden-master tests,
which stay byte-for-byte green). basedpyright errors on the file drop from
140 to 116, all from removed dict plumbing.
CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits
the existing validators/config unchanged (behavior preservation); only the
nested budget type is widened.
* refactor(proxy): annotate customer response mapper param as BaseModel
Address review nit: the mapper's untyped `record` added an ANN001 violation.
The incoming rows are pydantic v2 models, so type the param as BaseModel
rather than object (object has no model_dump, which would just move the
problem to basedpyright). This clears the ANN001 and also drops three
basedpyright unknown-type violations the untyped param was adding.
* style(test): ruff format customer endpoint tests
* test(proxy): give customer budget test update mocks a valid model_dump
The type-safe response refactor validates the update result via
_to_customer_response (CustomerResponse.model_validate(record.model_dump())).
These budget tests mocked the end-user update to return a bare MagicMock,
so model_dump() yielded a MagicMock that fails validation. Give each update
mock a minimal valid dict; the tests assert on the prisma calls, not the body.
* chore(ui): regenerate API types from proxy OpenAPI spec
* fix(ui): make generated API types stable across Python versions
Python 3.13 strips a docstring's common leading indentation at compile
time while 3.12 keeps it, so app.openapi() emits differently-indented
description strings depending on the interpreter. The dashboard type
generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted
and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed
Normalize every description through inspect.cleandoc in the spec dump so
the output is identical regardless of interpreter, then regenerate
litellm_overhead_latency_metric only covers the SDK wrapper window and excludes
proxy guardrails. Add a histogram that sums SDK overhead plus pre/post-call
guardrail durations (during-call excluded since it runs concurrently with the LLM
call, alongside logging_only and MCP modes that never block the response),
recorded next to the existing overhead metric with the same labels and buckets.
No existing metric's value is changed.
* chore: remove _experimental/out
* fix(ci): recreate _experimental/out before copying UI build output
The build scripts cp the Next.js output into litellm/proxy/_experimental/out,
which was removed from git. cp failed because the target directory no longer
existed; mkdir -p recreates it before the copy.
* fix(proxy): make UI serving resilient to a missing _experimental/out
Removing the committed UI export means the source/test tree no longer
ships litellm/proxy/_experimental/out. Three things assumed it was always
present and broke once it was gone:
- get_favicon hard-coded the built favicon path and 404'd without it; it
now falls back to the bundled swagger/favicon.ico
- the /_next and /ui static mounts raised at construction when the export
was absent, so the whole UI-setup block was swallowed and no mounts
registered; they now use check_dir=False
- _restructure_ui_html_files was a nested function only exposed as a
module attribute when that block happened to succeed; it is now a real
module-level function
test_admin_ui_export_serves_nested_extensionless_routes validated the
committed artifact, whose premise this PR removes; it now drives the same
MCP OAuth callback restructure guarantee through a synthetic export.
* chore(greptile): ignore generated _experimental/out so review fits the file limit
* Revert "chore(greptile): ignore generated _experimental/out so review fits the file limit"
ignorePatterns is applied after Greptile counts the files changed, so it
does not bring the diff under the file limit; the config had no effect.
* fix(anthropic): drop unsignable thinking blocks and allow null signature in logging (LIT-4007)
Open-source reasoning models (DeepSeek-R1 and distills, Qwen3/QwQ, IBM
Granite 3.2 via vLLM/Ollama/OpenRouter/DeepSeek) return reasoning_content
with no Anthropic-style signature, which LiteLLM represents as a thinking
block with a null signature.
Two failures resulted. First, ChatCompletionThinkingBlock.signature was a
required str, so building the StandardLoggingObject raised a ValidationError
on signature=None and the success log record was silently dropped while the
request still returned 200; relaxing it to Optional[str] lets the log build.
Second, replaying such a turn to a real Anthropic model forwarded the
null-signature thinking block unchanged and Anthropic rejected it with
400 thinking.signature.str; since Anthropic verifies the signature
cryptographically, a null, empty, or missing signature cannot be repaired,
so anthropic_messages_pt now drops the unsignable thinking block while
preserving the assistant text and keeping genuinely signed blocks.
* style: use builtin generics for thinking-block filter helpers
* fix(ui): regenerate schema.d.ts for nullable thinking-block signature
Reasoning-token cost was computed but folded into output_cost, and cache
cost was only populated from the top-level cache_read_input_tokens attribute,
so providers that report cache tokens under prompt_tokens_details (Gemini,
OpenAI, Vertex) never got a cache breakdown.
Adds a provider-agnostic get_token_type_cost_breakdown helper that derives
reasoning, cache-read and cache-creation cost from the normalized usage object
using the same rate-resolution primitives as the total-cost path, so the
breakdown reconciles with the totals. completion_cost stores these via
set_cost_breakdown, surfacing reasoning_cost (new), cache_read_cost and
cache_creation_cost in StandardLoggingPayload.cost_breakdown and the spend logs.
Co-authored-by: Kunal Nayyar <48790070+kunal2002@users.noreply.github.com>
The MCP gateway authenticated to upstream OAuth token endpoints only with
client_secret_post (client_secret placed in the POST body). Providers that
require HTTP Basic client authentication (client_secret_basic, the OIDC
default) reject that with invalid_client, which surfaced as a 500 on the
/<server>/token exchange and broke both the initial authorization_code
exchange and refresh.
Add a per-server token_endpoint_auth_method ("client_secret_basic" |
"client_secret_post") and a single helper that builds the right headers and
body for the configured method, then route every upstream token-endpoint POST
through it: the inbound exchange and refresh in discoverable_endpoints, the v1
per-user refresh in db, the v2 authorization_code refresher, the M2M
client_credentials fetch, and the RFC 8693 token exchange. The default stays
client_secret_post so existing servers are unaffected; basic sends
Authorization: Basic base64(form-urlencode(client_id):form-urlencode(secret))
per RFC 6749 section 2.3.1 and omits the secret from the body.
client_secret_basic is a confidential-client method, so a server configured for
it with a missing client_id/secret raises rather than silently downgrading to a
body request (no-silent-fallback); the inbound endpoint maps that to a 400 and
the refresh paths to a failed-refresh / needs-reauth. A secretless client_id
under the default method stays valid for public clients authenticating with PKCE.
Resolves LIT-4091
* fix(proxy): emit x-litellm-response-cost header on /messages and /generateContent (LIT-4076)
The Anthropic /v1/messages and Google native :generateContent routes return
TypedDict results (AnthropicMessagesResponse, GenerateContentResponseBody) that
are plain dicts at runtime and cannot hold a _hidden_params attribute. The cost
is computed by update_response_metadata, but ResponseMetadata.apply() only
persists _hidden_params back when the result object has that attribute, so for
those two routes the computed response_cost was dropped. The non-streaming
header build in base_process_llm_request then saw an empty response_cost and
get_custom_headers filtered the x-litellm-response-cost header out, even though
the other x-litellm-* headers still appeared.
The non-streaming success path now recovers the cost from the logging object
when the response cannot carry _hidden_params, preferring the value already
stored in model_call_details and recomputing from the same calculator only when
it has not been stored yet. Object responses (ModelResponse, ResponsesAPIResponse)
keep their existing behavior, so chat/completions, /responses, and the Anthropic
error path that intentionally emits a zero cost are unaffected. Streaming stays
out of scope because the header is emitted at stream start, before the cost is
known.
* fix(proxy): also recover response cost header for /generateContent responses with _hidden_params (LIT-4076)
* fix(proxy): compute generateContent response cost synchronously so cost header is emitted (LIT-4076)
* fix(lint): suppress BLE001 on generate_content cost normalization guard
The defensive blind except keeps cost normalization from ever breaking the
response path; mark it noqa so it does not breach the strict-rule budget.
* fix(bedrock): drop unmappable Responses tools instead of failing the request (LIT-3858)
When an OpenAI Responses request is routed to a Bedrock Converse Anthropic model,
litellm translates the tools array into Bedrock toolConfig. Responses built-in tool
types beyond function (web_search, image_generation, namespace, tool_search, custom)
have no Bedrock equivalent, and previously caused two failures.
A web_search tool is derived into a web_search_options param. Bedrock Anthropic
models do not list web_search_options in get_supported_openai_params, so the request
raised UnsupportedParamsError (HTTP 400) even though it never needed web search. The
derived param is now dropped on the Bedrock chat-completion bridge for models that
do not support it, scoped to Bedrock so other providers are untouched and without
requiring drop_params. Nova still keeps it since it maps to a nova_grounding systemTool.
The remaining non-function tools reached _bedrock_tools_pt and were emitted as junk
litellm_unnamed_tool_N toolSpecs with empty schemas, polluting toolConfig with tools
the model could hallucinate calls to. They are now dropped because they carry neither
an OpenAI function nor an Anthropic input_schema, while mappable function and
input_schema tools survive untouched.
* refactor(responses): drop derived web_search_options via provider config
Greptile flagged that the LIT-3858 fix put Bedrock-specific logic in the
generic Responses->Chat Completion bridge: it imported AmazonConverseConfig
and branched on custom_llm_provider.startswith("bedrock").
Read web_search_options support from each provider's own
get_supported_openai_params instead, so the bridge stays provider-agnostic
and Bedrock capability knowledge lives in the Bedrock config that already
owns it. Behavior is unchanged for the cases the PR targeted (Bedrock
Anthropic drops, Bedrock Nova and OpenAI keep) and now generalizes correctly
to any provider whose config does not support the derived param.
Add a Cohere regression test proving the drop is provider-agnostic; it fails
under the old bedrock-only check and passes now.
* fix(responses): drop derived web_search_options for bedrock_converse alias
Greptile/T-Rex caught that the provider-agnostic drop regressed the
bedrock_converse route: get_supported_openai_params did not map the
bedrock_converse alias (only "bedrock"), so it returned None (unmapped) and
the derived web_search_options was forwarded for
model="bedrock/converse/us.anthropic.claude-sonnet-4-6",
custom_llm_provider="bedrock_converse" instead of being dropped. The previous
startswith("bedrock") check happened to match the alias.
Map bedrock_converse through AmazonConverseConfig in get_supported_openai_params,
mirroring the existing ["bedrock", "bedrock_converse"] pairing in
_strip_model_name. Add regression tests at both levels: the alias now drops the
derived param for Anthropic Converse models, still keeps it for Nova, and the
helper resolves identically to "bedrock".
* fix: skip health check for semantic auto_router deployments
auto_router/<name> deployments are semantic meta-routers that select among
real LLM deployments at request time. They have no LLM endpoint to probe.
The health check was passing model=auto_router/router_1 to get_llm_provider(),
which raised BadRequestError: "Unmapped LLM provider for this endpoint" because
auto_router is not a real LLM provider, causing these deployments to always
appear unhealthy and curl requests to hang.
Detect semantic auto_router deployments in _run_model_health_check and return
{} (healthy) without calling litellm.ahealth_check. Sub-strategies
(complexity_router, adaptive_router, quality_router) are excluded from this
fast path and continue to be health-checked normally.
* ci: trigger circleci
The Model Armor guardrail only sent text extracted from user messages to
sanitizeUserPrompt, so harmful content inside attached PDFs, Office docs,
and CSVs reached the LLM unscanned. A file-only message had no extractable
text, so the pre-call and moderation hooks returned early and the document
was never submitted to Model Armor at all.
Wire inline document/file scanning into async_pre_call_hook and
async_moderation_hook. extract_file_attachments walks message content blocks
(OpenAI type:file file_data and Anthropic type:document source), decodes the
base64 bytes, maps the MIME type to a Model Armor byteDataType, and skips
remote URLs, bare file_id references, oversize files past the 4 MB limit, and
unsupported types. Each attachment is sent through the byte API and a
MATCH_FOUND blocks the request before it reaches the LLM.
Resolves LIT-4084
* fix(passthrough): drop top-level additional_drop_params on /v1/messages
On the Anthropic Messages pass-through path, additional_drop_params only
stripped nested dotted paths, so plain top-level keys like `thinking` and
`context_management` were forwarded to the provider. Bedrock rejects these
with "Extra inputs are not permitted", returning a 400 to Claude App/CLI
even when the user configured `additional_drop_params: ["thinking"]`.
delete_nested_value already handles plain top-level fields, so route every
drop param through it and remove the nested-only filter. Fixes#25931.
* fix(passthrough): drop thinking for bedrock inference-profile ARNs on /v1/messages
Opaque Bedrock Application Inference Profile ARNs contain neither "anthropic"
nor "claude", so is_anthropic_claude_model returned False and the thinking
param was rewritten to reasoning_effort before additional_drop_params ran.
That made additional_drop_params: ["thinking"] a no-op for the converse-ARN
form, and the Bedrock Converse transform re-expanded reasoning_effort back into
additionalModelRequestFields.thinking, so the request 400'd.
Extend the thinking-translation gates to also accept bedrock ARNs via the
existing is_bedrock_arn_model helper, mirroring the cache_control path, so
thinking is preserved as thinking and additional_drop_params can drop it.
* fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint
The OAuth token endpoint stored a user's per-server token under the identity
returned by _extract_user_id_from_request, which read only the Authorization
header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup
with no model_type rehydration and no DB fallback. That silently returned None
in two common cases on a multi-replica gateway: the LiteLLM key arrives on
x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code
send) rather than Authorization, and a cross-replica cache hit deserializes to
a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When
it returned None the token was not persisted.
This was survivable until the authorization_code v2 migration began stripping
the caller's Authorization for migrated per-user OAuth servers and routing the
preemptive 401 existence check through the stored token, so a persist miss now
hard-fails: the egress challenges with 401 on every reconnect (the client sees
"rejected them on reconnect" or a successful connect with zero tools).
Resolve identity through get_key_object, the canonical resolver that reads the
cache with model_type and falls back to the DB, and accept the key from
x-litellm-api-key as well as Authorization. The silent persist skip is now a
warning. The caller-Authorization stripping stays as is, since reinstating it
would reopen the cross-user credential override it was added to prevent.
* fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity
The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the
blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So
a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with
that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token
for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys
(blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed
that accidental shield.
Validate the resolved key before trusting its identity: return None when blocked or expired, so the
upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row.
Regression tests cover the blocked and expired cases and fail without the guard.
* fix(proxy): count only active users toward license seat limit
SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat
Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected
* fix(proxy): floor billable user count at zero
count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call
Addresses Greptile P1 on the PR
* refactor(proxy): count teams via TeamRepository in available_users
* style: ruff format changed files at line-length 120
* fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads
PR #31036 switched the vertex batch file upload from a single GCS media
upload to a chunked resumable session. The resumable path sends the body as
many sequential PUTs, each waiting a full round-trip to GCS before the next,
so a multi-GB upload accumulates hundreds of round-trips and overruns the
client/load-balancer request timeout, surfacing as 499s (client closed
connection) on files as small as 500MB. This was a regression from the
last-known-good commit, where the upload completed as one continuous request.
Revert the batch upload to a single uploadType=media request, but stage the
transformed payload to a temp file first so peak memory stays bounded (the
goal of the resumable rewrite) without the per-chunk round-trips. The temp
file is closed deterministically (TemporaryFile unlinks on close), not left
to the GC. The now-unused resumable chunked-upload plumbing is removed.
Also swap the per-row transform's stdlib json for orjson (parse + serialize),
which is ~4x faster on this hot path; the streaming body now emits compact
orjson bytes.
The request stays synchronous, so the returned file object is real and
POST /v1/batches keeps working immediately against the uploaded object.
Tests: single media request carries the whole payload with a real
Content-Length (no chunked transfer-encoding); failed upload raises; the
staged temp file is closed deterministically; byte-for-byte transform parity.
* test(vertex_ai/files): mock single media upload POST instead of removed resumable method
test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type.
* fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports
Forward the per-request timeout through _stage_and_upload_media /
_astage_and_upload_media to the GCS POST. Every other upload branch forwards
it; the new media path was dropping it, so a caller-provided timeout was
silently ignored (the files path passes 600s by default, but a custom
request_timeout would not have reached this upload). Regression test asserts
the resolved timeout reaches the request (mutation-verified).
Revert the orjson swap in the batch transform: importing orjson at module load
in this core-path file broke `import litellm` on environments without orjson
(the Windows import test). Back to stdlib json; the upload leg dominates large
uploads anyway, so the transform-side win was marginal.
Fix import ordering in llm_http_handler.py (I001) introduced by the new imports.
* fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file
Addresses a disk-exhaustion concern: staging the full transformed batch body to
a local temp file before the GCS request meant an authenticated user could fill
the proxy's temp volume with large concurrent uploads (on top of Starlette's
input spool).
GCS's simple/media upload accepts chunked transfer-encoding, so stream the
transform straight to the single media request instead. Each block is produced
on a worker thread (the transform never runs on the event loop) and sent
chunked, so the body is neither buffered in memory nor written to disk, and the
upload is still one continuous request (no per-chunk round-trips, no 499). Drops
the temp-file staging, the tempfile/IO imports, and Content-Length computation.
Regression test asserts the upload streams (chunked transfer-encoding, no
Content-Length) and creates no temp file; mutation-verified that reintroducing
staging fails it.
Mirror the scalar `max_budget` guard in `_common_key_generation_helper`
for the per-window check: a CLI session token caller (carrying
`max_budget=None`) cannot set `budget_limits` on a personal key. Pass
`team_table` into the helper so it can detect the personal-key shape;
reject before the `delegation_ceiling is None` early return.
Four new regression tests cover the personal-key reject, the team-key
happy path, the team-key over-team-budget path, and the proxy-admin
exemption.
Enforce that every `budget_limits[*].max_budget` is a finite number;
applies to every caller including proxy admin and runs before the
role / ceiling checks. Six parametrized regression tests cover NaN /
+inf / -inf for both non-admin and admin callers.
increment_spend_counters was parallelized in #31578, but the dominant
per-request cost under high concurrency is the pre-call budget enforcement
in common_checks, which still ran a Redis-first get_current_spend per scope
(team, team windows, key windows, org, tag, user, team member, end user)
one sequential await after another inside the auth span.
The per-scope reads target distinct counter keys with no cross-scope
ordering dependency, so they now run concurrently under asyncio.gather.
Key metadata.tags injection still runs before the gather so the tag budget
check sees it, and every scope settles before the first error in
scope-priority order propagates, preserving the previous rejection semantics.
Resolves LIT-4090
/key/generate validated the caller's delegation ceiling against
data.max_budget only. The per-window entries in data.budget_limits
bypassed the check, so a non-admin caller could mint a key whose
1-day window vastly exceeded their own max_budget. The data.permissions
dict also went unvalidated for non-admin callers, so they could
self-grant capabilities like allow_pii_controls (and on Enterprise,
get_spend_routes).
Both gates now live in _common_key_generation_helper, covering
/key/generate and /key/service-account/generate. The existing empty
{} default on permissions still passes for non-admin callers.
* fix(databricks): split parallel tool calls so each tool message follows tool_calls
Databricks OpenAI-compatible serving (e.g. GPT models) 400s with "messages with
role 'tool' must be a response to a preceeding message with 'tool_calls'" when an
assistant turn makes parallel tool calls. LiteLLM faithfully sends one assistant
message holding all tool_calls followed by one 'tool' message per result, so every
result after the first is preceded by another 'tool' message rather than the
assistant tool_calls message, which Databricks rejects.
Re-emit each result immediately after an assistant message that carries only its
matching tool_call, turning assistant(tool_calls=[A, B]), tool(A), tool(B) into
assistant(tool_calls=[A]), tool(A), assistant(tool_calls=[B]), tool(B). The
rewrite is a no-op when the turn is already valid (single call), the group is
incomplete, or ids don't line up, so no tool call is ever dropped. Scoped to
non-Claude models, matching the existing OpenAI-shaped transformation path.
* style(databricks): use builtin list generics in parallel tool-call split
Switch the List[...] annotations introduced by _split_parallel_tool_calls
to lowercase list[...] so the UP006 strict-rule budget stays within its
ceiling.
* feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents
Bump a2a-sdk to 1.x and wire send/stream through compat conversions so the proxy accepts A2A 1.0 JSON-RPC while preserving 0.3 wire clients.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add user controlled protocol version in agents
* Fix exeception mapping
* Fix a2a base url
* Add e2e test for a2a
* Fix lint
* Fix lint
* fix(a2a): harden card version detection and header isolation coverage
Use protocolVersion when inferring agent card wire format, assert distinct httpx cache keys in the header-isolation test, and suppress targeted basedpyright errors for optional SDK imports.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(a2a): suppress reportArgumentType for SDK compat types and fix streaming trace ID
- Add pyright: ignore[reportArgumentType] to SendMessageSuccessResponse id= and
result= args in _send_message, and SendStreamingMessageResponse root= in
_stream_messages, where a2a-sdk compat types diverge from basedpyright's
inferred signature, reducing the reportArgumentType count back within budget.
- Fix streaming trace ID in astream_a2a_message to use str(request.id) when
available instead of always generating a new uuid4(), restoring JSON-RPC
request-ID correlation for observability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style(a2a): expand SendStreamingMessageResponse for black formatting
Move pyright: ignore comment to the root= argument line so Black
accepts the expanded multi-line form.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(a2a): fix 2 reportArgumentType errors without suppression
- main.py: narrow logging_obj from object|None to Optional[Logging] via
isinstance check before A2AStreamingIterator call, fixing the
"Logging | object" argument type mismatch at line 699.
- a2a_endpoints.py: extract response_dict with explicit isinstance(dict)
guard before passing to normalize_jsonrpc_response, fixing the
"LLMResponseTypes | dict[str, Any]" type mismatch at line 835.
- Remove spurious pyright: ignore comments added in previous commits that
were not suppressing the actual errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(a2a): rewrite upstream URL for 1.0 agent cards in getAuthenticatedExtendedCard
1.0 upstream agent cards store the endpoint URL in supportedInterfaces[0].url
rather than a top-level url field. The previous guard only rewrote url when
it existed at the top level, so after normalize_agent_card lowered a 1.0 card
to 0.3 the upstream internal address leaked into the url field of the 0.3
response.
Fix: rewrite both url and supportedInterfaces[0].url to the proxy address
before calling normalize_agent_card, ensuring the upstream address is never
visible to downstream clients regardless of the upstream card's wire format.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: extend _served_version to all PascalCase methods; add direct httpx-client isolation proof
- _served_version now checks `_PASCAL_TO_WIRE` membership instead of two
hardcoded names, so GetTask/CancelTask/etc. are promoted to 1.0 wire format
alongside SendMessage — prevents mixed wire formats mid-session
- test_create_a2a_client_uses_fresh_httpx_client now asserts
a2a_client_a._litellm_httpx_client is not a2a_client_b._litellm_httpx_client
(direct proof that header bleed cannot occur), in addition to the cache-key
inequality check
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: id:0 silently dropped in version_convert; explicit continue in stream retry
- version_convert.py: replace `request_id or ""` with
`str(request_id) if request_id is not None else ""` in both
_send_result_to and _stream_result_to; id=0 is valid JSON-RPC and
must not be coerced to "" which breaks response correlation
- main.py: add explicit `continue` after the A2ALocalhostURLError retry
in _execute_a2a_stream_with_retry so the control flow (retry → next
iteration → stream_succeeded guard) is unambiguous
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: preserve a2a retry and discovery card urls
* Fix black
* Fix test
* fix(a2a): avoid KeyError in discovery log after 0.3→1.0 card normalization
When a 0.3-style agent card is normalized to 1.0, the top-level url key is
replaced by supportedInterfaces; log the already-computed proxy_url instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(a2a): preserve taskId when lowering push notification config set params
Flatten 1.x create envelope fields before parsing into TaskPushNotificationConfig so 1.0 clients forwarding to 0.3 upstream keep taskId and config.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(a2a): ignore unknown fields in message/send proto fallback
ParseDict in _build_message_send_params now matches other inbound paths so 1.0 clients with extra proto fields are not rejected with -32602.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(a2a): normalize tasks/list params and response across protocol versions
Convert list task entries on the response path and lower ListTasksRequest params including status filters when forwarding 1.0 clients to 0.3 upstream.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(a2a): avoid reportArgumentType in _lower_list_tasks_params; use local var instead of _parse return
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(a2a): drop private SDK symbol in tasks/list status lowering
_lower_list_tasks_params imported _CORE_TO_COMPAT_TASK_STATE, a private
a2a-sdk symbol that could disappear on a patch release and silently break
status-filter lowering. Derive the 0.3 wire string from the public
protobuf enum name instead (TASK_STATE_<NAME> maps to the 0.3 value once
the prefix is dropped and underscores become dashes) and validate the
result against the 0.3 TaskState enum's own values via a fully-typed pure
helper. Behavior is unchanged for every state; unspecified or unrecognized
states still drop the filter. Adds parametrized regression tests covering
dashed wire values (input-required, auth-required) and the unspecified drop.
* fix(a2a): drop redundant push-notification envelope key; unify MessageToDict import
_flatten_create_push_notification_params used `config or pushNotificationConfig`,
which short-circuits so a co-present pushNotificationConfig key was never popped and
leaked into the flattened params. Pop both keys unconditionally and prefer config
when present. Adds a regression test on the helper that fails on the old leak.
Also import MessageToDict from a2a.compat.v0_3.conversions in _lower_list_tasks_params
to match every other conversion helper in the module instead of pulling it straight
from google.protobuf.json_format.
* fix(a2a): reject invalid message/stream params early with -32602
_handle_stream_message built MessageSendParams lazily inside the
stream_response() generator, so malformed 1.0 params surfaced as a generic
-32603 after the 200 status line was already committed. The non-streaming
path validates up front and returns -32602 (Invalid params). Validate
eagerly before returning the StreamingResponse and emit -32602 on failure
so both paths reject malformed params identically. Adds a regression test
asserting the streamed error code is -32602.
* fix(a2a): raise clear error when non-streaming send ends on an update event
_send_message fed the SDK iterator's last event straight into
SendMessageSuccessResponse, whose result only accepts Message or Task. A
non-standard upstream whose final event is a TaskStatusUpdateEvent or
TaskArtifactUpdateEvent made the response construction raise an opaque
pydantic ValidationError. Guard the converted result and raise a clear
RuntimeError instead, consistent with the no-response guard above it.
Adds regression tests for the Message happy path and the update-event
rejection via an injected fake client.
* test(a2a): lock in clean merged agent-card URL without PROXY_BASE_URL
Regression coverage proving _build_merged_agent_card produces no double
slash in supportedInterfaces[0].url when PROXY_BASE_URL is unset and
request.base_url carries a trailing slash. get_custom_url routes through
join_paths, which rstrips the base, so the f-string join stays clean.
* style(a2a): modernize type annotations to satisfy strict ruff budget
After merging the black->ruff-format migration from base, the A2A files
owned by this PR still used Optional[X]/quoted annotations that pushed
UP037/UP045 over their lowered ceilings. Convert to X | None, drop the
now-unnecessary quoted local annotation in _send_message, and remove the
imports left unused by the rewrite. Type semantics are unchanged.
* style(a2a): type a2a_endpoints dict params as dict[str, Any]
The merge with the formatter-migration baseline tightened the
reportUnknownArgumentType ceiling; bare dict annotations made every value
Unknown and pushed the codebase total over cap. Annotate the JSON-RPC
params, body, metadata, and litellm_params dicts as dict[str, Any] so
their values are typed, dropping the unknown-argument count back under the
ceiling. No behavior change.
* fix(a2a): guard localhost retry against a missing agent card
handle_a2a_localhost_retry rewrote the card URL and called create_client
with whatever agent_card it received. The caller resolves the card from
the SDK client (Optional), so a None card reached set_agent_card_url and
create_client, surfacing an opaque SDK error instead of a clear one. Add
an early RuntimeError guard mirroring the httpx-client check, drop the now
always-true card None-check on the stash line, and cover it with a
regression test.
* style(a2a): disable reportUnknownArgumentType in a2a-sdk boundary modules
The lint env type-checks without the optional a2a-sdk/protobuf installed, so
every call into the protobuf-generated compat conversions counts as an
Unknown-typed argument and the new A2A code pushed the codebase
reportUnknownArgumentType total over its ceiling. These three modules are
the A2A SDK boundary; turn the rule off file-wide with a documented reason
instead of scattering dozens of per-line ignores across every SDK call.
* fix(a2a): tolerate unknown fields when lowering 1.0->0.3; align streaming trace id
Two issues greptile flagged:
version_convert: the 1.0->0.3 lowering paths (_send_result_to, _task_to,
_stream_result_to) called ParseDict without ignore_unknown_fields=True, so a
1.0 upstream response carrying vendor extensions raised and best-effort fell
back to passing the un-lowered 1.0 shape to a 0.3 client. Set the flag to match
the agent-card path and every inbound path; unknown fields are now dropped and
the result is correctly lowered.
main.py: asend_message_streaming derived X-LiteLLM-Trace-Id from the JSON-RPC
request id, unlike asend_message which uses the logging object's
litellm_trace_id. Prefer the logging trace id (then request id, then a uuid) so
streamed and non-streamed calls correlate under the same trace.
Adds regression tests for both, including the stream-event lowering path.
* style(a2a): apply ruff format to a2a protocol and proxy modules
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(batches): add 1:1 test file scaffold for batches component paths
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add harness test for create batch endpoint
* Add retrieve endpoint harness tests
* Add list endpoint harness tests
* Add cancel endpoint harness tests
* Add cancel endpoint harness tests
* Add test for litellm/batches/main.py
* Add test for litellm/tests/test_litellm/batches/test_batch_utils.py
* Add handler and transformation tests for all providers
* Fix: run batches tests in cicd
* fix(tests): remove azure/__init__.py that shadowed azure namespace package
Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to
insert tests/test_litellm/llms/ into sys.path[0], making our empty
azure/ dir shadow the real azure-identity namespace package. Any test
that patched azure.identity.* would then fail with AttributeError.
* style(tests): apply ruff format to test_batch_utils.py
Base migrated the formatter from black to ruff format (#31317); reformat the
batches scaffold test file to match.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Keep only video test files and CI workflow entries; drop unrelated
production code and non-video test changes from this branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
The error string is already produced by the f-string interpolation; the
trailing .format() call on it was redundant. Add a regression test that
the message renders the model name verbatim.
* feat: declarative fallback generalizations for unknown models
Unknown or newly-released models previously degraded (missed cost lookups,
wrong supports_* flags, broken provider routing) and were patched with one-off
hardcoded regexes scattered across Python. This adds a single data-driven source
of truth: a fallback_generalizations block in model_prices_and_context_window.json
holding ordered, case-insensitive regex rules that map a model name to the
metadata to apply when it has no exact entry.
A new fallback_generalizations module owns the rules and a compiled-regex cache
that is built once and invalidated on reload, so the O(n) scan runs only on a
cache miss. get_llm_provider now routes an otherwise-unknown model via the first
matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and
_matches_claude_model_pattern. _get_model_info_helper falls back to a matching
rule's model_info after the exact lookups miss, so get_model_info and the
supports_* helpers resolve unknown models from the same rule. get_model_cost_map
extracts the block out of the returned map, and the integrity check now counts
real model entries (excluding reserved meta keys) so the new key cannot mask a
genuinely shrunk upstream file.
The top level of the file stays a flat map of models so existing litellm releases
that fetch the live file keep working and keep receiving updates; the block ships
in both the root file and the bundled backup. An anthropic-claude rule reproduces
the old future-claude routing and additionally supplies capability flags and a
context window
https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo
* refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations
Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring
matchers with a single _claude_version_at_least predicate that parses the Claude
family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x
without a code change (the old matchers missed 4.8 entirely) while keeping an explicit
supports_adaptive_thinking flag authoritative when present, so there is one source of
truth. The two direct call sites in the chat transformation now route through
_is_adaptive_thinking_model instead of the deleted matchers.
Also address review feedback on the generalizations module: return a copy of the
matched model_info so a future caller cannot mutate the compiled-rule cache, document
that patterns are matched with re.search and must anchor with ^ and $, and reindent
the fallback_generalizations block to the file's 2-space style in both JSON files.
https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo
* fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse
supports_adaptive_thinking shipped in the model cost map but was never declared
on ModelInfo nor copied during construction, so get_model_info (and the supports_*
factory) silently dropped it for every provider-prefixed or generalized name; only
a bare base entry resolved. Wire it through ModelInfo like the other capability
flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across
providers so the data, not code, declares the capability. The anthropic-claude
fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so
an unmapped future Claude degrades to adaptive thinking without a code change.
Tighten the Claude version parser so an eight-digit date suffix
(claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor
4.20250514. The cost map stays authoritative; the version check is only a fallback
for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to
no mapped entry and so cannot be reached by an exact lookup or the bare-name rule.
https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo
* fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate
Reconcile adaptive-thinking detection after merging litellm_internal_staging.
Keep the cost-map resolver (_supports_model_capability) as the source of truth and
add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for
provider-prefixed ids the cost map cannot resolve (e.g.
bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an
eight-digit date suffix from being misread as a minor version, so the dated Claude
4.0 release stays non-adaptive
Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or
newly released Claude is over-costed rather than billed as free
Drop the module-level global state in fallback_generalizations (PLW0603) in favor of
a small registry object, and switch its annotations plus the new utils helper to
builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling
* refactor(anthropic): drive adaptive-thinking version gate from a declarative rule
Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit
The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor
* refactor(anthropic): dedupe adaptive-thinking rule via declarative extends
The version-gated anthropic-claude-adaptive-thinking rule duplicated the
broad anthropic-claude rule's entire Opus-tier price block because rules do
not merge: first match wins and returns one rule's whole model_info, so the
adaptive rule had to be self-contained.
Add a declarative extends field to fallback_generalizations: a rule names a
parent and inherits its model_info, with its own keys overriding. Inheritance
is resolved once at install time against each rule's raw model_info, so the
adaptive rule now carries only its delta (supports_adaptive_thinking) and
inherits pricing from the broad rule. Runtime matching, provider routing and
gating are unchanged; the broad rule stays anchored and first-match-wins still
holds.
* docs(anthropic): add ignored description key documenting each generalization regex
* fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule
Per review feedback, the base rule no longer carries input/output/cache costs, and the
adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an
unmapped model at a guessed tier reports a confidently-wrong cost without the caller
knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated
number) so a missing price stays visible. The rules still supply provider routing,
context window, and capability flags, so a brand-new Claude can still be called and its
capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests
updated to match
