* fix(ui): preserve Aria usage token breakdown and viewer model access
* fix(ui): make Aria conversations easier to read and reply to
* fix(ui): wrap long headings in assistant tables
* feat(ui): simplify Aria and add usage charts with PDF answer export
* fix(ui): retain full chart values in Aria PDF exports
* fix(deps): upgrade vulnerable Python dependencies
* fix(deps): lock patched Python dependency versions
* fix(otel): preserve JSON export with the updated SDK
* test(otel): cover JSON export with the updated SDK
* test(otel): update SDK log records and worker discovery
* fix(ui): align Lens peer dependency with patched Next.js
* fix(otel): hand the AgentOps exporter a session it owns for the JWT header
OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it
The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals
* fix(otel): skip FastAPI instrumentation once the app has started
opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects
* test(otel): assert exported requests and span flags instead of SDK internals
OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags
* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it
zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6
* test(ui): stub the tag summary fetch the client panel retry refreshes
Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing
* fix(deps): drop the duplicated Lens override from the dashboard package.json
* style(ui): format the client panel fetch stub
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat(ui): consolidate gateway and usage assistants as Aria Agent
* refactor(ui): give Aria an original blue sprite mascot
* feat(ui): prompt shared Aria from usage and retire duplicate usage agent
* fix(ui): keep Aria naming consistent and reachable across dashboard
* feat(ui): open Aria in a floating popup with suggested questions
* fix(ui): simplify the shared Aria Agent title
* feat(proxy): add LITELLM_DISABLE_LATEST_RELEASE_CHECK to stop the release check
The dashboard asks the proxy for the latest LiteLLM release, and the proxy fetches it from api.github.com on every cache miss. Deployments that block or audit outbound traffic had no way to turn that off. With LITELLM_DISABLE_LATEST_RELEASE_CHECK=true the endpoint returns null without making a request, which the dashboard already treats as no upgrade banner.
* fix(proxy): regenerate dashboard types for the release check flag
The endpoint docstring feeds the OpenAPI description, so the ui-api-types
check needs schema.d.ts regenerated. The docstring now says the flag skips
every outbound request, which stays true once the changelog feed is read
too
---------
Co-authored-by: Sujit Nirmal <nirmalsujit891@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(security): bump grpcio and boto3 so their bundled CA files drop the compromised eMudhra roots
grpc ignores SSL_CERT_FILE and verifies every channel against its own bundled roots.pem. In 1.78.0 that file still trusts emSign ECC Root CA - G3 and AAA Certificate Services, so Vertex AI, Google KMS, Chirp STT and OTLP gRPC exporters (Arize) accept all 14 advisory chains. 1.84.1 is the first release whose roots.pem drops both roots.
botocore uses certifi when it is installed and falls back to its bundled cacert.pem otherwise. Through 1.43.116 that file still trusts AAA Certificate Services. Requiring boto3 1.43.117 pulls in botocore 1.43.117, the first release whose cacert.pem rejects all 14 advisory chains, so the fallback stays safe if certifi is ever absent
* fix(docker): provide OpenSSL 3 when generating Prisma engines
The Admin UI upgrade notice read GitHub's latest release, which the release
pipeline sets on its own, so admins were offered releases the changelog had
not announced yet and sent to the auto-generated GitHub release page. The
notice now reads the newest stable release from the docs changelog feed and
links to its changelog page. GitHub release notes are still used for the
feature and fix counts of that version's minor line.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(interactions): assert the live status enum is parsable instead of pinning it
* test(interactions): read the spec from a vendored snapshot and canary the live one daily
* test(interactions): run the canary on PRs touching its inputs, file an issue on a red run, pin the regenerate command
* test(interactions): take the live spec URL as a pytest option, assert its source, split the canary's tracking-issue job
* test(interactions): file the canary issue whenever its compliance step did not pass, match the issue title exactly, cover UrlContextResult
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3
Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Every other crate names its error Error. The format crates already re-exported
it under that name, so only the inference crate and a few importers change
* refactor(types): replace Any with proven types in 100 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep untyped parsing where strict validation changed accepted shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep the OpenAI-compatible response parse untyped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep only annotation changes, drop new runtime validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): round-trip Anthropic compaction blocks as compaction items
/v1/responses on Anthropic routes now returns Claude's compaction block as a
compaction output item (and streams it as output_item.added/done), and a
compaction input item is replayed to Claude as its native compaction block,
so the next turn reuses the summary instead of compacting again. Other routes
receive the summary as assistant text. The Anthropic stream handler now keeps
the summary text that arrives in compaction_delta.
Fixes#41456
* fix(responses): replay compaction blocks natively only when the request compacts
* fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails
Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem
objects on replay, strip the encrypted_content affinity wrap before decoding,
scan and rewrite compaction summaries in Responses guardrails, drop the history
a summary replay replaces, send the summary as a user message so thinking
ordering holds, and send the streamed compaction .done with the newest block.
* fix(responses): count, route, guard, and redact compaction items consistently
A compaction item now reads as its summary everywhere the proxy counts or routes
a Responses request: pre-call context checks, the TPM limiter, the input token
endpoint, the stream usage estimate, and the previous_response_id session rebuild
all drop the history the item replaces. A Claude compaction item sent to a
non-Anthropic model (including a fallback) becomes a user summary message instead
of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform
takes the summary path. Output guardrails see the summary of a returned
compaction item, and logging redaction hides it.
* fix(responses): start previous_response_id replay at a stored compaction and count reservations from it
* fix(responses): keep a stored compaction summary whole in spend logs
* fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning
A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a
client-built item no longer reach the database whole. Message redaction also nulls the streamed
compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning
item that follows it. Adds the audit's integration tests for compaction items on the wire.
* test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order
Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order.
* fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes
A failed /v1/responses or /v1/messages call keeps its proxy request headers in
litellm_metadata, but the failure row read redaction headers from metadata only,
so x-litellm-enable-message-redaction never redacted those rows.
* fix(spend_logs): lift only the redaction opt-in header into failure rows
The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main
* test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes
* fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary
A previous_response_id rebuild that starts at a compaction now keeps the system and developer
messages of every earlier turn. Token estimates for raw Responses input count a compaction item
as its summary text instead of the serialized item.
* fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id
* fix(responses): count only message items with a string role as compaction instructions
* test(responses): flatten the last two-for comprehension in the compaction wire test
* fix(responses): replay from a compaction in stored Responses output and SDK compaction items
A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths.
* fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history
* fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary
A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a
replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from
rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and
MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound.
* fix(responses): replay a compacted session only through the referenced response
A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item
* fix(spend-logs): keep main's stored value for a prompt length limit under 2
At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where
main stores the marker followed by the whole value
* fix(spend-logs): keep main's stored tail for a negative prompt length limit
At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start
* fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages
* test(responses): run the compaction wire tests on the worker database under isolated workers
* fix(router): keep the encrypted content affinity pin out of provider requests
* fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction
* fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message
* fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses
* fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call
* test(proxy): run the dispatched-call-type failure test through the real request setup
* fix(proxy): count a failed Cursor stream's input from the newest compaction item
The Cursor route maps to no call type, so the stream generator's failure hook fell
back to the route and counted the whole history. Pass the processor's route_type.
* fix(proxy): count a broken Cursor stream's prompt from the newest compaction item
* fix(responses): keep a tool call a later output answers when trimming before a compaction item
* fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes
* perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time
* test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests
* fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item
* fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item
* fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content
* test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks
* fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once
* fix(responses): count input_tokens after a compaction item the way inference sends it
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* fix(llms): join api_base and route with one shared helper
A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base
* refactor(llms): move per-provider api_base strips onto join_api_base
Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join
* ci: run the trailing-slash URL builder test in the llms shard
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
* fix(llms): send every provider route through join_api_base
Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it
Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call
The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash
* fix(llms): keep the route skip only where a builder already had it
join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path
already ends in it, and it is used only where the merge base or the trailing-slash test PR head
already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI
model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the
path so a host named anthropic-* builds /anthropic/v1/messages
* fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL
* test(llms): run the Gemini RAG trailing-slash case through aingest
The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears
* test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base
* test(integration): cover api_base joins across providers, query strings and named hosts
* feat(url_utils): add join_api_base and ensure_api_route helpers
* fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host
join_api_base and ensure_api_route now keep the // of a base left with only a scheme,
so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client
base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it
* fix(azure_ai): build the count_tokens URL from the messages URL
* fix(azure_ai): match anthropic as a whole path segment in the messages URL
* fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints
Azure AI chat and messages go back to cutting the base at the first
/anthropic, as on main, and count_tokens keeps its path check, so a
gateway path like /anthropic-proxy builds the same URLs as main. A base
that already ends in /v1/messages still gets count_tokens next to it.
A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is
used as its text again in AgentCore, Bedrock responses and the RAGFlow
base, instead of falling back to the default host.
get_volcengine_base_url and GigaChat's get_api_base strip a trailing
slash again for outside callers. The cache-hit wire test is skipped as a
known proxy bug, since one cache hit writes two cache_hit spend rows.
* fix(llms): drop a slash before the api_base query and tighten the route test checks
* test(llms): annotate new test locals as Final and flatten the builder generator
* fix(url_utils): keep a scheme-only api_base hostless so it fails before sending
---------
Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>
A guardrail hook plants a litellm_metadata key on a chat completions request after the proxy
stamped litellm_received_at into metadata. get_litellm_metadata_from_kwargs prefers that key
when both exist and only copied the user_api_key_* fields across, so the response timing
headers lost their receive anchor and x-litellm-overhead-duration-ms reported 2 ms on
/v1/chat/completions against 58 ms on /v1/responses for the same proxy work. The receive stamp
now rides along with the user_api_key_* fields.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): add small_bodies_30rps and mixed_size_requests_4rps load cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): run the load cases at 50 rps small and 5 rps mixed, the pod's knee
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): ramp the load cases until the pod stops keeping up and report sustained_rps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): hand the ramp to the observation and widen the mixed ramp to 14 rps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): floor the small ramp at 20 rps and emit the per-step readings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): emit the ramp's hold bound and each step's unexpected statuses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): read each ramp step's cpu, redis latency and connections from the samplers' windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): floor mixed_size_requests_ramp at 8 rps from the lane's 10 rps sustained step
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): compare step readings with math.isclose so the e2e basedpyright gate stays clean
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage
A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object
across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception,
and a second drop never reached the failure handlers because the first had already marked the object as
logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the
router discards the stashed usage right before it opens the next attempt, on same-group retries and on
every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage.
* fix(router): dispatch chat stream failures through the attempt-scoped logging copy
The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types.
* test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy
* test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader
The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead.
* test(router): use real Logging in fallback cooldown test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): preserve superseded retry usage for billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill retry input after prior stream usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): track retry dispatch for terminal billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): use deterministic retry timestamp
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): avoid real clock in fallback tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): remove duplicate streaming logger import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): retain error-frame usage for exhausted retries
* test(proxy): fix retry billing expectations and helper annotations
* fix(proxy): preserve failure estimate without carried usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): audit retry helpers and fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): fix retry fixture timestamps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe retry stash at attempt boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): keep terminal failure attribution when dropped attempts log on copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish a dropped attempt's payload only while it is still the current attempt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover spend-log encoding error propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): cool down the final fallback attempt when its failure callbacks were blocked
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): import Final in adapter regression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop spend-log encoding expectation superseded by main's poison isolation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): finish the wrapped stream's failure log before surfacing the error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pin pass-through stream fixture start times
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish failed attempt payload before awaiting failure callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish failed attempt payload from the failure handler after baseline invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging): drop unused discard_partial_usage_for_failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(vertex-ai): support gemini 3.1 flash tts
* fix(vertex-ai): keep TTS metadata compatible with the cost map guard
* fix(vertex-ai): preserve existing Gemini TTS models and voice mappings
* fix(vertex-ai): recognize provider-stripped TTS model names
* test(vertex-ai): match current speech HTTP error response
* fix(vertex-ai): preserve nested speech locale and voice
* fix(vertex-ai): use provider speech media type in proxy
* fix(gemini): preserve TTS dispatch and nested speech settings
* test(gemini): run TTS regressions in CI shards
* fix(vertex-ai): account for Gemini Cloud TTS audio cost
* fix(vertex-ai): bill Gemini TTS without optional audio decoder
* test(gemini): cover TTS mapping and billing edge cases
* test(vertex-ai): cover Gemini voice normalization in provider shard
* fix(vertex-ai): keep Gemini TTS routing with generic provider metadata
* chore(vertex-ai): remove obsolete lint suppressions
* fix(gemini): strip audio format from speech configuration
* chore(vertex-ai): remove obsolete mutation suppressions
* fix(vertex-ai): preserve FLAC billing and wrapped audio headers
* fix(vertex-ai): reject unmeasurable FLAC before billing
* fix(vertex-ai): reconcile speech support with current provider mappings
* fix(vertex-ai): validate speech mappings and usage types
* fix(vertex-ai): preserve speech mapper override forwarding
* fix(gemini): map voice aliases across speech request forms
* fix(vertex-ai): send only formats the speech bridge cannot serve to Cloud TTS
Gemini TTS detection now goes by model name, so newer Vertex Gemini TTS
models keep working, and only mp3, opus, alaw, mulaw and other container
formats move to Cloud Text-to-Speech. wav and pcm requests stay on
generateContent as on main.
A Cloud TTS response whose duration cannot be measured now returns the
audio unbilled with a warning instead of raising after a paid call. FLAC
handling is removed because Cloud TTS has no FLAC encoding.
The cost map, the google_genai mappers and the proxy error path go back
to main, since those changes were outside this feature.
* fix(vertex_ai): keep Gemini TTS formats, region and instructions on routes that serve them
Only mp3, opus, alaw and mulaw go to Cloud Text-to-Speech. Every other
format gets a local 400 that lists the supported ones, before Google is
called.
Cloud TTS uses the deployment's us or eu regional host. Other regions
and a custom api_base get a 400 that says to request wav or pcm.
instructions is sent as the Cloud TTS style prompt, and a request with
no voice keeps the Gemini model.
get_usage_object is back to main's code, so dict usages still go
through Usage.__init__, and the proxy no longer labels pcm16 or
linear16 output as WAV.
* fix(vertex_ai): send Gemini TTS to every regional Cloud TTS host and keep speak tags as text
Cloud Text-to-Speech serves Gemini TTS on a host named after each Vertex
location, so a non-global vertex_location now uses
{location}-texttospeech.googleapis.com instead of only the us and eu hosts.
Google answers for regions it does not serve.
Gemini TTS voices reject SSML, so input that contains <speak> is sent as
text for Gemini TTS models. Chirp keeps the SSML auto-detect.
Removes the proxy alaw and mulaw media-type entries. Cloud TTS returns
those encodings with a WAV header, so the response is labeled audio/wav
and the entries never applied.
Adds a regression test that custom per-token pricing bills the usage a
speech response reports.
* fix(vertex_ai): map Gemini TTS regions to Google's Cloud TTS locations and keep a bad voice from cooling down the deployment
US and EU Vertex regions go to the us and eu Cloud TTS hosts, northamerica-northeast1 to its own, global or unset to global, and any other region gets a local 400. A Cloud TTS 404 on the Gemini path, such as a misspelled voice, comes back as a 400 so the router keeps the deployment serving wav. Usage reads the sent request from the response instead of the logging object.
* fix(vertex_ai): default Gemini TTS Cloud formats to a Gemini voice and validate speakers
Parse only usage fields from the sent Cloud TTS request, so a numeric speakingRate no longer fails after Google bills it.
Send the Gemini voice Kore when no voice is named, and reject Cloud TTS voice names unless drop_params is set.
Reject malformed speaker entries unless drop_params is set, and translate Cloud TTS speaker fields on wav and chat audio.
Use the Vertex default region when no location is set, so mp3 and wav share a region.
* test(vertex_ai): cover two-speaker Gemini TTS mp3 through aspeech with fake credentials
Replaces two tests that patched the Vertex TTS config's own token and URL methods.
* fix(vertex_ai): reject malformed Gemini TTS voices and keep beta Chirp and Lyria params unmapped
* fix(vertex_ai): drop unknown Gemini TTS voice keys on wav and pcm instead of rejecting them
* fix(vertex_ai): reject unknown Gemini TTS voice keys on wav, pcm, and chat audio unless drop_params is set
* fix(vertex_ai): honor every drop_params setting in the Gemini TTS voice key check
Global drop_params, batch row and deployment drop_params on Vertex batch
file uploads now drop unknown voice keys. A voice object with both
speechConfig and speech_config gets a 400 for the ignored speech_config.
* fix(gemini): drop unknown nested voice fields under drop_params and detect TTS by served model
* fix(gemini): filter unknown speech config fields with fixed-depth helpers and pass audio params through unchanged
* fix(gemini): camel-case snake replicated voice keys so drop_params keeps them
* fix(vertex-ai): reject voice fields Cloud TTS cannot send unless drop_params is set
* fix(vertex-ai): reject wrong-typed Cloud TTS voice values and check speakers as sent
Gemini TTS on mp3, opus, alaw, or mulaw now answers 400 for a voice field whose value Cloud TTS cannot send, such as a list languageCode or a string voiceConfig, and for nested voice fields under a speaker that also names speakerId. drop_params still drops them.
* fix(vertex-ai): reject wrong-typed speech config wrappers and speakerId overriding a prebuilt voice on Cloud TTS
* fix(vertex-ai): keep speakers and voice_config beside a speech_config wrapper on Cloud TTS
* fix(vertex-ai): pick Cloud TTS voices from the same speech config the WAV path sends to Gemini
* fix(files): carry the deployment's drop_params on model= file uploads so Vertex batch rows drop unknown voice keys
* fix(gemini): keep a structured speech voice when the model is a model_alias_map alias for a Gemini TTS model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): send pi's session id from lite pi so sessions group and stick
* docs(cli): drop the provider_block docstring lines the README already covers
* docs(cli): note the session id format the proxy reads for lite pi
* test(integration): keep autovacuum off the size cleanup test's scratch tables
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(circleci): keep the MCP SDK1 peer venv on mcp 1.x despite the project override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover billing every round of a streamed MCP auto-execute
* fix(responses): bill every round of a streamed MCP auto-execute on one spend row
* fix(responses): bill every round of a streamed MCP auto-execute on bridged providers
A streamed /v1/responses MCP auto-execute on a provider bridged through chat
completions (for example Anthropic) billed only round 1, because each round's
chat stream dispatched on the shared logging object and the first dispatch won.
Bridged rounds now go through the same one-row billing as native rounds. A
round is billed as soon as its chat stream ends, before its closing events
reach the client. The row carries the final response, usage and cost summed
over every round, and the fixed margin once. A round served from the response
cache is billed at zero. A follow-up chat stream that breaks bills the paid
rounds on its failure row. A single-round stream logs exactly as before.
* fix(responses): charge one fixed margin and keep bridged usage fields on streamed MCP rounds
A follow-up round that fails after earlier rounds were billed now charges the
per-request fixed margin once, and its failure row carries no cost breakdown
that would disagree with the summed cost.
A bridged multi-round stream logs a plain cache_hit flag like a single-round
stream does, and its usage keeps the summed provider usage fields of every
round.
A new test covers a stream dropped after its event loop closed, which must
release its parked log without an unawaited coroutine warning.
* fix(responses): price each streamed MCP round on its own and bill every round when a guardrail blocks
Each round of a streamed MCP auto-execute is now priced from its own
uncached cost, so a provider that states its cost (xAI) charges the stated
cost of every round plus the fixed margin once, and the cost breakdown
always agrees with the billed cost.
Summed usage goes through a JSON-safe dump, so a nested provider field such
as xAI's context_details no longer breaks a failed stream's response.failed
event.
A post-call guardrail that blocks a multi-round bridged stream now drops the
parked success log and bills every round on the failure row, and a failed
follow-up round's failure row carries the summed breakdown of both rounds.
* fix(responses): keep earlier MCP rounds on their success row when a later round fails
When a later round of a streamed MCP auto-execute fails, the rounds before
it are now logged on their success row, and the failure row bills only the
failed round with no second fixed margin. That failure log runs on its own
copy of the request's logging state, so its usage and cost never leak into
the earlier rounds' row.
A post-call guardrail that blocks a native multi-round stream keeps the
summed success row beside the $0 failure row, as at the merge base, while a
bridged stream still bills every round on its failure row.
* fix(responses): keep a failed later MCP round on its own failure row
A streamed MCP auto-execute whose later round failed wrote only the earlier
rounds' success row. The failure was logged on a copy of the logging state,
while the proxy builds its failure row from the request's own state, which by
then held the success payload, so the failure row took the success row's id
and was dropped.
Now the failure is logged on the request's own state as at the merge base,
and once a later round starts, the success row for the rounds before it is
logged from a copy of the state taken at that moment, so the two rows never
mix.
* fix(proxy): let an MCP tool call reuse its request's max_parallel_requests slot
* test(proxy): create the other-request task in a way Python 3.10 supports and mark its bindings Final
* fix(proxy): keep a reused max_parallel slot until the request's last round
An MCP tool call that reuses its request's max_parallel_requests slot no
longer loses it to the first round's late success callback, so the key
stays counted for the rounds still to run. The reused call also stops
replacing the request's rate limit headers.
* fix(responses): keep an unpriced streamed MCP round unpriced instead of billing it as free
* fix(responses): log a bridged MCP stream as a chat completion with every round's usage
* feat(ui): add Delete to the guardrail detail Settings card for DB guardrails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): hoist guardrail delete test fixtures to stay within the inline-object lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): share one guardrail delete modal between the table row and the detail page
The table-row delete now has the same dangling-reference warning and typed-name
confirmation as the detail page. Tests now assert the success and error toasts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): warn that deleting a guardrail used in a policy pipeline step blocks requests with a 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover hiding guardrail Delete while settings are being edited
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gemini): LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS sets the long decode budget for requests that send no limit
* test(gemini): cover a default long decode budget set by LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS
* fix(gemini): apply the default output budget only to requests asking for text output
* refactor(gemini): apply the default output budget once in the async request transform
* fix(gemini): leave the default output budget off non-chat models and a snake_case generation_config
* fix(gemini): skip the default output budget for audio-only models and continuation resumes
Lyria and other Gemini chat models whose cost map row lists no text output answer 400 when sent a maxOutputTokens, and a request that resumes from a continuation token answers 400 when a limit is added beside it. Both now go out without the default budget.
* fix(vertex_ai): skip the Gemini default budget for a model a deployment's model_info overrides, and pass odd generationConfig shapes through
* fix(vertex_ai): cap the Gemini default budget per deployment at the loaded cost map and its own model_info limit
* test(vertex_ai): wrap a long test signature under the 120 character limit
* fix(vertex_ai): keep a direct call's default output budget under the loaded cost map limit
A litellm.Router deployment's model_info writes onto the shared cost map row, so a later direct call with no model_info read that raised value and Vertex answered 400. The direct call now caps at the lower of the loaded row and the live row, so litellm.register_model can still lower the cap.
* fix(vertex_ai): ignore a deployment output limit that is not positive
A model_info.max_output_tokens of 0 or -1 capped the default budget at that value, and Google answers 400 to a limit that is not positive. It now counts as unreadable, so the cost map limit applies.
* fix(gemini): never continue a request asking for more than one candidate in camelCase
* fix(gemini): keep continuing a request whose candidate count is null or 0
* fix(router): unregister strategy selectors from global callbacks on discard
Router.discard only removed its own bound methods from litellm's callback
lists, so the strategy selectors it registered (least-busy, usage-based,
latency-based, cost-based and routing group selectors) stayed behind and
kept receiving events after the router was gone. Reuse the identity based
unregister that update_settings already uses so a discarded router takes
only its own selectors with it
* fix(router): purge discarded selectors from every event callback list
function_setup copies litellm.callbacks into the success and failure
lists on every request, so a router that served traffic still leaked its
selectors there after discard. Route the cleanup through
remove_callback_from_all_lists, which now also covers input_callback, and
fold the selector gathering into discard so the router coverage check
sees no untested helper
* fix(router): keep live routers, in-flight hooks and user_config limits intact on discard
discard() now rebuilds every global callback list without the router's selectors, optional pre-call checks and own usage callbacks instead of removing them in place, so a callback loop already running never skips a hook. It then re-registers what the remaining live routers need, so a same-strategy survivor keeps its deduped handler. The proxy's per-request user_config router is discarded only after its call returns.
* fix(router): bind user_config handlers to the request and tighten discard
When a user_config request carries a logging object, its router is
discarded before the request runs and its handlers ride on that
logging object, so v1 rpm and v2 tpm limits count every request and
concurrent requests count once. Without one, the router is discarded
after the request as before. The router's pre-call loops also read
the request-scoped handlers.
discard() now refills only the handler keys it vacated, guards the
callback lists and the live-router set with a lock, covers routers
built without a model_list, and drops its Slack alerting, adaptive
router hook and budget limiter sync task
* fix(proxy): reuse one user_config router per config and bind its handlers to each request
Every user_config request now goes through a cached router built once per
config. Its strategy handlers leave the global callback lists and are bound
to the request's logging object instead, so limits count each request once
and nothing is left behind after the request.
* fix(router): keep the proxy budget limiter on user_config requests
A user_config router now swaps in only its strategy selectors, so the proxy's
RouterBudgetLimiting stays on the request and keeps counting its spend. Usage
logged after a strategy is retired, such as a request still in flight when its
user_config router is evicted, now schedules one more push to Redis.
* fix(proxy): init the retire fields on the key model budget limiter
The key model budget limiter builds its own state instead of calling the base
init, so with the proxy's Redis attached every spend increment raised on the
missing retire flag and the key's model budget never counted spend.
* fix(router): bind the deployment callbacks to user_config requests
A cached user_config router serves requests through each request's logging
object but bound only its selectors and policies there, so a failing
deployment was never cooled down and its usage counters missed requests.
* fix(router): retry an evicted router's budget writes and keep the adaptive header on user_config requests
A retired budget limiter now keeps retrying its queued Redis increments every
sync interval until Redis accepts them, instead of giving up after one failed
write.
The x-litellm-adaptive-router-model header now comes from one stateless hook in
litellm.callbacks that holds no router, so user_config adaptive routers, whose
per-router hooks are request-scoped, still answer with the header.
* fix(router): lock request callback setup against router cleanup and drain in-flight budget writes on retire
function_setup now edits the global callback lists under the same lock that
router cleanup uses, so a router dropping its callbacks on another thread can
no longer shift the indexes function_setup pops or put a dropped handler back.
A budget limiter retired while a Redis write is still in flight now keeps
retrying that batch if the write fails, instead of leaving the spend queued
with no task left to send it.
* fix(utils): set up loggers after releasing the callback lists lock
function_setup held the callback lists lock while set_callbacks ran its one-time
logger setup, which can block on a Slack package install, so other threads and
Router.discard waited on it. The lock now covers only the list reads and writes.
* test(router): annotate the retire test's locals as Final
* fix(logging): skip replaced global callbacks on the input path too
replace_global_callback already kept a same-key global logger out of
success and failure dispatch. pre_call and post_call still sent it the
input events, so a logger with an input hook could count a request start
it never saw finish. Both now share one filter with the success path
* test(logging): type the input-path regression test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
