* fix(cli): run group subcommands in the lite shell and default to the logged-in gateway
Typing a command group alone in the lite shell printed 'Usage: lite lite',
the next subcommand was unknown, quoted arguments were split on spaces, and
a bare lite connected to localhost:4000 even after lite login to a gateway.
The shell now names the group in usage text, keeps the group as the prompt's
scope so its subcommands run next, accepts a leading 'lite', parses lines
with shlex, and a bare lite falls back to the server the stored lite login
credential was issued for before localhost:4000.
* refactor(cli): walk the shell's group path without recursion
* fix(cli): keep backslashes and compact JSON in the lite shell and pair the login server with its key
* fix(cli): keep the login server to the bare lite shell and accept lite options and = quoting at its prompt
* fix(cli): keep the shell's server for a lite options line and name the right login in the keychain notice
* fix(cli): keep compact JSON with a trailing = intact in the shell and pin options lines to the shell's server over LITELLM_PROXY_URL
Brings integration tests in line with intentional behavior changes from
#46331, #46291, #46341, #46355, #46368 and #46493, moves the Databricks
non-Claude cases off llama-3 names so they no longer download a Hugging Face
tokenizer, drops the unrealistic count_tokens tools=5 case and skips the
Mantle region-prefixed v1 base case as unrealistic input
* feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs
When the setting is on, spend log rows store requester_ip_address as null in
the column, the metadata and the stored request (which also drops client-IP
forwarding headers), and IP addresses in stored error messages are masked, so
the Logs page shows no caller IP. The setting is read the way the settings API
validates it. Capture is unchanged, so guardrails, logging callbacks and
metrics still receive the IP. Toggleable from the Admin UI General Settings.
Resolves LIT-7442
* fix(proxy): mask IPv6 addresses that end or start with :: across eight groups
The masking pattern allowed seven colons, so valid forms like 1:2:3:4:5:6:7::
and ::1:2:3:4:5:6:7 stayed in stored error text with
disable_requester_ip_address_logging on. Also lists the request metadata walker
in the recursion check, since it only walks the tree the spend log sanitizer
already returned.
* fix(proxy): mask IPs in every stored header value and after labels or underscores
With disable_requester_ip_address_logging on, client IPs carried in headers outside the forwarding list (do-connecting-ip, incap-client-ip, custom names) and addresses written as host:2001:db8::2, peer_2001:db8::2, socket_192.0.2.1 or an unbracketed IPv6 with a port were still stored in spend logs.
* fix(proxy): mask request header IPs before long values are truncated
A header value longer than the spend log string limit kept the part of an IP that fell before the truncation cut. Request headers in metadata and requester_metadata are now masked before the body is truncated, like error text already was. IPv4 followed by a dotted port (192.0.2.1.443) is masked again.
* fix(proxy): mask the IP piece a header truncation cut leaves instead of pre-masking headers
Masking header values before truncation scanned the whole client value,
including text the cut drops, and walked hook-made header cycles. Mask after
truncation instead, then mask the hex and digit run touching each side of the
truncation marker when it is short enough to be part of an IP address. Every
nested headers mapping gets the same treatment.
* docs(proxy): say the requester IP setting masks IPs in other stored header values
* fix(proxy): mask stored error text after truncation and harden the cut-piece pass
Error messages and tracebacks were regex-scanned in full before the DB cap,
which costs seconds on multi-megabyte provider errors. Mask them after the
cap like header values, with the same pass over the run beside each
truncation marker. That pass now handles every marker, skips long hex tokens
so an address beside one is still masked, keeps at most one address length,
and also masks colon-only IPv6 groups.
* fix(proxy): drop the requester IP from the originating request an auto router classifier row stores
* fix(proxy): drop the requester IP from the classifier input an auto router classifier row stores
* refactor(proxy): mask stored IPs with the content filter's ipv4 and ipv6 patterns
The requester IP setting now reuses the content filter guardrail's prebuilt
ipv4 and ipv6 patterns instead of its own address regexes, so spend logs mask
what that guardrail masks. Addresses glued to a label or underscore stay as
the content filter leaves them.
* fix(proxy): mask stored IPs with the spend-log matcher again
The content filter's ipv4 and ipv6 patterns left labelled IPv6 (host:2001:db8::2),
unbracketed IPv6 with a port and IPv4 after an underscore in stored error text and
header values, and masked parts of text like std::io::Error. Restore the
address-validating matcher and its regression cases.
This reverts commit 79537a8aa5.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(proxy): expose trace ID response header
* style(otel): format trace correlation middleware
* test(otel): cover websocket trace middleware
* fix(otel): disable correlation on setup failure
* refactor(otel): inject instrumentation setup
* fix(proxy): drop a relayed upstream traceresponse when the gateway sends its own
* fix(proxy): preserve upstream traceresponse without gateway injection
* fix(proxy): replace relayed streaming traceresponse
* test(otel): cover trace response fallback paths
* test(proxy): cover the streaming pass-through relay when the gateway propagator is off
* fix(proxy): keep a relayed upstream traceresponse and stamp the call id on the server span
The gateway's response propagator now leaves a response that already carries an upstream traceresponse untouched, so pass-through, file content and plugin proxy routes return one header and untraced pass-through routes keep the upstream's. The per-helper strips are gone. The propagator also sets litellm.call_id on the server span from the response's x-litellm-call-id.
* fix(proxy): add traceresponse only to sampled HTTP responses, tuple headers and unhandled 500s included
* fix(proxy): write only traceresponse and link a relayed header from a restarted upstream trace
* fix(proxy): send traceresponse only from the proxy app and expose it through CORS only with OTel V2
* test(proxy): check the CORS expose list contains the UI headers instead of pinning its identity
* test(integration): check traceresponse names the gateway server span across routes, errors, pass-through and chaos
* test(integration): read the cache-hit spend row by response id and assert only the traceresponse expose entry
---------
Co-authored-by: adityajoy-1902 <bittu19jain@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python
Rust's run_call built Python streams itself and imported
litellm.rust_bridge.streams by name, so sync calls were driven in Rust
while async calls came back as a Rust-made coroutine. Every native
inference entrypoint now returns one unstarted Execution and
public_execution.py drives it with lifecycle.run or lifecycle.drive,
wrapping streams and _hidden_params on the Python side. Rust only
imports the lifecycle step classes through StepTypes, and
python-bridge/src/lifecycle.rs, PythonLifecycle and streams.py are gone.
The adapter trusts the _native.pyi contract with a single cast instead
of a per-request runtime_checkable isinstance. Binding tests assert the
adapter starts the execution and returns its completed value rather
than checking callable identity.
* refactor(rust-bridge): fold lifecycle and public_execution into one execution module
The public stream wrappers only forwarded to the lifecycle streams and added the
header dict, so the streams now carry _hidden_params themselves. lifecycle.py no
longer held any call lifecycle, just the Execution driver, so it is renamed to
execution.py and absorbs the synchronous and asynchronous adapters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust-bridge): keep execution.py free of third-party imports
The host-python tests load execution.py into a bare interpreter, so the
pydantic header validation broke them. The stream head comes from the
binding's encode_stream_head, which always builds a str-keyed dict
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(host-python): preserve pre-async hook coverage after execution handoff
* style(rust-bridge): sort litellm_host_python imports
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): drive unstarted native calls through execution in the split lifecycle tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* fix(rust-bridge): mark execution TypeVar as Final
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): cover stream close, read failure, and sync suspension in execution
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): drive executions with generator-style send, throw and close
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): name stream steps StreamOpen and StreamChunk and reject out-of-phase steps
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): drive with ExitStack and require the stream factory
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(host-python): give the hook_chain lifecycle binding the stream factory
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust-bridge): pin Messages and Responses callback contracts on the native path
Each test runs the native route and asserts what the Python path does today, so a
known gap is a strict xfail that names its reason and the exception it fails with
Messages and Responses share one Surface and hook recorder in support/callbacks.py,
and each surface folder lists its own expected counts and gaps.
Body-level hooks from the function body (pre-request hook, websearch short-circuit,
Messages interceptors, planned and legacy agentic loops, streaming deployment hook)
are xfail until native runs them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-bridge): give every route folder the same one-concern-per-file layout
Route data, the route Surface and the server fixture move into each route folder, and support/ keeps only
route-neutral helpers. test_inference.py splits into chat_completions/ and responses/, ocr/test_requests.py
splits into requests, failures and settings, and Messages and Responses get streaming and body hook files.
Messages request tests that duplicate inference-messages request.rs are removed, and drop_params read
from litellm.drop_params is now covered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-bridge): expect the required-route decline error for unsupported Responses requests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): import route data from the route folders in the secrets unit tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): import NoReturn in the OCR dispatch tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): move route-neutral helpers out of support/callbacks.py
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): cover deployment rejection and concurrent callbacks
* test(rust-bridge): assert native serving via the x-litellm-rust marker in the route folder tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): keep the input/output split on streamed xai spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): split xAI's reported total across input and output by token math
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): let proxy admins open team member defaults to team admins
* fix(proxy): keep a team admin's never-resets duration on a team's first member budget
* test(proxy): pin the existing-row and stored never-resets cases for member defaults
* feat(proxy): let proxy admins open soft_budget to team admins
* feat(proxy): let proxy admins open team metadata to team admins, keeping proxy settings with proxy admins
* test(ui): name expected objects in team admin edit access tests
Keeps local/no-large-inline-object-arg within its eslint budget
* fix(proxy): compare team metadata without recursion and count key presence
A proxy-controlled metadata key that appears or disappears with a null value
is now a change, and the JSON comparison no longer recurses
* fix(proxy): compare team metadata numbers exactly
Numbers are compared as decimals, so integers past float precision and close
floats are no longer treated as equal
* fix(responses): keep the provider code and message on top-level stream error events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): log a mid-stream failure before the router falls back
CustomStreamWrapper now waits for failure logging before it raises the
error the router falls back on, so the cooldown lands on the deployment
that failed instead of the fallback that replaced it. Also keeps numeric
error codes and the top-level message on the WebSocket and ChatGPT paths.
* fix(streaming): run sync failure callbacks before a failed stream falls back to another deployment
The router cools a deployment down from a sync failure callback that reads the request's model_info. A failed stream submitted that callback to the logging executor without waiting, so it often ran after the router's fallback had overwritten model_info and cooled down the healthy fallback deployment instead, leaving the model group with no deployments
* test(responses): cover top-level OpenAI stream error events end to end
* test(streaming): read live worker pids so the second SIGKILL chaos variant sees two workers
* fix(streaming): run waited sync failure logging off the shared logging pool
* revert(streaming): keep failed-stream failure logging in the background
Waiting for sync failure callbacks before a failed stream raises fixed the router cooldown race, but it slowed healthy calls under many failing streams with a slow callback, since the wait shares a thread pool with every call's start. The race is pre-existing on main and is tracked separately, so this PR goes back to main's logging timing and keeps only the top-level error event changes
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Airia guardrail integration
Adds `airia` as a built-in guardrail provider. The hook sends the prompt under `pre_call` and the
model's response under `post_call` to the Airia AI Gateway, which answers allow, block, or redacted
content; the only configuration is `api_base`, `api_key`, and an optional `timeout`.
`during_call` is not offered: it runs concurrently with the model call, so a block could land after
the prompt has already reached the provider.
A BLOCKED verdict, or any action this version does not recognise, raises with
`blocked_content=True`; a transport error, a non-2xx, or a response body that is not a JSON object
raises with `blocked_content=False`, so callers can tell "could not evaluate" from "evaluated and
blocked" while both still fail closed. On GUARDRAIL_INTERVENED a copy of the inputs is returned with
every rewritten field substituted; an intervention carrying no applicable rewrite, a rewrite of the
wrong shape, or a rewrite whose item count does not match what was sent (which would misalign the
positional write-back downstream) blocks instead of letting the original through.
Streamed responses are moderated whole and then emitted redacted: the hook opts into the unified
hook's `incremental_diff` mode (the default `block_only` drops rewrites) with
`streaming_end_of_stream_only`, so a redaction can never span transform rounds and underflow. Covered
by a test that drives the real streaming iterator hook end to end and checks the assembled client
output, not just the guardrail's own config attributes.
The HTTP client is dependency-injected (an `async_handler` constructor parameter, defaulting to a
real client) rather than mutated post-construction, so tests inject a fake client instead of
monkeypatching an instance method.
Known limitation, not fixable from this guardrail alone: LiteLLM's unified guardrail framework
yields a tool-call chunk to the client immediately during `incremental_diff` streaming, inspecting
the assembled tool calls against the guardrail's verdict only once the stream ends
(`_run_incremental_transform_stream` in `unified_guardrail.py`). A blocked tool call can therefore
reach an executing client before the terminal error arrives. Switching to `block_only` with
`streaming_buffer_until_moderated` does not close this without a regression: that flag has no effect
under `incremental_diff` (the mode branches away before it is read), and `block_only` itself replays
the original, unredacted chunks on release, silently dropping any REDACT verdict on streamed text.
Relatedly, LiteLLM's OpenAI chat completions guardrail translation skips `post_call` guardrails
entirely for a response with no text content, so a tool-call-only response bypasses this guardrail
(and every other post-call guardrail) even outside streaming. Both are shared-framework behavior,
not specific to Airia; fixing either is out of scope for an isolated single-guardrail contribution.
* fix(guardrails): fail closed on an Airia rewrite for a field that was not sent
Bugbot finding on #40784. An INTERVENED verdict carrying `structured_messages: []` with
`texts` omitted passed `_rewritten` unchanged: an empty list has the same length as an
omitted field, so it counted as an applied rewrite while the `texts` that actually carried
the content went through untouched. post_call sends only `texts`, so this was reachable.
A rewrite for a field the request did not send is now a mismatch, same as a wrong length,
and the verdict blocks. Regression test covers both the empty and the non-empty case.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(guardrails): fail closed on streamed non-chat routes the Airia rewrite cannot reach
Veria finding on #40784. incremental_diff, the mode that carries a rewrite into the stream,
exists only for OpenAI chat completions; UnifiedLLMGuardrails sends every other streamed route
(/v1/responses, /v1/messages) to block_only, which streams the original text live and drops
rewrites. So with post_call on, a streamed Responses or Anthropic call could receive text that
Airia then blocked or redacted.
Two changes, both confined to this guardrail:
- streaming_buffer_until_moderated is set, so on the block_only fallback the framework holds
every chunk until the end-of-stream verdict instead of releasing it live. The flag is never
read on the incremental_diff path, so streamed chat completions are unchanged.
- A REDACT verdict on a streamed response whose route cannot carry a rewrite is enforced as a
block, since block_only would otherwise replay the original. The route check mirrors
_resolve_transform_call_type so the two cannot disagree; an unresolvable route fails closed.
Request-side rewrites and non-streamed calls are untouched.
The end-to-end test drives the real UnifiedLLMGuardrails iterator over an Anthropic
/v1/messages SSE stream and asserts not one original chunk is released for either verdict.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(guardrails): keep the Airia stream test's pytest.raises block to one statement
CI's test-tree ruff run flags PT012 on the non-chat streaming test: the `async for` that
collected released chunks sat inside `pytest.raises`, so a raise from the loop body would be
indistinguishable from the one the test is about. The loop now lives in a `_drain_into` helper
and the `with` block awaits it alone. Same assertion, same partial-leak capture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(guardrails): keep the Airia guardrail within the per-rule basedpyright ceilings
scripts/type_check_gate.py now gates every rule on its merge-base count (#40193), and the
guardrail added sixteen strict-rule errors it previously absorbed under the global budget.
- The parsed gateway response goes through `_as_object` once, so `action`, `blocked_reason`
and the rewrite fields are read from `dict[str, object]` rather than `dict[Unknown, Unknown]`;
`blocked_reason` is only used as the message when it is a non-empty string.
- `get_config_model` returns the concrete `type[AiriaGuardrailConfigModel]`, and the config model
carries the same pyright ignore as its siblings for the unparameterised base.
- The remaining untyped upstream members (the decorator, the client factory, `super().__init__`'s
kwargs, the handler's `post`) are ignored per line with the reason, as the Wingback guardrail
does. The registry reads `guardrail["guardrail_name"]`, a required TypedDict key.
- Tests reach the injected mock and the handler timeout through typed helpers instead of the
real handler's partially typed attributes.
No behaviour change: 46/46 tests, ruff default and strict, and both lint gates pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(guardrails): clear the Airia guardrail's type-discipline gate violations
Drop the mutable-ok comments that suppress nothing (LIT013), give the two
deferred-import noqa lines a reason (LIT003), and annotate the parsed JSON
object as Mapping instead of dict (LIT001).
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): propagate cache_control through responses->chat completion bridge
User/assistant messages, tool call outputs, and function calls built from
Responses API input items were dropping cache_control, breaking prompt
caching for Bedrock/Anthropic when routed through the litellm_completion
transform. Also propagate cache_control onto the last content element of
user/assistant messages, since Anthropic/Bedrock read it from the content
block rather than (or in addition to) the message level.
Adds the missing cache_control field to ChatCompletionToolMessage and
ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe.
* chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control
Adding cache_control to the ChatCompletionToolMessage TypedDict changed
the proxy's OpenAPI spec; regenerated via `npm run gen:api`.
* fix(responses): don't overwrite existing content-block cache_control; add list-content test
Per Greptile review on the PR:
- The message-level cache_control was unconditionally overwriting the
last content block's cache_control when that block already carried
its own (preserved by _transform_responses_api_content_to_chat_completion_content).
Now only propagates onto the last block when it doesn't already have one.
- Added the missing test coverage for the content-block propagation path
with list content, which no existing test exercised (all used plain
string content, so the isinstance(content, list) branch was never hit).
* style(responses): extract cache_control into a named variable, trim comment
Per style review: the message-building branch read cache_control 3
different ways in 5 lines (.get() twice, then [] indexing once), unlike
the sibling tool-call-output/function-call functions this PR touches,
which each extract it into a named variable up front
(output_cache_control, function_call_cache_control). Matched that
convention here too. Also trimmed the 3-line inline comment to one
line, matching the file's usual comment length for similarly subtle
behavior.
* chore: refresh lazy OpenAPI snapshot
* fix: type cache control propagation
* fix: place cache control type suppressions
* style: format cache control type conversion
* refactor(responses): build bridged cache_control messages in one shot
* fix(responses): keep cache_control marks on bridged tool calls and within the provider cap
A function_call item's mark now rides on the tool call the bridge builds, an assistant text
item merged behind a function_call keeps its own mark, a developer item keeps its mark when
it is mapped to a system message, and a previous_response_id replay yields its stale marks
to the newest ones so the provider cap holds. The Responses layer counts the marks on
function_call and function_call_output items before it places an ordinal injection point,
and the chat pass replaces that count with its own instead of adding to it.
* fix(responses): count each cache_control mark once and keep the bridge within the provider's limits
List content carries an item-level cache_control on its last block only, so the
provider sees one cache point and the breakpoint census counts one. Marked function
tools and SDK-typed input items are handled by the Responses-layer reserve, assistant
text merged into a function call turn keeps its mark on its own block, and the
previous_response_id replay budget drops marks per turn, newest kept, instead of
stripping every older turn once one overshoots.
The request's own marks now fall under the same budget, newest kept, so five marked
items no longer answer the provider's 400. A cache_control value that is not an
ephemeral mark with an optional ttl is left off the bridged message instead of
reaching the provider as sent, and a developer message maps to the system role with
its content, name, and cache_control only, so unknown keys never reach a strict
provider.
* fix(responses): send item cache_control only to cache-point targets and follow drop_params
Item marks reach the provider only when the target turns them into cache
points, and every other target keeps main's behavior of stripping them.
Without drop_params a mark the provider would refuse reaches it as sent,
so the client gets the same 400 as on /v1/chat/completions. With
drop_params a malformed mark is dropped and only the newest 4 are kept.
The shared input-to-messages helper and the developer mapping in
base_utils go back to main, and a marked developer item reaches a
cache-point target as a system message in the bridge.
* fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers
* fix(responses): validate replayed marks and count input tokens without item marks
* fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks
* test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets
* fix(responses): keep item marks off inspection paths and match each chat route's keep rule
Only the request the bridge sends gets item marks now. Guardrails, token
counting, rate limiting, and input_tokens read the input with no item marks,
as on main, so an off-spec mark cannot fail a guardrail call. Tool message
and tool call marks are typed inside the bridge, leaving the shared chat
types and guardrail schemas as on main.
Databricks keeps marks only for Claude, Snowflake uses its chat route's
Claude check, and every Anthropic deployment keeps them whatever its model
name. The injection census skips SDK-typed tools, and the owned-proxy test
expects the marked messages on /v1/responses.
* test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census
* fix(responses): leave a non-list input untouched when stripping item marks
* test(responses): cover string, list, object and tuple input with and without item marks kept
Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter
* test(cache): derive owned-proxy cell timeouts from the harness boot budget
The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
OpenRouter basic translation expected the pre-#44788 response shape. Read
refusal, logprobs and system_fingerprint from the fixture at their new
positions.
The team model TPM test required exactly one of three concurrent requests
to be served. The pipelined limiter can reject all of them, which is an
accepted trade-off, so assert no over-admission and that a lone retry is
served when the burst got nothing.
The Voyage host tests count every host their proxy dials. The responses
cost poller could dial api.openai.com for background-response rows that
other tests left in the shared worker database, so turn polling off in
those proxies.
Through a transaction-mode pooler Prisma takes pg_advisory_lock(72707369) on one
backend and releases it on another, so the first backend keeps the lock while idle
and the next boot waits on it. Migrations now run with
PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK=1 while LiteLLM holds the coordinator lock and
Prisma's key as transaction-level locks on one pinned connection, so nothing can
outlive the migration
Resolves LIT-9571
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cancelling a task mid-request closed the query engine socket, the engine
handed the still-busy pool connection back, the next transaction start
queued its BEGIN behind the orphaned statement, and when that waiter was
cancelled too the engine panicked on its dropped reply channel
(interactive_transactions/actors.rs) and exited under the shutdown spend
flush, losing daily user and team spend for requests that had returned 200.
Every engine request now runs in its own task behind asyncio.shield so only
the caller is cancelled, and a transaction that opens after its caller left
is rolled back.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): add fallback_on_unsupported_params so client-side param rejections can raise instead of falling back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(router): wrap the fallback gate so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): rename the flag to fallback_on_bad_request and cover provider 400s too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): narrow the wire test responses so the type gate count does not grow
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(openai): swap the model map with setattr so the type gate count does not grow
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): import the exception classes directly so the type gate count does not grow
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): type the fallbacks list in fallback_event_handlers to pay for the new Router kwarg in the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): keep the LIT001 count flat on the typed fallbacks parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): type the fallback stream wrapper weakref so the type gate stays level
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): make fallback_on_bad_request updatable at runtime so DB-stored router_settings apply it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for UpdateRouterConfig.fallback_on_bad_request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): rename the flag back to fallback_on_unsupported_params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(openai): drop the reasoning_effort=none client-side check, the Router flag covers the provider 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rerun docs checks now that the router_settings row is on litellm-docs main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): only stop fallbacks on real 400s and cover the context window and content policy exclusions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): move the no-fallback 400 check into is_bad_request_without_fallback and pin the 402 and Vertex 403 cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep falling back on router-raised 400s when fallback_on_unsupported_params is off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read llm_provider off the exception with getattr so the type gate stays flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response
* fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body
* test(anthropic): use Anthropic's documented refusal example as the unit-test fixture
* fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks
* fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields
* test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos
* fix(caching): skip a cached completion_tokens_details that fails validation on stream replay
* fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response
Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays
* test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(caching): collect the Redis circuit-breaker state gauge as one series per state across worker processes
The gauge declares multiprocess_mode=livesum, so PROMETHEUS_MULTIPROC_DIR apps get one series per state summed over
the live workers instead of one series per worker pid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): register Redis circuit-breaker metrics only when LiteLLM prometheus logging is on
The breaker collectors join the default registry once the prometheus or prometheus_system callback is configured or
a Prometheus logger starts, with earlier breaker events tallied and seeded in, and
litellm.disable_redis_circuit_breaker_metrics turns them off for apps that export their own metrics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): drop the disable_redis_circuit_breaker_metrics opt-out
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): keep the Redis breaker metrics gate to a per-state count and a publish hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): pin the Redis breaker metrics gate against an injected registry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): publish the breaker collectors before reading them in the existing metrics test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(integrations): build the Redis breaker collectors with typed prometheus_client constructors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): keep only the raising call inside pytest.raises for the pre-content stream tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): yield the primed first chunk from a helper generator so the stream iterators keep their complexity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): expect the empty invoke stream to be reported from router.completion now that the first chunk is read there
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: give router stream mocks a first chunk or an ending so the eager first read neither hangs nor retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): name the first-chunk helpers after what they do (_read_first_chunk, _FirstChunk, _chunks_with_first)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): drop the string annotation on _FirstChunk.chunk, ModelResponseStream is imported at runtime
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add known-bug translation cases for request fields LiteLLM drops
service_tier on /v1/messages, conversation on /v1/responses and metadata on
/v1/chat/completions never reach the provider. Each gets a TranslationTestCase
with the correct provider body, marked xfail(strict=True) through the new
TranslationTestCase.known_bug field, plus a passing service_tier case for
/v1/messages on the openai route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): widen the known-bug cases to azure_ai, bedrock_mantle, openai/responses and azure deployments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): forward service_tier on /v1/messages for anthropic, azure_ai and bedrock_mantle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): let proxy admins open team member defaults to team admins
* fix(proxy): keep a team admin's never-resets duration on a team's first member budget
* test(proxy): pin the existing-row and stored never-resets cases for member defaults
* fix(proxy): evict the cached user on every proxy for every /user/update and /user/bulk_update
Previously the cached user row was only evicted and broadcast when the
update touched max_budget, model_max_budget, tpm_limit, rpm_limit or
metadata, so a models or user_role change kept being served by peer
proxies until the cache TTL expired
* test(proxy): cover user cache eviction past the first bulk invalidation batch
* fix(config): treat blank numeric env values as unset instead of crashing at import
A numeric knob exported as an empty string, which is what a Helm value: "" or an
emptied .env line produces, raised ValueError inside litellm/constants.py and the
proxy crash-looped before serving traffic. Every int(os.getenv(...)) and
float(os.getenv(...)) in constants.py now goes through get_env_int / get_env_float,
which treat blank as unset and fall back to the default with a warning on garbage.
* fix(guardrails): treat blank litellm_params values as unset so the guardrail still loads
A guardrail whose litellm_params read os.environ/X resolves X to "" when the
variable is exported but empty. Pydantic rejected "" for bool and typed fields
such as default_on and fail_on_error, the boot log reported the guardrail as
skipped and the proxy came up without it. A before-validator on BaseLitellmParams
now drops blank string values so each field takes its default, and the
default_on shortcut in LitellmParams treats a blank value as False.
* fix(config): warn once about an unparsable numeric env value so logging cannot recurse
The LiteLLM loggers' filter reads MAX_STRING_LENGTH_STDOUT_LOG and MAX_BASE64_LENGTH_STDOUT_LOG through get_env_int on every record. Warning about a garbage value from inside that read re-entered the filter, so every log call raised RecursionError. The value is now marked as reported before the warning is logged, which also stops per-record readers from repeating it. get_env_int_or_none and the new get_env_float_or_none share the same parser
* fix(config): treat blank values as unset for the remaining import-time numeric knobs
LITELLM_MAX_STREAMING_DURATION_SECONDS, REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES, a whitespace CLI_JWT_EXPIRATION_HOURS, LITELLM_CONFIG_PARAM_CACHE_TTL_SECONDS, the agent health check timeouts and the Prisma reconnect knobs still crashed the proxy at boot when blank. A blank or unparsable REQUEST_TIMEOUT no longer counts as explicitly set, so chat completion() keeps its 600s fallback
* fix(guardrails): map blank litellm_params to the field default so PATCH can still clear a value
Dropping blank keys before validation made PATCH /guardrails/{id} with api_base "" a silent no-op. Blank values of optional fields now resolve to the field default and stay in the set fields, so the PATCH clears the stored value. Required fields and unknown keys keep the value as given
* test: cover proxy numeric knobs in-process and expect no warning for a blank Straiker api_version
The three proxy import-time numeric knobs move into litellm/constants.py so the existing in-process blank-env test covers them, replacing the subprocess test. A blank api_version now means unset, so the Straiker test asserts the key-prefix version with no warning.
* fix(guardrails): keep LitellmParams kwargs typed without adding argument-type errors
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(vllm): write spend log rows for /vllm passthrough calls to router models
The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call.
* ci: run the vllm passthrough unit tests in the llms-providers shard
* fix(proxy): log /vllm text completions and ask router models for stream usage
A chat stream relayed to a router model without stream_options now gets
include_usage added on the way out, and the usage chunk vLLM appends is
removed before the client sees it, so the spend row carries vLLM's own
token counts instead of a local estimate. /vllm/completions calls now
write a row too, stream and non-stream.
* fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams
Inject and strip stream usage on /vllm/completions too, hold every finish
frame for the fingerprint when n > 1, pass a forced-usage server's stream
through untouched, stop turning continuous_usage_stats alone into per-chunk
usage, close the upstream when the client hangs up so the row is written,
and assemble text choices with one sort instead of a scan per choice
* fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint
A held finish chunk now goes out as soon as any other chunk arrives, so a
choice that finishes while another still streams keeps its place. Only the
finish chunks right before the usage chunk get vLLM's system_fingerprint.
* fix(vllm): keep caller stream_options and log both success handlers
Forward a stream_options the proxy cannot read, or a non-boolean
include_usage, to vLLM as the caller sent it. Only a false
always_include_stream_usage changes the /vllm relay, so clients get the
same stream bytes under every setting.
Record the passthrough origin when logging starts so the sync and async
success handlers both parse the relayed body, hand callbacks a parsed
text completion, and read relayed messages and prompt separately so a
stray field no longer drops the local count.
Add the audit's integration tests for the relay, the settings, and the
Azure spend row.
* test(vllm): expect the caller's stream under always_include_stream_usage true
* fix(vllm): relay long SSE frames in linear time
* test(vllm): time the relay's usage and finish_reason prefilter on one large frame
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): close sync streams so the connection goes back to the pool
CustomStreamWrapper gets a sync close() (plus context manager support) that
closes the wrapped provider stream, and both the sync and async wrappers close
the provider stream when the stream reaches its end. The Anthropic stream
iterator keeps the httpx response it reads so close()/aclose() can release the
socket after message_stop instead of leaving it checked out of the pool
Co-authored-by: Blair Jordan <blair.j.jordan@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): keep the basedpyright ceiling flat in CustomStreamWrapper.close()
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): close the response on the shared sync stream path too
BaseModelResponseIterator gets a sync close() next to its aclose(), and the
sync branch of BaseLLMHTTPHandler hands it the httpx response the way the
async branch already does, so providers built on it release the socket on
close() instead of waiting for garbage collection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(streaming): unit tests for CustomStreamWrapper.close() and the shared iterator close()
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): close the provider stream before re-raising a mid-stream error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Blair Jordan <blair.j.jordan@gmail.com>
* fix(s3_v2): bound each upload attempt, add jitter, retry on 429 and 503
Each S3 PUT now carries a per-attempt httpx timeout read from
S3_UPLOAD_ATTEMPT_TIMEOUT_SECONDS (default 30s, clamped to 1..3600) instead
of inheriting the handler's unbounded default. Retries back off with jitter
instead of a fixed 1s then 2s, 429 joins 403, 500 and 503 in the in-call
retry set, and transport errors or litellm.Timeout on an attempt are retried
in the same call instead of ending it. Each flush logs one warning per batch
of dropped uploads that names how many were terminal and how many aged out.
* fix(s3_v2): retry uploads in a bounded loop and align tests with the new retry contract
The async and sync retry helpers called themselves, which the recursive-function check rejects. They are now a bounded loop over the attempt count with the same outcome.
The 429 integration test now expects in-call jittered backoff, and the aged-out test matches the new drop warning text. The sync transport test drives the real HTTPHandler so a dropped per-attempt timeout or a missed litellm.Timeout fails it, and the drop-reason test uses unequal counts so swapped labels fail.
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Drain the worker's fake-upstream request log when a test's gateway fixture starts, so traffic left by earlier tests on the same xdist worker no longer shows up in a test's first observation read
Generate the fine-tuning rig's provider file and job ids per fixture build, so a rebuilt module fixture in the same worker no longer collides on the managed object unique constraint
Wait for every wedged team_id and team_alias invalidation publish to fail before taking the recovery marker, now that team updates publish both keys
Pin the cooled-down priority scheduler request to the connection that observed the cooldown, since a sibling worker reads cooldowns from Redis on an interval
Simulate upstream outages and the held budget peer on ports the test keeps owning, instead of releasing a port another worker can bind
Start a peer proxy for the accounting group, which has a test that needs one
The gauge declares multiprocess_mode=livesum, so PROMETHEUS_MULTIPROC_DIR apps get one series per state summed over
the live workers instead of one series per worker pid
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude
Vertex AI Claude deployments that emulate response_format through a forced
tool sometimes nest the schema object under an extra key such as
`parameters` or `input`, and that envelope reached message.content with a
200. The converter now unwraps exactly one nested object that matches the
schema, leaving a payload that is itself schema-shaped alone.
Native structured-output rows sent the deprecated top-level `output_format`,
which Vertex rejects with a 400. The Vertex request now carries it as
`output_config.format`, with an explicit caller value winning.
* fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers
* test(anthropic): expect the json tool description through a shared constant
* fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys
* fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key
An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name
wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a
streamed answer whose first key is a schema property named name or tool_name waits for its second
key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided
past 4 KB is held to the end
* test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema
* fix(vertex_ai): unwrap claude json_schema answers nested under up to three wrapper keys
A tool-emulated json_schema answer was unwrapped only when the real object sat one key down. In live QA, 1 of 58 answers (/v1/responses, claude-sonnet-5) came back as {"parameter": {"value": {...}}} and reached the client still wrapped
The unwrap now follows single-key wrappers up to three levels down and stops at the first object that has a key the schema declares. That object is returned only when it has every required key. A payload wrapped deeper than three levels, a level with more than one key, and a level whose key is one of the schema's own property names all come back exactly as they arrived
The streamed path needs no change because the buffered answer goes through the same unwrap when the tool block closes, and a test now pins that both paths agree
* fix(bedrock): unwrap streamed claude json_schema answers on claude platform and mantle
A streamed json_schema answer on bedrock/claude_platform and Bedrock Mantle Claude reached the client as tool_calls deltas holding the json_tool_call wrapper, with finish_reason tool_calls and no content. The shared HTTP handler built the async stream iterator without json_mode, and neither stream path passed the response_format schema
The handler now asks the provider config for the iterator through get_model_response_iterator_for_request, which carries json_mode and the request's optional params on both the sync and async paths. Claude Platform and Mantle override it to hand the json tool schema to the Anthropic stream iterator, so the streamed answer is the bare schema object with finish_reason stop, the same as the non-streamed one
Databricks is the only other provider on this handler whose iterator reads json_mode, so its async Claude json mode stream now returns the answer as content like its sync stream already did
The stream family integration test covers claude_platform_unmapped and bedrock_mantle again
* fix(anthropic): unwrap json tool envelopes by structure instead of schema validation
The unwrap now reads only the schema's top-level property names. A lone
undeclared key holding an object, alone or beside a name equal to the json
tool's own name, is the envelope, and the unwrap repeats while the inner
object shares no key with the schema. jsonschema and referencing are no
longer imported on the request path. The stream holds only an answer that
opens with the tool's name or with an undeclared key holding an object, and a
schema without top-level properties streams as it arrives.
* fix(llms): keep other providers' async streams as they were
The request-aware iterator hook passed json_mode on async streams to every provider on the shared HTTP handler, which changed Databricks and any other provider whose iterator reads it. The default hook now makes the same two calls the handler made before, so only Claude Platform and Bedrock Mantle change, and the Databricks test that pinned the wider change is removed
* fix(anthropic): walk json tool envelopes iteratively
* test(llms): pin which stream iterators get json_mode from the shared handler
A provider config that does not override the request-aware iterator hook still gets json_mode on a sync stream and still does not get it on an async stream, the same as before the hook existed
* fix(anthropic): keep json tool arguments too deep to decode instead of raising
* fix(anthropic): unwrap json tool envelopes beside undeclared string labels
* fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent
A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter.
* fix(anthropic): join a json_schema answer the model splits into one tool block per field
With tool emulation the model sometimes answers with one json_tool_call block per schema field, each naming the field and holding its value. The non-stream path returned only the first block and the stream sent every block's text back to back, so the client got one field or invalid JSON. Both paths now build one schema object from those blocks, and any other set of blocks keeps today's output
* refactor(anthropic): set the streamed finish reason in one expression
* test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence
* fix(anthropic): unwrap a streamed values wrapper like the non-stream answer
* fix(anthropic): join streamed split answers on schemas without required keys, keep client tool calls and length on json_schema streams
* fix(anthropic): join streamed split answers that carry text labels on schemas without required keys
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sends
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sent raw
* fix(anthropic): join a split json_schema answer whose fields all hold the same string
* fix(anthropic): join a split json_schema answer sent as one partial object per field
* fix(anthropic): join field blocks after an empty streamed answer without a leading comma
* fix(proxy): check previous_response_id ownership on /v1/responses/compact
* test(proxy): cover the unmanaged response id escape hatches on every response id route
* test(proxy): audit compact previous_response_id ownership
Add integration cells for the native /v1/responses/compact routes covering owner,
admin, cross-tenant, unmanaged, malformed, duplicated and retained ids, provider
errors, the response cache, id-security settings, an upstream outage, a held
upstream, a worker kill and a proxy restart. Unskip the provider wire cell whose
bug this branch fixes.
* test(proxy): cover the no signing key path on every response id route
* feat(proxy): add deployment warning endpoints with instance-wide ignore and snooze
GET /health/deployment_warnings returns the active deployment warnings
(env credential login, license expiry, no Redis on multiple workers,
detailed debug logging) with a severity and ignore state, for proxy admins
and proxy admin viewers. PATCH /health/deployment_warnings/{warning_id}
lets a proxy admin ignore a warning forever or snooze it for up to 30 days,
or restore it. Ignores live in one LiteLLM_UISettings row, apply to every
admin, and are written to the config audit log. Expired snoozes drop out
on read, and LITELLM_DISABLE_NO_REDIS_WARNING is reported as an env var
ignore. /health/readiness/details is unchanged
* fix(proxy): keep readiness Redis short-circuit ahead of the env lookup and harden ignore storage
Readiness details again answers from Redis before reading
LITELLM_DISABLE_NO_REDIS_WARNING, so proxies with Redis and a secret
manager do not fetch it on every poll. Stored ignores require timezone
aware datetimes and are read from the writer before a rewrite
* fix(proxy): keep deployment warning ignores readable in the config audit log
The audit payload keyed ignores by warning id, so the secret matcher
redacted the env_credential_login entry and the audit row lost who
snoozed it and until when. Log a list of id and ignore entries instead
* fix(vertex_ai): pass the per-request timeout to Gemini streaming calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vertex_ai): pass timeout to make_call and make_sync_call in the http 201 tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): default timeout to None on make_call and make_sync_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: lower aiohttp keepalive timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): restore the 15s keep-alive default and apply it to the Rust client
15s is aiohttp's own default and what every release before v1.78.0 used. The Rust HTTP client reads the same AIOHTTP_KEEPALIVE_TIMEOUT variable, so its default moves with it.
* test(integration): cover upstream connections dropped after the keep-alive idle window
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): dedupe blank-question issues and label Decisions questions by key, not position
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): make the Response card header a flex row so title and badges share one line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): fail when duplicate blank-name issues stop being deduped
* fix(ui): drop the Decisions issue dedupe, which no reachable endpoint needs
* fix(ui): center the Response header and test the Decisions issue labels end to end
The card's border rule set 24px of bottom padding on the Response header, which put its content 8px above center. The header now keeps 8px on both sides. New tests cover the /v1/decisions position label and the score-level warning label for an all-digit key, which mutations of validatePayload left untested
* test(ui): check the Decisions Response header layout in the browser
* test(ui): register the Decisions header spec's model through the API so it runs in CI
The CI UI stack does not read fixtures/config.yml, so the config-only fake-decisions-model never existed there and the spec could not pick it. The spec now creates a uniquely named typesafe model against the mock server, waits until /model_group/info lists it as an evaluation model and /v1/systemone answers, and deletes it afterwards
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
