* feat(guardrails): add Airia guardrail integration
Adds `airia` as a built-in guardrail provider. The hook sends the prompt under `pre_call` and the
model's response under `post_call` to the Airia AI Gateway, which answers allow, block, or redacted
content; the only configuration is `api_base`, `api_key`, and an optional `timeout`.
`during_call` is not offered: it runs concurrently with the model call, so a block could land after
the prompt has already reached the provider.
A BLOCKED verdict, or any action this version does not recognise, raises with
`blocked_content=True`; a transport error, a non-2xx, or a response body that is not a JSON object
raises with `blocked_content=False`, so callers can tell "could not evaluate" from "evaluated and
blocked" while both still fail closed. On GUARDRAIL_INTERVENED a copy of the inputs is returned with
every rewritten field substituted; an intervention carrying no applicable rewrite, a rewrite of the
wrong shape, or a rewrite whose item count does not match what was sent (which would misalign the
positional write-back downstream) blocks instead of letting the original through.
Streamed responses are moderated whole and then emitted redacted: the hook opts into the unified
hook's `incremental_diff` mode (the default `block_only` drops rewrites) with
`streaming_end_of_stream_only`, so a redaction can never span transform rounds and underflow. Covered
by a test that drives the real streaming iterator hook end to end and checks the assembled client
output, not just the guardrail's own config attributes.
The HTTP client is dependency-injected (an `async_handler` constructor parameter, defaulting to a
real client) rather than mutated post-construction, so tests inject a fake client instead of
monkeypatching an instance method.
Known limitation, not fixable from this guardrail alone: LiteLLM's unified guardrail framework
yields a tool-call chunk to the client immediately during `incremental_diff` streaming, inspecting
the assembled tool calls against the guardrail's verdict only once the stream ends
(`_run_incremental_transform_stream` in `unified_guardrail.py`). A blocked tool call can therefore
reach an executing client before the terminal error arrives. Switching to `block_only` with
`streaming_buffer_until_moderated` does not close this without a regression: that flag has no effect
under `incremental_diff` (the mode branches away before it is read), and `block_only` itself replays
the original, unredacted chunks on release, silently dropping any REDACT verdict on streamed text.
Relatedly, LiteLLM's OpenAI chat completions guardrail translation skips `post_call` guardrails
entirely for a response with no text content, so a tool-call-only response bypasses this guardrail
(and every other post-call guardrail) even outside streaming. Both are shared-framework behavior,
not specific to Airia; fixing either is out of scope for an isolated single-guardrail contribution.
* fix(guardrails): fail closed on an Airia rewrite for a field that was not sent
Bugbot finding on #40784. An INTERVENED verdict carrying `structured_messages: []` with
`texts` omitted passed `_rewritten` unchanged: an empty list has the same length as an
omitted field, so it counted as an applied rewrite while the `texts` that actually carried
the content went through untouched. post_call sends only `texts`, so this was reachable.
A rewrite for a field the request did not send is now a mismatch, same as a wrong length,
and the verdict blocks. Regression test covers both the empty and the non-empty case.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(guardrails): fail closed on streamed non-chat routes the Airia rewrite cannot reach
Veria finding on #40784. incremental_diff, the mode that carries a rewrite into the stream,
exists only for OpenAI chat completions; UnifiedLLMGuardrails sends every other streamed route
(/v1/responses, /v1/messages) to block_only, which streams the original text live and drops
rewrites. So with post_call on, a streamed Responses or Anthropic call could receive text that
Airia then blocked or redacted.
Two changes, both confined to this guardrail:
- streaming_buffer_until_moderated is set, so on the block_only fallback the framework holds
every chunk until the end-of-stream verdict instead of releasing it live. The flag is never
read on the incremental_diff path, so streamed chat completions are unchanged.
- A REDACT verdict on a streamed response whose route cannot carry a rewrite is enforced as a
block, since block_only would otherwise replay the original. The route check mirrors
_resolve_transform_call_type so the two cannot disagree; an unresolvable route fails closed.
Request-side rewrites and non-streamed calls are untouched.
The end-to-end test drives the real UnifiedLLMGuardrails iterator over an Anthropic
/v1/messages SSE stream and asserts not one original chunk is released for either verdict.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(guardrails): keep the Airia stream test's pytest.raises block to one statement
CI's test-tree ruff run flags PT012 on the non-chat streaming test: the `async for` that
collected released chunks sat inside `pytest.raises`, so a raise from the loop body would be
indistinguishable from the one the test is about. The loop now lives in a `_drain_into` helper
and the `with` block awaits it alone. Same assertion, same partial-leak capture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(guardrails): keep the Airia guardrail within the per-rule basedpyright ceilings
scripts/type_check_gate.py now gates every rule on its merge-base count (#40193), and the
guardrail added sixteen strict-rule errors it previously absorbed under the global budget.
- The parsed gateway response goes through `_as_object` once, so `action`, `blocked_reason`
and the rewrite fields are read from `dict[str, object]` rather than `dict[Unknown, Unknown]`;
`blocked_reason` is only used as the message when it is a non-empty string.
- `get_config_model` returns the concrete `type[AiriaGuardrailConfigModel]`, and the config model
carries the same pyright ignore as its siblings for the unparameterised base.
- The remaining untyped upstream members (the decorator, the client factory, `super().__init__`'s
kwargs, the handler's `post`) are ignored per line with the reason, as the Wingback guardrail
does. The registry reads `guardrail["guardrail_name"]`, a required TypedDict key.
- Tests reach the injected mock and the handler timeout through typed helpers instead of the
real handler's partially typed attributes.
No behaviour change: 46/46 tests, ruff default and strict, and both lint gates pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(guardrails): clear the Airia guardrail's type-discipline gate violations
Drop the mutable-ok comments that suppress nothing (LIT013), give the two
deferred-import noqa lines a reason (LIT003), and annotate the parsed JSON
object as Mapping instead of dict (LIT001).
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): propagate cache_control through responses->chat completion bridge
User/assistant messages, tool call outputs, and function calls built from
Responses API input items were dropping cache_control, breaking prompt
caching for Bedrock/Anthropic when routed through the litellm_completion
transform. Also propagate cache_control onto the last content element of
user/assistant messages, since Anthropic/Bedrock read it from the content
block rather than (or in addition to) the message level.
Adds the missing cache_control field to ChatCompletionToolMessage and
ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe.
* chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control
Adding cache_control to the ChatCompletionToolMessage TypedDict changed
the proxy's OpenAPI spec; regenerated via `npm run gen:api`.
* fix(responses): don't overwrite existing content-block cache_control; add list-content test
Per Greptile review on the PR:
- The message-level cache_control was unconditionally overwriting the
last content block's cache_control when that block already carried
its own (preserved by _transform_responses_api_content_to_chat_completion_content).
Now only propagates onto the last block when it doesn't already have one.
- Added the missing test coverage for the content-block propagation path
with list content, which no existing test exercised (all used plain
string content, so the isinstance(content, list) branch was never hit).
* style(responses): extract cache_control into a named variable, trim comment
Per style review: the message-building branch read cache_control 3
different ways in 5 lines (.get() twice, then [] indexing once), unlike
the sibling tool-call-output/function-call functions this PR touches,
which each extract it into a named variable up front
(output_cache_control, function_call_cache_control). Matched that
convention here too. Also trimmed the 3-line inline comment to one
line, matching the file's usual comment length for similarly subtle
behavior.
* chore: refresh lazy OpenAPI snapshot
* fix: type cache control propagation
* fix: place cache control type suppressions
* style: format cache control type conversion
* refactor(responses): build bridged cache_control messages in one shot
* fix(responses): keep cache_control marks on bridged tool calls and within the provider cap
A function_call item's mark now rides on the tool call the bridge builds, an assistant text
item merged behind a function_call keeps its own mark, a developer item keeps its mark when
it is mapped to a system message, and a previous_response_id replay yields its stale marks
to the newest ones so the provider cap holds. The Responses layer counts the marks on
function_call and function_call_output items before it places an ordinal injection point,
and the chat pass replaces that count with its own instead of adding to it.
* fix(responses): count each cache_control mark once and keep the bridge within the provider's limits
List content carries an item-level cache_control on its last block only, so the
provider sees one cache point and the breakpoint census counts one. Marked function
tools and SDK-typed input items are handled by the Responses-layer reserve, assistant
text merged into a function call turn keeps its mark on its own block, and the
previous_response_id replay budget drops marks per turn, newest kept, instead of
stripping every older turn once one overshoots.
The request's own marks now fall under the same budget, newest kept, so five marked
items no longer answer the provider's 400. A cache_control value that is not an
ephemeral mark with an optional ttl is left off the bridged message instead of
reaching the provider as sent, and a developer message maps to the system role with
its content, name, and cache_control only, so unknown keys never reach a strict
provider.
* fix(responses): send item cache_control only to cache-point targets and follow drop_params
Item marks reach the provider only when the target turns them into cache
points, and every other target keeps main's behavior of stripping them.
Without drop_params a mark the provider would refuse reaches it as sent,
so the client gets the same 400 as on /v1/chat/completions. With
drop_params a malformed mark is dropped and only the newest 4 are kept.
The shared input-to-messages helper and the developer mapping in
base_utils go back to main, and a marked developer item reaches a
cache-point target as a system message in the bridge.
* fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers
* fix(responses): validate replayed marks and count input tokens without item marks
* fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks
* test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets
* fix(responses): keep item marks off inspection paths and match each chat route's keep rule
Only the request the bridge sends gets item marks now. Guardrails, token
counting, rate limiting, and input_tokens read the input with no item marks,
as on main, so an off-spec mark cannot fail a guardrail call. Tool message
and tool call marks are typed inside the bridge, leaving the shared chat
types and guardrail schemas as on main.
Databricks keeps marks only for Claude, Snowflake uses its chat route's
Claude check, and every Anthropic deployment keeps them whatever its model
name. The injection census skips SDK-typed tools, and the owned-proxy test
expects the marked messages on /v1/responses.
* test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census
* fix(responses): leave a non-list input untouched when stripping item marks
* test(responses): cover string, list, object and tuple input with and without item marks kept
Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter
* test(cache): derive owned-proxy cell timeouts from the harness boot budget
The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Cancelling a task mid-request closed the query engine socket, the engine
handed the still-busy pool connection back, the next transaction start
queued its BEGIN behind the orphaned statement, and when that waiter was
cancelled too the engine panicked on its dropped reply channel
(interactive_transactions/actors.rs) and exited under the shutdown spend
flush, losing daily user and team spend for requests that had returned 200.
Every engine request now runs in its own task behind asyncio.shield so only
the caller is cancelled, and a transaction that opens after its caller left
is rolled back.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response
* fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body
* test(anthropic): use Anthropic's documented refusal example as the unit-test fixture
* fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks
* fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields
* test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos
* fix(caching): skip a cached completion_tokens_details that fails validation on stream replay
* fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response
Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays
* test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): let proxy admins open team member defaults to team admins
* fix(proxy): keep a team admin's never-resets duration on a team's first member budget
* test(proxy): pin the existing-row and stored never-resets cases for member defaults
* fix(proxy): evict the cached user on every proxy for every /user/update and /user/bulk_update
Previously the cached user row was only evicted and broadcast when the
update touched max_budget, model_max_budget, tpm_limit, rpm_limit or
metadata, so a models or user_role change kept being served by peer
proxies until the cache TTL expired
* test(proxy): cover user cache eviction past the first bulk invalidation batch
* fix(config): treat blank numeric env values as unset instead of crashing at import
A numeric knob exported as an empty string, which is what a Helm value: "" or an
emptied .env line produces, raised ValueError inside litellm/constants.py and the
proxy crash-looped before serving traffic. Every int(os.getenv(...)) and
float(os.getenv(...)) in constants.py now goes through get_env_int / get_env_float,
which treat blank as unset and fall back to the default with a warning on garbage.
* fix(guardrails): treat blank litellm_params values as unset so the guardrail still loads
A guardrail whose litellm_params read os.environ/X resolves X to "" when the
variable is exported but empty. Pydantic rejected "" for bool and typed fields
such as default_on and fail_on_error, the boot log reported the guardrail as
skipped and the proxy came up without it. A before-validator on BaseLitellmParams
now drops blank string values so each field takes its default, and the
default_on shortcut in LitellmParams treats a blank value as False.
* fix(config): warn once about an unparsable numeric env value so logging cannot recurse
The LiteLLM loggers' filter reads MAX_STRING_LENGTH_STDOUT_LOG and MAX_BASE64_LENGTH_STDOUT_LOG through get_env_int on every record. Warning about a garbage value from inside that read re-entered the filter, so every log call raised RecursionError. The value is now marked as reported before the warning is logged, which also stops per-record readers from repeating it. get_env_int_or_none and the new get_env_float_or_none share the same parser
* fix(config): treat blank values as unset for the remaining import-time numeric knobs
LITELLM_MAX_STREAMING_DURATION_SECONDS, REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES, a whitespace CLI_JWT_EXPIRATION_HOURS, LITELLM_CONFIG_PARAM_CACHE_TTL_SECONDS, the agent health check timeouts and the Prisma reconnect knobs still crashed the proxy at boot when blank. A blank or unparsable REQUEST_TIMEOUT no longer counts as explicitly set, so chat completion() keeps its 600s fallback
* fix(guardrails): map blank litellm_params to the field default so PATCH can still clear a value
Dropping blank keys before validation made PATCH /guardrails/{id} with api_base "" a silent no-op. Blank values of optional fields now resolve to the field default and stay in the set fields, so the PATCH clears the stored value. Required fields and unknown keys keep the value as given
* test: cover proxy numeric knobs in-process and expect no warning for a blank Straiker api_version
The three proxy import-time numeric knobs move into litellm/constants.py so the existing in-process blank-env test covers them, replacing the subprocess test. A blank api_version now means unset, so the Straiker test asserts the key-prefix version with no warning.
* fix(guardrails): keep LitellmParams kwargs typed without adding argument-type errors
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(vllm): write spend log rows for /vllm passthrough calls to router models
The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call.
* ci: run the vllm passthrough unit tests in the llms-providers shard
* fix(proxy): log /vllm text completions and ask router models for stream usage
A chat stream relayed to a router model without stream_options now gets
include_usage added on the way out, and the usage chunk vLLM appends is
removed before the client sees it, so the spend row carries vLLM's own
token counts instead of a local estimate. /vllm/completions calls now
write a row too, stream and non-stream.
* fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams
Inject and strip stream usage on /vllm/completions too, hold every finish
frame for the fingerprint when n > 1, pass a forced-usage server's stream
through untouched, stop turning continuous_usage_stats alone into per-chunk
usage, close the upstream when the client hangs up so the row is written,
and assemble text choices with one sort instead of a scan per choice
* fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint
A held finish chunk now goes out as soon as any other chunk arrives, so a
choice that finishes while another still streams keeps its place. Only the
finish chunks right before the usage chunk get vLLM's system_fingerprint.
* fix(vllm): keep caller stream_options and log both success handlers
Forward a stream_options the proxy cannot read, or a non-boolean
include_usage, to vLLM as the caller sent it. Only a false
always_include_stream_usage changes the /vllm relay, so clients get the
same stream bytes under every setting.
Record the passthrough origin when logging starts so the sync and async
success handlers both parse the relayed body, hand callbacks a parsed
text completion, and read relayed messages and prompt separately so a
stray field no longer drops the local count.
Add the audit's integration tests for the relay, the settings, and the
Azure spend row.
* test(vllm): expect the caller's stream under always_include_stream_usage true
* fix(vllm): relay long SSE frames in linear time
* test(vllm): time the relay's usage and finish_reason prefilter on one large frame
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): add deployment warning endpoints with instance-wide ignore and snooze
GET /health/deployment_warnings returns the active deployment warnings
(env credential login, license expiry, no Redis on multiple workers,
detailed debug logging) with a severity and ignore state, for proxy admins
and proxy admin viewers. PATCH /health/deployment_warnings/{warning_id}
lets a proxy admin ignore a warning forever or snooze it for up to 30 days,
or restore it. Ignores live in one LiteLLM_UISettings row, apply to every
admin, and are written to the config audit log. Expired snoozes drop out
on read, and LITELLM_DISABLE_NO_REDIS_WARNING is reported as an env var
ignore. /health/readiness/details is unchanged
* fix(proxy): keep readiness Redis short-circuit ahead of the env lookup and harden ignore storage
Readiness details again answers from Redis before reading
LITELLM_DISABLE_NO_REDIS_WARNING, so proxies with Redis and a secret
manager do not fetch it on every poll. Stored ignores require timezone
aware datetimes and are read from the writer before a rewrite
* fix(proxy): keep deployment warning ignores readable in the config audit log
The audit payload keyed ignores by warning id, so the secret matcher
redacted the env_credential_login entry and the audit row lost who
snoozed it and until when. Log a list of id and ignore entries instead
* feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos
DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model
Studio's async task API: create hits
/api/v1/services/aigc/video-generation/video-synthesis and status and content
poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the
qwencloud and qwen_ai_platform brand aliases, following the same
get_dashscope_family_*_config pattern image generation uses. Unlike the
Sora-shaped providers the request body is nested, so prompt and media go under
input and everything else goes under parameters. X-DashScope-Async is always
sent, because DashScope rejects the call outright without it
OpenAI params translate to DashScope's contract. seconds becomes duration, and
size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus
720P), since sending only the ratio leaves the request on the default 1080P and
silently bills at that rate. input_reference becomes a single media entry,
base64 encoded into a data URI for file inputs. Which media type a model takes
(first_frame or, on the reference-to-video models, reference_image) is read
from provider_specific_entry.dashscope_video_reference_input via
get_model_info rather than hardcoded; models without it predate the media
array and get the flat img_url field. A caller-supplied media array always
wins, as that is the only way to express multi-role combinations
The create call is the only billed call, so it bills what DashScope will
deliver: the requested duration and tier, or the documented 5 second and 1080P
defaults when they are omitted. Smart duration (duration -1) lets the service
pick the length after that call, so it is rejected rather than recorded at
zero, with a 400. Task creation and task lookup both report failures as a 200 carrying
top-level code and message with no output, so that shape is raised instead of
surfacing as a queued video with an empty id. Status responses re-encode the
deployment model from the polled id, so follow-up polls and downloads route to
the same deployment. Timestamps parse as UTC+8. DashScope publishes no list,
delete or remix endpoint, so those raise rather than guess at a URL
The end-to-end tests drive the real avideo_generation() entrypoint through an
httpx MockTransport and assert on the JSON body httpx actually encoded. That is
what catches a mapper that drops input_reference before the request is built,
or a body fragment json cannot serialize, neither of which a test calling the
transform directly would see
Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v,
wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models,
for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output
second by resolution tier from Model Studio's USD rate cards. dashscope and
qwen_ai_platform default to the Beijing host and use its rate card; qwencloud
defaults to the international host and uses Singapore's. The three
*_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): fail loudly on video responses that cannot be read
Addresses the review on the video generation PR. A body that is not the
documented task shape, an output block missing task_id or task_status, an
undocumented task_status and a missing output all raise DashScopeVideoError
now, instead of defaulting into a queued video with an empty id that callers
poll forever. A JSONDecodeError from a gateway response no longer escapes as
a generic connection error.
task_id and task_status are required on _TaskOutput, and a missing output
raises even when the body carries no code or message.
Download failures move into the 4xx range, since a 5xx has the proxy retry
them: 409 while the task is still running, 400 for a task that failed, 404
for an id that expired. The status a provider reported is kept when it
reported one, so an in-body failure that arrives on a 200 is billed as the
400 it is rather than as a retryable connection error.
characters and image are only dropped when drop_params is set; the params
that are rebuilt into the body or consumed by the caller (model, prompt,
user, extra_*) stay silently skipped, since the proxy stamps user from the
caller's end-user header.
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): clear the lint gates after merging main
* fix(dashscope): accept file tuples and paths as video input_reference
video_generation() types input_reference as FileTypes, which includes
(filename, content[, content_type[, headers]]) tuples and pathlib paths,
but the DashScope encoder only read bytes and file objects, so those
image-to-video calls raised before reaching DashScope. Bare strings stay
URLs and are never opened as paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dashscope): snap size to a supported ratio and read file tuples and paths
* test(dashscope): assert whole mapped values and bill against hand-written rates
The tier pricing test read its expected rate from the same map the cost
calculator uses, so a wrong rate passed. Billing is now checked against
fixed rates driven through the status response usage, while the map test
only asserts every supported tier has its own rate. Size mapping tests
compare the full mapped dict, and the single-test wiring class is a
plain function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): download content through the caller's client and stop retrying dead tasks
Two Cursor Bugbot findings on the DashScope video PR.
The DashScope artifact lives on OSS, so transform_video_content_response makes a
second request. It built its own cached client, discarding an injected client and
its ssl_verify. Thread the client through the content response the way
transform_video_status_retrieve_response already does, and use it in the three
providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The
remaining providers accept it and read the response body, matching how they
already accept an unused client on status retrieve.
A FAILED or CANCELED task reported without a code or message fell past the
in-flight check into the 502 branch, so the proxy retried a task that can never
produce a video. Raise 400 for a terminal status instead.
* fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes
A download that lands while the task is still PENDING or RUNNING answered 409. That
reads as a client error, which was the intent, but litellm's retry predicate counts 409
as retryable, so the proxy polled DashScope twice more before giving up and then counted
the caller's own early download against the model's error rate
404 is what the OpenAI videos API documents for content that is not ready, and it is the
one status in that family the retry predicate leaves alone. The rest of the mapping is
unchanged: 400 for a task the provider reported as failed, 404 for an expired task id,
502 when a succeeded task carries no artifact
The download tests now drive avideo_content and assert the caller sees NotFoundError
rather than an internal server error
* fix(dashscope): route a video poll back to the model group that created the task
The create response encoded the upstream model name into the returned id. Both the
dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts,
so a proxy with a deployment per region had deployments that only the model group told
apart. Encoding the bare model name resolved a poll to whichever deployment the router
lists first, which sent it to the other region's host with that region's key and a task
id that region never issued. That is the reroute the earlier proof records as the
dashscope and qwen_ai_platform poll ids coming back tagged qwencloud
The id now carries the model group the create was routed to, which is what the proxy
resolves a poll back to, and falls back to the model name for a direct SDK call that has
no router
The regression test drives a real router with one deployment per region, under a
different prefix and under the same one, and asserts the poll carries the key of the
region that created the task
* fix(dashscope): read the routed model group from litellm_metadata
The router stamps the model group it picked into litellm_metadata. metadata only
carries a copy when the caller sent no metadata of its own, and every proxy request
sends some, so a proxied create still encoded the bare model name and the poll
resolved to whichever region's deployment the router lists first
The regression test now also creates the video with caller metadata, which fails
without this change
* fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients
* fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body
* fix(dashscope): check video request values before the billed create and honor per-call drop_params
Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task,
pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so
the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and
refuse to map size onto models litellm has no price for.
* fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll
A key scoped to an alias or to a name a wildcard serves could create a
video and then got a 403 on the poll and the download, because the id
carried the group that served the call (wan3, qwencloud/*) and the key
was never given that name. The id now carries the name the router was
asked for on that hop, and the router resolves an alias, routing group
or wildcard name in an id back to itself when no deployment name or
provider model matches it first
* fix(dashscope): keep a routing group's video polls on the member that created the task
A routing group spans several model groups, so an id carrying the group's name
load-balanced its polls, and some left for another region's host and key with a
task id that region never issued. The id now carries the member that served the
task when the request named a routing group, and keeps the requested name for a
model group, an alias, or a wildcard match.
* fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region
* fix(proxy): pin a video's served group only after key and team aliases resolve
* test(integration): audit DashScope video create, poll, download, routing and chaos paths
* fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group
* test(routing): expect a forged served group to be dropped from the polled video id
* test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos
---------
Co-authored-by: AaronHowell <237895480@qq.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): build configured logging callbacks at boot so a broken callback fails startup instead of the first request
Startup already built every configured callbacks, success_callback and
failure_callback entry, but through the per-request helper that swallows
every construction error. A callback whose SDK could not be imported (for
example langfuse outside the supported version range) stayed a string, so
the first request on each worker rebuilt it inside function_setup and
returned a 500.
Startup now builds them with a variant that re-raises ImportError naming
the callback, which fails boot. Other construction errors keep the old
non-blocking behavior, and callbacks added at runtime keep the lenient path.
* refactor(proxy): build the configured callback replacements in one pass
Collect the startup-built callbacks with a generator and a frozen mapping
instead of a mutable dict filled in a loop, and drop a docstring that
restated the function body
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(ui): add decisions and classifier providers to the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): use literal upstream decisions paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: drop duplicate TYPESAFE provider fields entry superseded by main's TypeSafe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): drop stray blank line in provider_info_helpers test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the OSS decisions provider fields test
* fix(decisions): keep Laya and Bespoke env keys on their configured server
Route laya and bespoke Decisions deployments through the same oss_connection
and validate_oss_model rules the pass-through and the classifier router use,
so LAYA_API_KEY and BESPOKE_API_KEY only go to LAYA_API_BASE and BESPOKE_API_BASE,
an api_base with credentials, a query, or a non-HTTP scheme answers 400, and
an unknown checkpoint name answers 400 before any upstream call
* fix(decisions): send the Laya and Bespoke env key to a form-added model on the env server
A deployment added through Add Model always carries an API base, so the
env key only reached deployments with no base. The env key now goes to a
deployment whose base matches LAYA_API_BASE or BESPOKE_API_BASE, an empty
deployment key counts as no key, and the Bespoke placeholder is the name
Ollama serves.
* test(decisions): cover the env-base classifier case in the complexity router test
* fix(decisions): match the env server with or without /v1 and point Ollama users at bespoke/nimble
* fix(decisions): call /v1/systemone once from classifiers and pass-through on a /v1 base
* fix(decisions): strip /v1 from the API base path only, so a host named v1 still works
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(mcp): separate ID-JAG leg-2 client identity, support public resource clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(mcp): format ID-JAG credential changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): satisfy lint gates and regenerate API snapshot for ID-JAG resource client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): require a confidential ID-JAG resource client, drop none auth method
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): drop formatting churn from ID-JAG resource client tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger after flaky test_mcp_tool_search timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rerun against main after duplicate gemini-4-argon price keys were fixed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): send images to Cloudflare Clef
OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise.
* fix(decisions): follow drop_params for images and use the call timeout for image downloads
* fix(decisions): drop a malformed images entry on its own under drop_params
* fix(decisions): refuse images next to input whatever drop_params says
* fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs
Refuse more than 4 images for Cloudflare clef before any download, keep the
caller's timeout as one deadline across image fetch retries, accept images
next to input under drop_params, validate image strings and objects, and pass
the caller's image URLs (not inlined bytes) to logging callbacks.
* fix(decisions): check input_image URLs and clef image sizes
* fix(decisions): accept data URL media type parameters and check clef image types
* fix(decisions): count clef images after dropping unsupported types and flag clef vision
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Non-admin callers previously received deployment-wide cache activity,
error breakdowns, every key alias on the proxy, and could filter the
aggregates by any other tenant's key alias. All four queries now take
the caller's read scope as bind parameters, which also shrinks the
key-alias and model filter options to the caller's own visibility.
The three-way scope resolution (admin / internal user / fallback) now
lives in resolve_spend_log_read_scope, shared with /spend/tags and
/spend/logs/session/ui instead of being duplicated per endpoint.
* feat(claude-code): emit installationPreference in marketplace.json entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): satisfy marketplace CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): qualify marketplace entry as read only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): suppress typed marketplace assembly warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): restrict installation_preference to the documented values
Accept only available, auto_install, or required so a typo returns 422 instead of being silently dropped, and cover the PUT full-replace clearing behavior with a test
* fix(claude-code): echo installation_preference on the public skill hub
* refactor(claude-code): read installation_preference with one membership check and cover hand-edited rows
* test(integration): audit installation_preference on the Claude Code marketplace
* test(integration): kill a serving worker in the marketplace chaos cell
* test(integration): give the marketplace worker-kill cell the two-boot timeout
* fix(proxy): keep installationPreference on the MarketplacePluginEntry model
* fix(proxy): refuse self-installing preferences off sha256 archives, accept the camelCase key, and echo the preference on POST and PUT
* fix(claude-code): serve installation_preference only on archive plugins and refuse both key spellings
Claude Code 2.1.75 and older reject a marketplace.json once any plugin carries installationPreference,
and every Claude Code release before archive support rejects a catalog with an archive plugin. Requiring
an archive source for every value, available included, keeps the field off catalogs those clients can read.
A request that sends installation_preference and installationPreference together now gets a 422.
* fix(claude-code): read the preference spelling that carries a value when the other is null
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): let team admins create keys for team members behind an opt-in setting
A team admin calling POST /key/generate with a team they administer and the
user_id of another member of that team got a 403, and the dashboard only
offered the "Another User" owner option to proxy admins
Add a create_member_keys entry to team_admin_editable_team_fields. When a
proxy admin turns it on, a team admin can create a team key owned by another
member of a team they administer, through the API and through the Create Key
form. It stays off by default, so nothing changes until a proxy admin opts in
With the setting off, the 403 a team admin gets for this call now names the
setting to enable instead of saying they can only create keys for themselves
* fix(ui): order the member owner picker after the team and drop the viewer from it
For a team admin the owner picker depends on the team, so it now sits under the Team field, lists only the other members of that team, and the Team field says why it is required
* fix(proxy): keep keys a team admin creates for a member from carrying more than LLM access
A key authenticates as its owner, so with create_member_keys on a team admin could create a key owned by a proxy admin who sits in their team and act as a proxy admin with it
The member now has to hold the internal_user or internal_user_viewer role, and the key is always an llm_api key, so the team admin never holds a secret that reaches management routes. Asking for a management or read_only key for a member is refused
The unit tests now check the permission decision directly instead of replacing generate_key_helper_fn, and the saved key is checked against a real proxy in the team admin gate matrix
* style(proxy): sort the team admin field permission imports
* refactor(proxy): force the member key type on the request instead of merging a second dict
* test(proxy): drive member key creation through the key generation helper
The pure helpers were covered but the wiring in _common_key_generation_helper was not. Dropping the owner role lookup, the refusal call, or the user_id handed to common_key_access_checks left every unit test green
* fix(proxy): skip the owner role lookup when create_member_keys is off
The owner role was read before the verdict checked the setting, so with the setting off a failing user table read surfaced as a 500 instead of the 403 a team admin got before, and every such request paid an extra writer read
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat(telemetry): count Admin UI page views and tab clicks through the proxy
* fix(ui): share one telemetry settings save across both tabs so a second toggle waits for the first
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): let every dashboard role reach the UI event routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): fail with the denial detail when a dashboard role cannot reach UI event routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): document POST /telemetry/ui_events with a description and a 200 response schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep the test-quality reason on the line pytest's checker reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): pick the telemetry destination on the Telemetry page and warn when none is chosen
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): send the destination along with groups in the telemetry settings hook test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add a top-level telemetry switch and gate turning telemetry on behind an enterprise modal
The page can only turn telemetry off. Turning it on from the switch, a group
or a first destination opens a modal: enterprise proxies are told to contact
their LiteLLM enterprise representative, others that telemetry is an
enterprise-only feature. Group switches and the destination stay visible but
disabled while telemetry is off.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): keep reports in a local table with a persistent instance id and an admin export route
* feat(telemetry): key deployment hashes from a random secret stored in the proxy DB
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): retry a locally stored window under the same id so it is never counted twice
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(telemetry): state the overlap and dedupe contract for paging stored reports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate dashboard API types for the telemetry reports paging note
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): require an explicit LITELLM_TELEMETRY_DESTINATION or saved destination before recording
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate dashboard API types for the explicit telemetry destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): register the attempt logger and create its hash secret only once a sink is built
A proxy with a database that never opts in now carries no telemetry callback
and writes no deployment hash secret to LiteLLM_Config.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): record proxy requests and provider attempts into the telemetry sink
* fix(telemetry): validate env settings, fix the window and only record the request's own provider calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): bind the call id from a typed local
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record a finished request into the sink current when it finishes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record every failed provider attempt, including retries and fallbacks on a shared logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): report timeouts and connection failures as their own provider status classes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record failed attempts from the per-deployment failure hook instead of a new core logging hook
Reverts the async_log_attempt_failure_event hook added to CustomLogger and
litellm_logging in 3421b38685. The router already calls
async_post_call_failure_deployment_hook once per failed attempt, retries and
fallbacks included, so every failed fallback still gets its own row without
touching the shared logging object.
* fix(telemetry): register the attempt logger only once a telemetry group is turned on
A proxy that never opts in now carries no telemetry callback. Once registered
the logger stays, returning early while every group is off.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors
Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy
modules with one pydantic validation at the point the payload enters the
module, so every value read afterwards carries a real type.
reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright,
ruff-strict or type-discipline rule is higher repo-wide or in any single
file, and no cast, ignore or suppression marker was added.
* chore(typing): replace 47 Any annotations that every caller already satisfies with object
Each annotation here is a parameter or return type whose value is only
stored, compared, logged or forwarded, so object describes it exactly and
no function body changes. A site was kept only if no basedpyright,
ruff-strict or type-discipline rule rose in any file that references it.
reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655.
* chore(typing): type upstream responses and loose annotations across providers, proxy and integrations
Second pass of the Any reduction. Decoded upstream JSON is validated once at
the provider boundary and read through typed views afterwards, annotations
that every caller already satisfies move from Any to a concrete type, and the
nine search transformations validate each result item on its own so error
text no longer carries the item index.
No cast, no type: ignore, no suppression marker and no budget file edit
* ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard
A JSON-RPC tools/call on /{server_name}/mcp (or /mcp with x-mcp-servers naming one server) that uses the upstream's bare tool name answered "Tool not found" on every proxy process that had not yet listed that server, because the per-process tool-name map stays empty for a server whose upstream only lists tools with the caller's own credentials, and mcp SDK 2 no longer lists before an uncached call. call_mcp_tool now hands the single scoped server to execute_mcp_tool as requested_server_id, so the existing REST contract lists the server with the caller's headers and routes the bare name to it, and a bare name shared by two servers goes to the one the path names
Co-authored-by: aayushbaluni <73417844+aayushbaluni@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast team cache invalidation on team writes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover cross-instance team cache invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): source Home What's new from day 0 support blog posts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): Home cards read card_title and card_summary from blog tags with provider logo
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files
Replace sleeps that raced background logging, cache writes, refreshes and
thread work with either awaiting that work directly (logging worker flush,
the batch logger's flush_queue, the tasks the call spawned, thread joins) or
an Event the test's own fake sets when it runs. Each wait keeps a 5s hang
guard that never decides pass or fail. Tests that need a fake clock, real
sockets, or measure latency are left unchanged, and no production code moves
* test(unit): await the tasks each completion spawns instead of yielding once
The logging tests yielded once with asyncio.sleep(0) so the task that
enqueues the success log could run before the worker flush. That relied
on the enqueue happening before the task's first await. Snapshot the
running tasks around each acompletion call and await the ones it spawned,
so the guarantee holds however many steps sit before the enqueue
* fix(proxy): return provider-matching 4xx for the LIT-8062 status code follow-ups
- OCR multipart errors are 400 invalid_request_error on the file param
- a missing model is a 400 on chat, responses and messages even with a wildcard deployment
- responses max_output_tokens <= 0 reaches the provider instead of being raised to 16
- daily activity rejects end_date before start_date with a 400
- vector store file attributes are sent in full so the provider enforces its key limit
- router-wide defaults no longer satisfy a param the Router takes positionally, or a model the router does not serve
- a JSON null user_config is treated as absent
* chore: keep the type gates at their merge-base counts
* fix(proxy): keep fixed-target wildcards serving a missing model and close router-default gaps
- a missing model is rejected only when the wildcard would forward it into the target
- moderation only counts router defaults for models get_model_list lists
- user_config defaults and deployment params never satisfy a positionally bound param
* test(integration): pin the fixed-target wildcard path and no provider call on positional-param 400s
* chore: validate pattern-matched deployments instead of reading untyped dicts
* fix(proxy): keep router defaults counting for a2a agent models
* test: drop the signature-walk test and type the new test helpers
* test(integration): audit matrix for the status-code follow-ups
Drives every mode the fix branches on through a real proxy, Postgres, Redis and
the scripted upstream: positional params against router, deployment and
user_config defaults; moderation listed vs unlisted; unknown and a2a models;
every user_config shape on chat, responses and messages, streaming and SDK
clients; forwarding vs fixed-target wildcards for missing, null and empty
models; Responses max_output_tokens edges; OCR parse errors; vector store file
attributes; every daily activity route; and a provider outage mid burst plus a
killed worker on a two-worker proxy. Product gaps found are skipped as BUG.
* style: ruff format the audit test files
* test: name the auth pre-check as the source of the JSON array 500
* test: build the audit helpers without mutation and share one sentinel
* test: build every audit request body as one literal
* fix(proxy): keep router defaults and model-less dispatch for requests a fallback chain serves
* fix(proxy): count request, key and team fallbacks when an unlisted model inherits router defaults
* fix(proxy): skip key and team fallbacks on the client-credential dispatch path
* fix(proxy): keep router defaults when the router's own * fallback serves an unlisted model
* fix(proxy): count client-style fallbacks, forward an empty model, report invalid usage chat dates
- The router-default pre-check now treats client-style request, key and team fallbacks
([{"model": ...}]) as serving an unlisted model, as the router does
- The forwarding-wildcard gate only rejects a missing or null model; an empty model
reaches the provider as before (vLLM serves its loaded model for it)
- The /usage/ai/chat tools hand the model the date-range error instead of an empty
dataset or a generic fetch error
* test(usage): type the invalid-date tool test
* fix(proxy): keep rejecting an empty model behind a forwarding wildcard
Key and team model checks skip an empty model, so forwarding it let a key
restricted to another model reach a server's default model (vLLM serves
its loaded model for ""). Pin that with a restricted-key integration cell.
* fix(usage): report the query layer's date 400 to the usage chat model
Checking dates before the fetch refused formats the global usage tool's query
accepts (2026/10/01 returned spend on main). Catch the daily activity query's
own 400 instead, so each tool rejects exactly what it rejected before and only
the message changes.
* fix(proxy): restore three model-less and unlisted-model paths main served
An empty model is looked up as sent, so next to a forwarding */* wildcard it
still reaches the fixed * deployment it matches. A null model with a prompt_id
is left to the prompt manager, which may pick the model. A Claude Code
subagent turn on a session-bound auto/complexity router counts as served, so
the router default max_tokens still applies.
* fix(proxy): only skip the model check for a prompt_id when a prompt manager is registered
Without a callback prompt manager nothing can pick the model, so a null model
with a prompt_id behind a forwarding wildcard gets the same 400 as any other
null model.
* fix(proxy): leave a model-less Claude Code subagent turn to its session router
A subagent turn on a session bound to an auto, complexity, adaptive or
quality router is answered by that router whatever model it names, so a
null or missing model behind a forwarding wildcard is no longer rejected.
An empty model stays rejected.
* fix(proxy): skip the wildcard model check only for a bound Claude Code session
Read the session's router binding, as the router does, instead of only
checking that the request could resume one, so a subagent turn without a
model on an unbound session is still rejected before a forwarding
wildcard sends it upstream.
* refactor(proxy): stop predicting routing in the proxy pre-checks
- drop the wildcard missing-model gate: a model-less request behind a forwarding wildcard goes to the router and gets the provider's own status, as on main
- drop the "model the router serves" check: router defaults count for any named model again, except moderation, where the Router merges them only for listed models
- keep the positional-param, moderation and null user_config fixes
* test(ocr): cover the form, size, empty body, invalid JSON and file-document 400s
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
test_cold_concurrent_schema_validation_accepts_valid_arguments pins the
process to a single CPU core and fans out 8 concurrent calls through
anyio.to_process's subprocess worker pool, each needing to cold-spawn a
fresh interpreter and import jsonschema. Under a slow or contended CI
runner that routinely exceeds the production 30-second budget before any
actual validation work runs, failing the test on a genuine timeout rather
than a validation bug.
_tool_argument_validation_error now takes an optional timeout_seconds,
defaulting to the existing constant, so production behavior is
unchanged. The test passes a timeout sized for its own single-core,
cold-start stress scenario, matching the margin the companion
integration test (test_schema_validation_stops_expensive_work_and_recovers)
already grants a similar concurrency=8 case.
* feat(guardrails): add Wingback guardrail integration
Mirrored from #38685 at d73b66544d (net diff against its merge base, applied on main)
* fix(guardrails): dispatch Wingback through the unified guardrail
The proxy only runs a unified guardrail when apply_guardrail is defined on
the guardrail's own class, so WingbackGuardrail now binds the parent's
decorated method in its class body. Chat, messages and responses pre, post
and during call checks reach Wingback again, and each check logs once.
Also types the initializer and config model for the lint gates and uses
monkeypatch.setenv in the unit test
* chore(guardrails): drop the Wingback dispatch comment
The unit and integration hook tests already fail when the apply_guardrail binding is removed, so the comment is redundant under the repo comment rule
---------
Co-authored-by: bsk-wb <bharath@wingback.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format otel excluded_services changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: use raw regex for excluded_services match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): reject non-string destination callback vars
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): harden destination callback var validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): compare parsed values in callback var conflict check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add user field to AI Guard execute-policy payload for dashboard attribution
Send the top-level OpenAI request `user` field in the execute-policy
JSON payload so AI Guard can surface usernames on its dashboard. This is
independent of the existing user_api_key_* headers which remain unchanged.
The upstream guardrail framework already passes the original request_data
(containing the `user` field) through to response-path guardrail calls,
so this works for both IN and OUT directions including streaming.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Add type annotations and tests for user field in execute-policy payload
- Add Final annotation to user_for_payload
- Add str | None type to user parameter in make_zscaler_ai_guard_api_call
- Add 3 tests: user included when present, omitted when absent,
and user in/out of JSON body sent to AI Guard
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(zscaler_ai_guard): inject the http client in user field tests
Pass an AsyncHTTPHandler into ZscalerAIGuard so the user field tests read the
real execute-policy body off a MockTransport instead of patching class methods,
and drop the comments that restated the lookup
* fix(zscaler_ai_guard): forward only non-empty string users
Read the request user through a typed helper and type the execute-policy call
params so basedpyright stays under the merge-base count, and skip non-string
values the vendor rejects with a 500
* test(zscaler_ai_guard): drop the test-only http_client
Run the user field tests against an in-process aiohttp server through api_base
so the guardrail keeps its original constructor
---------
Co-authored-by: Syamala Keerthana Chegu <schegu@zscaler.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
