Three findings from review of the previous commit.
max_messages applied the same number to two lists that count different things:
texts is fragment-based (a multimodal turn contributes several entries, a
tool-call-only turn contributes none) while structured_messages is message-based,
so the two views of the payload could cover different parts of the conversation,
sending text from an omitted turn or dropping text from a retained one. The
message window is now the source of truth and the text window is the number of
fragments those retained messages contain, counted with a role-agnostic helper so
an unenumerated role cannot silently shrink it.
/guardrails/apply_guardrail built request_data straight from the caller's
metadata, so a caller could claim a user_api_key_alias or user_api_key_team_id
that a guardrail exempts and get its text back unscanned. The authenticated
identity is now layered on top of the caller's metadata via the same sanitizer
the in-line path uses, so what authentication established wins. Caller metadata
that is not identity still passes through untouched, and every guardrail on that
route now sees the caller information it already sees in-line.
The per_session dedup cache was keyed only on the caller-supplied session id, so
whoever claimed an id first suppressed another tenant's entry for the same id.
The key is now namespaced by the authenticated key hash, falling back to the team
id.
Guardrail providers that are observe-only, that ignore parts of the payload, or
that only care about some traffic currently have no way to say so: LiteLLM sends
the whole payload for every call type, blocks the request until the guardrail
answers, and appends a logging entry on every single call. On long agent sessions
that means the entire transcript is re-sent every turn, synchronously, on the
user's critical path, and one spend-log row accumulates thousands of guardrail
entries.
Seven additive options, all defaulting to today's behavior:
- fire_and_forget dispatches the call as a background task and lets the request
proceed, which makes the guardrail observe-only (BLOCKED and
GUARDRAIL_INTERVENED can no longer be honored, and it says so at init).
fire_and_forget_max_inflight bounds the queue so a slow endpoint cannot pile up
tasks; in-flight tasks are strong-referenced so the event loop cannot collect
them mid-call. Streaming is forced to end-of-stream observation, so a stream
dispatches one call instead of one per sampled chunk.
- send_images, exclude_payload_fields, max_messages and max_text_chars declare
what the endpoint receives. send_images=false also replaces inline image data
in structured_messages with a marker, so the base64 payload really does stay in
LiteLLM. max_messages windows the flat texts list as well as structured_messages,
since the translation handlers build texts from the whole conversation and
windowing one without the other would leave payload size proportional to session
length. input_type and litellm_call_id cannot be excluded.
- strip_patterns removes configured regex matches from text before the payload is
sent, touching only string text fields, never JSON structure, tool schemas, ids
or metadata.
- skip_if_system_prompt_matches and skip_if_first_role_in skip a call entirely,
anchored to the system/developer message so pasted user content cannot trigger
them. Both read the request body, which the caller controls, so a caller that
knows the configured value can exempt itself; that is stated in the field
descriptions and warned about at init. skip_if_key_alias_in and
skip_if_team_id_in are the alternative that holds against the caller, matching
on what authentication established. The message-based decision is replayed on
the paired response through the shared logging object, with a bounded call-id
cache for paths that have none; the identity filters need no correlation, since
both hooks see the same metadata.
- run_only_on_call_types and skip_call_types skip whole call types (embeddings,
image, audio, rerank, moderation) before anything is sent. An unresolvable call
type still runs, so a misconfiguration cannot silently blind the guardrail.
- guardrail_information_scope picks how often a StandardLoggingGuardrailInformation
entry is recorded: per_call keeps today's behavior, per_session records only the
first call of a session, off records nothing on success. Session dedup needs a
resolvable session id and falls back to per_call without one, and the session set
is bounded by size and TTL. Suppression rides the log_guardrail_information
contract via a new named entry point, suppress_guardrail_information_record, and
is applied only after a normal return, so blocks and guardrail failures still
record under every scope.
Payload shaping is lossy, so each shaped payload carries the loss with it and the
guardrail refuses to write back text the endpoint never saw in full: a stripped,
truncated or windowed block keeps the caller's original content, and images or
tools that were not sent cannot be replaced. Shaping runs on the dumped JSON
because pydantic validates structured_messages content lazily into a
ValidatorIterator that is unsafe to walk or re-iterate.
The message-mapping helpers live in the shared guardrails content utils, next to
the existing walkers, so the Chat Completions and Responses content shapes are
handled in one place.
* fix(ptu): stop a PTU deployment billing for grounded search
A PTU deployment is billed by the flat cost of its reserved capacity, so the
model write endpoints refuse a rate the caller supplies and zero the ones already
stored. search_context_cost_per_query escaped both: it holds its rates in a table
keyed by context size, and the guard only recognised a number as a price, so a
grounded request on a PTU deployment kept billing per search on top of the flat
cost.
A table now counts as a price when it holds a non-zero rate. It is zeroed in
place rather than emptied the way tiered_pricing is, because an absent table
means the provider's own default rate rather than free, so dropping it would
start a charge instead of stopping one. For the same reason an all-zero table is
not read as a price: it is how an operator expresses free.
* fix(ptu): zero the search rate on every PTU deployment
A deployment that never stored its own search table is the normal case, and an
absent table means the provider's default rate, so the zeroing has to be written
unconditionally the way the per-token zeros already are. Writing it only where a
table was already stored left the default path billing per grounded search, which
is the charge this set out to stop.
The predicate that reads a table is split out rather than recursing, since the
repo's recursion gate rejects an unignored recursive function and one level is
all a rate table needs.
The TPM pre-call reservation read `max_tokens or max_completion_tokens`, so a
request declaring both was charged for whichever field came first. A caller
sending `max_tokens=1` with `max_completion_tokens=10000` reserved 2 tokens and
was then free to consume ten thousand, since the provider honours the modern
field and litellm's own param mapping drops the legacy one for the gpt-5 and
o-series families.
Reserve against the larger of the declared budgets instead. Over-reserving is
the safe direction for a limiter: post-call reconciliation refunds the
difference between the reservation and actual usage, while under-reserving lets
the window be exceeded before anything notices.
redis-py >= 7.4 decorates AbstractConnection.__init__ with @deprecated_args,
whose wrapper is declared (self, *args, **kwargs). _init_arg_names introspects
the wrapper directly, so from redis-py 7.4 the MRO walk loses every real
connection parameter and the from_url allowlist silently drops socket_timeout
and socket_connect_timeout again - the exact regression the allowlist rework
fixed, reintroduced one dependency version later. A url-configured Redis that
blackholes packets then blocks callers indefinitely instead of timing out.
Follow the __wrapped__ chain with inspect.unwrap before introspecting; a no-op
for undecorated __init__s.
Measured across redis-py lines (socket_timeout present in the allowlist):
6.4.0 before/after: yes/yes. 7.1.0: yes/yes. 7.4.1: NO/yes. 8.1.0: NO/yes.
tests/test_litellm/test_redis.py at redis-py 8.1.0: 10 failures before, 3
after (the residual trio is sentinel/cluster password handling, failing
identically without this change).
Two tests: a decorated-fake proving the unwrap mechanism, and a live-invariant
assertion that the installed redis-py's allowlist carries the socket timeouts -
the first thing to go red if a future redis-py changes signature declaration
again.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5
The Bedrock twins of both models already carry
supports_native_structured_output, but the Anthropic-direct entries do not,
so response_format requests to anthropic/claude-sonnet-5 and
anthropic/claude-haiku-4-5 fall back to the json_tool_call emulation and
inherit its nested-envelope failure modes (#8898) despite the API supporting
output_format natively.
Verified live against the Anthropic API on 2026-08-05: both models accept
output_format (structured outputs beta header) and return exact schema
instances, including a large nested production schema validated with
pydantic. Same two lines applied to the bundled backup map.
* fix(model_map): cover the versioned claude-haiku-4-5-20251001 alias
Exact-match capability lookup of anthropic/claude-haiku-4-5-20251001
resolved the versioned entry, which lacked the flag, so response_format
for that identifier still took the tool-emulation path. Flag it in both
the root and bundled maps, matching its unversioned alias.
* fix(anthropic): bound $defs inlining in output_format with the shared schema-bomb budget
map_response_format_to_anthropic_output_format called unpack_defs with
no max_inlined_bytes, so an authenticated caller could send a compact
schema whose repeated $refs expand without bound before reaching the
provider. Reuse the existing 10MB inlining budget (renamed from
_LEGACY_DEFS_MAX_INLINED_BYTES to DEFS_MAX_INLINED_BYTES now that two
call sites share it); overflow raises ValueError instead of
materialising the expansion.
Regression tests: a compact schema bomb is rejected, a normal $defs
schema still resolves; the bomb test fails when the bound is removed.
* chore: retrigger CI (benchmarks job flaked on a PyPI download timeout)
---------
Co-authored-by: Anmol Jaiswal <anmolg1997@users.noreply.github.com>
* fix(panw_prisma_airs): surface scan_id and scan metadata on allowed requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format panw guardrail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): expose scan id header only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(panw_prisma_airs): inject http client instead of patching private api
Adds an http_client seam so the scan-id tests drive the real AIRS request/parse path through a mock transport, plus direct coverage for the scan-id header helper.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): expose guardrail scan id header to browser clients
Keeps the panw optional_fields block untouched to avoid a needless conflict with a sibling PR that deletes it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Global Control Plane (formerly documented as the HA Control Plane) is
documented as an Enterprise feature, but `worker_registry` carried no premium
check, so any OSS install could run one. Gate it at config load, matching the
`enforced_params` precedent, and fail startup rather than ignoring the registry
silently: a silently dropped registry degrades a control plane into an ordinary
proxy with no signal to the operator.
Also declare `worker_registry` and `general_settings.control_plane_url`, both
load bearing today and neither previously declared, so they appear in the
generated config schema.
Google's Gemini deprecations page lists no shutdown date for the 2.5 GA models and May 14, 2028 for gemini-embedding-001; keep only the DeepSeek V4 max output correction.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Revert gemini-embedding-001 to its published 2028-05-14 shutdown, move
gpt-4-turbo-preview to the 2026-03-26 shutdown of the gpt-4-0125-preview
snapshot it aliases, and drop the unannounced Gemini 2.5 shutdown dates
* fix(proxy): requeue spend logs when the DB write fails with a transport error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hardcode the spend log queue cap and drop the stale re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): keep the spend log requeue within the type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply the spend log queue cap to producer appends too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): lower the spend log queue cap to 1k and make it env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the spend log queue by bytes instead of row count
A row cap cannot bound memory: a row carries the whole prompt under store_prompts_in_spend_logs, so a cap that rides out an outage of counter-only rows is an OOM once prompts are stored. Every enqueue and dequeue now goes through one pair that tracks what the queue costs and drops the oldest rows past a 64 MB budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte budget env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use a string default for the spend log queue byte budget env read
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte total a public attribute
The queue it accounts for is already public, and a private name only bought reportPrivateUsage errors at every call site.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Dropping it falls back to the public cost map's tier table, whose rates outrank the
zeros written beside them, so a PTU deployment on a tiered model keeps billing its
traffic per token. Stored empty, the tiers no longer apply and the zeros win
tiered_pricing is a list, so the 0.0 the flat-rate zeroing stores does not
even validate. Supplying tiers alongside PTU config gets the same 400 as a
flat rate; tiers already stored are dropped from both blobs
parse_prompt_tokens_details and parse_completion_tokens_details are imported
by four modules, so the leading underscore made every import a
reportPrivateUsage violation
The previous assertion read data-chips, which is derived from the anchor prop
being truthy, so it stayed true even when the ref never reached the DOM and the
popup was still anchored to the inner input.
Stub distinct widths on the chips container and the input, then read the width
the positioner resolved. Reverting the anchor wiring now reports the input's
width instead of the field's, which is the actual bug.
