Probe the column before the scheduler registers CheckBatchCost, closing the window where a retrieve that decided the poller was inactive billed a batch the first poll cycle then billed again. Also drop narration docstrings and section banners from the new tests.
The retrieve path now defers a managed batch's accounting to CheckBatchCost, which
bills the key, team, and tags stored on the managed object row. The /v1/batches
create hook never persisted api_key or request_tags there (only the passthrough
creates did), so the poller attributed the cost to the user alone and the creating
key's spend stayed at zero.
* fix(ptu): stop a PTU deployment billing for grounded search
A PTU deployment is billed by the flat cost of its reserved capacity, so the
model write endpoints refuse a rate the caller supplies and zero the ones already
stored. search_context_cost_per_query escaped both: it holds its rates in a table
keyed by context size, and the guard only recognised a number as a price, so a
grounded request on a PTU deployment kept billing per search on top of the flat
cost.
A table now counts as a price when it holds a non-zero rate. It is zeroed in
place rather than emptied the way tiered_pricing is, because an absent table
means the provider's own default rate rather than free, so dropping it would
start a charge instead of stopping one. For the same reason an all-zero table is
not read as a price: it is how an operator expresses free.
* fix(ptu): zero the search rate on every PTU deployment
A deployment that never stored its own search table is the normal case, and an
absent table means the provider's default rate, so the zeroing has to be written
unconditionally the way the per-token zeros already are. Writing it only where a
table was already stored left the default path billing per grounded search, which
is the charge this set out to stop.
The predicate that reads a table is split out rather than recursing, since the
repo's recursion gate rejects an unignored recursive function and one level is
all a rate table needs.
The TPM pre-call reservation read `max_tokens or max_completion_tokens`, so a
request declaring both was charged for whichever field came first. A caller
sending `max_tokens=1` with `max_completion_tokens=10000` reserved 2 tokens and
was then free to consume ten thousand, since the provider honours the modern
field and litellm's own param mapping drops the legacy one for the gpt-5 and
o-series families.
Reserve against the larger of the declared budgets instead. Over-reserving is
the safe direction for a limiter: post-call reconciliation refunds the
difference between the reservation and actual usage, while under-reserving lets
the window be exceeded before anything notices.
redis-py >= 7.4 decorates AbstractConnection.__init__ with @deprecated_args,
whose wrapper is declared (self, *args, **kwargs). _init_arg_names introspects
the wrapper directly, so from redis-py 7.4 the MRO walk loses every real
connection parameter and the from_url allowlist silently drops socket_timeout
and socket_connect_timeout again - the exact regression the allowlist rework
fixed, reintroduced one dependency version later. A url-configured Redis that
blackholes packets then blocks callers indefinitely instead of timing out.
Follow the __wrapped__ chain with inspect.unwrap before introspecting; a no-op
for undecorated __init__s.
Measured across redis-py lines (socket_timeout present in the allowlist):
6.4.0 before/after: yes/yes. 7.1.0: yes/yes. 7.4.1: NO/yes. 8.1.0: NO/yes.
tests/test_litellm/test_redis.py at redis-py 8.1.0: 10 failures before, 3
after (the residual trio is sentinel/cluster password handling, failing
identically without this change).
Two tests: a decorated-fake proving the unwrap mechanism, and a live-invariant
assertion that the installed redis-py's allowlist carries the socket timeouts -
the first thing to go red if a future redis-py changes signature declaration
again.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5
The Bedrock twins of both models already carry
supports_native_structured_output, but the Anthropic-direct entries do not,
so response_format requests to anthropic/claude-sonnet-5 and
anthropic/claude-haiku-4-5 fall back to the json_tool_call emulation and
inherit its nested-envelope failure modes (#8898) despite the API supporting
output_format natively.
Verified live against the Anthropic API on 2026-08-05: both models accept
output_format (structured outputs beta header) and return exact schema
instances, including a large nested production schema validated with
pydantic. Same two lines applied to the bundled backup map.
* fix(model_map): cover the versioned claude-haiku-4-5-20251001 alias
Exact-match capability lookup of anthropic/claude-haiku-4-5-20251001
resolved the versioned entry, which lacked the flag, so response_format
for that identifier still took the tool-emulation path. Flag it in both
the root and bundled maps, matching its unversioned alias.
* fix(anthropic): bound $defs inlining in output_format with the shared schema-bomb budget
map_response_format_to_anthropic_output_format called unpack_defs with
no max_inlined_bytes, so an authenticated caller could send a compact
schema whose repeated $refs expand without bound before reaching the
provider. Reuse the existing 10MB inlining budget (renamed from
_LEGACY_DEFS_MAX_INLINED_BYTES to DEFS_MAX_INLINED_BYTES now that two
call sites share it); overflow raises ValueError instead of
materialising the expansion.
Regression tests: a compact schema bomb is rejected, a normal $defs
schema still resolves; the bomb test fails when the bound is removed.
* chore: retrigger CI (benchmarks job flaked on a PyPI download timeout)
---------
Co-authored-by: Anmol Jaiswal <anmolg1997@users.noreply.github.com>
* fix(panw_prisma_airs): surface scan_id and scan metadata on allowed requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format panw guardrail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): expose scan id header only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(panw_prisma_airs): inject http client instead of patching private api
Adds an http_client seam so the scan-id tests drive the real AIRS request/parse path through a mock transport, plus direct coverage for the scan-id header helper.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): expose guardrail scan id header to browser clients
Keeps the panw optional_fields block untouched to avoid a needless conflict with a sibling PR that deletes it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Registering the five managed-batch fields in all_litellm_params stops them leaking into
extra_body, but two of them never reached the transformation that reads them.
CredentialLiteLLMParams is a whitelist, so get_deployment_credentials_with_provider
round-tripped the deployment and silently dropped s3_output_bucket_name and bedrock_tags
before the files/batch/passthrough callers saw them. s3_bucket_name, s3_region_name and
aws_batch_role_arn were added to that model for #25104; these two are the remainder of
the same deployment config
bedrock_tags is typed as a plain list rather than a stricter shape so a malformed value
still reaches _validate_bedrock_tags and gets its own error message instead of a Pydantic
one
The preservation assertion previously round-tripped through GenericLiteLLMParams, which is
extra="allow" and would hold even for a field nothing declares. It now also reproduces the
CredentialLiteLLMParams normalization the proxy actually performs, and fails naming
exactly the dropped fields without this change
A Bedrock managed-batch deployment carries aws_batch_role_arn, s3_bucket_name,
s3_region_name, s3_output_bucket_name and bedrock_tags in its litellm_params,
and the batch and files transformations read all five from there. None was
registered in all_litellm_params, so the param builder swept them into
extra_body on every other route that deployment serves: Bedrock answers
"aws_batch_role_arn: Extra inputs are not permitted" on Anthropic models and
"extraneous key [aws_batch_role_arn] is not permitted" on Nova, Llama and
Titan, so configuring batch turns every chat and embedding request to that
model into a 400.
Register them alongside the agentic-loop and callback-credential fields, which
are listed for exactly this reason. The batch path is unaffected because
GenericLiteLLMParams is extra="allow" and preserves them into litellm_params
for the transformations that consume them.
Before this, batch could only be configured on a deployment dedicated to
batch; the same model group could not serve both.
The Global Control Plane (formerly documented as the HA Control Plane) is
documented as an Enterprise feature, but `worker_registry` carried no premium
check, so any OSS install could run one. Gate it at config load, matching the
`enforced_params` precedent, and fail startup rather than ignoring the registry
silently: a silently dropped registry degrades a control plane into an ordinary
proxy with no signal to the operator.
Also declare `worker_registry` and `general_settings.control_plane_url`, both
load bearing today and neither previously declared, so they appear in the
generated config schema.
* fix(proxy): requeue spend logs when the DB write fails with a transport error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hardcode the spend log queue cap and drop the stale re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): keep the spend log requeue within the type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply the spend log queue cap to producer appends too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): lower the spend log queue cap to 1k and make it env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the spend log queue by bytes instead of row count
A row cap cannot bound memory: a row carries the whole prompt under store_prompts_in_spend_logs, so a cap that rides out an outage of counter-only rows is an OOM once prompts are stored. Every enqueue and dequeue now goes through one pair that tracks what the queue costs and drops the oldest rows past a 64 MB budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte budget env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use a string default for the spend log queue byte budget env read
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte total a public attribute
The queue it accounts for is already public, and a private name only bought reportPrivateUsage errors at every call site.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Dropping it falls back to the public cost map's tier table, whose rates outrank the
zeros written beside them, so a PTU deployment on a tiered model keeps billing its
traffic per token. Stored empty, the tiers no longer apply and the zeros win
tiered_pricing is a list, so the 0.0 the flat-rate zeroing stores does not
even validate. Supplying tiers alongside PTU config gets the same 400 as a
flat rate; tiers already stored are dropped from both blobs
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
