* fix(proxy): cache tag-name registry so unregistered request tags skip Postgres
Request tags are free-form attribution labels, so most have no LiteLLM_TagTable
row. get_tag_objects_batch never cached that absence: every tagged request ran
a find_many that came back empty, and under Prisma pool contention those
per-request queries queued for minutes inside user_api_key_auth.
Cache the bounded set of registered tag names under one aggregate key with the
management-object TTL. Uncached request tags are filtered against it before any
per-tag DB fetch, so unregistered tags cost zero DB reads on a warm path. An
empty registry is cached as a valid answer; DB errors are not cached and fall
back to the per-tag lookup; tables past TAG_REGISTRY_MAX_SIZE cache an overflow
sentinel that disables filtering. Tag create/update/delete endpoints now evict
the registry and per-tag keys and publish cross-worker invalidation (they
previously evicted nothing). The per-tag write-back also gains the management
TTL it was missing, and the hand-built tag:{name} key strings are replaced with
a shared builder.
* fix(proxy): skip per-request end-user DB reads via restricted-id registry
Every request carrying a user id ran get_end_user_object, and with high-cardinality
auto-created end-user rows (hundreds of thousands of ids, all restriction fields
NULL) the per-pod cache missed on nearly every request, so each one paid a Postgres
find_unique that queued behind the Prisma pool during background-job bursts. True
misses were never cached, and unknown ids paid the read twice per request.
Cache the bounded set of end-user ids that carry any restriction (blocked, budget,
region, default model, or object permission) under one aggregate key with the
management-object TTL. When an id misses the per-id cache and is absent from a
usable registry, get_end_user_object returns None with zero DB reads; restricted
ids keep today's fetch-and-cache path. The skip is bypassed whenever
litellm.max_end_user_budget_id is set (default budgets make unrestricted rows
behaviorally distinct from missing rows), validate_end_user_id_in_db is on
(existence checks need the row), or the token carries end_user_max_budget from
custom auth (the row's recorded spend seeds the budget counter). Empty registries
cache as a valid answer, DB errors are never cached, and oversized tables cache an
overflow sentinel that disables filtering. Customer create/update/block/delete now
evict the registry and per-id keys and publish cross-worker invalidation (they
previously evicted nothing), and the per-id write-back gains the management TTL it
was missing so Redis entries no longer live forever.
* refactor(proxy): single generic registry loader with error sentinel and single-flight
Code review follow-ups on the two registry caches. Registry DB errors now cache
the overflow sentinel for a short REGISTRY_ERROR_NEGATIVE_CACHE_TTL window and
log at warning, so a degraded Postgres stops paying the failing registry scan on
every request on top of the per-id fallback. Cold registry loads are single-flight
per worker behind per-registry locks with a recheck after acquire, so a TTL expiry
no longer fans out one full-table scan per in-flight request. The tag and end-user
loaders collapse into one _load_bounded_registry with per-entity fetch closures,
and the triplicated evict-then-broadcast protocol becomes one evict_and_broadcast
helper beside publish_auth_cache_invalidation, shared by the tag, customer, and
project eviction paths.
* chore(lint): suppress fail-safe registry excepts and ratchet BLE001 budget
* docs(proxy): trim registry cache commentary to single-line why docstrings
* fix(lint): move tag fetch return to else block to satisfy TRY300 budget
* fix(guardrails): scan text on /guardrails/apply_guardrail for Azure Content Safety
The two Azure Content Safety guardrails never implemented apply_guardrail, so the
endpoint fell through to the base no-op and answered 200 with the caller's text
echoed back, having scanned nothing.
Implementing that method also flips the proxy's unified-vs-native dispatch, which
would move request traffic off these guardrails' own hooks. Add an opt-out that
keeps every lifecycle event on the native hooks, so only the endpoint changes.
* test(guardrails): cover the remaining native-hook opt-out dispatch sites
Adds regression tests for the parallel post-call path, the MCP post-call hook, and
the policy engine step, so every read of the opt-out flag fails when removed.
- decode upstream first frame as utf-8 instead of ascii
- reject model-restricted keys at connect to match HTTP model enforcement
- log the actual request path for /openai_passthrough traffic
* fix(panw_prisma_airs): scan tool call args as plain text, not a tool_event
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): type the tool call argument extractor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(panw_prisma_airs): cover tool call error fallback and dict masking paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): scan tool names with args and tolerate custom tool calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): scan tool call arguments that arrive already parsed
The tool call slice types arguments as a string, so a client posting parsed JSON
failed validation and the whole tool call, name included, read as unscannable and
was skipped without ever reaching AIRS. The OpenAI request path forwards
client-supplied tool_calls verbatim, so that shape is reachable.
Coerce non-string arguments instead of rejecting them, so the content is scanned.
* fix(panw_prisma_airs): route tool-block masked data by scan side, not by key name
Merging #37036 (already on staging) with this PR produces no conflict and a
silent bug. #37036 withholds prompt_masked_data on response-side tool blocks,
which was right while tool calls went out as a request-side tool_event: AIRS
reported the model's arguments under that key. This PR scans tool calls as
ordinary prompt/response text, so the side of the scan now decides which key
holds what. The model's arguments arrive under response_masked_data, already
covered by _CLIENT_HIDDEN_SCAN_FIELDS, and prompt_masked_data goes back to
being the caller's own input -- one of the audit fields LIT-5638 asks for.
Left as merged, a response-side tool block drops that field with nothing to
flag it.
- Tool-path block branch calls _build_error_detail without also_hide
- also_hide parameter removed; after this change it has no callers
- Regression test asserts both directions: model output withheld, caller
input preserved. It fails against the auto-merged combination.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(panw_prisma_airs): a wrong-typed tool name must not suppress the scan
_ToolCallFunctionSlice types name as str, and _get_tool_call_function turns any
ValidationError into (None, None), which _scan_tool_calls_for_guardrail reads as
an unscannable tool call and skips. So a client posting "name": 123 keeps its
arguments off the wire to AIRS entirely -- no error, no log, no block. The
OpenAI request path forwards client tool_calls verbatim, so this is reachable by
any caller holding a valid key.
_coerce_arguments already existed for exactly this failure mode on the sibling
field. Widening it to cover name closes the gap:
name='transfer_funds' AIRS called: 1x args scanned: True
name=123 (int) AIRS called: 0x args scanned: False <- before
name=123 (int) AIRS called: 1x args scanned: True <- after
Reported by Cursor Bugbot on fd9f6396e5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(guardrails): return the full PANW AIRS scan response on blocked requests
The blocked-request error detail was assembled from a hardcoded allowlist, so audit fields like prompt_detection_details, prompt_masked_data, source, transaction_id and session_id never reached the client even though AIRS returned them.
Resolves LIT-5638
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(guardrails): drop redundant comment in AIRS error detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): withhold response_masked_data from the blocked-response error
The full AIRS passthrough also reached the response-side block path, where
response_masked_data carries the model's own generation. That branch is only
reached when mask_response_content is False, so the operator had explicitly
declined to deliver that text, and the error body handed it back anyway.
Withhold response_masked_data from the client-visible detail. prompt_masked_data
stays: it is the caller's own input and one of the fields the ticket asks for.
Every other AIRS field, including prompt_detection_details, source,
transaction_id and session_id, is unchanged.
* fix(panw_prisma_airs): withhold generated tool args from response-side blocks
_scan_tool_calls_for_guardrail calls AIRS with is_response=False because
tool_event is request-side in the AIRS schema, so AIRS returns the scanned
tool arguments under prompt_masked_data. When the tool calls being scanned
are the model's own output, that key holds generated content, and the
_CLIENT_HIDDEN_SCAN_FIELDS default (response_masked_data, empty on this
path) does not cover it. With the default mask_response_content=False the
block branch then shipped the model's masked tool arguments in the 400 --
the same content channel this PR closed for response_masked_data.
_build_error_detail takes an extra_hidden_fields argument so the withholding
stays in one place, and the tool-call block branch passes prompt_masked_data
when is_response is True. Request-side blocks are unchanged and still carry
prompt_masked_data, which is what LIT-5638 asks for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* style(panw_prisma_airs): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Probe the column before the scheduler registers CheckBatchCost, closing the window where a retrieve that decided the poller was inactive billed a batch the first poll cycle then billed again. Also drop narration docstrings and section banners from the new tests.
* fix(ptu): stop a PTU deployment billing for grounded search
A PTU deployment is billed by the flat cost of its reserved capacity, so the
model write endpoints refuse a rate the caller supplies and zero the ones already
stored. search_context_cost_per_query escaped both: it holds its rates in a table
keyed by context size, and the guard only recognised a number as a price, so a
grounded request on a PTU deployment kept billing per search on top of the flat
cost.
A table now counts as a price when it holds a non-zero rate. It is zeroed in
place rather than emptied the way tiered_pricing is, because an absent table
means the provider's own default rate rather than free, so dropping it would
start a charge instead of stopping one. For the same reason an all-zero table is
not read as a price: it is how an operator expresses free.
* fix(ptu): zero the search rate on every PTU deployment
A deployment that never stored its own search table is the normal case, and an
absent table means the provider's default rate, so the zeroing has to be written
unconditionally the way the per-token zeros already are. Writing it only where a
table was already stored left the default path billing per grounded search, which
is the charge this set out to stop.
The predicate that reads a table is split out rather than recursing, since the
repo's recursion gate rejects an unignored recursive function and one level is
all a rate table needs.
The TPM pre-call reservation read `max_tokens or max_completion_tokens`, so a
request declaring both was charged for whichever field came first. A caller
sending `max_tokens=1` with `max_completion_tokens=10000` reserved 2 tokens and
was then free to consume ten thousand, since the provider honours the modern
field and litellm's own param mapping drops the legacy one for the gpt-5 and
o-series families.
Reserve against the larger of the declared budgets instead. Over-reserving is
the safe direction for a limiter: post-call reconciliation refunds the
difference between the reservation and actual usage, while under-reserving lets
the window be exceeded before anything notices.
* fix(panw_prisma_airs): surface scan_id and scan metadata on allowed requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format panw guardrail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): expose scan id header only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(panw_prisma_airs): inject http client instead of patching private api
Adds an http_client seam so the scan-id tests drive the real AIRS request/parse path through a mock transport, plus direct coverage for the scan-id header helper.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): expose guardrail scan id header to browser clients
Keeps the panw optional_fields block untouched to avoid a needless conflict with a sibling PR that deletes it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Global Control Plane (formerly documented as the HA Control Plane) is
documented as an Enterprise feature, but `worker_registry` carried no premium
check, so any OSS install could run one. Gate it at config load, matching the
`enforced_params` precedent, and fail startup rather than ignoring the registry
silently: a silently dropped registry degrades a control plane into an ordinary
proxy with no signal to the operator.
Also declare `worker_registry` and `general_settings.control_plane_url`, both
load bearing today and neither previously declared, so they appear in the
generated config schema.
* fix(proxy): requeue spend logs when the DB write fails with a transport error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hardcode the spend log queue cap and drop the stale re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): keep the spend log requeue within the type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply the spend log queue cap to producer appends too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): lower the spend log queue cap to 1k and make it env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the spend log queue by bytes instead of row count
A row cap cannot bound memory: a row carries the whole prompt under store_prompts_in_spend_logs, so a cap that rides out an outage of counter-only rows is an OOM once prompts are stored. Every enqueue and dequeue now goes through one pair that tracks what the queue costs and drops the oldest rows past a 64 MB budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte budget env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use a string default for the spend log queue byte budget env read
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte total a public attribute
The queue it accounts for is already public, and a private name only bought reportPrivateUsage errors at every call site.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Dropping it falls back to the public cost map's tier table, whose rates outrank the
zeros written beside them, so a PTU deployment on a tiered model keeps billing its
traffic per token. Stored empty, the tiers no longer apply and the zeros win
tiered_pricing is a list, so the 0.0 the flat-rate zeroing stores does not
even validate. Supplying tiers alongside PTU config gets the same 400 as a
flat rate; tiers already stored are dropped from both blobs
* fix(mcp): drop caller host and configured upstream headers from logged metadata
The synthetic request that carries MCP client headers into
add_litellm_data_to_request forwarded the caller's Host header, and
Request.url is built from it, so a caller chose the proxy_server_request
url and the metadata endpoint that every logging callback records.
_upstream_credential_headers also only knew the configured client side
auth header and the x-mcp- prefix family, so a header name declared in
mcp_servers.<name>.extra_headers reached logging metadata in cleartext.
Those names are admin chosen, so no prefix rule can recognize them; read
them off the server registry instead. The header is still forwarded
upstream, which is what extra_headers is for. authorization is left out
because clean_headers already strips it and claiming it here would move
authenticated_with_header on the oauth passthrough config.
The Responses bridge tests stub the server manager, so their fakes gain
the registry accessor the sanitizer now reads.
* fix(mcp): drop caller host from the sanitized header mapping too
The synthetic request stopped forwarding host, but the parallel sanitizer
did not, so a forged hostname still reached the guardrail payload and the
list_tools spend row. Drop it there as well.
Exempt the configured identity headers from the upstream credential set.
get_user_from_headers resolves end user attribution off the same request
this module reconstructs, and it only fills end_user_id when auth left it
unset, so claiming user_header_name or a user_header_mappings name would
lose attribution on the MCP paths that authenticate upstream.
Drop the isinstance guard on extra_headers entries: the field is typed
list[str], so the check is dead and basedpyright scores it.
* fix(mcp): accept a bare user_header_mappings entry when exempting identity headers
get_internal_user_header_from_mapping and get_customer_user_header_from_mapping
both normalize a single mapping to a one element list, and config_settings.md
documents the key as a dict. Iterating the bare form yields its keys instead,
so the exemption silently matched nothing and an identity header also named in
an MCP server's extra_headers was dropped after all.
