The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
* fix(mcp): scan and mask MCP tool call arguments in unified guardrails
A guardrail configured with mode pre_mcp_call was handed only a synthetic
tool definition (name plus an empty parameters schema), so it never saw the
argument values it was configured to inspect, and any rewrite it returned was
discarded. Detection could not fire and masking could not take effect, while
the applied-guardrails metadata still reported the guardrail as having run.
Pass every string leaf of the tool call arguments as texts, and fold the
guardrail's rewritten leaves back into modified_arguments, which is the channel
the MCP call path reads to decide what to send upstream. The leaf walk reuses
the json_string_leaves / with_json_string_leaves helpers the tool result path
already uses, so both directions share one bounded traversal.
Two guardrails running concurrently under run_in_parallel scan the same payload
snapshot, so each returns a full replacement derived from the original leaf.
Rewrites of the same leaf to different values are rejected rather than silently
losing one redaction; a leaf that already holds this guardrail's own replacement
is convergent and still masks, which is what the bundled content filter does
when it rewrites the arguments itself as well as through texts.
* fix(mcp): annotate guardrail argument rewrites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): isolate MCP guardrail callback state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: ratchet LIT010 budget after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): remove duplicate Bedrock hook parameter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fail closed when guardrail rewrites cannot be mapped to MCP arguments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): patch the guardrail translation mappings cache where staging now keeps it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format prometheus_metrics_server
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): fail fast when the separate metrics server cannot start and force the multiproc dir whenever it is enabled
- wait for the child's /health before starting uvicorn; raise a ClickException if it exits first (port in use)
- create PROMETHEUS_MULTIPROC_DIR whenever --prometheus_metrics_port is set, so DB-configured prometheus callbacks work
- honour lowercase prometheus_multiproc_dir; validate the port before spawning
- cover main() entry point, readiness, bind failure and wildcard-host probing in tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pin metrics-server readiness to the child pid so another service on the port cannot pass the health check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): probe metrics-server readiness through the shared HTTPHandler instead of bare httpx.get
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): serve only /metrics on the prometheus metrics port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): validate metrics server CLI args with pydantic instead of typing.cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): satisfy metrics server lint gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Custom code guardrails could only allow(), block(reason) or modify(). This adds flag(reason, metadata={}) which lets the request or response through unchanged and records a guardrail_flagged entry carrying the guardrail name, configured mode, evaluated input_type (request or response), reason and structured metadata. The new status is threaded through the request-level guardrail_status aggregation, the Guardrails Monitor rollup (flagged_count), Request Logs (action=flagged, most severe phase wins when a guardrail runs pre and post call) and the Request Logs detail view in the dashboard, which now renders FLAGGED with warning styling instead of falling into FAILED.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture
Only the generic OIDC login path captures the IdP id_token that an oauth2_id_jag MCP
server spends as its RFC 8693 subject token. Under Google, Microsoft, SAML or no SSO at
all, registration succeeds and then every ID-JAG credential resolution fails for every
user, with nothing in the logs, the config or the API response to say why.
Report the mismatch from the two places it is knowable: when an oauth2_id_jag server is
created or updated through the management endpoint, and at SSO callback time when a login
hands the arm nothing while such a server is registered. Provider selection mirrors the
callback's precedence, so a generic client id sitting behind GOOGLE_CLIENT_ID does not
clear the warning.
* test(sso): update merged CLI diagnostic patch target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): warn about the ID-JAG capture gap for config-declared servers and on the SSO debug page (#39350)
* feat(sso): surface the ID-JAG capture gap on the SSO debug page
/sso/debug/callback is where an operator lands when they are already trying to work out
why ID-JAG is failing, so the reason belongs on it. The annotation appears only when the
active SSO provider captures no identity assertion AND an oauth2_id_jag server is
registered for that gap to break; a deployment without both renders the page it rendered
before, byte for byte. Only the provider name and the remedy are rendered, never a
configured value, and an unreachable MCP table costs the page its annotation rather than
the page itself.
The payload carries the one mutable-ok in this work. Conditionally including a member of a
JSON document has to construct a mapping, and the rejected alternatives are recorded on the
helper so the next reader does not rediscover them.
Held out of the diagnosability PR deliberately: that PR is already reviewed and green, and
this surface ships with the remaining config-load warning as one follow-up.
* feat(mcp): warn at config load when an oauth2_id_jag server outruns the SSO provider's assertion capture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(sso): trim comments on the ID-JAG debug page diagnostic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): clean up merged imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): satisfy type discipline for diagnostic payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(sso): keep the optional ID-JAG payload member on one line for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): use Python 3.10-compatible assert_never
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): keep the ID-JAG capture-gap diagnostic out of the unauthenticated debug page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sso): inject the retention check and log via caplog so the ID-JAG tests pass the test-quality gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sso): keep the debug-page outage test on the capture-gap path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): annotate the retention check type alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): renew the stored SSO identity assertion behind ID-JAG
The oauth2_id_jag arm asserts the id_token captured at the user's last
interactive SSO login, and nothing ever renewed it, so an agent holding a
brokered LiteLLM key could act for that user only until that token's exp.
The assertion already carried the IdP refresh token beside it; this
redeems it.
RefreshingSSOAssertionStore wraps the database reader and satisfies the
same protocol, so the egress arm is unchanged. Renewal is lazy and
single-flighted per user through the same RefreshCoordinator the
authorization_code arm uses, since an IdP that rotates refresh tokens
treats two concurrent redemptions as replay. A refusal leaves the expired
assertion in place so the reader still challenges the user; an
unreachable IdP surfaces as a store outage instead.
* fix(mcp): let a cross-replica loser settle the SSO assertion renewal itself
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): satisfy type discipline lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ci): rerun checks after docs main added the missing router setting row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): answer a cross-replica loser retryable instead of re-electing it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): bypass stale assertion cache during renewal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: ratchet type-discipline budget after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
* fix(cloudzero): preserve late resource tags
* style(cloudzero): remove redundant test comment
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
* fix(router): coordinate async and sync failure handlers at remaining router call sites
Five router failure paths still scheduled logging_obj.async_failure_handler
as a task while starting logging_obj.failure_handler on a raw thread, so
both handlers mutated the same logging object concurrently. Route them
through dispatch_failure_handlers like the streaming paths already do.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): wait on the real logging executor and justify the callbacks global patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): submit sync failure handler even when the dispatch task is cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): justify the executor submit patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the invalid-model 403 path cheap under a burst of rejections
Keep the wildcard pattern registry in specificity order at registration time
so route() no longer re-sorts every pattern per lookup, and reuse the
standardized failure payload across the async and threaded sync failure
handlers regardless of what a callback did to log_event_type. Rejections
are still logged and observable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(router): wrap the filtered pattern tuple the way ruff format wants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router,logging): assert registry order and callback awaits instead of patching a class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): inject the pattern sorter so the lookup test observes that route() never sorts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logging): scan large base64 payloads for log truncation off the event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logging): make base64 offload threshold a plain constant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(hide-secrets): stop redacting benign identifiers and make redaction deterministic
The OpenAI key detector matched `sk-` anywhere inside a word, so `<task-notification>`
became `<ta[REDACTED]>`, and the Base64 entropy limit of 3.0 flagged ordinary quoted
identifiers such as `"application/json"` and model ids. Redaction also iterated a
hash-seeded set, so the same request produced different bytes on different workers and
broke prompt caching.
- require a standalone `sk-`/`sk_` token with a digit (still catches sk-proj-/sk-ant-)
- raise Base64HighEntropyString limit from 3.0 to the detect-secrets default 4.5
- redact overlapping matches longest-first in a stable order
Resolves LIT-7049
* fix(hide-secrets): treat separators as key boundaries and defer sk_live_ to the stripe detector
The standalone-token boundary also rejected keys glued to a preceding `_`, `-`
or percent-encoded delimiter (`openai_sk-…`, `key-sk-…`, `Bearer%20sk-…`), which
the old pattern redacted, and `sk_live_…` was counted by both the OpenAI and the
Stripe detector.
* fix(hide-secrets): keep the openai key scan linear on repeated sk separators
The digit requirement was a lookahead, so every `sk` inside a long
`[a-zA-Z0-9_-]` run re-scanned the rest of that run looking for a digit.
100 KB of `-sk-` took over 5s in the worker's event loop and the proxy
closed the connection without a response. The check now runs once per
match in `analyze_string` instead.
* chore(hide-secrets): remove redundant performance test comment
* fix(hide-secrets): consume complete openai key tokens
* chore(hide-secrets): remove redundant fixture comment
* chore(hide-secrets): remove redundant test docstrings
* fix(hide-secrets): redact whole stripe live keys
* style(hide-secrets): wrap secret sorting key
* fix(proxy): retry deadlocks and requeue spend logs on any DB write error
update_spend_logs dequeued the batch and only retried/requeued on transport
errors. A 40P01 deadlock surfaced as a plain prisma DataError and went through
poison-row isolation, which dropped every row it hit; every other DB error was
re-raised with the batch already gone from the queue.
Treat deadlocks as transient (retry, then requeue), keep them out of poison-row
isolation, and requeue the batch at the head of the queue on any other prisma
error so it lands once the DB is healthy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop redundant docstrings and tighten test typing for spend-log requeue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert deadlock retries from mock call history instead of mutable lists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): hold max_parallel_requests slot until streaming response is exhausted or closed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): normalize deployment_slot once to keep stream_with_fallbacks under the C901 ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): close upstream stream before releasing max_parallel_requests slot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(responses): honor supported_endpoints /v1/responses opt-in for OpenAI-compatible deployments
custom_openai and other generic OpenAI-compatible deployments have no native
Responses API config, so every /v1/responses call is bridged through
/v1/chat/completions. When model_info.supported_endpoints lists /v1/responses,
resolve OpenAILikeResponsesConfig instead so the request is forwarded to
{api_base}/responses, for streaming, non-streaming and mode: responses
deployments alike. Providers with their own Responses config are unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop deployment supported_endpoints opt-in after cross-provider prompt swap
A prompt manager that moves the request to another provider leaves kwargs['model_info']
describing the original deployment; without this the swapped provider was sent an
OpenAI-like /responses request it does not serve.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): carry prompt-swap deployment metadata as a return value instead of a kwargs marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The module, its routing hook and its tests carried long prose rationale where the
repository allows only concise comments for genuinely complex logic. Trimmed to the
non-obvious reasons and dropped the rest; no logic or test behaviour changes.
test_router.py is not ruff-formatted on staging and CI's format check only scopes
litellm/*.py, so running ruff format over the whole file rewrote ~900 lines of
unrelated code. That reflow split long single-line patch() calls into multi-line
form, which the test-quality gate counts individually, pushing TQ008 four over its
ceiling. The file is back to staging's formatting with only the compression test
class added.
test_common_request_processing.py armed a model-side guardrail name with no such
guardrail registered, which stopped working once both hops began requiring the name
to resolve to an active compression guardrail.
per_user_usage.tsx conflicted with the server pagination that already landed on
staging (default 50 rows, stale-response guard, tag and page-size resets). Took
the staging version and dropped this PR's now-redundant 25-row test for it
Claude-Session: https://claude.ai/code/session_01HkaXiD6gssHnx3kqu1rR8C
The two policy fields are operator-supplied names and nothing else constrained them.
The routing hop calls apply_guardrail directly, which hands the guardrail the
conversation and POSTs it to whatever service backs that guardrail, and the model hop
is added to metadata["guardrails"], which runs it even when it is not default_on. So
naming an ordinary guardrail turned either hop into a way to invoke it and ship prompt
content to it. Both hops now refuse a name that does not resolve to an active
compression guardrail, and say so in the log rather than failing quietly.
Generalizes the heuristic_v2 ceiling from #39468 into a capability table whose
records own their in-process predicate, SQL spelling and refusal wording. The
existing heuristic_v2 capability keeps its own one-router ceiling. A single
customization capability combines operator-defined tier definitions with every
operator-written part of the classifier prompt. The prompt half only applies to
classifier types that call an LLM. The shipped default prompt, classification
rubric presets, tier-label renames and tier model choices remain ungated.
Scope every enforcement point to actual complexity routers. A model-less PATCH
or legacy update now decrypts the stored model before accepting strategy-router
settings, so a regular model cannot acquire a router config or spend a license
slot. Under the existing advisory lock, the cross-pod candidate query returns
only model scalars and the count decrypts and classifies them in process; old
non-router rows carrying a capability-shaped config no longer block a real
complexity router. The signed auto_router license feature makes both ceilings
unlimited.
The router reused the model hop's compression for routing whenever both hops named
the same guardrail, on the premise that arm_pre_call had already run it. Only the
proxy calls arm_pre_call, so through the SDK nothing armed the guardrail and nothing
had compressed anything: the shortcut skipped routing compression too and served the
request with no compression on either hop. The reuse is now conditional on the model
hop actually having been armed.
The Admin UI hydrated an absent auto_router_model_compression as same-as-routing,
while the backend reads it as no model-hop compression. Opening a router configured
with only auto_router_routing_compression and saving any unrelated edit wrote the
routing guardrail onto the model hop, silently starting to compress the model call.
Both carry a regression test that fails when the fix is reverted.
A slow or unreachable counter store made a pre-relay rejection wait behind
the reservation release before the client saw the error event and the close.
Close first and release in finally, mirroring the relay's own failure path,
so a client that already hung up still gets its reservation released.
The three pre-relay exits of realtime_websocket_endpoint (missing model,
key/model access denied, pre-call rejection such as a rate limit or a
guardrail) returned before the finally that releases the auth-time budget
reservation, so a rejected session pinned the key at the reserved amount
until the counter TTL expired and its next requests got budget_exceeded
while /key/info showed spend 0. A single _reject_realtime_session helper
now releases the reservation before sending the error event and closing,
and release_or_invalidate_budget_reservation shields the release from a
second cancellation and logs, rather than raises, a failing invalidate
fallback so it can never mask the session's own outcome.
The blanket finally release from the previous commit also zeroed the reservation
of successful sessions. Success settlement is enqueued on the logging worker, not
awaited, so the endpoint's finally ran first and released the reservation the cost
callback still had to reconcile, dropping the real spend from the key/team/user
counters.
The relay now stamps a synchronous marker (REALTIME_SESSION_SUCCESS_LOGGED_KEY) on
the shared logging object at the single success-dispatch site, and the endpoint
releases the reservation only when that marker is absent. Refused or failed
sessions, which never log success, still release; successful sessions leave the
reservation for the cost callback to settle to actual spend. Exactly one settler
touches each reservation, so the idempotent reconcile never double-adjusts.
