The hint copy described Bedrock's unit semantics and cost map entry even
though any provider's units reach this view, so it now explains the math
in provider-neutral terms. When units have no known price, the hint says
so and links to a prefilled GitHub feature request (provider and counter
names filled in) so the reader can ask for pricing. Per-unit prices below
$0.000001 now read "< $0.000001" instead of "$0".
Refs LIT-5652
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
* fix(cloudzero): preserve late resource tags
* style(cloudzero): remove redundant test comment
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
Adds a "How is this calculated?" hover to the Guardrail Cost card on the
overview and to the Cost and Usage Units cards on the detail page. The
overview hint lists each guardrail's cost and the total; the detail cost
hint shows units x per-unit price per counter with unpriced units called
out, and the units hint shows the per-counter sum. Also moves the Status
column to the front of the overview table.
Refs LIT-5652
#39680 made every admin table honor the selected page size, so the Tags and Model Hub tables now paginate at 25 by default and the two scroll specs, which seed 40 rows and expect them all on one page, fail on every litellm-e2e-ui run since (builds 206 to 208). Selecting 50 rows first, the way the Request Logs spec already does, keeps the overflow assertion meaningful
* fix(router): coordinate async and sync failure handlers at remaining router call sites
Five router failure paths still scheduled logging_obj.async_failure_handler
as a task while starting logging_obj.failure_handler on a raw thread, so
both handlers mutated the same logging object concurrently. Route them
through dispatch_failure_handlers like the streaming paths already do.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): wait on the real logging executor and justify the callbacks global patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): submit sync failure handler even when the dispatch task is cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): justify the executor submit patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_no_linear_scans_in_router: #39674 renamed heuristic_v2_router_limit_violation
to auto_router_capability_violation, so the allowlist entry stopped matching and the
same admin-only scan tripped the static check. Rename the entry to follow it.
tableScrolling.spec.ts: 9ba6cab889 (LIT-4738) gave the Tags and Model Hub tables
client-side pagination at 25 rows, so the 40 seeded rows no longer render on one
page. Select 50 rows per page before counting, as the Logs case already does.
With the selection in the URL, a mistyped or deleted guardrail id lands
on the info view's not-found branch, which rendered only the message and
left no way back to the table short of editing the address bar. The
not-found branch now shares the Back to Guardrails button with the
loaded view
* fix(proxy): make the invalid-model 403 path cheap under a burst of rejections
Keep the wildcard pattern registry in specificity order at registration time
so route() no longer re-sorts every pattern per lookup, and reuse the
standardized failure payload across the async and threaded sync failure
handlers regardless of what a callback did to log_event_type. Rejections
are still logged and observable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(router): wrap the filtered pattern tuple the way ruff format wants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router,logging): assert registry order and callback awaits instead of patching a class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): inject the pattern sorter so the lookup test observes that route() never sorts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logging): scan large base64 payloads for log truncation off the event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logging): make base64 offload threshold a plain constant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(hide-secrets): stop redacting benign identifiers and make redaction deterministic
The OpenAI key detector matched `sk-` anywhere inside a word, so `<task-notification>`
became `<ta[REDACTED]>`, and the Base64 entropy limit of 3.0 flagged ordinary quoted
identifiers such as `"application/json"` and model ids. Redaction also iterated a
hash-seeded set, so the same request produced different bytes on different workers and
broke prompt caching.
- require a standalone `sk-`/`sk_` token with a digit (still catches sk-proj-/sk-ant-)
- raise Base64HighEntropyString limit from 3.0 to the detect-secrets default 4.5
- redact overlapping matches longest-first in a stable order
Resolves LIT-7049
* fix(hide-secrets): treat separators as key boundaries and defer sk_live_ to the stripe detector
The standalone-token boundary also rejected keys glued to a preceding `_`, `-`
or percent-encoded delimiter (`openai_sk-…`, `key-sk-…`, `Bearer%20sk-…`), which
the old pattern redacted, and `sk_live_…` was counted by both the OpenAI and the
Stripe detector.
* fix(hide-secrets): keep the openai key scan linear on repeated sk separators
The digit requirement was a lookahead, so every `sk` inside a long
`[a-zA-Z0-9_-]` run re-scanned the rest of that run looking for a digit.
100 KB of `-sk-` took over 5s in the worker's event loop and the proxy
closed the connection without a response. The check now runs once per
match in `analyze_string` instead.
* chore(hide-secrets): remove redundant performance test comment
* fix(hide-secrets): consume complete openai key tokens
* chore(hide-secrets): remove redundant fixture comment
* chore(hide-secrets): remove redundant test docstrings
* fix(hide-secrets): redact whole stripe live keys
* style(hide-secrets): wrap secret sorting key
The Guardrails and Guardrails Monitor pages kept the selected guardrail
in local React state, so the detail view could not be shared, reloaded,
or reached with the browser back button. Both pages now read and write
the selection through the nuqs `guardrail` query param, matching how the
keys, teams, orgs, projects, users, models and logs pages deep link their
detail views. Opening a guardrail pushes a history entry and closing it
replaces the entry so back returns to the page the user came from
* fix(proxy): retry deadlocks and requeue spend logs on any DB write error
update_spend_logs dequeued the batch and only retried/requeued on transport
errors. A 40P01 deadlock surfaced as a plain prisma DataError and went through
poison-row isolation, which dropped every row it hit; every other DB error was
re-raised with the batch already gone from the queue.
Treat deadlocks as transient (retry, then requeue), keep them out of poison-row
isolation, and requeue the batch at the head of the queue on any other prisma
error so it lands once the DB is healthy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop redundant docstrings and tighten test typing for spend-log requeue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert deadlock retries from mock call history instead of mutable lists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): hold max_parallel_requests slot until streaming response is exhausted or closed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): normalize deployment_slot once to keep stream_with_fallbacks under the C901 ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): close upstream stream before releasing max_parallel_requests slot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(responses): honor supported_endpoints /v1/responses opt-in for OpenAI-compatible deployments
custom_openai and other generic OpenAI-compatible deployments have no native
Responses API config, so every /v1/responses call is bridged through
/v1/chat/completions. When model_info.supported_endpoints lists /v1/responses,
resolve OpenAILikeResponsesConfig instead so the request is forwarded to
{api_base}/responses, for streaming, non-streaming and mode: responses
deployments alike. Providers with their own Responses config are unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop deployment supported_endpoints opt-in after cross-provider prompt swap
A prompt manager that moves the request to another provider leaves kwargs['model_info']
describing the original deployment; without this the swapped provider was sent an
OpenAI-like /responses request it does not serve.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): carry prompt-swap deployment metadata as a return value instead of a kwargs marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The module, its routing hook and its tests carried long prose rationale where the
repository allows only concise comments for genuinely complex logic. Trimmed to the
non-obvious reasons and dropped the rest; no logic or test behaviour changes.
The marker walk rebound a loop-local on each iteration, which is the mutation the
repository's convention exists to discourage, but a `: Final` cannot express that
inside a loop body: basedpyright rejects it outright with 'A Final variable cannot
be assigned within a loop'. A lazy generator binds the name once per item and never
rebinds it, so the first marker carrying a policy still wins and the rest are never
read.
The backend treats either compression key on its own as an authoritative policy, but
hydrate returned the untouched inherit state whenever the routing key was absent. A
config carrying only auto_router_model_compression was therefore invisible in the
form, and picking a routing value then overwrote the stored model hop. Only neither
key set now reads as untouched, and an absent key on either hop hydrates as no
compression for that hop rather than same-as-the-other.
Four registry cells that had no e2e test. The cache_control suite gains a direct
Anthropic case (the same cache_control prefix the Bedrock and Vertex rows send)
and an OpenAI case, where caching is automatic so the prefix goes out as a plain
system string with a prompt_cache_key; both assert the second identical call
reports cache-read tokens. The shared second-call helper now takes the send
callable so the OpenAI shape fits without a second copy of the retry loop.
The embeddings suite gains a cohere/embed-v4.0 deployment that must return a
non-zero vector, and the passthrough suite gains an OpenAI-format chat through
the raw /openai/v1/chat/completions prefix that must relay a real completion and
log a costed pass_through_endpoint row whose token counts match the usage the
caller was served.
Three deterministic proxy-only cells from the coverage registry that had no e2e
test. A key over its max_budget is reset to 0 through /key/{key}/reset_spend and
must both read back 0 on /key/info and serve traffic again. /key/regenerate with
grace_period keeps the old key valid until the period elapses and rejects it 401
afterwards. A key whose allowed_routes is the llm_api_routes group must reach
/chat/completions and /embeddings while /model/new stays 403.
KeyRegenerateBody gains grace_period and the management client gains
reset_key_spend so the tests stay on the shared typed transport.
Adds TestAnthropicChatCompletions to the chat completions regression suite,
registering a claude-haiku-4-5 deployment via /model/new and asserting the
streamed call delivers real content deltas and a tool-forced call returns a
well-formed get_weather tool_call on both the non-streamed and streamed paths.
Covers three P0 registry cells that had no e2e test.
test_router.py is not ruff-formatted on staging and CI's format check only scopes
litellm/*.py, so running ruff format over the whole file rewrote ~900 lines of
unrelated code. That reflow split long single-line patch() calls into multi-line
form, which the test-quality gate counts individually, pushing TQ008 four over its
ceiling. The file is back to staging's formatting with only the compression test
class added.
test_common_request_processing.py armed a model-side guardrail name with no such
guardrail registered, which stopped working once both hops began requiring the name
to resolve to an active compression guardrail.
per_user_usage.tsx conflicted with the server pagination that already landed on
staging (default 50 rows, stale-response guard, tag and page-size resets). Took
the staging version and dropped this PR's now-redundant 25-row test for it
Claude-Session: https://claude.ai/code/session_01HkaXiD6gssHnx3kqu1rR8C
The two policy fields are operator-supplied names and nothing else constrained them.
The routing hop calls apply_guardrail directly, which hands the guardrail the
conversation and POSTs it to whatever service backs that guardrail, and the model hop
is added to metadata["guardrails"], which runs it even when it is not default_on. So
naming an ordinary guardrail turned either hop into a way to invoke it and ship prompt
content to it. Both hops now refuse a name that does not resolve to an active
compression guardrail, and say so in the log rather than failing quietly.
Generalizes the heuristic_v2 ceiling from #39468 into a capability table whose
records own their in-process predicate, SQL spelling and refusal wording. The
existing heuristic_v2 capability keeps its own one-router ceiling. A single
customization capability combines operator-defined tier definitions with every
operator-written part of the classifier prompt. The prompt half only applies to
classifier types that call an LLM. The shipped default prompt, classification
rubric presets, tier-label renames and tier model choices remain ungated.
Scope every enforcement point to actual complexity routers. A model-less PATCH
or legacy update now decrypts the stored model before accepting strategy-router
settings, so a regular model cannot acquire a router config or spend a license
slot. Under the existing advisory lock, the cross-pod candidate query returns
only model scalars and the count decrypts and classifies them in process; old
non-router rows carrying a capability-shaped config no longer block a real
complexity router. The signed auto_router license feature makes both ceilings
unlimited.
The router reused the model hop's compression for routing whenever both hops named
the same guardrail, on the premise that arm_pre_call had already run it. Only the
proxy calls arm_pre_call, so through the SDK nothing armed the guardrail and nothing
had compressed anything: the shortcut skipped routing compression too and served the
request with no compression on either hop. The reuse is now conditional on the model
hop actually having been armed.
The Admin UI hydrated an absent auto_router_model_compression as same-as-routing,
while the backend reads it as no model-hop compression. Opening a router configured
with only auto_router_routing_compression and saving any unrelated edit wrote the
routing guardrail onto the model hop, silently starting to compress the model call.
Both carry a regression test that fails when the fix is reverted.
A slow or unreachable counter store made a pre-relay rejection wait behind
the reservation release before the client saw the error event and the close.
Close first and release in finally, mirroring the relay's own failure path,
so a client that already hung up still gets its reservation released.
