Five findings, all in surface this PR added, none of them a regression to an
existing path.
The dry-run gate had no test at either form. Deleting the whole block from both
left all 613 tests green, so the mechanism the validate endpoint exists to serve
could have been removed invisibly. One case per form now asserts the save is not
attempted on a rejecting verdict, and one asserts it still is on a passing one.
An Admin Viewer could read the built-in prompt through the GET but got 403 on the
preview, because default-allow only covers safe methods and a POST needs listing.
Both methods on this path are pure reads, so the path joins admin_viewer_routes.
The preview normalized nothing while the write gate strips and caps at 2000, so
an untrimmed preamble previewed with whitespace the router removes, and an
over-long one previewed fine and then failed on save. Both now call one
normalize_classification_prompt, the same shape the prompt builder already uses,
and the hand-rolled negative-window branch gives way to the field's own bound.
A failed preview fetch rendered as a permanent "Loading", so a denied role saw a
spinner rather than an error. The state is now loading, error or ready.
Also: the built-in branch of the edit modal's dropped list was dead, since the
key is managed and filtered earlier, and its comment described the wrong
mechanism; and the prompt editor is renamed CustomTierPromptEditor so it is not
one character from the built-in ClassifierPromptEditor beside it.
The gate read the verdict's error as well as its valid flag, so a rejection that
carried no message fell through to the write and came back as the raw 400 the
dry-run exists to replace. valid is derived from error server side today, which
is why nothing had hit it, but the two arrive as independent fields and the UI
should not depend on that derivation.
Both forms now call one dryRunRejection, which gates on valid alone and falls
back to its own message when the backend sent none. A transport failure still
fails open as valid, leaving the write gate authoritative.
Reported by Bugbot.
The preview sent tier_definitions and classification_prompt as query params, so an
operator's own instructions and calibration examples reached every access log on
the path. They move to a POST on the same route with a typed body, and the GET
goes back to exactly the shape it already shipped, since it is a live route and
the fields it keeps are not the sensitive ones. Validating TierDefinition on the
model rather than parsing a JSON string drops the query-param helper with it.
The How Classification Works card also stops rendering under an edited tier set.
It exists to describe the seven-dimension scorer and its score ranges, and that
scorer never runs there, so a sentence saying the card does not apply was still a
card that did not apply.
The prompt editor previewed a rubric it rebuilt in the browser, which could not
show a built-in tier's criteria at all: a definition naming a built-in tier may
leave its description blank to track the shipped wording, and that wording lives
only in the backend. Blank rows rendered as an empty bullet, and the dashboard's
own tier copy is different text, so pasting that in would have shown a rubric the
classifier never receives.
default_prompt now takes tier_definitions and classification_prompt and returns
the assembled prompt, built by custom_tier_classification_prompt. The live
classifier calls the same function, so the preview cannot drift from what the
router sends: verified byte-identical against a running proxy.
The editor renders that response, debounced against the draft, so it previews
what is being typed rather than only what is saved. A failed fetch drops the
preview and leaves the edit usable, since nothing is saved from there.
Matches the edit auto-router modal at sm:max-w-4xl so the two do not jump size
when opened one from the other, and calls the block below the textarea what it
is rather than describing where it goes.
Restore defaults re-added only the missing built-in tiers and kept every custom
row, so a three-tier set became seven rather than four. That reads as a reset and
is not one.
It now does what the button beside it did: drop the edited set and rebuild the
four built-in rows. The two buttons had the same destination, so the additive
helper and its guard against overflowing the tier limit are gone with it, since
a reset can never exceed four.
The control is the one place an edited tier set writes calibration examples, so
the label and the line under it now say that rather than describing the text as
opening instructions. The label is fixed instead of switching on whether a
prompt is set, since the Reset to default link beside it already carries that.
The classifier prompt control was replaced by a sentence in custom mode, so an
edited tier set could not steer its classifier at all. That control edits
classifier_llm_config.system_prompt, a whole-prompt replacement the backend
rejects beside tier_definitions, so refusing it there is right.
classification_prompt is the field that fits: the router uses it as the opening
of the prompt and appends the tier bullets and the injection guard after it, so
an operator can state their own instructions and calibration examples without
being able to restate or remove either. It has always been accepted beside
tier_definitions and the dashboard just never exposed it.
Custom mode now gets its own editor for that field, showing the tier bullets it
will append underneath. Built-in routers keep the whole-prompt editor. The
classifierPrompt restriction goes away with the control it described, and the
Restricted helpers move out of ComplexityRouterConfig, which was at its
max-lines ceiling.
The auto-router form was fixed at the four built-in complexity tiers. The
backend has taken tier_definitions since #37226, so the only way to add a tier
was hand-written raw config plus a proxy restart.
Edit tiers turns the existing tier list into an editor: rename a built-in,
remove one, restore it, or add a tier with a name, the definition the LLM
classifier routes on, and its models. A Fallback Tier picks where classifier
failures land.
Every input an edited tier set forbids lives in one table beside the payload
keys it owns, so a control cannot grey out while its value still ships:
display names, escalation, adaptive routing, session pinning, the heuristic
classifier, the replacement classifier prompt, the rubric preset, and the
classifier-failure fallback. The scorer knobs go too, as inert rather than
rejected.
Per-model params live on the tier row that owns them, so a row's models and its
reasoning efforts cannot be keyed differently and travel together through a
rename, a removal, and a return to the built-in tiers.
Both forms dry-run POST /auto_router/validate_complexity_router_config against
the exact payload the save will send, and fail open on a transport error since
the write gate stays authoritative. A form that never opens the editor submits a
byte-identical payload.
Resolves LIT-5214
The keywords feed the scorer's technical dimension, so they change tier decisions
on any router that scores. The control rendered only for classifier_type
'heuristic', while the scoring knobs right below it already gated on
heuristicScoringRole(value) !== 'never'. The two disagreed, so an operator could
edit boundaries and weights on a router whose keywords they could neither see nor
set.
That hid the control on an LLM classifier using the default heuristic fallback,
and on heuristic_first, which runs the scorer on every request to decide whether
to short-circuit. Both now read the same predicate as the panel below them.
* feat(ui): add Teams list CSV export with budgets, model grants, and rate limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): neutralize formula-leading values in teams CSV export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(complexity_router): heuristic-first classifier chaining
Adds classifier_type 'heuristic_first', which scores locally on every request and
only calls the LLM classifier for traffic the scorer could not place at or below
heuristic_first_max_tier. A request short-circuits when the scorer landed at or
below the threshold and produced at least one signal; everything else escalates.
The signal requirement is load-bearing. A prompt where no dimension fires scores
exactly 0.0, which is under simple_medium, so the score-to-tier mapping calls it
SIMPLE by default rather than by evidence, and that is about half of general
traffic. Gating on the tier alone would route it to the cheapest model without
ever consulting the classifier.
Introduces uses_llm_classifier as the single owner of 'does this router call the
classifier model', replacing the classifier_type == 'llm' comparisons in the
config validator, the prompt prebuild, the health dependency graph, the
routing-test authorizer, and six dashboard sites.
* fix(complexity_router): reuse the heuristic verdict on classifier failure, load the threshold on edit
Three review findings, one push.
The heuristic-first fallback re-scored the prompt after a classifier failure,
which the README already documented as a reuse. The outcome computed before
escalation is now handed to the failure path, so the scorer runs once per request.
The edit modal never hydrated heuristic_first_max_tier, while save rebuilds every
managed key from form state, so opening a heuristic-first router and saving it
dropped a field the proxy requires. The dropdown's display fallback hid it. Both
are fixed, and the hydration is extracted into a pure function so a test can pin
the invariant: every managed key present in a stored config survives an untouched
open-and-save. That test also covers every field added later.
Classifier radio labels lost their em dashes, per the repo writing convention.
Two gaps the create form and the edit modal share today.
The submit gate never asked for a classifier model. Choosing the LLM classifier
and no model leaves Test Routing and Add Auto Router enabled, so Test Routing
posts a config the backend rejects and only the later save says why.
The keyword-rule gate only looked for empty keyword rows. A rule's tier has been
a free string since #37413, and the backend matches it exactly, so a rule naming
a tier the router does not have cleared the gate and failed the save as a raw
400.
Both gates now live in build_complexity_router_config.ts, and each form's submit
handler reads the same blocked reason the button reads instead of re-deriving
its own list, so a disabled button and a refused submit cannot disagree.
* fix(logging): stop billing and logging response reads as LLM calls
Retrieving, deleting or cancelling a stored response, and vector store management calls, run through the same logging lifecycle as inference. A retrieved response replays the usage of the call that created it, so every read priced it again and wrote a second spend log row for the same tokens. Non-inference calls now cost 0, report no usage, log no placeholder chat message, and get a litellm.responses_management operation name instead of reading as chat.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep billing background response jobs after the poll
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): use an empty list for read-call messages
A tuple matches no branch in the loggers that walk this value, so lunary's
parse_messages falls through to clean_message and raises AttributeError on the
success hook. An empty list reads as no messages everywhere: it satisfies the
isinstance(list) checks in newrelic, mlflow and datadog, iterates zero times in
traceloop and helicone, and is what StandardLoggingPayload.messages is typed to
hold. None would be type-legal too but is not iterable, so it trades one crash
for another in mlflow and traceloop.
* fix(otel): stop the legacy emitter reporting replayed tokens on response reads
The zeroing so far lands in the standard logging payload, which the legacy
OpenTelemetry emitter does not read for usage: it takes prompt, completion and
total tokens straight off the response object, so a retrieval span still carried
the token counts of the call that produced the response, and the token usage
histogram still recorded them. That emitter is the default, so the spend row said
zero while the trace said otherwise. The background cost poller keeps its counts,
the same exemption the pricing path already makes.
* fix(logging): keep billing a background response when its retrieval is read
A response created with background=true comes back queued and carries no usage, so
its create bills nothing. The retrieval that first sees the finished job is the only
place that job's tokens are ever visible, and pricing every read at zero therefore
loses the spend outright rather than deduplicating it. On a proxy without the
enterprise cost poller a background job ended up costing $0 end to end.
is_unbilled_non_inference_call now takes the response it is deciding about and treats
a background response the same way it already treats the poller's own read, which is
the same exemption seen from the other side. The legacy OpenTelemetry emitter's time
per output token metric picks up the read gate it was missing, so it stops dividing a
read's latency by the replayed completion token count.
* test(proxy): pass the read response to the non-inference predicate
The poller test called is_unbilled_non_inference_call with the pre-background signature, so it broke when the predicate gained the response it classifies. It now hands the predicate a foreground read, and asserts that the same read is free without the origin stamp, so the stamp is what the test proves.
* fix(otel): stop the v2 metrics recorder reporting replayed tokens on response reads
The v2 span builder sources usage from the standard logging payload, so the
earlier fix already zeroes it there. The metrics recorder reads response_obj
directly, so a responses-management read still recorded the original
generation's tokens into gen_ai.client.token.usage and divided generation time
by them for gen_ai.server.time_per_output_token.
The read still records operation and response duration, under the
litellm.responses_management operation, so it stays observable.
* fix(proxy): keep the response-cost headers on calls priced at zero
Pricing responses reads and vector-store management routes at zero dropped the whole
x-litellm-response-cost family off those replies. The header build reads a falsy zero as
a cost this response never recorded and filters it out, and a call that returns before
pricing stores no cost breakdown for the component headers to read, so a client parsing
the cost off a read got a KeyError where it had previously been handed a number.
Those calls now advertise the family at zero. Retrieving a background response, and the
cost poller's read of one, still report their real cost.
The params-taking form of the predicate moves from opentelemetry into
internal_call_metadata so the proxy header build and the OTEL recorders share one copy.
* fix(proxy): report a zero cost split only under a zero cost total
The component headers were filled from call-type membership alone, while the
total they sit beside keeps its real value when the read priced normally, so a
breakdown that had not landed by the time headers were built could advertise a
real total next to an all-zero split. The split is now reported as zero only
when the total agrees with it, and is otherwise left absent.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
For non-Anthropic models served over /v1/messages, the outer wrapper recomputes
cost over the adapter-translated Anthropic response dict. That dict dropped every
web search usage signal, so the recompute overwrote the correct cost breakdown
with a token-only one: x-litellm-response-cost-tool-usage read 0.0 and
x-litellm-response-cost-original excluded the search cost, while the total kept it.
The adapter now maps web search request counts (from Usage.server_tool_use or
Gemini's prompt_tokens_details) into usage.server_tool_use.web_search_requests,
matching the Anthropic API shape, and the Gemini web search cost calculator falls
back to server_tool_use when prompt_tokens_details carries no count. The shared
get_web_search_requests helper is now public since five modules consume it.
Resolves LIT-6288
The case was skipped because /budget/update 500d on any model_max_budget.
#38430 fixes that by serializing the update payload before the write, so
the case now passes against a proxy carrying that change and there is
nothing left for the skip to hide.
Merge this after #38430; on staging alone the case still fails with the
same 500 it was skipped for.
Reverts #37725. The field existed so SDK callers that cannot read
`x-litellm-model-id` could tell which tier an auto-router picked, and the
framework that motivated it was LangChain. `@langchain/openai` builds
`additional_kwargs` and `response_metadata` from fixed key allowlists and drops
unknown fields at both the chunk top level and inside `delta`, so no
proxy-side placement of a namespaced key can reach a LangChain caller.
The complexity router's existing `return_raw_model_name` already covers that
case: it puts the resolved model in the standard `model` field, which
LangChain does propagate (`model_name` is on its metadata allowlist), and the
proxy honors it on both the streaming and non-streaming paths.
Keeps the unrelated cleanup from #37725 that dropped the redundant
function-local `ProxyBaseLLMRequestProcessing` import shadowing the
module-level one in `async_data_generator`.
`TestModelGroupAliasReachesPreRoutingStrategies` asserted on the marker as a
proof of strategy dispatch; the surviving `response.model == "gemini-flash"`
assertion already proves it.
/budget/update handed prisma the raw update dict, so a model_max_budget
payload reached the Json? column as a nested python dict. prisma-client-py
renders that into the GraphQL mutation as bare object keys rather than a
JSON string, and the query engine rejects it, so every per-model budget
update returned a 500 and the cap was never stored. Model ids carrying
punctuation (glm-5.2) also produced an invalid GraphQL name.
/budget/new already ran its payload through jsonify_object for exactly this
reason. Do the same on the update path. Team member and organization member
budget updates route through this handler too, so they were failing the same
way.
The existing unit tests mocked the prisma table with an AsyncMock that
accepts any dict, which is why this never showed up outside a live proxy.
The new test asserts on what the endpoint hands prisma.
The breakdown priced reasoning tokens at the flat standard rate while the
total billed them tier-aware, so on flex requests the reasoning sub-cost
header could exceed the whole response cost. Route the breakdown's
reasoning rate through the same tier-aware resolver as the total.
On /v1/messages the response is a TypedDict that can never carry hidden
params, yet the client wrapper still recomputed cost on it, clobbering the
already-correct breakdown with a tier-less, reasoning-less one. Skip the
metadata pass for results that cannot hold hidden params, since apply()
discarded it anyway.
