The editor over the model layer beneath it. An Edit tiers button turns the tier
list into an editor: a tier takes a name, a classifier definition, and models,
between two and eight rows. Restore defaults resets to the built-in four rather
than stacking them on top. Keyword rules follow a rename, an orphaned rule
blocks the save, and both forms dry-run the exact payload against the backend
validator before writing.
The edit modal hydrates a stored custom set into rows, and an untouched
open-and-save round-trips byte-identically, per-model reasoning efforts
included. A form that never opens the editor submits the same bytes as before.
The cost-optimization tier chart renders arbitrary tier names: the guard that
returned no models for a non-built-in name is gone, and the fixed four-color
array gives way to the shared cycle.
* feat(ui): the model and wire layer for operator-defined auto-router tier sets
The data half of the custom tier set editor, with no visible UI change: the
editor lands separately on top of it.
One reader, activeTierRows, mints built-in rows with the canonical tier key as
their id, so the fallback pointer, the plan-mode floor and the per-model params
are row ids in both modes and nothing downstream branches on the mode. One
restrictions table carries each forbidden setting beside the reason shown for
it, so the greyed control and the omitted payload key cannot disagree. The
tier-set writes live in applyTierSetAction, where the fallback re-point and the
floor turn-off happen in one commit, unit-tested without a render.
buildComplexityRouterConfig emits tiers, tier_definitions and fallback_tier from
the rows, forces the LLM classifier, and strips what the backend rejects beside
tier_definitions. A payload built without a custom tier set is byte-identical to
what the form sends today.
* fix(ui): resolve frontend-lint failures on the tier-set model layer
* test(ui): drop a redundant explanatory comment per repo convention
* fix(ui): keyword rules follow their tier row through every tier-set action
The endpoint built messages=[{"role": "user", "content": prompt}], so a dry run
could not carry prior turns, the caller's system prompt, or the tool definitions
a request advertises. A real agentic turn reduced to its last sentence classified
as trivial, which is why a config sweep reported savings for every configuration.
Accept messages, system and tools, and forward them to the same pre-routing hook
untranslated, with the raw-body snapshot built by the serving path's own owner,
refresh_proxy_server_request_body_snapshot. Loose types are deliberate: the hook
reads whatever dialect the surface produced, so validating against one surface's
schema would reject the others.
prompt stays as the single-ask shorthand, normalized into one user turn inside the
request model so the handler carries no mode branch.
* fix(guardrails): add fail-open mode to CrowdStrike AIDR guardrail
Add a fail_on_error param (default True, preserving existing behaviour) to
the CrowdStrike AIDR guardrail, mirroring model_armor and generic_guardrail_api.
When fail_on_error=False the guard fails open only on server errors (5xx) and
connectivity failures, so the request proceeds unmodified. Caller-controlled
4xx responses and result.blocked policy blocks always fail closed. The
applied-guardrails header is recorded even on the fail-open path.
* fix(guardrails): fail open AIDR 4xx
* refactor(guardrails): isolate AIDR fail-open
* style(guardrails): format AIDR fail-open
* ci: satisfy unit workflow timeout invariant
* refactor(guardrails): accept AIDR mappings
* test(guardrails): inject AIDR HTTP client
* fix(guardrails): harden AIDR fail-open against delivered verdicts and record fail-open status
Reads the blocked verdict from the raw body before guard_output validation so schema drift or a changed verdict type cannot fail open past a delivered block. A transformed response that cannot be parsed fails closed so delivered redactions are never dropped. Fail-open runs record guardrail_status guardrail_failed_to_respond with timings instead of success. Restores the fail-open behavior tests dropped mid-PR and reverts the payload Mapping widening
* test(guardrails): cover fail_on_error wiring and fail-closed default for CrowdStrike AIDR
* chore(guardrails): annotate the transformed-drift detail payload for the LIT002 budget
---------
Co-authored-by: abrekhov <abrekhov@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): dry-run an auto-router config against the backend before saving it
Both auto-router forms built a payload and posted it, so anything the write gate
refused came back as a raw 400 with the backend's message buried in it. They now
POST the exact payload to /auto_router/validate_complexity_router_config first
and surface its verdict inline.
One dryRunRejection owns the gate, and it reads valid alone. The verdict's two
fields arrive independently, so gating on the error message would let a rejection
that carried none through to the write. A transport failure fails open as valid,
leaving the write gate authoritative rather than blocking a save on a flaky
network.
Applies to every auto-router, built-in tiers included.
* fix(ui): hold the auto-router create closed for the full dry-run and create sequence
A second submit while the dry-run round-trip was pending started another
create against the non-idempotent /model/new. The submit handler now
refuses re-entry and the button disables for the whole sequence, matching
the edit modal's loading guard. Also drops the explanatory comments this
PR had added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(ui): session-level cache observability in request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: guard cache_hit filter against non-string defaults in direct calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): drop redundant cache_hit field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two lazily loaded models changed without their generated artifacts being
regenerated, so check-ui-api-types has been red on every branch off staging.
The snapshot that /openapi.json serves for unloaded features was missing
ChatCompletionToolReferenceObject, and the dashboard types were missing
aws_external_id. The snapshot step runs first and short-circuits, so only the
first one was visible until it was fixed.
Both files are regenerated with `python -m litellm.proxy._lazy_openapi_snapshot`
and `npm run gen:api`, no hand edits.
* feat(alerting): add native Microsoft Teams alerting destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): preserve active destinations on MS Teams save and confirm health test delivery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): read persisted alerting destinations at MS Teams save time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An MCP server behind an API gateway needs two credentials on one request: the
gateway's own token on a private header, and a separate bearer on Authorization
for the server behind it. Every arm that minted or held a token hardcoded
Authorization, and the conflict rule then dropped the operator's static
Authorization to make room, so the second credential never arrived.
ApiKeyConfig already modelled this as header_name plus value_prefix behind a
header() method. Extend that carrier to the four minted-token configs, have each
resolver arm ask its config which header to use instead of naming one, and drop
only the header the resolved credential is about to occupy.
Operators set it per server via upstream_token_header, plumbed through
config.yaml, the credentials blob, the management API and the admin form, on the
M2M, token-exchange, authorization-code and ID-JAG arms. It is non-secret so it
stays plaintext and round-trips on admin reads. Unset keeps today's behaviour.
Moving a credential off Authorization means it stops inheriting what Authorization
gets for free, so the slot now carries those protections itself. httpx drops
Authorization when a redirect crosses origin and keeps every other header, so a
custom slot is dropped by the client on the same condition, mirroring httpx's own
scheme/host/port rule with an agreement test that fails if the two ever diverge.
The v1 path also mirrors the v2 conflict rule, so an injected header cannot shadow
the credential the gateway resolved for that slot.
Which header a credential occupies, and what counts as being that header, was
answered independently in nine places by four hand-rolled comparisons. same_header,
has_header and without_header in litellm/types/mcp.py are now the one owner, shared
by both MCP stacks, and the client derives its slot once instead of three times.
The header name reaches egress verbatim, so the RFC 7230 grammar lives in one
place and is checked where servers are built: a bad value fails the config load
and the management API returns 400, rather than raising while a spec is built
and emptying the aggregate tool list for every other server. A blank means unset,
matching what the endpoint already accepts.
Migrate the create-key user picker, add-member user search, and usage team filter onto the shared paginated selects, gate the logs error-code filter on input reasons, and add clearAllLabel, autoHighlight, and aria-required passthroughs the migrations need.
Select the picked label on focus and snapshot whether the pre-edit selection covered the whole input; when it did, the next input value is a full replacement, so skip the typedInsertion diff that mangles pastes sharing a prefix or suffix with the label.
* fix(ui): let the paginated search select keep what the user types
The combobox handed Base UI a freshly built option object for the current
selection every time a page of results came back. Base UI answers a changed
value by rewriting the input with that option's label, so every search response
wiped the query mid-typing and the list never narrowed. Once a user had been
picked in the Usage page filter box, no other user could be reached.
The component now owns the input text. It holds the query while the list is
open, falls back to the selected option's label once the list closes, and
remembers the picked option so its label survives later pages that no longer
carry it, the way the multi-select sibling already does.
* refactor(ui): name the paginated select's search state instead of commenting it
* fix(ui): start a fresh query when typing lands on the selected label
Focusing the filter box without clicking it leaves the caret at the end of the
selected option's label, so the next keystroke extended that label into a query
no server could match. Only a click cleared the box first.
A keystroke that arrives while the box is showing a label is now read as the
start of a new query, wherever in the label it landed.
The shared SelectContent wrapper defaulted alignItemWithTrigger to true,
which puts Base UI's positioner into item-aligned mode and places the
popup so the active item sits on top of the trigger. In that mode the
side and sideOffset the wrapper passes two lines above are ignored, and
the popup reports data-side="none".
The overlap only becomes visible once the items are tall enough to
matter, which is why the autorouter Template picker shows it clearly:
its options are three-line cards, so the popup covers both the select
box and its own label.
No call site in the dashboard asked for item-aligned mode. 21 of them
across 15 files already passed alignItemWithTrigger={false} by hand to
undo the default, and the remaining 127 inherited the bug. Flipping the
default makes side and sideOffset live, so collision handling works and
a select with no room below now flips above the trigger rather than
covering it. The 21 hand-written opt-outs are deleted as redundant.
* feat(health): opt-in model-group allowlist for background health checks and health-check routing
* fix(health): merge shared health states per writer scope instead of replacing
* refactor(health): drop restating comment and parameterize test scope annotations
* chore: remove stray generated prisma migration file
* fix(health): merge health states against the Redis snapshot, not the pod-local copy
* fix(health): fall back to the pod-local snapshot when the Redis read returns nothing
The preset put Fable 5 in REASONING, sitting above Opus in a Haiku to Sonnet to
Opus ladder even though Fable is the lighter model. Run Opus 5 there instead, at
high thinking, so the tier above COMPLEX is the same model thinking harder rather
than a different and lighter one.
This is the first bundled preset to carry tier_model_configs. The round trip was
already built and unit tested, but nothing between the bundled JSON and the
create payload asserted on it, so add that coverage here.
/v2/model/info returns llm_router.model_list, which carries no defined order: the DB
read has no order_by and an edited deployment is popped and re-appended. The Auto
routers table rendered that order verbatim behind a ten-row first page, so on a proxy
with more than ten auto routers a router created moments ago was drawn wherever the
API happened to return it, in practice last, and read as never created
Adopt the ordering the rest of the dashboard already uses, with the two cases this
table has and its siblings do not. created_at is enterprise-gated and config.yaml
routers never carry one, so seeding created_at desc alone leaves every comparison
tied on a non-premium proxy and the fix a no-op. The column now declares
sortUndefined last, which table-core applies before the desc flip so undated rows
stay last in both directions, and the row emits undefined rather than null so that
branch is reachable at all. Name is the secondary key, giving the undated block a
defined order too
Page size is deliberately unchanged: it exposes the missing order rather than
causing it
The savings card carried four numbers in two stacked halves: the headline
saving with its delta on the left over the two spend rows, and avg saved per
session on the right. Give the headline the whole left half, move the two
spend rows into a rail on the right, and drop avg saved per session into the
metric row below as its first tile, with the session count as an inline hint.
Each spend row stays a description list so assistive tech keeps the label to
value association, with the shadcn Separator between the two rows. Both hero
columns are minmax(0,1fr) so a large total wraps instead of overflowing the
card, which also fixes the clipping the old 1fr columns already had. Metric
grows one optional hint slot so the new tile reuses the same presenter as its
three siblings.
* fix(ui): carry a preset's per-tier litellm_params through the prefill
buildPresetPrefill rebuilt the complexity router config field by field and
never emitted tier_model_params, so a bundled preset that declares per-model
litellm_params (reasoning_effort, for instance) lost them before the create
form ever saw them. Both halves of the round trip already existed:
hydrateTierModelParams reads either storage shape, and serializeTierModelConfigs
writes them back on submit.
Hydrating alone is not enough. Tier entries get rewritten to the caller's
registered model spelling, which can differ from the preset's literal string by
version-separator punctuation, while the params stay keyed on what the preset
spelled. serializeTierModelConfigs then drops any param whose key is not in the
tier, silently. The param keys go through the same resolver as the tier entries.
* test(ui): catch a preset spelling the same model two ways in one tier
buildPresetPrefill resolves every model reference through normalizeModelName,
so two spellings of the same model in one tier (e.g. "claude-sonnet-4-5" and
"claude-sonnet-4.5") collapse to one key. For tier_model_configs that means one
model's litellm_params silently overwrites the other's - flagged by Greptile
on #38453 (P2, confirmed real via a throwaway repro, not a regression: on the
merge base both param sets were already dropped).
Nothing else validates preset authoring, and these are trusted, checked-in
JSON, so the fix is a static test over the bundled data rather than runtime
code. Exports normalizeModelName so the test exercises the actual resolution
rule instead of a hand-rolled copy of it. Verified the test fails when a
preset is mutated to spell one model two ways, and passes clean on the real
bundled presets.
The keywords feed the scorer's technical dimension, so they change tier decisions
on any router that scores. The control rendered only for classifier_type
'heuristic', while the scoring knobs right below it already gated on
heuristicScoringRole(value) !== 'never'. The two disagreed, so an operator could
edit boundaries and weights on a router whose keywords they could neither see nor
set.
That hid the control on an LLM classifier using the default heuristic fallback,
and on heuristic_first, which runs the scorer on every request to decide whether
to short-circuit. Both now read the same predicate as the panel below them.
* feat(ui): add Teams list CSV export with budgets, model grants, and rate limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): neutralize formula-leading values in teams CSV export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(complexity_router): heuristic-first classifier chaining
Adds classifier_type 'heuristic_first', which scores locally on every request and
only calls the LLM classifier for traffic the scorer could not place at or below
heuristic_first_max_tier. A request short-circuits when the scorer landed at or
below the threshold and produced at least one signal; everything else escalates.
The signal requirement is load-bearing. A prompt where no dimension fires scores
exactly 0.0, which is under simple_medium, so the score-to-tier mapping calls it
SIMPLE by default rather than by evidence, and that is about half of general
traffic. Gating on the tier alone would route it to the cheapest model without
ever consulting the classifier.
Introduces uses_llm_classifier as the single owner of 'does this router call the
classifier model', replacing the classifier_type == 'llm' comparisons in the
config validator, the prompt prebuild, the health dependency graph, the
routing-test authorizer, and six dashboard sites.
* fix(complexity_router): reuse the heuristic verdict on classifier failure, load the threshold on edit
Three review findings, one push.
The heuristic-first fallback re-scored the prompt after a classifier failure,
which the README already documented as a reuse. The outcome computed before
escalation is now handed to the failure path, so the scorer runs once per request.
The edit modal never hydrated heuristic_first_max_tier, while save rebuilds every
managed key from form state, so opening a heuristic-first router and saving it
dropped a field the proxy requires. The dropdown's display fallback hid it. Both
are fixed, and the hydration is extracted into a pure function so a test can pin
the invariant: every managed key present in a stored config survives an untouched
open-and-save. That test also covers every field added later.
Classifier radio labels lost their em dashes, per the repo writing convention.
Two gaps the create form and the edit modal share today.
The submit gate never asked for a classifier model. Choosing the LLM classifier
and no model leaves Test Routing and Add Auto Router enabled, so Test Routing
posts a config the backend rejects and only the later save says why.
The keyword-rule gate only looked for empty keyword rows. A rule's tier has been
a free string since #37413, and the backend matches it exactly, so a rule naming
a tier the router does not have cleared the gate and failed the save as a raw
400.
Both gates now live in build_complexity_router_config.ts, and each form's submit
handler reads the same blocked reason the button reads instead of re-deriving
its own list, so a disabled button and a refused submit cannot disagree.
* feat(langfuse): support langfuse_environment as a per-key dynamic callback param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the langfuse_environment constructor param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): only pass environment when the SDK client supports it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): drop the request-body metadata test for langfuse_environment
The proxy bans request-body callback params by default (derived from
_supported_callback_params in auth_utils), so the metadata channel this
test asserted is rejected with a 401 on the proxy. The supported channel
is admin-set key/team callback_vars, with LANGFUSE_TRACING_ENVIRONMENT
as the deployment-wide fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(langfuse): validate langfuse_environment, avoid redundant clients, honor it in langfuse_otel
Closes the review gaps on the langfuse_environment param:
- Validate values against Langfuse's environment pattern at save time
(/key/generate, /key/update, /team callback all 400 on e.g. 'Production'
instead of 200-then-silently-dropping every trace server-side) and at
logger init; non-string values are str()-coerced instead of crashing
the SDK's regex check per event.
- Treat empty/whitespace values and values equal to the deployment-wide
LANGFUSE_TRACING_ENVIRONMENT as non-dynamic so an environment-only
override that changes nothing no longer mints a duplicate SDK client
against MAX_LANGFUSE_INITIALIZED_CLIENTS.
- langfuse_otel now reads the per-key/team langfuse_environment from
standard_callback_dynamic_params instead of only the env var.
- Advertise the param on the discovery surfaces: callback_configs.json
(langfuse + langfuse_otel), the dashboard callback registry, and the
/team/{team_id}/callback docstring (schema.d.ts regenerated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format langfuse files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lint): remove duplicate test import, LIT002 dict literal, and mock-echo otel test
- drop redundant in-function import of callback_config_error (F811)
- avoid the `or {}` mutable literal in _set_langfuse_specific_attributes (LIT002)
- rewrite the dynamic-env otel test to observe span.set_attribute output
instead of patching litellm internals (TQ002/TQ008)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Gemini API Maps-grounded prompts were billed as web search and Vertex AI Maps-grounded prompts were not billed at all. Classify grounding metadata per candidate into web search vs Maps requests, carry a distinct google_maps_grounding_requests usage counter through non-streaming and streaming paths, and price it via the new google_maps_grounding_cost_per_query cost map key with per-query and per-prompt defaults keyed off web_search_billing_unit. Fixes#35906
The dashboard resolved the complexity-router tier set three different ways: a
private TIER_KEYS in build_complexity_router_config.ts, TIER_ORDER in
complexity_router_tiers.ts, and TIER_KEYS in ComplexityRouterConfig.tsx. The
edit modal went further and re-implemented the whole create payload builder,
kept in sync only by a comment reading "Mirrors buildComplexityRouterConfig".
tier_rows.ts now owns the tier set. Every consumer reads activeTierRows(value)
and a row carries its own id, so the plan-mode floor and per-model params point
at a row rather than at a position, and the leaves that already wanted entries
(buildAutoRouterTestTargets, getRequiredModels, model_info_view) take them.
buildUpdatedComplexityRouterConfig becomes preserve-unmanaged-keys around the
shared builder instead of a second copy of it.
Also drops the literal ", ]" that renders as visible text in two DialogFooter
blocks on the auto-router routing-test and connection-test dialogs, left over
from a JSX array-to-fragment conversion.
No behaviour change: all 566 tests over the touched modules pass with fixture
shape changes only, no assertion edited.
