AWS bills a Bedrock GPT-5.5 or GPT-5.4 prompt past 272K tokens under the long-context usage types for the
whole prompt, at 2x input, 2x cache read, and 1.5x output, and the cost map only had the flat rates, so a
300K prompt was logged at half of what the invoice charges. The map's promo rates for gpt-5.6-sol are 20%
under the $5.50 input, $33.00 output, $0.55 cache read, and $6.88 cache write per million the invoice bills.
Adds the *_above_272k_tokens fields to gpt-5.5 and gpt-5.4, moves sol's base and tier rates to the invoiced
ones, replaces the test that pinned the flat behaviour with one that pins the invoiced numbers, and updates
the sol pins in the mantle transformation tests
* fix(anthropic): resolve /v1/messages effort tiers through the capability owner
The bridge normalizer read three supports_*_reasoning_effort booleans of its own, so it
answered "which levels does this deployment take" independently of the resolver behind
/model_group/info. The two disagreed: a proxy advertising kimi-k3 max forwarded high.
Degrade against resolve_supported_reasoning_efforts instead, with the chains as a declared
table. When no step of a chain is accepted, the fallback is read off that same resolved set
rather than assumed, since an entry naming its levels outright can exclude the tiers the
per-level flags treat as unconditional. none is never chosen as that fallback, being an off
switch rather than a tier, and a deployment accepting no tier at all keeps the floor every
deployment degraded to before.
* test(anthropic): pin the normalized effort at the /v1/messages request boundary
The existing coverage stopped at normalize_reasoning_effort_value, so nothing failed if the
handler dropped or overwrote the normalized tier on its way into completion_kwargs. Drive
_prepare_completion_kwargs instead and assert on the kwargs handed to acompletion, in both the
string and the dict effort shapes, including the provider-prefixed model name the handler is
actually called with.
Against the pre-fix normalizer the fallback case fails, and against the baseline before a map
entry could declare its levels 7 of the 12 fail, so the boundary is pinned rather than restated.
A gpt-5 model accepts a non-default temperature only while its effective reasoning
effort resolves to "none". litellm had no representation of the effort a model applies
when the request omits reasoning_effort, so it substituted supports_none_reasoning_effort,
which is a different fact. Every model that supports "none" without defaulting to it
therefore had temperature forwarded and rejected upstream, and because the carve-out
returned before the drop_params branch, drop_params: true could not save it.
Declare the fact instead. A new cost-map key, default_reasoning_effort, states the effort
the provider applies when the request omits one, and one shared predicate resolves the
effective effort from it: an explicit reasoning_effort wins, otherwise the declared
default, otherwise the catalogue decides.
That last step matters because the cost map is fetched from the published branch at import
time, so it can be OLDER than the code reading it. On such a map every model looks
undeclared, and reading that as "reasoning is active" would strip temperature from the 39
gpt-5.1/5.2/5.4 entries that accept it, a regression caused by data lag rather than by
anything about the model. So an absent declaration is only meaningful once the catalogue
carries the key at all; a map that predates the feature keeps the answer litellm gave
before it existed, and the conservative answer applies from the moment the data lands.
The top_p/logprobs/top_logprobs gate carried the same assumption spelled differently and
now shares the predicate, as does the Responses API, which reimplemented the rule and is
what the default /v1/messages bridge routes openai models through. Azure normalises its
routing names in one resolver that every capability lookup goes through, which replaces
its bespoke per-lookup rewrite.
Declared on the 37 gpt-5.1/5.2/5.4 entries measured to accept temperature=0 today, so
their behaviour is unchanged. The 23 gpt-5.5/5.6 entries that reject it stay undeclared
and are fixed once the catalogue carries the key.
Resolves LIT-3797
Resolves LIT-5028
A shadow eval whose judge_model is one of the router's tier models, the router's
default model, or a reverse job's baseline_model was accepted with no warning. An
LLM judge scores its own output higher than a rival's, so that tier's win rate
measures the judge instead of the models, and the job's whole budget buys a result
that has to be thrown away.
start_shadow_eval now rejects it with a 400 naming the colliding arm.
`judge_target` is the single answer to "where does a call to this name go for this
caller, and what answers it", and the resolvability gate, the collision gate and
the judge dispatch all read it. It has three outcomes and no others: the router
serves the name, the SDK serves it, or nothing does. Splitting that question is
what every bug here came from, so `router_resolves_model` and `answering_models`
are gone rather than joined by a third.
Two spellings of one model are one identity. A name is compared by what would
answer it, resolved through every channel `get_model_list` composes and then put
in the provider-qualified form litellm itself uses, so a judge given as `gpt-4o`
collides with a tier deployment serving `openai/gpt-4o`, and a judge given as
`openai/gpt-4o` collides with a deployment configured as bare `gpt-4o`. Both ends
are normalised because an admin writes them at different times.
Answering is also per-caller. The shadow and judge calls carry the shadowed key's
`user_api_key_team_id`, which is what the router selects deployments with, so the
endpoint derives the job's teams once from the keys it already looks up and every
check runs under them, and the judge dispatch picks its arm under the same team.
A team's public model name resolves to nothing for everyone else and a team's own
deployment resolves for nobody else, so a check that omits the team answers for a
caller who does not exist. A collision under any one team fails the job, because
every key's verdicts land in the same win rates.
Three sites were separately re-deriving "the provider models this name resolves
to", with unexplained divergence in whether they fell back to the literal name.
`Router.resolved_litellm_models` is now the one owner; the routing-plugin
candidate list and the stream-options check both delegate to it, and
`_deployment_litellm_model` is gone.
The router's arms come from `strategy_router_dependencies`, the same enumeration
the health check reads. Only the roles that serve are arms: a classifier or
embedding model picks the tier and never produces a response anyone judges. A
semantic auto-router keeps its routes in an opaque config blob, so only its
default model is enumerable and the guard is incomplete there by design, able to
miss a collision but never to invent one
The two regenerated artifacts carry `presidio_analyze_chunk_size_bytes` from
alters the spec; the sync gate runs on any PR touching litellm/proxy, so this one
has to carry the base's drift to go green
The endpoint built messages=[{"role": "user", "content": prompt}], so a dry run
could not carry prior turns, the caller's system prompt, or the tool definitions
a request advertises. A real agentic turn reduced to its last sentence classified
as trivial, which is why a config sweep reported savings for every configuration.
Accept messages, system and tools, and forward them to the same pre-routing hook
untranslated, with the raw-body snapshot built by the serving path's own owner,
refresh_proxy_server_request_body_snapshot. Loose types are deliberate: the hook
reads whatever dialect the surface produced, so validating against one surface's
schema would reject the others.
prompt stays as the single-ask shorthand, normalized into one user turn inside the
request model so the handler carries no mode branch.
The top bar's theme control needed a click on the sun/moon, then a menu, then
a choice, to do something every other product does in one click. It is now a
plain button that flips between light and dark, with the beta marker moved into
the label of the click that turns dark on. An explicit "system" choice is gone,
but next-themes still follows the OS for anyone who has it stored and has not
clicked yet.
Docs and Blog also drifted apart in the gateway header: Blog rendered through
the shared product-link class while Docs was a muted ghost button one size
down, so Docs read as dimmer and sat 4px shorter. Both now go through a shared
DocsLink component, which is also what the legacy navbar uses, so the pair
cannot drift again.
* fix(langfuse): warn and drop invalid LANGFUSE_TRACING_ENVIRONMENT instead of failing requests
* fix(langfuse): treat a dynamic environment equal to the raw deployment value as redundant
* fix(guardrails): add fail-open mode to CrowdStrike AIDR guardrail
Add a fail_on_error param (default True, preserving existing behaviour) to
the CrowdStrike AIDR guardrail, mirroring model_armor and generic_guardrail_api.
When fail_on_error=False the guard fails open only on server errors (5xx) and
connectivity failures, so the request proceeds unmodified. Caller-controlled
4xx responses and result.blocked policy blocks always fail closed. The
applied-guardrails header is recorded even on the fail-open path.
* fix(guardrails): fail open AIDR 4xx
* refactor(guardrails): isolate AIDR fail-open
* style(guardrails): format AIDR fail-open
* ci: satisfy unit workflow timeout invariant
* refactor(guardrails): accept AIDR mappings
* test(guardrails): inject AIDR HTTP client
* fix(guardrails): harden AIDR fail-open against delivered verdicts and record fail-open status
Reads the blocked verdict from the raw body before guard_output validation so schema drift or a changed verdict type cannot fail open past a delivered block. A transformed response that cannot be parsed fails closed so delivered redactions are never dropped. Fail-open runs record guardrail_status guardrail_failed_to_respond with timings instead of success. Restores the fail-open behavior tests dropped mid-PR and reverts the payload Mapping widening
* test(guardrails): cover fail_on_error wiring and fail-closed default for CrowdStrike AIDR
* chore(guardrails): annotate the transformed-drift detail payload for the LIT002 budget
---------
Co-authored-by: abrekhov <abrekhov@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): dry-run an auto-router config against the backend before saving it
Both auto-router forms built a payload and posted it, so anything the write gate
refused came back as a raw 400 with the backend's message buried in it. They now
POST the exact payload to /auto_router/validate_complexity_router_config first
and surface its verdict inline.
One dryRunRejection owns the gate, and it reads valid alone. The verdict's two
fields arrive independently, so gating on the error message would let a rejection
that carried none through to the write. A transport failure fails open as valid,
leaving the write gate authoritative rather than blocking a save on a flaky
network.
Applies to every auto-router, built-in tiers included.
* fix(ui): hold the auto-router create closed for the full dry-run and create sequence
A second submit while the dry-run round-trip was pending started another
create against the non-idempotent /model/new. The submit handler now
refuses re-entry and the button disables for the whole sequence, matching
the edit modal's loading guard. Also drops the explanatory comments this
PR had added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Route records below WARNING to stdout (WARNING and above stay on stderr),
emit ANSI color codes only when both streams are a TTY (honoring NO_COLOR),
and parse JSON_LOGS strictly so JSON_LOGS=false no longer enables JSON logs.
* fix(presidio): chunk oversized text before /analyze so large content blocks do not fail
The Presidio PII guardrail sent each content block to the analyzer as a
single /analyze call with no size check. Analyzer deployments commonly cap
the request body (the reporting deployment rejects bodies over 1,000,000
bytes with HTTP 413), so large blocks failed closed, and analyzer latency
grew linearly with payload size.
analyze_text now splits texts larger than presidio_analyze_chunk_size_bytes
(default 500,000 UTF-8 bytes, configurable per guardrail) into overlapping
chunks, analyzes them concurrently, remaps each detection's start/end onto
the original text, and deduplicates detections from the overlap regions.
Anonymization, blocked-entity checks, score filtering, numbered-token
unmasking, telemetry, and the dashboard entity positions all consume the
remapped global offsets unchanged.
Resolves LIT-4785
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(presidio): review-round hardening for chunked analyze
- measure the chunk budget on the JSON-serialized text (non-ASCII escapes
expand beyond raw UTF-8, so a raw-byte budget could still exceed the
analyzer body limit)
- share the chunk fan-out semaphore per event loop and instance instead of
per call, so many oversized blocks cannot multiply concurrent analyzer
calls
- apply configured score thresholds and deny list per chunk BEFORE overlap
resolution, so a below-threshold span cannot displace a detection the
thresholds keep
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The /v1/messages bridge decided a Claude target could take `reasoning_effort` from
the model name, which says nothing about the params the provider in front of it
accepts. Snowflake serves Claude over the Anthropic dialect and declares `thinking`
alone, so `get_optional_params` raised `UnsupportedParamsError` before the request
reached the wire: every adaptive request carrying an effort tier turned a 200 into
a 400 for all seven of its Claude entries.
The tier is now offered only where the target declares the param, reading the same
`get_supported_openai_params` the sibling `_supports_prompt_cache_key` reads twelve
lines up. A target declaring neither carrier keeps its bare `thinking` block, which
is what this bridge sent before it carried a tier at all.
Without a resolved provider the tier stays behind rather than being offered blind.
Resolving one from the model's prefix instead would run an OAuth device flow for
github_copilot and chatgpt, blocking for minutes, and one of the two callers in that
position is a logging callback. The copilot case is pinned by a test.
The bundled-path check was an unanchored substring match, so a
user-supplied logo URL that happened to carry /assets/logos/ or
/_next/static/media/ in its path, and whose filename collided with one of
the 36 manifest entries, would pick up a dark-mode treatment meant only
for assets we ship.
assetPaths already draws this line for resolveLogoSrc, which returns an
external src untouched. Export that predicate instead of writing a second
one, and require a treated src to clear it.
The existing test only covered a remote URL with a bare filename, which
passed either way. The new ones fail without the guard.
* feat(ui): session-level cache observability in request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: guard cache_hit filter against non-string defaults in direct calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): drop redundant cache_hit field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A complexity-router setting placed beside complexity_router_config, or inside a
tier entry's litellm_params, is read by nobody: the router loads its settings only
from litellm_params.complexity_router_config. It does not stay inert. The
alias-marker forwarding and the per-tier param spread carry every unrecognized key
onto the outbound request, and all_litellm_params only knows the outer names, so
the key reaches the provider as an unknown body field and every call through that
model group fails with an error naming an internal config key.
Guard the whole set, derived from ComplexityRouterConfig.model_fields so a field
added later is covered, and scoped to complexity-router deployments because the
names only mean this there (embedding_model is a legitimate flat param on an
s3_vectors vector store). Scope is read from the same merged field view the naming
check is judged on, so a router named only by its default model is in scope and a
field added to the required-field table is covered without another edit. The write
endpoints reject with a 400 naming the keys and where they belong, config.yaml
refuses to start for the same reason max_agentic_loops does, and a tier entry is
judged by the config model itself.
An already-stored deployment keeps loading, so an upgrade cannot take a running
gateway down over a row that was written before the gate existed.
Two lazily loaded models changed without their generated artifacts being
regenerated, so check-ui-api-types has been red on every branch off staging.
The snapshot that /openapi.json serves for unloaded features was missing
ChatCompletionToolReferenceObject, and the dashboard types were missing
aws_external_id. The snapshot step runs first and short-circuits, so only the
first one was visible until it was fixed.
Both files are regenerated with `python -m litellm.proxy._lazy_openapi_snapshot`
and `npm run gen:api`, no hand edits.
* feat(alerting): add native Microsoft Teams alerting destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): preserve active destinations on MS Teams save and confirm health test delivery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): read persisted alerting destinations at MS Teams save time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The dashboard's dark theme left a chunk of the bundled provider logos
unreadable: 25 of them are pure black marks on a transparent background,
so on a near-black surface they disappeared entirely, and another 11 are
dark multicolor marks drawn for a white page.
This adds the seam the rest of the work hangs off: a per-asset treatment
manifest in logoTreatments.ts, and a Logo component that applies the
treatment it names. Two treatments exist today. "invert" flattens a mark
to solid white with brightness(0) invert(1), which is what the vendor's
own white mark looks like for a pure-black transparent glyph. "plate"
puts a white surface behind the mark so it reads exactly as it does on a
light page.
Both are dark-only, and only assets named in the manifest are touched, so
light mode is unchanged and the other 96 bundled logos keep rendering
byte for byte as they do today. The className an untreated logo receives
is passed through verbatim rather than routed through cn(), so even the
class string is unchanged.
The split between invert and plate was measured per asset, not guessed:
luminance, saturation and alpha coverage sampled off a canvas render. Two
assets that look monochrome, aiml_api and repelloai, carry a light
knockout inside dark artwork, so inversion would flatten the knockout
into the mark and erase it. They get a plate instead, and a test pins
that.
Six assets whose artwork is an opaque dark box (aim_logo, aim_security,
deepgram, jina, lakeraai, openmeter) are deliberately left untreated. A
plate cannot show through an opaque image, so the only honest fix for
them is a replacement asset.
Two loading/empty transitions the disabled empty state got wrong.
A selection made in a range that had options survives a move to a range that
has none, and it still scopes the data below, so disabling the combobox
outright took away the only control that could clear it. Disable it only when
there is nothing selected to clear.
The customer list defaulted to an empty array while its query was in flight,
so the filter announced a range with no customers before anything had been
read. Leave it undefined until the query resolves, as the tag list now does.
/v1/messages forwarded `thinking` verbatim for a Claude-family model and then returned,
carrying `output_config.effort` only when the model string started with a Bedrock prefix.
Every other bridged provider got a bare adaptive thinking block, so the caller's effort did
nothing: max and minimal produced byte-identical upstream bodies.
Send those targets the tier as `reasoning_effort`, which is the param they take. Bedrock keeps
taking `output_config`, since the two are not interchangeable there: an application inference
profile ARN resolves to no chat config, so `reasoning_effort` is dropped and the tier vanishes,
and a provider that rebuilds `output_config` from it overwrites a caller-set `thinking.display`
on the way. The tier stays a plain string, the summary already travelling inside the forwarded
`thinking` block. Adaptive with no tier, and budgeted thinking, both stay exactly as they were.
