* feat(ui): edit the auto-router tier set with custom classifier-defined tiers
The editor over the model layer beneath it. An Edit tiers button turns the tier
list into an editor: a tier takes a name, a classifier definition, and models,
between two and eight rows. Restore defaults resets to the built-in four rather
than stacking them on top. Keyword rules follow a rename, an orphaned rule
blocks the save, and both forms dry-run the exact payload against the backend
validator before writing.
The edit modal hydrates a stored custom set into rows, and an untouched
open-and-save round-trips byte-identically, per-model reasoning efforts
included. A form that never opens the editor submits the same bytes as before.
The cost-optimization tier chart renders arbitrary tier names: the guard that
returned no models for a non-built-in name is gone, and the fixed four-color
array gives way to the shared cycle.
* refactor(ui): extract tier editor sections to clear new lint warnings
* test(ui): drop narration comments per repo convention
* fix(ui): default editingTiers so the build's type check passes
* fix(ui): restore the mid-dry-run submit guard and its regression tests
Every test in the file seeds, reads and deletes the same fixed group and team
ids, and auth_ui_unit_tests runs pytest with -n 2. Two tests landing on the two
workers at once tread on each other: one worker's _clean_db DELETE wipes rows
the other just seeded, and its sync writes land in the other's read.
Both shapes showed up on 13a0976bb6, a commit that renames a passthrough test
and nothing else. test_reconcile_is_idempotent... read back an empty table, and
test_reconcile_handles_a_null_array_column read the idempotent test's team on
its own second group.
Scoping the ids to PYTEST_XDIST_WORKER keeps each worker in its own rows. Tests
on one worker still run in sequence, so no isolation is lost.
Reproduced against a local Postgres: -n 2 failed 6 out of 6 runs before, passed
6 out of 6 after, and serial runs are green either way. Stripping the COALESCE
guard from the mirror's SQL still fails the suite, so the ids are all that
changed.
Greptile flagged this test as coupled to OpenAI's availability. The coupling was
not the status assertion it pointed at, and it predates this PR: the test it
replaced called /v1/assistants live the same way, and pass_through_endpoints gates
success logging on `response.status_code < 400`, so an upstream outage has always
meant no log fires and the payload assertions fail regardless.
The target is now a local HTTP server on an ephemeral port, so the test is offline
either way. It still exercises the generic passthrough handler, since
_is_supported_openai_endpoint does not claim a 127.0.0.1 URL any more than it
claimed /v1/moderations, and it now also asserts what the upstream actually
received rather than only what came back.
respx was the obvious approach and does not work here: it patches httpx transports,
and the passthrough issues its request through the custom aiohttp transport, so the
call went to the real api.openai.com and returned 401 while respx sat unused.
Mutation checked: gating off the success enqueue fails the test, and tampering with
the logged response body fails it.
`_count_content_list` accepted text, image_url, tool_use, tool_result,
thinking and tool_reference, and raised on anything else, so an
Anthropic-native `{"type": "image", "source": {...}}` block aborted the
whole count. That is the documented Anthropic image format and exactly
what /v1/messages receives.
Three user-visible effects. /v1/messages/count_tokens and
/utils/token_counter return 500, and the router's context-window
pre-call check swallows the ValueError and returns every deployment
unfiltered, so an oversized prompt carrying an image is dispatched to
the provider instead of being rejected locally with a 400.
Prices the block through the existing image path: a base64 source
becomes a data URI, a url source passes through, and a file source
falls back to the default image token count. Blocks nested inside
tool_result.content are covered too, because _count_anthropic_content
recurses back into _count_content_list.
Fixes#36604
batch_completion collects per-request failures into its result list rather than
raising them; its own source says "return exceptions if any". So the test's
`except Timeout` and `except litellm.InternalServerError` arms could never fire for
the case they were written for. An upstream 500 instead reached
`response.choices`, raised AttributeError on the exception object, and fell through
to the bare `except Exception` that calls pytest.fail. That is what CircleCI hit.
The tolerance now reads the returned values, which is where the failures actually
are. The same two exception types are tolerated as before, nothing broader.
Checked against four injected outcomes: three InternalServerErrors pass, three
Timeouts pass, an AuthenticationError fails, and a response whose content is None
fails. So it is not tolerating its way to a vacuous green.
* feat(spend): report prompt caching savings as total and gateway-attributed
`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.
Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.
Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.
The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.
Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.
* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)
The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.
Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
Both anchoring tests read the trigger's box before the click and the popup's box the
instant it turns visible. Base UI places the popup asynchronously and opening it can
shift the trigger, so both boxes could be sampled before the layout settled. The run
on 1eedaa3a43 missed by 4.2px (expected >= 446.015, got 441.799) on a tree with no UI
changes at all, having passed on 21092d633b, which differs only in a deleted python
test and a budget json.
Each assertion now re-reads both boxes under expect.poll. The conditions themselves
are unchanged: the popup must sit at or below the trigger's bottom edge in the first
test and must not overlap it in the second. Polling cannot mask a genuinely misplaced
popup, since one that never lands correctly still fails when the poll times out.
Azure now answers the create call with 410 and code assistants_api_deprecated:
"The Assistants API has been retired. Follow the migration guide to update your
workloads." Microsoft retired it on the same day OpenAI retired theirs, which is
why this landed with the OpenAI ones rather than before them.
This job runs pytest with -x, so the test was also hiding everything after it in
tests/pass_through_tests.
test_pass_through_file_operations stays. It only asks /v1/files for
purpose="assistants", which still answers 200 on both upload and delete.
test_opik_logging_http_request asserted "nothing has been POSTed yet" roughly one
second into a window governed by OpikLogger's 5-second periodic flush. On a loaded
CI worker the five preceding acompletion calls eat that budget, the periodic flush
fires, and the assertion flips. Reproduced with no product changes at all: letting
5.5 seconds pass before the assertion drains the queue and sets mock_post.called,
which is exactly the failure CircleCI reports.
The test now pins flush_interval past anything the test can reach, so the two
batching assertions measure batching instead of wall clock, and drives the flush
path explicitly at the end rather than sleeping the interval. That last phase used
to be near-vacuous, since the size-triggered flush had already emptied the queue.
Assertions now match only calls to Opik's own /traces/batch and /spans/batch.
get_async_httpx_client caches one client per special provider, so the mock is
process-wide and any other logging callback's POST would otherwise count.
Dropped the teardown that closed that shared client, which broke every later test
in the same worker that logs through it, and the try/except that turned assertion
failures into a pytest.fail with no traceback.
Mutation checked: flushing on every event and never flushing on size both fail the
test.
Two unrelated causes, both leaving staging red with tests that no longer describe
anything true.
#38264 gave LangFuseLogger a langfuse_environment argument and started carrying it
in the credentials dict. The handler test's fake logger did not accept the new
keyword, so constructing it raised TypeError, and four cases in
test_langfuse_unit_tests rebuilt the cache key by hand from three fields and missed
on the four-field key production now writes. Caching itself was never broken: the
handler sets and gets with the same dict. The fake now takes the argument and
asserts it is forwarded, and the cache assertion issues a second identical request
and expects the same logger back, which is the behaviour that matters and cannot
rot the next time a credential field is added.
OpenAI has retired the Assistants API. /v1/assistants and /v1/threads both answer
404 with a valid key, where every live route answers 401, so nothing calling them
can pass again. test_custom_logger_passthrough covered generic passthrough logging
and only used assistants because it is a route with no provider-specific handler;
it moves to /v1/moderations, which is still unclaimed by
_is_supported_openai_endpoint, so the same generic branch is exercised. The two
tests there asserted the same thing against different dead routes, so they collapse
into one. The Ruby suite existed solely to drive assistants, threads, messages and
runs, so it goes along with the RVM and bundler steps that were installed only to
run it, and the two dead OpenAI assistants cases leave
test_openai_assistants_passthrough.
The Azure assistants case in that file stays. Azure runs its own lifecycle and I
could not reach the CI deployment to check whether that API is still there.
Resolve tests/llm_translation/test_together_ai.py in favor of staging:
bcb6a0a998 already landed the fail-open assertion for models missing from
the registry, so both models now list response_format and tools. This
branch's narrower gating of response_format no longer matches behavior.
A coding agent names each conversation by quoting the whole session and asking
for a title. The classifier rated the quoted session rather than the request, so
the cheapest call the client makes routed to the most expensive tier: 11 of 17
title generations in one day of real traffic came back COMPLEX.
Recognize those prompts by literal sentinel on the newest ask and route them to
the cheapest configured tier without classifying them, so the call costs nothing
to route. The placement is scoped to the one request that carries the sentinel:
it never displaces an operator's classifier plugin, the bandit cannot reach above
the tier as raised, it never becomes the session pin, and the sentinel that
matched is recorded on the routing decision. Detection reads the newest ask
alone, so a title request quoted into a later turn cannot cheapen the work that
follows it, and a keyword rule, an escalation keyword or the plan-mode floor all
still decide over it.
Regenerates the lazy OpenAPI snapshot, which was already stale on the base for an
unrelated Presidio guardrail field and failed the schema check on every PR.
Resolves LIT-6349
* fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning
When a model has only per-level flags (e.g. supports_minimal_reasoning_effort: true)
but no explicit supports_reasoning flag, treat it as implicitly reasoning-capable.
This fixes gpt-5-search-api which declares minimal support but was incorrectly
degraded to low/minimal floor due to missing explicit supports_reasoning flag.
Test: verify per-level flag enables resolution path even without supports_reasoning.
Note: This change indirectly causes 20 azure deployments to forward max/xhigh
instead of degrading to high when requested, as these models now correctly
resolve their supported efforts through declared capability flags. This is
intended behavior (avoiding unnecessary degradation) but silent; operators
seeing increased latency/cost should check reasoning effort changes in logs.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(anthropic): explicit supports_reasoning=False wins over per-level flags
Greptile P1: the implicit-True branch bypassed the operator's explicit
supports_reasoning: false escape hatch when per-level flags were present
or inherited through the bare-twin lookup. Return () first on explicit
False, then apply the per-level implication only when the flag is unset.
Also drops a test comment that restated the test name (P2).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(anthropic): drop and self-heal empty thinking blocks on /v1/messages
* test(anthropic): pin early-signature carry across the blank thinking chunk skip
#38318 taught exception_type to map upstream status codes for providers with
no branch of their own. It reads the status code off the exception, but
_handle_error stamps 500 onto every failure that never carried one, so a
refused connection reached the mapper wearing a status code nothing upstream
had sent, and came back as InternalServerError instead of APIConnectionError.
The two are not interchangeable to a caller: a 5xx says the provider answered
and failed, which the router treats as a reason to cool the deployment down,
while a connection error says the request never landed.
BaseLLMException now records whether its status code was received or
synthesized, _handle_error sets that when it invents the 500, and the status
mapper declines to act on a code litellm made up, so those failures fall
through to the APIConnectionError the branch was always meant to produce.
Genuine upstream 5xx responses are untouched, which the second test pins.
The search transformation assertion #38318 had loosened to InternalServerError
goes back to APIConnectionError for the same reason.
The Blog trigger carried `bg-transparent!`, which emits an important
background-color and so beat the non-important `hover:bg-accent` the shared
product-link class supplies. Docs lit up on hover and Blog stayed flat, the
same Docs/Blog inconsistency this branch is about on a different axis.
Dropping the override lets the shared hover through. `border-0!` stays, since
it keeps the trigger's box identical to the plain Docs anchor. Verified in the
browser: both now paint lab(96.1596 -0.0823438 -1.13575) on hover at 36px tall.
Docs went from a ghost Button to a plain anchor, which dropped the focus
treatment the Button was supplying, so tabbing to Docs showed nothing while
tabbing to Blog showed a ring. The ring now lives on the shared class both
sides use, matching the Button primitive's values.
Kept Docs as a real anchor rather than routing it back through Button:
nativeButton={false} stamps role="button" onto the element, so the old
DashboardHeader markup announced Docs as a button and lost its link
semantics. Tests pin both the ring and the link role.
Project auth expands all-proxy-models, * patterns, and access-group names
to many concrete models, but the rate limiter looks quotas up by the exact
requested model name, so a quota keyed on one of those entries is never
applied. Fail loudly with a 400 instead of storing an unenforceable quota
* fix(moonshot, together_ai): send the reasoning effort Kimi K3 accepts
Moonshot documents reasoning_effort as a top-level chat completions field for its reasoning
models, and defaults it to max, but MoonshotChatConfig builds its supported params by
subtracting from the OpenAI base list, which never carried that param. An explicit level
raised UnsupportedParamsError before the request left the proxy, so low and high were
unreachable and every call ran at the provider default
Together accepts low, high and max on Kimi K3. The per-model clamp added for the gpt-oss
family folds max down to high for every model except deepseek-ai/DeepSeek-V4-Pro, so a caller
asking for max silently got roughly half the reasoning budget they paid for
Moonshot now offers reasoning_effort whenever the registry says the model reasons. Together
sends a level the map entry declares unchanged, and keeps its existing table for every level
an entry does not name, so the only value that moves is Kimi K3 at max
* fix(moonshot): unwrap the bridges' effort object to the level string
* feat(ui): the model and wire layer for operator-defined auto-router tier sets
The data half of the custom tier set editor, with no visible UI change: the
editor lands separately on top of it.
One reader, activeTierRows, mints built-in rows with the canonical tier key as
their id, so the fallback pointer, the plan-mode floor and the per-model params
are row ids in both modes and nothing downstream branches on the mode. One
restrictions table carries each forbidden setting beside the reason shown for
it, so the greyed control and the omitted payload key cannot disagree. The
tier-set writes live in applyTierSetAction, where the fallback re-point and the
floor turn-off happen in one commit, unit-tested without a render.
buildComplexityRouterConfig emits tiers, tier_definitions and fallback_tier from
the rows, forces the LLM classifier, and strips what the backend rejects beside
tier_definitions. A payload built without a custom tier set is byte-identical to
what the form sends today.
* fix(ui): resolve frontend-lint failures on the tier-set model layer
* test(ui): drop a redundant explanatory comment per repo convention
* fix(ui): keyword rules follow their tier row through every tier-set action
* fix(anthropic): resolve /v1/messages effort tiers through the capability owner
The bridge normalizer read three supports_*_reasoning_effort booleans of its own, so it
answered "which levels does this deployment take" independently of the resolver behind
/model_group/info. The two disagreed: a proxy advertising kimi-k3 max forwarded high.
Degrade against resolve_supported_reasoning_efforts instead, with the chains as a declared
table. When no step of a chain is accepted, the fallback is read off that same resolved set
rather than assumed, since an entry naming its levels outright can exclude the tiers the
per-level flags treat as unconditional. none is never chosen as that fallback, being an off
switch rather than a tier, and a deployment accepting no tier at all keeps the floor every
deployment degraded to before.
* test(anthropic): pin the normalized effort at the /v1/messages request boundary
The existing coverage stopped at normalize_reasoning_effort_value, so nothing failed if the
handler dropped or overwrote the normalized tier on its way into completion_kwargs. Drive
_prepare_completion_kwargs instead and assert on the kwargs handed to acompletion, in both the
string and the dict effort shapes, including the provider-prefixed model name the handler is
actually called with.
Against the pre-fix normalizer the fallback case fails, and against the baseline before a map
entry could declare its levels 7 of the 12 fail, so the boundary is pinned rather than restated.
A shadow eval whose judge_model is one of the router's tier models, the router's
default model, or a reverse job's baseline_model was accepted with no warning. An
LLM judge scores its own output higher than a rival's, so that tier's win rate
measures the judge instead of the models, and the job's whole budget buys a result
that has to be thrown away.
start_shadow_eval now rejects it with a 400 naming the colliding arm.
`judge_target` is the single answer to "where does a call to this name go for this
caller, and what answers it", and the resolvability gate, the collision gate and
the judge dispatch all read it. It has three outcomes and no others: the router
serves the name, the SDK serves it, or nothing does. Splitting that question is
what every bug here came from, so `router_resolves_model` and `answering_models`
are gone rather than joined by a third.
Two spellings of one model are one identity. A name is compared by what would
answer it, resolved through every channel `get_model_list` composes and then put
in the provider-qualified form litellm itself uses, so a judge given as `gpt-4o`
collides with a tier deployment serving `openai/gpt-4o`, and a judge given as
`openai/gpt-4o` collides with a deployment configured as bare `gpt-4o`. Both ends
are normalised because an admin writes them at different times.
Answering is also per-caller. The shadow and judge calls carry the shadowed key's
`user_api_key_team_id`, which is what the router selects deployments with, so the
endpoint derives the job's teams once from the keys it already looks up and every
check runs under them, and the judge dispatch picks its arm under the same team.
A team's public model name resolves to nothing for everyone else and a team's own
deployment resolves for nobody else, so a check that omits the team answers for a
caller who does not exist. A collision under any one team fails the job, because
every key's verdicts land in the same win rates.
Three sites were separately re-deriving "the provider models this name resolves
to", with unexplained divergence in whether they fell back to the literal name.
`Router.resolved_litellm_models` is now the one owner; the routing-plugin
candidate list and the stream-options check both delegate to it, and
`_deployment_litellm_model` is gone.
The router's arms come from `strategy_router_dependencies`, the same enumeration
the health check reads. Only the roles that serve are arms: a classifier or
embedding model picks the tier and never produces a response anyone judges. A
semantic auto-router keeps its routes in an opaque config blob, so only its
default model is enumerable and the guard is incomplete there by design, able to
miss a collision but never to invent one
The two regenerated artifacts carry `presidio_analyze_chunk_size_bytes` from
alters the spec; the sync gate runs on any PR touching litellm/proxy, so this one
has to carry the base's drift to go green
The endpoint built messages=[{"role": "user", "content": prompt}], so a dry run
could not carry prior turns, the caller's system prompt, or the tool definitions
a request advertises. A real agentic turn reduced to its last sentence classified
as trivial, which is why a config sweep reported savings for every configuration.
Accept messages, system and tools, and forward them to the same pre-routing hook
untranslated, with the raw-body snapshot built by the serving path's own owner,
refresh_proxy_server_request_body_snapshot. Loose types are deliberate: the hook
reads whatever dialect the surface produced, so validating against one surface's
schema would reject the others.
prompt stays as the single-ask shorthand, normalized into one user turn inside the
request model so the handler carries no mode branch.
The top bar's theme control needed a click on the sun/moon, then a menu, then
a choice, to do something every other product does in one click. It is now a
plain button that flips between light and dark, with the beta marker moved into
the label of the click that turns dark on. An explicit "system" choice is gone,
but next-themes still follows the OS for anyone who has it stored and has not
clicked yet.
Docs and Blog also drifted apart in the gateway header: Blog rendered through
the shared product-link class while Docs was a muted ghost button one size
down, so Docs read as dimmer and sat 4px shorter. Both now go through a shared
DocsLink component, which is also what the legacy navbar uses, so the pair
cannot drift again.
* fix(langfuse): warn and drop invalid LANGFUSE_TRACING_ENVIRONMENT instead of failing requests
* fix(langfuse): treat a dynamic environment equal to the raw deployment value as redundant
