subprocess.run leaves stderr as bytes on TimeoutExpired even under
text=True, unlike CalledProcessError. Classifying both in one handler
meant a real `prisma db push` timeout died on a TypeError, which
proxy_cli.py's `except RuntimeError` does not catch, so the migrations
Job container ended on an unhandled traceback instead of a clean exit.
Give the timeout its own handler and retry it, matching what the migrate
deploy loop beside it already does. That puts a fallthrough back into the
loop, so the trailing raise removed in the previous commit is reachable
again and comes back with it.
Also drop a comment restating why the resolver cases exist and widen the
db push test's docstring, which had stopped describing what it covers.
The retry loop already raises on the final attempt, so the raise that
followed the loop could never run. Drop it and cover the exhaustion path
with a test that pins the attempt count and keeps the prisma error in the
message, which is the only thing that tells an operator why the boot
stopped.
The moved fail-fast test carried a sys.path.insert that the uv workspace
makes unnecessary, and one pre-existing case asserted nothing beyond "did
not raise", so it could not tell a swallowed error from a skipped query.
Give it a liveness gate on the connect count instead.
Fold the resolver default/opt-out matrix into the existing db-push flag
test rather than standing up another patched test, so the flag pair, the
env var, and their precedence are covered without new mock scaffolding.
Moving the resolver tests under tests/ brings them under ruff-tests.toml,
which the package-internal directory they came from was never linted by, so a
pre-existing pytest.raises pattern now needs to be a raw string (RUF043).
Also corrects the comment on proxy_cli's RuntimeError handler: both resolvers
raise on permission failures, not just v2.
`prisma db push` under v2 raised on the first failure while v1 retried it four
times, so making v2 the default silently cost --use_prisma_db_push its
retries. It now uses the same transient classification as migrate deploy.
The classifier moves onto ProxyExtrasDBManager next to _is_permission_error
and _is_idempotent_error, which do the same kind of stderr matching.
Replaces a test that claimed to pin the transient classification but fed it a
P3009 stderr, which an earlier branch catches, so it passed even when the
classifier was mutated to treat everything as transient. The replacement uses
an unclassified error and fails on that mutant. Drops a v1 test that duplicated
test_v1_default_still_calls_resolve_all_migrations.
The v2 resolver skips the diff-and-force recovery that caused schema
thrashing when two LiteLLM versions contend for one database during a
rolling deploy. The standalone migration Job already defaulted to v2; this
aligns the proxy-server path.
v1 stays reachable two ways: --use_legacy_migration_resolver on the CLI, and
USE_V2_MIGRATION_RESOLVER=false for containerised deploys, where
prisma_migration.py calls run_server with a fixed argv and the env var is the
only route in. --use_v2_migration_resolver still parses, so existing commands
do not die on an unknown option.
Because v2 fails fast where v1 retried every failed deploy, a database that is
not accepting connections yet, or another instance holding the migration
advisory lock, would now kill a boot that used to ride it out. Those two
failures are retried, with Prisma's stderr logged each round, and still raise
once the attempts are spent.
Moves the resolver tests from litellm-proxy-extras/tests, which no CI job
runs, into tests/litellm-proxy-extras, and repoints the dedicated Postgres
CircleCI job at the legacy path so v1 keeps real-DB and proxy-boot coverage.
Removes restating comments added with the Teams alerting destination and the
lazy OpenAPI snapshot refactor, types three signatures that shipped untyped or
with bare dict, and ratchets the strict and basedpyright budgets down.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
batch_completion collects per-request failures into its result list rather than
raising them; its own source says "return exceptions if any". So the test's
`except Timeout` and `except litellm.InternalServerError` arms could never fire for
the case they were written for. An upstream 500 instead reached
`response.choices`, raised AttributeError on the exception object, and fell through
to the bare `except Exception` that calls pytest.fail. That is what CircleCI hit.
The tolerance now reads the returned values, which is where the failures actually
are. The same two exception types are tolerated as before, nothing broader.
Checked against four injected outcomes: three InternalServerErrors pass, three
Timeouts pass, an AuthenticationError fails, and a response whose content is None
fails. So it is not tolerating its way to a vacuous green.
* feat(spend): report prompt caching savings as total and gateway-attributed
`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.
Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.
Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.
The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.
Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.
* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)
The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.
Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
Both anchoring tests read the trigger's box before the click and the popup's box the
instant it turns visible. Base UI places the popup asynchronously and opening it can
shift the trigger, so both boxes could be sampled before the layout settled. The run
on 1eedaa3a43 missed by 4.2px (expected >= 446.015, got 441.799) on a tree with no UI
changes at all, having passed on 21092d633b, which differs only in a deleted python
test and a budget json.
Each assertion now re-reads both boxes under expect.poll. The conditions themselves
are unchanged: the popup must sit at or below the trigger's bottom edge in the first
test and must not overlap it in the second. Polling cannot mask a genuinely misplaced
popup, since one that never lands correctly still fails when the poll times out.
/v1/batches now rejects a missing input_file_id with a 400 through
raise_if_required_body_param_missing, so the contract negative that was
skipped for "500s instead of 400" passes as written. Verified against a
live proxy.
The three Datadog MCP tests were skipped because each one sent a
`telemetry` argument that search_datadog_logs rejects with "unexpected
additional properties". That argument was never a documented Datadog
parameter and no assertion reads it, so it is dropped and the tests run
again unchanged otherwise.
Azure now answers the create call with 410 and code assistants_api_deprecated:
"The Assistants API has been retired. Follow the migration guide to update your
workloads." Microsoft retired it on the same day OpenAI retired theirs, which is
why this landed with the OpenAI ones rather than before them.
This job runs pytest with -x, so the test was also hiding everything after it in
tests/pass_through_tests.
test_pass_through_file_operations stays. It only asks /v1/files for
purpose="assistants", which still answers 200 on both upload and delete.
test_opik_logging_http_request asserted "nothing has been POSTed yet" roughly one
second into a window governed by OpikLogger's 5-second periodic flush. On a loaded
CI worker the five preceding acompletion calls eat that budget, the periodic flush
fires, and the assertion flips. Reproduced with no product changes at all: letting
5.5 seconds pass before the assertion drains the queue and sets mock_post.called,
which is exactly the failure CircleCI reports.
The test now pins flush_interval past anything the test can reach, so the two
batching assertions measure batching instead of wall clock, and drives the flush
path explicitly at the end rather than sleeping the interval. That last phase used
to be near-vacuous, since the size-triggered flush had already emptied the queue.
Assertions now match only calls to Opik's own /traces/batch and /spans/batch.
get_async_httpx_client caches one client per special provider, so the mock is
process-wide and any other logging callback's POST would otherwise count.
Dropped the teardown that closed that shared client, which broke every later test
in the same worker that logs through it, and the try/except that turned assertion
failures into a pytest.fail with no traceback.
Mutation checked: flushing on every event and never flushing on size both fail the
test.
Two unrelated causes, both leaving staging red with tests that no longer describe
anything true.
#38264 gave LangFuseLogger a langfuse_environment argument and started carrying it
in the credentials dict. The handler test's fake logger did not accept the new
keyword, so constructing it raised TypeError, and four cases in
test_langfuse_unit_tests rebuilt the cache key by hand from three fields and missed
on the four-field key production now writes. Caching itself was never broken: the
handler sets and gets with the same dict. The fake now takes the argument and
asserts it is forwarded, and the cache assertion issues a second identical request
and expects the same logger back, which is the behaviour that matters and cannot
rot the next time a credential field is added.
OpenAI has retired the Assistants API. /v1/assistants and /v1/threads both answer
404 with a valid key, where every live route answers 401, so nothing calling them
can pass again. test_custom_logger_passthrough covered generic passthrough logging
and only used assistants because it is a route with no provider-specific handler;
it moves to /v1/moderations, which is still unclaimed by
_is_supported_openai_endpoint, so the same generic branch is exercised. The two
tests there asserted the same thing against different dead routes, so they collapse
into one. The Ruby suite existed solely to drive assistants, threads, messages and
runs, so it goes along with the RVM and bundler steps that were installed only to
run it, and the two dead OpenAI assistants cases leave
test_openai_assistants_passthrough.
The Azure assistants case in that file stays. Azure runs its own lifecycle and I
could not reach the CI deployment to check whether that API is still there.
Resolve tests/llm_translation/test_together_ai.py in favor of staging:
bcb6a0a998 already landed the fail-open assertion for models missing from
the registry, so both models now list response_format and tools. This
branch's narrower gating of response_format no longer matches behavior.
A coding agent names each conversation by quoting the whole session and asking
for a title. The classifier rated the quoted session rather than the request, so
the cheapest call the client makes routed to the most expensive tier: 11 of 17
title generations in one day of real traffic came back COMPLEX.
Recognize those prompts by literal sentinel on the newest ask and route them to
the cheapest configured tier without classifying them, so the call costs nothing
to route. The placement is scoped to the one request that carries the sentinel:
it never displaces an operator's classifier plugin, the bandit cannot reach above
the tier as raised, it never becomes the session pin, and the sentinel that
matched is recorded on the routing decision. Detection reads the newest ask
alone, so a title request quoted into a later turn cannot cheapen the work that
follows it, and a keyword rule, an escalation keyword or the plan-mode floor all
still decide over it.
Regenerates the lazy OpenAPI snapshot, which was already stale on the base for an
unrelated Presidio guardrail field and failed the schema check on every PR.
Resolves LIT-6349
* fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning
When a model has only per-level flags (e.g. supports_minimal_reasoning_effort: true)
but no explicit supports_reasoning flag, treat it as implicitly reasoning-capable.
This fixes gpt-5-search-api which declares minimal support but was incorrectly
degraded to low/minimal floor due to missing explicit supports_reasoning flag.
Test: verify per-level flag enables resolution path even without supports_reasoning.
Note: This change indirectly causes 20 azure deployments to forward max/xhigh
instead of degrading to high when requested, as these models now correctly
resolve their supported efforts through declared capability flags. This is
intended behavior (avoiding unnecessary degradation) but silent; operators
seeing increased latency/cost should check reasoning effort changes in logs.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(anthropic): explicit supports_reasoning=False wins over per-level flags
Greptile P1: the implicit-True branch bypassed the operator's explicit
supports_reasoning: false escape hatch when per-level flags were present
or inherited through the bare-twin lookup. Return () first on explicit
False, then apply the per-level implication only when the flag is unset.
Also drops a test comment that restated the test name (P2).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(anthropic): drop and self-heal empty thinking blocks on /v1/messages
* test(anthropic): pin early-signature carry across the blank thinking chunk skip
Model chips on the team info and virtual key info pages (overview and settings tabs) now link to /models-and-endpoints?model_group=<name>, which the All Models tab reads through a new nuqs-backed model group filter. Grant sentinels such as all-proxy-models stay plain badges. The selected group is also sent as the server-side search so the matching deployments are fetched even when they are beyond the first page.
#38318 taught exception_type to map upstream status codes for providers with
no branch of their own. It reads the status code off the exception, but
_handle_error stamps 500 onto every failure that never carried one, so a
refused connection reached the mapper wearing a status code nothing upstream
had sent, and came back as InternalServerError instead of APIConnectionError.
The two are not interchangeable to a caller: a 5xx says the provider answered
and failed, which the router treats as a reason to cool the deployment down,
while a connection error says the request never landed.
BaseLLMException now records whether its status code was received or
synthesized, _handle_error sets that when it invents the 500, and the status
mapper declines to act on a code litellm made up, so those failures fall
through to the APIConnectionError the branch was always meant to produce.
Genuine upstream 5xx responses are untouched, which the second test pins.
The search transformation assertion #38318 had loosened to InternalServerError
goes back to APIConnectionError for the same reason.
The Blog trigger carried `bg-transparent!`, which emits an important
background-color and so beat the non-important `hover:bg-accent` the shared
product-link class supplies. Docs lit up on hover and Blog stayed flat, the
same Docs/Blog inconsistency this branch is about on a different axis.
Dropping the override lets the shared hover through. `border-0!` stays, since
it keeps the trigger's box identical to the plain Docs anchor. Verified in the
browser: both now paint lab(96.1596 -0.0823438 -1.13575) on hover at 36px tall.
Docs went from a ghost Button to a plain anchor, which dropped the focus
treatment the Button was supplying, so tabbing to Docs showed nothing while
tabbing to Blog showed a ring. The ring now lives on the shared class both
sides use, matching the Button primitive's values.
Kept Docs as a real anchor rather than routing it back through Button:
nativeButton={false} stamps role="button" onto the element, so the old
DashboardHeader markup announced Docs as a button and lost its link
semantics. Tests pin both the ring and the link role.
Project auth expands all-proxy-models, * patterns, and access-group names
to many concrete models, but the rate limiter looks quotas up by the exact
requested model name, so a quota keyed on one of those entries is never
applied. Fail loudly with a 400 instead of storing an unenforceable quota
* fix(moonshot, together_ai): send the reasoning effort Kimi K3 accepts
Moonshot documents reasoning_effort as a top-level chat completions field for its reasoning
models, and defaults it to max, but MoonshotChatConfig builds its supported params by
subtracting from the OpenAI base list, which never carried that param. An explicit level
raised UnsupportedParamsError before the request left the proxy, so low and high were
unreachable and every call ran at the provider default
Together accepts low, high and max on Kimi K3. The per-model clamp added for the gpt-oss
family folds max down to high for every model except deepseek-ai/DeepSeek-V4-Pro, so a caller
asking for max silently got roughly half the reasoning budget they paid for
Moonshot now offers reasoning_effort whenever the registry says the model reasons. Together
sends a level the map entry declares unchanged, and keeps its existing table for every level
an entry does not name, so the only value that moves is Kimi K3 at max
* fix(moonshot): unwrap the bridges' effort object to the level string
* feat(ui): the model and wire layer for operator-defined auto-router tier sets
The data half of the custom tier set editor, with no visible UI change: the
editor lands separately on top of it.
One reader, activeTierRows, mints built-in rows with the canonical tier key as
their id, so the fallback pointer, the plan-mode floor and the per-model params
are row ids in both modes and nothing downstream branches on the mode. One
restrictions table carries each forbidden setting beside the reason shown for
it, so the greyed control and the omitted payload key cannot disagree. The
tier-set writes live in applyTierSetAction, where the fallback re-point and the
floor turn-off happen in one commit, unit-tested without a render.
buildComplexityRouterConfig emits tiers, tier_definitions and fallback_tier from
the rows, forces the LLM classifier, and strips what the backend rejects beside
tier_definitions. A payload built without a custom tier set is byte-identical to
what the form sends today.
* fix(ui): resolve frontend-lint failures on the tier-set model layer
* test(ui): drop a redundant explanatory comment per repo convention
* fix(ui): keyword rules follow their tier row through every tier-set action
AWS bills a Bedrock GPT-5.5 or GPT-5.4 prompt past 272K tokens under the long-context usage types for the
whole prompt, at 2x input, 2x cache read, and 1.5x output, and the cost map only had the flat rates, so a
300K prompt was logged at half of what the invoice charges. The map's promo rates for gpt-5.6-sol are 20%
under the $5.50 input, $33.00 output, $0.55 cache read, and $6.88 cache write per million the invoice bills.
Adds the *_above_272k_tokens fields to gpt-5.5 and gpt-5.4, moves sol's base and tier rates to the invoiced
ones, replaces the test that pinned the flat behaviour with one that pins the invoiced numbers, and updates
the sol pins in the mantle transformation tests
* fix(anthropic): resolve /v1/messages effort tiers through the capability owner
The bridge normalizer read three supports_*_reasoning_effort booleans of its own, so it
answered "which levels does this deployment take" independently of the resolver behind
/model_group/info. The two disagreed: a proxy advertising kimi-k3 max forwarded high.
Degrade against resolve_supported_reasoning_efforts instead, with the chains as a declared
table. When no step of a chain is accepted, the fallback is read off that same resolved set
rather than assumed, since an entry naming its levels outright can exclude the tiers the
per-level flags treat as unconditional. none is never chosen as that fallback, being an off
switch rather than a tier, and a deployment accepting no tier at all keeps the floor every
deployment degraded to before.
* test(anthropic): pin the normalized effort at the /v1/messages request boundary
The existing coverage stopped at normalize_reasoning_effort_value, so nothing failed if the
handler dropped or overwrote the normalized tier on its way into completion_kwargs. Drive
_prepare_completion_kwargs instead and assert on the kwargs handed to acompletion, in both the
string and the dict effort shapes, including the provider-prefixed model name the handler is
actually called with.
Against the pre-fix normalizer the fallback case fails, and against the baseline before a map
entry could declare its levels 7 of the 12 fail, so the boundary is pinned rather than restated.
A gpt-5 model accepts a non-default temperature only while its effective reasoning
effort resolves to "none". litellm had no representation of the effort a model applies
when the request omits reasoning_effort, so it substituted supports_none_reasoning_effort,
which is a different fact. Every model that supports "none" without defaulting to it
therefore had temperature forwarded and rejected upstream, and because the carve-out
returned before the drop_params branch, drop_params: true could not save it.
Declare the fact instead. A new cost-map key, default_reasoning_effort, states the effort
the provider applies when the request omits one, and one shared predicate resolves the
effective effort from it: an explicit reasoning_effort wins, otherwise the declared
default, otherwise the catalogue decides.
That last step matters because the cost map is fetched from the published branch at import
time, so it can be OLDER than the code reading it. On such a map every model looks
undeclared, and reading that as "reasoning is active" would strip temperature from the 39
gpt-5.1/5.2/5.4 entries that accept it, a regression caused by data lag rather than by
anything about the model. So an absent declaration is only meaningful once the catalogue
carries the key at all; a map that predates the feature keeps the answer litellm gave
before it existed, and the conservative answer applies from the moment the data lands.
The top_p/logprobs/top_logprobs gate carried the same assumption spelled differently and
now shares the predicate, as does the Responses API, which reimplemented the rule and is
what the default /v1/messages bridge routes openai models through. Azure normalises its
routing names in one resolver that every capability lookup goes through, which replaces
its bespoke per-lookup rewrite.
Declared on the 37 gpt-5.1/5.2/5.4 entries measured to accept temperature=0 today, so
their behaviour is unchanged. The 23 gpt-5.5/5.6 entries that reject it stay undeclared
and are fixed once the catalogue carries the key.
Resolves LIT-3797
Resolves LIT-5028
A shadow eval whose judge_model is one of the router's tier models, the router's
default model, or a reverse job's baseline_model was accepted with no warning. An
LLM judge scores its own output higher than a rival's, so that tier's win rate
measures the judge instead of the models, and the job's whole budget buys a result
that has to be thrown away.
start_shadow_eval now rejects it with a 400 naming the colliding arm.
`judge_target` is the single answer to "where does a call to this name go for this
caller, and what answers it", and the resolvability gate, the collision gate and
the judge dispatch all read it. It has three outcomes and no others: the router
serves the name, the SDK serves it, or nothing does. Splitting that question is
what every bug here came from, so `router_resolves_model` and `answering_models`
are gone rather than joined by a third.
Two spellings of one model are one identity. A name is compared by what would
answer it, resolved through every channel `get_model_list` composes and then put
in the provider-qualified form litellm itself uses, so a judge given as `gpt-4o`
collides with a tier deployment serving `openai/gpt-4o`, and a judge given as
`openai/gpt-4o` collides with a deployment configured as bare `gpt-4o`. Both ends
are normalised because an admin writes them at different times.
Answering is also per-caller. The shadow and judge calls carry the shadowed key's
`user_api_key_team_id`, which is what the router selects deployments with, so the
endpoint derives the job's teams once from the keys it already looks up and every
check runs under them, and the judge dispatch picks its arm under the same team.
A team's public model name resolves to nothing for everyone else and a team's own
deployment resolves for nobody else, so a check that omits the team answers for a
caller who does not exist. A collision under any one team fails the job, because
every key's verdicts land in the same win rates.
Three sites were separately re-deriving "the provider models this name resolves
to", with unexplained divergence in whether they fell back to the literal name.
`Router.resolved_litellm_models` is now the one owner; the routing-plugin
candidate list and the stream-options check both delegate to it, and
`_deployment_litellm_model` is gone.
The router's arms come from `strategy_router_dependencies`, the same enumeration
the health check reads. Only the roles that serve are arms: a classifier or
embedding model picks the tier and never produces a response anyone judges. A
semantic auto-router keeps its routes in an opaque config blob, so only its
default model is enumerable and the guard is incomplete there by design, able to
miss a collision but never to invent one
The two regenerated artifacts carry `presidio_analyze_chunk_size_bytes` from
alters the spec; the sync gate runs on any PR touching litellm/proxy, so this one
has to carry the base's drift to go green
The endpoint built messages=[{"role": "user", "content": prompt}], so a dry run
could not carry prior turns, the caller's system prompt, or the tool definitions
a request advertises. A real agentic turn reduced to its last sentence classified
as trivial, which is why a config sweep reported savings for every configuration.
Accept messages, system and tools, and forward them to the same pre-routing hook
untranslated, with the raw-body snapshot built by the serving path's own owner,
refresh_proxy_server_request_body_snapshot. Loose types are deliberate: the hook
reads whatever dialect the surface produced, so validating against one surface's
schema would reject the others.
prompt stays as the single-ask shorthand, normalized into one user turn inside the
request model so the handler carries no mode branch.
