A deployment carrying reasoning_effort in its litellm_params on the
/v1/messages passthrough mapped the effort to a legacy thinking block
whose budget_tokens was forwarded as is, so any request whose max_tokens
sat at or below that budget was rejected upstream with a 400. The mapped
budget now runs through the same cap the adaptive-to-legacy branch and
the chat path already use: it is clamped to max_tokens - 1, and dropped
with a warning when even the minimum budget cannot fit.
The cap helper becomes public since three call sites outside
AnthropicConfig use it.
GPT-5 and later accept a top-level anyOf natively and call tools better with it intact, so the flattening now runs only for the gpt-4, gpt-3.5, chatgpt-4o, o1, o3, and o4 families. Non-dict tool entries pass through untouched, a typeless root that carries properties counts as an object, and the bounded $ref walker is listed in the recursion detector allowlist.
Reprice ten more retired xAI slugs (grok-3 and grok-3-mini families,
grok-4-1-fast) to the grok-4.3 rates they now bill at, with family-correct
deprecation dates. Restore cache_read_input_token_cost on the Bedrock Grok 4.6
entries so implicit cache hits bill at the cache-read rate while explicit
cachePoint stays unsupported. Drop the unsourced 1080p video rate and the
gemini/ live native-audio entry the Gemini API 404s on. Add Groq qwen3.8-27b
tool-use flags per Groq docs. Extend the xai and gemini tests to lock all of
this in
OpenAI's function-calling validator rejects tool parameters carrying
oneOf/anyOf/allOf/enum/const/not at the top level, while the ChatGPT
backend Codex talks to natively accepts them, so an MCP tool declaring a
top-level union 400s through the proxy. Merge the branches into the
object schema for OpenAI itself only, walking the namespace-nested tools
current Codex builds send, on both /v1/responses and /v1/responses/compact
Adds litellm/litellm_core_utils/aws_partition.py mapping a region to its
AWS partition (aws, aws-cn, aws-us-gov, and the iso partitions), its DNS
suffix, and its ARN prefix, and uses it at every AWS host and ARN build
site: bedrock (runtime, agent, agentcore, legacy client, batches, files,
realtime), sagemaker, polly, secrets manager, s3 log uploads, bedrock
passthrough routes, and rag ingestion. ARN detection now accepts
arn:aws-cn: and arn:aws-us-gov: prefixes.
STS region resolution now falls back to the configured aws_region_name
after the aws_sts_endpoint host and the AWS_REGION/AWS_DEFAULT_REGION env
vars, so cn and gov role assumption no longer silently signs against
us-west-2.
A partition sweep test walks every endpoint builder with cn regions and
asserts no amazonaws.com host or arn:aws: prefix comes out, plus an AST
guard that fails on any new f-string hardcoding either literal.
Mutation testing surfaced three factory functions whose tests ran against them
but asserted nothing that a mutation could break, so every planted bug survived.
- litellm/llms/litellm_proxy/skills/code_execution.py: the OpenAI and Anthropic
tool schemas were unpinned (the Anthropic one was not reached by any test at
all) and the handler's default fallbacks were unchecked
- litellm/containers/endpoint_factory.py: the endpoints.json contract, the
generated sync/async function set and the response-type mapping were unpinned
- litellm/llms/openai_like/dynamic_config.py: the generated Responses API config
class had no coverage of auth header, URL resolution or the store override
The openai_like tests clear _responses_config_cache around each test. Without
that, the module-level cache hands back a class built before the mutation and
the tests pass against mutated code.
Verified by re-running mutmut per scope:
llms/litellm_proxy 45.2% -> 62.8% (70 mutants newly killed)
containers 36.8% -> 84.3% (45 mutants newly killed)
llms/openai_like 55.7% -> 66.9% (34 mutants newly killed)
* feat(spend): report prompt caching savings as total and gateway-attributed
`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.
Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.
Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.
The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.
Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.
* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)
The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.
Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
* fix(anthropic): drop and self-heal empty thinking blocks on /v1/messages
* test(anthropic): pin early-signature carry across the blank thinking chunk skip
#38318 taught exception_type to map upstream status codes for providers with
no branch of their own. It reads the status code off the exception, but
_handle_error stamps 500 onto every failure that never carried one, so a
refused connection reached the mapper wearing a status code nothing upstream
had sent, and came back as InternalServerError instead of APIConnectionError.
The two are not interchangeable to a caller: a 5xx says the provider answered
and failed, which the router treats as a reason to cool the deployment down,
while a connection error says the request never landed.
BaseLLMException now records whether its status code was received or
synthesized, _handle_error sets that when it invents the 500, and the status
mapper declines to act on a code litellm made up, so those failures fall
through to the APIConnectionError the branch was always meant to produce.
Genuine upstream 5xx responses are untouched, which the second test pins.
The search transformation assertion #38318 had loosened to InternalServerError
goes back to APIConnectionError for the same reason.
* fix(moonshot, together_ai): send the reasoning effort Kimi K3 accepts
Moonshot documents reasoning_effort as a top-level chat completions field for its reasoning
models, and defaults it to max, but MoonshotChatConfig builds its supported params by
subtracting from the OpenAI base list, which never carried that param. An explicit level
raised UnsupportedParamsError before the request left the proxy, so low and high were
unreachable and every call ran at the provider default
Together accepts low, high and max on Kimi K3. The per-model clamp added for the gpt-oss
family folds max down to high for every model except deepseek-ai/DeepSeek-V4-Pro, so a caller
asking for max silently got roughly half the reasoning budget they paid for
Moonshot now offers reasoning_effort whenever the registry says the model reasons. Together
sends a level the map entry declares unchanged, and keeps its existing table for every level
an entry does not name, so the only value that moves is Kimi K3 at max
* fix(moonshot): unwrap the bridges' effort object to the level string
AWS bills a Bedrock GPT-5.5 or GPT-5.4 prompt past 272K tokens under the long-context usage types for the
whole prompt, at 2x input, 2x cache read, and 1.5x output, and the cost map only had the flat rates, so a
300K prompt was logged at half of what the invoice charges. The map's promo rates for gpt-5.6-sol are 20%
under the $5.50 input, $33.00 output, $0.55 cache read, and $6.88 cache write per million the invoice bills.
Adds the *_above_272k_tokens fields to gpt-5.5 and gpt-5.4, moves sol's base and tier rates to the invoiced
ones, replaces the test that pinned the flat behaviour with one that pins the invoiced numbers, and updates
the sol pins in the mantle transformation tests
* fix(anthropic): resolve /v1/messages effort tiers through the capability owner
The bridge normalizer read three supports_*_reasoning_effort booleans of its own, so it
answered "which levels does this deployment take" independently of the resolver behind
/model_group/info. The two disagreed: a proxy advertising kimi-k3 max forwarded high.
Degrade against resolve_supported_reasoning_efforts instead, with the chains as a declared
table. When no step of a chain is accepted, the fallback is read off that same resolved set
rather than assumed, since an entry naming its levels outright can exclude the tiers the
per-level flags treat as unconditional. none is never chosen as that fallback, being an off
switch rather than a tier, and a deployment accepting no tier at all keeps the floor every
deployment degraded to before.
* test(anthropic): pin the normalized effort at the /v1/messages request boundary
The existing coverage stopped at normalize_reasoning_effort_value, so nothing failed if the
handler dropped or overwrote the normalized tier on its way into completion_kwargs. Drive
_prepare_completion_kwargs instead and assert on the kwargs handed to acompletion, in both the
string and the dict effort shapes, including the provider-prefixed model name the handler is
actually called with.
Against the pre-fix normalizer the fallback case fails, and against the baseline before a map
entry could declare its levels 7 of the 12 fail, so the boundary is pinned rather than restated.
A gpt-5 model accepts a non-default temperature only while its effective reasoning
effort resolves to "none". litellm had no representation of the effort a model applies
when the request omits reasoning_effort, so it substituted supports_none_reasoning_effort,
which is a different fact. Every model that supports "none" without defaulting to it
therefore had temperature forwarded and rejected upstream, and because the carve-out
returned before the drop_params branch, drop_params: true could not save it.
Declare the fact instead. A new cost-map key, default_reasoning_effort, states the effort
the provider applies when the request omits one, and one shared predicate resolves the
effective effort from it: an explicit reasoning_effort wins, otherwise the declared
default, otherwise the catalogue decides.
That last step matters because the cost map is fetched from the published branch at import
time, so it can be OLDER than the code reading it. On such a map every model looks
undeclared, and reading that as "reasoning is active" would strip temperature from the 39
gpt-5.1/5.2/5.4 entries that accept it, a regression caused by data lag rather than by
anything about the model. So an absent declaration is only meaningful once the catalogue
carries the key at all; a map that predates the feature keeps the answer litellm gave
before it existed, and the conservative answer applies from the moment the data lands.
The top_p/logprobs/top_logprobs gate carried the same assumption spelled differently and
now shares the predicate, as does the Responses API, which reimplemented the rule and is
what the default /v1/messages bridge routes openai models through. Azure normalises its
routing names in one resolver that every capability lookup goes through, which replaces
its bespoke per-lookup rewrite.
Declared on the 37 gpt-5.1/5.2/5.4 entries measured to accept temperature=0 today, so
their behaviour is unchanged. The 23 gpt-5.5/5.6 entries that reject it stay undeclared
and are fixed once the catalogue carries the key.
Resolves LIT-3797
Resolves LIT-5028
