CI runs without LITELLM_LOCAL_MODEL_COST_MAP=True, so litellm.model_cost
is loaded from main-branch JSON (default model_cost_map_url) instead of
the PR's checked-out model_prices_and_context_window.json. Tests that
assert per-model flags added in this PR (supports_max_reasoning_effort,
supports_xhigh_reasoning_effort) therefore pass locally but fail in CI
with 'AssertionError: assert False is True' on 5 cases:
- test_anthropic_model_supports_effort_param_recognizes_supporting_models
[anthropic.claude-mythos-preview, bedrock/.../mythos-preview,
claude-opus-4-5-20251101]
- test_supports_effort_level_handles_provider_prefixes
[bedrock/invoke/us.anthropic.claude-sonnet-4-6-max-True,
claude-sonnet-4-6-max-True]
Add an autouse fixture at tests/test_litellm/llms/anthropic/chat/conftest.py
that monkey-patches litellm.model_cost to the PR-local JSON for every test
in this directory. The parent conftest already snapshots+restores
litellm.model_cost per-function, so the mutation is contained.
This is a scoped workaround. The proper fix is to set the env var
globally in the test workflow once the ~10 inline self-set test files
are audited; tracking that as a follow-up issue.
The chat completion path (`_apply_output_config`) and the /v1/messages
pass-through (`AnthropicMessagesConfig._translate_reasoning_effort_to_anthropic`)
both gate `max` / `xhigh` per model. The two sites had diverged from
near-identical copies into separately maintained blocks, creating a real
drift risk when a new model tier (e.g. Claude 4.8) lands -- a contributor
could update one site and miss the other.
Centralise the gating in `AnthropicConfig._validate_effort_for_model`,
which returns an error message string or `None`. Each call site keeps
its own provider-appropriate exception type (`BadRequestError` for the
chat path, `AnthropicError` for the /v1/messages pass-through) but the
gating decision now comes from one place. Net -11 LOC.
Adds a parametrised unit test exercising the helper directly across
4.5 / 4.6 / 4.7 model families and `max` / `xhigh` / lower-effort
inputs. Existing tests at both call sites continue to pass unchanged.
Addresses Greptile finding on PR #27074.
`azure_ai` is registered in `litellm.openai_compatible_providers`, so
`add_provider_specific_params_to_optional_params` (litellm/utils.py)
auto-stuffs any non-OpenAI kwarg (e.g. `output_config={"effort": "..."}`)
into `optional_params["extra_body"]`. `AzureAnthropicConfig.transform_request`
then strips `extra_body` entirely on the way out, silently dropping the
param — and `AnthropicConfig._apply_output_config` never sees it, so
`effort="invalid"` / `effort="xhigh"` on a non-supporting model
quietly reaches the model with default behavior instead of returning a
clean 400 (as the native `anthropic` provider does).
Promote the keys back to top-level `optional_params` (using `setdefault`
so explicit top-level values win) before delegating to the parent
`AnthropicConfig`. Apply in both `validate_environment` and
`transform_request` so flag detection (`is_mcp_server_used`, etc.) and
output-config validation both run.
Surfaced by the QA matrix expansion on PR #27074: 20 cells where Azure
returned 200 while `anthropic` returned 400 — all `output_config` mode
across haiku_4_5, sonnet_4_5, opus_4_5, sonnet_4_6, opus_4_6, opus_4_7
families with `effort` in {invalid, xhigh, max, low, medium, high}.
Tests:
* `test_output_config_promoted_from_extra_body`: valid effort reaches data
* `test_invalid_output_config_effort_raises_via_extra_body`: 400 on bad effort
* `test_unsupported_effort_xhigh_raises_via_extra_body`: 400 on xhigh-on-Sonnet-4.6
* `test_extra_body_promotion_does_not_clobber_top_level`: setdefault semantics
- Image generation tests patch HTTPHandler.post / get_async_httpx_client so
make_*_azure_httpx_request runs and wire json is asserted on call kwargs.
- Azure image edit: strip model in finalize_image_edit_multipart_data using the
same URL string the handler passes to POST (no second get_complete_url in
transform). BaseImageEditConfig default finalize is a no-op.
Co-authored-by: Cursor <cursoragent@cursor.com>
Azure OpenAI routes image gen/edit by deployment in the URL; sending the
deployment id in model breaks gpt-image-2 (invalid_value). Strip model from
JSON for deployments/.../images/generations and from multipart data for
.../images/edits. Non-deployment URLs (e.g. Azure AI FLUX) unchanged.
Fixes#26316.
Co-authored-by: Cursor <cursoragent@cursor.com>
- claude-sonnet-4-6 + reasoning_effort=max no longer 400s. Renamed
_is_opus_4_6_model to _is_claude_4_6_model at three sites and added
supports_max_reasoning_effort: true to 12 model entries in the JSON
cost map (10 sonnet 4.6 ids + OpenRouter opus 4.6/4.7).
- _map_reasoning_effort now raises BadRequestError(400) directly with
llm_provider, instead of letting Databricks (and similar callers)
surface its raw ValueError as a 500.
- output_config.effort on Opus 4.5 over Bedrock no longer 400s for
missing effort-2025-11-24 beta. Flipped JSON to "effort-2025-11-24"
for bedrock + bedrock_converse and added an auto-attach branch in
_process_tools_and_beta for non-adaptive Anthropic + output_config
on Converse.
- reasoning_effort=xhigh / =max on legacy budget-mode models
(Haiku 4.5, Sonnet 4.5, Opus 4.5) now map to thinking.budget_tokens
8192 / 16384 instead of returning 400. Added two constants in
litellm/constants.py.
Tests updated for all four flips. Validated end-to-end via 306-cell
live proxy matrix (6 model families x 3 routes x 17 effort cases),
all pass.
Replace hardcoded _EFFORT_SUPPORTING_MODEL_PATTERNS with a JSON-backed
check that uses supports_*_reasoning_effort flags from the model map.
Add supports_minimal_reasoning_effort: true to opus-4-5 and mythos-preview
entries (which previously only carried supports_reasoning) so the JSON
remains the single source of truth for effort capability.
When a proxy fronts Claude Code (which always sends `output_config.effort`)
at a pre-4.5 Anthropic model — haiku-3, sonnet-3.5, opus-3, sonnet-4 — the
forwarded knob causes a forced 400 the client can't fix. Gating a strip
behind the existing `drop_params` flag lets operators opt into silent
fixup once and stop worrying about per-model param hygiene.
Default (`drop_params=False`) still forwards and surfaces the provider's
error, preserving the strict, debuggable contract from #27074.
Per https://platform.claude.com/docs/en/build-with-claude/effort the
supporting set is Opus 4.5+, Sonnet 4.6+, and Mythos Preview; everything
else is dropped (with a verbose_logger warning so the strip is visible).
Recognition uses model-name patterns plus a fallback to any
`supports_*_reasoning_effort` flag in the model map for forward
compatibility with new entries.
https://claude.ai/code/session_01WjHq31rvXT6xYNdVmSJvRp
(cherry picked from commit 1233943e78)
The reasoning-effort mapping dict was a public class attribute on
AnthropicConfig, so BaseConfig.get_config returned it as a request
parameter and every Anthropic-backed call (Anthropic / Azure / Vertex /
Bedrock Invoke) hit a 400 'REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT:
Extra inputs are not permitted' from the provider. Move the mapping
to a module-level constant.
_supports_effort_level only looked the model up under
custom_llm_provider='anthropic', so bedrock-prefixed model ids
(e.g. bedrock/invoke/us.anthropic.claude-opus-4-7) returned False
for both 'max' and 'xhigh' even when the underlying model entry has
the flag set. Strip known provider prefixes and retry the lookup
against litellm.model_cost directly so per-model gating works on
every route.
Mirror the per-model xhigh/max gate from
AnthropicConfig._apply_output_config in
AnthropicMessagesConfig._translate_reasoning_effort_to_anthropic so
the /v1/messages route also raises a clean 400 instead of forwarding
the unsupported tier.
Closes the remaining QA-sweep gap on PR #27074: Bedrock Invoke
/v1/messages was silently ignoring ``reasoning_effort`` because the
shared param filter only kept native Anthropic keys, so every effort
tier collapsed to the same behavior on the wire (27/231 cells failing
across opus-4-5 / opus-4-6 / sonnet-4-6).
Map ``reasoning_effort`` to native Anthropic ``thinking`` /
``output_config.effort`` at the ``AnthropicMessagesConfig`` layer so
all four /v1/messages routes (direct Anthropic, Azure AI, Vertex AI,
Bedrock Invoke) inherit the same translation:
- Add ``reasoning_effort`` to ``AnthropicMessagesRequestOptionalParams``
so the param filter in
``AnthropicMessagesRequestUtils.get_requested_anthropic_messages_optional_param``
no longer drops it before the transformation runs.
- Add ``_translate_reasoning_effort_to_anthropic`` and call it from
``transform_anthropic_messages_request``. Mirrors
``AnthropicConfig.map_openai_params`` on the chat completion path
(re-uses ``_map_reasoning_effort`` and
``REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT``) so the two routes
cannot drift. Pops ``reasoning_effort`` so it never reaches the wire.
- Caller-supplied native ``thinking`` / ``output_config.effort`` always
win — same precedence as
``_translate_legacy_thinking_for_adaptive_model``.
- Garbage values (``""``, ``"disabled"``, ``"invalid"``) raise
``AnthropicError(status_code=400)`` instead of falling through and
surfacing as 500s from the provider.
- ``"none"`` clears thinking + output_config so callers can opt out
per request.
Also restores the non-adaptive-model test coverage on Bedrock Invoke
/v1/messages that the previous commit lost when
``test_bedrock_messages_strips_output_config`` was renamed to the
``forwards`` variant on Opus 4.7.
Adds a new test file
``test_reasoning_effort_translation.py`` covering the translation at
the shared config level (adaptive + non-adaptive models, none, garbage,
caller precedence) so all four /v1/messages routes are exercised by a
single suite.
Adds parametrized + behavioral tests on the Bedrock Invoke /v1/messages
suite covering: minimal/low/medium/high/xhigh/max mapping for adaptive
models, thinking-budget mapping for non-adaptive Opus 4.5, ``none``
clears both, garbage raises 400, explicit ``output_config`` wins.
Refs: https://github.com/BerriAI/litellm/pull/27074
Follow-up bugs surfaced by the QA sweep on PR #27039
(https://github.com/BerriAI/litellm/pull/27039#issuecomment-4363363610).
1. Stop stripping output_config.effort on Bedrock + Vertex adaptive routes.
- Vertex AI Claude 4.6/4.7 accepts output_config.effort on rawPredict
(verified end-to-end against us-east5 / global). The strip helper now
no-ops for effort.
- Bedrock Converse routes output_config into additionalModelRequestFields
for anthropic base models so the requested adaptive tier (low/medium/
high/xhigh/max) actually reaches the wire instead of all collapsing to
identical thinking.
- Bedrock Invoke chat transformation (AmazonAnthropicClaudeConfig) stops
popping output_config from the post-AnthropicConfig request body.
- Bedrock Invoke /v1/messages allowlist (BedrockInvokeAnthropicMessagesRequest)
now lists output_config so the runtime allowlist filter forwards it.
2. Validate effort across Bedrock Converse so 'disabled' / 'invalid' / '' /
unsupported tiers (xhigh/max on Sonnet 4.6 or budget-mode 4.5 models)
surface as a clean 400 BadRequestError instead of 500.
3. ValueError -> BadRequestError throughout (AnthropicConfig.map_openai_params,
_apply_output_config, AmazonConverseConfig._handle_reasoning_effort_parameter).
Empty-string effort is now rejected (was silently passing the
'if effort and ...' short-circuit).
4. Floor reasoning_effort='minimal' at the Anthropic provider minimum
(1024 budget_tokens) via new ANTHROPIC_MIN_THINKING_BUDGET_TOKENS so it's
a usable tier on direct Anthropic / Azure AI Anthropic / Vertex AI Anthropic /
Bedrock Invoke (all of which 400 below 1024).
5. model_prices: dedupe duplicate supports_max_reasoning_effort key on
claude-opus-4-7 / claude-opus-4-7-20260416.
Adds regression tests across all five affected paths; existing tests asserting
the silent-strip behavior were updated to reflect the new pass-through and
clean 400 surfaces.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Hold the resolved config in a process-memory TTL cache so the
request-handling path doesn't run litellm_proxymodeltable.find_first
on every vector-store call.
The vector store create/update path previously called
``_resolve_embedding_config`` against the admin-configured router/DB
model and persisted the resolved ``litellm_embedding_config`` dict
(``api_key`` / ``api_base`` / ``api_version``) into the
``litellm_managedvectorstorestable.litellm_params`` column. Because the
resolver expanded ``os.environ/...`` references via ``get_secret``, the
DB row carried cleartext provider credentials, and the
``/vector_store/{new,info,update,list}`` responses returned them to any
authenticated caller who could supply a known admin model name.
Move the auto-resolve out of ``create_vector_store_in_db`` and out of
the update path. Persist only the user-supplied ``litellm_embedding_model``
reference. Resolve at request-handling time inside
``_update_request_data_with_litellm_managed_vector_store_registry`` so
the resolved config lives in the per-request ``data`` dict and is
garbage-collected after the response. Legacy rows that were created by
an earlier proxy version and already carry a resolved
``litellm_embedding_config`` skip the re-resolution and pass through
unchanged so embedding calls keep working.
The ``new_vector_store`` response now also runs the existing
``_redact_sensitive_litellm_params`` masker (already used by ``info``,
``update``, and ``list``), defending against caller-supplied cleartext
on the create path and against legacy rows whose persisted credentials
are still in the database.
Existing tests that asserted the old write-time-resolve behaviour are
updated to assert the new persistence shape (no embedding config
stored, just the model reference). Two new tests cover the use-time
path: one asserting fresh resolution happens when a row carries only
the model reference, the other asserting legacy rows with persisted
config skip re-resolution and continue to work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A pre-existing logic bug in ``_check_banned_params``: when the
deployment-level ``configurable_clientside_auth_params`` permitted one
banned field, the loop ``return``-ed on the first match instead of
``continue``-ing, so any other banned param later in the same body or
metadata dict was never checked. This PR's metadata walk multiplies the
surface where that bypass matters — a body pairing an allowed
``api_base`` with an observability credential like ``langfuse_host``
would silently pass.
Proxy-wide ``allow_client_side_credentials`` keeps ``return`` (it's a
global opt-in for every banned param). The per-param branch becomes
``continue`` so only the one explicitly-permitted field is skipped.
Adds a regression test that exercises the api_base + langfuse_host pair.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related gaps in the proxy's request bouncer:
1. ``is_request_body_safe`` (auth_utils.py) walked the request-body root
and the ``litellm_embedding_config`` nested dict, but not ``metadata``
or ``litellm_metadata``. The same fields it bans at root — Langfuse /
Langsmith / Arize / PostHog / Braintrust / Phoenix / W&B Weave / GCS /
Humanloop / Lunary credentials and routing — were silently accepted
when the caller put them inside metadata, retargeting observability
callbacks to a caller-controlled host with caller-supplied creds.
Walk both metadata containers (and parse the JSON-string form sent via
multipart / ``extra_body``) through the same banned-params helper, so
the existing ``allow_client_side_credentials`` opt-in covers both
paths consistently.
2. The banned-params list was hand-maintained and lagged the canonical
``_supported_callback_params`` allow-list in
``initialize_dynamic_callback_params``. Derive the observability bans
from that allow-list (minus a small ``_SAFE_CLIENT_CALLBACK_PARAMS``
set for informational fields like ``langfuse_prompt_version`` and
``langsmith_sampling_rate``) so future integrations are covered
automatically; ``_EXTRA_BANNED_OBSERVABILITY_PARAMS`` carries the
handful of fields integrations read but the allow-list hasn't caught
up to. A guard test fails CI if a new entry is added to
``_supported_callback_params`` without an explicit safe-list decision.
Separately in ``litellm_pre_call_utils.py``: add ``callbacks``,
``service_callback``, ``logger_fn``, and ``litellm_disabled_callbacks``
to ``_UNTRUSTED_ROOT_CONTROL_FIELDS``. The first three are appended to
worker-wide ``litellm.{input,success,failure,_async_*,service}_callback``
lists / ``litellm.user_logger_fn`` from inside ``function_setup`` — one
request poisons every subsequent caller in that worker. The last is the
inverse primitive: the legitimate path reads it from key/team metadata,
the request-body version silently disables admin-configured audit /
observability for the call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Anthropic's main API no longer resolves the non-canonical 'claude-4-sonnet-20250514'
alias for freshly issued keys, returning 404 not_found_error. PR #27031 already
swept three other live tests pinned to this alias to claude-haiku-4-5-20251001
but missed test_multiturn_tool_calls in the responses API suite, which is now
failing reliably on PR CI runs (e.g. PR #27074, job 1603363).
Bump the two model references in test_multiturn_tool_calls to the same
claude-haiku-4-5-20251001 snapshot used by PR #27031 -- it covers everything
this test exercises (tool calling, multi-turn) and isn't on a deprecation
schedule.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The strip ran before the proxy parses ``litellm_metadata`` from a JSON
string into a dict (a path used by multipart/form-data and ``extra_body``
callers), so ``isinstance(metadata, dict)`` was False and ``model_info``
survived the strip. Move the call to the same post-parse position the
``user_api_key_*`` strip already uses for the same reason. Adds a
regression test exercising the JSON-string ``litellm_metadata`` path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operators upgrading would otherwise see client-supplied pricing overrides
silently stop applying with no diagnostic. Emit a debug-level line listing
the dropped fields and pointing at the opt-in flag when any are stripped;
stay silent on the no-op path so the log isn't filled with noise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The proxy currently forwards request-body pricing parameters (the fields
on `CustomPricingLiteLLMParams`, plus `metadata.model_info`) into the
core call path. Those fields belong to deployment configuration, not to
per-request input — sending them from a client mutates the request's
recorded cost and, via `litellm.completion` → `register_model`, the
process-wide `litellm.model_cost` map for every later caller in the
worker. Strip them at the boundary.
The strip set is built from `CustomPricingLiteLLMParams.model_fields` so
pricing fields added later are covered automatically. Operators who do
want clients to supply per-request pricing can opt back in per key or
team via `metadata.allow_client_pricing_override = true`, mirroring the
existing `allow_client_mock_response` and
`allow_client_message_redaction_opt_out` flags.
Tests cover the strip set's coverage, root and metadata strips, the
opt-in skip on both key and team metadata, and a regression check that
the global `litellm.model_cost` map is unmutated after a stripped
request.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DotpromptManager was hardened to render through
ImmutableSandboxedEnvironment. The three sibling managers (gitlab,
arize, bitbucket) were missed and still instantiate plain
jinja2.Environment(), leaving the same attribute-traversal SSTI
primitive open: a template fetched from a GitLab/BitBucket repo or
Arize Phoenix workspace can reach __class__.__init__.__globals__ and
execute arbitrary Python on the proxy host.
Match the dotprompt pattern by switching all three to
ImmutableSandboxedEnvironment. The sandbox blocks the dunder-traversal
chain while leaving normal {{ var }} substitution intact, so the
template surface is unchanged for legitimate use.
Adds tests/test_litellm/integrations/test_prompt_manager_ssti.py
(18 cases) verifying each manager's jinja_env is a sandbox, that
classic SSTI payloads raise SecurityError, and that ordinary variable
rendering still works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Vertex batch output transformer was emitting both a populated 'response' and 'error' for failed batch entries. The OpenAI Batch output spec defines them as mutually exclusive: on error 'response' MUST be null. This broke any consumer using 'result["response"] is None' to detect failures.
Setting reasoning_effort="none" on Anthropic chat models (direct, Bedrock
Invoke, Bedrock Converse, Vertex AI Anthropic, Azure AI Anthropic) crashed
LiteLLM with:
litellm.APIConnectionError: 'NoneType' object has no attribute 'get'
Both the Anthropic chat transformation and Bedrock Converse called
``AnthropicConfig._map_reasoning_effort`` and assigned the ``None`` it returns
for ``"none"`` directly to ``optional_params["thinking"]``. Downstream
``is_thinking_enabled`` then did ``optional_params["thinking"].get("type")``
and crashed.
Pop ``thinking`` (and on Claude 4.6/4.7, ``output_config``) instead of
assigning ``None``, restoring the documented contract that
``reasoning_effort="none"`` means "do not enable thinking". This also
prevents downstream Anthropic 400s ("thinking: Input should be an object",
"output_config.effort: Input should be ...") if the bug were ever masked.
Verified end-to-end against the live Anthropic API and Bedrock Converse
on claude-opus-4-{5,6,7} and claude-sonnet-4-6, plus Bedrock Invoke for
Claude 4.5/4.6. Vertex AI Anthropic and Azure AI Anthropic inherit the
fixed ``map_openai_params`` from ``AnthropicConfig`` and need no further
changes.
Adds litellm.disable_vertex_batch_output_transformation (default False).
When True, afile_content returns raw Vertex predictions.jsonl untouched
so users that parse candidates/modelVersion directly are not broken.
The method was overwriting logging_obj.optional_params, logging_obj.model,
and logging_obj.start_time on the caller's Logging instance. When invoked
from llm_http_handler.py's generic framework path, the framework's own
logging_obj (which already went through pre_call) had its properties
clobbered, causing model and start_time to reflect the last batch line's
values rather than the original call context.
Fix: create a fresh local Logging instance for the per-line transformation
instead of mutating the incoming logging_obj. The caller's object is now
left entirely untouched regardless of whether a logging_obj was passed in
or not.
Regression tests added to verify model, start_time, and optional_params
are not mutated on the caller's logging_obj.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text'
instead of 'response' on streamed chunks. The non-streaming path was
already updated to fall back to 'response_text' (#26385), but the
streaming chunk parser still only read 'response', which caused
streaming requests against those models to silently produce empty
content.
Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser
and add a streaming test for the response_text shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
A SCIM PUT may legally omit `active` (full-replace with the field
absent). Pydantic fills the SCIMUser.active default of True, so the PUT
handler was overwriting metadata.scim_active with True even when the
client never sent it — silently reactivating a previously SCIM-blocked
user and unblocking their keys.
Use model_fields_set to detect whether the client actually sent
`active`. If omitted, preserve the prior scim_active value and skip
the cascade to virtual keys.
Also drop comments added in this PR that just narrate what the code
does; keep only the docstrings and the SQL-NULL pitfall note that
explain non-obvious behaviour.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
