* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models
* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping
* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models
Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params
The Anthropic /v1/messages -> Responses API adapter always wrapped
tool_choice in an object ({"type": "auto"}, {"type": "required"}), but
the Responses API's tool_choice schema for these cases is a bare
string ("auto"/"required"/"none"). Sending the object shape to an
OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a
400. The "none" case also fell through to "auto" instead of mapping to
"none".
Separately, litellm_params configured directly on a router-alias
deployment (auto_router/complexity_router, adaptive_router,
quality_router, or semantic auto_router) - e.g.
cache_control_injection_points, drop_params - were silently dropped
for every request through that alias. async_pre_routing_hook swaps
`model` from the alias name to the selected tier/route's model before
the deployment lookup runs, so the outbound call only ever merged in
the tier deployment's own litellm_params, never the alias's. Register
non-routing-config litellm_params from the alias deployment and apply
them to the request whenever a pre-routing hook substitutes the model.
* fix: satisfy ruff-strict-budget UP006 and router coverage checker
Use builtin dict[...] generics instead of typing.Dict for the two new
annotations introduced in the previous commit, since they pushed
UP006 over the codebase ceiling in ruff-strict-budget.json. Add a
direct unit test for _register_pre_routing_alias_overrides so the
text-based router_code_coverage.py checker sees it exercised by name.
* fix(router): replace alias-param denylist with a tight allowlist
_PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from
the alias's litellm_params before forwarding the rest as request
kwargs, but GenericLiteLLMParams also holds deployment-management
fields (tpm, rpm, weight, tags, max_budget, budget_duration,
use_in_pass_through, litellm_credential_name, ...) on the same object.
Any of those left off the denylist would get silently forwarded as if
they were request kwargs.
Replace the denylist with a tight allowlist of exactly the two
request-shaping params this feature exists for - drop_params and
cache_control_injection_points - so unrelated management fields never
reach the outbound call regardless of what else GenericLiteLLMParams
grows to hold.
* fix(router): re-register adaptive-alias overrides on set_model_list reload
set_model_list() unconditionally clears pre_routing_alias_overrides on
every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured()
skips rebuilding an AdaptiveRouter whose model_name already exists in
self.adaptive_routers - so _register_pre_routing_alias_overrides() never
ran again for an auto_router/adaptive_router alias after a reload,
silently dropping its drop_params/cache_control_injection_points.
Build the Deployment unconditionally and re-register its overrides even
on the skip-existing-router path; only the (expensive) AdaptiveRouter
construction itself stays skipped.
* style: ruff format after merging litellm_internal_staging
* fix(router): drop the alias-param allowlist, exclude only model
Per review discussion: instead of a router.py-local allowlist of exactly
which litellm_params an alias (auto_router/complexity_router,
adaptive_router, quality_router, semantic auto_router) can forward to
the request it routes, _register_pre_routing_alias_overrides now
forwards everything except `model` (the alias marker itself, e.g.
auto_router/complexity_router, never a real provider model).
Router-init-only fields (complexity_router_config,
complexity_router_default_model, auto_router_config,
auto_router_config_path, auto_router_default_model,
auto_router_embedding_model, adaptive_router_config,
adaptive_router_default_model, quality_router_config,
quality_router_default_model) now flow into request_kwargs unfiltered
too. That's safe because litellm.completion()/acompletion() already
strips anything in litellm.types.utils.all_litellm_params before
building the provider request - added these 10 keys there, alongside
the deployment-management fields (tpm, rpm, weight, ...) already listed.
Verified live: without that addition, complexity_router_config lands in
extra_body and ships raw to the provider; with it, it's stripped.
This moves the "which fields aren't real LLM params" list from a
router.py-local allowlist to the single existing global list every
completion() call already depends on, instead of maintaining two.
* refactor(router): look up alias litellm_params on demand instead of caching them
_register_pre_routing_alias_overrides cached each alias's litellm_params
into self.pre_routing_alias_overrides at deployment-init time, which
required keeping that cache in sync with set_model_list() reloads - the
exact bug the previous adaptive-router-reload fix was patching around
(AdaptiveRouter survives a reload, but the cache didn't always get
refreshed to match).
Delete the cache and the registration method entirely. async_pre_routing_hook
now looks up the alias's own litellm_params directly from self.model_list
via self.model_name_to_deployment_indices at request time, the same
model_list that's already correctly rebuilt on every set_model_list()
call. No second piece of state to invalidate, so the reload staleness
bug class isn't possible anymore, and it's less code than before.
* fix(openai/responses): clamp max_output_tokens below API minimum
Claude Code sends a max_tokens=1 warmup probe when running /model, which
the Anthropic Messages -> Responses adapter forwards as max_output_tokens=1.
OpenAI's Responses API rejects values below 16, so the probe failed with a
400. Clamp anything below the minimum up to 16 in map_openai_params so all
Responses API entrypoints (direct, chat->responses, anthropic->responses)
are covered.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(openai/responses): extract _enforce_min_max_output_tokens helper
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Adds the regression test for the warning-drop branch in the Converse
adaptive-thinking translation, mirroring the chat completions path's
test_raw_adaptive_thinking_dropped_when_max_tokens_too_small.
Follow-up to #32867 (native /v1/messages) and the /chat/completions
commit earlier on this branch, extending the same adaptive-thinking
translation to the Bedrock Converse path.
Clients like Claude Code send thinking={type: "adaptive"} on every
request. When routed via Bedrock Converse to pre-4.6 models
(claude-haiku-4-5, claude-sonnet-4-5), this was forwarded as-is and
rejected by the model. Mirrors the translation already applied on the
/chat/completions and /v1/messages paths: map to legacy
thinking={type: enabled, budget_tokens}, capped below max_tokens.
Also fixes the missing custom_llm_provider arg in the chat completions
path's call to AnthropicConfig._map_reasoning_effort.
Clients that pass thinking={"type": "adaptive"} directly (not via the
reasoning_effort alias) on the /chat/completions interface had it forwarded
unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the
translation already applied on the native /v1/messages passthrough (#32867):
translate to legacy thinking={type: enabled, budget_tokens}, capped below
max_tokens, dropping thinking when max_tokens can't fit even the minimum
budget. Hoists the shared budget-capping helper onto AnthropicConfig so both
paths use one implementation.
YAML-parsed tier costs can arrive as strings (e.g. "4e-07"), which broke
arithmetic in the graduated tiered-pricing calculation. Coerce per-token
costs to float in both the in-range and remaining-tokens paths.
Move the tiered-cost helper out of the Dashscope module into a
provider-neutral home so the proxy budget reservation no longer depends on
a provider-specific module.
Co-authored-by: Cursor <cursoragent@cursor.com>
Exact cost-map hits resolve before fallback-generalization rules, so the
mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the
bedrock-anthropic-claude-mid-conversation-system rule and hoisted
mid-conversation system messages, invalidating the prompt cache.
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support
AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.
This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.
* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests
Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.
Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.
* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts
The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.
Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.
Updates the tests to assert the silent behavior and residual output_config
preservation.
* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)
Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.
thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.
Adds regression tests for Opus 4.5 with and without adaptive thinking.
* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models
Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic
* fix(anthropic): fall back to legacy thinking when effort level unsupported
Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels
* fix(anthropic): keep effort-only requests untouched for provider normalization
The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
vertex_ai/claude-opus-4-8@default (and sibling @default models) were
misclassified as non-adaptive because _model_map_lookup_candidates only
stripped provider prefixes but never the @<suffix> portion. The lookup
produced candidates like ["vertex_ai/claude-opus-4-8@default",
"claude-opus-4-8@default"], neither of which exists in model_cost, so
_is_adaptive_thinking_model returned False. LiteLLM then sent
thinking.type=enabled to a @default Vertex AI endpoint that requires
thinking.type=adaptive, resulting in a 400.
_strip_version_suffix now removes @<suffix> from each candidate,
adding the bare model name (e.g. "claude-opus-4-8") to the lookup
chain. Also adds supports_adaptive_thinking: true to the three
@default model_cost entries that were missing it as belt-and-suspenders.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke
* feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule
Hoisting every role system entry into the top-level system field mutates
the cache prefix whenever a client such as Claude Code appends a new
mid-conversation system message, invalidating the prompt cache for the
entire message history on Bedrock Invoke. Bedrock only rejects a system
entry at messages.0, so hoist just the leading run and forward the rest
in place
* fix(bedrock-invoke): retain clear_tool_uses_20250919 context_management edits and emit context-management-2025-06-27 beta (LIT-3393)
Copy of #29206 by oss-agent-shin, rebased onto litellm_internal_staging so CircleCI can run.
Bedrock InvokeModel supports automatic tool-call clearing (clear_tool_uses_20250919) under the context-management-2025-06-27 beta, but LiteLLM stripped the edit and dropped the beta header, causing a Bedrock 400. This maps bedrock.context-management-2025-06-27 to itself in anthropic_beta_headers_config.json (bedrock_converse stays null) and rewrites _filter_context_management_for_bedrock_invoke around an allowlist of supported edit types that keeps each supported edit and adds its matching beta.
* test(bedrock-invoke): restore beta-headers config cache with a shared fixture in LIT-3393 tests
Greptile flagged that three of the four new tests reloaded the module-level
beta-headers config into local mode without restoring it on teardown, leaking
state into later tests in the same process. Move setup/teardown into a
local_beta_headers_config fixture used by all four tests.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Bedrock Converse supports cachePoint ttl (1h GA for Claude 4.5+), and
_get_cache_point_block maps cache_control.ttl -> cachePoint.ttl, but the
model parameter its allow-list gate requires was only threaded through
the system-message path. Every message-level path either called
_get_cache_point_block without model= (8 call sites in
_bedrock_converse_messages_pt / _pt_async) or hardcoded
CachePointBlock(type="default") (tool-result blocks and
_convert_to_bedrock_tool_call_invoke), so a requested 1h ttl silently
degraded to the 5-minute default - exactly on the conversation-tail
breakpoint that long-running agents need to survive tool calls longer
than 5 minutes.
- pass model= at the 8 _get_cache_point_block call sites
- tool-result blocks: capture the cache_control dict (was a boolean)
and route through _get_cache_point_block so ttl survives
- _convert_to_bedrock_tool_call_invoke: accept optional model and route
per-tool-call cache_control through _get_cache_point_block
Completes the ttl support added for system messages (#19848, #20326):
message-level cache_control now behaves identically.
Note: message-level cache_control on a content-less assistant message
emits no cachePoint at all today; that pre-existing gap is orthogonal
to ttl and left out of scope (per-tool-call placement covers it).
Co-authored-by: Arash <arashne@glia-ai.com>
* fix(bedrock): stop stale SigV4 headers clobbering fresh signature on re-sign
When the Anthropic /v1/messages strip-thinking-and-retry path re-signs a
Bedrock request, _sign_request received attempt 1's already-signed headers
and copied the old Authorization and X-Amz-Date back over the freshly
computed SigV4 signature, so the retry POSTed the stripped body with a
signature for the original body and AWS returned 403 SignatureDoesNotMatch.
Skip SigV4-computed headers (authorization, x-amz-date,
x-amz-security-token, date) when restoring caller headers after signing,
and only preserve a caller-supplied Authorization that is not itself a
SigV4 header so bearer-token setups keep working.
* fix(bedrock): apply the same stale-header guard to get_request_headers
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
* fix(llm_http_handler): send dict transcription request data as a JSON body
httpx form-encodes dicts passed via data= and silently ignores json=, so the
generic audio transcription path never actually sent a JSON body. No provider
hit this before; JSON-body speech APIs need it.
* feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support
Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so
vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the
Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential
resolution (vertex_project/vertex_location/vertex_credentials or ADC); the
location defaults to the us multi-region since chirp_3 is only served from the
us and eu multi-regions, and non-global locations use the regional
<location>-speech.googleapis.com host. Maps language to languageCodes (auto
language detection by default), joins all result alternatives into the
transcript, and tracks cost from totalBilledDuration with a
vertex_ai/chirp_3 price entry at Google's published $0.016/min.
* fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text
OpenAI clients send language codes like "en", which Google rejects with 400
("not supported by the model chirp_3 in the location us"); Speech-to-Text
wants region-qualified BCP-47 like "en-US". Adds a shared
normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's
transcription config already hand-rolled the same table privately) that maps
common bare codes and passes region-qualified ones through, and applies it in
the Vertex transcription request. Also narrows the response JSON parse guard
to ValueError.
* fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works
cost_per_second prefers output_cost_per_second whenever it is not None, so the
0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using
input_cost_per_second. Remove it from both cost maps and pin the behavior with
a regression test computing 18s of chirp_3 audio to ~$0.0048.
* fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text
get_complete_url interpolated vertex_location straight into the request host,
and vertex_location is client-controllable on the proxy (it flows from the
request body and is not on the request-body blocklist). An authenticated caller
could send vertex_location="attacker.example/" to point the host at their own
server, so the proxy would POST the audio plus its admin-minted Google bearer
token and x-goog-user-project header to the attacker, exfiltrating a
cloud-platform-scoped OAuth token minted from the admin's credentials.
Factor the location validation the rest of vertex_ai already applied in
get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared
validate_vertex_location helper in common_utils and call it from both the chat
host builder and the new speech host builder. Invalid locations now raise a 400
VertexAIError instead of building a host. Also reject vertex_project values that
carry URL-structural characters, since it lands in the URL path.
Regression tests assert on the parsed netloc so the security property is pinned:
valid locations always resolve to a *speech.googleapis.com host and injection
inputs are rejected.
* fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring
* fix(streaming): surface in-body error payloads on OpenAI-compatible streams
vLLM and sglang return HTTP 200 streams whose SSE body carries the error,
e.g. data: {"error": {"message": "...", "code": 400}}. The OpenAI-compatible
chunk parser had no detection for this shape: since #23931 the payload parsed
into an empty chunk (choices=[]) and the stream ended silently with 200,
losing the provider's error and never attempting configured fallbacks.
Detect the payload in OpenAIChatCompletionStreamingHandler.chunk_parser and
raise OpenAIError with the upstream message and status code. The existing
mid-stream gate then applies: 4xx surface directly to the client, 5xx wrap
into MidStreamFallbackError so the router can run configured fallbacks.
Fixes#25492
* fix(streaming): serialize messageless error payloads as JSON
Address review feedback: an error dict without a message field now
serializes via json.dumps instead of Python dict repr
* fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop
* fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors
The bytes branch of _is_message_stop_chunk used a plain substring match,
so a content_block_delta whose partial_json contained the literal text
message_stop would look like a real terminal event and suppress the
synthetic incomplete-stream error. Match the SSE event header line
instead.
Also treat a provider-emitted error event as terminal so a stream that
ends with an upstream error is not followed by a second, contradictory
synthetic incomplete-stream error.
* test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses
* fix(anthropic_messages): forward aclose to inner streaming iterator
* fix(anthropic_messages): forward aclose through the streaming response wrapper
The proxy's streaming cleanup closes the handler's return value via
hasattr(response, "aclose"); the new wrapper hid the upstream
generator's aclose, so provider connections could linger on client
disconnect. The wrapper now delegates aclose to the wrapped stream and
AgenticAnthropicStreamingIterator closes its inner and follow-up
streams. Also adds test coverage for the agentic streaming branch
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix: zero out crash-class basedpyright rules across litellm/
* feat(lint): add LIT009 banning inert type: ignore comments
* docs: require bracketed rule and reason on every suppression
* chore(lint): ratchet budgets down and zero crash-class pyright limits
* fix: narrow auto router routelayer through a local before calling
* test: add regression tests for crash-class fixes
* fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool
The advisor_20260301 interceptor honored a caller-supplied api_base once
allow_client_side_credentials was enabled, even without a caller-supplied
api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's
own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials
plus the conversation history got sent to a caller-chosen destination
_resolve_advisor_credentials() now only honors api_base alongside a
non-empty caller-supplied api_key, requires the https scheme, and validates
api_base via validate_url() before use, mirroring check_complete_credentials
in auth_utils.py. https is required because validate_url only DNS-pins the
connection for http; for https with TLS verification on it returns the URL
unchanged and relies on certificate validation to block DNS rebinding
* fix(anthropic): also reject advisor api_base when ssl_verify is disabled
validate_url only DNS-pins the connection for http, or for https with
litellm.ssl_verify disabled; the previous https-only check missed the
ssl_verify=False case, where validate_url's rewritten URL was still being
discarded, per Greptile's review of this PR. Reject api_base outright when
ssl_verify is False so the discarded rewrite can no longer matter
* fix(anthropic): keep context_management working when drop_params is enabled
drop_params (proxy-wide or per-request) silently disabled the in-gateway
context_management polyfill on the /v1/messages -> chat completions adapter
path, even though context_management is a LiteLLM-supported param (native on
Anthropic, polyfilled elsewhere). Gate the polyfill on an explicit
additional_drop_params: ["context_management"] opt-out instead, which also
makes that escape hatch actually work on the adapter path.
* test(anthropic): cover sync adapter polyfill gate for global drop_params and additional_drop_params
* feat(tinyfish): make search provider permissive, attribute errors
Reshapes the TinyFish search provider so LiteLLM mirrors the TinyFish
Search API surface instead of maintaining a parallel cherry-pick.
Request side:
- Drop misleading request TypedDict
- Stop sending max_results on wire (TinyFish ignores it); clamp to [1,10]
client-side via self-threaded state
- Guard non-numeric max_results from bare ValueError
- Auto-JSON-encode dict params; lowercase bool serialization for ux-labs
Response side:
- Drop both Pydantic response models; parse directly into SearchResponse
so per-result extras flow through via extra="allow"
- Default missing title/url/snippet to "" instead of failing the call
- Read top-level parameter_warnings and re-fire as verbose_logger.warning
(pre-wired for upcoming TinyFish-side rollout; no-op today)
Error handling:
- Attributed _wrap_error helper at 3 call sites in transform_search_response
("TinyFish Search: <msg>. See https://docs.tinyfish.ai/search-api for
details.")
- Dispatch non-2xx responses through _wrap_error (fixes pre-existing bug
where 4xx/5xx silently returned empty SearchResponse)
- Wrap json.JSONDecodeError on 200 bodies
- Wrap pydantic.ValidationError for envelope-shape mismatches
Bug fix worth flagging: 4xx/5xx responses now raise an attributed
BaseLLMException instead of silently returning SearchResponse(results=[]).
Follow-up to #30634.
* fix(tinyfish): apply ruff format; guard OverflowError in max_results clamp
- Run ruff format on the touched files (CI lint job rejected the prior
commit's formatting).
- Add OverflowError to the except clause in the max_results clamp so
callers passing math.inf (or other non-finite floats) get the same
warn-and-ignore behavior as other malformed values. Greptile spotted
this in the first-pass review.
- Add test_max_results_infinity_float_warns_and_skips covering the
inf case.
* fix(tinyfish): apply --line-length 88 ruff format to match CI
CI uses 'ruff format --check --line-length 88'; my prior format pass
used the default line length, leaving several lines unwrapped. No
behavior change — purely whitespace.
* fix(tinyfish): reduce transform_search_response complexity; sort imports
CI's ruff strict-rule budget rejected the prior commit with:
- C901: transform_search_response complexity 16 > 10 (cap exceeded by 1)
- I001: import sort violation (cap exceeded by 1)
Extract two module-level helpers from transform_search_response to drop
its cyclomatic complexity:
- _default_missing_result_fields: in-place title/url/snippet defaulting
- _emit_parameter_warnings: defensive parameter_warnings reader
Auto-fix the import sort via ruff --fix.
No behavior change; the 59 existing tests still pass.
* test(tinyfish): cover defensive branches in _default_missing_result_fields
Codecov flagged 97.61% patch coverage (2 lines missing). The uncovered
lines were the non-dict raw_json and non-dict per-result item early-exits
in _default_missing_result_fields. Add two unit tests on the helper
directly to bring patch coverage to 100%.
* chore(tinyfish): apply ruff format to fix lint after staging merge
---------
Co-authored-by: Chenlu Ji <jichenlulu@gmail.com>
* fix(azure_ai): preserve content, tables, and keyValuePairs in doc-intelligence /v1/ocr
Azure Document Intelligence analyzeResult.content, .tables, and
.keyValuePairs were dropped when normalizing to the Mistral OCR schema.
They are now passed through verbatim as top-level response fields, and
the duplicated sync/async response parsing is consolidated into one
pydantic-validated helper.
Also adds the Azure DI features query param (list[str] or
comma-separated string, e.g. features=keyValuePairs) which Azure
requires for keyValuePairs extraction.
* test(azure_ai): replace fastapi jsonable_encoder with model_dump in ocr unit tests
* fix(bedrock): map guardrailConfig to InvokeModel guardrail headers
The InvokeModel API takes the guardrail identifier, version and trace as
X-Amzn-Bedrock-* request headers, unlike Converse which takes them in the
request body. The invoke transformer never set these headers, so
guardrailConfig was silently dropped (or leaked into the request body)
and Bedrock guardrails never ran on invoke-route models. Pop
guardrailConfig in AmazonInvokeConfig.validate_environment, validate it,
and set the headers before SigV4 signing; explicitly passed headers keep
winning over guardrailConfig so existing workarounds are unaffected
* fix(bedrock): reject guardrailConfig missing guardrailIdentifier
A guardrailConfig without guardrailIdentifier (e.g. an empty dict) would
validate, produce no guardrail headers, and let the request proceed with
guardrails silently not applied; that silent skip is the exact failure
mode this fix exists to remove, so fail fast with a 400 instead
* feat(tencent): add Tencent TokenHub as a provider
Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a
new provider: TencentChatConfig routes /v1/chat/completions and gates the
thinking/reasoning_effort params behind supports_reasoning, and
TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API.
Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider
endpoint support metadata.
* test(tencent): add unit tests for Tencent TokenHub provider
Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig
(messages API) across transformation, param mapping, URL building, and header
validation, plus get_optional_params routing. Tests mock supports_reasoning to
stay independent of remote model cost data.
* fix(tencent): correct max_output_tokens and reuse parent messages env validation
Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent
Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden
Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling
* fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix
* fix(tencent): register tencent in models_by_provider
The provider was added to the LlmProviders enum and cost map but not to the
models_by_provider lookup, so test_models_by_provider (which asserts every
litellm_provider present in the cost map is registered) failed once the tencent
models were loaded. Add the tencent_models set, populate it from the cost map,
and expose it under the tencent key, mirroring deepseek.
* fix(tencent): import generic_cost_per_token from its canonical module
Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils
instead of the top-level litellm.cost_calculator dispatcher, which imports the
tencent cost module at load time. Removing the back-reference avoids the circular
import and matches how deepseek and the other providers source the helper.
---------
Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(bedrock): honor ttl for tool_config cache injection points
Pass cache_control_injection_points control.ttl through to Bedrock
toolConfig cachePoint blocks, matching message/system cache behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex
is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a
manual update for every new Claude release (it already silently missed
Sonnet 5 and Fable 5). Replace it with a lookup against
cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json,
which AWS docs confirm tracks the same 1h-TTL-capable model set.
Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried
that pricing field (their own regional variants didn't have it), which
would have made the JSON-driven check wrongly grant them 1h TTL support.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests
test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and
test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id
(...-20250514-v1:0) that never shipped. This passed under the old regex-based
is_claude_4_5_on_bedrock, which matched on substring alone, but fails now
that it looks up cache_creation_input_token_cost_above_1hr in
litellm.model_cost, since the fake id has no pricing entry.
Also force the bundled local cost map in both tests so ttl eligibility reads
this branch's pricing data instead of the network-fetched main copy, which
lacks the fix until merge.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): restore cache and tool config compatibility
* fix(bedrock): preserve Sonnet 5 parallel tool config
* fix(bedrock): decouple parallel tool support from cache ttl
* refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns
Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and
bedrock_converse_supports_strict_tool_schemas (dead code) with a
supports_parallel_tool_use_config key in model_prices_and_context_window.json,
matching how is_claude_4_5_on_bedrock already reads
cache_creation_input_token_cost_above_1hr from the pricing JSON.
New models pick up parallel tool use support automatically when their
pricing entry ships with the key set, with no code change required
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test
anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in
model_prices_and_context_window.json, so
bedrock_converse_supports_parallel_tool_use_config returned False and the
test died with KeyError on additionalModelRequestFields. Use
jp.anthropic.claude-opus-4-7, a real entry that carries
supports_parallel_tool_use_config without 1h-TTL cache pricing, which is
exactly the decoupling this test exists to cover
* test(utils): allow supports_parallel_tool_use_config in pricing schema
The misc unit test job validates model_prices_and_context_window.json
against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects
unknown keys. Add the supports_parallel_tool_use_config key this PR
introduced so test_aaamodel_prices_and_context_window_json_is_valid
passes again
* fix(bedrock): preserve ttl for regional claude models
* fix(bedrock): fall back to base model entry when regional pricing lacks capability fields
Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit
cache_creation_input_token_cost_above_1hr shadowed the base entry that has it,
so is_claude_4_5_on_bedrock returned False and requested cache ttl values were
dropped for those deployments. Both capability lookups now consult the full
model id and the region-stripped base entry, matching the coverage of the old
name-pattern list. Also restores ToolBlock keyword construction for the
tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every
supported Python version
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(vertex_ai): pass full imageConfig dict for Gemini image generation
Support all ImageConfig fields (aspectRatio, imageSize, personGeneration,
imageOutputOptions) when calling Vertex AI Gemini image generation endpoints.
Previously only aspectRatio and imageSize were extracted; other fields were
silently dropped.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format vertex_gemini_transformation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): warn on non-dict imageConfig instead of silently dropping
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (LIT-2239)
* fix(bedrock): reopen audio content at client sample rate after trigger block
* test(bedrock): cover realtime handler disconnect flush and stream-end guard
* fix(bedrock): always close realtime input stream even if close flush fails
* fix(lint): use contextlib.suppress in bedrock realtime cleanup to satisfy BLE001 budget
* fix(bedrock): suppress bedrock close send errors per-message so promptEnd/sessionEnd still flush
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
