Adds the regression test for the warning-drop branch in the Converse
adaptive-thinking translation, mirroring the chat completions path's
test_raw_adaptive_thinking_dropped_when_max_tokens_too_small.
Follow-up to #32867 (native /v1/messages) and the /chat/completions
commit earlier on this branch, extending the same adaptive-thinking
translation to the Bedrock Converse path.
Clients like Claude Code send thinking={type: "adaptive"} on every
request. When routed via Bedrock Converse to pre-4.6 models
(claude-haiku-4-5, claude-sonnet-4-5), this was forwarded as-is and
rejected by the model. Mirrors the translation already applied on the
/chat/completions and /v1/messages paths: map to legacy
thinking={type: enabled, budget_tokens}, capped below max_tokens.
Also fixes the missing custom_llm_provider arg in the chat completions
path's call to AnthropicConfig._map_reasoning_effort.
Clients that pass thinking={"type": "adaptive"} directly (not via the
reasoning_effort alias) on the /chat/completions interface had it forwarded
unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the
translation already applied on the native /v1/messages passthrough (#32867):
translate to legacy thinking={type: enabled, budget_tokens}, capped below
max_tokens, dropping thinking when max_tokens can't fit even the minimum
budget. Hoists the shared budget-capping helper onto AnthropicConfig so both
paths use one implementation.
YAML-parsed tier costs can arrive as strings (e.g. "4e-07"), which broke
arithmetic in the graduated tiered-pricing calculation. Coerce per-token
costs to float in both the in-range and remaining-tokens paths.
Move the tiered-cost helper out of the Dashscope module into a
provider-neutral home so the proxy budget reservation no longer depends on
a provider-specific module.
Co-authored-by: Cursor <cursoragent@cursor.com>
Exact cost-map hits resolve before fallback-generalization rules, so the
mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the
bedrock-anthropic-claude-mid-conversation-system rule and hoisted
mid-conversation system messages, invalidating the prompt cache.
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support
AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.
This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.
* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests
Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.
Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.
* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts
The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.
Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.
Updates the tests to assert the silent behavior and residual output_config
preservation.
* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)
Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.
thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.
Adds regression tests for Opus 4.5 with and without adaptive thinking.
* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models
Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic
* fix(anthropic): fall back to legacy thinking when effort level unsupported
Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels
* fix(anthropic): keep effort-only requests untouched for provider normalization
The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
vertex_ai/claude-opus-4-8@default (and sibling @default models) were
misclassified as non-adaptive because _model_map_lookup_candidates only
stripped provider prefixes but never the @<suffix> portion. The lookup
produced candidates like ["vertex_ai/claude-opus-4-8@default",
"claude-opus-4-8@default"], neither of which exists in model_cost, so
_is_adaptive_thinking_model returned False. LiteLLM then sent
thinking.type=enabled to a @default Vertex AI endpoint that requires
thinking.type=adaptive, resulting in a 400.
_strip_version_suffix now removes @<suffix> from each candidate,
adding the bare model name (e.g. "claude-opus-4-8") to the lookup
chain. Also adds supports_adaptive_thinking: true to the three
@default model_cost entries that were missing it as belt-and-suspenders.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke
* feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule
Hoisting every role system entry into the top-level system field mutates
the cache prefix whenever a client such as Claude Code appends a new
mid-conversation system message, invalidating the prompt cache for the
entire message history on Bedrock Invoke. Bedrock only rejects a system
entry at messages.0, so hoist just the leading run and forward the rest
in place
* fix(bedrock-invoke): retain clear_tool_uses_20250919 context_management edits and emit context-management-2025-06-27 beta (LIT-3393)
Copy of #29206 by oss-agent-shin, rebased onto litellm_internal_staging so CircleCI can run.
Bedrock InvokeModel supports automatic tool-call clearing (clear_tool_uses_20250919) under the context-management-2025-06-27 beta, but LiteLLM stripped the edit and dropped the beta header, causing a Bedrock 400. This maps bedrock.context-management-2025-06-27 to itself in anthropic_beta_headers_config.json (bedrock_converse stays null) and rewrites _filter_context_management_for_bedrock_invoke around an allowlist of supported edit types that keeps each supported edit and adds its matching beta.
* test(bedrock-invoke): restore beta-headers config cache with a shared fixture in LIT-3393 tests
Greptile flagged that three of the four new tests reloaded the module-level
beta-headers config into local mode without restoring it on teardown, leaking
state into later tests in the same process. Move setup/teardown into a
local_beta_headers_config fixture used by all four tests.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Bedrock Converse supports cachePoint ttl (1h GA for Claude 4.5+), and
_get_cache_point_block maps cache_control.ttl -> cachePoint.ttl, but the
model parameter its allow-list gate requires was only threaded through
the system-message path. Every message-level path either called
_get_cache_point_block without model= (8 call sites in
_bedrock_converse_messages_pt / _pt_async) or hardcoded
CachePointBlock(type="default") (tool-result blocks and
_convert_to_bedrock_tool_call_invoke), so a requested 1h ttl silently
degraded to the 5-minute default - exactly on the conversation-tail
breakpoint that long-running agents need to survive tool calls longer
than 5 minutes.
- pass model= at the 8 _get_cache_point_block call sites
- tool-result blocks: capture the cache_control dict (was a boolean)
and route through _get_cache_point_block so ttl survives
- _convert_to_bedrock_tool_call_invoke: accept optional model and route
per-tool-call cache_control through _get_cache_point_block
Completes the ttl support added for system messages (#19848, #20326):
message-level cache_control now behaves identically.
Note: message-level cache_control on a content-less assistant message
emits no cachePoint at all today; that pre-existing gap is orthogonal
to ttl and left out of scope (per-tool-call placement covers it).
Co-authored-by: Arash <arashne@glia-ai.com>
* fix(bedrock): stop stale SigV4 headers clobbering fresh signature on re-sign
When the Anthropic /v1/messages strip-thinking-and-retry path re-signs a
Bedrock request, _sign_request received attempt 1's already-signed headers
and copied the old Authorization and X-Amz-Date back over the freshly
computed SigV4 signature, so the retry POSTed the stripped body with a
signature for the original body and AWS returned 403 SignatureDoesNotMatch.
Skip SigV4-computed headers (authorization, x-amz-date,
x-amz-security-token, date) when restoring caller headers after signing,
and only preserve a caller-supplied Authorization that is not itself a
SigV4 header so bearer-token setups keep working.
* fix(bedrock): apply the same stale-header guard to get_request_headers
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
* fix(llm_http_handler): send dict transcription request data as a JSON body
httpx form-encodes dicts passed via data= and silently ignores json=, so the
generic audio transcription path never actually sent a JSON body. No provider
hit this before; JSON-body speech APIs need it.
* feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support
Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so
vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the
Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential
resolution (vertex_project/vertex_location/vertex_credentials or ADC); the
location defaults to the us multi-region since chirp_3 is only served from the
us and eu multi-regions, and non-global locations use the regional
<location>-speech.googleapis.com host. Maps language to languageCodes (auto
language detection by default), joins all result alternatives into the
transcript, and tracks cost from totalBilledDuration with a
vertex_ai/chirp_3 price entry at Google's published $0.016/min.
* fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text
OpenAI clients send language codes like "en", which Google rejects with 400
("not supported by the model chirp_3 in the location us"); Speech-to-Text
wants region-qualified BCP-47 like "en-US". Adds a shared
normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's
transcription config already hand-rolled the same table privately) that maps
common bare codes and passes region-qualified ones through, and applies it in
the Vertex transcription request. Also narrows the response JSON parse guard
to ValueError.
* fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works
cost_per_second prefers output_cost_per_second whenever it is not None, so the
0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using
input_cost_per_second. Remove it from both cost maps and pin the behavior with
a regression test computing 18s of chirp_3 audio to ~$0.0048.
* fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text
get_complete_url interpolated vertex_location straight into the request host,
and vertex_location is client-controllable on the proxy (it flows from the
request body and is not on the request-body blocklist). An authenticated caller
could send vertex_location="attacker.example/" to point the host at their own
server, so the proxy would POST the audio plus its admin-minted Google bearer
token and x-goog-user-project header to the attacker, exfiltrating a
cloud-platform-scoped OAuth token minted from the admin's credentials.
Factor the location validation the rest of vertex_ai already applied in
get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared
validate_vertex_location helper in common_utils and call it from both the chat
host builder and the new speech host builder. Invalid locations now raise a 400
VertexAIError instead of building a host. Also reject vertex_project values that
carry URL-structural characters, since it lands in the URL path.
Regression tests assert on the parsed netloc so the security property is pinned:
valid locations always resolve to a *speech.googleapis.com host and injection
inputs are rejected.
* fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring
* fix(streaming): surface in-body error payloads on OpenAI-compatible streams
vLLM and sglang return HTTP 200 streams whose SSE body carries the error,
e.g. data: {"error": {"message": "...", "code": 400}}. The OpenAI-compatible
chunk parser had no detection for this shape: since #23931 the payload parsed
into an empty chunk (choices=[]) and the stream ended silently with 200,
losing the provider's error and never attempting configured fallbacks.
Detect the payload in OpenAIChatCompletionStreamingHandler.chunk_parser and
raise OpenAIError with the upstream message and status code. The existing
mid-stream gate then applies: 4xx surface directly to the client, 5xx wrap
into MidStreamFallbackError so the router can run configured fallbacks.
Fixes#25492
* fix(streaming): serialize messageless error payloads as JSON
Address review feedback: an error dict without a message field now
serializes via json.dumps instead of Python dict repr
* fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop
* fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors
The bytes branch of _is_message_stop_chunk used a plain substring match,
so a content_block_delta whose partial_json contained the literal text
message_stop would look like a real terminal event and suppress the
synthetic incomplete-stream error. Match the SSE event header line
instead.
Also treat a provider-emitted error event as terminal so a stream that
ends with an upstream error is not followed by a second, contradictory
synthetic incomplete-stream error.
* test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses
* fix(anthropic_messages): forward aclose to inner streaming iterator
* fix(anthropic_messages): forward aclose through the streaming response wrapper
The proxy's streaming cleanup closes the handler's return value via
hasattr(response, "aclose"); the new wrapper hid the upstream
generator's aclose, so provider connections could linger on client
disconnect. The wrapper now delegates aclose to the wrapped stream and
AgenticAnthropicStreamingIterator closes its inner and follow-up
streams. Also adds test coverage for the agentic streaming branch
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix: zero out crash-class basedpyright rules across litellm/
* feat(lint): add LIT009 banning inert type: ignore comments
* docs: require bracketed rule and reason on every suppression
* chore(lint): ratchet budgets down and zero crash-class pyright limits
* fix: narrow auto router routelayer through a local before calling
* test: add regression tests for crash-class fixes
* fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool
The advisor_20260301 interceptor honored a caller-supplied api_base once
allow_client_side_credentials was enabled, even without a caller-supplied
api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's
own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials
plus the conversation history got sent to a caller-chosen destination
_resolve_advisor_credentials() now only honors api_base alongside a
non-empty caller-supplied api_key, requires the https scheme, and validates
api_base via validate_url() before use, mirroring check_complete_credentials
in auth_utils.py. https is required because validate_url only DNS-pins the
connection for http; for https with TLS verification on it returns the URL
unchanged and relies on certificate validation to block DNS rebinding
* fix(anthropic): also reject advisor api_base when ssl_verify is disabled
validate_url only DNS-pins the connection for http, or for https with
litellm.ssl_verify disabled; the previous https-only check missed the
ssl_verify=False case, where validate_url's rewritten URL was still being
discarded, per Greptile's review of this PR. Reject api_base outright when
ssl_verify is False so the discarded rewrite can no longer matter
* fix(anthropic): keep context_management working when drop_params is enabled
drop_params (proxy-wide or per-request) silently disabled the in-gateway
context_management polyfill on the /v1/messages -> chat completions adapter
path, even though context_management is a LiteLLM-supported param (native on
Anthropic, polyfilled elsewhere). Gate the polyfill on an explicit
additional_drop_params: ["context_management"] opt-out instead, which also
makes that escape hatch actually work on the adapter path.
* test(anthropic): cover sync adapter polyfill gate for global drop_params and additional_drop_params
* feat(tinyfish): make search provider permissive, attribute errors
Reshapes the TinyFish search provider so LiteLLM mirrors the TinyFish
Search API surface instead of maintaining a parallel cherry-pick.
Request side:
- Drop misleading request TypedDict
- Stop sending max_results on wire (TinyFish ignores it); clamp to [1,10]
client-side via self-threaded state
- Guard non-numeric max_results from bare ValueError
- Auto-JSON-encode dict params; lowercase bool serialization for ux-labs
Response side:
- Drop both Pydantic response models; parse directly into SearchResponse
so per-result extras flow through via extra="allow"
- Default missing title/url/snippet to "" instead of failing the call
- Read top-level parameter_warnings and re-fire as verbose_logger.warning
(pre-wired for upcoming TinyFish-side rollout; no-op today)
Error handling:
- Attributed _wrap_error helper at 3 call sites in transform_search_response
("TinyFish Search: <msg>. See https://docs.tinyfish.ai/search-api for
details.")
- Dispatch non-2xx responses through _wrap_error (fixes pre-existing bug
where 4xx/5xx silently returned empty SearchResponse)
- Wrap json.JSONDecodeError on 200 bodies
- Wrap pydantic.ValidationError for envelope-shape mismatches
Bug fix worth flagging: 4xx/5xx responses now raise an attributed
BaseLLMException instead of silently returning SearchResponse(results=[]).
Follow-up to #30634.
* fix(tinyfish): apply ruff format; guard OverflowError in max_results clamp
- Run ruff format on the touched files (CI lint job rejected the prior
commit's formatting).
- Add OverflowError to the except clause in the max_results clamp so
callers passing math.inf (or other non-finite floats) get the same
warn-and-ignore behavior as other malformed values. Greptile spotted
this in the first-pass review.
- Add test_max_results_infinity_float_warns_and_skips covering the
inf case.
* fix(tinyfish): apply --line-length 88 ruff format to match CI
CI uses 'ruff format --check --line-length 88'; my prior format pass
used the default line length, leaving several lines unwrapped. No
behavior change — purely whitespace.
* fix(tinyfish): reduce transform_search_response complexity; sort imports
CI's ruff strict-rule budget rejected the prior commit with:
- C901: transform_search_response complexity 16 > 10 (cap exceeded by 1)
- I001: import sort violation (cap exceeded by 1)
Extract two module-level helpers from transform_search_response to drop
its cyclomatic complexity:
- _default_missing_result_fields: in-place title/url/snippet defaulting
- _emit_parameter_warnings: defensive parameter_warnings reader
Auto-fix the import sort via ruff --fix.
No behavior change; the 59 existing tests still pass.
* test(tinyfish): cover defensive branches in _default_missing_result_fields
Codecov flagged 97.61% patch coverage (2 lines missing). The uncovered
lines were the non-dict raw_json and non-dict per-result item early-exits
in _default_missing_result_fields. Add two unit tests on the helper
directly to bring patch coverage to 100%.
* chore(tinyfish): apply ruff format to fix lint after staging merge
---------
Co-authored-by: Chenlu Ji <jichenlulu@gmail.com>
* fix(azure_ai): preserve content, tables, and keyValuePairs in doc-intelligence /v1/ocr
Azure Document Intelligence analyzeResult.content, .tables, and
.keyValuePairs were dropped when normalizing to the Mistral OCR schema.
They are now passed through verbatim as top-level response fields, and
the duplicated sync/async response parsing is consolidated into one
pydantic-validated helper.
Also adds the Azure DI features query param (list[str] or
comma-separated string, e.g. features=keyValuePairs) which Azure
requires for keyValuePairs extraction.
* test(azure_ai): replace fastapi jsonable_encoder with model_dump in ocr unit tests
* fix(bedrock): map guardrailConfig to InvokeModel guardrail headers
The InvokeModel API takes the guardrail identifier, version and trace as
X-Amzn-Bedrock-* request headers, unlike Converse which takes them in the
request body. The invoke transformer never set these headers, so
guardrailConfig was silently dropped (or leaked into the request body)
and Bedrock guardrails never ran on invoke-route models. Pop
guardrailConfig in AmazonInvokeConfig.validate_environment, validate it,
and set the headers before SigV4 signing; explicitly passed headers keep
winning over guardrailConfig so existing workarounds are unaffected
* fix(bedrock): reject guardrailConfig missing guardrailIdentifier
A guardrailConfig without guardrailIdentifier (e.g. an empty dict) would
validate, produce no guardrail headers, and let the request proceed with
guardrails silently not applied; that silent skip is the exact failure
mode this fix exists to remove, so fail fast with a 400 instead
* feat(tencent): add Tencent TokenHub as a provider
Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a
new provider: TencentChatConfig routes /v1/chat/completions and gates the
thinking/reasoning_effort params behind supports_reasoning, and
TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API.
Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider
endpoint support metadata.
* test(tencent): add unit tests for Tencent TokenHub provider
Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig
(messages API) across transformation, param mapping, URL building, and header
validation, plus get_optional_params routing. Tests mock supports_reasoning to
stay independent of remote model cost data.
* fix(tencent): correct max_output_tokens and reuse parent messages env validation
Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent
Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden
Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling
* fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix
* fix(tencent): register tencent in models_by_provider
The provider was added to the LlmProviders enum and cost map but not to the
models_by_provider lookup, so test_models_by_provider (which asserts every
litellm_provider present in the cost map is registered) failed once the tencent
models were loaded. Add the tencent_models set, populate it from the cost map,
and expose it under the tencent key, mirroring deepseek.
* fix(tencent): import generic_cost_per_token from its canonical module
Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils
instead of the top-level litellm.cost_calculator dispatcher, which imports the
tencent cost module at load time. Removing the back-reference avoids the circular
import and matches how deepseek and the other providers source the helper.
---------
Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(bedrock): honor ttl for tool_config cache injection points
Pass cache_control_injection_points control.ttl through to Bedrock
toolConfig cachePoint blocks, matching message/system cache behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex
is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a
manual update for every new Claude release (it already silently missed
Sonnet 5 and Fable 5). Replace it with a lookup against
cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json,
which AWS docs confirm tracks the same 1h-TTL-capable model set.
Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried
that pricing field (their own regional variants didn't have it), which
would have made the JSON-driven check wrongly grant them 1h TTL support.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests
test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and
test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id
(...-20250514-v1:0) that never shipped. This passed under the old regex-based
is_claude_4_5_on_bedrock, which matched on substring alone, but fails now
that it looks up cache_creation_input_token_cost_above_1hr in
litellm.model_cost, since the fake id has no pricing entry.
Also force the bundled local cost map in both tests so ttl eligibility reads
this branch's pricing data instead of the network-fetched main copy, which
lacks the fix until merge.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): restore cache and tool config compatibility
* fix(bedrock): preserve Sonnet 5 parallel tool config
* fix(bedrock): decouple parallel tool support from cache ttl
* refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns
Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and
bedrock_converse_supports_strict_tool_schemas (dead code) with a
supports_parallel_tool_use_config key in model_prices_and_context_window.json,
matching how is_claude_4_5_on_bedrock already reads
cache_creation_input_token_cost_above_1hr from the pricing JSON.
New models pick up parallel tool use support automatically when their
pricing entry ships with the key set, with no code change required
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test
anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in
model_prices_and_context_window.json, so
bedrock_converse_supports_parallel_tool_use_config returned False and the
test died with KeyError on additionalModelRequestFields. Use
jp.anthropic.claude-opus-4-7, a real entry that carries
supports_parallel_tool_use_config without 1h-TTL cache pricing, which is
exactly the decoupling this test exists to cover
* test(utils): allow supports_parallel_tool_use_config in pricing schema
The misc unit test job validates model_prices_and_context_window.json
against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects
unknown keys. Add the supports_parallel_tool_use_config key this PR
introduced so test_aaamodel_prices_and_context_window_json_is_valid
passes again
* fix(bedrock): preserve ttl for regional claude models
* fix(bedrock): fall back to base model entry when regional pricing lacks capability fields
Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit
cache_creation_input_token_cost_above_1hr shadowed the base entry that has it,
so is_claude_4_5_on_bedrock returned False and requested cache ttl values were
dropped for those deployments. Both capability lookups now consult the full
model id and the region-stripped base entry, matching the coverage of the old
name-pattern list. Also restores ToolBlock keyword construction for the
tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every
supported Python version
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(vertex_ai): pass full imageConfig dict for Gemini image generation
Support all ImageConfig fields (aspectRatio, imageSize, personGeneration,
imageOutputOptions) when calling Vertex AI Gemini image generation endpoints.
Previously only aspectRatio and imageSize were extracted; other fields were
silently dropped.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format vertex_gemini_transformation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): warn on non-dict imageConfig instead of silently dropping
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (LIT-2239)
* fix(bedrock): reopen audio content at client sample rate after trigger block
* test(bedrock): cover realtime handler disconnect flush and stream-end guard
* fix(bedrock): always close realtime input stream even if close flush fails
* fix(lint): use contextlib.suppress in bedrock realtime cleanup to satisfy BLE001 budget
* fix(bedrock): suppress bedrock close send errors per-message so promptEnd/sessionEnd still flush
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(gdc): add Google Distributed Cloud Gemini provider support
Introduce support for the Google Distributed Cloud (GDC) Gemini provider by adding "gdc" to the list of chat providers and enabling the gdc/ model prefix. The implementation defines a new GDCGeminiConfig class which handles authentication via Google Distributed Cloud service account credentials, manages token generation, formats GDC Gemini request URLs, and transforms request structures accordingly
The PreProcessNonDefaultParams class is also updated to exclude vertex parameters from filtering when the custom LLM provider is GDC, allowing vertex parameters to be passed properly during GDC initialization
* fix: resolve issues identified in PR #30702
* fix(gdc): harden credentials, fix vertex param filtering, add tests
The supports_vertex_params branch regressed vertex_ai and vertex_ai_beta: the `if custom_llm_provider in [...]: pass` was a no-op, so those providers fell through to the config lookup, found no supports_vertex_params, and had their vertex_ params stripped. The check is now a single _provider_supports_vertex_params helper that keeps vertex_ params for the vertex family and for any config that opts in, and only swallows the expected ValueError from an unknown provider string instead of a blanket except
GDC project and location now resolve from the deployment's litellm_params and the litellm.vertex_project / litellm.vertex_location globals before falling back to request optional_params, matching how vertex_ai resolves them, so a proxy caller can no longer route a request to a project the deployment did not expose
A request api_key is no longer treated as a filesystem path, so a caller can't make the host open a local service-account file; api_key must be a literal service-account JSON string or a bearer token
The opt-in token cache is hardened: the lock and cache dict are created in __init__ instead of via a racy hasattr lazy-init, the token is read inside the lock, and the audience is stripped of a trailing slash once so the cached and non-cached paths agree
Also declares gdc_api_base, switches the lazy-import entry to the relative path every other entry uses, adds the missing trailing comma in the provider config map, and drops the api_base fallback that only ran when api_key was None
Adds unit tests covering the vertex-param filter, deployment-over-request precedence, the api_key file-path rejection, URL construction branches, environment validation, token caching, and the gdc completion dispatch; transformation.py is fully covered
* fix(gdc): prefer GDC-specific config, honor vertex_ai aliases, harden URL and bool parsing
* fix(gdc): mint the GDCH token audience from the host, not the full base
When api_base embedded /v1/projects/... and the deployment set project/location, get_complete_url rebuilt the request URL from the host while validate_environment still derived the token audience from the full original api_base, so the bearer token could target a different audience than the URL actually called. The audience is now the scheme://host of api_base in every case, matching the host get_complete_url builds against
* fix(gdc): restrict JSON api_key to GDCH service accounts
Only accept a credential whose type is gdch_service_account before
calling google.auth.load_credentials_from_dict, so a caller-supplied
external_account/identity_pool/pluggable credential carrying arbitrary
token or credential_source endpoints is rejected before any token
refresh runs. GDC only ever uses GDCH service accounts, and non-GDCH
credentials could not have completed auth anyway (with_gdch_audience is
GDCH-only), so this narrows the credential-refresh surface without
changing valid GDC behavior.
* fix(gdc): validate project and location as plain identifiers
vertex_project and vertex_location can come from request params and were
interpolated as raw path text into the GDC request URL and the
x-goog-user-project header. A caller-supplied value containing / ? # or
.. could reshape the path and make the proxy send its GDC-authorized
request to a different endpoint under the configured host. Validate both
against a strict identifier pattern before building the URL or header and
raise an auth error otherwise; GCP project ids and locations are plain
identifiers so valid deployments are unaffected.
* fix(gdc): bind x-goog-user-project quota header to the deployment
The quota project header was resolved with request-level vertex_project
taking effect, so with a preformed deployment api_base a caller could set
vertex_project to a different project and have it sent under the proxy's
GDC credential, misattributing quota or billing. Resolve the header
project the same way the URL is resolved: a preformed api_base without a
deployment override binds to the project embedded in the URL, otherwise
deployment and global config win over request params. This keeps the URL
and the quota header consistent.
* fix(gdc): always rebind x-goog-user-project, stripping caller-forwarded values
The quota project header was only set when absent, so with client header
forwarding an authenticated caller could send their own
x-goog-user-project (any casing) and have it ride on the proxy's GDC
credential, bypassing the deployment-derived binding. Strip every casing
of the header and always set it from _effective_project before the
request is signed.
* fix(gdc): make a preformed api_base authoritative for project routing
get_litellm_params copies caller-supplied vertex_project and vertex_location into litellm_params via OPTIONAL_KWARGS_KEYS, so litellm_params cannot be treated as a deployment-only source. The previous _deployment_overrides_path inference let an authenticated caller flip a pinned preformed api_base such as /v1/projects/pinned/... to /v1/projects/attacker/..., driving requests to a caller-chosen project with the proxy's configured GDC credentials and quota header
A preformed /v1/projects/ api_base is now authoritative; get_complete_url returns it unchanged and _effective_project binds the x-goog-user-project quota header to the project embedded in that URL, so a caller can no longer redirect a pinned deployment or move the quota header off it. The two tests that asserted the override behavior are now regression tests that fail if the rewrite is reintroduced
* fix(gdc): make a preformed api_base self-sufficient in get_complete_url
get_complete_url resolved and required a params-derived vertex_project before returning a preformed /v1/projects/ api_base, so a deployment that pins its project in the api_base path was forced to also pass vertex_project or hit 'project is required'. validate_environment already extracts the project from a preformed URL and needs no such param, so the two paths disagreed
The preformed-URL early return now runs before project/location resolution, matching validate_environment: a preformed api_base is returned as-is with no redundant param, and non-preformed bases still require vertex_project and vertex_location as before. Adds a regression test that a preformed base with no project/location params returns the URL unchanged
---------
Co-authored-by: Paige O'Connor <lostpaige@google.com>
Co-authored-by: Tim Laubach <tlaubach@google.com>
* feat(github_copilot): route /v1/messages to Copilot native Anthropic endpoint
Add a GitHub Copilot Anthropic Messages transformation that routes supported Claude models through the native /v1/messages endpoint. This covers request URL construction, default headers, and supported model metadata.
* fix(github_copilot): address PR review feedback
Tighten the Anthropic Messages environment validation and web search interception behavior after review feedback. Avoid treating non-web-search requests as web-search-only paths.
* style(github_copilot): apply black formatting
Apply Black formatting to the GitHub Copilot Anthropic Messages tests.
* test(github_copilot): cover ProviderConfigManager dispatch for Anthropic Messages
Add coverage for ProviderConfigManager dispatch when GitHub Copilot models use the Anthropic Messages API, including non-Anthropic models returning no config.
* fix(github_copilot): apply messages-proxy intent header to /v1/messages
Set the messages-proxy interaction header for GitHub Copilot Anthropic Messages requests so /v1/messages uses the expected Copilot intent.
* fix(github_copilot): use modern generic annotations
Replace legacy typing generics in the GitHub Copilot Anthropic Messages transformation so the strict Ruff budget gate stays within its ceiling.
* refactor(github_copilot): decouple web-search short-circuit and harden /v1/messages URL
Address review feedback on the Copilot native Anthropic messages path.
Replace the hardcoded LlmProviders.GITHUB_COPILOT check in the web-search
interception handler with a handles_web_search_natively() method on
BaseAnthropicMessagesConfig (default True), overridden to False in
GithubCopilotAnthropicMessagesConfig. Provider-specific behavior now lives in
llms/ and the handler stays provider-agnostic, so a future provider in the same
situation needs no carve-out here.
In get_complete_url, reuse the already-resolved api_base returned by
validate_anthropic_messages_environment instead of reading the authenticator a
second time, removing redundant I/O and the mid-request inconsistency window.
The caller-supplied base is still discarded in validate, which is the security
boundary. Normalize a trailing slash on the base in both methods so a
tenant-specific host never yields a double-slash //v1/messages URL.
* fix(github_copilot): forward anthropic-beta headers on /v1/messages
The Copilot config inherited should_filter_anthropic_beta_headers()==True
from BaseAnthropicMessagesConfig, so update_headers_with_filtered_beta
stripped every anthropic-beta value after validate_anthropic_messages_environment
injected them (github_copilot has no mapping in
anthropic_beta_headers_config.json). That silently disabled header-gated
features like context_management and structured outputs on the native
passthrough. Override the hook to False, matching OpenAILikeAnthropicMessagesConfig.
* test(github_copilot): remove dead branch in beta-header regression test
The anthropic-beta filtering test guarded the fix with an if branch on
should_filter_anthropic_beta_headers(), which is always False, so the branch was
unreachable. Replace it with a direct assertion that running the provider-scoped
filter for github_copilot strips every beta value, proving why the override is
load-bearing and catching a regression that flips it back on.
---------
Co-authored-by: ririnto <ririnto@kakao.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Register bedrock_mantle/xai.grok-4.3 with /v1/responses in
supported_endpoints so the data-driven gate routes it through
BedrockMantleResponsesAPIConfig (which inherits SigV4 signing via
BedrockMantleAuthMixin). Without this entry the model falls through to
None and forces bearer-token-only auth.
Pricing sourced from AWS Bedrock pricing page.
Closes#31196
Co-authored-by: unknown <>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch): wire chat completion agentic loop to correct hooks
maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller.
Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path.
Regression test added.
* fix(websearch): strip tool_choice from follow-up request
When the original request forces tool_choice to litellm_web_search,
the follow-up request after search execution inherited that tool_choice,
causing the model to call the search tool again instead of synthesizing
an answer from the results.
* fix(websearch): inject api_key into agentic hook kwargs for anthropic messages
Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's
synthesis call after executing Exa/Perplexity searches) were missing api_key
because the named api_key param in async_anthropic_messages_handler was never
merged into the kwargs dict forwarded downstream. Result: every /v1/messages
websearch follow-up failed with "x-api-key header is required" and the caller
received the raw tool_use response instead of the synthesized answer.
* ci: trigger CI run
* fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks
CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with
_agentic_loop_api_surface to handle both surfaces from one hook. The chat
completion loop must also check _gate_overridden so callbacks using the
unified hook pattern still fire for chat completions.
* fix(websearch): strip tool_choice from legacy chat completion follow-up call
The _execute_chat_completion_agentic_loop path merged original optional_params
(which includes forced tool_choice) into follow-up params without explicit
removal. _build_chat_completion_request_patch already excluded tool_choice from
its optional_params output, but dict.update() with a missing key leaves the
original value intact. Explicit pop after the merge removes it.
* fix(websearch): always strip tool_choice from plan-path follow-up params
The tool_choice removal was gated on patch.tools is not None. WebSearch sets
tools via patch.optional_params not patch.tools, so the gate was False and
forced tool_choice from the original request survived into the synthesis call.
Move the pop outside the patch.tools branch so it applies unconditionally.
