* fix(bedrock): decouple STS region from Bedrock aws_region_name
STS AssumeRole now resolves signing region from aws_sts_endpoint (parsed
host) or AWS_REGION/AWS_DEFAULT_REGION instead of aws_region_name, fixing
air-gapped cross-region Bedrock setups and endpoint/signature mismatches.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): add regression coverage for _build_sts_client_kwargs
Parametrize _resolve_sts_region and _build_sts_client_kwargs matrix cases,
and assert IRSA/web-identity paths use aligned STS endpoint and region_name.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(bedrock): tighten STS region helpers and drop redundant web-identity endpoint synthesis
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): cover FIPS, GovCloud, and China STS endpoints
Addresses greptile P2: regex sts(?:-fips)? supported sts-fips hosts but
was not exercised by the parametrized parse test.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): strip bedrock/ prefix and URL-encode ARNs in get_bedrock_model_id for invoke path
The invoke path (used by /v1/messages → Anthropic SDK / Claude Code) called
get_bedrock_model_id() which, when falling back to the raw model string, did
not strip the 'bedrock/' routing prefix and did not URL-encode ARNs.
For a model like:
bedrock/arn:aws:bedrock:us-east-1:<ACCOUNT>:inference-profile/global.anthropic...
the URL built was:
/model/bedrock/arn:aws:bedrock:…/invoke-with-response-stream ❌
Bedrock returned a JSON error body. LiteLLM's AWSEventStreamDecoder passed
those bytes into botocore's EventStreamBuffer which expects binary event-stream
framing. Checksum validation failed on the JSON prelude (0x223a7b22 == ':{"')
producing a misleading botocore.eventstream.ChecksumMismatch instead of the
actual Bedrock error.
Fix: strip 'bedrock/' (and 'invoke/') routing prefix from model string, then
URL-encode if the result is an ARN — matching what the converse path already
does in converse_handler.py.
Fixes: LIT-3274
* fix(bedrock): use strip_bedrock_routing_prefix to handle compound prefixes
Address greptile review: the original fix used a loop with break, so
bedrock/invoke/arn:... only stripped bedrock/ leaving invoke/arn:...
which is not an ARN → fell through to .replace('invoke/','',1) →
bare unencoded ARN → same malformed-URL bug.
strip_bedrock_routing_prefix() iterates without break, correctly
stripping bedrock/ then invoke/ in sequence. Also adds test case
for the compound-prefix scenario.
* style: apply black formatting to fix lint CI (LIT-3274)
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: LiteLLM Bot <bot@berri.ai>
* fix(bedrock): use model info lookup for output_config support instead of hardcoded check
Replace hardcoded _is_claude_4_6_model() string matching with
supports_output_config flag in model_prices_and_context_window.json,
accessed via _supports_factory(). This follows the project's established
pattern for model capability checks (per AGENTS.md rule #8).
Bedrock Invoke now conditionally preserves output_config for models
that declare supports_output_config=true (currently Claude 4.6 models),
while stripping it for older models to avoid request rejection.
Ref: https://github.com/BerriAI/litellm/issues/22797
* fix(vertex_ai): single-flight credential refresh to prevent thundering herd (#26024)
* fix(vertex_ai): single-flight credential refresh to prevent thundering herd
When GCP credentials expire under high concurrency, all requests
simultaneously call credentials.refresh() via asyncify, saturating the
40-thread anyio pool and blocking the proxy for 20+ seconds.
This adds:
- Per-credential asyncio.Lock in get_access_token_async for single-flight
refresh (1 coroutine refreshes, others wait on the lock)
- Background refresh when token_state is STALE (usable but near expiry),
returning the current token immediately with zero added latency
- threading.Lock on the sync get_access_token path
- Uses google-auth's TokenState enum (FRESH/STALE/INVALID) instead of
reimplementing expiry logic
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR review comments
- Use asyncio.create_task() instead of deprecated get_event_loop().create_task()
- Track in-flight background refresh tasks to prevent duplicate refreshes
when multiple STALE-path callers pass through the lock before the first
background task completes
- Add token validation in the STALE branch (consistent with FRESH/INVALID)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: lazy-import TokenState to avoid breaking when google-auth is not installed
Also extract helper methods to bring get_access_token_async under the
PLR0915 statement limit (50).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: apply Black formatting to test file and update uv.lock
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove user-provided project_id from log messages (CodeQL log injection)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: avoid leaking token value in error message, log type instead
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: restore uv.lock to match litellm_oss_branch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove project_id from remaining log message (CodeQL log injection)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove remaining project_id from log and error messages
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reuse cached credentials in VertexAIPartnerModels (#26065)
* fix: reuse cached credentials in VertexAIPartnerModels instead of creating new VertexLLM per request
VertexAIPartnerModels.completion() was creating a throwaway VertexLLM()
instance on every call to get an access token, bypassing the credential
cache inherited from VertexBase. This caused a fresh token fetch for
every single request, adding significant latency overhead.
Fix: call super().__init__() to initialize VertexBase's credential cache,
and use self._ensure_access_token() instead of a new VertexLLM instance.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: apply same credential caching fix to VertexAIGemmaModels and VertexAIModelGardenModels
Same bug as VertexAIPartnerModels: both classes had `pass` in __init__
instead of `super().__init__()`, and created throwaway VertexLLM()
instances per request instead of using self._ensure_access_token().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(fireworks): add glm-5p1 metadata and parallel_tool_calls (#26069)
* fix(chatgpt): preserve responses routing and recover empty output (#25403) (#26219)
- preserve existing shared backend `mode` when router deployment registration
reuses a provider/model key already in `litellm.model_cost` (prevents alias
with `mode: chat` from downgrading shared `chatgpt/gpt-5.4` from `responses`
to `chat` and triggering 403s on /v1/chat/completions)
- teach the ChatGPT Responses parser to recover `response.output_item.done`
entries when `response.completed.output` is empty
- add defensive /responses -> /chat/completions bridge fallback that
reconstructs output items from raw SSE when `raw_response.output` is empty
- regression coverage for shared alias routing, empty completed.output
parsing, and SSE bridge recovery
Closes#25403
Co-authored-by: afoninsky <andrey.afoninsky@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(deps): relax core runtime dependency pins from exact == to ranges
When litellm migrated from Poetry to uv (PR #24905, v1.83.1), the core
dependency specifications in pyproject.toml changed from Poetry bare-version
strings (e.g. openai = "2.30.0") to PEP 621 exact pins (openai==2.24.0).
Poetry bare-version strings are actually caret ranges (^X.Y.Z == >=X.Y.Z,<X+1),
but PEP 621 == is exact. This means every downstream package that installs
litellm as a library dependency is now forced to downgrade aiohttp, pydantic,
openai, click, and 8 other common packages to exact old versions.
Fix: restore range specifiers for the 12 core runtime dependencies. The
optional extras (proxy, proxy-runtime, etc.) are consumed primarily by
Docker images where exact pins are appropriate and are left unchanged.
The uv.lock file continues to provide exact reproducibility for Docker
builds and CI.
Fixes: #26154
* Add Rubrik as officially-supported guardrail plugin (#25305)
* Add Rubrik as officially-supported guardrail plugin
Adds tool blocking and batch logging integration with an external Rubrik
webhook service. The plugin validates LLM tool calls against a policy
service (fail-open on errors) and batch-logs all requests/responses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Update Rubrik docs: config.yaml as primary, env vars as fallback
Restructures the Quick Start to present config.yaml as the recommended
approach with tabbed UI, and environment variables as an alternative
fallback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add Rubrik env vars to config_settings reference
Fixes documentation validation by adding RUBRIK_API_KEY,
RUBRIK_BATCH_SIZE, RUBRIK_SAMPLING_RATE, and RUBRIK_WEBHOOK_URL
to the environment settings reference table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add fallback message when blocking service returns empty explanation
Prevents whitespace-only violation message when the tool blocking
service blocks tools but returns an empty content field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(ocr): add Reducto parse OCR support (#26068)
* feat(ocr): add Reducto parse OCR support
* fix(reducto): address OCR review feedback
* chore: refresh uv lockfile
* Revert "chore: refresh uv lockfile"
This reverts commit 47200c0e60.
* Fix failing tests
* Fix code qa
* Replaced the async client violation
* Replaced black formatting
* Fix failing tests
* Fix failing tests
* Fix failing tests
* Fix failing tests
* Fix tests
* Fix vertex ai cred test
* Fix test
* fix(xai): normalize usage total_tokens for prompt caching
xAI can return total_tokens inconsistent with prompt_tokens +
completion_tokens when caching is enabled. Align with OpenAI-style
usage so shared LLM tests and downstream consumers see coherent totals.
Apply to non-streaming responses and streaming usage chunks.
Made-with: Cursor
* Fix stale Vertex token refresh fallback
* Fix OCR zero credit and Bedrock support checks
* Fix OCR and Fireworks capability handling
* fix: evict completed background refresh tasks from _background_refresh_tasks
Completed asyncio.Task objects were never removed from
_background_refresh_tasks. In long-running proxies with many distinct
credential keys the dict grows indefinitely, retaining references to
finished tasks and their results.
Fix:
- Pop the existing (done) entry before creating a replacement task.
- Attach a done_callback to each new task that removes its entry from
the dict once the task finishes (success or failure).
Tests:
- test_background_refresh_task_removed_after_completion: verifies the
done-callback cleans up a single entry after the task completes.
- test_background_refresh_tasks_no_accumulation_across_many_keys:
drives 20 distinct credential keys and confirms the dict is empty
after all background refreshes finish.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix: guard asyncio.create_task in RubrikLogger.__init__ against missing event loop
asyncio.create_task() raises RuntimeError when called outside a running
event loop. Wrap the call in a try/except RuntimeError so that RubrikLogger
can be instantiated in synchronous contexts (e.g. during startup, testing)
without crashing. The periodic_flush background task simply won't start in
those cases; it starts normally when the constructor is called inside an
event loop.
Add a test that verifies instantiation outside an event loop does not raise
(does not patch asyncio.create_task).
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix: preserve async batch and reauth coordination
* Fix mypy
* Fix xAI usage and Fireworks parallel tool params
* Fix Rubrik batch drain and SSE recovery mutation
* Fix router mode preservation and Rubrik batch flushing
* fix(responses): merge text-only items with output items in SSE recovery
When recovering output from raw SSE, OUTPUT_ITEM_DONE and OUTPUT_TEXT_DONE
events were treated as mutually exclusive fallbacks. If a stream emitted
OUTPUT_ITEM_DONE for some output indices and only OUTPUT_TEXT_DONE for
others, the text-only items at the missing indices were silently dropped.
Merge both dicts before returning, with OUTPUT_ITEM_DONE entries taking
precedence at any shared index (preserving the existing behavior covered
by test_transform_response_preserves_output_item_when_text_done_arrives_later).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(rubrik): preserve events on batch send failure
Previously, _log_batch_to_rubrik swallowed all HTTP errors and exceptions,
and the parent flush_queue unconditionally drained the queue afterwards.
On Rubrik 5xx responses, network errors, or timeouts the in-flight events
were silently dropped without ever being delivered.
- Re-raise from _log_batch_to_rubrik so failures surface to the caller.
- In CustomBatchLogger.flush_queue, catch exceptions from async_send_batch
and leave the queue intact for retry on the next flush. Existing loggers
that override flush_queue (e.g. Datadog) or that swallow their own errors
inside async_send_batch (e.g. Langsmith, GCS, Argilla) are unaffected.
- Tests now assert events are preserved on HTTP errors, network errors,
and that mid-flush appended events are also preserved on failure.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(chatgpt/responses): strip whitespace before parsing SSE chunks
_parse_sse_json_chunk in ChatGPTResponsesAPIConfig passed the raw chunk
directly to _strip_sse_data_from_chunk, which only matches the 'data:'
prefix at position 0. Chunks with leading whitespace (e.g. ' data: {...}')
were returned unchanged and silently failed JSON parsing, dropping the
contained event.
Mirror the existing fix in LiteLLMResponsesTransformationHandler._parse_raw_sse_chunk
by calling chunk.strip() before stripping the SSE prefix.
Adds a regression test using whitespace-padded data: lines and verifies
that the response.output_item.done payload is recovered into the final
ResponsesAPIResponse output.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(rubrik): override flush_queue so a single snapshot drives send and drain
Previously RubrikLogger relied on CustomBatchLogger.flush_queue, which
captured len(self.log_queue) separately from the snapshot taken inside
async_send_batch. Although both happen without an intervening await today
(so they agree in practice), they are semantically disconnected: a future
refactor that adds an await between the two captures, or that changes the
async_send_batch contract, could cause the parent to delete a different
number of items than were actually sent and trigger duplicate deliveries
to Rubrik.
Override flush_queue on RubrikLogger so a single snapshot drives both the
HTTP POST and the queue truncation. async_send_batch is preserved for
direct callers/tests but no longer participates in the canonical flush
path. Existing tests (including the one that explicitly invokes the base
CustomBatchLogger.flush_queue path) still pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: register reducto/parse-v3 and reducto/parse-legacy in active model pricing file
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bedrock): restore output_config forwarding and black formatting
Use model-map lookup with _model_supports_effort_param fallback so Bedrock
Invoke keeps output_config for Claude 4.6/4.7 when pricing flags are missing.
Revert custom_llm_provider=bedrock for supports_output_config checks, fix
allowlist test model, and apply black to xai/vertex files failing lint CI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(greptile): address remaining review concerns
- fireworks: resolve supports_reasoning lookup for short model names by also
trying the full accounts/fireworks/models/ path in model_cost
- ocr_cost: drop reducto-specific guard in shared utility; treat missing
pages_processed as zero cost when no per-page pricing is configured
- docs: remove reducto/rubrik markdown stubs from this repo (canonical docs
live in litellm-docs)
* fix(model_prices): register mistral/ministral-8b-2512
Mistral's API now returns model='ministral-8b-2512' when 'mistral-tiny' is requested. Adding the entry so completion_cost can resolve the cost for that response.
* fix(greptile): prune async refresh locks and lazy-start rubrik flush
- vertex: back `_async_refresh_locks` with a WeakValueDictionary so a per-key
Lock is auto-evicted once no coroutine holds it, preventing unbounded growth
in deployments with many credential combinations while keeping single-flight
semantics intact.
- rubrik: defer the periodic flush task to the first log event when the logger
is constructed without a running event loop, so low-traffic batches still
get drained instead of being silently stranded by a swallowed RuntimeError.
* Remove duplicate supports_max_reasoning_effort key in claude-opus-4-7 entries
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(vertex_ai): stabilize background refresh task tracking
- Guard background refresh done_callback with an identity check so a
stale callback cannot remove a newer task that already replaced it in
the tracking dict (done_callbacks are scheduled via call_soon, so a
fresh task can be stored for the same credential key before the old
callback fires).
- Replace WeakValueDictionary with a regular dict for
_async_refresh_locks so the per-key asyncio.Lock identity is stable
across concurrent callers; otherwise a lock can be GC'd between two
coroutines arriving for the same key, breaking single-flight.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: surface OCR pricing gaps and recover OUTPUT_TEXT_DONE in ChatGPT SSE
- cost_calculator.ocr_cost: log a warning when pages_processed is reported
but no ocr_cost_per_page is configured, instead of silently billing zero
via an implicit '(... or 0.0) * pages_processed' fallback. Behavior is
preserved (zero cost) so free-tier / unpriced models still work, but
configuration gaps are now visible in logs.
- ChatGPTResponsesAPIConfig._extract_completed_response_from_sse: also
collect response.output_text.done events into a text-only items map and
merge them into the recovered output (OUTPUT_ITEM_DONE wins on duplicate
output_index), mirroring the LiteLLMResponses handler. This recovers
text content when a provider only emits OUTPUT_TEXT_DONE and the final
response.completed event has an empty output list.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(cicd): drop obsolete async refresh locks auto-prune test
Commit dfb2524 intentionally reverted _async_refresh_locks from a
WeakValueDictionary back to a regular Dict so the per-key asyncio.Lock
identity is stable across concurrent callers — preserving
single-flight semantics. The test asserting that the dict shrinks
back to 0 after refreshes was added when the WeakValueDictionary
backing was still in place; it now contradicts the deliberate design
and is failing CI.
* fix(rubrik): sanitize proxy_server_request and harden tool_calls parsing
Address bugbot review concerns:
- Sanitize proxy_server_request before forwarding to the Rubrik webhook.
The previous code passed the entire inbound HTTP context (Authorization,
Cookie, x-api-key, and the raw request body) through to a third-party
endpoint, which exfiltrates proxy credentials and upstream secrets. The
new _sanitize_proxy_server_request allowlists only url and method.
(Cursor Bugbot HIGH severity #3192354895)
- Treat a null choices[0].message.tool_calls as 'all blocked' rather than
letting iteration raise and silently fall through the outer except in
apply_guardrail (which would fail open). Iterate over a defensive
fallback list instead of relying on the dict default.
(Cursor Bugbot MEDIUM severity #3192349538)
Co-authored-by: Cursor Bugbot <bugbot@cursor.com>
* fix: restore Fireworks substring matching and use RLock for Vertex sync refresh
- Fireworks _get_model_cost_capability: after exact-key lookups, fall back
to substring matching against fireworks_ai/* entries in model_cost so
model name variants (e.g. fine-tuned suffixes) continue to inherit
capability flags like supports_reasoning.
- Vertex vertex_llm_base: replace non-reentrant threading.Lock with RLock
on the sync refresh path so the reauthentication retry, which recurses
into get_access_token while still holding the lock, does not deadlock
when reloaded credentials are also expired.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(rubrik): collapse BlockedToolsResult dead-code into Optional[str]
The `allowed_tools` field on `BlockedToolsResult` was computed in
`_extract_blocked_tools` but never read by the only caller — when any
tool was blocked the integration unconditionally raised
`ModifyResponseException` to reject the full response, never doing
partial filtering. Drop the dataclass and return the blocking
explanation directly as `Optional[str]` so there's no misleading shape
hinting at unused partial-filter capability.
Co-authored-by: Greptile <greptile-apps[bot]@users.noreply.github.com>
* fix(greptile): prune vertex async refresh lock dict after release
Address greptile's open thread on _async_refresh_locks growing
unboundedly in high-cardinality deployments.
- Add _maybe_prune_async_refresh_lock: drops the per-key Lock from
the registry once no coroutine holds it and no coroutine is queued
in lock._waiters. The check-then-pop sequence is safe under
asyncio's cooperative scheduler — a waiter that arrives after the
pop simply creates a fresh lock under the same key, which is fine
because the previous batch is already done.
- Wrap the slow-path async with lock in a try/finally so the prune
runs on every exit (return, exception, reauth retry).
- Extract the existing background-refresh task scheduling into
_schedule_background_refresh so get_access_token_async stays under
ruff's PLR0915 ("Too many statements") limit. No behaviour change.
- Regression tests cover both pruning after release (the dict
shrinks back to zero after each call) and the safeguard that
keeps the lock alive while a waiter is still queued.
* fix(greptile): pass explicit bedrock provider to _supports_factory
Bedrock Invoke transformation files (chat and messages) called
_supports_factory(custom_llm_provider=None, ...) which relies on
auto-detection. For short Bedrock model names (e.g. 'anthropic.claude-opus-4-6'
without the version suffix) auto-detection fails and the lookup falls back
through the exception path. Passing the known 'bedrock' provider explicitly
makes the lookup deterministic for all Bedrock model variants, including
cross-region inference profile IDs.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(greptile): warn when OCR cost silently returns 0.0
Address greptile's P2 thread (#3144753707) about ocr_cost silently
under-reporting billing when response.usage_info.pages_processed is
missing. The credit-priced and unpriced fallback still has to return
0.0 (we don't know how to bill without usage), but emit a warning so
the missing-data case is visible in logs instead of disappearing.
The per-page-priced branch still raises, preserving the original
ValueError signal callers may catch.
* fix(greptile): reorder bedrock output_config strip comment labels
Swap the # 5a / # 5b step labels so they appear in numerical order
within the file. The new output_config-strip block was added with
label # 5b above the pre-existing # 5a 'remove custom field from
tools' block; rename the new block to # 5a and the pre-existing
block to # 5b so the labels match the order of the steps in the
file.
No behavior change.
Co-authored-by: Greptile Reviewer <greptile-apps@users.noreply.github.com>
* Fix substring matching specificity and remove mutable Reducto OCR config state
- Fireworks: _get_model_cost_capability fallback now picks the longest
substring match in model_cost so more specific entries win over less
specific ones (instead of returning the first match by insertion order).
- Reducto OCR: drop per-request _api_key/_api_base instance attributes on
_BaseReductoOCRConfig and instead thread api_key/api_base through
transform_ocr_request/async_transform_ocr_request kwargs from the
shared OCR HTTP handler. Makes the config safe to share/cache across
concurrent requests with different credentials.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(greptile): drain background refresh + warn on router mode override
Address the two new findings from greptile's 19:45 review of the
vertex+router surfaces.
- vertex_llm_base: when the slow path sees TokenState.INVALID, await any
in-flight background refresh task before invoking refresh_auth
ourselves. google-auth's Credentials.refresh() is not safe to call
concurrently on the same credentials object, and the background task
runs outside the per-key lock. After the wait, re-check the cached
token so we can short-circuit if the background refresh already
restored it. Extracted the helper into
_await_in_flight_background_refresh so get_access_token_async stays
under ruff's PLR0915 statement budget.
- router.py: when alias registration would overwrite the deployment's
declared `mode` to keep the shared backend mode stable, emit a
verbose_router_logger.warning so the override is visible to operators
instead of silently winning. The existing fix (preventing alias
registration from downgrading a shared `mode: responses` to chat) is
preserved; the warning just surfaces it.
* fix(cicd): apply black formatting to vertex_llm_base.py
* fix(greptile): guard Reducto upload helpers against missing file_id
Raise a clear ValueError when Reducto /upload returns 200 without a
file_id key (or with a non-JSON body), instead of letting downstream
callers see a confusing KeyError.
* fireworks_ai: cache fireworks model_cost index and use hyphen-boundary matching
- Build a memoized index of fireworks_ai/* entries from litellm.model_cost,
invalidated by (id, len) of the model_cost dict. Avoids re-scanning the
full ~30k-entry model_cost dictionary on every get_provider_info call.
- Replace plain substring containment with hyphen-aligned boundary matching
so a known short model name (e.g. 'some-model') cannot falsely match an
unrelated longer query (e.g. 'awesome-model').
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(greptile): refcount vertex async refresh lock pruning
Replace the asyncio.Lock._waiters inspection in
_maybe_prune_async_refresh_lock with an explicit refcount so the entry
is pruned exactly when no coroutine is holding or waiting on the lock,
without depending on any private asyncio internals.
* fix(vertex): serialize credentials.refresh() across threads via _sync_refresh_lock
refresh_auth is invoked from three call sites that can run on different
threads (sync get_access_token, async slow path via asyncify, and the
background proactive refresh task). Only the sync path was protected
by _sync_refresh_lock, so a concurrent sync + async/background call
could invoke google-auth's Credentials.refresh() on the same object
from two threads simultaneously, mutating internal credential state.
Move the lock acquisition into refresh_auth itself; the lock is an
RLock so reentrant acquisition from the sync path remains safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(responses): extract shared SSE output-item recovery helpers
Both ChatGPTResponsesAPIConfig and LiteLLMResponsesTransformationHandler
duplicated the same OUTPUT_ITEM_DONE / OUTPUT_TEXT_DONE recovery
algorithm. Move that logic into litellm.responses.sse_output_recovery
and have both call sites use the shared helpers, so future fixes apply
in one place.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(greptile): tie fireworks index cache to model_cost mutation generation
* fix: address three bug detection findings
- rubrik: use 'is not None' check for tool call IDs to allow empty-string IDs
- router: indent mode preservation mutation to match warning conditional
- responses transformation: add missing 'continue' after OUTPUT_TEXT_DONE handler
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(router): always preserve existing shared backend mode when deployment mode is None
Previously the inner guard 'if _deployment_mode is not None' prevented
_shared_model_info['mode'] from being set back to the existing shared
mode when the deployment mode was None, which then overwrote the shared
backend's mode with None via register_model.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: address three bug detection findings
- vertex_llm_base: guard background refresh's cache write with an
identity check so a stale write cannot overwrite a credentials
reference replaced by a concurrent reauthentication path.
- router: make shared backend mode preservation directional - only
preserve when an existing 'responses' mode would be downgraded to
'chat', or when the deployment mode is None (which would otherwise
clear the existing mode). Legitimate upgrades now apply.
- rubrik: remove unused preserve_events_added_during_flush attribute;
RubrikLogger overrides flush_queue, so the base-class flag never
applied. Drop the test that exercised the parent path on a Rubrik
instance since it does not reflect real flush behavior.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(veria): scope reducto file IDs to current request + register pricing
- Reject reducto:// file IDs sent through the proxy /v1/ocr JSON API.
The IDs are not bound to a LiteLLM key, so an authenticated user
could submit another user's file ID and receive OCR text via the
proxy's shared Reducto credentials. Force fresh uploads (multipart
form or inline base64 data URI) so every OCR call is server-mediated
and implicitly bound to the originating request.
- Add ocr_cost_per_credit=0.015 to reducto/parse-v3 and
reducto/parse-legacy in both pricing JSONs so successful Reducto OCR
calls debit key/team spend instead of recording zero.
* fix(vertex): always overwrite resolved cache key with fresh credentials
After reauthentication or fresh load, the resolved (cache_credentials, project_id)
cache key may point to stale credentials from a prior load. Skipping the write
when the key existed forced the next request to go through a redundant
refresh/reauth cycle. Always overwrite so callers using the resolved project_id
hit the fresh credentials object.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(xai): fold reasoning tokens before normalizing usage in streaming chunks
The non-streaming transform_response folds xAI's reasoning_tokens into
completion_tokens before calling _normalize_openai_compatible_usage_totals,
preserving the OpenAI invariant total = prompt + completion. The streaming
chunk_parser only ran the normalization, so when xAI streamed usage with
reasoning tokens (total = prompt + completion + reasoning), the normalize
check (total < prompt + completion) was a no-op and the invariant remained
violated.
Refactor _fold_reasoning_tokens_into_completion to also accept a raw usage
dict (in addition to ModelResponse / Usage) and call it from the streaming
chunk_parser before normalization, so streaming and non-streaming paths
report usage consistently for reasoning models.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(greptile): cap SSE content_index padding and use multiset tool-id check
* fix(rubrik): apply event_hook default when caller passes None
initialize_guardrail always passes event_hook=litellm_params.mode, so
setdefault never applied its default. When mode is omitted from the
guardrail config, event_hook ended up as None instead of post_call.
Use 'or' to fall back to the intended default when the value is None.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(rubrik): cover event_hook default coercion
Regression tests for the case where the upstream caller (initialize_guardrail)
passes event_hook=None and the logger should still fall back to post_call,
and the sanity case where an explicitly-set non-None event_hook is preserved.
* fix: address autofix bugs in chatgpt SSE, vertex token cache, rubrik aclose
- chatgpt responses: don't overwrite a meaningful error_message with None
when a later RESPONSE_FAILED/ERROR event lacks an error object.
- vertex_ai: serve STALE tokens from the lock-free fast path and only
schedule a deduplicated background refresh, eliminating per-key lock
contention near token expiry.
- rubrik: aclose() now closes both async_httpx_client and
tool_blocking_client to avoid leaking connections from the dedicated
client when the logger shuts down.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(vertex): drop redundant resolved_project rebind in slow path
Reusing resolved_project (typed str from the fast path's tuple unpack)
for an Optional[str] assignment tripped mypy. Use project_id directly
after the None check.
* test(team_members): skip flaky test_add_multiple_members
The test creates a team via /team/new, adds a member via /team/member_add,
then queries /team/info — and intermittently gets a 404 for a team that
was just successfully created and mutated. The basic happy path is
already covered by test_add_single_member; we only lose the 10-iteration
stress loop.
* fix(rubrik): cancel periodic flush task on aclose
The aclose() method closed both HTTP clients but did not cancel the
periodic flush task. After close, the task would wake up every
flush_interval seconds and try to POST via the now-closed
async_httpx_client, generating recurring errors.
Cancel the task and await its termination before closing the clients.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(rubrik): coerce None default_on to True at init
* fix: tighten SSE done parser + rubrik /v1/messages match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bedrock): warn when invoke transformation strips output_config
The Bedrock Invoke chat and messages transformations strip output_config
when neither supports_output_config nor any supports_*_reasoning_effort
flag is set in the model JSON. This was silent; emit a verbose_logger
warning when the strip actually removes a present output_config so newly
released models (where the JSON entry hasn't caught up yet) surface a
clear log line instead of dropping the effort parameter without notice.
* fix(rubrik): drop tool_call repr from normalize error to avoid leaking args
The TypeError raised in _normalize_tool_calls is caught by apply_guardrail's
broad except, which logs the message plus exc_info. Including repr(tc) in
the message could expose function arguments (potentially sensitive user
data) in the proxy log stream. Type name alone is enough for debugging.
* fix: dedupe SSE chunk parser and warn on Fireworks tool drop
- Centralize SSE 'data:' chunk parsing in litellm.responses.sse_output_recovery
so the ChatGPT Responses transformer and the Responses->Chat-Completions bridge
share a single implementation.
- Log a warning when get_supported_openai_params drops 'tools' for a
fireworks_ai model whose JSON entry sets supports_function_calling=false,
so users notice the behavioral change instead of silently losing tools.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(fireworks_ai): demote per-request tool drop warning to debug
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(veria): cap Rubrik retry queue at 10k events with drop-oldest
A persistent Rubrik webhook outage previously let authenticated traffic
accumulate prompt/response payloads in the in-memory retry queue
without bound. The PR-introduced retry-on-failure behavior in
flush_queue() never trims the queue, so under sustained outage and
high request volume the proxy can run out of memory.
Cap the queue at RUBRIK_MAX_QUEUE_SIZE events (default 10_000) and
drop the oldest events when the cap is exceeded. Emit a throttled
verbose_logger warning so operators can detect a stuck webhook.
* fix(tests): accept either initial event type from xAI realtime
xAI's Grok Voice Agent API used to emit 'conversation.created' as the
first event over the WebSocket. It has since shipped a fully
OpenAI-compatible 'session.created' event (and may still emit the
legacy 'conversation.created' on some routes), which breaks the
strict-equality assertion in the realtime e2e test:
AssertionError: Expected conversation.created, got session.created
This is an upstream behavior change, not a regression in our code.
Loosen the base realtime test so get_initial_event_type() may return a
tuple of acceptable event types, and have the xAI subclass accept both
'conversation.created' and 'session.created'. The OpenAI subclasses
keep their single-string contract unchanged.
* fix(rubrik): drop RUBRIK_MAX_QUEUE_SIZE env knob, hardcode 10k cap
The doc-validation CI scans for os.getenv() calls and requires each key
to appear in litellm-docs config_settings.md. Adding the env var here
without a matching docs PR fails the docs and code-quality checks, and
the extra env-parsing block in __init__ also tripped ruff PLR0915.
The hard cap at 10k still bounds memory on a Rubrik webhook outage,
which is the actual bug being fixed -- operators don't need to tune
this knob to get the safety guarantee.
* test(team_members): skip flaky test_duplicate_user_addition
Same /team/info 404-after-add_team_member race that already led to
test_add_multiple_members being skipped in dedc4022. Duplicate-prevention
behavior is covered by test_update_team_members_list_duplicate_prevention
in tests/test_litellm/proxy/management_endpoints/test_team_endpoints.py,
so the e2e proxy variant doesn't add coverage.
* fix: bound CustomBatchLogger queue and call super().__init__ in ContextCachingEndpoints
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(rubrik): distinguish malformed tool-blocking response from transient errors
Raise a dedicated _MalformedToolBlockingResponseError when the tool
blocking service returns an empty 'choices' list, instead of a bare
Exception. Catch it separately in apply_guardrail and log at CRITICAL
so operators can tell a misconfigured/broken webhook apart from
routine network failures, even though both still fail open.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* router: clarify shared backend mode preservation flow
Add a blank line and a brief comment before the _backend_alias_cost
assignment to make it clear that registration runs unconditionally
after the optional mode-preservation mutation.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(ci): skip chronically flaky test_spend_logs_with_org_id
Same write-then-read race against the spend logs DB as test_spend_logs
(already skipped above). /spend/logs?request_id=... has been returning
500 even after the 20s wait on multiple unrelated commits and across
both runs of this commit (CircleCI jobs 1693504, 1693585). The PR
itself does not touch spend logs.
Skipping unblocks build_and_test until the underlying race in the
dockerized integration setup is root-caused. Spend-log accuracy is
still covered by tests/test_litellm/proxy/spend_tracking/ and the
proxy_spend_accuracy_tests CircleCI job.
---------
Co-authored-by: Kevin Zhao <zkm8093@gmail.com>
Co-authored-by: Matthew Lapointe <lapointe683@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Elon Azoulay <elon.azoulay@gmail.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: afoninsky <andrey.afoninsky@gmail.com>
Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Maruti Agarwal <88403147+marutilai@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Cursor Bugbot <bugbot@cursor.com>
Co-authored-by: Greptile <greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Greptile Reviewer <greptile-apps@users.noreply.github.com>
* refactor(bedrock/sagemaker): switch to lazy loading for response stream shapes
- Replace eager loading of BEDROCK_RESPONSE_STREAM_SHAPE and SAGEMAKER_RESPONSE_STREAM_SHAPE with lazy loading via get_bedrock_response_stream_shape() and get_sagemaker_response_stream_shape() respectively.
- This change optimizes performance by avoiding unnecessary imports and logging warnings unless the response stream shapes are actually needed.
- Update relevant classes and tests to utilize the new lazy loading functions, ensuring consistent behavior across the codebase.
* test(bedrock/sagemaker): add fixtures to clear response stream shape cache
- Introduced `_reset_bedrock_response_stream_shape_cache` and `_reset_sagemaker_response_stream_shape_cache` fixtures to prevent lru_cache leakage between tests in their respective modules.
- Updated tests to utilize these fixtures, ensuring that the response stream shape cache is cleared before and after each test run.
- Added `pytest.importorskip("botocore")` to ensure that tests are skipped if the botocore library is not available.
* fix(bedrock): sanitize batch metadata to prevent Pydantic ValidationError
Proxy guardrail hooks (Model Armor, OpenAI Moderations) and internal
processing inject non-string values (dicts, floats) into the request
metadata. When the Bedrock batch handler passes this metadata directly
to LiteLLMBatch (which inherits OpenAI's Batch Pydantic model with
metadata: Dict[str, str]), Pydantic raises a ValidationError. This
causes the router retry loop to re-submit the same Bedrock job
multiple times before ultimately failing.
Add _get_openai_compatible_batch_metadata() that serializes non-string
values to JSON strings via safe_dumps, skips None values and internal
logging keys, ensuring the response object always validates.
* test(bedrock): add tests for batch metadata sanitization
Covers _get_openai_compatible_batch_metadata: string passthrough, dict/float
serialization, None/internal key exclusion, and LiteLLMBatch compatibility.
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* fix(bedrock/cohere): wrap embedding_types as list in map_openai_params
Bedrock Cohere expects embedding_types as a JSON array but
encoding_format was passed through as a raw string, causing:
Malformed input request: #/embedding_types: expected type: JSONArray, found: String
* test(bedrock/cohere): assert embedding_types is sent as JSON array
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943)
* docs: add one-line docstring to _disable_debugging (#27894)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831)
Squash-merged by litellm-agent from Cyberfilo's PR.
* Sanitize empty text content blocks on /v1/messages (#27832)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint
The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic
Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not
Found. Both AmazonMantleConfig (chat/completions caller route) and
AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded
the wrong path, so every Mantle request 404'd before reaching the model.
Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages
API at /anthropic/v1/messages with SSE streaming."
https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock
Confirmed independently against the live endpoint:
/v1/chat/completions -> 200 OK
/v1/messages -> 404 Not Found (what litellm used)
/anthropic/v1/messages -> 200 OK (Claude only)
Adds a regression test asserting both Mantle configs build the
/anthropic/v1/messages path, and updates the existing assertions that
encoded the wrong path.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
* fix: sanitize empty text blocks in sync anthropic_messages_handler path
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(bedrock): support retrieve for model-invocation-job batch ARNs
`bedrock.retrieve_batch` previously only handled `:async-invoke/` ARNs
(Twelve Labs Marengo embeddings). The `:model-invocation-job/` ARNs
returned by `CreateModelInvocationJob` (the bulk batch inference API
behind `bedrock.create_batch`) fell through and returned a misleading
data-plane error, leaving created jobs unretrievable through the
LiteLLM batches API.
The two ARN families live on different AWS service endpoints
(`bedrock-runtime` data plane vs `bedrock` control plane), so they need
distinct handlers. This adds:
* `BedrockBatchesHandler._handle_model_invocation_job_status` — calls
the control plane via boto3 (`bedrock:GetModelInvocationJob`),
reusing `BaseAWSLLM.get_credentials` for credential resolution so
model_list / env / role-assumption configs continue to apply. The
response is reshaped into a `LiteLLMBatch` with the same status
mapping `transform_create_batch_response` already uses.
* Output-file-URI prediction. Bedrock surfaces the user-supplied
`s3OutputDataConfig.s3Uri` *prefix* in `GetModelInvocationJob`, but
results actually land at `<prefix>/<job-id>/<basename(input)>.out`.
We compute that single-file URI client-side and surface it as
`output_file_id`, so OpenAI-style `client.files.content(...)` works
without an extra `ListObjectsV2` round-trip. The bare prefix stays
in metadata for callers that want the manifest.
* Dispatch in `litellm/batches/main.py` for the new ARN family,
alongside the existing async-invoke branch.
* Unit tests covering ARN parsing, output-URI prediction (incl. edge
cases), the full status mapping, region resolution precedence, and
failure-message propagation.
Note: `request_counts` is intentionally `(0, 0, 0)` —
`GetModelInvocationJob` does not report per-record counts; getting
accurate numbers requires parsing `manifest.json.out` from the output
S3 prefix, which is left to callers.
Made-with: Cursor
* fix(bedrock): address PR feedback on model-invocation-job retrieve
Addresses Greptile P2 findings on #26834:
1. Use the bare job id (not the full ARN) when constructing the
`api_base` URL for `pre_call` logging. Passing the full ARN double-
counts the `model-invocation-job/` segment and embeds colons in the
path, producing misleading log lines.
2. Drop the `or output_prefix` fallback when `_predict_output_file_uri`
returns None. A bare prefix is not a downloadable object and surfacing
it as `output_file_id` re-creates the very NoSuchKey bug this handler
exists to fix. The bare prefix is still preserved in
`metadata["output_s3_uri"]` for callers that want to do their own S3
listing or read `manifest.json.out`.
`metadata["output_file_uri"]` uses "" rather than None to satisfy the
OpenAI Batch metadata schema (`dict[str, str]`); callers should branch
on the typed `output_file_id` field instead.
Also expands test coverage on the new code path:
- new "stay None" regression test for the prediction-fail case
- pre_call/post_call logging hook assertions (incl. the bare-id URL)
- explicit cancelled_at / expired_at coverage
- _to_epoch type-handling matrix and the boto3 ImportError branch
- defensive _extract_region_from_bedrock_arn exception path
- empty-basename case for _predict_output_file_uri
Patch coverage on the changed lines is now 100% (the only remaining
uncovered lines in the file belong to the pre-existing
`_handle_async_invoke_status` method, which this PR does not touch).
Made-with: Cursor
* test(bedrock): cover retrieve_batch dispatch for both ARN families
Codecov flagged 8 uncovered lines on `litellm/batches/main.py` after
this PR refactored the Bedrock dispatch into a single guard with two
sub-branches (`async-invoke` + `model-invocation-job`). Existing tests
exercised the handlers directly but not the dispatch in `main.py`.
Adds `tests/test_litellm/batches/test_retrieve_batch_bedrock_dispatch.py`
with 6 mocked tests that exercise `litellm.retrieve_batch` end-to-end
for the dispatch logic:
- async-invoke ARN routes to `_handle_async_invoke_status`
- async-invoke ARN with no region falls back to "us-east-1" (preserves
prior behavior on this branch)
- model-invocation-job ARN routes to the new
`_handle_model_invocation_job_status` handler
- model-invocation-job ARN with no region forwards None (so the new
handler can sniff region from the ARN itself, rather than getting
silently routed to us-east-1)
- unrelated bedrock ARN family falls through to the generic
provider-config retrieve path (neither special handler invoked)
- non-bedrock batch ids skip the bedrock dispatch entirely
Both handlers are mocked at the import site so the tests don't hit
AWS — the focus here is purely the new dispatch logic in main.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): move retrieve_batch dispatch test to tests/test_litellm/
The dispatch test landed under `tests/test_litellm/batches/`, a new
directory that no upstream `test-unit-*.yml` workflow's `test-path`
allow-list includes. As a result, the test was never executed in CI
and codecov reported `litellm/batches/main.py` patch coverage at
11.11% (8 lines uncovered) — the lines belonging to this PR's
dispatch refactor itself.
Move the file up one level so it matches the
`tests/test_litellm/test_*.py` glob that `test-unit-misc.yml`
already runs, and adjust `sys.path.insert` for the new depth.
The companion handler tests under
`tests/test_litellm/llms/bedrock/batches/test_handler.py` are
unaffected — they're picked up by the `llms` directory in
`test-unit-llm-providers.yml`.
Made-with: Cursor
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Refactor Bedrock response stream shape handling
- Introduced a module-level constant `BEDROCK_RESPONSE_STREAM_SHAPE` to cache the response stream shape, eliminating the need for per-instance caching in `BedrockEventStreamDecoderBase`.
- Updated relevant methods to utilize the new constant, improving performance by avoiding redundant loading of the shape.
- Added tests to ensure the shape is loaded correctly at import time and is consistent across different modules.
- Added a new mock server script for testing Bedrock pass-through functionality.
* Refactor response parsing for Bedrock and SageMaker
- Improved code readability by formatting the parsing method calls in `AWSEventStreamDecoder` for both Bedrock and SageMaker response stream shapes.
- Added blank lines for better separation of code blocks in `invoke_handler.py` and `common_utils.py` to enhance maintainability.
* Enhance error handling for Bedrock and SageMaker response stream shape loading
- Wrapped the loading logic in `_load_bedrock_response_stream_shape` and `_load_sagemaker_response_stream_shape` with try-except blocks to gracefully handle exceptions.
- Added logging to warn when the response stream shape cannot be pre-loaded, ensuring the module imports cleanly.
- Updated tests to verify that loading failures return `None` instead of propagating exceptions.
* Implement error handling for missing response stream shapes in Bedrock and SageMaker
- Added checks in `_parse_message_from_event` methods to raise appropriate errors when `BEDROCK_RESPONSE_STREAM_SHAPE` or `SAGEMAKER_RESPONSE_STREAM_SHAPE` is None, ensuring clearer error reporting.
- Updated logging messages to reflect the unavailability of event-stream decoding for both Bedrock and SageMaker.
- Enhanced unit tests to verify that the correct exceptions are raised when the response stream shapes are not loaded.
* refactor(BaseAWSLLM): implement shared IAM cache and static credential caching
- Introduced a process-wide shared IAM cache to optimize credential management across instances.
- Added a method to handle caching of static credentials, ensuring only long-lived credentials are cached.
- Updated the get_credentials method to utilize the new caching mechanism for static credential flows.
- Enhanced unit tests to verify the correct behavior of the shared cache and static credential usage.
* refactor(BaseAWSLLM): enhance IAM credential caching and update related tests
- Improved the process-wide IAM credential caching mechanism to better handle static and AssumeRole credentials.
- Renamed the caching method for clarity and updated comments to reflect the new caching behavior.
- Added a fixture to ensure the IAM cache is flushed between tests to prevent leakage of cached entries.
- Updated unit tests to verify the correct behavior of the shared IAM cache, particularly for static credentials and role assumptions.
* refactor(BaseAWSLLM): clarify IAM credential caching behavior and enhance tests
- Updated documentation to specify that only static and ambient environment credentials are cached, excluding AssumeRole and other credential types.
- Modified the caching logic to ensure that AssumeRole credentials are not stored in the IAM cache, requiring STS calls for each request.
- Enhanced unit tests to verify that AssumeRole credentials are not cached and to ensure proper behavior of the IAM cache across different scenarios.
* Code Readability improvement for aws auth path
* refactor(BaseAWSLLM): enhance IAM credential caching documentation and add tests
- Updated comments to clarify the behavior of the in-process IAM credential cache, specifying the TTL for static and ambient credentials.
- Added new unit tests to verify the caching behavior for ambient environment credentials across instances and ensure that static access key sessions are constructed only once when cached.
- Ensured that temporary session tokens and AWS profiles are not cached, validating the expected behavior through additional tests.
* refactor(BaseAWSLLM): improve IAM credential handling and add tests for role assumption
- Updated comments to clarify the behavior of IAM credential caching, particularly regarding the handling of ambient credentials and role assumptions.
- Enhanced unit tests to verify that the caching mechanism correctly distinguishes between already running roles and new role assumptions, ensuring that cached environment credentials are not reused incorrectly.
- Added a new test case to validate the behavior when switching roles, confirming that the system correctly uses AssumeRole when the role changes.
Strip out the explanatory and historical comments that don't carry
business-logic justification. Comments that simply narrate what code
does — or that explain prior behavior, what was changed, or which PR
introduced a fix — are removed. Docstrings are reduced to a one-line
summary where the long form repeated information already evident from
the code or test data.
No code-behavior changes. All 643 affected unit tests still pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- claude-sonnet-4-6 + reasoning_effort=max no longer 400s. Renamed
_is_opus_4_6_model to _is_claude_4_6_model at three sites and added
supports_max_reasoning_effort: true to 12 model entries in the JSON
cost map (10 sonnet 4.6 ids + OpenRouter opus 4.6/4.7).
- _map_reasoning_effort now raises BadRequestError(400) directly with
llm_provider, instead of letting Databricks (and similar callers)
surface its raw ValueError as a 500.
- output_config.effort on Opus 4.5 over Bedrock no longer 400s for
missing effort-2025-11-24 beta. Flipped JSON to "effort-2025-11-24"
for bedrock + bedrock_converse and added an auto-attach branch in
_process_tools_and_beta for non-adaptive Anthropic + output_config
on Converse.
- reasoning_effort=xhigh / =max on legacy budget-mode models
(Haiku 4.5, Sonnet 4.5, Opus 4.5) now map to thinking.budget_tokens
8192 / 16384 instead of returning 400. Added two constants in
litellm/constants.py.
Tests updated for all four flips. Validated end-to-end via 306-cell
live proxy matrix (6 model families x 3 routes x 17 effort cases),
all pass.
When a proxy fronts Claude Code (which always sends `output_config.effort`)
at a pre-4.5 Anthropic model — haiku-3, sonnet-3.5, opus-3, sonnet-4 — the
forwarded knob causes a forced 400 the client can't fix. Gating a strip
behind the existing `drop_params` flag lets operators opt into silent
fixup once and stop worrying about per-model param hygiene.
Default (`drop_params=False`) still forwards and surfaces the provider's
error, preserving the strict, debuggable contract from #27074.
Per https://platform.claude.com/docs/en/build-with-claude/effort the
supporting set is Opus 4.5+, Sonnet 4.6+, and Mythos Preview; everything
else is dropped (with a verbose_logger warning so the strip is visible).
Recognition uses model-name patterns plus a fallback to any
`supports_*_reasoning_effort` flag in the model map for forward
compatibility with new entries.
https://claude.ai/code/session_01WjHq31rvXT6xYNdVmSJvRp
(cherry picked from commit 1233943e78)
Closes the remaining QA-sweep gap on PR #27074: Bedrock Invoke
/v1/messages was silently ignoring ``reasoning_effort`` because the
shared param filter only kept native Anthropic keys, so every effort
tier collapsed to the same behavior on the wire (27/231 cells failing
across opus-4-5 / opus-4-6 / sonnet-4-6).
Map ``reasoning_effort`` to native Anthropic ``thinking`` /
``output_config.effort`` at the ``AnthropicMessagesConfig`` layer so
all four /v1/messages routes (direct Anthropic, Azure AI, Vertex AI,
Bedrock Invoke) inherit the same translation:
- Add ``reasoning_effort`` to ``AnthropicMessagesRequestOptionalParams``
so the param filter in
``AnthropicMessagesRequestUtils.get_requested_anthropic_messages_optional_param``
no longer drops it before the transformation runs.
- Add ``_translate_reasoning_effort_to_anthropic`` and call it from
``transform_anthropic_messages_request``. Mirrors
``AnthropicConfig.map_openai_params`` on the chat completion path
(re-uses ``_map_reasoning_effort`` and
``REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT``) so the two routes
cannot drift. Pops ``reasoning_effort`` so it never reaches the wire.
- Caller-supplied native ``thinking`` / ``output_config.effort`` always
win — same precedence as
``_translate_legacy_thinking_for_adaptive_model``.
- Garbage values (``""``, ``"disabled"``, ``"invalid"``) raise
``AnthropicError(status_code=400)`` instead of falling through and
surfacing as 500s from the provider.
- ``"none"`` clears thinking + output_config so callers can opt out
per request.
Also restores the non-adaptive-model test coverage on Bedrock Invoke
/v1/messages that the previous commit lost when
``test_bedrock_messages_strips_output_config`` was renamed to the
``forwards`` variant on Opus 4.7.
Adds a new test file
``test_reasoning_effort_translation.py`` covering the translation at
the shared config level (adaptive + non-adaptive models, none, garbage,
caller precedence) so all four /v1/messages routes are exercised by a
single suite.
Adds parametrized + behavioral tests on the Bedrock Invoke /v1/messages
suite covering: minimal/low/medium/high/xhigh/max mapping for adaptive
models, thinking-budget mapping for non-adaptive Opus 4.5, ``none``
clears both, garbage raises 400, explicit ``output_config`` wins.
Refs: https://github.com/BerriAI/litellm/pull/27074
Follow-up bugs surfaced by the QA sweep on PR #27039
(https://github.com/BerriAI/litellm/pull/27039#issuecomment-4363363610).
1. Stop stripping output_config.effort on Bedrock + Vertex adaptive routes.
- Vertex AI Claude 4.6/4.7 accepts output_config.effort on rawPredict
(verified end-to-end against us-east5 / global). The strip helper now
no-ops for effort.
- Bedrock Converse routes output_config into additionalModelRequestFields
for anthropic base models so the requested adaptive tier (low/medium/
high/xhigh/max) actually reaches the wire instead of all collapsing to
identical thinking.
- Bedrock Invoke chat transformation (AmazonAnthropicClaudeConfig) stops
popping output_config from the post-AnthropicConfig request body.
- Bedrock Invoke /v1/messages allowlist (BedrockInvokeAnthropicMessagesRequest)
now lists output_config so the runtime allowlist filter forwards it.
2. Validate effort across Bedrock Converse so 'disabled' / 'invalid' / '' /
unsupported tiers (xhigh/max on Sonnet 4.6 or budget-mode 4.5 models)
surface as a clean 400 BadRequestError instead of 500.
3. ValueError -> BadRequestError throughout (AnthropicConfig.map_openai_params,
_apply_output_config, AmazonConverseConfig._handle_reasoning_effort_parameter).
Empty-string effort is now rejected (was silently passing the
'if effort and ...' short-circuit).
4. Floor reasoning_effort='minimal' at the Anthropic provider minimum
(1024 budget_tokens) via new ANTHROPIC_MIN_THINKING_BUDGET_TOKENS so it's
a usable tier on direct Anthropic / Azure AI Anthropic / Vertex AI Anthropic /
Bedrock Invoke (all of which 400 below 1024).
5. model_prices: dedupe duplicate supports_max_reasoning_effort key on
claude-opus-4-7 / claude-opus-4-7-20260416.
Adds regression tests across all five affected paths; existing tests asserting
the silent-strip behavior were updated to reflect the new pass-through and
clean 400 surfaces.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Setting reasoning_effort="none" on Anthropic chat models (direct, Bedrock
Invoke, Bedrock Converse, Vertex AI Anthropic, Azure AI Anthropic) crashed
LiteLLM with:
litellm.APIConnectionError: 'NoneType' object has no attribute 'get'
Both the Anthropic chat transformation and Bedrock Converse called
``AnthropicConfig._map_reasoning_effort`` and assigned the ``None`` it returns
for ``"none"`` directly to ``optional_params["thinking"]``. Downstream
``is_thinking_enabled`` then did ``optional_params["thinking"].get("type")``
and crashed.
Pop ``thinking`` (and on Claude 4.6/4.7, ``output_config``) instead of
assigning ``None``, restoring the documented contract that
``reasoning_effort="none"`` means "do not enable thinking". This also
prevents downstream Anthropic 400s ("thinking: Input should be an object",
"output_config.effort: Input should be ...") if the bug were ever masked.
Verified end-to-end against the live Anthropic API and Bedrock Converse
on claude-opus-4-{5,6,7} and claude-sonnet-4-6, plus Bedrock Invoke for
Claude 4.5/4.6. Vertex AI Anthropic and Azure AI Anthropic inherit the
fixed ``map_openai_params`` from ``AnthropicConfig`` and need no further
changes.
Resolves merge conflict in tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py
by keeping both the new bedrock tool-result file/document tests and the
transform_response body-leak regression test.
Also addresses Greptile P2 comment: when BedrockImageProcessor returns a
block with neither 'image' nor 'document' keys on the tool-result path
(image_url and file content types), log a warning instead of silently
dropping the block.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`,
returning 404s with "This model version has reached the end of its life."
Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability
surface: thinking, tools, prompt caching, PDF input, vision, computer use).
The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5
is converse-only on Bedrock.
- Tighten _is_anthropic_document_data_uri to match the mimes Anthropic
actually accepts as base64 `document` source ({application/pdf,
text/plain}). The previous application/* + text/* prefix match would
route e.g. data:application/json URIs through the document path,
producing blocks the Anthropic API rejects. Unsupported mimes now
stay on the existing image code path (same failure mode as before the
fix — no regression, just stops introducing a new one).
- On the Bedrock tool-result `type: "file"` branch, accept either
file_data or file_id and raise BadRequestError on both-None, mirroring
the user-message _process_file_message pattern. Previously a file
block with only file_id was silently dropped.
- Consolidate the six new PDF tool-result tests under tests/test_litellm/
only (the PR template's required location and where the unit-test CI
workflow runs with coverage). The duplicate copies under
tests/llm_translation/ added drift risk with no additional coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Duplicate the three Bedrock and three Anthropic tool-result tests into
tests/test_litellm/ so they're picked up by `make test-unit` (and its
coverage report). The originals in tests/llm_translation/ stay — they
run under integration and remain the canonical translation-suite
regression cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Route vector store search `extra_body` into provider transformers and handle Bedrock `retrievalConfiguration` explicitly so only intended provider-specific fields are forwarded.
Made-with: Cursor
Bedrock enforces non-increasing TTL ordering across cache_control blocks
(tools → system → messages). The tool cache_control TTL was being
unconditionally dropped to the default 5m, while system blocks preserved
the user-specified TTL for Claude 4.5+ models. This mismatch caused
"a ttl='1h' block must not come after a ttl='5m' block" errors when
users set ttl='1h' on both tools and system.
Converse path: add_cache_point_tool_block() now accepts a model param
and preserves TTL for Claude 4.5+, matching _get_cache_point_block().
Invoke path: _remove_ttl_from_cache_control() now also processes tools
(was only processing system and messages).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fail-safes for the /v1/messages → Bedrock Invoke pass-through so new
Anthropic-only extensions Claude Code starts sending can't reach Bedrock
and trigger a 400 "Extra inputs are not permitted":
1. Top-level body fields are filtered to a typed allowlist. New
`BedrockInvokeAnthropicMessagesRequest` TypedDict (in
`litellm/types/llms/bedrock.py`) captures the Bedrock Invoke Anthropic
Messages body schema; the runtime allowlist is derived from its
`__annotations__` so the type and the filter can't drift. Anchored to
the AWS reference page in docstrings + transform comment. An
exact-set test pins the resolved allowlist so any future edit forces
conscious review.
Drops context_management, output_config, speed, mcp_servers,
container, inference_geo, internal litellm_metadata, and any future
Anthropic addition. output_format stays as an active inline-schema
conversion (not just a strip).
2. The anthropic-beta header list is filtered + transformed against the
bedrock mapping for ALL betas, not just auto-injected ones. The
previous code union'd user-provided betas back in unfiltered, so a
client on a new Anthropic-direct beta (e.g. advisor-tool-…,
context-management-…) could still pin the request to fail. In a proxy
context the client can't know the backend is Bedrock; the provider
mapping is authoritative. User-provided drops are logged at WARNING
so intentional overrides leave a breadcrumb.
Updates one existing test that happened to assert on the old buggy
pass-through (it used output-128k-2025-02-19, which is null in the
bedrock mapping and would 400 at runtime); rewrote it against a
bedrock-supported beta.
Scope: messages/invoke only. The same user-beta bypass exists in
chat/invoke but that's a different code path with different
user-expectation trade-offs — follow-up.
Fixes SyntaxError at pytest collection time caused by leftover
<<<<<<<, =======, >>>>>>> markers in test_bedrock_common_utils.py.
Keeps the assertion matching the model under test
(claude-haiku-4-5-20251001-v1:0).
Drop test_bedrock_invoke_messages_injects_thinking_for_clear_thinking_context_management.
Its assertion 'interleaved-thinking-2025-05-14' in betas cannot hold because
anthropic_beta_headers_config.json maps that header to null for the bedrock
provider, so filter_and_transform_beta_headers drops it from the auto-added
beta set before anthropic_beta is written to the request.
The adjacent test_bedrock_invoke_messages_skips_thinking_injection_when_already_enabled
already covers the inverse behavior for the same model, so no coverage is lost.
