Fifty six of the deleted tests turn out to assert the output of litellm code rather than the catalog lookup itself, things like map_openai_params, get_supported_openai_params, should_fake_stream, transform_request bodies, cost_per_token arithmetic, get_llm_provider routing, and provider config dispatch. They only happen to read shipped entries as inputs, so they belong in the later rewrite that injects a local model_cost, not in this deletion
Each one is restored verbatim from origin/main along with the fixtures, helpers, constants and imports it needs, and tests/test_litellm/test_sambanova_model_metadata.py is restored wholesale
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Regenerated every touched file from origin/main applying only the B1 test deletions and the unused import and helper cleanup they leave behind, without running the formatter across untouched code. CI only checks ruff format under litellm/, so the earlier reflows of test files were pure diff noise for reviewers
Also drops the tests/local_testing/test_prompt_caching.py entry from the caching-local shard in test-unit.yml since that file is deleted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The repo rule is that a test must only fail when litellm code changes, never when a vendor updates a price, renames a field, or drops a model. These tests asserted shipped catalog entries directly, comparing lookup results to literals copied from model_prices_and_context_window.json or requiring named entries to exist or be absent, so every cost map sync could break them without any litellm code changing
Tests that exercise real litellm behavior with an injected local model_cost, invariants like backup parity, and assertions on non-lookup code paths are untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Bedrock InvokeModel transformations decided whether to send the
tool-search-tool-2025-10-19 beta from hardcoded model name lists (a pattern
list on the messages path, an "opus-4" substring on the chat path), so Opus 4.8,
Opus 5 and Sonnet 5 never got the beta on the messages path, Opus 5 and Sonnet 5
never got it on the chat path, Opus 4.1 got it without support, and
/v1/model/info reported supports_tool_search as unset for all three.
Both paths now read the model map through one shared helper: the Bedrock
entries for Opus 4.8, Opus 5 and Sonnet 5 carry supports_tool_search
explicitly, and a claude-tool-search fallback rule flags Claude 4.5 and newer
for unmapped ids, inference-profile ARNs and mapped entries with no opinion,
so the next Claude gets the beta with no code change. An explicit false on a
resolved entry still wins.
Each route now has litellm/rust_bridge/<route>/{entrypoints,callbacks}.py and a
public dispatch module (litellm/chat_completions/dispatch.py,
litellm/responses/dispatch.py, litellm/messages/dispatch.py) that binds the
public call to the legacy Python signature, builds a frozen request, and asks
the runtime to pick Rust or Python from the catalog. The legacy implementations
stay in litellm/main.py, litellm/responses/main.py and the anthropic messages
handler, and litellm/__init__.py re-exports the dispatch names over them the
same way it already does for ocr
The per-handler shims in rust_bridge/chat_completions/native.py and
rust_bridge/messages/native.py are removed along with their call sites in the
anthropic and bedrock chat handlers and the http handler. The exception
mapping that every callbacks module repeated moves to rust_bridge/failures.py
and the signature binding helpers to rust_bridge/public_call.py
Move each route's bridge module under litellm/rust_bridge/<route>/ so a folder
means a Rust implementation exists while the catalog row says whether it is
used. OCR now keeps the Python implementation in litellm/ocr/main.py and the
Rust selection in litellm/ocr/rust.py, removing litellm/ocr/legacy.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Nova InvokeModel rejects the standalone cachePoint blocks the shared Converse transform emits, so each one is folded into the block it caches and tool_config injection points are dropped before the transform runs, since this route has no tool caching to credit. Usage reads Bedrock's Count-suffixed cache keys and adds cached tokens into prompt_tokens, streaming routes every wrapped InvokeModel event through the Converse chunk parser and tolerates the missing totalTokens, and the Nova 1 cost-map entries gain cache_read_input_token_cost at a quarter of the input rate
The converse reasoning gate only matched openai.gpt-5, so gpt-6-astra fell through to
Anthropic's thinking block and Bedrock rejected the call with 400 Unknown parameter:
'thinking'. Match any openai.gpt-<digit> model at the three gate sites instead.
Nova 2 lite and pro accept forced tool_choice on Converse (verified live on
us.amazon.nova-2-lite-v1:0), so the nine Nova 2 registry keys now advertise
supports_tool_choice. The invoke dispatcher also forwards json_mode to Nova like it
already does for Anthropic and TwelveLabs.
Operators can now set aws_session_tags on a Bedrock or SageMaker
deployment and every AssumeRole call carries them as STS session tags,
so trust policies gated on sts:TagSession admit the session and
CloudTrail and cost reports see the tags. Tags are validated up front,
sorted into the credential cache key so tag order does not fork
sessions, stripped from invoke and embedding request bodies, and
blocked from client request bodies like the other AWS identity params.
Based on #34073.
Co-authored-by: Satya Yedida <222552552+satya-rubrik@users.noreply.github.com>
SigV4 signing resolves AWS credentials, and botocore refreshes expiring
credentials inside that signing with a blocking HTTP call. Every async
Bedrock path that still signed on the event loop (/v1/messages, Converse,
count tokens, the agent-runtime and Comprehend Medical pass-throughs,
async-invoke status polling, realtime, AgentCore, SQS, S3) now signs on a
worker thread, so one Bedrock request no longer stalls the whole worker.
Fixes#40165
* fix(bedrock): keep x-amzn-RequestId on chat error responses
Bedrock chat error paths built BedrockError from only a status code and a
message, so the provider response headers were gone before exception mapping
ran and the proxy had nothing to forward. AWS support needs x-amzn-RequestId
to investigate a server-side error.
- converse and invoke chat handlers pass the real headers and response when
they turn an httpx.HTTPStatusError into a BedrockError, and read the body
through error_response_text so a streamed body nobody read does not throw
- every bedrock chat get_error_class honors the headers it is already handed:
invoke, moonshot, bedrock-hosted openai, agentcore and the invoke agent
- BedrockError carries those headers into the response it synthesizes when a
caller has headers but no response, skipping values httpx cannot carry
- the bedrock 500 mapping forwards the provider response like its 4xx and 503
siblings instead of fabricating a blank one
The proxy now returns llm_provider-x-amzn-requestid on Bedrock chat errors.
* fix(bedrock): keep request-id on text-classified errors
The context-window and image branches of _map_bedrock_exception built their
litellm exception without the provider response, so a Bedrock 400 classified
by its body text lost x-amzn-RequestId while the sibling branches kept it.
Also narrows the new BedrockError types and trims its docstrings.
* chore(bedrock): drop the docstrings on the new error helpers
* fix(bedrock): keep request-id on every error path that has one
The ticket's root cause is that every BedrockError raise site under
litellm/llms/bedrock/ was built from status and message alone. The first
commits covered the chat and invoke handlers; this covers the rest.
Embeddings, rerank, image generation, image edit, count tokens, search and
the transformation layers now hand on the provider response or its headers,
and both bedrock_mantle configs return a BedrockError instead of the
OpenAI error that drops them.
Two blockers surfaced while verifying the streaming path. The trailing
`except Exception` in make_call and make_sync_call swallowed the BedrockError
raised a few lines above, relabelling a provider status as a 500, and the
non-200 branch read an unread streamed body, which throws.
The raise sites left alone have no provider response to carry: timeouts,
credential and config errors, and mid-stream event frames.
* fix(bedrock): forward provider headers from the count tokens route
The count tokens route converts BedrockError into an HTTPException, and dropped
the headers the handler had just kept, so that route still lost the request id.
get_response_headers now takes a Mapping so an httpx.Headers can be handed to it
without a copy.
* fix(bedrock): classify every bedrock surface through BedrockError
Eleven bedrock configs still inherited a provider-agnostic get_error_class
that builds a blank response, so the request id was gone before the proxy
read it. Claude platform, bedrock anthropic-messages, both image edit
configs, passthrough, realtime, vector stores and agentcore search now
return BedrockError, and a parametrized audit drives all 36 configs.
* fix(proxy): keep provider headers on the httpx status error branch
_handle_llm_api_exception forwards safe_headers on every branch except the
httpx.HTTPStatusError one, which the bedrock passthrough route reaches, so
the request id was dropped before the client saw the response.
* fix(bedrock): keep the request id on the timeout mappings
Timeout takes no response argument, so the three bedrock timeout branches
dropped the provider headers even when the upstream answered 408 or 504
with an x-amzn-RequestId. They now ride on the exception, already
llm_provider-prefixed, which is the form the proxy emits.
* fix(bedrock): keep the provider response on mapped timeouts
The previous round attached llm_provider-prefixed headers directly to the
Timeout. That shadowed the raw upstream headers for _get_response_headers,
so router cooldown and fallback cooldown stopped honouring retry-after on
bedrock 408/504 replies.
Give Timeout an optional response instead, the way every other mapped
bedrock exception already carries one. Retry logic reads the raw
retry-after off the response, and the proxy prefixes those headers on the
way out, so clients still see llm_provider-x-amzn-requestid.
* chore(bedrock): drop the explanatory comment on Timeout.response
AmazonMoonshotConfig.transform_request called
_get_boto_credentials_from_optional_params purely for its side effect of
popping the aws_* keys off optional_params, then threw the result away. On
a box whose default AWS profile uses login_session without botocore[crt],
that call raises, so a bearer-token bedrock/invoke/moonshot.* deployment
still 500s with MissingDependencyException even after the rest of this
branch skips the chain.
It now filters the aws_* keys into a local dict the way the Qwen, OpenAI
and Claude 3 invoke transformations already do, so no credentials are
resolved and the caller's optional_params keeps the keys sign_request
reads afterwards.
A deployment authenticating with api_key or AWS_BEARER_TOKEN_BEDROCK still ran
boto3's credential chain before every call, so an unloadable default profile
(a login_session profile without botocore[crt]) made Converse, embeddings,
image generation, image edit, and the Bedrock guardrail hook fail with
MissingDependencyException even though the bearer token alone signs the
request. The chain now runs only when no bearer token is configured
