The agent-runtime branch of /bedrock/{endpoint} (agents, knowledgebases, flows,
retrieveAndGenerate, rerank, generateQuery, optimize-prompt) forwarded every
caller header to AWS next to the SigV4 signature, so a LiteLLM key presented in
x-api-key or x-litellm-api-key reached bedrock-agent-runtime verbatim. Build the
upstream header set explicitly: drop LiteLLM credential headers by name and any
authenticated secret by value, keep the remaining caller headers, and let the
signed headers win on collisions.
Without a master key the auth layer echoes whatever key the caller presented as the authenticated key, so the passthrough's strip-by-value matched the caller's own Anthropic key and dropped it: a bring-your-own-key request that returned 200 on main answered 401 telling the caller to send the key they had just sent. Only the auth module's own no-auth dev-mode definition, shared through is_no_auth_dev_mode, decides that nothing was authenticated, and only when no custom auth is installed; JWTs, OAuth2 tokens, and custom-auth credentials are still stripped there. The sk- prefix heuristic goes with it.
The Vertex credential-less test now sets a master key, since a virtual key can only authenticate under one: the auth layer returns before any key lookup when the master key is unset.
Pass user_api_key_dict.parent_otel_span into the outgoing W3C injection so the
legacy otel callback propagates its litellm_request span, falling back to the
otel_v2 request root span and then the ambient span. Extend the mapped unit
tests to assert the propagated trace and span ids over real captured headers
for HTTP and WebSocket passthrough with forwarding on and off.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The /anthropic/{endpoint} route forwarded every incoming header upstream, so
the header carrying the caller's LiteLLM virtual key (Authorization, x-api-key,
x-litellm-api-key, or the operator-configured key header) reached Anthropic and
was rejected there as an invalid credential, with or without a proxy-side
Anthropic key layered on top.
Share the Vertex credential-less header filter: drop the proxy-only credential
headers by name, drop the value that authenticated the caller (virtual key,
master key, or JWT) from Authorization / x-api-key, keep a caller's own
Anthropic credential, layer the proxy's Anthropic credential on top, and fail
with a clean 401 when neither the proxy nor the caller supplied one.
Resolves LIT-3550
Route Vertex and Gemini /interactions create responses through the shared
Interactions usage transformation so SpendLogs carry per-modality tokens and
cost instead of zeros, and carry parameters.sampleCount into Veo usage as
video_count so duration-based video cost is multiplied by the number of
generated videos
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Vertex passthrough route resolved a router deployment only to rewrite the
upstream URL and dropped its model_info, so the standard logging payload and
the Prometheus litellm_deployment_success_responses_total counter carried
model_id="". Carry the deployment's model_info through request.state into the
passthrough logging metadata, where it overrides any client-supplied model_info.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A client that named a bare gateway alias logged the session as "unknown" and billed nothing,
because the model was read off the raw setup frame and the extractor only yields a name when the
string already contains "/models/". The rewriter qualifies that same model a few lines later for
the upstream, so the supported client form, an alias, was the one that went unbilled.
Resolving through the rewriter first means the real model reaches the logging object, and from
there the cost map. A route with no rewriter, which is every non-Live passthrough, hands the frame
over untouched.
(cherry picked from commit 573982803df612fd94144e2e06dd647f8530d4e8)
error_status_code only read status_code, so a ProxyException raised
before routing (which stores its status as the string code) answered
500 with its 4xx type through the rerank, images, realtime, files, and
pass-through tails. It now falls back to a decimal code. A 408 maps to
timeout_error instead of invalid_request_error.
Tail regressions for rerank, images, realtime calls, and the chat
pass-through fail at the merge base with ('None', 'None'); the new
files-test helpers are fully typed.
SigV4 signing resolves AWS credentials, and botocore refreshes expiring
credentials inside that signing with a blocking HTTP call. Every async
Bedrock path that still signed on the event loop (/v1/messages, Converse,
count tokens, the agent-runtime and Comprehend Medical pass-throughs,
async-invoke status polling, realtime, AgentCore, SQS, S3) now signs on a
worker thread, so one Bedrock request no longer stalls the whole worker.
Fixes#40165
Wrap the context-management editors, the end-of-stream chunk builder,
acount_tokens, the compression interception hook, the passthrough
interrupted-stream recovery, the A2A usage counters, and the semantic
cache embedding truncation in asyncify so a multi-megabyte payload no
longer stalls the worker's event loop while it is tokenized
The pass-through suite now drains the process-global logging worker
from an autouse conftest fixture so work queued on one test's loop
cannot fire against the next test's callbacks
Resolves LIT-7190
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep x-amzn-RequestId on chat error responses
Bedrock chat error paths built BedrockError from only a status code and a
message, so the provider response headers were gone before exception mapping
ran and the proxy had nothing to forward. AWS support needs x-amzn-RequestId
to investigate a server-side error.
- converse and invoke chat handlers pass the real headers and response when
they turn an httpx.HTTPStatusError into a BedrockError, and read the body
through error_response_text so a streamed body nobody read does not throw
- every bedrock chat get_error_class honors the headers it is already handed:
invoke, moonshot, bedrock-hosted openai, agentcore and the invoke agent
- BedrockError carries those headers into the response it synthesizes when a
caller has headers but no response, skipping values httpx cannot carry
- the bedrock 500 mapping forwards the provider response like its 4xx and 503
siblings instead of fabricating a blank one
The proxy now returns llm_provider-x-amzn-requestid on Bedrock chat errors.
* fix(bedrock): keep request-id on text-classified errors
The context-window and image branches of _map_bedrock_exception built their
litellm exception without the provider response, so a Bedrock 400 classified
by its body text lost x-amzn-RequestId while the sibling branches kept it.
Also narrows the new BedrockError types and trims its docstrings.
* chore(bedrock): drop the docstrings on the new error helpers
* fix(bedrock): keep request-id on every error path that has one
The ticket's root cause is that every BedrockError raise site under
litellm/llms/bedrock/ was built from status and message alone. The first
commits covered the chat and invoke handlers; this covers the rest.
Embeddings, rerank, image generation, image edit, count tokens, search and
the transformation layers now hand on the provider response or its headers,
and both bedrock_mantle configs return a BedrockError instead of the
OpenAI error that drops them.
Two blockers surfaced while verifying the streaming path. The trailing
`except Exception` in make_call and make_sync_call swallowed the BedrockError
raised a few lines above, relabelling a provider status as a 500, and the
non-200 branch read an unread streamed body, which throws.
The raise sites left alone have no provider response to carry: timeouts,
credential and config errors, and mid-stream event frames.
* fix(bedrock): forward provider headers from the count tokens route
The count tokens route converts BedrockError into an HTTPException, and dropped
the headers the handler had just kept, so that route still lost the request id.
get_response_headers now takes a Mapping so an httpx.Headers can be handed to it
without a copy.
* fix(bedrock): classify every bedrock surface through BedrockError
Eleven bedrock configs still inherited a provider-agnostic get_error_class
that builds a blank response, so the request id was gone before the proxy
read it. Claude platform, bedrock anthropic-messages, both image edit
configs, passthrough, realtime, vector stores and agentcore search now
return BedrockError, and a parametrized audit drives all 36 configs.
* fix(proxy): keep provider headers on the httpx status error branch
_handle_llm_api_exception forwards safe_headers on every branch except the
httpx.HTTPStatusError one, which the bedrock passthrough route reaches, so
the request id was dropped before the client saw the response.
* fix(bedrock): keep the request id on the timeout mappings
Timeout takes no response argument, so the three bedrock timeout branches
dropped the provider headers even when the upstream answered 408 or 504
with an x-amzn-RequestId. They now ride on the exception, already
llm_provider-prefixed, which is the form the proxy emits.
* fix(bedrock): keep the provider response on mapped timeouts
The previous round attached llm_provider-prefixed headers directly to the
Timeout. That shadowed the raw upstream headers for _get_response_headers,
so router cooldown and fallback cooldown stopped honouring retry-after on
bedrock 408/504 replies.
Give Timeout an optional response instead, the way every other mapped
bedrock exception already carries one. Retry logic reads the raw
retry-after off the response, and the proxy prefixes those headers on the
way out, so clients still see llm_provider-x-amzn-requestid.
* chore(bedrock): drop the explanatory comment on Timeout.response
Drop the second pricing pass on interrupted /v1/messages streams: clearing
the stale usage.cost and hidden response_cost is enough for the existing
success and failure logging to price the recovered usage. Add an iterator
test for the upstream-close path the proxy takes on a client disconnect.
Streaming chat relays on Azure and azure_ai deployments rebuild the response from
the SSE chunks through the OpenAI passthrough assembler, so the spend log carries
usage. The router relays keep the JSON body when the Content-Type carries a
charset, return the upstream status and body instead of a 500 when the deployment
rejects the call, and fall back to the caller's api-version when the deployment
sets none. Lint budgets ratcheted to the measured totals
