* fix(bedrock): keep x-amzn-RequestId on chat error responses
Bedrock chat error paths built BedrockError from only a status code and a
message, so the provider response headers were gone before exception mapping
ran and the proxy had nothing to forward. AWS support needs x-amzn-RequestId
to investigate a server-side error.
- converse and invoke chat handlers pass the real headers and response when
they turn an httpx.HTTPStatusError into a BedrockError, and read the body
through error_response_text so a streamed body nobody read does not throw
- every bedrock chat get_error_class honors the headers it is already handed:
invoke, moonshot, bedrock-hosted openai, agentcore and the invoke agent
- BedrockError carries those headers into the response it synthesizes when a
caller has headers but no response, skipping values httpx cannot carry
- the bedrock 500 mapping forwards the provider response like its 4xx and 503
siblings instead of fabricating a blank one
The proxy now returns llm_provider-x-amzn-requestid on Bedrock chat errors.
* fix(bedrock): keep request-id on text-classified errors
The context-window and image branches of _map_bedrock_exception built their
litellm exception without the provider response, so a Bedrock 400 classified
by its body text lost x-amzn-RequestId while the sibling branches kept it.
Also narrows the new BedrockError types and trims its docstrings.
* chore(bedrock): drop the docstrings on the new error helpers
* fix(bedrock): keep request-id on every error path that has one
The ticket's root cause is that every BedrockError raise site under
litellm/llms/bedrock/ was built from status and message alone. The first
commits covered the chat and invoke handlers; this covers the rest.
Embeddings, rerank, image generation, image edit, count tokens, search and
the transformation layers now hand on the provider response or its headers,
and both bedrock_mantle configs return a BedrockError instead of the
OpenAI error that drops them.
Two blockers surfaced while verifying the streaming path. The trailing
`except Exception` in make_call and make_sync_call swallowed the BedrockError
raised a few lines above, relabelling a provider status as a 500, and the
non-200 branch read an unread streamed body, which throws.
The raise sites left alone have no provider response to carry: timeouts,
credential and config errors, and mid-stream event frames.
* fix(bedrock): forward provider headers from the count tokens route
The count tokens route converts BedrockError into an HTTPException, and dropped
the headers the handler had just kept, so that route still lost the request id.
get_response_headers now takes a Mapping so an httpx.Headers can be handed to it
without a copy.
* fix(bedrock): classify every bedrock surface through BedrockError
Eleven bedrock configs still inherited a provider-agnostic get_error_class
that builds a blank response, so the request id was gone before the proxy
read it. Claude platform, bedrock anthropic-messages, both image edit
configs, passthrough, realtime, vector stores and agentcore search now
return BedrockError, and a parametrized audit drives all 36 configs.
* fix(proxy): keep provider headers on the httpx status error branch
_handle_llm_api_exception forwards safe_headers on every branch except the
httpx.HTTPStatusError one, which the bedrock passthrough route reaches, so
the request id was dropped before the client saw the response.
* fix(bedrock): keep the request id on the timeout mappings
Timeout takes no response argument, so the three bedrock timeout branches
dropped the provider headers even when the upstream answered 408 or 504
with an x-amzn-RequestId. They now ride on the exception, already
llm_provider-prefixed, which is the form the proxy emits.
* fix(bedrock): keep the provider response on mapped timeouts
The previous round attached llm_provider-prefixed headers directly to the
Timeout. That shadowed the raw upstream headers for _get_response_headers,
so router cooldown and fallback cooldown stopped honouring retry-after on
bedrock 408/504 replies.
Give Timeout an optional response instead, the way every other mapped
bedrock exception already carries one. Retry logic reads the raw
retry-after off the response, and the proxy prefixes those headers on the
way out, so clients still see llm_provider-x-amzn-requestid.
* chore(bedrock): drop the explanatory comment on Timeout.response
* feat(skills): semantic search over the LiteLLM-hosted skill registry
Adds GET /v1/skills?query= (custom_llm_provider=litellm_proxy) and a
skill_search MCP virtual tool, ranking the caller's accessible skills by
semantic similarity, mirroring the A2A agent registry search (LIT-6309).
Also fixes a pre-existing bug where create_skill() dropped description and
instructions for the litellm_proxy provider, which left every LiteLLM-hosted
skill with no searchable text.
* fix(mcp): coerce skill_search top_k instead of raising 500 on malformed input
The MCP-REST skill_search dispatch validated raw tool arguments through a
pydantic model directly, so a non-numeric top_k raised a ValidationError
that the endpoint's catch-all turned into an HTTP 500. Mirrors the
agent_search branch's tolerant coerce_top_k handling instead.
* fix(skills): enforce key limits on search embeddings and bound the semantic index
Semantic search embeddings now run the same pre_call_hook the /embeddings
route runs, so key rate limits, budgets and guardrails apply before the
embedding model is called. The shared SemanticTextIndex caps cached vectors
and evicts the least recently searched entries, and each skill's embedded
text is capped so one skill cannot inflate the embedding batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): surface proxy 429s from search embeddings instead of a 503
ProxyRateLimitError is also an OpenAIError, so the search engine was folding
a key rate limit into skill_search_unavailable. Proxy HTTPExceptions now
propagate so the caller gets the same 429 the /embeddings route returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): import assert_never from typing_extensions for Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): embed the request as the pre-call hooks returned it, not the original text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(skills): keep the litellm_proxy provider check for GET /v1/skills?query= inside llms/
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(skills): move the GET /v1/skills?query= endpoint tests under tests/test_litellm/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A fake-streamed provider hands the adapter one chunk carrying both the
delta payload and the finish_reason, which is exactly what the combined
chunk splitter exists for, but its content check never listed the refusal.
The translation short-circuits on finish_reason, so that refusal text was
dropped and the client got `stop_reason: refusal` over an empty content
array, the symptom this PR set out to fix.
Both refusal accumulators also drop their `mutable-ok` lists for a plain
string attribute
The first-delta guard read `delta.refusal` directly, while the translation
three lines later goes through `openai_chat_refusal_text`, which also reads
the `provider_specific_fields` LiteLLM parks unrecognized fields in. A
provider that sends the refusal that way had its only refusal delta skipped
as blank, so the client got `stop_reason: refusal` over an empty content
array, which is the symptom this PR set out to fix
The /v1/messages adapter lowers a tier the entry does not accept, so dropping max from the astra
rows moves that path from Foundry's 400 to a request at xhigh. Nothing pinned that, and the guard
test's docstring named gpt-6-astra as the only gpt-5 name with an azure_ai row, which 11 rows
contradict.
The new Lyria passthrough branch runs before the image-generation branch
and keys on the same `predictions[0].bytesBase64Encoded` shape imagen
returns, so only the cost-map lookup separates them. Cover an imagen
predict response end to end so a future change that drops that lookup
fails here instead of misbilling images as audio.
Google prices Lyria per generated clip, so every Vertex Lyria entry in the
price map now carries a single output_cost_per_image and both the speech
and the passthrough cost paths read that one field. The old
output_cost_per_second and audio_seconds_per_prediction pair assumed a
30 second clip, which does not match the 32.768 second WAV Vertex returns,
and no other model in the map priced audio that way
Drops max_audio_length_hours and max_audio_per_prompt from the price map,
its schema, the generator, and ModelInfo, since nothing reads them, and
drops the audio_mime_type hidden param for the same reason: the response
already carries the resolved content type on its own header
Folds the per-model bundled catalog lookups into one cached parse of the
local cost map, validated with a TypeAdapter over a ReadOnly TypedDict
gpt-6-astra is the only gpt-5-family name with an azure_ai row. Prefixing the rest
cost them every effort flag, since get_llm_provider sends an azure_ai name down the
azure provider when a global AZURE_AI_API_BASE points at an openai.azure.com host and
azure/<model> is not a key either, which turned temperature, top_p and logprobs on
azure_ai/gpt-5.1-chat-latest from accepted into an UnsupportedParamsError.
Both bridges opened an empty text block on a refused streaming turn and
closed it without a single delta, so a client replaying that assistant
turn got HTTP 400 "text content blocks must be non-empty" from Anthropic.
The safeguard-refusal fallback that motivated withholding the text only
runs on the awaited non-streaming response, so nothing needed it withheld
Move the refusal readers into the shared messages/utils helpers so the
adapters stop reaching into each other's private statics, which is also
what put reportPrivateUsage over its budget
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card
for the reasoning_effort none gates, so temperature and top_p were refused while the
azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series
params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/
prefix the way the azure route does
Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a
flex service tier cost test that fails at the merge base, and keeps the wildcard test
from stripping azure_ai/gpt-6-astra out of the provider set
The chat and Responses bridges serialize tool_use blocks with model_dump(), so every
bridged /v1/messages response carried LiteLLM's internal provider_specific_fields key
(null, or a Gemini thought signature). Clients replay the block verbatim, and the next
turn that lands on a native Anthropic deployment (auto-router tier change, model swap)
is rejected with "tool_use.provider_specific_fields: Extra inputs are not permitted"
Strip the key from replayed content blocks at the single native Anthropic dispatch so
already-poisoned transcripts self-heal on every native provider, and stop emitting the
null on new responses. The bridges keep reading the signature for the Gemini round trip
Closes#19739
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
A live cost map older than this release, or a proxy whose map fetch lags, could
strip `reasoning` from a model this release knows accepts it. The bundled map is
now the floor: any OpenAI entry it flags as reasoning keeps the param whatever
the live map says. Fine-tuned ids with an empty suffix (`ft:gpt-4o-2024-08-06:org::id`)
now resolve to their base entry instead of failing open, `chat-latest` carries
the flag, and the schema test keeps every codex, deep-research, and chat-latest
entry flagged. The none-effort check goes through a public wrapper so the
responses config stops importing a private helper.
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
OCI Cohere restates the whole assistant text on the chunk that carries
the tool calls and again on the terminal chunk that carries chatHistory.
Only the terminal restatement was dropped, so a tool-calling turn streamed
the text twice. Treat a toolCalls-bearing chunk as a restatement too and
drop its text once deltas were already emitted.
Resolves LIT-6819
The image edit handler now awaits BaseImageEditConfig.async_transform_image_edit_request, and
Black Forest Labs overrides it so URL images and masks download through async_safe_get instead of
the blocking safe_get on the event loop. Rejected image fetches raise a fixed policy message with
the user_url_allowed_hosts hint rather than echoing the resolver's verdict (resolved IP, DNS
failure) back to the caller. The test fixture also fails any request-path call of the sync
convert_url_to_base64 so a regression cannot pass unnoticed.
Health checks probed every OCR deployment with a PDF, which Cohere Parse
rejects, so /health, background health checks, and the UI Test Connection
button marked Cohere Parse deployments unhealthy. BaseOCRConfig gains a
get_health_check_document hook (PDF by default) that CohereParseConfig
overrides with a 1x1 PNG data URI. cohere also gains ocr in the provider
endpoint matrix
* fix(responses): decode JSON-string tool schemas before sending to the provider
A caller that hands a tool schema over already JSON-encoded reached the
Responses API with a string `parameters`, and the provider rejected the
request with a 400 naming the routed model instead of the offending tool.
Decode it at the one place every Responses request converges, and refuse
anything that is neither an object nor a string encoding one.
Collapses the duplicated input/tool sanitization block shared by the
request and compact-request builders into a single owner, so the decode
cannot be wired into one path and not the other.
* test(responses): pin null tool schemas as accepted, and type the parametrized cases
The Responses API serves `parameters: null` and an omitted schema alike, so
neither may raise. Pin both against a future tightening, annotate the
parametrized inputs, and trim the docstrings back to what the code does not
already say.
