* feat(responses): honor supported_endpoints /v1/responses opt-in for OpenAI-compatible deployments
custom_openai and other generic OpenAI-compatible deployments have no native
Responses API config, so every /v1/responses call is bridged through
/v1/chat/completions. When model_info.supported_endpoints lists /v1/responses,
resolve OpenAILikeResponsesConfig instead so the request is forwarded to
{api_base}/responses, for streaming, non-streaming and mode: responses
deployments alike. Providers with their own Responses config are unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop deployment supported_endpoints opt-in after cross-provider prompt swap
A prompt manager that moves the request to another provider leaves kwargs['model_info']
describing the original deployment; without this the swapped provider was sent an
OpenAI-like /responses request it does not serve.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): carry prompt-swap deployment metadata as a return value instead of a kwargs marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep provider response headers in streaming logging callbacks
The responses streaming iterator captures the provider's HTTP response headers into
its own _hidden_params, but never puts them on the completed response, and the
model_validate(model_dump()) copy made for logging drops pydantic private attributes.
Success callbacks and StandardLoggingPayload.hidden_params.additional_headers therefore
saw an empty dict for streaming /v1/responses, so Azure's apim-request-id was unreadable
from the callback payload.
Restore the headers on the nested response of the logging copy, preferring any the
provider transform already set (the fake_stream path) and falling back to the ones the
iterator captured from the stream. Skipped when the copy fell back to the original event,
so a serialization failure never leaves logging-only state on the caller's object.
* fix: satisfy LIT002 mutable-collection gate in header restore
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Any pre_call guardrail on /v1/responses flattened Codex namespace tools
into ns__member functions and wrote the flattened list back to the
request, so the model called mcp__server__tool with no namespace and
Codex rejected the call as unsupported.
The handler now keeps the client's original tools, hands the guardrail a
deep copy of the flattened ones, and rebuilds data["tools"] by matching
the guardrail's output to the originals by type and name. Unchanged
tools go back as the original objects, a dropped or edited namespace
member changes only that member, and tools the guardrail injects are
still appended.
Fixes#39183
The chat-completions bridge emitted Responses output items whose item ids
were raw Anthropic tool ids (toolu_/srvtoolu_), which OpenAI rejects on
replay with "Expected an ID that begins with 'fc'", breaking router
fallback conversations from gpt-5 to claude models.
Four fixes, composable and independently useful:
- emission: bridge output items get fc_/ctc_-prefixed item ids while
call_id stays raw so tool_result pairing keeps working (streaming and
non-streaming share the same helpers)
- openai replay: request transformation drops tool call item ids that do
not match OpenAI's own shapes instead of forwarding them, gated to
OpenAI and Azure, since the API accepts the items with no id at all
- anthropic replay: a replayed srvtoolu_ call whose paired server tool
result is unavailable degrades to a plain client tool_use instead of a
dangling server_tool_use that 400s the client's tool_result
- tool-only turns no longer emit a message output item with output_text
text null, matching native OpenAI output
Streamed responses through the proxy previously exposed no usable cost:
the x-litellm-response-cost header is unreadable mid-stream and the final
usage chunk carried only tokens, priced against an alias model name the
client cannot resolve. The include_cost_in_streaming_usage flag existed
but was off by default and only fixed the wire, not SDK clients.
Stamp usage.cost into the joined streaming response by default wherever a
final usage object is built: the chat-completions stream_chunk_builder,
the native /v1/responses RESPONSE_COMPLETED event, and synthetic response
events. Provider-reported cost always wins over the computed value, and
only positive computed costs are stamped so unpriceable alias responses
keep deferring to the logging object's own calculation. Per-chunk SSE
cost injection (/v1/messages, generateContent, passthrough) stays behind
the flag.
Also normalize non-litellm usage objects in stream_chunk_builder: openai
CompletionUsage lacks Usage.__contains__, so membership probes silently
returned False and client-side rebuilds dropped the wire cost and
recounted token usage locally. Wire token counts and cost now survive.
Resolves LIT-6427
The Responses-to-chat transform dropped the filename OpenAI requires next to
file_data, so a request carrying an inline PDF counted 13 tokens instead of 36
and a real completion through the chat bridge got a 400.
- _write_back_structured_messages now patches only the rewritten rows back
into the original input items, so reasoning items (encrypted_content),
function_call ids, and web_search_call items survive a guardrail rewrite
verbatim; rewrites that cannot be row-mapped fall back to the previous
full conversion
- the responses bridge stream snapshot restores usage hidden in
_hidden_params when stream_options is unset, so converted fake streams
report real input_tokens instead of 0
`AnthropicCacheControlHook` spends the configured injection points on the first
message list it is shown and drops the message points that matched nothing. That is
right when the messages it sees are the ones going upstream. It is wrong for
/v1/responses: the system prompt lives in `instructions`, which only becomes a system
message once the chat-completion bridge builds one, so a role-targeted point matched
nothing and was thrown away before the message it wanted existed. Injection silently
did nothing across the whole surface.
Hand those points back instead, stamped as judged, when the caller says its message
list is provisional. The stamp is what makes carrying them safe: without it the next
pass re-judges the points against messages this pass has already marked and stands the
whole configuration down. Callers holding the final messages -- /chat/completions and
/v1/messages -- do not raise the signal and keep dropping unmatched points as before.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
On /v1/responses the prompt template ran inside litellm.aresponses, after the
router had already resolved a deployment and injected its api_key/api_base, so a
prompt whose metadata.model pointed at another provider sent the old
deployment's credentials cross-provider (401). The proxy now runs the prompt
template for aresponses in the pre-call hook, before routing, so the router
picks the deployment that matches the swapped model. As a backstop, the SDK
refuses a cross-provider swap when explicit credentials are already present
instead of forwarding them.
ignore_prompt_manager_model and ignore_prompt_manager_optional_params saved on
a prompt were only read by the generic manager, so dotprompt prompts ignored
them on every endpoint. PromptManagementBase now merges the prompt spec's flags
with the per-request ones for every manager, and the generic manager no longer
drops caller flags when no spec is present.
An empty content list, or one holding only opaque blocks, still lets the
provider-bound branch replay the summary text. The inspection path treated
any non-None content as final, so that replayed text stayed invisible to
guardrails and token counting.
A reasoning input item that carries only summary text is replayed to the
provider as reasoning_content, so inspection-only callers must see that
text too. They used to fall through to the generic content branch, which
reads content and drops a summary-only item, leaving guardrails and token
counters blind to text the model still receives.
The session lookup reads spend logs straight out of the database, so a
follow-up sent seconds after the turn it chains off found nothing while the
row was still queued in the worker that served it, and the conversation was
dropped without an error. Responses calls now ask the spend-log writer to
flush on its next pass instead of waiting out its poll interval, and the
lookup gives a just-finished turn a short second chance.
Replaying a session also accepted `input` only as a string or a single dict,
so the standard list shape dropped every user turn and left the model with
assistant messages alone.
Write the fallback reasoning item id back to the cache so the
reasoning-done path and the completed snapshot cannot drift apart, and
cover the shared delta id and the snapshot alignment with tests.
