mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
f7323dc681
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
|
||
|---|---|---|
| .. | ||
| azure-chat.svg | ||
| bedrock_messages-flow.svg | ||
| budgets.svg | ||
| chat-streaming.svg | ||
| hooks-pipeline.svg | ||
| loadtest-rig.svg | ||
| model-capacity.svg | ||
| openai_chat-flow.svg | ||
| ops-drain.svg | ||
| ops-metrics.svg | ||
| prometheus.svg | ||
| rate-limits.svg | ||
| request-hooks.svg | ||
| routing-cooldown.svg | ||
| routing-request-flow.svg | ||
| s3-log.svg | ||
| shared_state-redis.svg | ||
| spend-writer.svg | ||
| virtual_keys-auth-flow.svg | ||
