mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-06 08:16:43 +00:00
docs(anthropic_background_mode.md): guide on when to use responses api background mode via litellm
This commit is contained in:
parent
f1dc9a4ab8
commit
904a7b85ac
2 changed files with 20 additions and 1 deletions
|
|
@ -131,4 +131,19 @@
|
|||
"Claude Code",
|
||||
"LiteLLM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Stream OpenAI/Anthropic/Gemini/etc. in Ruby",
|
||||
"description": "This is a guide to streaming OpenAI/Anthropic/Gemini/etc. in Ruby or other single-threaded frameworks.",
|
||||
"url": "https://docs.litellm.ai/docs/guides/anthropic_background_mode",
|
||||
"date": "2026-01-22",
|
||||
"version": "1.0.0",
|
||||
"tags": [
|
||||
"OpenAI",
|
||||
"Anthropic",
|
||||
"Gemini",
|
||||
"Ruby",
|
||||
"Responses API",
|
||||
"Streaming"
|
||||
]
|
||||
}]
|
||||
|
|
@ -3,6 +3,10 @@
|
|||
Imaging the case where services are running frameworks that does not support multithreading (for example, your company runs ruby service on single thread framework), you want to stream responses, but don't want to block I/O for others since it is a single thread. Native SSE streaming won't work in this case. LiteLLM supports background mode where user can use litellm to poll partial responses periodically to simulate streaming.
|
||||
|
||||
|
||||
Requirements:
|
||||
- Redis cache must be enabled
|
||||
- /responses API endpoint must be used
|
||||
|
||||
## How it works
|
||||
|
||||
Think of this as a producer–consumer pattern:
|
||||
|
|
@ -77,7 +81,7 @@ curl http://0.0.0.0:4000/v1/responses/litellm_poll_adff0089-f9d3-4135-b49e-f5825
|
|||
|
||||
Response:
|
||||
|
||||
Keep polling until the response is completed.
|
||||
**Keep polling until the response is completed.**
|
||||
|
||||
```bash
|
||||
{
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue