docs(anthropic_background_mode.md): guide on when to use responses api background mode via litellm

This commit is contained in:
Krrish Dholakia 2026-01-22 19:18:36 -08:00
parent f1dc9a4ab8
commit 904a7b85ac
2 changed files with 20 additions and 1 deletions

View file

@ -131,4 +131,19 @@
"Claude Code",
"LiteLLM"
]
},
{
"title": "Stream OpenAI/Anthropic/Gemini/etc. in Ruby",
"description": "This is a guide to streaming OpenAI/Anthropic/Gemini/etc. in Ruby or other single-threaded frameworks.",
"url": "https://docs.litellm.ai/docs/guides/anthropic_background_mode",
"date": "2026-01-22",
"version": "1.0.0",
"tags": [
"OpenAI",
"Anthropic",
"Gemini",
"Ruby",
"Responses API",
"Streaming"
]
}]

View file

@ -3,6 +3,10 @@
Imaging the case where services are running frameworks that does not support multithreading (for example, your company runs ruby service on single thread framework), you want to stream responses, but don't want to block I/O for others since it is a single thread. Native SSE streaming won't work in this case. LiteLLM supports background mode where user can use litellm to poll partial responses periodically to simulate streaming.
Requirements:
- Redis cache must be enabled
- /responses API endpoint must be used
## How it works
Think of this as a producerconsumer pattern:
@ -77,7 +81,7 @@ curl http://0.0.0.0:4000/v1/responses/litellm_poll_adff0089-f9d3-4135-b49e-f5825
Response:
Keep polling until the response is completed.
**Keep polling until the response is completed.**
```bash
{