mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-08 03:08:45 +00:00
doc on streaming usage litellm proxy
This commit is contained in:
parent
83879d2a3d
commit
60bdfb437f
3 changed files with 50 additions and 9 deletions
|
|
@ -3,9 +3,11 @@ import TabItem from '@theme/TabItem';
|
|||
|
||||
# Streaming + Async
|
||||
|
||||
- [Streaming Responses](#streaming-responses)
|
||||
- [Async Completion](#async-completion)
|
||||
- [Async + Streaming Completion](#async-streaming)
|
||||
| Feature | LiteLLM SDK | LiteLLM Proxy |
|
||||
|---------|-------------|---------------|
|
||||
| Streaming | ✅ [start here](#streaming-responses) | ✅ [start here](../proxy/user_keys#streaming) |
|
||||
| Async | ✅ [start here](#async-completion) | ✅ [start here](../proxy/user_keys#streaming) |
|
||||
| Async Streaming | ✅ [start here](#async-streaming) | ✅ [start here](../proxy/user_keys#streaming) |
|
||||
|
||||
## Streaming Responses
|
||||
LiteLLM supports streaming the model response back by passing `stream=True` as an argument to the completion function
|
||||
|
|
|
|||
|
|
@ -381,6 +381,51 @@ assert user.age == 25
|
|||
|
||||
```
|
||||
|
||||
### **Streaming**
|
||||
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="curl">
|
||||
|
||||
```bash
|
||||
curl http://0.0.0.0:4000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $OPTIONAL_YOUR_PROXY_KEY" \
|
||||
-d '{
|
||||
"model": "gpt-4-turbo",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "this is a test request, write a short poem"
|
||||
}
|
||||
],
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(
|
||||
api_key="sk-1234", # [OPTIONAL] set if you set one on proxy, else set ""
|
||||
base_url="http://0.0.0.0:4000",
|
||||
)
|
||||
|
||||
messages = [{"role": "user", "content": "this is a test request, write a short poem"}]
|
||||
completion = client.chat.completions.create(
|
||||
model="gpt-4o",
|
||||
messages=messages,
|
||||
stream=True
|
||||
)
|
||||
|
||||
print(completion)
|
||||
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
### Function Calling
|
||||
|
||||
Here's some examples of doing function calling with the proxy.
|
||||
|
|
|
|||
|
|
@ -43,12 +43,6 @@ const config = {
|
|||
id: 'release_notes',
|
||||
path: './release_notes', // Folder where your release notes are stored
|
||||
routeBasePath: '/release_notes', // URL path for the release notes
|
||||
sortPosts: (a, b) => {
|
||||
// Extract folder names from the file paths
|
||||
const folderA = a.metadata.permalink.split('/')[2]; // Get folder name from permalink
|
||||
const folderB = b.metadata.permalink.split('/')[2];
|
||||
return folderA.localeCompare(folderB); // Compare folder names
|
||||
},
|
||||
include: ['**/*.md', '**/*.mdx'], // Files to include
|
||||
// Other blog options
|
||||
},
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue