scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-08 03:08:45 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 115

docs(index.md): write up on session management

Browse source
This commit is contained in:
Krrish Dholakia 2025-08-16 17:55:18 -07:00
parent cb95f2d5d7
commit 21549a3463
3 changed files with 21 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
docs/my-website/docs/response_api.md
View file

@ -803,11 +803,18 @@ LiteLLM Proxy supports session management for non-OpenAI models. This allows you
1. Enable storing request / response content in the database
Set `store_prompts_in_spend_logs: true` in your proxy config.yaml. When this is enabled, LiteLLM will store the request and response content in the database.
Set `store_prompts_in_cold_storage: true` in your proxy config.yaml. When this is enabled, LiteLLM will store the request and response content in the s3 bucket you specify.
```yaml
litellm_settings:
callbacks: ["s3_v2"]
s3_callback_params: # learn more https://docs.litellm.ai/docs/proxy/logging#s3-buckets
s3_bucket_name: litellm-logs # AWS Bucket Name for S3
s3_region_name: us-west-2
general_settings:
store_prompts_in_spend_logs: true
cold_storage_custom_logger: s3_v2
store_prompts_in_cold_storage: true
```
2. Make request 1 with no `previous_response_id` (new session)

BIN
docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 472 KiB

15
docs/my-website/release_notes/v1.75.5-stable/index.md
View file

@ -59,22 +59,31 @@ Users of our Docker image, are **not** affected by this change.
---
## Redis - Latency Improvements
## Redis Latency Improvements
<Image
img={require('../../img/release_notes/faster_caching_calls.png')}
style={{width: '100%', display: 'block', margin: '2rem auto'}}
/>
<br/>
This release adds in-memory caching for Redis requests, enabling faster response times in high-traffic. Now, LiteLLM instances will check their in-memory cache for a cache hit, before checking Redis. This reduces caching-related latency from 100ms for LLM API calls to sub-1ms, on cache hits.
---
## Responses API Session Management w/ Images
<Image
img={require('../../img/release_notes/responses_api_session_mgt_images.jpg')}
style={{width: '100%', display: 'block', margin: '2rem auto'}}
/>
<br/>
LiteLLM now supports session management for Responses API requests with images. This is great for use-cases like chatbots, that are using the Responses API to track the state of a conversation. LiteLLM session management works across **ALL** LLM API's (including Anthropic, Bedrock, OpenAI, etc). LiteLLM session management works by storing the request and response content in an s3 bucket, you can specify.
---
## New Models / Updated Models