diff --git a/docs/my-website/docs/response_api.md b/docs/my-website/docs/response_api.md index e64f922ac80..c0bb003c096 100644 --- a/docs/my-website/docs/response_api.md +++ b/docs/my-website/docs/response_api.md @@ -803,11 +803,18 @@ LiteLLM Proxy supports session management for non-OpenAI models. This allows you 1. Enable storing request / response content in the database -Set `store_prompts_in_spend_logs: true` in your proxy config.yaml. When this is enabled, LiteLLM will store the request and response content in the database. +Set `store_prompts_in_cold_storage: true` in your proxy config.yaml. When this is enabled, LiteLLM will store the request and response content in the s3 bucket you specify. ```yaml +litellm_settings: + callbacks: ["s3_v2"] + s3_callback_params: # learn more https://docs.litellm.ai/docs/proxy/logging#s3-buckets + s3_bucket_name: litellm-logs # AWS Bucket Name for S3 + s3_region_name: us-west-2 + general_settings: - store_prompts_in_spend_logs: true + cold_storage_custom_logger: s3_v2 + store_prompts_in_cold_storage: true ``` 2. Make request 1 with no `previous_response_id` (new session) diff --git a/docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg b/docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg new file mode 100644 index 00000000000..852d2fdd6d0 Binary files /dev/null and b/docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg differ diff --git a/docs/my-website/release_notes/v1.75.5-stable/index.md b/docs/my-website/release_notes/v1.75.5-stable/index.md index 499b08a5b0f..8eedccce9ea 100644 --- a/docs/my-website/release_notes/v1.75.5-stable/index.md +++ b/docs/my-website/release_notes/v1.75.5-stable/index.md @@ -59,22 +59,31 @@ Users of our Docker image, are **not** affected by this change. --- -## Redis - Latency Improvements +## Redis Latency Improvements +
+ This release adds in-memory caching for Redis requests, enabling faster response times in high-traffic. Now, LiteLLM instances will check their in-memory cache for a cache hit, before checking Redis. This reduces caching-related latency from 100ms for LLM API calls to sub-1ms, on cache hits. +--- +## Responses API Session Management w/ Images + +
+LiteLLM now supports session management for Responses API requests with images. This is great for use-cases like chatbots, that are using the Responses API to track the state of a conversation. LiteLLM session management works across **ALL** LLM API's (including Anthropic, Bedrock, OpenAI, etc). LiteLLM session management works by storing the request and response content in an s3 bucket, you can specify. - - +--- ## New Models / Updated Models