Merge pull request #9279 from BerriAI/litellm_stable_release_1_63_11

v1.63.11-stable release notes
2026-09-07 08:26:10 +00:00 · 2025-03-15 11:16:13 -07:00 · 2025-03-15 11:16:13 -07:00 · 78194f5fb6
commit 78194f5fb6
parent d4caaae1be b6f3d124d9
12 changed files with 266 additions and 34 deletions
--- a/docs/my-website/docs/completion/prompt_caching.md
+++ b/docs/my-website/docs/completion/prompt_caching.md
@ -3,7 +3,13 @@ import TabItem from '@theme/TabItem';

 # Prompt Caching 

-For OpenAI + Anthropic + Deepseek, LiteLLM follows the OpenAI prompt caching usage object format:
+Supported Providers:
+- OpenAI (`deepseek/`)
+- Anthropic API (`anthropic/`)
+- Bedrock (`bedrock/`, `bedrock/invoke/`, `bedrock/converse`) ([All models bedrock supports prompt caching on](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html))
+- Deepseek API (`deepseek/`)
+
+For the supported providers, LiteLLM follows the OpenAI prompt caching usage object format:

 ```bash
 "usage": {
--- a/docs/my-website/docs/proxy/access_control.md
+++ b/docs/my-website/docs/proxy/access_control.md
@ -10,17 +10,13 @@ Role-based access control (RBAC) is based on Organizations, Teams and Internal U

 ## Roles

-**Admin Roles**
-  - `proxy_admin`: admin over the platform
-  - `proxy_admin_viewer`: can login, view all keys, view all spend. **Cannot** create keys/delete keys/add new users
-
-**Organization Roles**
-  - `org_admin`: admin over the organization. Can create teams and users within their organization
-
-**Internal User Roles**
-  - `internal_user`: can login, view/create/delete their own keys, view their spend. **Cannot** add new users.
-  - `internal_user_viewer`: can login, view their own keys, view their own spend. **Cannot** create/delete keys, add new users.
-
+| Role Type | Role Name | Permissions |
+|-----------|-----------|-------------|
+| **Admin** | `proxy_admin` | Admin over the platform |
+| | `proxy_admin_viewer` | Can login, view all keys, view all spend. **Cannot** create keys/delete keys/add new users |
+| **Organization** | `org_admin` | Admin over the organization. Can create teams and users within their organization |
+| **Internal User** | `internal_user` | Can login, view/create/delete their own keys, view their spend. **Cannot** add new users |
+| | `internal_user_viewer` | Can login, view their own keys, view their own spend. **Cannot** create/delete keys, add new users |

 ## Onboarding Organizations 

--- a/docs/my-website/docs/proxy/token_auth.md
+++ b/docs/my-website/docs/proxy/token_auth.md
@ -102,7 +102,19 @@ curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
 </TabItem>
 </Tabs>

-## Advanced - Set Accepted JWT Scope Names 
+## Advanced
+
+### Multiple OIDC providers
+
+Use this if you want LiteLLM to validate your JWT against multiple OIDC providers (e.g. Google Cloud, GitHub Auth)
+
+Set `JWT_PUBLIC_KEY_URL` in your environment to a comma-separated list of URLs for your OIDC providers.
+
+```bash
+export JWT_PUBLIC_KEY_URL="https://demo.duendesoftware.com/.well-known/openid-configuration/jwks,https://accounts.google.com/.well-known/openid-configuration/jwks"
+```
+
+### Set Accepted JWT Scope Names 

 Change the string in JWT 'scopes', that litellm evaluates to see if a user has admin access.

@ -114,7 +126,7 @@ general_settings:
    admin_jwt_scope: "litellm-proxy-admin"
 ```

-## Tracking End-Users / Internal Users / Team / Org
+### Tracking End-Users / Internal Users / Team / Org

 Set the field in the jwt token, which corresponds to a litellm user / team / org.

@ -156,7 +168,7 @@ scope: ["litellm-proxy-admin",...]
 scope: "litellm-proxy-admin ..."
 ```

-## Control model access with Teams
+### Control model access with Teams


 1. Specify the JWT field that contains the team ids, that the user belongs to. 
@ -207,11 +219,11 @@ OIDC Auth for API: [**See Walkthrough**](https://www.loom.com/share/00fe2deab59a
 - If all checks pass, allow the request


-## Advanced - Custom Validate
+### Custom JWT Validate

 Validate a JWT Token using custom logic, if you need an extra way to verify if tokens are valid for LiteLLM Proxy.

-### 1. Setup custom validate function
+#### 1. Setup custom validate function

 ```python
 from typing import Literal
@ -230,7 +242,7 @@ def my_custom_validate(token: str) -> Literal[True]:
  return True
 ```

-### 2. Setup config.yaml
+#### 2. Setup config.yaml

 ```yaml
 general_settings:
@ -243,7 +255,7 @@ general_settings:
    custom_validate: custom_validate.my_custom_validate # 👈 custom validate function
 ```

-### 3. Test the flow
+#### 3. Test the flow

 **Expected JWT**

@ -265,7 +277,7 @@ general_settings:



-## Advanced - Allowed Routes 
+### Allowed Routes 

 Configure which routes a JWT can access via the config.

@ -297,7 +309,7 @@ general_settings:
    team_allowed_routes: ["/v1/chat/completions"] # 👈 Set accepted routes
 ```

-## Advanced - Caching Public Keys 
+### Caching Public Keys 

 Control how long public keys are cached for (in seconds).

@ -311,7 +323,7 @@ general_settings:
    public_key_ttl: 600 # 👈 KEY CHANGE
 ```

-## Advanced - Custom JWT Field 
+### Custom JWT Field 

 Set a custom field in which the team_id exists. By default, the 'client_id' field is checked. 

@ -323,14 +335,7 @@ general_settings:
    team_id_jwt_field: "client_id" # 👈 KEY CHANGE
 ```

-## All Params
-
-[**See Code**](https://github.com/BerriAI/litellm/blob/b204f0c01c703317d812a1553363ab0cb989d5b6/litellm/proxy/_types.py#L95)
-
-
-
-
-## Advanced - Block Teams 
+### Block Teams 

 To block all requests for a certain team id, use `/team/block`

@ -357,7 +362,7 @@ curl --location 'http://0.0.0.0:4000/team/unblock' \
 ```


-## Advanced - Upsert Users + Allowed Email Domains 
+### Upsert Users + Allowed Email Domains 

 Allow users who belong to a specific email domain, automatic access to the proxy.
 
@ -494,4 +499,10 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
    }
  ]
 }'
-```
+```
+
+## All JWT Params
+
+[**See Code**](https://github.com/BerriAI/litellm/blob/b204f0c01c703317d812a1553363ab0cb989d5b6/litellm/proxy/_types.py#L95)
+
+
--- a/docs/my-website/docs/proxy/ui_logs.md
+++ b/docs/my-website/docs/proxy/ui_logs.md
@ -0,0 +1,52 @@
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# UI Logs Page
+
+View Spend, Token Usage, Key, Team Name for Each Request to LiteLLM
+
+
+<Image img={require('../../img/ui_request_logs.png')}/>
+
+
+## Overview
+
+| Log Type | Tracked by Default |
+|----------|-------------------|
+| Success Logs | ✅ Yes |
+| Error Logs | ✅ Yes |
+| Request/Response Content Stored | ❌ No by Default, **opt in with `store_prompts_in_spend_logs`** |
+
+
+
+**By default LiteLLM does not track the request and response content.**
+
+## Tracking - Request / Response Content in Logs Page 
+
+If you want to switch on request and response content tracking, you need to opt in with this setting
+
+```yaml
+general_settings:
+  store_prompts_in_spend_logs: true
+```
+
+## Stop storing Error Logs in DB
+
+If you do not want to store error logs in DB, you can opt out with this setting
+
+```yaml
+general_settings:
+  disable_error_logs: True   # Only disable writing error logs to DB, regular spend logs will still be written unless `disable_spend_logs: True`
+```
+
+## Stop storing Spend Logs in DB
+
+If you do not want to store spend logs in DB, you can opt out with this setting
+
+```yaml
+general_settings:
+  disable_spend_logs: True   # Disable writing spend logs to DB
+```
+
--- a/docs/my-website/docs/reasoning_content.md
+++ b/docs/my-website/docs/reasoning_content.md
@ -14,6 +14,9 @@ Supported Providers:
 - Anthropic API (`anthropic/`)
 - Bedrock (Anthropic + Deepseek) (`bedrock/`)
 - Vertex AI (Anthropic) (`vertexai/`)
+- OpenRouter (`openrouter/`)
+
+LiteLLM will standardize the `reasoning_content` in the response and `thinking_blocks` in the assistant message.

 ```python
 "message": {
--- a/docs/my-website/docs/secret.md
+++ b/docs/my-website/docs/secret.md
@ -96,7 +96,7 @@ litellm --config /path/to/config.yaml
 ```


-### Using K/V pairs in 1 AWS Secret
+#### Using K/V pairs in 1 AWS Secret

 You can read multiple keys from a single AWS Secret using the `primary_secret_name` parameter:

--- a/docs/my-website/img/release_notes/credentials.jpg
+++ b/docs/my-website/img/release_notes/credentials.jpg
--- a/docs/my-website/img/release_notes/litellm_test_connection.gif
+++ b/docs/my-website/img/release_notes/litellm_test_connection.gif
--- a/docs/my-website/img/release_notes/responses_api.png
+++ b/docs/my-website/img/release_notes/responses_api.png
--- a/docs/my-website/img/ui_request_logs.png
+++ b/docs/my-website/img/ui_request_logs.png
--- a/docs/my-website/release_notes/v1.63.11-stable/index.md
+++ b/docs/my-website/release_notes/v1.63.11-stable/index.md
@ -0,0 +1,163 @@
+---
+title: v1.63.11-stable
+slug: v1.63.11-stable
+date: 2025-03-15T10:00:00
+authors:
+  - name: Krrish Dholakia
+    title: CEO, LiteLLM
+    url: https://www.linkedin.com/in/krish-d/
+    image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI
+  - name: Ishaan Jaffer
+    title: CTO, LiteLLM
+    url: https://www.linkedin.com/in/reffajnaahsi/
+    image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+tags: [credential management, thinking content, responses api, snowflake]
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+
+These are the changes since `v1.63.2-stable`.
+
+This release is primarily focused on:
+- [Beta] Responses API Support
+- Snowflake Cortex Support
+- UI - Credential Management, re-use credentials when adding new models
+- UI - Test Connection to LLM Provider before adding a model
+
+:::info
+
+This release will be live on 03/16/2025
+
+:::
+
+<!-- <Image img={require('../../img/release_notes/v16311_release.jpg')} /> -->
+
+## Demo Instance
+
+Here's a Demo Instance to test changes:
+- Instance: https://demo.litellm.ai/
+- Login Credentials:
+    - Username: admin
+    - Password: sk-1234
+
+
+## LLM Translation
+
+<Image img={require('../../img/release_notes/responses_api.png')} />
+
+1. **New Endpoints**
+- [Beta] POST `/responses` API. [Get Started](https://docs.litellm.ai/docs/response_api)
+
+2. **New LLM Providers**
+- Snowflake Cortex [Get Started](https://docs.litellm.ai/docs/providers/snowflake)
+
+3. **New models**
+
+- Support OpenRouter `reasoning_content` on streaming [Get Started](https://docs.litellm.ai/docs/reasoning_content)
+- Support Bedrock converse cache token tracking [Get Started](https://docs.litellm.ai/docs/completion/prompt_caching)
+
+4. **Bug Fixes**
+
+- Return `code`, `param` and `type` on OpenAI bad request error [More information on litellm exceptions](https://docs.litellm.ai/docs/exception_mapping)
+- Fix Bedrock converse chunk parsing to only return empty dict on tool use [PR](https://github.com/BerriAI/litellm/pull/9166)
+- Fix Azure Function Calling Bug & Update Default API Version to `2025-02-01-preview` [PR](https://github.com/BerriAI/litellm/pull/9191)
+- Fix Perplexity incorrect streaming response [PR](https://github.com/BerriAI/litellm/pull/9081)
+- Fix Triton streaming completions bug [PR](https://github.com/BerriAI/litellm/pull/8386)
+- Fix: String `data:` stripped from entire content in streamed Gemini responses [PR](https://github.com/BerriAI/litellm/pull/9070)
+- Fix: Support bytes.IO when handling audio files for transcription [PR](https://github.com/BerriAI/litellm/pull/9071)
+- Fix: "system" role has become unacceptable in Ollama [PR](https://github.com/BerriAI/litellm/pull/9261)
+- Handle HTTP 201 status code in Vertex AI response [PR](https://github.com/BerriAI/litellm/pull/9193)
+
+
+### New Models Added to Model Cost Map
+- Add support for Amazon Nova Canvas model [PR](https://github.com/BerriAI/litellm/pull/7838)
+- Add pricing for Jamba new models [PR](https://github.com/BerriAI/litellm/pull/9032)
+- Add pricing for Amazon EU models [PR](https://github.com/BerriAI/litellm/pull/9056)
+- Add Bedrock Deepseek R1 model pricing [PR](https://github.com/BerriAI/litellm/pull/9108)
+- Update Gemini pricing: Gemma 3, Flash 2 thinking update, LearnLM [PR](https://github.com/BerriAI/litellm/pull/9190)
+- Mark Cohere Embedding 3 models as Multimodal [PR](https://github.com/BerriAI/litellm/pull/9176)
+
+
+## Spend Tracking Improvements
+
+1. Add Azure Data Zone pricing [PR](https://github.com/BerriAI/litellm/pull/9185)
+2. Cost Tracking for Responses API
+3. Fix Azure Whisper cost tracking [PR](https://github.com/BerriAI/litellm/pull/9166)
+
+
+## UI
+
+### Re-Use Credentials on UI
+
+You can now onboard LLM provider credentials on LiteLLM UI. Once these credentials are added you can re-use them when adding new models
+
+<Image img={require('../../img/release_notes/credentials.jpg')} />
+
+
+### Test Connections before adding models
+
+Before adding a model you can test the connection to the LLM provider to verify you have setup your API Base + API Key correctly
+
+<Image img={require('../../img/release_notes/litellm_test_connection.gif')} />
+
+### General UI Improvements
+1. Add Models Page
+   - Allow adding Cerebras, Sambanova, Perplexity, Fireworks, Openrouter, TogetherAI Models, Text-Completion OpenAI on Admin UI
+   - Allow adding EU OpenAI models
+   - Fix: Instantly show edit + deletes to models
+2. Keys Page
+   - Fix: Instantly show newly created keys on Admin UI (don't require refresh)
+   - Fix: Allow clicking into Top Keys when showing users Top API Key
+   - Fix: Allow Filter Keys by Team Alias, Key Alias and Org
+   - UI Improvements: Show 100 Keys Per Page, Use full height, increase width of key alias
+3. Users Page
+   - Fix: Show correct count of internal user keys on Users Page
+   - Fix: Metadata not updating in Team UI
+4. Logs Page
+   - UI Improvements: Keep expanded log in focus on LiteLLM UI
+   - UI Improvements: Minor improvements to logs page
+   - Fix: Allow internal user to query their own logs
+
+
+## Security
+
+1. Support for Rotating Master Keys [Getting Started](https://docs.litellm.ai/docs/proxy/master_key_rotations)
+2. Fix: Internal User Viewer Permissions, don't allow `internal_user_viewer` role to see `Test Key Page` or `Create Key Button` [Role based access controls](https://docs.litellm.ai/docs/proxy/access_control)
+3. Emit audit logs on All user + model Create/Update/Delete endpoints [Get Started](https://docs.litellm.ai/docs/proxy/multiple_admins)
+4. JWT
+    - Support multiple JWT OIDC providers [Get Started](https://docs.litellm.ai/docs/proxy/token_auth)
+    - Fix JWT access with Groups not working when team is assigned All Proxy Models access
+5. Using K/V pairs in 1 AWS Secret [Get Started](https://docs.litellm.ai/docs/secret#using-kv-pairs-in-1-aws-secret)
+
+
+## Logging Integrations
+
+1. Prometheus: Track Azure LLM API latency metric [Get Started here](https://docs.litellm.ai/docs/proxy/prometheus#request-latency-metrics)
+2. Allow switching off storing Error Logs in DB **MISSING DOC**
+3. Added tags, user_feedback and model_options to additional_keys which can be sent to Athina [Get Started here](https://docs.litellm.ai/docs/observability/athina_integration)
+
+## OpenWebUI Integration - display `thinking` tokens
+- Guide on getting started with LiteLLM x OpenWebUI. [Get Started](https://docs.litellm.ai/docs/tutorials/openweb_ui)
+- Display `thinking` tokens on OpenWebUI (Bedrock, Anthropic, Deepseek) [Get Started](https://docs.litellm.ai/docs/tutorials/openweb_ui#render-thinking-content-on-openweb-ui)
+
+<Image img={require('../../img/litellm_thinking_openweb.gif')} />
+
+## Performance / Reliability improvements
+
+1. Fix Redis cluster mode for routers [PR](https://github.com/BerriAI/litellm/pull/9010)
+2. Delegate router Azure client init logic to Azure provider [PR](https://github.com/BerriAI/litellm/pull/9140)
+3. Fix Azure AI services URL [PR](https://github.com/BerriAI/litellm/pull/9185)
+4. Support extra_headers on Bedrock [PR](https://github.com/BerriAI/litellm/pull/9113)
+
+
+## General Improvements
+1. UI API Playground for testing LiteLLM translation [PR](https://github.com/BerriAI/litellm/pull/9073)
+2. Fix: Correctly use `PROXY_LOGOUT_URL` when set [PR](https://github.com/BerriAI/litellm/pull/9117)
+3. Bing Search Pass Through endpoint [PR](https://github.com/BerriAI/litellm/pull/8019)
+
+
+## Complete Git Diff
+
+[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.2-stable...v1.63.11-stable)
--- a/docs/my-website/sidebars.js
+++ b/docs/my-website/sidebars.js
@ -101,7 +101,8 @@ const sidebars = {
            "proxy/admin_ui_sso",
            "proxy/self_serve",
            "proxy/public_teams",
-            "proxy/custom_sso"
+            "proxy/custom_sso",
+            "proxy/ui_logs"
          ],
        },
        {