diff --git a/docs/my-website/docs/completion/prompt_caching.md b/docs/my-website/docs/completion/prompt_caching.md index 5c795778ef9..6fbf89bd6df 100644 --- a/docs/my-website/docs/completion/prompt_caching.md +++ b/docs/my-website/docs/completion/prompt_caching.md @@ -3,7 +3,13 @@ import TabItem from '@theme/TabItem'; # Prompt Caching -For OpenAI + Anthropic + Deepseek, LiteLLM follows the OpenAI prompt caching usage object format: +Supported Providers: +- OpenAI (`deepseek/`) +- Anthropic API (`anthropic/`) +- Bedrock (`bedrock/`, `bedrock/invoke/`, `bedrock/converse`) ([All models bedrock supports prompt caching on](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html)) +- Deepseek API (`deepseek/`) + +For the supported providers, LiteLLM follows the OpenAI prompt caching usage object format: ```bash "usage": { diff --git a/docs/my-website/docs/proxy/access_control.md b/docs/my-website/docs/proxy/access_control.md index 3d335380f4f..69b8a3ff6de 100644 --- a/docs/my-website/docs/proxy/access_control.md +++ b/docs/my-website/docs/proxy/access_control.md @@ -10,17 +10,13 @@ Role-based access control (RBAC) is based on Organizations, Teams and Internal U ## Roles -**Admin Roles** - - `proxy_admin`: admin over the platform - - `proxy_admin_viewer`: can login, view all keys, view all spend. **Cannot** create keys/delete keys/add new users - -**Organization Roles** - - `org_admin`: admin over the organization. Can create teams and users within their organization - -**Internal User Roles** - - `internal_user`: can login, view/create/delete their own keys, view their spend. **Cannot** add new users. - - `internal_user_viewer`: can login, view their own keys, view their own spend. **Cannot** create/delete keys, add new users. - +| Role Type | Role Name | Permissions | +|-----------|-----------|-------------| +| **Admin** | `proxy_admin` | Admin over the platform | +| | `proxy_admin_viewer` | Can login, view all keys, view all spend. **Cannot** create keys/delete keys/add new users | +| **Organization** | `org_admin` | Admin over the organization. Can create teams and users within their organization | +| **Internal User** | `internal_user` | Can login, view/create/delete their own keys, view their spend. **Cannot** add new users | +| | `internal_user_viewer` | Can login, view their own keys, view their own spend. **Cannot** create/delete keys, add new users | ## Onboarding Organizations diff --git a/docs/my-website/docs/proxy/token_auth.md b/docs/my-website/docs/proxy/token_auth.md index c6d280cb822..c562c7fb713 100644 --- a/docs/my-website/docs/proxy/token_auth.md +++ b/docs/my-website/docs/proxy/token_auth.md @@ -102,7 +102,19 @@ curl --location 'http://0.0.0.0:4000/v1/chat/completions' \ -## Advanced - Set Accepted JWT Scope Names +## Advanced + +### Multiple OIDC providers + +Use this if you want LiteLLM to validate your JWT against multiple OIDC providers (e.g. Google Cloud, GitHub Auth) + +Set `JWT_PUBLIC_KEY_URL` in your environment to a comma-separated list of URLs for your OIDC providers. + +```bash +export JWT_PUBLIC_KEY_URL="https://demo.duendesoftware.com/.well-known/openid-configuration/jwks,https://accounts.google.com/.well-known/openid-configuration/jwks" +``` + +### Set Accepted JWT Scope Names Change the string in JWT 'scopes', that litellm evaluates to see if a user has admin access. @@ -114,7 +126,7 @@ general_settings: admin_jwt_scope: "litellm-proxy-admin" ``` -## Tracking End-Users / Internal Users / Team / Org +### Tracking End-Users / Internal Users / Team / Org Set the field in the jwt token, which corresponds to a litellm user / team / org. @@ -156,7 +168,7 @@ scope: ["litellm-proxy-admin",...] scope: "litellm-proxy-admin ..." ``` -## Control model access with Teams +### Control model access with Teams 1. Specify the JWT field that contains the team ids, that the user belongs to. @@ -207,11 +219,11 @@ OIDC Auth for API: [**See Walkthrough**](https://www.loom.com/share/00fe2deab59a - If all checks pass, allow the request -## Advanced - Custom Validate +### Custom JWT Validate Validate a JWT Token using custom logic, if you need an extra way to verify if tokens are valid for LiteLLM Proxy. -### 1. Setup custom validate function +#### 1. Setup custom validate function ```python from typing import Literal @@ -230,7 +242,7 @@ def my_custom_validate(token: str) -> Literal[True]: return True ``` -### 2. Setup config.yaml +#### 2. Setup config.yaml ```yaml general_settings: @@ -243,7 +255,7 @@ general_settings: custom_validate: custom_validate.my_custom_validate # 👈 custom validate function ``` -### 3. Test the flow +#### 3. Test the flow **Expected JWT** @@ -265,7 +277,7 @@ general_settings: -## Advanced - Allowed Routes +### Allowed Routes Configure which routes a JWT can access via the config. @@ -297,7 +309,7 @@ general_settings: team_allowed_routes: ["/v1/chat/completions"] # 👈 Set accepted routes ``` -## Advanced - Caching Public Keys +### Caching Public Keys Control how long public keys are cached for (in seconds). @@ -311,7 +323,7 @@ general_settings: public_key_ttl: 600 # 👈 KEY CHANGE ``` -## Advanced - Custom JWT Field +### Custom JWT Field Set a custom field in which the team_id exists. By default, the 'client_id' field is checked. @@ -323,14 +335,7 @@ general_settings: team_id_jwt_field: "client_id" # 👈 KEY CHANGE ``` -## All Params - -[**See Code**](https://github.com/BerriAI/litellm/blob/b204f0c01c703317d812a1553363ab0cb989d5b6/litellm/proxy/_types.py#L95) - - - - -## Advanced - Block Teams +### Block Teams To block all requests for a certain team id, use `/team/block` @@ -357,7 +362,7 @@ curl --location 'http://0.0.0.0:4000/team/unblock' \ ``` -## Advanced - Upsert Users + Allowed Email Domains +### Upsert Users + Allowed Email Domains Allow users who belong to a specific email domain, automatic access to the proxy. @@ -494,4 +499,10 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ } ] }' -``` \ No newline at end of file +``` + +## All JWT Params + +[**See Code**](https://github.com/BerriAI/litellm/blob/b204f0c01c703317d812a1553363ab0cb989d5b6/litellm/proxy/_types.py#L95) + + diff --git a/docs/my-website/docs/proxy/ui_logs.md b/docs/my-website/docs/proxy/ui_logs.md new file mode 100644 index 00000000000..6ab53db3b6f --- /dev/null +++ b/docs/my-website/docs/proxy/ui_logs.md @@ -0,0 +1,52 @@ + +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# UI Logs Page + +View Spend, Token Usage, Key, Team Name for Each Request to LiteLLM + + + + + +## Overview + +| Log Type | Tracked by Default | +|----------|-------------------| +| Success Logs | ✅ Yes | +| Error Logs | ✅ Yes | +| Request/Response Content Stored | ❌ No by Default, **opt in with `store_prompts_in_spend_logs`** | + + + +**By default LiteLLM does not track the request and response content.** + +## Tracking - Request / Response Content in Logs Page + +If you want to switch on request and response content tracking, you need to opt in with this setting + +```yaml +general_settings: + store_prompts_in_spend_logs: true +``` + +## Stop storing Error Logs in DB + +If you do not want to store error logs in DB, you can opt out with this setting + +```yaml +general_settings: + disable_error_logs: True # Only disable writing error logs to DB, regular spend logs will still be written unless `disable_spend_logs: True` +``` + +## Stop storing Spend Logs in DB + +If you do not want to store spend logs in DB, you can opt out with this setting + +```yaml +general_settings: + disable_spend_logs: True # Disable writing spend logs to DB +``` + diff --git a/docs/my-website/docs/reasoning_content.md b/docs/my-website/docs/reasoning_content.md index f0bcd30a9b1..1cce3f05701 100644 --- a/docs/my-website/docs/reasoning_content.md +++ b/docs/my-website/docs/reasoning_content.md @@ -14,6 +14,9 @@ Supported Providers: - Anthropic API (`anthropic/`) - Bedrock (Anthropic + Deepseek) (`bedrock/`) - Vertex AI (Anthropic) (`vertexai/`) +- OpenRouter (`openrouter/`) + +LiteLLM will standardize the `reasoning_content` in the response and `thinking_blocks` in the assistant message. ```python "message": { diff --git a/docs/my-website/docs/secret.md b/docs/my-website/docs/secret.md index 76761642594..9f0ff7059cd 100644 --- a/docs/my-website/docs/secret.md +++ b/docs/my-website/docs/secret.md @@ -96,7 +96,7 @@ litellm --config /path/to/config.yaml ``` -### Using K/V pairs in 1 AWS Secret +#### Using K/V pairs in 1 AWS Secret You can read multiple keys from a single AWS Secret using the `primary_secret_name` parameter: diff --git a/docs/my-website/img/release_notes/credentials.jpg b/docs/my-website/img/release_notes/credentials.jpg new file mode 100644 index 00000000000..1f11c67f054 Binary files /dev/null and b/docs/my-website/img/release_notes/credentials.jpg differ diff --git a/docs/my-website/img/release_notes/litellm_test_connection.gif b/docs/my-website/img/release_notes/litellm_test_connection.gif new file mode 100644 index 00000000000..2c8ea45ab43 Binary files /dev/null and b/docs/my-website/img/release_notes/litellm_test_connection.gif differ diff --git a/docs/my-website/img/release_notes/responses_api.png b/docs/my-website/img/release_notes/responses_api.png new file mode 100644 index 00000000000..045d86825de Binary files /dev/null and b/docs/my-website/img/release_notes/responses_api.png differ diff --git a/docs/my-website/img/ui_request_logs.png b/docs/my-website/img/ui_request_logs.png new file mode 100644 index 00000000000..912123522bd Binary files /dev/null and b/docs/my-website/img/ui_request_logs.png differ diff --git a/docs/my-website/release_notes/v1.63.11-stable/index.md b/docs/my-website/release_notes/v1.63.11-stable/index.md new file mode 100644 index 00000000000..4e4ca35d483 --- /dev/null +++ b/docs/my-website/release_notes/v1.63.11-stable/index.md @@ -0,0 +1,163 @@ +--- +title: v1.63.11-stable +slug: v1.63.11-stable +date: 2025-03-15T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg + +tags: [credential management, thinking content, responses api, snowflake] +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; + +These are the changes since `v1.63.2-stable`. + +This release is primarily focused on: +- [Beta] Responses API Support +- Snowflake Cortex Support +- UI - Credential Management, re-use credentials when adding new models +- UI - Test Connection to LLM Provider before adding a model + +:::info + +This release will be live on 03/16/2025 + +::: + + + +## Demo Instance + +Here's a Demo Instance to test changes: +- Instance: https://demo.litellm.ai/ +- Login Credentials: + - Username: admin + - Password: sk-1234 + + +## LLM Translation + + + +1. **New Endpoints** +- [Beta] POST `/responses` API. [Get Started](https://docs.litellm.ai/docs/response_api) + +2. **New LLM Providers** +- Snowflake Cortex [Get Started](https://docs.litellm.ai/docs/providers/snowflake) + +3. **New models** + +- Support OpenRouter `reasoning_content` on streaming [Get Started](https://docs.litellm.ai/docs/reasoning_content) +- Support Bedrock converse cache token tracking [Get Started](https://docs.litellm.ai/docs/completion/prompt_caching) + +4. **Bug Fixes** + +- Return `code`, `param` and `type` on OpenAI bad request error [More information on litellm exceptions](https://docs.litellm.ai/docs/exception_mapping) +- Fix Bedrock converse chunk parsing to only return empty dict on tool use [PR](https://github.com/BerriAI/litellm/pull/9166) +- Fix Azure Function Calling Bug & Update Default API Version to `2025-02-01-preview` [PR](https://github.com/BerriAI/litellm/pull/9191) +- Fix Perplexity incorrect streaming response [PR](https://github.com/BerriAI/litellm/pull/9081) +- Fix Triton streaming completions bug [PR](https://github.com/BerriAI/litellm/pull/8386) +- Fix: String `data:` stripped from entire content in streamed Gemini responses [PR](https://github.com/BerriAI/litellm/pull/9070) +- Fix: Support bytes.IO when handling audio files for transcription [PR](https://github.com/BerriAI/litellm/pull/9071) +- Fix: "system" role has become unacceptable in Ollama [PR](https://github.com/BerriAI/litellm/pull/9261) +- Handle HTTP 201 status code in Vertex AI response [PR](https://github.com/BerriAI/litellm/pull/9193) + + +### New Models Added to Model Cost Map +- Add support for Amazon Nova Canvas model [PR](https://github.com/BerriAI/litellm/pull/7838) +- Add pricing for Jamba new models [PR](https://github.com/BerriAI/litellm/pull/9032) +- Add pricing for Amazon EU models [PR](https://github.com/BerriAI/litellm/pull/9056) +- Add Bedrock Deepseek R1 model pricing [PR](https://github.com/BerriAI/litellm/pull/9108) +- Update Gemini pricing: Gemma 3, Flash 2 thinking update, LearnLM [PR](https://github.com/BerriAI/litellm/pull/9190) +- Mark Cohere Embedding 3 models as Multimodal [PR](https://github.com/BerriAI/litellm/pull/9176) + + +## Spend Tracking Improvements + +1. Add Azure Data Zone pricing [PR](https://github.com/BerriAI/litellm/pull/9185) +2. Cost Tracking for Responses API +3. Fix Azure Whisper cost tracking [PR](https://github.com/BerriAI/litellm/pull/9166) + + +## UI + +### Re-Use Credentials on UI + +You can now onboard LLM provider credentials on LiteLLM UI. Once these credentials are added you can re-use them when adding new models + + + + +### Test Connections before adding models + +Before adding a model you can test the connection to the LLM provider to verify you have setup your API Base + API Key correctly + + + +### General UI Improvements +1. Add Models Page + - Allow adding Cerebras, Sambanova, Perplexity, Fireworks, Openrouter, TogetherAI Models, Text-Completion OpenAI on Admin UI + - Allow adding EU OpenAI models + - Fix: Instantly show edit + deletes to models +2. Keys Page + - Fix: Instantly show newly created keys on Admin UI (don't require refresh) + - Fix: Allow clicking into Top Keys when showing users Top API Key + - Fix: Allow Filter Keys by Team Alias, Key Alias and Org + - UI Improvements: Show 100 Keys Per Page, Use full height, increase width of key alias +3. Users Page + - Fix: Show correct count of internal user keys on Users Page + - Fix: Metadata not updating in Team UI +4. Logs Page + - UI Improvements: Keep expanded log in focus on LiteLLM UI + - UI Improvements: Minor improvements to logs page + - Fix: Allow internal user to query their own logs + + +## Security + +1. Support for Rotating Master Keys [Getting Started](https://docs.litellm.ai/docs/proxy/master_key_rotations) +2. Fix: Internal User Viewer Permissions, don't allow `internal_user_viewer` role to see `Test Key Page` or `Create Key Button` [Role based access controls](https://docs.litellm.ai/docs/proxy/access_control) +3. Emit audit logs on All user + model Create/Update/Delete endpoints [Get Started](https://docs.litellm.ai/docs/proxy/multiple_admins) +4. JWT + - Support multiple JWT OIDC providers [Get Started](https://docs.litellm.ai/docs/proxy/token_auth) + - Fix JWT access with Groups not working when team is assigned All Proxy Models access +5. Using K/V pairs in 1 AWS Secret [Get Started](https://docs.litellm.ai/docs/secret#using-kv-pairs-in-1-aws-secret) + + +## Logging Integrations + +1. Prometheus: Track Azure LLM API latency metric [Get Started here](https://docs.litellm.ai/docs/proxy/prometheus#request-latency-metrics) +2. Allow switching off storing Error Logs in DB **MISSING DOC** +3. Added tags, user_feedback and model_options to additional_keys which can be sent to Athina [Get Started here](https://docs.litellm.ai/docs/observability/athina_integration) + +## OpenWebUI Integration - display `thinking` tokens +- Guide on getting started with LiteLLM x OpenWebUI. [Get Started](https://docs.litellm.ai/docs/tutorials/openweb_ui) +- Display `thinking` tokens on OpenWebUI (Bedrock, Anthropic, Deepseek) [Get Started](https://docs.litellm.ai/docs/tutorials/openweb_ui#render-thinking-content-on-openweb-ui) + + + +## Performance / Reliability improvements + +1. Fix Redis cluster mode for routers [PR](https://github.com/BerriAI/litellm/pull/9010) +2. Delegate router Azure client init logic to Azure provider [PR](https://github.com/BerriAI/litellm/pull/9140) +3. Fix Azure AI services URL [PR](https://github.com/BerriAI/litellm/pull/9185) +4. Support extra_headers on Bedrock [PR](https://github.com/BerriAI/litellm/pull/9113) + + +## General Improvements +1. UI API Playground for testing LiteLLM translation [PR](https://github.com/BerriAI/litellm/pull/9073) +2. Fix: Correctly use `PROXY_LOGOUT_URL` when set [PR](https://github.com/BerriAI/litellm/pull/9117) +3. Bing Search Pass Through endpoint [PR](https://github.com/BerriAI/litellm/pull/8019) + + +## Complete Git Diff + +[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.2-stable...v1.63.11-stable) \ No newline at end of file diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 385983fd339..cae4dee9526 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -101,7 +101,8 @@ const sidebars = { "proxy/admin_ui_sso", "proxy/self_serve", "proxy/public_teams", - "proxy/custom_sso" + "proxy/custom_sso", + "proxy/ui_logs" ], }, {