diff --git a/docs/my-website/docs/image_variations.md b/docs/my-website/docs/image_variations.md new file mode 100644 index 00000000000..23c7d8cb167 --- /dev/null +++ b/docs/my-website/docs/image_variations.md @@ -0,0 +1,31 @@ +# [BETA] Image Variations + +OpenAI's `/image/variations` endpoint is now supported. + +## Quick Start + +```python +from litellm import image_variation +import os + +# set env vars +os.environ["OPENAI_API_KEY"] = "" +os.environ["TOPAZ_API_KEY"] = "" + +# openai call +response = image_variation( + model="dall-e-2", image=image_url +) + +# topaz call +response = image_variation( + model="topaz/Standard V2", image=image_url +) + +print(response) +``` + +## Supported Providers + +- OpenAI +- Topaz diff --git a/docs/my-website/docs/providers/bedrock.md b/docs/my-website/docs/providers/bedrock.md index e6f05a4ed55..61e6e804871 100644 --- a/docs/my-website/docs/providers/bedrock.md +++ b/docs/my-website/docs/providers/bedrock.md @@ -2,7 +2,7 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # AWS Bedrock -ALL Bedrock models (Anthropic, Meta, Mistral, Amazon, etc.) are Supported +ALL Bedrock models (Anthropic, Meta, Deepseek, Mistral, Amazon, etc.) are Supported | Property | Details | |-------|-------| @@ -1331,6 +1331,65 @@ Here's an example of using a bedrock model with LiteLLM. For a complete list, re | Mistral 7B Instruct | `completion(model='bedrock/mistral.mistral-7b-instruct-v0:2', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` | | Mixtral 8x7B Instruct | `completion(model='bedrock/mistral.mixtral-8x7b-instruct-v0:1', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` | +### Deepseek + +Native Deepseek support will be [available soon](https://github.com/BerriAI/litellm/pull/8132) + +Till then, use this: + + + + +```python +import litellm +response = litellm.completion( + model="bedrock/meta.DeepSeek-R1-Distill-Llama-8B", + model_id="provisioned-model-arn", + messages=[{"content": "Hello, how are you?", "role": "user"}] +) +``` + + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: bedrock-deepseek + litellm_params: + model: bedrock/meta.DeepSeek-R1-Distill-Llama-8B + model_id: provisioned-model-arn +``` + +2. Start proxy server + +```bash +litellm --config config.yaml + +# RUNNING on http://0.0.0.0:4000 +``` + +3. Test it! + +```bash +curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ +-H 'Content-Type: application/json' \ +-H 'Authorization: Bearer sk-1234' \ +-d '{ + "model": "bedrock-deepseek", + "messages": [ + { + "role": "user", + "content": "What'\''s the weather like in Boston today 236589?" + } + ] +}' +``` + + + + ## Bedrock Embedding ### API keys diff --git a/docs/my-website/docs/providers/deepseek.md b/docs/my-website/docs/providers/deepseek.md index 9f48e87123d..31efb36c21f 100644 --- a/docs/my-website/docs/providers/deepseek.md +++ b/docs/my-website/docs/providers/deepseek.md @@ -76,7 +76,7 @@ resp = completion( ) print( - resp.choices[0].message.provider_specific_fields["reasoning_content"] + resp.choices[0].message.reasoning_content ) ``` diff --git a/docs/my-website/docs/providers/ollama.md b/docs/my-website/docs/providers/ollama.md index d850b8ae6e1..848be2beb7c 100644 --- a/docs/my-website/docs/providers/ollama.md +++ b/docs/my-website/docs/providers/ollama.md @@ -238,6 +238,76 @@ Ollama supported models: https://github.com/ollama/ollama | Nous-Hermes 13B | `completion(model='ollama/nous-hermes:13b', messages, api_base="http://localhost:11434", stream=True)` | | Wizard Vicuna Uncensored | `completion(model='ollama/wizard-vicuna', messages, api_base="http://localhost:11434", stream=True)` | + +### JSON Schema support + + + + +```python +from litellm import completion + +response = completion( + model="ollama_chat/deepseek-r1", + messages=[{ "content": "respond in 20 words. who are you?","role": "user"}], + response_format={"type": "json_schema", "json_schema": {"schema": {"type": "object", "properties": {"name": {"type": "string"}}}}}, +) +print(response) +``` + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: "deepseek-r1" + litellm_params: + model: "ollama_chat/deepseek-r1" + api_base: "http://localhost:11434" +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml + +# RUNNING ON http://0.0.0.0:4000 +``` + +3. Test it! + +```python +from pydantic import BaseModel +from openai import OpenAI + +client = OpenAI( + api_key="anything", # 👈 PROXY KEY (can be anything, if master_key not set) + base_url="http://0.0.0.0:4000" # 👈 PROXY BASE URL +) + +class Step(BaseModel): + explanation: str + output: str + +class MathReasoning(BaseModel): + steps: list[Step] + final_answer: str + +completion = client.beta.chat.completions.parse( + model="deepseek-r1", + messages=[ + {"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step."}, + {"role": "user", "content": "how can I solve 8x + 7 = -23"} + ], + response_format=MathReasoning, +) + +math_reasoning = completion.choices[0].message.parsed +``` + + + ## Ollama Vision Models | Model Name | Function Call | |------------------|--------------------------------------| diff --git a/docs/my-website/docs/providers/topaz.md b/docs/my-website/docs/providers/topaz.md new file mode 100644 index 00000000000..018d269684d --- /dev/null +++ b/docs/my-website/docs/providers/topaz.md @@ -0,0 +1,27 @@ +# Topaz + +| Property | Details | +|-------|-------| +| Description | Professional-grade photo and video editing powered by AI. | +| Provider Route on LiteLLM | `topaz/` | +| Provider Doc | [Topaz ↗](https://www.topazlabs.com/enhance-api) | +| API Endpoint for Provider | https://api.topazlabs.com | +| Supported OpenAI Endpoints | `/image/variations` | + + +## Quick Start + +```python +from litellm import image_variation +import os + +os.environ["TOPAZ_API_KEY"] = "" +response = image_variation( + model="topaz/Standard V2", image=image_url +) +``` + +## Supported OpenAI Params + +- `response_format` +- `size` (widthxheight) diff --git a/docs/my-website/docs/proxy/health.md b/docs/my-website/docs/proxy/health.md index c9f67394bd0..52321a38457 100644 --- a/docs/my-website/docs/proxy/health.md +++ b/docs/my-website/docs/proxy/health.md @@ -314,6 +314,17 @@ Example Response: "I'm alive!" ``` +## `/health/services` + +Use this admin-only endpoint to check if a connected service (datadog/slack/langfuse/etc.) is healthy. + +```bash +curl -L -X GET 'http://0.0.0.0:4000/health/services?service=datadog' -H 'Authorization: Bearer sk-1234' +``` + +[**API Reference**](https://litellm-api.up.railway.app/#/health/health_services_endpoint_health_services_get) + + ## Advanced - Call specific models To check health of specific models, here's how to call them: diff --git a/docs/my-website/docs/proxy/virtual_keys.md b/docs/my-website/docs/proxy/virtual_keys.md index 183beee4a63..04be4ade482 100644 --- a/docs/my-website/docs/proxy/virtual_keys.md +++ b/docs/my-website/docs/proxy/virtual_keys.md @@ -559,6 +559,21 @@ curl 'http://localhost:4000/key/sk-1234/regenerate' \ [**👉 API REFERENCE DOCS**](https://litellm-api.up.railway.app/#/key%20management/regenerate_key_fn_key__key__regenerate_post) + +### Temporary Budget Increase + +Use the `/key/update` endpoint to increase the budget of an existing key. + +```bash +curl -L -X POST 'http://localhost:4000/key/update' \ +-H 'Authorization: Bearer sk-1234' \ +-H 'Content-Type: application/json' \ +-d '{"key": "sk-b3Z3Lqdb_detHXSUp4ol4Q", "temp_budget_increase": 100, "temp_budget_expiry": "10d"}' +``` + +[API Reference](https://litellm-api.up.railway.app/#/key%20management/update_key_fn_key_update_post) + + ### Restricting Key Generation Use this to control who can generate keys. Useful when letting others create keys on the UI. diff --git a/docs/my-website/release_notes/v1.55.10/index.md b/docs/my-website/release_notes/v1.55.10/index.md index 2d509a5d535..7f9839c2b53 100644 --- a/docs/my-website/release_notes/v1.55.10/index.md +++ b/docs/my-website/release_notes/v1.55.10/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.55.8-stable/index.md b/docs/my-website/release_notes/v1.55.8-stable/index.md index 7887192eca5..7e82e947475 100644 --- a/docs/my-website/release_notes/v1.55.8-stable/index.md +++ b/docs/my-website/release_notes/v1.55.8-stable/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.56.1/index.md b/docs/my-website/release_notes/v1.56.1/index.md index 1a1b1aaa1a5..7c4ccc74eaf 100644 --- a/docs/my-website/release_notes/v1.56.1/index.md +++ b/docs/my-website/release_notes/v1.56.1/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.56.3/index.md b/docs/my-website/release_notes/v1.56.3/index.md index 6ec6a6e2ec6..95205633ea6 100644 --- a/docs/my-website/release_notes/v1.56.3/index.md +++ b/docs/my-website/release_notes/v1.56.3/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.56.4/index.md b/docs/my-website/release_notes/v1.56.4/index.md index ea0cf0e73cf..93f87256321 100644 --- a/docs/my-website/release_notes/v1.56.4/index.md +++ b/docs/my-website/release_notes/v1.56.4/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.57.3/index.md b/docs/my-website/release_notes/v1.57.3/index.md index 0a5fba3e52b..3bee71a8e14 100644 --- a/docs/my-website/release_notes/v1.57.3/index.md +++ b/docs/my-website/release_notes/v1.57.3/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.57.7/index.md b/docs/my-website/release_notes/v1.57.7/index.md index 747aaba4bdd..ce987baf779 100644 --- a/docs/my-website/release_notes/v1.57.7/index.md +++ b/docs/my-website/release_notes/v1.57.7/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.57.8-stable/index.md b/docs/my-website/release_notes/v1.57.8-stable/index.md index 4c54e35cba0..9787444fde5 100644 --- a/docs/my-website/release_notes/v1.57.8-stable/index.md +++ b/docs/my-website/release_notes/v1.57.8-stable/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.59.0/index.md b/docs/my-website/release_notes/v1.59.0/index.md index 515ff464e29..5343ba49adb 100644 --- a/docs/my-website/release_notes/v1.59.0/index.md +++ b/docs/my-website/release_notes/v1.59.0/index.md @@ -6,7 +6,7 @@ authors: - name: Krrish Dholakia title: CEO, LiteLLM url: https://www.linkedin.com/in/krish-d/ - image_url: https://media.licdn.com/dms/image/v2/C5103AQHYMXJfHTf4Ng/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517455593871?e=1741824000&v=beta&t=udmat6jS-s3EQZp1DTykf7NZmf-3sefD_I9B1aMjE5Y + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI - name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ diff --git a/docs/my-website/release_notes/v1.59.8-stable/index.md b/docs/my-website/release_notes/v1.59.8-stable/index.md new file mode 100644 index 00000000000..ac180546ee5 --- /dev/null +++ b/docs/my-website/release_notes/v1.59.8-stable/index.md @@ -0,0 +1,161 @@ +--- +title: v1.59.8-stable +slug: v1.59.8-stable +date: 2025-01-31T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGiM7ZrUwqu_Q/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1675971026692?e=1741824000&v=beta&t=eQnRdXPJo4eiINWTZARoYTfqh064pgZ-E21pQTSy8jc +tags: [admin ui, logging, db schema] +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; + +# v1.59.8-stable + + + +:::info + +Get a 7 day free trial for LiteLLM Enterprise [here](https://litellm.ai/#trial). + +**no call needed** + +::: + + +## New Models / Updated Models + +1. New OpenAI `/image/variations` endpoint BETA support [Docs](../../docs/image_variations) +2. Topaz API support on OpenAI `/image/variations` BETA endpoint [Docs](../../docs/providers/topaz) +3. Deepseek - r1 support w/ reasoning_content ([Deepseek API](../../docs/providers/deepseek#reasoning-models), [Vertex AI](../../docs/providers/vertex.md#model-garden), [Bedrock](../../docs/providers/bedrock#deepseek)) +4. Azure - Add azure o1 pricing [See Here](https://github.com/BerriAI/litellm/blob/b8b927f23bc336862dacb89f59c784a8d62aaa15/model_prices_and_context_window.json#L952) +5. Anthropic - handle `-latest` tag in model for cost calculation +6. Gemini-2.0-flash-thinking - add model pricing (it’s 0.0) [See Here](https://github.com/BerriAI/litellm/blob/b8b927f23bc336862dacb89f59c784a8d62aaa15/model_prices_and_context_window.json#L3393) +7. Bedrock - add stability sd3 model pricing [See Here](https://github.com/BerriAI/litellm/blob/b8b927f23bc336862dacb89f59c784a8d62aaa15/model_prices_and_context_window.json#L6814) (s/o [Marty Sullivan](https://github.com/marty-sullivan)) +8. Bedrock - add us.amazon.nova-lite-v1:0 to model cost map [See Here](https://github.com/BerriAI/litellm/blob/b8b927f23bc336862dacb89f59c784a8d62aaa15/model_prices_and_context_window.json#L5619) +9. TogetherAI - add new together_ai llama3.3 models [See Here](https://github.com/BerriAI/litellm/blob/b8b927f23bc336862dacb89f59c784a8d62aaa15/model_prices_and_context_window.json#L6985) + +## LLM Translation + +1. LM Studio -> fix async embedding call +2. Gpt 4o models - fix response_format translation +3. Bedrock nova - expand supported document types to include .md, .csv, etc. [Start Here](../../docs/providers/bedrock#usage---pdf--document-understanding) +4. Bedrock - docs on IAM role based access for bedrock - [Start Here](https://docs.litellm.ai/docs/providers/bedrock#sts-role-based-auth) +5. Bedrock - cache IAM role credentials when used +6. Google AI Studio (`gemini/`) - support gemini 'frequency_penalty' and 'presence_penalty' +7. Azure O1 - fix model name check +8. WatsonX - ZenAPIKey support for WatsonX [Docs](../../docs/providers/watsonx.md) +9. Ollama Chat - support json schema response format [Start Here](../../docs/providers/ollama.md#json-schema-support) +10. Bedrock - return correct bedrock status code and error message if error during streaming +11. Anthropic - Supported nested json schema on anthropic calls +12. OpenAI - `metadata` param preview support + 1. SDK - enable via `litellm.enable_preview_features = True` + 2. PROXY - enable via `litellm_settings::enable_preview_features: true` +13. Replicate - retry completion response on status=processing + +## Spend Tracking Improvements + +1. Bedrock - QA asserts all bedrock regional models have same `supported_` as base model +2. Bedrock - fix bedrock converse cost tracking w/ region name specified +3. Spend Logs reliability fix - when `user` passed in request body is int instead of string +4. Ensure ‘base_model’ cost tracking works across all endpoints +5. Fixes for Image generation cost tracking +6. Anthropic - fix anthropic end user cost tracking +7. JWT / OIDC Auth - add end user id tracking from jwt auth + +## Management Endpoints / UI + +1. allows team member to become admin post-add (ui + endpoints) +2. New edit/delete button for updating team membership on UI +3. If team admin - show all team keys +4. Model Hub - clarify cost of models is per 1m tokens +5. Invitation Links - fix invalid url generated +6. New - SpendLogs Table Viewer - allows proxy admin to view spend logs on UI + 1. New spend logs - allow proxy admin to ‘opt in’ to logging request/response in spend logs table - enables easier abuse detection + 2. Show country of origin in spend logs + 3. Add pagination + filtering by key name/team name +7. `/key/delete` - allow team admin to delete team keys +8. Internal User ‘view’ - fix spend calculation when team selected +9. Model Analytics is now on Free +10. Usage page - shows days when spend = 0, and round spend on charts to 2 sig figs +11. Public Teams - allow admins to expose teams for new users to ‘join’ on UI - [Start Here](https://docs.litellm.ai/docs/proxy/public_teams) +12. Guardrails + 1. set/edit guardrails on a virtual key + 2. Allow setting guardrails on a team + 3. Set guardrails on team create + edit page +13. Support temporary budget increases on `/key/update` - new `temp_budget_increase` and `temp_budget_expiry` fields - [Start Here](../../docs/proxy/virtual_keys.md#temporary-budget-increase) +14. Support writing new key alias to AWS Secret Manager - on key rotation [Start Here](../../docs/secret.md#aws-secret-manager) + +## Helm + +1. add securityContext and pull policy values to migration job (s/o https://github.com/Hexoplon) +2. allow specifying envVars on values.yaml +3. new helm lint test + +## Logging / Guardrail Integrations + +1. Log the used prompt when prompt management used. [Start Here](../../docs/proxy/prompt_management.md) +2. Support s3 logging with team alias prefixes - [Start Here](https://docs.litellm.ai/docs/proxy/logging#team-alias-prefix-in-object-key) +3. Prometheus [Start Here](../../docs/proxy/prometheus.md) + 1. fix litellm_llm_api_time_to_first_token_metric not populating for bedrock models + 2. emit remaining team budget metric on regular basis (even when call isn’t made) - allows for more stable metrics on Grafana/etc. + 3. add key and team level budget metrics + 4. emit `litellm_overhead_latency_metric` + 5. Emit `litellm_team_budget_reset_at_metric` and `litellm_api_key_budget_remaining_hours_metric` +4. Datadog - support logging spend tags to Datadog. [Start Here](../../docs/proxy/enterprise.md#tracking-spend-for-custom-tags) +5. Langfuse - fix logging request tags, read from standard logging payload +6. GCS - don’t truncate payload on logging +7. New GCS Pub/Sub logging support [Start Here](https://docs.litellm.ai/docs/proxy/logging#google-cloud-storage---pubsub-topic) +8. Add AIM Guardrails support [Start Here](../../docs/proxy/guardrails/aim_security.md) + +## Security + +1. New Enterprise SLA for patching security vulnerabilities. [See Here](../../docs/enterprise.md#slas--professional-support) +2. Hashicorp - support using vault namespace for TLS auth. [Start Here](../../docs/secret.md#hashicorp-vault) +3. Azure - DefaultAzureCredential support + +## Health Checks + +1. Cleanup pricing-only model names from wildcard route list - prevent bad health checks +2. Allow specifying a health check model for wildcard routes - https://docs.litellm.ai/docs/proxy/health#wildcard-routes +3. New ‘health_check_timeout ‘ param with default 1min upperbound to prevent bad model from health check to hang and cause pod restarts. [Start Here](../../docs/proxy/health.md#health-check-timeout) +4. Datadog - add data dog service health check + expose new `/health/services` endpoint. [Start Here](../../docs/proxy/health.md#healthservices) + +## Performance / Reliability improvements + +1. 3x increase in RPS - moving to orjson for reading request body +2. LLM Routing speedup - using cached get model group info +3. SDK speedup - using cached get model info helper - reduces CPU work to get model info +4. Proxy speedup - only read request body 1 time per request +5. Infinite loop detection scripts added to codebase +6. Bedrock - pure async image transformation requests +7. Cooldowns - single deployment model group if 100% calls fail in high traffic - prevents an o1 outage from impacting other calls +8. Response Headers - return + 1. `x-litellm-timeout` + 2. `x-litellm-attempted-retries` + 3. `x-litellm-overhead-duration-ms` + 4. `x-litellm-response-duration-ms` +9. ensure duplicate callbacks are not added to proxy +10. Requirements.txt - bump certifi version + +## General Proxy Improvements + +1. JWT / OIDC Auth - new `enforce_rbac` param,allows proxy admin to prevent any unmapped yet authenticated jwt tokens from calling proxy. [Start Here](../../docs/proxy/token_auth.md#enforce-role-based-access-control-rbac) +2. fix custom openapi schema generation for customized swagger’s +3. Request Headers - support reading `x-litellm-timeout` param from request headers. Enables model timeout control when using Vercel’s AI SDK + LiteLLM Proxy. [Start Here](../../docs/proxy/request_headers.md#litellm-headers) +4. JWT / OIDC Auth - new `role` based permissions for model authentication. [See Here](https://docs.litellm.ai/docs/proxy/jwt_auth_arch) + +## Complete Git Diff + +This is the diff between v1.57.8-stable and v1.59.8-stable. + +Use this to see the changes in the codebase. + +[**Git Diff**](https://github.com/BerriAI/litellm/compare/v1.57.8-stable...v1.59.8-stable) diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 41febfd5642..5324d50ed40 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -206,6 +206,7 @@ const sidebars = { "providers/perplexity", "providers/friendliai", "providers/galadriel", + "providers/topaz", "providers/groq", "providers/github", "providers/deepseek", @@ -278,7 +279,14 @@ const sidebars = { }, "text_completion", "embedding/supported_embedding", - "image_generation", + { + type: "category", + label: "Image", + items: [ + "image_generation", + "image_variations", + ] + }, { type: "category", label: "Audio",