From 2394cd465e9c23795b575f3c192b01ee3bf93863 Mon Sep 17 00:00:00 2001 From: Ishaan Jaff Date: Sat, 12 Apr 2025 17:26:38 -0700 Subject: [PATCH] stable release note fixes (#9954) * docs fix * docs metrics * docs fix release notes * docs 1.66.0-stable --- docs/my-website/docs/proxy/prometheus.md | 9 +++- .../release_notes/v1.66.0-stable/index.md | 52 ++++++++++--------- 2 files changed, 35 insertions(+), 26 deletions(-) diff --git a/docs/my-website/docs/proxy/prometheus.md b/docs/my-website/docs/proxy/prometheus.md index 3666c6f7386..0ce94ab9627 100644 --- a/docs/my-website/docs/proxy/prometheus.md +++ b/docs/my-website/docs/proxy/prometheus.md @@ -95,7 +95,14 @@ Use this for for tracking per [user, key, team, etc.](virtual_keys) ### Initialize Budget Metrics on Startup -If you want to initialize the key/team budget metrics on startup, you can set the `prometheus_initialize_budget_metrics` to `true` in the `config.yaml` +If you want litellm to emit the budget metrics for all keys, teams irrespective of whether they are getting requests or not, set `prometheus_initialize_budget_metrics` to `true` in the `config.yaml` + +**How this works:** + +- If the `prometheus_initialize_budget_metrics` is set to `true` + - Every 5 minutes litellm runs a cron job to read all keys, teams from the database + - It then emits the budget metrics for each key, team + - This is used to populate the budget metrics on the `/metrics` endpoint ```yaml litellm_settings: diff --git a/docs/my-website/release_notes/v1.66.0-stable/index.md b/docs/my-website/release_notes/v1.66.0-stable/index.md index 65024792cd4..f54dfb8190d 100644 --- a/docs/my-website/release_notes/v1.66.0-stable/index.md +++ b/docs/my-website/release_notes/v1.66.0-stable/index.md @@ -46,7 +46,7 @@ v1.66.0-stable is live now, here are the key highlights of this release ## Key Highlights - **Microsoft SSO Auto-sync**: Auto-sync groups and group members from Azure Entra ID to LiteLLM - **Unified File IDs**: Use the same file id across LLM API providers. -- **Realtime API Cost Tracking**: Track cost of realtime api calls +- **Realtime API Cost Tracking**: Track cost of realtime API calls - **xAI grok-3**: Added support for `xai/grok-3` models - **Security Fixes**: Fixed [CVE-2025-0330](https://www.cve.org/CVERecord?id=CVE-2025-0330) and [CVE-2024-6825](https://www.cve.org/CVERecord?id=CVE-2024-6825) vulnerabilities @@ -62,10 +62,10 @@ Let's dive in. Auto-sync groups and members from Azure Entra ID to LiteLLM

-This release adds support for auto-syncing groups and members on Microsoft Entra ID with LiteLLM. This means that litellm proxy administrators can spend less time managing teams and members and LiteLLM handles the following: +This release adds support for auto-syncing groups and members on Microsoft Entra ID with LiteLLM. This means that LiteLLM proxy administrators can spend less time managing teams and members and LiteLLM handles the following: -- Auto-create Teams that existing on Microsoft Entra ID -- Sync team members on Microsoft Entra ID with LiteLLM Teams +- Auto-create teams that exist on Microsoft Entra ID +- Sync team members on Microsoft Entra ID with LiteLLM teams Get started with this [here](https://docs.litellm.ai/docs/tutorials/msft_sso) @@ -76,42 +76,42 @@ Get started with this [here](https://docs.litellm.ai/docs/tutorials/msft_sso) ## New Models / Updated Models - xAI - 1. Added cost tracking for `xai/grok-3` models [PR](https://github.com/BerriAI/litellm/pull/9920) - 2. Added reasoning_effort support for `xai/grok-3-mini-beta` model family [PR](https://github.com/BerriAI/litellm/pull/9932) + 1. Added reasoning_effort support for `xai/grok-3-mini-beta` [Get Started](https://docs.litellm.ai/docs/providers/xai#reasoning-usage) + 2. Added cost tracking for `xai/grok-3` models [PR](https://github.com/BerriAI/litellm/pull/9920) - Hugging Face - 1. Hugging Face - Added inference providers support [Getting Started](https://docs.litellm.ai/docs/providers/huggingface#serverless-inference-providers) + 1. Added inference providers support [Get Started](https://docs.litellm.ai/docs/providers/huggingface#serverless-inference-providers) - Azure - 1. Azure - Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893) + 1. Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893) - VertexAI - 1. VertexAI - Added enterpriseWebSearch tool support [PR](https://github.com/BerriAI/litellm/pull/9856) - 2. VertexAI - Moved to only passing in accepted keys by vertex ai response schema [PR](https://github.com/BerriAI/litellm/pull/8992) + 1. Added enterpriseWebSearch tool support [Get Started](https://docs.litellm.ai/docs/providers/vertex#grounding---web-search) + 2. Moved to only passing keys accepted by the Vertex AI response schema [PR](https://github.com/BerriAI/litellm/pull/8992) - Google AI Studio - 1. Google AI Studio - Added cost tracking for `gemini-2.5-pro` [PR](https://github.com/BerriAI/litellm/pull/9837) - 2. Google AI Studio - Fixed pricing for 'gemini/gemini-2.5-pro-preview-03-25' [PR](https://github.com/BerriAI/litellm/pull/9896) - 3. Google AI Studio - Fixed handling file_data being passed in [PR](https://github.com/BerriAI/litellm/pull/9786) + 1. Added cost tracking for `gemini-2.5-pro` [PR](https://github.com/BerriAI/litellm/pull/9837) + 2. Fixed pricing for 'gemini/gemini-2.5-pro-preview-03-25' [PR](https://github.com/BerriAI/litellm/pull/9896) + 3. Fixed handling file_data being passed in [PR](https://github.com/BerriAI/litellm/pull/9786) - Azure - 1. Azure - Updated Azure Phi-4 pricing [PR](https://github.com/BerriAI/litellm/pull/9862) - 2. Azure - Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893) + 1. Updated Azure Phi-4 pricing [PR](https://github.com/BerriAI/litellm/pull/9862) + 2. Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893) - Databricks - 1. Databricks - Removed reasoning_effort from parameters [PR](https://github.com/BerriAI/litellm/pull/9811) + 1. Removed reasoning_effort from parameters [PR](https://github.com/BerriAI/litellm/pull/9811) 2. Fixed custom endpoint check for Databricks [PR](https://github.com/BerriAI/litellm/pull/9925) - General - 1. Function Calling - Handle pydantic base model in message tool calls, handle tools = [], and support fake streaming on tool calls for meta.llama3-3-70b-instruct-v1:0 [PR](https://github.com/BerriAI/litellm/pull/9774) - 2. LiteLLM Proxy - Allow passing `thinking` param to litellm proxy via client sdk [PR](https://github.com/BerriAI/litellm/pull/9386) - 3. Reasoning - Added litellm.supports_reasoning() util to track if an llm supports reasoning [PR](https://github.com/BerriAI/litellm/pull/9923) + 1. Added litellm.supports_reasoning() util to track if an llm supports reasoning [Get Started](https://docs.litellm.ai/docs/providers/anthropic#reasoning) + 2. Function Calling - Handle pydantic base model in message tool calls, handle tools = [], and support fake streaming on tool calls for meta.llama3-3-70b-instruct-v1:0 [PR](https://github.com/BerriAI/litellm/pull/9774) + 3. LiteLLM Proxy - Allow passing `thinking` param to litellm proxy via client sdk [PR](https://github.com/BerriAI/litellm/pull/9386) 4. Fixed correctly translating 'thinking' param for litellm [PR](https://github.com/BerriAI/litellm/pull/9904) ## Spend Tracking Improvements - OpenAI, Azure - 1. Realtime API Cost tracking with token usage metrics in spend logs [PR](https://github.com/BerriAI/litellm/pull/9795) + 1. Realtime API Cost tracking with token usage metrics in spend logs [Get Started](https://docs.litellm.ai/docs/realtime) - Anthropic 1. Fixed Claude Haiku cache read pricing per token [PR](https://github.com/BerriAI/litellm/pull/9834) 2. Added cost tracking for Claude responses with base_model [PR](https://github.com/BerriAI/litellm/pull/9897) @@ -134,24 +134,26 @@ Get started with this [here](https://docs.litellm.ai/docs/tutorials/msft_sso) View input, output, reasoning tokens, ttft metrics.

2. Tag / Policy Management: - 1. Added Tag/Policy Management [PR](https://github.com/BerriAI/litellm/pull/9813) + 1. Added Tag/Policy Management. Create routing rules based on request metadata. This allows you to enforce that requests with `tags="private"` only go to specific models. [Get Started](https://docs.litellm.ai/docs/tutorials/tag_management) + +

- Tag / Policy Management + Create and manage tags.

3. Redesigned Login Screen: 1. Polished login screen [PR](https://github.com/BerriAI/litellm/pull/9778) 2. Microsoft SSO Auto-Sync: 1. Added debug route to allow admins to debug SSO JWT fields [PR](https://github.com/BerriAI/litellm/pull/9835) 2. Added ability to use MSFT Graph API to assign users to teams [PR](https://github.com/BerriAI/litellm/pull/9865) - 3. Connected LiteLLM to Azure Entra ID Enterprise Application [PR](https://github.com/BerriAI/litellm/pull/9872) + 3. Connected litellm to Azure Entra ID Enterprise Application [PR](https://github.com/BerriAI/litellm/pull/9872) 4. Added ability for admins to set `default_team_params` for when litellm SSO creates default teams [PR](https://github.com/BerriAI/litellm/pull/9895) 5. Fixed MSFT SSO to use correct field for user email [PR](https://github.com/BerriAI/litellm/pull/9886) - 6. Added UI support for setting Default Team setting when LiteLLM SSO auto creates teams [PR](https://github.com/BerriAI/litellm/pull/9918) + 6. Added UI support for setting Default Team setting when litellm SSO auto creates teams [PR](https://github.com/BerriAI/litellm/pull/9918) 5. UI Bug Fixes: 1. Prevented team, key, org, model numerical values changing on scrolling [PR](https://github.com/BerriAI/litellm/pull/9776) 2. Instantly reflect key and team updates in UI [PR](https://github.com/BerriAI/litellm/pull/9825) @@ -159,7 +161,7 @@ Get started with this [here](https://docs.litellm.ai/docs/tutorials/msft_sso) ## Logging / Guardrail Improvements 1. Prometheus: - - Emit Key and Team Budget metrics on a cron job schedule [PR](https://github.com/BerriAI/litellm/pull/9528) + - Emit Key and Team Budget metrics on a cron job schedule [Get Started](https://docs.litellm.ai/docs/proxy/prometheus#initialize-budget-metrics-on-startup) ## Security Fixes