scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-10 22:41:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 83

Merge branch 'main' into litellm_ishaan_march_17

Browse source
This commit is contained in:
Ishaan Jaff 2026-03-18 15:08:34 -07:00 • committed by GitHub
commit 8c23f9f6a1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
85 changed files with 4813 additions and 1290 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
docs/my-website/blog/anthropic_opus_4_5_and_advanced_features/index.md
View file

@ -3,18 +3,9 @@ slug: anthropic_advanced_features
title: "Day 0 Support: Claude 4.5 Opus (+Advanced Features)"
date: 2025-11-25T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to Claude Opus 4.5 and advanced features in LiteLLM: Tool Search, Programmatic Tool Calling, and Effort Parameter."
tags: [anthropic, claude, tool search, programmatic tool calling, effort, advanced features]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
This guide covers Anthropic's latest model (Claude Opus 4.5) and its advanced features now available in LiteLLM: Tool Search, Programmatic Tool Calling, Tool Input Examples, and the Effort Parameter.
{/* truncate */}
---
| Feature | Supported Models |

15
docs/my-website/blog/anthropic_wildcard_model_access_incident/index.md
View file

@ -3,18 +3,9 @@ slug: anthropic-wildcard-model-access-incident
title: "Incident Report: Wildcard Blocking New Models After Cost Map Reload"
date: 2026-02-23T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
tags: [incident-report, proxy, auth, model-access]
hide_table_of_contents: false
---

24
docs/my-website/blog/authors.yml
View file

@ -4,6 +4,12 @@ litellm:
url: https://github.com/BerriAI/litellm
image_url: https://github.com/BerriAI.png
sameer:
name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
krrish:
name: Krrish Dholakia
title: CEO, LiteLLM
@ -22,3 +28,21 @@ ishaan-alt:
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
ryan:
name: Ryan Crabbe
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
image_url: https://media.licdn.com/dms/image/v2/D5603AQHt1t9Z4BJ6Gw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1724453682340?e=1772064000&v=beta&t=VXdmr13rsNB05wyA2F1TENOB5UuDHUZ0FCHTolNyR5M
alexsander:
name: Alexsander Hamir
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
yuneng:
name: Yuneng Jiang
title: SWE @ LiteLLM (Full Stack)
url: https://www.linkedin.com/in/yuneng-david-jiang-455676139/
image_url: https://avatars.githubusercontent.com/u/171294688?v=4

17
docs/my-website/blog/claude_code_beta_headers/index.md
View file

@ -3,18 +3,9 @@ slug: claude-code-beta-headers-incident
title: "Incident Report: Invalid beta headers with Claude Code"
date: 2026-02-16T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- sameer
- ishaan-alt
- krrish
tags: [incident-report, anthropic, stability]
hide_table_of_contents: false
---
@ -173,5 +164,5 @@ curl -X POST "https://your-proxy-url/reload/anthropic_beta_headers" \
## Related documentation
- [Managing Anthropic Beta Headers](../proxy/sync_anthropic_beta_headers.md) - Complete configuration guide
- [Managing Anthropic Beta Headers](../../docs/proxy/sync_anthropic_beta_headers) - Complete configuration guide
- [`anthropic_beta_headers_config.json`](https://github.com/BerriAI/litellm/blob/main/litellm/anthropic_beta_headers_config.json) - Current configuration file

17
docs/my-website/blog/claude_opus_4_6/index.md
View file

@ -3,18 +3,9 @@ slug: claude_opus_4_6
title: "Day 0 Support: Claude Opus 4.6"
date: 2026-02-05T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- sameer
- ishaan-alt
- krrish
description: "Day 0 support for Claude Opus 4.6 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
tags: [anthropic, claude, opus 4.6]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports Claude Opus 4.6 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
{/* truncate */}
## Docker Image
```bash

12
docs/my-website/blog/claude_sonnet_4_6/index.md
View file

@ -3,14 +3,8 @@ slug: claude_sonnet_4_6
title: "Day 0 Support: Claude Sonnet 4.6"
date: 2026-02-17T10:00:00
authors:
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- ishaan-alt
- krrish
description: "Day 0 support for Claude Sonnet 4.6 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
tags: [anthropic, claude, sonnet 4.6]
hide_table_of_contents: false
@ -21,6 +15,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports Claude Sonnet 4.6 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
{/* truncate */}
## Docker Image
```bash

15
docs/my-website/blog/fastapi_middleware_performance/index.mdx
View file

@ -3,18 +3,9 @@ slug: fastapi-middleware-performance
title: "Your Middleware Could Be a Bottleneck"
date: 2026-02-07T10:00:00
authors:
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Ryan Crabbe
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
image_url: https://media.licdn.com/dms/image/v2/D5603AQHt1t9Z4BJ6Gw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1724453682340?e=1772064000&v=beta&t=VXdmr13rsNB05wyA2F1TENOB5UuDHUZ0FCHTolNyR5M
- krrish
- ishaan-alt
- ryan
description: "How we improved LiteLLM proxy latency and throughput by replacing a single middleware base class"
tags: [performance, fastapi, middleware]
hide_table_of_contents: false

20
docs/my-website/blog/gemin_3.1/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3_1_pro
title: "DAY 0 Support: Gemini 3.1 Pro on LiteLLM"
date: 2026-02-19T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to using Gemini 3.1 Pro on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
@ -28,6 +19,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports `gemini-3.1-pro-preview` and all the new API changes along with it.
{/* truncate */}
## Deploy this version
<Tabs>
@ -67,7 +60,7 @@ LiteLLM provides **full end-to-end support** for Gemini 3.1 Pro on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
@ -147,4 +140,3 @@ curl -X POST http://localhost:4000/v1/chat/completions \
| `high` | `high` |
| `disable` | `minimal` |
| `none` | `minimal` |

26
docs/my-website/blog/gemini_3/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3
title: "DAY 0 Support: Gemini 3 on LiteLLM"
date: 2025-11-19T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Common questions and best practices for using gemini-3-pro-preview with LiteLLM Proxy and SDK."
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
@ -29,6 +20,8 @@ This guide covers common questions and best practices for using `gemini-3-pro-pr
:::
{/* truncate */}
## Quick Start
<Tabs>
@ -976,8 +969,7 @@ messages.append(response.choices[0].message) # ✅ Includes thought signatures
## Additional Resources
- [Gemini Provider Documentation](../gemini.md)
- [Thought Signatures Guide](../gemini.md#thought-signatures)
- [Reasoning Content Documentation](../../reasoning_content.md)
- [Function Calling Guide](../../function_calling.md)
- [Gemini Provider Documentation](../../docs/providers/gemini)
- [Thought Signatures Guide](../../docs/providers/gemini#thought-signatures)
- [Reasoning Content Documentation](../../docs/reasoning_content)
- [Function Calling Guide](../../docs/completion/function_call)

21
docs/my-website/blog/gemini_3_1_flash_lite/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3_1_flash_lite_preview
title: "DAY 0 Support: Gemini 3.1 Flash Lite Preview on LiteLLM"
date: 2026-03-03T08:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to using Gemini 3.1 Flash Lite Preview on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms, supernova]
hide_table_of_contents: false
@ -32,6 +23,8 @@ LiteLLM now supports `gemini-3.1-flash-lite-preview` with full day 0 support!
If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
:::
{/* truncate */}
## Deploy this version
<Tabs>
@ -150,7 +143,7 @@ LiteLLM provides **full end-to-end support** for Gemini 3.1 Flash Lite Preview o
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
@ -172,4 +165,4 @@ LiteLLM automatically maps OpenAI's `reasoning_effort` parameter to Gemini's `th
| `medium` | `medium` | Balanced reasoning for moderate complexity |
| `high` | `high` | Maximum reasoning depth, complex problems |
| `disable` | `minimal` | Disable extended reasoning |
| `none` | `minimal` | No extended reasoning |
| `none` | `minimal` | No extended reasoning |

20
docs/my-website/blog/gemini_3_flash/index.md
View file

@ -3,18 +3,9 @@ slug: gemini_3_flash
title: "DAY 0 Support: Gemini 3 Flash on LiteLLM"
date: 2025-12-17T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Guide to using Gemini 3 Flash on LiteLLM Proxy and SDK with day 0 support."
tags: [gemini, day 0 support, llms]
hide_table_of_contents: false
@ -32,6 +23,8 @@ LiteLLM now supports `gemini-3-flash-preview` and all the new API changes along
If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
:::
{/* truncate */}
## Deploy this version
<Tabs>
@ -80,7 +73,7 @@ LiteLLM provides **full end-to-end support** for Gemini 3 Flash on:
- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent.md) compatible endpoint
- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint
All endpoints support:
- Streaming and non-streaming responses
- Function calling with thought signatures
@ -252,4 +245,3 @@ If using this model via vertex_ai, keep the location as global as this is the on
| `high` | `high` |
| `disable` | `minimal` |
| `none` | `minimal` |

7
docs/my-website/blog/gemini_embedding_2_multimodal/index.md
View file

@ -3,10 +3,7 @@ slug: gemini_embedding_2_multimodal
title: "Gemini Embedding 2 Preview: Multimodal Embeddings on LiteLLM"
date: 2025-03-11T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- sameer
description: "Generate embeddings from text, images, audio, video, and PDFs with gemini-embedding-2-preview on LiteLLM via Gemini API and Vertex AI."
tags: [gemini, embeddings, multimodal, vertex ai]
hide_table_of_contents: false
@ -19,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports **multimodal embeddings** with `gemini-embedding-2-preview`—generating a single embedding from a mix of text, images, audio, video, and PDF content. Available via both the **Gemini API** (API key) and **Vertex AI** (GCP credentials).
{/* truncate */}
## Supported Input Types
| Modality | Supported Formats |

17
docs/my-website/blog/gpt_5_3_codex/index.md
View file

@ -3,18 +3,9 @@ slug: gpt_5_3_codex
title: "Day 0 Support: GPT-5.3-Codex"
date: 2026-02-24T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Day 0 support for GPT-5.3-Codex on LiteLLM, including phase parameter handling for Responses API."
tags: [openai, gpt-5.3-codex, codex, day 0 support]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports GPT-5.3-Codex on Day 0, including support for the new assistant `phase` metadata on Responses API output items.
{/* truncate */}
## Why `phase` matters for GPT-5.3-Codex
`phase` appears on assistant output items and helps distinguish preamble/commentary turns from final closeout responses.

17
docs/my-website/blog/gpt_5_4/index.md
View file

@ -3,18 +3,9 @@ slug: gpt_5_4
title: "Day 0 Support: GPT-5.4"
date: 2026-03-05T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "GPT-5.4 model support in LiteLLM"
tags: [openai, gpt-5.4, completion]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports fully GPT-5.4!
{/* truncate */}
## Docker Image
```bash

16
docs/my-website/blog/httpx_cache_eviction_incident/index.md
View file

@ -3,17 +3,9 @@ slug: httpx-cache-eviction-incident
title: "Incident Report: Cache Eviction Closes In-Use httpx Clients"
date: 2026-02-27T10:00:00
authors:
- name: Ryan Crabbe
title: Performance Engineer, LiteLLM
url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- ryan
- ishaan-alt
- krrish
tags: [incident-report, caching, stability]
hide_table_of_contents: false
---
@ -31,6 +23,8 @@ A change to improve Redis connection pool cleanup introduced a regression that c
**Impact:** Any proxy instance that hit the cache TTL (default 10 minutes) or capacity limit (200 entries) would have its httpx clients closed out from under it, causing requests to LLM providers to fail with connection errors.
{/* truncate */}
---
## Background

18
docs/my-website/blog/litellm_observatory/index.md
View file

@ -3,18 +3,9 @@ slug: litellm-observatory
title: "Improve release stability with 24 hour load tests"
date: 2026-02-06T10:00:00
authors:
- name: Alexsander Hamir
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- alexsander
- krrish
- ishaan-alt
description: "How we built a long-running, release-validation system to catch regressions before they reach users."
tags: [testing, observability, reliability, releases]
hide_table_of_contents: false
@ -28,6 +19,8 @@ As LiteLLM adoption has grown, so have expectations around reliability, performa
This post introduces **LiteLLM Observatory**, a long-running release-validation system we built to catch regressions before they reach users.
{/* truncate */}
---
## Why We Built the Observatory
@ -133,4 +126,3 @@ Reliability is an ongoing investment.
LiteLLM Observatory is one of several systems we’re building to continuously raise the bar on release quality and operational safety. As LiteLLM evolves, so will our validation tooling, informed by real-world usage and lessons learned.
We’ll continue to share those improvements openly as we go.

17
docs/my-website/blog/minimax_m2_5/index.md
View file

@ -3,18 +3,9 @@ slug: minimax_m2_5
title: "Day 0 Support: MiniMax-M2.5"
date: 2026-02-12T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Day 0 support for MiniMax-M2.5 on LiteLLM"
tags: [minimax, M2.5, llm]
hide_table_of_contents: false
@ -25,6 +16,8 @@ import TabItem from '@theme/TabItem';
LiteLLM now supports MiniMax-M2.5 on Day 0. Use it across OpenAI-compatible and Anthropic-compatible APIs through the LiteLLM AI Gateway.
{/* truncate */}
## Supported Models
LiteLLM supports the following MiniMax models:

5
docs/my-website/blog/model_cost_map_incident/index.md
View file

@ -3,10 +3,7 @@ slug: model-cost-map-incident
title: "Incident Report: Invalid model cost map on main"
date: 2026-02-10T10:00:00
authors:
- name: Ishaan Jaffer
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/ishaanjaffer/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- ishaan
tags: [incident-report, stability]
hide_table_of_contents: false
---

18
docs/my-website/blog/realtime_webrtc_http_endpoints/index.md
View file

@ -3,18 +3,9 @@ slug: realtime_webrtc_http_endpoints
title: "Realtime WebRTC HTTP Endpoints"
date: 2026-03-12T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "Use the LiteLLM proxy to route OpenAI-style WebRTC realtime via HTTP: client_secrets and SDP exchange."
tags: [realtime, webrtc, proxy, openai]
hide_table_of_contents: false
@ -24,6 +15,8 @@ import WebRTCTester from '@site/src/components/WebRTCTester';
Connect to the Realtime API via WebRTC from browser/mobile clients. LiteLLM handles auth and key management.
{/* truncate */}
## How it works
![WebRTC flow: Browser, LiteLLM Proxy, and OpenAI/Azure](../../img/webrtc_flow.png)
@ -116,4 +109,3 @@ A: Set the correct `api_version` in `litellm_params` (or via the `AZURE_API_VERS
**Q: What if I get no audio?**
A: Make sure you grant microphone permission, ensure `pc.ontrack` assigns the audio element with `autoplay` enabled, check your network/firewall for WebRTC traffic, and inspect the browser console for ICE or SDP errors.

15
docs/my-website/blog/responses_api_encrypted_content_incident/index.md
View file

@ -3,18 +3,9 @@ slug: responses-api-encrypted-content-incident
title: "Incident Report: Encrypted Content Failures in Multi-Region Responses API Load Balancing"
date: 2026-02-24T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
tags: [incident-report, proxy, responses-api, load-balancing]
hide_table_of_contents: false
---

14
docs/my-website/blog/server_root_path/index.md
View file

@ -3,17 +3,9 @@ slug: server-root-path-incident
title: "Incident Report: SERVER_ROOT_PATH regression broke UI routing"
date: 2026-02-21T10:00:00
authors:
- name: Yuneng Jiang
title: SWE @ LiteLLM (Full Stack)
url: https://www.linkedin.com/in/yunengjiang/
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- yuneng
- ishaan-alt
- krrish
tags: [incident-report, ui, stability]
hide_table_of_contents: false
---

17
docs/my-website/blog/sub_millisecond_proxy_overhead/index.md
View file

@ -3,18 +3,9 @@ slug: sub-millisecond-proxy-overhead
title: "Achieving Sub-Millisecond Proxy Overhead"
date: 2026-02-02T10:00:00
authors:
- name: Alexsander Hamir
title: "Performance Engineer, LiteLLM"
url: https://www.linkedin.com/in/alexsander-baptista/
image_url: https://github.com/AlexsanderHamir.png
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- alexsander
- krrish
- ishaan-alt
description: "Our Q1 performance target and architectural direction for achieving sub-millisecond proxy overhead on modest hardware."
tags: [performance, architecture]
hide_table_of_contents: false
@ -32,6 +23,8 @@ Proxy overhead refers to the latency introduced by LiteLLM itself, independent o
To measure it, we run the same workload directly against the provider and through LiteLLM at identical QPS (for example, 1,000 QPS) and compare the latency delta. To reduce noise, the load generator, LiteLLM, and a mock LLM endpoint all run on the same machine, ensuring the difference reflects proxy overhead rather than network latency.
{/* truncate */}
---
## Where We're Coming From

19
docs/my-website/blog/video_characters_litellm/index.md
View file

@ -3,18 +3,9 @@ slug: video_characters_api
title: "New Video Characters, Edit and Extension API support"
date: 2026-03-16T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
description: "LiteLLM now supports creating, retrieving, and managing reusable video characters across multiple video generations."
tags: [videos, characters, proxy, routing]
hide_table_of_contents: false
@ -22,6 +13,8 @@ hide_table_of_contents: false
LiteLLM now supoports videos character, edit and extension apis.
{/* truncate */}
## What's New
Four new endpoints for video character operations:
@ -125,4 +118,4 @@ Router knows exactly which deployment to use
**Behind the scenes:**
- Character ID format: `character_<base64_encoded_metadata>`
- Metadata includes: provider, model_id, original_character_id
- Transparent to you - just use the ID, LiteLLM handles routing
- Transparent to you - just use the ID, LiteLLM handles routing

15
docs/my-website/blog/vllm_embeddings_incident/index.md
View file

@ -3,18 +3,9 @@ slug: vllm-embeddings-incident
title: "Incident Report: vLLM Embeddings Broken by encoding_format Parameter"
date: 2026-02-18T10:00:00
authors:
- name: Sameer Kankute
title: SWE @ LiteLLM (LLM Translation)
url: https://www.linkedin.com/in/sameer-kankute/
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
- name: Krrish Dholakia
title: "CEO, LiteLLM"
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
- sameer
- krrish
- ishaan-alt
tags: [incident-report, embeddings, vllm]
hide_table_of_contents: false
---

78
docs/my-website/docs/guides/index.md Normal file
View file

@ -0,0 +1,78 @@
---
title: Guides
sidebar_label: Overview
---
import NavigationCards from '@site/src/components/NavigationCards';
**Guides** are focused references organized by the job you are trying to do with LiteLLM: make requests, use tools, handle media, manage context, or operate the gateway safely.
> New to LiteLLM or not sure whether you need the SDK or Gateway path first? Start at [Learn →](/docs/learn)
---
## Build With LiteLLM
<NavigationCards
columns={3}
items={[
{
icon: "⚡",
title: "Core Requests",
description: "Streaming, batching, structured outputs, and reasoning behavior.",
to: "/docs/guides/core_request_response_patterns",
},
{
icon: "🛠️",
title: "Tool Calling",
description: "Function calling, web tools, interception patterns, computer use, code interpreter, and tool-call hygiene.",
to: "/docs/guides/tools_integrations",
},
{
icon: "🖼️",
title: "Multimodal I/O",
description: "Vision, audio, PDFs, image generation, and video generation.",
to: "/docs/guides/multimodal_io",
},
{
icon: "📚",
title: "Retrieval & Knowledge",
description: "Vector stores, file search, citations, and knowledge-base routing.",
to: "/docs/guides/retrieval_knowledge",
},
{
icon: "🧠",
title: "Prompts & Context",
description: "Prompt caching, trimming, formatting, assistant prefill, and predicted outputs.",
to: "/docs/guides/prompts_context",
},
]}
/>
---
## Operate & Extend
<NavigationCards
columns={3}
items={[
{
icon: "🎛️",
title: "Compatibility & Extensibility",
description: "Provider-specific params, model aliases, fine-tuned models, and adapters.",
to: "/docs/guides/compatibility_extensibility",
},
{
icon: "🧪",
title: "Reliability, Testing & Spend",
description: "Retries, fallbacks, mock responses, and budget controls.",
to: "/docs/guides/reliability_testing_spend",
},
{
icon: "🔒",
title: "Security & Network",
description: "SSL, custom CA bundles, HTTP proxy settings, and per-service verification.",
to: "/docs/guides/security_network",
},
]}
/>

1051
docs/my-website/docs/index.md
View file

File diff suppressed because it is too large Load diff

340
docs/my-website/docs/integrations/index.md
View file

@ -1,18 +1,336 @@
# Integrations
---
title: Integrations
sidebar_label: Overview
---
import NavigationCards from '@site/src/components/NavigationCards';
This section covers integrations with various tools and services that can be used with LiteLLM (either Proxy or SDK).
## AI Agent Frameworks
- **[Letta](./letta.md)** - Build stateful LLM agents with persistent memory using LiteLLM Proxy
---
## Development Tools
- **[OpenWebUI](../tutorials/openweb_ui.md)** - Self-hosted ChatGPT-style interface
## Observability
## Observability & Monitoring
- **[Langfuse](../observability/langfuse_integration.md)** - LLM observability and analytics
- **[Prometheus](../proxy/prometheus.md)** - Metrics collection and monitoring
- **[PagerDuty](../proxy/pagerduty.md)** - Incident response and alerting
- **[Datadog](../observability/datadog.md)**
Track, debug, and analyze LLM calls with observability platforms.
<NavigationCards
columns={3}
items={[
{
icon: "🪢",
title: "Langfuse",
description: "LLM observability and analytics.",
to: "/docs/observability/langfuse_integration",
},
{
icon: "🐶",
title: "Datadog",
description: "Metrics, traces, and dashboards.",
to: "/docs/observability/datadog",
},
{
icon: "📡",
title: "OpenTelemetry",
description: "Vendor-neutral tracing.",
to: "/docs/observability/opentelemetry_integration",
},
{
icon: "🔗",
title: "LangSmith",
description: "LLM debugging and evaluation.",
to: "/docs/observability/langsmith_integration",
},
{
icon: "🔥",
title: "Arize / Phoenix",
description: "ML observability and evaluation.",
to: "/docs/observability/arize_integration",
},
{
icon: "🌀",
title: "Helicone",
description: "LLM request logging and analytics.",
to: "/docs/observability/helicone_integration",
},
{
icon: "📊",
title: "MLflow",
description: "Experiment tracking.",
to: "/docs/observability/mlflow",
},
{
icon: "🏋️",
title: "Weights & Biases",
description: "ML experiment tracking.",
to: "/docs/observability/wandb_integration",
},
{
icon: "📉",
title: "PostHog",
description: "Product analytics.",
to: "/docs/observability/posthog_integration",
},
]}
/>
Click into each section to learn more about the integrations.
[View all observability integrations →](/docs/integrations/observability_integrations)
---
## Alerting & Monitoring
Set up alerts, metrics collection, and infrastructure monitoring.
<NavigationCards
columns={2}
items={[
{
icon: "📈",
title: "Prometheus",
description: "Metrics collection and monitoring.",
to: "../proxy/prometheus",
},
{
icon: "🚨",
title: "PagerDuty",
description: "Incident response and alerting.",
to: "../proxy/pagerduty",
},
{
icon: "🔔",
title: "Alerting",
description: "Slack, Teams, and webhook alerts.",
to: "../proxy/alerting",
},
{
icon: "🔍",
title: "Pyroscope",
description: "Continuous profiling.",
to: "../proxy/pyroscope_profiling",
},
]}
/>
---
## Guardrail Providers
Add safety and content filtering to LLM calls.
<NavigationCards
columns={3}
items={[
{
icon: "🛡️",
title: "Lakera AI",
description: "Prompt injection detection.",
to: "/docs/proxy/guardrails/lakera_ai",
},
{
icon: "☁️",
title: "Azure Content Safety",
description: "Content moderation.",
to: "/docs/proxy/guardrails/azure_content_guardrail",
},
{
icon: "🛏️",
title: "Bedrock Guardrails",
description: "AWS Bedrock safety.",
to: "/docs/proxy/guardrails/bedrock",
},
{
icon: "🤖",
title: "OpenAI Moderation",
description: "OpenAI content policy.",
to: "/docs/proxy/guardrails/openai_moderation",
},
{
icon: "🔐",
title: "Secret Detection",
description: "Prevent credential leaks.",
to: "/docs/proxy/guardrails/secret_detection",
},
{
icon: "🕵️",
title: "PII Masking",
description: "Mask sensitive data.",
to: "/docs/proxy/guardrails/pii_masking_v2",
},
]}
/>
[View all guardrail providers →](/docs/guardrail_providers)
---
## Policies
Define and enforce usage policies across your LLM deployment.
<NavigationCards
columns={3}
items={[
{
icon: "📋",
title: "Guardrail Policies",
description: "Policy-based guardrail rules.",
to: "../proxy/guardrails/guardrail_policies",
},
{
icon: "🔀",
title: "Policy Flow Builder",
description: "Visual policy configuration.",
to: "../proxy/guardrails/policy_flow_builder",
},
{
icon: "📄",
title: "Policy Templates",
description: "Pre-built policy templates.",
to: "../proxy/guardrails/policy_templates",
},
]}
/>
---
## AI Tools
Connect LiteLLM to AI-powered coding and productivity tools.
<NavigationCards
columns={3}
items={[
{
icon: "💬",
title: "OpenWebUI",
description: "Self-hosted ChatGPT-style interface.",
to: "../tutorials/openweb_ui",
},
{
icon: "🤖",
title: "Claude Code",
description: "Use LiteLLM with Claude Code.",
to: "../tutorials/claude_responses_api",
},
{
icon: "🖱️",
title: "Cursor",
description: "AI code editor integration.",
to: "../tutorials/cursor_integration",
},
{
icon: "🐙",
title: "GitHub Copilot",
description: "GitHub Copilot integration.",
to: "../tutorials/github_copilot_integration",
},
{
icon: "💻",
title: "OpenCode",
description: "Open source coding assistant.",
to: "../tutorials/opencode_integration",
},
{
icon: "🔧",
title: "Retool Assist",
description: "Retool AI assistant.",
to: "../tutorials/retool_assist",
},
]}
/>
---
## Agent SDKs
Use LiteLLM with agent frameworks and SDKs.
<NavigationCards
columns={3}
items={[
{
icon: "🤖",
title: "OpenAI Agents SDK",
description: "Build agents with OpenAI's SDK.",
to: "../tutorials/openai_agents_sdk",
},
{
icon: "🧠",
title: "Claude Agent SDK",
description: "Build agents with Anthropic's SDK.",
to: "../tutorials/claude_agent_sdk",
},
{
icon: "🌐",
title: "Google ADK",
description: "Google Agent Development Kit.",
to: "../tutorials/google_adk",
},
{
icon: "🚀",
title: "CopilotKit",
description: "In-app AI copilots.",
to: "../tutorials/copilotkit_sdk",
},
{
icon: "🧬",
title: "Letta",
description: "Build stateful LLM agents with persistent memory.",
to: "./letta",
},
{
icon: "🎙️",
title: "LiveKit",
description: "Real-time voice and video AI agents.",
to: "../tutorials/livekit_xai_realtime",
},
]}
/>
---
## Prompt Management
Manage, version, and deploy prompts.
<NavigationCards
columns={3}
items={[
{
icon: "📝",
title: "LiteLLM Prompt Management",
description: "Built-in prompt management.",
to: "../proxy/litellm_prompt_management",
},
{
icon: "🔌",
title: "Custom Prompt Management",
description: "Bring your own prompt store.",
to: "../proxy/custom_prompt_management",
},
{
icon: "🔥",
title: "Arize Phoenix Prompts",
description: "Prompt management with Phoenix.",
to: "../proxy/arize_phoenix_prompts",
},
]}
/>
---
## Manage with AI Agents
Use AI agents to manage your LiteLLM deployment — create users, teams, keys, models, and more via natural language.
<NavigationCards
columns={1}
items={[
{
icon: "🤖",
title: "LiteLLM Skills",
description: "Manage LiteLLM via Claude Code — create keys, teams, models, and more using natural language commands.",
to: "../tutorials/claude_code_skills",
},
]}
/>

12
docs/my-website/docs/integrations/letta.md
View file

@ -920,9 +920,9 @@ for model in models:
## Resources
- [Letta Documentation](https://docs.letta.ai/)
- [LiteLLM Proxy Documentation](../proxy/quick_start.md)
- [LiteLLM SDK Documentation](../completion/input.md)
- [Function Calling Guide](../completion/function_call.md)
- [Observability Setup](../observability/langfuse_integration.md)
- [Router Configuration](../routing.md)
- [Letta Documentation](https://docs.letta.com/)
- [LiteLLM Proxy Documentation](/docs/simple_proxy)
- [LiteLLM SDK Documentation](/docs/#litellm-python-sdk)
- [Function Calling Guide](/docs/completion/function_call)
- [Observability Setup](/docs/integrations/observability_integrations)
- [Router Configuration](/docs/routing)

28
docs/my-website/docs/integrations/observability_index.md Normal file
View file

@ -0,0 +1,28 @@
---
title: Observability
sidebar_label: Overview
slug: observability_integrations
---
Track, debug, and analyze LLM calls with observability platforms.
import NavigationCards from '@site/src/components/NavigationCards';
## Observability Integrations
<NavigationCards
columns={3}
items={[
{ icon: "🪢", title: "Langfuse", description: "LLM observability and analytics.", to: "/docs/observability/langfuse_integration" },
{ icon: "🐶", title: "Datadog", description: "Metrics, traces, and dashboards.", to: "/docs/observability/datadog" },
{ icon: "📡", title: "OpenTelemetry", description: "Vendor-neutral tracing.", to: "/docs/observability/opentelemetry_integration" },
{ icon: "🔗", title: "LangSmith", description: "LLM debugging and evaluation.", to: "/docs/observability/langsmith_integration" },
{ icon: "🔥", title: "Arize / Phoenix", description: "ML observability and evaluation.", to: "/docs/observability/arize_integration" },
{ icon: "🌀", title: "Helicone", description: "LLM request logging and analytics.", to: "/docs/observability/helicone_integration" },
{ icon: "📊", title: "MLflow", description: "Experiment tracking.", to: "/docs/observability/mlflow" },
{ icon: "🏋️", title: "Weights & Biases", description: "ML experiment tracking.", to: "/docs/observability/wandb_integration" },
{ icon: "📉", title: "PostHog", description: "Product analytics.", to: "/docs/observability/posthog_integration" },
]}
/>
[View all observability integrations →](/docs/observability/callbacks)

2
docs/my-website/docs/integrations/websearch_interception.md
View file

@ -375,7 +375,7 @@ search_tools:
- [Search Providers](../search/index.md) - Detailed search provider setup
- [Claude Code WebSearch](../tutorials/claude_code_websearch.md) - Using with Claude Code
- [Tool Calling](../completion/function_call.md) - General tool calling documentation
- [Callbacks](./custom_callback.md) - Custom callback documentation
- [Callbacks](../observability/custom_callback.md) - Custom callback documentation
## Technical Details

174
docs/my-website/docs/learn/gateway_quickstart.md Normal file
View file

@ -0,0 +1,174 @@
---
title: Gateway Quickstart
sidebar_label: Gateway Quickstart
description: Start LiteLLM Gateway, add models and keys, then connect applications and SDKs to one shared endpoint.
---
import NavigationCards from '@site/src/components/NavigationCards';
Use this path if you need one shared OpenAI-compatible endpoint for a team or platform.
If you need a Docker or database-first setup, use the [Docker + Database tutorial](/docs/proxy/docker_quick_start). Otherwise, use the steps below to get to a working request fast.
## 1. Install The Gateway
```bash
pip install 'litellm[proxy]'
```
## 2. Set One Provider Key
```bash
export OPENAI_API_KEY="your-api-key"
```
## 3. Create `config.yaml`
```yaml
model_list:
- model_name: gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
general_settings:
master_key: sk-1234
```
## 4. Start The Gateway
```bash
litellm --config config.yaml
```
You should see the proxy start on `http://0.0.0.0:4000`.
## 5. Send Your First Request
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Hello from LiteLLM Gateway"}
]
}'
```
## 6. Check The Response
If the request succeeds, the proxy returns `200 OK` with an OpenAI-style response.
The assistant text will be in:
```json
choices[0].message.content
```
If your gateway is routing to OpenAI, a real response can look like this:
```json
{
"id": "chatcmpl-abc123",
"created": 1677858242,
"model": "gpt-4o-mini-2024-07-18",
"object": "chat.completion",
"system_fingerprint": "fp_406d6473f8",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! How can I assist you today?",
"tool_calls": null,
"function_call": null,
"annotations": []
}
}
],
"usage": {
"completion_tokens": 9,
"prompt_tokens": 13,
"total_tokens": 22,
"completion_tokens_details": {
"accepted_prediction_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 0,
"rejected_prediction_tokens": 0
},
"prompt_tokens_details": {
"audio_tokens": 0,
"cached_tokens": 0
}
},
"service_tier": "default"
}
```
`id`, `created`, the resolved model version, token counts, and message text will vary by request. Other providers may return a smaller or slightly different set of fields, but `choices[0].message.content` is the main field to read.
## 7. Add Keys And The UI
If you need virtual keys, spend tracking, or the admin UI, add a database next.
- Add `database_url` under `general_settings`
- Use [Virtual keys](/docs/proxy/virtual_keys) for key creation and budgets
- Use [Admin UI](/docs/proxy/ui) to manage models and keys
- Use the [Docker + Database tutorial](/docs/proxy/docker_quick_start) if you want a fuller setup
## 8. Pick Your Next Step
<NavigationCards
columns={3}
items={[
{
icon: "🖥️",
title: "Make LLM Requests",
description: "Point LiteLLM or OpenAI-compatible clients to the gateway.",
to: "/docs/proxy/user_keys",
},
{
icon: "🎛️",
title: "Model Config",
description: "Add more models and gateway settings.",
to: "/docs/proxy/configs",
},
{
icon: "🔑",
title: "Virtual Keys",
description: "Create keys, budgets, and access controls.",
to: "/docs/proxy/virtual_keys",
},
{
icon: "📈",
title: "Add Logging",
description: "Capture logs, spend, and traces.",
to: "/docs/proxy/logging",
},
{
icon: "🔀",
title: "Load Balance",
description: "Route across deployments, regions, or providers.",
to: "/docs/proxy/load_balancing",
},
{
icon: "🛡️",
title: "Add Guardrails",
description: "Add safety checks and policy enforcement.",
to: "/docs/proxy/guardrails/quick_start",
},
{
icon: "📊",
title: "Reliability",
description: "Configure retries, fallbacks, and timeouts.",
to: "/docs/proxy/reliability",
},
]}
/>
## When To Use The SDK Path Instead
If you only need to call models from one application and do not need centralized auth or shared infrastructure, start with the [SDK Quickstart](/docs/learn/sdk_quickstart) instead.

117
docs/my-website/docs/learn/index.md Normal file
View file

@ -0,0 +1,117 @@
---
title: Learn LiteLLM
sidebar_label: Learn
slug: /learn
---
import NavigationCards from '@site/src/components/NavigationCards';
LiteLLM gives you one OpenAI-compatible interface for 100+ LLM providers. Start with the path that matches your setup.
---
## Start Here
Pick one path first.
<NavigationCards
columns={2}
items={[
{
icon: "🐍",
title: "SDK Quickstart",
description: "Use LiteLLM directly in application code.",
listDescription: [
"Install",
"First request",
"Next SDK features",
],
to: "/docs/learn/sdk_quickstart",
},
{
icon: "🖥️",
title: "Gateway Quickstart",
description: "Run LiteLLM as a shared gateway.",
listDescription: [
"Start proxy",
"Add models and keys",
"Connect clients",
],
to: "/docs/learn/gateway_quickstart",
},
]}
/>
---
## Common Tasks
Jump to a specific task.
<NavigationCards
columns={3}
items={[
{
icon: "⚡",
title: "Stream Responses",
description: "Return tokens as they are generated.",
to: "/docs/guides/core_request_response_patterns",
},
{
icon: "🧰",
title: "Use Tools",
description: "Add function calling to your app.",
to: "/docs/guides/tools_integrations",
},
{
icon: "🔀",
title: "Add Routing",
description: "Retries, fallbacks, and load balancing.",
to: "/docs/routing-load-balancing",
},
{
icon: "🔑",
title: "Set Up Keys",
description: "Gateway auth, virtual keys, and access control.",
to: "/docs/proxy/virtual_keys",
},
{
icon: "📈",
title: "Add Logging",
description: "Capture request logs and spend data.",
to: "/docs/proxy/logging",
},
{
icon: "🌐",
title: "Choose A Provider",
description: "Find provider-specific auth and params.",
to: "/docs/providers",
},
]}
/>
---
## Docs Map
Use these when you already know the type of doc you want.
<NavigationCards
columns={2}
items={[
{
icon: "📚",
title: "Guides",
description: "Feature reference.",
to: "/docs/guides",
},
{
icon: "🛠️",
title: "Tutorials",
description: "Step-by-step integrations.",
to: "/docs/tutorials",
},
]}
/>
Not sure where to start? Use [SDK Quickstart](/docs/learn/sdk_quickstart) for app code or [Gateway Quickstart](/docs/learn/gateway_quickstart) for shared infrastructure.

174
docs/my-website/docs/learn/sdk_quickstart.md Normal file
View file

@ -0,0 +1,174 @@
---
title: SDK Quickstart
sidebar_label: SDK Quickstart
description: Make your first LiteLLM SDK call, then jump to the right docs for the next feature you need.
---
import NavigationCards from '@site/src/components/NavigationCards';
Use this path if you are integrating LiteLLM directly into application code.
## 1. Install LiteLLM
```bash
pip install litellm
```
## 2. Set Provider Credentials
Start with one provider and set its environment variables.
- OpenAI: `OPENAI_API_KEY`
- Anthropic: `ANTHROPIC_API_KEY`
- Azure OpenAI: `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`
- Bedrock: standard AWS credentials
- Vertex AI: `VERTEXAI_PROJECT`, `VERTEXAI_LOCATION`
If you have not picked a provider yet, browse [all supported providers](/docs/providers).
## 3. Make Your First Call
```python
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
response = completion(
model="openai/gpt-4o",
messages=[{"role": "user", "content": "Hello, how are you?"}],
)
print(response.choices[0].message.content)
```
## 4. Check The Response
The line below:
```python
print(response.choices[0].message.content)
```
prints the assistant text, for example:
```text
Hello! I'm doing well, thanks for asking.
```
If you print the full object with:
```python
print(response)
```
you will see a Python `ModelResponse(...)` object. For an OpenAI-backed model, it can look like this:
```python
ModelResponse(
id='chatcmpl-abc123',
created=1773782130,
model='gpt-4o-2024-08-06',
object='chat.completion',
system_fingerprint='fp_4ff89bf575',
choices=[
Choices(
finish_reason='stop',
index=0,
message=Message(
content="Hello! I'm just a program, but I'm here to help you. How can I assist you today?",
role='assistant',
tool_calls=None,
function_call=None,
provider_specific_fields={'refusal': None},
annotations=[]
),
provider_specific_fields={}
)
],
usage=Usage(
completion_tokens=21,
prompt_tokens=13,
total_tokens=34,
completion_tokens_details=CompletionTokensDetailsWrapper(...),
prompt_tokens_details=PromptTokensDetailsWrapper(...)
),
service_tier='default'
)
```
The same response follows an OpenAI-style shape. Conceptually, it looks like this:
```json
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1677858242,
"model": "gpt-4o",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! I'm doing well, thanks for asking."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 13,
"completion_tokens": 12,
"total_tokens": 25
}
}
```
`id`, `created`, token counts, and message text will vary by request.
If you call an OpenAI-backed model, you may also see extra fields such as `system_fingerprint`, `service_tier`, `tool_calls`, `function_call`, `annotations`, `provider_specific_fields`, and detailed token usage. For the full output reference, see [completion output](/docs/completion/output).
Need more provider examples? See the main [Getting Started](/docs/#quick-start) page.
## 5. Pick Your Next Step
<NavigationCards
columns={3}
items={[
{
icon: "⚡",
title: "Stream Responses",
description: "Receive tokens incrementally with stream=True.",
to: "/docs/completion/stream",
},
{
icon: "🧰",
title: "Use Tools",
description: "Add function calling in a provider-agnostic way.",
to: "/docs/completion/function_call",
},
{
icon: "📦",
title: "Return JSON",
description: "Constrain responses to structured JSON output.",
to: "/docs/completion/json_mode",
},
{
icon: "🔀",
title: "Add Routing",
description: "Use retries, fallbacks, and load balancing in app code.",
to: "/docs/routing",
},
{
icon: "🌐",
title: "Choose A Provider",
description: "Find provider-specific auth, model naming, and params.",
to: "/docs/providers",
},
]}
/>
## When To Use Gateway Instead
Use LiteLLM Gateway if you need centralized auth, virtual keys, spend tracking, shared logging, or one OpenAI-compatible endpoint for multiple apps.
[Go to Gateway Quickstart →](/docs/learn/gateway_quickstart)

4
docs/my-website/docs/load_test_advanced.md
View file

@ -11,7 +11,7 @@ Tutorial on how to get to 1K+ RPS with LiteLLM Proxy on locust
- [Github releases](https://github.com/BerriAI/litellm/releases)
- [litellm docker containers](https://github.com/BerriAI/litellm/pkgs/container/litellm)
- [litellm database docker container](https://github.com/BerriAI/litellm/pkgs/container/litellm-database)
- [ ] Ensure you're following **ALL** [best practices for production](./proxy/production_setup.md)
- [ ] Ensure you're following **ALL** [best practices for production](./proxy/prod.md)
- [ ] Locust - Ensure you're Locust instance can create 1K+ requests per second
- 👉 You can use our **[maintained locust instance here](https://locust-load-tester-production.up.railway.app/)**
- If you're self hosting locust
@ -222,4 +222,4 @@ class MyUser(HttpUser):
def on_start(self):
self.api_key = os.getenv('API_KEY', 'sk-1234')
self.client.headers.update({'Authorization': f'Bearer {self.api_key}'})
```
```

6
docs/my-website/docs/observability/langfuse_otel_integration.md
View file

@ -83,6 +83,9 @@ os.environ["LANGFUSE_OTEL_HOST"] = "https://cloud.langfuse.com" # EU region
# Or use self-hosted instance
# os.environ["LANGFUSE_OTEL_HOST"] = "https://my-langfuse.company.com"
# Optional: Ignore otel context propagation to prevent parent-child relationships with spans from other providers
# os.environ["OTEL_IGNORE_CONTEXT_PROPAGATION"] = "true"
litellm.callbacks = ["langfuse_otel"]
```
@ -124,6 +127,9 @@ export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_OTEL_HOST="https://us.cloud.langfuse.com" # Default US region
# export LANGFUSE_OTEL_HOST="https://otel.my-langfuse.company.com" # custom OTEL endpoint
# Optional: Ignore otel context propagation to prevent parent-child relationships with spans from other providers
# export OTEL_IGNORE_CONTEXT_PROPAGATION="true"
```
2. Setup config.yaml

2
docs/my-website/docs/providers/azure/azure.md
View file

@ -1032,7 +1032,7 @@ print("list_batches_response=", list_batches_response)
</TabItem>
</Tabs>
### [Health Check Azure Batch models](./proxy/health.md#batch-models-azure-only)
### [Health Check Azure Batch models](../../proxy/health.md#batch-models-azure-only)
### [BETA] Loadbalance Multiple Azure Deployments

5
docs/my-website/docs/providers/azure/azure_anthropic.md
View file

@ -372,7 +372,6 @@ response = completion(
## Related Documentation
- [Anthropic Provider Documentation](./anthropic.md) - For standard Anthropic API usage
- [Anthropic Provider Documentation](../anthropic.md) - For standard Anthropic API usage
- [Azure OpenAI Documentation](./azure.md) - For Azure OpenAI models
- [Azure Authentication Guide](../secret_managers/azure_key_vault.md) - For Azure AD token setup
- [Azure Authentication Guide](../../secret_managers/azure_key_vault.md) - For Azure AD token setup

2
docs/my-website/docs/providers/openai.md
View file

@ -1153,4 +1153,4 @@ response = completion(
LiteLLM supports OpenAI's video generation models including Sora.
For detailed documentation on video generation, see [OpenAI Video Generation →](./openai/video_generation.md)
For detailed documentation on video generation, see [OpenAI Video Generation →](./openai/videos.md)

2
docs/my-website/docs/proxy/call_hooks.md
View file

@ -7,7 +7,7 @@ import Image from '@theme/IdealImage';
- Enforce 'user' param for all openai endpoint calls
:::tip
**Understanding Callback Hooks?** Check out our [Callback Management Guide](../observability/callback_management.md) to understand the differences between proxy-specific hooks like `async_pre_call_hook` and general logging hooks like `async_log_success_event`.
**Understanding Callback Hooks?** Check out our [Callback Guide](../observability/callbacks.md) to understand the differences between proxy-specific hooks like `async_pre_call_hook` and general logging hooks like `async_log_success_event`.
:::
## Which Hook Should I Use?

2
docs/my-website/docs/proxy/config_settings.md
View file

@ -364,7 +364,7 @@ router_settings:
| optional_pre_call_checks | List[str] | List of pre-call checks to add to the router. Supported: `router_budget_limiting`, `prompt_caching`, `responses_api_deployment_check`, `encrypted_content_affinity`, `deployment_affinity`, `session_affinity`, `forward_client_headers_by_model_group` |
| deployment_affinity_ttl_seconds | int | TTL (seconds) for user-key → deployment affinity mapping when `deployment_affinity` is enabled (configured at Router init / proxy startup). Defaults to `3600` (1 hour). |
| ignore_invalid_deployments | boolean | If true, ignores invalid deployments. Default for proxy is True - to prevent invalid models from blocking other models from being loaded. |
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search.md) |
| search_tools | List[SearchToolTypedDict] | List of search tool configurations for Search API integration. Each tool specifies a search_tool_name and litellm_params with search_provider, api_key, api_base, etc. [Further Docs](../search/index.md) |
| guardrail_list | List[GuardrailTypedDict] | List of guardrail configurations for guardrail load balancing. Enables load balancing across multiple guardrail deployments with the same guardrail_name. [Further Docs](./guardrails/guardrail_load_balancing.md) |

234
docs/my-website/docs/proxy/docker_quick_start.md
View file

@ -1,6 +1,6 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# Getting Started Tutorial
@ -78,13 +78,13 @@ You can skip the curl install and run `litellm --setup` directly after `pip inst
## Pre-Requisites
- Install LiteLLM Docker Image **OR** LiteLLM CLI (pip package)
Choose your install method. **Docker Compose** users complete their full setup inside the tab and are done. **Docker** and **pip** users continue with the steps below the tabs.
<Tabs>
<TabItem value="docker" label="Docker">
```
```bash
docker pull docker.litellm.ai/berriai/litellm:main-latest
```
@ -102,7 +102,25 @@ $ pip install 'litellm[proxy]'
<TabItem value="docker-compose" label="Docker Compose (Proxy + DB)">
Use this docker compose to spin up the proxy with a postgres database running locally.
Docker Compose bundles LiteLLM with a Postgres database. Follow the steps below — the proxy will be fully running by the end.
### Step 1 — Pull the LiteLLM database image
LiteLLM provides a dedicated `litellm-database` image for proxy deployments that connect to Postgres.
```bash
docker pull ghcr.io/berriai/litellm-database:main-latest
```
See all available tags on the [GitHub Container Registry](https://github.com/BerriAI/litellm/pkgs/container/litellm-database).
---
### Step 2 — Set up a database
Complete all three config files **before** running `docker compose up`. The proxy server will not start correctly if any of these are missing.
#### 2.1 — Get `docker-compose.yml` and create `.env`
```bash
# Get the docker compose file
@ -111,26 +129,154 @@ curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/docker-compose.ym
# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommend - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
# Add the litellm salt key — cannot be changed after adding a model
# Used to encrypt/decrypt your LLM API key credentials
# Generate a strong random value: https://1password.com/password-generator/
echo 'LITELLM_SALT_KEY="sk-1234"' >> .env
# Start
# Add your model credentials
echo 'AZURE_API_BASE="https://openai-***********/"' >> .env
echo 'AZURE_API_KEY="your-azure-api-key"' >> .env
```
#### 2.2 — Create `config.yaml`
The default `docker-compose.yml` starts a Postgres container at `db:5432`. Your `config.yaml` must include `database_url` pointing to it:
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: azure/my_azure_deployment
api_base: os.environ/AZURE_API_BASE
api_key: os.environ/AZURE_API_KEY
api_version: "2025-01-01-preview"
general_settings:
master_key: sk-1234 # 🔑 your proxy admin key (must start with sk-)
database_url: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
```
:::tip
`database_url` enables virtual keys, spend tracking, and the UI. Replace it with your [Supabase](https://supabase.com/) or [Neon](https://neon.tech/) connection string if you prefer a managed database.
:::
#### 2.3 — Create `prometheus.yml`
This file **must exist as a file** before `docker compose up`. If it is missing, Docker auto-creates it as an empty directory and the Prometheus container fails to start.
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "litellm"
static_configs:
- targets: ["litellm:4000"]
```
Also verify that the `config.yaml` volume mount and `--config` flag are **not commented out** in `docker-compose.yml`:
```yaml
services:
litellm:
volumes:
- ./config.yaml:/app/config.yaml # ✅ must be uncommented
command:
- "--config=/app/config.yaml" # ✅ must be uncommented
```
:::warning
All three files (`.env`, `config.yaml`, `prometheus.yml`) must be present before running `docker compose up`. See [Troubleshooting](#troubleshooting) if you run into issues.
:::
---
### Step 3 — Start the proxy server and test it
After `config.yaml`, `prometheus.yml`, and `.env` are complete, start the proxy:
```bash
docker compose up
```
Once running, test it with a curl request:
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
**Expected response:**
```json
{
"id": "chatcmpl-abcd",
"created": 1773817678,
"model": "gpt-4o",
"object": "chat.completion",
"system_fingerprint": "fp_6b1ef07cda",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Hello! How can I assist you today?",
"role": "assistant",
"annotations": []
}
}
],
"usage": {
"completion_tokens": 9,
"prompt_tokens": 9,
"total_tokens": 18,
"completion_tokens_details": {
"accepted_prediction_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 0,
"rejected_prediction_tokens": 0
},
"prompt_tokens_details": {
"audio_tokens": 0,
"cached_tokens": 0
}
},
"service_tier": "default"
}
```
---
### Optional — Navigate to the LiteLLM UI and generate a virtual key
Open [http://localhost:4000/ui](http://localhost:4000/ui) in your browser and log in with your master key (`sk-1234`).
Navigate to **Virtual Keys** and click **+ Create New Key**:
<Image img={require('../../img/litellm_ui_create_key.png')} alt="LiteLLM UI — Create Virtual Key" />
Virtual keys let you track spend, set rate limits, and control model access per user or team.
</TabItem>
</Tabs>
## 1. Add a model
:::note Docker Compose users
Your setup is complete — the steps below are for **Docker** and **pip** users only.
:::
Control LiteLLM Proxy with a config.yaml file.
---
Setup your config.yaml with your azure model.
## Step 1 — Add a model
Note: When using the proxy with a database, you can also **just add models via UI** (UI is available on `/ui` route).
Control LiteLLM Proxy with a `config.yaml` file. Create one with your Azure model:
```yaml
model_list:
@ -154,8 +300,6 @@ You can read more about how model resolution works in the [Model Configuration](
- **`api_base`** (`str`) - The API base for your azure deployment.
- **`api_version`** (`str`) - The API Version to use when calling Azure's OpenAI API. Get the latest Inference API version [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation?source=recommendations#latest-preview-api-releases).
---
---
@ -203,19 +347,19 @@ $ litellm --config /app/config.yaml --detailed_debug
</Tabs>
Confirm your config was loaded correctly — you should see this in the logs:
Confirm your config.yaml got mounted correctly
```bash
```
Loaded config YAML (api_key and environment_variables are not shown):
{
"model_list": [
{
"model_name ...
"model_list": [
{
"model_name": ...
```
### 2.2 Make Call
LiteLLM Proxy is 100% OpenAI-compatible. Test your model via `/chat/completions`:
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
@ -309,15 +453,17 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
- [Other/Non-Chat Completion Endpoints](../embedding/supported_embedding.md)
- [Pass-through for VertexAI, Bedrock, etc.](../pass_through/vertex_ai.md)
## 3. Generate a virtual key
## Optional: Generate a virtual key
Track Spend, and control model access via virtual keys for the proxy
Track spend and control model access via virtual keys for the proxy.
### 3.1 Set up a Database
### Prerequisite — Set up a database
**Requirements**
- Need a postgres database (e.g. [Supabase](https://supabase.com/), [Neon](https://neon.tech/), etc)
:::note Docker Compose users
Your Postgres container is already running — skip ahead to [Create Key w/ RPM Limit](#create-key-w-rpm-limit) below.
:::
**Docker / pip users** — you need a Postgres database (e.g. [Supabase](https://supabase.com/), [Neon](https://neon.tech/), or self-hosted). Add `general_settings` to your `config.yaml`:
```yaml
model_list:
@ -333,7 +479,9 @@ general_settings:
database_url: "postgresql://<user>:<password>@<host>:<port>/<dbname>" # 👈 KEY CHANGE
```
Save config.yaml as `litellm_config.yaml` (used in 3.2).
Save config.yaml as `litellm_config.yaml` before continuing.
You must finish this setup before starting the proxy server.
---
@ -359,7 +507,7 @@ See All General Settings [here](http://localhost:3000/docs/proxy/configs#all-set
`database_url: "postgresql://..."`
- Set `DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname>` in your env
### 3.2 Start Proxy
### Start Proxy
```bash
docker run \
@ -367,12 +515,11 @@ docker run \
-e AZURE_API_KEY=d6*********** \
-e AZURE_API_BASE=https://openai-***********/ \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:main-latest \
ghcr.io/berriai/litellm-database:main-latest \
--config /app/config.yaml --detailed_debug
```
### 3.3 Create Key w/ RPM Limit
### Create Key w/ RPM Limit
Create a key with `rpm_limit: 1`. This will only allow 1 request per minute for calls to proxy with this key.
@ -395,9 +542,9 @@ curl -L -X POST 'http://0.0.0.0:4000/key/generate' \
}
```
### 3.4 Test it!
### Test it!
**Use your virtual key from step 3.3**
**Use the virtual key you just created.**
1st call - Expect to work!
@ -611,6 +758,24 @@ model_list:
## Troubleshooting
### `prometheus.yml` mount error — "not a directory"
If you see:
```bash
Error: cannot create subdirectories in ".../prometheus.yml": not a directory
```
Docker created `prometheus.yml` as an **empty directory** instead of a file. This happens when the file is missing at `docker compose up` time.
Fix it:
Then create the file (see [Step 2.3 — Create `prometheus.yml`](#23--create-prometheusyml)) and run `docker compose up` again.
```bash
rm -rf prometheus.yml
```
Then create the file (see [Step 2.4](#step-24--create-prometheusyml)) and run `docker compose up` again.
### Non-root docker image?
If you need to run the docker image as a non-root user, use [this](https://github.com/BerriAI/litellm/pkgs/container/litellm-non_root).
@ -710,6 +875,3 @@ LiteLLM Proxy uses the [LiteLLM Python SDK](https://docs.litellm.ai/docs/routing
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
[![Chat on WhatsApp](https://img.shields.io/static/v1?label=Chat%20on&message=WhatsApp&color=success&logo=WhatsApp&style=flat-square)](https://wa.link/huol9n) [![Chat on Discord](https://img.shields.io/static/v1?label=Chat%20on&message=Discord&color=blue&logo=Discord&style=flat-square)](https://discord.gg/wuPM9dRgDw)

2
docs/my-website/docs/proxy/endpoint_activity.md
View file

@ -114,4 +114,4 @@ Understand spend distribution across endpoints:
- [Customer Usage](./customer_usage.md) - Track spend and usage for individual customers
- [Cost Tracking](./cost_tracking.md) - Comprehensive cost tracking and analytics
- [Spend Logs](./spend_logs.md) - Detailed request-level spend logs
- [Spend Logs](./cost_tracking.md#-spend-logs-api---individual-transaction-logs) - Detailed request-level spend logs

4
docs/my-website/docs/proxy/keys_teams_router_settings.md
View file

@ -146,5 +146,5 @@ Test new router settings on specific keys or teams before applying globally:
- [Router Settings Reference](./config_settings.md#router_settings---reference) - Complete reference of all router settings
- [Load Balancing](./load_balancing.md) - Learn about routing strategies and load balancing
- [Reliability](./reliability.md) - Configure fallbacks, retries, and error handling
- [Keys](./keys.md) - Manage API keys and their settings
- [Teams](./teams.md) - Organize keys into teams
- [Keys](./virtual_keys.md) - Manage API keys and their settings
- [Teams](./multi_tenant_architecture.md) - Organize keys into teams

4
docs/my-website/docs/proxy/model_compare_ui.md
View file

@ -187,7 +187,7 @@ Use tags and multiple comparisons to run structured A/B tests:
## Related Features
- [Playground Chat UI](./playground.md) - Single model testing interface
- [Playground Chat UI](./ui.md) - Single model testing interface
- [Model Management](./model_management.md) - Configure and manage models
- [Guardrails](./guardrails.md) - Set up safety filters
- [Guardrails](./guardrails/quick_start.md) - Set up safety filters
- [AI Hub](./ai_hub.md) - Share models and agents with your organization

2
docs/my-website/docs/proxy/sync_anthropic_beta_headers.md
View file

@ -125,4 +125,4 @@ curl -X DELETE "https://your-proxy-url/schedule/anthropic_beta_headers_reload" \
## Related
- [Model Cost Map Sync](./sync_models_github.md) - Auto-sync model pricing data
- [Anthropic Beta Headers](../completion/anthropic.md#beta-features) - Using Anthropic beta features
- [Anthropic Beta Headers](../providers/anthropic.md) - Using Anthropic beta features

4
docs/my-website/docs/proxy/ui_store_model_db_setting.md
View file

@ -87,6 +87,6 @@ Change the setting from the UI and have it take effect immediately—perfect for
## Related Documentation
- [Admin UI Overview](./ui_overview.md) – General guide to the LiteLLM Admin UI
- [Models and Endpoints](./models_and_endpoints.md) – Managing models and API endpoints
- [Admin UI Overview](./ui.md) – General guide to the LiteLLM Admin UI
- [Models and Endpoints](./model_management.md) – Managing models and API endpoints
- [Config Settings](./config_settings.md) – `store_model_in_db` in `general_settings`

2
docs/my-website/docs/proxy/user_onboarding.md
View file

@ -79,4 +79,4 @@ curl -X POST http://localhost:4000/v1/chat/completions \
## See Also
- [Proxy Quick Start](./quick_start.md)
- [User Management](./users.md)
- [Key Management](./key_management.md)
- [Key Management](./virtual_keys.md)

98
docs/my-website/docs/tutorials/index.md Normal file
View file

@ -0,0 +1,98 @@
---
title: Tutorials
sidebar_label: Overview
---
import NavigationCards from '@site/src/components/NavigationCards';
**Tutorials** are step-by-step walkthroughs for integrating LiteLLM with external tools, frameworks, and services — or building complete end-to-end workflows.
> Need help choosing the right path before you start? See [Learn →](/docs/learn)
---
## Getting Started
<NavigationCards
columns={2}
items={[
{
icon: "⚡",
title: "Getting Started",
description: "Installation, playground, text completion, and mock completions.",
to: "/docs/tutorials/getting_started",
},
]}
/>
---
## Integrations
<NavigationCards
columns={2}
items={[
{
icon: "🤖",
title: "Agent SDKs & Frameworks",
description: "OpenAI Agents SDK, Claude Agent SDK, Google ADK, CopilotKit, Letta, LiveKit, Instructor.",
to: "/docs/agent_sdks",
},
{
icon: "🛠️",
title: "AI Coding Tools",
description: "Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Qwen Code, OpenAI Codex.",
to: "/docs/ai_tools",
},
{
icon: "🐍",
title: "Python SDK",
description: "Gradio, fallbacks, provider-specific params — no proxy required.",
to: "/docs/tutorials/python_sdk",
},
{
icon: "🔌",
title: "Provider Setup",
description: "Azure OpenAI, HuggingFace, TogetherAI, local models, and more.",
to: "/docs/tutorials/provider_tutorials",
},
]}
/>
---
## Proxy
<NavigationCards
columns={2}
items={[
{
icon: "👥",
title: "Proxy: Admin & Access",
description: "User and team management, SSO, SCIM, and routing rules.",
to: "/docs/tutorials/proxy_admin_access",
},
{
icon: "🛡️",
title: "Proxy: Features & Safety",
description: "Prompt caching, passthrough APIs, realtime, guardrails, and PII masking.",
to: "/docs/tutorials/proxy_features_safety",
},
]}
/>
---
## Observability & Evaluation
<NavigationCards
columns={2}
items={[
{
icon: "🔍",
title: "Observability & Evaluation",
description: "Logging to Elasticsearch, benchmarking, and evaluation suites.",
to: "/docs/tutorials/observability_evaluation",
},
]}
/>

8
docs/my-website/docs/tutorials/installation.md
View file

@ -1,7 +1,3 @@
---
displayed_sidebar: tutorialSidebar
---
# Set up environment
Let's get the necessary keys to set up our demo environment.
@ -11,7 +7,5 @@ Every LLM provider needs API keys (e.g. `OPENAI_API_KEY`). You can get API keys
Let's get them for our demo!
**OpenAI**: https://platform.openai.com/account/api-keys
**Cohere**: https://dashboard.cohere.com/welcome/login?redirect_uri=%2Fapi-keys (no credit card required)
**Cohere**: https://dashboard.cohere.com/welcome/login?redirect_uri=%2Fapi-keys (no credit card required)
**AI21**: https://studio.ai21.com/account/api-key (no credit card required)

6
docs/my-website/docs/vertex_batch_passthrough.md
View file

@ -155,6 +155,6 @@ Common error scenarios and their solutions:
## Related Documentation
- [Vertex AI Provider Documentation](./vertex.md)
- [General Batches API Documentation](../batches.md)
- [Cost Tracking and Monitoring](../observability/telemetry.md)
- [Vertex AI Provider Documentation](./providers/vertex.md)
- [General Batches API Documentation](./batches.md)
- [Cost Tracking and Monitoring](./observability/telemetry.md)

144
docs/my-website/docusaurus.config.js
View file

@ -2,9 +2,9 @@
// Note: type annotations allow type checking and IDEs autocompletion
// @ts-ignore
const lightCodeTheme = require('prism-react-renderer/themes/github');
const lightCodeTheme = require('prism-react-renderer/themes/vsLight');
// @ts-ignore
const darkCodeTheme = require('prism-react-renderer/themes/dracula');
const darkCodeTheme = require('prism-react-renderer/themes/nightOwl');
const inkeepConfig = {
baseSettings: {
@ -87,18 +87,88 @@ const config = {
},
],
[
'@docusaurus/plugin-content-blog',
'@docusaurus/plugin-content-docs',
{
id: 'release_notes',
id: 'release-notes',
path: './release_notes',
routeBasePath: 'release_notes',
blogTitle: 'Release Notes',
blogSidebarTitle: 'Releases',
blogSidebarCount: 'ALL',
postsPerPage: 'ALL',
showReadingTime: false,
sortPosts: 'descending',
include: ['**/*.{md,mdx}'],
sidebarPath: require.resolve('./sidebars-release-notes.js'),
async sidebarItemsGenerator({defaultSidebarItemsGenerator, docs, ...args}) {
const items = await defaultSidebarItemsGenerator({docs, ...args});
// Build map of doc id -> year from frontmatter date
const docYearMap = {};
for (const doc of docs) {
const date = doc.frontMatter && doc.frontMatter.date;
if (date) {
const year = new Date(date).getFullYear();
docYearMap[doc.id] = year;
}
}
function parseVersion(str) {
const match = (str || '').match(/v?(\d+)\.(\d+)\.(\d+)/);
if (!match) return [0, 0, 0];
return [parseInt(match[1]), parseInt(match[2]), parseInt(match[3])];
}
function compareVersionsDesc(a, b) {
const [aMaj, aMin, aPatch] = parseVersion(a.label || a.id || '');
const [bMaj, bMin, bPatch] = parseVersion(b.label || b.id || '');
if (bMaj !== aMaj) return bMaj - aMaj;
if (bMin !== aMin) return bMin - aMin;
return bPatch - aPatch;
}
// Flatten and transform doc items (filter index, shorten labels)
function flattenDocs(list) {
const result = [];
for (const item of list) {
if (item.type === 'doc' && item.id === 'index') continue;
if (item.type === 'doc') {
const label = item.id.replace(/\/index$/, '');
result.push({...item, label});
} else if (item.type === 'category') {
if (item.link && item.link.type === 'doc' && item.link.id !== 'index') {
const id = item.link.id;
const label = id.replace(/\/index$/, '');
result.push({type: 'doc', id, label});
} else {
result.push(...flattenDocs(item.items));
}
}
}
return result;
}
const docItems = flattenDocs(items);
// Group by year
const byYear = {};
for (const item of docItems) {
const year = docYearMap[item.id] || 'Other';
if (!byYear[year]) byYear[year] = [];
byYear[year].push(item);
}
// Sort each year's items by version descending
for (const year of Object.keys(byYear)) {
byYear[year].sort(compareVersionsDesc);
}
// Build categories sorted by year descending
const years = Object.keys(byYear).sort((a, b) => {
// Object.keys() returns strings; avoid numeric subtraction type errors.
const na = Number.parseInt(a, 10);
const nb = Number.parseInt(b, 10);
return nb - na;
});
return years.map(year => ({
type: 'category',
label: String(year),
collapsed: year !== String(years[0]),
items: byYear[year],
}));
},
},
],
[
@ -130,6 +200,20 @@ const config = {
};
},
}),
// Ensure gtag exists before the GA script loads.
() => ({
name: 'gtag-shim',
injectHtmlTags() {
return {
headTags: [
{
tagName: 'script',
innerHTML: `window.dataLayer=window.dataLayer||[];function gtag(){dataLayer.push(arguments);}if(!window.gtag){window.gtag=gtag;}`,
},
],
};
},
}),
],
presets: [
@ -137,10 +221,13 @@ const config = {
'classic',
/** @type {import('@docusaurus/preset-classic').Options} */
({
gtag: {
trackingID: 'G-K7K215ZVNC',
anonymizeIP: true,
},
gtag:
process.env.NODE_ENV === 'production'
? {
trackingID: 'G-K7K215ZVNC',
anonymizeIP: true,
}
: undefined,
docs: {
sidebarPath: require.resolve('./sidebars.js'),
},
@ -181,34 +268,39 @@ const config = {
label: 'Docs',
},
{
type: 'docSidebar',
sidebarId: 'learnSidebar',
position: 'left',
label: 'Learn',
},
{
type: 'docSidebar',
sidebarId: 'integrationsSidebar',
position: 'left',
label: 'Integrations',
to: "docs/integrations"
},
{
sidebarId: 'tutorialSidebar',
position: 'left',
label: 'Enterprise',
to: "docs/enterprise"
},
{ to: '/release_notes', label: 'Release Notes', position: 'left' },
{ to: '/blog', label: 'Blog', position: 'left' },
{
href: 'https://models.litellm.ai/',
label: '💸 LLM Model Cost Map',
position: 'right',
},
{
href: 'https://github.com/BerriAI/litellm',
label: 'GitHub',
position: 'right',
className: 'header-github-link',
'aria-label': 'GitHub repository',
},
{
href: 'https://www.litellm.ai/support',
label: 'Slack/Discord',
position: 'right',
}
className: 'header-discord-link',
'aria-label': 'Discord / Slack community',
},
{
type: 'search',
position: 'right',
},
],
},
footer: {

BIN
docs/my-website/img/hero.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 6.2 MiB

52
docs/my-website/release_notes/index.md Normal file
View file

@ -0,0 +1,52 @@
---
title: Release Notes
sidebar_label: Overview
slug: /
---
# Release Notes
LiteLLM ships new releases regularly with new provider support, performance improvements, and enterprise features. Use the sidebar to browse all releases.
## Latest Release
### [v1.82.3 — Nebius AI, gpt-5.4, Gemini 3.x, FLUX Kontext, and 116 New Models](/release_notes/v1.82.3/v1-82-3)
_March 16, 2026_
116 new models including Nebius AI, gpt-5.4, Gemini 3.x, and FLUX Kontext.
---
## Recent Releases
| Version | Date | Highlights |
| ----------------------------------- | ------------ | ---------------------------------------------------------- |
| [v1.82.0](/release_notes/v1.82.0/v1-82-0) | Feb 28, 2026 | Realtime Guardrails, Projects Management, and 10+ Performance Optimizations |
| [v1.81.14](/release_notes/v1.81.14/v1-81-14) | Feb 21, 2026 | New Gateway Level Guardrails & Compliance Playground |
| [v1.81.12](/release_notes/v1.81.12/v1-81-12) | Feb 14, 2026 | Guardrail Policy Templates & Action Builder |
| [v1.81.9](/release_notes/v1.81.9/v1-81-9) | Feb 7, 2026 | Control which MCP Servers are exposed on the Internet |
| [v1.81.6](/release_notes/v1.81.6/v1-81-6) | Jan 31, 2026 | Logs v2 with Tool Call Tracing |
| [v1.81.3](/release_notes/v1.81.3-stable/v1-81-3) | Jan 26, 2026 | Performance — 25% CPU Usage Reduction |
| [v1.81.0](/release_notes/v1.81.0/v1-81-0) | Jan 18, 2026 | Claude Code — Web Search Across All Providers |
| [v1.80.15](/release_notes/v1.80.15/v1-80-15) | Jan 10, 2026 | Manus API Support |
| [v1.80.8](/release_notes/v1.80.8-stable/v1-80-8) | Dec 6, 2025 | Introducing A2A Agent Gateway |
| [v1.80.5](/release_notes/v1.80.5-stable/v1-80-5) | Nov 22, 2025 | Gemini 3.0 Support |
| [v1.80.0](/release_notes/v1.80.0-stable/v1-80-0) | Nov 15, 2025 | Introducing Agent Hub: Register, Publish, and Share Agents |
| [v1.79.3](/release_notes/v1.79.3-stable/v1-79-3) | Nov 8, 2025 | Built-in Guardrails on AI Gateway |
| [v1.79.0](/release_notes/v1.79.0-stable/v1-79-0) | Oct 26, 2025 | Search APIs |
| [v1.78.5](/release_notes/v1.78.5-stable/v1-78-5) | Oct 18, 2025 | Native OCR Support |
| [v1.78.0](/release_notes/v1.78.0-stable/v1-78-0) | Oct 11, 2025 | MCP Gateway: Control Tool Access by Team, Key |
| [v1.77.7](/release_notes/v1.77.7-stable/v1-77-7) | Oct 4, 2025 | 2.9x Lower Median Latency |
| [v1.77.5](/release_notes/v1.77.5-stable/v1-77-5) | Sep 29, 2025 | MCP OAuth 2.0 Support |
| [v1.77.3](/release_notes/v1.77.3-stable/v1-77-3) | Sep 21, 2025 | Priority Based Rate Limiting |
---
## Stay Updated
- **GitHub**: Watch the [BerriAI/litellm](https://github.com/BerriAI/litellm) repository for release notifications
- **Discord**: Join our [community](https://discord.com/invite/wuPM9dRgDw) for announcements
- **Twitter**: Follow [@LiteLLM](https://twitter.com/LiteLLM)
Use the sidebar to browse the full release history.

8
docs/my-website/release_notes/v1.63.14/index.md
View file

@ -62,10 +62,10 @@ Here's a Demo Instance to test changes:
- Infer aws region from bedrock application profile id - (`arn:aws:bedrock:us-east-1:...`)
- Ollama - support calling via `/v1/completions` [Get Started](../../docs/providers/ollama#using-ollama-fim-on-v1completions)
- Bedrock - support `us.deepseek.r1-v1:0` model name [Docs](../../docs/providers/bedrock#supported-aws-bedrock-models)
- OpenRouter - `OPENROUTER_API_BASE` env var support [Docs](../../docs/providers/openrouter.md)
- OpenRouter - `OPENROUTER_API_BASE` env var support [Docs](../../docs/providers/openrouter)
- Azure - add audio model parameter support - [Docs](../../docs/providers/azure#azure-audio-model)
- OpenAI - PDF File support [Docs](../../docs/completion/document_understanding#openai-file-message-type)
- OpenAI - o1-pro Responses API streaming support [Docs](../../docs/response_api.md#streaming)
- OpenAI - o1-pro Responses API streaming support [Docs](../../docs/response_api#streaming)
- [BETA] MCP - Use MCP Tools with LiteLLM SDK [Docs](../../docs/mcp)
2. **Bug Fixes**
@ -102,7 +102,7 @@ Here's a Demo Instance to test changes:
- fix logging to just log the LLM I/O [PR](https://github.com/BerriAI/litellm/pull/9353)
- Dynamic API Key/Space param support [Get Started](../../docs/observability/arize_integration#pass-arize-spacekey-per-request)
- StandardLoggingPayload - Log litellm_model_name in payload. Allows knowing what the model sent to API provider was [Get Started](../../docs/proxy/logging_spec#standardlogginghiddenparams)
- Prompt Management - Allow building custom prompt management integration [Get Started](../../docs/proxy/custom_prompt_management.md)
- Prompt Management - Allow building custom prompt management integration [Get Started](../../docs/proxy/custom_prompt_management)
## Performance / Reliability improvements
@ -128,4 +128,4 @@ Here's a Demo Instance to test changes:
## Complete Git Diff
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.11-stable...v1.63.14.rc)
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.11-stable...v1.63.14.rc)

8
docs/my-website/release_notes/v1.63.2-stable/index.md
View file

@ -64,15 +64,15 @@ Here's a Demo Instance to test changes:
9. Bedrock - handle thinking blocks in assistant message. [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---thinking--reasoning-content)
10. Anthropic - Return `signature` on streaming. [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---thinking--reasoning-content)
- Note: We've also migrated from `signature_delta` to `signature`. [Read more](https://docs.litellm.ai/release_notes/v1.63.0)
11. Support format param for specifying image type. [Get Started](../../docs/completion/vision.md#explicitly-specify-image-type)
12. Anthropic - `/v1/messages` endpoint - `thinking` param support. [Get Started](../../docs/anthropic_unified.md)
11. Support format param for specifying image type. [Get Started](../../docs/completion/vision#explicitly-specify-image-type)
12. Anthropic - `/v1/messages` endpoint - `thinking` param support. [Get Started](../../docs/anthropic_unified)
- Note: this refactors the [BETA] unified `/v1/messages` endpoint, to just work for the Anthropic API.
13. Vertex AI - handle $id in response schema when calling vertex ai. [Get Started](https://docs.litellm.ai/docs/providers/vertex#json-schema)
## Spend Tracking Improvements
1. Batches API - Fix cost calculation to run on retrieve_batch. [Get Started](https://docs.litellm.ai/docs/batches)
2. Batches API - Log batch models in spend logs / standard logging payload. [Get Started](../../docs/proxy/logging_spec.md#standardlogginghiddenparams)
2. Batches API - Log batch models in spend logs / standard logging payload. [Get Started](../../docs/proxy/logging_spec#standardlogginghiddenparams)
## Management Endpoints / UI
@ -109,4 +109,4 @@ Here's a Demo Instance to test changes:
## Complete Git Diff
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.61.20-stable...v1.63.2-stable)
[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.61.20-stable...v1.63.2-stable)

3
docs/my-website/release_notes/v1.80.15/index.md
View file

@ -53,7 +53,7 @@ pip install litellm==1.80.15
- **MCP Global Mode** - [Configure MCP servers globally with visibility controls](../../docs/mcp)
- **Interactions API Bridge** - [Use all LiteLLM providers with the Interactions API](../../docs/interactions)
- **RAG Query Endpoint** - [New RAG Search/Query endpoint for retrieval-augmented generation](../../docs/search/index)
- **UI Usage - Endpoint Activity** - [Users can now see Endpoint Activity Metrics in the UI](../../docs/proxy/endpoint_activity.md)
- **UI Usage - Endpoint Activity** - [Users can now see Endpoint Activity Metrics in the UI](../../docs/proxy/endpoint_activity)
- **50% Overhead Reduction** - LiteLLM now sends 2.5× more requests to LLM providers
@ -640,4 +640,3 @@ Users can now see Endpoint Activity Metrics in the UI.
**[View complete changelog on GitHub](https://github.com/BerriAI/litellm/compare/v1.80.11.rc.1...v1.80.15-stable.1)**

4
docs/my-website/release_notes/v1.81.0/index.md
View file

@ -48,7 +48,7 @@ pip install litellm==1.81.0
- **Claude Code** - Support for using web search across Bedrock, Vertex AI, and all LiteLLM providers
- **Major Change** - [50MB limit on image URL downloads](#major-change---chatcompletions-image-url-download-size-limit) to improve reliability
- **Performance** - [25% CPU Usage Reduction](#performance---25-cpu-usage-reduction) by removing premature model.dump() calls from the hot path
- **Deleted Keys Audit Table on UI** - [View deleted keys and teams for audit purposes](../../docs/proxy/deleted_keys_teams.md) with spend and budget information at the time of deletion
- **Deleted Keys Audit Table on UI** - [View deleted keys and teams for audit purposes](../../docs/proxy/deleted_keys_teams) with spend and budget information at the time of deletion
---
@ -166,7 +166,7 @@ LiteLLM now reduces CPU usage by removing premature `model.dump()` calls from th
<Image img={require('../../img/ui_deleted_keys_table.png')} />
LiteLLM now provides a comprehensive audit table for deleted API keys and teams directly in the UI. This feature allows you to easily track the spend of deleted keys, view their associated team information, and maintain accurate financial records for auditing and compliance purposes. The table displays key details including key aliases, team associations, and spend information captured at the time of deletion. For more information on how to use this feature, see the [Deleted Keys & Teams documentation](../../docs/proxy/deleted_keys_teams.md).
LiteLLM now provides a comprehensive audit table for deleted API keys and teams directly in the UI. This feature allows you to easily track the spend of deleted keys, view their associated team information, and maintain accurate financial records for auditing and compliance purposes. The table displays key details including key aliases, team associations, and spend information captured at the time of deletion. For more information on how to use this feature, see the [Deleted Keys & Teams documentation](../../docs/proxy/deleted_keys_teams).
---

4
docs/my-website/release_notes/v1.81.12.md → docs/my-website/release_notes/v1.81.12/index.md
View file

@ -62,13 +62,13 @@ This release fixes out-of-memory (OOM) risks from unbounded `asyncio.Queue()` us
This release adds a visual action builder for guardrail policies with conditional execution support. You can now chain guardrails into multi-step pipelines — if a simple guardrail fails, route to an advanced one instead of immediately blocking. Each step has configurable ON PASS and ON FAIL actions (Next Step, Block, or Allow), and you can test the full pipeline with a sample message before saving.
![Guardrail Action Builder](../img/release_notes/guard_actions.png)
![Guardrail Action Builder](../../img/release_notes/guard_actions.png)
### Access Groups
Access Groups simplify defining resource access across your organization. One group can grant access to models, MCP servers, and agents—simply attach it to a key or team. Create groups in the Admin UI, define which resources each group includes, then assign the group when creating keys or teams. Updates to a group apply automatically to all attached keys and teams.
<Image img={require('../img/ui_access_groups.png')} />
<Image img={require('../../img/ui_access_groups.png')} />
## New Providers and Endpoints

8
docs/my-website/release_notes/v1.81.14.md → docs/my-website/release_notes/v1.81.14/index.md
View file

@ -56,7 +56,7 @@ pip install litellm==1.81.14
AI Platform Admins can now browse built-in and partner guardrails from the Guardrail Garden. Guardrails are organized by use case — blocking financial advice, filtering insults, detecting competitor mentions, and more — so you can find the right one and deploy it in a few clicks.
![Guardrail Garden](../img/release_notes/guardrail_garden.png)
![Guardrail Garden](../../img/release_notes/guardrail_garden.png)
### 3 New Built-in Guardrails
@ -72,7 +72,7 @@ These guardrails are built for production and on our benchmarks had a 100% Recal
Previously, the `store_model_in_db` setting could only be configured in `proxy_config.yaml` under `general_settings`, requiring a proxy restart to take effect. Now you can enable or disable this setting directly from the Admin UI without any restarts. This is especially useful for cloud deployments where you don't have direct access to config files or want to avoid downtime. Enable `store_model_in_db` to move model definitions from your YAML into the database—reducing config complexity, improving scalability, and enabling dynamic model management across multiple proxy instances.
![Store model in DB Setting](../img/ui_store_model_in_db.png)
![Store model in DB Setting](../../img/ui_store_model_in_db.png)
#### Eval results
@ -91,14 +91,14 @@ We benchmarked our new built-in guardrails against labeled datasets before shipp
The Compliance Playground lets you test any guardrail against our pre-built eval datasets or your own custom datasets, so you can see precision, recall, and false positive rate before rolling it out to production.
![Compliance Playground](../img/release_notes/compliance_playground.png)
![Compliance Playground](../../img/release_notes/compliance_playground.png)
---
## Performance & Reliability — Up to 13% Lower Latency
<Image img={require('../img/release_notes/v1_81_14_perf.png')} />
<Image img={require('../../img/release_notes/v1_81_14_perf.png')} />
This release cuts latency across all percentiles through 20+ micro-optimizations across logging, cost calculation, routing, and connection management. See [benchmarking](../../docs/benchmarks) for more info about how to benchmark yourself.

0
docs/my-website/release_notes/v1.81.6.md → docs/my-website/release_notes/v1.81.6/index.md
View file

4
docs/my-website/release_notes/v1.81.9.md → docs/my-website/release_notes/v1.81.9/index.md
View file

@ -81,7 +81,7 @@ This release makes it safe to expose MCP servers on the public internet by addin
[Get started](../../docs/mcp_public_internet)
<Image
img={require('../img/release_notes/mcp_internet.png')}
img={require('../../img/release_notes/mcp_internet.png')}
style={{ maxWidth: '900px', width: '100%' }}
/>
@ -92,7 +92,7 @@ Set a soft budget on any team to receive email alerts when spending crosses the
[Get started](../../docs/proxy/ui_team_soft_budget_alerts)
<Image
img={require('../img/ui_team_soft_budget_alerts.png')}
img={require('../../img/ui_team_soft_budget_alerts.png')}
style={{ maxWidth: '900px', width: '100%' }}
/>

0
docs/my-website/release_notes/v1.82.0.md → docs/my-website/release_notes/v1.82.0/index.md
View file

374
docs/my-website/release_notes/v1.82.3/index.md Normal file
View file

@ -0,0 +1,374 @@
---
title: "v1.82.3 - Nebius AI, gpt-5.4, Gemini 3.x, FLUX Kontext, and 116 New Models"
slug: "v1-82-3"
date: 2026-03-16T00:00:00
authors:
- name: Krrish Dholakia
title: CEO, LiteLLM
url: https://www.linkedin.com/in/krish-d/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
- name: Ishaan Jaff
title: CTO, LiteLLM
url: https://www.linkedin.com/in/reffajnaahsi/
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
hide_table_of_contents: false
---
## Deploy this version
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
<Tabs>
<TabItem value="docker" label="Docker">
``` showLineNumbers title="docker run litellm"
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-1.82.3-stable
```
</TabItem>
<TabItem value="pip" label="Pip">
``` showLineNumbers title="pip install litellm"
pip install litellm==1.82.3
```
</TabItem>
</Tabs>
## Key Highlights
- **Nebius AI — new provider** — [30 models across DeepSeek, Qwen, Llama, Mistral, NVIDIA, and BAAI available via Nebius AI cloud](../../docs/providers/nebius) - [PR #22614](https://github.com/BerriAI/litellm/pull/22614)
- **OpenAI gpt-5.4 / gpt-5.4-pro — day 0** — Full pricing and routing support for `gpt-5.4` (1M context, $2.50/$15.00) and `gpt-5.4-pro` ($30.00/$180.00) on OpenAI and Azure
- **Gemini 3.x models** — `gemini-3-flash-preview`, `gemini-3.1-pro-preview`, `gemini-3.1-flash-image-preview`, and `gemini-embedding-2-preview` added to cost map for Google AI and Vertex AI
- **FLUX Kontext image editing** — `flux-kontext-pro` and `flux-kontext-max` added to Black Forest Labs, alongside `flux-pro-1.0-fill` and `flux-pro-1.0-expand` for inpainting and outpainting
- **116 new models, 132 deprecated models cleaned up** — Major model map refresh including Mistral Magistral, Dashscope Qwen3 VL, xAI Grok via Azure AI, ZAI GLM-5, Serper Search; removal of OpenAI GPT-3.5/GPT-4 legacy variants, Gemini 1.5, and Vertex AI PaLM2
- **SageMaker Nova provider** — [New `sagemaker_nova` provider for Amazon Nova models on SageMaker](../../docs/providers/aws_sagemaker) - [PR #21542](https://github.com/BerriAI/litellm/pull/21542)
- **Secret redaction in logs** — API keys, tokens, and credentials automatically scrubbed from all proxy log output. Enabled by default; opt out with `LITELLM_DISABLE_REDACT_SECRETS=true` - [PR #23668](https://github.com/BerriAI/litellm/pull/23668)
- **Streaming stability fix** — Critical fix for `RuntimeError: Cannot send a request, as the client has been closed.` crashes after ~1 hour in production - [PR #22926](https://github.com/BerriAI/litellm/pull/22926)
---
## New Providers and Endpoints
### New Providers (5 new providers)
| Provider | Supported LiteLLM Endpoints | Description |
| -------- | --------------------------- | ----------- |
| [Nebius AI](../../docs/providers/nebius) (`nebius/`) | `/chat/completions`, `/embeddings` | EU-based AI cloud with 30+ open models — DeepSeek, Qwen3, Llama 3.1/3.3, NVIDIA Nemotron, BAAI embeddings |
| [ZAI](../../docs/providers/zai) (`zai/`) | `/chat/completions` | ZhipuAI GLM-5 models via ZAI cloud |
| [Black Forest Labs](../../docs/providers/black_forest_labs) (`black_forest_labs/`) | `/images/generations`, `/images/edits` | FLUX image generation and editing — Kontext Pro/Max, Pro 1.0 Fill/Expand |
| [Serper](../../docs/providers/serper) (`serper/`) | `/search` | Web search via Serper API |
| [SageMaker Nova](../../docs/providers/aws_sagemaker) (`sagemaker_nova/`) | `/chat/completions` | Amazon Nova models via SageMaker endpoint |
---
## New Models / Updated Models
#### New Model Support (116 new models)
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
| OpenAI | `gpt-5.4` | 1.05M | $2.50 | $15.00 | chat, vision, tools, reasoning |
| OpenAI | `gpt-5.4-pro` | 1.05M | $30.00 | $180.00 | responses, vision, tools, reasoning |
| OpenAI | `gpt-5.3-chat-latest` | 128K | $1.75 | $14.00 | chat, vision, tools, reasoning |
| Azure OpenAI | `azure/gpt-5.4` | 1.05M | $2.50 | $15.00 | chat, vision, tools, reasoning |
| Azure OpenAI | `azure/gpt-5.4-pro` | 1.05M | $30.00 | $180.00 | responses, vision, tools, reasoning |
| Azure OpenAI | `azure/gpt-5.3-chat` | 128K | $1.75 | $14.00 | chat, vision, tools, reasoning |
| Google Gemini | `gemini/gemini-3-flash-preview` | 1M | $0.50 | $3.00 | chat, vision, tools, reasoning |
| Google Gemini | `gemini/gemini-3.1-pro-preview` | 1M | $2.00 | $12.00 | chat, vision, tools, reasoning |
| Google Gemini | `gemini/gemini-3.1-flash-image-preview` | 65K | $0.25 | $1.50 | image generation, vision |
| Google Gemini | `gemini/gemini-3.1-flash-lite-preview` | - | - | - | chat |
| Google Gemini | `gemini/gemini-3-pro-image-preview` | - | - | - | image generation |
| Google Gemini | `gemini/gemini-embedding-2-preview` | 8K | $0.20 | - | embeddings |
| Google Vertex AI | `vertex_ai/gemini-3-flash-preview` | - | - | - | chat |
| Google Vertex AI | `vertex_ai/gemini-3.1-pro-preview` | - | - | - | chat |
| Google Vertex AI | `vertex_ai/gemini-3.1-flash-lite-preview` | - | - | - | chat |
| Google Vertex AI | `vertex_ai/gemini-embedding-2-preview` | - | $0.20 | - | embeddings |
| Mistral | `mistral/magistral-medium-1-2-2509` | 40K | $2.00 | $5.00 | chat, tools, reasoning |
| Mistral | `mistral/magistral-small-1-2-2509` | 40K | $0.50 | $1.50 | chat, tools, reasoning |
| Mistral | `mistral/mistral-large-2512` | 262K | $0.50 | $1.50 | chat, vision, tools |
| Mistral | `mistral/mistral-medium-3-1-2508` | - | - | - | chat |
| Mistral | `mistral/mistral-small-3-2-2506` | - | - | - | chat |
| Mistral | `mistral/ministral-3-3b-2512` | - | - | - | chat |
| Mistral | `mistral/ministral-3-8b-2512` | - | - | - | chat |
| Mistral | `mistral/ministral-3-14b-2512` | - | - | - | chat |
| Black Forest Labs | `black_forest_labs/flux-kontext-pro` | - | - | - | image editing |
| Black Forest Labs | `black_forest_labs/flux-kontext-max` | - | - | - | image editing |
| Black Forest Labs | `black_forest_labs/flux-pro-1.0-fill` | - | - | - | image editing (inpaint) |
| Black Forest Labs | `black_forest_labs/flux-pro-1.0-expand` | - | - | - | image editing (outpaint) |
| Black Forest Labs | `black_forest_labs/flux-pro-1.1` | - | - | - | image generation |
| Black Forest Labs | `black_forest_labs/flux-pro-1.1-ultra` | - | - | - | image generation |
| Black Forest Labs | `black_forest_labs/flux-dev` | - | - | - | image generation |
| Black Forest Labs | `black_forest_labs/flux-pro` | - | - | - | image generation |
| Azure AI | `azure_ai/grok-4-1-fast-non-reasoning` | 131K | $0.20 | $0.50 | chat, tools |
| Azure AI | `azure_ai/grok-4-1-fast-reasoning` | 131K | $0.20 | $0.50 | chat, tools, reasoning |
| Azure AI | `azure_ai/mistral-document-ai-2512` | - | - | - | OCR |
| Dashscope | `dashscope/qwen3-next-80b-a3b-instruct` | 262K | $0.15 | $1.20 | chat |
| Dashscope | `dashscope/qwen3-next-80b-a3b-thinking` | 262K | $0.15 | $1.20 | chat, reasoning |
| Dashscope | `dashscope/qwen3-vl-235b-a22b-instruct` | 131K | $0.40 | $1.60 | chat, vision |
| Dashscope | `dashscope/qwen3-vl-235b-a22b-thinking` | 131K | $0.40 | $4.00 | chat, vision, reasoning |
| Dashscope | `dashscope/qwen3-vl-32b-instruct` | 131K | $0.16 | $0.64 | chat, vision |
| Dashscope | `dashscope/qwen3-vl-32b-thinking` | 131K | $0.16 | $2.87 | chat, vision, reasoning |
| Dashscope | `dashscope/qwen3-vl-plus` | 260K | - | - | chat, vision |
| Dashscope | `dashscope/qwen3.5-plus` | 992K | - | - | chat |
| Dashscope | `dashscope/qwen3-max-2026-01-23` | 258K | - | - | chat |
| Nebius AI | `nebius/deepseek-ai/DeepSeek-R1` | 128K | $0.80 | $2.40 | chat, reasoning |
| Nebius AI | `nebius/deepseek-ai/DeepSeek-R1-0528` | 164K | $0.80 | $2.40 | chat, reasoning |
| Nebius AI | `nebius/deepseek-ai/DeepSeek-V3` | 128K | $0.50 | $1.50 | chat |
| Nebius AI | `nebius/deepseek-ai/DeepSeek-V3-0324` | 128K | $0.50 | $1.50 | chat |
| Nebius AI | `nebius/deepseek-ai/DeepSeek-R1-Distill-Llama-70B` | 128K | $0.25 | $0.75 | chat |
| Nebius AI | `nebius/Qwen/Qwen3-235B-A22B` | 262K | $0.20 | $0.60 | chat |
| Nebius AI | `nebius/Qwen/Qwen3-32B` | 32K | $0.10 | $0.30 | chat |
| Nebius AI | `nebius/Qwen/Qwen3-30B-A3B` | 32K | $0.10 | $0.30 | chat |
| Nebius AI | `nebius/Qwen/Qwen3-14B` | 32K | $0.08 | $0.24 | chat |
| Nebius AI | `nebius/Qwen/Qwen3-4B` | 32K | $0.08 | $0.24 | chat |
| Nebius AI | `nebius/Qwen/QwQ-32B` | 32K | $0.15 | $0.45 | chat |
| Nebius AI | `nebius/Qwen/Qwen2.5-72B-Instruct` | 128K | $0.13 | $0.40 | chat |
| Nebius AI | `nebius/Qwen/Qwen2.5-32B-Instruct` | 128K | $0.06 | $0.20 | chat |
| Nebius AI | `nebius/Qwen/Qwen2.5-VL-72B-Instruct` | 131K | $0.13 | $0.40 | chat, vision |
| Nebius AI | `nebius/Qwen/Qwen2-VL-72B-Instruct` | 131K | $0.13 | $0.40 | chat, vision |
| Nebius AI | `nebius/Qwen/Qwen2-VL-7B-Instruct` | 131K | $0.02 | $0.06 | chat, vision |
| Nebius AI | `nebius/meta-llama/Meta-Llama-3.1-405B-Instruct` | 128K | $1.00 | $3.00 | chat |
| Nebius AI | `nebius/meta-llama/Meta-Llama-3.1-70B-Instruct` | 128K | $0.13 | $0.40 | chat |
| Nebius AI | `nebius/meta-llama/Meta-Llama-3.1-8B-Instruct` | 128K | $0.02 | $0.06 | chat |
| Nebius AI | `nebius/meta-llama/Llama-3.3-70B-Instruct` | 128K | $0.13 | $0.40 | chat |
| Nebius AI | `nebius/meta-llama/Llama-Guard-3-8B` | 128K | $0.02 | $0.06 | chat |
| Nebius AI | `nebius/nvidia/Llama-3.1-Nemotron-Ultra-253B-v1` | 128K | $0.60 | $1.80 | chat |
| Nebius AI | `nebius/nvidia/Llama-3.3-Nemotron-Super-49B-v1` | 131K | $0.10 | $0.40 | chat |
| Nebius AI | `nebius/NousResearch/Hermes-3-Llama-3.1-405B` | 128K | $1.00 | $3.00 | chat |
| Nebius AI | `nebius/google/gemma-3-27b-it` | 128K | $0.06 | $0.20 | chat |
| Nebius AI | `nebius/mistralai/Mistral-Nemo-Instruct-2407` | 128K | $0.04 | $0.12 | chat |
| Nebius AI | `nebius/Qwen/Qwen2.5-Coder-7B` | 32K | $0.01 | $0.03 | chat |
| Nebius AI | `nebius/BAAI/bge-en-icl` | 32K | $0.01 | - | embeddings |
| Nebius AI | `nebius/BAAI/bge-multilingual-gemma2` | 8K | $0.01 | - | embeddings |
| Nebius AI | `nebius/intfloat/e5-mistral-7b-instruct` | 32K | $0.01 | - | embeddings |
| AWS Bedrock | `mistral.devstral-2-123b` | 256K | $0.40 | $2.00 | chat, tools |
| AWS Bedrock | `zai.glm-4.7-flash` | 200K | $0.07 | $0.40 | chat, tools, reasoning |
| ZAI | `zai/glm-5` | 200K | $1.00 | $3.20 | chat, tools, reasoning |
| ZAI | `zai/glm-5-code` | 200K | $1.20 | $5.00 | chat, tools, reasoning |
| OpenRouter | `openrouter/anthropic/claude-sonnet-4.6` | - | - | - | chat |
| OpenRouter | `openrouter/google/gemini-3.1-pro-preview` | - | - | - | chat |
| OpenRouter | `openrouter/openai/gpt-5.1-codex-max` | - | - | - | chat |
| OpenRouter | `openrouter/qwen/qwen3-coder-plus` | - | - | - | chat |
| OpenRouter | `openrouter/qwen/qwen3.5-*` (5 models) | - | - | - | chat |
| OpenRouter | `openrouter/z-ai/glm-5` | - | - | - | chat |
| Together AI | `together_ai/Qwen/Qwen3.5-397B-A17B` | - | - | - | chat |
| Perplexity | `perplexity/pplx-embed-v1-0.6b` | 32K | $0.00 | - | embeddings |
| Perplexity | `perplexity/pplx-embed-v1-4b` | 32K | $0.03 | - | embeddings |
| Serper | `serper/search` | - | - | - | search |
#### Updated Models
- **[AWS Bedrock](../../docs/providers/bedrock)**
- Add `cache_read_input_token_cost` and `cache_creation_input_token_cost` to Bedrock-hosted Anthropic models (`claude-3-opus`, `claude-3-sonnet`, `claude-3-haiku`, and APAC/EU variants) — prompt caching is now tracked for cost estimation
- Rename `apac.anthropic.claude-sonnet-4-6` → `au.anthropic.claude-sonnet-4-6` to reflect correct regional identifier
- **[Azure OpenAI](../../docs/providers/azure)**
- Add `supports_none_reasoning_effort` to all `gpt-5.1-chat`, `gpt-5.1-codex`, and `gpt-5.4` variants (global, EU, standard deployments) — allows passing `reasoning_effort: null` to disable reasoning
- **[Azure OpenAI](../../docs/providers/azure)** — Removed deprecated models
- Remove `azure/gpt-35-turbo-0301` (deprecated 2025-02-13)
- Remove `azure/gpt-35-turbo-0613` (deprecated 2025-02-13)
#### Features
- **[OpenAI](../../docs/providers/openai)**
- Day 0 support for `gpt-5.4` and `gpt-5.4-pro` on OpenAI and Azure
- **[Google Gemini](../../docs/providers/gemini)**
- Add Gemini 3.x model cost map entries — `gemini-3-flash-preview`, `gemini-3.1-pro-preview`, `gemini-3.1-flash-lite-preview`, `gemini-3-pro-image-preview`, `gemini-embedding-2-preview`
- Add Gemini 2.0 Flash and Flash Lite to cost map (re-added with updated pricing)
- **[Google Vertex AI](../../docs/providers/vertex)**
- Add `gemini-3-flash-preview`, `gemini-3.1-flash-lite-preview`, `gemini-flash-experimental`, and `gemini-embedding-2-preview` to Vertex AI model cost map
- **[Mistral](../../docs/providers/mistral)**
- Add Magistral reasoning models (`magistral-medium-1-2-2509`, `magistral-small-1-2-2509`)
- Add `mistral-large-2512`, `mistral-medium-3-1-2508`, `mistral-small-3-2-2506`, `ministral-3-*` variants
- **[Dashscope / Qwen](../../docs/providers/dashscope)**
- Add Qwen3 VL multimodal models (`qwen3-vl-235b`, `qwen3-vl-32b` — instruct and thinking variants)
- Add `qwen3-next-80b-a3b` (instruct + thinking), `qwen3.5-plus`, `qwen3-max-2026-01-23`
- **[Black Forest Labs](../../docs/providers/black_forest_labs)**
- Add FLUX Kontext image editing models (`flux-kontext-pro`, `flux-kontext-max`)
- Add FLUX Pro 1.0 Fill (inpainting) and Expand (outpainting)
- Add `flux-pro-1.1`, `flux-pro-1.1-ultra`, `flux-dev`, `flux-pro`
- **[Azure AI](../../docs/providers/azure_ai)**
- Add xAI Grok models via Azure AI Foundry (`grok-4-1-fast-non-reasoning`, `grok-4-1-fast-reasoning`)
- Add Mistral Document AI (`mistral-document-ai-2512`) — OCR mode
- **[AWS Bedrock](../../docs/providers/bedrock)**
- Add `mistral.devstral-2-123b` (256K context, tools)
- Add `zai.glm-4.7-flash` via Bedrock Converse (200K context, tools, reasoning)
- **[SageMaker](../../docs/providers/aws_sagemaker)**
- Add `sagemaker_nova` provider for Amazon Nova models on SageMaker - [PR #21542](https://github.com/BerriAI/litellm/pull/21542)
#### Deprecated / Removed Models
**OpenAI** — Legacy models removed from cost map:
- `gpt-3.5-turbo-0301`, `gpt-3.5-turbo-0613`, `gpt-3.5-turbo-16k-0613`
- `gpt-4-0314`, `gpt-4-32k`, `gpt-4-32k-0314`, `gpt-4-32k-0613`, `gpt-4-1106-vision-preview`, `gpt-4-vision-preview`
- `gpt-4.5-preview`, `gpt-4.5-preview-2025-02-27`
- `gpt-4o-audio-preview-2024-10-01`, `gpt-4o-realtime-preview-2024-10-01`
- `o1-mini`, `o1-mini-2024-09-12`, `o1-preview`, `o1-preview-2024-09-12`
**Google Gemini** — Gemini 1.5 and legacy 2.0 variants removed:
- All `gemini-1.5-*` variants (flash, flash-8b, pro, and dated versions)
- `gemini-2.0-flash-exp`, `gemini-2.0-pro-exp-02-05`, `gemini-2.5-flash-preview-04-17`, `gemini-2.5-flash-preview-05-20`
**Google Vertex AI** — PaLM 2 / legacy models removed:
- All `chat-bison`, `text-bison`, `codechat-bison`, `code-bison`, `code-gecko` variants
- Gemini 1.0 Pro, 1.5 Flash/Pro, 2.0 Flash experimental, and preview variants
**Perplexity** — Legacy Llama-sonar models removed:
- `llama-3.1-sonar-huge-128k-online`, `llama-3.1-sonar-large/small-128k-chat/online`
---
## LLM API Endpoints
#### Features
- **[Responses API](../../docs/response_api)**
- Handle `response.failed`, `response.incomplete`, and `response.cancelled` terminal event types in background streaming — previously only `response.completed` was handled - [PR #23492](https://github.com/BerriAI/litellm/pull/23492)
#### Bug Fixes
- **[Anthropic](../../docs/providers/anthropic)**
- Preserve native tool format (web_search, bash, tool_search, etc.) when guardrails convert tools for the Anthropic Messages API - [PR #23526](https://github.com/BerriAI/litellm/pull/23526)
- **[Moonshot / Kimi](../../docs/providers/openai_compatible)**
- Auto-fill `reasoning_content` for Moonshot Kimi reasoning models - [PR #23580](https://github.com/BerriAI/litellm/pull/23580)
- **[HuggingFace](../../docs/providers/huggingface)**
- Forward `extra_headers` to HuggingFace embedding API - [PR #23525](https://github.com/BerriAI/litellm/pull/23525)
- **General**
- Normalize `content_filtered` finish reason across providers - [PR #23564](https://github.com/BerriAI/litellm/pull/23564)
- Fix custom cost tracking on deployments for `/v1/messages` and `/v1/responses` - [PR #23647](https://github.com/BerriAI/litellm/pull/23647)
- Fix per-request custom pricing when `router_model_id` has no pricing data — now falls back to model name
---
## Management Endpoints / UI
#### Features
- **Virtual Keys**
- Add Organization dropdown to Create/Edit Key form — `organization_id` is now a first-class field in Key Ownership - [PR #23595](https://github.com/BerriAI/litellm/pull/23595)
- Allow setting `organization_id` on `/key/update` — keys can be assigned or moved to a different organization after creation - [PR #23557](https://github.com/BerriAI/litellm/pull/23557)
- **Internal Users**
- Add/Remove Team Membership directly from the Internal Users info page — includes searchable dropdown and role selector; no longer requires navigating to each team - [PR #23638](https://github.com/BerriAI/litellm/pull/23638)
- **Default Team Settings**
- Modernize page to antd (consistent with rest of app) - [PR #23614](https://github.com/BerriAI/litellm/pull/23614)
- Fix: default team params (budget, duration, tpm, rpm, permissions) now correctly applied on `/team/new` - [PR #23614](https://github.com/BerriAI/litellm/pull/23614)
- Fix: settings persist across proxy restarts (`default_team_params` added to `LITELLM_SETTINGS_SAFE_DB_OVERRIDES`) - [PR #23614](https://github.com/BerriAI/litellm/pull/23614)
- Fix: resolved race condition in `_update_litellm_setting` where `get_config()` could overwrite freshly saved values - [PR #23614](https://github.com/BerriAI/litellm/pull/23614)
- **Usage**
- Auto-paginate daily spend data — all entity views (teams, orgs, customers, tags, agents, users) fetch pages progressively with charts updating after each page - [PR #23622](https://github.com/BerriAI/litellm/pull/23622)
- **Models / Cost**
- Azure Model Router cost breakdown in UI — show per-sub-model `additional_costs` from `hidden_params` in `CostBreakdownViewer` - [PR #23550](https://github.com/BerriAI/litellm/pull/23550)
- **User Management**
- New `/user/info/v2` endpoint — scoped, paginated replacement for the existing god endpoint that caused memory and stability issues on large installs - [PR #23437](https://github.com/BerriAI/litellm/pull/23437)
#### Bugs
- Fix Tag list endpoint returning 500 due to invalid Prisma `group_by` kwargs - [PR #23606](https://github.com/BerriAI/litellm/pull/23606)
- Fix Team Admin getting 403 on `/user/filter/ui` when `scope_user_search_to_org` is enabled - [PR #23671](https://github.com/BerriAI/litellm/pull/23671)
- Fix Public Model Hub not showing config-defined models after save - [PR #23501](https://github.com/BerriAI/litellm/pull/23501)
- Fix fallback popup model dropdown z-index issue - [PR #23516](https://github.com/BerriAI/litellm/pull/23516)
- Fix double-counting bug in org/team key limit checks on `/key/update`
---
## AI Integrations
### Logging
- **[Vantage](https://vantage.sh)**
- Add Vantage integration for FOCUS 1.2 CSV export — export LiteLLM proxy spend data as FinOps Open Cost & Usage Specification reports, with time-windowed filenames to prevent overwrites - [PR #23333](https://github.com/BerriAI/litellm/pull/23333)
- **General**
- Fix silent metrics race condition causing metric collision across experiments - [PR #23542](https://github.com/BerriAI/litellm/pull/23542)
### Guardrails
No major guardrail changes in this release.
### Prompt Management
No major prompt management changes in this release.
### Secret Managers
No major secret manager changes in this release.
---
## Performance / Loadbalancing / Reliability improvements
- **Fix streaming crashes after ~1 hour** — `LLMClientCache._remove_key()` no longer calls `close()`/`aclose()` on evicted HTTP/SDK clients. In-flight requests were crashing with `RuntimeError: Cannot send a request, as the client has been closed.` after the 1-hour TTL expired. Cleanup now happens only at shutdown via `close_litellm_async_clients()` - [PR #22926](https://github.com/BerriAI/litellm/pull/22926)
- **Fix OOM / Prisma connection loss** on large installs — unbounded managed-object poll was exhausting Prisma connections after ~60–70 minutes on instances with 336K+ queued response rows - [PR #23472](https://github.com/BerriAI/litellm/pull/23472)
- **Centralize logging kwarg updates** — root cause fix migrating all logging updates to a single function, eliminating kwarg inconsistencies across logging paths - [PR #23659](https://github.com/BerriAI/litellm/pull/23659)
- **Fix tiktoken cache for non-root offline containers** — tiktoken cache now works correctly in offline environments running as non-root users - [PR #23498](https://github.com/BerriAI/litellm/pull/23498)
- **Add CodSpeed continuous performance benchmarks** — automated performance regression tracking on CI - [PR #23676](https://github.com/BerriAI/litellm/pull/23676)
---
## Security
- **Secret redaction in proxy logs** — Adds a `SecretRedactionFilter` to all LiteLLM loggers that scrubs API keys, tokens, and credentials from log messages, format args, exception tracebacks, and extra fields. Enabled by default; opt out with `LITELLM_DISABLE_REDACT_SECRETS=true` - [PR #23668](https://github.com/BerriAI/litellm/pull/23668), [PR #23667](https://github.com/BerriAI/litellm/pull/23667)
- **Bump PyJWT to `^2.12.0`** — addresses security vulnerability in `^2.10.1` - [PR #23678](https://github.com/BerriAI/litellm/pull/23678)
- **Bump `tar` to 7.5.11 and `tornado` to 6.5.5** — addresses CVEs in transitive dependencies - [PR #23602](https://github.com/BerriAI/litellm/pull/23602)
---
## Database / Proxy Operations
- **Fix Prisma migrate deploy on pre-existing instances** — resolved multiple bugs in migration recovery logic: missing return in the P3018 idempotent error handler and unhandled exceptions in `_roll_back_migration` that caused silent failures even after successful recovery - [PR #23655](https://github.com/BerriAI/litellm/pull/23655)
- **Make DB migration failure exit opt-in** — proxy no longer exits on `prisma migrate deploy` failure by default; enable with `--enforce_prisma_migration_check` - [PR #23675](https://github.com/BerriAI/litellm/pull/23675)
---
## New Contributors
* @ryanh-ai made their first contribution in [PR #21542](https://github.com/BerriAI/litellm/pull/21542)
* @ryan-crabbe made their first contribution in [PR #23668](https://github.com/BerriAI/litellm/pull/23668)
* @Jah-yee made their first contribution in [PR #23525](https://github.com/BerriAI/litellm/pull/23525)
* @gambletan made their first contribution in [PR #23516](https://github.com/BerriAI/litellm/pull/23516)
* @awais786 made their first contribution in [PR #23183](https://github.com/BerriAI/litellm/pull/23183)
* @pradyyadav made their first contribution in [PR #23580](https://github.com/BerriAI/litellm/pull/23580)
* @xianzongxie-stripe made their first contribution in [PR #23492](https://github.com/BerriAI/litellm/pull/23492)
* @Harshit28j made their first contribution in [PR #23333](https://github.com/BerriAI/litellm/pull/23333)
* @codspeed-hq[bot] made their first contribution in [PR #23676](https://github.com/BerriAI/litellm/pull/23676)
---
## Diff Summary
## 03/16/2026
* New Providers: 5
* New Models / Updated Models: 116 new, 132 removed
* LLM API Endpoints: 5
* Management Endpoints / UI: 11
* AI Integrations: 2
* Performance / Reliability: 5
* Security: 3
* Database / Proxy Operations: 2
---
## Full Changelog
[v1.82.0-stable...v1.82.3-stable](https://github.com/BerriAI/litellm/compare/v1.82.0-stable...v1.82.3-stable)

14
docs/my-website/sidebars-release-notes.js Normal file
View file

@ -0,0 +1,14 @@
// @ts-check
/** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */
const sidebars = {
releaseNotesSidebar: [
{ type: 'doc', id: 'index', label: 'Release Notes' },
{
type: 'autogenerated',
dirName: '.',
},
],
};
module.exports = sidebars;

622
docs/my-website/sidebars.js
View file

@ -20,6 +20,7 @@ const sidebars = {
{
type: "category",
label: "Observability",
link: { type: "doc", id: "integrations/observability_index" },
items: [
{
type: "category",
@ -29,80 +30,55 @@ const sidebars = {
type: "autogenerated",
dirName: "contribute_integration"
}
]
],
},
{
type: "autogenerated",
dirName: "observability"
}
],
},
{
type: "category",
label: "Guardrails",
items: [
"proxy/guardrails/quick_start",
"proxy/guardrails/team_based_guardrails",
"proxy/guardrails/guardrail_load_balancing",
"proxy/guardrails/test_playground",
"proxy/guardrails/litellm_content_filter",
"proxy/guardrails/realtime_guardrails",
{
type: "category",
label: "Providers",
items: [
...[
"proxy/guardrails/qualifire",
"proxy/guardrails/aim_security",
"proxy/guardrails/onyx_security",
"proxy/guardrails/aporia_api",
"proxy/guardrails/azure_content_guardrail",
"proxy/guardrails/bedrock",
"proxy/guardrails/crowdstrike_aidr",
"proxy/guardrails/enkryptai",
"proxy/guardrails/ibm_guardrails",
"proxy/guardrails/grayswan",
"proxy/guardrails/hiddenlayer",
"proxy/guardrails/lasso_security",
"proxy/guardrails/guardrails_ai",
"proxy/guardrails/lakera_ai",
"proxy/guardrails/model_armor",
"proxy/guardrails/noma_security",
"proxy/guardrails/dynamoai",
"proxy/guardrails/openai_moderation",
"proxy/guardrails/pangea",
"proxy/guardrails/pillar_security",
"proxy/guardrails/pii_masking_v2",
"proxy/guardrails/panw_prisma_airs",
"proxy/guardrails/secret_detection",
"proxy/guardrails/custom_guardrail",
"proxy/guardrails/custom_code_guardrail",
"proxy/guardrails/prompt_injection",
"proxy/guardrails/tool_permission",
"proxy/guardrails/zscaler_ai_guard",
"proxy/guardrails/javelin"
].sort(),
],
},
{
type: "category",
label: "Contributing to Guardrails",
items: [
"adding_provider/generic_guardrail_api",
"adding_provider/simple_guardrail_tutorial",
"adding_provider/adding_guardrail_support",
]
},
],
},
{
type: "category",
label: "Policies",
label: "Guardrail Providers",
link: {
type: "generated-index",
title: "Guardrail Providers",
description: "Add safety and content filtering to LLM calls",
slug: "/guardrail_providers"
},
items: [
"proxy/guardrails/guardrail_policies",
"proxy/guardrails/policy_flow_builder",
"proxy/guardrails/policy_templates",
"proxy/guardrails/policy_tags",
...[
"proxy/guardrails/qualifire",
"proxy/guardrails/aim_security",
"proxy/guardrails/onyx_security",
"proxy/guardrails/aporia_api",
"proxy/guardrails/azure_content_guardrail",
"proxy/guardrails/bedrock",
"proxy/guardrails/crowdstrike_aidr",
"proxy/guardrails/enkryptai",
"proxy/guardrails/ibm_guardrails",
"proxy/guardrails/grayswan",
"proxy/guardrails/hiddenlayer",
"proxy/guardrails/lasso_security",
"proxy/guardrails/guardrails_ai",
"proxy/guardrails/lakera_ai",
"proxy/guardrails/model_armor",
"proxy/guardrails/noma_security",
"proxy/guardrails/dynamoai",
"proxy/guardrails/openai_moderation",
"proxy/guardrails/pangea",
"proxy/guardrails/pillar_security",
"proxy/guardrails/pii_masking_v2",
"proxy/guardrails/panw_prisma_airs",
"proxy/guardrails/secret_detection",
"proxy/guardrails/custom_guardrail",
"proxy/guardrails/custom_code_guardrail",
"proxy/guardrails/prompt_injection",
"proxy/guardrails/tool_permission",
"proxy/guardrails/zscaler_ai_guard",
"proxy/guardrails/javelin"
].sort(),
],
},
{
@ -115,11 +91,6 @@ const sidebars = {
"proxy/pyroscope_profiling"
]
},
{
type: "doc",
id: "integrations/websearch_interception",
label: "Web Search Integration"
},
{
type: "category",
label: "[Beta] Prompt Management",
@ -140,7 +111,7 @@ const sidebars = {
},
{
type: "category",
label: "AI Tools (OpenWebUI, Claude Code, etc.)",
label: "AI Tools",
link: {
type: "generated-index",
title: "AI Tools",
@ -167,14 +138,13 @@ const sidebars = {
},
"tutorials/opencode_integration",
"tutorials/openclaw_integration",
"tutorials/cost_tracking_coding",
"tutorials/cursor_integration",
"tutorials/github_copilot_integration",
"tutorials/litellm_gemini_cli",
"tutorials/google_genai_sdk",
"tutorials/litellm_qwen_code_cli",
"tutorials/openai_codex",
"tutorials/retool_assist"
"tutorials/retool_assist",
"tutorials/cost_tracking_coding"
]
},
{
@ -191,7 +161,11 @@ const sidebars = {
"tutorials/claude_agent_sdk",
"tutorials/copilotkit_sdk",
"tutorials/google_adk",
"tutorials/google_genai_sdk",
"tutorials/livekit_xai_realtime",
"integrations/letta",
{ type: "doc", id: "tutorials/instructor", label: "Instructor with LiteLLM" },
{ type: "doc", id: "langchain/langchain", label: "LangChain with LiteLLM" },
"projects/openai-agents"
]
},
@ -212,7 +186,20 @@ const sidebars = {
],
// But you can create a sidebar manually
tutorialSidebar: [
{ type: "doc", id: "index", label: "Getting Started" },
// ════════════════════════════════════════════════════════════
// GET STARTED
// ════════════════════════════════════════════════════════════
{
type: "category",
label: "Get Started",
collapsible: false,
collapsed: false,
items: [
{ type: "doc", id: "index", label: "Quickstart" },
{ type: "link", label: "Models & Pricing", href: "https://models.litellm.ai" },
{ type: "link", label: "Changelog", href: "/release_notes" },
],
},
{
type: "category",
@ -280,11 +267,6 @@ const sidebars = {
},
"completion/token_usage",
"exception_mapping",
{
type: "category",
label: "LangChain, LlamaIndex, Instructor",
items: ["langchain/langchain", "tutorials/instructor"],
}
],
},
{
@ -297,16 +279,46 @@ const sidebars = {
slug: "/simple_proxy",
},
items: [
"proxy/docker_quick_start",
{ type: "doc", id: "proxy/docker_quick_start", label: "Getting Started Tutorial" },
{
type: "link",
label: "A2A Agent Gateway",
href: "https://docs.litellm.ai/docs/a2a",
},
{
type: "link",
label: "MCP Gateway",
href: "https://docs.litellm.ai/docs/mcp",
type: "category",
label: "Agent & MCP Gateway",
items: [
{
type: "category",
label: "A2A Agent Gateway",
items: [
"a2a",
"a2a_invoking_agents",
"a2a_agent_headers",
"a2a_cost_tracking",
"a2a_agent_permissions",
"a2a_iteration_budgets",
],
},
{
type: "category",
label: "MCP Gateway",
items: [
"mcp",
"mcp_usage",
"mcp_openapi",
"mcp_oauth",
"mcp_aws_sigv4",
"mcp_zero_trust",
"mcp_public_internet",
"mcp_semantic_filter",
"mcp_control",
"mcp_cost",
"mcp_guardrail",
{
type: "link",
label: "MCP Troubleshooting Guide",
href: "/docs/mcp_troubleshoot"
},
],
},
],
},
{
"type": "category",
@ -365,7 +377,11 @@ const sidebars = {
type: "category",
label: "Teams & Organizations",
items: [
"proxy/access_control",
{
type: "link",
label: "Role-based Access Controls (RBAC) →",
href: "/docs/proxy/access_control"
},
"proxy/self_serve",
"proxy/public_teams",
"proxy/ui_project_management",
@ -448,14 +464,40 @@ const sidebars = {
},
"proxy/caching",
{
type: "link",
type: "category",
label: "Guardrails",
href: "https://docs.litellm.ai/docs/proxy/guardrails/quick_start",
items: [
"proxy/guardrails/quick_start",
"proxy/guardrails/team_based_guardrails",
"proxy/guardrails/guardrail_load_balancing",
"proxy/guardrails/test_playground",
"proxy/guardrails/litellm_content_filter",
"proxy/guardrails/realtime_guardrails",
{
type: "link",
label: "Providers →",
href: "/docs/guardrail_providers",
},
{
type: "category",
label: "Contributing to Guardrails",
items: [
"adding_provider/generic_guardrail_api",
"adding_provider/simple_guardrail_tutorial",
"adding_provider/adding_guardrail_support",
]
},
],
},
{
type: "link",
type: "category",
label: "Policies",
href: "https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies",
items: [
"proxy/guardrails/guardrail_policies",
"proxy/guardrails/policy_flow_builder",
"proxy/guardrails/policy_templates",
"proxy/guardrails/policy_tags",
],
},
{
type: "category",
@ -551,16 +593,9 @@ const sidebars = {
},
items: [
{
type: "category",
type: "link",
label: "/a2a - A2A Agent Gateway",
items: [
"a2a",
"a2a_invoking_agents",
"a2a_agent_headers",
"a2a_cost_tracking",
"a2a_agent_permissions",
"a2a_iteration_budgets"
],
href: "/docs/a2a",
},
"assistants",
"audio_transcription",
@ -975,51 +1010,14 @@ const sidebars = {
"providers/zai",
],
},
{
type: "category",
label: "Guides",
items: [
"budget_manager",
"completion/computer_use",
"completion/web_search",
"completion/web_fetch",
"completion/function_call",
"completion/audio",
"completion/document_understanding",
"completion/drop_params",
"completion/image_generation_chat",
"completion/json_mode",
"completion/knowledgebase",
"providers/anthropic_tool_search",
"guides/code_interpreter",
"completion/message_trimming",
"completion/message_sanitization",
"completion/model_alias",
"completion/mock_requests",
"completion/predict_outputs",
"completion/prefix",
"completion/prompt_caching",
"completion/prompt_formatting",
"completion/reliable_completions",
"completion/stream",
"completion/provider_specific_params",
"completion/vision",
"exception_mapping",
"completion/batching",
"guides/finetuned_models",
"guides/security_settings",
"proxy/veo_video_generation",
"reasoning_content",
"extras/creating_adapters",
]
},
{
type: "category",
label: "Routing, Loadbalancing & Fallbacks",
label: "Routing & Load Balancing",
link: {
type: "generated-index",
title: "Routing, Loadbalancing & Fallbacks",
title: "Routing & Load Balancing",
description: "Learn how to load balance, route, and set fallbacks for your LLM requests",
slug: "/routing-load-balancing",
},
@ -1047,43 +1045,6 @@ const sidebars = {
"load_test_rpm",
]
},
{
type: "category",
label: "Tutorials",
items: [
{
type: "link",
label: "AI Coding Tools (OpenWebUI, Claude Code, Gemini CLI, OpenAI Codex, etc.)",
href: "/docs/ai_tools",
},
"tutorials/anthropic_file_usage",
"tutorials/default_team_self_serve",
"tutorials/msft_sso",
"tutorials/prompt_caching",
"tutorials/tag_management",
'tutorials/litellm_proxy_aporia',
"tutorials/presidio_pii_masking",
"tutorials/elasticsearch_logging",
"tutorials/gemini_realtime_with_audio",
"tutorials/claude_code_beta_headers",
{
type: "category",
label: "LiteLLM Python SDK Tutorials",
items: [
'tutorials/azure_openai',
'tutorials/instructor',
"tutorials/gradio_integration",
"tutorials/huggingface_codellama",
"tutorials/huggingface_tutorial",
"tutorials/TogetherAI_liteLLM",
"tutorials/finetuned_chat_gpt",
"tutorials/text_completion",
"tutorials/first_playground",
"tutorials/model_fallbacks",
],
},
]
},
{
type: "category",
label: "Contributing",
@ -1140,12 +1101,11 @@ const sidebars = {
"projects/Codium PR Agent",
"projects/Prompt2Model",
"projects/SalesGPT",
"projects/Softgen",
"projects/Quivr",
"projects/Langstream",
"projects/Otter",
"projects/GPT Migrate",
"projects/YiVal",
"projects/LiteLLM Proxy",
"projects/llm_cord",
"projects/pgai",
"projects/GPTLocalhost",
@ -1183,23 +1143,309 @@ const sidebars = {
"troubleshoot",
],
},
],
};
const learnSidebar = {
learnSidebar: [
// ── Landing page ──────────────────────────────────────────────────
{ type: "doc", id: "learn/index", label: "Learn" },
{
type: "category",
label: "Blog",
label: "Start Here",
collapsible: true,
collapsed: false,
items: [
"learn/sdk_quickstart",
"learn/gateway_quickstart",
],
},
// ── Guides ────────────────────────────────────────────────────────
{
type: "category",
label: "Guides",
collapsible: true,
collapsed: false,
link: { type: "doc", id: "guides/index" },
items: [
{
type: "link",
label: "Day 0 Support: Claude Sonnet 4.6",
href: "/blog/claude_sonnet_4_6",
type: "category",
label: "Core Requests",
collapsible: true,
collapsed: false,
link: {
type: "generated-index",
title: "Core Requests",
description: "Streaming, batching, structured outputs, and reasoning behavior",
slug: "/guides/core_request_response_patterns"
},
items: [
"completion/stream",
"completion/batching",
"completion/json_mode",
"reasoning_content",
],
},
{
type: "category",
label: "Tool Calling",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Tool Calling",
description: "Function calling, web tools, interception patterns, computer use, code interpreter, and tool-call hygiene",
slug: "/guides/tools_integrations"
},
items: [
"completion/function_call",
"completion/web_search",
{
type: "doc",
id: "integrations/websearch_interception",
label: "Web Search Interception",
},
"completion/web_fetch",
"completion/computer_use",
"guides/code_interpreter",
"completion/message_sanitization",
],
},
{
type: "category",
label: "Multimodal I/O",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Multimodal I/O",
description: "Vision, audio, PDFs, image generation, and video generation",
slug: "/guides/multimodal_io"
},
items: [
"completion/vision",
"completion/audio",
"completion/document_understanding",
"completion/image_generation_chat",
"proxy/veo_video_generation",
],
},
{
type: "category",
label: "Retrieval & Knowledge",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Retrieval & Knowledge",
description: "Vector stores, file search, citations, and knowledge-base routing",
slug: "/guides/retrieval_knowledge"
},
items: [
"completion/knowledgebase",
],
},
{
type: "category",
label: "Prompts & Context",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Prompts & Context",
description: "Prompt caching, trimming, formatting, assistant prefill, and predicted outputs",
slug: "/guides/prompts_context"
},
items: [
"completion/prefix",
"completion/predict_outputs",
"completion/message_trimming",
"completion/prompt_caching",
"completion/prompt_formatting",
],
},
{
type: "category",
label: "Compatibility & Extensibility",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Compatibility & Extensibility",
description: "Provider-specific params, model aliases, fine-tuned models, and adapters",
slug: "/guides/compatibility_extensibility"
},
items: [
"completion/provider_specific_params",
"completion/drop_params",
"completion/model_alias",
"guides/finetuned_models",
"extras/creating_adapters",
],
},
{
type: "category",
label: "Reliability, Testing & Spend",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Reliability, Testing & Spend",
description: "Retries, fallbacks, mock responses, and budget controls",
slug: "/guides/reliability_testing_spend"
},
items: [
"completion/mock_requests",
"completion/reliable_completions",
"budget_manager",
],
},
{
type: "category",
label: "Security & Network",
collapsible: true,
collapsed: true,
link: {
type: "generated-index",
title: "Security & Network",
description: "SSL, custom CA bundles, HTTP proxy settings, and per-service verification",
slug: "/guides/security_network"
},
items: [
"guides/security_settings",
],
},
],
},
// ── Tutorials ─────────────────────────────────────────────────────
{
type: "category",
label: "Tutorials",
collapsible: true,
collapsed: false,
link: { type: "doc", id: "tutorials/index" },
items: [
{
type: "category",
label: "Getting Started",
collapsed: false,
link: {
type: "generated-index",
title: "Getting Started",
description: "Installation, playground, text completion, and mock completions",
slug: "/tutorials/getting_started"
},
items: [
"tutorials/installation",
"tutorials/first_playground",
"tutorials/text_completion",
"tutorials/mock_completion",
],
},
{
type: "link",
label: "Incident: Broken Model Cost Map",
href: "/blog/model-cost-map-incident",
label: "Agent SDKs & Frameworks",
href: "/docs/agent_sdks",
},
{
type: "link",
label: "AI Coding Tools",
href: "/docs/ai_tools",
},
{
type: "category",
label: "Python SDK",
collapsed: true,
link: {
type: "generated-index",
title: "Python SDK",
description: "Tutorials using only the Python SDK — no proxy server required",
slug: "/tutorials/python_sdk"
},
items: [
"tutorials/gradio_integration",
"tutorials/provider_specific_params",
"tutorials/model_fallbacks",
"tutorials/fallbacks",
],
},
{
type: "category",
label: "Provider Setup",
collapsed: true,
link: {
type: "generated-index",
title: "Provider Setup",
description: "Connect LiteLLM to Azure OpenAI, HuggingFace, TogetherAI, local models, and more",
slug: "/tutorials/provider_tutorials"
},
items: [
"tutorials/azure_openai",
"tutorials/TogetherAI_liteLLM",
"tutorials/huggingface_tutorial",
"tutorials/huggingface_codellama",
"tutorials/finetuned_chat_gpt",
"tutorials/oobabooga",
],
},
{
type: "category",
label: "Proxy: Admin & Access",
collapsed: true,
link: {
type: "generated-index",
title: "Proxy: Admin & Access",
description: "User and team management, SSO, SCIM, and routing rules",
slug: "/tutorials/proxy_admin_access"
},
items: [
"tutorials/default_team_self_serve",
"tutorials/msft_sso",
"tutorials/scim_litellm",
"tutorials/tag_management",
],
},
{
type: "category",
label: "Proxy: Features & Safety",
collapsed: true,
link: {
type: "generated-index",
title: "Proxy: Features & Safety",
description: "Prompt caching, passthrough APIs, realtime, guardrails, and PII masking",
slug: "/tutorials/proxy_features_safety"
},
items: [
"tutorials/prompt_caching",
"tutorials/anthropic_file_usage",
"tutorials/gemini_realtime_with_audio",
"tutorials/litellm_proxy_aporia",
"tutorials/presidio_pii_masking",
],
},
{
type: "category",
label: "Observability & Evaluation",
collapsed: true,
link: {
type: "generated-index",
title: "Observability & Evaluation",
description: "Logging, monitoring, benchmarking, and evaluation suites",
slug: "/tutorials/observability_evaluation"
},
items: [
"tutorials/elasticsearch_logging",
"tutorials/compare_llms",
"tutorials/litellm_Test_Multiple_Providers",
"tutorials/eval_suites",
"tutorials/lm_evaluation_harness",
],
},
],
},
],
};
module.exports = sidebars;
module.exports = { ...sidebars, ...learnSidebar };

44
docs/my-website/src/components/NavigationCards/index.js Normal file
View file

@ -0,0 +1,44 @@
import React from 'react';
import Link from '@docusaurus/Link';
import styles from './styles.module.css';
export default function NavigationCards({ items, columns = 2 }) {
return (
<div
className={styles.grid}
style={{ '--nav-columns': columns }}
>
{items.map((item, i) => {
const isExternal =
item.to && (item.to.startsWith('http://') || item.to.startsWith('https://'));
return (
<Link
key={i}
to={item.to}
className={styles.card}
target={isExternal ? '_blank' : undefined}
rel={isExternal ? 'noopener noreferrer' : undefined}
>
{item.icon && (
<div className={styles.icon}>{item.icon}</div>
)}
<div className={styles.title}>{item.title}</div>
{item.description && (
<div className={styles.description}>{item.description}</div>
)}
{item.listDescription && (
<ul className={styles.list}>
{item.listDescription.map((line, j) => (
<li key={j}>{line}</li>
))}
</ul>
)}
{isExternal && (
<span className={styles.externalIcon}>↗</span>
)}
</Link>
);
})}
</div>
);
}

82
docs/my-website/src/components/NavigationCards/styles.module.css Normal file
View file

@ -0,0 +1,82 @@
.grid {
display: grid;
grid-template-columns: repeat(var(--nav-columns, 2), 1fr);
gap: 0.75rem;
margin: 1.25rem 0;
}
@media (max-width: 768px) {
.grid {
grid-template-columns: 1fr;
}
}
.card {
position: relative;
display: flex;
flex-direction: column;
padding: 1rem 1.1rem;
border: 1px solid var(--ifm-color-emphasis-200);
border-radius: 6px;
text-decoration: none !important;
color: inherit !important;
background: var(--ifm-background-surface-color);
transition: border-color 0.15s ease, box-shadow 0.15s ease;
}
.card:hover {
border-color: var(--ifm-color-primary);
box-shadow: 0 0 0 1px var(--ifm-color-primary);
text-decoration: none !important;
}
[data-theme='dark'] .card {
background: var(--ifm-background-surface-color);
border-color: #2d3748;
}
[data-theme='dark'] .card:hover {
border-color: var(--ifm-color-primary);
box-shadow: 0 0 0 1px var(--ifm-color-primary);
}
.icon {
font-size: 1.4rem;
margin-bottom: 0.5rem;
line-height: 1;
}
.title {
font-size: 14px;
font-weight: 600;
margin-bottom: 0.35rem;
color: var(--ifm-heading-color);
}
.description {
font-size: 13px;
line-height: 1.5;
color: var(--ifm-color-emphasis-700);
margin-bottom: 0.5rem;
}
.list {
margin: 0.35rem 0 0 0;
padding-left: 1.1rem;
list-style: disc;
}
.list li {
font-size: 12.5px;
color: var(--ifm-color-emphasis-700);
line-height: 1.6;
margin-bottom: 0;
}
.externalIcon {
position: absolute;
top: 0.75rem;
right: 0.75rem;
font-size: 12px;
color: var(--ifm-color-emphasis-500);
}

757
docs/my-website/src/css/custom.css
View file

@ -1,10 +1,16 @@
/**
* Any CSS included here will be global. The classic template
* bundles Infima by default. Infima is a CSS framework designed to
* work well for content-centric websites.
* Global CSS overrides for LiteLLM docs.
* Infima (Docusaurus CSS framework) variables + custom styling.
*/
/* You can override the default Infima variables here. */
/* =========================================
FONTS
========================================= */
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap');
/* =========================================
ROOT — Light Mode Variables
========================================= */
:root {
--ifm-color-primary: #2e8555;
--ifm-color-primary-dark: #29784c;
@ -13,11 +19,22 @@
--ifm-color-primary-light: #33925d;
--ifm-color-primary-lighter: #359962;
--ifm-color-primary-lightest: #3cad6e;
--ifm-code-font-size: 95%;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
--ifm-code-font-size: 85%;
--ifm-menu-color: #6b7280;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.08);
--ifm-font-family-base: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
--ifm-heading-font-weight: 600;
--ifm-font-size-base: 15px;
--ifm-line-height-base: 1.65;
--ifm-border-radius: 6px;
/* Wider reading column — reduces excessive whitespace on large monitors */
--ifm-container-width: 1380px;
--ifm-container-width-xl: 1560px;
}
/* For readability concerns, you should choose a lighter palette in dark mode. */
/* =========================================
DARK MODE Variables
========================================= */
[data-theme='dark'] {
--ifm-color-primary: #25c2a0;
--ifm-color-primary-dark: #21af90;
@ -26,10 +43,710 @@
--ifm-color-primary-light: #29d5b0;
--ifm-color-primary-lighter: #32d8b4;
--ifm-color-primary-lightest: #4fddbf;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
--ifm-background-color: #0d1117;
--ifm-background-surface-color: #161b22;
--docusaurus-highlighted-code-line-bg: rgba(255, 255, 255, 0.07);
}
/* Levo logo sizing and theme switching */
/* =========================================
TYPOGRAPHY
========================================= */
.theme-doc-markdown h1 {
font-size: 2.2rem;
letter-spacing: -0.02em;
line-height: 1.2;
}
.theme-doc-markdown h2 {
font-size: 1.6rem;
letter-spacing: -0.01em;
line-height: 1.3;
}
.theme-doc-markdown h3 {
font-size: 1.25rem;
line-height: 1.4;
}
.theme-doc-markdown p,
.theme-doc-markdown ul,
.theme-doc-markdown ol {
font-size: 0.9rem;
}
.theme-doc-markdown table {
font-size: 0.875rem;
}
.theme-doc-markdown td {
font-size: 0.85rem;
}
/* =========================================
NAVBAR
========================================= */
[data-theme='light'] .navbar {
background-color: #ffffff;
box-shadow: 0 1px 0 0 #e5e7eb;
}
[data-theme='dark'] .navbar {
background-color: var(--ifm-background-color);
border-bottom: 1px solid #21262d;
box-shadow: none;
}
.navbar__link {
font-weight: 400 !important;
font-size: 14px !important;
border-bottom: 2px solid transparent !important;
padding-bottom: 2px;
}
.navbar__link--active {
font-weight: 500 !important;
border-bottom: 2px solid var(--ifm-color-primary) !important;
}
@media (max-width: 1330px) {
.navbar__link {
font-size: 13px !important;
}
}
/* Three-column navbar: logo | center nav | right icons */
@media (min-width: 997px) {
.navbar__inner {
display: flex !important;
align-items: center;
justify-content: space-between;
}
.navbar__brand-col {
display: flex;
align-items: center;
flex: 0 0 auto;
margin-left: 1rem;
}
.navbar__brand-col .navbar__brand {
font-size: 1.25rem;
}
.navbar__brand-col .navbar__logo {
height: 2rem;
width: auto;
}
.navbar__center-col {
display: flex;
align-items: center;
justify-content: center;
flex: 1;
}
.navbar__right-col {
display: flex;
align-items: center;
flex: 0 0 auto;
gap: 0.25rem;
margin-right: 2rem;
}
}
/* =========================================
ALERTS / ADMONITIONS
========================================= */
.alert {
padding: 0.75rem 1rem;
font-size: 14px;
border-radius: var(--ifm-border-radius);
border-left-width: 3px;
}
/* Light mode */
.alert--info {
--ifm-alert-background-color: #f0f7ff !important;
--ifm-alert-border-color: #2264ab !important;
}
.alert--success {
--ifm-alert-background-color: #f0fff8 !important;
--ifm-alert-border-color: #09bda8 !important;
}
.alert--secondary {
--ifm-alert-background-color: #f8fafc !important;
--ifm-alert-border-color: #64748b !important;
}
.alert--danger {
--ifm-alert-background-color: #fff0f5 !important;
--ifm-alert-border-color: #e11d48 !important;
}
.alert--warning {
--ifm-alert-background-color: #fffbeb !important;
--ifm-alert-border-color: #d97706 !important;
}
/* Dark mode */
[data-theme='dark'] .alert--info {
--ifm-alert-background-color: #0c1e30 !important;
--ifm-alert-border-color: #3b82f6 !important;
color: #bfdbfe !important;
}
[data-theme='dark'] .alert--success {
--ifm-alert-background-color: #022c22 !important;
--ifm-alert-border-color: #10b981 !important;
}
[data-theme='dark'] .alert--secondary {
--ifm-alert-background-color: #0f172a !important;
--ifm-alert-border-color: #475569 !important;
}
[data-theme='dark'] .alert--danger {
--ifm-alert-background-color: #2d0a14 !important;
--ifm-alert-border-color: #f43f5e !important;
}
[data-theme='dark'] .alert--warning {
--ifm-alert-background-color: #1c1200 !important;
--ifm-alert-border-color: #f59e0b !important;
}
/* =========================================
COLLAPSIBLE / DETAILS
========================================= */
details {
color: #1d232e;
background-color: #ffffff;
border: 1px solid #e9eef2 !important;
border-radius: var(--ifm-border-radius) !important;
padding: 0.75rem !important;
margin-bottom: 1rem !important;
margin-top: 1.5rem !important;
box-shadow: 0 1px 4px 0 rgba(0, 0, 0, 0.05) !important;
}
details summary {
font-weight: 600 !important;
}
details [class*='collapsibleContent'] {
border-top: 1px solid #e2e8f0 !important;
}
details [class*='collapsibleContent'] p,
details [class*='collapsibleContent'] ul {
font-size: 13px !important;
line-height: 1.75;
}
[data-theme='dark'] details {
background-color: #1c2130 !important;
color: #e5e7eb !important;
border: 1px solid #2d3748 !important;
box-shadow: 0 1px 6px 0 rgba(0, 0, 0, 0.3) !important;
}
[data-theme='dark'] details summary {
color: #f3f4f6 !important;
}
[data-theme='dark'] details [class*='collapsibleContent'] {
border-top: 1px solid #2d3748 !important;
}
/* =========================================
TABS
========================================= */
.tabs-container > div {
padding: 1rem;
border: 1px solid #e2e8f0;
border-radius: var(--ifm-border-radius);
}
/* Remove styling from nested tabs containers */
.tabs-container .tabs-container > div {
padding: 0;
border: none;
border-radius: 0;
}
[data-theme='dark'] .tabs-container > div {
border-color: #2d3748;
}
ul.tabs {
border-bottom: 1px solid #e2e8f0 !important;
column-gap: 0.5rem !important;
}
[data-theme='dark'] ul.tabs {
border-bottom-color: #2d3748 !important;
}
li.tabs__item {
padding: 0.5rem !important;
font-weight: 500 !important;
font-size: 14px !important;
border-bottom: 2px solid transparent !important;
}
li.tabs__item--active {
border-bottom: 2px solid var(--ifm-color-primary) !important;
}
/* =========================================
CODE BLOCKS
========================================= */
.prism-code {
border-radius: var(--ifm-border-radius);
font-size: 12.5px !important;
line-height: 1.6;
}
[data-theme='dark'] .prism-code {
border: 1px solid #21262d;
}
[class*='codeLineNumber']::before {
font-size: 12px !important;
display: flex !important;
align-items: center !important;
justify-content: center !important;
}
[class*='codeBlockTitle'] {
font-size: 13px !important;
font-weight: 500 !important;
padding: 0.5rem 1rem !important;
border-bottom: 1px solid #334155 !important;
}
.theme-code-block-highlighted-line {
background-color: rgba(0, 0, 0, 0.1) !important;
}
[data-theme='dark'] .theme-code-block-highlighted-line {
background-color: rgba(255, 255, 255, 0.06) !important;
}
.theme-code-block-highlighted-line > span {
background-color: transparent !important;
}
/* =========================================
SIDEBAR / MENU
========================================= */
.menu {
font-weight: 400;
padding: 0.5rem 0.25rem !important;
background-image: radial-gradient(rgba(0, 0, 0, 0.07) 1px, transparent 1px);
background-size: 24px 24px;
}
[data-theme='dark'] .menu {
background-image: radial-gradient(rgba(255, 255, 255, 0.04) 1px, transparent 1px);
background-size: 24px 24px;
}
.menu__link {
font-size: 14px;
padding: 0.22rem 0.75rem !important;
border-radius: 4px;
}
.menu__link--active {
font-weight: 600 !important;
background-color: rgba(46, 133, 85, 0.08) !important;
}
[data-theme='dark'] .menu__link--active {
background-color: rgba(37, 194, 160, 0.1) !important;
}
/* ─── Sidebar collapse arrows — uniform size & alignment ────── */
/* 1. Categories WITHOUT a link prop:
The button itself holds the text + ::after arrow.
Make it flex so the arrow never wraps to a new line. */
.menu__link--sublist-caret {
display: flex !important;
align-items: center !important;
justify-content: space-between !important;
gap: 0.5rem;
padding-right: 0.625rem !important;
}
.menu__link--sublist-caret::after {
content: '' !important;
display: block !important;
flex-shrink: 0 !important;
width: 1.25rem !important;
height: 1.25rem !important;
min-width: 1.25rem !important;
background: var(--ifm-menu-link-sublist-icon) center / 1.25rem 1.25rem no-repeat !important;
margin: 0 !important;
}
/* 2. Categories WITH a link prop:
A separate <a> + <button class="menu__caret"> sit in a flex row. */
.menu__list-item-collapsible {
display: flex !important;
align-items: center !important;
border-radius: 4px;
}
.menu__list-item-collapsible > .menu__link {
flex: 1 1 auto;
min-width: 0; /* allow text to shrink/wrap without pushing arrow */
}
.menu__caret {
flex-shrink: 0 !important;
width: 2rem !important;
height: 2rem !important;
padding: 0 !important;
display: flex !important;
align-items: center !important;
justify-content: center !important;
}
.menu__caret::before {
content: '' !important;
display: block !important;
width: 1.25rem !important;
height: 1.25rem !important;
min-width: 1.25rem !important;
background: var(--ifm-menu-link-sublist-icon) center / 1.25rem 1.25rem no-repeat !important;
}
/* Top-level sidebar category labels */
.theme-doc-sidebar-item-category-level-1
> .menu__list-item-collapsible
> .menu__link:first-of-type {
color: #111827;
font-weight: 600 !important;
font-size: 14px;
}
[data-theme='dark']
.theme-doc-sidebar-item-category-level-1
> .menu__list-item-collapsible
> .menu__link:first-of-type {
color: #f9fafb;
font-weight: 600 !important;
font-size: 14px;
}
.theme-doc-sidebar-item-category-level-1 {
margin-bottom: 0.25rem !important;
}
/* Consistent row height for all top-level items */
.theme-doc-sidebar-item-category-level-1 > .menu__list-item-collapsible,
.theme-doc-sidebar-item-category-level-1 > .menu__list-item-collapsible--active {
min-height: 2.5rem;
padding: 0 0.25rem !important;
background-color: transparent !important;
border-radius: 6px;
}
.theme-doc-sidebar-item-category-level-1 > .menu__list-item-collapsible:hover {
background-color: var(--ifm-menu-color-background-hover) !important;
}
/* Suppress the auto active-state background so all items look the same height */
.menu__list-item-collapsible--active {
background-color: transparent !important;
}
.menu__list-item-collapsible--active:hover {
background-color: var(--ifm-menu-color-background-hover) !important;
}
/* Nested menu list */
.menu__list .menu__list {
padding-left: 4px !important;
}
/* Nested sidebar border line */
[class*='sidebar-item-'] ul:not(:first-child) {
margin-left: 16px !important;
border-left: 1px solid #e5e7eb;
}
[data-theme='dark'] [class*='sidebar-item-'] ul:not(:first-child) {
border-left-color: #2d3748;
}
/* ─── Sidebar link colour & hover fixes ─────────────────────── */
/* Doc links: ensure a visible colour in both themes */
.menu__link {
color: var(--ifm-menu-color);
}
.menu__link:hover {
color: var(--ifm-color-primary);
background-color: var(--ifm-menu-color-background-hover);
text-decoration: none;
}
/* External / type:"link" items — subtle right-arrow hint */
.menu__list-item--collapsed ~ .menu__link[href^="http"],
.menu__link[href^="http"] {
color: var(--ifm-menu-color);
}
[data-theme='dark'] .menu__link {
color: #9ca3af;
}
[data-theme='dark'] .menu__link:hover {
color: var(--ifm-color-primary);
}
/* =========================================
TABLE OF CONTENTS
========================================= */
.table-of-contents {
font-size: 13px;
}
.table-of-contents__link {
color: #6b7280;
line-height: 1.5;
}
.table-of-contents__link:hover {
color: var(--ifm-color-primary);
}
.table-of-contents__link--active {
font-weight: 600;
color: var(--ifm-color-primary);
}
[data-theme='dark'] .table-of-contents__link {
color: #9ca3af;
}
/* =========================================
IMAGES & MEDIA
========================================= */
img,
video {
border-radius: var(--ifm-border-radius);
}
/* Prevent video iframes (Loom, YouTube, etc.) from overflowing the
content column and overlapping the right-hand TOC sidebar. */
.theme-doc-markdown iframe {
max-width: 100%;
}
/* =========================================
FOOTER
========================================= */
.footer {
background: #0d1117;
}
.footer__copyright {
font-size: 13px;
color: #6b7280;
}
.footer__link-item {
color: #8b949e;
font-size: 14px;
}
.footer__link-item:hover {
text-decoration: none;
color: #c9d1d9;
}
.footer__title {
color: #f0f6fc;
font-size: 14px;
font-weight: 600;
}
.footer__bottom {
margin: 40px auto 10px auto;
}
/* =========================================
AVATAR
========================================= */
.avatar__name > a {
color: inherit;
}
/* =========================================
NAVBAR ICON LINKS (GitHub, Discord)
========================================= */
.header-github-link,
.header-discord-link {
display: flex;
align-items: center;
}
.header-github-link:hover,
.header-discord-link:hover {
opacity: 0.65;
}
/* Hide text label, show only the icon */
.header-github-link::before,
.header-discord-link::before {
content: '';
width: 22px;
height: 22px;
display: flex;
}
.header-github-link::before {
background: url("data:image/svg+xml,%3Csvg viewBox='0 0 24 24' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath d='M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12'/%3E%3C/svg%3E")
no-repeat center / contain;
}
[data-theme='dark'] .header-github-link::before {
background: url("data:image/svg+xml,%3Csvg viewBox='0 0 24 24' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath fill='white' d='M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12'/%3E%3C/svg%3E")
no-repeat center / contain;
}
/* Discord icon */
.header-discord-link::before {
background: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24'%3E%3Cpath fill='%23111' d='M20.317 4.37a19.791 19.791 0 0 0-4.885-1.515.074.074 0 0 0-.079.037c-.21.375-.444.864-.608 1.25a18.27 18.27 0 0 0-5.487 0 12.64 12.64 0 0 0-.617-1.25.077.077 0 0 0-.079-.037A19.736 19.736 0 0 0 3.677 4.37a.07.07 0 0 0-.032.027C.533 9.046-.32 13.58.099 18.057a.082.082 0 0 0 .031.057 19.9 19.9 0 0 0 5.993 3.03.078.078 0 0 0 .084-.028c.462-.63.874-1.295 1.226-1.994a.076.076 0 0 0-.041-.106 13.107 13.107 0 0 1-1.872-.892.077.077 0 0 1-.008-.128 10.2 10.2 0 0 0 .372-.292.074.074 0 0 1 .077-.01c3.928 1.793 8.18 1.793 12.062 0a.074.074 0 0 1 .078.01c.12.098.246.198.373.292a.077.077 0 0 1-.006.127 12.299 12.299 0 0 1-1.873.892.077.077 0 0 0-.041.107c.36.698.772 1.362 1.225 1.993a.076.076 0 0 0 .084.028 19.839 19.839 0 0 0 6.002-3.03.077.077 0 0 0 .032-.054c.5-5.177-.838-9.674-3.549-13.66a.061.061 0 0 0-.031-.03zM8.02 15.33c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.956-2.419 2.157-2.419 1.21 0 2.176 1.096 2.157 2.42 0 1.333-.956 2.418-2.157 2.418zm7.975 0c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.955-2.419 2.157-2.419 1.21 0 2.176 1.096 2.157 2.42 0 1.333-.946 2.418-2.157 2.418z'/%3E%3C/svg%3E")
no-repeat center / contain;
}
[data-theme='dark'] .header-discord-link::before {
background: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24'%3E%3Cpath fill='white' d='M20.317 4.37a19.791 19.791 0 0 0-4.885-1.515.074.074 0 0 0-.079.037c-.21.375-.444.864-.608 1.25a18.27 18.27 0 0 0-5.487 0 12.64 12.64 0 0 0-.617-1.25.077.077 0 0 0-.079-.037A19.736 19.736 0 0 0 3.677 4.37a.07.07 0 0 0-.032.027C.533 9.046-.32 13.58.099 18.057a.082.082 0 0 0 .031.057 19.9 19.9 0 0 0 5.993 3.03.078.078 0 0 0 .084-.028c.462-.63.874-1.295 1.226-1.994a.076.076 0 0 0-.041-.106 13.107 13.107 0 0 1-1.872-.892.077.077 0 0 1-.008-.128 10.2 10.2 0 0 0 .372-.292.074.074 0 0 1 .077-.01c3.928 1.793 8.18 1.793 12.062 0a.074.074 0 0 1 .078.01c.12.098.246.198.373.292a.077.077 0 0 1-.006.127 12.299 12.299 0 0 1-1.873.892.077.077 0 0 0-.041.107c.36.698.772 1.362 1.225 1.993a.076.076 0 0 0 .084.028 19.839 19.839 0 0 0 6.002-3.03.077.077 0 0 0 .032-.054c.5-5.177-.838-9.674-3.549-13.66a.061.061 0 0 0-.031-.03zM8.02 15.33c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.956-2.419 2.157-2.419 1.21 0 2.176 1.096 2.157 2.42 0 1.333-.956 2.418-2.157 2.418zm7.975 0c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.955-2.419 2.157-2.419 1.21 0 2.176 1.096 2.157 2.42 0 1.333-.946 2.418-2.157 2.418z'/%3E%3C/svg%3E")
no-repeat center / contain;
}
/* =========================================
SEARCH BAR — hide from navbar (it lives in sidebar)
========================================= */
[class^='navbarSearchContainer'],
[class*='navbarSearchContainer'] {
display: none !important;
}
/* =========================================
INKEEP AI CHAT BUTTON — minimal floating style
========================================= */
.ikp-chat-button__button {
width: 42px !important;
height: 42px !important;
border-radius: 50% !important;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.18) !important;
margin-right: 1rem !important;
margin-bottom: 1rem !important;
}
/* =========================================
NAVBAR — mobile fixes
========================================= */
/* On small screens: hide icon links (GitHub, Discord) and text links
from the right section. Keep only the color mode toggle.
All nav items remain accessible inside the hamburger drawer. */
@media (max-width: 996px) {
/* Hide GitHub and Discord icon links */
.header-github-link,
.header-discord-link {
display: none !important;
}
/* Hide anchor links in the right section (e.g. Cost Map) */
.navbar__items--right > a.navbar__link {
display: none !important;
}
/* Keep the right section visible (color mode toggle lives here) */
.navbar__items--right {
display: flex !important;
flex: 0 0 auto;
}
/* Force navbar to stay on a single row, no wrapping */
.navbar__inner {
flex-wrap: nowrap !important;
align-items: center;
}
/* Keep hamburger and logo side by side (not stacked) */
.navbar__brand-col {
display: flex !important;
align-items: center;
gap: 0.5rem;
}
/* Ensure hamburger is always shown */
.navbar__toggle {
display: flex !important;
}
}
/* Hide theme toggle in mobile sidebar (slide-out drawer) */
.navbar-sidebar .navbar-sidebar__brand [class*="toggle"] {
display: none !important;
}
/* =========================================
BASH / SHELL CODE BLOCKS
========================================= */
/* Subtle grey bash/shell blocks — light mode */
[data-theme='light'] [class*='language-bash'] .prism-code,
[data-theme='light'] [class*='language-sh'] .prism-code,
[data-theme='light'] [class*='language-shell'] .prism-code,
[data-theme='light'] [class*='language-zsh'] .prism-code {
background-color: #f3f4f6 !important;
border: 1px solid #d1d5db !important;
border-left: 3px solid #6b7280 !important;
color: #1f2937 !important;
}
[data-theme='light'] [class*='language-bash'] .prism-code span,
[data-theme='light'] [class*='language-sh'] .prism-code span,
[data-theme='light'] [class*='language-shell'] .prism-code span,
[data-theme='light'] [class*='language-zsh'] .prism-code span {
color: #1f2937 !important;
}
/* Dark mode — slightly lighter than the default near-black */
[data-theme='dark'] [class*='language-bash'] .prism-code,
[data-theme='dark'] [class*='language-sh'] .prism-code,
[data-theme='dark'] [class*='language-shell'] .prism-code,
[data-theme='dark'] [class*='language-zsh'] .prism-code {
background-color: #1c2130 !important;
border: 1px solid #2d3748 !important;
border-left: 3px solid #4b5563 !important;
}
/* "$ " terminal prompt prefix on bash title bar */
[class*='language-bash'] [class*='codeBlockTitle']::before,
[class*='language-sh'] [class*='codeBlockTitle']::before,
[class*='language-shell'] [class*='codeBlockTitle']::before,
[class*='language-zsh'] [class*='codeBlockTitle']::before {
content: '$ ';
color: #6b7280;
font-weight: 700;
}
/* =========================================
LOGO (existing logo toggle logic)
========================================= */
.levo-logo-container {
position: relative;
}
@ -42,7 +759,6 @@
height: auto !important;
}
/* Show light logo by default, hide dark logo */
.levo-logo-dark {
display: none !important;
}
@ -51,7 +767,6 @@
display: block !important;
}
/* In dark mode, hide light logo and show dark logo */
[data-theme='dark'] .levo-logo-light {
display: none !important;
}
@ -59,3 +774,23 @@
[data-theme='dark'] .levo-logo-dark {
display: block !important;
}
/* =========================================
DOC CONTENT — wider reading column
========================================= */
/* On large screens the default col--7 grid leaves too much dead space.
Expand the content column; the TOC column fills the remainder via flex. */
@media (min-width: 1200px) {
.docItemCol {
--ifm-col-width: calc(8 / 12 * 100%) !important;
max-width: calc(8 / 12 * 100%) !important;
}
}
@media (min-width: 1600px) {
.docItemCol {
--ifm-col-width: calc(9 / 12 * 100%) !important;
max-width: calc(9 / 12 * 100%) !important;
}
}

29
docs/my-website/src/theme/DocSidebar/index.js Normal file
View file

@ -0,0 +1,29 @@
import React from 'react';
import { useWindowSize } from '@docusaurus/theme-common';
import DocSidebarDesktop from '@theme/DocSidebar/Desktop';
import DocSidebarMobile from '@theme/DocSidebar/Mobile';
import SearchBar from '@theme/SearchBar';
import styles from './styles.module.css';
export default function DocSidebar(props) {
const windowSize = useWindowSize();
const shouldRenderSidebarDesktop =
windowSize === 'desktop' || windowSize === 'ssr';
const shouldRenderSidebarMobile = windowSize === 'mobile';
return (
<>
{shouldRenderSidebarDesktop && (
<div className={styles.sidebarContainer}>
<div className={styles.searchBarContainer}>
<SearchBar />
</div>
<DocSidebarDesktop {...props} />
</div>
)}
{shouldRenderSidebarMobile && <DocSidebarMobile {...props} />}
</>
);
}

30
docs/my-website/src/theme/DocSidebar/styles.module.css Normal file
View file

@ -0,0 +1,30 @@
.sidebarContainer {
display: flex;
flex-direction: column;
height: 100%;
max-height: 100vh;
}
/* Make the sidebar content scroll inside the container */
.sidebarContainer > aside,
.sidebarContainer > div:last-child {
flex: 1;
min-height: 0;
overflow-y: auto;
}
.searchBarContainer {
flex-shrink: 0;
padding: 0.75rem 1rem;
padding-top: calc(var(--ifm-navbar-height) + 0.75rem);
border-bottom: 1px solid var(--ifm-toc-border-color);
}
.searchBarContainer > div {
width: 100%;
}
/* Remove the default top padding from the sidebar content that follows */
.searchBarContainer + div {
padding-top: 0 !important;
}

71
docs/my-website/src/theme/Navbar/Content/index.js Normal file
View file

@ -0,0 +1,71 @@
import React from 'react';
import {
useThemeConfig,
ErrorCauseBoundary,
} from '@docusaurus/theme-common';
import {
splitNavbarItems,
useNavbarMobileSidebar,
} from '@docusaurus/theme-common/internal';
import NavbarItem from '@theme/NavbarItem';
import NavbarColorModeToggle from '@theme/Navbar/ColorModeToggle';
import NavbarMobileSidebarToggle from '@theme/Navbar/MobileSidebar/Toggle';
import NavbarLogo from '@theme/Navbar/Logo';
import NavbarSearch from '@theme/Navbar/Search';
import SearchBar from '@theme/SearchBar';
function useNavbarItems() {
return useThemeConfig().navbar.items;
}
function NavbarItems({ items }) {
return (
<>
{items.map((item, i) => (
<ErrorCauseBoundary
key={i}
onError={(error) =>
new Error(
`A theme navbar item failed to render.\n${JSON.stringify(item, null, 2)}`,
{ cause: error },
)
}>
<NavbarItem {...item} />
</ErrorCauseBoundary>
))}
</>
);
}
export default function NavbarContent() {
const mobileSidebar = useNavbarMobileSidebar();
const items = useNavbarItems();
const [leftItems, rightItems] = splitNavbarItems(items);
const searchBarItem = items.find((item) => item.type === 'search');
return (
<div className="navbar__inner">
{/* Left: Logo only */}
<div className="navbar__brand-col">
{!mobileSidebar.disabled && <NavbarMobileSidebarToggle />}
<NavbarLogo />
</div>
{/* Center: nav links */}
<div className="navbar__center-col">
<NavbarItems items={leftItems} />
</div>
{/* Right: icons + color toggle */}
<div className="navbar__right-col">
<NavbarItems items={rightItems} />
<NavbarColorModeToggle />
{!searchBarItem && (
<NavbarSearch>
<SearchBar />
</NavbarSearch>
)}
</div>
</div>
);
}

36
docs/my-website/src/theme/TOC/index.js Normal file
View file

@ -0,0 +1,36 @@
import React from 'react';
import clsx from 'clsx';
import TOCItems from '@theme/TOCItems';
import Link from '@docusaurus/Link';
import styles from './styles.module.css';
const LINK_CLASS_NAME = 'table-of-contents__link toc-highlight';
const LINK_ACTIVE_CLASS_NAME = 'table-of-contents__link--active';
export default function TOC({ className, ...props }) {
return (
<div className={clsx(styles.tableOfContents, className)}>
{/* Scrollable TOC items */}
<div className={clsx(styles.tocItemsContainer, 'thin-scrollbar')}>
<TOCItems
{...props}
linkClassName={LINK_CLASS_NAME}
linkActiveClassName={LINK_ACTIVE_CLASS_NAME}
/>
</div>
{/* Enterprise promo card pinned at the bottom */}
<div className={styles.promoCard}>
<div className={styles.promoEmoji}>🚅</div>
<div className={styles.promoHeading}>LiteLLM Enterprise</div>
<div className={styles.promoDescription}>
SSO/SAML, audit logs, spend tracking, multi-team management, and
guardrails — built for production.
</div>
<Link to="/docs/enterprise" className={styles.promoButton}>
Learn more →
</Link>
</div>
</div>
);
}

74
docs/my-website/src/theme/TOC/styles.module.css Normal file
View file

@ -0,0 +1,74 @@
.tableOfContents {
max-height: calc(100vh - (var(--ifm-navbar-height) + 2rem));
position: sticky;
top: calc(var(--ifm-navbar-height) + 1rem);
display: flex;
flex-direction: column;
}
.tocItemsContainer {
overflow-y: auto;
flex: 1 1 auto;
min-height: 0;
}
@media (max-width: 996px) {
.tableOfContents {
display: none;
}
}
/* =========================================
Enterprise Promo Card
========================================= */
.promoCard {
flex-shrink: 0;
margin-top: 1.5rem;
padding: 0.875rem 1rem;
border-radius: 6px;
border: 1px solid rgba(46, 133, 85, 0.2);
background-color: rgba(46, 133, 85, 0.05);
}
[data-theme='dark'] .promoCard {
border-color: rgba(37, 194, 160, 0.2);
background-color: rgba(37, 194, 160, 0.05);
}
.promoEmoji {
font-size: 18px;
margin-bottom: 0.4rem;
line-height: 1;
}
.promoHeading {
font-size: 13px;
font-weight: 600;
margin-bottom: 0.35rem;
color: var(--ifm-heading-color);
}
.promoDescription {
font-size: 11.5px;
line-height: 1.55;
margin-bottom: 0.75rem;
color: var(--ifm-color-emphasis-700);
}
.promoButton {
display: inline-block;
background-color: var(--ifm-color-primary);
color: #fff !important;
padding: 0.28rem 0.65rem;
border-radius: 4px;
font-size: 12px;
font-weight: 500;
text-decoration: none !important;
transition: background-color 0.15s ease;
}
.promoButton:hover {
background-color: var(--ifm-color-primary-dark);
color: #fff !important;
text-decoration: none !important;
}

21
litellm/integrations/opentelemetry.py
View file

@ -11,7 +11,7 @@ from litellm.integrations._types.open_inference import (
)
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.secret_managers.main import get_secret_bool
from litellm.secret_managers.main import get_secret_bool, str_to_bool
from litellm.types.services import ServiceLoggerPayload
from litellm.types.utils import (
ChatCompletionMessageToolCall,
@ -68,6 +68,7 @@ class OpenTelemetryConfig:
service_name: Optional[str] = None
deployment_environment: Optional[str] = None
model_id: Optional[str] = None
ignore_context_propagation: Optional[bool] = None
def __post_init__(self) -> None:
# If endpoint is specified but exporter is still the default "console",
@ -89,6 +90,10 @@ class OpenTelemetryConfig:
)
if not self.model_id:
self.model_id = os.getenv("OTEL_MODEL_ID", self.service_name)
if self.ignore_context_propagation is None:
self.ignore_context_propagation = str_to_bool(
os.getenv("OTEL_IGNORE_CONTEXT_PROPAGATION")
)
@classmethod
def from_env(cls):
@ -710,12 +715,7 @@ class OpenTelemetry(CustomLogger):
)
ctx, parent_span = self._get_span_context(kwargs)
# CRITICAL FIX: For langfuse_otel, ALWAYS create primary spans
# Don't use parent spans from other providers as they cause trace corruption
is_langfuse_otel = (
hasattr(self, "callback_name") and self.callback_name == "langfuse_otel"
)
if is_langfuse_otel:
if self.config.ignore_context_propagation:
parent_span = None # Ignore parent spans from other providers
ctx = None
@ -1256,12 +1256,7 @@ class OpenTelemetry(CustomLogger):
)
_parent_context, parent_otel_span = self._get_span_context(kwargs)
# CRITICAL FIX: For langfuse_otel, ALWAYS create primary spans
# Don't use parent spans from other providers as they cause trace corruption
is_langfuse_otel = (
hasattr(self, "callback_name") and self.callback_name == "langfuse_otel"
)
if is_langfuse_otel:
if self.config.ignore_context_propagation:
parent_otel_span = None # Ignore parent spans from other providers
_parent_context = None

5
litellm/litellm_core_utils/litellm_logging.py
View file

@ -2956,7 +2956,10 @@ class Logging(LiteLLMLoggingBaseClass):
callback_func=callback,
)
if (
isinstance(callback, CustomLogger) and is_sync_request
isinstance(callback, CustomLogger)
and is_sync_request
and self.call_type
!= CallTypes.pass_through.value
): # custom logger class
callback.log_failure_event(
start_time=start_time,

6
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
View file

@ -961,6 +961,8 @@ async def pass_through_request( # noqa: PLR0915
if kwargs:
for key, value in kwargs.items():
request_payload[key] = value
if logging_obj is not None:
request_payload["litellm_logging_obj"] = logging_obj
if (
"model" not in request_payload
@ -1703,6 +1705,8 @@ async def websocket_passthrough_request( # noqa: PLR0915
if kwargs:
for key, value in kwargs.items():
request_payload[key] = value
if logging_obj is not None:
request_payload["litellm_logging_obj"] = logging_obj
# Log the connection failure using the same pattern as HTTP
await proxy_logging_obj.post_call_failure_hook(
@ -1729,6 +1733,8 @@ async def websocket_passthrough_request( # noqa: PLR0915
if kwargs:
for key, value in kwargs.items():
request_payload[key] = value
if logging_obj is not None:
request_payload["litellm_logging_obj"] = logging_obj
# Log the unexpected error using the same pattern as HTTP
await proxy_logging_obj.post_call_failure_hook(

86
litellm/proxy/route_llm_request.py
View file

@ -1,3 +1,4 @@
import asyncio
from typing import TYPE_CHECKING, Any, Literal, Optional
from fastapi import HTTPException, status
@ -123,30 +124,99 @@ def get_team_id_from_data(data: dict) -> Optional[str]:
return None
def add_shared_session_to_data(data: dict) -> None:
_shared_session_lock: Optional[asyncio.Lock] = None
def _get_shared_session_lock() -> asyncio.Lock:
"""Lazily create the shared session lock (must be called within a running event loop).
WARNING: Do not reset _shared_session_lock to None while any coroutine may be
executing the session-recovery path; doing so breaks the double-checked locking
guarantee and can cause duplicate session creation.
"""
global _shared_session_lock
if _shared_session_lock is None:
_shared_session_lock = asyncio.Lock()
return _shared_session_lock
async def add_shared_session_to_data(data: dict) -> None:
"""
Add shared aiohttp session for connection reuse (prevents cold starts).
If the session was closed (e.g. due to network interruption or idle timeout),
automatically recreates it so connection pooling is restored.
Uses an asyncio.Lock to prevent race conditions where multiple concurrent
requests could each create a new session, leaking intermediate ones.
Silently continues without session reuse if import fails or session is unavailable.
Args:
data: Dictionary to add the shared session to
"""
try:
import litellm.proxy.proxy_server as proxy_server
from litellm._logging import verbose_proxy_logger
from litellm.proxy.proxy_server import shared_aiohttp_session
if shared_aiohttp_session is not None and not shared_aiohttp_session.closed:
data["shared_session"] = shared_aiohttp_session
session = proxy_server.shared_aiohttp_session
if session is not None and not session.closed:
data["shared_session"] = session
verbose_proxy_logger.info(
f"SESSION REUSE: Attached shared aiohttp session to request (ID: {id(shared_aiohttp_session)})"
f"SESSION REUSE: Attached shared aiohttp session to request (ID: {id(session)})"
)
elif session is not None and session.closed:
# Session was created at startup but has since closed — recreate it
# Use lock to prevent concurrent recreation (avoids session/connector leak)
lock = _get_shared_session_lock()
async with lock:
# Double-check under lock — another coroutine may have already recreated it
session = proxy_server.shared_aiohttp_session
if session is not None and not session.closed:
data["shared_session"] = session
return
# session could be None here (if another coroutine set it to None)
# or closed — either way we need to recreate
if session is not None:
verbose_proxy_logger.warning(
f"SESSION REUSE: Shared aiohttp session is closed (ID: {id(session)}), recreating..."
)
else:
verbose_proxy_logger.warning(
"SESSION REUSE: Shared aiohttp session is None after re-check, recreating..."
)
try:
new_session = (
await proxy_server._initialize_shared_aiohttp_session()
)
except Exception:
verbose_proxy_logger.exception(
"SESSION REUSE: Exception during shared session recreation"
)
new_session = None
if new_session is not None:
proxy_server.shared_aiohttp_session = new_session
data["shared_session"] = new_session
else:
verbose_proxy_logger.info(
"SESSION REUSE: Failed to recreate shared session, continuing without session reuse"
)
else:
verbose_proxy_logger.info(
"SESSION REUSE: No shared session available for this request"
)
except Exception:
# Silently continue without session reuse if import fails or session unavailable
pass
# Continue without session reuse — this outer handler covers import failures
# and other unexpected errors to avoid breaking the request path.
# Inner recovery logic has its own specific exception handling.
try:
from litellm._logging import verbose_proxy_logger
verbose_proxy_logger.debug(
"SESSION REUSE: Unexpected error in session setup, continuing without reuse",
exc_info=True,
)
except Exception:
pass
async def route_request( # noqa: PLR0915 - Complex routing function, refactoring tracked separately
@ -248,7 +318,7 @@ async def route_request( # noqa: PLR0915 - Complex routing function, refactorin
"""
Common helper to route the request
"""
add_shared_session_to_data(data)
await add_shared_session_to_data(data)
team_id = get_team_id_from_data(data)
router_model_names = llm_router.model_names if llm_router is not None else []

28
litellm/proxy/utils.py
View file

@ -1744,6 +1744,9 @@ class ProxyLogging:
original_exception=original_exception,
)
# Remove before callbacks iterate — not serialisable
request_data.pop("litellm_logging_obj", None)
# Track the first HTTPException returned or raised by any callback
transformed_exception: Optional[HTTPException] = None
@ -1864,7 +1867,7 @@ class ProxyLogging:
for k, v in request_data.items():
if k in litellm_param_keys:
_litellm_params[k] = v
elif k != "model" and k != "user":
elif k not in ("model", "user", "litellm_logging_obj"):
_optional_params[k] = v
litellm_logging_obj.update_environment_variables(
@ -1880,15 +1883,23 @@ class ProxyLogging:
):
input = request_data["messages"]
litellm_logging_obj.model_call_details["messages"] = input
litellm_logging_obj.call_type = CallTypes.acompletion.value
if litellm_logging_obj.call_type != CallTypes.pass_through.value:
litellm_logging_obj.call_type = CallTypes.acompletion.value
elif "prompt" in request_data and isinstance(request_data["prompt"], str):
input = request_data["prompt"]
litellm_logging_obj.model_call_details["prompt"] = input
litellm_logging_obj.call_type = CallTypes.atext_completion.value
if litellm_logging_obj.call_type != CallTypes.pass_through.value:
litellm_logging_obj.call_type = CallTypes.atext_completion.value
elif "input" in request_data and isinstance(request_data["input"], list):
input = request_data["input"]
litellm_logging_obj.model_call_details["input"] = input
litellm_logging_obj.call_type = CallTypes.aembedding.value
if litellm_logging_obj.call_type != CallTypes.pass_through.value:
litellm_logging_obj.call_type = CallTypes.aembedding.value
# Pass-through endpoints are logged via the callback loop's
# async_post_call_failure_hook — skip pre_call and failure handlers.
if litellm_logging_obj.call_type == CallTypes.pass_through.value:
return
litellm_logging_obj.pre_call(
input=input,
api_key="",
@ -1924,6 +1935,7 @@ class ProxyLogging:
4. /files
"""
from litellm.proxy.proxy_server import llm_router
from litellm.types.guardrails import GuardrailEventHooks
guardrail_callbacks: List[CustomGuardrail] = []
@ -1946,12 +1958,18 @@ class ProxyLogging:
############## Handle Guardrails ########################################
#############################################################################
# Merge model-level guardrails before checking which guardrails to run
guardrail_data = _check_and_merge_model_level_guardrails(
data=data, llm_router=llm_router
)
for callback in guardrail_callbacks:
# Main - V2 Guardrails implementation
if (
callback.should_run_guardrail(
data=data, event_type=GuardrailEventHooks.post_call
data=guardrail_data,
event_type=GuardrailEventHooks.post_call,
)
is not True
):

17
poetry.lock generated
View file

@ -4461,6 +4461,21 @@ sql-other = ["SQLAlchemy (>=2.0.0)", "adbc-driver-postgresql (>=0.8.0)", "adbc-d
test = ["hypothesis (>=6.46.1)", "pytest (>=7.3.2)", "pytest-xdist (>=2.2.0)"]
xml = ["lxml (>=4.9.2)"]
[[package]]
name = "parameterized"
version = "0.9.0"
description = "Parameterized testing with any Python test framework"
optional = false
python-versions = ">=3.7"
groups = ["dev"]
files = [
{file = "parameterized-0.9.0-py2.py3-none-any.whl", hash = "sha256:4e0758e3d41bea3bbd05ec14fc2c24736723f243b28d702081aef438c9372b1b"},
{file = "parameterized-0.9.0.tar.gz", hash = "sha256:7fc905272cefa4f364c1a3429cbbe9c0f98b793988efb5bf90aac80f08db09b1"},
]
[package.extras]
dev = ["jinja2"]
[[package]]
name = "pathspec"
version = "0.12.1"
@ -8003,4 +8018,4 @@ utils = ["numpydoc"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.9,<4.0"
content-hash = "22cdc8096e0c8296827734393f5ab6e66088f397b5295caa1d277466d1fde1e8"
content-hash = "0002021a7733b370a9855b26198e8e6dc49a62b67f1eface6f2fbe406ff3c3ac"

1
pyproject.toml
View file

@ -167,6 +167,7 @@ langfuse = "^2.45.0"
fastapi-offline = "^1.7.3"
fakeredis = "^2.27.1"
pytest-rerunfailures = "^14.0"
parameterized = "^0.9.0"
[tool.poetry.group.proxy-dev.dependencies]
prisma = "0.11.0"

131
tests/proxy_unit_tests/test_proxy_utils.py
View file

@ -2385,3 +2385,134 @@ async def test_during_call_hook_parallel_execution_with_error():
assert "Guardrail violation detected!" in str(exc_info.value)
finally:
litellm.callbacks = original_callbacks
@pytest.mark.asyncio
async def test_handle_logging_proxy_only_error_preserves_pass_through_call_type():
"""Ensure _handle_logging_proxy_only_error does not overwrite call_type
when the logging object is already marked as pass_through_endpoint.
"""
from litellm.caching.caching import DualCache
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.proxy.utils import ProxyLogging
from litellm.types.utils import CallTypes
logging_obj = Logging(
model="unknown",
messages=[{"role": "user", "content": "test"}],
stream=False,
call_type="pass_through_endpoint",
start_time=datetime.now(),
litellm_call_id="test-call-id",
function_id="test-function-id",
)
request_data = {
"litellm_logging_obj": logging_obj,
"messages": [{"role": "user", "content": "test"}],
"model": "claude-3-5-sonnet",
}
cache = DualCache()
proxy_logging = ProxyLogging(user_api_key_cache=cache)
with patch.object(logging_obj, "async_failure_handler", new_callable=AsyncMock):
with patch.object(logging_obj, "failure_handler"):
await proxy_logging._handle_logging_proxy_only_error(
request_data=request_data,
user_api_key_dict=UserAPIKeyAuth(
api_key="test_key", token="test_token"
),
original_exception=Exception("test error"),
)
assert logging_obj.call_type == CallTypes.pass_through.value
@pytest.mark.asyncio
async def test_litellm_logging_obj_excluded_from_optional_params():
"""Ensure litellm_logging_obj is excluded from _optional_params to prevent
circular references in model_call_details.
"""
from litellm.caching.caching import DualCache
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.proxy.utils import ProxyLogging
logging_obj = Logging(
model="unknown",
messages=[{"role": "user", "content": "test"}],
stream=False,
call_type="pass_through_endpoint",
start_time=datetime.now(),
litellm_call_id="test-call-id",
function_id="test-function-id",
)
request_data = {
"litellm_logging_obj": logging_obj,
"messages": [{"role": "user", "content": "test"}],
"model": "claude-3-5-sonnet",
}
cache = DualCache()
proxy_logging = ProxyLogging(user_api_key_cache=cache)
with patch.object(logging_obj, "async_failure_handler", new_callable=AsyncMock):
with patch.object(logging_obj, "failure_handler"):
await proxy_logging._handle_logging_proxy_only_error(
request_data=request_data,
user_api_key_dict=UserAPIKeyAuth(
api_key="test_key", token="test_token"
),
original_exception=Exception("test error"),
)
assert "litellm_logging_obj" not in logging_obj.model_call_details
@pytest.mark.asyncio
async def test_handle_logging_proxy_only_error_skips_handlers_for_pass_through():
"""Ensure _handle_logging_proxy_only_error skips async_failure_handler and
failure_handler for pass-through endpoint errors, so only
async_post_call_failure_hook fires (avoiding duplicate logs).
Regression test for duplicate Datadog/Arize logs on pass-through failures.
"""
from litellm.caching.caching import DualCache
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.proxy.utils import ProxyLogging
from litellm.types.utils import CallTypes
logging_obj = Logging(
model="unknown",
messages=[{"role": "user", "content": "test"}],
stream=False,
call_type="pass_through_endpoint",
start_time=datetime.now(),
litellm_call_id="test-call-id",
function_id="test-function-id",
)
cache = DualCache()
proxy_logging = ProxyLogging(user_api_key_cache=cache)
request_data = {
"litellm_logging_obj": logging_obj,
"messages": [{"role": "user", "content": "test"}],
"model": "claude-3-5-sonnet",
}
with patch.object(logging_obj, "async_failure_handler", new_callable=AsyncMock) as mock_async:
with patch.object(logging_obj, "failure_handler") as mock_sync:
await proxy_logging._handle_logging_proxy_only_error(
request_data=request_data,
user_api_key_dict=UserAPIKeyAuth(
api_key="test_key", token="test_token"
),
original_exception=Exception("test error"),
)
# Neither handler should fire for pass-through requests
mock_async.assert_not_called()
mock_sync.assert_not_called()
assert logging_obj.call_type == CallTypes.pass_through.value

131
tests/test_litellm/integrations/test_opentelemetry.py
View file

@ -3,7 +3,8 @@ import os
import sys
import time
import unittest
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from parameterized import parameterized
from unittest.mock import MagicMock, patch
# Adds the grandparent directory to sys.path to allow importing project modules
@ -425,6 +426,7 @@ class TestOpenTelemetry(unittest.TestCase):
) as mock_get_headers, patch.object(
otel, "_get_tracer_with_dynamic_headers"
) as mock_get_tracer:
# Test case 1: With dynamic headers
mock_get_headers.return_value = {
"arize-space-id": "test-space",
@ -2165,25 +2167,30 @@ class TestOpenTelemetrySemanticConventions138(unittest.TestCase):
mock_span.set_attribute.assert_any_call("gen_ai.operation.name", "chat")
def test_handle_failure_langfuse_otel_nulls_parent_span(self):
@parameterized.expand([("_handle_success",), ("_handle_failure",)])
def test_handle_success_failure_nulls_parent_span_if_ignore_context_propagation(
self, handle_method: str
):
"""
For langfuse_otel, _handle_failure should ignore parent spans from other providers
and create a root-level error span (symmetric with _handle_success).
If ignore_context_propagation is True, _handle_success should ignore any parent span
and create a root-level span. This could be useful for langfuse_otel where
_handle_success may ignore parent spans from other providers and create a root-level
span (symmetric with _handle_failure).
"""
span_exporter = InMemorySpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
otel = OpenTelemetry(
callback_name="langfuse_otel",
config=OpenTelemetryConfig(ignore_context_propagation=True),
tracer_provider=tracer_provider,
)
otel.tracer = tracer_provider.get_tracer("litellm")
other_tracer = tracer_provider.get_tracer("other_provider")
other_span = other_tracer.start_span("other_provider_span")
other_span = other_tracer.start_span("parent_span")
start = datetime.utcnow()
start = datetime.now(timezone.utc)
end = start + timedelta(seconds=1)
kwargs = {
@ -2202,23 +2209,33 @@ class TestOpenTelemetrySemanticConventions138(unittest.TestCase):
"exception": Exception("test error"),
}
otel._handle_failure(kwargs, None, start, end)
with patch.dict(os.environ, {"USE_OTEL_LITELLM_REQUEST_SPAN": "true"}):
if handle_method == "_handle_success":
otel._handle_success(kwargs, None, start, end)
elif handle_method == "_handle_failure":
otel._handle_failure(kwargs, None, start, end)
else:
self.fail(f"Invalid handle_method: {handle_method}")
other_span.end()
spans = span_exporter.get_finished_spans()
failure_spans = [s for s in spans if s.name != "other_provider_span"]
child_spans = [s for s in spans if s.name != "parent_span"]
child_span_ids = {s.context.span_id for s in child_spans if s.context}
self.assertTrue(failure_spans, "Expected at least one failure span")
for span in failure_spans:
self.assertIsNone(
span.parent,
f"langfuse_otel failure span should be a root span, but has parent: {span.parent}",
)
self.assertTrue(child_spans, "Expected at least one child span")
for span in child_spans:
assert (
span.parent is None or span.parent.span_id in child_span_ids
), f"if ignore_context_propagation is True, span should not have parent from other providers, but got parent: {span.parent}"
def test_handle_failure_non_langfuse_preserves_parent_span(self):
@parameterized.expand([("_handle_success",), ("_handle_failure",)])
def test_handle_success_failure_default_preserves_parent_span(
self, handle_method: str
):
"""
For non-langfuse_otel callbacks, _handle_failure should still use parent spans normally.
For default otel callbacks, _handle_success should use parent spans normally.
(symmetric with _handle_failure)
"""
span_exporter = InMemorySpanExporter()
tracer_provider = TracerProvider()
@ -2229,7 +2246,7 @@ class TestOpenTelemetrySemanticConventions138(unittest.TestCase):
parent_span = otel.tracer.start_span("parent_span")
start = datetime.utcnow()
start = datetime.now(timezone.utc)
end = start + timedelta(seconds=1)
kwargs = {
@ -2249,19 +2266,81 @@ class TestOpenTelemetrySemanticConventions138(unittest.TestCase):
}
with patch.dict(os.environ, {"USE_OTEL_LITELLM_REQUEST_SPAN": "true"}):
otel._handle_failure(kwargs, None, start, end)
if handle_method == "_handle_success":
otel._handle_success(kwargs, None, start, end)
elif handle_method == "_handle_failure":
otel._handle_failure(kwargs, None, start, end)
else:
self.fail(f"Invalid handle_method: {handle_method}")
parent_span.end()
spans = span_exporter.get_finished_spans()
child_spans = [s for s in spans if s.name != "parent_span"]
self.assertTrue(child_spans, "Expected at least one child failure span")
self.assertTrue(child_spans, "Expected at least one child span")
for span in child_spans:
self.assertIsNotNone(
span.parent,
"Non-langfuse_otel failure span should have a parent",
)
assert (
span.parent is not None
), f"By default parent span should be preserved, but got None parent for span: {span.name}"
@parameterized.expand([("_handle_success",), ("_handle_failure",)])
def test_handle_success_failure_with_context_propagation_preserves_parent_span(
self, handle_method: str
):
"""
For otel callbacks with context propagation enabled, _handle_success should
use parent spans normally. (symmetric with _handle_failure)
"""
span_exporter = InMemorySpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
otel = OpenTelemetry(
config=OpenTelemetryConfig(ignore_context_propagation=False),
tracer_provider=tracer_provider,
)
otel.tracer = tracer_provider.get_tracer("litellm")
parent_span = otel.tracer.start_span("parent_span")
start = datetime.now(timezone.utc)
end = start + timedelta(seconds=1)
kwargs = {
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}],
"optional_params": {},
"litellm_params": {
"custom_llm_provider": "openai",
"metadata": {"litellm_parent_otel_span": parent_span},
},
"standard_logging_object": {
"id": "test-id",
"call_type": "completion",
"metadata": {},
},
"exception": Exception("test error"),
}
with patch.dict(os.environ, {"USE_OTEL_LITELLM_REQUEST_SPAN": "true"}):
if handle_method == "_handle_success":
otel._handle_success(kwargs, None, start, end)
elif handle_method == "_handle_failure":
otel._handle_failure(kwargs, None, start, end)
else:
self.fail(f"Invalid handle_method: {handle_method}")
parent_span.end()
spans = span_exporter.get_finished_spans()
child_spans = [s for s in spans if s.name != "parent_span"]
self.assertTrue(child_spans, "Expected at least one child span")
for span in child_spans:
assert (
span.parent is not None
), f"If ignore_context_propagation is False, parent span should be preserved, but got None parent for span: {span.name}"
def test_handle_failure_hasattr_guard_on_parent_name(self):
"""
@ -2431,9 +2510,7 @@ class TestNoParentSpanDuplication(unittest.TestCase):
otel._handle_success(kwargs, response_obj, start, end)
spans = span_exporter.get_finished_spans()
proxy_spans = [
s for s in spans if s.name == LITELLM_PROXY_REQUEST_SPAN_NAME
]
proxy_spans = [s for s in spans if s.name == LITELLM_PROXY_REQUEST_SPAN_NAME]
self.assertEqual(len(proxy_spans), 1, "Should have exactly one proxy span")
proxy_attrs = proxy_spans[0].attributes or {}

65
tests/test_litellm/litellm_core_utils/test_litellm_logging.py
View file

@ -1969,3 +1969,68 @@ def test_function_setup_empty_metadata_falls_back_to_litellm_metadata():
assert metadata is not None
assert metadata.get("user_api_key_hash") == "sk-hashed-empty-test"
assert metadata.get("user_api_key_team_id") == "team-empty-test"
def test_failure_handler_skips_sync_callbacks_for_pass_through_requests(logging_obj):
"""Ensure sync failure callbacks are skipped for pass-through endpoint requests.
Regression test for duplicate Datadog/Arize logs on pass-through endpoint failures.
The async_failure_handler fires async_log_failure_event; the sync failure_handler
must NOT also fire log_failure_event for pass-through requests.
"""
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.utils import CallTypes
class DummyLogger(CustomLogger):
pass
logging_obj.call_type = CallTypes.pass_through.value
logging_obj.stream = False
logging_obj.model_call_details["litellm_params"] = {}
logging_obj.litellm_params = {}
dummy_logger = DummyLogger()
dummy_logger.log_failure_event = MagicMock()
with patch.object(
logging_obj,
"get_combined_callback_list",
return_value=[dummy_logger],
):
logging_obj.failure_handler(
exception=Exception("test error"),
traceback_exception="",
)
dummy_logger.log_failure_event.assert_not_called()
@pytest.mark.parametrize("call_type", ["completion", "acompletion"])
def test_failure_handler_runs_sync_callbacks_for_non_pass_through_requests(
logging_obj, call_type
):
"""Ensure sync failure callbacks still fire for normal (non-pass-through) requests."""
from litellm.integrations.custom_logger import CustomLogger
class DummyLogger(CustomLogger):
pass
logging_obj.call_type = call_type
logging_obj.stream = False
logging_obj.model_call_details["litellm_params"] = {}
logging_obj.litellm_params = {}
dummy_logger = DummyLogger()
dummy_logger.log_failure_event = MagicMock()
with patch.object(
logging_obj,
"get_combined_callback_list",
return_value=[dummy_logger],
):
logging_obj.failure_handler(
exception=Exception("test error"),
traceback_exception="",
)
dummy_logger.log_failure_event.assert_called_once()

182
tests/test_litellm/proxy/test_aiohttp_session_recovery.py Normal file
View file

@ -0,0 +1,182 @@
"""
Tests for shared aiohttp session auto-recovery.
When the shared session closes (e.g. network interruption, idle timeout),
add_shared_session_to_data should recreate it instead of permanently
falling back to per-request connections.
Fixes: https://github.com/BerriAI/litellm/issues/23806
"""
import asyncio
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@pytest.mark.asyncio
async def test_add_shared_session_attaches_open_session():
"""When the shared session is open, it should be attached to data."""
from litellm.proxy.route_llm_request import add_shared_session_to_data
mock_session = MagicMock()
mock_session.closed = False
with patch("litellm.proxy.proxy_server.shared_aiohttp_session", mock_session):
data = {}
await add_shared_session_to_data(data)
assert data["shared_session"] is mock_session
@pytest.mark.asyncio
async def test_add_shared_session_recreates_closed_session():
"""When the shared session is closed, it should be recreated."""
import litellm.proxy.route_llm_request as route_module
from litellm.proxy import proxy_server as proxy_server_module
from litellm.proxy.route_llm_request import add_shared_session_to_data
# Reset the module-level lock so each test uses the current event loop
route_module._shared_session_lock = None
closed_session = MagicMock()
closed_session.closed = True
new_session = MagicMock()
new_session.closed = False
with patch.object(
proxy_server_module,
"shared_aiohttp_session",
closed_session,
):
with patch.object(
proxy_server_module,
"_initialize_shared_aiohttp_session",
new_callable=AsyncMock,
return_value=new_session,
) as mock_init:
data = {}
await add_shared_session_to_data(data)
mock_init.assert_called_once()
assert data["shared_session"] is new_session
assert proxy_server_module.shared_aiohttp_session is new_session
@pytest.mark.asyncio
async def test_add_shared_session_handles_recreation_failure():
"""When recreation fails, data should not contain shared_session."""
import litellm.proxy.route_llm_request as route_module
from litellm.proxy import proxy_server as proxy_server_module
from litellm.proxy.route_llm_request import add_shared_session_to_data
# Reset the module-level lock so each test uses the current event loop
route_module._shared_session_lock = None
closed_session = MagicMock()
closed_session.closed = True
with patch.object(
proxy_server_module,
"shared_aiohttp_session",
closed_session,
):
with patch.object(
proxy_server_module,
"_initialize_shared_aiohttp_session",
new_callable=AsyncMock,
return_value=None,
):
data = {}
await add_shared_session_to_data(data)
assert "shared_session" not in data
@pytest.mark.asyncio
async def test_add_shared_session_handles_recreation_exception():
"""When _initialize_shared_aiohttp_session raises, data should not contain shared_session."""
import litellm.proxy.route_llm_request as route_module
from litellm.proxy import proxy_server as proxy_server_module
from litellm.proxy.route_llm_request import add_shared_session_to_data
# Reset the module-level lock so each test uses the current event loop
route_module._shared_session_lock = None
closed_session = MagicMock()
closed_session.closed = True
with patch.object(
proxy_server_module,
"shared_aiohttp_session",
closed_session,
):
with patch.object(
proxy_server_module,
"_initialize_shared_aiohttp_session",
new_callable=AsyncMock,
side_effect=RuntimeError("connection pool exhausted"),
):
data = {}
await add_shared_session_to_data(data)
# Should gracefully handle exception — no shared_session attached
assert "shared_session" not in data
@pytest.mark.asyncio
async def test_add_shared_session_no_session_available():
"""When no session was ever created, data should not contain shared_session."""
from litellm.proxy.route_llm_request import add_shared_session_to_data
with patch("litellm.proxy.proxy_server.shared_aiohttp_session", None):
data = {}
await add_shared_session_to_data(data)
assert "shared_session" not in data
@pytest.mark.asyncio
async def test_add_shared_session_concurrent_recreation_uses_lock():
"""When multiple coroutines detect a closed session concurrently,
only one should recreate it (double-checked locking via asyncio.Lock)."""
import litellm.proxy.route_llm_request as route_module
from litellm.proxy import proxy_server as proxy_server_module
from litellm.proxy.route_llm_request import add_shared_session_to_data
# Reset the module-level lock so each test is isolated
route_module._shared_session_lock = None
closed_session = MagicMock()
closed_session.closed = True
new_session = MagicMock()
new_session.closed = False
call_count = 0
async def mock_init():
nonlocal call_count
call_count += 1
# Simulate some async work
await asyncio.sleep(0.01)
return new_session
with patch.object(
proxy_server_module,
"shared_aiohttp_session",
closed_session,
):
with patch.object(
proxy_server_module,
"_initialize_shared_aiohttp_session",
new_callable=AsyncMock,
side_effect=mock_init,
):
# Launch 5 concurrent calls
results = [{} for _ in range(5)]
await asyncio.gather(*(add_shared_session_to_data(d) for d in results))
# Only 1 coroutine should have called _initialize (the rest see the
# re-checked session as open under the lock)
assert call_count == 1, f"Expected 1 init call, got {call_count}"
# All should have the new session
for d in results:
assert d.get("shared_session") is new_session

300
tests/test_litellm/proxy/test_model_level_guardrails.py Normal file
View file

@ -0,0 +1,300 @@
"""
Unit tests for model-level guardrails in post_call paths.
Tests verify that guardrails configured via litellm_params.guardrails on a
deployment are merged into request metadata and trigger execution for both
streaming and non-streaming post_call hooks.
"""
import os
import sys
import pytest
from unittest.mock import MagicMock, patch
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../..")))
from litellm.proxy.utils import (
_check_and_merge_model_level_guardrails,
_merge_guardrails_with_existing,
)
# ---------------------------------------------------------------------------
# Unit tests for _check_and_merge_model_level_guardrails
# ---------------------------------------------------------------------------
class TestCheckAndMergeModelLevelGuardrails:
"""Tests for the _check_and_merge_model_level_guardrails function."""
def test_merge_adds_model_guardrails_to_metadata(self):
"""Model-level guardrails are added to metadata.guardrails."""
data = {
"model": "gpt-4",
"metadata": {"model_info": {"id": "model-uuid-123"}},
}
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = ["openai-moderation"]
mock_router.get_deployment.return_value = mock_deployment
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
assert "openai-moderation" in result["metadata"]["guardrails"]
mock_router.get_deployment.assert_called_once_with(model_id="model-uuid-123")
def test_merge_combines_with_existing_guardrails(self):
"""Model-level guardrails merge with existing request guardrails."""
data = {
"model": "gpt-4",
"metadata": {
"model_info": {"id": "model-uuid-123"},
"guardrails": ["existing-guardrail"],
},
}
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = ["model-guardrail"]
mock_router.get_deployment.return_value = mock_deployment
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
assert "existing-guardrail" in result["metadata"]["guardrails"]
assert "model-guardrail" in result["metadata"]["guardrails"]
def test_no_duplicates_when_guardrail_already_in_metadata(self):
"""No duplicates when the same guardrail is in both model and request."""
data = {
"model": "gpt-4",
"metadata": {
"model_info": {"id": "model-uuid-123"},
"guardrails": ["openai-moderation"],
},
}
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = ["openai-moderation"]
mock_router.get_deployment.return_value = mock_deployment
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
assert result["metadata"]["guardrails"].count("openai-moderation") == 1
def test_returns_data_unchanged_when_no_router(self):
"""Returns data unchanged when llm_router is None."""
data = {"model": "gpt-4", "metadata": {}}
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=None
)
assert result is data
def test_returns_data_unchanged_when_no_model_info(self):
"""Returns data unchanged when metadata has no model_info."""
data = {"model": "gpt-4", "metadata": {}}
mock_router = MagicMock()
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
assert result is data
def test_returns_data_unchanged_when_deployment_has_no_guardrails(self):
"""Returns data unchanged when deployment has no guardrails configured."""
data = {
"model": "gpt-4",
"metadata": {"model_info": {"id": "model-uuid-123"}},
}
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = None
mock_router.get_deployment.return_value = mock_deployment
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
assert result is data
def test_returns_data_unchanged_when_deployment_not_found(self):
"""Returns data unchanged when router can't find the deployment."""
data = {
"model": "gpt-4",
"metadata": {"model_info": {"id": "nonexistent-id"}},
}
mock_router = MagicMock()
mock_router.get_deployment.return_value = None
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
assert result is data
def test_returns_new_data_dict(self):
"""Returns a new top-level dict (shallow copy), not the same object."""
data = {
"model": "gpt-4",
"metadata": {
"model_info": {"id": "model-uuid-123"},
"guardrails": ["existing"],
},
}
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = ["new-guardrail"]
mock_router.get_deployment.return_value = mock_deployment
result = _check_and_merge_model_level_guardrails(
data=data, llm_router=mock_router
)
# Result is a different top-level dict
assert result is not data
# Result should have the merged guardrail
assert "new-guardrail" in result["metadata"]["guardrails"]
assert "existing" in result["metadata"]["guardrails"]
# ---------------------------------------------------------------------------
# Integration test: post_call_success_hook with model-level guardrails
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_post_call_success_hook_runs_model_level_guardrail():
"""
Model-level guardrails configured on a deployment should execute in
post_call_success_hook (non-streaming path).
"""
from litellm.caching.caching import DualCache
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.utils import ProxyLogging
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.utils import Choices, Message, ModelResponse, Usage
class TestGuardrail(CustomGuardrail):
def __init__(self):
super().__init__(
guardrail_name="test-model-guardrail",
event_hook=GuardrailEventHooks.post_call,
)
self.was_called = False
async def async_post_call_success_hook(
self, data, user_api_key_dict, response
):
self.was_called = True
return response
guardrail = TestGuardrail()
# Mock router that returns a deployment with guardrails configured
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = ["test-model-guardrail"]
mock_router.get_deployment.return_value = mock_deployment
with patch("litellm.callbacks", [guardrail]), patch(
"litellm.proxy.proxy_server.llm_router", mock_router
):
proxy_logging = ProxyLogging(user_api_key_cache=DualCache())
data = {
"model": "gpt-4",
"metadata": {"model_info": {"id": "model-uuid-123"}},
}
response = ModelResponse(
id="resp-1",
choices=[
Choices(
message=Message(content="Hello", role="assistant"),
index=0,
finish_reason="stop",
)
],
model="gpt-4",
usage=Usage(prompt_tokens=5, completion_tokens=5, total_tokens=10),
)
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
await proxy_logging.post_call_success_hook(
data=data,
response=response,
user_api_key_dict=user_api_key_dict,
)
assert guardrail.was_called is True
@pytest.mark.asyncio
async def test_post_call_success_hook_skips_guardrail_not_on_model():
"""
Guardrails NOT configured on the model should not execute when
no other source (request body, key, team) enables them.
"""
from litellm.caching.caching import DualCache
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.utils import ProxyLogging
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.utils import Choices, Message, ModelResponse, Usage
class TestGuardrail(CustomGuardrail):
def __init__(self):
super().__init__(
guardrail_name="unrelated-guardrail",
event_hook=GuardrailEventHooks.post_call,
)
self.was_called = False
async def async_post_call_success_hook(
self, data, user_api_key_dict, response
):
self.was_called = True
return response
guardrail = TestGuardrail()
# Deployment has a DIFFERENT guardrail configured
mock_router = MagicMock()
mock_deployment = MagicMock()
mock_deployment.litellm_params.get.return_value = ["some-other-guardrail"]
mock_router.get_deployment.return_value = mock_deployment
with patch("litellm.callbacks", [guardrail]), patch(
"litellm.proxy.proxy_server.llm_router", mock_router
):
proxy_logging = ProxyLogging(user_api_key_cache=DualCache())
data = {
"model": "gpt-4",
"metadata": {"model_info": {"id": "model-uuid-123"}},
}
response = ModelResponse(
id="resp-1",
choices=[
Choices(
message=Message(content="Hello", role="assistant"),
index=0,
finish_reason="stop",
)
],
model="gpt-4",
usage=Usage(prompt_tokens=5, completion_tokens=5, total_tokens=10),
)
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
await proxy_logging.post_call_success_hook(
data=data,
response=response,
user_api_key_dict=user_api_key_dict,
)
assert guardrail.was_called is False