Merge remote-tracking branch 'origin' into litellm_blog_dropdown

.github/ISSUE_TEMPLATE/config.yml

@@ -1,7 +1,7 @@
 blank_issues_enabled: true
 contact_links:
   - name: Schedule Demo
-    url: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
+    url: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions
     about: Speak directly with Krrish and Ishaan, the founders, to discuss issues, share feedback, or explore improvements for LiteLLM
   - name: Discord
     url: https://discord.com/invite/wuPM9dRgDw

.github/workflows/interpret_load_test.py

@@ -123,7 +123,7 @@ if __name__ == "__main__":
         + docker_run_command
         + "\n\n"
         + "### Don't want to maintain your internal proxy? get in touch 🎉"
-        + "\nHosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
+        + "\nHosted Proxy Alpha: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions"
         + "\n\n"
         + "## Load Test LiteLLM Proxy Results"
         + "\n\n"

README.md

@@ -399,7 +399,7 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature
 # Enterprise
 For companies that need better security, user management and professional support
 
-[Talk to founders](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Talk to founders](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 This covers:
 - ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):**

cookbook/benchmark/readme.md

@@ -178,4 +178,4 @@ Benchmark Results for 'When will BerriAI IPO?':
 ```
 
 ## Support 
-**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
+**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.

cookbook/gollem_go_agent_framework/README.md

@@ -0,0 +1,119 @@
+# Gollem Go Agent Framework with LiteLLM
+
+A working example showing how to use [gollem](https://github.com/fugue-labs/gollem), a production-grade Go agent framework, with LiteLLM as a proxy gateway. This lets Go developers access 100+ LLM providers through a single proxy while keeping compile-time type safety for tools and structured output.
+
+## Quick Start
+
+### 1. Start LiteLLM Proxy
+
+```bash
+# Simple start with a single model
+litellm --model gpt-4o
+
+# Or with the example config for multi-provider access
+litellm --config proxy_config.yaml
+```
+
+### 2. Run the examples
+
+```bash
+# Install Go dependencies
+go mod tidy
+
+# Basic agent
+go run ./basic
+
+# Agent with type-safe tools
+go run ./tools
+
+# Streaming responses
+go run ./streaming
+```
+
+## Configuration
+
+The included `proxy_config.yaml` sets up three providers through LiteLLM:
+
+```yaml
+model_list:
+  - model_name: gpt-4o          # OpenAI
+  - model_name: claude-sonnet    # Anthropic
+  - model_name: gemini-pro       # Google Vertex AI
+```
+
+Switch providers in Go by changing a single string — no code changes needed:
+
+```go
+model := openai.NewLiteLLM("http://localhost:4000",
+    openai.WithModel("gpt-4o"),        // OpenAI
+    // openai.WithModel("claude-sonnet"),  // Anthropic
+    // openai.WithModel("gemini-pro"),     // Google
+)
+```
+
+## Examples
+
+### `basic/` — Basic Agent
+
+Connects gollem to LiteLLM and runs a simple prompt. Demonstrates the `NewLiteLLM` constructor and basic agent creation.
+
+### `tools/` — Type-Safe Tools
+
+Shows gollem's compile-time type-safe tool framework working through LiteLLM's tool-use passthrough. The tool parameters are Go structs with JSON tags — the schema is generated automatically at compile time.
+
+### `streaming/` — Streaming Responses
+
+Real-time token streaming using Go 1.23+ range-over-function iterators, proxied through LiteLLM's SSE passthrough.
+
+## How It Works
+
+Gollem's `openai.NewLiteLLM()` constructor creates an OpenAI-compatible provider pointed at your LiteLLM proxy. Since LiteLLM speaks the OpenAI API protocol, everything works out of the box:
+
+- **Chat completions** — standard request/response
+- **Tool use** — LiteLLM passes tool definitions and calls through transparently
+- **Streaming** — Server-Sent Events proxied through LiteLLM
+- **Structured output** — JSON schema response format works with supporting models
+
+```
+Go App (gollem) → LiteLLM Proxy → OpenAI / Anthropic / Google / ...
+```
+
+## Why Use This?
+
+- **Type-safe Go**: Compile-time type checking for tools, structured output, and agent configuration — no runtime surprises
+- **Single proxy, many models**: Switch between OpenAI, Anthropic, Google, and 100+ other providers by changing a model name string
+- **Zero-dependency core**: gollem's core has no external dependencies — just stdlib
+- **Single binary deployment**: `go build` produces one binary, no pip/venv/Docker needed
+- **Cost tracking & rate limiting**: LiteLLM handles cost tracking, rate limits, and fallbacks at the proxy layer
+
+## Environment Variables
+
+```bash
+# Required for providers you want to use (set in LiteLLM config or env)
+export OPENAI_API_KEY="sk-..."
+export ANTHROPIC_API_KEY="sk-ant-..."
+
+# Optional: point to a non-default LiteLLM proxy
+export LITELLM_PROXY_URL="http://localhost:4000"
+```
+
+## Troubleshooting
+
+**Connection errors?**
+- Make sure LiteLLM is running: `litellm --model gpt-4o`
+- Check the URL is correct (default: `http://localhost:4000`)
+
+**Model not found?**
+- Verify the model name matches what's configured in LiteLLM
+- Run `curl http://localhost:4000/models` to see available models
+
+**Tool calls not working?**
+- Ensure the underlying model supports tool use (GPT-4o, Claude, Gemini)
+- Check LiteLLM logs for any provider-specific errors
+
+## Learn More
+
+- [gollem GitHub](https://github.com/fugue-labs/gollem)
+- [gollem API Reference](https://pkg.go.dev/github.com/fugue-labs/gollem/core)
+- [LiteLLM Proxy Docs](https://docs.litellm.ai/docs/simple_proxy)
+- [LiteLLM Supported Models](https://docs.litellm.ai/docs/providers)

cookbook/gollem_go_agent_framework/basic/main.go

@@ -0,0 +1,41 @@
+// Basic gollem agent connected to a LiteLLM proxy.
+//
+// Usage:
+//
+//	litellm --model gpt-4o   # start proxy in another terminal
+//	go run ./basic
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"os"
+
+	"github.com/fugue-labs/gollem/core"
+	"github.com/fugue-labs/gollem/provider/openai"
+)
+
+func main() {
+	proxyURL := "http://localhost:4000"
+	if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
+		proxyURL = u
+	}
+
+	// Connect to LiteLLM proxy. NewLiteLLM creates an OpenAI-compatible
+	// provider pointed at the given URL.
+	model := openai.NewLiteLLM(proxyURL,
+		openai.WithModel("gpt-4o"), // any model name configured in LiteLLM
+	)
+
+	// Create and run a simple agent.
+	agent := core.NewAgent[string](model,
+		core.WithSystemPrompt[string]("You are a helpful assistant. Be concise."),
+	)
+
+	result, err := agent.Run(context.Background(), "Explain quantum computing in two sentences.")
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println(result.Output)
+}

cookbook/gollem_go_agent_framework/go.mod

@@ -0,0 +1,5 @@
+module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
+
+go 1.25.1
+
+require github.com/fugue-labs/gollem v0.1.0

cookbook/gollem_go_agent_framework/go.sum

@@ -0,0 +1,2 @@
+github.com/fugue-labs/gollem v0.1.0 h1:QexYnvkb44QZFEljgAePqMIGZjgsbk0Y5GJ2jYYgfa8=
+github.com/fugue-labs/gollem v0.1.0/go.mod h1:htW1YO81uysSKVOkYJtxhGCFrzm+36HBFxEWuECoHKQ=

cookbook/gollem_go_agent_framework/proxy_config.yaml

@@ -0,0 +1,16 @@
+model_list:
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+  - model_name: claude-sonnet
+    litellm_params:
+      model: anthropic/claude-sonnet-4-20250514
+      api_key: os.environ/ANTHROPIC_API_KEY
+
+  - model_name: gemini-pro
+    litellm_params:
+      model: vertex_ai/gemini-2.0-flash
+      vertex_project: my-project
+      vertex_location: us-central1

cookbook/gollem_go_agent_framework/streaming/main.go

@@ -0,0 +1,56 @@
+// Streaming responses from gollem through LiteLLM.
+//
+// Uses Go 1.23+ range-over-function iterators for real-time token
+// streaming via LiteLLM's SSE passthrough.
+//
+// Usage:
+//
+//	litellm --model gpt-4o
+//	go run ./streaming
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"os"
+
+	"github.com/fugue-labs/gollem/core"
+	"github.com/fugue-labs/gollem/provider/openai"
+)
+
+func main() {
+	proxyURL := "http://localhost:4000"
+	if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
+		proxyURL = u
+	}
+
+	model := openai.NewLiteLLM(proxyURL,
+		openai.WithModel("gpt-4o"),
+	)
+
+	agent := core.NewAgent[string](model)
+
+	// RunStream returns a streaming result that yields tokens as they arrive.
+	stream, err := agent.RunStream(context.Background(), "Write a haiku about distributed systems")
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	// StreamText yields text chunks in real-time.
+	// The boolean argument controls whether deltas (true) or accumulated
+	// text (false) is returned.
+	fmt.Print("Response: ")
+	for text, err := range stream.StreamText(true) {
+		if err != nil {
+			log.Fatal(err)
+		}
+		fmt.Print(text)
+	}
+	fmt.Println()
+
+	// After streaming completes, the final response is available.
+	resp := stream.Response()
+	fmt.Printf("\nTokens used: input=%d, output=%d\n",
+		resp.Usage.InputTokens, resp.Usage.OutputTokens)
+}

cookbook/gollem_go_agent_framework/tools/main.go

@@ -0,0 +1,64 @@
+// Gollem agent with type-safe tools through LiteLLM.
+//
+// The tool parameters are Go structs — gollem generates the JSON schema
+// automatically at compile time. LiteLLM passes tool definitions through
+// transparently to the underlying provider.
+//
+// Usage:
+//
+//	litellm --model gpt-4o
+//	go run ./tools
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"os"
+
+	"github.com/fugue-labs/gollem/core"
+	"github.com/fugue-labs/gollem/provider/openai"
+)
+
+// WeatherParams defines the tool's input schema via struct tags.
+// The JSON schema is generated at compile time — no runtime reflection needed.
+type WeatherParams struct {
+	City string `json:"city" description:"City name to get weather for"`
+	Unit string `json:"unit,omitempty" description:"Temperature unit: celsius or fahrenheit"`
+}
+
+func main() {
+	proxyURL := "http://localhost:4000"
+	if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
+		proxyURL = u
+	}
+
+	model := openai.NewLiteLLM(proxyURL,
+		openai.WithModel("gpt-4o"),
+	)
+
+	// Define a type-safe tool. The function signature enforces correct types.
+	weatherTool := core.FuncTool[WeatherParams](
+		"get_weather",
+		"Get current weather for a city",
+		func(ctx context.Context, p WeatherParams) (string, error) {
+			unit := p.Unit
+			if unit == "" {
+				unit = "fahrenheit"
+			}
+			// In production, call a real weather API here.
+			return fmt.Sprintf("Weather in %s: 72°F (22°C), sunny", p.City), nil
+		},
+	)
+
+	agent := core.NewAgent[string](model,
+		core.WithTools[string](weatherTool),
+		core.WithSystemPrompt[string]("You are a helpful weather assistant. Use the get_weather tool to answer weather questions."),
+	)
+
+	result, err := agent.Run(context.Background(), "What's the weather like in San Francisco and Tokyo?")
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println(result.Output)
+}

docs/my-website/blog/anthropic_wildcard_model_access_incident/index.md

@@ -0,0 +1,147 @@
+---
+slug: anthropic-wildcard-model-access-incident
+title: "Incident Report: Wildcard Blocking New Models After Cost Map Reload"
+date: 2026-02-23T10:00:00
+authors:
+  - name: Sameer Kankute
+    title: SWE @ LiteLLM (LLM Translation)
+    url: https://www.linkedin.com/in/sameer-kankute/
+    image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
+  - name: Krrish Dholakia
+    title: "CEO, LiteLLM"
+    url: https://www.linkedin.com/in/krish-d/
+    image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+  - name: Ishaan Jaff
+    title: "CTO, LiteLLM"
+    url: https://www.linkedin.com/in/reffajnaahsi/
+    image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+tags: [incident-report, proxy, auth, model-access]
+hide_table_of_contents: false
+---
+
+**Date:** Feb 23, 2026  
+**Duration:** ~3 hours  
+**Severity:** High (for users with provider wildcard access rules)  
+**Status:** Resolved
+
+## Summary
+
+When a new Anthropic model (e.g. `claude-sonnet-4-6`) was added to the LiteLLM model cost map and a cost map reload was triggered, requests to the new model were rejected with:
+
+```
+key not allowed to access model. This key can only access models=['anthropic/*']. Tried to access claude-sonnet-4-6.
+```
+
+The reload updated `litellm.model_cost` correctly but never re-ran `add_known_models()`, so `litellm.anthropic_models` (the in-memory set used by the wildcard resolver) remained stale. The new model was invisible to the `anthropic/*` wildcard even though the cost map knew about it.
+
+- **LLM calls:** All requests to newly-added Anthropic models were blocked with a 401.
+- **Existing models:** Unaffected — only models missing from the stale provider set were impacted.
+- **Other providers:** Same bug class existed for any provider wildcard (e.g. `openai/*`, `gemini/*`).
+
+{/* truncate */}
+
+---
+
+## Background
+
+LiteLLM supports provider-level wildcard access rules. When an admin configures a key or team with `models=['anthropic/*']`, any model whose provider resolves to `anthropic` should be allowed. The resolution happens in `_model_custom_llm_provider_matches_wildcard_pattern`:
+
+```mermaid
+flowchart TD
+    A["1. Request arrives for claude-sonnet-4-6"] --> B["2. Auth check: can this key call this model?
+    proxy/auth/auth_checks.py"]
+    B --> C["3. Key has models=['anthropic/*']
+    → wildcard match attempted"]
+    C --> D["4. get_llm_provider('claude-sonnet-4-6')
+    checks litellm.anthropic_models set"]
+    D -->|"model IN set"| E["5a. ✅ Provider = 'anthropic'
+    → 'anthropic/claude-sonnet-4-6' matches 'anthropic/*'"]
+    D -->|"model NOT IN set"| F["5b. ❌ Provider unknown
+    → exception raised → wildcard returns False"]
+    E --> G["6. Request allowed"]
+    F --> H["6. 401: key not allowed to access model"]
+
+    style E fill:#d4edda,stroke:#28a745
+    style F fill:#f8d7da,stroke:#dc3545
+    style H fill:#f8d7da,stroke:#dc3545
+    style D fill:#fff3cd,stroke:#ffc107
+```
+
+`litellm.anthropic_models` is a Python `set` populated at import time by `add_known_models()`. It is the source `get_llm_provider()` consults to map a bare model name like `claude-sonnet-4-6` to the provider string `"anthropic"`.
+
+---
+
+## Root Cause
+
+`add_known_models()` is called **once** at module import time. Both reload paths in `proxy_server.py` updated `litellm.model_cost` with the fresh map but never called `add_known_models()` again:
+
+```python
+# Before the fix — both reload paths looked like this:
+new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
+litellm.model_cost = new_model_cost_map          # ✅ cost map updated
+_invalidate_model_cost_lowercase_map()           # ✅ cache cleared
+# ❌ add_known_models() never called
+#    → litellm.anthropic_models still has the old set
+#    → new model not in the set
+#    → get_llm_provider() raises for the new model
+#    → wildcard match returns False
+#    → 401 for every request to the new model
+```
+
+The gap existed in two places:
+1. `_check_and_reload_model_cost_map` — the periodic automatic reload (every 10 s)
+2. The `/reload/model_cost_map` admin endpoint — the manual reload
+
+**Timeline:**
+
+1. New model (`claude-sonnet-4-6`) added to `model_prices_and_context_window.json`
+2. Admin triggers cost map reload via UI → `litellm.model_cost` updated
+3. Users with `anthropic/*` wildcard keys attempt requests to `claude-sonnet-4-6`
+4. `get_llm_provider('claude-sonnet-4-6')` raises → wildcard returns False → 401
+5. Admin reloads cost map again — same result (root cause not addressed)
+6. ~3 hours of investigation → root cause identified → fix deployed
+
+---
+
+## The Fix
+
+After each reload, `add_known_models()` is called with the freshly fetched map passed explicitly. Passing the map directly (rather than relying on the module-level reference) removes any ambiguity about which dict is iterated:
+
+```python
+# After the fix — both reload paths now do:
+new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
+litellm.model_cost = new_model_cost_map
+_invalidate_model_cost_lowercase_map()
+litellm.add_known_models(model_cost_map=new_model_cost_map)  # ✅ sets repopulated
+```
+
+`add_known_models()` was also updated to accept an optional explicit map so callers cannot accidentally iterate a stale module-level reference:
+
+```python
+# Before
+def add_known_models():
+    for key, value in model_cost.items():   # reads module global — ambiguous after reload
+        ...
+
+# After
+def add_known_models(model_cost_map: Optional[Dict] = None):
+    _map = model_cost_map if model_cost_map is not None else model_cost
+    for key, value in _map.items():         # always iterates the map you just fetched
+        ...
+```
+
+After the fix, the provider sets (`anthropic_models`, `open_ai_chat_completion_models`, etc.) are always consistent with `litellm.model_cost` immediately after every reload. New models become accessible via wildcard rules without any proxy restart.
+
+---
+
+## Remediation
+
+| # | Action | Status | Code |
+|---|---|---|---|
+| 1 | Call `add_known_models(model_cost_map=...)` in the periodic reload path | ✅ Done | [`proxy_server.py#L4393`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L4393) |
+| 2 | Call `add_known_models(model_cost_map=...)` in the `/reload/model_cost_map` endpoint | ✅ Done | [`proxy_server.py#L11904`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L11904) |
+| 3 | Update `add_known_models()` to accept an explicit map parameter | ✅ Done | [`__init__.py#L617`](https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py#L617) |
+| 4 | Regression test: `add_known_models(model_cost_map=...)` populates provider sets | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
+| 5 | Regression test: `anthropic/*` wildcard grants/denies access correctly after reload | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
+
+---

docs/my-website/blog/server_root_path/index.md

@@ -0,0 +1,154 @@
+---
+slug: server-root-path-incident
+title: "Incident Report: SERVER_ROOT_PATH regression broke UI routing"
+date: 2026-02-21T10:00:00
+authors:
+  - name: Yuneng Jiang
+    title: SWE @ LiteLLM (Full Stack)
+    url: https://www.linkedin.com/in/yunengjiang/
+  - name: Ishaan Jaff
+    title: "CTO, LiteLLM"
+    url: https://www.linkedin.com/in/reffajnaahsi/
+    image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+  - name: Krrish Dholakia
+    title: "CEO, LiteLLM"
+    url: https://www.linkedin.com/in/krish-d/
+    image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+tags: [incident-report, ui, stability]
+hide_table_of_contents: false
+---
+
+**Date:** January 22, 2026
+**Duration:** ~4 days (until fix merged January 26, 2026)
+**Severity:** High
+**Status:** Resolved
+
+> **Note:** This fix is available starting from LiteLLM `v1.81.3.rc.6` or higher.
+
+## Summary
+
+A PR ([`#19467`](https://github.com/BerriAI/litellm/pull/19467)) accidentally removed the `root_path=server_root_path` parameter from the FastAPI app initialization in `proxy_server.py`. This caused the proxy to ignore the `SERVER_ROOT_PATH` environment variable when serving the UI. Users who deploy LiteLLM behind a reverse proxy with a path prefix (e.g., `/api/v1` or `/llmproxy`) found that all UI pages returned 404 Not Found.
+
+- **LLM API calls:** No impact. API routing was unaffected.
+- **UI pages:** All UI pages returned 404 for deployments using `SERVER_ROOT_PATH`.
+- **Swagger/OpenAPI docs:** Broken when accessed through the configured root path.
+
+{/* truncate */}
+
+---
+
+## Background
+
+Many LiteLLM deployments run behind a reverse proxy (e.g., Nginx, Traefik, AWS ALB) that routes traffic to LiteLLM under a path prefix. FastAPI's `root_path` parameter tells the application about this prefix so it can correctly serve static files, generate URLs, and handle routing.
+
+```mermaid
+sequenceDiagram
+    participant User as User Browser
+    participant RP as Reverse Proxy
+    participant LP as LiteLLM Proxy
+
+    User->>RP: GET /llmproxy/ui/
+    RP->>LP: GET /ui/ (X-Forwarded-Prefix: /llmproxy)
+
+    Note over LP: Before regression:<br/>FastAPI root_path="/llmproxy"<br/>→ Serves UI correctly
+
+    Note over LP: After regression:<br/>FastAPI root_path=""<br/>→ UI assets resolve to wrong paths<br/>→ 404 Not Found
+```
+
+The `root_path` parameter was present in `proxy_server.py` since early versions of LiteLLM. It was removed as a side effect of PR [#19467](https://github.com/BerriAI/litellm/pull/19467), which was intended to fix a different UI 404 issue.
+
+---
+
+## Root cause
+
+PR [#19467](https://github.com/BerriAI/litellm/pull/19467) (`73d49f8`) removed the `root_path=server_root_path` line from the `FastAPI()` constructor in `proxy_server.py`:
+
+```diff
+ app = FastAPI(
+     docs_url=_get_docs_url(),
+     redoc_url=_get_redoc_url(),
+     title=_title,
+     description=_description,
+     version=version,
+-    root_path=server_root_path,
+     lifespan=proxy_startup_event,
+ )
+```
+
+Without `root_path`, FastAPI treated all requests as if the application was mounted at `/`, causing path mismatches for any deployment using `SERVER_ROOT_PATH`.
+
+The regression went undetected because:
+
+1. **No automated test** verified that `root_path` was set on the FastAPI app.
+2. **No manual test procedure** existed for `SERVER_ROOT_PATH` functionality.
+3. **Default deployments** (without `SERVER_ROOT_PATH`) were unaffected, so most CI tests passed.
+
+---
+
+## Remediation
+
+| #   | Action                                                                                            | Status  | Code                                                                                                                       |
+| --- | ------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
+| 1   | Restore `root_path=server_root_path` in FastAPI app initialization                                | ✅ Done | [`#19790`](https://github.com/BerriAI/litellm/pull/19790) (`5426b3c`)                                                      |
+| 2   | Add unit tests for `get_server_root_path()` and FastAPI app initialization                        | ✅ Done | [`test_server_root_path.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_server_root_path.py) |
+| 3   | Add CI workflow that builds Docker image and tests UI routing with `SERVER_ROOT_PATH` on every PR | ✅ Done | [`test_server_root_path.yml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test_server_root_path.yml)    |
+| 4   | Document manual test procedure for `SERVER_ROOT_PATH`                                             | ✅ Done | [Discussion #8495](https://github.com/BerriAI/litellm/discussions/8495)                                                    |
+
+---
+
+## CI workflow details
+
+The new [`test_server_root_path.yml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test_server_root_path.yml) workflow runs on every PR against `main`. It:
+
+1. Builds the LiteLLM Docker image
+2. Starts a container with `SERVER_ROOT_PATH` set (tests both `/api/v1` and `/llmproxy`)
+3. Verifies the UI returns valid HTML at `{ROOT_PATH}/ui/`
+4. Fails the workflow if the UI is unreachable
+
+```mermaid
+flowchart TD
+    A["PR opened/updated"] --> B["Build Docker image"]
+    B --> C["Start container with SERVER_ROOT_PATH=/api/v1"]
+    B --> D["Start container with SERVER_ROOT_PATH=/llmproxy"]
+    C --> E["curl {ROOT_PATH}/ui/ → expect HTML"]
+    D --> F["curl {ROOT_PATH}/ui/ → expect HTML"]
+    E -->|"HTML found"| G["✅ Pass"]
+    E -->|"404 or no HTML"| H["❌ Fail Workflow"]
+    F -->|"HTML found"| G
+    F -->|"404 or no HTML"| H
+
+    style G fill:#d4edda,stroke:#28a745
+    style H fill:#f8d7da,stroke:#dc3545
+```
+
+This prevents future regressions where changes to `proxy_server.py` accidentally break `SERVER_ROOT_PATH` support.
+
+---
+
+## Timeline
+
+| Time (UTC)         | Event                                                                                                                                                        |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Jan 22, 2026 04:20 | PR [#19467](https://github.com/BerriAI/litellm/pull/19467) merged, removing `root_path=server_root_path`                                                     |
+| Jan 22–26          | Users on nightly builds report UI 404 errors when using `SERVER_ROOT_PATH`                                                                                   |
+| Jan 26, 2026 17:48 | Fix PR [#19790](https://github.com/BerriAI/litellm/pull/19790) merged, restoring `root_path=server_root_path`                                                |
+| Feb 18, 2026       | CI workflow [`test_server_root_path.yml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test_server_root_path.yml) added to run on every PR |
+
+---
+
+## Resolution steps for users
+
+For users still experiencing issues, update to the latest LiteLLM version:
+
+```bash
+pip install --upgrade litellm
+```
+
+Verify your `SERVER_ROOT_PATH` is correctly set:
+
+```bash
+# In your environment or docker-compose.yml
+SERVER_ROOT_PATH="/your-prefix"
+```
+
+Then confirm the UI is accessible at `http://your-host:4000/your-prefix/ui/`.

docs/my-website/docs/benchmarks.md

@@ -5,6 +5,44 @@ import Image from '@theme/IdealImage';
 
 Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.
 
+## Setting Up Benchmarking with Network Mock
+
+The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider. 
+
+**1. Create a proxy config:**
+
+```yaml
+model_list:
+  - model_name: db-openai-endpoint
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: "sk-fake-key"
+      api_base: "https://api.openai.com"
+
+litellm_settings:
+  network_mock: true
+  callbacks: []
+  num_retries: 0
+  request_timeout: 30
+
+general_settings:
+  master_key: "sk-1234"
+```
+
+**2. Start the proxy:**
+
+```bash
+litellm --config benchmark_config.yaml --port 4000 --num_workers 8
+```
+
+**3. Run the benchmark script:**
+
+```bash
+python scripts/benchmark_mock.py --requests 2000 --max-concurrent 200 --runs 3
+```
+
+This measures pure proxy overhead on the hot path without any network latency to a real or fake provider.
+
 ## Setting Up a Fake OpenAI Endpoint
 
 For load testing and benchmarking, you can use a fake OpenAI proxy server. LiteLLM provides:

docs/my-website/docs/caching/all_caches.md

@@ -297,6 +297,7 @@ litellm.cache = Cache(
     similarity_threshold=0.7, # similarity threshold for cache hits, 0 == no similarity, 1 = exact matches, 0.5 == 50% similarity
     qdrant_quantization_config ="binary", # can be one of 'binary', 'product' or 'scalar' quantizations that is supported by qdrant
     qdrant_semantic_cache_embedding_model="text-embedding-ada-002", # this model is passed to litellm.embedding(), any litellm.embedding() model is supported here
+    qdrant_semantic_cache_vector_size=1536, # vector size for the embedding model, must match the dimensionality of the embedding model used
 )
 
 response1 = completion(
@@ -635,6 +636,7 @@ def __init__(
     qdrant_quantization_config: Optional[str] = None,
     qdrant_semantic_cache_embedding_model="text-embedding-ada-002",
 
+    qdrant_semantic_cache_vector_size: Optional[int] = None,
     **kwargs
 ):
 ```

docs/my-website/docs/contributing.md

@@ -79,7 +79,27 @@ cp -r out/* ../../litellm/proxy/_experimental/out/
 
 Then restart the proxy and access the UI at `http://localhost:4000/ui`
 
-## 4. Submitting a PR
+## 4. Pre-PR Checklist
+
+Before submitting your pull request, make sure the following pass locally from `ui/litellm-dashboard/`:
+
+**Run tests related to your changes:**
+
+```bash
+npx vitest run src/components/path/to/YourComponent.test.tsx
+```
+
+Tests are co-located with components (e.g., `TeamInfo.tsx` → `TeamInfo.test.tsx`). If you add a new component, add a corresponding `.test.tsx` file next to it.
+
+**Run the build:**
+
+```bash
+npm run build
+```
+
+These map to the `ui_tests` and `ui_build` CI checks.
+
+## 5. Submitting a PR
 
 1. Create a new branch for your changes:
 ```bash

docs/my-website/docs/enterprise.md

@@ -4,7 +4,7 @@ import Image from '@theme/IdealImage';
 
 :::info
 - ✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
-- Who is Enterprise for? Companies giving access to 100+ users **OR** 10+ AI use-cases. If you're not sure, [get in touch with us](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) to discuss your needs.
+- Who is Enterprise for? Companies giving access to 100+ users **OR** 10+ AI use-cases. If you're not sure, [get in touch with us](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) to discuss your needs.
 :::
 
 For companies that need SSO, user management and professional support for LiteLLM Proxy
@@ -36,7 +36,7 @@ Manage Yourself - you can deploy our Docker Image or build a custom image from o
 
 ### What’s the cost of the Self-Managed Enterprise edition?
 
-Self-Managed Enterprise deployments require our team to understand your exact needs. [Get in touch with us to learn more](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+Self-Managed Enterprise deployments require our team to understand your exact needs. [Get in touch with us to learn more](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 
 ### How does deployment with Enterprise License work? 
@@ -106,7 +106,7 @@ Professional Support can assist with LLM/Provider integrations, deployment, upgr
 
 Pricing is based on usage. We can figure out a price that works for your team, on the call. 
 
-[**Contact Us to learn more**](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[**Contact Us to learn more**](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 
 

docs/my-website/docs/fine_tuning.md

@@ -6,7 +6,7 @@ import TabItem from '@theme/TabItem';
 
 :::info
 
-This is an Enterprise only endpoint [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+This is an Enterprise only endpoint [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/interactions.md

@@ -130,13 +130,12 @@ Point the Google GenAI SDK to LiteLLM Proxy:
 
 ```python showLineNumbers title="Google GenAI SDK with LiteLLM Proxy"
 from google import genai
-import os
 
 # Point SDK to LiteLLM Proxy
-os.environ["GOOGLE_GENAI_BASE_URL"] = "http://localhost:4000"
-os.environ["GEMINI_API_KEY"] = "sk-1234"  # Your LiteLLM API key
-
-client = genai.Client()
+client = genai.Client(
+    api_key="sk-1234",  # Your LiteLLM API key
+    http_options={"base_url": "http://localhost:4000"},
+)
 
 # Create an interaction
 interaction = client.interactions.create(
@@ -151,12 +150,11 @@ print(interaction.outputs[-1].text)
 
 ```python showLineNumbers title="Google GenAI SDK Streaming"
 from google import genai
-import os
 
-os.environ["GOOGLE_GENAI_BASE_URL"] = "http://localhost:4000"
-os.environ["GEMINI_API_KEY"] = "sk-1234"
-
-client = genai.Client()
+client = genai.Client(
+    api_key="sk-1234",  # Your LiteLLM API key
+    http_options={"base_url": "http://localhost:4000"},
+)
 
 for chunk in client.interactions.create_stream(
     model="gemini/gemini-2.5-flash",

docs/my-website/docs/observability/gcs_bucket_integration.md

@@ -6,7 +6,7 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
 
 :::info
 
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/pass_through/google_ai_studio.md

@@ -35,26 +35,25 @@ curl 'http://0.0.0.0:4000/gemini/v1beta/models/gemini-1.5-flash:countTokens?key=
 ```
 
 </TabItem>
-<TabItem value="js" label="Google AI Node.js SDK">
+<TabItem value="js" label="Google GenAI JS SDK">
 
 ```javascript
-const { GoogleGenerativeAI } = require("@google/generative-ai");
+const { GoogleGenAI } = require("@google/genai");
 
-const modelParams = {
-    model: 'gemini-pro',
-};
-  
-const requestOptions = {
-    baseUrl: 'http://localhost:4000/gemini', // http://<proxy-base-url>/gemini
-};
-  
-const genAI = new GoogleGenerativeAI("sk-1234"); // litellm proxy API key
-const model = genAI.getGenerativeModel(modelParams, requestOptions);
+const ai = new GoogleGenAI({
+    apiKey: "sk-1234", // litellm proxy API key
+    httpOptions: {
+        baseUrl: "http://localhost:4000/gemini", // http://<proxy-base-url>/gemini
+    },
+});
 
 async function main() {
     try {
-        const result = await model.generateContent("Explain how AI works");
-        console.log(result.response.text());
+        const response = await ai.models.generateContent({
+            model: "gemini-2.5-flash",
+            contents: "Explain how AI works",
+        });
+        console.log(response.text);
     } catch (error) {
         console.error('Error:', error);
     }
@@ -63,12 +62,13 @@ async function main() {
 // For streaming responses
 async function main_streaming() {
     try {
-        const streamingResult = await model.generateContentStream("Explain how AI works");
-        for await (const chunk of streamingResult.stream) {
-            console.log('Stream chunk:', JSON.stringify(chunk));
+        const response = await ai.models.generateContentStream({
+            model: "gemini-2.5-flash",
+            contents: "Explain how AI works",
+        });
+        for await (const chunk of response) {
+            process.stdout.write(chunk.text);
         }
-        const aggregatedResponse = await streamingResult.response;
-        console.log('Aggregated response:', JSON.stringify(aggregatedResponse));
     } catch (error) {
         console.error('Error:', error);
     }
@@ -321,29 +321,28 @@ curl 'http://0.0.0.0:4000/gemini/v1beta/models/gemini-1.5-flash:generateContent?
 ```
 
 </TabItem>
-<TabItem value="js" label="Google AI Node.js SDK">
+<TabItem value="js" label="Google GenAI JS SDK">
 
 ```javascript
-const { GoogleGenerativeAI } = require("@google/generative-ai");
+const { GoogleGenAI } = require("@google/genai");
 
-const modelParams = {
-    model: 'gemini-pro',
-};
-  
-const requestOptions = {
-    baseUrl: 'http://localhost:4000/gemini', // http://<proxy-base-url>/gemini
-    customHeaders: {
-        "tags": "gemini-js-sdk,pass-through-endpoint"
-    }
-};
-  
-const genAI = new GoogleGenerativeAI("sk-1234");
-const model = genAI.getGenerativeModel(modelParams, requestOptions);
+const ai = new GoogleGenAI({
+    apiKey: "sk-1234",
+    httpOptions: {
+        baseUrl: "http://localhost:4000/gemini", // http://<proxy-base-url>/gemini
+        headers: {
+            "tags": "gemini-js-sdk,pass-through-endpoint",
+        },
+    },
+});
 
 async function main() {
     try {
-        const result = await model.generateContent("Explain how AI works");
-        console.log(result.response.text());
+        const response = await ai.models.generateContent({
+            model: "gemini-2.5-flash",
+            contents: "Explain how AI works",
+        });
+        console.log(response.text);
     } catch (error) {
         console.error('Error:', error);
     }

docs/my-website/docs/proxy/alerting.md

@@ -438,6 +438,59 @@ curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \
 
 - `event_message` *str*: A human-readable description of the event.
 
+### Digest Mode (Reducing Alert Noise)
+
+By default, LiteLLM sends a separate Slack message for **every** alert event. For high-frequency alert types like `llm_requests_hanging` or `llm_too_slow`, this can produce hundreds of duplicate messages per day.
+
+**Digest mode** aggregates duplicate alerts within a configurable time window and emits a single summary message with the total count and time range.
+
+#### Configuration
+
+Use `alert_type_config` in `general_settings` to enable digest mode per alert type:
+
+```yaml
+general_settings:
+  alerting: ["slack"]
+  alert_type_config:
+    llm_requests_hanging:
+      digest: true
+      digest_interval: 86400  # 24 hours (default)
+    llm_too_slow:
+      digest: true
+      digest_interval: 3600   # 1 hour
+    llm_exceptions:
+      digest: true
+      # uses default interval (86400 seconds / 24 hours)
+```
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `digest` | bool | `false` | Enable digest mode for this alert type |
+| `digest_interval` | int | `86400` (24h) | Time window in seconds. Alerts are aggregated within this interval. |
+
+#### How It Works
+
+1. When an alert fires for a digest-enabled type, it is **grouped** by `(alert_type, request_model, api_base)` instead of being sent immediately
+2. A counter tracks how many times the alert fires within the interval
+3. When the interval expires, a **single summary message** is sent:
+
+```
+Alert type: `llm_requests_hanging` (Digest)
+Level: `Medium`
+Start: `2026-02-19 03:27:39`
+End: `2026-02-20 03:27:39`
+Count: `847`
+
+Message: `Requests are hanging - 600s+ request time`
+Request Model: `gemini-2.5-flash`
+API Base: `None`
+```
+
+#### Limitations
+
+- **Per-instance**: Digest state is held in memory per proxy instance. If you run multiple instances (e.g., Cloud Run with autoscaling), each instance maintains its own digest and emits its own summary.
+- **Not durable**: If an instance is terminated before the digest interval expires, the aggregated alerts for that instance are lost.
+
 ## Region-outage alerting (✨ Enterprise feature)
 
 :::info

docs/my-website/docs/proxy/budget_reset_and_tz.md

@@ -22,6 +22,8 @@ litellm_settings:
 This ensures that all budget resets happen at midnight in your specified timezone rather than in UTC.
 If no timezone is specified, UTC will be used by default.
 
+Any valid [IANA timezone string](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) is supported (powered by Python's `zoneinfo` module). DST transitions are handled automatically.
+
 Common timezone values:
 
 - `UTC` - Coordinated Universal Time

docs/my-website/docs/proxy/caching.md

@@ -340,6 +340,7 @@ litellm_settings:
     qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
     qdrant_collection_name: test_collection
     qdrant_quantization_config: binary
+    qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
     similarity_threshold: 0.8 # similarity threshold for semantic cache
 ```
 

docs/my-website/docs/proxy/config_settings.md

@@ -73,6 +73,7 @@ litellm_settings:
     qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
     qdrant_collection_name: test_collection
     qdrant_quantization_config: binary
+    qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
     similarity_threshold: 0.8 # similarity threshold for semantic cache
 
     # Optional - S3 Cache Settings
@@ -574,6 +575,8 @@ router_settings:
 | DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO | Default minimal reasoning effort thinking budget for Gemini 2.5 Pro. Default is 512
 | DEFAULT_REDIS_MAJOR_VERSION | Default Redis major version to assume when version cannot be determined. Default is 7
 | DEFAULT_REDIS_SYNC_INTERVAL | Default Redis synchronization interval in seconds. Default is 1
+| DEFAULT_SEMANTIC_GUARD_EMBEDDING_MODEL | Default embedding model for Semantic Guard (route-matching guardrail). Default is "text-embedding-3-small"
+| DEFAULT_SEMANTIC_GUARD_SIMILARITY_THRESHOLD | Default similarity threshold for Semantic Guard route matching. Default is 0.75
 | DEFAULT_REPLICATE_GPU_PRICE_PER_SECOND | Default price per second for Replicate GPU. Default is 0.001400
 | DEFAULT_REPLICATE_POLLING_DELAY_SECONDS | Default delay in seconds for Replicate polling. Default is 1
 | DEFAULT_REPLICATE_POLLING_RETRIES | Default number of retries for Replicate polling. Default is 5
@@ -756,6 +759,7 @@ router_settings:
 | LITELLM_CLI_JWT_EXPIRATION_HOURS | Expiration time in hours for CLI-generated JWT tokens. Default is 24 hours
 | LITELLM_DD_AGENT_HOST | Hostname or IP of DataDog agent for LiteLLM-specific logging. When set, logs are sent to agent instead of direct API
 | LITELLM_DEPLOYMENT_ENVIRONMENT | Environment name for the deployment (e.g., "production", "staging"). Used as a fallback when OTEL_ENVIRONMENT_NAME is not set. Sets the `environment` tag in telemetry data
+| LITELLM_DETAILED_TIMING | When true, adds detailed per-phase timing headers to responses (`x-litellm-timing-{pre-processing,llm-api,post-processing,message-copy}-ms`). Default is false. See [latency overhead docs](../troubleshoot/latency_overhead.md)
 | LITELLM_DD_AGENT_PORT | Port of DataDog agent for LiteLLM-specific log intake. Default is 10518
 | LITELLM_DD_LLM_OBS_PORT | Port for Datadog LLM Observability agent. Default is 8126
 | LITELLM_DONT_SHOW_FEEDBACK_BOX | Flag to hide feedback box in LiteLLM UI
@@ -807,6 +811,7 @@ router_settings:
 | LOGGING_WORKER_MAX_QUEUE_SIZE | Maximum size of the logging worker queue. When the queue is full, the worker aggressively clears tasks to make room instead of dropping logs. Default is 50,000
 | LOGGING_WORKER_MAX_TIME_PER_COROUTINE | Maximum time in seconds allowed for each coroutine in the logging worker before timing out. Default is 20.0
 | LOGGING_WORKER_CLEAR_PERCENTAGE | Percentage of the queue to extract when clearing. Default is 50% 
+| MAX_BASE64_LENGTH_FOR_LOGGING | Maximum number of base64 characters to keep in logging payloads. Data URIs exceeding this are replaced with a size placeholder. Set to 0 to disable truncation. Default is 64
 | MAX_COMPETITOR_NAMES | Maximum number of competitor names allowed in policy template enrichment. Default is 100
 | MAX_EXCEPTION_MESSAGE_LENGTH | Maximum length for exception messages. Default is 2000
 | MAX_ITERATIONS_TO_CLEAR_QUEUE | Maximum number of iterations to attempt when clearing the logging worker queue during shutdown. Default is 200
@@ -830,6 +835,7 @@ router_settings:
 | MAX_LANGFUSE_INITIALIZED_CLIENTS | Maximum number of Langfuse clients to initialize on proxy. Default is 50. This is set since langfuse initializes 1 thread everytime a client is initialized. We've had an incident in the past where we reached 100% cpu utilization because Langfuse was initialized several times.
 | MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH | Maximum header length for MCP semantic filter tools. Default is 150
 | MAX_POLICY_ESTIMATE_IMPACT_ROWS | Maximum number of rows returned when estimating the impact of a policy. Default is 1000
+| MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG | Maximum payload size in bytes for full DEBUG serialization. Payloads exceeding this will be truncated in logs. Default is 102400 (100 KB)
 | MIN_NON_ZERO_TEMPERATURE | Minimum non-zero temperature value. Default is 0.0001
 | MINIMUM_PROMPT_CACHE_TOKEN_COUNT | Minimum token count for caching a prompt. Default is 1024
 | MISTRAL_API_BASE | Base URL for Mistral API. Default is https://api.mistral.ai
@@ -893,6 +899,13 @@ router_settings:
 | POSTHOG_API_URL | Base URL for PostHog API (defaults to https://us.i.posthog.com)
 | POSTHOG_MOCK | Enable mock mode for PostHog integration testing. When set to true, intercepts PostHog API calls and returns mock responses without making actual network calls. Default is false
 | POSTHOG_MOCK_LATENCY_MS | Mock latency in milliseconds for PostHog API calls when mock mode is enabled. Simulates network round-trip time. Default is 100ms
+| PRISMA_AUTH_RECONNECT_LOCK_TIMEOUT_SECONDS | Lock timeout in seconds for Prisma auth reconnection. Default is 0.1
+| PRISMA_AUTH_RECONNECT_TIMEOUT_SECONDS | Timeout in seconds for Prisma auth reconnection attempts. Default is 2.0
+| PRISMA_HEALTH_WATCHDOG_ENABLED | Enable the Prisma DB health watchdog that monitors and reconnects on connection loss. Default is true
+| PRISMA_HEALTH_WATCHDOG_INTERVAL_SECONDS | Interval in seconds for Prisma health watchdog probes. Default is 30
+| PRISMA_HEALTH_WATCHDOG_PROBE_TIMEOUT_SECONDS | Timeout in seconds for each Prisma health probe. Default is 5.0
+| PRISMA_RECONNECT_COOLDOWN_SECONDS | Cooldown in seconds between Prisma reconnection attempts. Default is 15
+| PRISMA_WATCHDOG_RECONNECT_TIMEOUT_SECONDS | Timeout in seconds for Prisma watchdog-initiated reconnection. Default is 30.0
 | PREDIBASE_API_BASE | Base URL for Predibase API
 | PRESIDIO_ANALYZER_API_BASE | Base URL for Presidio Analyzer service
 | PRESIDIO_ANONYMIZER_API_BASE | Base URL for Presidio Anonymizer service

docs/my-website/docs/proxy/cost_tracking.md

@@ -161,7 +161,7 @@ Use this when you want non-proxy admins to access `/spend` endpoints
 
 :::info
 
-Schedule a [meeting with us to get your Enterprise License](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+Schedule a [meeting with us to get your Enterprise License](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/email.md

@@ -203,7 +203,7 @@ After regenerating the key, the user will receive an email notification with:
 
 :::info
 
-Customizing Email Branding is an Enterprise Feature [Get in touch with us for a Free Trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+Customizing Email Branding is an Enterprise Feature [Get in touch with us for a Free Trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/enterprise.md

@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
 # ✨ Enterprise Features
 :::tip
 
-To get a license, get in touch with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+To get a license, get in touch with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/guardrails/aporia_api.md

@@ -139,7 +139,7 @@ curl -i http://localhost:4000/v1/chat/completions \
 
 :::info
 
-✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/guardrails/custom_guardrail.md

@@ -409,7 +409,7 @@ curl -i  -X POST http://localhost:4000/v1/chat/completions \
 
 :::info
 
-✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/guardrails/guardrails_ai.md

@@ -59,7 +59,7 @@ curl -i http://localhost:4000/v1/chat/completions \
 
 :::info
 
-✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Contact us to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/guardrails/noma_security.md

@@ -6,6 +6,108 @@ import TabItem from '@theme/TabItem';
 
 Use [Noma Security](https://noma.security/) to protect your LLM applications with comprehensive AI content moderation and safety guardrails.
 
+:::warning Deprecated: `guardrail: noma` (Legacy)
+`guardrail: noma` is deprecated and users should migrate to `guardrail: noma_v2`.
+The legacy `guardrail: noma` API will no longer be supported after March 31, 2026.
+
+For easier migration of existing integrations, keep `guardrail: noma` and set `use_v2: true`.
+With `use_v2: true`, requests route to `noma_v2`; `monitor_mode` and `block_failures` still apply, while `anonymize_input` is ignored.
+:::
+
+## Noma v2 guardrails (Recommended)
+
+### Quick Start
+
+```yaml showLineNumbers title="litellm config.yaml"
+guardrails:
+  - guardrail_name: "noma-v2-guard"
+    litellm_params:
+      guardrail: noma_v2
+      mode: "pre_call"
+      api_key: os.environ/NOMA_API_KEY
+      api_base: os.environ/NOMA_API_BASE
+```
+
+If you want to migrate gradually without changing guardrail names yet:
+
+```yaml showLineNumbers title="litellm config.yaml"
+guardrails:
+  - guardrail_name: "noma-guard"
+    litellm_params:
+      guardrail: noma
+      use_v2: true
+      mode: "pre_call"
+      api_key: os.environ/NOMA_API_KEY
+      api_base: os.environ/NOMA_API_BASE
+```
+
+### Supported Params
+
+- **`guardrail`**: Use `noma_v2` (recommended), or `noma` with `use_v2: true` for migration
+- **`mode`**: `pre_call`, `post_call`, `during_call`, `pre_mcp_call`, `during_mcp_call`
+- **`api_key`**: Noma API key (required for Noma SaaS, optional for self-managed deployments)
+- **`api_base`**: Noma API base URL (defaults to `https://api.noma.security/`)
+- **`application_id`**: Application identifier. If omitted, v2 checks dynamic `extra_body.application_id`, then configured/env `application_id`; otherwise it is omitted.
+- **`monitor_mode`**: If `true`, runs in monitor-only mode without blocking (defaults to `false`)
+- **`block_failures`**: If `true`, fail-closed on guardrail technical failures (defaults to `true`)
+- **`use_v2`**: Migration toggle when `guardrail: noma` is used
+
+### Environment Variables
+
+```shell
+export NOMA_API_KEY="your-api-key-here"
+export NOMA_API_BASE="https://api.noma.security/"       # Optional
+export NOMA_APPLICATION_ID="my-app"                     # Optional
+export NOMA_MONITOR_MODE="false"                        # Optional
+export NOMA_BLOCK_FAILURES="true"                       # Optional
+```
+
+### Multiple Guardrails
+
+Apply different v2 configurations for input and output:
+
+```yaml showLineNumbers title="litellm config.yaml"
+guardrails:
+  - guardrail_name: "noma-v2-input"
+    litellm_params:
+      guardrail: noma_v2
+      mode: "pre_call"
+      api_key: os.environ/NOMA_API_KEY
+
+  - guardrail_name: "noma-v2-output"
+    litellm_params:
+      guardrail: noma_v2
+      mode: "post_call"
+      api_key: os.environ/NOMA_API_KEY
+```
+
+### Pass Additional Parameters
+
+This is supported in v2 via `extra_body`.  
+Currently, `noma_v2` consumes dynamic `application_id`.
+
+```shell showLineNumbers title="Curl Request"
+curl 'http://0.0.0.0:4000/v1/chat/completions' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "gpt-4o-mini",
+    "messages": [
+      {
+        "role": "user",
+        "content": "Hello, how are you?"
+      }
+    ],
+    "guardrails": {
+      "noma-v2-guard": {
+        "extra_body": {
+          "application_id": "my-specific-app-id"
+        }
+      }
+    }
+  }'
+```
+## Noma guardrails (Legacy)
+
 ## Quick Start
 
 ### 1. Define Guardrails on your LiteLLM config.yaml

docs/my-website/docs/proxy/ip_address.md

@@ -3,7 +3,7 @@
 
 :::info
 
-You need a LiteLLM License to unlock this feature. [Grab time](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat), to get one today!
+You need a LiteLLM License to unlock this feature. [Grab time](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions), to get one today!
 
 :::
 

docs/my-website/docs/proxy/logging.md

@@ -1109,7 +1109,7 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
 
 :::info
 
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 
@@ -1194,7 +1194,7 @@ Log LLM Logs/SpendLogs to [Google Cloud Storage PubSub Topic](https://cloud.goog
 
 :::info
 
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 
@@ -1497,7 +1497,7 @@ Log LLM Logs to [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azur
 
 :::info
 
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/multiple_admins.md

@@ -20,7 +20,7 @@ LiteLLM tracks changes to the following entities and actions:
 
 :::tip
 
-Requires Enterprise License, Get in touch with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+Requires Enterprise License, Get in touch with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/oauth2.md

@@ -4,7 +4,7 @@ Use this if you want to use an Oauth2.0 token to make `/chat`, `/embeddings` req
 
 :::info
 
-This is an Enterprise Feature - [get in touch with us if you want a free trial to test if this feature meets your needs]((https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat))
+This is an Enterprise Feature - [get in touch with us if you want a free trial to test if this feature meets your needs]((https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions))
 
 :::
 

docs/my-website/docs/proxy/prod.md

@@ -47,7 +47,7 @@ export LITELLM_LOG="ERROR"
 
 :::info
 
-Need Help or want dedicated support ? Talk to a founder [here]: (https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+Need Help or want dedicated support ? Talk to a founder [here]: (https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/public_routes.md

@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
 
 :::info
 
-Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat).
+Requires a LiteLLM Enterprise License. [Get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions).
 
 :::
 

docs/my-website/docs/proxy/tag_routing.md

@@ -215,7 +215,7 @@ LiteLLM Proxy supports team-based tag routing, allowing you to associate specifi
 
 :::info
 
-This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+This is an enterprise feature, [Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/team_logging.md

@@ -26,7 +26,7 @@ Team 3 -> Disabled Logging (for GDPR compliance)
 
 :::info
 
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 
@@ -248,7 +248,7 @@ Use the `/key/generate` or `/key/update` endpoints to add logging callbacks to a
 
 :::info
 
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+✨ This is an Enterprise only feature [Get Started with Enterprise here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/team_model_add.md

@@ -5,7 +5,7 @@
 This is an Enterprise feature.
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/token_auth.md

@@ -11,7 +11,7 @@ Use JWT's to auth admins / users / projects into the proxy.
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/proxy/ui_store_model_db_setting.md

@@ -0,0 +1,92 @@
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Store Model in DB Settings
+
+Enable or disable storing model definitions in the database directly from the Admin UI—no config file edits or proxy restart required. This is especially useful for cloud deployments where updating the config is difficult or requires a long release process.
+
+## Overview
+
+Previously, the `store_model_in_db` setting had to be configured in `proxy_config.yaml` under `general_settings`. Changing it required editing the config and restarting the proxy, which was problematic for cloud users who don't have direct access to the config file or who want to avoid the downtime caused by restarts.
+
+<Image img={require('../../img/ui_store_model_in_db.png')} />
+
+**Store Model in DB Settings** lets you:
+
+- **Enable or disable storing models in the database** – Control whether model definitions are cached in your database (useful for reducing config file size and improving scalability)
+- **Apply changes immediately** – No proxy restart needed; settings take effect for new model operations as soon as you save
+
+:::warning UI overrides config
+Settings changed in the UI **override** the values in your config file. For example, if `store_model_in_db` is set to `false` in `general_settings`, enabling it in the UI will still persist model definitions to the database. Use the UI when you want runtime control without redeploying.
+:::
+
+## How Store Model in DB Works
+
+When `store_model_in_db` is enabled, the LiteLLM proxy stores model definitions in the database instead of relying solely on your `proxy_config.yaml`. This provides several benefits:
+
+- **Reduced config size** – Move model definitions out of YAML for easier maintenance
+- **Scalability** – Database storage scales better than large YAML files
+- **Dynamic updates** – Models can be added or updated without editing config files
+- **Persistence** – Model definitions persist across proxy instances and restarts
+
+The setting applies to all new model operations from the moment you save it.
+
+## How to Configure Store Model in DB in the UI
+
+### 1. Access Models + Endpoints Settings
+
+Navigate to the Admin UI (e.g. `http://localhost:4000/ui` or your `PROXY_BASE_URL/ui`) and go to the **Models + Endpoints** page.
+
+![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/55bc71f5-730f-4b2c-8539-8a4f46b8bd10/ascreenshot_0f7ba8f1c2694e94938996fd1b4adfcc_text_export.jpeg)
+
+### 2. Open Settings
+
+Click **Models + Endpoints** from the navigation menu.
+
+![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/55bc71f5-730f-4b2c-8539-8a4f46b8bd10/ascreenshot_fc2b9e4812a9480087f4eb350fa0a792_text_export.jpeg)
+
+### 3. Click the Settings Icon
+
+Look for the settings (gear) icon on the Models + Endpoints page to open the configuration panel.
+
+![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/7b394364-c281-4db8-8cad-ee322c76c935/ascreenshot_d7c8a6b234bc4e4d92aa7f09aefb13d3_text_export.jpeg)
+
+### 4. Enable or Disable Store Model in DB
+
+Toggle the **Store Model in DB** setting based on your preference:
+
+- **Enabled**: Model definitions will be stored in the database
+- **Disabled**: Models are read from the config file only
+
+![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/54a263ec-ad67-4b16-ba9f-2be57c3e4cb8/ascreenshot_501abda2a6c847f79d085efce814265d_text_export.jpeg)
+
+### 5. Save Settings
+
+Click **Save Settings** to apply the change. No proxy restart is required; the new setting takes effect immediately for subsequent model operations.
+
+![](https://colony-recorder.s3.amazonaws.com/files/2026-02-22/7d13559a-d4e4-41f7-993b-cb20fbfa1f6e/ascreenshot_3245f3c5bd0d43cb96c5f5ff0ccb461d_text_export.jpeg)
+
+## Use Cases
+
+### Cloud and Managed Deployments
+
+When the proxy runs in a managed or cloud environment, config may be in a separate repo, require a long release cycle, or be controlled by another team. Using the UI lets you change the `store_model_in_db` setting without going through a deployment process.
+
+### Reducing Configuration Complexity
+
+For large deployments with hundreds of models, storing model definitions in the database reduces the size and complexity of your `proxy_config.yaml`, making it easier to maintain and version control.
+
+### Dynamic Model Management
+
+Enable `store_model_in_db` to support dynamic model additions and updates without editing your config file. Teams can manage models through the UI or API without needing to redeploy the proxy.
+
+### Zero-Downtime Updates
+
+Change the setting from the UI and have it take effect immediately—perfect for production environments where downtime must be minimized.
+
+## Related Documentation
+
+- [Admin UI Overview](./ui_overview.md) – General guide to the LiteLLM Admin UI
+- [Models and Endpoints](./models_and_endpoints.md) – Managing models and API endpoints
+- [Config Settings](./config_settings.md) – `store_model_in_db` in `general_settings`

docs/my-website/docs/secret.md

@@ -6,7 +6,7 @@
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/aws_kms.md

@@ -6,7 +6,7 @@
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/aws_secret_manager.md

@@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem';
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/azure_key_vault.md

@@ -6,7 +6,7 @@
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/cyberark.md

@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/google_kms.md

@@ -6,7 +6,7 @@
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/google_secret_manager.md

@@ -6,7 +6,7 @@
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/hashicorp_vault.md

@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/secret_managers/overview.md

@@ -8,7 +8,7 @@ import Image from '@theme/IdealImage';
 
 [Enterprise Pricing](https://www.litellm.ai/#pricing)
 
-[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
+[Contact us here to get a free trial](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions)
 
 :::
 

docs/my-website/docs/tutorials/compare_llms.md

@@ -82,7 +82,7 @@ Benchmark Results for 'When will BerriAI IPO?':
 +-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
 ```
 ## Support
-**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
+**🤝 Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
 
 
 <!-- 

docs/my-website/docs/tutorials/google_genai_sdk.md

@@ -0,0 +1,406 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Google GenAI SDK with LiteLLM
+
+Use Google's official GenAI SDK (JavaScript/TypeScript and Python) with any LLM provider through LiteLLM Proxy.
+
+The Google GenAI SDK (`@google/genai` for JS, `google-genai` for Python) provides a native interface for calling Gemini models. By pointing it to LiteLLM, you can use the same SDK with OpenAI, Anthropic, Bedrock, Azure, Vertex AI, or any other provider — while keeping the native Gemini request/response format.
+
+## Why Use LiteLLM with Google GenAI SDK?
+
+**Developer Benefits:**
+- **Universal Model Access**: Use any LiteLLM-supported model (Anthropic, OpenAI, Vertex AI, Bedrock, etc.) through the Google GenAI SDK interface
+- **Higher Rate Limits & Reliability**: Load balance across multiple models and providers to avoid hitting individual provider limits, with fallbacks to ensure you get responses even if one provider fails
+
+**Proxy Admin Benefits:**
+- **Centralized Management**: Control access to all models through a single LiteLLM proxy instance without giving developers API keys to each provider
+- **Budget Controls**: Set spending limits and track costs across all SDK usage
+- **Logging & Observability**: Track all requests with cost tracking, logging, and analytics
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| Cost Tracking | ✅ | All models on `/generateContent` endpoint |
+| Logging | ✅ | Works across all integrations |
+| Streaming | ✅ | `streamGenerateContent` supported |
+| Virtual Keys | ✅ | Use LiteLLM keys instead of Google keys |
+| Load Balancing | ✅ | Via native router endpoints |
+| Fallbacks | ✅ | Via native router endpoints |
+
+## Quick Start
+
+### 1. Install the SDK
+
+<Tabs>
+<TabItem value="js" label="JavaScript/TypeScript">
+
+```bash
+npm install @google/genai
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```bash
+pip install google-genai
+```
+
+</TabItem>
+</Tabs>
+
+### 2. Start LiteLLM Proxy
+
+```yaml title="config.yaml" showLineNumbers
+model_list:
+  - model_name: gemini-2.5-flash
+    litellm_params:
+      model: gemini/gemini-2.5-flash
+      api_key: os.environ/GEMINI_API_KEY
+```
+
+```bash
+litellm --config config.yaml
+```
+
+### 3. Call the SDK through LiteLLM
+
+<Tabs>
+<TabItem value="js" label="JavaScript/TypeScript">
+
+```javascript title="index.js" showLineNumbers
+const { GoogleGenAI } = require("@google/genai");
+
+const ai = new GoogleGenAI({
+  apiKey: "sk-1234",  // LiteLLM virtual key (not a Google key)
+  httpOptions: {
+    baseUrl: "http://localhost:4000/gemini",  // LiteLLM proxy URL
+  },
+});
+
+async function main() {
+  const response = await ai.models.generateContent({
+    model: "gemini-2.5-flash",
+    contents: "Explain how AI works",
+  });
+  console.log(response.text);
+}
+
+main();
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python title="main.py" showLineNumbers
+from google import genai
+
+client = genai.Client(
+    api_key="sk-1234",  # LiteLLM virtual key (not a Google key)
+    http_options={"base_url": "http://localhost:4000/gemini"},  # LiteLLM proxy URL
+)
+
+response = client.models.generate_content(
+    model="gemini-2.5-flash",
+    contents="Explain how AI works",
+)
+print(response.text)
+```
+
+</TabItem>
+<TabItem value="curl" label="curl">
+
+```bash
+curl "http://localhost:4000/gemini/v1beta/models/gemini-2.5-flash:generateContent?key=sk-1234" \
+  -H 'Content-Type: application/json' \
+  -X POST \
+  -d '{
+    "contents": [{
+      "parts": [{"text": "Explain how AI works"}]
+    }]
+  }'
+```
+
+</TabItem>
+</Tabs>
+
+## Streaming
+
+<Tabs>
+<TabItem value="js" label="JavaScript/TypeScript">
+
+```javascript title="streaming.js" showLineNumbers
+const { GoogleGenAI } = require("@google/genai");
+
+const ai = new GoogleGenAI({
+  apiKey: "sk-1234",
+  httpOptions: {
+    baseUrl: "http://localhost:4000/gemini",
+  },
+});
+
+async function main() {
+  const response = await ai.models.generateContentStream({
+    model: "gemini-2.5-flash",
+    contents: "Write a short poem about the ocean",
+  });
+
+  for await (const chunk of response) {
+    process.stdout.write(chunk.text);
+  }
+}
+
+main();
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python title="streaming.py" showLineNumbers
+from google import genai
+
+client = genai.Client(
+    api_key="sk-1234",
+    http_options={"base_url": "http://localhost:4000/gemini"},
+)
+
+response = client.models.generate_content_stream(
+    model="gemini-2.5-flash",
+    contents="Write a short poem about the ocean",
+)
+
+for chunk in response:
+    print(chunk.text, end="")
+```
+
+</TabItem>
+</Tabs>
+
+## Multi-turn Chat
+
+<Tabs>
+<TabItem value="js" label="JavaScript/TypeScript">
+
+```javascript title="chat.js" showLineNumbers
+const { GoogleGenAI } = require("@google/genai");
+
+const ai = new GoogleGenAI({
+  apiKey: "sk-1234",
+  httpOptions: {
+    baseUrl: "http://localhost:4000/gemini",
+  },
+});
+
+async function main() {
+  const chat = ai.chats.create({
+    model: "gemini-2.5-flash",
+  });
+
+  const response1 = await chat.sendMessage({ message: "I have 2 dogs and 3 cats." });
+  console.log(response1.text);
+
+  const response2 = await chat.sendMessage({ message: "How many pets is that in total?" });
+  console.log(response2.text);
+}
+
+main();
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python title="chat.py" showLineNumbers
+from google import genai
+
+client = genai.Client(
+    api_key="sk-1234",
+    http_options={"base_url": "http://localhost:4000/gemini"},
+)
+
+chat = client.chats.create(model="gemini-2.5-flash")
+
+response1 = chat.send_message("I have 2 dogs and 3 cats.")
+print(response1.text)
+
+response2 = chat.send_message("How many pets is that in total?")
+print(response2.text)
+```
+
+</TabItem>
+</Tabs>
+
+
+## Advanced: Use Any Model with the GenAI SDK
+
+By default, the GenAI SDK talks to Gemini models. But with LiteLLM's router, you can route GenAI SDK requests to **any provider** — Anthropic, OpenAI, Bedrock, etc.
+
+This works by using `model_group_alias` to map Gemini model names to your desired provider models. LiteLLM handles the format translation internally.
+
+:::info
+
+For this to work, point the SDK `baseUrl` to `http://localhost:4000` (without `/gemini`). This routes requests through LiteLLM's native Google endpoints, which go through the router and support model aliasing.
+
+:::
+
+<Tabs>
+<TabItem value="anthropic" label="Anthropic">
+
+Route `gemini-2.5-flash` requests to Claude Sonnet:
+
+```yaml title="config.yaml" showLineNumbers
+model_list:
+  - model_name: claude-sonnet
+    litellm_params:
+      model: anthropic/claude-sonnet-4-20250514
+      api_key: os.environ/ANTHROPIC_API_KEY
+
+router_settings:
+  model_group_alias: {"gemini-2.5-flash": "claude-sonnet"}
+```
+
+</TabItem>
+<TabItem value="openai" label="OpenAI">
+
+Route `gemini-2.5-flash` requests to GPT-4o:
+
+```yaml title="config.yaml" showLineNumbers
+model_list:
+  - model_name: gpt-4o-model
+    litellm_params:
+      model: gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+router_settings:
+  model_group_alias: {"gemini-2.5-flash": "gpt-4o-model"}
+```
+
+</TabItem>
+<TabItem value="bedrock" label="Bedrock">
+
+Route `gemini-2.5-flash` requests to Claude on Bedrock:
+
+```yaml title="config.yaml" showLineNumbers
+model_list:
+  - model_name: bedrock-claude
+    litellm_params:
+      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
+      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
+      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
+      aws_region_name: us-east-1
+
+router_settings:
+  model_group_alias: {"gemini-2.5-flash": "bedrock-claude"}
+```
+
+</TabItem>
+<TabItem value="multi" label="Multi-Provider Load Balancing">
+
+Load balance across Anthropic and OpenAI:
+
+```yaml title="config.yaml" showLineNumbers
+model_list:
+  - model_name: my-model
+    litellm_params:
+      model: anthropic/claude-sonnet-4-20250514
+      api_key: os.environ/ANTHROPIC_API_KEY
+  - model_name: my-model
+    litellm_params:
+      model: gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+router_settings:
+  model_group_alias: {"gemini-2.5-flash": "my-model"}
+```
+
+</TabItem>
+</Tabs>
+
+Then use the SDK with `baseUrl` pointing to LiteLLM (without `/gemini`):
+
+<Tabs>
+<TabItem value="js" label="JavaScript/TypeScript">
+
+```javascript title="any_model.js" showLineNumbers
+const { GoogleGenAI } = require("@google/genai");
+
+const ai = new GoogleGenAI({
+  apiKey: "sk-1234",
+  httpOptions: {
+    baseUrl: "http://localhost:4000",  // No /gemini — goes through the router
+  },
+});
+
+async function main() {
+  // This calls Claude/GPT-4o/Bedrock under the hood via model_group_alias
+  const response = await ai.models.generateContent({
+    model: "gemini-2.5-flash",
+    contents: "Hello from any model!",
+  });
+  console.log(response.text);
+}
+
+main();
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python title="any_model.py" showLineNumbers
+from google import genai
+
+client = genai.Client(
+    api_key="sk-1234",
+    http_options={"base_url": "http://localhost:4000"},  # No /gemini
+)
+
+# This calls Claude/GPT-4o/Bedrock under the hood via model_group_alias
+response = client.models.generate_content(
+    model="gemini-2.5-flash",
+    contents="Hello from any model!",
+)
+print(response.text)
+```
+
+</TabItem>
+</Tabs>
+
+
+## Pass-through vs Native Router Endpoints
+
+LiteLLM offers two ways to handle GenAI SDK requests:
+
+| | Pass-through (`/gemini`) | Native Router (`/`) |
+|---|---|---|
+| **baseUrl** | `http://localhost:4000/gemini` | `http://localhost:4000` |
+| **Models** | Gemini only | Any provider via `model_group_alias` |
+| **Translation** | None — proxies directly to Google | Translates internally |
+| **Cost Tracking** | ✅ | ✅ |
+| **Virtual Keys** | ✅ | ✅ |
+| **Load Balancing** | ❌ | ✅ |
+| **Fallbacks** | ❌ | ✅ |
+| **Best for** | Simple Gemini proxy | Multi-provider routing |
+
+## Environment Variable Configuration
+
+You can also configure the SDK via environment variables instead of code:
+
+```bash
+# For JavaScript SDK (@google/genai)
+export GOOGLE_GEMINI_BASE_URL="http://localhost:4000/gemini"
+export GEMINI_API_KEY="sk-1234"
+
+# For Python SDK (google-genai)
+# Note: The Python SDK does not support a base URL env var.
+# Configure it in code with http_options={"base_url": "..."} instead.
+export GEMINI_API_KEY="sk-1234"
+```
+
+This is especially useful for tools built on top of the GenAI SDK (like [Gemini CLI](./litellm_gemini_cli.md)).
+
+## Related Resources
+
+- [Gemini CLI with LiteLLM](./litellm_gemini_cli.md)
+- [Google AI Studio Pass-Through](../pass_through/google_ai_studio)
+- [Google ADK with LiteLLM](./google_adk.md)
+- [LiteLLM Proxy Quick Start](../proxy/quick_start)
+- [`@google/genai` npm package](https://www.npmjs.com/package/@google/genai)
+- [`google-genai` PyPI package](https://pypi.org/project/google-genai/)

docs/my-website/docs/tutorials/openai_agents_sdk.md

@@ -0,0 +1,373 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# OpenAI Agents SDK with LiteLLM
+
+Use OpenAI's Agents SDK with any LLM provider through LiteLLM Proxy.
+
+This tutorial shows you how to build AI agents using the OpenAI Agents SDK with support for multiple LLM providers through LiteLLM.
+
+## Overview
+
+The OpenAI Agents SDK provides a high-level interface for building AI agents. By integrating with LiteLLM, you can:
+
+- Use multiple LLM providers (Bedrock, Azure, Vertex AI, etc.) with the same agent code
+- Switch easily between models from different providers
+- Connect to a LiteLLM proxy for centralized model management
+
+:::tip Built-in LiteLLM Extension
+
+The OpenAI Agents SDK includes an official LiteLLM extension (`LitellmModel`) that works without a proxy. If you don't need centralized proxy features (cost tracking, rate limiting, load balancing), you can use it directly:
+
+```python
+from agents import Agent, Runner
+from agents.extensions.models.litellm_model import LitellmModel
+
+
+agent = Agent(
+    name="Assistant",
+    instructions="You are a helpful assistant.",
+    model=LitellmModel(model="anthropic/claude-sonnet-4-20250514"),
+)
+
+result = Runner.run_sync(agent, "Hello!")
+print(result.final_output)
+```
+
+See the [Docs](https://openai.github.io/openai-agents-python/models/litellm/) for more details. The rest of this tutorial focuses on the **proxy-based approach** for teams that need centralized model management.
+
+:::
+
+## Prerequisites
+
+- Python environment setup
+- API keys for your LLM providers
+- Basic understanding of LLMs and agent concepts
+
+## Installation
+
+```bash showLineNumbers title="Install dependencies"
+pip install openai-agents litellm
+```
+
+## 1. Start LiteLLM Proxy
+
+Configure and start the LiteLLM proxy with the models you want to use:
+
+```yaml title="config.yaml" showLineNumbers
+model_list:
+  - model_name: bedrock-claude-sonnet-4
+    litellm_params:
+      model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
+      aws_region_name: "us-east-1"
+
+  - model_name: gpt-4o
+    litellm_params:
+      model: "openai/gpt-4o"
+
+  - model_name: claude-sonnet-4
+    litellm_params:
+      model: "anthropic/claude-sonnet-4-20250514"
+
+  - model_name: bedrock-claude-haiku
+    litellm_params:
+      model: "bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0"
+      aws_region_name: "us-east-1"
+
+  - model_name: bedrock-nova-premier
+    litellm_params:
+      model: "bedrock/amazon.nova-premier-v1:0"
+      aws_region_name: "us-east-1"
+```
+
+```bash
+litellm --config config.yaml
+```
+
+Required environment variables:
+
+| Variable | Value | Description |
+|----------|-------|-------------|
+| `LITELLM_BASE_URL` | `http://localhost:4000` | LiteLLM proxy URL |
+| `LITELLM_API_KEY` | `sk-1234` | Your LiteLLM API key (not your provider's key) |
+
+## 2. Setting Up Environment
+
+Import the necessary libraries and configure your LiteLLM proxy connection:
+
+```python showLineNumbers title="Setup environment"
+from __future__ import annotations
+
+import asyncio
+import os
+
+from openai import AsyncOpenAI
+
+from agents import (
+    Agent,
+    Model,
+    ModelProvider,
+    OpenAIChatCompletionsModel,
+    RunConfig,
+    Runner,
+    function_tool,
+    set_tracing_disabled,
+)
+
+# Point to LiteLLM proxy
+BASE_URL = os.getenv("LITELLM_BASE_URL") or "http://localhost:4000"
+API_KEY = os.getenv("LITELLM_API_KEY") or "sk-1234"
+
+# Define model constants for cleaner code
+MODEL_BEDROCK_SONNET = "bedrock-claude-sonnet-4"
+MODEL_BEDROCK_HAIKU = "bedrock-claude-haiku"
+MODEL_GPT_4O = "gpt-4o"
+
+# Create the OpenAI client pointed at LiteLLM
+client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
+
+# Disable tracing since we're not using OpenAI's platform directly
+set_tracing_disabled(disabled=True)
+```
+
+## 3. Create a Custom Model Provider
+
+The Agents SDK uses a `ModelProvider` to resolve model names. Create a custom provider that routes all requests through LiteLLM:
+
+```python showLineNumbers title="Custom LiteLLM model provider"
+class LiteLLMModelProvider(ModelProvider):
+    def get_model(self, model_name: str | None) -> Model:
+        return OpenAIChatCompletionsModel(
+            model=model_name or MODEL_BEDROCK_SONNET,
+            openai_client=client,
+        )
+
+
+LITELLM_MODEL_PROVIDER = LiteLLMModelProvider()
+```
+
+## 4. Define a Simple Tool
+
+Create a tool that your agent can use:
+
+```python showLineNumbers title="Weather tool implementation"
+@function_tool
+def get_weather(city: str) -> str:
+    """Retrieves the current weather report for a specified city.
+
+    Args:
+        city: The name of the city (e.g., "New York", "London", "Tokyo").
+
+    Returns:
+        A string containing the weather information for the city.
+    """
+    print(f"[debug] getting weather for {city}")
+
+    mock_weather_db = {
+        "new york": "The weather in New York is sunny with a temperature of 25°C.",
+        "london": "It's cloudy in London with a temperature of 15°C.",
+        "tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
+    }
+
+    city_normalized = city.lower()
+
+    if city_normalized in mock_weather_db:
+        return mock_weather_db[city_normalized]
+    else:
+        return f"Sorry, I don't have weather information for '{city}'."
+```
+
+## 5. Using Different Models with Agents
+
+### 5.1 Using Bedrock Models
+
+```python showLineNumbers title="Bedrock model via LiteLLM proxy"
+async def test_bedrock_agent():
+    print("\n--- Testing Bedrock Claude Agent ---")
+
+    agent = Agent(
+        name="weather_agent_bedrock",
+        instructions="You are a helpful weather assistant powered by Claude. "
+                     "Use the 'get_weather' tool for city weather requests. "
+                     "Present information clearly.",
+        tools=[get_weather],
+    )
+
+    result = await Runner.run(
+        agent,
+        "What's the weather in Tokyo?",
+        run_config=RunConfig(
+            model_provider=LITELLM_MODEL_PROVIDER,
+            model="bedrock-claude-sonnet-4",  # Uses the model name from your LiteLLM config
+        ),
+    )
+    print(f"<<< Agent Response: {result.final_output}")
+
+
+asyncio.run(test_bedrock_agent())
+```
+
+### 5.2 Using OpenAI Models
+
+```python showLineNumbers title="OpenAI model via LiteLLM proxy"
+async def test_openai_agent():
+    print("\n--- Testing OpenAI GPT Agent ---")
+
+    agent = Agent(
+        name="weather_agent_gpt",
+        instructions="You are a helpful weather assistant powered by GPT-4o. "
+                     "Use the 'get_weather' tool for city weather requests. "
+                     "Present information clearly.",
+        tools=[get_weather],
+    )
+
+    result = await Runner.run(
+        agent,
+        "What's the weather in London?",
+        run_config=RunConfig(
+            model_provider=LITELLM_MODEL_PROVIDER,
+            model="gpt-4o",  # Uses the model name from your LiteLLM config
+        ),
+    )
+    print(f"<<< Agent Response: {result.final_output}")
+
+
+asyncio.run(test_openai_agent())
+```
+
+### 5.3 Using Anthropic Models
+
+```python showLineNumbers title="Anthropic model via LiteLLM proxy"
+async def test_anthropic_agent():
+    print("\n--- Testing Anthropic Claude Agent ---")
+
+    agent = Agent(
+        name="weather_agent_claude",
+        instructions="You are a helpful weather assistant powered by Claude. "
+                     "Use the 'get_weather' tool for city weather requests. "
+                     "Present information clearly.",
+        tools=[get_weather],
+    )
+
+    result = await Runner.run(
+        agent,
+        "What's the weather in New York?",
+        run_config=RunConfig(
+            model_provider=LITELLM_MODEL_PROVIDER,
+            model="claude-sonnet-4",  # Uses the model name from your LiteLLM config
+        ),
+    )
+    print(f"<<< Agent Response: {result.final_output}")
+
+
+asyncio.run(test_anthropic_agent())
+```
+
+## 6. Complete Working Example
+
+Here's a full end-to-end script you can copy and run:
+
+```python showLineNumbers title="complete_agent.py"
+from __future__ import annotations
+
+import asyncio
+import os
+
+from openai import AsyncOpenAI
+
+from agents import (
+    Agent,
+    Model,
+    ModelProvider,
+    OpenAIChatCompletionsModel,
+    RunConfig,
+    Runner,
+    function_tool,
+    set_tracing_disabled,
+)
+
+# Point to LiteLLM proxy
+BASE_URL = os.getenv("LITELLM_BASE_URL") or "http://localhost:4000"
+API_KEY = os.getenv("LITELLM_API_KEY") or "sk-1234"
+MODEL_NAME = os.getenv("MODEL_NAME") or "bedrock-claude-sonnet-4"
+
+client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
+set_tracing_disabled(disabled=True)
+
+
+class LiteLLMModelProvider(ModelProvider):
+    def get_model(self, model_name: str | None) -> Model:
+        return OpenAIChatCompletionsModel(
+            model=model_name or MODEL_NAME,
+            openai_client=client,
+        )
+
+
+LITELLM_MODEL_PROVIDER = LiteLLMModelProvider()
+
+
+@function_tool
+def get_weather(city: str) -> str:
+    """Retrieves the current weather report for a specified city."""
+    print(f"[debug] getting weather for {city}")
+
+    mock_weather_db = {
+        "new york": "The weather in New York is sunny with a temperature of 25°C.",
+        "london": "It's cloudy in London with a temperature of 15°C.",
+        "tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
+    }
+
+    city_normalized = city.lower()
+    if city_normalized in mock_weather_db:
+        return mock_weather_db[city_normalized]
+    else:
+        return f"Sorry, I don't have weather information for '{city}'."
+
+
+async def main():
+    agent = Agent(
+        name="Assistant",
+        instructions="You are a helpful weather assistant. "
+                     "Use the 'get_weather' tool for city weather requests. "
+                     "Present information clearly and concisely.",
+        tools=[get_weather],
+    )
+
+    # Run with the default model (bedrock-claude-sonnet-4)
+    result = await Runner.run(
+        agent,
+        "What's the weather in Tokyo?",
+        run_config=RunConfig(model_provider=LITELLM_MODEL_PROVIDER),
+    )
+    print(result.final_output)
+
+    # Switch to a different model by passing model in RunConfig
+    result = await Runner.run(
+        agent,
+        "What's the weather in London?",
+        run_config=RunConfig(
+            model_provider=LITELLM_MODEL_PROVIDER,
+            model="gpt-4o",
+        ),
+    )
+    print(result.final_output)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## Why Use LiteLLM with Agents SDK?
+
+| Feature | Benefit |
+|---------|---------|
+| **Multi-Provider** | Use the same agent code with OpenAI, Bedrock, Azure, Vertex AI, etc. |
+| **Cost Tracking** | Track spending across all agent conversations |
+| **Rate Limiting** | Set budgets and limits on agent usage |
+| **Load Balancing** | Distribute requests across multiple API keys or regions |
+| **Fallbacks** | Automatically retry with different models if one fails |
+
+## Related Resources
+
+- [OpenAI Agents SDK Documentation](https://openai.github.io/openai-agents-python/)
+- [LiteLLM Proxy Quick Start](../proxy/quick_start)

docs/my-website/docs/tutorials/openclaw_integration.md

@@ -0,0 +1,210 @@
+---
+sidebar_label: "OpenClaw"
+---
+
+# OpenClaw + LiteLLM Integration
+
+[OpenClaw](https://openclaw.ai) is a self-hosted AI assistant that connects chat apps (WhatsApp, Telegram, Discord, and more) to LLM providers. By routing OpenClaw through LiteLLM Proxy, you get access to 100+ providers, cost tracking, spend limits, and automatic failover — all from a single gateway.
+
+## What you'll set up
+
+```
+Chat apps → OpenClaw Gateway → LiteLLM Proxy → LLM Providers (OpenAI, Anthropic, etc.)
+```
+
+## Prerequisites
+
+| Requirement | How to get it |
+|---|---|
+| **Node.js 22+** | `node --version` — install from [nodejs.org](https://nodejs.org) if needed |
+| **Python 3.8+** | `python --version` |
+| **At least one LLM API key** | OpenAI, Anthropic, Gemini, etc. |
+
+## Step 1 — Install LiteLLM Proxy
+
+```bash
+pip install 'litellm[proxy]'
+```
+
+## Step 2 — Create a LiteLLM config file
+
+Create a config file  `litellm_config.yaml` with the models you want to use. Here's an example with OpenAI:
+
+```yaml title="litellm_config.yaml"
+model_list:
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+general_settings:
+  master_key: sk-your-secret-key  # pick any value — this is YOUR proxy password
+```
+
+:::tip Multi-provider example
+You can add as many models as you want from different providers:
+
+```yaml title="litellm_config.yaml"
+model_list:
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+  - model_name: claude-sonnet
+    litellm_params:
+      model: anthropic/claude-sonnet-4-20250514
+      api_key: os.environ/ANTHROPIC_API_KEY
+
+  - model_name: gemini-flash
+    litellm_params:
+      model: gemini/gemini-2.0-flash
+      api_key: os.environ/GEMINI_API_KEY
+
+general_settings:
+  master_key: sk-your-secret-key
+```
+
+See [LiteLLM proxy config docs](https://docs.litellm.ai/docs/proxy/configs) for all options.
+:::
+
+## Step 3 — Start the proxy
+
+Make sure your API key(s) are available as environment variables (via `export`, `.env` file, or however you manage secrets), then start the proxy:
+
+```bash
+litellm --config litellm_config.yaml --port 4000
+```
+
+## Step 4 — Install OpenClaw
+
+```bash
+# macOS / Linux
+curl -fsSL https://openclaw.ai/install.sh | bash
+```
+
+:::note Windows
+On Windows, use PowerShell: `iwr -useb https://openclaw.ai/install.ps1 | iex`
+
+WSL2 is recommended over native Windows.
+:::
+
+## Step 5 — Connect OpenClaw to LiteLLM
+
+Run the onboarding wizard:
+
+```bash
+openclaw onboard --install-daemon
+```
+
+When prompted:
+
+1. Choose **QuickStart** or **Manual** as the onboarding mode (both work — Manual gives you more options for gateway settings)
+2. Select **LiteLLM** as the model/auth provider
+3. Enter your LiteLLM `master_key` from Step 2 and set the base URL to your proxy address (e.g., `http://localhost:4000`)
+4. When asked for the default model, choose **Enter model manually** and type the model name from your `litellm_config.yaml` (e.g., `litellm/gpt-4o`)
+
+You can also set or change the model after onboarding:
+
+```bash
+openclaw models set litellm/gpt-4o
+```
+
+For scripted / CI environments, you can skip the prompts entirely:
+
+```bash
+openclaw onboard --non-interactive --accept-risk \
+  --auth-choice litellm-api-key \
+  --litellm-api-key "sk-your-secret-key" \
+  --custom-base-url "http://localhost:4000" \
+  --install-daemon --skip-channels --skip-skills
+```
+
+## Step 6 — Verify
+
+Check the gateway is healthy:
+
+```bash
+openclaw health
+```
+
+Then send a test message:
+
+```bash
+openclaw dashboard                                           # web UI
+openclaw tui                                                 # terminal UI
+openclaw agent --agent main -m "Hello, what model are you?"  # one-shot CLI
+```
+
+If you get a response from your model, the integration is working.
+
+Check which model is active:
+
+```bash
+openclaw models status
+```
+
+## Config reference
+
+After onboarding, OpenClaw stores the LiteLLM provider config in `~/.openclaw/openclaw.json`. The relevant sections are something like this:
+
+```json5 title="~/.openclaw/openclaw.json (excerpt)"
+{
+  "models": {
+    "providers": {
+      "litellm": {
+        "baseUrl": "http://localhost:4000",
+        "apiKey": "sk-your-secret-key",
+        "api": "openai-completions",
+        "models": [
+          {
+            "id": "gpt-4o",
+            "name": "GPT-4o via LiteLLM"
+          }
+        ]
+      }
+    }
+  },
+  "agents": {
+    "defaults": {
+      "model": { "primary": "litellm/gpt-4o" }
+    }
+  }
+}
+```
+
+You can edit this file directly to add more models or change the `baseUrl`. OpenClaw hot-reloads changes automatically.
+
+## Troubleshooting
+
+**Connection refused / proxy not reachable**
+
+Make sure the LiteLLM proxy is running and that the `baseUrl` in your OpenClaw config matches:
+
+```bash
+curl http://localhost:4000/health -H "Authorization: Bearer sk-your-secret-key"
+```
+
+**Wrong model or "Invalid model name"**
+
+The model name in OpenClaw must match a `model_name` from your `litellm_config.yaml`. Switch the active model with:
+
+```bash
+openclaw models set litellm/gpt-4o
+```
+
+**Gateway pairing issues after reinstall**
+
+If the CLI can't connect to the gateway after a reinstall, stop the service and reinstall it:
+
+```bash
+openclaw gateway stop
+openclaw gateway install
+```
+
+## References
+
+- [OpenClaw docs](https://docs.openclaw.ai)
+- [OpenClaw LiteLLM provider docs](https://docs.openclaw.ai/providers/litellm)
+- [OpenClaw model providers](https://docs.openclaw.ai/concepts/model-providers)
+- [LiteLLM proxy configuration](https://docs.litellm.ai/docs/proxy/configs)

docs/my-website/release_notes/v1.81.14.md

@@ -1,5 +1,5 @@
 ---
-title: "v1.81.14-stable - Claude Sonnet 4.6, Guardrail Garden & Major Performance Improvements"
+title: "[Preview] v1.81.14 - New Gateway Level Guardrails & Compliance Playground"
 slug: "v1-81-14"
 date: 2026-02-21T00:00:00
 authors:
@@ -27,7 +27,7 @@ import Image from '@theme/IdealImage';
 docker run \
 -e STORE_MODEL_IN_DB=True \
 -p 4000:4000 \
-ghcr.io/berriai/litellm:main-v1.81.14-stable
+ghcr.io/berriai/litellm:main-v1.81.14.rc.1
 ```
 
 </TabItem>
@@ -42,39 +42,93 @@ pip install litellm==1.81.14
 
 ## Key Highlights
 
-- **Guardrail Model Garden** - [Easily browse built-in and partner guardrails by use case — competitor blockers, topic filters, keyword lists, GDPR PII, EU AI Act, prompt injection, and more](../../docs/proxy/guardrails/policy_templates)
-- **3 new built-in guardrails** - [Competitor name blocker, topic blocker, and insults/keyword filter — all run at the gateway level, no external API, configurable per-team or key](../../docs/proxy/guardrails)
-- **Compliance Playground** - [Run your guardrails against your own datasets or pre-built ones to measure performance before going live](../../docs/proxy/guardrails/policy_templates)
-- **Claude Sonnet 4.6 — day 0** - [Full support across Anthropic and Vertex AI: reasoning, computer use, prompt caching, 200K context](../../docs/providers/anthropic)
-- **Major performance batch** - 20+ targeted optimizations across router algorithms, logging overhead, cost calculator, and connection management — meaningfully lower latency and CPU overhead on every request
+- **Guardrail Garden** — [Browse built-in and partner guardrails by use case — competitor blocking, topic filtering, GDPR, prompt injection, and more. Pick a template, customize it, attach it to a team or key.](../../docs/proxy/guardrails/policy_templates)
+- **Compliance Playground** — [Test any guardrail policy against your own traffic before it goes live. See precision, recall, and false positive rate — so you know how it'll behave in production.](../../docs/proxy/guardrails/policy_templates)
+- **3 new zero-cost built-in guardrails** — [Competitor name blocker, topic blocker, and insults filter — all gateway-level, &lt;0.1ms latency, no external API, configurable per-team or key](../../docs/proxy/guardrails)
+- **Store Model in DB Settings via UI** - [Configure model storage directly in the Admin UI without editing config files or restarting the proxy—perfect for cloud deployments](../../docs/proxy/ui_store_model_db_setting)
+- **Claude Sonnet 4.6 — day 0** — [Full support across Anthropic and Vertex AI: reasoning, computer use, prompt caching, 200K context](../../docs/providers/anthropic)
+- **20+ performance optimizations** — Faster routing, lower logging overhead, reduced cost-calculator latency, and connection pool fixes — meaningfully less CPU and latency on every request
 
 ---
 
-## Guardrail Model Garden
 
-Browse built-in and partner guardrails organized by use case — competitor blocking, topic filtering, keyword lists, GDPR/EU AI Act compliance, prompt injection detection, and more. Pick a template, customize the parameters (keyword lists, blocked topics, score thresholds), and attach it to a team or key.
+### Guardrail Garden
 
-## 3 New Built-in Guardrails
+AI Platform Admins can now browse built-in and partner guardrails from the Guardrail Garden. Guardrails are organized by use case — blocking financial advice, filtering insults, detecting competitor mentions, and more — so you can find the right one and deploy it in a few clicks.
 
-A competitor name blocker, a topic blocker (keyword and embedding-based), and an insults/keyword filter. All run at the gateway level with no external API call. They're configurable per-team or per-key, and you can swap in AWS Bedrock Guardrails or Azure Content Safety on the same endpoint without changing your application code.
+![Guardrail Garden](../img/release_notes/guardrail_garden.png)
 
-## Compliance Playground
+### 3 New Built-in Guardrails
 
-Upload your own dataset or use a pre-built one to measure how a guardrail policy performs before it goes live. See precision, recall, and false positive rate on your actual traffic patterns — so you know how the policy will behave in production before you deploy it.
+This release brings 3 new built-in guardrails that run directly on the gateway. This is great for AI Gateway Admins who need low latency, zero cost guardrails for their scenarios.
 
-### Eval results
+- **Denied Financial Advice** — detects requests for personalized financial advice, investment recommendations, or financial planning
+- **Denied Insults** — detects insults, name-calling, and personal attacks directed at the chatbot, staff, or other people
+- **Competitor Name Blocker** — detects mentions of competitor brands in responses
 
-We benchmark every built-in guardrail against labeled datasets before shipping. Results for the two policies most relevant to topic and keyword blocking (207 investment-question cases, 299 insult cases):
+These guardrails are built for production and on our benchmarks had a 100% Recall and Precision.
+
+### Store Model in DB Settings via UI
+
+Previously, the `store_model_in_db` setting could only be configured in `proxy_config.yaml` under `general_settings`, requiring a proxy restart to take effect. Now you can enable or disable this setting directly from the Admin UI without any restarts. This is especially useful for cloud deployments where you don't have direct access to config files or want to avoid downtime. Enable `store_model_in_db` to move model definitions from your YAML into the database—reducing config complexity, improving scalability, and enabling dynamic model management across multiple proxy instances.
+
+![Store model in DB Setting](../img/ui_store_model_in_db.png)
+
+
+#### Eval results
+
+We benchmarked our new built-in guardrails against labeled datasets before shipping. You can see the results for Denied Financial Advice (207 cases) and Denied Insults (299 cases):
 
 | Guardrail | Precision | Recall | F1 | Latency p50 | Cost/req |
 |-----------|-----------|--------|----|-------------|----------|
-| Block investment questions | 100% | 100% | 100% | &lt;0.1ms | $0 |
-| Block insults / keywords | 100% | 100% | 100% | &lt;0.1ms | $0 |
+| Denied Financial Advice | 100% | 100% | 100% | &lt;0.1ms | $0 |
+| Denied Insults | 100% | 100% | 100% | &lt;0.1ms | $0 |
+
+100% precision means zero false positives — no legitimate messages were incorrectly blocked. 100% recall means zero false negatives — every message that should have been blocked was caught.
+
+
+### Compliance Playground
+
+The Compliance Playground lets you test any guardrail against our pre-built eval datasets or your own custom datasets, so you can see precision, recall, and false positive rate before rolling it out to production.
+
+![Compliance Playground](../img/release_notes/compliance_playground.png)
 
-For reference, ONNX embedding approaches on the same eval set hit 95–98% precision at 2–20ms latency and require additional dependencies. The built-in content filter uses no ML model — just structured YAML rules with layered matching — so there's nothing to download, no API key needed, and latency is effectively zero.
 
 ---
 
+## Performance & Reliability — Up to 13% Lower Latency
+
+<Image img={require('../img/release_notes/v1_81_14_perf.png')} />
+
+This release cuts latency across all percentiles through 20+ micro-optimizations across logging, cost calculation, routing, and connection management. See [benchmarking](../../docs/benchmarks) for more info about how to benchmark yourself. 
+
+- **Mean latency:** 78.4 ms → **70.3 ms** (−10.3%)
+- **p50 latency:** 64.8 ms → **57.3 ms** (−11.7%)
+- **p99 latency:** 288.9 ms → **250.0 ms** (−13.4%)
+
+**Streaming Connection Pool Fix**
+
+Fixed a 3-fold connection leak that caused TCP connection starvation under streaming workloads: the aiohttp transport wasn't closing connections, no `finally` blocks were calling close on disconnect, and a Uvicorn bug prevented disconnect signaling. [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
+
+```mermaid
+graph LR
+    A[Client Disconnects] --> B[Stream Abandoned]
+    B --> C{Connection cleaned up?}
+    C -->|Before| D["❌ No — connection leaked"]
+    C -->|After| E["✅ Yes — connection returned to pool"]
+```
+
+**Redis Connection Pool Reliability**
+
+Fixed 4 separate connection pool bugs to make how we use Redis more reliable. The most important change was on pools being leaked on cache expiry and the other fixes are detailed here in [PR #21717](https://github.com/BerriAI/litellm/pull/21717).
+
+```mermaid
+graph LR
+    A[Cache Entry Expires] --> B{Pool cleanup?}
+    B -->|Before| C["❌ New untracked pool created — leaked"]
+    B -->|After| D["✅ Pool closed on eviction"]
+```
+
 ---
 
 ## New Providers and Endpoints
@@ -417,6 +471,7 @@ For reference, ONNX embedding approaches on the same eval set hit 95–98% preci
 
 - Fix Redis connection pool reliability — prevent connection exhaustion under load - [PR #21717](https://github.com/BerriAI/litellm/pull/21717)
 - Fix Prisma connection self-heal for auth and runtime reconnection (reverted, will be re-introduced with fixes) - [PR #21706](https://github.com/BerriAI/litellm/pull/21706)
+- Close streaming connections to prevent connection pool exhaustion - [PR #21213](https://github.com/BerriAI/litellm/pull/21213)
 - Make `PodLockManager.release_lock` atomic compare-and-delete - [PR #21226](https://github.com/BerriAI/litellm/pull/21226)
 
 ---

docs/my-website/sidebars.js

@@ -161,10 +161,12 @@ const sidebars = {
           ]
         },
         "tutorials/opencode_integration",
+        "tutorials/openclaw_integration",
         "tutorials/cost_tracking_coding",
         "tutorials/cursor_integration",
         "tutorials/github_copilot_integration",
         "tutorials/litellm_gemini_cli",
+        "tutorials/google_genai_sdk",
         "tutorials/litellm_qwen_code_cli",
         "tutorials/openai_codex"
       ]
@@ -179,6 +181,7 @@ const sidebars = {
         slug: "/agent_sdks"
       },
       items: [
+        "tutorials/openai_agents_sdk",
         "tutorials/claude_agent_sdk",
         "tutorials/copilotkit_sdk",
         "tutorials/google_adk",
@@ -334,6 +337,7 @@ const sidebars = {
                 "proxy/ui_credentials",
                 "proxy/ai_hub",
                 "proxy/model_compare_ui",
+                "proxy/ui_store_model_db_setting",
               ]
             },
             {
@@ -417,6 +421,7 @@ const sidebars = {
             "proxy/dynamic_rate_limit",
             "proxy/rate_limit_tiers",
             "proxy/temporary_budget_increase",
+            "proxy/budget_reset_and_tz",
           ],
         },
         "proxy/caching",

enterprise/LICENSE.md

@@ -7,7 +7,7 @@ With regard to the BerriAI Software:
 This software and associated documentation files (the "Software") may only be
 used in production, if you (and any entity that you represent) have agreed to,
 and are in compliance with, the BerriAI Subscription Terms of Service, available
-via [call](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) or email (info@berri.ai) (the "Enterprise Terms"), or other
+via [call](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions) or email (info@berri.ai) (the "Enterprise Terms"), or other
 agreement governing the use of the Software, as agreed by you and BerriAI,
 and otherwise have a valid BerriAI Enterprise license for the
 correct number of user seats. Subject to the foregoing sentence, you are free to

enterprise/README.md

@@ -4,6 +4,6 @@ Code in this folder is licensed under a commercial license. Please review the [L
 
 **These features are covered under the LiteLLM Enterprise contract**
 
-👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat?month=2024-02)
+👉 **Using in an Enterprise / Need specific features ?** Meet with us [here](https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions?month=2024-02)
 
 See all Enterprise Features here 👉 [Docs](https://docs.litellm.ai/docs/proxy/enterprise)

enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py

@@ -16,6 +16,10 @@ from litellm_enterprise.types.enterprise_callbacks.send_emails import (
 
 from litellm._logging import verbose_proxy_logger
 from litellm.caching.caching import DualCache
+from litellm.constants import (
+    EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
+    EMAIL_BUDGET_ALERT_TTL,
+)
 from litellm.integrations.custom_logger import CustomLogger
 from litellm.integrations.email_templates.email_footer import EMAIL_FOOTER
 from litellm.integrations.email_templates.key_created_email import (
@@ -24,14 +28,14 @@ from litellm.integrations.email_templates.key_created_email import (
 from litellm.integrations.email_templates.key_rotated_email import (
     KEY_ROTATED_EMAIL_TEMPLATE,
 )
-from litellm.integrations.email_templates.user_invitation_email import (
-    USER_INVITATION_EMAIL_TEMPLATE,
-)
 from litellm.integrations.email_templates.templates import (
     MAX_BUDGET_ALERT_EMAIL_TEMPLATE,
     SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
     TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE,
 )
+from litellm.integrations.email_templates.user_invitation_email import (
+    USER_INVITATION_EMAIL_TEMPLATE,
+)
 from litellm.proxy._types import (
     CallInfo,
     InvitationNew,
@@ -41,10 +45,6 @@ from litellm.proxy._types import (
 )
 from litellm.secret_managers.main import get_secret_bool
 from litellm.types.integrations.slack_alerting import LITELLM_LOGO_URL
-from litellm.constants import (
-    EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE,
-    EMAIL_BUDGET_ALERT_TTL,
-)
 
 
 class BaseEmailLogger(CustomLogger):
@@ -121,10 +121,16 @@ class BaseEmailLogger(CustomLogger):
         )
 
         # Check if API key should be included in email
-        include_api_key = get_secret_bool(secret_name="EMAIL_INCLUDE_API_KEY", default_value=True)
+        include_api_key = get_secret_bool(
+            secret_name="EMAIL_INCLUDE_API_KEY", default_value=True
+        )
         if include_api_key is None:
             include_api_key = True  # Default to True if not set
-        key_token_display = send_key_created_email_event.virtual_key if include_api_key else "[Key hidden for security - retrieve from dashboard]"
+        key_token_display = (
+            send_key_created_email_event.virtual_key
+            if include_api_key
+            else "[Key hidden for security - retrieve from dashboard]"
+        )
 
         email_html_content = KEY_CREATED_EMAIL_TEMPLATE.format(
             email_logo_url=email_params.logo_url,
@@ -162,10 +168,16 @@ class BaseEmailLogger(CustomLogger):
         )
 
         # Check if API key should be included in email
-        include_api_key = get_secret_bool(secret_name="EMAIL_INCLUDE_API_KEY", default_value=True)
+        include_api_key = get_secret_bool(
+            secret_name="EMAIL_INCLUDE_API_KEY", default_value=True
+        )
         if include_api_key is None:
             include_api_key = True  # Default to True if not set
-        key_token_display = send_key_rotated_email_event.virtual_key if include_api_key else "[Key hidden for security - retrieve from dashboard]"
+        key_token_display = (
+            send_key_rotated_email_event.virtual_key
+            if include_api_key
+            else "[Key hidden for security - retrieve from dashboard]"
+        )
 
         email_html_content = KEY_ROTATED_EMAIL_TEMPLATE.format(
             email_logo_url=email_params.logo_url,
@@ -201,7 +213,9 @@ class BaseEmailLogger(CustomLogger):
         )
 
         # Format budget values
-        soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
+        soft_budget_str = (
+            f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
+        )
         spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
         max_budget_info = ""
         if event.max_budget is not None:
@@ -231,13 +245,13 @@ class BaseEmailLogger(CustomLogger):
         """
         # Collect all recipient emails
         recipient_emails: List[str] = []
-        
+
         # Add additional alert emails from team metadata.soft_budget_alert_emails
         if hasattr(event, "alert_emails") and event.alert_emails:
             for email in event.alert_emails:
                 if email and email not in recipient_emails:  # Avoid duplicates
                     recipient_emails.append(email)
-        
+
         # If no recipients found, skip sending
         if not recipient_emails:
             verbose_proxy_logger.warning(
@@ -268,7 +282,9 @@ class BaseEmailLogger(CustomLogger):
         )
 
         # Format budget values
-        soft_budget_str = f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
+        soft_budget_str = (
+            f"${event.soft_budget}" if event.soft_budget is not None else "N/A"
+        )
         spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
         max_budget_info = ""
         if event.max_budget is not None:
@@ -286,7 +302,7 @@ class BaseEmailLogger(CustomLogger):
             base_url=email_params.base_url,
             email_support_contact=email_params.support_contact,
         )
-        
+
         # Send email to all recipients
         await self.send_email(
             from_email=self.DEFAULT_LITELLM_EMAIL,
@@ -313,11 +329,17 @@ class BaseEmailLogger(CustomLogger):
 
         # Format budget values
         spend_str = f"${event.spend}" if event.spend is not None else "$0.00"
-        max_budget_str = f"${event.max_budget}" if event.max_budget is not None else "N/A"
-        
+        max_budget_str = (
+            f"${event.max_budget}" if event.max_budget is not None else "N/A"
+        )
+
         # Calculate percentage and alert threshold
         percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
-        alert_threshold_str = f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}" if event.max_budget is not None else "N/A"
+        alert_threshold_str = (
+            f"${event.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE:.2f}"
+            if event.max_budget is not None
+            else "N/A"
+        )
 
         email_html_content = MAX_BUDGET_ALERT_EMAIL_TEMPLATE.format(
             email_logo_url=email_params.logo_url,
@@ -382,7 +404,10 @@ class BaseEmailLogger(CustomLogger):
                 # For non-team alerts, require either max_budget or soft_budget
                 if user_info.max_budget is None and user_info.soft_budget is None:
                     return
-            if user_info.soft_budget is not None and user_info.spend >= user_info.soft_budget:
+            if (
+                user_info.soft_budget is not None
+                and user_info.spend >= user_info.soft_budget
+            ):
                 # Generate cache key based on event type and identifier
                 # Use appropriate ID based on event_group to ensure unique cache keys per entity type
                 if user_info.event_group == Litellm_EntityType.TEAM:
@@ -395,7 +420,7 @@ class BaseEmailLogger(CustomLogger):
                     # For KEY and other types, use token or user_id
                     _id = user_info.token or user_info.user_id or "default_id"
                 _cache_key = f"email_budget_alerts:soft_budget_crossed:{_id}"
-                
+
                 # Check if we've already sent this alert
                 result = await _cache.async_get_cache(key=_cache_key)
                 if result is None:
@@ -420,14 +445,14 @@ class BaseEmailLogger(CustomLogger):
                         event_group=user_info.event_group,
                         alert_emails=user_info.alert_emails,
                     )
-                    
+
                     try:
                         # Use team-specific function for team alerts, otherwise use standard function
                         if user_info.event_group == Litellm_EntityType.TEAM:
                             await self.send_team_soft_budget_alert_email(webhook_event)
                         else:
                             await self.send_soft_budget_alert_email(webhook_event)
-                        
+
                         # Cache the alert to prevent duplicate sends
                         await _cache.async_set_cache(
                             key=_cache_key,
@@ -444,20 +469,27 @@ class BaseEmailLogger(CustomLogger):
         # For max_budget_alert, check if we've already sent an alert
         if type == "max_budget_alert":
             if user_info.max_budget is not None and user_info.spend is not None:
-                alert_threshold = user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
-                
+                alert_threshold = (
+                    user_info.max_budget * EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE
+                )
+
                 # Only alert if we've crossed the threshold but haven't exceeded max_budget yet
-                if user_info.spend >= alert_threshold and user_info.spend < user_info.max_budget:
+                if (
+                    user_info.spend >= alert_threshold
+                    and user_info.spend < user_info.max_budget
+                ):
                     # Generate cache key based on event type and identifier
                     _id = user_info.token or user_info.user_id or "default_id"
                     _cache_key = f"email_budget_alerts:max_budget_alert:{_id}"
-                    
+
                     # Check if we've already sent this alert
                     result = await _cache.async_get_cache(key=_cache_key)
                     if result is None:
                         # Calculate percentage
-                        percentage = int(EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100)
-                        
+                        percentage = int(
+                            EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE * 100
+                        )
+
                         # Create WebhookEvent for max budget alert
                         event_message = f"Max Budget Alert - {percentage}% of Maximum Budget Reached"
                         webhook_event = WebhookEvent(
@@ -478,10 +510,10 @@ class BaseEmailLogger(CustomLogger):
                             projected_spend=user_info.projected_spend,
                             event_group=user_info.event_group,
                         )
-                        
+
                         try:
                             await self.send_max_budget_alert_email(webhook_event)
-                            
+
                             # Cache the alert to prevent duplicate sends
                             await _cache.async_set_cache(
                                 key=_cache_key,
@@ -525,9 +557,14 @@ class BaseEmailLogger(CustomLogger):
         unused_custom_fields = []
 
         # Function to safely get custom value or default
-        def get_custom_or_default(custom_value: Optional[str], default_value: str, field_name: str) -> str:
-            if custom_value is not None:  # Only check premium if trying to use custom value
+        def get_custom_or_default(
+            custom_value: Optional[str], default_value: str, field_name: str
+        ) -> str:
+            if (
+                custom_value is not None
+            ):  # Only check premium if trying to use custom value
                 from litellm.proxy.proxy_server import premium_user
+
                 if premium_user is not True:
                     unused_custom_fields.append(field_name)
                     return default_value
@@ -536,38 +573,48 @@ class BaseEmailLogger(CustomLogger):
 
         # Get parameters, falling back to defaults if custom values aren't allowed
         logo_url = get_custom_or_default(custom_logo, LITELLM_LOGO_URL, "logo URL")
-        support_contact = get_custom_or_default(custom_support, self.DEFAULT_SUPPORT_EMAIL, "support contact")
-        base_url = os.getenv("PROXY_BASE_URL", "http://0.0.0.0:4000")  # Not a premium feature
-        signature = get_custom_or_default(custom_signature, EMAIL_FOOTER, "email signature")
+        support_contact = get_custom_or_default(
+            custom_support, self.DEFAULT_SUPPORT_EMAIL, "support contact"
+        )
+        base_url = os.getenv(
+            "PROXY_BASE_URL", "http://0.0.0.0:4000"
+        )  # Not a premium feature
+        signature = get_custom_or_default(
+            custom_signature, EMAIL_FOOTER, "email signature"
+        )
 
         # Get custom subject template based on email event type
         if email_event == EmailEvent.new_user_invitation:
             subject_template = get_custom_or_default(
                 custom_subject_invitation,
                 self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.new_user_invitation],
-                "invitation subject template"
+                "invitation subject template",
             )
         elif email_event == EmailEvent.virtual_key_created:
             subject_template = get_custom_or_default(
                 custom_subject_key_created,
                 self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.virtual_key_created],
-                "key created subject template"
+                "key created subject template",
             )
         elif email_event == EmailEvent.virtual_key_rotated:
             custom_subject_key_rotated = os.getenv("EMAIL_SUBJECT_KEY_ROTATED", None)
             subject_template = get_custom_or_default(
                 custom_subject_key_rotated,
                 self.DEFAULT_SUBJECT_TEMPLATES[EmailEvent.virtual_key_rotated],
-                "key rotated subject template"
+                "key rotated subject template",
             )
         else:
             subject_template = "LiteLLM: {event_message}"
 
-        subject = subject_template.format(event_message=event_message) if event_message else "LiteLLM Notification"
+        subject = (
+            subject_template.format(event_message=event_message)
+            if event_message
+            else "LiteLLM Notification"
+        )
 
-        recipient_email: Optional[
-            str
-        ] = user_email or await self._lookup_user_email_from_db(user_id=user_id)
+        recipient_email: Optional[str] = (
+            user_email or await self._lookup_user_email_from_db(user_id=user_id)
+        )
         if recipient_email is None:
             raise ValueError(
                 f"User email not found for user_id: {user_id}. User email is required to send email."
@@ -585,11 +632,9 @@ class BaseEmailLogger(CustomLogger):
             warning_msg = (
                 f"Email sent with default values instead of custom values for: {fields_str}. "
                 "This is an Enterprise feature. To use custom email fields, please upgrade to LiteLLM Enterprise. "
-                "Schedule a meeting here: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
-            )
-            verbose_proxy_logger.warning(
-                f"{warning_msg}"
+                "Schedule a meeting here: https://calendly.com/d/cx9p-5yf-2nm/litellm-introductions"
             )
+            verbose_proxy_logger.warning(f"{warning_msg}")
 
         return EmailParams(
             logo_url=logo_url,
@@ -636,44 +681,49 @@ class BaseEmailLogger(CustomLogger):
         if not user_id:
             verbose_proxy_logger.debug("No user_id provided for invitation link")
             return base_url
-            
+
         if not await self._is_prisma_client_available():
             return base_url
-            
+
         # Wait for any concurrent invitation creation to complete
         await self._wait_for_invitation_creation()
-        
+
         # Get or create invitation
         invitation = await self._get_or_create_invitation(user_id)
         if not invitation:
-            verbose_proxy_logger.warning(f"Failed to get/create invitation for user_id: {user_id}")
+            verbose_proxy_logger.warning(
+                f"Failed to get/create invitation for user_id: {user_id}"
+            )
             return base_url
-            
+
         return self._construct_invitation_link(invitation.id, base_url)
 
     async def _is_prisma_client_available(self) -> bool:
         """Check if Prisma client is available"""
         from litellm.proxy.proxy_server import prisma_client
-        
+
         if prisma_client is None:
-            verbose_proxy_logger.debug("Prisma client not found. Unable to lookup invitation")
+            verbose_proxy_logger.debug(
+                "Prisma client not found. Unable to lookup invitation"
+            )
             return False
         return True
 
     async def _wait_for_invitation_creation(self) -> None:
         """
         Wait for any concurrent invitation creation to complete.
-        
+
         The UI calls /invitation/new to generate the invitation link.
         We wait to ensure any pending invitation creation is completed.
         """
         import asyncio
+
         await asyncio.sleep(10)
 
     async def _get_or_create_invitation(self, user_id: str):
         """
         Get existing invitation or create a new one for the user
-        
+
         Returns:
             Invitation object with id attribute, or None if failed
         """
@@ -681,31 +731,41 @@ class BaseEmailLogger(CustomLogger):
             create_invitation_for_user,
         )
         from litellm.proxy.proxy_server import prisma_client
-        
+
         if prisma_client is None:
-            verbose_proxy_logger.error("Prisma client is None in _get_or_create_invitation")
+            verbose_proxy_logger.error(
+                "Prisma client is None in _get_or_create_invitation"
+            )
             return None
-            
+
         try:
             # Try to get existing invitation
-            existing_invitations = await prisma_client.db.litellm_invitationlink.find_many(
-                where={"user_id": user_id},
-                order={"created_at": "desc"},
+            existing_invitations = (
+                await prisma_client.db.litellm_invitationlink.find_many(
+                    where={"user_id": user_id},
+                    order={"created_at": "desc"},
+                )
             )
-            
+
             if existing_invitations and len(existing_invitations) > 0:
-                verbose_proxy_logger.debug(f"Found existing invitation for user_id: {user_id}")
+                verbose_proxy_logger.debug(
+                    f"Found existing invitation for user_id: {user_id}"
+                )
                 return existing_invitations[0]
-            
+
             # Create new invitation if none exists
-            verbose_proxy_logger.debug(f"Creating new invitation for user_id: {user_id}")
+            verbose_proxy_logger.debug(
+                f"Creating new invitation for user_id: {user_id}"
+            )
             return await create_invitation_for_user(
                 data=InvitationNew(user_id=user_id),
                 user_api_key_dict=UserAPIKeyAuth(user_id=user_id),
             )
-            
+
         except Exception as e:
-            verbose_proxy_logger.error(f"Error getting/creating invitation for user_id {user_id}: {e}")
+            verbose_proxy_logger.error(
+                f"Error getting/creating invitation for user_id {user_id}: {e}"
+            )
             return None
 
     def _construct_invitation_link(self, invitation_id: str, base_url: str) -> str:

enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py

@@ -13,6 +13,9 @@ if TYPE_CHECKING:
     from litellm.router import Router
 
 
+CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
+
+
 class CheckBatchCost:
     def __init__(
         self,
@@ -27,6 +30,25 @@ class CheckBatchCost:
         self.prisma_client: PrismaClient = prisma_client
         self.llm_router: Router = llm_router
 
+    async def _get_user_info(self, batch_id, user_id) -> dict:
+        """
+        Look up user email and key alias by user_id for enriching the S3 callback metadata.
+        Returns a dict with user_api_key_user_email and user_api_key_alias (both may be None).
+        """
+        try:
+            user_row = await self.prisma_client.db.litellm_usertable.find_unique(
+                where={"user_id": user_id}
+            )
+            if user_row is None:
+                return {}
+            return {
+                "user_api_key_user_email": getattr(user_row, "user_email", None),
+                "user_api_key_alias": getattr(user_row, "user_alias", None),
+            }
+        except Exception as e:
+            verbose_proxy_logger.error(f"CheckBatchCost: could not look up user {user_id} for batch {batch_id}: {e}")
+            return {}
+
     async def check_batch_cost(self):
         """
         Check if the batch JOB has been tracked.
@@ -48,10 +70,12 @@ class CheckBatchCost:
             get_model_id_from_unified_batch_id,
         )
 
+        # Look for all batches that have not yet been processed by CheckBatchCost
         jobs = await self.prisma_client.db.litellm_managedobjecttable.find_many(
             where={
-                "status": {"in": ["validating", "in_progress", "finalizing"]},
                 "file_purpose": "batch",
+                "batch_processed" : False,
+                "status": {"not_in": ["failed", "expired", "cancelled"]}
             }
         )
         completed_jobs = []
@@ -107,6 +131,21 @@ class CheckBatchCost:
                     f"Batch ID: {batch_id} is complete, tracking cost and usage"
                 )
 
+                # aretrieve_batch is called with the raw provider batch ID, so response.id
+                # is the raw provider value (e.g. "batch_20260223-0518.234"). We need the
+                # unified base64 ID in the S3 log so downstream consumers can correlate it
+                # back to the batch they submitted via the proxy.
+                #
+                # CheckBatchCost builds its own LiteLLMLogging object (logging_obj below) and
+                # calls async_success_handler(result=response) directly. That handler calls
+                # _build_standard_logging_payload(response, ...) which reads response.id at
+                # that point — so setting response.id here is sufficient.
+                #
+                # The HTTP endpoint does this substitution via the managed files hook
+                # (async_post_call_success_hook). CheckBatchCost bypasses that hook entirely,
+                # so we do it explicitly here.
+                response.id = job.unified_object_id
+
                 # This background job runs as default_user_id, so going through the HTTP endpoint
                 # would trigger check_managed_file_id_access and get 403. Instead, extract the raw
                 # provider file ID and call afile_content directly with deployment credentials.
@@ -171,11 +210,21 @@ class CheckBatchCost:
                     function_id=str(uuid.uuid4()),
                 )
 
+                creator_user_id = job.created_by
+                user_info = await self._get_user_info(batch_id, job.created_by)
+
                 logging_obj.update_environment_variables(
                     litellm_params={
+                        # set the user-agent header so that S3 callback consumers can easily identify CheckBatchCost callbacks
+                        "proxy_server_request": {
+                            "headers": {
+                                "user-agent": CHECK_BATCH_COST_USER_AGENT,
+                            }
+                        },
                         "metadata": {
-                            "user_api_key_user_id": job.created_by or "default-user-id",
-                        }
+                            "user_api_key_user_id": creator_user_id,
+                            **user_info,
+                        },
                     },
                     optional_params={},
                 )
@@ -191,8 +240,7 @@ class CheckBatchCost:
                 completed_jobs.append(job)
 
             if len(completed_jobs) > 0:
-                # mark the jobs as complete
                 await self.prisma_client.db.litellm_managedobjecttable.update_many(
                     where={"id": {"in": [job.id for job in completed_jobs]}},
-                    data={"status": "complete"},
+                    data={"batch_processed": True, "status": "complete"},
                 )

enterprise/litellm_enterprise/proxy/hooks/managed_files.py

@@ -1086,11 +1086,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
         self, file_id: str
     ) -> List[Dict[str, Any]]:
         """
-        Find batches in non-terminal states that reference this file.
-        
-        Non-terminal states: validating, in_progress, finalizing
-        Terminal states: completed, complete, failed, expired, cancelled
-        
+        Find batches that reference this file and still need cost tracking.
+        Find batches that are in non-terminal state and have not yet been processed by CheckBatchCost.
         Args:
             file_id: The unified file ID to check
             
@@ -1121,7 +1118,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
         batches = await self.prisma_client.db.litellm_managedobjecttable.find_many(
             where={
                 "file_purpose": "batch",
-                "status": {"in": ["validating", "in_progress", "finalizing"]},
+                "batch_processed": False,
+                "status": {"not_in": ["failed", "expired", "cancelled"]}
             },
             take=MAX_MATCHES_TO_RETURN,
             order={"created_at": "desc"},
@@ -1205,7 +1203,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
             
             error_message += (
                 f"To delete this file before complete cost tracking, please delete or cancel the referencing batch(es) first. "
-                f"Alternatively, wait for all batches to complete processing."
+                f"Alternatively, wait for all batches to complete and for cost to be computed (batch_processed=true)."
             )
             
             raise HTTPException(

license_cache.json

@@ -0,0 +1,9 @@
+{
+  "tornado:6.5.3": "Apache-2.0",
+  "redisvl:0.4.1": "MIT",
+  "google-cloud-iam:2.19.1": "Apache 2.0",
+  "google-genai:1.37.0": "Apache-2.0",
+  "azure-keyvault:4.2.0": "MIT License",
+  "soundfile:0.12.1": "BSD 3-Clause License",
+  "openapi-core:0.21.0": "BSD-3-Clause"
+}

litellm-proxy-extras/litellm_proxy_extras/migrations/20260214124140_baseline_diff/migration.sql

@@ -1,2 +0,0 @@
--- This is an empty migration.
-

litellm-proxy-extras/litellm_proxy_extras/migrations/20260219181415_baseline_diff/migration.sql

@@ -0,0 +1,60 @@
+-- CreateTable
+CREATE TABLE "LiteLLM_DailyGuardrailMetrics" (
+    "guardrail_id" TEXT NOT NULL,
+    "date" TEXT NOT NULL,
+    "requests_evaluated" BIGINT NOT NULL DEFAULT 0,
+    "passed_count" BIGINT NOT NULL DEFAULT 0,
+    "blocked_count" BIGINT NOT NULL DEFAULT 0,
+    "flagged_count" BIGINT NOT NULL DEFAULT 0,
+    "avg_score" DOUBLE PRECISION,
+    "avg_latency_ms" DOUBLE PRECISION,
+    "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updated_at" TIMESTAMP(3) NOT NULL,
+
+    CONSTRAINT "LiteLLM_DailyGuardrailMetrics_pkey" PRIMARY KEY ("guardrail_id","date")
+);
+
+-- CreateTable
+CREATE TABLE "LiteLLM_DailyPolicyMetrics" (
+    "policy_id" TEXT NOT NULL,
+    "date" TEXT NOT NULL,
+    "requests_evaluated" BIGINT NOT NULL DEFAULT 0,
+    "passed_count" BIGINT NOT NULL DEFAULT 0,
+    "blocked_count" BIGINT NOT NULL DEFAULT 0,
+    "flagged_count" BIGINT NOT NULL DEFAULT 0,
+    "avg_score" DOUBLE PRECISION,
+    "avg_latency_ms" DOUBLE PRECISION,
+    "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updated_at" TIMESTAMP(3) NOT NULL,
+
+    CONSTRAINT "LiteLLM_DailyPolicyMetrics_pkey" PRIMARY KEY ("policy_id","date")
+);
+
+-- CreateTable
+CREATE TABLE "LiteLLM_SpendLogGuardrailIndex" (
+    "request_id" TEXT NOT NULL,
+    "guardrail_id" TEXT NOT NULL,
+    "policy_id" TEXT,
+    "start_time" TIMESTAMP(3) NOT NULL,
+
+    CONSTRAINT "LiteLLM_SpendLogGuardrailIndex_pkey" PRIMARY KEY ("request_id","guardrail_id")
+);
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DailyGuardrailMetrics_date_idx" ON "LiteLLM_DailyGuardrailMetrics"("date");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DailyGuardrailMetrics_guardrail_id_idx" ON "LiteLLM_DailyGuardrailMetrics"("guardrail_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DailyPolicyMetrics_date_idx" ON "LiteLLM_DailyPolicyMetrics"("date");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_DailyPolicyMetrics_policy_id_idx" ON "LiteLLM_DailyPolicyMetrics"("policy_id");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_SpendLogGuardrailIndex_guardrail_id_start_time_idx" ON "LiteLLM_SpendLogGuardrailIndex"("guardrail_id", "start_time");
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_SpendLogGuardrailIndex_policy_id_start_time_idx" ON "LiteLLM_SpendLogGuardrailIndex"("policy_id", "start_time");
+

litellm-proxy-extras/litellm_proxy_extras/migrations/20260221183800_add_policy_versioning/migration.sql

@@ -0,0 +1,17 @@
+-- DropIndex
+DROP INDEX "LiteLLM_PolicyTable_policy_name_key";
+
+-- AlterTable
+ALTER TABLE "LiteLLM_PolicyTable" ADD COLUMN     "is_latest" BOOLEAN NOT NULL DEFAULT true,
+ADD COLUMN     "parent_version_id" TEXT,
+ADD COLUMN     "production_at" TIMESTAMP(3),
+ADD COLUMN     "published_at" TIMESTAMP(3),
+ADD COLUMN     "version_number" INTEGER NOT NULL DEFAULT 1,
+ADD COLUMN     "version_status" TEXT NOT NULL DEFAULT 'production';
+
+-- CreateIndex
+CREATE INDEX "LiteLLM_PolicyTable_policy_name_version_status_idx" ON "LiteLLM_PolicyTable"("policy_name", "version_status");
+
+-- CreateIndex
+CREATE UNIQUE INDEX "LiteLLM_PolicyTable_policy_name_version_number_key" ON "LiteLLM_PolicyTable"("policy_name", "version_number");
+

litellm-proxy-extras/litellm_proxy_extras/migrations/20260222000000_add_batch_processed_to_managed_object_table/migration.sql

@@ -0,0 +1,3 @@
+-- Add batch_processed column to LiteLLM_ManagedObjectTable
+-- Set to true by CheckBatchCost after cost has been computed for a completed batch
+ALTER TABLE "LiteLLM_ManagedObjectTable" ADD COLUMN "batch_processed" BOOLEAN NOT NULL DEFAULT false;

litellm-proxy-extras/litellm_proxy_extras/schema.prisma

@@ -213,53 +213,6 @@ model LiteLLM_DeletedTeamTable {
     @@index([created_at])
 }
 
-// Audit table for deleted teams - preserves spend and team information for historical tracking
-model LiteLLM_DeletedTeamTable {
-    id                      String   @id @default(uuid())
-    team_id                 String   // Original team_id
-    team_alias              String?
-    organization_id         String?
-    object_permission_id    String?
-    admins                  String[]
-    members                 String[]
-    members_with_roles      Json     @default("{}")
-    metadata                Json     @default("{}")
-    max_budget              Float?
-    soft_budget             Float?
-    spend                   Float    @default(0.0)
-    models                  String[]
-    max_parallel_requests   Int?
-    tpm_limit               BigInt?
-    rpm_limit               BigInt?
-    budget_duration         String?
-    budget_reset_at         DateTime?
-    blocked                 Boolean  @default(false)
-    model_spend             Json     @default("{}")
-    model_max_budget        Json     @default("{}")
-    router_settings  Json? @default("{}")
-    team_member_permissions String[] @default([])
-    access_group_ids String[] @default([])
-    policies                String[] @default([])
-    model_id                Int?     // id for LiteLLM_ModelTable -> stores team-level model aliases
-    allow_team_guardrail_config Boolean @default(false)
-
-    // Original timestamps from team creation/updates
-    created_at              DateTime? @map("created_at")
-    updated_at              DateTime? @map("updated_at")
-    
-    // Deletion metadata
-    deleted_at              DateTime  @default(now()) @map("deleted_at")
-    deleted_by              String?   @map("deleted_by") // User who deleted the team
-    deleted_by_api_key      String?   @map("deleted_by_api_key") // API key hash that performed the deletion
-    litellm_changed_by      String?   @map("litellm_changed_by") // From litellm-changed-by header if provided
-    
-    @@index([team_id])
-    @@index([deleted_at])
-    @@index([organization_id])
-    @@index([team_alias])
-    @@index([created_at])
-}
-
 // Track spend, rate limit, budget Users
 model LiteLLM_UserTable {
 		user_id    String @id
@@ -320,6 +273,7 @@ model LiteLLM_MCPServerTable {
   alias        String?
   description  String?
   url          String?
+  spec_path    String?
   transport    String         @default("sse")
   auth_type    String?        
   credentials  Json?          @default("{}")
@@ -859,6 +813,7 @@ model LiteLLM_ManagedObjectTable { // for batches or finetuning jobs which use t
   file_object Json // Stores the OpenAIFileObject
   file_purpose String // either 'batch' or 'fine-tune'
   status String? // check if batch cost has been tracked  
+  batch_processed Boolean @default(false) // set to true by CheckBatchCost after cost is computed
   created_at DateTime @default(now())
   created_by String?
   updated_at DateTime @updatedAt
@@ -912,6 +867,54 @@ model LiteLLM_GuardrailsTable {
   updated_at DateTime @updatedAt
 }
 
+// Daily guardrail metrics for usage dashboard (one row per guardrail per day)
+model LiteLLM_DailyGuardrailMetrics {
+  guardrail_id        String   // logical id; may not FK if guardrail from config
+  date                String   // YYYY-MM-DD
+  requests_evaluated  BigInt   @default(0)
+  passed_count        BigInt   @default(0)
+  blocked_count       BigInt   @default(0)
+  flagged_count       BigInt   @default(0)
+  avg_score           Float?
+  avg_latency_ms      Float?
+  created_at          DateTime @default(now())
+  updated_at          DateTime @updatedAt
+
+  @@id([guardrail_id, date])
+  @@index([date])
+  @@index([guardrail_id])
+}
+
+// Daily policy metrics for usage dashboard (one row per policy per day)
+model LiteLLM_DailyPolicyMetrics {
+  policy_id           String
+  date                String   // YYYY-MM-DD
+  requests_evaluated  BigInt   @default(0)
+  passed_count        BigInt   @default(0)
+  blocked_count       BigInt   @default(0)
+  flagged_count       BigInt   @default(0)
+  avg_score           Float?
+  avg_latency_ms      Float?
+  created_at          DateTime @default(now())
+  updated_at          DateTime @updatedAt
+
+  @@id([policy_id, date])
+  @@index([date])
+  @@index([policy_id])
+}
+
+// Index for fast "last N logs for guardrail/policy" from SpendLogs
+model LiteLLM_SpendLogGuardrailIndex {
+  request_id   String
+  guardrail_id String
+  policy_id    String?   // set when run as part of a policy pipeline
+  start_time   DateTime
+
+  @@id([request_id, guardrail_id])
+  @@index([guardrail_id, start_time])
+  @@index([policy_id, start_time])
+}
+
 // Prompt table for storing prompt configurations
 model LiteLLM_PromptTable {
   id String @id @default(uuid())
@@ -1009,20 +1012,29 @@ model LiteLLM_SkillsTable {
   updated_by     String?
 }
 
-// Policy table for storing guardrail policies
+// Policy table for storing guardrail policies (versioned)
 model LiteLLM_PolicyTable {
-  policy_id        String    @id @default(uuid())
-  policy_name      String    @unique
-  inherit          String?   // Name of parent policy to inherit from
-  description      String?
-  guardrails_add   String[]  @default([])
-  guardrails_remove String[] @default([])
-  condition        Json?     @default("{}")  // Policy conditions (e.g., model matching)
-  pipeline         Json?     // Optional guardrail pipeline (mode + steps[])
-  created_at       DateTime  @default(now())
-  created_by       String?
-  updated_at       DateTime  @default(now()) @updatedAt
-  updated_by       String?
+  policy_id          String    @id @default(uuid())
+  policy_name        String    // No longer @unique; use @@unique([policy_name, version_number])
+  version_number     Int       @default(1)
+  version_status     String    @default("production")  // "draft" | "published" | "production"
+  parent_version_id  String?
+  is_latest          Boolean   @default(true)
+  published_at       DateTime?
+  production_at      DateTime?
+  inherit            String?   // Name of parent policy to inherit from
+  description        String?
+  guardrails_add     String[]  @default([])
+  guardrails_remove  String[]  @default([])
+  condition          Json?     @default("{}")  // Policy conditions (e.g., model matching)
+  pipeline           Json?     // Optional guardrail pipeline (mode + steps[])
+  created_at         DateTime  @default(now())
+  created_by         String?
+  updated_at         DateTime  @default(now()) @updatedAt
+  updated_by         String?
+
+  @@unique([policy_name, version_number])
+  @@index([policy_name, version_status])
 }
 
 // Policy attachment table for defining where policies apply

litellm/__init__.py

@@ -409,6 +409,7 @@ disable_aiohttp_trust_env: bool = (
 force_ipv4: bool = (
     False  # when True, litellm will force ipv4 for all LLM requests. Some users have seen httpx ConnectionError when using ipv6.
 )
+network_mock: bool = False  # When True, use mock transport — no real network calls
 
 ####### STOP SEQUENCE LIMIT #######
 disable_stop_sequence_limit: bool = False  # when True, stop sequence limit is disabled
@@ -618,8 +619,9 @@ def is_openai_finetune_model(key: str) -> bool:
     return key.startswith("ft:") and not key.count(":") > 1
 
 
-def add_known_models():
-    for key, value in model_cost.items():
+def add_known_models(model_cost_map: Optional[Dict] = None):
+    _map = model_cost_map if model_cost_map is not None else model_cost
+    for key, value in _map.items():
         if value.get("litellm_provider") == "openai" and not is_openai_finetune_model(
             key
         ):

litellm/anthropic_beta_headers_config.json

@@ -127,7 +127,7 @@
     "compact-2026-01-12": null,
     "computer-use-2025-01-24": "computer-use-2025-01-24",
     "computer-use-2025-11-24": "computer-use-2025-11-24",
-    "context-1m-2025-08-07": null,
+    "context-1m-2025-08-07": "context-1m-2025-08-07",
     "context-management-2025-06-27": "context-management-2025-06-27",
     "effort-2025-11-24": null,
     "fast-mode-2026-02-01": null,
@@ -179,4 +179,4 @@
     "web-fetch-2025-09-10": "web-fetch-2025-09-10",
     "web-search-2025-03-05": "web-search-2025-03-05"
   }
-}
+}

litellm/caching/caching.py

@@ -108,6 +108,7 @@ class Cache:
         qdrant_collection_name: Optional[str] = None,
         qdrant_quantization_config: Optional[str] = None,
         qdrant_semantic_cache_embedding_model: str = "text-embedding-ada-002",
+        qdrant_semantic_cache_vector_size: Optional[int] = None,
         # GCP IAM authentication parameters
         gcp_service_account: Optional[str] = None,
         gcp_ssl_ca_certs: Optional[str] = None,
@@ -207,6 +208,7 @@ class Cache:
                 similarity_threshold=similarity_threshold,
                 quantization_config=qdrant_quantization_config,
                 embedding_model=qdrant_semantic_cache_embedding_model,
+                vector_size=qdrant_semantic_cache_vector_size,
             )
         elif type == LiteLLMCacheType.LOCAL:
             self.cache = InMemoryCache()

litellm/caching/qdrant_semantic_cache.py

@@ -31,6 +31,7 @@ class QdrantSemanticCache(BaseCache):
         quantization_config=None,
         embedding_model="text-embedding-ada-002",
         host_type=None,
+        vector_size=None,
     ):
         import os
 
@@ -53,6 +54,7 @@ class QdrantSemanticCache(BaseCache):
             raise Exception("similarity_threshold must be provided, passed None")
         self.similarity_threshold = similarity_threshold
         self.embedding_model = embedding_model
+        self.vector_size = vector_size if vector_size is not None else QDRANT_VECTOR_SIZE
         headers = {}
 
         # check if defined as os.environ/ variable
@@ -138,7 +140,7 @@ class QdrantSemanticCache(BaseCache):
             new_collection_status = self.sync_client.put(
                 url=f"{self.qdrant_api_base}/collections/{self.collection_name}",
                 json={
-                    "vectors": {"size": QDRANT_VECTOR_SIZE, "distance": "Cosine"},
+                    "vectors": {"size": self.vector_size, "distance": "Cosine"},
                     "quantization_config": quantization_params,
                 },
                 headers=self.headers,

litellm/cost_calculator.py

@@ -480,6 +480,7 @@ def cost_per_token(  # noqa: PLR0915
                 model=model_without_prefix,
                 custom_llm_provider=custom_llm_provider,
                 usage=usage_block,
+                service_tier=service_tier,
             )
     elif custom_llm_provider == "anthropic":
         return anthropic_cost_per_token(model=model, usage=usage_block)
@@ -500,7 +501,9 @@ def cost_per_token(  # noqa: PLR0915
             model=model, usage=usage_block, response_time_ms=response_time_ms
         )
     elif custom_llm_provider == "gemini":
-        return gemini_cost_per_token(model=model, usage=usage_block)
+        return gemini_cost_per_token(
+            model=model, usage=usage_block, service_tier=service_tier
+        )
     elif custom_llm_provider == "deepseek":
         return deepseek_cost_per_token(model=model, usage=usage_block)
     elif custom_llm_provider == "perplexity":
@@ -704,6 +707,36 @@ def _get_response_model(completion_response: Any) -> Optional[str]:
     return None
 
 
+_GEMINI_TRAFFIC_TYPE_TO_SERVICE_TIER: dict = {
+    # ON_DEMAND_PRIORITY maps to "priority" — selects input_cost_per_token_priority, etc.
+    "ON_DEMAND_PRIORITY": "priority",
+    # FLEX / BATCH maps to "flex" — selects input_cost_per_token_flex, etc.
+    "FLEX": "flex",
+    "BATCH": "flex",
+    # ON_DEMAND is standard pricing — no service_tier suffix applied
+    "ON_DEMAND": None,
+}
+
+
+def _map_traffic_type_to_service_tier(traffic_type: Optional[str]) -> Optional[str]:
+    """
+    Map a Gemini usageMetadata.trafficType value to a LiteLLM service_tier string.
+
+    This allows the same `_priority` / `_flex` cost-key suffix logic used for
+    OpenAI/Azure to work for Gemini and Vertex AI models.
+
+    trafficType values seen in practice
+    ------------------------------------
+    ON_DEMAND          -> standard pricing  (service_tier = None)
+    ON_DEMAND_PRIORITY -> priority pricing  (service_tier = "priority")
+    FLEX / BATCH       -> batch/flex pricing (service_tier = "flex")
+    """
+    if traffic_type is None:
+        return None
+    service_tier = _GEMINI_TRAFFIC_TYPE_TO_SERVICE_TIER.get(traffic_type.upper())
+    return service_tier
+
+
 def _get_usage_object(
     completion_response: Any,
 ) -> Optional[Usage]:
@@ -1145,6 +1178,20 @@ def completion_cost(  # noqa: PLR0915
                             "custom_llm_provider", custom_llm_provider or None
                         )
                         region_name = hidden_params.get("region_name", region_name)
+
+                        # For Gemini/Vertex AI responses, trafficType is stored in
+                        # provider_specific_fields.  Map it to the service_tier used
+                        # by the cost key lookup (_priority / _flex suffixes) so that
+                        # ON_DEMAND_PRIORITY requests are billed at priority prices.
+                        if service_tier is None:
+                            provider_specific = (
+                                hidden_params.get("provider_specific_fields") or {}
+                            )
+                            raw_traffic_type = provider_specific.get("traffic_type")
+                            if raw_traffic_type:
+                                service_tier = _map_traffic_type_to_service_tier(
+                                    raw_traffic_type
+                                )
                 else:
                     if model is None:
                         raise ValueError(

litellm/integrations/SlackAlerting/hanging_request_check.py

@@ -172,4 +172,6 @@ Team Alias: `{hanging_request_data.team_alias}`"""
             level="Medium",
             alert_type=AlertType.llm_requests_hanging,
             alerting_metadata=hanging_request_data.alerting_metadata or {},
+            request_model=hanging_request_data.model,
+            api_base=hanging_request_data.api_base,
         )

litellm/integrations/SlackAlerting/slack_alerting.py

@@ -70,6 +70,7 @@ class SlackAlerting(CustomBatchLogger):
         ] = None,  # if user wants to separate alerts to diff channels
         alerting_args={},
         default_webhook_url: Optional[str] = None,
+        alert_type_config: Optional[Dict[str, dict]] = None,
         **kwargs,
     ):
         if alerting_threshold is None:
@@ -92,6 +93,12 @@ class SlackAlerting(CustomBatchLogger):
         self.hanging_request_check = AlertingHangingRequestCheck(
             slack_alerting_object=self,
         )
+        self.alert_type_config: Dict[str, AlertTypeConfig] = {}
+        if alert_type_config:
+            for key, val in alert_type_config.items():
+                self.alert_type_config[key] = AlertTypeConfig(**val) if isinstance(val, dict) else val
+        self.digest_buckets: Dict[str, DigestEntry] = {}
+        self.digest_lock = asyncio.Lock()
         super().__init__(**kwargs, flush_lock=self.flush_lock)
 
     def update_values(
@@ -102,6 +109,7 @@ class SlackAlerting(CustomBatchLogger):
         alert_to_webhook_url: Optional[Dict[AlertType, Union[List[str], str]]] = None,
         alerting_args: Optional[Dict] = None,
         llm_router: Optional[Router] = None,
+        alert_type_config: Optional[Dict[str, dict]] = None,
     ):
         if alerting is not None:
             self.alerting = alerting
@@ -116,6 +124,9 @@ class SlackAlerting(CustomBatchLogger):
             if not self.periodic_started:
                 asyncio.create_task(self.periodic_flush())
                 self.periodic_started = True
+        if alert_type_config is not None:
+            for key, val in alert_type_config.items():
+                self.alert_type_config[key] = AlertTypeConfig(**val) if isinstance(val, dict) else val
 
         if alert_to_webhook_url is not None:
             # update the dict
@@ -284,6 +295,8 @@ class SlackAlerting(CustomBatchLogger):
                 level="Low",
                 alert_type=AlertType.llm_too_slow,
                 alerting_metadata=alerting_metadata,
+                request_model=model,
+                api_base=api_base,
             )
 
     async def async_update_daily_reports(
@@ -1354,13 +1367,15 @@ Model Info:
 
         return False
 
-    async def send_alert(
+    async def send_alert( # noqa: PLR0915
         self,
         message: str,
         level: Literal["Low", "Medium", "High"],
         alert_type: AlertType,
         alerting_metadata: dict,
         user_info: Optional[WebhookEvent] = None,
+        request_model: Optional[str] = None,
+        api_base: Optional[str] = None,
         **kwargs,
     ):
         """
@@ -1376,6 +1391,8 @@ Model Info:
         Parameters:
             level: str - Low|Medium|High - if calls might fail (Medium) or are failing (High); Currently, no alerts would be 'Low'.
             message: str - what is the alert about
+            request_model: Optional[str] - model name for digest grouping
+            api_base: Optional[str] - api base for digest grouping
         """
         if self.alerting is None:
             return
@@ -1413,6 +1430,44 @@ Model Info:
 
         from datetime import datetime
 
+        # Check if digest mode is enabled for this alert type
+        alert_type_name_str = getattr(alert_type, "value", str(alert_type))
+        _atc = self.alert_type_config.get(alert_type_name_str)
+        if _atc is not None and _atc.digest:
+            # Resolve webhook URL for this alert type (needed for digest entry)
+            if (
+                self.alert_to_webhook_url is not None
+                and alert_type in self.alert_to_webhook_url
+            ):
+                _digest_webhook: Optional[Union[str, List[str]]] = self.alert_to_webhook_url[alert_type]
+            elif self.default_webhook_url is not None:
+                _digest_webhook = self.default_webhook_url
+            else:
+                _digest_webhook = os.getenv("SLACK_WEBHOOK_URL", None)
+            if _digest_webhook is None:
+                raise ValueError("Missing SLACK_WEBHOOK_URL from environment")
+
+            digest_key = f"{alert_type_name_str}:{request_model or ''}:{api_base or ''}"
+
+            async with self.digest_lock:
+                now = datetime.now()
+                if digest_key in self.digest_buckets:
+                    self.digest_buckets[digest_key]["count"] += 1
+                    self.digest_buckets[digest_key]["last_time"] = now
+                else:
+                    self.digest_buckets[digest_key] = DigestEntry(
+                        alert_type=alert_type_name_str,
+                        request_model=request_model or "",
+                        api_base=api_base or "",
+                        first_message=message,
+                        level=level,
+                        count=1,
+                        start_time=now,
+                        last_time=now,
+                        webhook_url=_digest_webhook,
+                    )
+            return  # Suppress immediate alert; will be emitted by _flush_digest_buckets
+
         # Get the current timestamp
         current_time = datetime.now().strftime("%H:%M:%S")
         _proxy_base_url = os.getenv("PROXY_BASE_URL", None)
@@ -1488,6 +1543,72 @@ Model Info:
         await asyncio.gather(*tasks)
         self.log_queue.clear()
 
+    async def _flush_digest_buckets(self):
+        """Flush any digest buckets whose interval has expired.
+
+        For each expired bucket, formats a digest summary message and
+        appends it to the log_queue for delivery via the normal batching path.
+        """
+        from datetime import datetime
+
+        now = datetime.now()
+        flushed_keys: List[str] = []
+
+        async with self.digest_lock:
+            for key, entry in self.digest_buckets.items():
+                alert_type_name = entry["alert_type"]
+                _atc = self.alert_type_config.get(alert_type_name)
+                if _atc is None:
+                    continue
+                elapsed = (now - entry["start_time"]).total_seconds()
+                if elapsed < _atc.digest_interval:
+                    continue
+
+                # Build digest summary message
+                start_ts = entry["start_time"].strftime("%H:%M:%S")
+                end_ts = entry["last_time"].strftime("%H:%M:%S")
+                start_date = entry["start_time"].strftime("%Y-%m-%d")
+                end_date = entry["last_time"].strftime("%Y-%m-%d")
+                formatted_message = (
+                    f"Alert type: `{alert_type_name}` (Digest)\n"
+                    f"Level: `{entry['level']}`\n"
+                    f"Start: `{start_date} {start_ts}`\n"
+                    f"End: `{end_date} {end_ts}`\n"
+                    f"Count: `{entry['count']}`\n\n"
+                    f"Message: {entry['first_message']}"
+                )
+                _proxy_base_url = os.getenv("PROXY_BASE_URL", None)
+                if _proxy_base_url is not None:
+                    formatted_message += f"\n\nProxy URL: `{_proxy_base_url}`"
+
+                payload = {"text": formatted_message}
+                headers = {"Content-type": "application/json"}
+                webhook_url = entry["webhook_url"]
+
+                if isinstance(webhook_url, list):
+                    for url in webhook_url:
+                        self.log_queue.append(
+                            {"url": url, "headers": headers, "payload": payload, "alert_type": alert_type_name}
+                        )
+                else:
+                    self.log_queue.append(
+                        {"url": webhook_url, "headers": headers, "payload": payload, "alert_type": alert_type_name}
+                    )
+                flushed_keys.append(key)
+
+            for key in flushed_keys:
+                del self.digest_buckets[key]
+
+    async def periodic_flush(self):
+        """Override base periodic_flush to also flush digest buckets."""
+        while True:
+            await asyncio.sleep(self.flush_interval)
+            try:
+                await self._flush_digest_buckets()
+            except Exception as e:
+                verbose_proxy_logger.debug(f"Error flushing digest buckets: {str(e)}")
+            await self.flush_queue()
+
     async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
         """Log deployment latency"""
         try:

litellm/integrations/custom_guardrail.py

@@ -587,9 +587,10 @@ class CustomGuardrail(CustomLogger):
         elif "litellm_metadata" in request_data:
             _append_guardrail_info(request_data["litellm_metadata"])
         else:
-            verbose_logger.warning(
-                "unable to log guardrail information. No metadata found in request_data"
-            )
+            # Ensure guardrail info is always logged (e.g. proxy may not have set
+            # metadata yet). Attach to "metadata" so spend log / standard logging see it.
+            request_data["metadata"] = {}
+            _append_guardrail_info(request_data["metadata"])
 
     async def apply_guardrail(
         self,

litellm/integrations/opentelemetry.py

@@ -40,9 +40,7 @@ if TYPE_CHECKING:
     Context = Union[_Context, Any]
     SpanExporter = Union[_SpanExporter, Any]
     UserAPIKeyAuth = Union[_UserAPIKeyAuth, Any]
-    ManagementEndpointLoggingPayload = Union[
-        _ManagementEndpointLoggingPayload, Any
-    ]
+    ManagementEndpointLoggingPayload = Union[_ManagementEndpointLoggingPayload, Any]
 else:
     Span = Any
     Tracer = Any
@@ -76,7 +74,11 @@ class OpenTelemetryConfig:
         # automatically infer "otlp_http" to send traces to the endpoint.
         # This fixes an issue where UI-configured OTEL settings would default
         # to console output instead of sending traces to the configured endpoint.
-        if self.endpoint and isinstance(self.exporter, str) and self.exporter == "console":
+        if (
+            self.endpoint
+            and isinstance(self.exporter, str)
+            and self.exporter == "console"
+        ):
             self.exporter = "otlp_http"
 
         if not self.service_name:
@@ -104,16 +106,12 @@ class OpenTelemetryConfig:
         exporter = os.getenv(
             "OTEL_EXPORTER_OTLP_PROTOCOL", os.getenv("OTEL_EXPORTER", "console")
         )
-        endpoint = os.getenv(
-            "OTEL_EXPORTER_OTLP_ENDPOINT", os.getenv("OTEL_ENDPOINT")
-        )
+        endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", os.getenv("OTEL_ENDPOINT"))
         headers = os.getenv(
             "OTEL_EXPORTER_OTLP_HEADERS", os.getenv("OTEL_HEADERS")
         )  # example: OTEL_HEADERS=x-honeycomb-team=B85YgLm96***"
         enable_metrics: bool = (
-            os.getenv(
-                "LITELLM_OTEL_INTEGRATION_ENABLE_METRICS", "false"
-            ).lower()
+            os.getenv("LITELLM_OTEL_INTEGRATION_ENABLE_METRICS", "false").lower()
             == "true"
         )
         enable_events: bool = (
@@ -121,9 +119,7 @@ class OpenTelemetryConfig:
             == "true"
         )
         service_name = os.getenv("OTEL_SERVICE_NAME", "litellm")
-        deployment_environment = os.getenv(
-            "OTEL_ENVIRONMENT_NAME", "production"
-        )
+        deployment_environment = os.getenv("OTEL_ENVIRONMENT_NAME", "production")
         model_id = os.getenv("OTEL_MODEL_ID", service_name)
 
         if exporter == "in_memory":
@@ -172,9 +168,7 @@ class OpenTelemetry(CustomLogger):
             logging.getLogger(__name__)
 
             # Enable OpenTelemetry logging
-            otel_exporter_logger = logging.getLogger(
-                "opentelemetry.sdk.trace.export"
-            )
+            otel_exporter_logger = logging.getLogger("opentelemetry.sdk.trace.export")
             otel_exporter_logger.setLevel(logging.DEBUG)
 
         # init CustomLogger params
@@ -229,6 +223,7 @@ class OpenTelemetry(CustomLogger):
         sdk_provider_class,
         create_new_provider_fn,
         set_provider_fn,
+        skip_set_global: bool = False,
     ):
         """
         Generic helper to get or create an OpenTelemetry provider (Tracer, Meter, or Logger).
@@ -240,6 +235,7 @@ class OpenTelemetry(CustomLogger):
             sdk_provider_class: The SDK provider class to check for (e.g., TracerProvider from SDK)
             create_new_provider_fn: Function to create a new provider instance
             set_provider_fn: Function to set the provider globally
+            skip_set_global: If True, don't set the provider globally (for dynamic-only providers)
 
         Returns:
             The provider to use (either existing, new, or explicitly provided)
@@ -270,11 +266,15 @@ class OpenTelemetry(CustomLogger):
                 # Don't call set_provider to preserve existing context
             else:
                 # Default proxy provider or unknown type, create our own
-                verbose_logger.debug(
-                    "OpenTelemetry: Creating new %s", provider_name
-                )
+                verbose_logger.debug("OpenTelemetry: Creating new %s", provider_name)
                 provider = create_new_provider_fn()
-                set_provider_fn(provider)
+                if not skip_set_global:
+                    set_provider_fn(provider)
+                else:
+                    verbose_logger.info(
+                        "OpenTelemetry: Created %s but NOT setting it globally (will use dynamic providers per-request)",
+                        provider_name,
+                    )
         except Exception as e:
             # Fallback: create a new provider if something goes wrong
             verbose_logger.debug(
@@ -283,7 +283,8 @@ class OpenTelemetry(CustomLogger):
                 str(e),
             )
             provider = create_new_provider_fn()
-            set_provider_fn(provider)
+            if not skip_set_global:
+                set_provider_fn(provider)
 
         return provider
 
@@ -293,12 +294,15 @@ class OpenTelemetry(CustomLogger):
         from opentelemetry.trace import SpanKind
 
         def create_tracer_provider():
-            provider = TracerProvider(
-                resource=self._get_litellm_resource(self.config)
-            )
+            provider = TracerProvider(resource=self._get_litellm_resource(self.config))
             provider.add_span_processor(self._get_span_processor())
             return provider
 
+        # CRITICAL FIX: For Langfuse OTEL, skip setting global provider to prevent interference
+        skip_global = (
+            hasattr(self, "callback_name") and self.callback_name == "langfuse_otel"
+        )
+
         tracer_provider = self._get_or_create_provider(
             provider=tracer_provider,
             provider_name="TracerProvider",
@@ -306,6 +310,7 @@ class OpenTelemetry(CustomLogger):
             sdk_provider_class=TracerProvider,
             create_new_provider_fn=create_tracer_provider,
             set_provider_fn=trace.set_tracer_provider,
+            skip_set_global=skip_global,
         )
 
         # Grab our tracer from the TracerProvider (not from global context)
@@ -409,14 +414,10 @@ class OpenTelemetry(CustomLogger):
     def log_failure_event(self, kwargs, response_obj, start_time, end_time):
         self._handle_failure(kwargs, response_obj, start_time, end_time)
 
-    async def async_log_success_event(
-        self, kwargs, response_obj, start_time, end_time
-    ):
+    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
         self._handle_success(kwargs, response_obj, start_time, end_time)
 
-    async def async_log_failure_event(
-        self, kwargs, response_obj, start_time, end_time
-    ):
+    async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
         self._handle_failure(kwargs, response_obj, start_time, end_time)
 
     async def async_service_success_hook(
@@ -613,14 +614,37 @@ class OpenTelemetry(CustomLogger):
 
         if dynamic_headers is not None:
             # Create spans using a temporary tracer with dynamic headers
-            tracer_to_use = self._get_tracer_with_dynamic_headers(
-                dynamic_headers
-            )
+            tracer_to_use = self._get_tracer_with_dynamic_headers(dynamic_headers)
             verbose_logger.debug(
-                "Using dynamic headers for this request: %s", dynamic_headers
+                "[OTEL DEBUG] Using DYNAMIC tracer with headers: %s", dynamic_headers
             )
         else:
-            tracer_to_use = self.tracer
+            # For langfuse_otel without dynamic headers, create a provider with env var credentials
+            if hasattr(self, "callback_name") and self.callback_name == "langfuse_otel":
+                # Use the headers from config (which were set from env vars during init)
+                env_var_headers = (
+                    self._get_headers_dictionary(self.OTEL_HEADERS)
+                    if self.OTEL_HEADERS
+                    else {}
+                )
+                if env_var_headers:
+                    tracer_to_use = self._get_tracer_with_dynamic_headers(
+                        env_var_headers
+                    )
+                    verbose_logger.debug(
+                        "[OTEL DEBUG] Using env var credentials for langfuse_otel (master key request)"
+                    )
+                else:
+                    # No env vars set, use global tracer (will be NoOp)
+                    tracer_to_use = self.tracer
+                    verbose_logger.debug(
+                        "[OTEL DEBUG] No credentials available for langfuse_otel"
+                    )
+            else:
+                tracer_to_use = self.tracer
+                verbose_logger.debug(
+                    "[OTEL DEBUG] Using GLOBAL tracer (no dynamic headers)"
+                )
 
         return tracer_to_use
 
@@ -651,9 +675,7 @@ class OpenTelemetry(CustomLogger):
             )
 
         # Create a temporary tracer provider with dynamic headers
-        temp_provider = TracerProvider(
-            resource=self._get_litellm_resource(self.config)
-        )
+        temp_provider = TracerProvider(resource=self._get_litellm_resource(self.config))
         temp_provider.add_span_processor(
             self._get_span_processor(dynamic_headers=dynamic_headers)
         )
@@ -688,6 +710,15 @@ class OpenTelemetry(CustomLogger):
         )
         ctx, parent_span = self._get_span_context(kwargs)
 
+        # CRITICAL FIX: For langfuse_otel, ALWAYS create primary spans
+        # Don't use parent spans from other providers as they cause trace corruption
+        is_langfuse_otel = (
+            hasattr(self, "callback_name") and self.callback_name == "langfuse_otel"
+        )
+        if is_langfuse_otel:
+            parent_span = None  # Ignore parent spans from other providers
+            ctx = None
+
         # Decide whether to create a primary span
         # Always create if no parent span exists (backward compatibility)
         # OR if USE_OTEL_LITELLM_REQUEST_SPAN is explicitly enabled
@@ -707,6 +738,7 @@ class OpenTelemetry(CustomLogger):
             # Ensure proxy-request parent span is annotated with the actual operation kind
             if (
                 parent_span is not None
+                and hasattr(parent_span, "name")
                 and parent_span.name == LITELLM_PROXY_REQUEST_SPAN_NAME
             ):
                 self.set_attributes(parent_span, kwargs, response_obj)
@@ -717,8 +749,9 @@ class OpenTelemetry(CustomLogger):
             span = None
             # Only set attributes if the span is still recording (not closed)
             # Note: parent_span is guaranteed to be not None here
-            parent_span.set_status(Status(StatusCode.OK))
-            self.set_attributes(parent_span, kwargs, response_obj)
+            if hasattr(parent_span, "set_status"):
+                parent_span.set_status(Status(StatusCode.OK))
+                self.set_attributes(parent_span, kwargs, response_obj)
             # Raw-request as direct child of parent_span
             self._maybe_log_raw_request(
                 kwargs, response_obj, start_time, end_time, parent_span
@@ -741,6 +774,7 @@ class OpenTelemetry(CustomLogger):
         # However, proxy-created spans should be closed here
         if (
             parent_span is not None
+            and hasattr(parent_span, "name")
             and parent_span.name == LITELLM_PROXY_REQUEST_SPAN_NAME
         ):
             parent_span.end(end_time=self._to_ns(end_time))
@@ -784,9 +818,7 @@ class OpenTelemetry(CustomLogger):
         metadata = litellm_params.get("metadata") or {}
         generation_name = metadata.get("generation_name")
 
-        raw_span_name = (
-            generation_name if generation_name else RAW_REQUEST_SPAN_NAME
-        )
+        raw_span_name = generation_name if generation_name else RAW_REQUEST_SPAN_NAME
 
         otel_tracer: Tracer = self.get_tracer_to_use_for_request(kwargs)
         raw_span = otel_tracer.start_span(
@@ -811,9 +843,7 @@ class OpenTelemetry(CustomLogger):
         }
 
         std_log = kwargs.get("standard_logging_object")
-        md = getattr(std_log, "metadata", None) or (std_log or {}).get(
-            "metadata", {}
-        )
+        md = getattr(std_log, "metadata", None) or (std_log or {}).get("metadata", {})
         for key in [
             "user_api_key_hash",
             "user_api_key_alias",
@@ -835,9 +865,9 @@ class OpenTelemetry(CustomLogger):
                 common_attrs[f"metadata.{key}"] = str(md[key])
 
         # get hidden params
-        hidden_params = getattr(std_log, "hidden_params", None) or (
-            std_log or {}
-        ).get("hidden_params", {})
+        hidden_params = getattr(std_log, "hidden_params", None) or (std_log or {}).get(
+            "hidden_params", {}
+        )
         if hidden_params:
             common_attrs["hidden_params"] = safe_dumps(hidden_params)
 
@@ -890,9 +920,7 @@ class OpenTelemetry(CustomLogger):
             except ValueError:
                 return None
 
-    def _record_time_to_first_token_metric(
-        self, kwargs: dict, common_attrs: dict
-    ):
+    def _record_time_to_first_token_metric(self, kwargs: dict, common_attrs: dict):
         """Record Time to First Token (TTFT) metric for streaming requests."""
         optional_params = kwargs.get("optional_params", {})
         is_streaming = optional_params.get("stream", False)
@@ -905,10 +933,7 @@ class OpenTelemetry(CustomLogger):
         api_call_start_time = kwargs.get("api_call_start_time", None)
         completion_start_time = kwargs.get("completion_start_time", None)
 
-        if (
-            api_call_start_time is not None
-            and completion_start_time is not None
-        ):
+        if api_call_start_time is not None and completion_start_time is not None:
             # Convert to timestamps if needed (handles datetime, float, and string)
             api_call_start_ts = self._to_timestamp(api_call_start_time)
             completion_start_ts = self._to_timestamp(completion_start_time)
@@ -916,9 +941,7 @@ class OpenTelemetry(CustomLogger):
             if api_call_start_ts is None or completion_start_ts is None:
                 return  # Skip recording if conversion failed
 
-            time_to_first_token_seconds = (
-                completion_start_ts - api_call_start_ts
-            )
+            time_to_first_token_seconds = completion_start_ts - api_call_start_ts
             self._time_to_first_token_histogram.record(
                 time_to_first_token_seconds, attributes=common_attrs
             )
@@ -988,9 +1011,7 @@ class OpenTelemetry(CustomLogger):
             generation_time_seconds = duration_s
 
         if generation_time_seconds > 0:
-            time_per_output_token_seconds = (
-                generation_time_seconds / completion_tokens
-            )
+            time_per_output_token_seconds = generation_time_seconds / completion_tokens
             self._time_per_output_token_histogram.record(
                 time_per_output_token_seconds, attributes=common_attrs
             )
@@ -1052,6 +1073,7 @@ class OpenTelemetry(CustomLogger):
         # TODO: Refactor to use the proper OTEL Logs API instead of directly creating SDK LogRecords
 
         from opentelemetry._logs import SeverityNumber, get_logger
+
         try:
             from opentelemetry.sdk._logs import (  # type: ignore[attr-defined]  # OTEL < 1.39.0
                 LogRecord as SdkLogRecord,
@@ -1188,9 +1210,7 @@ class OpenTelemetry(CustomLogger):
                 value=guardrail_information.get("guardrail_mode"),
             )
 
-            masked_entity_count = guardrail_information.get(
-                "masked_entity_count"
-            )
+            masked_entity_count = guardrail_information.get("masked_entity_count")
             if masked_entity_count is not None:
                 guardrail_span.set_attribute(
                     "masked_entity_count", safe_dumps(masked_entity_count)
@@ -1214,12 +1234,20 @@ class OpenTelemetry(CustomLogger):
         )
         _parent_context, parent_otel_span = self._get_span_context(kwargs)
 
+        # CRITICAL FIX: For langfuse_otel, ALWAYS create primary spans
+        # Don't use parent spans from other providers as they cause trace corruption
+        is_langfuse_otel = (
+            hasattr(self, "callback_name") and self.callback_name == "langfuse_otel"
+        )
+        if is_langfuse_otel:
+            parent_otel_span = None  # Ignore parent spans from other providers
+            _parent_context = None
+
         # Decide whether to create a primary span
         # Always create if no parent span exists (backward compatibility)
         # OR if USE_OTEL_LITELLM_REQUEST_SPAN is explicitly enabled
-        should_create_primary_span = (
-            parent_otel_span is None
-            or get_secret_bool("USE_OTEL_LITELLM_REQUEST_SPAN")
+        should_create_primary_span = parent_otel_span is None or get_secret_bool(
+            "USE_OTEL_LITELLM_REQUEST_SPAN"
         )
 
         if should_create_primary_span:
@@ -1245,9 +1273,7 @@ class OpenTelemetry(CustomLogger):
             if parent_otel_span.is_recording():
                 parent_otel_span.set_status(Status(StatusCode.ERROR))
                 self.set_attributes(parent_otel_span, kwargs, response_obj)
-                self._record_exception_on_span(
-                    span=parent_otel_span, kwargs=kwargs
-                )
+                self._record_exception_on_span(span=parent_otel_span, kwargs=kwargs)
 
         # Create span for guardrail information
         self._create_guardrail_span(kwargs=kwargs, context=_parent_context)
@@ -1257,6 +1283,7 @@ class OpenTelemetry(CustomLogger):
         # However, proxy-created spans should be closed here
         if (
             parent_otel_span is not None
+            and hasattr(parent_otel_span, "name")
             and parent_otel_span.name == LITELLM_PROXY_REQUEST_SPAN_NAME
         ):
             parent_otel_span.end(end_time=self._to_ns(end_time))
@@ -1282,17 +1309,15 @@ class OpenTelemetry(CustomLogger):
                 span.record_exception(exception)
 
             # Get StandardLoggingPayload for structured error information
-            standard_logging_payload: Optional[StandardLoggingPayload] = (
-                kwargs.get("standard_logging_object")
+            standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
+                "standard_logging_object"
             )
 
             if standard_logging_payload is None:
                 return
 
             # Extract error_information from StandardLoggingPayload
-            error_information = standard_logging_payload.get(
-                "error_information"
-            )
+            error_information = standard_logging_payload.get("error_information")
 
             if error_information is None:
                 # Fallback to error_str if error_information is not available
@@ -1382,9 +1407,7 @@ class OpenTelemetry(CustomLogger):
             )
             pass
 
-    def cast_as_primitive_value_type(
-        self, value
-    ) -> Union[str, bool, int, float]:
+    def cast_as_primitive_value_type(self, value) -> Union[str, bool, int, float]:
         """
         Casts the value to a primitive OTEL type if it is not already a primitive type.
 
@@ -1454,8 +1477,8 @@ class OpenTelemetry(CustomLogger):
 
             optional_params = kwargs.get("optional_params", {})
             litellm_params = kwargs.get("litellm_params", {}) or {}
-            standard_logging_payload: Optional[StandardLoggingPayload] = (
-                kwargs.get("standard_logging_object")
+            standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
+                "standard_logging_object"
             )
             if standard_logging_payload is None:
                 raise ValueError("standard_logging_object not found in kwargs")
@@ -1482,8 +1505,8 @@ class OpenTelemetry(CustomLogger):
                     value=safe_dumps(hidden_params),
                 )
             # Cost breakdown tracking
-            cost_breakdown: Optional[CostBreakdown] = (
-                standard_logging_payload.get("cost_breakdown")
+            cost_breakdown: Optional[CostBreakdown] = standard_logging_payload.get(
+                "cost_breakdown"
             )
             if cost_breakdown:
                 for key, value in cost_breakdown.items():
@@ -1696,9 +1719,7 @@ class OpenTelemetry(CustomLogger):
                 "OpenTelemetry logging error in set_attributes %s", str(e)
             )
 
-    def _cast_as_primitive_value_type(
-        self, value
-    ) -> Union[str, bool, int, float]:
+    def _cast_as_primitive_value_type(self, value) -> Union[str, bool, int, float]:
         """
         Casts the value to a primitive OTEL type if it is not already a primitive type.
 
@@ -1776,11 +1797,9 @@ class OpenTelemetry(CustomLogger):
             message = choice.get("message") or {}
             finish_reason = choice.get("finish_reason")
 
-            transformed_msg = (
-                self._transform_messages_to_otel_semantic_conventions(
-                    [message]
-                )[0]
-            )
+            transformed_msg = self._transform_messages_to_otel_semantic_conventions(
+                [message]
+            )[0]
             if finish_reason:
                 transformed_msg["finish_reason"] = finish_reason
 
@@ -1792,9 +1811,7 @@ class OpenTelemetry(CustomLogger):
             self.set_attributes(span, kwargs, response_obj)
             kwargs.get("optional_params", {})
             litellm_params = kwargs.get("litellm_params", {}) or {}
-            custom_llm_provider = litellm_params.get(
-                "custom_llm_provider", "Unknown"
-            )
+            custom_llm_provider = litellm_params.get("custom_llm_provider", "Unknown")
 
             _raw_response = kwargs.get("original_response")
             _additional_args = kwargs.get("additional_args", {}) or {}
@@ -1882,9 +1899,7 @@ class OpenTelemetry(CustomLogger):
         )
 
         litellm_params = kwargs.get("litellm_params", {}) or {}
-        proxy_server_request = (
-            litellm_params.get("proxy_server_request", {}) or {}
-        )
+        proxy_server_request = litellm_params.get("proxy_server_request", {}) or {}
         headers = proxy_server_request.get("headers", {}) or {}
         traceparent = headers.get("traceparent", None)
         _metadata = litellm_params.get("metadata", {}) or {}
@@ -1951,6 +1966,19 @@ class OpenTelemetry(CustomLogger):
             headers=dynamic_headers or self.OTEL_HEADERS
         )
 
+        if dynamic_headers:
+            verbose_logger.debug(
+                "[OTEL DEBUG] Creating span processor with DYNAMIC headers: %s",
+                {
+                    k: v[:20] + "..." if len(str(v)) > 20 else v
+                    for k, v in _split_otel_headers.items()
+                },
+            )
+        else:
+            verbose_logger.debug(
+                "[OTEL DEBUG] Creating span processor with GLOBAL headers"
+            )
+
         if hasattr(
             self.OTEL_EXPORTER, "export"
         ):  # Check if it has the export method that SpanExporter requires
@@ -2034,14 +2062,10 @@ class OpenTelemetry(CustomLogger):
             self.OTEL_HEADERS,
         )
 
-        _split_otel_headers = OpenTelemetry._get_headers_dictionary(
-            self.OTEL_HEADERS
-        )
+        _split_otel_headers = OpenTelemetry._get_headers_dictionary(self.OTEL_HEADERS)
 
         # Normalize endpoint for logs - ensure it points to /v1/logs instead of /v1/traces
-        normalized_endpoint = self._normalize_otel_endpoint(
-            self.OTEL_ENDPOINT, "logs"
-        )
+        normalized_endpoint = self._normalize_otel_endpoint(self.OTEL_ENDPOINT, "logs")
 
         verbose_logger.debug(
             "OpenTelemetry: Log endpoint normalized from %s to %s",
@@ -2129,18 +2153,14 @@ class OpenTelemetry(CustomLogger):
             self.OTEL_HEADERS,
         )
 
-        _split_otel_headers = OpenTelemetry._get_headers_dictionary(
-            self.OTEL_HEADERS
-        )
+        _split_otel_headers = OpenTelemetry._get_headers_dictionary(self.OTEL_HEADERS)
         normalized_endpoint = self._normalize_otel_endpoint(
             self.OTEL_ENDPOINT, "metrics"
         )
 
         if self.OTEL_EXPORTER == "console":
             exporter = ConsoleMetricExporter()
-            return PeriodicExportingMetricReader(
-                exporter, export_interval_millis=5000
-            )
+            return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
 
         elif (
             self.OTEL_EXPORTER == "otlp_http"
@@ -2156,9 +2176,7 @@ class OpenTelemetry(CustomLogger):
                 headers=_split_otel_headers,
                 preferred_temporality={Histogram: AggregationTemporality.DELTA},
             )
-            return PeriodicExportingMetricReader(
-                exporter, export_interval_millis=5000
-            )
+            return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
 
         elif self.OTEL_EXPORTER == "otlp_grpc" or self.OTEL_EXPORTER == "grpc":
             try:
@@ -2176,9 +2194,7 @@ class OpenTelemetry(CustomLogger):
                 headers=_split_otel_headers,
                 preferred_temporality={Histogram: AggregationTemporality.DELTA},
             )
-            return PeriodicExportingMetricReader(
-                exporter, export_interval_millis=5000
-            )
+            return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
 
         else:
             verbose_logger.warning(
@@ -2186,9 +2202,7 @@ class OpenTelemetry(CustomLogger):
                 self.OTEL_EXPORTER,
             )
             exporter = ConsoleMetricExporter()
-            return PeriodicExportingMetricReader(
-                exporter, export_interval_millis=5000
-            )
+            return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
 
     def _normalize_otel_endpoint(
         self, endpoint: Optional[str], signal_type: str

litellm/litellm_core_utils/duration_parser.py

@@ -8,8 +8,9 @@ duration_in_seconds is used in diff parts of the code base, example
 
 import re
 import time
-from datetime import datetime, timedelta, timezone
+from datetime import datetime, timedelta, timezone, tzinfo
 from typing import Optional, Tuple
+from zoneinfo import ZoneInfo
 
 
 def _extract_from_regex(duration: str) -> Tuple[int, str]:
@@ -116,7 +117,7 @@ def get_next_standardized_reset_time(
     - Next reset time at a standardized interval in the specified timezone
     """
     # Set up timezone and normalize current time
-    current_time, timezone = _setup_timezone(current_time, timezone_str)
+    current_time, tz = _setup_timezone(current_time, timezone_str)
 
     # Parse duration
     value, unit = _parse_duration(duration)
@@ -131,7 +132,7 @@ def get_next_standardized_reset_time(
 
     # Handle different time units
     if unit == "d":
-        return _handle_day_reset(current_time, base_midnight, value, timezone)
+        return _handle_day_reset(current_time, base_midnight, value, tz)
     elif unit == "h":
         return _handle_hour_reset(current_time, base_midnight, value)
     elif unit == "m":
@@ -147,22 +148,13 @@ def get_next_standardized_reset_time(
 
 def _setup_timezone(
     current_time: datetime, timezone_str: str = "UTC"
-) -> Tuple[datetime, timezone]:
+) -> Tuple[datetime, tzinfo]:
     """Set up timezone and normalize current time to that timezone."""
     try:
         if timezone_str is None:
-            tz = timezone.utc
+            tz: tzinfo = timezone.utc
         else:
-            # Map common timezone strings to their UTC offsets
-            timezone_map = {
-                "US/Eastern": timezone(timedelta(hours=-4)),  # EDT
-                "US/Pacific": timezone(timedelta(hours=-7)),  # PDT
-                "Asia/Kolkata": timezone(timedelta(hours=5, minutes=30)),  # IST
-                "Asia/Bangkok": timezone(timedelta(hours=7)),  # ICT (Indochina Time)
-                "Europe/London": timezone(timedelta(hours=1)),  # BST
-                "UTC": timezone.utc,
-            }
-            tz = timezone_map.get(timezone_str, timezone.utc)
+            tz = ZoneInfo(timezone_str)
     except Exception:
         # If timezone is invalid, fall back to UTC
         tz = timezone.utc
@@ -190,7 +182,7 @@ def _parse_duration(duration: str) -> Tuple[Optional[int], Optional[str]]:
 
 
 def _handle_day_reset(
-    current_time: datetime, base_midnight: datetime, value: int, timezone: timezone
+    current_time: datetime, base_midnight: datetime, value: int, tz: tzinfo
 ) -> datetime:
     """Handle day-based reset times."""
     # Handle zero value - immediate expiration
@@ -215,7 +207,7 @@ def _handle_day_reset(
                 minute=0,
                 second=0,
                 microsecond=0,
-                tzinfo=timezone,
+                tzinfo=tz,
             )
         else:
             next_reset = datetime(
@@ -226,7 +218,7 @@ def _handle_day_reset(
                 minute=0,
                 second=0,
                 microsecond=0,
-                tzinfo=timezone,
+                tzinfo=tz,
             )
         return next_reset
     else:  # Custom day value - next interval is value days from current

litellm/litellm_core_utils/get_model_cost_map.py

@@ -11,6 +11,7 @@ export LITELLM_LOCAL_MODEL_COST_MAP=True
 import json
 import os
 from importlib.resources import files
+from typing import Optional
 
 import httpx
 
@@ -151,6 +152,37 @@ class GetModelCostMap:
         return response.json()
 
 
+class ModelCostMapSourceInfo:
+    """Tracks the source of the currently loaded model cost map."""
+
+    source: str = "local"  # "local" or "remote"
+    url: Optional[str] = None
+    is_env_forced: bool = False
+    fallback_reason: Optional[str] = None
+
+
+# Module-level singleton tracking the source of the current cost map
+_cost_map_source_info = ModelCostMapSourceInfo()
+
+
+def get_model_cost_map_source_info() -> dict:
+    """
+    Return metadata about where the current model cost map was loaded from.
+
+    Returns a dict with:
+    - source: "local" or "remote"
+    - url: the remote URL attempted (or None for local-only)
+    - is_env_forced: True if LITELLM_LOCAL_MODEL_COST_MAP=True forced local usage
+    - fallback_reason: human-readable reason if remote failed and local was used
+    """
+    return {
+        "source": _cost_map_source_info.source,
+        "url": _cost_map_source_info.url,
+        "is_env_forced": _cost_map_source_info.is_env_forced,
+        "fallback_reason": _cost_map_source_info.fallback_reason,
+    }
+
+
 def get_model_cost_map(url: str) -> dict:
     """
     Public entry point — returns the model cost map dict.
@@ -166,8 +198,15 @@ def get_model_cost_map(url: str) -> dict:
     # Note: can't use get_secret_bool here — this runs during litellm.__init__
     # before litellm._key_management_settings is set.
     if os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", "").lower() == "true":
+        _cost_map_source_info.source = "local"
+        _cost_map_source_info.url = None
+        _cost_map_source_info.is_env_forced = True
+        _cost_map_source_info.fallback_reason = None
         return GetModelCostMap.load_local_model_cost_map()
 
+    _cost_map_source_info.url = url
+    _cost_map_source_info.is_env_forced = False
+
     try:
         content = GetModelCostMap.fetch_remote_model_cost_map(url)
     except Exception as e:
@@ -177,6 +216,8 @@ def get_model_cost_map(url: str) -> dict:
             url,
             str(e),
         )
+        _cost_map_source_info.source = "local"
+        _cost_map_source_info.fallback_reason = f"Remote fetch failed: {str(e)}"
         return GetModelCostMap.load_local_model_cost_map()
 
     # Validate using cached count (cheap int comparison, no file I/O)
@@ -189,6 +230,10 @@ def get_model_cost_map(url: str) -> dict:
             "Using local backup instead. url=%s",
             url,
         )
+        _cost_map_source_info.source = "local"
+        _cost_map_source_info.fallback_reason = "Remote data failed integrity validation"
         return GetModelCostMap.load_local_model_cost_map()
 
+    _cost_map_source_info.source = "remote"
+    _cost_map_source_info.fallback_reason = None
     return content

litellm/litellm_core_utils/llm_cost_calc/utils.py

@@ -200,8 +200,14 @@ def _get_token_base_cost(
     ## CHECK IF ABOVE THRESHOLD
     # Optimization: collect threshold keys first to avoid sorting all model_info keys.
     # Most models don't have threshold pricing, so we can return early.
+    # Exclude service_tier-specific variants (e.g. input_cost_per_token_above_200k_tokens_priority)
+    # so that the threshold detection loop only processes standard keys.  The
+    # service_tier-specific above-threshold key is resolved later via _get_service_tier_cost_key.
     threshold_keys = [
-        k for k in model_info if k.startswith("input_cost_per_token_above_")
+        k
+        for k in model_info
+        if k.startswith("input_cost_per_token_above_")
+        and not any(k.endswith(f"_{st.value}") for st in ServiceTier)
     ]
     if not threshold_keys:
         return (
@@ -224,14 +230,34 @@ def _get_token_base_cost(
                     1000 if "k" in threshold_str else 1
                 )
                 if usage.prompt_tokens > threshold:
+                    # Prefer a service_tier-specific above-threshold key when available,
+                    # e.g. input_cost_per_token_priority_above_200k_tokens for Gemini
+                    # ON_DEMAND_PRIORITY.  Falls back to the standard key automatically
+                    # via _get_cost_per_unit's service_tier fallback logic.
+                    tiered_input_key = (
+                        _get_service_tier_cost_key(
+                            f"input_cost_per_token_above_{threshold_str}_tokens",
+                            service_tier,
+                        )
+                        if service_tier
+                        else key
+                    )
                     prompt_base_cost = cast(
-                        float, _get_cost_per_unit(model_info, key, prompt_base_cost)
+                        float, _get_cost_per_unit(model_info, tiered_input_key, prompt_base_cost)
+                    )
+                    tiered_output_key = (
+                        _get_service_tier_cost_key(
+                            f"output_cost_per_token_above_{threshold_str}_tokens",
+                            service_tier,
+                        )
+                        if service_tier
+                        else f"output_cost_per_token_above_{threshold_str}_tokens"
                     )
                     completion_base_cost = cast(
                         float,
                         _get_cost_per_unit(
                             model_info,
-                            f"output_cost_per_token_above_{threshold_str}_tokens",
+                            tiered_output_key,
                             completion_base_cost,
                         ),
                     )
@@ -517,6 +543,7 @@ def _calculate_input_cost(
     cache_read_cost: float,
     cache_creation_cost: float,
     cache_creation_cost_above_1hr: float,
+    service_tier: Optional[str] = None,
 ) -> float:
     """
     Calculates the input cost for a given model, prompt tokens, and completion tokens.
@@ -528,8 +555,11 @@ def _calculate_input_cost(
 
     ### AUDIO COST
     if prompt_tokens_details["audio_tokens"]:
+        audio_cost_key = _get_service_tier_cost_key(
+            "input_cost_per_audio_token", service_tier
+        )
         prompt_cost += calculate_cost_component(
-            model_info, "input_cost_per_audio_token", prompt_tokens_details["audio_tokens"]
+            model_info, audio_cost_key, prompt_tokens_details["audio_tokens"]
         )
 
     ### IMAGE TOKEN COST
@@ -659,6 +689,7 @@ def generic_cost_per_token(  # noqa: PLR0915
         cache_read_cost=cache_read_cost,
         cache_creation_cost=cache_creation_cost,
         cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
+        service_tier=service_tier,
     )
 
     ## CALCULATE OUTPUT COST

litellm/litellm_core_utils/prompt_templates/common_utils.py

@@ -1136,6 +1136,7 @@ def add_system_prompt_to_messages(
     if merge_with_first_system and messages and messages[0].get("role") == "system":
         first = dict(messages[0])
         existing_content = first.get("content", "")
+        merged_content: Union[str, List[Dict[str, str]]]
         if isinstance(existing_content, str):
             merged_content = f"{system_prompt.strip()}\n\n{existing_content}"
         elif isinstance(existing_content, list):

litellm/litellm_core_utils/streaming_handler.py

@@ -6,7 +6,17 @@ import logging
 import threading
 import time
 import traceback
-from typing import Any, Callable, Dict, List, Optional, Union, cast
+from typing import (
+    Any,
+    AsyncIterator,
+    Callable,
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Union,
+    cast,
+)
 
 import anyio
 import httpx
@@ -151,10 +161,10 @@ class CustomStreamWrapper:
         self.is_function_call = self.check_is_function_call(logging_obj=logging_obj)
         self.created: Optional[int] = None
 
-    def __iter__(self):
+    def __iter__(self) -> Iterator["ModelResponseStream"]:
         return self
 
-    def __aiter__(self):
+    def __aiter__(self) -> AsyncIterator["ModelResponseStream"]:
         return self
 
     async def aclose(self):
@@ -1726,7 +1736,7 @@ class CustomStreamWrapper:
             model_response.choices[0].finish_reason = "tool_calls"
         return model_response
 
-    def __next__(self):  # noqa: PLR0915
+    def __next__(self) -> "ModelResponseStream":  # noqa: PLR0915
         cache_hit = False
         if (
             self.custom_llm_provider is not None
@@ -1748,7 +1758,7 @@ class CustomStreamWrapper:
                     chunk = next(self.completion_stream)
                 if chunk is not None and chunk != b"":
                     print_verbose(
-                        f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk}; custom_llm_provider: {self.custom_llm_provider}"
+                        f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk.decode('utf-8', errors='replace') if isinstance(chunk, bytes) else chunk}; custom_llm_provider: {self.custom_llm_provider}"
                     )
                     response: Optional[ModelResponseStream] = self.chunk_creator(
                         chunk=chunk
@@ -1900,7 +1910,7 @@ class CustomStreamWrapper:
 
         return self.completion_stream
 
-    async def __anext__(self):  # noqa: PLR0915
+    async def __anext__(self) -> "ModelResponseStream":  # noqa: PLR0915
         cache_hit = False
         if (
             self.custom_llm_provider is not None
@@ -1996,9 +2006,7 @@ class CustomStreamWrapper:
                     else:
                         chunk = next(self.completion_stream)
                     if chunk is not None and chunk != b"":
-                        processed_chunk: Optional[
-                            ModelResponseStream
-                        ] = self.chunk_creator(chunk=chunk)
+                        processed_chunk = self.chunk_creator(chunk=chunk)
                         if processed_chunk is None:
                             continue
 

litellm/llms/anthropic/cost_calculation.py

@@ -5,10 +5,50 @@ Helper util for handling anthropic-specific cost calculation
 
 from typing import TYPE_CHECKING, Optional, Tuple
 
-from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
+from litellm.litellm_core_utils.llm_cost_calc.utils import (
+    _get_token_base_cost,
+    _parse_prompt_tokens_details,
+    calculate_cache_writing_cost,
+    generic_cost_per_token,
+)
 
 if TYPE_CHECKING:
     from litellm.types.utils import ModelInfo, Usage
+import litellm
+
+
+def _compute_cache_only_cost(model_info: "ModelInfo", usage: "Usage") -> float:
+    """
+    Return only the cache-related portion of the prompt cost (cache read + cache write).
+
+    These costs must NOT be scaled by geo/speed multipliers because the old
+    explicit ``fast/`` model entries carried unchanged cache rates while
+    multiplying only the regular input/output token costs.
+    """
+    if usage.prompt_tokens_details is None:
+        return 0.0
+
+    prompt_tokens_details = _parse_prompt_tokens_details(usage)
+    _, _, cache_creation_cost, cache_creation_cost_above_1hr, cache_read_cost = (
+        _get_token_base_cost(model_info=model_info, usage=usage)
+    )
+
+    cache_cost = float(prompt_tokens_details["cache_hit_tokens"]) * cache_read_cost
+
+    if (
+        prompt_tokens_details["cache_creation_tokens"]
+        or prompt_tokens_details["cache_creation_token_details"] is not None
+    ):
+        cache_cost += calculate_cache_writing_cost(
+            cache_creation_tokens=prompt_tokens_details["cache_creation_tokens"],
+            cache_creation_token_details=prompt_tokens_details[
+                "cache_creation_token_details"
+            ],
+            cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
+            cache_creation_cost=cache_creation_cost,
+        )
+
+    return cache_cost
 
 
 def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
@@ -22,20 +62,34 @@ def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
     Returns:
         Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
     """
-    model_with_prefix = model
-    
-    # First, prepend inference_geo if present
-    if hasattr(usage, "inference_geo") and usage.inference_geo and usage.inference_geo.lower() not in ["global", "not_available"]:
-        model_with_prefix = f"{usage.inference_geo}/{model_with_prefix}"
-    
-    # Then, prepend speed if it's "fast"
-    if hasattr(usage, "speed") and usage.speed == "fast":
-        model_with_prefix = f"fast/{model_with_prefix}"
-    
     prompt_cost, completion_cost = generic_cost_per_token(
-        model=model_with_prefix, usage=usage, custom_llm_provider="anthropic"
+        model=model, usage=usage, custom_llm_provider="anthropic"
     )
 
+    # Apply provider_specific_entry multipliers for geo/speed routing
+    try:
+        model_info = litellm.get_model_info(model=model, custom_llm_provider="anthropic")
+        provider_specific_entry: dict = model_info.get("provider_specific_entry") or {}
+
+        multiplier = 1.0
+        if (
+            hasattr(usage, "inference_geo")
+            and usage.inference_geo
+            and usage.inference_geo.lower() not in ["global", "not_available"]
+        ):
+            multiplier *= provider_specific_entry.get(
+                usage.inference_geo.lower(), 1.0
+            )
+        if hasattr(usage, "speed") and usage.speed == "fast":
+            multiplier *= provider_specific_entry.get("fast", 1.0)
+
+        if multiplier != 1.0:
+            cache_cost = _compute_cache_only_cost(model_info=model_info, usage=usage)
+            prompt_cost = (prompt_cost - cache_cost) * multiplier + cache_cost
+            completion_cost *= multiplier
+    except Exception:
+        pass
+
     return prompt_cost, completion_cost
 
 

litellm/llms/base_llm/videos/transformation.py

@@ -118,10 +118,11 @@ class BaseVideoConfig(ABC):
         api_base: str,
         litellm_params: GenericLiteLLMParams,
         headers: dict,
+        variant: Optional[str] = None,
     ) -> Tuple[str, Dict]:
         """
         Transform the video content request into a URL and data/params
-        
+
         Returns:
             Tuple[str, Dict]: (url, params) for the video content request
         """

litellm/llms/bedrock/base_aws_llm.py

@@ -234,6 +234,8 @@ class BaseAWSLLM:
                     aws_session_token=aws_session_token,
                     aws_role_name=aws_role_name,
                     aws_session_name=aws_session_name,
+                    aws_region_name=aws_region_name,
+                    aws_sts_endpoint=aws_sts_endpoint,
                     aws_external_id=aws_external_id,
                     ssl_verify=ssl_verify,
                 )
@@ -733,6 +735,7 @@ class BaseAWSLLM:
         region: str,
         web_identity_token_file: str,
         aws_external_id: Optional[str] = None,
+        aws_sts_endpoint: Optional[str] = None,
         ssl_verify: Optional[Union[bool, str]] = None,
     ) -> dict:
         """Handle cross-account role assumption for IRSA."""
@@ -744,11 +747,13 @@ class BaseAWSLLM:
         with open(web_identity_token_file, "r") as f:
             web_identity_token = f.read().strip()
 
+        irsa_sts_kwargs: dict = {"region_name": region, "verify": self._get_ssl_verify(ssl_verify)}
+        if aws_sts_endpoint is not None:
+            irsa_sts_kwargs["endpoint_url"] = aws_sts_endpoint
+
         # Create an STS client without credentials
         with tracer.trace("boto3.client(sts) for manual IRSA"):
-            sts_client = boto3.client(
-                "sts", region_name=region, verify=self._get_ssl_verify(ssl_verify)
-            )
+            sts_client = boto3.client("sts", **irsa_sts_kwargs)
 
         # Manually assume the IRSA role with the session name
         verbose_logger.debug(
@@ -767,11 +772,10 @@ class BaseAWSLLM:
         with tracer.trace("boto3.client(sts) with manual IRSA credentials"):
             sts_client_with_creds = boto3.client(
                 "sts",
-                region_name=region,
                 aws_access_key_id=irsa_creds["AccessKeyId"],
                 aws_secret_access_key=irsa_creds["SecretAccessKey"],
                 aws_session_token=irsa_creds["SessionToken"],
-                verify=self._get_ssl_verify(ssl_verify),
+                **irsa_sts_kwargs,
             )
 
         # Get current caller identity for debugging
@@ -804,16 +808,19 @@ class BaseAWSLLM:
         aws_session_name: str,
         region: str,
         aws_external_id: Optional[str] = None,
+        aws_sts_endpoint: Optional[str] = None,
         ssl_verify: Optional[Union[bool, str]] = None,
     ) -> dict:
         """Handle same-account role assumption for IRSA."""
         import boto3
 
+        irsa_sts_kwargs: dict = {"region_name": region, "verify": self._get_ssl_verify(ssl_verify)}
+        if aws_sts_endpoint is not None:
+            irsa_sts_kwargs["endpoint_url"] = aws_sts_endpoint
+
         verbose_logger.debug("Same account role assumption, using automatic IRSA")
         with tracer.trace("boto3.client(sts) with automatic IRSA"):
-            sts_client = boto3.client(
-                "sts", region_name=region, verify=self._get_ssl_verify(ssl_verify)
-            )
+            sts_client = boto3.client("sts", **irsa_sts_kwargs)
 
         # Get current caller identity for debugging
         try:
@@ -867,6 +874,8 @@ class BaseAWSLLM:
         aws_session_token: Optional[str],
         aws_role_name: str,
         aws_session_name: str,
+        aws_region_name: Optional[str] = None,
+        aws_sts_endpoint: Optional[str] = None,
         aws_external_id: Optional[str] = None,
         ssl_verify: Optional[Union[bool, str]] = None,
     ) -> Tuple[Credentials, Optional[int]]:
@@ -880,6 +889,8 @@ class BaseAWSLLM:
         web_identity_token_file = os.getenv("AWS_WEB_IDENTITY_TOKEN_FILE")
         irsa_role_arn = os.getenv("AWS_ROLE_ARN")
 
+        region = aws_region_name or os.getenv("AWS_REGION") or os.getenv("AWS_DEFAULT_REGION")
+
         # If we have IRSA environment variables and no explicit credentials,
         # we need to use the web identity token flow
         if (
@@ -895,12 +906,8 @@ class BaseAWSLLM:
             )
 
             try:
-                # Get region from environment
-                region = (
-                    os.getenv("AWS_REGION")
-                    or os.getenv("AWS_DEFAULT_REGION")
-                    or "us-east-1"
-                )
+                # Use passed-in region when set, else env, else default (align with AssumeRole path)
+                region = region or "us-east-1"
 
                 # Check if we need to do cross-account role assumption
                 if aws_role_name != irsa_role_arn:
@@ -911,6 +918,7 @@ class BaseAWSLLM:
                         region,
                         web_identity_token_file,
                         aws_external_id,
+                        aws_sts_endpoint=aws_sts_endpoint,
                         ssl_verify=ssl_verify,
                     )
                 else:
@@ -919,6 +927,7 @@ class BaseAWSLLM:
                         aws_session_name,
                         region,
                         aws_external_id,
+                        aws_sts_endpoint=aws_sts_endpoint,
                         ssl_verify=ssl_verify,
                     )
 
@@ -940,11 +949,14 @@ class BaseAWSLLM:
 
         # In EKS/IRSA environments, use ambient credentials (no explicit keys needed)
         # This allows the web identity token to work automatically
+        sts_client_kwargs: dict = {"verify": self._get_ssl_verify(ssl_verify)}
+        if region is not None:
+            sts_client_kwargs["region_name"] = region
+        if aws_sts_endpoint is not None:
+            sts_client_kwargs["endpoint_url"] = aws_sts_endpoint
         if aws_access_key_id is None and aws_secret_access_key is None:
             with tracer.trace("boto3.client(sts)"):
-                sts_client = boto3.client(
-                    "sts", verify=self._get_ssl_verify(ssl_verify)
-                )
+                sts_client = boto3.client("sts", **sts_client_kwargs)
         else:
             with tracer.trace("boto3.client(sts)"):
                 sts_client = boto3.client(
@@ -952,7 +964,7 @@ class BaseAWSLLM:
                     aws_access_key_id=aws_access_key_id,
                     aws_secret_access_key=aws_secret_access_key,
                     aws_session_token=aws_session_token,
-                    verify=self._get_ssl_verify(ssl_verify),
+                    **sts_client_kwargs,
                 )
 
         assume_role_params = {

litellm/llms/bedrock/chat/invoke_transformations/amazon_openai_transformation.py

@@ -14,6 +14,7 @@ import httpx
 from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
 from litellm.llms.bedrock.common_utils import BedrockError
 from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+from litellm.passthrough.utils import CommonUtils
 from litellm.types.llms.openai import AllMessageValues
 
 if TYPE_CHECKING:
@@ -94,6 +95,9 @@ class AmazonBedrockOpenAIConfig(OpenAIGPTConfig, BaseAWSLLM):
             aws_bedrock_runtime_endpoint=aws_bedrock_runtime_endpoint,
             aws_region_name=aws_region_name,
         )
+
+        # Encode model ID for ARNs (e.g., :imported-model/ -> :imported-model%2F)
+        model_id = CommonUtils.encode_bedrock_runtime_modelid_arn(model_id)
         
         # Build the invoke URL
         if stream:

litellm/llms/bedrock/files/transformation.py

@@ -202,52 +202,84 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
         return optional_params
 
 
+    # Providers whose InvokeModel body uses the Converse API format
+    # (messages + inferenceConfig + image blocks). Nova is the primary
+    # example; add others here as they adopt the same schema.
+    CONVERSE_INVOKE_PROVIDERS = ("nova",)
+
     def _map_openai_to_bedrock_params(
         self,
         openai_request_body: Dict[str, Any],
         provider: Optional[str] = None,
     ) -> Dict[str, Any]:
         """
-        Transform OpenAI request body to Bedrock-compatible modelInput parameters using existing transformation logic
+        Transform OpenAI request body to Bedrock-compatible modelInput
+        parameters using existing transformation logic.
+
+        Routes to the correct per-provider transformation so that the
+        resulting dict matches the InvokeModel body that Bedrock expects
+        for batch inference.
         """
         from litellm.types.utils import LlmProviders
+
         _model = openai_request_body.get("model", "")
         messages = openai_request_body.get("messages", [])
-        
-        # Use existing Anthropic transformation logic for Anthropic models
+        optional_params = {
+            k: v
+            for k, v in openai_request_body.items()
+            if k not in ["model", "messages"]
+        }
+
+        # --- Anthropic: use existing AmazonAnthropicClaudeConfig ---
         if provider == LlmProviders.ANTHROPIC:
             from litellm.llms.bedrock.chat.invoke_transformations.anthropic_claude3_transformation import (
                 AmazonAnthropicClaudeConfig,
             )
-            
-            anthropic_config = AmazonAnthropicClaudeConfig()
-            
-            # Extract optional params (everything except model and messages)
-            optional_params = {k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
-            mapped_params = anthropic_config.map_openai_params(
+
+            config = AmazonAnthropicClaudeConfig()
+            mapped_params = config.map_openai_params(
                 non_default_params={},
                 optional_params=optional_params,
                 model=_model,
-                drop_params=False
+                drop_params=False,
             )
-            
-            # Transform using existing Anthropic logic
-            bedrock_params = anthropic_config.transform_request(
+            return config.transform_request(
                 model=_model,
                 messages=messages,
                 optional_params=mapped_params,
                 litellm_params={},
-                headers={}
+                headers={},
             )
 
-            return bedrock_params
-        else:
-            # For other providers, use basic mapping
-            bedrock_params = {
-                "messages": messages,
-                **{k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
-            }
-            return bedrock_params
+        # --- Converse API providers (e.g. Nova): use AmazonConverseConfig
+        #     to correctly convert image_url blocks to Bedrock image format
+        #     and wrap inference params inside inferenceConfig. ---
+        if provider in self.CONVERSE_INVOKE_PROVIDERS:
+            from litellm.llms.bedrock.chat.converse_transformation import (
+                AmazonConverseConfig,
+            )
+
+            converse_config = AmazonConverseConfig()
+            mapped_params = converse_config.map_openai_params(
+                non_default_params=optional_params,
+                optional_params={},
+                model=_model,
+                drop_params=False,
+            )
+            return converse_config.transform_request(
+                model=_model,
+                messages=messages,
+                optional_params=mapped_params,
+                litellm_params={},
+                headers={},
+            )
+
+        # --- All other providers: passthrough (OpenAI-compatible models
+        #     like openai.gpt-oss-*, qwen, deepseek, etc.) ---
+        return {
+            "messages": messages,
+            **optional_params,
+        }
 
     def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
         self, openai_jsonl_content: List[Dict[str, Any]]

litellm/llms/custom_httpx/llm_http_handler.py

@@ -3014,8 +3014,11 @@ class BaseLLMHTTPHandler:
             raise ValueError(f"Unsupported transformed_request type: {type(transformed_request)}")
 
         # Store the upload URL in litellm_params for the transformation method
+        # Honour the URL already set by transform_create_file_request (e.g. Bedrock pre-signed S3 uploads),
+        # fall back to api_base for providers that do not set it.
         litellm_params_with_url = dict(litellm_params)
-        litellm_params_with_url["upload_url"] = api_base
+        if "upload_url" not in litellm_params:
+            litellm_params_with_url["upload_url"] = api_base
 
         return provider_config.transform_create_file_response(
             model=None,
@@ -5397,6 +5400,7 @@ class BaseLLMHTTPHandler:
         api_key: Optional[str] = None,
         client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
         _is_async: bool = False,
+        variant: Optional[str] = None,
     ) -> Union[bytes, Coroutine[Any, Any, bytes]]:
         """
         Handle video content download requests.
@@ -5412,6 +5416,7 @@ class BaseLLMHTTPHandler:
                 extra_headers=extra_headers,
                 api_key=api_key,
                 client=client,
+                variant=variant,
             )
 
         if client is None or not isinstance(client, HTTPHandler):
@@ -5443,6 +5448,7 @@ class BaseLLMHTTPHandler:
             api_base=api_base,
             litellm_params=litellm_params,
             headers=headers,
+            variant=variant,
         )
 
         try:
@@ -5485,6 +5491,7 @@ class BaseLLMHTTPHandler:
         extra_headers: Optional[Dict[str, Any]] = None,
         api_key: Optional[str] = None,
         client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
+        variant: Optional[str] = None,
     ) -> bytes:
         """
         Async version of the video content download handler.
@@ -5519,6 +5526,7 @@ class BaseLLMHTTPHandler:
             api_base=api_base,
             litellm_params=litellm_params,
             headers=headers,
+            variant=variant,
         )
 
         try:

litellm/llms/custom_httpx/mock_transport.py

@@ -0,0 +1,92 @@
+"""
+Mock httpx transport that returns valid OpenAI ChatCompletion responses.
+
+Activated via `litellm_settings: { network_mock: true }`.
+Intercepts at the httpx transport layer — the lowest point before bytes hit the wire —
+so the full proxy -> router -> OpenAI SDK -> httpx path is exercised.
+"""
+
+import json
+import time
+import uuid
+from typing import Tuple
+
+import httpx
+
+
+# ---------------------------------------------------------------------------
+# Pre-built response templates
+# ---------------------------------------------------------------------------
+
+def _mock_id() -> str:
+    return f"chatcmpl-mock-{uuid.uuid4().hex[:8]}"
+
+
+def _chat_completion_json(model: str) -> dict:
+    """Return a minimal valid ChatCompletion object."""
+    return {
+        "id": _mock_id(),
+        "object": "chat.completion",
+        "created": int(time.time()),
+        "model": model,
+        "choices": [
+            {
+                "index": 0,
+                "message": {
+                    "role": "assistant",
+                    "content": "Mock response",
+                },
+                "finish_reason": "stop",
+            }
+        ],
+        "usage": {
+            "prompt_tokens": 1,
+            "completion_tokens": 1,
+            "total_tokens": 2,
+        },
+    }
+
+
+# ---------------------------------------------------------------------------
+# Transport
+# ---------------------------------------------------------------------------
+
+_JSON_HEADERS = {
+    "content-type": "application/json",
+}
+
+
+class MockOpenAITransport(httpx.AsyncBaseTransport, httpx.BaseTransport):
+    """
+    httpx transport that returns canned OpenAI ChatCompletion responses.
+
+    Supports both async (AsyncOpenAI) and sync (OpenAI) SDK paths.
+    """
+
+    @staticmethod
+    def _parse_request(request: httpx.Request) -> Tuple[str, bool]:
+        """Extract model from the request body."""
+        try:
+            body = json.loads(request.content)
+        except (json.JSONDecodeError, ValueError):
+            return ("mock-model", False)
+        model = body.get("model", "mock-model")
+        return (model, False)
+
+    async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
+        model, _ = self._parse_request(request)
+        body = json.dumps(_chat_completion_json(model)).encode()
+        return httpx.Response(
+            status_code=200,
+            headers=_JSON_HEADERS,
+            content=body,
+        )
+
+    def handle_request(self, request: httpx.Request) -> httpx.Response:
+        model, _ = self._parse_request(request)
+        body = json.dumps(_chat_completion_json(model)).encode()
+        return httpx.Response(
+            status_code=200,
+            headers=_JSON_HEADERS,
+            content=body,
+        )

litellm/llms/gemini/cost_calculator.py

@@ -4,13 +4,15 @@ This file is used to calculate the cost of the Gemini API.
 Handles the context caching for Gemini API.
 """
 
-from typing import TYPE_CHECKING, Tuple
+from typing import TYPE_CHECKING, Optional, Tuple
 
 if TYPE_CHECKING:
     from litellm.types.utils import ModelInfo, Usage
 
 
-def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
+def cost_per_token(
+    model: str, usage: "Usage", service_tier: Optional[str] = None
+) -> Tuple[float, float]:
     """
     Calculates the cost per token for a given model, prompt tokens, and completion tokens.
 
@@ -19,7 +21,7 @@ def cost_per_token(model: str, usage: "Usage") -> Tuple[float, float]:
     from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
 
     return generic_cost_per_token(
-        model=model, usage=usage, custom_llm_provider="gemini"
+        model=model, usage=usage, custom_llm_provider="gemini", service_tier=service_tier
     )
 
 

litellm/llms/gemini/videos/transformation.py

@@ -393,10 +393,11 @@ class GeminiVideoConfig(BaseVideoConfig):
         api_base: str,
         litellm_params: GenericLiteLLMParams,
         headers: dict,
+        variant: Optional[str] = None,
     ) -> Tuple[str, Dict]:
         """
         Transform the video content request for Veo API.
-        
+
         For Veo, we need to:
         1. Get operation status to extract video URI
         2. Return download URL for the video

litellm/llms/openai/common_utils.py

@@ -205,6 +205,11 @@ class BaseOpenAILLM:
         if litellm.aclient_session is not None:
             return litellm.aclient_session
 
+        if getattr(litellm, "network_mock", False):
+            from litellm.llms.custom_httpx.mock_transport import MockOpenAITransport
+
+            return httpx.AsyncClient(transport=MockOpenAITransport())
+
         # Get unified SSL configuration
         ssl_config = get_ssl_configuration()
 
@@ -225,6 +230,11 @@ class BaseOpenAILLM:
         if litellm.client_session is not None:
             return litellm.client_session
 
+        if getattr(litellm, "network_mock", False):
+            from litellm.llms.custom_httpx.mock_transport import MockOpenAITransport
+
+            return httpx.Client(transport=MockOpenAITransport())
+
         # Get unified SSL configuration
         ssl_config = get_ssl_configuration()
 

litellm/llms/openai/videos/transformation.py

@@ -172,18 +172,22 @@ class OpenAIVideoConfig(BaseVideoConfig):
         api_base: str,
         litellm_params: GenericLiteLLMParams,
         headers: dict,
+        variant: Optional[str] = None,
     ) -> Tuple[str, Dict]:
         """
         Transform the video content request for OpenAI API.
-        
+
         OpenAI API expects the following request:
         - GET /v1/videos/{video_id}/content
+        - GET /v1/videos/{video_id}/content?variant=thumbnail
         """
         original_video_id = extract_original_video_id(video_id)
-        
+
         # Construct the URL for video content download
         url = f"{api_base.rstrip('/')}/{original_video_id}/content"
-        
+        if variant is not None:
+            url = f"{url}?variant={variant}"
+
         # No additional data needed for GET content request
         data: Dict[str, Any] = {}