Merge branch 'main' into fix/bedrock-strip-client-metadata

.github/actions/helm-oci-chart-releaser/action.yml

@@ -1,94 +0,0 @@
-name: Helm OCI Chart Releaser
-description: Push Helm charts to OCI-based (Docker) registries
-author: sergeyshaykhullin
-branding:
-  color: yellow
-  icon: upload-cloud
-inputs:
-  name:
-    required: true
-    description: Chart name
-  repository:
-    required: true
-    description: Chart repository name
-  tag:
-    required: true
-    description: Chart version
-  app_version:
-    required: true
-    description: App version
-  path:
-    required: false
-    description: Chart path (Default 'charts/{name}')
-  registry:
-    required: true
-    description: OCI registry
-  registry_username:
-    required: true
-    description: OCI registry username
-  registry_password:
-    required: true
-    description: OCI registry password
-  update_dependencies:
-    required: false
-    default: 'false'
-    description: Update chart dependencies before packaging (Default 'false')
-outputs:
-  image:
-    value: ${{ steps.output.outputs.image }}
-    description: Chart image (Default '{registry}/{repository}/{image}:{tag}')
-runs:
-  using: composite
-  steps:
-    - name: Helm | Setup
-      uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1
-      with:
-        version: v3.20.0
-
-    - name: Helm | Login
-      shell: bash
-      env:
-        REGISTRY_PASSWORD: ${{ inputs.registry_password }}
-        REGISTRY_USERNAME: ${{ inputs.registry_username }}
-        REGISTRY: ${{ inputs.registry }}
-      run: echo "$REGISTRY_PASSWORD" | helm registry login -u "$REGISTRY_USERNAME" --password-stdin "$REGISTRY"
-
-    - name: Helm | Dependency
-      if: inputs.update_dependencies == 'true'
-      shell: bash
-      env:
-        CHART_PATH: ${{ inputs.path == null && format('{0}/{1}', 'charts', inputs.name) || inputs.path }}
-      run: helm dependency update "$CHART_PATH"
-
-    - name: Helm | Package
-      shell: bash
-      env:
-        CHART_PATH: ${{ inputs.path == null && format('{0}/{1}', 'charts', inputs.name) || inputs.path }}
-        TAG: ${{ inputs.tag }}
-        APP_VERSION: ${{ inputs.app_version }}
-      run: helm package "$CHART_PATH" --version "$TAG" --app-version "$APP_VERSION"
-
-    - name: Helm | Push
-      shell: bash
-      env:
-        NAME: ${{ inputs.name }}
-        TAG: ${{ inputs.tag }}
-        REGISTRY: ${{ inputs.registry }}
-        REPOSITORY: ${{ inputs.repository }}
-      run: helm push "${NAME}-${TAG}.tgz" "oci://${REGISTRY}/${REPOSITORY}"
-
-    - name: Helm | Logout
-      shell: bash
-      env:
-        REGISTRY: ${{ inputs.registry }}
-      run: helm registry logout "$REGISTRY"
-
-    - name: Helm | Output
-      id: output
-      shell: bash
-      env:
-        REGISTRY: ${{ inputs.registry }}
-        REPOSITORY: ${{ inputs.repository }}
-        NAME: ${{ inputs.name }}
-        TAG: ${{ inputs.tag }}
-      run: echo "image=${REGISTRY}/${REPOSITORY}/${NAME}:${TAG}" >> $GITHUB_OUTPUT

.github/pull_request_template.md

@@ -2,6 +2,10 @@
 
 <!-- e.g. "Fixes #000" -->
 
+## Linear ticket
+
+<!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR -->
+
 ## Pre-Submission checklist
 
 **Please complete all items before asking a LiteLLM maintainer to review your PR**

.github/workflows/README.md

@@ -1,35 +0,0 @@
-# Simple PyPI Publishing
-
-A GitHub workflow to manually publish LiteLLM packages to PyPI with a specified version.
-
-## How to Use
-
-1. Go to the **Actions** tab in the GitHub repository
-2. Select **Simple PyPI Publish** from the workflow list
-3. Click **Run workflow**
-4. Enter the version to publish (e.g., `1.74.10`)
-
-## What the Workflow Does
-
-1. **Updates** the version in `pyproject.toml`
-2. **Copies** the model prices backup file
-3. **Builds** the Python package
-4. **Publishes** to PyPI
-
-## Prerequisites
-
-Make sure the following secret is configured in the repository:
-- `PYPI_PUBLISH_PASSWORD`: PyPI API token for authentication
-
-## Example Usage
-
-- Version: `1.74.11` → Publishes as v1.74.11
-- Version: `1.74.10-hotfix1` → Publishes as v1.74.10-hotfix1
-
-## Features
-
-- ✅ Manual trigger with version input
-- ✅ Automatic version updates in `pyproject.toml`
-- ✅ Repository safety check (only runs on official repo)
-- ✅ Clean package building and publishing
-- ✅ Success confirmation with PyPI package link 

.github/workflows/_test-unit-base.yml

@@ -91,7 +91,7 @@ jobs:
             --reruns-delay 1 \
             --dist=loadscope \
             --durations=20 \
-            --cov=litellm \
+            --cov=./litellm \
             --cov-report=xml:coverage.xml \
             --cov-config=pyproject.toml
 
@@ -132,4 +132,5 @@ jobs:
           use_oidc: true
           directory: coverage-reports
           root_dir: ${{ github.workspace }}
+          flags: ${{ inputs.artifact-name }}
           fail_ci_if_error: false

.github/workflows/_test-unit-services-base.yml

@@ -32,41 +32,39 @@ on:
         required: false
         type: boolean
         default: false
+      dist:
+        description: "pytest-xdist distribution mode (loadscope|load|worksteal|loadfile|no)"
+        required: false
+        type: string
+        default: "loadscope"
       artifact-name:
         description: "Unique name for the coverage artifact (must be unique per run)"
         required: false
         type: string
         default: "run"
-    secrets:
-      DATABASE_URL:
-        required: false
-      POSTGRES_USER:
-        required: false
-      POSTGRES_PASSWORD:
-        required: false
 
 permissions:
   contents: read
 
+# The postgres service container below is spawned per-job on localhost and
+# destroyed with the job. Nothing outside the runner can reach it. The
+# user/password/database here are not secrets — they're bootstrap values
+# for a throwaway container — so we hardcode them instead of attaching
+# every matrix shard to a GHA environment just to read three "secrets"
+# (which also produces a "temporarily deployed to …" notification on the
+# PR timeline per shard per push).
 jobs:
   run:
     name: Run tests
     runs-on: ubuntu-latest
     timeout-minutes: ${{ inputs.timeout-minutes }}
-    # Environment is derived from the enable-* flags, not caller-controllable.
-    # This prevents callers from passing arbitrary environment names to bypass secret scoping.
-    environment: >-
-      ${{
-        inputs.enable-postgres && 'integration-postgres' ||
-        ''
-      }}
 
     services:
       postgres:
         image: postgres@sha256:705a5d5b5836f3fcba0d02c4d281e6a7dd9ed2dd4078640f08a1e1e9896e097d # postgres:14
         env:
-          POSTGRES_USER: ${{ secrets.POSTGRES_USER }}
-          POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
+          POSTGRES_USER: litellm
+          POSTGRES_PASSWORD: litellm
           POSTGRES_DB: litellm_test
         ports:
           - 5432:5432
@@ -114,7 +112,7 @@ jobs:
       - name: Run Prisma migrations
         if: ${{ inputs.enable-postgres }}
         env:
-          DATABASE_URL: ${{ secrets.DATABASE_URL }}
+          DATABASE_URL: "postgresql://litellm:litellm@localhost:5432/litellm_test"
         run: |
           uv run --no-sync prisma db push --schema litellm/proxy/schema.prisma --accept-data-loss
 
@@ -124,7 +122,8 @@ jobs:
           MAX_FAILURES: ${{ inputs.max-failures }}
           WORKERS: ${{ inputs.workers }}
           RERUNS: ${{ inputs.reruns }}
-          DATABASE_URL: ${{ inputs.enable-postgres && secrets.DATABASE_URL || '' }}
+          DIST: ${{ inputs.dist }}
+          DATABASE_URL: ${{ inputs.enable-postgres && 'postgresql://litellm:litellm@localhost:5432/litellm_test' || '' }}
         run: |
           if [ "${WORKERS}" = "0" ]; then
             uv run --no-sync pytest ${TEST_PATH:?} \
@@ -133,7 +132,7 @@ jobs:
               --reruns "${RERUNS}" \
               --reruns-delay 1 \
               --durations=20 \
-              --cov=litellm \
+              --cov=./litellm \
               --cov-report=xml:coverage.xml \
               --cov-config=pyproject.toml
           else
@@ -143,9 +142,9 @@ jobs:
               -n "${WORKERS}" \
               --reruns "${RERUNS}" \
               --reruns-delay 1 \
-              --dist=loadscope \
+              --dist="${DIST}" \
               --durations=20 \
-              --cov=litellm \
+              --cov=./litellm \
               --cov-report=xml:coverage.xml \
               --cov-config=pyproject.toml
           fi
@@ -187,4 +186,5 @@ jobs:
           use_oidc: true
           directory: coverage-reports
           root_dir: ${{ github.workspace }}
+          flags: ${{ inputs.artifact-name }}
           fail_ci_if_error: false

.github/workflows/check_duplicate_issues.yml

@@ -39,7 +39,7 @@ jobs:
         if: github.event.action == 'opened'
         uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
         with:
-          python-version: "3.11"
+          python-version: "3.12"
 
       - name: Auto-close if high-confidence duplicate
         if: github.event.action == 'opened'

.github/workflows/codeql.yml

@@ -53,3 +53,31 @@ jobs:
         uses: github/codeql-action/analyze@ebcb5b36ded6beda4ceefea6a8bc4cc885255bb3 # v3
         with:
           category: "/language:${{ matrix.language }}"
+          output: sarif-results
+          upload: failure-only
+
+      # py/weak-sensitive-data-hashing (CWE-328) fires on the OCI signing call at
+      # litellm/llms/oci/common_utils.py, which hashes the HTTP request body to
+      # produce the x-content-sha256 header required by the OCI HTTP signing spec —
+      # a content-integrity hash, not a password or secret hash. SHA-256 is mandated
+      # by Oracle for this header; see
+      # https://docs.oracle.com/en-us/iaas/Content/API/Concepts/signingrequests.htm
+      # The `usedforsecurity=False` flag on the hashlib.sha256 call already declares
+      # non-security intent, but CodeQL's taint flow still re-fires when callers
+      # further up the stack are modified. The suppression is scoped to this one
+      # file/rule pair via SARIF post-filtering so every other callsite of
+      # py/weak-sensitive-data-hashing in the repository continues to be analyzed.
+      - name: Filter SARIF (OCI sha256)
+        if: matrix.language == 'python'
+        uses: advanced-security/filter-sarif@2da736ff05ef065cb2894ac6892e47b5eac2c3c0 # v1.1
+        with:
+          patterns: |
+            -litellm/llms/oci/common_utils.py:py/weak-sensitive-data-hashing
+          input: sarif-results/python.sarif
+          output: sarif-results/python.sarif
+
+      - name: Upload SARIF
+        uses: github/codeql-action/upload-sarif@ebcb5b36ded6beda4ceefea6a8bc4cc885255bb3 # v3
+        with:
+          sarif_file: sarif-results
+          category: "/language:${{ matrix.language }}"

.github/workflows/create-release-branch.yml

@@ -0,0 +1,65 @@
+name: Create Release Branch
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Release tag (e.g. 1.84.0, 1.84.0rc1, 1.84.0.dev42, 1.84.0.post1; legacy v1.83.10-stable still accepted) — branch will be named release/<tag>"
+        required: true
+        type: string
+      commit_hash:
+        description: "Full 40-char commit SHA the branch should point to"
+        required: true
+        type: string
+  workflow_call:
+    inputs:
+      tag:
+        description: "Release tag (e.g. 1.84.0, 1.84.0rc1, 1.84.0.dev42, 1.84.0.post1; legacy v1.83.10-stable still accepted)"
+        required: true
+        type: string
+      commit_hash:
+        description: "Full 40-char commit SHA the branch should point to"
+        required: true
+        type: string
+
+permissions: {}
+
+jobs:
+  create-branch:
+    name: Create Release Branch
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Validate inputs
+        env:
+          TAG: ${{ inputs.tag }}
+          COMMIT_HASH: ${{ inputs.commit_hash }}
+        run: |
+          if ! echo "${COMMIT_HASH}" | grep -qE '^[0-9a-f]{40}$'; then
+            echo "::error::commit_hash must be a full 40-character commit SHA"
+            exit 1
+          fi
+          if ! echo "${TAG}" | grep -qE '^v?[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::tag must start with X.Y.Z (optional leading v), e.g. 1.84.0, 1.84.0rc1, 1.84.0.dev42, or v1.83.10-stable"
+            exit 1
+          fi
+
+      - name: Create release branch
+        env:
+          TAG: ${{ inputs.tag }}
+          COMMIT_HASH: ${{ inputs.commit_hash }}
+        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
+        with:
+          script: |
+            const tag = process.env.TAG;
+            const commitHash = process.env.COMMIT_HASH;
+            const branchName = `release/${tag}`;
+
+            await github.rest.git.createRef({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              ref: `refs/heads/${branchName}`,
+              sha: commitHash,
+            });
+            core.info(`Created branch ${branchName} at ${commitHash}`);

.github/workflows/create-release.yml

@@ -4,7 +4,7 @@ on:
   workflow_dispatch:
     inputs:
       tag:
-        description: "Release tag (e.g. v1.83.0-stable)"
+        description: "Release tag (e.g. 1.84.0, 1.84.0rc1, 1.84.0.dev42, 1.84.0-dev.2, 1.84.0.post1; legacy v1.83.10-stable still accepted)"
         required: true
         type: string
       commit_hash:
@@ -30,8 +30,8 @@ jobs:
             echo "::error::commit_hash must be a full 40-character commit SHA"
             exit 1
           fi
-          if ! echo "${TAG}" | grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+'; then
-            echo "::error::tag must start with vX.Y.Z"
+          if ! echo "${TAG}" | grep -qE '^v?[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::tag must start with X.Y.Z (optional leading v), e.g. 1.84.0, 1.84.0rc1, 1.84.0.dev42, or v1.83.10-stable"
             exit 1
           fi
 
@@ -45,6 +45,13 @@ jobs:
             const tag = process.env.TAG;
             const commitHash = process.env.COMMIT_HASH;
 
+            // Mark RC / dev / nightly / alpha / beta tags as GitHub pre-releases.
+            // Accept both PEP 440 (`.dev`) and SemVer (`-dev`) separators so tags
+            // like `1.84.0.dev2` and `1.84.0-dev.2` are both detected.
+            // PEP 440 post-releases (e.g. `1.84.0.post1`) and legacy `-stable[.patch.N]`
+            // are stable maintenance releases, not pre-releases.
+            const isPrerelease = /(?:rc|nightly|alpha|beta|[-.]dev)/i.test(tag);
+
             const cosignSection = [
               `## Verify Docker Image Signature`,
               ``,
@@ -89,7 +96,7 @@ jobs:
                 target_commitish: commitHash,
                 name: tag,
                 owner: context.repo.owner,
-                prerelease: false,
+                prerelease: isPrerelease,
                 repo: context.repo.repo,
                 tag_name: tag,
               });
@@ -102,6 +109,17 @@ jobs:
                 body: updatedBody,
                 draft: false,
               });
+
             } catch (error) {
               core.setFailed(error.message);
             }
+
+  create-branch:
+    name: Create Release Branch
+    needs: release
+    permissions:
+      contents: write
+    uses: ./.github/workflows/create-release-branch.yml
+    with:
+      tag: ${{ inputs.tag }}
+      commit_hash: ${{ inputs.commit_hash }}

.github/workflows/guard-fork-dependencies.yml

@@ -0,0 +1,140 @@
+name: Guard fork dependency changes
+
+on:
+  pull_request:
+    branches:
+      - main
+      - litellm_internal_staging
+      - litellm_oss_branch
+      - "litellm_**"
+    paths:
+      - "uv.lock"
+      - "pyproject.toml"
+      - "litellm-proxy-extras/pyproject.toml"
+      - "enterprise/pyproject.toml"
+
+permissions: {}
+
+jobs:
+  guard:
+    name: Block fork dependency changes
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    if: github.event.pull_request.head.repo.full_name != github.repository
+    steps:
+      - name: Checkout base branch
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          ref: ${{ github.base_ref }}
+          persist-credentials: false
+          path: base
+
+      - name: Checkout PR head (read-only)
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+          path: pr
+
+      - name: Reject uv.lock changes
+        run: |
+          if ! diff -q base/uv.lock pr/uv.lock >/dev/null 2>&1; then
+            echo "::error::Fork PRs must not modify uv.lock. Dependency lockfile changes must come from a branch in the canonical repository."
+            exit 1
+          fi
+          echo "uv.lock is unchanged."
+
+      - name: Reject new dependencies in pyproject.toml files
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          # Write the checker script to a temp file to avoid shell quoting issues.
+          cat > /tmp/extract_deps.py << 'SCRIPT'
+          import re
+          import sys
+          import tomllib
+
+
+          def normalize(name: str) -> str:
+              return re.sub(r"[-_.]+", "-", name).lower()
+
+
+          def extract_dep_names(path: str) -> set[str]:
+              with open(path, "rb") as f:
+                  data = tomllib.load(f)
+
+              deps: set[str] = set()
+              pep508 = re.compile(r"^([A-Za-z0-9](?:[A-Za-z0-9._-]*[A-Za-z0-9])?)")
+
+              # [project].dependencies
+              for spec in data.get("project", {}).get("dependencies", []):
+                  m = pep508.match(spec)
+                  if m:
+                      deps.add(normalize(m.group(1)))
+
+              # [project.optional-dependencies]
+              for group in data.get("project", {}).get("optional-dependencies", {}).values():
+                  for spec in group:
+                      m = pep508.match(spec)
+                      if m:
+                          deps.add(normalize(m.group(1)))
+
+              # [dependency-groups]
+              for group in data.get("dependency-groups", {}).values():
+                  for item in group:
+                      if isinstance(item, str):
+                          m = pep508.match(item)
+                          if m:
+                              deps.add(normalize(m.group(1)))
+
+              return deps
+
+
+          if __name__ == "__main__":
+              for name in sorted(extract_dep_names(sys.argv[1])):
+                  print(name)
+          SCRIPT
+
+          had_error=0
+
+          check_deps() {
+            local base_file="$1"
+            local pr_file="$2"
+            local label="$3"
+
+            if [ ! -f "$base_file" ] && [ ! -f "$pr_file" ]; then
+              return 0
+            fi
+
+            if [ ! -f "$base_file" ] && [ -f "$pr_file" ]; then
+              echo "::error::Fork PR introduces a new $label that does not exist on the base branch."
+              return 1
+            fi
+
+            base_deps=$(python3 /tmp/extract_deps.py "$base_file")
+            pr_deps=$(python3 /tmp/extract_deps.py "$pr_file")
+
+            new_deps=$(comm -13 <(echo "$base_deps") <(echo "$pr_deps"))
+
+            if [ -n "$new_deps" ]; then
+              echo "::error::Fork PR adds new dependencies in $label: $(echo $new_deps | tr '\n' ', ')"
+              echo "New packages detected:"
+              echo "$new_deps"
+              return 1
+            fi
+
+            echo "$label: no new dependencies."
+            return 0
+          }
+
+          check_deps base/pyproject.toml pr/pyproject.toml "pyproject.toml" || had_error=1
+          check_deps base/litellm-proxy-extras/pyproject.toml pr/litellm-proxy-extras/pyproject.toml "litellm-proxy-extras/pyproject.toml" || had_error=1
+          check_deps base/enterprise/pyproject.toml pr/enterprise/pyproject.toml "enterprise/pyproject.toml" || had_error=1
+
+          if [ "$had_error" -ne 0 ]; then
+            echo ""
+            echo "::error::Fork PRs must not add new dependencies. Please open an issue or coordinate with a maintainer to update dependencies from within the canonical repository."
+            exit 1
+          fi
+
+          echo "All pyproject.toml files passed dependency check."

.github/workflows/llm-translation-testing.yml

@@ -1,92 +0,0 @@
-name: LLM Translation Tests
-
-on:
-  workflow_dispatch:
-    inputs:
-      release_candidate_tag:
-        description: "Release candidate tag/version"
-        required: true
-        type: string
-  push:
-    tags:
-      - "v*-rc*" # Triggers on release candidate tags like v1.0.0-rc1
-
-permissions:
-  contents: read
-
-jobs:
-  run-llm-translation-tests:
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-          ref: ${{ github.event.inputs.release_candidate_tag || github.ref }}
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.11"
-
-      - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
-        with:
-          version: "0.10.9"
-          enable-cache: false
-
-      - name: Restore uv dependencies cache
-        uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
-        with:
-          path: |
-            ~/.cache/uv
-            .venv
-          key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-uv-
-
-      - name: Install dependencies
-        run: |
-          uv sync --frozen
-
-      - name: Create test results directory
-        run: mkdir -p test-results
-
-      - name: Run LLM Translation Tests
-        env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
-          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
-          AZURE_API_KEY: ${{ secrets.AZURE_API_KEY }}
-          AZURE_API_BASE: ${{ secrets.AZURE_API_BASE }}
-          AZURE_API_VERSION: ${{ secrets.AZURE_API_VERSION }}
-          RC_TAG: ${{ github.event.inputs.release_candidate_tag || github.ref_name }}
-          COMMIT_SHA: ${{ github.sha }}
-        run: |
-          python .github/workflows/run_llm_translation_tests.py \
-            --tag "$RC_TAG" \
-            --commit "$COMMIT_SHA" \
-            || true  # Continue even if tests fail
-
-      - name: Display test summary
-        if: always()
-        run: |
-          if [ -f "test-results/llm_translation_report.md" ]; then
-            echo "Test report generated successfully!"
-            echo "Artifact will contain:"
-            echo "- test-results/junit.xml (JUnit XML results)"
-            echo "- test-results/llm_translation_report.md (Beautiful markdown report)"
-          else
-            echo "Warning: Test report was not generated"
-          fi
-
-      - name: Upload test artifacts
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
-        if: always()
-        with:
-          name: LLM-Translation-Artifact-${{ github.event.inputs.release_candidate_tag || github.ref_name }}
-          path: test-results/
-          retention-days: 30

.github/workflows/mutation-test.yml

@@ -0,0 +1,131 @@
+name: "Mutation Test (manual)"
+
+# Manually-triggered mutation testing. Runs mutmut against the scope
+# configured in [tool.mutmut] in pyproject.toml (currently the
+# litellm/proxy/management_endpoints/ folder). Intended cadence is roughly
+# weekly — clicked from the Actions tab when someone wants a fresh report.
+#
+# Uploads a structured `mutation-report.md` (Meta ACH-style: original +
+# mutated function with `# MUTANT START`/`# MUTANT END` delimiters + the
+# existing tests + a task instruction) as a workflow artifact. Failures
+# do not block anything because nothing depends on this workflow.
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: mutation-test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  mutation:
+    name: Run mutmut
+    runs-on: ubuntu-latest
+    # Whole-folder mutation against ~15 files / ~7.5k LOC can take hours.
+    # 350 minutes is just under the GitHub-hosted job cap of 360 minutes.
+    timeout-minutes: 350
+
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        with:
+          version: "0.10.9"
+
+      - name: Cache uv dependencies
+        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+        with:
+          path: |
+            ~/.cache/uv
+            .venv
+          key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-uv-
+
+      - name: Install dependencies
+        run: |
+          uv sync --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
+
+      - name: Generate Prisma client
+        env:
+          PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
+        run: |
+          uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
+
+      # mutmut 3.x runs tests inside a `mutants/` sandbox where it injects
+      # mutation trampolines. uv installs the project as editable by default,
+      # which puts the original source dir on sys.path via a .pth file and
+      # shadows the sandbox copy — so tests would never exercise the mutated
+      # code. Reinstalling non-editable removes the .pth shadow.
+      - name: Reinstall litellm non-editable (so mutants/ is not shadowed)
+        run: |
+          uv pip uninstall litellm
+          uv pip install . --no-deps
+
+      # pytest-retry's pytest_configure hook crashes with
+      # `INTERNALERROR: no option named 'filtered_exceptions'` when invoked
+      # via mutmut's in-process pytest.main() call. The entry-point name
+      # doesn't normalize cleanly with `-p no:<name>`, so just remove the
+      # package outright. Reruns are wrong for mutation testing anyway —
+      # rerunning a "failed" mutant test would mask which mutants are killed.
+      - name: Remove pytest plugins that conflict with mutmut
+        run: |
+          uv pip uninstall pytest-retry || true
+
+      - name: Run mutmut
+        env:
+          # Make the mutants/ sandbox win over site-packages on sys.path so the
+          # trampolined files are imported instead of the installed copy.
+          PYTHONPATH: ${{ github.workspace }}/mutants
+        run: |
+          set -o pipefail
+          mkdir -p mutants
+          uv run --no-sync --with mutmut==3.5.0 mutmut run 2>&1 | tee mutmut-run.log
+
+      # Generate the structured report. The script embeds the enclosing
+      # function source for each survivor (via Python AST) and includes the
+      # existing test files, so an LLM agent has enough context to write
+      # killing tests without further file lookups. Modeled on Meta's ACH
+      # prompt template (arXiv 2501.12862).
+      - name: Generate detailed mutation report
+        if: always()
+        run: |
+          set +e
+          uv run --no-sync --with mutmut==3.5.0 mutmut export-cicd-stats > /dev/null 2>&1
+          uv run --no-sync --with mutmut==3.5.0 mutmut results > mutmut-results.txt 2>&1
+          uv run --no-sync python scripts/mutation_report.py
+          # The full report can be very long for big test files; the run-page
+          # summary cuts off at 1 MB. Append the head of the report (summary
+          # + survivor list) and link out to the artifact for the full body.
+          {
+            head -c 900000 mutation-report.md
+            echo ""
+            echo ""
+            echo "_Full report (with embedded function bodies and test files) is in the workflow artifact._"
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload mutmut artifacts
+        if: always()
+        uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
+        with:
+          name: mutmut-${{ github.run_id }}-${{ github.run_attempt }}
+          path: |
+            mutation-report.md
+            mutmut-results.txt
+            mutmut-run.log
+            mutants/mutmut-stats.json
+            mutants/mutmut-cicd-stats.json
+            mutants/litellm/proxy/management_endpoints/**/*.py
+          if-no-files-found: warn
+          retention-days: 14

.github/workflows/publish_to_pypi.yml

@@ -1,153 +0,0 @@
-name: Publish to PyPI
-
-on:
-  workflow_dispatch:
-
-jobs:
-  preflight-checks:
-    name: Preflight Checks
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-    # No environment — read-only checks, no approval needed
-    outputs:
-      needs_publish: ${{ steps.check-litellm.outputs.needs_publish }}
-      version: ${{ steps.check-litellm.outputs.version }}
-
-    steps:
-      - name: Checkout repo
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
-        with:
-          version: "0.10.9"
-          enable-cache: false
-
-      - name: Check litellm version on PyPI
-        id: check-litellm
-        run: |
-          VERSION=$(python - <<'PY'
-          import tomllib
-
-          with open("pyproject.toml", "rb") as f:
-              print(tomllib.load(f)["project"]["version"])
-          PY
-          )
-          echo "version=$VERSION" >> "$GITHUB_OUTPUT"
-          echo "Checking if litellm $VERSION exists on PyPI..."
-
-          HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://pypi.org/pypi/litellm/$VERSION/json")
-          if [ "$HTTP_STATUS" = "200" ]; then
-            echo "litellm $VERSION already exists on PyPI. Skipping publish."
-            echo "needs_publish=false" >> "$GITHUB_OUTPUT"
-          else
-            echo "litellm $VERSION not found on PyPI. Publish needed."
-            echo "needs_publish=true" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Sanity check proxy-extras version
-        run: |
-          # Read pinned version from project optional dependencies
-          PYPROJECT_VERSION=$(python3 - <<'PY'
-          import sys
-          import tomllib
-
-          with open("pyproject.toml", "rb") as f:
-              proxy_requirements = tomllib.load(f)["project"]["optional-dependencies"]["proxy"]
-
-          version = None
-          for requirement in proxy_requirements:
-              normalized = requirement.split(";", 1)[0].strip()
-              if not normalized.startswith("litellm-proxy-extras"):
-                  continue
-              parts = normalized.split("==", 1)
-              if len(parts) == 2 and parts[0].strip() == "litellm-proxy-extras":
-                  candidate = parts[1].strip()
-                  if candidate:
-                      version = candidate
-                      break
-
-          if version is None:
-              print(
-                  "::error::Could not find an exact litellm-proxy-extras pin in project.optional-dependencies.proxy",
-                  file=sys.stderr,
-              )
-              sys.exit(1)
-
-          print(version)
-          PY
-          )
-          echo "pyproject.toml pins litellm-proxy-extras version: $PYPROJECT_VERSION"
-
-          # Check that the pinned version exists on PyPI
-          echo "Checking if litellm-proxy-extras $PYPROJECT_VERSION exists on PyPI..."
-          HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://pypi.org/pypi/litellm-proxy-extras/$PYPROJECT_VERSION/json")
-          if [ "$HTTP_STATUS" != "200" ]; then
-            echo "::error::litellm-proxy-extras $PYPROJECT_VERSION is not published on PyPI yet. Publish it before releasing litellm."
-            exit 1
-          fi
-          echo "litellm-proxy-extras $PYPROJECT_VERSION exists on PyPI. Sanity check passed."
-
-  publish-litellm:
-    name: Publish litellm to PyPI
-    needs: preflight-checks
-    if: needs.preflight-checks.outputs.needs_publish == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      id-token: write
-      contents: read
-    environment: pypi-publish
-
-    steps:
-      - name: Checkout repo
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
-        with:
-          version: "0.10.9"
-          enable-cache: false
-
-      - name: Copy model prices backup
-        run: cp model_prices_and_context_window.json litellm/model_prices_and_context_window_backup.json
-
-      - name: Build package
-        run: |
-          rm -rf build dist
-          uv build
-
-      - name: Verify build artifacts
-        env:
-          EXPECTED_VERSION: ${{ needs.preflight-checks.outputs.version }}
-        run: |
-          echo "Contents of dist/:"
-          ls -la dist/
-          # Ensure we have both sdist and wheel
-          ls dist/*.tar.gz
-          ls dist/*.whl
-          # Verify built version matches expected
-          ls dist/ | grep -q "litellm-${EXPECTED_VERSION}" || {
-            echo "::error::Built artifacts do not match expected version $EXPECTED_VERSION"
-            ls dist/
-            exit 1
-          }
-
-      - name: Validate package metadata
-        run: |
-          uv tool run --from 'twine==6.2.0' twine check dist/*
-
-      - name: Publish to PyPI
-        uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0

.github/workflows/read_pyproject_version.yml

@@ -1,28 +0,0 @@
-name: Read Version from pyproject.toml
-
-on:
-  push:
-    branches:
-      - main # Change this to the default branch of your repository
-
-permissions:
-  contents: read
-
-jobs:
-  read-version:
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Read version from pyproject.toml
-        id: read-version
-        run: |
-          version=$(grep -m1 '^version' pyproject.toml | sed 's/version = "\(.*\)"/\1/')
-          printf "LITELLM_VERSION=%s" "$version" >> $GITHUB_ENV
-
-      - name: Display version
-        run: echo "Current version is $LITELLM_VERSION"

.github/workflows/results_stats.csv

@@ -1,27 +0,0 @@
-Date,"Ben 
-Ashley",Tom Brooks,Jimmy Cooney,"Sue 
-Daniels",Berlinda Fong,Terry Jones,Angelina Little,Linda Smith
-10/1,FALSE,TRUE,TRUE,TRUE,TRUE,TRUE,FALSE,TRUE
-10/2,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/3,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/4,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/5,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/6,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/7,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/8,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/9,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/10,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/11,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/12,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/13,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/14,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/15,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/16,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/17,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/18,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/19,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/20,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/21,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/22,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-10/23,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE
-Total,0,1,1,1,1,1,0,1

.github/workflows/run_observatory_tests.yml

@@ -1,229 +0,0 @@
-name: Run Observatory Tests
-on:
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: "Docker image tag to test (e.g. v1.61.0.rc1)"
-        required: true
-        type: string
-      commit_hash:
-        description: "Commit hash (defaults to HEAD of current branch)"
-        required: false
-        type: string
-  workflow_call:
-    inputs:
-      tag:
-        description: "Docker image tag to test"
-        required: true
-        type: string
-      commit_hash:
-        description: "Commit hash of the release"
-        required: true
-        type: string
-
-permissions:
-  contents: read
-
-env:
-  LITELLM_MASTER_KEY: ${{ secrets.LITELLM_MASTER_KEY_STAGING }}
-
-jobs:
-  observatory-tests:
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Validate tag input
-        env:
-          TAG: ${{ inputs.tag }}
-        run: |
-          if [[ ! "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
-            echo "Invalid tag format: $TAG (expected vX.Y.Z...)"
-            exit 1
-          fi
-
-      - name: Start LiteLLM container
-        env:
-          TAG: ${{ inputs.tag }}
-          AZURE_API_KEY: ${{ secrets.AZURE_API_KEY }}
-          AZURE_API_BASE: ${{ secrets.AZURE_API_BASE }}
-          WORKSPACE: ${{ github.workspace }}
-        run: |
-          docker run -d \
-            --name litellm-rc \
-            -p 4000:4000 \
-            -v "${WORKSPACE}/.github/observatory/litellm_config.yaml:/app/config.yaml" \
-            -e LITELLM_MASTER_KEY="${LITELLM_MASTER_KEY}" \
-            -e AZURE_API_KEY="${AZURE_API_KEY}" \
-            -e AZURE_API_BASE="${AZURE_API_BASE}" \
-            "litellm/litellm:${TAG}" \
-            --config /app/config.yaml --port 4000
-
-      - name: Wait for LiteLLM health check
-        run: |
-          echo "Waiting for LiteLLM to be ready..."
-          for i in $(seq 1 30); do
-            if curl -s -f http://localhost:4000/health/liveliness > /dev/null 2>&1; then
-              echo "LiteLLM is healthy"
-              exit 0
-            fi
-            echo "Attempt $i/30 - not ready yet, waiting 10s..."
-            sleep 10
-          done
-          echo "LiteLLM failed to start within 5 minutes"
-          docker logs litellm-rc
-          exit 1
-
-      - name: Start cloudflared tunnel
-        run: |
-          # Install cloudflared (pinned version + checksum)
-          curl -sL https://github.com/cloudflare/cloudflared/releases/download/2025.2.1/cloudflared-linux-amd64 -o /usr/local/bin/cloudflared
-          echo "afdfadd1ef552e66bffc35246fe30a9bd578356d2d386de95585ccfc432472b8  /usr/local/bin/cloudflared" | sha256sum -c -
-          chmod +x /usr/local/bin/cloudflared
-
-          # Start a quick tunnel (no account needed) and capture the URL
-          cloudflared tunnel --url http://localhost:4000 --no-autoupdate > /tmp/cloudflared.log 2>&1 &
-          CLOUDFLARED_PID=$!
-          echo "CLOUDFLARED_PID=$CLOUDFLARED_PID" >> $GITHUB_ENV
-
-          # Wait for tunnel URL to appear in logs
-          echo "Waiting for tunnel URL..."
-          for i in $(seq 1 30); do
-            TUNNEL_URL=$(grep -oP 'https://[a-z0-9-]+\.trycloudflare\.com' /tmp/cloudflared.log | head -1 || true)
-            if [ -n "$TUNNEL_URL" ]; then
-              echo "Tunnel URL: $TUNNEL_URL"
-              echo "TUNNEL_URL=$TUNNEL_URL" >> $GITHUB_ENV
-              exit 0
-            fi
-            sleep 2
-          done
-          echo "Failed to get tunnel URL"
-          cat /tmp/cloudflared.log
-          exit 1
-
-      - name: Verify tunnel connectivity
-        run: |
-          echo "Testing tunnel at ${TUNNEL_URL}..."
-          # Quick tunnels need time for DNS propagation; retry to avoid
-          # transient NXDOMAIN (curl exit code 6) on first attempt.
-          for i in $(seq 1 10); do
-            if curl -sf "${TUNNEL_URL}/health/liveliness" > /dev/null 2>&1; then
-              echo "Tunnel is working (attempt $i)"
-              exit 0
-            fi
-            echo "Attempt $i/10 - tunnel not routable yet, waiting 5s..."
-            sleep 5
-          done
-          echo "Tunnel failed to become reachable after 50s"
-          cat /tmp/cloudflared.log
-          exit 1
-
-      - name: Trigger observatory test run
-        id: trigger
-        env:
-          OBSERVATORY_URL: ${{ secrets.OBSERVATORY_URL }}
-          OBSERVATORY_API_KEY: ${{ secrets.OBSERVATORY_API_KEY }}
-        run: |
-          PAYLOAD=$(jq -n \
-            --arg url "${TUNNEL_URL}" \
-            --arg key "${LITELLM_MASTER_KEY}" \
-            '{
-              deployment_url: $url,
-              api_key: $key,
-              test_suite: "TestOAIAzureRelease",
-              models: ["gpt-4o-mini", "gpt-4o"]
-            }')
-          RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "${OBSERVATORY_URL}/run-test" \
-            -H "Content-Type: application/json" \
-            -H "X-LiteLLM-Observatory-API-Key: ${OBSERVATORY_API_KEY}" \
-            -d "$PAYLOAD")
-          HTTP_CODE=$(echo "$RESPONSE" | tail -1)
-          BODY=$(echo "$RESPONSE" | head -n -1)
-          echo "Response ($HTTP_CODE): $BODY"
-          if [ "$HTTP_CODE" -ge 400 ]; then
-            echo "Failed to trigger test run"
-            exit 1
-          fi
-
-          # Extract request_id for polling this specific run
-          REQUEST_ID=$(echo "$BODY" | jq -r '.results.request_id')
-          if [ -z "$REQUEST_ID" ] || [ "$REQUEST_ID" = "null" ]; then
-            echo "Failed to extract request_id from response"
-            exit 1
-          fi
-          echo "Request ID: $REQUEST_ID"
-          echo "request_id=$REQUEST_ID" >> $GITHUB_OUTPUT
-
-      - name: Poll for test completion
-        id: poll
-        env:
-          OBSERVATORY_URL: ${{ secrets.OBSERVATORY_URL }}
-          OBSERVATORY_API_KEY: ${{ secrets.OBSERVATORY_API_KEY }}
-          REQUEST_ID: ${{ steps.trigger.outputs.request_id }}
-        run: |
-          TIMEOUT=900  # 15 minutes
-          INTERVAL=30
-          ELAPSED=0
-          while [ $ELAPSED -lt $TIMEOUT ]; do
-            STATUS=$(curl -s "${OBSERVATORY_URL}/run-status/${REQUEST_ID}" \
-              -H "X-LiteLLM-Observatory-API-Key: ${OBSERVATORY_API_KEY}")
-            RUN_STATUS=$(echo "$STATUS" | jq -r '.status')
-            echo "Run status (${ELAPSED}s elapsed): $RUN_STATUS"
-
-            if [ "$RUN_STATUS" = "completed" ] || [ "$RUN_STATUS" = "failed" ]; then
-              echo "Test finished with status: $RUN_STATUS"
-              echo "$STATUS" > /tmp/observatory_result.json
-              exit 0
-            fi
-
-            sleep $INTERVAL
-            ELAPSED=$((ELAPSED + INTERVAL))
-          done
-          echo "Timed out waiting for test to complete after ${TIMEOUT}s"
-          exit 1
-
-      - name: Verify test results
-        run: |
-          RESULT=$(cat /tmp/observatory_result.json)
-          echo "Full result: $RESULT"
-
-          STATUS=$(echo "$RESULT" | jq -r '.status')
-          TEST_PASSED=$(echo "$RESULT" | jq -r '.result.test_passed // false')
-          FAILURE_RATE=$(echo "$RESULT" | jq -r '.result.failure_rate // "N/A"')
-          ERROR=$(echo "$RESULT" | jq -r '.error // empty')
-
-          echo "Status: $STATUS"
-          echo "Test passed: $TEST_PASSED"
-          echo "Failure rate: $FAILURE_RATE"
-
-          if [ -n "$ERROR" ]; then
-            echo "Error: $ERROR"
-          fi
-
-          if [ "$STATUS" = "failed" ]; then
-            echo "Test run failed"
-            exit 1
-          fi
-
-          if [ "$TEST_PASSED" != "true" ]; then
-            echo "Tests did not pass (failure rate: $FAILURE_RATE)"
-            exit 1
-          fi
-
-          echo "All tests passed!"
-
-      - name: Print LiteLLM logs on failure
-        if: failure()
-        run: |
-          docker logs litellm-rc 2>/dev/null || true
-          cat /tmp/cloudflared.log 2>/dev/null || true
-
-      - name: Cleanup
-        if: always()
-        run: |
-          kill "$CLOUDFLARED_PID" 2>/dev/null || true
-          docker rm -f litellm-rc 2>/dev/null || true

.github/workflows/scan_duplicate_issues.yml

@@ -1,48 +0,0 @@
-name: Scan Duplicate Issues (One-Time)
-
-on:
-  workflow_dispatch:
-    inputs:
-      threshold:
-        description: "Similarity threshold (0-1)"
-        required: false
-        default: "0.85"
-      close:
-        description: "Actually close duplicates (false = dry run)"
-        required: false
-        type: boolean
-        default: false
-
-jobs:
-  scan:
-    runs-on: ubuntu-latest
-    permissions:
-      issues: write
-      contents: read
-    steps:
-      - name: Checkout scripts
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          sparse-checkout: .github/scripts
-          persist-credentials: false
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.13"
-
-      - name: Scan for duplicate issues
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          INPUT_THRESHOLD: ${{ inputs.threshold }}
-          INPUT_CLOSE: ${{ inputs.close }}
-        run: |
-          CLOSE_FLAG=""
-          if [ "$INPUT_CLOSE" = "true" ]; then
-            CLOSE_FLAG="--close"
-          fi
-          python3 .github/scripts/close_duplicate_issues.py \
-            --scan \
-            --repo ${{ github.repository }} \
-            --threshold "$INPUT_THRESHOLD" \
-            $CLOSE_FLAG

.github/workflows/test-code-quality.yml

@@ -0,0 +1,128 @@
+name: Code Quality Checks
+
+on:
+  pull_request:
+    branches:
+      - main
+      - litellm_internal_staging
+      - litellm_oss_branch
+      - "litellm_**"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  code-quality:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Checkout litellm-docs into docs/my-website (for documentation_tests)
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          repository: BerriAI/litellm-docs
+          path: docs/my-website
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        with:
+          version: "0.10.9"
+
+      - name: Cache uv dependencies
+        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+        with:
+          path: |
+            ~/.cache/uv
+            .venv
+          key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-uv-
+
+      - name: Install dependencies
+        run: uv sync --frozen --all-groups --all-extras
+
+      - name: check_licenses
+        run: uv run --no-sync python ./tests/code_coverage_tests/check_licenses.py
+
+      - name: check_provider_folders_documented
+        run: uv run --no-sync python ./tests/code_coverage_tests/check_provider_folders_documented.py
+
+      - name: router_code_coverage
+        run: uv run --no-sync python ./tests/code_coverage_tests/router_code_coverage.py
+
+      - name: test_chat_completion_imports
+        run: uv run --no-sync python ./tests/code_coverage_tests/test_chat_completion_imports.py
+
+      - name: info_log_check
+        run: uv run --no-sync python ./tests/code_coverage_tests/info_log_check.py
+
+      - name: check_guardrail_apply_decorator
+        run: uv run --no-sync python ./tests/code_coverage_tests/check_guardrail_apply_decorator.py
+
+      - name: test_ban_set_verbose
+        run: uv run --no-sync python ./tests/code_coverage_tests/test_ban_set_verbose.py
+
+      - name: code_qa_check_tests
+        run: uv run --no-sync python ./tests/code_coverage_tests/code_qa_check_tests.py
+
+      - name: check_get_model_cost_key_performance
+        run: uv run --no-sync python ./tests/code_coverage_tests/check_get_model_cost_key_performance.py
+
+      - name: test_proxy_types_import
+        run: uv run --no-sync python ./tests/code_coverage_tests/test_proxy_types_import.py
+
+      - name: callback_manager_test
+        run: uv run --no-sync python ./tests/code_coverage_tests/callback_manager_test.py
+
+      - name: recursive_detector
+        run: uv run --no-sync python ./tests/code_coverage_tests/recursive_detector.py
+
+      - name: test_router_strategy_async
+        run: uv run --no-sync python ./tests/code_coverage_tests/test_router_strategy_async.py
+
+      - name: litellm_logging_code_coverage
+        run: uv run --no-sync python ./tests/code_coverage_tests/litellm_logging_code_coverage.py
+
+      - name: ensure_async_clients_test
+        run: uv run --no-sync python ./tests/code_coverage_tests/ensure_async_clients_test.py
+
+      - name: enforce_llms_folder_style
+        run: uv run --no-sync python ./tests/code_coverage_tests/enforce_llms_folder_style.py
+
+      - name: prevent_key_leaks_in_exceptions
+        run: uv run --no-sync python ./tests/code_coverage_tests/prevent_key_leaks_in_exceptions.py
+
+      - name: check_unsafe_enterprise_import
+        run: uv run --no-sync python ./tests/code_coverage_tests/check_unsafe_enterprise_import.py
+
+      - name: ban_copy_deepcopy_kwargs
+        run: uv run --no-sync python ./tests/code_coverage_tests/ban_copy_deepcopy_kwargs.py
+
+      - name: check_fastuuid_usage
+        run: uv run --no-sync python ./tests/code_coverage_tests/check_fastuuid_usage.py
+
+      - name: memory_test
+        run: uv run --no-sync python ./tests/code_coverage_tests/memory_test.py
+
+      - name: documentation_test_env_keys
+        run: uv run --no-sync python ./tests/documentation_tests/test_env_keys.py
+
+      - name: documentation_test_router_settings
+        run: uv run --no-sync python ./tests/documentation_tests/test_router_settings.py
+
+      - name: documentation_test_api_docs
+        run: uv run --no-sync python ./tests/documentation_tests/test_api_docs.py

.github/workflows/test-litellm.yml

@@ -1,45 +0,0 @@
-name: LiteLLM Mock Tests (folder - tests/test_litellm)
-
-# DEPRECATED: This workflow is replaced by test-litellm-matrix.yml which runs
-# the same tests in parallel across 10 jobs for faster CI times.
-# Kept for manual debugging only.
-on:
-  workflow_dispatch: # Manual trigger only
-  # pull_request:
-  #   branches: [ main ]
-
-permissions:
-  contents: read
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    timeout-minutes: 25
-
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Thank You Message
-        run: |
-          echo "### 🙏 Thank you for contributing to LiteLLM!" >> $GITHUB_STEP_SUMMARY
-          echo "Your PR is being tested now. We appreciate your help in making LiteLLM better!" >> $GITHUB_STEP_SUMMARY
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
-        with:
-          version: "0.10.9"
-
-      - name: Install dependencies
-        run: |
-          uv lock --check
-          uv sync --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
-      - name: Run tests
-        run: |
-          uv run --no-sync pytest tests/test_litellm --tb=short -vv --maxfail=10 -n 4 --durations=50

.github/workflows/test-mcp.yml

@@ -43,4 +43,4 @@ jobs:
 
       - name: Run MCP tests
         run: |
-          uv run --no-sync pytest tests/mcp_tests -x -vv -n 4 --cov=litellm --cov-report=xml --durations=5
+          uv run --no-sync pytest tests/mcp_tests -x -vv -n 4 --cov=./litellm --cov-report=xml --durations=5

.github/workflows/test-semgrep.yml

@@ -0,0 +1,39 @@
+name: Semgrep
+
+on:
+  pull_request:
+    branches:
+      - main
+      - litellm_internal_staging
+      - litellm_oss_branch
+      - "litellm_**"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  semgrep:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        with:
+          version: "0.10.9"
+
+      - name: Run Semgrep (custom rules)
+        run: uv tool run --from 'semgrep==1.157.0' semgrep scan --config .semgrep/rules . --error

.github/workflows/test-unit-documentation.yml

@@ -25,6 +25,13 @@ jobs:
         with:
           persist-credentials: false
 
+      - name: Checkout litellm-docs into docs/my-website (for documentation_tests)
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          repository: BerriAI/litellm-docs
+          path: docs/my-website
+          persist-credentials: false
+
       - name: Set up Python
         uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
         with:

.github/workflows/test-unit-proxy-db.yml

@@ -12,8 +12,74 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
+# Semantic matrix: each shard groups tests by concern (auth, server, logging, …)
+# rather than alphabetical letter ranges. Adding a new test file means adding it
+# to whichever group it belongs to, not reshuffling slices.
+#
+# Design targets:
+#   * Every shard runs in <= 7 minutes of wall-clock on the default runner.
+#     Most of a shard's time is pytest plugin load + xdist worker imports +
+#     pytest-cov instrumentation, not the tests themselves. Keeping per-shard
+#     work low and matching worker count to runner cores is what controls it.
+#   * workers: 4 matches the 4-core ubuntu-latest runner. -n 8 on 4 cores
+#     oversubscribes 2x and workers fight for CPU during their cold-start
+#     imports (measured ~441% CPU for -n 8 locally, i.e. ~55% effective).
+#   * test_key_generate_prisma.py stays serial (workers=0) — it has event-loop
+#     conflicts with the logging worker when run in parallel.
+#   * test_proxy_utils.py runs as a single shard with --dist=worksteal so
+#     xdist balances its 188 parametrized cases across workers instead of
+#     pinning the whole file to one worker (the default --dist=loadscope
+#     behavior for single-file targets).
+#   * test_db_schema_migration.py is isolated because one test in it
+#     (test_aaaasschema_migration_check) takes ~170s — by itself it
+#     determines the shard's wall-clock floor.
 jobs:
+  # Fast guard — fails the workflow if a test_*.py file under
+  # tests/proxy_unit_tests/ is not referenced by any matrix entry below.
+  # The semantic-shard design (no catch-all "remaining" bucket) relies on
+  # every test file being explicitly assigned; this guard prevents a new
+  # file from silently dropping out of CI.
+  assert-shard-coverage:
+    runs-on: ubuntu-latest
+    timeout-minutes: 2
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+      - name: Assert every test_*.py is in a matrix shard
+        run: |
+          python3 - <<'PY'
+          import pathlib, sys, yaml
+          wf = yaml.safe_load(open(".github/workflows/test-unit-proxy-db.yml"))
+          matrix = wf["jobs"]["proxy-db"]["strategy"]["matrix"]["include"]
+          referenced = set()
+          for entry in matrix:
+              for token in entry["test-path"].split():
+                  if token.startswith("tests/proxy_unit_tests/"):
+                      referenced.add(pathlib.PurePosixPath(token).name)
+          actual = {p.name for p in pathlib.Path("tests/proxy_unit_tests").iterdir()
+                    if p.name.startswith("test_") and (p.suffix == ".py" or p.is_dir())
+                    and p.name != "test_configs"}
+          orphans = sorted(actual - referenced)
+          if orphans:
+              print("ERROR: the following files/dirs under tests/proxy_unit_tests/")
+              print("       are not assigned to any shard in test-unit-proxy-db.yml:")
+              for o in orphans:
+                  print(f"  - {o}")
+              print()
+              print("Add each to whichever semantic shard it belongs to.")
+              sys.exit(1)
+          print(f"OK: all {len(actual)} files assigned to a shard.")
+          PY
+
   proxy-db:
+    needs: assert-shard-coverage
+    # Display only the semantic shard name in the checks UI instead of GHA's
+    # default "proxy-db (key-generation, tests/proxy_unit_tests/…, 0, loadscope, 20)"
+    # which includes every matrix field and gets truncated past the test-path.
+    name: ${{ matrix.test-group }}
     permissions:
       contents: read
       id-token: write
@@ -22,19 +88,150 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          # Key generation tests must NOT run in parallel (event loop conflicts with logging worker)
+          # Must run serially — event-loop conflict with the logging worker.
           - test-group: key-generation
             test-path: "tests/proxy_unit_tests/test_key_generate_prisma.py"
             workers: 0
-            timeout: 30
-          - test-group: auth-checks
-            test-path: "tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py"
-            workers: 8
+            dist: loadscope
             timeout: 20
-          - test-group: remaining
-            test-path: "tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py"
-            workers: 8
-            timeout: 30
+
+          # ---- auth: split into 2 shards ----
+          - test-group: auth-checks
+            test-path: >-
+              tests/proxy_unit_tests/test_auth_checks.py
+              tests/proxy_unit_tests/test_user_api_key_auth.py
+              tests/proxy_unit_tests/test_deprecated_key_grace_period.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+          - test-group: jwt-and-keys
+            test-path: >-
+              tests/proxy_unit_tests/test_jwt.py
+              tests/proxy_unit_tests/test_jwt_key_mapping.py
+              tests/proxy_unit_tests/test_proxy_custom_auth.py
+              tests/proxy_unit_tests/test_key_generate_dynamodb.py
+              tests/proxy_unit_tests/test_deployed_proxy_keygen.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+
+          # ---- test_proxy_utils.py, single shard, worksteal distribution ----
+          - test-group: proxy-utils
+            test-path: "tests/proxy_unit_tests/test_proxy_utils.py"
+            workers: 4
+            dist: worksteal
+            timeout: 15
+
+          # ---- proxy server: split into 2 shards ----
+          - test-group: proxy-server-core
+            test-path: >-
+              tests/proxy_unit_tests/test_proxy_server.py
+              tests/proxy_unit_tests/test_proxy_server_keys.py
+              tests/proxy_unit_tests/test_proxy_server_caching.py
+              tests/proxy_unit_tests/test_proxy_server_langfuse.py
+              tests/proxy_unit_tests/test_proxy_server_spend.py
+              tests/proxy_unit_tests/test_aproxy_startup.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+          - test-group: proxy-runtime
+            test-path: >-
+              tests/proxy_unit_tests/test_proxy_config_unit_test.py
+              tests/proxy_unit_tests/test_proxy_routes.py
+              tests/proxy_unit_tests/test_proxy_gunicorn.py
+              tests/proxy_unit_tests/test_server_root_path.py
+              tests/proxy_unit_tests/test_proxy_pass_user_config.py
+              tests/proxy_unit_tests/test_proxy_token_counter.py
+              tests/proxy_unit_tests/test_request_size_limit_middleware.py
+              tests/proxy_unit_tests/test_multipart_bypass_repro.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+
+          # ---- logging: split into 2 shards ----
+          - test-group: custom-logging
+            test-path: >-
+              tests/proxy_unit_tests/test_custom_callback_input.py
+              tests/proxy_unit_tests/test_custom_logger_s3_gcs.py
+              tests/proxy_unit_tests/test_proxy_custom_logger.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+          - test-group: logging-misc
+            test-path: >-
+              tests/proxy_unit_tests/test_proxy_reject_logging.py
+              tests/proxy_unit_tests/test_audit_logs_proxy.py
+              tests/proxy_unit_tests/test_search_api_logging.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+
+          # ---- db-and-spend: isolate the 170s schema-migration test ----
+          # test_db_schema_migration.py has exactly one test, and that test
+          # is mostly waiting on `prisma migrate deploy` / `prisma migrate
+          # diff` subprocesses (~170s). It does no CPU-bound Python work
+          # inside the test. Running with workers=0 (serial, no xdist)
+          # skips the 4-worker cold-start cost we'd otherwise pay for a
+          # single test, saving ~4 minutes of wall-clock.
+          - test-group: schema-migration
+            test-path: "tests/proxy_unit_tests/test_db_schema_migration.py"
+            workers: 0
+            dist: loadscope
+            timeout: 15
+          - test-group: db-and-spend
+            test-path: >-
+              tests/proxy_unit_tests/test_prisma_client_backoff_retry.py
+              tests/proxy_unit_tests/test_db_schema_changes.py
+              tests/proxy_unit_tests/test_e2e_pod_lock_manager.py
+              tests/proxy_unit_tests/test_skills_db.py
+              tests/proxy_unit_tests/test_update_daily_tag_spend.py
+              tests/proxy_unit_tests/test_update_spend.py
+              tests/proxy_unit_tests/test_proxy_encrypt_decrypt.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+
+          # ---- guardrails + budget + hooks: split into 2 ----
+          - test-group: guardrails-hooks
+            test-path: >-
+              tests/proxy_unit_tests/test_proxy_setting_guardrails.py
+              tests/proxy_unit_tests/test_banned_keyword_list.py
+              tests/proxy_unit_tests/test_unit_test_proxy_hooks.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+          - test-group: budgets
+            test-path: >-
+              tests/proxy_unit_tests/test_default_end_user_budget_simple.py
+              tests/proxy_unit_tests/test_unit_test_max_model_budget_limiter.py
+              tests/proxy_unit_tests/test_zero_cost_model_budget_bypass.py
+            workers: 4
+            dist: loadscope
+            timeout: 15
+
+          - test-group: endpoints-and-responses
+            test-path: >-
+              tests/proxy_unit_tests/test_blog_posts_endpoint.py
+              tests/proxy_unit_tests/test_models_fallback_endpoint.py
+              tests/proxy_unit_tests/test_google_endpoint_routing.py
+              tests/proxy_unit_tests/test_google_gemini_proxy_request.py
+              tests/proxy_unit_tests/test_gemini_agents_endpoints.py
+              tests/proxy_unit_tests/test_get_favicon.py
+              tests/proxy_unit_tests/test_get_image.py
+              tests/proxy_unit_tests/test_reducto_ocr_route.py
+              tests/proxy_unit_tests/test_ui_path_detection.py
+              tests/proxy_unit_tests/test_prompt_test_endpoint.py
+              tests/proxy_unit_tests/test_check_batch_cost.py
+              tests/proxy_unit_tests/test_check_responses_cost.py
+              tests/proxy_unit_tests/test_response_polling_handler.py
+              tests/proxy_unit_tests/test_response_polling_pre_call_checks.py
+              tests/proxy_unit_tests/test_realtime_cache.py
+              tests/proxy_unit_tests/test_proxy_exception_mapping.py
+              tests/proxy_unit_tests/test_custom_tokenizer_bug.py
+              tests/proxy_unit_tests/test_model_response_typing
+            workers: 4
+            dist: loadscope
+            timeout: 15
     uses: ./.github/workflows/_test-unit-services-base.yml
     with:
       test-path: ${{ matrix.test-path }}
@@ -42,8 +239,5 @@ jobs:
       reruns: 2
       timeout-minutes: ${{ matrix.timeout }}
       enable-postgres: true
+      dist: ${{ matrix.dist }}
       artifact-name: proxy-db-${{ matrix.test-group }}
-    secrets:
-      DATABASE_URL: ${{ secrets.DATABASE_URL }}
-      POSTGRES_USER: ${{ secrets.POSTGRES_USER }}
-      POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}

.github/workflows/test-unit-proxy-endpoints.yml

@@ -36,6 +36,8 @@ jobs:
         tests/test_litellm/proxy/health_endpoints
         tests/test_litellm/proxy/public_endpoints
         tests/test_litellm/proxy/prompts
+        tests/test_litellm/proxy/rag_endpoints
+        tests/test_litellm/proxy/realtime_endpoints
         tests/test_litellm/proxy/ui_crud_endpoints
       workers: 2
       reruns: 2

.github/workflows/test-unit-proxy-legacy.yml

@@ -28,7 +28,7 @@ jobs:
           - name: "key-generation"
             path: "tests/proxy_unit_tests/test_[k-o]*.py"
           - name: "proxy-config"
-            path: "tests/proxy_unit_tests/test_prisma*.py tests/proxy_unit_tests/test_project*.py tests/proxy_unit_tests/test_prompt*.py tests/proxy_unit_tests/test_proxy_[c-r]*.py"
+            path: "tests/proxy_unit_tests/test_prisma*.py tests/proxy_unit_tests/test_prompt*.py tests/proxy_unit_tests/test_proxy_[c-r]*.py"
           - name: "proxy-server"
             path: "tests/proxy_unit_tests/test_proxy_server.py"
           - name: "proxy-server-extras"

.github/workflows/test-unit-proxy-mgmt-behavior.yml

@@ -0,0 +1,34 @@
+name: "Unit Tests: Proxy Management-Endpoint Behavior Pinning"
+
+on:
+  pull_request:
+    branches:
+      - main
+      - litellm_internal_staging
+      - litellm_oss_branch
+      - "litellm_**"
+
+permissions:
+  contents: read
+  id-token: write
+  pull-requests: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  proxy-mgmt-behavior:
+    uses: ./.github/workflows/_test-unit-services-base.yml
+    with:
+      test-path: tests/proxy_behavior
+      # workers=0 (no xdist): the world seed is a single shared Postgres
+      # state — two xdist workers both call seed_world() and race on the
+      # ``behavior-pin-budget`` row, producing UniqueViolation + cascading
+      # missing-membership FK failures. The whole suite is ~7s sequentially,
+      # so the cost of disabling parallelism here is negligible.
+      workers: 0
+      reruns: 0
+      enable-postgres: true
+      artifact-name: proxy-mgmt-behavior
+      timeout-minutes: 15

.github/workflows/test-unit-security.yml

@@ -1,6 +1,8 @@
 name: "Unit Tests: Security"
 
-# Uses DATABASE_URL secret — only runs on trusted branches, not PRs.
+# Kept push-only (was previously required by DATABASE_URL secret scoping;
+# now the postgres credentials are ephemeral localhost values but the
+# push-trigger stays to match the proxy-db workflow cadence).
 on:
   push:
     branches: [main, "litellm_**"]
@@ -24,7 +26,3 @@ jobs:
       timeout-minutes: 20
       enable-postgres: true
       artifact-name: security
-    secrets:
-      DATABASE_URL: ${{ secrets.DATABASE_URL }}
-      POSTGRES_USER: ${{ secrets.POSTGRES_USER }}
-      POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}

.github/workflows/update_release.py

@@ -1,54 +0,0 @@
-import os
-import requests
-from datetime import datetime
-
-# GitHub API endpoints
-GITHUB_API_URL = "https://api.github.com"
-REPO_OWNER = "BerriAI"
-REPO_NAME = "litellm"
-
-# GitHub personal access token (required for uploading release assets)
-GITHUB_ACCESS_TOKEN = os.environ.get("GITHUB_ACCESS_TOKEN")
-
-# Headers for GitHub API requests
-headers = {
-    "Accept": "application/vnd.github+json",
-    "Authorization": f"Bearer {GITHUB_ACCESS_TOKEN}",
-    "X-GitHub-Api-Version": "2022-11-28",
-}
-
-# Get the latest release
-releases_url = f"{GITHUB_API_URL}/repos/{REPO_OWNER}/{REPO_NAME}/releases/latest"
-response = requests.get(releases_url, headers=headers)
-latest_release = response.json()
-print("Latest release:", latest_release)
-
-# Upload an asset to the latest release
-upload_url = latest_release["upload_url"].split("{?")[0]
-asset_name = "results_stats.csv"
-asset_path = os.path.join(os.getcwd(), asset_name)
-print("upload_url:", upload_url)
-
-with open(asset_path, "rb") as asset_file:
-    asset_data = asset_file.read()
-
-upload_payload = {
-    "name": asset_name,
-    "label": "Load test results",
-    "created_at": datetime.utcnow().isoformat() + "Z",
-}
-
-upload_headers = headers.copy()
-upload_headers["Content-Type"] = "application/octet-stream"
-
-upload_response = requests.post(
-    upload_url,
-    headers=upload_headers,
-    data=asset_data,
-    params=upload_payload,
-)
-
-if upload_response.status_code == 201:
-    print(f"Asset '{asset_name}' uploaded successfully to the latest release.")
-else:
-    print(f"Failed to upload asset. Response: {upload_response.text}")

.gitignore

@@ -90,7 +90,6 @@ test.py
 litellm_config.yaml
 !.github/observatory/litellm_config.yaml
 .cursor
-.vscode/launch.json
 litellm/proxy/to_delete_loadtest_work/*
 update_model_cost_map.py
 tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py
@@ -100,4 +99,25 @@ STABILIZATION_TODO.md
 **/test-results
 **/playwright-report
 **/*.storageState.json
-**/coverage
+**/coverage
+test-config
+
+# ---------- Terraform ----------
+# Provider binaries + module cache — regenerated by `terraform init`.
+**/.terraform/
+# State files often contain secrets (DB passwords, API keys snapshotted from
+# data sources). Keep state in a remote backend, never in git.
+*.tfstate
+*.tfstate.*
+*.tfstate.backup
+# Plan files can also contain sensitive values (variables in plaintext).
+*.tfplan
+# User-specific variable inputs — example files (terraform.tfvars.example) are
+# tracked because they end in .example, which doesn't match the glob below.
+*.tfvars
+*.auto.tfvars
+crash.log
+crash.*.log
+# .terraform.lock.hcl is intentionally NOT ignored — it pins provider versions
+# and should be committed.
+.vscode

.npmrc

@@ -2,4 +2,4 @@
 # Packages needing lifecycle scripts: npm rebuild <pkg>
 ignore-scripts=true
 # Protects local npm install only — npm ci (used in CI) ignores this
-min-release-age=3d
+min-release-age=3

AGENTS.md

@@ -23,10 +23,11 @@ LiteLLM is a unified interface for 100+ LLMs that:
 
 ### Key Directories
 - `tests/` - Comprehensive test suites
-- `docs/my-website/` - Documentation website
 - `ui/litellm-dashboard/` - Admin dashboard UI
 - `enterprise/` - Enterprise-specific features
 
+Documentation lives in the separate [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs) repository and is served at [docs.litellm.ai](https://docs.litellm.ai).
+
 ## DEVELOPMENT GUIDELINES
 
 ### MAKING CODE CHANGES
@@ -218,8 +219,8 @@ When opening issues or pull requests, follow these templates:
 
 ## HELPFUL RESOURCES
 
-- Main documentation: https://docs.litellm.ai/
-- Provider-specific docs in `docs/my-website/docs/providers/`
+- Main documentation: https://docs.litellm.ai/ (source: [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs))
+- Provider-specific docs: https://docs.litellm.ai/docs/providers/
 - Admin UI for testing proxy features
 
 ## WHEN IN DOUBT
@@ -240,10 +241,27 @@ When opening issues or pull requests, follow these templates:
 
 ### Running the proxy server
 
-Start the proxy with a config file:
+Create a minimal config file and start the proxy:
+
+```yaml
+# config.yaml
+model_list:
+  - model_name: fake-openai-endpoint
+    litellm_params:
+      model: openai/fake-model
+      api_key: fake-key
+      api_base: https://fake-api.example.com
+
+general_settings:
+  master_key: sk-1234
+
+litellm_settings:
+  drop_params: True
+  telemetry: False
+```
 
 ```bash
-uv run litellm --config dev_config.yaml --port 4000
+uv run litellm --config config.yaml --port 4000
 ```
 
 The proxy takes ~15-20 seconds to fully start (it runs Prisma migrations on boot). Wait for `/health` to return before sending requests. Without a PostgreSQL `DATABASE_URL`, the proxy connects to a default Neon dev database embedded in the `litellm-proxy-extras` package.

CLAUDE.md

@@ -2,6 +2,10 @@
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
+## Documentation
+
+Documentation lives in a separate repository: [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs). It is served at [docs.litellm.ai](https://docs.litellm.ai). Do not create or edit documentation files in this repository — open doc PRs against `BerriAI/litellm-docs` instead.
+
 ## Development Commands
 
 ### Installation
@@ -113,6 +117,7 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
 - **Always use `antd` for new UI components** — we are migrating off of `@tremor/react`. Do not introduce new `Badge`, `Text`, `Card`, `Grid`, `Title`, or other imports from `@tremor/react` in any new or modified file. Use `antd` equivalents: `Tag` for labels, `Typography.Text` / `Typography.Title` / `Typography.Paragraph` for textual content (avoid plain text-only `<span>`, `<p>`, `<h*>` when Typography fits), and `Card` from `antd`. Note that `antd` has no `"yellow"` Tag color — use `"gold"` for amber/yellow.
 
 ### MCP OAuth / OpenAPI Transport Mapping
+- **`available_on_public_internet: false` with `delegate_auth_to_upstream: true` (oauth2, interactive — not `client_credentials`)** — LiteLLM still allows the anonymous upstream PKCE path (no proxy API key for `/authorize` and matching MCP routes). The internal-only flag mainly affects other surfaces (e.g. IP-based discovery). Rely on the upstream IdP and network policy; the dashboard shows a warning when both are set, and the proxy logs a warning when the server is loaded from config or the database.
 - `TRANSPORT.OPENAPI` is a UI-only concept. The backend only accepts `"http"`, `"sse"`, or `"stdio"`. Always map it to `"http"` before any API call (including pre-OAuth temp-session calls).
 - FastAPI validation errors return `detail` as an array of `{loc, msg, type}` objects. Error extractors must handle: array (map `.msg`), string, nested `{error: string}`, and fallback.
 - When an MCP server already has `authorization_url` stored, skip OAuth discovery (`_discovery_metadata`) — the server URL for OpenAPI MCPs is the spec file, not the API base, and fetching it causes timeouts.
@@ -142,7 +147,7 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
 - **Bound large result sets.** Prisma materializes full results in memory. For results over ~10 MB, paginate with `take`/`skip` or `cursor`/`take`, always with an explicit `order`. Prefer cursor-based pagination (`skip` is O(n)). Don't paginate naturally small result sets.
 - **Limit fetched columns on wide tables.** Use `select` to fetch only needed fields — returns a partial object, so downstream code must not access unselected fields.
 - **Check index coverage.** For new or modified queries, check `schema.prisma` for a supporting index. Prefer extending an existing index (e.g. `@@index([a])` → `@@index([a, b])`) over adding a new one, unless it's a `@@unique`. Only add indexes for large/frequent queries.
-- **Keep schema files in sync.** Apply schema changes to all `schema.prisma` copies (`schema.prisma`, `litellm/proxy/`, `litellm-proxy-extras/`, `litellm-js/spend-logs/` for SpendLogs) with a migration under `litellm-proxy-extras/litellm_proxy_extras/migrations/`.
+- **Keep schema files in sync.** Apply schema changes to all `schema.prisma` copies (`schema.prisma`, `litellm/proxy/`, `litellm-proxy-extras/`) with a migration under `litellm-proxy-extras/litellm_proxy_extras/migrations/`.
 
 ### Setup Wizard (`litellm/setup_wizard.py`)
 - The wizard is implemented as a single `SetupWizard` class with `@staticmethod` methods — keep it that way. No module-level functions except `run_setup_wizard()` (the public entrypoint) and pure helpers (color, ANSI).

Dockerfile

@@ -1,9 +1,9 @@
 # Base image for building
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:a5a619c1793039dcf92f02178f37c94bb3d6001403716da59d6092dfe8d9b502
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
 
 # Runtime image
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:a5a619c1793039dcf92f02178f37c94bb3d6001403716da59d6092dfe8d9b502
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin
 
@@ -27,10 +27,8 @@ RUN apk add --no-cache \
     npm \
     libsndfile
 
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    UV_PROJECT_ENVIRONMENT=/app/.venv \
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
     UV_LINK_MODE=copy \
-    XDG_CACHE_HOME=/app/.cache \
     PATH="/app/.venv/bin:${PATH}"
 
 # Copy dependency metadata first for layer caching
@@ -70,42 +68,34 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
 
 USER root
 
-RUN apk add --no-cache bash openssl tzdata nodejs npm python3 libsndfile supervisor && \
-    npm install -g npm@11.12.1 tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
+RUN apk add --no-cache bash openssl tzdata nodejs npm python3 libsndfile && \
+    npm install -g npm@11.14.0 tar@7.5.11 glob@13.0.6 @isaacs/brace-expansion@5.0.1 brace-expansion@5.0.5 minimatch@10.2.4 diff@8.0.3 picomatch@4.0.4 && \
     GLOBAL="$(npm root -g)" && \
-    find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
+    for pkg in tar glob @isaacs/brace-expansion brace-expansion minimatch diff picomatch; do \
+        name="${pkg##*/}"; \
+        find "$GLOBAL/npm" -type d -name "$name" -path "*/node_modules/$pkg" | while read d; do \
+            rm -rf "$d" && cp -rL "$GLOBAL/$pkg" "$d"; \
+        done; \
     done && \
-    find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
-    done && \
-    find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
-        sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
     npm cache clean --force && \
     { apk del --no-cache npm 2>/dev/null || true; }
 
 WORKDIR /app
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    XDG_CACHE_HOME=/app/.cache \
-    PATH="/app/.venv/bin:${PATH}"
+ENV PATH="/app/.venv/bin:${PATH}"
 
 COPY --from=builder /app /app
+# Prisma binaries live in $HOME/.cache (default prisma-python location),
+# which is /root/.cache here. Copy only the Prisma subdirs — copying the
+# whole /root/.cache drags in the uv build cache (~660 MB, includes a
+# setuptools wheel that surfaces as a CVE finding even though it's not
+# on the runtime sys.path).
+COPY --from=builder /root/.cache/prisma /root/.cache/prisma
+COPY --from=builder /root/.cache/prisma-python /root/.cache/prisma-python
 
 RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
     find /app/.venv -type d -path "*/tornado/test" -delete
 
 EXPOSE 4000/tcp
 
-COPY docker/supervisord.conf /etc/supervisord.conf
-
 ENTRYPOINT ["docker/prod_entrypoint.sh"]
 CMD ["--port", "4000"]

Makefile

@@ -185,3 +185,6 @@ test-llm-translation-single: install-test-deps
 	$(UV_RUN) pytest tests/llm_translation/$(FILE) \
 		--junitxml=test-results/junit.xml \
 		-v --tb=short --maxfail=100 --timeout=300
+
+test-llm-translation-flush-vcr-cache:
+	$(UV_RUN) python tests/_flush_vcr_cache.py

README.md

@@ -68,7 +68,7 @@ Managing LLM calls across providers gets complicated fast — different SDKs, au
     <td><img height="60" alt="Stripe" src="https://github.com/user-attachments/assets/f7296d4f-9fbd-460d-9d05-e4df31697c4b" /></td>
     <td><img height="60" alt="image" src="https://github.com/user-attachments/assets/436fca71-988b-40bb-b5fe-8450c80fdbd0" /></td>
     <td><img height="60" alt="Google ADK" src="https://github.com/user-attachments/assets/caf270a2-5aee-45c4-8222-41a2070c4f19" /></td>
-    <td><img height="60" alt="Greptile" src="https://github.com/user-attachments/assets/0be4bd8a-7cfa-48d3-9090-f415fe948280" /></td>
+    <td><img height="60" alt="Greptile" src="https://github.com/user-attachments/assets/3db0ae72-0843-4005-a56d-bba1dde2193d" /></td>
     <td><img height="60" alt="OpenHands" src="https://github.com/user-attachments/assets/a6150c4c-149e-4cae-888b-8b92be6e003f" /></td>
     <td><h2>Netflix</h2></td>
     <td><img height="60" alt="OpenAI Agents SDK" src="https://github.com/user-attachments/assets/c02f7be0-8c2e-4d27-aea7-7c024bfaebc0" /></td>
@@ -292,7 +292,7 @@ curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
 | [CompactifAI (`compactifai`)](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Custom (`custom`)](https://docs.litellm.ai/docs/providers/custom_llm_server) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Custom OpenAI (`custom_openai`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ | ✅ | ✅ |  |
-| [Dashscope (`dashscope`)](https://docs.litellm.ai/docs/providers/dashscope) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
+| [Dashscope (`dashscope`)](https://docs.litellm.ai/docs/providers/dashscope) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  | ✅ |
 | [Databricks (`databricks`)](https://docs.litellm.ai/docs/providers/databricks) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [DataRobot (`datarobot`)](https://docs.litellm.ai/docs/providers/datarobot) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Deepgram (`deepgram`)](https://docs.litellm.ai/docs/providers/deepgram) | ✅ | ✅ | ✅ |  |  | ✅ |  |  |  |  |
@@ -484,6 +484,8 @@ make format-check   # Check formatting only
 
 For detailed contributing guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).
 
+> **📖 Contributing to documentation?** The LiteLLM docs have moved to a separate repository: [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs). Please open doc PRs there. Docs are served at [docs.litellm.ai](https://docs.litellm.ai).
+
 ## Code Quality / Linting
 
 LiteLLM follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html).

backend/Dockerfile

@@ -0,0 +1,83 @@
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
+
+FROM $UV_IMAGE AS uvbin
+
+# ---------- Builder ----------
+FROM $LITELLM_BUILD_IMAGE AS builder
+
+WORKDIR /app
+USER root
+
+COPY --from=uvbin /uv /uvx /usr/local/bin/
+
+RUN apk add --no-cache bash gcc python3 python3-dev openssl openssl-dev libsndfile
+
+# UV_COMPILE_BYTECODE=1   precompiles .pyc at install time → faster cold start.
+# UV_LINK_MODE=copy       avoids hardlink warnings when uv installs from a
+#                         BuildKit cache mount (different filesystem).
+# UV_PYTHON_DOWNLOADS=0   force uv to use the apk-installed CPython instead of
+#                         silently pulling a managed interpreter.
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
+    UV_LINK_MODE=copy \
+    UV_COMPILE_BYTECODE=1 \
+    UV_PYTHON_DOWNLOADS=0 \
+    PATH="/app/.venv/bin:${PATH}"
+
+# Stage 1 — install dependencies only.
+RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
+    --mount=type=bind,source=uv.lock,target=uv.lock \
+    --mount=type=bind,source=enterprise/pyproject.toml,target=enterprise/pyproject.toml \
+    --mount=type=bind,source=litellm-proxy-extras/pyproject.toml,target=litellm-proxy-extras/pyproject.toml \
+    uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
+        --extra proxy \
+        --extra proxy-runtime \
+        --extra extra_proxy \
+        --extra semantic-router \
+        --python python3
+
+# Stage 2 — copy source and install the project + workspace members.
+COPY . .
+
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-default-groups --no-editable \
+        --extra proxy \
+        --extra proxy-runtime \
+        --extra extra_proxy \
+        --extra semantic-router \
+        --python python3
+
+RUN mkdir -p /home/nonroot && \
+    HOME=/home/nonroot prisma generate --schema=./schema.prisma && \
+    chown -R nonroot:nonroot /home/nonroot/.cache
+
+# ---------- Runtime ----------
+FROM $LITELLM_RUNTIME_IMAGE AS runtime
+
+USER root
+
+RUN apk add --no-cache bash openssl tzdata python3 libsndfile libatomic
+
+# wolfi-base ships an unprivileged `nonroot` account (UID/GID 65532) with
+# /home/nonroot. We run the backend as that user
+WORKDIR /app
+ENV HOME=/home/nonroot \
+    PATH="/app/.venv/bin:${PATH}" \
+    PYTHONPATH="/app" \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
+
+COPY --from=builder --chown=nonroot:nonroot /app /app
+COPY --from=builder --chown=nonroot:nonroot /home/nonroot/.cache /home/nonroot/.cache
+
+RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
+    find /app/.venv -type d -path "*/tornado/test" -delete
+
+USER nonroot
+
+EXPOSE 4001/tcp
+
+ENTRYPOINT ["uvicorn", "backend.main:app"]
+CMD ["--host", "0.0.0.0", "--port", "4001"]

backend/main.py

@@ -0,0 +1,51 @@
+"""UI backend entrypoint.
+
+Reuses the existing FastAPI app from `litellm.proxy.proxy_server` and trims its
+route table to just the management/admin surface used by the dashboard. Purely
+additive — no existing module is modified.
+
+Run with:
+    uvicorn backend.main:app --host 0.0.0.0 --port 4001
+"""
+
+from contextlib import asynccontextmanager
+
+from fastapi.routing import Mount
+
+# See gateway/main.py for why we assemble DATABASE_URL(s) here before
+# importing proxy_server.
+from litellm.proxy.db.db_url_settings import DatabaseURLSettings
+
+DatabaseURLSettings.from_env().apply_to_env()
+
+from litellm.proxy.proxy_server import app
+
+from backend.routes.allowlist import BACKEND_EXACT_PATHS, BACKEND_PATH_PREFIXES
+
+
+def _is_backend_route(route) -> bool:
+    """Keep the route on the backend if its path is in the management surface."""
+    path = getattr(route, "path", None)
+    if path is None:
+        return False
+    if isinstance(route, Mount):
+        # Static UI mounts are served by the dedicated UI container, not here.
+        return False
+    if path in BACKEND_EXACT_PATHS:
+        return True
+    return any(path.startswith(prefix) for prefix in BACKEND_PATH_PREFIXES)
+
+
+# See gateway/main.py for why the trim runs inside the lifespan instead of at
+# module scope.
+_proxy_lifespan = app.router.lifespan_context
+
+
+@asynccontextmanager
+async def _backend_lifespan(app_):
+    async with _proxy_lifespan(app_):
+        app_.router.routes = [r for r in app_.router.routes if _is_backend_route(r)]
+        yield
+
+
+app.router.lifespan_context = _backend_lifespan

backend/routes/allowlist.py

@@ -0,0 +1,135 @@
+"""Path allowlist for the UI backend (control plane) component.
+
+The backend exposes management/admin endpoints consumed by the UI: keys, users,
+teams, orgs, customers, budgets, tags, workflows, model management, spend &
+analytics, settings (router/cache/cost-tracking/fallbacks), SSO/onboarding,
+audit logs, debug, enterprise admin, and UI bootstrap helpers (logo, favicon,
+.well-known config).
+
+Anything LLM data-plane is dropped — those run on the gateway component.
+"""
+
+BACKEND_PATH_PREFIXES: tuple[str, ...] = (
+    # Identity / access
+    "/key/",
+    "/v2/key/",
+    "/user/",
+    "/v2/user/",
+    "/team/",
+    "/v2/team/",
+    "/organization/",
+    "/customer/",
+    "/end_user/",
+    "/sso/",
+    "/login",
+    "/v2/login",
+    "/v3/login",
+    "/logout",
+    "/token",
+    "/onboarding/",
+    "/audit",
+    "/oauth/",
+    "/invitation/",
+    "/jwt/",
+    # Models & routing config
+    "/model/",
+    "/v1/model/info",
+    "/v2/model/",
+    "/model_group",
+    "/model_access_group/",
+    "/model_hub/",
+    "/v1/access_group",
+    "/access_group/",
+    "/router/",
+    "/router_settings",
+    "/adaptive_router/",
+    "/fallback",
+    "/fallbacks",
+    "/cache_settings",
+    "/cost_tracking",
+    "/cost/",
+    "/credentials",
+    "/credential",
+    "/provider/budgets",
+    # Tools / agents (registry & policy admin)
+    "/v1/tool/",
+    "/v1/agents",
+    # Guardrails admin
+    "/v2/guardrails/",
+    # MCP server admin + BYOK OAuth flow (UI-initiated) + dynamic per-server endpoints
+    "/v1/mcp/",
+    "/test/",
+    "/{mcp_server_name}/",
+    # Budgets / tags / workflows / memory mgmt
+    "/budget/",
+    "/tag/",
+    "/workflow/",
+    "/v1/workflows/",
+    "/project/",
+    "/memory/",
+    "/mcp/",
+    # Spend / analytics
+    "/spend/",
+    "/analytics/",
+    "/global/",
+    "/user_agent",
+    "/usage/",
+    "/daily/",
+    # CloudZero cost-export admin (init / settings / export / dry-run / delete)
+    "/cloudzero/",
+    # Caching admin
+    "/cache/",
+    "/caching/",
+    # Callbacks / hooks
+    "/active/callbacks",
+    "/callbacks",
+    "/team_callback",
+    # Alerting / email / IP allowlist
+    "/alerting/",
+    "/email/",
+    "/add/allowed_ip",
+    "/delete/allowed_ip",
+    "/get/",
+    # Enterprise admin
+    "/enterprise/",
+    # Debug / config / profiling
+    "/debug/",
+    "/config/",
+    "/memory-usage-in-mem-cache",
+    "/otel-spans",
+    "/lazy/",
+    "/in_product_nudges",
+    # Admin reload / schedule
+    "/reload/",
+    "/schedule/",
+    "/settings",
+    "/update/",
+    "/upload/",
+    # Dev / admin utilities
+    "/utils/",
+    # UI bootstrap helpers (assets the dashboard fetches)
+    "/get_logo_url",
+    "/get_image",
+    "/get_favicon",
+    "/.well-known/",
+    "/litellm/.well-known/",
+    "/ui_discovery/",
+    "/ui-config",
+    "/sso_settings",
+    "/public/",
+    "/robots.txt",
+    # Health (k8s probes)
+    "/health",
+)
+
+BACKEND_EXACT_PATHS: frozenset[str] = frozenset(
+    {
+        "/",
+        "/routes",
+        "/openapi.json",
+        "/docs",
+        "/docs/oauth2-redirect",
+        "/redoc",
+        "/fallback/login",
+    }
+)

ci_cd/run_migration.py

@@ -1,22 +1,231 @@
+import argparse
 import os
-import subprocess
-from pathlib import Path
-from datetime import datetime
-import testing.postgresql
+import re
 import shutil
+import subprocess
+import sys
+from datetime import datetime
+from pathlib import Path
+
+import testing.postgresql
 
 
-def create_migration(migration_name: str = None):
+DESTRUCTIVE_PATTERN = re.compile(r"\bDROP\s+(COLUMN|TABLE|INDEX)\b", re.IGNORECASE)
+DEFAULT_BASE_BRANCH = "litellm_internal_staging"
+
+
+def _find_destructive_statements(sql: str) -> list:
+    """Return SQL lines containing DROP COLUMN, DROP TABLE, or DROP INDEX."""
+    return [
+        line.strip() for line in sql.splitlines() if DESTRUCTIVE_PATTERN.search(line)
+    ]
+
+
+def _print_freshness_failure(
+    base_branch: str, reason: str, stderr_text: str = ""
+) -> None:
+    """Loudly refuse to run when the freshness check can't be completed."""
+    banner = "=" * 72
+    out = sys.stderr
+    print(banner, file=out)
+    print(f"  FRESHNESS CHECK FAILED — COULD NOT VERIFY origin/{base_branch}", file=out)
+    print(banner, file=out)
+    print("", file=out)
+    print(f"Reason: {reason}", file=out)
+    if stderr_text:
+        print("", file=out)
+        print("git stderr:", file=out)
+        for line in stderr_text.rstrip().splitlines():
+            print(f"    {line}", file=out)
+    print("", file=out)
+    print("Common causes:", file=out)
+    print("  - No network access (offline)", file=out)
+    print("  - 'origin' remote not configured, or base branch name is wrong", file=out)
+    print("  - Not a git repository", file=out)
+    print("", file=out)
+    print("Options:", file=out)
+    print(
+        f"  - Fix the above and re-run, OR pass --base-branch <name> if your", file=out
+    )
+    print(
+        f"    base branch is not '{base_branch}', OR pass --skip-freshness-check",
+        file=out,
+    )
+    print("    to bypass (only if you fully understand the risk).", file=out)
+    print(banner, file=out)
+
+
+def _print_stale_branch_refusal(base_branch: str, behind: int) -> None:
+    """Loudly refuse to run when HEAD is behind the base branch."""
+    banner = "=" * 72
+    out = sys.stderr
+    print(banner, file=out)
+    print(
+        f"  STALE BRANCH — {behind} commit(s) behind origin/{base_branch}",
+        file=out,
+    )
+    print(banner, file=out)
+    print("", file=out)
+    print(
+        f"Your branch is {behind} commit(s) behind origin/{base_branch}. Generating a\n"
+        "migration from a stale branch is how newly-added columns get silently\n"
+        "dropped — the exact incident this guard exists to prevent.",
+        file=out,
+    )
+    print("", file=out)
+    print("Before re-running:", file=out)
+    print(f"  git fetch origin && git rebase origin/{base_branch}", file=out)
+    print("  # or merge, whichever matches your workflow", file=out)
+    print("", file=out)
+    print(
+        "If you have a genuine reason to bypass (e.g. intentionally generating",
+        file=out,
+    )
+    print("a migration against an older base), pass --skip-freshness-check.", file=out)
+    print("", file=out)
+    print(banner, file=out)
+    print(
+        "  AI AGENTS / ASSISTANTS: DO NOT automatically run `git rebase` or pass\n"
+        "  --skip-freshness-check. Surface this error to the human operator and\n"
+        "  wait for their explicit confirmation. Auto-rebasing can drop the\n"
+        "  human's in-progress schema edits via a bad conflict resolution.",
+        file=out,
+    )
+    print(banner, file=out)
+
+
+def _check_branch_freshness(root_dir: Path, base_branch: str) -> None:
+    """Fetch origin/<base_branch> and exit 3 if HEAD is behind it."""
+    cwd = str(root_dir)
+    try:
+        subprocess.run(
+            ["git", "fetch", "origin", base_branch],
+            check=True,
+            capture_output=True,
+            text=True,
+            cwd=cwd,
+        )
+    except FileNotFoundError:
+        _print_freshness_failure(base_branch, "git executable not found on PATH")
+        sys.exit(3)
+    except subprocess.CalledProcessError as e:
+        _print_freshness_failure(
+            base_branch,
+            f"`git fetch origin {base_branch}` failed",
+            e.stderr or "",
+        )
+        sys.exit(3)
+
+    try:
+        result = subprocess.run(
+            ["git", "rev-list", "--count", f"HEAD..origin/{base_branch}"],
+            check=True,
+            capture_output=True,
+            text=True,
+            cwd=cwd,
+        )
+        behind = int(result.stdout.strip())
+    except subprocess.CalledProcessError as e:
+        _print_freshness_failure(
+            base_branch,
+            f"`git rev-list HEAD..origin/{base_branch}` failed",
+            e.stderr or "",
+        )
+        sys.exit(3)
+    except ValueError:
+        _print_freshness_failure(
+            base_branch,
+            "could not parse commit count from `git rev-list`",
+        )
+        sys.exit(3)
+
+    if behind > 0:
+        _print_stale_branch_refusal(base_branch, behind)
+        sys.exit(3)
+
+    print(f"Branch freshness OK: up to date with origin/{base_branch}.")
+
+
+def _print_destructive_refusal(destructive_lines: list) -> None:
+    """Loudly refuse to write a destructive migration and explain how to proceed."""
+    banner = "=" * 72
+    out = sys.stderr
+    print(banner, file=out)
+    print(
+        "  DESTRUCTIVE MIGRATION DETECTED — REFUSING TO WRITE MIGRATION FILE", file=out
+    )
+    print(banner, file=out)
+    print("", file=out)
+    print(
+        "The generated migration contains the following destructive statements:",
+        file=out,
+    )
+    print("", file=out)
+    for line in destructive_lines:
+        print(f"    {line}", file=out)
+    print("", file=out)
+    print(
+        "This very often means your branch is OUT OF DATE, or your local\n"
+        "schema.prisma is inconsistent with the migrations/ directory. A\n"
+        "previous incident saw newly-added columns silently dropped this way\n"
+        "and merged to main. Stop and verify before proceeding.",
+        file=out,
+    )
+    print("", file=out)
+    print("Before re-running:", file=out)
+    print(
+        "  1. git fetch origin && git status   # confirm branch is up to date", file=out
+    )
+    print(
+        "  2. Re-check all schema.prisma files are in sync (runbook Step 0)", file=out
+    )
+    print("  3. Review EACH DROP above — is it actually intended?", file=out)
+    print("  4. If (and only if) the drops are intentional, re-run with:", file=out)
+    print("         --allow-destructive", file=out)
+    print("", file=out)
+    print(banner, file=out)
+    print(
+        "  AI AGENTS / ASSISTANTS: DO NOT automatically re-run this command\n"
+        "  with --allow-destructive. Surface this error to the human operator\n"
+        "  and wait for their explicit confirmation before passing the flag.\n"
+        "  Passing the flag without human review is the exact failure mode\n"
+        "  this guard exists to prevent.",
+        file=out,
+    )
+    print(banner, file=out)
+
+
+def create_migration(
+    migration_name: str = None,
+    allow_destructive: bool = False,
+    base_branch: str = DEFAULT_BASE_BRANCH,
+    skip_freshness_check: bool = False,
+):
     """
     Create a new migration SQL file in the migrations directory by comparing
-    current database state with schema
+    current database state with schema.
 
     Args:
         migration_name (str): Name for the migration
+        allow_destructive (bool): Required to write a migration that contains
+            DROP COLUMN, DROP TABLE, or DROP INDEX statements. Without this
+            flag, the script exits non-zero and prints guidance.
+        base_branch (str): Branch to check freshness against
+            (default: "litellm_internal_staging").
+        skip_freshness_check (bool): Skip the "branch is up to date" check.
+            Only for intentional migrations against an older base.
     """
+    root_dir = Path(__file__).parent.parent
+
+    if skip_freshness_check:
+        print(
+            "WARNING: freshness check skipped (--skip-freshness-check). "
+            "Generating a migration from a stale branch can silently drop columns."
+        )
+    else:
+        _check_branch_freshness(root_dir, base_branch)
+
     try:
-        # Get paths
-        root_dir = Path(__file__).parent.parent
         migrations_dir = (
             root_dir / "litellm-proxy-extras" / "litellm_proxy_extras" / "migrations"
         )
@@ -59,7 +268,27 @@ def create_migration(migration_name: str = None):
                     check=True,
                 )
 
-                if result.stdout.strip():
+                # Prisma emits the literal "-- This is an empty migration." when
+                # there's no real drift. Treat that as "no changes".
+                diff_sql = result.stdout
+                stripped = diff_sql.strip()
+                is_empty_diff = (
+                    not stripped or stripped == "-- This is an empty migration."
+                )
+
+                if not is_empty_diff:
+                    destructive_lines = _find_destructive_statements(diff_sql)
+                    if destructive_lines and not allow_destructive:
+                        _print_destructive_refusal(destructive_lines)
+                        sys.exit(2)
+                    if destructive_lines and allow_destructive:
+                        print(
+                            "WARNING: writing destructive migration "
+                            "(--allow-destructive passed). Statements:"
+                        )
+                        for line in destructive_lines:
+                            print(f"    {line}")
+
                     # Generate timestamp and create migration directory
                     timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
                     migration_name = migration_name or "unnamed_migration"
@@ -68,7 +297,7 @@ def create_migration(migration_name: str = None):
 
                     # Write the SQL to migration.sql
                     migration_file = migration_dir / "migration.sql"
-                    migration_file.write_text(result.stdout)
+                    migration_file.write_text(diff_sql)
 
                     print(f"Created migration in {migration_dir}")
                     return True
@@ -90,8 +319,48 @@ def create_migration(migration_name: str = None):
 
 
 if __name__ == "__main__":
-    # If running directly, can optionally pass migration name as argument
-    import sys
-
-    migration_name = sys.argv[1] if len(sys.argv) > 1 else None
-    create_migration(migration_name)
+    parser = argparse.ArgumentParser(
+        description=(
+            "Generate a Prisma migration by diffing the temp DB "
+            "(existing migrations applied) against schema.prisma."
+        )
+    )
+    parser.add_argument(
+        "migration_name",
+        nargs="?",
+        default=None,
+        help="Name for the migration (used in the generated directory name).",
+    )
+    parser.add_argument(
+        "--allow-destructive",
+        action="store_true",
+        help=(
+            "Required to write a migration that contains DROP COLUMN, "
+            "DROP TABLE, or DROP INDEX. Without this flag, destructive "
+            "diffs are refused."
+        ),
+    )
+    parser.add_argument(
+        "--base-branch",
+        default=DEFAULT_BASE_BRANCH,
+        help=(
+            f"Branch to check freshness against (default: {DEFAULT_BASE_BRANCH}). "
+            "The script fetches origin/<base-branch> and refuses to run if HEAD "
+            "is behind it."
+        ),
+    )
+    parser.add_argument(
+        "--skip-freshness-check",
+        action="store_true",
+        help=(
+            "Bypass the 'branch is up to date' check. Only for intentional "
+            "migrations against an older base. Pairs poorly with automation."
+        ),
+    )
+    args = parser.parse_args()
+    create_migration(
+        args.migration_name,
+        allow_destructive=args.allow_destructive,
+        base_branch=args.base_branch,
+        skip_freshness_check=args.skip_freshness_check,
+    )

codecov.yaml

@@ -1,3 +1,18 @@
+codecov:
+  require_ci_to_pass: false # post coverage status even if CI has unrelated failures
+  notify:
+    wait_for_ci: false # post as soon as expected uploads arrive, don't wait on CI
+
+# Uploads are flagged per workflow/shard (GHA) or "circleci". carryforward makes
+# a re-upload of a flag replace its prior session instead of accumulating a
+# conflicting one, and lets a commit reuse a flag from its parent when that flag
+# was not re-uploaded. Required because the same commit can receive the
+# push-triggered workflows more than once (re-runs / branches cut at the same
+# SHA); flagless overlapping sessions made Codecov drop the largest files.
+flag_management:
+  default_rules:
+    carryforward: true
+
 component_management:
   individual_components:
     - component_id: "Router"
@@ -28,7 +43,7 @@ coverage:
     project:
       default:
         target: auto
-        threshold: 1% # at maximum allow project coverage to drop by 1%
+        threshold: 0% # do not allow project coverage to drop
     patch:
       default:
         target: auto

cookbook/litellm-ollama-docker-image/requirements.txt

@@ -1 +1 @@
-litellm==1.83.5
+litellm==1.83.14

deploy/Dockerfile.ghcr_base

@@ -1,18 +0,0 @@
-# Use the provided base image
-FROM ghcr.io/berriai/litellm:main-latest@sha256:7c311546c25e7bb6e8cafede9fcd3d0d622ac636b5c9418befaa32e85dfb0186
-
-# Set the working directory to /app
-WORKDIR /app
-
-# Copy the configuration file into the container at /app
-COPY config.yaml .
-
-# Make sure your docker/entrypoint.sh is executable
-# Convert Windows line endings to Unix
-RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh
-
-# Expose the necessary port
-EXPOSE 4000/tcp
-
-# Override the CMD instruction with your desired command and arguments
-CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug", "--run_gunicorn"]

deploy/charts/litellm-helm/Chart.yaml

@@ -24,7 +24,7 @@ version: 1.1.0
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: v1.80.12
+appVersion: v1.85.1
 
 annotations:
   org.opencontainers.image.source: "https://github.com/BerriAI/litellm"

deploy/charts/litellm-helm/templates/deployment.yaml

@@ -53,7 +53,7 @@ spec:
         - name: {{ include "litellm.name" . }}
           securityContext:
             {{- toYaml .Values.securityContext | nindent 12 }}
-          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default (printf "main-%s" .Chart.AppVersion) }}"
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           env:
             - name: HOST
@@ -100,6 +100,16 @@ spec:
             - name: DATABASE_URL
               value: {{ .Values.db.url | quote }}
             {{- end }}
+            {{- if and .Values.db.useExisting .Values.db.secret.readReplicaUrlKey }}
+            - name: DATABASE_URL_READ_REPLICA
+              valueFrom:
+                secretKeyRef:
+                  name: {{ .Values.db.secret.name }}
+                  key: {{ .Values.db.secret.readReplicaUrlKey }}
+            {{- else if .Values.db.readReplicaUrl }}
+            - name: DATABASE_URL_READ_REPLICA
+              value: {{ .Values.db.readReplicaUrl | quote }}
+            {{- end }}
             - name: PROXY_MASTER_KEY
               valueFrom:
                 secretKeyRef:
@@ -116,21 +126,32 @@ spec:
                   name: {{ include "redis.secretName" .Subcharts.redis }}
                   key: {{include "redis.secretPasswordKey" .Subcharts.redis }}
             {{- end }}
+            {{- /*
+              Inject LITELLM_LOG only when envVars does not already define it.
+            */}}
+            {{- if and .Values.logLevel (not (hasKey (default dict .Values.envVars) "LITELLM_LOG")) }}
+            - name: LITELLM_LOG
+              value: {{ .Values.logLevel | quote }}
+            {{- end }}
             {{- if .Values.envVars }}
             {{- range $key, $val := .Values.envVars }}
             - name: {{ $key }}
               value: {{ $val | quote }}
             {{- end }}
             {{- end }}
-            {{- if .Values.separateHealthApp }}
-            - name: SEPARATE_HEALTH_APP
-              value: "1"
-            - name: SEPARATE_HEALTH_PORT
-              value: {{ .Values.separateHealthPort | default "8081" | quote }}
-            {{- end }}
             {{- with .Values.extraEnvVars }}
               {{- toYaml . | nindent 12 }}
             {{- end }}
+            {{- if .Values.migrationJob.enabled }}
+            # Schema updates are owned by the dedicated migrations Job; skip
+            # the proxy's startup `prisma db push` so N replicas don't race
+            # one DB on every rollout. Placed last (after envVars and
+            # extraEnvVars) so this override can't be silently shadowed by a
+            # user-supplied DISABLE_SCHEMA_UPDATE under last-wins duplicate-env
+            # semantics — same pattern the migrations Job uses.
+            - name: DISABLE_SCHEMA_UPDATE
+              value: "true"
+            {{- end }}
           envFrom:
           {{- range .Values.environmentSecrets }}
             - secretRef:
@@ -158,15 +179,10 @@ spec:
             - name: http
               containerPort: {{ .Values.service.port }}
               protocol: TCP
-            {{- if .Values.separateHealthApp }}
-            - name: health
-              containerPort: {{ .Values.separateHealthPort | default 8081 }}
-              protocol: TCP
-            {{- end }}
           livenessProbe:
             httpGet:
               path: {{ .Values.livenessProbe.path | quote }}
-              port: {{ if .Values.separateHealthApp }}"health"{{ else }}"http"{{ end }}
+              port: "http"
             initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
             periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
             timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
@@ -175,7 +191,7 @@ spec:
           readinessProbe:
             httpGet:
               path: {{ .Values.readinessProbe.path | quote }}
-              port: {{ if .Values.separateHealthApp }}"health"{{ else }}"http"{{ end }}
+              port: "http"
             initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
             periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
             timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
@@ -184,7 +200,7 @@ spec:
           startupProbe:
             httpGet:
               path: {{ .Values.startupProbe.path | quote }}
-              port: {{ if .Values.separateHealthApp }}"health"{{ else }}"http"{{ end }}
+              port: "http"
             initialDelaySeconds: {{ .Values.startupProbe.initialDelaySeconds }}
             periodSeconds: {{ .Values.startupProbe.periodSeconds }}
             timeoutSeconds: {{ .Values.startupProbe.timeoutSeconds }}

deploy/charts/litellm-helm/templates/hpa.yaml

@@ -12,6 +12,10 @@ spec:
     name: {{ include "litellm.fullname" . }}
   minReplicas: {{ .Values.autoscaling.minReplicas }}
   maxReplicas: {{ .Values.autoscaling.maxReplicas }}
+  {{- if .Values.autoscaling.behavior }}
+  behavior:
+    {{- toYaml .Values.autoscaling.behavior | nindent 4 }}
+  {{- end }}
   metrics:
     {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
     - type: Resource

deploy/charts/litellm-helm/templates/migrations-job.yaml

@@ -41,7 +41,7 @@ spec:
       {{- end }} 
       containers:
         - name: prisma-migrations
-          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default (printf "main-%s" .Chart.AppVersion) }}"
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           securityContext:
             {{- toYaml .Values.securityContext | nindent 12 }}

deploy/charts/litellm-helm/tests/deployment_tests.yaml

@@ -257,16 +257,16 @@ tests:
           value: 0
       - equal:
           path: spec.template.spec.containers[0].livenessProbe.periodSeconds
-          value: 10
+          value: 15
       - equal:
           path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
-          value: 1
+          value: 5
       - equal:
           path: spec.template.spec.containers[0].livenessProbe.successThreshold
           value: 1
       - equal:
           path: spec.template.spec.containers[0].livenessProbe.failureThreshold
-          value: 3
+          value: 5
       - equal:
           path: spec.template.spec.containers[0].readinessProbe.httpGet.path
           value: /health/readiness
@@ -278,7 +278,7 @@ tests:
           value: 10
       - equal:
           path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
-          value: 1
+          value: 5
       - equal:
           path: spec.template.spec.containers[0].readinessProbe.successThreshold
           value: 1
@@ -296,7 +296,7 @@ tests:
           value: 10
       - equal:
           path: spec.template.spec.containers[0].startupProbe.timeoutSeconds
-          value: 1
+          value: 5
       - equal:
           path: spec.template.spec.containers[0].startupProbe.successThreshold
           value: 1

deploy/charts/litellm-helm/tests/hpa_tests.yaml

@@ -0,0 +1,36 @@
+suite: "hpa with behavior"
+templates:
+  - hpa.yaml
+tests:
+  - it: "renders behavior when set"
+    set:
+      autoscaling.enabled: true
+      autoscaling.behavior:
+        scaleUp:
+          stabilizationWindowSeconds: 60
+          policies:
+            - type: Pods
+              value: 2
+              periodSeconds: 60
+        scaleDown:
+          stabilizationWindowSeconds: 90
+          policies:
+            - type: Pods
+              value: 1
+              periodSeconds: 60
+    asserts:
+      - isKind: { of: HorizontalPodAutoscaler }
+      - equal: { path: spec.behavior.scaleUp.stabilizationWindowSeconds, value: 60 }
+      - equal: { path: spec.behavior.scaleDown.stabilizationWindowSeconds, value: 90 }
+
+---
+suite: "hpa without behavior"
+templates:
+  - hpa.yaml
+tests:
+  - it: "does not render behavior when not set"
+    set:
+      autoscaling.enabled: true
+    asserts:
+      - isKind: { of: HorizontalPodAutoscaler }
+      - isNull: { path: spec.behavior }

deploy/charts/litellm-helm/values.yaml

@@ -10,7 +10,7 @@ image:
   repository: ghcr.io/berriai/litellm-database
   pullPolicy: Always
   # Overrides the image tag whose default is the chart appVersion.
-  # tag: "main-latest"
+  # tag: "latest"
   tag: ""
 
 imagePullSecrets: []
@@ -88,26 +88,20 @@ service:
   # optionally specify loadBalancerClass
   # loadBalancerClass: tailscale
 
-# Separate health app configuration
-# When enabled, health checks will use a separate port and the application
-# will receive SEPARATE_HEALTH_APP=1 and SEPARATE_HEALTH_PORT from environment variables
-separateHealthApp: false
-separateHealthPort: 8081
-
-# Probe tuning for proxy container
+# Probes for LiteLLM gateway container
 livenessProbe:
   path: /health/liveliness
   initialDelaySeconds: 0
-  periodSeconds: 10
-  timeoutSeconds: 1
+  periodSeconds: 15
+  timeoutSeconds: 5
   successThreshold: 1
-  failureThreshold: 3
+  failureThreshold: 5
 
 readinessProbe:
   path: /health/readiness
   initialDelaySeconds: 0
   periodSeconds: 10
-  timeoutSeconds: 1
+  timeoutSeconds: 5
   successThreshold: 1
   failureThreshold: 3
 
@@ -115,7 +109,7 @@ startupProbe:
   path: /health/readiness
   initialDelaySeconds: 0
   periodSeconds: 10
-  timeoutSeconds: 1
+  timeoutSeconds: 5
   successThreshold: 1
   failureThreshold: 30
 
@@ -190,6 +184,7 @@ autoscaling:
   maxReplicas: 100
   targetCPUUtilizationPercentage: 80
   # targetMemoryUtilizationPercentage: 80
+  # behavior: {}
 
 # Autoscaling with keda is mutually exclusive with hpa
 keda:
@@ -258,6 +253,26 @@ db:
     passwordKey: password
     # Optional: when set, DATABASE_HOST will be sourced from this secret key instead of db.endpoint
     endpointKey: ""
+    # Optional: when set, DATABASE_URL_READ_REPLICA will be sourced from this
+    # secret key instead of db.readReplicaUrl. Prefer this over the plain
+    # value: read-replica URLs typically embed credentials, and a value
+    # written to db.readReplicaUrl ends up visible in the rendered pod spec
+    # and the Helm release secret.
+    readReplicaUrlKey: ""
+
+  # Optional read-replica routing. When set, the proxy sends read-only
+  # queries (find_*, count, group_by, query_raw/_first) to this URL while
+  # writes continue to go to db.url. Useful for Aurora-style clusters with
+  # separate reader/writer endpoints. Leave empty to keep single-DB behavior.
+  # When IAM_TOKEN_DB_AUTH is enabled, the reader URL is auto-refreshed
+  # alongside the writer (host/port/user/db are parsed from this URL once
+  # at startup; only the IAM token rotates).
+  #
+  # If the URL embeds credentials, prefer db.secret.readReplicaUrlKey over
+  # this field — the plain value is rendered into the pod spec and the
+  # Helm release secret. This field is intended for credential-less URLs
+  # only (e.g. when IAM_TOKEN_DB_AUTH supplies the token at runtime).
+  readReplicaUrl: ""
 
   # Use the Stackgres Helm chart to deploy an instance of a Stackgres cluster.
   #  The Stackgres Operator must already be installed within the target
@@ -329,6 +344,17 @@ migrationJob:
     helm:
       enabled: false
 
+# Log level for the litellm proxy (sets LITELLM_LOG in the deployment env).
+# Rendered as a direct `env:` entry, which in Kubernetes takes precedence over
+# any `envFrom:` source. If you currently source LITELLM_LOG from an
+# environmentSecret or environmentConfigMap, set `logLevel: ""` here to
+# disable injection — otherwise this value silently overrides your secret /
+# configmap entry.
+#
+# Setting LITELLM_LOG inside `envVars:` below also wins: the template skips
+# this injection entirely when envVars already defines LITELLM_LOG.
+logLevel: INFO
+
 # Additional environment variables to be added to the deployment as a map of key-value pairs
 envVars: {}
 

deploy/kubernetes/kub.yaml

@@ -1,56 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: litellm-deployment
-spec:
-  replicas: 3
-  selector:
-    matchLabels:
-      app: litellm
-  template:
-    metadata:
-      labels:
-        app: litellm
-    spec:
-      containers:
-        - name: litellm-container
-          image: ghcr.io/berriai/litellm:main-latest
-          imagePullPolicy: Always
-          env:
-            - name: AZURE_API_KEY
-              value: "d6f****"
-            - name: AZURE_API_BASE
-              value: "https://openai"
-            - name: LITELLM_MASTER_KEY
-              value: "sk-1234"
-            - name: DATABASE_URL
-              value: "postgresql://ishaan*********"
-          args:
-            - "--config"
-            - "/app/proxy_config.yaml"  # Update the path to mount the config file
-          volumeMounts:                 # Define volume mount for proxy_config.yaml
-            - name: config-volume
-              mountPath: /app
-              readOnly: true
-          livenessProbe:
-            httpGet:
-              path: /health/liveliness
-              port: 4000
-            initialDelaySeconds: 120
-            periodSeconds: 15
-            successThreshold: 1
-            failureThreshold: 3
-            timeoutSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /health/readiness
-              port: 4000
-            initialDelaySeconds: 120
-            periodSeconds: 15
-            successThreshold: 1
-            failureThreshold: 3
-            timeoutSeconds: 10
-      volumes:  # Define volume to mount proxy_config.yaml
-        - name: config-volume
-          configMap:
-            name: litellm-config  

deploy/kubernetes/service.yaml

@@ -1,12 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: litellm-service
-spec:
-  selector:
-    app: litellm
-  ports:
-    - protocol: TCP
-      port: 4000
-      targetPort: 4000
-  type: LoadBalancer

dev_config.yaml

@@ -1,13 +0,0 @@
-model_list:
-  - model_name: fake-openai-endpoint
-    litellm_params:
-      model: openai/fake-model
-      api_key: fake-key
-      api_base: https://exampleopenaiendpoint-production.up.railway.app/
-
-general_settings:
-  master_key: sk-1234
-
-litellm_settings:
-  drop_params: True
-  telemetry: False

docker-compose.yml

@@ -16,6 +16,11 @@ services:
       - "4000:4000" # Map the container port to the host, change the host port if necessary
     environment:
       DATABASE_URL: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
+      # Optional: route read-only queries (find_*, count, group_by, query_raw/_first)
+      # to a separate reader endpoint, e.g. an Aurora reader. Leave unset for
+      # single-DB deployments. With IAM_TOKEN_DB_AUTH enabled, the reader URL
+      # is auto-refreshed alongside the writer.
+      # DATABASE_URL_READ_REPLICA: "postgresql://llmproxy:dbpassword9090@db-reader:5432/litellm"
       STORE_MODEL_IN_DB: "True" # allows adding models to proxy via UI
     env_file:
       - .env # Load local .env file

docker/Dockerfile.alpine

@@ -1,68 +0,0 @@
-# Base image for building
-ARG LITELLM_BUILD_IMAGE=python:3.11-alpine@sha256:f07e2ace46f560f09a6eeec7b4913b80ee99546e749ef82342a419a326620856
-
-# Runtime image
-ARG LITELLM_RUNTIME_IMAGE=python:3.11-alpine@sha256:f07e2ace46f560f09a6eeec7b4913b80ee99546e749ef82342a419a326620856
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
-
-FROM $UV_IMAGE AS uvbin
-
-FROM $LITELLM_BUILD_IMAGE AS builder
-
-WORKDIR /app
-
-COPY --from=uvbin /uv /usr/local/bin/uv
-COPY --from=uvbin /uvx /usr/local/bin/uvx
-
-RUN apk add --no-cache gcc python3-dev musl-dev nodejs npm libsndfile
-
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    UV_PROJECT_ENVIRONMENT=/app/.venv \
-    UV_LINK_MODE=copy \
-    XDG_CACHE_HOME=/app/.cache \
-    PATH="/app/.venv/bin:${PATH}"
-
-# Copy dependency metadata first for layer caching
-COPY pyproject.toml uv.lock ./
-COPY enterprise/pyproject.toml enterprise/
-COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
-
-# Install third-party dependencies (cached unless pyproject.toml/uv.lock change)
-RUN uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
-    --extra proxy \
-    --extra proxy-runtime \
-    --extra extra_proxy \
-    --extra semantic-router \
-    --python python3
-
-# Copy full source tree
-COPY . .
-
-# Install project and workspace packages (fast - deps already cached)
-RUN uv sync --frozen --no-default-groups --no-editable \
-    --extra proxy \
-    --extra proxy-runtime \
-    --extra extra_proxy \
-    --extra semantic-router \
-    --python python3
-
-RUN prisma generate --schema=./schema.prisma
-
-RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh && \
-    sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
-
-FROM $LITELLM_RUNTIME_IMAGE AS runtime
-
-RUN apk upgrade --no-cache && apk add --no-cache libsndfile nodejs npm
-
-WORKDIR /app
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    XDG_CACHE_HOME=/app/.cache \
-    PATH="/app/.venv/bin:${PATH}"
-
-COPY --from=builder /app /app
-
-EXPOSE 4000/tcp
-
-ENTRYPOINT ["docker/prod_entrypoint.sh"]
-CMD ["--port", "4000"]

docker/Dockerfile.custom_ui

@@ -1,86 +0,0 @@
-# Use the provided base image
-# NOTE: This is a dev/branch-specific tag. Update digest when the base image is rebuilt.
-FROM ghcr.io/berriai/litellm:litellm_fwd_server_root_path-dev
-
-# Set the working directory to /app
-WORKDIR /app
-
-# Install Node.js and npm (adjust version as needed)
-RUN apt-get update && apt-get upgrade -y \
-        libxml2 \
-        libexpat1 \
-        openssl \
-        libssl3 \
-        git \
-        libkrb5-3 \
-        libglib2.0-0 \
-        wget \
-        libaom3 \
-        libxslt1.1 \
-        libgnutls30 \
-        libc6 && \
-    apt-get install -y --no-install-recommends nodejs npm && \
-    npm install -g npm@11.12.1 tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
-    GLOBAL="$(npm root -g)" && \
-    find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
-        rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
-    done && \
-    find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
-        sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
-    npm cache clean --force && \
-    apt-get purge -y npm
-
-# Copy the UI source into the container
-COPY ./ui/litellm-dashboard /app/ui/litellm-dashboard
-
-# Set an environment variable for UI_BASE_PATH
-# This can be overridden at build time
-# set UI_BASE_PATH to "<your server root path>/ui"
-ENV UI_BASE_PATH="/prod/ui"
-
-# Build the UI with the specified UI_BASE_PATH
-WORKDIR /app/ui/litellm-dashboard
-RUN npm ci
-RUN UI_BASE_PATH=$UI_BASE_PATH npm run build
-
-# Create the destination directory
-RUN mkdir -p /app/litellm/proxy/_experimental/out
-
-# Move the built files to the appropriate location
-# Assuming the build output is in ./out directory
-RUN rm -rf /app/litellm/proxy/_experimental/out/* && \
-    mv ./out/* /app/litellm/proxy/_experimental/out/
-
-# Switch back to the main app directory
-WORKDIR /app
-
-# Make sure your docker/entrypoint.sh is executable
-# Convert Windows line endings to Unix for entrypoint scripts
-RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh
-RUN sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
-
-# Run as non-root user
-RUN groupadd --gid 1000 appuser && useradd --uid 1000 --gid 1000 --no-create-home appuser \
-    && chown -R appuser:appuser /app
-USER appuser
-
-# Expose the necessary port
-EXPOSE 4000/tcp
-
-HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
-  CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health')"]
-
-# Override the CMD instruction with your desired command and arguments
-CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug"]

docker/Dockerfile.database

@@ -1,9 +1,9 @@
 # Base image for building
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:a5a619c1793039dcf92f02178f37c94bb3d6001403716da59d6092dfe8d9b502
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
 
 # Runtime image
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:a5a619c1793039dcf92f02178f37c94bb3d6001403716da59d6092dfe8d9b502
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin
 
@@ -26,10 +26,8 @@ RUN apk add --no-cache \
     npm \
     libsndfile
 
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    UV_PROJECT_ENVIRONMENT=/app/.venv \
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
     UV_LINK_MODE=copy \
-    XDG_CACHE_HOME=/app/.cache \
     PATH="/app/.venv/bin:${PATH}"
 
 # Copy dependency metadata first for layer caching
@@ -68,7 +66,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
 
 USER root
 
-RUN apk add --no-cache bash openssl tzdata nodejs npm python3 libsndfile supervisor && \
+RUN apk add --no-cache bash openssl tzdata nodejs npm python3 libsndfile && \
     npm install -g npm@11.12.1 tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
     GLOBAL="$(npm root -g)" && \
     find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
@@ -86,24 +84,23 @@ RUN apk add --no-cache bash openssl tzdata nodejs npm python3 libsndfile supervi
     find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
         rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
     done && \
-    find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
-        sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
     npm cache clean --force && \
     { apk del --no-cache npm 2>/dev/null || true; }
 
 WORKDIR /app
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    XDG_CACHE_HOME=/app/.cache \
-    PATH="/app/.venv/bin:${PATH}"
+ENV PATH="/app/.venv/bin:${PATH}"
 
 COPY --from=builder /app /app
+# Prisma binaries live in $HOME/.cache (default prisma-python location),
+# which is /root/.cache here. Copy them from the builder so they survive
+# deployments that volume-mount /app/.cache (e.g. readOnlyRootFilesystem
+# + emptyDir) — otherwise the mount would shadow the baked-in query engine.
+COPY --from=builder /root/.cache /root/.cache
 
 RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
     find /app/.venv -type d -path "*/tornado/test" -delete
 
 EXPOSE 4000/tcp
 
-COPY docker/supervisord.conf /etc/supervisord.conf
-
 ENTRYPOINT ["docker/prod_entrypoint.sh"]
 CMD ["--port", "4000"]

docker/Dockerfile.dev

@@ -1,121 +0,0 @@
-# Base image for building
-ARG LITELLM_BUILD_IMAGE=python:3.13-slim@sha256:739e7213785e88c0f702dcdc12c0973afcbd606dbf021a589cab77d6b00b579d
-
-# Runtime image
-ARG LITELLM_RUNTIME_IMAGE=python:3.13-slim@sha256:739e7213785e88c0f702dcdc12c0973afcbd606dbf021a589cab77d6b00b579d
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
-
-FROM $UV_IMAGE AS uvbin
-
-FROM $LITELLM_BUILD_IMAGE AS builder
-
-WORKDIR /app
-USER root
-
-COPY --from=uvbin /uv /usr/local/bin/uv
-COPY --from=uvbin /uvx /usr/local/bin/uvx
-
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    gcc \
-    g++ \
-    python3-dev \
-    libssl-dev \
-    pkg-config \
-    nodejs \
-    npm \
-    && rm -rf /var/lib/apt/lists/*
-
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    UV_PROJECT_ENVIRONMENT=/app/.venv \
-    UV_LINK_MODE=copy \
-    XDG_CACHE_HOME=/app/.cache \
-    PATH="/app/.venv/bin:${PATH}"
-
-# Copy dependency metadata first for layer caching
-COPY pyproject.toml uv.lock ./
-COPY enterprise/pyproject.toml enterprise/
-COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
-
-# Install third-party dependencies (cached unless pyproject.toml/uv.lock change)
-RUN uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
-    --extra proxy \
-    --extra proxy-runtime \
-    --extra extra_proxy \
-    --extra semantic-router \
-    --python python
-
-# Copy full source tree
-COPY . .
-
-# Build Admin UI before final sync
-RUN sed -i 's/\r$//' docker/build_admin_ui.sh && chmod +x docker/build_admin_ui.sh && ./docker/build_admin_ui.sh
-
-# Install project and workspace packages (fast - deps already cached)
-RUN uv sync --frozen --no-default-groups --no-editable \
-    --extra proxy \
-    --extra proxy-runtime \
-    --extra extra_proxy \
-    --extra semantic-router \
-    --python python
-
-RUN prisma generate --schema=./schema.prisma
-
-RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh && \
-    sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
-
-FROM $LITELLM_RUNTIME_IMAGE AS runtime
-
-USER root
-
-RUN apt-get update && apt-get upgrade -y \
-        libxml2 \
-        libexpat1 \
-        openssl \
-        libssl3 \
-        git \
-        libkrb5-3 \
-        libglib2.0-0 \
-        wget \
-        libaom3 \
-        libxslt1.1 \
-        libgnutls30 \
-        libc6 \
-    && apt-get install -y --no-install-recommends \
-        libssl3 \
-        libatomic1 \
-        nodejs \
-        npm \
-    && rm -rf /var/lib/apt/lists/* \
-    && npm install -g npm@11.12.1 tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 \
-    && GLOBAL="$(npm root -g)" \
-    && find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
-          rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
-       done \
-    && find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
-          rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
-       done \
-    && find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
-          rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
-       done \
-    && find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
-          rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
-       done \
-    && find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
-          rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
-       done \
-    && find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
-        sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null \
-    && npm cache clean --force \
-    && apt-get purge -y npm
-
-WORKDIR /app
-ENV PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    XDG_CACHE_HOME=/app/.cache \
-    PATH="/app/.venv/bin:${PATH}"
-
-COPY --from=builder /app /app
-
-EXPOSE 4000/tcp
-
-ENTRYPOINT ["docker/prod_entrypoint.sh"]
-CMD ["--port", "4000"]

docker/Dockerfile.health_check

@@ -1,30 +0,0 @@
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
-FROM $UV_IMAGE AS uvbin
-
-FROM python:3.13-slim@sha256:739e7213785e88c0f702dcdc12c0973afcbd606dbf021a589cab77d6b00b579d
-
-WORKDIR /app
-
-# Copy the uv binary and the health check script.
-COPY --from=uvbin /uv /usr/local/bin/uv
-COPY pyproject.toml uv.lock /app/
-COPY scripts/health_check/health_check_client.py /app/health_check_client.py
-
-# Resolve and install the health-check dependencies from the project lockfile
-# so the runtime image stays self-contained and reproducible.
-RUN uv export --frozen --no-default-groups --only-group healthcheck --no-emit-project --no-hashes --output-file /tmp/health-check-requirements.txt \
-  && uv pip install --system -r /tmp/health-check-requirements.txt \
-  && rm /tmp/health-check-requirements.txt \
-  && rm /app/pyproject.toml /app/uv.lock \
-  && chmod +x /app/health_check_client.py
-
-# Run as non-root user
-RUN groupadd --gid 1000 appuser && useradd --uid 1000 --gid 1000 --no-create-home appuser
-USER appuser
-
-# Health check
-HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
-  CMD ["python", "/app/health_check_client.py", "--help"]
-
-# Set entrypoint
-ENTRYPOINT ["python", "/app/health_check_client.py"]

docker/Dockerfile.non_root

@@ -1,8 +1,8 @@
 # Base images
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:a5a619c1793039dcf92f02178f37c94bb3d6001403716da59d6092dfe8d9b502
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:a5a619c1793039dcf92f02178f37c94bb3d6001403716da59d6092dfe8d9b502
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
 ARG PROXY_EXTRAS_SOURCE=published
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin
 
@@ -15,32 +15,24 @@ COPY --from=uvbin /uv /usr/local/bin/uv
 COPY --from=uvbin /uvx /usr/local/bin/uvx
 
 RUN for i in 1 2 3; do \
-    apk add --no-cache \
-      python3 \
-      python3-dev \
-      clang \
-      llvm \
-      lld \
-      gcc \
-      linux-headers \
-      build-base \
-      bash \
-      coreutils \
-      curl \
-      openssl \
-      openssl-dev \
-      nodejs \
-      npm \
-      libsndfile && break || sleep 5; \
+      apk add --no-cache \
+        python3 \
+        python3-dev \
+        gcc \
+        bash \
+        coreutils \
+        curl \
+        openssl \
+        libsndfile \
+        nodejs \
+        npm && break || sleep 5; \
     done
 
 ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
     UV_LINK_MODE=copy \
-    NVM_DIR=/root/.nvm \
-    PATH="/root/.nvm/versions/node/v20.20.2/bin:/app/.venv/bin:${PATH}" \
+    PATH="/app/.venv/bin:${PATH}" \
     LITELLM_NON_ROOT=true \
     PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    PRISMA_CLI_BINARY_TARGETS="debian-openssl-3.0.x" \
     XDG_CACHE_HOME=/app/.cache
 
 # Copy dependency metadata first for layer caching
@@ -49,7 +41,8 @@ COPY enterprise/pyproject.toml enterprise/
 COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
 
 # Install third-party dependencies (cached unless pyproject.toml/uv.lock change)
-RUN uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
+RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
+    uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
     --extra proxy \
     --extra proxy-runtime \
     --extra extra_proxy \
@@ -62,38 +55,12 @@ COPY . .
 # Set non-root flag for build time consistency
 ENV LITELLM_NON_ROOT=true
 
-# Build Admin UI once and stage the static output for the runtime image.
-# NOTE: .npmrc files (which may set ignore-scripts=true and min-release-age=3d)
-# are temporarily renamed during npm install/ci so they don't block lifecycle
-# scripts needed by the build. This is safe because npm ci installs from
-# package-lock.json with pinned versions + integrity hashes.
+# Stage the pre-built Admin UI from the checked-in Next.js static export.
+# _experimental/out/ is regenerated as part of the release runbook.
+# Restructure extensionless routes (foo.html -> foo/index.html) to match the layout
+# proxy_server.py expects, and drop a readiness marker.
 RUN mkdir -p /var/lib/litellm/ui /var/lib/litellm/assets && \
-    ([ -f /app/.npmrc ] && mv /app/.npmrc /app/.npmrc.bak || true) && \
-    NVM_VERSION="v0.40.4" && \
-    NVM_CHECKSUM="4b7412c49960c7d31e8df72da90c1fb5b8cccb419ac99537b737028d497aba4f" && \
-    NODE_VERSION="v20.20.2" && \
-    NVM_SCRIPT="/tmp/install-nvm.sh" && \
-    curl -fsSL "https://raw.githubusercontent.com/nvm-sh/nvm/${NVM_VERSION}/install.sh" -o "$NVM_SCRIPT" && \
-    echo "${NVM_CHECKSUM}  ${NVM_SCRIPT}" | sha256sum -c - && \
-    bash "$NVM_SCRIPT" && \
-    export NVM_DIR="$HOME/.nvm" && \
-    . "$NVM_DIR/nvm.sh" && \
-    nvm install "${NODE_VERSION}" && \
-    nvm use "${NODE_VERSION}" && \
-    npm install -g npm@11.12.1 && \
-    npm install -g node-gyp@12.2.0 && \
-    ln -sf "$(npm root -g)/node-gyp" "$(npm root -g)/npm/node_modules/node-gyp" && \
-    npm cache clean --force && \
-    cd /app/ui/litellm-dashboard && \
-    if [ -f "/app/enterprise/enterprise_ui/enterprise_colors.json" ]; then \
-      cp /app/enterprise/enterprise_ui/enterprise_colors.json ./ui_colors.json; \
-    fi && \
-    ([ -f .npmrc ] && mv .npmrc .npmrc.bak || true) && \
-    npm ci --no-audit --no-fund && \
-    ([ -f .npmrc.bak ] && mv .npmrc.bak .npmrc || true) && \
-    ([ -f /app/.npmrc.bak ] && mv /app/.npmrc.bak /app/.npmrc || true) && \
-    npm run build && \
-    cp -r /app/ui/litellm-dashboard/out/* /var/lib/litellm/ui/ && \
+    cp -r /app/litellm/proxy/_experimental/out/. /var/lib/litellm/ui/ && \
     cp /app/litellm/proxy/logo.jpg /var/lib/litellm/assets/logo.jpg && \
     ( cd /var/lib/litellm/ui && \
       for html_file in *.html; do \
@@ -103,10 +70,10 @@ RUN mkdir -p /var/lib/litellm/ui /var/lib/litellm/assets && \
           mv "$html_file" "$folder_name/index.html"; \
         fi; \
       done && \
-      touch .litellm_ui_ready ) && \
-    cd /app/ui/litellm-dashboard && rm -rf ./out
+      touch .litellm_ui_ready )
 
-RUN if [ "$PROXY_EXTRAS_SOURCE" = "published" ]; then \
+RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
+    if [ "$PROXY_EXTRAS_SOURCE" = "published" ]; then \
       uv sync --frozen --no-default-groups --no-editable \
         --extra proxy \
         --extra proxy-runtime \
@@ -123,10 +90,7 @@ RUN if [ "$PROXY_EXTRAS_SOURCE" = "published" ]; then \
         --python python3; \
     fi
 
-RUN mkdir -p /app/.cache/npm && \
-    prisma generate --schema=./schema.prisma && \
-    prisma --version && \
-    prisma migrate diff --from-empty --to-schema-datamodel ./schema.prisma --script > /dev/null 2>&1 || true
+RUN prisma generate --schema=./schema.prisma
 
 RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh && \
     sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
@@ -137,57 +101,28 @@ WORKDIR /app
 USER root
 
 RUN for i in 1 2 3; do \
-    apk upgrade --no-cache && break || sleep 5; \
+      apk upgrade --no-cache && break || sleep 5; \
     done && \
     for i in 1 2 3; do \
-      apk add --no-cache python3 bash openssl tzdata nodejs npm supervisor libsndfile && break || sleep 5; \
-    done && \
-    apk upgrade --no-cache nodejs && \
-    npm install -g npm@11.12.1 tar@7.5.11 glob@11.1.0 @isaacs/brace-expansion@5.0.1 minimatch@10.2.4 diff@8.0.3 && \
-    GLOBAL="$(npm root -g)" && \
-    find "$GLOBAL/npm" -type d -name "tar" -path "*/node_modules/tar" | while read d; do \
-      rm -rf "$d" && cp -rL "$GLOBAL/tar" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "glob" -path "*/node_modules/glob" | while read d; do \
-      rm -rf "$d" && cp -rL "$GLOBAL/glob" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "brace-expansion" -path "*/node_modules/@isaacs/brace-expansion" | while read d; do \
-      rm -rf "$d" && cp -rL "$GLOBAL/@isaacs/brace-expansion" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "minimatch" -path "*/node_modules/minimatch" | while read d; do \
-      rm -rf "$d" && cp -rL "$GLOBAL/minimatch" "$d"; \
-    done && \
-    find "$GLOBAL/npm" -type d -name "diff" -path "*/node_modules/diff" | while read d; do \
-      rm -rf "$d" && cp -rL "$GLOBAL/diff" "$d"; \
-    done && \
-    find /usr/local/lib /usr/lib -path "*/node_modules/npm/package.json" -exec \
-      sed -i 's/"tar": "\^7\.5\.[0-9]*"/"tar": "^7.5.10"/g; s/"minimatch": "\^10\.[0-9.]*"/"minimatch": "^10.2.4"/g' {} + 2>/dev/null && \
-    npm cache clean --force && \
-    { apk del --no-cache npm 2>/dev/null || true; }
+      apk add --no-cache python3 bash openssl tzdata libsndfile nodejs && break || sleep 5; \
+    done
 
 COPY --from=builder /app /app
 COPY --from=builder /var/lib/litellm/ui /var/lib/litellm/ui
 COPY --from=builder /var/lib/litellm/assets /var/lib/litellm/assets
-COPY --from=builder /app/docker/supervisord.conf /etc/supervisord.conf
 
 ENV PATH="/app/.venv/bin:${PATH}" \
     PRISMA_BINARY_CACHE_DIR=/app/.cache/prisma-python/binaries \
-    PRISMA_CLI_BINARY_TARGETS="debian-openssl-3.0.x" \
     HOME=/app \
     LITELLM_NON_ROOT=true \
     XDG_CACHE_HOME=/app/.cache \
     PRISMA_SKIP_POSTINSTALL_GENERATE=1 \
     PRISMA_HIDE_UPDATE_MESSAGE=1 \
     PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING=1 \
-    NPM_CONFIG_CACHE=/app/.cache/npm \
-    NPM_CONFIG_PREFER_OFFLINE=true \
     PRISMA_OFFLINE_MODE=true
 
-RUN sed -i 's/\r$//' docker/entrypoint.sh && \
-    sed -i 's/\r$//' docker/prod_entrypoint.sh && \
-    chmod +x docker/entrypoint.sh docker/prod_entrypoint.sh && \
-    mkdir -p /nonexistent /.npm /var/lib/litellm/assets /var/lib/litellm/ui /tmp/.npm && \
-    chown -R nobody:nogroup /app /var/lib/litellm/ui /var/lib/litellm/assets /nonexistent /.npm /tmp/.npm && \
+RUN mkdir -p /nonexistent /var/lib/litellm/assets /var/lib/litellm/ui && \
+    chown -R nobody:nogroup /app /var/lib/litellm/ui /var/lib/litellm/assets /nonexistent && \
     PRISMA_PATH=$(python -c "import os, prisma; print(os.path.dirname(prisma.__file__))") && \
     chown -R nobody:nogroup "$PRISMA_PATH" && \
     LITELLM_PKG_MIGRATIONS_PATH="$(python -c 'import os, litellm_proxy_extras; print(os.path.dirname(litellm_proxy_extras.__file__))' 2>/dev/null || echo '')/migrations" && \
@@ -201,7 +136,7 @@ RUN sed -i 's/\r$//' docker/entrypoint.sh && \
     [ -n "$LITELLM_PROXY_EXTRAS_PATH" ] && chmod -R g+w "$LITELLM_PROXY_EXTRAS_PATH" || true && \
     chmod -R g+rX "$PRISMA_PATH" /var/lib/litellm/ui /var/lib/litellm/assets /app/.cache
 
-USER nobody
+USER 65534
 
 RUN prisma generate --schema=./schema.prisma
 

docker/prod_entrypoint.sh

@@ -1,14 +1,8 @@
 #!/bin/sh
 
-if [ "$SEPARATE_HEALTH_APP" = "1" ]; then
-    export LITELLM_ARGS="$@"
-    export SUPERVISORD_STOPWAITSECS="${SUPERVISORD_STOPWAITSECS:-3600}"
-    exec supervisord -c /etc/supervisord.conf
-fi
-
 if [ "$USE_DDTRACE" = "true" ]; then
     export DD_TRACE_OPENAI_ENABLED="False"
     exec ddtrace-run litellm "$@"
 else
     exec litellm "$@"
-fi
+fi

docker/supervisord.conf

@@ -1,46 +0,0 @@
-[supervisord]
-nodaemon=true
-loglevel=info
-logfile=/tmp/supervisord.log
-pidfile=/tmp/supervisord.pid
-
-[group:litellm]
-programs=main,health
-
-[program:main]
-command=sh -c 'if [ "$USE_DDTRACE" = "true" ]; then export DD_TRACE_OPENAI_ENABLED="False"; exec ddtrace-run python -m litellm.proxy.proxy_cli --host 0.0.0.0 --port=4000 $LITELLM_ARGS; else exec python -m litellm.proxy.proxy_cli --host 0.0.0.0 --port=4000 $LITELLM_ARGS; fi'
-autostart=true
-autorestart=true
-startretries=3
-priority=1
-exitcodes=0
-stopasgroup=true
-killasgroup=true
-stopwaitsecs=%(ENV_SUPERVISORD_STOPWAITSECS)s
-stdout_logfile=/dev/stdout
-stderr_logfile=/dev/stderr
-stdout_logfile_maxbytes = 0
-stderr_logfile_maxbytes = 0
-environment=PYTHONUNBUFFERED=true
-
-[program:health]
-command=sh -c '[ "$SEPARATE_HEALTH_APP" = "1" ] && exec uvicorn litellm.proxy.health_endpoints.health_app_factory:build_health_app --factory --host 0.0.0.0 --port=${SEPARATE_HEALTH_PORT:-4001} || exit 0'
-autostart=true
-autorestart=true
-startretries=3
-priority=2
-exitcodes=0
-stopasgroup=true
-killasgroup=true
-stopwaitsecs=%(ENV_SUPERVISORD_STOPWAITSECS)s
-stdout_logfile=/dev/stdout
-stderr_logfile=/dev/stderr
-stdout_logfile_maxbytes = 0
-stderr_logfile_maxbytes = 0
-environment=PYTHONUNBUFFERED=true
-
-[eventlistener:process_monitor]
-command=python -c "from supervisor import childutils; import os, signal; [os.kill(os.getppid(), signal.SIGTERM) for h,p in iter(lambda: childutils.listener.wait(), None) if h['eventname'] in ['PROCESS_STATE_FATAL', 'PROCESS_STATE_EXITED'] and dict([x.split(':') for x in p.split(' ')])['processname'] in ['main', 'health'] or childutils.listener.ok()]"
-events=PROCESS_STATE_EXITED,PROCESS_STATE_FATAL
-autostart=true
-autorestart=true

docker/tests/nonroot.yaml

@@ -2,7 +2,7 @@ schemaVersion: 2.0.0
 
 metadataTest:
   entrypoint: ["docker/prod_entrypoint.sh"]
-  user: "nobody"
+  user: "65534"
   workdir: "/app"
 
 fileExistenceTests:

docs/my-website/.gitignore

@@ -1,23 +0,0 @@
-# Dependencies
-/node_modules
-
-# Production
-/build
-
-# Generated files
-.docusaurus
-.cache-loader
-
-# Misc
-.DS_Store
-.env
-.env.local
-.env.development.local
-.env.test.local
-.env.production.local
-
-npm-debug.log*
-yarn-debug.log*
-yarn-error.log*
-yarn.lock
-pnpm-lock.yaml

docs/my-website/Dockerfile

@@ -1,32 +0,0 @@
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9
-
-FROM $UV_IMAGE AS uvbin
-
-FROM python:3.14.0a3-slim
-
-COPY --from=uvbin /uv /usr/local/bin/uv
-COPY --from=uvbin /uvx /usr/local/bin/uvx
-COPY . /app
-WORKDIR /app
-
-ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
-    UV_LINK_MODE=copy \
-    PATH="/app/.venv/bin:${PATH}"
-
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    gcc \
-    python3-dev \
-    libssl-dev \
-    pkg-config \
-    && rm -rf /var/lib/apt/lists/*
-
-RUN uv sync --frozen --no-default-groups --no-editable \
-    --extra proxy \
-    --extra proxy-runtime \
-    --extra extra_proxy \
-    --extra semantic-router \
-    --python python
-
-EXPOSE $PORT 
-
-CMD ["sh", "-c", "litellm --host 0.0.0.0 --port $PORT --workers 10 --config config.yaml"]

docs/my-website/README.md

@@ -1,41 +0,0 @@
-# Website
-
-This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
-
-### Installation
-
-```
-$ yarn
-```
-
-### Local Development
-
-```
-$ yarn start
-```
-
-This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
-
-### Build
-
-```
-$ yarn build
-```
-
-This command generates static content into the `build` directory and can be served using any static contents hosting service.
-
-### Deployment
-
-Using SSH:
-
-```
-$ USE_SSH=true yarn deploy
-```
-
-Not using SSH:
-
-```
-$ GIT_USER=<Your GitHub username> yarn deploy
-```
-
-If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.

docs/my-website/babel.config.js

@@ -1,3 +0,0 @@
-module.exports = {
-  presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
-};

docs/my-website/blog/anthropic_wildcard_model_access_incident/index.md

@@ -1,138 +0,0 @@
----
-slug: anthropic-wildcard-model-access-incident
-title: "Incident Report: Wildcard Blocking New Models After Cost Map Reload"
-date: 2026-02-23T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-tags: [incident-report, proxy, auth, model-access]
-hide_table_of_contents: false
----
-
-**Date:** Feb 23, 2026  
-**Duration:** ~3 hours  
-**Severity:** High (for users with provider wildcard access rules)  
-**Status:** Resolved
-
-## Summary
-
-When a new Anthropic model (e.g. `claude-sonnet-4-6`) was added to the LiteLLM model cost map and a cost map reload was triggered, requests to the new model were rejected with:
-
-```
-key not allowed to access model. This key can only access models=['anthropic/*']. Tried to access claude-sonnet-4-6.
-```
-
-The reload updated `litellm.model_cost` correctly but never re-ran `add_known_models()`, so `litellm.anthropic_models` (the in-memory set used by the wildcard resolver) remained stale. The new model was invisible to the `anthropic/*` wildcard even though the cost map knew about it.
-
-- **LLM calls:** All requests to newly-added Anthropic models were blocked with a 401.
-- **Existing models:** Unaffected — only models missing from the stale provider set were impacted.
-- **Other providers:** Same bug class existed for any provider wildcard (e.g. `openai/*`, `gemini/*`).
-
-{/* truncate */}
-
----
-
-## Background
-
-LiteLLM supports provider-level wildcard access rules. When an admin configures a key or team with `models=['anthropic/*']`, any model whose provider resolves to `anthropic` should be allowed. The resolution happens in `_model_custom_llm_provider_matches_wildcard_pattern`:
-
-```mermaid
-flowchart TD
-    A["1. Request arrives for claude-sonnet-4-6"] --> B["2. Auth check: can this key call this model?
-    proxy/auth/auth_checks.py"]
-    B --> C["3. Key has models=['anthropic/*']
-    → wildcard match attempted"]
-    C --> D["4. get_llm_provider('claude-sonnet-4-6')
-    checks litellm.anthropic_models set"]
-    D -->|"model IN set"| E["5a. ✅ Provider = 'anthropic'
-    → 'anthropic/claude-sonnet-4-6' matches 'anthropic/*'"]
-    D -->|"model NOT IN set"| F["5b. ❌ Provider unknown
-    → exception raised → wildcard returns False"]
-    E --> G["6. Request allowed"]
-    F --> H["6. 401: key not allowed to access model"]
-
-    style E fill:#d4edda,stroke:#28a745
-    style F fill:#f8d7da,stroke:#dc3545
-    style H fill:#f8d7da,stroke:#dc3545
-    style D fill:#fff3cd,stroke:#ffc107
-```
-
-`litellm.anthropic_models` is a Python `set` populated at import time by `add_known_models()`. It is the source `get_llm_provider()` consults to map a bare model name like `claude-sonnet-4-6` to the provider string `"anthropic"`.
-
----
-
-## Root Cause
-
-`add_known_models()` is called **once** at module import time. Both reload paths in `proxy_server.py` updated `litellm.model_cost` with the fresh map but never called `add_known_models()` again:
-
-```python
-# Before the fix — both reload paths looked like this:
-new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
-litellm.model_cost = new_model_cost_map          # ✅ cost map updated
-_invalidate_model_cost_lowercase_map()           # ✅ cache cleared
-# ❌ add_known_models() never called
-#    → litellm.anthropic_models still has the old set
-#    → new model not in the set
-#    → get_llm_provider() raises for the new model
-#    → wildcard match returns False
-#    → 401 for every request to the new model
-```
-
-The gap existed in two places:
-1. `_check_and_reload_model_cost_map` — the periodic automatic reload (every 10 s)
-2. The `/reload/model_cost_map` admin endpoint — the manual reload
-
-**Timeline:**
-
-1. New model (`claude-sonnet-4-6`) added to `model_prices_and_context_window.json`
-2. Admin triggers cost map reload via UI → `litellm.model_cost` updated
-3. Users with `anthropic/*` wildcard keys attempt requests to `claude-sonnet-4-6`
-4. `get_llm_provider('claude-sonnet-4-6')` raises → wildcard returns False → 401
-5. Admin reloads cost map again — same result (root cause not addressed)
-6. ~3 hours of investigation → root cause identified → fix deployed
-
----
-
-## The Fix
-
-After each reload, `add_known_models()` is called with the freshly fetched map passed explicitly. Passing the map directly (rather than relying on the module-level reference) removes any ambiguity about which dict is iterated:
-
-```python
-# After the fix — both reload paths now do:
-new_model_cost_map = get_model_cost_map(url=model_cost_map_url)
-litellm.model_cost = new_model_cost_map
-_invalidate_model_cost_lowercase_map()
-litellm.add_known_models(model_cost_map=new_model_cost_map)  # ✅ sets repopulated
-```
-
-`add_known_models()` was also updated to accept an optional explicit map so callers cannot accidentally iterate a stale module-level reference:
-
-```python
-# Before
-def add_known_models():
-    for key, value in model_cost.items():   # reads module global — ambiguous after reload
-        ...
-
-# After
-def add_known_models(model_cost_map: Optional[Dict] = None):
-    _map = model_cost_map if model_cost_map is not None else model_cost
-    for key, value in _map.items():         # always iterates the map you just fetched
-        ...
-```
-
-After the fix, the provider sets (`anthropic_models`, `open_ai_chat_completion_models`, etc.) are always consistent with `litellm.model_cost` immediately after every reload. New models become accessible via wildcard rules without any proxy restart.
-
----
-
-## Remediation
-
-| # | Action | Status | Code |
-|---|---|---|---|
-| 1 | Call `add_known_models(model_cost_map=...)` in the periodic reload path | ✅ Done | [`proxy_server.py#L4393`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L4393) |
-| 2 | Call `add_known_models(model_cost_map=...)` in the `/reload/model_cost_map` endpoint | ✅ Done | [`proxy_server.py#L11904`](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L11904) |
-| 3 | Update `add_known_models()` to accept an explicit map parameter | ✅ Done | [`__init__.py#L617`](https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py#L617) |
-| 4 | Regression test: `add_known_models(model_cost_map=...)` populates provider sets | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
-| 5 | Regression test: `anthropic/*` wildcard grants/denies access correctly after reload | ✅ Done | [`test_auth_checks.py`](https://github.com/BerriAI/litellm/blob/main/tests/proxy_unit_tests/test_auth_checks.py) |
-
----

docs/my-website/blog/april_townhall_announcement/index.md

@@ -1,40 +0,0 @@
----
-slug: april-townhall-announcement
-title: "April Townhall: Security + Product Roadmap"
-date: 2026-04-02T07:30:00
-authors:
-  - krrish
-  - ishaan-alt
-description: "Join the LiteLLM April townhall on Friday, 10 April at 7:30 AM to learn about LiteLLM's security and product roadmap."
-tags: [announcement, townhall]
-hide_table_of_contents: true
----
-
-import Image from '@theme/IdealImage';
-
-We are hosting our April townhall on **Friday, 10 April at 7:30 AM PST**.
-
-<Image
-  img={require('../../img/april_townhall_banner.png')}
-  style={{width: '900px', height: 'auto', display: 'block'}}
-/>
-
-{/* truncate */}
-
-## Agenda
-
-- Product updates and roadmap progress
-- Reliability and security updates
-- Open Q&A with the team
-
-## How to contribute
-
-Add your thoughts to this [ticket](https://github.com/BerriAI/litellm/issues/24825) to help us shape the agenda.
-
-## Register
-
-Register here: [LiteLLM April Townhall Form](https://forms.gle/hvyVXwbFjzJQE7dEA)
-
-We will hold the townhall from **7:30 AM to 8:30 AM PST on Zoom**.
-
-For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.

docs/my-website/blog/april_townhall_updates/index.md

@@ -1,162 +0,0 @@
----
-slug: april-townhall-updates
-title: "April Townhall Updates: CI/CD v2, Stability, and Product Roadmap"
-date: 2026-04-10T12:00:00
-authors:
-  - krrish
-  - ishaan-alt
-description: "A recap of the April LiteLLM town hall covering CI/CD v2, product stability work, and the near-term roadmap."
-tags: [townhall, security, reliability, product]
-hide_table_of_contents: false
----
-
-import Image from '@theme/IdealImage';
-
-Thank you to everyone who joined our April town hall.
-
-We used the session to share our CI/CD v2 improvements, product stability work, and what we are prioritizing next across reliability and product roadmap.
-
-{/* truncate */}
-
-## CI/CD v2 improvements
-
-Our CI/CD v2 work is centered around four goals:
-
-1. **Limit** what each package can access
-2. **Reduce** the number of sensitive environment variables
-3. **Avoid** compromised packages
-4. **Reduce the risk of** release tampering
-
-#### New architecture: isolated environments
-
-We have begun moving to isolated environments for distinct CI/CD stages to reduce the chance that a single compromised step can inherit broad access across the entire pipeline.
-
-<Image
-  img={require('../../img/april_townhall_isolated_environments.png')}
-  style={{width: '900px', height: 'auto', display: 'block'}}
-/>
-
-#### Current rollout status
-
-These changes are deployed in our current release workflow. [See here](https://github.com/BerriAI/litellm/tags)
-
-#### Independently verify releases
-
-A key part of CI/CD v2 is supporting independent verification of release artifacts using our published verification process, while reducing reliance on any single credential or release path.
-
-[**Learn more about how to verify releases**](https://docs.litellm.ai/docs/proxy/docker_image_security)
-
-<Image
-  img={require('../../img/verify_releases.png')}
-  style={{width: '900px', height: 'auto', display: 'block'}}
-/>
-
-## Stability improvements
-
-### SDLC improvements
-
-This month, we're focusing on process stability improvements around:
-- Improving main-branch stability
-- Mapping UI QA to built Docker images for 1:1 environment parity
-- Consistent release tags across PyPI and Docker
-- Fixing release notes publication
-
-#### Improving main-branch stability
-
-We're introducing a staging-gated flow:
-
-<Image
-  img={require('../../img/stable_main.png')}
-  style={{width: '900px', height: 'auto', display: 'block'}}
-/>
-
-- Only an internal staging branch can push to `main`.
-- PRs to that staging branch must pass CircleCI LLM API testing.
-- Collision handling happens on staging, which is designed to reduce unstable changes reaching `main`.
-
-#### UI QA in Docker environment
-
-Moving forward, all UI QA will be performed in the built Docker image that users run.
-
-Previously, some UI QA paths were run in local environments that did not fully replicate Docker runtime conditions.
-
-That contributed to release-specific issues, including MCP registration problems in `v1.82.3`.
-
-#### Consistent release tags
-
-Today we publish releases for multiple scenarios:
-- Dev (Built of a PR for a customer-specific scenario)
-- Nightly (Passes all CI/CD checks)
-- Release Candidate (Passes all CI/CD checks + manual UI QA)
-- Stable (intended to pass all CI/CD checks + manual UI QA + 7 days of production testing)
-
-We are targeting a consistent naming convention across PyPI and Docker by the end of April.
-
-#### Release notes
-
-CI/CD v2 changes moved release notes to a manual path. This is a temporary solution while we investigate a better automated workflow. We are targeting a more consistent process by the end of April.
-
-### Product stability improvements
-
-#### Stable Prisma migrations
-
-Today, we have observed several migration failure classes:
-- Migration not applied
-- Migration marked applied but incomplete
-- Migration not applied due to non-root image issues
-
-We're prioritizing this work this month and have assigned an engineering owner to the effort. Our target is to resolve these error classes by the end of April.
-
-#### UI type safety
-
-Another area of focus is improving the stability of the UI. Today, one cause of errors is that the UI maintains its own assumptions about backend API types. This can lead to issues when backend responses differ from UI assumptions.
-
-We aim to move to having the UI and Backend be in sync with each other, and are exploring OpenAPI-driven mapping to achieve this.
-
-## Product roadmap
-
-### Our Assumptions
-
-Over the next few years, we expect:
-- Companies will give employees more AI tools.
-- More AI agents will move into production workflows across HR, finance, support, and operations.
-
-### Our Inferences
-#### Near-term
-
-- AI spend will increase.
-- Uptime and latency will become even more important.
-- More AI resources (skills, CLIs, and related assets) will require governance.
-- Agent and MCP usage patterns will require deeper controls.
-- Broader developer adoption will increase the need for simpler, more discoverable tooling.
-
-#### Long-term 
-
-- We expect many organizations to treat agent auditability (how decisions were made across LLM + MCP + sub-agent inputs/outputs) as a compliance expectation.
-- Permission management will get more complex as user-agent interaction chains deepen.
-
-Roadmap timelines in this post are targets and may evolve based on validation and user feedback.
-
-## April investments
-
-### Reliability
-
-- Increase uptime for 10k+ RPS scenarios.
-- Investigate latency overhead for long-running Claude Code requests.
-
-### Feature reliability
-
-- Polish MCP authentication.
-- Better understand how teams are using agents through LiteLLM.
-
-### Governance
-
-- Launch Skills as a first-class citizen in LiteLLM.
-
-## Q&A
-
-Thank you again for all the questions and direct feedback. We will keep sharing concrete progress updates as these efforts ship.
-
-## Hiring
-
-We are actively hiring across several roles, please apply [here](https://jobs.ashbyhq.com/litellm) if you're interested!

docs/my-website/blog/authors.yml

@@ -1,48 +0,0 @@
-litellm:
-  name: LiteLLM Team
-  title: LiteLLM Core Team
-  url: https://github.com/BerriAI/litellm
-  image_url: https://github.com/BerriAI.png
-
-sameer:
-  name: Sameer Kankute
-  title: SWE @ LiteLLM (LLM Translation)
-  url: https://www.linkedin.com/in/sameer-kankute/
-  image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
-
-krrish:
-  name: Krrish Dholakia
-  title: CEO, LiteLLM
-  url: https://www.linkedin.com/in/krish-d/
-  image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
-
-ishaan:
-  name: Ishaan Jaffer
-  title: CTO, LiteLLM
-  url: https://www.linkedin.com/in/reffajnaahsi/
-  image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
-
-# Alias for typo in name
-ishaan-alt:
-  name: Ishaan Jaffer
-  title: CTO, LiteLLM
-  url: https://www.linkedin.com/in/reffajnaahsi/
-  image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
-
-ryan:
-  name: Ryan Crabbe
-  title: Performance Engineer, LiteLLM
-  url: https://www.linkedin.com/in/ryan-crabbe-0b9687214
-  image_url: https://media.licdn.com/dms/image/v2/D5603AQHt1t9Z4BJ6Gw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1724453682340?e=1772064000&v=beta&t=VXdmr13rsNB05wyA2F1TENOB5UuDHUZ0FCHTolNyR5M
-
-alexsander:
-  name: Alexsander Hamir
-  title: Performance Engineer, LiteLLM
-  url: https://www.linkedin.com/in/alexsander-baptista/
-  image_url: https://github.com/AlexsanderHamir.png
-
-yuneng:
-  name: Yuneng Jiang
-  title: SWE @ LiteLLM (Full Stack)
-  url: https://www.linkedin.com/in/yuneng-david-jiang-455676139/
-  image_url: https://avatars.githubusercontent.com/u/171294688?v=4

docs/my-website/blog/ci_cd_v2_improvements/index.md

@@ -1,90 +0,0 @@
----
-slug: ci-cd-v2-improvements
-title: "Announcing CI/CD v2 for LiteLLM"
-date: 2026-03-30T21:30:00
-authors:
-  - krrish
-description: "CI/CD v2 introduces isolated environments, stronger security gates, and safer release separation for LiteLLM."
-tags: [engineering, ci-cd, security]
-hide_table_of_contents: false
----
-
-import Image from '@theme/IdealImage';
-
-The CI/CD v2 is now live for LiteLLM.
-
-<Image
-  img={require('../../img/ci_cd_architecture.png')}
-  style={{width: '700px', height: 'auto', display: 'block'}}
-/>
-
-<br/>
-Building on the roadmap from our [security incident](https://docs.litellm.ai/blog/security-townhall-updates#roadmap), CI/CD v2 introduces isolated environments, stronger security gates, and safer release separation for LiteLLM.
-
-## What changed
-
-- Security scans and unit tests run in isolated environments.
-- Validation and release are separated into different repositories, making it harder for an attacker to reach release credentials.
-- Trusted Publishing for PyPI releases - this means no long-lived credentials are used to publish releases.
-- Immutable Docker release tags - this means no tampering of Docker release tags after they are published [Learn more](https://docs.docker.com/docker-hub/repos/manage/hub-images/immutable-tags/). Note: work for GHCR docker releases is planned as well.
-- Docker image signing with [Cosign](https://github.com/sigstore/cosign) - all release images are signed so users can independently verify they came from us.
-
-## Verify Docker image signatures
-
-Starting from `v1.83.0-nightly`, all LiteLLM Docker images published to GHCR are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0).
-
-**Verify using the pinned commit hash (recommended):**
-
-A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:
-
-```bash
-cosign verify \
-  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
-  ghcr.io/berriai/litellm:<release-tag>
-```
-
-**Verify using a release tag (convenience):**
-
-Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:
-
-```bash
-cosign verify \
-  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
-  ghcr.io/berriai/litellm:<release-tag>
-```
-
-Replace `<release-tag>` with the version you are deploying (e.g. `v1.83.0-stable`).
-
-Expected output:
-
-```
-The following checks were performed on each of these signatures:
-  - The cosign claims were validated
-  - The signatures were verified against the specified public key
-```
-
-## What's next
-
-Moving forward, we plan on:
-- Adopting OpenSSF (this is a set of security criteria that projects should meet to demonstrate a strong security posture - [Learn more](https://baseline.openssf.org/versions/2026-02-19.html))
-  - We've added Scorecard and Allstar to our Github
-
-- Adding SLSA Build Provenance to our CI/CD pipeline - this means we allow users to independently verify that a release came from us and prevent silent modifications of releases after they are published.
-
-
-We hope that this will mean you can be confident that the releases you are using are safe and from us.
-
-
-## The principle
-
-The new CI/CD pipeline reflects the principles, outlined below, and is designed to be more secure and reliable:
-
-- **Limit** what each package can access
-- **Reduce** the number of sensitive environment variables
-- **Avoid** compromised packages
-- **Prevent** release tampering
-
-
-## How to help: 
-
-Help us plan April's stability sprint - https://github.com/BerriAI/litellm/issues/24825

docs/my-website/blog/claude_code_beta_headers/index.md

@@ -1,168 +0,0 @@
----
-slug: claude-code-beta-headers-incident
-title: "Incident Report: Invalid beta headers with Claude Code"
-date: 2026-02-16T10:00:00
-authors:
-  - sameer
-  - ishaan-alt
-  - krrish
-tags: [incident-report, anthropic, stability]
-hide_table_of_contents: false
----
-
-**Date:** February 13, 2026
-**Duration:** ~3 hours
-**Severity:** High
-**Status:** Resolved
-
-> **Note:** This fix will be available starting from `v1.81.13-nightly` or higher of LiteLLM.
-
-## Summary
-
-Claude Code began sending unsupported Anthropic beta headers to non-Anthropic providers (Bedrock, Azure AI, Vertex AI), causing `invalid beta flag` errors. LiteLLM was forwarding all beta headers without provider-specific validation. Users experienced request failures when routing Claude Code requests through LiteLLM to these providers.
-
-- **LLM calls to Anthropic:** No impact.
-- **LLM calls to Bedrock/Azure/Vertex:** Failed with `invalid beta flag` errors when unsupported headers were present.
-- **Cost tracking and routing:** No impact.
-
-{/* truncate */}
-
----
-
-## Background
-
-Anthropic uses beta headers to enable experimental features in Claude. When Claude Code makes API requests, it includes headers like `anthropic-beta: prompt-caching-scope-2026-01-05,advanced-tool-use-2025-11-20`. However, not all providers support all Anthropic beta features.
-
-Before this incident, LiteLLM forwarded all beta headers to all providers without validation:
-
-```mermaid
-sequenceDiagram
-    participant CC as Claude Code
-    participant LP as LiteLLM (old behavior)
-    participant Provider as Provider (Bedrock/Azure/Vertex)
-
-    CC->>LP: Request with beta headers
-    Note over CC,LP: anthropic-beta: header1,header2,header3
-
-    LP->>Provider: Forward ALL headers (no validation)
-    Note over LP,Provider: anthropic-beta: header1,header2,header3
-
-    Provider-->>LP: ❌ Error: invalid beta flag
-    LP-->>CC: Request fails
-```
-
-Requests succeeded for Anthropic (native support) but failed for other providers when Claude Code sent headers those providers didn't support.
-
----
-
-## Root cause
-
-LiteLLM lacked provider-specific beta header validation. When Claude Code introduced new beta features or sent headers that specific providers didn't support, those headers were blindly forwarded, causing provider API errors.
-
----
-
-## Remediation
-
-| # | Action | Status | Code |
-|---|---|---|---|
-| 1 | Create `anthropic_beta_headers_config.json` with provider-specific mappings | ✅ Done | [`anthropic_beta_headers_config.json`](https://github.com/BerriAI/litellm/blob/main/litellm/anthropic_beta_headers_config.json) |
-| 2 | Implement strict validation: headers must be explicitly mapped to be forwarded | ✅ Done | [`litellm_logging.py`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/litellm_logging.py) |
-| 3 | Add `/reload/anthropic_beta_headers` endpoint for dynamic config updates | ✅ Done | Proxy management endpoints |
-| 4 | Add `/schedule/anthropic_beta_headers_reload` for automatic periodic updates | ✅ Done | Proxy management endpoints |
-| 5 | Support `LITELLM_ANTHROPIC_BETA_HEADERS_URL` for custom config sources | ✅ Done | Environment configuration |
-| 6 | Support `LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS` for air-gapped deployments | ✅ Done | Environment configuration |
-
-Now LiteLLM validates and transforms headers per-provider:
-
-```mermaid
-sequenceDiagram
-    participant CC as Claude Code
-    participant LP as LiteLLM (new behavior)
-    participant Config as Beta Headers Config
-    participant Provider as Provider (Bedrock/Azure/Vertex)
-
-    CC->>LP: Request with beta headers
-    Note over CC,LP: anthropic-beta: header1,header2,header3
-
-    LP->>Config: Load header mapping for provider
-    Config-->>LP: Returns mapping (header→value or null)
-
-    Note over LP: Validate & Transform:<br/>1. Check if header exists in mapping<br/>2. Filter out null values<br/>3. Map to provider-specific names
-
-    LP->>Provider: Request with filtered & mapped headers
-    Note over LP,Provider: anthropic-beta: mapped-header2<br/>(header1, header3 filtered out)
-
-    Provider-->>LP: ✅ Success response
-    LP-->>CC: Response
-```
-
----
-
-## Dynamic configuration updates
-
-A key improvement is zero-downtime configuration updates. When Anthropic releases new beta features, users can update their configuration without restarting:
-
-```bash
-# Manually trigger reload (no restart needed)
-curl -X POST "https://your-proxy-url/reload/anthropic_beta_headers" \
-  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"
-
-# Or schedule automatic reloads every 24 hours
-curl -X POST "https://your-proxy-url/schedule/anthropic_beta_headers_reload?hours=24" \
-  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"
-```
-
-This prevents future incidents where Claude Code introduces new headers before LiteLLM configuration is updated.
-
----
-
-## Configuration format
-
-The `anthropic_beta_headers_config.json` file maps input headers to provider-specific output headers:
-
-```json
-{
-  "description": "Mapping of Anthropic beta headers for each provider.",
-  "anthropic": {
-    "advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
-    "computer-use-2025-01-24": "computer-use-2025-01-24"
-  },
-  "bedrock_converse": {
-    "advanced-tool-use-2025-11-20": null,
-    "computer-use-2025-01-24": "computer-use-2025-01-24"
-  },
-  "azure_ai": {
-    "advanced-tool-use-2025-11-20": "advanced-tool-use-2025-11-20",
-    "computer-use-2025-01-24": "computer-use-2025-01-24"
-  }
-}
-```
-
-**Validation rules:**
-1. Headers must exist in the mapping for the target provider
-2. Headers with `null` values are filtered out (unsupported)
-3. Header names can be transformed per-provider (e.g., Bedrock uses different names for some features)
-
----
-
-## Resolution steps for users
-
-For users still experiencing issues, update to the latest LiteLLM version if < v1.81.11-nightly:
-
-```bash
-pip install --upgrade litellm
-```
-
-Or manually reload the configuration without restarting:
-
-```bash
-curl -X POST "https://your-proxy-url/reload/anthropic_beta_headers" \
-  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"
-```
-
----
-
-## Related documentation
-
-- [Managing Anthropic Beta Headers](../../docs/proxy/sync_anthropic_beta_headers) - Complete configuration guide
-- [`anthropic_beta_headers_config.json`](https://github.com/BerriAI/litellm/blob/main/litellm/anthropic_beta_headers_config.json) - Current configuration file

docs/my-website/blog/claude_opus_4_6/index.md

@@ -1,723 +0,0 @@
----
-slug: claude_opus_4_6
-title: "Day 0 Support: Claude Opus 4.6"
-date: 2026-02-05T10:00:00
-authors:
-  - sameer
-  - ishaan-alt
-  - krrish
-description: "Day 0 support for Claude Opus 4.6 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
-tags: [anthropic, claude, opus 4.6]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports Claude Opus 4.6 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
-
-{/* truncate */}
-
-## Docker Image
-
-```bash
-docker pull ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.80.0-stable.opus-4-6
-```
-
-## Usage - Anthropic
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-6
-    litellm_params:
-      model: anthropic/claude-opus-4-6
-      api_key: os.environ/ANTHROPIC_API_KEY
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.80.0-stable.opus-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Azure
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-6
-    litellm_params:
-      model: azure_ai/claude-opus-4-6
-      api_key: os.environ/AZURE_AI_API_KEY
-      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
-  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.80.0-stable.opus-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Vertex AI
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-6
-    litellm_params:
-      model: vertex_ai/claude-opus-4-6
-      vertex_project: os.environ/VERTEX_PROJECT
-      vertex_location: us-east5
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e VERTEX_PROJECT=$VERTEX_PROJECT \
-  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  -v $(pwd)/credentials.json:/app/credentials.json \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.80.0-stable.opus-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Bedrock
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-6
-    litellm_params:
-      model: bedrock/anthropic.claude-opus-4-6-v1
-      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
-      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
-      aws_region_name: us-east-1
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.80.0-stable.opus-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Advanced Features
-
-### Compaction
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-Litellm supports enabling compaction for the new claude-opus-4-6.
-
-**Enabling Compaction**
-
-To enable compaction, add the `context_management` parameter with the `compact_20260112` edit type:
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "What is the weather in San Francisco?"
-    }
-  ],
-  "context_management": {
-    "edits": [
-      {
-        "type": "compact_20260112"
-      }
-    ]
-  },
-  "max_tokens": 100
-}'
-```
-All the parameters supported for context_management by anthropic are supported and can be directly added. Litellm automatically adds the `compact-2026-01-12` beta header in the request.
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-Enable compaction to reduce context size while preserving key information. LiteLLM automatically adds the `compact-2026-01-12` beta header when compaction is enabled.
-
-:::info
-**Provider Support:** Compaction is supported on Anthropic, Azure AI, and Vertex AI. It is **not supported** on Bedrock (Invoke or Converse APIs).
-:::
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-6",
-    "max_tokens": 4096,
-    "messages": [
-        {
-            "role": "user",
-            "content": "Hi"
-        }
-    ],
-    "context_management": {
-        "edits": [
-            {
-                "type": "compact_20260112"
-            }
-        ]
-    }
-}'
-```
-
-</TabItem>
-</Tabs>
-
-
-**Response with Compaction Block**
-
-The response will include the compaction summary in `provider_specific_fields.compaction_blocks`:
-
-```json
-{
-  "id": "chatcmpl-a6c105a3-4b25-419e-9551-c800633b6cb2",
-  "created": 1770357619,
-  "model": "claude-opus-4-6",
-  "object": "chat.completion",
-  "choices": [
-    {
-      "finish_reason": "length",
-      "index": 0,
-      "message": {
-        "content": "I don't have access to real-time data, so I can't provide the current weather in San Francisco. To get up-to-date weather information, I'd recommend checking:\n\n- **Weather websites** like weather.com, accuweather.com, or wunderground.com\n- **Search engines** – just Google \"San Francisco weather\"\n- **Weather apps** on your phone (e.g., Apple Weather, Google Weather)\n- **National",
-        "role": "assistant",
-        "provider_specific_fields": {
-          "compaction_blocks": [
-            {
-              "type": "compaction",
-              "content": "Summary of the conversation: The user requested help building a web scraper..."
-            }
-          ]
-        }
-      }
-    }
-  ],
-  "usage": {
-    "completion_tokens": 100,
-    "prompt_tokens": 86,
-    "total_tokens": 186
-  }
-}
-```
-
-**Using Compaction Blocks in Follow-up Requests**
-
-To continue the conversation with compaction, include the compaction block in the assistant message's `provider_specific_fields`:
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "How can I build a web scraper?"
-    },
-    {
-      "role": "assistant",
-      "content": [
-        {
-          "type": "text",
-          "text": "Certainly! To build a basic web scraper, you'll typically use a programming language like Python along with libraries such as `requests` (for fetching web pages) and `BeautifulSoup` (for parsing HTML). Here's a basic example:\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\n\nurl = 'https://example.com'\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.text, 'html.parser')\n\n# Extract and print all text\ntext = soup.get_text()\nprint(text)\n```\n\nLet me know what you're interested in scraping or if you need help with a specific website!"
-        }
-      ],
-      "provider_specific_fields": {
-        "compaction_blocks": [
-          {
-            "type": "compaction",
-            "content": "Summary of the conversation: The user asked how to build a web scraper, and the assistant gave an overview using Python with requests and BeautifulSoup."
-          }
-        ]
-      }
-    },
-    {
-      "role": "user",
-      "content": "How do I use it to scrape product prices?"
-    }
-  ],
-  "context_management": {
-    "edits": [
-      {
-        "type": "compact_20260112"
-      }
-    ]
-  },
-  "max_tokens": 100
-}'
-```
-
-**Streaming Support**
-
-Compaction blocks are also supported in streaming mode. You'll receive:
-- `compaction_start` event when a compaction block begins
-- `compaction_delta` events with the compaction content
-- The accumulated `compaction_blocks` in `provider_specific_fields`
-
-### Adaptive Thinking
-
-:::note
-When using `reasoning_effort` with Claude Opus 4.6, all values (`low`, `medium`, `high`) are mapped to `thinking: {type: "adaptive"}`. To use explicit thinking budgets with `type: "enabled"`, pass the native `thinking` parameter directly (see "Native thinking param" tab below).
-:::
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-LiteLLM supports adaptive thinking through the `reasoning_effort` parameter:
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Solve this complex problem: What is the optimal strategy for..."
-    }
-  ],
-  "reasoning_effort": "high"
-}'
-```
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-Use the `thinking` parameter with `type: "adaptive"` to enable adaptive thinking mode:
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-6",
-    "max_tokens": 16000,
-    "thinking": {
-        "type": "adaptive"
-    },
-    "messages": [
-        {
-            "role": "user",
-            "content": "Explain why the sum of two even numbers is always even."
-        }
-    ]
-}'
-```
-
-</TabItem>
-<TabItem value="native" label="Native thinking param">
-
-Use the `thinking` parameter directly for adaptive thinking via the SDK:
-
-```python
-import litellm
-
-response = litellm.completion(
-  model="anthropic/claude-opus-4-6",
-  messages=[{"role": "user", "content": "Solve this complex problem: What is the optimal strategy for..."}],
-  thinking={"type": "adaptive"},
-)
-```
-
-</TabItem>
-</Tabs>
-
-### Effort Levels
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-Four effort levels available: `low`, `medium`, `high` (default), and `max`. Pass directly via the `output_config` parameter:
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Explain quantum computing"
-    }
-  ],
-  "output_config": {
-        "effort": "medium"
-    }
-}'
-```
-
-You can use reasoning effort plus output_config to have more control on the model.
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-Four effort levels available: `low`, `medium`, `high` (default), and `max`. Pass directly via the `output_config` parameter:
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-6",
-    "max_tokens": 4096,
-    "messages": [
-        {
-            "role": "user",
-            "content": "Explain quantum computing"
-        }
-    ],
-    "output_config": {
-        "effort": "medium"
-    }
-}'
-```
-
-</TabItem>
-</Tabs>
-
-### 1M Token Context (Beta)
-
-Opus 4.6 supports 1M token context. Premium pricing applies for prompts exceeding 200k tokens ($10/$37.50 per million input/output tokens). LiteLLM supports cost calculations for 1M token contexts.
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-To use the 1M token context window, you need to forward the `anthropic-beta` header from your client to the LLM provider.
-
-**Step 1: Enable header forwarding in your config**
-
-```yaml
-general_settings:
-  forward_client_headers_to_llm_api: true
-```
-
-**Step 2: Send requests with the beta header**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---header 'anthropic-beta: context-1m-2025-08-07' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Analyze this large document..."
-    }
-  ]
-}'
-```
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-To use the 1M token context window, you need to forward the `anthropic-beta` header from your client to the LLM provider.
-
-**Step 1: Enable header forwarding in your config**
-
-```yaml
-general_settings:
-  forward_client_headers_to_llm_api: true
-```
-
-**Step 2: Send requests with the beta header**
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'anthropic-beta: context-1m-2025-08-07' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-6",
-    "max_tokens": 16000,
-    "messages": [
-        {
-            "role": "user",
-            "content": "Analyze this large document..."
-        }
-    ]
-}'
-```
-
-:::tip
-You can combine multiple beta headers by separating them with commas:
-```bash
---header 'anthropic-beta: context-1m-2025-08-07,compact-2026-01-12'
-```
-:::
-
-</TabItem>
-</Tabs>
-
-### US-Only Inference
-
-Available at 1.1× token pricing. LiteLLM automatically tracks costs for US-only inference.
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-Use the `inference_geo` parameter to specify US-only inference:
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "What is the capital of France?"
-    }
-  ],
-  "inference_geo": "us"
-}'
-```
-
-LiteLLM will automatically apply the 1.1× pricing multiplier for US-only inference in cost tracking.
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-Use the `inference_geo` parameter to specify US-only inference:
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-6",
-    "max_tokens": 4096,
-    "messages": [
-        {
-            "role": "user",
-            "content": "What is the capital of France?"
-        }
-    ],
-    "inference_geo": "us"
-}'
-```
-
-LiteLLM will automatically apply the 1.1× pricing multiplier for US-only inference in cost tracking.
-
-</TabItem>
-</Tabs>
-
-### Fast Mode
-
-:::info
-Fast mode is **only supported on the Anthropic provider** (`anthropic/claude-opus-4-6`). It is not available on Azure AI, Vertex AI, or Bedrock.
-:::
-
-**Pricing:**
-- Standard: $5 input / $25 output per MTok
-- Fast: $30 input / $150 output per MTok (6× premium)
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Refactor this module..."
-    }
-  ],
-  "max_tokens": 4096,
-  "speed": "fast"
-}'
-```
-
-**Using OpenAI SDK:**
-
-```python
-import openai
-
-client = openai.OpenAI(
-    api_key="your-litellm-key",
-    base_url="http://0.0.0.0:4000"
-)
-
-response = client.chat.completions.create(
-    model="claude-opus-4-6",
-    messages=[{"role": "user", "content": "Refactor this module..."}],
-    max_tokens=4096,
-    extra_body={"speed": "fast"}
-)
-```
-
-**Using LiteLLM SDK:**
-
-```python
-from litellm import completion
-
-response = completion(
-    model="anthropic/claude-opus-4-6",
-    messages=[{"role": "user", "content": "Refactor this module..."}],
-    max_tokens=4096,
-    speed="fast"
-)
-```
-
-LiteLLM automatically tracks the higher costs for fast mode in usage and cost calculations.
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-6",
-    "max_tokens": 4096,
-    "speed": "fast",
-    "messages": [
-        {
-            "role": "user",
-            "content": "Refactor this module..."
-        }
-    ]
-}'
-```
-
-LiteLLM automatically:
-- Adds the `fast-mode-2026-02-01` beta header
-- Tracks the 6× premium pricing in cost calculations
-
-</TabItem>
-</Tabs>

docs/my-website/blog/claude_opus_4_7/index.md

@@ -1,366 +0,0 @@
----
-slug: claude_opus_4_7
-title: "Day 0 Support: Claude Opus 4.7"
-date: 2026-04-16T10:00:00
-authors:
-  - sameer
-  - ishaan-alt
-  - krrish
-description: "Day 0 support for Claude Opus 4.7 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
-tags: [anthropic, claude, opus 4.7]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports [Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7) on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
-
-{/* truncate */}
-
-## Docker Image
-
-```bash
-docker pull ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.83.3-stable.opus-4.7
-```
-
-## Usage - Anthropic
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-7
-    litellm_params:
-      model: anthropic/claude-opus-4-7
-      api_key: os.environ/ANTHROPIC_API_KEY
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.83.3-stable.opus-4.7 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-7",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Azure
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-7
-    litellm_params:
-      model: azure_ai/claude-opus-4-7
-      api_key: os.environ/AZURE_AI_API_KEY
-      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
-  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.83.3-stable.opus-4.7 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-7",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Vertex AI
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-7
-    litellm_params:
-      model: vertex_ai/claude-opus-4-7
-      vertex_project: os.environ/VERTEX_PROJECT
-      vertex_location: us-east5
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e VERTEX_PROJECT=$VERTEX_PROJECT \
-  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  -v $(pwd)/credentials.json:/app/credentials.json \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.83.3-stable.opus-4.7 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-7",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Bedrock
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-opus-4-7
-    litellm_params:
-      model: bedrock/anthropic.claude-opus-4-7
-      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
-      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
-      aws_region_name: us-east-1
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.83.3-stable.opus-4.7 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-7",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-## Advanced Features
-
-### Adaptive Thinking
-
-:::note
-When using `reasoning_effort` with Claude Opus 4.7, all values (`low`, `medium`, `high`, `xhigh`) are mapped to `thinking: {type: "adaptive"}`. To use explicit thinking budgets with `type: "enabled"`, pass the native `thinking` parameter directly.
-:::
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-LiteLLM supports adaptive thinking through the `reasoning_effort` parameter:
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-7",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Solve this complex problem: What is the optimal strategy for..."
-    }
-  ],
-  "reasoning_effort": "high"
-}'
-```
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-Use the `thinking` parameter with `type: "adaptive"` to enable adaptive thinking mode:
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-7",
-    "max_tokens": 16000,
-    "thinking": {
-        "type": "adaptive"
-    },
-    "messages": [
-        {
-            "role": "user",
-            "content": "Explain why the sum of two even numbers is always even."
-        }
-    ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-### Effort Levels
-
-Claude Opus 4.7 supports four effort levels: `low`, `medium`, `high` (default), and `xhigh`. These give you finer-grained control over how much reasoning the model applies to a task. Pass the effort level via the `output_config` parameter.
-
-`xhigh` is a new effort level introduced with Opus 4.7 that sits above `high`. The `max` effort level is Claude Opus 4.6 only and is not available on 4.7.
-
-<Tabs>
-<TabItem value="completions" label="/chat/completions">
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-opus-4-7",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Explain quantum computing"
-    }
-  ],
-  "output_config": {
-    "effort": "xhigh"
-  }
-}'
-```
-
-**Using OpenAI SDK:**
-
-```python
-import openai
-
-client = openai.OpenAI(
-    api_key="your-litellm-key",
-    base_url="http://0.0.0.0:4000"
-)
-
-response = client.chat.completions.create(
-    model="claude-opus-4-7",
-    messages=[{"role": "user", "content": "Explain quantum computing"}],
-    extra_body={"output_config": {"effort": "xhigh"}}
-)
-```
-
-**Using LiteLLM SDK:**
-
-```python
-from litellm import completion
-
-response = completion(
-    model="anthropic/claude-opus-4-7",
-    messages=[{"role": "user", "content": "Explain quantum computing"}],
-    output_config={"effort": "xhigh"},
-)
-```
-
-You can combine `reasoning_effort` with `output_config` for even more fine-grained control over the model's behavior.
-
-</TabItem>
-<TabItem value="messages" label="/v1/messages">
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'x-api-key: sk-12345' \
---header 'content-type: application/json' \
---data '{
-    "model": "claude-opus-4-7",
-    "max_tokens": 4096,
-    "messages": [
-        {
-            "role": "user",
-            "content": "Explain quantum computing"
-        }
-    ],
-    "output_config": {
-        "effort": "xhigh"
-    }
-}'
-```
-
-</TabItem>
-</Tabs>
-
-**Effort level guide:**
-
-| Effort | When to use |
-|--------|-------------|
-| `low` | Short, fast responses — simple lookups, formatting, classification |
-| `medium` | Balanced tradeoff for everyday Q&A and light reasoning |
-| `high` (default) | Complex reasoning, code generation, analysis |
-| `xhigh` | Hardest problems — multi-step math, deep research, agentic planning |
-

docs/my-website/blog/claude_sonnet_4_6/index.md

@@ -1,279 +0,0 @@
----
-slug: claude_sonnet_4_6
-title: "Day 0 Support: Claude Sonnet 4.6"
-date: 2026-02-17T10:00:00
-authors:
-  - ishaan-alt
-  - krrish
-description: "Day 0 support for Claude Sonnet 4.6 on LiteLLM AI Gateway - use across Anthropic, Azure, Vertex AI, and Bedrock."
-tags: [anthropic, claude, sonnet 4.6]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports Claude Sonnet 4.6 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway.
-
-{/* truncate */}
-
-## Docker Image
-
-```bash
-docker pull ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6
-```
-
-## Usage - Anthropic
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-sonnet-4-6
-    litellm_params:
-      model: anthropic/claude-sonnet-4-6
-      api_key: os.environ/ANTHROPIC_API_KEY
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-sonnet-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-
-<TabItem value="sdk" label="LiteLLM SDK">
-
-```python
-from litellm import completion
-
-response = completion(
-    model="anthropic/claude-sonnet-4-6",
-    messages=[{"role": "user", "content": "what llm are you"}]
-)
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Azure
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-sonnet-4-6
-    litellm_params:
-      model: azure_ai/claude-sonnet-4-6
-      api_key: os.environ/AZURE_AI_API_KEY
-      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
-  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-sonnet-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-
-<TabItem value="sdk" label="LiteLLM SDK">
-
-```python
-from litellm import completion
-
-response = completion(
-    model="azure_ai/claude-sonnet-4-6",
-    api_key="your-azure-api-key",
-    api_base="https://<resource>.services.ai.azure.com",
-    messages=[{"role": "user", "content": "what llm are you"}]
-)
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Vertex AI
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-sonnet-4-6
-    litellm_params:
-      model: vertex_ai/claude-sonnet-4-6
-      vertex_project: os.environ/VERTEX_PROJECT
-      vertex_location: us-east5
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e VERTEX_PROJECT=$VERTEX_PROJECT \
-  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  -v $(pwd)/credentials.json:/app/credentials.json \
-  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-sonnet-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-
-<TabItem value="sdk" label="LiteLLM SDK">
-
-```python
-from litellm import completion
-
-response = completion(
-    model="vertex_ai/claude-sonnet-4-6",
-    vertex_project="your-project-id",
-    vertex_location="us-east5",
-    messages=[{"role": "user", "content": "what llm are you"}]
-)
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-</Tabs>
-
-## Usage - Bedrock
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: claude-sonnet-4-6
-    litellm_params:
-      model: bedrock/anthropic.claude-sonnet-4-6-v1
-      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
-      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
-      aws_region_name: us-east-1
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.3-stable.sonnet-4-6 \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "claude-sonnet-4-6",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-
-<TabItem value="sdk" label="LiteLLM SDK">
-
-```python
-from litellm import completion
-
-response = completion(
-    model="bedrock/anthropic.claude-sonnet-4-6-v1",
-    aws_access_key_id="your-access-key",
-    aws_secret_access_key="your-secret-key",
-    aws_region_name="us-east-1",
-    messages=[{"role": "user", "content": "what llm are you"}]
-)
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-</Tabs>

docs/my-website/blog/fastapi_middleware_performance/index.mdx

@@ -1,211 +0,0 @@
----
-slug: fastapi-middleware-performance
-title: "Your Middleware Could Be a Bottleneck"
-date: 2026-02-07T10:00:00
-authors:
-  - krrish
-  - ishaan-alt
-  - ryan
-description: "How we improved LiteLLM proxy latency and throughput by replacing a single middleware base class"
-tags: [performance, fastapi, middleware]
-hide_table_of_contents: false
----
-
-import { BaseHTTPMiddlewareAnimation, PureASGIAnimation, BenchmarkVisualization } from '@site/src/components/MiddlewareDiagrams';
-
-> How we improved LiteLLM proxy latency and throughput by replacing a single, simple middleware base class
-
----
-
-## Our Setup
-
-The LiteLLM proxy server has two middleware layers. The first is Starlette's `CORSMiddleware` (re-exported by FastAPI), which is a pure ASGI middleware. Then we have a simple BaseHTTPMiddleware called PrometheusAuthMiddleware.
-
-The job of `PrometheusAuthMiddleware` is to authenticate requests to the `/metrics` endpoint. It's not on by default, you enable it with a flag in your proxy config:
-
-<details>
-<summary>Proxy config flag</summary>
-
-```yaml
-litellm_settings:
-    require_auth_for_metrics_endpoint: true
-```
-
-</details>
-
-The middleware checks two things: is the request hitting `/metrics`, and is auth even enabled? If both checks fail, which they do for the vast majority of requests, it just passes the request through unchanged.
-
-<details>
-<summary>PrometheusAuthMiddleware source</summary>
-
-```python
-class PrometheusAuthMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        if self._is_prometheus_metrics_endpoint(request):
-            if self._should_run_auth_on_metrics_endpoint() is True:
-                try:
-                    await user_api_key_auth(request=request, api_key=...)
-                except Exception as e:
-                    return JSONResponse(status_code=401, content=...)
-        response = await call_next(request)
-        return response
-
-    @staticmethod
-    def _is_prometheus_metrics_endpoint(request: Request):
-        if "/metrics" in request.url.path:
-            return True
-        return False
-```
-
-</details>
-
-Looks harmless. Subclass `BaseHTTPMiddleware`, implement `dispatch()`, done. This is what you will see in Starlette's documentation<sup>[1](#footnote-1)</sup>.
-
-{/* truncate */}
-
----
-
-## What BaseHTTPMiddleware Actually Does
-
-When you write a `dispatch()` method, you'd expect the request to flow straight through your function and out the other side. What actually happens is much more involved. 
-
-On every request, even a pure passthrough (meaning nothing happens), `BaseHTTPMiddleware` creates **7 intermediate objects and tasks**:
-
-<BaseHTTPMiddlewareAnimation />
-
-It wraps the request in a new object to track body state, creates a synchronization event, allocates an in-memory channel to pass messages between your middleware and the inner app, sets up a task group to manage the lifecycle, and then runs your actual route handler in a *separate background task* when you call `call_next()`. The response body then flows back through that in-memory channel, gets re-wrapped in a streaming response object, and finally reaches the caller. That's a lot.
-
-For a middleware that for us, does nothing on 99.9% of requests, paying this cost doesn't make sense.
-
-Compare that to a pure ASGI middleware, which we can have just check the request path and continue along. 
-
-<PureASGIAnimation />
-
-Our middleware is doing something really simple. For the vast majority of requests it doesn't need to do anything at all but just let the request pass through. It doesn't need task groups, memory streams, or cancel scopes. It needs a function call.
-
----
-
-## Comparing Both
-
-We replaced the `BaseHTTPMiddleware` subclass with a pure ASGI middleware. To benchmark the difference, we used Apache Bench<sup>[2](#footnote-2)</sup> to compare both configurations of LiteLLM's middleware stack: the old setup (1 pure ASGI + 1 `BaseHTTPMiddleware`) against the new setup (2 pure ASGI).
-
-A minimal FastAPI app serves `GET /health` → `PlainTextResponse("ok")`. The endpoint does zero work to isolate the middleware overhead: any difference between configs is purely the cost of the middleware plumbing itself. Both middlewares are just calling the next layer. Same work, different base class.
-
-Apache Bench (`ab`) fires requests at the server with 1,000 concurrent connections and a single uvicorn worker. One worker means one event loop, so the benchmark directly measures how each middleware design handles concurrent load on a single thread.
-
-<BenchmarkVisualization />
-
-<details>
-<summary>Try it yourself</summary>
-
-Save the script below as `benchmark_middleware.py`, then run:
-
-```bash
-# Terminal 1 — start the "before" server (1 ASGI + 1 BaseHTTPMiddleware)
-python benchmark_middleware.py --middleware mixed
-
-# Terminal 2 — benchmark it
-ab -n 50000 -c 1000 http://localhost:8000/health
-
-# Stop the server, then start the "after" server (2x pure ASGI)
-python benchmark_middleware.py --middleware asgi
-
-# Terminal 2 — benchmark again
-ab -n 50000 -c 1000 http://localhost:8000/health
-```
-
-```python
-import argparse
-import uvicorn
-from fastapi import FastAPI
-from fastapi.responses import PlainTextResponse
-from starlette.middleware.base import BaseHTTPMiddleware
-from starlette.requests import Request
-from starlette.types import ASGIApp, Receive, Scope, Send
-
-
-class NoOpBaseHTTPMiddleware(BaseHTTPMiddleware):
-    async def dispatch(self, request: Request, call_next):
-        return await call_next(request)
-
-
-class NoOpPureASGIMiddleware:
-    def __init__(self, app: ASGIApp) -> None:
-        self.app = app
-
-    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
-        await self.app(scope, receive, send)
-
-
-def create_app(middleware_type: str | None = None, layers: int = 2) -> FastAPI:
-    app = FastAPI()
-
-    @app.get("/health")
-    async def health():
-        return PlainTextResponse("ok")
-
-    if middleware_type == "mixed":
-        app.add_middleware(NoOpBaseHTTPMiddleware)
-        app.add_middleware(NoOpPureASGIMiddleware)
-    elif middleware_type == "asgi":
-        for _ in range(layers):
-            app.add_middleware(NoOpPureASGIMiddleware)
-
-    return app
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--middleware", choices=["asgi", "mixed"], default=None)
-    parser.add_argument("--layers", type=int, default=2)
-    parser.add_argument("--port", type=int, default=8000)
-    args = parser.parse_args()
-
-    app = create_app(middleware_type=args.middleware, layers=args.layers)
-    uvicorn.run(app, host="0.0.0.0", port=args.port, workers=1, log_level="warning")
-```
-
-</details>
-
----
-
-## Our Change
-
-Here's what we replaced it with:
-
-```python
-class PrometheusAuthMiddleware:
-    def __init__(self, app: ASGIApp) -> None:
-        self.app = app
-
-    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
-        if scope["type"] != "http" or "/metrics" not in scope.get("path", ""):
-            await self.app(scope, receive, send)
-            return
-
-        if litellm.require_auth_for_metrics_endpoint is True:
-            request = Request(scope, receive)
-            api_key = request.headers.get("Authorization") or ""
-            try:
-                await user_api_key_auth(request=request, api_key=api_key)
-            except Exception as e:
-                # send 401 directly via ASGI protocol
-                ...
-                return
-
-        await self.app(scope, receive, send)
-```
-
-For the 99.9% of requests that aren't hitting `/metrics`, the middleware is now one dict lookup, one string check, and one function call. No objects allocated, no tasks spawned. 
-
-It's important to evaluate if the tools you're using are the right fit for the job as your software grows and handles more responsiblity. We're now putting in a static analysis check to prevent this from happening again with any newly introduced middlewares. If we find the use case is necessary then that's okay and we'll reevalute but for everything LiteLLM needs to do at the moment it's not. 
-
-This middleware change was one part of a broader optimization effort on the LiteLLM proxy. Across all optimizations combined, we've measured about a **30% reduction in proxy overhead** over the past two weeks. 
-
----
-
-<a id="footnote-1"></a>
-<sup>1</sup> [Starlette Middleware — BaseHTTPMiddleware](https://starlette.dev/middleware/#basehttpmiddleware)
-
-<a id="footnote-2"></a>
-<sup>2</sup> [Apache HTTP server benchmarking tool (`ab`)](https://httpd.apache.org/docs/2.4/programs/ab.html)

docs/my-website/blog/gemin_3.1/index.md

@@ -1,142 +0,0 @@
----
-slug: gemini_3_1_pro
-title: "DAY 0 Support: Gemini 3.1 Pro on LiteLLM"
-date: 2026-02-19T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Guide to using Gemini 3.1 Pro on LiteLLM Proxy and SDK with day 0 support."
-tags: [gemini, day 0 support, llms]
-hide_table_of_contents: false
----
-
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-# Gemini 3.1 Pro Day 0 Support 
-
-LiteLLM now supports `gemini-3.1-pro-preview` and all the new API changes along with it.
-
-{/* truncate */}
-
-## Deploy this version
-
-<Tabs>
-<TabItem value="docker" label="Docker">
-
-``` showLineNumbers title="docker run litellm"
-docker run \
--e STORE_MODEL_IN_DB=True \
--p 4000:4000 \
-ghcr.io/berriai/litellm:main-v1.81.9-stable.gemini.3.1-pro
-```
-
-</TabItem>
-
-<TabItem value="pip" label="Pip">
-
-``` showLineNumbers title="pip install litellm"
-pip install litellm==v1.81.9-stable.gemini.3.1-pro
-```
-
-</TabItem>
-</Tabs>
-
-## What's New
-
-### 1. New Thinking Levels: `thinkingLevel` with MINIMAL & MEDIUM
-
-Gemini 3.1 Pro introduces support for **medium** thinking level
-
-LiteLLM automatically maps the OpenAI `reasoning_effort` parameter to Gemini's `thinkingLevel`, so you can use familiar `reasoning_effort` values (`minimal`, `low`, `medium`, `high`) without changing your code!
-
----
-## Supported Endpoints
-
-LiteLLM provides **full end-to-end support** for Gemini 3.1 Pro on:
-
-- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
-- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
-- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
-- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint 
-
-All endpoints support:
-- Streaming and non-streaming responses
-- Function calling with thought signatures
-- Multi-turn conversations
-- All Gemini 3-specific features
-- Conversion of provider specific thinking related param to thinkingLevel
-
-## Quick Start
-
-<Tabs>
-<TabItem value="sdk" label="SDK">
-
-**Basic Usage with MEDIUM thinking (NEW)**
-
-```python
-from litellm import completion
-
-# No need to make any changes to your code as we map openai reasoning param to thinkingLevel
-response = completion(
-    model="gemini/gemini-3.1-pro-preview",
-    messages=[{"role": "user", "content": "Solve this complex math problem: 25 * 4 + 10"}],
-    reasoning_effort="medium",  # NEW: MEDIUM thinking level
-)
-
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-
-<TabItem value="proxy" label="PROXY">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: gemini-3.1-pro-preview
-    litellm_params:
-      model: gemini/gemini-3.1-pro-preview
-      api_key: os.environ/GEMINI_API_KEY
-  - model_name: vertex-gemini-3.1-pro-preview
-    litellm_params:
-      model: vertex_ai/gemini-3.1-pro-preview
-```
-
-**2. Start proxy**
-
-```bash
-litellm --config /path/to/config.yaml
-```
-
-**3. Call with MEDIUM thinking**
-
-```bash
-curl -X POST http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-  -d '{
-    "model": "gemini-3.1-pro-preview",
-    "messages": [{"role": "user", "content": "Complex reasoning task"}],
-    "reasoning_effort": "medium"
-  }'
-```
-
-</TabItem>
-</Tabs>
-
----
-
-## `reasoning_effort` Mapping for Gemini 3+
-
-| reasoning_effort | thinking_level | 
-|------------------|----------------|
-| `minimal` | `minimal` |
-| `low` | `low` |
-| `medium` | `medium` |
-| `high` | `high` |
-| `disable` | `minimal` |
-| `none` | `minimal` |

docs/my-website/blog/gemini_3/index.md

@@ -1,975 +0,0 @@
----
-slug: gemini_3
-title: "DAY 0 Support: Gemini 3 on LiteLLM"
-date: 2025-11-19T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Common questions and best practices for using gemini-3-pro-preview with LiteLLM Proxy and SDK."
-tags: [gemini, day 0 support, llms]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-:::info
-
-This guide covers common questions and best practices for using `gemini-3-pro-preview` with LiteLLM Proxy and SDK.
-
-:::
-
-{/* truncate */}
-
-## Quick Start
-
-<Tabs>
-<TabItem value="sdk" label="Python SDK">
-
-```python
-from litellm import completion
-import os
-
-os.environ["GEMINI_API_KEY"] = "your-api-key"
-
-response = completion(
-    model="gemini/gemini-3-pro-preview",
-    messages=[{"role": "user", "content": "Hello!"}],
-    reasoning_effort="low"
-)
-
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Add to config.yaml:**
-
-```yaml
-model_list:
-  - model_name: gemini-3-pro-preview
-    litellm_params:
-      model: gemini/gemini-3-pro-preview
-      api_key: os.environ/GEMINI_API_KEY
-```
-
-**2. Start proxy:**
-
-```bash
-litellm --config /path/to/config.yaml
-```
-
-**3. Make request:**
-
-```bash
-curl http://0.0.0.0:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-3-pro-preview",
-    "messages": [{"role": "user", "content": "Hello!"}],
-    "reasoning_effort": "low"
-  }'
-```
-
-</TabItem>
-</Tabs>
-
-## Supported Endpoints
-
-LiteLLM provides **full end-to-end support** for Gemini 3 Pro Preview on:
-
-- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
-- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
-- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
-- ✅ `/v1/generateContent` – [Google Gemini API](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#rest) compatible endpoint (for code, see: `client.models.generate_content(...)`)
-
-All endpoints support:
-- Streaming and non-streaming responses
-- Function calling with thought signatures
-- Multi-turn conversations
-- All Gemini 3-specific features
-
-## Thought Signatures
-
-#### What are Thought Signatures?
-
-Thought signatures are encrypted representations of the model's internal reasoning process. They're essential for maintaining context across multi-turn conversations, especially with function calling.
-
-#### How Thought Signatures Work
-
-1. **Automatic Extraction**: When Gemini 3 returns a function call, LiteLLM automatically extracts the `thought_signature` from the response
-2. **Storage**: Thought signatures are stored in `provider_specific_fields.thought_signature` of tool calls
-3. **Automatic Preservation**: When you include the assistant's message in conversation history, LiteLLM automatically preserves and returns thought signatures to Gemini
-
-## Example: Multi-Turn Function Calling
-
-#### Streaming with Thought Signatures
-
-When using streaming mode with `stream_chunk_builder()`, thought signatures are now automatically preserved:
-
-<Tabs>
-<TabItem value="streaming" label="Streaming SDK">
-
-```python
-import os
-import litellm
-from litellm import completion
-
-os.environ["GEMINI_API_KEY"] = "your-api-key"
-
-MODEL = "gemini/gemini-3-pro-preview"
-
-messages = [
-    {"role": "system", "content": "You are a helpful assistant. Use the calculate tool."},
-    {"role": "user", "content": "What is 2+2?"},
-]
-
-tools = [{
-    "type": "function",
-    "function": {
-        "name": "calculate",
-        "description": "Calculate a mathematical expression",
-        "parameters": {
-            "type": "object",
-            "properties": {"expression": {"type": "string"}},
-            "required": ["expression"],
-        },
-    },
-}]
-
-print("Step 1: Sending request with stream=True...")
-response = completion(
-    model=MODEL,
-    messages=messages,
-    stream=True,
-    tools=tools,
-    reasoning_effort="low"
-)
-
-# Collect all chunks
-chunks = []
-for part in response:
-    chunks.append(part)
-
-# Reconstruct message using stream_chunk_builder
-# Thought signatures are now preserved automatically!
-full_response = litellm.stream_chunk_builder(chunks, messages=messages)
-print(f"Full response: {full_response}")
-
-assistant_msg = full_response.choices[0].message
-
-# ✅ Thought signature is now preserved in provider_specific_fields
-if assistant_msg.tool_calls and assistant_msg.tool_calls[0].provider_specific_fields:
-    thought_sig = assistant_msg.tool_calls[0].provider_specific_fields.get("thought_signature")
-    print(f"Thought signature preserved: {thought_sig is not None}")
-
-# Append assistant message (includes thought signatures automatically)
-messages.append(assistant_msg)
-
-# Mock tool execution
-messages.append({
-    "role": "tool",
-    "content": "4",
-    "tool_call_id": assistant_msg.tool_calls[0].id
-})
-
-print("\nStep 2: Sending tool result back to model...")
-response_2 = completion(
-    model=MODEL,
-    messages=messages,
-    stream=True,
-    tools=tools,
-    reasoning_effort="low"
-)
-
-for part in response_2:
-    if part.choices[0].delta.content:
-        print(part.choices[0].delta.content, end="")
-print()  # New line
-```
-
-**Key Points:**
-- ✅ `stream_chunk_builder()` now preserves `provider_specific_fields` including thought signatures
-- ✅ Thought signatures are automatically included when appending `assistant_msg` to conversation history
-- ✅ Multi-turn conversations work seamlessly with streaming
-
-</TabItem>
-<TabItem value="sdk" label="Non-Streaming SDK">
-
-```python
-from openai import OpenAI
-import json
-
-client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000")
-
-# Define tools
-tools = [
-    {
-        "type": "function",
-        "function": {
-            "name": "get_weather",
-            "description": "Get the current weather",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "location": {"type": "string"}
-                },
-                "required": ["location"]
-            }
-        }
-    }
-]
-
-# Step 1: Initial request
-messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]
-
-response = client.chat.completions.create(
-    model="gemini-3-pro-preview",
-    messages=messages,
-    tools=tools,
-    reasoning_effort="low"
-)
-
-# Step 2: Append assistant message (thought signatures automatically preserved)
-messages.append(response.choices[0].message)
-
-# Step 3: Execute tool and append result
-for tool_call in response.choices[0].message.tool_calls:
-    if tool_call.function.name == "get_weather":
-        result = {"temperature": 30, "unit": "celsius"}
-        messages.append({
-            "role": "tool",
-            "content": json.dumps(result),
-            "tool_call_id": tool_call.id
-        })
-
-# Step 4: Follow-up request (thought signatures automatically included)
-response2 = client.chat.completions.create(
-    model="gemini-3-pro-preview",
-    messages=messages,
-    tools=tools,
-    reasoning_effort="low"
-)
-
-print(response2.choices[0].message.content)
-```
-
-**Key Points:**
-- ✅ Thought signatures are automatically extracted from `response.choices[0].message.tool_calls[].provider_specific_fields.thought_signature`
-- ✅ When you append `response.choices[0].message` to your conversation history, thought signatures are automatically preserved
-- ✅ You don't need to manually extract or manage thought signatures
-
-</TabItem>
-<TabItem value="proxy" label="cURL">
-
-```bash
-# Step 1: Initial request
-curl http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-3-pro-preview",
-    "messages": [
-      {"role": "user", "content": "What'\''s the weather in Tokyo?"}
-    ],
-    "tools": [
-      {
-        "type": "function",
-        "function": {
-          "name": "get_weather",
-          "description": "Get the current weather",
-          "parameters": {
-            "type": "object",
-            "properties": {
-              "location": {"type": "string"}
-            },
-            "required": ["location"]
-          }
-        }
-      }
-    ],
-    "reasoning_effort": "low"
-  }'
-```
-
-**Response includes thought signature:**
-
-```json
-{
-  "choices": [{
-    "message": {
-      "role": "assistant",
-      "tool_calls": [{
-        "id": "call_abc123",
-        "type": "function",
-        "function": {
-          "name": "get_weather",
-          "arguments": "{\"location\": \"Tokyo\"}"
-        },
-        "provider_specific_fields": {
-          "thought_signature": "CpcHAdHtim9+q4rstcbvQC0ic4x1/vqQlCJWgE+UZ6dTLYGHMMBkF/AxqL5UmP6SY46uYC8t4BTFiXG5zkw6EMJ..."
-        }
-      }]
-    }
-  }]
-}
-```
-
-```bash
-# Step 2: Follow-up request (include assistant message with thought signature)
-curl http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-3-pro-preview",
-    "messages": [
-      {"role": "user", "content": "What'\''s the weather in Tokyo?"},
-      {
-        "role": "assistant",
-        "content": null,
-        "tool_calls": [{
-          "id": "call_abc123",
-          "type": "function",
-          "function": {
-            "name": "get_weather",
-            "arguments": "{\"location\": \"Tokyo\"}"
-          },
-          "provider_specific_fields": {
-            "thought_signature": "CpcHAdHtim9+q4rstcbvQC0ic4x1/vqQlCJWgE+UZ6dTLYGHMMBkF/AxqL5UmP6SY46uYC8t4BTFiXG5zkw6EMJ..."
-          }
-        }]
-      },
-      {
-        "role": "tool",
-        "content": "{\"temperature\": 30, \"unit\": \"celsius\"}",
-        "tool_call_id": "call_abc123"
-      }
-    ],
-    "tools": [...],
-    "reasoning_effort": "low"
-  }'
-```
-
-</TabItem>
-</Tabs>
-
-#### Important Notes on Thought Signatures
-
-1. **Automatic Handling**: LiteLLM automatically extracts and preserves thought signatures. You don't need to manually manage them.
-
-2. **Parallel Function Calls**: When the model makes parallel function calls, only the **first function call** has a thought signature.
-
-3. **Sequential Function Calls**: In multi-step function calling, each step's first function call has its own thought signature that must be preserved.
-
-4. **Required for Context**: Thought signatures are essential for maintaining reasoning context. Without them, the model may lose context of its previous reasoning.
-
-## Conversation History: Switching from Non-Gemini-3 Models
-
-#### Common Question: Will switching from a non-Gemini-3 model to Gemini-3 break conversation history?
-
-**Answer: No!** LiteLLM automatically handles this by adding dummy thought signatures when needed.
-
-#### How It Works
-
-When you switch from a model that doesn't use thought signatures (e.g., `gemini-2.5-flash`) to Gemini 3, LiteLLM:
-
-1. **Detects missing signatures**: Identifies assistant messages with tool calls that lack thought signatures
-2. **Adds dummy signature**: Automatically injects a dummy thought signature (`skip_thought_signature_validator`) for compatibility
-3. **Maintains conversation flow**: Your conversation history continues to work seamlessly
-
-#### Example: Switching Models Mid-Conversation
-
-<Tabs>
-<TabItem value="sdk" label="Python SDK">
-
-```python
-from openai import OpenAI
-
-client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000")
-
-# Step 1: Start with gemini-2.5-flash (no thought signatures)
-messages = [{"role": "user", "content": "What's the weather?"}]
-
-response1 = client.chat.completions.create(
-    model="gemini-2.5-flash",
-    messages=messages,
-    tools=[...],
-    reasoning_effort="low"
-)
-
-# Append assistant message (no tool call thought signature from gemini-2.5-flash)
-messages.append(response1.choices[0].message)
-
-# Step 2: Switch to gemini-3-pro-preview
-# LiteLLM automatically adds dummy thought signature to the previous assistant message
-response2 = client.chat.completions.create(
-    model="gemini-3-pro-preview",  # 👈 Switched model
-    messages=messages,  # 👈 Same conversation history
-    tools=[...],
-    reasoning_effort="low"
-)
-
-# ✅ Works seamlessly! No errors, no breaking changes
-print(response2.choices[0].message.content)
-```
-
-</TabItem>
-<TabItem value="proxy" label="cURL">
-
-```bash
-# Step 1: Start with gemini-2.5-flash
-curl http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-2.5-flash",
-    "messages": [{"role": "user", "content": "What'\''s the weather?"}],
-    "tools": [...],
-    "reasoning_effort": "low"
-  }'
-
-# Step 2: Switch to gemini-3-pro-preview with same conversation history
-# LiteLLM automatically handles the missing thought signature
-curl http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-3-pro-preview",  # 👈 Switched model
-    "messages": [
-      {"role": "user", "content": "What'\''s the weather?"},
-      {
-        "role": "assistant",
-        "tool_calls": [...]  # 👈 No thought_signature from gemini-2.5-flash
-      }
-    ],
-    "tools": [...],
-    "reasoning_effort": "low"
-  }'
-# ✅ Works! LiteLLM adds dummy signature automatically
-```
-
-</TabItem>
-</Tabs>
-
-#### Dummy Signature Details
-
-The dummy signature used is: `base64("skip_thought_signature_validator")`
-
-This is the recommended approach by Google for handling conversation history from models that don't support thought signatures. It allows Gemini 3 to:
-- Accept the conversation history without validation errors
-- Continue the conversation seamlessly
-- Maintain context across model switches
-
-## Thinking Level Parameter
-
-#### How `reasoning_effort` Maps to `thinking_level`
-
-For Gemini 3 Pro Preview, LiteLLM automatically maps `reasoning_effort` to the new `thinking_level` parameter:
-
-| `reasoning_effort` | `thinking_level` | Notes |
-|-------------------|------------------|-------|
-| `"minimal"` | `"low"` | Maps to low thinking level |
-| `"low"` | `"low"` | Default for most use cases |
-| `"medium"` | `"high"` | Medium not available yet, maps to high |
-| `"high"` | `"high"` | Maximum reasoning depth |
-| `"disable"` | `"low"` | Gemini 3 cannot fully disable thinking |
-| `"none"` | `"low"` | Gemini 3 cannot fully disable thinking |
-
-#### Default Behavior
-
-If you don't specify `reasoning_effort`, LiteLLM automatically sets `thinking_level="low"` for Gemini 3 models, to avoid high costs. 
-
-### Example Usage
-
-<Tabs>
-<TabItem value="sdk" label="Python SDK">
-
-```python
-from litellm import completion
-
-# Low thinking level (faster, lower cost)
-response = completion(
-    model="gemini/gemini-3-pro-preview",
-    messages=[{"role": "user", "content": "What's the weather?"}],
-    reasoning_effort="low"  # Maps to thinking_level="low"
-)
-
-# High thinking level (deeper reasoning, higher cost)
-response = completion(
-    model="gemini/gemini-3-pro-preview",
-    messages=[{"role": "user", "content": "Solve this complex math problem step by step."}],
-    reasoning_effort="high"  # Maps to thinking_level="high"
-)
-```
-
-</TabItem>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-```bash
-# Low thinking level
-curl http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-3-pro-preview",
-    "messages": [{"role": "user", "content": "What'\''s the weather?"}],
-    "reasoning_effort": "low"
-  }'
-
-# High thinking level
-curl http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer sk-1234" \
-  -d '{
-    "model": "gemini-3-pro-preview",
-    "messages": [{"role": "user", "content": "Solve this complex problem."}],
-    "reasoning_effort": "high"
-  }'
-```
-
-</TabItem>
-</Tabs>
-
-## Important Notes
-
-1. **Gemini 3 Cannot Disable Thinking**: Unlike Gemini 2.5 models, Gemini 3 cannot fully disable thinking. Even when you set `reasoning_effort="none"` or `"disable"`, it maps to `thinking_level="low"`.
-
-2. **Temperature Recommendation**: For Gemini 3 models, LiteLLM defaults `temperature` to `1.0` and strongly recommends keeping it at this default. Setting `temperature < 1.0` can cause:
-   - Infinite loops
-   - Degraded reasoning performance
-   - Failure on complex tasks
-
-3. **Automatic Defaults**: If you don't specify `reasoning_effort`, LiteLLM automatically sets `thinking_level="low"` for optimal performance.
-
-## Cost Tracking: Prompt Caching & Context Window
-
-LiteLLM provides comprehensive cost tracking for Gemini 3 Pro Preview, including support for prompt caching and tiered pricing based on context window size.
-
-### Prompt Caching Cost Tracking
-
-Gemini 3 supports prompt caching, which allows you to cache frequently used prompt prefixes to reduce costs. LiteLLM automatically tracks and calculates costs for:
-
-- **Cache Hit Tokens**: Tokens that are read from cache (charged at a lower rate)
-- **Cache Creation Tokens**: Tokens that are written to cache (one-time cost)
-- **Text Tokens**: Regular prompt tokens that are processed normally
-
-#### How It Works
-
-LiteLLM extracts caching information from the `prompt_tokens_details` field in the usage object:
-
-```python
-{
-  "usage": {
-    "prompt_tokens": 50000,
-    "completion_tokens": 1000,
-    "total_tokens": 51000,
-    "prompt_tokens_details": {
-      "cached_tokens": 30000,  # Cache hit tokens
-      "cache_creation_tokens": 5000,  # Tokens written to cache
-      "text_tokens": 15000  # Regular processed tokens
-    }
-  }
-}
-```
-
-### Context Window Tiered Pricing
-
-Gemini 3 Pro Preview supports up to 1M tokens of context, with tiered pricing that automatically applies when your prompt exceeds 200k tokens.
-
-#### Automatic Tier Detection
-
-LiteLLM automatically detects when your prompt exceeds the 200k token threshold and applies the appropriate tiered pricing:
-
-```python
-from litellm import completion_cost
-
-# Example: Small prompt (< 200k tokens)
-response_small = completion(
-    model="gemini/gemini-3-pro-preview",
-    messages=[{"role": "user", "content": "Hello!"}]
-)
-# Uses base pricing: $0.000002/input token, $0.000012/output token
-
-# Example: Large prompt (> 200k tokens)
-response_large = completion(
-    model="gemini/gemini-3-pro-preview",
-    messages=[{"role": "user", "content": "..." * 250000}]  # 250k tokens
-)
-# Automatically uses tiered pricing: $0.000004/input token, $0.000018/output token
-```
-
-#### Cost Breakdown
-
-The cost calculation includes:
-
-1. **Text Processing Cost**: Regular tokens processed at base or tiered rate
-2. **Cache Read Cost**: Cached tokens read at discounted rate
-3. **Cache Creation Cost**: One-time cost for writing tokens to cache (applies tiered rate if above 200k)
-4. **Output Cost**: Generated tokens at base or tiered rate
-
-### Example: Viewing Cost Breakdown
-
-You can view the detailed cost breakdown using LiteLLM's cost tracking:
-
-```python
-from litellm import completion, completion_cost
-
-response = completion(
-    model="gemini/gemini-3-pro-preview",
-    messages=[{"role": "user", "content": "Explain prompt caching"}],
-    caching=True  # Enable prompt caching
-)
-
-# Get total cost
-total_cost = completion_cost(completion_response=response)
-print(f"Total cost: ${total_cost:.6f}")
-
-# Access usage details
-usage = response.usage
-print(f"Prompt tokens: {usage.prompt_tokens}")
-print(f"Completion tokens: {usage.completion_tokens}")
-
-# Access caching details
-if usage.prompt_tokens_details:
-    print(f"Cache hit tokens: {usage.prompt_tokens_details.cached_tokens}")
-    print(f"Cache creation tokens: {usage.prompt_tokens_details.cache_creation_tokens}")
-    print(f"Text tokens: {usage.prompt_tokens_details.text_tokens}")
-```
-
-### Cost Optimization Tips
-
-1. **Use Prompt Caching**: For repeated prompt prefixes, enable caching to reduce costs by up to 90% for cached portions
-2. **Monitor Context Size**: Be aware that prompts above 200k tokens use tiered pricing (2x for input, 1.5x for output)
-3. **Cache Management**: Cache creation tokens are charged once when writing to cache, then subsequent reads are much cheaper
-4. **Track Usage**: Use LiteLLM's built-in cost tracking to monitor spending across different token types
-
-### Integration with LiteLLM Proxy
-
-When using LiteLLM Proxy, all cost tracking is automatically logged and available through:
-
-- **Usage Logs**: Detailed token and cost breakdowns in proxy logs
-- **Budget Management**: Set budgets and alerts based on actual usage
-- **Analytics Dashboard**: View cost trends and breakdowns by token type
-
-```yaml
-# config.yaml
-model_list:
-  - model_name: gemini-3-pro-preview
-    litellm_params:
-      model: gemini/gemini-3-pro-preview
-      api_key: os.environ/GEMINI_API_KEY
-
-litellm_settings:
-  # Enable detailed cost tracking
-  success_callback: ["langfuse"]  # or your preferred logging service
-```
-
-## Using with Claude Code CLI
-
-You can use `gemini-3-pro-preview` with **Claude Code CLI** - Anthropic's command-line interface. This allows you to use Gemini 3 Pro Preview with Claude Code's native syntax and workflows.
-
-### Setup
-
-**1. Add Gemini 3 Pro Preview to your `config.yaml`:**
-
-```yaml
-model_list:
-  - model_name: gemini-3-pro-preview
-    litellm_params:
-      model: gemini/gemini-3-pro-preview
-      api_key: os.environ/GEMINI_API_KEY
-
-litellm_settings:
-  master_key: os.environ/LITELLM_MASTER_KEY
-```
-
-**2. Set environment variables:**
-
-```bash
-export GEMINI_API_KEY="your-gemini-api-key"
-export LITELLM_MASTER_KEY="sk-1234567890"  # Generate a secure key
-```
-
-**3. Start LiteLLM Proxy:**
-
-```bash
-litellm --config /path/to/config.yaml
-
-# RUNNING on http://0.0.0.0:4000
-```
-
-**4. Configure Claude Code to use LiteLLM Proxy:**
-
-```bash
-export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
-export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
-```
-
-**5. Use Gemini 3 Pro Preview with Claude Code:**
-
-```bash
-# Claude Code will use gemini-3-pro-preview from your LiteLLM proxy
-claude --model gemini-3-pro-preview
-
-```
-
-### Example Usage
-
-Once configured, you can interact with Gemini 3 Pro Preview using Claude Code's native interface:
-
-```bash
-$ claude --model gemini-3-pro-preview
-> Explain how thought signatures work in multi-turn conversations.
-
-# Gemini 3 Pro Preview responds through Claude Code interface
-```
-
-### Benefits
-
-- ✅ **Native Claude Code Experience**: Use Gemini 3 Pro Preview with Claude Code's familiar CLI interface
-- ✅ **Unified Authentication**: Single API key for all models through LiteLLM proxy
-- ✅ **Cost Tracking**: All usage tracked through LiteLLM's centralized logging
-- ✅ **Seamless Model Switching**: Easily switch between Claude and Gemini models
-- ✅ **Full Feature Support**: All Gemini 3 features (thought signatures, function calling, etc.) work through Claude Code
-
-### Troubleshooting
-
-**Claude Code not finding the model:**
-- Ensure the model name in Claude Code matches exactly: `gemini-3-pro-preview`
-- Verify your proxy is running: `curl http://0.0.0.0:4000/health`
-- Check that `ANTHROPIC_BASE_URL` points to your LiteLLM proxy
-
-**Authentication errors:**
-- Verify `ANTHROPIC_AUTH_TOKEN` matches your LiteLLM master key
-- Ensure `GEMINI_API_KEY` is set correctly
-- Check LiteLLM proxy logs for detailed error messages
-
-## Responses API Support
-
-LiteLLM fully supports the OpenAI Responses API for Gemini 3 Pro Preview, including both streaming and non-streaming modes. The Responses API provides a structured way to handle multi-turn conversations with function calling, and LiteLLM automatically preserves thought signatures throughout the conversation.
-
-### Example: Using Responses API with Gemini 3
-
-<Tabs>
-<TabItem value="sdk" label="Non-Streaming">
-
-```python
-from openai import OpenAI
-import json
-
-client = OpenAI()
-
-# 1. Define a list of callable tools for the model
-tools = [
-    {
-        "type": "function",
-        "name": "get_horoscope",
-        "description": "Get today's horoscope for an astrological sign.",
-        "parameters": {
-            "type": "object",
-            "properties": {
-                "sign": {
-                    "type": "string",
-                    "description": "An astrological sign like Taurus or Aquarius",
-                },
-            },
-            "required": ["sign"],
-        },
-    },
-]
-
-def get_horoscope(sign):
-    return f"{sign}: Next Tuesday you will befriend a baby otter."
-
-# Create a running input list we will add to over time
-input_list = [
-    {"role": "user", "content": "What is my horoscope? I am an Aquarius."}
-]
-
-# 2. Prompt the model with tools defined
-response = client.responses.create(
-    model="gemini-3-pro-preview",
-    tools=tools,
-    input=input_list,
-)
-
-# Save function call outputs for subsequent requests
-input_list += response.output
-
-for item in response.output:
-    if item.type == "function_call":
-        if item.name == "get_horoscope":
-            # 3. Execute the function logic for get_horoscope
-            horoscope = get_horoscope(json.loads(item.arguments))
-            
-            # 4. Provide function call results to the model
-            input_list.append({
-                "type": "function_call_output",
-                "call_id": item.call_id,
-                "output": json.dumps({
-                  "horoscope": horoscope
-                })
-            })
-
-print("Final input:")
-print(input_list)
-
-response = client.responses.create(
-    model="gemini-3-pro-preview",
-    instructions="Respond only with a horoscope generated by a tool.",
-    tools=tools,
-    input=input_list,
-)
-
-# 5. The model should be able to give a response!
-print("Final output:")
-print(response.model_dump_json(indent=2))
-print("\n" + response.output_text)
-```
-
-**Key Points:**
-- ✅ Thought signatures are automatically preserved in function calls
-- ✅ Works seamlessly with multi-turn conversations
-- ✅ All Gemini 3-specific features are fully supported
-
-</TabItem>
-<TabItem value="streaming" label="Streaming">
-
-```python
-from openai import OpenAI
-import json
-
-client = OpenAI()
-
-tools = [
-    {
-        "type": "function",
-        "name": "get_horoscope",
-        "description": "Get today's horoscope for an astrological sign.",
-        "parameters": {
-            "type": "object",
-            "properties": {
-                "sign": {
-                    "type": "string",
-                    "description": "An astrological sign like Taurus or Aquarius",
-                },
-            },
-            "required": ["sign"],
-        },
-    },
-]
-
-def get_horoscope(sign):
-    return f"{sign}: Next Tuesday you will befriend a baby otter."
-
-input_list = [
-    {"role": "user", "content": "What is my horoscope? I am an Aquarius."}
-]
-
-# Streaming mode
-response = client.responses.create(
-    model="gemini-3-pro-preview",
-    tools=tools,
-    input=input_list,
-    stream=True,
-)
-
-# Collect all chunks
-chunks = []
-for chunk in response:
-    chunks.append(chunk)
-    # Process streaming chunks as they arrive
-    print(chunk)
-
-# Thought signatures are automatically preserved in streaming mode
-```
-
-**Key Points:**
-- ✅ Streaming mode fully supported
-- ✅ Thought signatures preserved across streaming chunks
-- ✅ Real-time processing of function calls and responses
-
-</TabItem>
-</Tabs>
-
-### Responses API Benefits
-
-- ✅ **Structured Output**: Responses API provides a clear structure for handling function calls and multi-turn conversations
-- ✅ **Thought Signature Preservation**: LiteLLM automatically preserves thought signatures in both streaming and non-streaming modes
-- ✅ **Seamless Integration**: Works with existing OpenAI SDK patterns
-- ✅ **Full Feature Support**: All Gemini 3 features (thought signatures, function calling, reasoning) are fully supported
-
-
-## Best Practices
-
-#### 1. Always Include Thought Signatures in Conversation History
-
-When building multi-turn conversations with function calling:
-
-✅ **Do:**
-```python
-# Append the full assistant message (includes thought signatures)
-messages.append(response.choices[0].message)
-```
-
-❌ **Don't:**
-```python
-# Don't manually construct assistant messages without thought signatures
-messages.append({
-    "role": "assistant",
-    "tool_calls": [...]  # Missing thought signatures!
-})
-```
-
-#### 2. Use Appropriate Thinking Levels
-
-- **`reasoning_effort="low"`**: For simple queries, quick responses, cost optimization
-- **`reasoning_effort="high"`**: For complex problems requiring deep reasoning
-
-#### 3. Keep Temperature at Default
-
-For Gemini 3 models, always use `temperature=1.0` (default). Lower temperatures can cause issues.
-
-#### 4. Handle Model Switches Gracefully
-
-When switching from non-Gemini-3 to Gemini-3:
-- ✅ LiteLLM automatically handles missing thought signatures
-- ✅ No manual intervention needed
-- ✅ Conversation history continues seamlessly
-
-
-## Troubleshooting
-
-#### Issue: Missing Thought Signatures
-
-**Symptom**: Error when including assistant messages in conversation history
-
-**Solution**: Ensure you're appending the full assistant message from the response:
-```python
-messages.append(response.choices[0].message)  # ✅ Includes thought signatures
-```
-
-#### Issue: Conversation Breaks When Switching Models
-
-**Symptom**: Errors when switching from gemini-2.5-flash to gemini-3-pro-preview
-
-**Solution**: This should work automatically! LiteLLM adds dummy signatures. If you see errors, ensure you're using the latest LiteLLM version.
-
-#### Issue: Infinite Loops or Poor Performance
-
-**Symptom**: Model gets stuck or produces poor results
-
-**Solution**: 
-- Ensure `temperature=1.0` (default for Gemini 3)
-- Check that `reasoning_effort` is set appropriately
-- Verify you're using the correct model name: `gemini/gemini-3-pro-preview`
-
-## Additional Resources
-
-- [Gemini Provider Documentation](../../docs/providers/gemini)
-- [Thought Signatures Guide](../../docs/providers/gemini#thought-signatures)
-- [Reasoning Content Documentation](../../docs/reasoning_content)
-- [Function Calling Guide](../../docs/completion/function_call)

docs/my-website/blog/gemini_3_1_flash_lite/index.md

@@ -1,168 +0,0 @@
----
-slug: gemini_3_1_flash_lite_preview
-title: "DAY 0 Support: Gemini 3.1 Flash Lite Preview on LiteLLM"
-date: 2026-03-03T08:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Guide to using Gemini 3.1 Flash Lite Preview on LiteLLM Proxy and SDK with day 0 support."
-tags: [gemini, day 0 support, llms, supernova]
-hide_table_of_contents: false
----
-
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-# Gemini 3.1 Flash Lite Preview Day 0 Support 
-
-LiteLLM now supports `gemini-3.1-flash-lite-preview` with full day 0 support!
-
-:::note
-If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
-:::
-
-{/* truncate */}
-
-## Deploy this version
-
-<Tabs>
-<TabItem value="docker" label="Docker">
-
-``` showLineNumbers title="docker run litellm"
-docker run \
--e STORE_MODEL_IN_DB=True \
--p 4000:4000 \
-ghcr.io/berriai/litellm:main-v1.80.8-stable.1
-```
-
-</TabItem>
-
-<TabItem value="pip" label="Pip">
-
-``` showLineNumbers title="pip install litellm"
-pip install litellm==v1.80.8-stable.1
-```
-
-</TabItem>
-</Tabs>
-
-## What's New
-
-Supports all four thinking levels:
-- **MINIMAL**: Ultra-fast responses with minimal reasoning
-- **LOW**: Simple instruction following
-- **MEDIUM**: Balanced reasoning for complex tasks
-- **HIGH**: Maximum reasoning depth (dynamic)
-
----
-
-## Quick Start
-
-<Tabs>
-<TabItem value="sdk" label="SDK">
-
-**Basic Usage**
-
-```python
-from litellm import completion
-
-response = completion(
-    model="gemini/gemini-3.1-flash-lite-preview",
-    messages=[{"role": "user", "content": "Extract key entities from this text: ..."}],
-)
-
-print(response.choices[0].message.content)
-```
-
-**With Thinking Levels**
-
-```python
-from litellm import completion
-
-# Use MEDIUM thinking for complex reasoning tasks
-response = completion(
-    model="gemini/gemini-3.1-flash-lite-preview",
-    messages=[{"role": "user", "content": "Analyze this dataset and identify patterns"}],
-    reasoning_effort="medium",  # low, medium , high
-)
-
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-
-<TabItem value="proxy" label="PROXY">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: gemini-3.1-flash-lite
-    litellm_params:
-      model: gemini/gemini-3.1-flash-lite-preview
-      api_key: os.environ/GEMINI_API_KEY
-  
-  # Or use Vertex AI
-  - model_name: vertex-gemini-3.1-flash-lite
-    litellm_params:
-      model: vertex_ai/gemini-3.1-flash-lite-preview
-      vertex_project: your-project-id
-      vertex_location: us-central1
-```
-
-**2. Start proxy**
-
-```bash
-litellm --config /path/to/config.yaml
-```
-
-**3. Make requests**
-
-```bash
-curl -X POST http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-  -d '{
-    "model": "gemini-3.1-flash-lite",
-    "messages": [{"role": "user", "content": "Extract structured data from this text"}],
-    "reasoning_effort": "low"
-  }'
-```
-
-</TabItem>
-</Tabs>
-
----
-
-## Supported Endpoints
-
-LiteLLM provides **full end-to-end support** for Gemini 3.1 Flash Lite Preview on:
-
-- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
-- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
-- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
-- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint 
-
-All endpoints support:
-- Streaming and non-streaming responses
-- Function calling with thought signatures
-- Multi-turn conversations
-- All Gemini 3-specific features (thinking levels, thought signatures)
-- Full multimodal support (text, image, audio, video)
-
----
-
-## `reasoning_effort` Mapping for Gemini 3.1
-
-LiteLLM automatically maps OpenAI's `reasoning_effort` parameter to Gemini's `thinkingLevel`:
-
-| reasoning_effort | thinking_level | Use Case |
-|------------------|----------------|----------|
-| `minimal` | `minimal` | Ultra-fast responses, simple queries |
-| `low` | `low` | Basic instruction following |
-| `medium` | `medium` | Balanced reasoning for moderate complexity |
-| `high` | `high` | Maximum reasoning depth, complex problems |
-| `disable` | `minimal` | Disable extended reasoning |
-| `none` | `minimal` | No extended reasoning |

docs/my-website/blog/gemini_3_flash/index.md

@@ -1,247 +0,0 @@
----
-slug: gemini_3_flash
-title: "DAY 0 Support: Gemini 3 Flash on LiteLLM"
-date: 2025-12-17T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Guide to using Gemini 3 Flash on LiteLLM Proxy and SDK with day 0 support."
-tags: [gemini, day 0 support, llms]
-hide_table_of_contents: false
----
-
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-# Gemini 3 Flash Day 0 Support 
-
-LiteLLM now supports `gemini-3-flash-preview` and all the new API changes along with it.
-
-:::note
-If you only want cost tracking, you need no change in your current Litellm version. But if you want the support for new features introduced along with it like thinking levels, you will need to use v1.80.8-stable.1 or above.
-:::
-
-{/* truncate */}
-
-## Deploy this version
-
-<Tabs>
-<TabItem value="docker" label="Docker">
-
-``` showLineNumbers title="docker run litellm"
-docker run \
--e STORE_MODEL_IN_DB=True \
--p 4000:4000 \
-ghcr.io/berriai/litellm:main-v1.80.8-stable.1
-```
-
-</TabItem>
-
-<TabItem value="pip" label="Pip">
-
-``` showLineNumbers title="pip install litellm"
-pip install litellm==1.80.8.post1
-```
-
-</TabItem>
-</Tabs>
-
-## What's New
-
-### 1. New Thinking Levels: `thinkingLevel` with MINIMAL & MEDIUM
-
-Gemini 3 Flash introduces granular thinking control with `thinkingLevel` instead of `thinkingBudget`.
-- **MINIMAL**: Ultra-lightweight thinking for fast responses
-- **MEDIUM**: Balanced thinking for complex reasoning  
-- **HIGH**: Maximum reasoning depth
-
-LiteLLM automatically maps the OpenAI `reasoning_effort` parameter to Gemini's `thinkingLevel`, so you can use familiar `reasoning_effort` values (`minimal`, `low`, `medium`, `high`) without changing your code!
-
-### 2. Thought Signatures
-
-Like `gemini-3-pro`, this model also includes thought signatures for tool calls. LiteLLM handles signature extraction and embedding internally. [Learn more about thought signatures](../gemini_3/index.md#thought-signatures).
-
-**Edge Case Handling**: If thought signatures are missing in the request, LiteLLM adds a dummy signature ensuring the API call doesn't break
-
----
-## Supported Endpoints
-
-LiteLLM provides **full end-to-end support** for Gemini 3 Flash on:
-
-- ✅ `/v1/chat/completions` - OpenAI-compatible chat completions endpoint
-- ✅ `/v1/responses` - OpenAI Responses API endpoint (streaming and non-streaming)
-- ✅ [`/v1/messages`](../../docs/anthropic_unified) - Anthropic-compatible messages endpoint
-- ✅ `/v1/generateContent` – [Google Gemini API](../../docs/generateContent) compatible endpoint 
-All endpoints support:
-- Streaming and non-streaming responses
-- Function calling with thought signatures
-- Multi-turn conversations
-- All Gemini 3-specific features
-- Converstion of provider specific thinking related param to thinkingLevel
-
-## Quick Start
-
-<Tabs>
-<TabItem value="sdk" label="SDK">
-
-**Basic Usage with MEDIUM thinking (NEW)**
-
-```python
-from litellm import completion
-
-# No need to make any changes to your code as we map openai reasoning param to thinkingLevel
-response = completion(
-    model="gemini/gemini-3-flash-preview",
-    messages=[{"role": "user", "content": "Solve this complex math problem: 25 * 4 + 10"}],
-    reasoning_effort="medium",  # NEW: MEDIUM thinking level
-)
-
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-
-<TabItem value="proxy" label="PROXY">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: gemini-3-flash
-    litellm_params:
-      model: gemini/gemini-3-flash-preview
-      api_key: os.environ/GEMINI_API_KEY
-```
-
-**2. Start proxy**
-
-```bash
-litellm --config /path/to/config.yaml
-```
-
-**3. Call with MEDIUM thinking**
-
-```bash
-curl -X POST http://localhost:4000/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-  -d '{
-    "model": "gemini-3-flash",
-    "messages": [{"role": "user", "content": "Complex reasoning task"}],
-    "reasoning_effort": "medium"
-  }'
-``'
-
-</TabItem>
-</Tabs>
-
----
-
-## All `reasoning_effort` Levels
-
-<Tabs>
-<TabItem value="minimal" label="MINIMAL">
-
-**Ultra-fast, minimal reasoning**
-
-```python
-from litellm import completion
-
-response = completion(
-    model="gemini/gemini-3-flash-preview",
-    messages=[{"role": "user", "content": "What's 2+2?"}],
-    reasoning_effort="minimal",
-)
-```
-
-</TabItem>
-
-<TabItem value="low" label="LOW">
-
-**Simple instruction following**
-
-```python
-response = completion(
-    model="gemini/gemini-3-flash-preview",
-    messages=[{"role": "user", "content": "Write a haiku about coding"}],
-    reasoning_effort="low",
-)
-```
-
-</TabItem>
-
-<TabItem value="medium" label="MEDIUM (NEW)">
-
-**Balanced reasoning for complex tasks** ✨
-
-```python
-response = completion(
-    model="gemini/gemini-3-flash-preview",
-    messages=[{"role": "user", "content": "Analyze this dataset and find patterns"}],
-    reasoning_effort="medium",  # NEW!
-)
-```
-
-</TabItem>
-
-<TabItem value="high" label="HIGH">
-
-**Maximum reasoning depth**
-
-```python
-response = completion(
-    model="gemini/gemini-3-flash-preview",
-    messages=[{"role": "user", "content": "Prove this mathematical theorem"}],
-    reasoning_effort="high",
-)
-```
-
-</TabItem>
-</Tabs>
-
----
-
-## Key Features
-
-✅ **Thinking Levels**: MINIMAL, LOW, MEDIUM, HIGH  
-✅ **Thought Signatures**: Track reasoning with unique identifiers  
-✅ **Seamless Integration**: Works with existing OpenAI-compatible client  
-✅ **Backward Compatible**: Gemini 2.5 models continue using `thinkingBudget`  
-
----
-
-## Installation
-
-```bash
-pip install litellm --upgrade
-```
-
-```python
-import litellm
-from litellm import completion
-
-response = completion(
-    model="gemini/gemini-3-flash-preview",
-    messages=[{"role": "user", "content": "Your question here"}],
-    reasoning_effort="medium",  # Use MEDIUM thinking
-)
-print(response)
-```
-
-:::note
-If using this model via vertex_ai, keep the location as global as this is the only supported location as of now.
-:::
-
-
-## `reasoning_effort` Mapping for Gemini 3+
-
-| reasoning_effort | thinking_level | 
-|------------------|----------------|
-| `minimal` | `minimal` |
-| `low` | `low` |
-| `medium` | `medium` |
-| `high` | `high` |
-| `disable` | `minimal` |
-| `none` | `minimal` |

docs/my-website/blog/gemini_embedding_2_multimodal/index.md

@@ -1,168 +0,0 @@
----
-slug: gemini_embedding_2_multimodal
-title: "Gemini Embedding 2 Preview: Multimodal Embeddings on LiteLLM"
-date: 2025-03-11T10:00:00
-authors:
-  - sameer
-description: "Generate embeddings from text, images, audio, video, and PDFs with gemini-embedding-2-preview on LiteLLM via Gemini API and Vertex AI."
-tags: [gemini, embeddings, multimodal, vertex ai]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-# Gemini Embedding 2 Preview: Multimodal Embeddings
-
-LiteLLM now supports **multimodal embeddings** with `gemini-embedding-2-preview`—generating a single embedding from a mix of text, images, audio, video, and PDF content. Available via both the **Gemini API** (API key) and **Vertex AI** (GCP credentials).
-
-{/* truncate */}
-
-## Supported Input Types
-
-| Modality | Supported Formats | 
-|----------|-------------------|
-| **Text** | Plain text |
-| **Image** | PNG, JPEG | 
-| **Audio** | MP3, WAV | 
-| **Video** | MP4, MOV | 
-| **Documents** | PDF | 
-
-## Input Formats
-
-LiteLLM accepts three input formats for multimodal content:
-
-1. **Data URIs** – Base64-encoded inline: `data:image/png;base64,<encoded_data>`
-2. **GCS URLs** – Cloud Storage paths (Vertex AI): `gs://bucket/path/to/file.png`
-3. **Gemini File References** – Pre-uploaded files (Gemini API): `files/abc123`
-
-## Quick Start
-
-<Tabs>
-<TabItem value="gemini" label="Gemini API">
-
-```python
-from litellm import embedding
-import os
-
-os.environ["GEMINI_API_KEY"] = "your-api-key"
-
-# Text + Image (base64)
-response = embedding(
-    model="gemini/gemini-embedding-2-preview",
-    input=[
-        "The food was delicious and the waiter...",
-        "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"
-    ],
-)
-print(response)
-```
-
-</TabItem>
-
-<TabItem value="vertex" label="Vertex AI">
-
-```python
-import litellm
-from litellm import embedding
-
-litellm.vertex_project = "your-project-id"
-litellm.vertex_location = "us-central1"
-
-# Text + Image (GCS URL)
-response = embedding(
-    model="vertex_ai/gemini-embedding-2-preview",
-    input=[
-        "Describe this image",
-        "gs://my-bucket/images/photo.png"
-    ],
-)
-print(response)
-```
-
-</TabItem>
-
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Config (config.yaml)**
-
-```yaml
-model_list:
-  - model_name: gemini-embedding-2-preview
-    litellm_params:
-      model: gemini/gemini-embedding-2-preview
-      api_key: os.environ/GEMINI_API_KEY
-  - model_name: vertex-gemini-embedding-2-preview
-    litellm_params:
-      model: vertex_ai/gemini-embedding-2-preview
-      vertex_project: os.environ/VERTEXAI_PROJECT
-      vertex_location: os.environ/VERTEXAI_LOCATION
-
-general_settings:
-  master_key: sk-1234
-```
-
-**2. Start proxy**
-
-```bash
-litellm --config config.yaml
-```
-
-**3. Call embeddings**
-
-```bash
-curl -X POST http://localhost:4000/embeddings \
-  -H "Authorization: Bearer sk-1234" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "model": "gemini-embedding-2-preview",
-    "input": [
-      "The food was delicious and the waiter...",
-      "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"
-    ]
-  }'
-```
-
-</TabItem>
-</Tabs>
-
-## Input Format Examples
-
-| Format | Example | Provider |
-|--------|---------|----------|
-| **Data URI** | `data:image/png;base64,...` | Gemini, Vertex AI |
-| **GCS URL** | `gs://bucket/path/image.png` | Vertex AI |
-| **File reference** | `files/abc123` | Gemini API only |
-
-### Supported MIME Types for Data URIs
-
-- **Images:** `image/png`, `image/jpeg`
-- **Audio:** `audio/mpeg`, `audio/wav`
-- **Video:** `video/mp4`, `video/quicktime`
-- **Documents:** `application/pdf`
-
-### GCS URL MIME Inference
-
-For Vertex AI, MIME types are inferred from file extensions:
-
-- `.png` → `image/png`
-- `.jpg` / `.jpeg` → `image/jpeg`
-- `.mp3` → `audio/mpeg`
-- `.wav` → `audio/wav`
-- `.mp4` → `video/mp4`
-- `.mov` → `video/quicktime`
-- `.pdf` → `application/pdf`
-
-## Optional Parameters
-
-| Parameter | Description | Maps to |
-|-----------|-------------|---------|
-| `dimensions` | Output embedding size | `outputDimensionality` |
-
-```python
-response = embedding(
-    model="gemini/gemini-embedding-2-preview",
-    input=["text to embed"],
-    dimensions=768,  # Optional: control output vector size
-)
-```

docs/my-website/blog/gpt_5_3_codex/index.md

@@ -1,138 +0,0 @@
----
-slug: gpt_5_3_codex
-title: "Day 0 Support: GPT-5.3-Codex"
-date: 2026-02-24T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Day 0 support for GPT-5.3-Codex on LiteLLM, including phase parameter handling for Responses API."
-tags: [openai, gpt-5.3-codex, codex, day 0 support]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports GPT-5.3-Codex on Day 0, including support for the new assistant `phase` metadata on Responses API output items.
-
-{/* truncate */}
-
-## Why `phase` matters for GPT-5.3-Codex
-
-`phase` appears on assistant output items and helps distinguish preamble/commentary turns from final closeout responses.
-
-Reference: [Phase parameter docs](https://developers.openai.com/api/reference/overview)
-
-Supported values:
-- `null`
-- `"commentary"`
-- `"final_answer"`
-
-Important:
-- Persist assistant output items with `phase` exactly as returned.
-- Send those assistant items back on the next turn.
-- Do **not** add `phase` to user messages.
-
-## Docker Image
-
-```bash
-docker pull ghcr.io/berriai/litellm:v1.81.12-stable.gpt-5.3
-```
-
-## Usage 
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: gpt-5.3-codex
-    litellm_params:
-      model: openai/gpt-5.3-codex
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e ANTHROPIC_API_KEY=$OPENAI_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.12-stable.gpt-5.3 \
-  --config /app/config.yaml
-```
-
-
-**3. Test it**
-
-```bash
-curl -X POST "http://0.0.0.0:4000/v1/responses" \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer $LITELLM_KEY" \
-  -d '{
-    "model": "gpt-5.3-codex",
-    "input": "Write a Python script that checks if a number is prime."
-  }'
-```
-
-</TabItem>
-</Tabs>
-
-## Python Example: Persist `phase` with OpenAI Client + LiteLLM Base URL
-
-```python
-from openai import OpenAI
-
-client = OpenAI(
-    base_url="http://0.0.0.0:4000/v1",  # LiteLLM Proxy
-    api_key="your-litellm-api-key",
-)
-
-items = []  # Persist this per conversation/thread
-
-
-def _item_get(item, key, default=None):
-    if isinstance(item, dict):
-        return item.get(key, default)
-    return getattr(item, key, default)
-
-
-def run_turn(user_text: str):
-    global items
-
-    # User message: no phase field
-    items.append(
-        {
-            "type": "message",
-            "role": "user",
-            "content": [{"type": "input_text", "text": user_text}],
-        }
-    )
-
-    resp = client.responses.create(
-        model="gpt-5.3-codex",
-        input=items,
-    )
-
-    # Persist assistant output items verbatim, including phase
-    for out_item in (resp.output or []):
-        items.append(out_item)
-
-    # Optional: inspect latest phase for UI/telemetry routing
-    latest_phase = None
-    for out_item in reversed(resp.output or []):
-        if _item_get(out_item, "type") == "output_item.done" and _item_get(out_item, "phase") is not None:
-            latest_phase = _item_get(out_item, "phase")
-            break
-
-    return resp, latest_phase
-```
-
-## Notes
-
-- Use `/v1/responses` for GPT Codex models.
-- Preserve full assistant output history for best multi-turn behavior.
-- If `phase` metadata is dropped during history reconstruction, output quality can degrade on long-running tasks.

docs/my-website/blog/gpt_5_4/index.md

@@ -1,90 +0,0 @@
----
-slug: gpt_5_4
-title: "Day 0 Support: GPT-5.4"
-date: 2026-03-05T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "GPT-5.4 model support in LiteLLM"
-tags: [openai, gpt-5.4, completion]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports fully GPT-5.4!
-
-{/* truncate */}
-
-## Docker Image
-
-```bash
-docker pull ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch
-```
-
-## Usage
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: gpt-5.4
-    litellm_params:
-      model: openai/gpt-5.4
-      api_key: os.environ/OPENAI_API_KEY
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e OPENAI_API_KEY=$OPENAI_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch \
-  --config /app/config.yaml
-```
-
-**3. Test it**
-
-```bash
-curl -X POST "http://0.0.0.0:4000/chat/completions" \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer $LITELLM_KEY" \
-  -d '{
-    "model": "gpt-5.4",
-    "messages": [
-      {"role": "user", "content": "Write a Python function to check if a number is prime."}
-    ]
-  }'
-```
-
-</TabItem>
-<TabItem value="sdk" label="LiteLLM SDK">
-
-```python
-from litellm import completion
-
-response = completion(
-    model="openai/gpt-5.4",
-    messages=[
-        {"role": "user", "content": "Write a Python function to check if a number is prime."}
-    ],
-)
-
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-</Tabs>
-
-## Notes
-
-- Restart your container to get the cost tracking for this model.
-- Use `/responses` for better model performance.
-- GPT-5.4 supports reasoning, function calling, vision, and tool-use — see the [OpenAI provider docs](../../docs/providers/openai) for advanced usage.

docs/my-website/blog/gpt_5_4_mini_nano/index.md

@@ -1,106 +0,0 @@
----
-slug: gpt_5_4_mini_nano
-title: "Day 0 Support: GPT-5.4-mini and GPT-5.4-nano"
-date: 2026-03-17T10:00:00
-authors:
-  - name: Sameer Kankute
-    title: SWE @ LiteLLM (LLM Translation)
-    url: https://www.linkedin.com/in/sameer-kankute/
-    image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
-  - name: Krrish Dholakia
-    title: "CEO, LiteLLM"
-    url: https://www.linkedin.com/in/krish-d/
-    image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
-  - name: Ishaan Jaff
-    title: "CTO, LiteLLM"
-    url: https://www.linkedin.com/in/reffajnaahsi/
-    image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
-description: "GPT-5.4-mini and GPT-5.4-nano model support in LiteLLM"
-tags: [openai, gpt-5.4-mini, gpt-5.4-nano, completion]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports GPT-5.4-mini and GPT-5.4-nano — cost-effective models for simple completions and high-throughput workloads.
-
-:::note
-If you're on **v1.82.3-stable** or above, you don't need any update to use these models.
-:::
-
-## Usage
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: gpt-5.4-mini
-    litellm_params:
-      model: openai/gpt-5.4-mini
-      api_key: os.environ/OPENAI_API_KEY
-  - model_name: gpt-5.4-nano
-    litellm_params:
-      model: openai/gpt-5.4-nano
-      api_key: os.environ/OPENAI_API_KEY
-```
-
-**2. Start the proxy**
-
-```bash
-litellm --config /path/to/config.yaml
-```
-
-**3. Test it**
-
-```bash
-# GPT-5.4-mini
-curl -X POST "http://localhost:4000/v1/chat/completions" \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer $LITELLM_KEY" \
-  -d '{
-    "model": "gpt-5.4-mini",
-    "messages": [{"role": "user", "content": "What is the capital of France?"}]
-  }'
-
-# GPT-5.4-nano
-curl -X POST "http://localhost:4000/v1/chat/completions" \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer $LITELLM_KEY" \
-  -d '{
-    "model": "gpt-5.4-nano",
-    "messages": [{"role": "user", "content": "What is 2 + 2?"}]
-  }'
-```
-
-</TabItem>
-<TabItem value="sdk" label="LiteLLM SDK">
-
-```python
-from litellm import completion
-
-# GPT-5.4-mini
-response = completion(
-    model="openai/gpt-5.4-mini",
-    messages=[{"role": "user", "content": "What is the capital of France?"}],
-)
-print(response.choices[0].message.content)
-
-# GPT-5.4-nano
-response = completion(
-    model="openai/gpt-5.4-nano",
-    messages=[{"role": "user", "content": "What is 2 + 2?"}],
-)
-print(response.choices[0].message.content)
-```
-
-</TabItem>
-</Tabs>
-
-## Notes
-
-- Both models support function calling, vision, and tool-use — see the [OpenAI provider docs](../../docs/providers/openai) for advanced usage.
-- GPT-5.4-nano is the most cost-effective option for simple tasks; GPT-5.4-mini offers a balance of speed and capability.

docs/my-website/blog/guardrail_logging_secret_exposure_incident/index.md

@@ -1,78 +0,0 @@
----
-slug: guardrail-logging-secret-exposure-incident
-title: "Incident Report: Guardrail logging exposed secret headers in spend logs and traces"
-date: 2026-03-18T10:00:00
-authors:
-  - litellm
-tags: [incident-report, security, guardrails]
-hide_table_of_contents: false
----
-
-**Date:** March 18, 2026
-**Duration:** Unknown
-**Severity:** High
-**Status:** Resolved
-
-## Summary
-
-When a custom guardrail returned the full LiteLLM request/data dictionary, the guardrail response logged by LiteLLM could include `secret_fields.raw_headers`, including plaintext `Authorization` headers containing API keys or other credentials.
-
-This information could then propagate to logging and observability surfaces that consume guardrail metadata, including:
-
-- **Spend logs in the LiteLLM UI:** visible to admins with access to spend-log data
-- **OpenTelemetry traces:** visible to anyone with access to the relevant telemetry backend
-
-LLM calls, proxy routing, and provider execution were not blocked by this bug. The impact was exposure of sensitive request headers in observability and logging paths.
-
-{/* truncate */}
-
----
-
-## Background
-
-LiteLLM keeps internal request data (including request headers) for use during the call. That data is not meant to be written to logs or telemetry.
-
-When custom guardrails run, their outcomes are logged so they can appear in spend logs, OpenTelemetry traces, and other observability backends. If a guardrail returned the full request payload instead of a minimal result, that internal request data could be included in what was logged. Before the fix, the guardrail logging path did not strip that data before sending it to those systems.
-
-```mermaid
-flowchart TD
-    inboundRequest["1. Incoming proxy request"] --> storeSecrets["2. Store internal request data"]
-    storeSecrets --> guardrailRuns["3. Custom guardrail runs"]
-    guardrailRuns --> fullDataReturn["4. Guardrail returns full request payload"]
-    fullDataReturn --> loggingBuild["5. Build guardrail log payload"]
-    loggingBuild --> spendLogs["6a. Persist to spend logs / UI"]
-    loggingBuild --> otelTraces["6b. Attach to OTEL guardrail spans"]
-```
-
----
-
-## Root Cause
-
-The root cause was incomplete sanitization in the guardrail logging path. When building the payload that gets sent to spend logs and traces, LiteLLM prepared guardrail responses for logging but did not strip internal request data (such as headers) from them. If a guardrail returned a response that included that data, it was passed through to the logging and observability systems unchanged.
-
----
-
-## Impact
-
-This issue required all of the following:
-
-1. A custom guardrail returned the full LiteLLM request/data dictionary, or another response object containing `secret_fields`.
-2. LiteLLM logged that guardrail response through the standard guardrail logging path.
-3. An operator, admin, or telemetry consumer had access to the resulting logs or traces.
-
-When those conditions were met, sensitive values could become visible through:
-
-- **Spend logs / UI responses:** guardrail metadata could be included in spend-log payloads rendered in the admin UI.
-- **OpenTelemetry traces:** `guardrail_response` could be written as a span attribute on guardrail spans.
-- **Other downstream observability backends:** any integration consuming the same guardrail metadata could receive the leaked values.
-
-This was a logging and telemetry exposure bug. It did not let callers bypass auth, access other tenants directly, or change model behavior, but it could expose plaintext credentials to people with access to those observability systems.
-
----
-
-## Guidance For Users
-
-- Upgrade to LiteLLM 1.82.3+.
-- If you operated custom guardrails that return the full request/data dict, review whether spend logs or telemetry traces were retained during the affected period.
-- Rotate any credentials that may have appeared in `Authorization` or other forwarded request headers in those systems.
-- Apply least-privilege access controls to spend-log views and telemetry backends that may contain request-derived metadata.

docs/my-website/blog/httpx_cache_eviction_incident/index.md

@@ -1,126 +0,0 @@
----
-slug: httpx-cache-eviction-incident
-title: "Incident Report: Cache Eviction Closes In-Use httpx Clients"
-date: 2026-02-27T10:00:00
-authors:
-  - ryan
-  - ishaan-alt
-  - krrish
-tags: [incident-report, caching, stability]
-hide_table_of_contents: false
----
-
-**Date:** February 27, 2026
-**Duration:** ~6 days (Feb 21 merge -> Feb 27 fix)
-**Severity:** High
-**Status:** Resolved
-
-> **Note:** This fix is available starting from LiteLLM `v1.81.14.rc.2` or higher.
-
-## Summary
-
-A change to improve Redis connection pool cleanup introduced a regression that closed **httpx clients** that were still actively being used by the proxy. The `LLMClientCache` (an in-memory TTL cache) stores both Redis clients *and* httpx clients under the same eviction policy. When a cache entry expired or was evicted, the new cleanup code called `aclose()`/`close()` on the evicted value which worked correctly for Redis clients, but destroyed httpx clients that other parts of the system still held references to and were actively using for LLM API calls.
-
-**Impact:** Any proxy instance that hit the cache TTL (default 10 minutes) or capacity limit (200 entries) would have its httpx clients closed out from under it, causing requests to LLM providers to fail with connection errors.
-
-{/* truncate */}
-
----
-
-## Background
-
-`LLMClientCache` extends `InMemoryCache` and is used to cache SDK clients (OpenAI, Anthropic, etc.) to avoid re-creating them on every request. These clients are keyed by configuration + event loop ID. The cache has:
-
-- **Max size:** 200 entries
-- **Default TTL:** 10 minutes
-
-When the cache is full or entries expire, `InMemoryCache.evict_cache()` calls `_remove_key()` to drop entries.
-
-The cached values are a mix of:
-- **Redis/async Redis clients** — owned exclusively by the cache, safe to close on eviction
-- **httpx-backed SDK clients** (OpenAI, Anthropic, etc.) — shared references, still in use by router/model instances
-
----
-
-## Root Cause
-
-[PR #21717](https://github.com/BerriAI/litellm/pull/21717) overrode `_remove_key()` in `LLMClientCache` to close async clients on eviction:
-
-<details>
-<summary>Problematic code added in PR #21717</summary>
-
-```python
-class LLMClientCache(InMemoryCache):
-    def _remove_key(self, key: str) -> None:
-        value = self.cache_dict.get(key)
-        super()._remove_key(key)
-        if value is not None:
-            close_fn = getattr(value, "aclose", None) or getattr(value, "close", None)
-            if close_fn and asyncio.iscoroutinefunction(close_fn):
-                try:
-                    asyncio.get_running_loop().create_task(close_fn())
-                except RuntimeError:
-                    pass
-            elif close_fn and callable(close_fn):
-                try:
-                    close_fn()
-                except Exception:
-                    pass
-```
-
-</details>
-
-The intent was correct for Redis clients — prevent connection pool leaks when cached Redis clients expire. But `LLMClientCache` also stores httpx-backed SDK clients (e.g., `AsyncOpenAI`, `AsyncAnthropic`). These clients:
-
-1. Have an `aclose()` method (inherited from httpx)
-2. Are still held by references elsewhere in the codebase (router, model instances)
-3. Were being closed without any check on whether they were still in use
-
-So when the cache evicted an entry, it would call `aclose()` on an httpx client that was still being used for active LLM requests → closed transport → connection errors.
-
----
-
-## The Fix
-
-[PR #22247](https://github.com/BerriAI/litellm/pull/22247) removed the `_remove_key` override entirely:
-
-<details>
-<summary>The fix (PR #22247)</summary>
-
-```diff
- class LLMClientCache(InMemoryCache):
--    def _remove_key(self, key: str) -> None:
--        """Close async clients before evicting them to prevent connection pool leaks."""
--        value = self.cache_dict.get(key)
--        super()._remove_key(key)
--        if value is not None:
--            close_fn = getattr(value, "aclose", None) or getattr(
--                value, "close", None
--            )
--            ...
--
-     def update_cache_key_with_event_loop(self, key):
-```
-
-</details>
-
-The eviction now simply drops the reference and lets Python's GC handle cleanup, which is safe because:
-- httpx clients that are still referenced elsewhere stay alive
-- Unreferenced clients get cleaned up by GC naturally
-
-The other improvements from PR #21717 were kept:
-- **`max_connections` respected for URL-based Redis configs**, previously silently dropped
-- **`disconnect()` now closes both sync and async Redis clients**, sync client was previously leaked
-- **Connection pool passthrough**, when a pool is provided with a URL config, it's used directly instead of creating a duplicate
-
----
-
-## Remediation
-
-| Action | Status | Code |
-|--------|--------|------|
-| Remove `_remove_key` override that closes shared clients on eviction | ✅ Done | [PR #22247](https://github.com/BerriAI/litellm/pull/22247) |
-| Add e2e test: evicted client still usable (capacity) | ✅ Done | [PR #22313](https://github.com/BerriAI/litellm/pull/22313) |
-| Add e2e test: expired client still usable (TTL) | ✅ Done | [PR #22313](https://github.com/BerriAI/litellm/pull/22313) |
-
-The e2e tests go through `get_async_httpx_client()` the same code path the proxy uses in production and assert the client is still functional after eviction. These run in CI on every PR against `main`. If anyone modifies `LLMClientCache` eviction behavior, overrides `_remove_key`, or adds any form of client cleanup on eviction, these tests will fail regardless of the implementation approach.

docs/my-website/blog/litellm_observatory/index.md

@@ -1,128 +0,0 @@
----
-slug: litellm-observatory
-title: "Improve release stability with 24 hour load tests"
-date: 2026-02-06T10:00:00
-authors:
-  - alexsander
-  - krrish
-  - ishaan-alt
-description: "How we built a long-running, release-validation system to catch regressions before they reach users."
-tags: [testing, observability, reliability, releases]
-hide_table_of_contents: false
----
-
-![LiteLLM Observatory](https://raw.githubusercontent.com/AlexsanderHamir/assets/main/Screenshot%202026-01-31%20175355.png)
-
-# Improve release stability with 24 hour load tests
-
-As LiteLLM adoption has grown, so have expectations around reliability, performance, and operational safety. Meeting those expectations requires more than correctness-focused tests, it requires validating how the system behaves over time, under real-world conditions.
-
-This post introduces **LiteLLM Observatory**, a long-running release-validation system we built to catch regressions before they reach users.
-
-{/* truncate */}
-
----
-
-## Why We Built the Observatory
-
-LiteLLM operates at the intersection of external providers, long-lived network connections, and high-throughput workloads. While our unit and integration tests do an excellent job validating correctness, they are not designed to surface issues that only appear after extended operation.
-
-A subtle lifecycle edge case discovered in v1.81.3 reinforced the need for stronger release validation in this area.
-
----
-
-## A Real-World Lifecycle Edge Case
-
-In v1.81.3, we shipped a fix for an HTTP client memory leak. The change passed unit and integration tests and behaved correctly in short-lived runs.
-
-The issue that surfaced was not caused by a single incorrect line of logic, but by how multiple components interacted over time:
-
-- A cached `httpx` client was configured with a 1-hour TTL
-- When the cache expired, the underlying HTTP connection was closed as expected
-- A higher-level client continued to hold a reference to that connection
-- Subsequent requests failed with:
-
-```
-Cannot send a request, as the client has been closed
-```
-
-**Before (with bug):**
-
-| Provider | Requests | Success | Failures | Fail % |
-|----------|----------|---------|----------|--------|
-| OpenAI   | 720,000  | 432,000 | 288,000  | 40%    |
-| Azure    | 692,000  | 415,200 | 276,800  | 40%    |
-
-**After (fixed):**
-
-| Provider | Requests   | Success   | Failures | Fail %  |
-|----------|------------|-----------|----------|---------|
-| OpenAI   | 1,200,000  | 1,199,988 | 12       | 0.001%  |
-| Azure    | 1,150,000  | 1,149,982 | 18       | 0.002%  |
-
-Our focus moving forward is on being the first to detect issues, even when they aren’t covered by unit tests. LiteLLM Observatory is designed to surface latency regressions, OOMs, and failure modes that only appear under real traffic patterns in **our own production deployments** during release validation.
-
-
----
-
-### How the Observatory Works
-
-[LiteLLM Observatory](https://github.com/BerriAI/litellm-observatory) is a testing service that runs long-running tests against our LiteLLM deployments. We trigger tests by sending API requests, and results are automatically sent to Slack when tests complete.
-
-#### How Tests Run
-
-1. **Start a Test**: We send a request to the Observatory API with:
-   - Which LiteLLM deployment to test (URL and API key)
-   - Which test to run (e.g., `TestOAIAzureRelease`)
-   - Test settings (which models to test, how long to run, failure thresholds)
-
-2. **Smart Queueing**:
-   - The system checks whether we are attempting to run the exact same test more than once
-   - If a duplicate test is already running or queued, we receive an error to avoid wasting resources
-   - Otherwise, the test is added to a queue and runs when capacity is available (up to 5 tests can run concurrently by default)
-
-3. **Instant Response**: The API responds immediately—we do not wait for the test to finish. Tests may run for hours, but the request itself completes in milliseconds.
-
-4. **Background Execution**:
-   - The test runs in the background, issuing requests against our LiteLLM deployment
-   - It tracks request success and failure rates over time
-   - When the test completes, results are automatically posted to our Slack channel
-
-#### Example: The OpenAI / Azure Reliability Test
-
-The `TestOAIAzureRelease` test is designed to catch a class of bugs that only surface after sustained runtime:
-
-- **Duration**: Runs continuously for 3 hours
-- **Behavior**: Cycles through specified models (such as `gpt-4` and `gpt-3.5-turbo`), issuing requests continuously
-- **Why 3 Hours**: This helps catch issues where HTTP clients degrade or fail after extended use (for example, a bug observed in LiteLLM v1.81.3)
-- **Pass / Fail Criteria**: The test passes if fewer than 1% of requests fail. If the failure rate exceeds 1%, the test fails and we are notified in Slack
-- **Key Detail**: The same HTTP client is reused for the entire run, allowing us to detect lifecycle-related bugs that only appear under prolonged reuse
-
-#### When We Use It
-
-- **Before Deployments**: Run tests before promoting a new LiteLLM version to production
-- **Routine Validation**: Schedule regular runs (daily or weekly) to catch regressions early
-- **Issue Investigation**: Run tests on demand when we suspect a deployment issue
-- **Long-Running Failure Detection**: Identify bugs that only appear under sustained load, beyond what short smoke tests can reveal
-
-
-### Complementing Unit Tests
-
-Unit tests remain a foundational part of our development process. They are fast and precise, but they don’t cover:
-
-- Real provider behavior
-- Long-lived network interactions
-- Resource lifecycle edge cases
-- Time-dependent regressions
-
-LiteLLM Observatory complements unit tests by validating the system as it actually runs in production-like environments.
-
----
-
-### Looking Ahead
-
-Reliability is an ongoing investment.
-
-LiteLLM Observatory is one of several systems we’re building to continuously raise the bar on release quality and operational safety. As LiteLLM evolves, so will our validation tooling, informed by real-world usage and lessons learned.
-
-We’ll continue to share those improvements openly as we go.

docs/my-website/blog/minimax_m2_5/index.md

@@ -1,387 +0,0 @@
----
-slug: minimax_m2_5
-title: "Day 0 Support: MiniMax-M2.5"
-date: 2026-02-12T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Day 0 support for MiniMax-M2.5 on LiteLLM"
-tags: [minimax, M2.5, llm]
-hide_table_of_contents: false
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-LiteLLM now supports MiniMax-M2.5 on Day 0. Use it across OpenAI-compatible and Anthropic-compatible APIs through the LiteLLM AI Gateway.
-
-{/* truncate */}
-
-## Supported Models
-
-LiteLLM supports the following MiniMax models:
-
-| Model | Description | Input Cost | Output Cost | Context Window |
-|-------|-------------|------------|-------------|----------------|
-| **MiniMax-M2.5** | Advanced reasoning, Agentic capabilities | $0.3/M tokens | $1.2/M tokens | 1M tokens |
-| **MiniMax-M2.5-lightning** | Faster and More Agile (~100 tps) | $0.3/M tokens | $2.4/M tokens | 1M tokens |
-
-## Features Supported
-
-- **Prompt Caching**: Reduce costs with cached prompts ($0.03/M tokens for cache read, $0.375/M tokens for cache write)
-- **Function Calling**: Built-in tool calling support
-- **Reasoning**: Advanced reasoning capabilities with thinking support
-- **System Messages**: Full system message support
-- **Cost Tracking**: Automatic cost calculation for all requests
-
-## Docker Image
-
-```bash
-docker pull litellm/litellm:v1.81.3-stable
-```
-
-## Usage - OpenAI Compatible API (/v1/chat/completions)
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: minimax-m2-5
-    litellm_params:
-      model: minimax/MiniMax-M2.5
-      api_key: os.environ/MINIMAX_API_KEY
-      api_base: https://api.minimax.io/v1
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e MINIMAX_API_KEY=$MINIMAX_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.3-stable \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "minimax-m2-5",
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-### With Reasoning Split
-
-```bash
-curl --location 'http://0.0.0.0:4000/chat/completions' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "minimax-m2-5",
-  "messages": [
-    {
-      "role": "user",
-      "content": "Solve: 2+2=?"
-    }
-  ],
-  "extra_body": {
-    "reasoning_split": true
-  }
-}'
-```
-
-## Usage - Anthropic Compatible API (/v1/messages)
-
-<Tabs>
-<TabItem value="proxy" label="LiteLLM Proxy">
-
-**1. Setup config.yaml**
-
-```yaml
-model_list:
-  - model_name: minimax-m2-5
-    litellm_params:
-      model: minimax/MiniMax-M2.5
-      api_key: os.environ/MINIMAX_API_KEY
-      api_base: https://api.minimax.io/anthropic/v1/messages
-```
-
-**2. Start the proxy**
-
-```bash
-docker run -d \
-  -p 4000:4000 \
-  -e MINIMAX_API_KEY=$MINIMAX_API_KEY \
-  -v $(pwd)/config.yaml:/app/config.yaml \
-  ghcr.io/berriai/litellm:v1.81.3-stable \
-  --config /app/config.yaml
-```
-
-**3. Test it!**
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "minimax-m2-5",
-  "max_tokens": 1000,
-  "messages": [
-    {
-      "role": "user",
-      "content": "what llm are you"
-    }
-  ]
-}'
-```
-
-</TabItem>
-</Tabs>
-
-### With Thinking
-
-```bash
-curl --location 'http://0.0.0.0:4000/v1/messages' \
---header 'Content-Type: application/json' \
---header 'Authorization: Bearer $LITELLM_KEY' \
---data '{
-  "model": "minimax-m2-5",
-  "max_tokens": 1000,
-  "thinking": {
-    "type": "enabled",
-    "budget_tokens": 1000
-  },
-  "messages": [
-    {
-      "role": "user",
-      "content": "Solve: 2+2=?"
-    }
-  ]
-}'
-```
-
-## Usage - LiteLLM SDK
-
-### OpenAI-compatible API
-
-```python
-import litellm
-
-response = litellm.completion(
-    model="minimax/MiniMax-M2.5",
-    messages=[
-        {"role": "user", "content": "Hello, how are you?"}
-    ],
-    api_key="your-minimax-api-key",
-    api_base="https://api.minimax.io/v1"
-)
-
-print(response.choices[0].message.content)
-```
-
-### Anthropic-compatible API
-
-```python
-import litellm
-
-response = litellm.anthropic.messages.acreate(
-    model="minimax/MiniMax-M2.5",
-    messages=[{"role": "user", "content": "Hello, how are you?"}],
-    api_key="your-minimax-api-key",
-    api_base="https://api.minimax.io/anthropic/v1/messages",
-    max_tokens=1000
-)
-
-print(response.choices[0].message.content)
-```
-
-### With Thinking
-
-```python
-response = litellm.anthropic.messages.acreate(
-    model="minimax/MiniMax-M2.5",
-    messages=[{"role": "user", "content": "Solve: 2+2=?"}],
-    thinking={"type": "enabled", "budget_tokens": 1000},
-    api_key="your-minimax-api-key"
-)
-
-# Access thinking content
-for block in response.choices[0].message.content:
-    if hasattr(block, 'type') and block.type == 'thinking':
-        print(f"Thinking: {block.thinking}")
-```
-
-### With Reasoning Split (OpenAI API)
-
-```python
-response = litellm.completion(
-    model="minimax/MiniMax-M2.5",
-    messages=[
-        {"role": "user", "content": "Solve: 2+2=?"}
-    ],
-    extra_body={"reasoning_split": True},
-    api_key="your-minimax-api-key",
-    api_base="https://api.minimax.io/v1"
-)
-
-# Access thinking and response
-if hasattr(response.choices[0].message, 'reasoning_details'):
-    print(f"Thinking: {response.choices[0].message.reasoning_details}")
-print(f"Response: {response.choices[0].message.content}")
-```
-
-## Cost Tracking
-
-LiteLLM automatically tracks costs for MiniMax-M2.5 requests. The pricing is:
-
-- **Input**: $0.3 per 1M tokens
-- **Output**: $1.2 per 1M tokens
-- **Cache Read**: $0.03 per 1M tokens
-- **Cache Write**: $0.375 per 1M tokens
-
-### Accessing Cost Information
-
-```python
-response = litellm.completion(
-    model="minimax/MiniMax-M2.5",
-    messages=[{"role": "user", "content": "Hello!"}],
-    api_key="your-minimax-api-key"
-)
-
-# Access cost information
-print(f"Cost: ${response._hidden_params.get('response_cost', 0)}")
-```
-
-## Streaming Support
-
-### OpenAI API
-
-```python
-response = litellm.completion(
-    model="minimax/MiniMax-M2.5",
-    messages=[{"role": "user", "content": "Tell me a story"}],
-    stream=True,
-    api_key="your-minimax-api-key",
-    api_base="https://api.minimax.io/v1"
-)
-
-for chunk in response:
-    if chunk.choices[0].delta.content:
-        print(chunk.choices[0].delta.content, end="")
-```
-
-### Streaming with Reasoning Split
-
-```python
-stream = litellm.completion(
-    model="minimax/MiniMax-M2.5",
-    messages=[
-        {"role": "user", "content": "Tell me a story"},
-    ],
-    extra_body={"reasoning_split": True},
-    stream=True,
-    api_key="your-minimax-api-key",
-    api_base="https://api.minimax.io/v1"
-)
-
-reasoning_buffer = ""
-text_buffer = ""
-
-for chunk in stream:
-    if hasattr(chunk.choices[0].delta, "reasoning_details") and chunk.choices[0].delta.reasoning_details:
-        for detail in chunk.choices[0].delta.reasoning_details:
-            if "text" in detail:
-                reasoning_text = detail["text"]
-                new_reasoning = reasoning_text[len(reasoning_buffer):]
-                if new_reasoning:
-                    print(new_reasoning, end="", flush=True)
-                    reasoning_buffer = reasoning_text
-
-    if chunk.choices[0].delta.content:
-        content_text = chunk.choices[0].delta.content
-        new_text = content_text[len(text_buffer):] if text_buffer else content_text
-        if new_text:
-            print(new_text, end="", flush=True)
-            text_buffer = content_text
-```
-
-## Using with Native SDKs
-
-### Anthropic SDK via LiteLLM Proxy
-
-```python
-import os
-os.environ["ANTHROPIC_BASE_URL"] = "http://localhost:4000"
-os.environ["ANTHROPIC_API_KEY"] = "sk-1234"  # Your LiteLLM proxy key
-
-import anthropic
-
-client = anthropic.Anthropic()
-
-message = client.messages.create(
-    model="minimax-m2-5",
-    max_tokens=1000,
-    system="You are a helpful assistant.",
-    messages=[
-        {
-            "role": "user",
-            "content": [
-                {
-                    "type": "text",
-                    "text": "Hi, how are you?"
-                }
-            ]
-        }
-    ]
-)
-
-for block in message.content:
-    if block.type == "thinking":
-        print(f"Thinking:\n{block.thinking}\n")
-    elif block.type == "text":
-        print(f"Text:\n{block.text}\n")
-```
-
-### OpenAI SDK via LiteLLM Proxy
-
-```python
-import os
-os.environ["OPENAI_BASE_URL"] = "http://localhost:4000"
-os.environ["OPENAI_API_KEY"] = "sk-1234"  # Your LiteLLM proxy key
-
-from openai import OpenAI
-
-client = OpenAI()
-
-response = client.chat.completions.create(
-    model="minimax-m2-5",
-    messages=[
-        {"role": "system", "content": "You are a helpful assistant."},
-        {"role": "user", "content": "Hi, how are you?"},
-    ],
-    extra_body={"reasoning_split": True},
-)
-
-# Access thinking and response
-if hasattr(response.choices[0].message, 'reasoning_details'):
-    print(f"Thinking:\n{response.choices[0].message.reasoning_details[0]['text']}\n")
-print(f"Text:\n{response.choices[0].message.content}\n")
-```

docs/my-website/blog/model_cost_map_incident/index.md

@@ -1,92 +0,0 @@
----
-slug: model-cost-map-incident
-title: "Incident Report: Invalid model cost map on main"
-date: 2026-02-10T10:00:00
-authors:
-  - ishaan
-tags: [incident-report, stability]
-hide_table_of_contents: false
----
-
-**Date:** January 27, 2026
-**Duration:** ~20 minutes
-**Severity:** Low
-**Status:** Resolved
-
-## Summary
-
-A malformed JSON entry in `model_prices_and_context_window.json` was merged to `main` ([`562f0a0`](https://github.com/BerriAI/litellm/commit/562f0a028251750e3d75386bee0e630d9796d0df)). This caused LiteLLM to silently fall back to a stale local copy of the model cost map. Users on older package versions lost cost tracking for newer models only (e.g. `azure/gpt-5.2`). No LLM calls were blocked.
-
-- **LLM calls and proxy routing:** No impact.
-- **Cost tracking:** Impacted for newer models not present in the local backup. Older models were unaffected. The incident lasted ~20 minutes until the commit was reverted.
-
-{/* truncate */}
-
----
-
-## Background
-
-The model cost map is not in the request path. It is used after the LLM response comes back, inside a try/catch, to calculate spend. A missing entry never blocks a call.
-
-```mermaid
-flowchart TD
-    A["1. litellm.completion() receives request
-    litellm/main.py"] --> B["2. Route to provider
-    litellm/litellm_core_utils/get_llm_provider_logic.py"]
-    B --> C["3. LLM returns response
-    litellm/main.py"]
-    C --> D["4. Post-call: look up model in cost map
-    litellm/cost_calculator.py"]
-    D -->|"found"| E["5a. Attach cost to response"]
-    D -->|"not found (try/catch)"| F["5b. Log warning, set cost=0"]
-    E --> G["6. Return response to caller"]
-    F --> G
-
-    style D fill:#fff3cd,stroke:#ffc107
-    style F fill:#fff3cd,stroke:#ffc107
-    style E fill:#d4edda,stroke:#28a745
-    style G fill:#d4edda,stroke:#28a745
-```
-
-Both paths return a response to the caller. When the cost map lookup fails, the only difference is `cost=0` on that request.
-
----
-
-## Root cause
-
-LiteLLM fetches the model cost map from GitHub `main` at import time. If the fetch fails, it falls back to a local backup bundled with the package. Before this incident, the fallback was completely silent -- no warning was logged.
-
-A contributor PR introduced an extra `{` bracket, producing invalid JSON. The remote fetch failed with `JSONDecodeError`, triggering the silent fallback. Users on older package versions had backup files missing newer models.
-
-**Timeline:**
-
-1. Malformed JSON merged to `main`
-2. LiteLLM installations fall back to local backup on next import
-3. Users report `"This model isn't mapped yet"` for newer models
-4. Bad commit identified and reverted (~20 minutes)
-
----
-
-## Remediation
-
-| # | Action | Status | Code |
-|---|---|---|---|
-| 1 | CI validation on `model_prices_and_context_window.json` | ✅ Done | [`test-model-map.yaml`](https://github.com/BerriAI/litellm/blob/main/.github/workflows/test-model-map.yaml) |
-| 2 | Warning log on fallback to local backup | ✅ Done | [`get_model_cost_map.py#L57-L68`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/get_model_cost_map.py#L57-L68) |
-| 3 | `GetModelCostMap` class with integrity validation helpers | ✅ Done | [`get_model_cost_map.py#L24-L149`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/get_model_cost_map.py#L24-L149) |
-| 4 | Resilience test suite (bad hosted map, fallback, completion) | ✅ Done | [`test_model_cost_map_resilience.py#L150-L291`](https://github.com/BerriAI/litellm/blob/main/tests/llm_translation/test_model_cost_map_resilience.py#L150-L291) |
-| 5 | Test that backup model cost map always exists and contains common models | ✅ Done | [`test_model_cost_map_resilience.py#L213-L228`](https://github.com/BerriAI/litellm/blob/main/tests/llm_translation/test_model_cost_map_resilience.py#L213-L228) |
-
-Enterprises that require zero external dependencies at import time can set `LITELLM_LOCAL_MODEL_COST_MAP=True` to skip the GitHub fetch entirely.
-
----
-
-## Other dependencies on external resources
-
-| Dependency | Impact if unavailable | Fallback |
-|---|---|---|
-| Model cost map (GitHub) | Cost tracking for newer models | Local backup (now with warning) |
-| JWT public keys (IDP/SSO) | Auth fails | None |
-| OIDC UserInfo (IDP/SSO) | Auth fails | None |
-| HuggingFace model API | HF provider calls fail | None |
-| Ollama tags (localhost) | Ollama model list stale | Static list |

docs/my-website/blog/realtime_webrtc_http_endpoints/index.md

@@ -1,111 +0,0 @@
----
-slug: realtime_webrtc_http_endpoints
-title: "Realtime WebRTC HTTP Endpoints"
-date: 2026-03-12T10:00:00
-authors:
-  - sameer
-  - krrish
-  - ishaan-alt
-description: "Use the LiteLLM proxy to route OpenAI-style WebRTC realtime via HTTP: client_secrets and SDP exchange."
-tags: [realtime, webrtc, proxy, openai]
-hide_table_of_contents: false
----
-
-import WebRTCTester from '@site/src/components/WebRTCTester';
-
-Connect to the Realtime API via WebRTC from browser/mobile clients. LiteLLM handles auth and key management.
-
-{/* truncate */}
-
-## How it works
-
-![WebRTC flow: Browser, LiteLLM Proxy, and OpenAI/Azure](../../img/webrtc_flow.png)
-
-**Flow of generating ephemeral token**
-
-![Ephemeral token flow: Browser requests token, LiteLLM gets real token from OpenAI, returns encrypted token](../../img/ephemeral_token.png)
-
-
-## Proxy Setup
-
-```yaml
-model_list:
-  - model_name: gpt-4o-realtime
-    litellm_params:
-      model: openai/gpt-4o-realtime-preview-2024-12-17
-      api_key: os.environ/OPENAI_API_KEY
-    model_info:
-      mode: realtime
-```
-
-**Azure:** use `model: azure/gpt-4o-realtime-preview`, `api_key`, `api_base`.
-
-```bash
-litellm --config /path/to/config.yaml
-```
-
-## Try it live
-
-<WebRTCTester />
-
-## Client Usage
-
-**1. Get token** - `POST /v1/realtime/client_secrets` with LiteLLM API key and `{ model }`.
-
-**2. WebRTC handshake** - Create `RTCPeerConnection`, add mic track, create data channel `oai-events`, send SDP offer to `POST /v1/realtime/calls` with `Authorization: Bearer <encrypted_token>` and `Content-Type: application/sdp`.
-
-**3. Events** - Use the data channel for `session.update` and other events.
-
-<details>
-<summary>Full code example</summary>
-
-```javascript
-// 1. Token
-const r = await fetch("http://proxy:4000/v1/realtime/client_secrets", {
-  method: "POST",
-  headers: { "Authorization": "Bearer sk-litellm-key", "Content-Type": "application/json" },
-  body: JSON.stringify({ model: "gpt-4o-realtime" }),
-});
-const { client_secret } = await r.json();
-const token = client_secret.value;
-
-// 2. WebRTC
-const pc = new RTCPeerConnection();
-const audio = document.createElement("audio");
-audio.autoplay = true;
-pc.ontrack = (e) => (audio.srcObject = e.streams[0]);
-const ms = await navigator.mediaDevices.getUserMedia({ audio: true });
-pc.addTrack(ms.getTracks()[0]);
-const dc = pc.createDataChannel("oai-events");
-const offer = await pc.createOffer();
-await pc.setLocalDescription(offer);
-
-const sdpRes = await fetch("http://proxy:4000/v1/realtime/calls", {
-  method: "POST",
-  headers: { "Authorization": `Bearer ${token}`, "Content-Type": "application/sdp" },
-  body: offer.sdp,
-});
-await pc.setRemoteDescription({ type: "answer", sdp: await sdpRes.text() });
-
-// 3. Events
-dc.send(JSON.stringify({ type: "session.update", session: { instructions: "..." } }));
-```
-
-</details>
-
-## FAQ
-
-**Q: What do I do if I get a 401 Token expired error?**  
-A: Tokens are short-lived. Get a fresh token right before creating the WebRTC offer.
-
-**Q: Which key should I use for `/v1/realtime/calls`?**  
-A: Use the **encrypted token** from `client_secrets`, not your raw API key.
-
-**Q: Should I pass the `model` parameter when making the call?**  
-A: No, the encrypted token already encodes all routing information including model.
-
-**Q: How do I resolve Azure `api-version` errors?**  
-A: Set the correct `api_version` in `litellm_params` (or via the `AZURE_API_VERSION` environment variable), along with the right `api_base` and deployment values.
-
-**Q: What if I get no audio?**  
-A: Make sure you grant microphone permission, ensure `pc.ontrack` assigns the audio element with `autoplay` enabled, check your network/firewall for WebRTC traffic, and inspect the browser console for ICE or SDP errors.

docs/my-website/blog/redis_circuit_breaker/diagrams.js

@@ -1,159 +0,0 @@
-import React from 'react';
-
-const s = {
-  fig: {margin: '2.5rem 0', fontFamily: 'inherit'},
-  box: {borderRadius: 12, border: '1px solid #e5e7eb', background: '#fff', padding: '2rem 2.5rem'},
-  label: {fontSize: 11, fontWeight: 700, textTransform: 'uppercase', letterSpacing: '0.12em', color: '#9ca3af', textAlign: 'center', marginBottom: '1.5rem'},
-  caption: {textAlign: 'center', fontSize: 12, color: '#9ca3af', marginTop: 12},
-  node: (border='#d1d5db', bg='#f9fafb') => ({
-    border: `1px solid ${border}`, borderRadius: 6, padding: '8px 20px',
-    fontSize: 13, background: bg, display: 'inline-block',
-  }),
-  arrow: {display: 'flex', flexDirection: 'column', alignItems: 'center'},
-};
-
-const SmallArrow = ({color='#9ca3af'}) => (
-  <svg width="2" height="28" style={{display:'block'}}>
-    <line x1="1" y1="0" x2="1" y2="22" stroke={color} strokeWidth="1.5"/>
-    <polygon points="1,28 -2,21 4,21" fill={color}/>
-  </svg>
-);
-
-export function CascadeFailure() {
-  return (
-    <figure style={s.fig}>
-      <div style={s.box}>
-        <p style={s.label}>Without circuit breaker — cascade failure</p>
-        <div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:0}}>
-          <div style={s.node()}>LiteLLM Pod (×100)</div>
-          <SmallArrow />
-          <div style={s.node()}>Rate limit / cache check</div>
-          <div style={{position:'relative', display:'flex', flexDirection:'column', alignItems:'center'}}>
-            <SmallArrow color="#f87171"/>
-            <span style={{position:'absolute', left:8, top:4, fontSize:11, color:'#f87171', whiteSpace:'nowrap'}}>hangs 30s per request</span>
-          </div>
-          <div style={s.node('#fca5a5','#fef2f2')}><span style={{color:'#b91c1c', fontWeight:600}}>Redis — degraded, timing out</span></div>
-          <SmallArrow color="#fb923c"/>
-          <div style={s.node('#fdba74','#fff7ed')}><span style={{color:'#c2410c', fontWeight:600}}>Postgres — 100× normal read load</span></div>
-          <SmallArrow />
-          <div style={{...s.node('#111827','#111827'), color:'#fff', fontWeight:600}}>Total outage — gateway down</div>
-        </div>
-      </div>
-      <figcaption style={s.caption}>Slow Redis → every auth check times out → database overwhelmed → full cascade</figcaption>
-    </figure>
-  );
-}
-
-export function CircuitBreakerStates() {
-  const circle = (border, color, label, sub) => (
-    <div style={{display:'flex', flexDirection:'column', alignItems:'center', width: 140}}>
-      <div style={{width:88, height:88, borderRadius:'50%', border:`2px solid ${border}`, background:'#fff', display:'flex', flexDirection:'column', alignItems:'center', justifyContent:'center'}}>
-        <span style={{fontSize:11, fontWeight:700, color, letterSpacing:'0.06em'}}>{label}</span>
-        <span style={{fontSize:10, color:'#9ca3af', marginTop:2}}>{sub}</span>
-      </div>
-      <p style={{fontSize:11, color:'#6b7280', textAlign:'center', marginTop:10, lineHeight:1.5}}>{'\u00a0'}</p>
-    </div>
-  );
-  const arrow = (label) => (
-    <div style={{display:'flex', flexDirection:'column', alignItems:'center', marginTop:36, marginLeft:4, marginRight:4}}>
-      <span style={{fontSize:10, color:'#6b7280', marginBottom:4}}>{label}</span>
-      <div style={{display:'flex', alignItems:'center'}}>
-        <div style={{height:1, width:48, background:'#9ca3af'}}/>
-        <svg width="8" height="8" style={{marginLeft:-1}}><polygon points="0,0 8,4 0,8" fill="#6b7280"/></svg>
-      </div>
-    </div>
-  );
-  return (
-    <figure style={s.fig}>
-      <div style={s.box}>
-        <p style={s.label}>Circuit breaker state machine</p>
-        <div style={{display:'flex', justifyContent:'center', alignItems:'flex-start'}}>
-          {circle('#1f2937','#111827','CLOSED','normal')}
-          {arrow('5 failures')}
-          {circle('#f87171','#dc2626','OPEN','fast-fail')}
-          {arrow('60s timeout')}
-          {circle('#fbbf24','#b45309','HALF-OPEN','probing')}
-        </div>
-        <div style={{display:'flex', justifyContent:'center', gap:32, marginTop:24}}>
-          <div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:4}}>
-            <div style={{display:'flex', alignItems:'center', gap:4}}>
-              <svg width="8" height="8"><polygon points="8,0 0,4 8,8" fill="#16a34a"/></svg>
-              <div style={{height:1, width:100, background:'#16a34a'}}/>
-            </div>
-            <span style={{fontSize:10, color:'#16a34a'}}>probe success → CLOSED</span>
-          </div>
-          <div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:4}}>
-            <div style={{display:'flex', alignItems:'center', gap:4}}>
-              <svg width="8" height="8"><polygon points="8,0 0,4 8,8" fill="#ef4444"/></svg>
-              <div style={{height:1, width:100, borderTop:'2px dashed #f87171'}}/>
-            </div>
-            <span style={{fontSize:10, color:'#ef4444'}}>probe failure → OPEN again</span>
-          </div>
-        </div>
-      </div>
-    </figure>
-  );
-}
-
-export function CircuitBreakerFlow() {
-  return (
-    <figure style={s.fig}>
-      <div style={s.box}>
-        <p style={s.label}>With circuit breaker — graceful degradation</p>
-        <div style={{display:'flex', flexDirection:'column', alignItems:'center'}}>
-          <div style={s.node()}>Incoming request</div>
-          <SmallArrow />
-          <div style={{...s.node('#111827'), border:'2px solid #111827', fontWeight:600}}>Circuit Breaker</div>
-          <div style={{display:'flex', gap:80, marginTop:20, alignItems:'flex-start'}}>
-            <div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:8}}>
-              <SmallArrow />
-              <span style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.06em', color:'#6b7280', border:'1px solid #e5e7eb', borderRadius:4, padding:'2px 8px'}}>Closed</span>
-              <div style={{...s.node(), textAlign:'center', fontSize:13}}>Redis call<br/><span style={{fontSize:11, color:'#9ca3af'}}>normal latency</span></div>
-            </div>
-            <div style={{display:'flex', flexDirection:'column', alignItems:'center', gap:8}}>
-              <SmallArrow />
-              <span style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.06em', color:'#ef4444', border:'1px solid #fca5a5', borderRadius:4, padding:'2px 8px'}}>Open</span>
-              <div style={{...s.node('#fca5a5'), textAlign:'center', fontSize:13}}>Fast-fail — 0ms<br/><span style={{fontSize:11, color:'#9ca3af'}}>no network call</span></div>
-              <SmallArrow />
-              <div style={{...s.node(), textAlign:'center', fontSize:13}}>DB fallback<br/><span style={{fontSize:11, color:'#9ca3af'}}>bounded load</span></div>
-            </div>
-          </div>
-          <div style={{...s.node('#111827','#111827'), color:'#fff', fontWeight:600, marginTop:24}}>Request completes — gateway stays up</div>
-        </div>
-      </div>
-      <figcaption style={s.caption}>Redis down → circuit opens → 0ms rejection → DB absorbs bounded fallback traffic</figcaption>
-    </figure>
-  );
-}
-
-export function IncidentTimeline() {
-  const row = (color, text) => (
-    <div style={{display:'flex', alignItems:'flex-start', gap:10, marginBottom:12}}>
-      <div style={{marginTop:5, width:6, height:6, borderRadius:'50%', background:color, flexShrink:0}}/>
-      <p style={{fontSize:13, color:'#4b5563', margin:0, lineHeight:1.5}}>{text}</p>
-    </div>
-  );
-  return (
-    <figure style={s.fig}>
-      <div style={s.box}>
-        <p style={s.label}>Redis degrades — before vs. after</p>
-        <div style={{display:'grid', gridTemplateColumns:'1fr 1fr', gap:20}}>
-          <div style={{border:'1px solid #e5e7eb', borderRadius:8, padding:20}}>
-            <p style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.1em', color:'#9ca3af', marginBottom:16}}>Without circuit breaker</p>
-            {row('#f87171','All 100 pods hang for 30s on each auth check')}
-            {row('#f87171','Threadpools fill up, requests queue')}
-            {row('#f87171','100× simultaneous DB fallbacks overwhelm Postgres')}
-            {row('#f87171','Requires manual intervention to recover')}
-          </div>
-          <div style={{border:'1px solid #111827', borderRadius:8, padding:20}}>
-            <p style={{fontSize:10, fontWeight:700, textTransform:'uppercase', letterSpacing:'0.1em', color:'#9ca3af', marginBottom:16}}>With circuit breaker</p>
-            {row('#111827','Circuit opens after 5 failures — 0ms fast-fail')}
-            {row('#111827','Auth falls back to DB — bounded, not 100× load')}
-            {row('#111827','Cache miss rate temporarily elevated — gateway stays up')}
-            {row('#111827','Auto-recovers when Redis comes back — no intervention needed')}
-          </div>
-        </div>
-      </div>
-    </figure>
-  );
-}

docs/my-website/blog/redis_circuit_breaker/index.md

@@ -1,141 +0,0 @@
----
-slug: redis-circuit-breaker
-title: "Making the AI Gateway Resilient to Redis Failures"
-date: 2026-04-11T09:00:00
-authors:
-  - ishaan
-description: "How LiteLLM's production AI Gateway handles Redis degradation at scale without cascading failures — circuit breaker pattern, 0ms fast-fail, automatic recovery."
-tags: [reliability, redis, infrastructure, engineering, ai-gateway]
-hide_table_of_contents: true
----
-
-import { CascadeFailure, CircuitBreakerStates, CircuitBreakerFlow, IncidentTimeline } from './diagrams';
-
-*Last Updated: April 2026*
-
-Enterprise AI Gateway deployments put Redis in the hot path for nearly every request: rate limiting, cache lookups, spend tracking. When Redis is healthy, the latency contribution is single-digit milliseconds — invisible to end users. When it degrades, a production AI Gateway needs to stay up regardless.
-
-Running LiteLLM at scale across 100+ pods means designing for failure modes before they appear. The easy case is Redis going fully down: fail fast, fall through to the database, continue serving requests. The hard case — the one that takes down gateways — is a *slow* Redis: still accepting connections, still responding, but timing out after 20-30 seconds per operation.
-
-{/* truncate */}
-
-## Why slow Redis is harder than a full outage
-
-<CascadeFailure />
-
-With 100 pods each hanging 30 seconds on every auth check, threadpools fill up and requests queue. By the time Redis times out and falls through to Postgres, the database receives 100× its normal load from simultaneous fallbacks. A slow Redis becomes a database outage becomes a full gateway outage. A production-grade AI Gateway cannot allow one degraded dependency to cascade into total failure.
-
-## The fix: circuit breaker pattern
-
-The circuit breaker pattern tracks consecutive failures and cuts off the unhealthy dependency before it cascades. Instead of hanging 30 seconds on each Redis call, the circuit opens after 5 consecutive failures and fast-fails at 0ms — no network call, no wait.
-
-<CircuitBreakerStates />
-
-Three states:
-
-- **CLOSED** — normal. All Redis calls pass through.
-- **OPEN** — Redis is unhealthy. Every call fast-fails instantly. Requests continue with degraded-but-functional behavior: auth and rate limiting fall back to the database.
-- **HALF-OPEN** — after 60 seconds, one probe request tests recovery. Success closes the circuit; failure resets the timer.
-
-This is how a reliable AI Gateway handles infrastructure degradation: stay up, degrade gracefully, recover automatically.
-
-## How requests flow through the AI Gateway
-
-<CircuitBreakerFlow />
-
-When the circuit is open, the gateway does not stall. Auth checks fall back to Postgres — slower, but bounded. The database absorbs the load because it receives *some* requests via DB fallback, not *all* 100 pods simultaneously dumping their queued requests after a 30-second timeout.
-
-The difference between a resilient AI Gateway and a fragile one: controlled degradation vs. uncontrolled cascade.
-
-## The implementation
-
-```python
-class RedisCircuitBreaker:
-    def __init__(self, failure_threshold: int, recovery_timeout: int):
-        self.failure_threshold = failure_threshold  # default: 5
-        self.recovery_timeout = recovery_timeout    # default: 60s
-        self._failure_count = 0
-        self._state = self.CLOSED
-
-    def is_open(self) -> bool:
-        if self._state == self.OPEN:
-            if time.time() - self._opened_at > self.recovery_timeout:
-                self._state = self.HALF_OPEN
-                return False  # this caller is the recovery probe
-            return True       # fast-fail
-        return False
-
-    def record_failure(self):
-        self._failure_count += 1
-        self._opened_at = time.time()
-        if self._failure_count >= self.failure_threshold:
-            self._state = self.OPEN  # open the circuit
-
-    def record_success(self):
-        self._failure_count = 0
-        self._state = self.CLOSED   # Redis recovered
-```
-
-Every async Redis operation goes through a decorator that checks the breaker before touching the network. When open, it raises immediately:
-
-```python
-@_redis_circuit_breaker_guard
-async def async_get_cache(self, key: str):
-    ...
-```
-
-The decorator handles all bookkeeping — success resets nothing, failures increment the counter, exceptions trigger `record_failure()`. The caller sees a clean exception and falls through to its normal non-Redis path. No changes required in calling code.
-
-## AI Gateway resilience in production
-
-<IncidentTimeline />
-
-Redis degradation events no longer cascade in production. The observable symptom during a Redis slowdown is a temporary bump in cache miss rate — the right failure mode for a resilient AI Gateway. Auth still works. Rate limiting still works. Spend tracking still works, at slightly higher DB cost. Recovery is fully automatic when Redis comes back.
-
-```bash
-# configure via environment variables
-REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5   # failures before opening
-REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT=60  # seconds before probe
-```
-
-The circuit breaker ships on by default in all LiteLLM versions since `v1.82.0`. No configuration needed for most deployments.
-
-## Key Takeaways
-
-- A slow Redis is more dangerous than a downed one: 30-second timeouts across 100+ pods overwhelm Postgres at 100× normal load
-- LiteLLM's AI Gateway uses a circuit breaker that fast-fails Redis calls at 0ms after 5 consecutive failures
-- Three states: CLOSED (normal), OPEN (fast-fail + DB fallback), HALF-OPEN (probe recovery)
-- Auth, rate limiting, and spend tracking continue working during Redis outages
-- Resilient, production-grade behavior — enabled by default since `v1.82.0`, no configuration required
-
----
-
-### Frequently Asked Questions
-
-### Does the circuit breaker affect normal Redis performance?
-
-No. When Redis is healthy (circuit CLOSED), every call passes through with zero overhead. The breaker only activates after 5 consecutive failures — transparent under normal conditions.
-
-### What happens to rate limiting when the circuit is open?
-
-Rate limiting falls back to Postgres with bounded load. Limits remain enforced at slightly higher DB cost until Redis recovers and the circuit closes automatically.
-
-### How is this different from basic Redis retry logic?
-
-Retry logic still waits for each timeout (30s × retries). The circuit breaker cuts the connection immediately at 0ms after the failure threshold, preventing threadpool exhaustion across all pods simultaneously. Retries make slow-Redis worse; the circuit breaker contains it.
-
-### Is this available in LiteLLM OSS?
-
-Yes. The circuit breaker ships in LiteLLM OSS (Apache 2.0) by default since `v1.82.0`. [LiteLLM Enterprise](https://litellm.ai/enterprise) adds SSO/SCIM, air-gapped deployment, 24/7 SLA support, and advanced guardrails on top of the OSS foundation.
-
----
-
-## Conclusion
-
-Redis resilience is one layer of what makes LiteLLM a production-grade, reliable AI Gateway at scale. The circuit breaker pattern ensures infrastructure degradation stays contained — the right failure mode is a temporary cache miss rate bump, not a full outage. This is how AI Gateway infrastructure should behave under pressure: degrade gracefully, recover automatically, keep serving traffic. For teams with strict uptime and compliance requirements, [LiteLLM Enterprise](https://litellm.ai/enterprise) provides the additional controls needed for regulated production environments.
-
-## Recommended Reading
-
-- [LiteLLM AI Gateway — full feature overview](https://docs.litellm.ai/docs/simple_proxy)
-- [Load balancing and routing across 100+ LLM providers](https://docs.litellm.ai/docs/routing)
-- [Spend tracking and budget controls](https://docs.litellm.ai/docs/proxy/cost_tracking)