.agents/skills/database-access/SKILL.md

@@ -1,73 +0,0 @@
----
-name: database-access
-description: Organize database repositories and proxy dependency wiring, manage connection ownership, or port Prisma operations to Rust and SQLx without expanding migration scope
-metadata:
-  internal: true
----
-
-# Database access
-
-## Situations
-
-- Adding a query or an operation to a repository, read [adding a query](references/python-repository.md#adding-a-query)
-- Moving existing Prisma calls behind a repository, read [moving existing access](references/python-repository.md#moving-existing-access-into-a-repository)
-- Porting a repository from Prisma to Rust and SQLx, read [porting to SQLx](references/python-repository.md#porting-a-repository-to-sqlx)
-- Injecting a repository into a route, read [Python and FastAPI](references/python-fastapi.md) or [Rust and Axum](references/rust-axum.md)
-
-## Adding new database access
-
-- Put Python database access behind an operation in `litellm/repositories`
-- Give each repository its own FastAPI provider, so a route builds only the repositories it uses
-- A provider borrows existing handles and performs no database I/O during construction
-- Define the operations a route calls as a Protocol in `litellm/models/<feature>.py`, next to its domain types
-  - The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
-  - Parsing and serialization of stored values live in the repository
-  - Add a Protocol when a repository gets a provider, not up front for every repository
-- Keep query execution inside the repository, with one implementation per repository and no backend switch or fallback
-- A repository is either all Prisma or all native, never a mix of both
-- Prisma is fine for the implementation, adding a query does not require a Rust port
-- Native Rust callers use `litellm-rust/crates/db` directly
-
-## Framework wiring
-
-- Python routes receive a per-repository dependency such as `UserBannerRepositoryDep` and call its operations, such as `banners.get_banner()`
-- `DatabaseDep` lives in `litellm/proxy/deps/database.py`, repository providers in `litellm/proxy/deps/repositories.py`
-  - A provider returns the feature's Protocol, so routes never see which backend implements it
-  - Dependencies used by one feature only, like the banner audit binding, stay next to that feature's routes
-- Repositories never import FastAPI, so jobs and helpers can use them without a request
-- Use named FastAPI providers with `Annotated[..., Depends(...)]`, ordinary helpers and jobs receive explicit arguments
-- Preserve each route's no-database behavior, authorization precedence and background-task scheduling
-- Read [Python and FastAPI](references/python-fastapi.md) for provider examples, lifespan state and route verification
-- Read [Rust and Axum](references/rust-axum.md) when adding database access to Rust handlers
-
-## Connection ownership
-
-- Keep connection setup and shutdown separate from repositories, which own no pools
-- Reuse existing handles, never connect per request or disconnect from a repository operation
-- Give each pool one owner for setup and shutdown, borrowed handles never close it
-- Finish requests, jobs and queued database work before closing connections
-- Clean up resources opened before a startup failure, without a second disconnect fallback
-- Add a backend to the existing lifecycle owner and pass its handle only to repositories that need it
-- Preserve cancellation, timeout, error and transaction semantics across the native bridge
-
-## Changing ownership or reloading resources
-
-- Inventory initialization, reload, tests, background jobs and shutdown together
-- Move every writer when changing authoritative storage, copying a replaceable global into state is insufficient
-- A frozen repository does not make its clients immutable
-- Decide whether each request captures one resource generation or reads it again during processing
-- Build and validate replacements before publishing them, preserving the old generation on failure
-- Serialize reloads and coordinate publication with shutdown
-- Drain streams and background users before retiring old resources, retaining every generation grows resource use
-- Define multi-worker coordination separately from an in-process swap
-- Choose mechanisms such as `ArcSwap` only after defining publication, capture and retirement
-
-## Migration context and boundaries
-
-- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations with Rust APIs one repository at a time
-- Prisma and native repositories coexist during the transition, unrelated existing calls can stay as they are
-- Prisma and SQLx keep separate client pools to PgBouncer, which pools connections to PostgreSQL
-  - Size both layers together and preserve reader/writer routing and pooling-mode compatibility
-- PgBouncer does not share transactions between Prisma and SQLx, keep each atomic unit of work on one backend and transaction
-- Never retry a failed Rust write through Prisma when it may already have committed
-- Prisma owns schema migrations today, eventual SQLx migration ownership and Prisma removal are separate planned steps, not part of a query port

.agents/skills/database-access/references/python-fastapi.md

@@ -1,101 +0,0 @@
-# Python and FastAPI
-
-## Existing pattern
-
-- `UserBannerStore` in `litellm/models/user_banner.py` is the Protocol the banner routes call
-- `get_database` in `litellm/proxy/deps/database.py` reads the lifespan-owned handle from request state
-- `get_user_banner_repository` in `litellm/proxy/deps/repositories.py` builds the banner repository from that handle
-- Choose each repository's replica routing once in its provider, never at call sites
-- A handler receives `UserBannerStore | None`, never a Prisma client or a concrete repository
-- `UserBannerRepository` implements `UserBannerStore` structurally, keeping the existing Prisma table, serialization and configuration-sync behavior
-- The banner audit provider binds the legacy audit callable at request time, without performing audit I/O
-- A native repository can later implement the same Protocol without changing handler signatures, see [porting a repository to SQLx](python-repository.md#porting-a-repository-to-sqlx)
-
-## Dependency injection
-
-- Add one provider and one `Annotated` alias per repository in `litellm/proxy/deps/repositories.py`
-- Annotate the provider and alias with the feature's Protocol, not the concrete repository
-- A route depends only on the repositories it calls, so a request never builds repositories it does not use
-- Do not build repositories once at lifespan, a reconnect can replace the Prisma client they wrap
-
-```python
-class UserBannerStore(Protocol):
-    async def get_banner(self) -> UserBanner: ...
-
-    async def save_banner(self, banner: UserBanner) -> None: ...
-```
-
-```python
-async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
-    return None if database is None else UserBannerRepository(database.prisma)
-
-
-UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)]
-```
-
-- FastAPI caches each provider within a request, so routes and sub-dependencies share one instance
-- Ordinary helpers and jobs receive explicit arguments, never unresolved `Depends` defaults
-- A route still reading the `prisma_client` global or building a repository inline switches to the repository's `...RepositoryDep`, keeping its no-database behavior and error precedence
-
-```python
-async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner:
-    if banners is None:
-        return UserBanner()
-    return await banners.get_banner()
-```
-
-## Lifespan state
-
-- Prisma startup owns connection setup, authentication refresh, health checks and migrations
-- `Database` holds the initialized Prisma client and the `NativeDatabase` handle, without holding fields per feature
-  - The native handle opens no pool yet, its `connect` and `close` are noops
-- The proxy lifespan builds the handle with `Database.open`, registers `Database.aclose` on its exit stack and publishes it in request state
-  - Callers only open and close `Database`, which closes Prisma and the native handle together, and `open` closes Prisma itself when the native connect fails
-- Starlette shallow-copies lifespan state into request state, which is distinct from `app.state`
-- The database module never imports a repository, legacy globals borrow its handle without owning shutdown
-- Drain requests, jobs, logging, spend and audit work before database teardown
-- The exit stack closes the handle on exit, including startup failures after the client is created
-- When adding native resources, initialize them in the serving process, respecting runtime fork restrictions
-- Await native shutdown instead of depending on Python garbage collection
-- Apply the [connection ownership rules](../SKILL.md#connection-ownership) when changing the owner
-
-## Timing and errors
-
-- FastAPI resolves dependencies before entering the handler
-- Handler exception handling cannot catch failures raised during dependency resolution
-- A universal required-database dependency can override authorization and validation behavior
-- Banner GET without a database returns the default disabled banner
-- Banner PATCH checks the admin role before returning the existing missing-database error
-- Invalid stored banner data falls back to the default, but storage failures still propagate
-- Banner writes finish before the success audit is queued as a FastAPI background task
-- Queue post-response work with `BackgroundTasks`, not bare `asyncio.create_task`, so the in-flight drain waits for it before the database closes
-- Do not move background work into an awaited provider merely to remove an import
-
-## Verification
-
-- Exercise real FastAPI handlers using `dependency_overrides` for external boundaries
-- Keep production parsing and persistence code real
-- Restore overrides after each test
-- Verify complete response bodies, status codes and OpenAPI against the base
-- Include combined failure conditions to pin error precedence
-- Check saved values, fresh revisions, audit contents and failed-write behavior
-- Check lifespan state with a running lifespan, not only a TestClient that skips startup
-- Verify providers do not connect, query, disconnect or audit eagerly
-- Use real PostgreSQL and live proxy requests when changing persistence behavior
-
-## Scope
-
-- Preserve direct Python call sites or migrate them explicitly
-- Inventory imports, state writers, side effects and error ordering before migrating a feature
-- Preserve configuration sync and serialization, and state which legacy reads remain
-- Do not migrate every route in a large module just to establish the pattern
-- Do not invent providers for resources without consumers
-
-## Framework references
-
-- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/) and [Starlette lifespan state](https://starlette.dev/lifespan/)
-- [`SessionDep`-style `Annotated` aliases](https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-session-dependency), which `DatabaseDep` and `UserBannerRepositoryDep` follow
-- [Dependency composition and caching](https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/)
-- [Background tasks and dependencies with `yield`](https://fastapi.tiangolo.com/advanced/advanced-dependencies/#background-tasks-and-dependencies-with-yield-technical-details)
-- [Dependency overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
-- [Testing lifespan](https://fastapi.tiangolo.com/advanced/testing-events/)

.agents/skills/database-access/references/python-repository.md

@@ -1,119 +0,0 @@
-# Python repositories
-
-## Adding a query
-
-Use this when a feature needs a read or write that no repository offers yet, including one more operation on an existing repository
-
-### Before starting
-
-- Search `litellm/repositories` for a repository that already owns the table, and extend it instead of adding a second one
-- Check `litellm/models/<feature>.py` for an existing Protocol and domain types
-- Decide the route's behavior when no database is configured, and where that check sits relative to authorization and validation
-
-### Steps
-
-1. Domain types and Protocol in `litellm/models/<feature>.py`
-   - The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
-   - Add the Protocol together with the provider, not for repositories nothing injects
-2. Repository in `litellm/repositories/<feature>_repository.py`
-   - Implement the operation with Prisma, a Rust port is not required
-   - Compose the Prisma client and type the tables it touches with `TableActions` from `prisma_protocols.py`, never inherit `PrismaTableRepository`
-   - Parse and serialize stored values here, so callers only see domain types
-   - Never import FastAPI, connect, disconnect or choose replicas per call
-3. Provider and `Annotated` alias in `litellm/proxy/deps/repositories.py`, see [dependency injection](python-fastapi.md#dependency-injection)
-4. The route takes `<Feature>RepositoryDep`, handles `None` with the behavior decided above, and calls the operation
-5. Jobs and helpers that need the operation receive the repository as an explicit argument, built by their caller from the handle it already holds
-
-### Done when
-
-- Handler tests drive the real route with `dependency_overrides` and the real repository code, see [verification](python-fastapi.md#verification)
-- Tests cover the no-database path, error precedence, and what was actually stored
-- Persistence changes are checked against real PostgreSQL through a live proxy
-
-### Out of scope
-
-- Porting the query to Rust, see [porting a repository to SQLx](#porting-a-repository-to-sqlx)
-- Moving unrelated direct Prisma calls in the same module
-
-## Moving existing access into a repository
-
-Use this when code calls Prisma directly, through `prisma_client.db.<table>` or a helper, and the calls should live behind a repository. The backend stays Prisma
-
-### Before starting
-
-- List every call being moved, its callers, and the transaction or batch it runs in
-- List the module's imports, global state writers, side effects such as audit and config sync, and error ordering
-- Record current return values, raised errors and response bodies, these are what the change must preserve
-
-### Steps
-
-1. Move the existing Prisma code into the relevant repository in `litellm/repositories` without rewriting the queries
-2. Add domain types and a Protocol in `litellm/models/<feature>.py` when a route will inject the repository
-3. Replace each direct call with the repository operation, through a provider for routes and explicit arguments for jobs and helpers, see [Python and FastAPI](python-fastapi.md)
-4. Calls sharing a transaction move into the same repository or unit of work in one change
-5. State which legacy reads remain and why, when the change does not move all of them
-
-### Done when
-
-- Focused regression tests pin the preserved return values, errors and transaction behavior
-- Response bodies, status codes and OpenAPI match the base
-- Configuration sync and serialization behave the same, including invalid stored data
-
-### Out of scope
-
-- Switching the backend, even when an equivalent Rust query exists
-- Converting other routes in the module just to establish the pattern
-- Cleanups unrelated to the moved calls
-
-## Porting a repository to SQLx
-
-Status: TBD. No repository has been ported yet. `database.native` is a `NativeDatabase` from `crates/python-bridge/src/db.rs` whose `connect` and `close` are noops that open no pool, and the PyO3 repository surface does not exist. A real pool takes typed writer/reader settings and a token provider, never the Prisma `DATABASE_URL`, which carries Prisma-only params and rotating IAM tokens. The constraints below are settled, the steps are a sketch to revise after the first port lands
-
-Use this only when a port is explicitly requested. Repository extraction does not require one
-
-### When not to port
-
-- An equivalent Rust or SQLx query existing is not a reason to switch a Prisma caller
-- Leave existing backend selection unchanged unless integration is explicitly requested
-- Never port part of a repository, it is either all Prisma or all native
-
-### Before starting
-
-- Identify the repository's inputs, outputs, errors and transaction boundary
-- List repositories that share a transaction with it, they move together
-- Check `litellm-rust/crates/db` for an equivalent query and reuse it when its behavior matches
-
-### Steps
-
-1. Rust queries, following `litellm-rust/crates/db/AGENTS.md`
-   - Static SQL in `queries/`, compile-checked with SQLx, typed inputs and row projections in `src/`
-   - Dynamic filters and sort keys become closed Rust enums that select static SQL, every value is bound
-   - Refresh SQLx query metadata after SQL changes
-2. PyO3 surface exposing typed operations, never SQL text, table names or column names from Python. TBD: where it lives and how the handle reaches it
-3. Native repository implementing the same Protocol in `litellm/repositories`
-   - Converts typed PyO3 results to domain types, preserving defaults, invalid-data handling and storage errors
-   - Python conversion stays in the bridge, not in `crates/db`
-4. Switch the provider in `litellm/proxy/deps/repositories.py`, the Protocol and routes stay unchanged
-
-```diff
- async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
--    return None if database is None else UserBannerRepository(database.prisma)
-+    return None if database is None else NativeUserBannerRepository(database.native)
-```
-
-5. Delete the Prisma repository in the same change and switch every caller, including jobs and helpers
-
-### Done when
-
-- Behavior is verified against real PostgreSQL, covering null handling, conflicts, ordering, errors and transaction semantics
-- `DATABASE_URL=... cargo test -p litellm-db --locked -- --include-ignored` passes against a Prisma-migrated database
-- Live proxy requests show unchanged responses. TBD: a Python-side real-PostgreSQL test harness
-
-### Rollback
-
-- Roll back the release, never retry an uncertain native write through Prisma
-
-### Out of scope
-
-- SQLx owning schema migrations, and removing Prisma
-- Converting unrelated routes to dependency injection

.agents/skills/database-access/references/rust-axum.md

@@ -1,4 +0,0 @@
-# Rust and Axum
-
-- This should use same `crates/db` what is exposed to Python world.
-- Details are TBD.

.circleci/AGENTS.md

@@ -1,7 +0,0 @@
-# .circleci
-
-The `integration` workflow on `main` runs from a scheduled pipeline in the CircleCI project settings, cron `9 0-8,14-23 * * *` (UTC) on `main` with no pipeline parameters. It has no `triggers:` block in this file because a workflow with one runs only on its schedule, so the API pipelines that set `run_migration_tests` or `routing_parity_base` would stop running it. If main stops getting integration runs, check that the schedule still exists and recreate it with these values
-
-Jobs clone with the `checkout_over_https` command, never the built-in `checkout` step, because CircleCI falls back to the project's SSH checkout key without its GitHub App access, GitHub rejects that key, and the public repo needs no key. It fetches the job's commit with full history and no file contents (`--filter=blob:none`), like the built-in step did, so `scripts/path_filter.sh` can find a merge base
-
-A step that checks out files from another commit fetches that commit with `--no-filter`, as `routing_parity` does, because the clone has no file contents and git would otherwise download each missing file in its own request, which took 210 to 896 seconds for the routing parity base

.circleci/config.yml

@@ -21,30 +21,19 @@ orbs:
   win: circleci/windows@5.0 # Add Windows orb
 
 commands:
-  checkout_over_https:
+  checkout_migration_source:
-    parameters:
-      revision:
-        type: string
-        default: ""
-      shell:
-        type: string
-        default: /bin/bash -eo pipefail
     steps:
       - run:
-          name: Check out the public repository over HTTPS
+          name: Select the requested migration test revision
-          shell: << parameters.shell >>
           environment:
-            REQUESTED_REVISION: << parameters.revision >>
+            MIGRATION_SOURCE_SHA: << pipeline.parameters.migration_source_sha >>
           command: |
-            revision="${REQUESTED_REVISION:-$CIRCLE_SHA1}"
+            revision="${MIGRATION_SOURCE_SHA:-$CIRCLE_SHA1}"
             [[ "$revision" =~ ^[0-9a-f]{40}$ ]] || exit 1
             git init
             git remote add origin https://github.com/BerriAI/litellm.git
-            for attempt in 1 2 3; do
+            git fetch --depth 1 origin "$revision"
-              git fetch --filter=blob:none origin "$revision" && git checkout --detach FETCH_HEAD && break
+            git checkout --detach FETCH_HEAD
-              [[ $attempt -lt 3 ]] || exit 1
-              sleep 5
-            done
   skip_if_unrelated_changes:
     parameters:
       category:
@@ -358,7 +347,7 @@ commands:
       - when:
           condition: << parameters.checkout >>
           steps:
-            - checkout_over_https
+            - checkout
       - setup_google_dns
       - install_uv
       - install_rust
@@ -388,8 +377,7 @@ jobs:
       CARGO_HTTP_MULTIPLEXING: "false"
       CARGO_NET_RETRY: "5"
     steps:
-      - checkout_over_https:
+      - checkout
-          shell: bash.exe
       - run:
           name: Skip job when no windows-release-relevant files changed
           shell: bash.exe
@@ -463,7 +451,7 @@ jobs:
           password: ${DOCKERHUB_PASSWORD}
     working_directory: ~/project
     steps:
-      - checkout_over_https
+      - checkout
       - setup_google_dns
       - install_uv
       - install_rust
@@ -494,7 +482,7 @@ jobs:
     working_directory: ~/project
 
     steps:
-      - checkout_over_https
+      - checkout
       - skip_if_unrelated_changes
       - attach_workspace:
           at: ~/project
@@ -531,36 +519,31 @@ jobs:
             IMAGE_TAG=${CIRCLE_SHA1:-ci}
             kind load docker-image litellm-ci:${IMAGE_TAG} --name litellm-test
 
+      # Run helm lint
       - run:
           name: Run helm lint
           command: |
-            helm lint ./helm/litellm -f ./helm/litellm/ci/test-values.yaml
+            helm lint ./helm/litellm-helm
-
-      - run:
-          name: Start PostgreSQL for the chart
-          command: |
-            kubectl apply -f ./helm/litellm/ci/postgres.yaml
-            kubectl rollout status deployment/litellm-ci-postgres --timeout=3m
 
+      # Run helm tests
       - run:
           name: Run helm tests
           command: |
             IMAGE_TAG=${CIRCLE_SHA1:-ci}
-            helm install litellm ./helm/litellm -f ./helm/litellm/ci/test-values.yaml \
+            helm install litellm ./helm/litellm-helm -f ./helm/litellm-helm/ci/test-values.yaml \
               --set image.repository=litellm-ci \
               --set image.tag=${IMAGE_TAG} \
-              --set image.pullPolicy=Never \
+              --set image.pullPolicy=Never
-              --wait --wait-for-jobs --timeout 10m || {
+            if ! kubectl rollout status deployment/litellm --timeout=180s; then
-                kubectl get pods -o wide
+              kubectl describe pods -l app.kubernetes.io/name=litellm
-                kubectl describe pods -l app.kubernetes.io/instance=litellm
+              kubectl logs -l app.kubernetes.io/name=litellm --tail=200
-                kubectl logs -l app.kubernetes.io/component=proxy --tail=200 || true
+              exit 1
-                kubectl logs -l app.kubernetes.io/component=migrations --tail=200 || true
+            fi
-                exit 1
-              }
 
-            echo "Printing proxy pod logs..."
+            echo "Printing pod logs..."
-            kubectl logs -l app.kubernetes.io/component=proxy --tail=200
+            kubectl logs $(kubectl get pods -l app.kubernetes.io/name=litellm -o jsonpath="{.items[0].metadata.name}")
 
+            # Run the helm tests
             helm test litellm --logs
 
       # Cleanup
@@ -576,7 +559,7 @@ jobs:
     resource_class: medium
     working_directory: ~/project
     steps:
-      - checkout_over_https
+      - checkout
       - run:
           name: Generate LiteLLM master key
           command: |
@@ -676,7 +659,7 @@ jobs:
     resource_class: large
     working_directory: ~/project
     steps:
-      - checkout_over_https
+      - checkout
       - run:
           name: Generate LiteLLM master key
           command: |
@@ -757,7 +740,7 @@ jobs:
     resource_class: large
     working_directory: ~/project
     steps:
-      - checkout_over_https
+      - checkout
       - skip_if_unrelated_changes:
           category: client
       - setup_google_dns
@@ -811,7 +794,7 @@ jobs:
       # the rest of the suite: nothing else performs a logout.
       PROXY_LOGOUT_URL: "https://www.example.com"
     steps:
-      - checkout_over_https
+      - checkout
       - run:
           name: Generate LiteLLM master key
           command: |
@@ -957,7 +940,7 @@ jobs:
       # and by migration.serverRootPath.config.ts, which refuses to run without it.
       SERVER_ROOT_PATH: "/litellm"
     steps:
-      - checkout_over_https
+      - checkout
       - run:
           name: Generate LiteLLM master key
           command: |
@@ -1083,12 +1066,11 @@ jobs:
       - when:
           condition: << parameters.migration_qualification >>
           steps:
-            - checkout_over_https:
+            - checkout_migration_source
-                revision: << pipeline.parameters.migration_source_sha >>
       - unless:
           condition: << parameters.migration_qualification >>
           steps:
-            - checkout_over_https
+            - checkout
             - skip_if_unrelated_changes
 
       - run:
@@ -1104,7 +1086,7 @@ jobs:
               docker build \
                 --label org.opencontainers.image.revision="$(git rev-parse HEAD)" \
                 -t litellm-docker-database:ci \
-                -f Dockerfile .
+                -f docker/Dockerfile.database .
             fi
             python3 .circleci/scripts/run_migration_tests.py record-image
 
@@ -1140,8 +1122,7 @@ jobs:
       MIGRATION_TEST_OUTPUT: /tmp/migration-results
       PYTHONPATH: tests/e2e
     steps:
-      - checkout_over_https:
+      - checkout_migration_source
-          revision: << pipeline.parameters.migration_source_sha >>
       - run:
           name: Generate LiteLLM master key
           command: |
@@ -1216,7 +1197,7 @@ jobs:
     resource_class: medium
     working_directory: ~/project
     steps:
-      - checkout_over_https
+      - checkout
       - skip_if_unrelated_changes
       - attach_workspace:
           at: ~/project
@@ -1310,21 +1291,8 @@ jobs:
                 name: Install MCP SDK1 peer
                 command: |
                   uv venv --python 3.12 .venv-mcp-peer
-                  uv pip install --no-config --exclude-newer "3 days" --python .venv-mcp-peer 'mcp==1.28.1' 'langchain-mcp-adapters==0.2.1'
+                  uv pip install --python .venv-mcp-peer 'mcp==1.28.1' 'langchain-mcp-adapters==0.2.1'
                   echo "export MCP_TEST_PEER_PYTHON=$PWD/.venv-mcp-peer/bin/python" >> "$BASH_ENV"
-      - when:
-          condition:
-            equal: [sdk, << parameters.suite >>]
-          steps:
-            - run:
-                name: Install redis-server 7.2.16
-                command: |
-                  mkdir -p /tmp/redis-bin
-                  cid="$(docker create redis:7.2.16@sha256:0637954999d01b7c9ce9167db2da50656e2590d3b884f1c600c5f63bb6e6773c)"
-                  docker cp "${cid}:/usr/local/bin/redis-server" /tmp/redis-bin/redis-server
-                  docker rm "$cid"
-                  /tmp/redis-bin/redis-server --version | grep -q 'v=7.2.16'
-                  echo 'export PATH="/tmp/redis-bin:$PATH"' >> "$BASH_ENV"
       - start_postgres:
           image: postgres:16@sha256:e17e86066e5ef83e0952a9347f5c792b7ece00972e2aa787a6986f471b3dd3d5
           server_args: "-c max_connections=300 -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all -c pg_stat_statements.max=20000"
@@ -1390,7 +1358,7 @@ jobs:
     environment:
       UV_PYTHON_DOWNLOADS: never
     steps:
-      - checkout_over_https
+      - checkout
       - install_uv
       - run:
           name: Install the locked shard verifier and coverage dependencies
@@ -1398,6 +1366,7 @@ jobs:
             uv export --frozen --all-groups --all-extras --no-emit-project --no-emit-workspace --no-header --no-annotate \
               | awk '/^[^ ]/ { keep = $0 ~ /^(annotated-types|coverage|pydantic|pydantic-core|typing-extensions|typing-inspection)==/ } keep' \
               > /tmp/fan-in-requirements.txt
+            test "$(grep -c '^[^ ]' /tmp/fan-in-requirements.txt)" -eq 6
             uv venv .venv --python 3.12
             uv pip install --no-config --python .venv --require-hashes -r /tmp/fan-in-requirements.txt
       - attach_workspace:
@@ -1438,7 +1407,7 @@ jobs:
             ROUTING_PARITY_BASE: << pipeline.parameters.routing_parity_base >>
           command: |
             [[ "$ROUTING_PARITY_BASE" =~ ^[0-9a-f]{40}$ ]] || exit 1
-            git fetch --depth 1 --no-filter origin "$ROUTING_PARITY_BASE"
+            git fetch --depth 1 origin "$ROUTING_PARITY_BASE"
             git rm -r -f --quiet litellm enterprise litellm-proxy-extras
             git checkout "$ROUTING_PARITY_BASE" -- litellm enterprise litellm-proxy-extras
             git reset --quiet
@@ -1513,7 +1482,7 @@ jobs:
     environment:
       DATABASE_URL: postgresql://postgres:postgres@localhost:5432/litellm_test
     steps:
-      - checkout_over_https
+      - checkout
       - skip_if_unrelated_changes
       - setup_litellm_test_deps:
           checkout: false

.circleci/scripts/classify_changes.sh

@@ -33,7 +33,7 @@ while IFS= read -r file || [ -n "$file" ]; do
     *) has_backend=true ;;
   esac
   case "$file" in
-    model_prices_and_context_window.json | litellm/model_prices_and_context_window_backup.json)
+    model_prices_and_context_window.json | litellm/model_prices_and_context_window_backup.json | model_prices_and_context_window.schema.json)
       has_cost_map=true ;;
     tests/unit/proxy/*) : ;;
     *) outside_cost_map_set=true ;;

.circleci/scripts/run_integration.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
+if [ "${GITHUB_ACTIONS:-}" = true ]; then
+  echo "Integration contracts are owned by CircleCI" >&2
+  exit 1
+fi
+
 suite="${1:?integration suite required}"
 mode="${2:-standard}"
 side="${3:-}"
-requested_workers="${INTEGRATION_WORKERS:-}"
-license_env=()
-if [ -n "${LITELLM_LICENSE:-}" ]; then
-  license_env=("LITELLM_LICENSE=$LITELLM_LICENSE")
-fi
 if [ "$mode" = replica ]; then
   results="test-results/integration-${suite}-replica"
 elif [ "$mode" = parity ]; then
@@ -84,7 +84,7 @@ if [ "$suite" = browser ]; then
   export LITELLM_UI_PATH="$PWD/ui/litellm-dashboard/out"
   test -f "$LITELLM_UI_PATH/index.html"
 fi
-export INTEGRATION_SEED="$(.venv/bin/python -c 'import hashlib,os; print(int(hashlib.sha256(((os.environ.get("CIRCLE_SHA1") or os.environ.get("GITHUB_SHA") or "local") + (os.environ.get("CIRCLE_WORKFLOW_ID") or os.environ.get("GITHUB_RUN_ID") or "local")).encode()).hexdigest()[:8],16))')"
+export INTEGRATION_SEED="$(.venv/bin/python -c 'import hashlib,os; print(int(hashlib.sha256((os.environ.get("CIRCLE_SHA1", "local") + os.environ.get("CIRCLE_WORKFLOW_ID", "local")).encode()).hexdigest()[:8],16))')"
 export INTEGRATION_ORDER_SEED="$INTEGRATION_SEED"
 
 uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma > "$results/prisma-generate.log" 2>&1
@@ -107,7 +107,7 @@ sudo iptables -N integration_only
 guard_created=true
 sudo iptables -A integration_only -o lo -j ACCEPT
 sudo iptables -A integration_only -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-for service in "${INTEGRATION_POSTGRES_CONTAINER:-postgres-db}" "${INTEGRATION_REDIS_CONTAINER:-redis-cache}"; do
+for service in postgres-db redis-cache; do
   address="$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$service")"
   sudo iptables -A integration_only -d "$address" -j ACCEPT
 done
@@ -128,8 +128,6 @@ fi
 sudo iptables -L integration_only -n -v -x > "$results/egress-guard.txt"
 awk '$3 == "REJECT" && $1 > 0 { rejected=1 } END { exit !rejected }' "$results/egress-guard.txt"
 
-export UV_OFFLINE=true
-
 if [ "$mode" = parity ]; then
   export INTEGRATION_WORKERS=1
 else
@@ -147,9 +145,6 @@ fi
 if [ "$suite" = mcp ]; then
   export INTEGRATION_WORKERS=2 INTEGRATION_COVERAGE=1
 fi
-if [ -n "$requested_workers" ]; then
-  export INTEGRATION_WORKERS="$requested_workers"
-fi
 isolated_workers=false
 if [ "${INTEGRATION_WORKERS:-1}" -gt 1 ]; then
   isolated_workers=true
@@ -190,7 +185,7 @@ start_proxy() {
     "${database_env[@]}" REDIS_HOST="$REDIS_HOST" REDIS_PORT="$REDIS_PORT" \
     INTEGRATION_UPSTREAM_URL="$INTEGRATION_UPSTREAM_URL" \
     LITELLM_MASTER_KEY="$LITELLM_MASTER_KEY" LITELLM_SALT_KEY="$LITELLM_SALT_KEY" LITELLM_UI_PATH="$LITELLM_UI_PATH" PROXY_BASE_URL="http://127.0.0.1:$port" \
-    "${license_env[@]}" \
+    LITELLM_LICENSE="${LITELLM_LICENSE:-}" \
     LITELLM_MODE=PRODUCTION STORE_MODEL_IN_DB=True LITELLM_ENABLE_MCP_STDIO=true "${cost_map_env[@]}" \
     AWS_EC2_METADATA_DISABLED=true DO_NOT_TRACK=1 COVERAGE_FILE="$coverage_data" \
     "${proxy_command[@]}" --config tests/integration/proxy_config.yaml \
@@ -262,7 +257,7 @@ env -i PATH="$PATH" HOME="$HOME" PYTHONPATH="$PYTHONPATH" \
   "${shared_services[@]}" "${worker_coverage[@]}" \
   INTEGRATION_WORKERS="${INTEGRATION_WORKERS:-1}" \
   LITELLM_MODE=PRODUCTION \
-  "${license_env[@]}" \
+  LITELLM_LICENSE="${LITELLM_LICENSE:-}" \
   INTEGRATION_SEED="$INTEGRATION_SEED" \
   INTEGRATION_ORDER_SEED="$INTEGRATION_ORDER_SEED" \
   INTEGRATION_CASE_SHARDS="${CIRCLE_NODE_TOTAL:-1}" INTEGRATION_CASE_SHARD_INDEX="${CIRCLE_NODE_INDEX:-0}" \

.dockerignore

@@ -7,6 +7,7 @@ tests
 .devcontainer
 *.tgz
 log.txt
+docker/Dockerfile.*
 
 # Claude Flow generated files (must be excluded from Docker build)
 .claude/

.githooks/pre-push

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+#
+# pre-push — enforce Conventional Branches
+#   https://conventional-branch.github.io/
+#
+# Branch format:  <type>/<description>
+#   <type> must be one of: feature, bugfix, hotfix, release, chore
+#
+# Protected branches (always allowed):
+#   - main
+#   - dependabot/*
+#   - gh-readonly-queue/*
+#
+# Tag pushes and branch deletions are skipped.
+# Bypass: git push --no-verify.
+
+set -eu
+
+ZERO_OID="0000000000000000000000000000000000000000"
+ZERO_OID_SHA256="0000000000000000000000000000000000000000000000000000000000000000"
+ALLOWED_TYPES="feature|bugfix|hotfix|release|chore"
+BRANCH_PATTERN="^(${ALLOWED_TYPES})/.+"
+
+PROTECTED_NAMES="main"
+PROTECTED_PREFIXES="dependabot/ gh-readonly-queue/"
+
+is_protected() {
+    branch="$1"
+    for name in $PROTECTED_NAMES; do
+        if [ "$branch" = "$name" ]; then
+            return 0
+        fi
+    done
+    for prefix in $PROTECTED_PREFIXES; do
+        case "$branch" in "$prefix"*) return 0 ;; esac
+    done
+    return 1
+}
+
+invalid=""
+
+while read -r local_ref local_oid remote_ref remote_oid; do
+    # Branch deletion (no local commit being pushed).
+    if [ "$local_oid" = "$ZERO_OID" ] || [ "$local_oid" = "$ZERO_OID_SHA256" ]; then
+        continue
+    fi
+
+    # Only validate branch pushes; ignore tags and other ref namespaces.
+    case "$remote_ref" in
+        refs/heads/*) ;;
+        *) continue ;;
+    esac
+
+    branch="${remote_ref#refs/heads/}"
+
+    if is_protected "$branch"; then
+        continue
+    fi
+
+    if ! printf '%s' "$branch" | grep -Eq "$BRANCH_PATTERN"; then
+        invalid="$invalid $branch"
+    fi
+done
+
+if [ -n "$invalid" ]; then
+    cat >&2 <<EOF
+✗ Branch name does not follow Conventional Branches.
+
+  Invalid:$invalid
+
+  Expected: <type>/<description>
+
+  Allowed types: feature, bugfix, hotfix, release, chore
+  Examples:
+    feature/weighted-round-robin
+    bugfix/streaming-empty-chunks
+    chore/bump-deps
+    hotfix/auth-bypass
+
+  Protected (always allowed): main, dependabot/*, gh-readonly-queue/*.
+
+See https://conventional-branch.github.io/
+
+Rename with: git branch -m <new-name>
+To bypass (use sparingly): git push --no-verify
+EOF
+    exit 1
+fi
+
+exit 0

.github/CODEOWNERS

@@ -1,6 +1,2 @@
-/model_prices_and_context_window.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
+/model_prices_and_context_window.json @mateo-berri @ryan-crabbe-berri @kerry-berri
-/litellm/model_prices_and_context_window_backup.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
+/litellm/model_prices_and_context_window_backup.json @mateo-berri @ryan-crabbe-berri @kerry-berri
-/ci_cd/cost_map_pins.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
-/helm/ @yassin-berriai
-/terraform/ @yassin-berriai
-/.github/workflows/zizmor.yml @yuneng-berri

.github/PULL_REQUEST_TEMPLATE/general.md

@@ -4,18 +4,6 @@
      Drop every section you have nothing to put in, heading included: a bare "## Relevant issues" or
      "## Affected release" with nothing under it must not appear in the final description -->
 
-## Handwritten 1-3 sentence description
-
-<!-- The author writes this in their own words, and it can be left empty. AI agents never write here and leave the
-     placeholder line below as is, and when updating an existing PR they keep whatever the author wrote. Keep this heading
-     Say what the PR does for users and why, then anything that keeps it working over time (a sync job, a config someone owns)
-     No bullets, tables, or code names here. Everything else goes inside the Details block below -->
-
-[insert handwritten description here]
-
-<details>
-<summary>Details</summary>
-
 ## TLDR
 
 <!-- Fill in the bullets below and keep each one short and concrete: one line per bullet, roughly 10 words max
@@ -89,11 +77,9 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
 ## Screenshots / Proof of Fix
 
 <!-- Include screenshots, screen recordings, or command (e.g., curl) + output demonstrating that your changes work as expected
-     Simple PR exception (2026-10-28, #49030). A mechanical change fully proven by focused checks may skip before/after e2e. Follow AGENTS.md's eligibility rule, trace callers, and verify bug-fix regression sensitivity. Record E2E exception at <tip SHA>, scope, rationale, commands and results here. Tests/static checks suffice only for this exception. Keep CI and bot gates, reassess changed diffs, and honor explicit e2e requests. Omit the Before/After scaffold for an exception
+     The proof must be completely e2e with no mocks, using actual LLM calls costing real $$$ if applicable. `pytest` commands are not enough
-     Otherwise proof must be fully e2e, no mocks, with real provider calls where applicable. Pytest alone is insufficient. The following format applies to live proof
+     Show ONLY the latest run: capture Before at the merge base and After at the PR's current tip, and when new commits change behavior, replace this whole section with the fresh run instead of stacking it on top of older ones. The run must be up to date. As soon as a new commit is made and it makes this PR description's after sha stale (it's no longer tip of PR), you must re-run the QA
-     Show only the latest proof. Before uses the merge base and After the tested tip. Rerun affected QA after behavior changes. Verified neutral diffs may carry proof with the tested SHA, current tip, equivalence reason and focused checks, following AGENTS.md and /qa
+     Structure the section exactly as below: Before and After one heading level below this section, each naming the commit hash it was captured at, one lower-level heading per case inside each, the same case names in the same order on both sides, and numbered steps (command, observed output) under every case, never loose prose; shared setup (config, payloads) goes above Before, and with a single case, drop the case headings and number the steps directly
-     Use matching Before/After case headings with numbered commands and observed output. Put shared setup above them. For one case, omit case headings
-     A reviewer must be able to rerun every case from the body alone, so the shared setup gives the proxy start command, the config.yaml, env var names (never values) and the client code or curl used, all exactly as run. Wrap any block longer than about 20 lines in a collapsed <details> with a one-line summary
 
 ### Before (<hash>)
 
@@ -178,5 +164,3 @@ Example checklists:
 ## Final Attestation
 
 - [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
-
-</details>

.github/ci-coverage-allowlist.yml

@@ -5,7 +5,7 @@ description: >-
 
 test_paths:
   - reason: >-
-      litellm.agent() end-to-end suite. It drives the real claude, codex, opencode and pi CLIs and
+      litellm.agent() end-to-end suite. It drives the real claude, codex and opencode CLIs and
       deepagents against a live LiteLLM AI Gateway, so it needs those binaries on PATH plus
       LITELLM_PROXY_API_BASE / LITELLM_PROXY_API_KEY, and skips without them. Run manually
       before changing litellm/harness; the mocked coverage runs in tests/unit/harness and
@@ -57,6 +57,15 @@ test_paths:
       - tests/integration/test_oci_proxy_integration.py
 
 dockerfiles:
+  - reason: >-
+      The dashboard container is a static Next.js export served by nginx, and the dashboard build
+      and lint workflows already exercise that output, so building the image adds no signal about it
+    paths:
+      - ui/Dockerfile
+  - reason: >-
+      An example image under cookbook/ that is documentation rather than a shipped artifact
+    paths:
+      - cookbook/litellm-ollama-docker-image/Dockerfile
   - reason: >-
       The Rust gateway image compiles the whole workspace in release mode, which is too slow for
       a per-pull-request job while the gateway binary is still being assembled; the Rust lint,

.github/e2e-stack/assert_tests_ran.py

@@ -0,0 +1,69 @@
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+from pathlib import Path
+from typing import Final
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "tests/e2e"))
+from coverage_registry.management_cases import MANAGEMENT_CASES
+
+
+def main() -> int:
+    selected: Final = tuple(sys.argv[2:])
+    try:
+        report: Final = ET.parse(Path(sys.argv[1])).getroot()
+    except (ET.ParseError, OSError):
+        _ = sys.stdout.write("::error::could not read the test execution report\n")
+        return 1
+    cases: Final = tuple(report.iter("testcase"))
+    expected_count: Final = os.environ.get("E2E_REQUIRED_TEST_COUNT")
+    passed: Final = frozenset(
+        case.get("file") for case in cases if all(case.find(tag) is None for tag in ("skipped", "failure", "error"))
+    )
+    missing: Final = tuple(path for path in selected if path not in passed)
+    required_nodes: Final = frozenset(case.node for case in MANAGEMENT_CASES if case.node.split("::", 1)[0] in selected)
+    passed_nodes: Final = frozenset(
+        prop.get("value")
+        for case in cases
+        if all(case.find(tag) is None for tag in ("skipped", "failure", "error"))
+        for prop in case.findall("./properties/property")
+        if prop.get("name") == "management_node"
+    )
+    missing_nodes: Final = required_nodes - passed_nodes
+    for node in sorted(missing_nodes):
+        _ = sys.stdout.write(f"::error::required management case did not pass: {node}\n")
+    for path in selected:
+        collected: Final = sum(case.get("file") == path for case in cases)
+        skipped: Final = sum(case.get("file") == path and case.find("skipped") is not None for case in cases)
+        _ = sys.stdout.write(f"{path}: {collected} collected, {skipped} skipped\n")
+        for case in cases:
+            if case.get("file") != path or all(case.find(tag) is None for tag in ("failure", "error", "skipped")):
+                continue
+            outcome = "skipped" if case.find("skipped") is not None else "failed"
+            _ = sys.stdout.write(f"  {outcome}: {case.get('classname', '')}::{case.get('name', '')}\n")
+            for prop in case.findall("./properties/property"):
+                name = prop.get("name", "")
+                value = prop.get("value", "")
+                if name in ("oauth_failure_phase", "oauth_exception_type", "oauth_frame") and re.fullmatch(
+                    r"[A-Za-z0-9_.:<>-]{1,240}", value
+                ):
+                    _ = sys.stdout.write(f"    {name}: {value}\n")
+    if expected_count is not None and (
+        len(cases) != int(expected_count) or any(case.find("skipped") is not None for case in cases)
+    ):
+        _ = sys.stdout.write("::error::required test count was not met or a required case was skipped\n")
+        return 1
+    if (
+        selected
+        and not missing
+        and not missing_nodes
+        and not any(case.find(tag) is not None for case in cases for tag in ("failure", "error"))
+    ):
+        return 0
+    _ = sys.stdout.write("::error::every selected file must execute a passing test, with no failures or errors\n")
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/scripts/assert_ci_coverage.py

@@ -32,7 +32,6 @@ IGNORE_ARG_RE: Final = re.compile(r"--ignore(?:-glob)?[= ](\S+)")
 DOCKERFILE_TOKEN_RE = re.compile(r"[A-Za-z0-9_./-]*Dockerfile[A-Za-z0-9_.-]*")
 COMMENT_RE = re.compile(r"^\s*#.*$", re.MULTILINE)
 GLOB_CHARS = frozenset("*?")
-DEDICATED_INTEGRATION_WORKFLOWS: Final = MappingProxyType({"image-scan.yml": "tests/integration/database/image/"})
 
 # Trees whose jobs are sharded with no catch-all bucket, so every child that holds
 # tests has to be named by some shard or it runs nowhere. A child listed here is
@@ -561,17 +560,6 @@ def _integration_github_files(runner: pathlib.Path) -> frozenset[str]:
     return frozenset(ast.literal_eval(values))
 
 
-def _owns(workflow: str, relative_path: str) -> bool:
-    owned_root: Final = DEDICATED_INTEGRATION_WORKFLOWS.get(workflow)
-    return owned_root is not None and relative_path.startswith(owned_root)
-
-
-def _tokens_outside_owner(relative_path: str, workflow_tokens: Mapping[str, frozenset[str]]) -> frozenset[str]:
-    return frozenset[str]().union(
-        *(tokens for workflow, tokens in workflow_tokens.items() if not _owns(workflow, relative_path))
-    )
-
-
 def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozenset[str], tuple[Finding, ...]]:
     runner: Final = repo_root / "tests/integration/run.py"
     if not runner.exists():
@@ -623,20 +611,15 @@ def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozens
         )
         != 1
     )
-    workflow_tokens: Final = MappingProxyType(
+    gha_tokens: Final = _invoked_test_tokens(
-        {
+        scalar
-            path.name: _invoked_test_tokens(_scalars(yaml.safe_load(path.read_text()), path.name))
+        for path in (repo_root / ".github/workflows").glob("*.y*ml")
-            for path in (repo_root / ".github/workflows").glob("*.y*ml")
+        for scalar in _scalars(yaml.safe_load(path.read_text()), path.name)
-        }
-    )
-    gha_tokens: Final = frozenset[str]().union(*workflow_tokens.values())
-    circle_owned: Final = (
-        frozenset(str(path.relative_to(repo_root)) for path in integration_root.rglob("test_*.py")) - github_files
     )
     findings: Final = (
         tuple(
-            Finding(path, "integration file outside GITHUB_FILES is selected by GitHub Actions")
+            Finding(path, "integration contract is also selected by GitHub Actions")
-            for path in sorted(circle_owned)
+            for path in paths
             if any(_token_covers(token, path) for token in gha_tokens)
         )
         + tuple(
@@ -649,11 +632,6 @@ def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozens
             for path in sorted(github_files)
             if not (repo_root / path).is_file()
         )
-        + tuple(
-            Finding(path, "GitHub-owned integration contract runs outside a dedicated integration workflow")
-            for path in sorted(github_files)
-            if any(_token_covers(token, path) for token in _tokens_outside_owner(path, workflow_tokens))
-        )
     )
     browser_commands: Final = tuple(
         scalar.value
@@ -710,9 +688,13 @@ def main() -> int:
     scalars = _all_scalars()
 
     integration_paths, ownership_findings = _integration_ownership()
-    test_findings = _uncovered_tests(
+    test_findings = (
-        allowlist,
+        _uncovered_tests(
-        _invoked_selections(scalars) + (Selection(included=integration_paths, ignored=frozenset()),),
+            allowlist,
+            _invoked_selections(scalars)
+            + (Selection(included=integration_paths, ignored=frozenset()),),
+        )
+        + ownership_findings
     )
     dockerfile_findings = _uncovered_dockerfiles(allowlist, _built_dockerfile_tokens(scalars))
     stale_findings = _stale_allowlist_paths(allowlist, test_files=_test_files(), dockerfiles=_dockerfiles())
@@ -729,25 +711,13 @@ def main() -> int:
             test_findings,
             "Add each to a job's test path, or list it in .github/ci-coverage-allowlist.yml with a reason.",
         )
-    if ownership_findings:
-        browser_remedy = (
-            " CircleCI's browser group owns the integrationCritical Playwright contracts, so no GitHub workflow may "
-            "select them and the default Playwright config must exclude them."
-            if any("browser" in finding.detail or "Playwright" in finding.detail for finding in ownership_findings)
-            else ""
-        )
-        _report(
-            "integration contracts with the wrong CI owner",
-            ownership_findings,
-            "Follow tests/integration/AGENTS.md, which says which CI job owns each integration file." + browser_remedy,
-        )
     if dockerfile_findings:
         _report(
             "Dockerfiles that no CI job builds",
             dockerfile_findings,
             "Build each in a workflow, or list it in .github/ci-coverage-allowlist.yml with a reason.",
         )
-    if stale_findings or test_findings or ownership_findings or dockerfile_findings:
+    if stale_findings or test_findings or dockerfile_findings:
         return 1
 
     _write(

.github/scripts/assert_secret_absent.py

@@ -1,119 +0,0 @@
-import base64
-import io
-import os
-import sys
-import urllib.parse
-import zipfile
-from collections.abc import Callable, Iterator, Mapping
-from dataclasses import dataclass
-from functools import reduce
-from itertools import chain
-from pathlib import Path
-from types import MappingProxyType
-from typing import Final, TextIO
-
-SECRET_VARIABLE: Final = "SCANNED_SECRET"
-MIN_ENCODED_LENGTH: Final = 12
-MAX_ARCHIVE_DEPTH: Final = 3
-MASK: Final = "***"
-
-Forms = tuple[tuple[str, str], ...]
-
-
-@dataclass(frozen=True, slots=True)
-class Hit:
-    location: str
-    form: str
-
-
-def _aligned_encodings(raw: bytes, name: str, encode: Callable[[bytes], bytes]) -> Iterator[tuple[str, str]]:
-    for offset in range(3):
-        chunk: Final = raw[offset:]
-        yield f"{name} offset {offset}", encode(chunk[: len(chunk) - len(chunk) % 3]).decode()
-
-
-def secret_forms(secret: str) -> Forms:
-    encoded: Final = (
-        ("url-encoded", urllib.parse.quote(secret, safe="")),
-        *_aligned_encodings(secret.encode(), "base64", base64.b64encode),
-        *_aligned_encodings(secret.encode(), "base64url", base64.urlsafe_b64encode),
-    )
-    by_value: Final = MappingProxyType(
-        {value: name for name, value in encoded if value != secret and len(value) >= MIN_ENCODED_LENGTH}
-    )
-    return (("raw", secret), *((name, value) for value, name in by_value.items()))
-
-
-def mask(text: str, forms: Forms) -> str:
-    return reduce(lambda masked, form: masked.replace(form[1], MASK), forms, text)
-
-
-def _name_hits(location: str, name: str, forms: Forms) -> Iterator[Hit]:
-    for form, value in forms:
-        if value in name:
-            yield Hit(mask(location, forms), f"{form} in name")
-
-
-def _content_hits(location: str, data: bytes, forms: Forms, depth: int) -> Iterator[Hit]:
-    for form, value in forms:
-        if value.encode() in data:
-            yield Hit(mask(location, forms), form)
-    if depth >= MAX_ARCHIVE_DEPTH or not zipfile.is_zipfile(io.BytesIO(data)):
-        return
-    with zipfile.ZipFile(io.BytesIO(data)) as archive:
-        for member in archive.infolist():
-            if member.is_dir():
-                continue
-            yield from _content_hits(f"{location}!{member.filename}", archive.read(member), forms, depth + 1)
-
-
-def _walk(root: Path) -> Iterator[tuple[Path, bool]]:
-    for directory, subdirectories, files in os.walk(root):
-        yield from ((Path(directory, name), True) for name in subdirectories if Path(directory, name).is_symlink())
-        yield from ((Path(directory, name), False) for name in files)
-
-
-def _path_hits(root: Path, path: Path, linked_directory: bool, forms: Forms) -> Iterator[Hit]:
-    location: Final = str(path.relative_to(root))
-    if linked_directory:
-        yield Hit(mask(location, forms), "directory symlink, not scanned")
-        return
-    yield from _name_hits(location, location, forms)
-    yield from _content_hits(location, path.read_bytes(), forms, 0)
-
-
-def scan(root: Path, secret: str) -> tuple[int, tuple[Hit, ...]]:
-    forms: Final = secret_forms(secret)
-    entries: Final = tuple(_walk(root))
-    hits: Final = tuple(chain.from_iterable(_path_hits(root, path, linked, forms) for path, linked in entries))
-    return len(entries), hits
-
-
-def main(argv: tuple[str, ...], environ: Mapping[str, str], out: TextIO) -> int:
-    if len(argv) != 1:
-        print(f"usage: assert_secret_absent.py <directory>, with the secret in ${SECRET_VARIABLE}", file=out)
-        return 2
-    secret: Final = environ.get(SECRET_VARIABLE, "")
-    root: Final = Path(argv[0])
-    if not secret:
-        print(f"{SECRET_VARIABLE} is empty, nothing to scan for", file=out)
-        return 0
-    if not root.is_dir():
-        print(f"{root} is not a directory, nothing to scan", file=out)
-        return 0
-    try:
-        scanned, hits = scan(root, secret)
-    except Exception as error:
-        print(f"scan failed with {type(error).__name__}, treating the results as unsafe", file=out)
-        return 1
-    for hit in hits:
-        print(f"secret found: {hit.location} ({hit.form})", file=out)
-    if hits:
-        print(f"{len(hits)} copies of the secret under {root}, refusing to upload", file=out)
-        return 1
-    print(f"no copy of the secret in {scanned} files under {root}", file=out)
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main(tuple(sys.argv[1:]), os.environ, sys.stdout))

.github/scripts/auto_update_price_and_context_window_file.py

@@ -0,0 +1,319 @@
+import asyncio
+import aiohttp
+import json
+import math
+from typing import Any
+
+# Asynchronously fetch data from a given URL
+async def fetch_data(url):
+    try:
+        # Create an asynchronous session
+        async with aiohttp.ClientSession() as session:
+            # Send a GET request to the URL
+            async with session.get(url) as resp:
+                # Raise an error if the response status is not OK
+                resp.raise_for_status()
+                # Parse the response JSON
+                resp_json = await resp.json()
+                print("Fetch the data from URL.")
+                # Return the 'data' field from the JSON response
+                return resp_json['data']
+    except Exception as e:
+        # Print an error message if fetching data fails
+        print("Error fetching data from URL:", e)
+        return None
+
+
+FRIENDLI_API_URL = "https://api.friendli.ai/serverless/v1/models"
+FRIENDLI_PROVIDER = "friendliai"
+
+INHERITABLE_BASE_KEYS = (
+    "supports_pdf_input",
+    "supports_assistant_prefill",
+    "supports_adaptive_thinking",
+    "supports_output_config",
+)
+
+REASONING_EFFORT_LEVEL_ORDER = ("none", "minimal", "low", "medium", "high", "xhigh", "max")
+
+
+def _find_base_model_entry(base_model: str, local_data: dict) -> str | None:
+    if not base_model:
+        return None
+    bm_tail = base_model.split("/")[-1].lower()
+    if base_model in local_data:
+        return base_model
+    for key in local_data:
+        if key.startswith("sample_spec") or key == "fallback_generalizations":
+            continue
+        if key.split("/")[-1].lower() == bm_tail:
+            return key
+    return None
+
+
+def _reasoning_effort_levels(reasoning_options: list) -> list:
+    offered = {
+        val
+        for opt in reasoning_options or []
+        if opt.get("type") == "effort"
+        for val in opt.get("values", [])
+    }
+    return [level for level in REASONING_EFFORT_LEVEL_ORDER if level in offered]
+
+
+def _valid_token_price(value: object) -> bool:
+    try:
+        price = float(value)  # pyright: ignore[reportArgumentType]  # non-numeric values are rejected via the except
+    except (TypeError, ValueError):
+        return False
+    return math.isfinite(price) and price >= 0
+
+
+def _has_valid_token_prices(pricing: dict | None) -> bool:
+    prices = pricing or {}
+    return _valid_token_price(prices.get("input")) and _valid_token_price(prices.get("output"))
+
+
+def _pricing(pricing: dict) -> dict:
+    out: dict[str, Any] = {}
+    if not pricing:
+        return out
+    if "input" in pricing:
+        out["input_cost_per_token"] = float(pricing["input"])
+    if "output" in pricing:
+        out["output_cost_per_token"] = float(pricing["output"])
+    if "input_cache_read" in pricing and pricing["input_cache_read"] is not None:
+        out["cache_read_input_token_cost"] = float(pricing["input_cache_read"])
+    return out
+
+
+def _modality_flags(input_mods: list) -> dict:
+    mods = input_mods or []
+    has_image = "image" in mods
+    return {
+        "supports_vision": has_image,
+        "supports_image_input": has_image,
+        "supports_video_input": "video" in mods,
+    }
+
+
+def transform_friendli_data(data: list, local_data: dict) -> dict:
+    transformed: dict[str, dict] = {}
+    if not data:
+        return transformed
+    for model in data:
+        # An unpriced row must never wholesale-replace an already priced local entry:
+        # missing prices cost-calculate as zero, silently zeroing tracked spend
+        if not _has_valid_token_prices(model.get("pricing")):
+            continue
+        model_id = model["id"]
+        base_model = model.get("base_model") or ""
+        entry: dict[str, Any] = {
+            "litellm_provider": FRIENDLI_PROVIDER,
+        }
+
+        base_key = _find_base_model_entry(base_model, local_data)
+        if base_key:
+            base_entry = local_data[base_key]
+            for k in INHERITABLE_BASE_KEYS:
+                if k in base_entry:
+                    entry[k] = base_entry[k]
+
+        ctx = model.get("context_length")
+        if ctx is not None:
+            entry["max_input_tokens"] = int(ctx)
+        max_out = model.get("max_completion_tokens")
+        if max_out is not None:
+            entry["max_output_tokens"] = int(max_out)
+            entry["max_tokens"] = int(max_out)
+
+        pricing = _pricing(model.get("pricing", {}))
+        entry.update(pricing)
+        entry["supports_prompt_caching"] = "cache_read_input_token_cost" in pricing
+
+        reasoning = model.get("reasoning") is True
+        entry["supports_reasoning"] = reasoning
+        if reasoning:
+            entry["reasoning_effort_levels"] = _reasoning_effort_levels(
+                model.get("reasoning_options", [])
+            )
+
+        func = model.get("functionality", {})
+        entry["supports_function_calling"] = func.get("tool_call") is True
+        entry["supports_parallel_function_calling"] = func.get("parallel_tool_call") is True
+        is_struct = func.get("structured_output") is True
+        entry["supports_response_schema"] = is_struct
+        entry["supports_native_structured_output"] = is_struct
+        entry["supports_system_messages"] = func.get("system_messages") is True
+        entry["supports_tool_choice"] = func.get("tool_choice") is True
+
+        entry.update(_modality_flags(model.get("input_modalities", [])))
+
+        entry["mode"] = model.get("mode", "chat")
+
+        desc = model.get("description")
+        if desc:
+            entry["comment"] = desc
+
+        dep = model.get("deprecation_date")
+        if dep:
+            entry["deprecation_date"] = dep.split("T")[0]
+
+        entry["source"] = FRIENDLI_API_URL
+
+        transformed[f"{FRIENDLI_PROVIDER}/{model_id}"] = entry
+    return transformed
+
+# Synchronize local data with remote data
+def sync_local_data_with_remote(local_data, remote_data, replace_keys=frozenset()):
+    # Update existing keys in local_data with values from remote_data
+    # (replace_keys entries are swapped wholesale so a field the remote catalog
+    #  dropped, e.g. cache pricing, cannot survive as a stale value)
+    for key in (set(local_data) & set(remote_data)):
+        if key in replace_keys:
+            local_data[key] = remote_data[key]
+        else:
+            local_data[key].update(remote_data[key])
+
+    # Add new keys from remote_data to local_data
+    for key in (set(remote_data) - set(local_data)):
+        local_data[key] = remote_data[key]
+
+# Write data to the json file
+def write_to_file(file_path, data):
+    try:
+        # Open the file in write mode
+        with open(file_path, "w") as file:
+            # Dump the data as JSON into the file
+            json.dump(data, file, indent=4)
+        print("Values updated successfully.")
+    except Exception as e:
+        # Print an error message if writing to file fails
+        print("Error updating JSON file:", e)
+
+# Update the existing models and add the missing models for OpenRouter
+def transform_openrouter_data(data):
+    transformed = {}
+    if not data:
+        return transformed
+    for row in data:
+        # Add the fields 'max_tokens' and 'input_cost_per_token'
+        obj = {
+            "max_tokens": row["context_length"],
+            "input_cost_per_token": float(row["pricing"]["prompt"]),
+        }
+
+        # Add 'max_output_tokens' as a field if it is not None
+        if "top_provider" in row and "max_completion_tokens" in row["top_provider"] and row["top_provider"]["max_completion_tokens"] is not None:
+            obj['max_output_tokens'] = int(row["top_provider"]["max_completion_tokens"])
+
+        # Add the field 'output_cost_per_token'
+        obj.update({
+            "output_cost_per_token": float(row["pricing"]["completion"]),
+        })
+
+        # Add field 'input_cost_per_image' if it exists and is non-zero
+        if "pricing" in row and "image" in row["pricing"] and float(row["pricing"]["image"]) != 0.0:
+            obj['input_cost_per_image'] = float(row["pricing"]["image"])
+
+        # Add the fields 'litellm_provider' and 'mode'
+        obj.update({
+            "litellm_provider": "openrouter",
+            "mode": "chat"
+        })
+
+        # Add the 'supports_vision' field if the modality is 'multimodal'
+        if row.get('architecture', {}).get('modality') == 'multimodal':
+            obj['supports_vision'] = True
+
+        # Use a composite key to store the transformed object
+        transformed[f'openrouter/{row["id"]}'] = obj
+
+    return transformed
+
+# Update the existing models and add the missing models for Vercel AI Gateway
+def transform_vercel_ai_gateway_data(data):
+    transformed = {}
+    if not data:
+        return transformed
+    for row in data:
+        # Rows without token pricing or token limits (video/embedding models) previously KeyError'd the whole sync
+        if any(row.get(k) is None for k in ("context_window", "max_tokens")) or any(
+            row.get("pricing", {}).get(k) is None for k in ("input", "output")
+        ):
+            continue
+        obj = {
+            "max_tokens": row["context_window"],
+            "input_cost_per_token": float(row["pricing"]["input"]),
+            "output_cost_per_token": float(row["pricing"]["output"]),
+            'max_output_tokens': row['max_tokens'],
+            'max_input_tokens': row["context_window"],
+        }
+
+        # Handle cache pricing if available
+        if "pricing" in row:
+            if "input_cache_read" in row["pricing"] and row["pricing"]["input_cache_read"] is not None:
+                obj['cache_read_input_token_cost'] = float(f"{float(row['pricing']['input_cache_read']):e}")
+            
+            if "input_cache_write" in row["pricing"] and row["pricing"]["input_cache_write"] is not None:
+                obj['cache_creation_input_token_cost'] = float(f"{float(row['pricing']['input_cache_write']):e}")
+
+        mode = "embedding" if "embedding" in row["id"].lower() else "chat"
+        
+        obj.update({"litellm_provider": "vercel_ai_gateway", "mode": mode})
+
+        transformed[f'vercel_ai_gateway/{row["id"]}'] = obj
+
+    return transformed
+
+
+# Load local data from a specified file
+def load_local_data(file_path):
+    try:
+        # Open the file in read mode
+        with open(file_path, "r") as file:
+            # Load and return the JSON data
+            return json.load(file)
+    except FileNotFoundError:
+        # Print an error message if the file is not found
+        print("File not found:", file_path)
+        return None
+    except json.JSONDecodeError as e:
+        # Print an error message if JSON decoding fails
+        print("Error decoding JSON:", e)
+        return None
+
+def main():
+    local_file_path = "model_prices_and_context_window.json"  # Path to the local data file
+    openrouter_url = "https://openrouter.ai/api/v1/models"  # URL to fetch OpenRouter data
+    vercel_ai_gateway_url = "https://ai-gateway.vercel.sh/v1/models"  # URL to fetch Vercel AI Gateway data
+
+    # Load local data from file
+    local_data = load_local_data(local_file_path)
+    
+    # Fetch OpenRouter data
+    openrouter_data = asyncio.run(fetch_data(openrouter_url))
+    # Transform the fetched OpenRouter data
+    openrouter_data = transform_openrouter_data(openrouter_data)
+    
+    # Fetch Vercel AI Gateway data
+    vercel_data = asyncio.run(fetch_data(vercel_ai_gateway_url))
+    # Transform the fetched Vercel AI Gateway data
+    vercel_data = transform_vercel_ai_gateway_data(vercel_data)
+
+    friendli_data = asyncio.run(fetch_data(FRIENDLI_API_URL))
+    friendli_data = transform_friendli_data(friendli_data, local_data)
+    
+    # Combine both datasets
+    all_remote_data = {**openrouter_data, **vercel_data, **friendli_data}
+
+    # If both local and openrouter data are available, synchronize and save
+    if local_data and all_remote_data:
+        sync_local_data_with_remote(local_data, all_remote_data, replace_keys=frozenset(friendli_data))
+        write_to_file(local_file_path, local_data)
+    else:
+        print("Failed to fetch model data from either local file or URL.")
+
+# Entry point of the script
+if __name__ == "__main__":
+    main()

.github/scripts/smoke_test_native_wheel.py

@@ -7,8 +7,6 @@ import zipfile
 from pathlib import Path
 from typing import Final
 
-COST_MAP: Final = Path(__file__).resolve().parents[2] / "model_prices_and_context_window.json"
-
 CHILD_SCRIPT: Final = """
 from importlib.util import module_from_spec, spec_from_file_location
 from pathlib import Path
@@ -36,9 +34,6 @@ else:
 after = module.gil_stats()
 if not isinstance(after.get("releases"), int):
     raise AssertionError(f"native module unusable after panic: {after!r}")
-
-if module.bundled_model_cost_map() != Path(sys.argv[2]).read_bytes():
-    raise AssertionError("the wheel embeds a different model_prices_and_context_window.json than this checkout")
 """
 
 
@@ -60,9 +55,7 @@ def main() -> int:
 
         native_path: Final = Path(temporary_directory) / Path(native_members[0].filename).name
         native_path.write_bytes(archive.read(native_members[0]))
-        result: Final = subprocess.run(
+        result: Final = subprocess.run((sys.executable, "-c", CHILD_SCRIPT, str(native_path)), check=False)
-            (sys.executable, "-c", CHILD_SCRIPT, str(native_path), str(COST_MAP)), check=False
-        )
 
     if result.returncode != 0:
         sys.stderr.write(f"native wheel smoke test exited with status {result.returncode}\n")

.github/scripts/verify_linux_native_wheel.py

@@ -214,7 +214,7 @@ def main(
     native_module: Final = load_native_module(native_path)
     native_module_loads: Final = native_module is not None
     panic_test_hook_absent: Final = native_module is not None and not hasattr(native_module, "_panic_for_test")
-    native_size_limit: Final = 55_000_000
+    native_size_limit: Final = 48_000_000
     native_size_within_limit: Final = native_member.file_size <= native_size_limit
     validations: Final = (
         (f"Python tag is {EXPECTED_PYTHON_TAG}", python_tag == EXPECTED_PYTHON_TAG),

.github/workflows/auto_update_price_and_context_window.yml

@@ -0,0 +1,39 @@
+name: Updates model_prices_and_context_window.json and Create Pull Request
+
+on:
+  schedule:
+    - cron: "0 0 * * 0" # Run every Sundays at midnight
+    #- cron: "0 0 * * *" # Run daily at midnight
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  auto_update_price_and_context_window:
+    if: github.repository == 'BerriAI/litellm'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+      - name: Set up uv
+        uses: ./.github/actions/setup-uv-with-retries
+        with:
+          version: "0.10.9"
+      - name: Update JSON Data
+        run: |
+          uv run --frozen --with 'aiohttp==3.13.3' python ".github/scripts/auto_update_price_and_context_window_file.py"
+      - name: Regenerate JSON Schema
+        run: |
+          uv run --frozen python ci_cd/generate_model_prices_schema.py
+      - name: Create Pull Request
+        run: |
+          git add model_prices_and_context_window.json model_prices_and_context_window.schema.json
+          git commit -m "Update model_prices_and_context_window.json file: $(date +'%Y-%m-%d')" 
+          gh pr create --title "Update model_prices_and_context_window.json file" \
+            --body "Automated update for model_prices_and_context_window.json" \
+            --head auto-update-price-and-context-window-$(date +'%Y-%m-%d') \
+            --base main
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN }}

.github/workflows/compat-matrix-image.yml

@@ -4,7 +4,7 @@ on:
   pull_request:
     paths:
       - tests/e2e/claude_code/cron_vm/**
-      - tests/e2e/cli_harness/npm_version_resolver.py
+      - tests/e2e/claude_code/pr_gate_version_resolver.py
       - .github/workflows/compat-matrix-image.yml
   workflow_dispatch:
 
@@ -26,24 +26,17 @@ jobs:
         with:
           persist-credentials: false
 
-      - name: Install the pinned uv the cron image ships
+      - name: Build the Render cron image
-        env:
+        run: docker build -f tests/e2e/claude_code/cron_vm/Dockerfile -t compat-matrix:${{ github.sha }} tests/e2e
-          UV_VERSION: 0.10.9
-          UV_SHA256: 20d79708222611fa540b5c9ed84f352bcd3937740e51aacc0f8b15b271c57594
-        run: |
-          set -euo pipefail
-          curl -fsSLo "$RUNNER_TEMP/uv.tar.gz" "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-x86_64-unknown-linux-gnu.tar.gz"
-          echo "${UV_SHA256}  $RUNNER_TEMP/uv.tar.gz" | sha256sum -c -
-          mkdir -p "$RUNNER_TEMP/bin"
-          tar -xzf "$RUNNER_TEMP/uv.tar.gz" -C "$RUNNER_TEMP/bin" --strip-components=1 uv-x86_64-unknown-linux-gnu/uv
-          echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
 
-      - name: Resolve and install the Claude Code CLI as an unprivileged user
+      - name: Resolve and install the Claude Code CLI as the cron user
         run: |
-          set -euo pipefail
+          docker run --rm compat-matrix:${{ github.sha }} bash -c '
-          whoami
+            set -euo pipefail
-          gh --version
+            whoami
-          uv --version
+            gh --version
-          version="$(uv run --no-project --python 3.12 python tests/e2e/cli_harness/npm_version_resolver.py @anthropic-ai/claude-code)"
+            uv --version
-          tests/e2e/claude_code/cron_vm/install_claude_code.sh "${version}" "$RUNNER_TEMP/claude-cli"
+            version="$(uv run --no-project --python 3.12 python /opt/litellm/tests/e2e/claude_code/pr_gate_version_resolver.py)"
-          "$RUNNER_TEMP/claude-cli/claude" --version
+            /opt/litellm/tests/e2e/claude_code/cron_vm/install_claude_code.sh "${version}" /tmp/claude-cli
+            /tmp/claude-cli/claude --version
+          '

.github/workflows/cost-map-guard.yml

@@ -1,14 +1,13 @@
 name: Cost map guard
 
-on:
+on: # zizmor: ignore[dangerous-triggers] runs the base branch's code only; the PR's cost map files are read as data and never executed
-  pull_request:
+  pull_request_target:
     branches:
       - main
       - "litellm_**"
 
 permissions:
   contents: read
-  pull-requests: read
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
@@ -20,23 +19,17 @@ jobs:
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
-          ref: ${{ github.event.pull_request.base.sha }}
           persist-credentials: false
-      - name: Check out the base branch tip and fetch the pull request head and its merge base
+      - name: Fetch the pull request head and its merge base
         id: revisions
         env:
           GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
-          base_ref="$(gh api "repos/${GITHUB_REPOSITORY}/branches/${EVENT_BASE_REF}" --jq .name)" || base_ref="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq .base.ref)"
+          merge_base="$(gh api "repos/${GITHUB_REPOSITORY}/compare/${BASE_SHA}...${HEAD_SHA}" --jq '.merge_base_commit.sha')"
-          target="$(gh api "repos/${GITHUB_REPOSITORY}/branches/${base_ref}" --jq '.commit.sha')"
+          git fetch --no-tags --depth=1 origin "$merge_base" "$HEAD_SHA"
-          merge_base="$(gh api "repos/${GITHUB_REPOSITORY}/compare/${target}...${HEAD_SHA}" --jq '.merge_base_commit.sha')"
-          git fetch --no-tags --depth=1 origin "$merge_base" "$HEAD_SHA" "$target"
-          git checkout -q --detach "$target"
           echo "merge_base=$merge_base" >> "$GITHUB_OUTPUT"
-          echo "target=$target" >> "$GITHUB_OUTPUT"
       - name: Set up uv
         uses: ./.github/actions/setup-uv-with-retries
         with:
@@ -44,30 +37,7 @@ jobs:
       - name: Run the guard
         env:
           MERGE_BASE: ${{ steps.revisions.outputs.merge_base }}
-          COST_MAP_GUARD_TARGET: ${{ steps.revisions.outputs.target }}
-          COST_MAP_GUARD_AUTHOR_TYPE: ${{ github.event.pull_request.user.type }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
           HEAD_REF: ${{ github.event.pull_request.head.ref }}
         run: |
-          uv run --no-project python ci_cd/cost_map_guard.py --base "$MERGE_BASE" --head "$HEAD_SHA" --head-ref "$HEAD_REF"
+          uv run --frozen python ci_cd/cost_map_guard.py --base "$MERGE_BASE" --head "$HEAD_SHA" --head-ref "$HEAD_REF"
-      - name: Check whether the pull request changes the cost map
-        id: changed
-        env:
-          MERGE_BASE: ${{ steps.revisions.outputs.merge_base }}
-          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-        run: |
-          if git diff --quiet "$MERGE_BASE" "$HEAD_SHA" -- model_prices_and_context_window.json; then
-            echo "cost_map=false" >> "$GITHUB_OUTPUT"
-          else
-            echo "cost_map=true" >> "$GITHUB_OUTPUT"
-          fi
-      - if: steps.changed.outputs.cost_map == 'true'
-        run: rustup toolchain install --no-self-update
-      - name: Load the cost map through model-catalog
-        if: steps.changed.outputs.cost_map == 'true'
-        env:
-          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-        run: |
-          git show "${HEAD_SHA}:model_prices_and_context_window.json" > "$RUNNER_TEMP/model_prices_and_context_window.json"
-          cargo run --locked --quiet --manifest-path litellm-rust/Cargo.toml -p litellm-model-catalog --bin check-registry \
-            -- "$RUNNER_TEMP/model_prices_and_context_window.json"

.github/workflows/cost-map-pins-recheck.yml

@@ -1,91 +0,0 @@
-name: Cost map pins recheck
-
-on:
-  push:
-    branches:
-      - main
-      - "litellm_**"
-    paths:
-      - model_prices_and_context_window.json
-      - litellm/model_prices_and_context_window_backup.json
-      - ci_cd/cost_map_pins.json
-
-permissions: {}
-
-jobs:
-  rerun-cost-map-guard:
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    permissions:
-      actions: write
-      contents: read
-      pull-requests: read
-    steps:
-      - name: Re-run the cost map guard on open pull requests the push could break
-        env:
-          GH_TOKEN: ${{ github.token }}
-          BASE_REF: ${{ github.ref_name }}
-          BEFORE_SHA: ${{ github.event.before }}
-          AFTER_SHA: ${{ github.event.after }}
-          FORCED: ${{ github.event.forced }}
-          COST_MAP_PATHS: '["model_prices_and_context_window.json", "litellm/model_prices_and_context_window_backup.json", "ci_cd/cost_map_pins.json"]'
-          PINS_PATHS: '["ci_cd/cost_map_pins.json"]'
-        run: |
-          if [[ "$BEFORE_SHA" =~ ^0+$ || "$FORCED" == true ]] \
-            || ! gh api "repos/${GITHUB_REPOSITORY}/compare/${BEFORE_SHA}...${AFTER_SHA}?per_page=1" \
-              --jq '(env.PINS_PATHS | fromjson) as $pins | (.files | length) < 300 and all(.files[]; [.filename, .previous_filename] | any(IN($pins[])) | not)' \
-              | grep -qx true; then
-            export RECHECK_PATHS="$COST_MAP_PATHS"
-          else
-            export RECHECK_PATHS="$PINS_PATHS"
-          fi
-          rerun_cutoff="$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)"
-          candidates="$(gh api graphql --paginate -F owner="${GITHUB_REPOSITORY_OWNER}" -F name="${GITHUB_REPOSITORY#*/}" -F base="$BASE_REF" \
-            -f query='query($owner: String!, $name: String!, $base: String!, $endCursor: String) {
-              repository(owner: $owner, name: $name) {
-                pullRequests(states: OPEN, baseRefName: $base, first: 25, after: $endCursor) {
-                  nodes { number headRefOid changedFiles files(first: 100) { nodes { path } } }
-                  pageInfo { hasNextPage endCursor }
-                }
-              }
-            }' \
-            --jq '(env.RECHECK_PATHS | fromjson) as $paths | .data.repository.pullRequests.nodes[]
-              | {number, headRefOid, listed: any(.files.nodes[].path; IN($paths[])), unlisted: (.changedFiles > 100)}
-              | select(.listed or .unlisted) | "\(.number) \(.headRefOid) \(.listed)"')"
-          failed=0
-          declare -A rerun_heads
-          while read -r number head_sha listed; do
-            [ -n "$number" ] || continue
-            if [ "$listed" = false ]; then
-              if ! changed="$(gh api --paginate "repos/${GITHUB_REPOSITORY}/pulls/${number}/files?per_page=100" \
-                --jq '(env.RECHECK_PATHS | fromjson) as $paths | .[].filename | select(IN($paths[]))')"; then
-                echo "::error::could not list the files of #$number"
-                failed=1
-                continue
-              fi
-              [ -n "$changed" ] || continue
-            fi
-            [ -z "${rerun_heads[$head_sha]:-}" ] || continue
-            rerun_heads[$head_sha]=1
-            if ! runs="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow cost-map-guard.yml --commit "$head_sha" --limit 100 --json databaseId,createdAt --jq '.[] | "\(.databaseId) \(.createdAt)"')"; then
-              echo "::error::could not list the cost map guard runs of #$number"
-              failed=1
-              continue
-            fi
-            if [ -z "$runs" ]; then
-              echo "::warning::no cost map guard run found for #$number"
-              continue
-            fi
-            while read -r run_id created_at; do
-              if [[ "$created_at" < "$rerun_cutoff" ]]; then
-                echo "::warning::cost map guard run $run_id for #$number is over 30 days old, so GitHub cannot re-run it; #$number gets the pin checks on its next push"
-                continue
-              fi
-              gh run watch "$run_id" --repo "$GITHUB_REPOSITORY" > /dev/null || true
-              if ! gh run rerun "$run_id" --repo "$GITHUB_REPOSITORY"; then
-                echo "::error::could not re-run cost map guard run $run_id for #$number"
-                failed=1
-              fi
-            done <<< "$runs"
-          done <<< "$candidates"
-          exit "$failed"

.github/workflows/create-rc-branch.yml

@@ -2,7 +2,7 @@ name: Create RC Branch
 
 on:
   schedule:
-    - cron: "0 3 * * 1"
+    - cron: "0 3 * * 5"
       timezone: "America/Los_Angeles"
   workflow_dispatch:
 

.github/workflows/deploy-dev-on-main.yml

@@ -0,0 +1,30 @@
+name: Deploy main to gateway-dev
+
+on:
+  push:
+    branches: [main]
+
+permissions: {}
+
+concurrency:
+  group: deploy-dev-dispatch
+  cancel-in-progress: true
+
+jobs:
+  dispatch:
+    if: github.repository == 'BerriAI/litellm'
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2.2.2
+        id: token
+        with:
+          app-id: ${{ secrets.LITELLM_OPS_DISPATCH_APP_ID }}
+          private-key: ${{ secrets.LITELLM_OPS_DISPATCH_APP_PRIVATE_KEY }}
+          owner: BerriAI
+          repositories: litellm-ops
+          permission-actions: write
+      - name: Dispatch the litellm-ops dev deploy
+        env:
+          GH_TOKEN: ${{ steps.token.outputs.token }}
+        run: gh api --method POST repos/BerriAI/litellm-ops/actions/workflows/deploy-dev.yml/dispatches -f ref=main

.github/workflows/image-scan.yml

@@ -8,25 +8,29 @@ on:
       - "litellm_**"
     paths:
       - Dockerfile
-      - .dockerignore
+      - docker/Dockerfile.non_root
-      - docker-entrypoint.sh
+      - docker/Dockerfile.database
+      - migrations/Dockerfile
       - migrations/run.py
+      - gateway/Dockerfile
       - gateway/main.py
-      - gateway/launch.py
       - gateway/routes/allowlist.py
-      - litellm_cli/**
+      - backend/Dockerfile
       - backend/main.py
-      - backend/routes/allowlist.py
       - litellm-rust/**
       - litellm/proxy/lens/**
+      - docker/component_entrypoint.sh
+      - docker/entrypoint.sh
       - litellm/proxy/prisma_migration.py
       - litellm/proxy/admin_mcp.py
       - litellm/proxy/proxy_server.py
+      - backend/routes/allowlist.py
       - pyproject.toml
       - litellm-proxy-extras/**
       - tests/integration/database/image/**
       - uv.lock
       - ui/litellm-dashboard/package-lock.json
+      - ui/Dockerfile
       - ui/nginx.conf
       - .github/workflows/image-scan.yml
       - .grype.yaml
@@ -40,18 +44,14 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
-env:
-  IMAGE: litellm-image-scan:${{ github.sha }}
-  ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
-
 jobs:
-  build:
+  image-scan:
-    name: build
+    name: image-scan
     runs-on: ubuntu-latest
     if: >-
       github.event_name != 'pull_request' ||
       github.event.pull_request.head.repo.full_name == github.repository
-    timeout-minutes: 60
+    timeout-minutes: 30
     permissions:
       contents: read
     steps:
@@ -59,129 +59,7 @@ jobs:
         with:
           persist-credentials: false
 
-      - name: Build the image
-        run: docker build --build-arg MATURIN_PEP517_ARGS="--profile ci" --build-arg LITELLM_RELEASE_TAG=v0.0.0-lens-ci -t "$IMAGE" .
-
-      - name: Save the image for the verify matrix
-        run: docker save "$IMAGE" | zstd -T0 -3 -o "$RUNNER_TEMP/litellm-image.tar.zst"
-
-      - uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
-        with:
-          name: litellm-image
-          path: ${{ runner.temp }}/litellm-image.tar.zst
-          retention-days: 1
-          compression-level: 0
-
-      - name: Build the Admin MCP schema image
-        run: docker build --build-arg MATURIN_PEP517_ARGS="--profile ci" --build-arg LITELLM_RELEASE_TAG=v0.0.0-lens-ci --target builder -t "$ADMIN_MCP_SCHEMA_IMAGE" .
-
-      - name: Save the Admin MCP schema image for the verify matrix
-        run: docker save "$ADMIN_MCP_SCHEMA_IMAGE" | zstd -T0 -3 -o "$RUNNER_TEMP/litellm-admin-mcp-schema.tar.zst"
-
-      - uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
-        with:
-          name: litellm-admin-mcp-schema
-          path: ${{ runner.temp }}/litellm-admin-mcp-schema.tar.zst
-          retention-days: 1
-          compression-level: 0
-
-  verify:
-    name: verify (${{ matrix.check }})
-    needs: build
-    runs-on: ubuntu-latest
-    timeout-minutes: 45
-    permissions:
-      contents: read
-    strategy:
-      fail-fast: false
-      matrix:
-        check: [scan, migrations, gateway, backend, ui, collector]
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
-        with:
-          name: litellm-image
-          path: ${{ runner.temp }}
-
-      - name: Load the image
-        run: zstd -d --stdout "$RUNNER_TEMP/litellm-image.tar.zst" | docker load
-
-      - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
-        if: matrix.check == 'migrations' || matrix.check == 'backend'
-        with:
-          name: litellm-admin-mcp-schema
-          path: ${{ runner.temp }}
-
-      - name: Load the Admin MCP schema image
-        if: matrix.check == 'migrations' || matrix.check == 'backend'
-        run: zstd -d --stdout "$RUNNER_TEMP/litellm-admin-mcp-schema.tar.zst" | docker load
-
-      - name: Set up Python
-        if: matrix.check != 'scan'
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Install pytest
-        if: matrix.check != 'scan'
-        run: python -m pip install "pytest==9.0.3"
-
-      # The prisma bake must migrate a fresh DB with no egress as an arbitrary
-      # non-root uid (OpenShift restricted-v2 / air-gapped / readOnlyFilesystem).
-      # `docker run` as the default uid with network hides a broken bake because
-      # the migration entrypoint exits 0 even when it applied nothing; asserting
-      # the schema was created is what catches it.
-      - name: Verify offline migration as a non-root uid
-        if: matrix.check == 'migrations'
-        env:
-          LITELLM_IMAGE: ${{ env.IMAGE }}
-          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: ${{ env.ADMIN_MCP_SCHEMA_IMAGE }}
-        run: |
-          python -m pytest --noconftest -v \
-            tests/integration/database/image/test_offline_image_migration.py \
-            tests/integration/database/image/test_image_bedrock_realtime_extra.py \
-            tests/integration/database/image/test_image_admin_mcp.py
-
-      - name: Verify the gateway component serves offline as a non-root uid
-        if: matrix.check == 'gateway'
-        env:
-          LITELLM_IMAGE: ${{ env.IMAGE }}
-          LITELLM_COMPONENT: gateway
-          LITELLM_IMAGE_COMPONENT: gateway
-        run: |
-          python -m pytest --noconftest -v \
-            tests/integration/database/image/test_component_image_serves_offline.py \
-            tests/integration/database/image/test_image_admin_mcp.py
-
-      - name: Verify the backend component serves offline as a non-root uid
-        if: matrix.check == 'backend'
-        env:
-          LITELLM_IMAGE: ${{ env.IMAGE }}
-          LITELLM_COMPONENT: backend
-          LITELLM_IMAGE_COMPONENT: backend
-          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: ${{ env.ADMIN_MCP_SCHEMA_IMAGE }}
-        run: |
-          python -m pytest --noconftest -v \
-            tests/integration/database/image/test_component_image_serves_offline.py \
-            tests/integration/database/image/test_image_admin_mcp.py
-
-      - name: Verify the ui component serves as an arbitrary uid with a read-only root fs
-        if: matrix.check == 'ui'
-        env:
-          LITELLM_IMAGE: ${{ env.IMAGE }}
-        run: python -m pytest --noconftest -v tests/integration/database/image/test_ui_image_serves_offline.py
-
-      - name: Verify the collector creates its socket as the image user
-        if: matrix.check == 'collector'
-        env:
-          LITELLM_IMAGE: ${{ env.IMAGE }}
-        run: python -m pytest --noconftest -v tests/integration/database/image/test_collector_image_serves_socket.py
-
       - name: Download Grype v0.114.0
-        if: matrix.check == 'scan'
         run: |
           curl -fsSL --retry 3 -o "$RUNNER_TEMP/grype.tar.gz" \
             https://github.com/anchore/grype/releases/download/v0.114.0/grype_0.114.0_linux_amd64.tar.gz
@@ -189,6 +67,33 @@ jobs:
           tar xzf "$RUNNER_TEMP/grype.tar.gz" -C "$RUNNER_TEMP" grype
           chmod +x "$RUNNER_TEMP/grype"
 
+      # Dockerfile.non_root is the rootless variant we ship. The other
+      # Dockerfiles share the same wolfi base and apk set, so OS-layer coverage
+      # is the same; matrix-scan if those variants ever diverge.
+      - name: Build runtime image
+        run: docker build -f docker/Dockerfile.non_root -t litellm-image-scan:${{ github.sha }} .
+
+      - name: Tag the cached builder for Admin MCP schema setup
+        run: docker build --target builder -f docker/Dockerfile.non_root -t litellm-admin-mcp-schema:${{ github.sha }} .
+
+      # The prisma bake must migrate a fresh DB with no egress as an arbitrary
+      # non-root uid (OpenShift restricted-v2 / air-gapped / readOnlyRootFilesystem).
+      # `docker run` as the default uid with network hides a broken bake because
+      # the migration entrypoint exits 0 even when it applied nothing; asserting
+      # the schema was created is what catches it.
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Verify offline migration as a non-root uid
+        env:
+          LITELLM_IMAGE: litellm-image-scan:${{ github.sha }}
+          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
+        run: |
+          python -m pip install "pytest==9.0.3"
+          python -m pytest --noconftest tests/integration/database/image/test_offline_image_migration.py tests/integration/database/image/test_image_bedrock_realtime_extra.py tests/integration/database/image/test_image_admin_mcp.py -v
+
       # Scans the whole shipped artifact: OS/apk plus every language package
       # baked into the image, including ones no lockfile declares (e.g. prisma's
       # vendored node engine) that osv-scan cannot see. osv-scan stays the fast
@@ -196,12 +101,175 @@ jobs:
       # free OSS, run as a pinned, checksum-verified binary; no GitHub Action
       # dependency and no vendor SaaS callout.
       - name: Scan image for fixable HIGH/CRITICAL CVEs
-        if: matrix.check == 'scan'
         env:
           GRYPE_MATCH_PYTHON_USING_CPES: "true"
         run: |
-          "$RUNNER_TEMP/grype" "$IMAGE" \
+          "$RUNNER_TEMP/grype" litellm-image-scan:${{ github.sha }} \
             --config .grype.yaml \
             --only-fixed \
             --fail-on high \
             --output table
+
+  runtime-image:
+    name: runtime-image (${{ matrix.dockerfile }})
+    runs-on: ubuntu-latest
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    timeout-minutes: 45
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        dockerfile: [Dockerfile, docker/Dockerfile.database]
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Build runtime image
+        run: docker build --build-arg LITELLM_RELEASE_TAG=v0.0.0-lens-ci -f "${{ matrix.dockerfile }}" -t litellm-runtime-scan:${{ github.sha }} .
+
+      - name: Tag the cached builder for Admin MCP schema setup
+        run: docker build --target builder -f "${{ matrix.dockerfile }}" -t litellm-admin-mcp-schema:${{ github.sha }} .
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Verify offline migration as a non-root uid
+        env:
+          LITELLM_IMAGE: litellm-runtime-scan:${{ github.sha }}
+          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
+        run: |
+          python -m pip install "pytest==9.0.3"
+          python -m pytest --noconftest tests/integration/database/image/test_offline_image_migration.py tests/integration/database/image/test_image_bedrock_realtime_extra.py tests/integration/database/image/test_image_admin_mcp.py -v
+
+
+  migrations-image:
+    name: migrations-image
+    runs-on: ubuntu-latest
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Build migrations image
+        run: docker build -f migrations/Dockerfile -t litellm-migrations-scan:${{ github.sha }} .
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Verify offline migration as a non-root uid
+        env:
+          LITELLM_IMAGE: litellm-migrations-scan:${{ github.sha }}
+          LITELLM_MIGRATION_INTERPRETER: python3
+          LITELLM_MIGRATION_SCRIPT: /app/run.py
+        run: |
+          python -m pip install "pytest==9.0.3"
+          python -m pytest --noconftest tests/integration/database/image/test_offline_image_migration.py -v
+
+  gateway-image:
+    name: gateway-image
+    runs-on: ubuntu-latest
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Build gateway image
+        run: docker build -f gateway/Dockerfile -t litellm-gateway-scan:${{ github.sha }} .
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Verify the gateway serves offline as a non-root uid
+        env:
+          LITELLM_IMAGE: litellm-gateway-scan:${{ github.sha }}
+          LITELLM_COMPONENT_PORT: "4000"
+          LITELLM_IMAGE_COMPONENT: gateway
+        run: |
+          python -m pip install "pytest==9.0.3"
+          python -m pytest --noconftest tests/integration/database/image/test_component_image_serves_offline.py tests/integration/database/image/test_image_bedrock_realtime_extra.py tests/integration/database/image/test_image_admin_mcp.py -v
+
+  ui-image:
+    name: ui-image
+    runs-on: ubuntu-latest
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Build UI image
+        run: docker build -f ui/Dockerfile -t litellm-ui-scan:${{ github.sha }} .
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Verify the UI serves offline as an arbitrary uid with a read-only root fs
+        env:
+          LITELLM_IMAGE: litellm-ui-scan:${{ github.sha }}
+        run: |
+          python -m pip install "pytest==9.0.3"
+          python -m pytest --noconftest tests/integration/database/image/test_ui_image_serves_offline.py -v
+
+  backend-image:
+    name: backend-image
+    runs-on: ubuntu-latest
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Build backend image
+        run: docker build -f backend/Dockerfile -t litellm-backend-scan:${{ github.sha }} .
+
+      - name: Tag the cached builder for Admin MCP schema setup
+        run: docker build --target builder -f backend/Dockerfile -t litellm-admin-mcp-schema:${{ github.sha }} .
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Verify the backend serves offline as a non-root uid
+        env:
+          LITELLM_IMAGE: litellm-backend-scan:${{ github.sha }}
+          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
+          LITELLM_COMPONENT_PORT: "4001"
+          LITELLM_IMAGE_COMPONENT: backend
+        run: |
+          python -m pip install "pytest==9.0.3"
+          python -m pytest --noconftest tests/integration/database/image/test_component_image_serves_offline.py tests/integration/database/image/test_image_admin_mcp.py -v

.github/workflows/interactions-spec-canary.yml

@@ -1,94 +0,0 @@
-name: Interactions spec canary
-
-on:
-  schedule:
-    - cron: "17 7 * * *"
-  pull_request:
-    paths:
-      - .github/workflows/interactions-spec-canary.yml
-      - tests/unit/interactions/**
-      - litellm/types/interactions/**
-  workflow_dispatch:
-
-permissions: {}
-
-concurrency: interactions-spec-canary
-
-jobs:
-  interactions-spec-canary:
-    name: interactions-spec-canary
-    if: github.repository == 'BerriAI/litellm'
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    permissions:
-      contents: read
-    outputs:
-      compliance: ${{ steps.compliance.outcome }}
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Set up uv
-        uses: ./.github/actions/setup-uv-with-retries
-        with:
-          version: "0.10.9"
-
-      - name: Set up the Rust bridge
-        uses: ./.github/actions/rust-bridge
-
-      - name: Install dependencies
-        run: .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra proxy
-
-      - name: Run the compliance tests against Google's live spec
-        id: compliance
-        run: |
-          uv run --no-sync pytest tests/unit/interactions/test_openapi_compliance.py \
-            --interactions-spec-url=https://aistudio.google.com/api-reference/interactions.openapi.json \
-            --tb=short -vv -p no:rerunfailures -p no:pytest-retry \
-            --allow-hosts=127.0.0.1,::1,localhost,ai.google.dev,aistudio.google.com
-
-  tracking-issue:
-    name: tracking-issue
-    needs: interactions-spec-canary
-    if: always() && github.event_name != 'pull_request' && github.ref_name == github.event.repository.default_branch && github.repository == 'BerriAI/litellm'
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
-    permissions:
-      issues: write
-    steps:
-      - name: File the tracking issue unless the compliance step passed, close it when it did
-        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
-        env:
-          COMPLIANCE_OUTCOME: ${{ needs.interactions-spec-canary.outputs.compliance }}
-          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
-        with:
-          script: |
-            const title = "Interactions spec canary is red";
-            const outcome = process.env.COMPLIANCE_OUTCOME;
-            const runUrl = process.env.RUN_URL;
-            const query = `repo:${context.repo.owner}/${context.repo.repo} is:issue is:open in:title "${title}"`;
-            const { data: found } = await github.rest.search.issuesAndPullRequests({ q: query, per_page: 20 });
-            const open = found.items.find((issue) => issue.title === title);
-            if (outcome === "success") {
-              if (open === undefined) return;
-              await github.rest.issues.createComment({ ...context.repo, issue_number: open.number, body: `Green again: ${runUrl}` });
-              await github.rest.issues.update({ ...context.repo, issue_number: open.number, state: "closed" });
-              return;
-            }
-            const state = outcome === "failure" ? "red" : `not run (compliance step outcome: ${outcome || "none"})`;
-            if (open !== undefined) {
-              await github.rest.issues.createComment({ ...context.repo, issue_number: open.number, body: `Still ${state}: ${runUrl}` });
-              return;
-            }
-            await github.rest.issues.create({
-              ...context.repo,
-              title,
-              assignees: ["mateo-berri"],
-              body: `The scheduled run of tests/unit/interactions/test_openapi_compliance.py against Google's live Interactions spec is ${state}: ${runUrl}\n\nlitellm/types/interactions/README.md says how to refresh the snapshot and regenerate the types.`,
-            });

.github/workflows/lens-install-smoke.yml

@@ -0,0 +1,152 @@
+name: Lens installation smoke
+
+on:
+  workflow_dispatch:
+    inputs:
+      lens_image:
+        description: Verified baseline Lens image at ghcr.io/berriai/lens@sha256
+        required: true
+        type: string
+      lens_upgrade_image:
+        description: Compatible Lens upgrade image at ghcr.io/berriai/lens@sha256
+        required: true
+        type: string
+      gateway_baseline_ref:
+        description: Previously qualified gateway source commit, full SHA
+        required: true
+        type: string
+
+permissions:
+  contents: read
+
+concurrency:
+  group: lens-install-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build-images:
+    runs-on: ubuntu-latest-16-cores
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - component: gateway
+            dockerfile: gateway/Dockerfile
+          - component: backend
+            dockerfile: backend/Dockerfile
+          - component: ui
+            dockerfile: ui/Dockerfile
+          - component: migrations
+            dockerfile: migrations/Dockerfile
+          - component: monolith
+            dockerfile: Dockerfile
+          - component: gateway
+            dockerfile: gateway/Dockerfile
+            baseline: true
+          - component: backend
+            dockerfile: backend/Dockerfile
+            baseline: true
+          - component: monolith
+            dockerfile: Dockerfile
+            baseline: true
+    env:
+      COMPONENT: ${{ matrix.component }}
+      DOCKERFILE: ${{ matrix.dockerfile }}
+      SOURCE_REF: ${{ matrix.baseline && inputs.gateway_baseline_ref || github.sha }}
+      IMAGE_TAG: ${{ matrix.baseline && 'v0.0.0-lens-ci-baseline' || 'v0.0.0-lens-ci' }}
+      CURRENT_REF: ${{ github.sha }}
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+          ref: ${{ env.SOURCE_REF }}
+      - name: Verify the selected source
+        shell: bash -euo pipefail {0}
+        run: |
+          [[ "$SOURCE_REF" =~ ^[0-9a-f]{40}$ ]]
+          test "$(git rev-parse HEAD)" = "$SOURCE_REF"
+          if [[ "$IMAGE_TAG" == v0.0.0-lens-ci-baseline ]]; then
+            test "$SOURCE_REF" != "$CURRENT_REF"
+          fi
+      - name: Build the matching release image
+        run: |
+          docker build --build-arg "LITELLM_RELEASE_TAG=$IMAGE_TAG" \
+            -f "$DOCKERFILE" -t "lens-ci-$COMPONENT:$IMAGE_TAG" .
+      - name: Save the matching release image
+        run: |
+          docker save "lens-ci-$COMPONENT:$IMAGE_TAG" \
+            | gzip -1 > "$RUNNER_TEMP/lens-install-$COMPONENT-$IMAGE_TAG.tar.gz"
+      - uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
+        with:
+          name: lens-install-${{ matrix.component }}-${{ env.IMAGE_TAG }}-${{ github.sha }}
+          path: ${{ runner.temp }}/lens-install-${{ matrix.component }}-${{ env.IMAGE_TAG }}.tar.gz
+          compression-level: 0
+          retention-days: 3
+          if-no-files-found: error
+          overwrite: true
+
+  helm-install:
+    needs: build-images
+    runs-on: ubuntu-latest-16-cores
+    timeout-minutes: 25
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+      - uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e # v4.2.1
+        with:
+          pattern: lens-install-*-${{ github.sha }}
+          merge-multiple: true
+          path: ${{ runner.temp }}/lens-install-images
+      - name: Load the matching release images
+        run: |
+          for archive in "$RUNNER_TEMP"/lens-install-images/lens-install-*.tar.gz; do
+            gzip -dc "$archive" | docker load
+            rm "$archive"
+          done
+      - uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v3.8.2
+        with:
+          cosign-release: v2.4.3
+      - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        with:
+          registry: ghcr.io
+          username: ${{ vars.LENS_READ_USER || github.actor }}
+          password: ${{ secrets.LENS_READ_TOKEN }}
+      - name: Verify independent Lens images before loading them
+        env:
+          LENS_IMAGE: ${{ inputs.lens_image }}
+          LENS_UPGRADE_IMAGE: ${{ inputs.lens_upgrade_image }}
+          LENS_PUBLIC_KEY: ${{ vars.LENS_COSIGN_PUBLIC_KEY }}
+          LENS_PUBLIC_KEY_SHA256: ${{ vars.LENS_COSIGN_PUBLIC_KEY_SHA256 }}
+        shell: bash -euo pipefail {0}
+        run: |
+          [[ "$LENS_PUBLIC_KEY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          printf '%s' "$LENS_PUBLIC_KEY" > "$RUNNER_TEMP/lens.pub"
+          echo "$LENS_PUBLIC_KEY_SHA256  $RUNNER_TEMP/lens.pub" | sha256sum --check
+          for image in "$LENS_IMAGE" "$LENS_UPGRADE_IMAGE"; do
+            [[ "$image" =~ ^ghcr.io/berriai/lens@sha256:[0-9a-f]{64}$ ]]
+            cosign verify --key "$RUNNER_TEMP/lens.pub" "$image" > /dev/null
+            cosign verify-attestation --key "$RUNNER_TEMP/lens.pub" --type spdxjson "$image" > /dev/null
+            docker pull "$image"
+          done
+          docker tag "$LENS_IMAGE" lens-ci-worker:baseline
+          docker tag "$LENS_UPGRADE_IMAGE" lens-ci-worker:upgrade
+      - name: Install pinned Kubernetes test tools
+        run: |
+          curl --fail --location --output "$RUNNER_TEMP/kind" \
+            https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-amd64
+          echo "a6875aaea358acf0ac07786b1a6755d08fd640f4c79b7a2e46681cc13f49a04b  $RUNNER_TEMP/kind" | sha256sum --check
+          chmod +x "$RUNNER_TEMP/kind"
+          curl --fail --location --output "$RUNNER_TEMP/kubectl" \
+            https://dl.k8s.io/release/v1.32.2/bin/linux/amd64/kubectl
+          echo "4f6a959dcc5b702135f8354cc7109b542a2933c46b808b248a214c1f69f817ea  $RUNNER_TEMP/kubectl" | sha256sum --check
+          chmod +x "$RUNNER_TEMP/kubectl"
+          curl --fail --location --output "$RUNNER_TEMP/helm.tar.gz" \
+            https://get.helm.sh/helm-v3.19.0-linux-amd64.tar.gz
+          echo "a7f81ce08007091b86d8bd696eb4d86b8d0f2e1b9f6c714be62f82f96a594496  $RUNNER_TEMP/helm.tar.gz" | sha256sum --check
+          tar -xzf "$RUNNER_TEMP/helm.tar.gz" -C "$RUNNER_TEMP"
+          echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
+          echo "$RUNNER_TEMP/linux-amd64" >> "$GITHUB_PATH"
+      - name: Install, ingest, upgrade, and restart both charts
+        run: bash tests/e2e/migrations/lens_helm_smoke.sh

.github/workflows/osv-scan.yml

@@ -5,9 +5,6 @@ on:
     branches:
       - main
       - "litellm_**"
-      - "rc/**"
-      - "stable/**"
-      - "lts/**"
   schedule:
     - cron: "23 6 * * *"
   workflow_dispatch:

.github/workflows/spend-logs-guard.yml

@@ -1,45 +0,0 @@
-name: Spend logs guard
-
-on:
-  pull_request:
-    branches:
-      - main
-
-permissions:
-  contents: read
-  pull-requests: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
-  cancel-in-progress: true
-
-jobs:
-  spend-logs-guard:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          ref: ${{ github.event.pull_request.base.ref }}
-          persist-credentials: false
-      - name: Fetch the pull request head, its merge base and its reviews
-        id: revisions
-        env:
-          GH_TOKEN: ${{ github.token }}
-          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
-          BASE_REF: ${{ github.event.pull_request.base.ref }}
-        run: |
-          merge_base="$(gh api "repos/${GITHUB_REPOSITORY}/compare/${BASE_REF}...${HEAD_SHA}" --jq '.merge_base_commit.sha')"
-          git fetch --no-tags --depth=1 origin "$merge_base" "$HEAD_SHA"
-          gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/reviews" --paginate --slurp | jq 'add // []' > "$RUNNER_TEMP/reviews.json"
-          echo "merge_base=$merge_base" >> "$GITHUB_OUTPUT"
-      - name: Set up uv
-        uses: ./.github/actions/setup-uv-with-retries
-        with:
-          version: "0.10.9"
-      - name: Run the guard
-        env:
-          MERGE_BASE: ${{ steps.revisions.outputs.merge_base }}
-          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-        run: |
-          uv run --no-project --with pydantic==2.14.1 python ci_cd/spend_logs_guard.py --base "$MERGE_BASE" --head "$HEAD_SHA" --approver kerry-berri --reviews "$RUNNER_TEMP/reviews.json"

.github/workflows/sync-together-ai-models.yml

@@ -0,0 +1,70 @@
+name: Sync Together AI model registry
+
+on:
+  schedule:
+    - cron: "30 6 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  sync_together_ai_models:
+    if: github.repository == 'BerriAI/litellm'
+    runs-on: ubuntu-latest
+    env:
+      BASE_BRANCH: ${{ github.event.repository.default_branch }}
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          ref: ${{ env.BASE_BRANCH }}
+          persist-credentials: false
+      - name: Set up uv
+        uses: ./.github/actions/setup-uv-with-retries
+        with:
+          version: "0.10.9"
+      - name: Look for an already-open sync PR
+        id: existing
+        run: |
+          open_pr="$(gh pr list --repo "$GITHUB_REPOSITORY" --state open --limit 1000 --json headRefName \
+            --jq '[.[].headRefName | select(startswith("litellm_together_registry_sync_"))] | first // empty')"
+          echo "open_pr=$open_pr" >> "$GITHUB_OUTPUT"
+          if [ -n "$open_pr" ]; then
+            echo "An open sync PR already exists on branch $open_pr; skipping this run."
+          fi
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN || github.token }}
+      - name: Run the sync
+        if: steps.existing.outputs.open_pr == ''
+        run: |
+          uv run --frozen python scripts/sync_together_ai_models.py --write --pr-body-file "$RUNNER_TEMP/pr_body.md"
+        env:
+          TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
+      - name: Regenerate the JSON schema
+        if: steps.existing.outputs.open_pr == ''
+        run: |
+          uv run --frozen python ci_cd/generate_model_prices_schema.py
+      - name: Create a pull request when the registry changed
+        if: steps.existing.outputs.open_pr == ''
+        run: |
+          if git diff --quiet; then
+            echo "Registry already in sync; no PR needed."
+            exit 0
+          fi
+          branch="litellm_together_registry_sync_$(date +'%Y-%m-%d')"
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git checkout -b "$branch"
+          git add model_prices_and_context_window.json \
+            litellm/model_prices_and_context_window_backup.json \
+            model_prices_and_context_window.schema.json
+          git commit -m "feat(models): sync together_ai model registry $(date +'%Y-%m-%d')"
+          gh auth setup-git
+          git push origin "$branch"
+          gh pr create --title "feat(models): sync together_ai model registry" \
+            --body-file "$RUNNER_TEMP/pr_body.md" \
+            --head "$branch" \
+            --base "$BASE_BRANCH"
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN || github.token }}

.github/workflows/test-integration.yml

@@ -1,199 +0,0 @@
-name: integration
-
-on:
-  pull_request:
-    types: [labeled]
-  schedule:
-    - cron: "5 8-20/3 * * 1-6"
-      timezone: America/Los_Angeles
-  workflow_dispatch:
-    inputs:
-      suite:
-        description: Integration group to run
-        type: choice
-        options:
-          - all
-          - providers
-          - extensions
-          - accounting
-          - management
-          - management-replica
-          - database
-          - database-replica
-          - mcp
-          - security
-          - cost
-          - sdk
-          - browser
-        default: all
-      workers:
-        description: Workers per group, blank keeps each group's default
-        type: string
-        default: ""
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}-${{ github.event.label.name }}-${{ inputs.suite }}-${{ inputs.workers }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  integration:
-    name: integration ${{ matrix.suite }}
-    if: github.event_name != 'pull_request' || github.event.label.name == 'run-ci'
-    environment: integration-license
-    runs-on: ${{ fromJSON('{"providers":"ubuntu-latest-64-cores","extensions":"ubuntu-latest-64-cores","accounting":"ubuntu-latest-32-cores","management":"ubuntu-latest-32-cores","management-replica":"ubuntu-latest-32-cores"}')[matrix.suite] || 'ubuntu-latest-16-cores' }}
-    timeout-minutes: 60
-    strategy:
-      fail-fast: false
-      matrix:
-        suite: ${{ fromJSON((inputs.suite || 'all') == 'all' && '["providers","extensions","accounting","management","management-replica","database","database-replica","mcp","security","cost","sdk","browser"]' || format('["{0}"]', inputs.suite)) }}
-    env:
-      UV_PYTHON: "3.12"
-      INTEGRATION_WORKERS: ${{ inputs.workers || fromJSON('{"providers":"32","extensions":"48","accounting":"16","management":"16","management-replica":"16","cost":"16","mcp":"8","security":"8","database":"4","database-replica":"4"}')[matrix.suite] || '' }}
-    services:
-      postgres:
-        image: postgres:16@sha256:e17e86066e5ef83e0952a9347f5c792b7ece00972e2aa787a6986f471b3dd3d5
-        command: -c max_connections=1000 -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all -c pg_stat_statements.max=20000
-        env:
-          POSTGRES_USER: postgres
-          POSTGRES_PASSWORD: postgres
-          POSTGRES_DB: circle_test
-        ports:
-          - 5432:5432
-        options: >-
-          --shm-size 1g
-          --health-cmd "pg_isready -U postgres"
-          --health-interval 5s
-          --health-timeout 5s
-          --health-retries 20
-      redis:
-        image: redis:7-alpine@sha256:7aec734b2bb298a1d769fd8729f13b8514a41bf90fcdd1f38ec52267fbaa8ee6
-        ports:
-          - 6379:6379
-        options: >-
-          --health-cmd "redis-cli ping"
-          --health-interval 5s
-          --health-timeout 5s
-          --health-retries 20
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Set up Python
-        timeout-minutes: 3
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: ${{ env.UV_PYTHON }}
-
-      - name: Set up uv
-        timeout-minutes: 3
-        uses: ./.github/actions/setup-uv-with-retries
-        with:
-          version: "0.10.9"
-
-      - name: Cache the Rust build
-        timeout-minutes: 5
-        uses: ./.github/actions/cache-cargo-build
-
-      - name: Cache Prisma binaries
-        timeout-minutes: 5
-        uses: ./.github/actions/cache-prisma-binaries
-
-      - name: Install dependencies
-        timeout-minutes: 20
-        run: .github/scripts/uv_sync_with_retries.sh --frozen --all-groups --all-extras
-
-      - name: Install redis-server 7.2.16
-        timeout-minutes: 3
-        run: |
-          mkdir -p "$RUNNER_TEMP/redis-bin"
-          cid="$(docker create redis:7.2.16@sha256:0637954999d01b7c9ce9167db2da50656e2590d3b884f1c600c5f63bb6e6773c)"
-          docker cp "${cid}:/usr/local/bin/redis-server" "$RUNNER_TEMP/redis-bin/redis-server"
-          docker rm "$cid"
-          "$RUNNER_TEMP/redis-bin/redis-server" --version | grep -q 'v=7.2.16'
-          echo "$RUNNER_TEMP/redis-bin" >> "$GITHUB_PATH"
-
-      - name: Install MCP SDK1 peer
-        if: matrix.suite == 'mcp'
-        timeout-minutes: 5
-        run: |
-          uv venv --python 3.12 .venv-mcp-peer
-          uv pip install --no-config --exclude-newer "3 days" --python .venv-mcp-peer 'mcp==1.28.1' 'langchain-mcp-adapters==0.2.1'
-          echo "MCP_TEST_PEER_PYTHON=$PWD/.venv-mcp-peer/bin/python" >> "$GITHUB_ENV"
-
-      - name: Set up Node.js
-        if: matrix.suite == 'browser'
-        timeout-minutes: 5
-        uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
-        with:
-          node-version-file: ui/litellm-dashboard/.nvmrc
-          cache: "npm"
-          cache-dependency-path: |
-            ui/litellm-dashboard/package-lock.json
-            tests/e2e/ui/package-lock.json
-
-      - name: Install browser dependencies and build the dashboard
-        if: matrix.suite == 'browser'
-        timeout-minutes: 20
-        run: |
-          (cd ui/litellm-dashboard && npm ci)
-          (cd tests/e2e/ui && npm ci)
-          sudo env PATH="$PATH" DEBIAN_FRONTEND=noninteractive \
-            node tests/e2e/ui/node_modules/@playwright/test/cli.js install-deps chromium
-          node tests/e2e/ui/node_modules/@playwright/test/cli.js install chromium
-          (cd ui/litellm-dashboard && NEXT_TELEMETRY_DISABLED=1 npm run build)
-
-      - name: Run integration contracts
-        timeout-minutes: 45
-        env:
-          SUITE: ${{ matrix.suite }}
-          INTEGRATION_POSTGRES_CONTAINER: ${{ job.services.postgres.id }}
-          INTEGRATION_REDIS_CONTAINER: ${{ job.services.redis.id }}
-          LITELLM_LICENSE: ${{ secrets.LITELLM_LICENSE }}
-        run: |
-          case "$SUITE" in
-            *-replica) bash .circleci/scripts/run_integration.sh "${SUITE%-replica}" replica ;;
-            *) bash .circleci/scripts/run_integration.sh "$SUITE" ;;
-          esac
-
-      - name: Verify every case ran exactly once
-        if: ${{ !cancelled() && matrix.suite != 'browser' }}
-        timeout-minutes: 2
-        run: |
-          record="$(find test-results -mindepth 2 -maxdepth 2 -name execution.json -print -quit)"
-          test -n "$record"
-          mkdir -p "$RUNNER_TEMP/shards/0"
-          cp "$record" "$RUNNER_TEMP/shards/0/execution.json"
-          .venv/bin/python -m tests.integration._support.sharding "$RUNNER_TEMP/shards" --shards 1
-
-      - name: Report MCP coverage
-        if: ${{ !cancelled() && matrix.suite == 'mcp' }}
-        timeout-minutes: 5
-        run: |
-          coverage_dir=test-results/integration-mcp/coverage
-          test -f "$coverage_dir/coverage.txt" && exit 0
-          export COVERAGE_FILE="$PWD/$coverage_dir/data"
-          .venv/bin/python -m coverage combine --rcfile=tests/integration/mcp_coverage.toml
-          .venv/bin/python -m coverage report --rcfile=tests/integration/mcp_coverage.toml > "$coverage_dir/coverage.txt"
-          .venv/bin/python -m coverage html --rcfile=tests/integration/mcp_coverage.toml -d "$coverage_dir/html"
-          tail -n 1 "$coverage_dir/coverage.txt"
-
-      - name: Check results for the license
-        id: secret-scan
-        if: ${{ !cancelled() }}
-        timeout-minutes: 5
-        env:
-          SCANNED_SECRET: ${{ secrets.LITELLM_LICENSE }}
-        run: python .github/scripts/assert_secret_absent.py test-results
-
-      - name: Upload results
-        if: ${{ !cancelled() && steps.secret-scan.outcome == 'success' }}
-        timeout-minutes: 10
-        uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
-        with:
-          name: integration-${{ matrix.suite }}-${{ github.run_id }}-${{ github.run_attempt }}
-          path: test-results/
-          retention-days: 14

.github/workflows/test-linting.yml

@@ -21,7 +21,7 @@ jobs:
       pull-requests: read
       actions: read
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 20
     services:
       redis:
         image: redis@sha256:e2debfb7956fa12c7ddc79d7e645c8cf26b30c99a6e9161ea9bf4171e1668a5f
@@ -48,13 +48,11 @@ jobs:
         if: steps.changes.outputs.decision != 'skip'
         env:
           GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
           retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
-          BASE_REF=$(gh api "repos/${GITHUB_REPOSITORY}/branches/${EVENT_BASE_REF}" --jq .name) || BASE_REF=$(retry gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq .base.ref)
+          MERGE_BASE=$(retry gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
-          MERGE_BASE=$(retry gh api "repos/${GITHUB_REPOSITORY}/compare/${BASE_REF}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
           test -n "$MERGE_BASE"
           retry git fetch --no-tags --depth=1 origin "$MERGE_BASE"
           echo "GATE_BASE_SHA=$MERGE_BASE" >> "$GITHUB_ENV"
@@ -208,7 +206,7 @@ jobs:
             exit 0
           fi
           retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
-          CLAUDE_VERSION="$(retry uv run --no-sync python tests/e2e/cli_harness/npm_version_resolver.py @anthropic-ai/claude-code)"
+          CLAUDE_VERSION="$(retry uv run --no-sync python tests/e2e/claude_code/pr_gate_version_resolver.py)"
           tests/e2e/claude_code/cron_vm/install_claude_code.sh "$CLAUDE_VERSION" "$RUNNER_TEMP/claude-cli"
           PATH="$RUNNER_TEMP/claude-cli:$PATH" uv run --no-sync pytest -q tests/e2e_harness
           uv run --no-sync pytest --tb=short -q --noconftest -o addopts= -o "pythonpath=tests/e2e tests/e2e_harness" -p no:rerunfailures \
@@ -266,7 +264,6 @@ jobs:
     name: ui
     permissions:
       contents: read
-      pull-requests: read
     runs-on: ubuntu-latest
     timeout-minutes: 8
     defaults:
@@ -284,12 +281,16 @@ jobs:
         id: changed
         env:
           GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
-          base_ref=$(gh api "repos/${GITHUB_REPOSITORY}/branches/${EVENT_BASE_REF}" --jq .name) || base_ref=$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq .base.ref)
+          # base.sha is the base branch tip from when the PR was opened, while
-          merge_base=$(gh api "repos/${GITHUB_REPOSITORY}/compare/${base_ref}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
+          # actions/checkout leaves HEAD on a merge of the PR into the *current*
+          # base tip. "$BASE_SHA"...HEAD therefore spans every base-branch commit
+          # landed since, so a PR that touches no UI file still gets linted
+          # against hundreds of other people's files. Diff the PR head against its
+          # own merge base instead, which is exactly what this PR changed.
+          merge_base=$(gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
           test -n "$merge_base"
           git fetch --no-tags --depth=1 origin "$merge_base" "$HEAD_SHA"
           : > "$RUNNER_TEMP/prettier_files.txt"
@@ -375,10 +376,10 @@ jobs:
             exit 0
           fi
           files="$(git diff --name-only "$base" HEAD)"
-          if grep -Eq '^(litellm/|enterprise/|(pyproject\.toml|uv\.lock)$|ui/litellm-dashboard/(src/lib/http/schema\.d\.ts|scripts/gen-api-types\.mjs|package(-lock)?\.json)$|\.github/workflows/test-linting\.yml$)' <<< "$files"; then
+          if grep -Eq '^(litellm/(proxy|types)/|ui/litellm-dashboard/(src/lib/http/schema\.d\.ts|scripts/gen-api-types\.mjs|package(-lock)?\.json)$|\.github/workflows/test-linting\.yml$)' <<< "$files"; then
             echo "relevant=true" >> "$GITHUB_OUTPUT"
           else
-            echo "No litellm, enterprise, dependency or generator changes in this pull request, nothing to verify."
+            echo "No proxy, types or generator changes in this pull request, nothing to verify."
             echo "relevant=false" >> "$GITHUB_OUTPUT"
           fi
 
@@ -523,12 +524,6 @@ jobs:
       - name: check_provider_folders_documented
         run: uv run --no-sync python ./tests/code_coverage_tests/check_provider_folders_documented.py
 
-      - name: check_provider_endpoints_support_generated
-        run: uv run --no-sync python scripts/generate_provider_endpoints_support.py --check
-
-      - name: test_generate_provider_endpoints_support
-        run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_generate_provider_endpoints_support.py
-
       - name: check_prisma_binary_cache
         run: uv run --no-sync python ./tests/code_coverage_tests/check_prisma_binary_cache.py
 
@@ -544,8 +539,8 @@ jobs:
       - name: test_unit_passed_gate
         run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_unit_passed_gate.py
 
-      - name: test_e2e_idp_stack
+      - name: test_e2e_changed_gate
-        run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_e2e_idp_stack.py
+        run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_e2e_changed_gate.py tests/code_coverage_tests/test_e2e_idp_stack.py
 
       - name: test_e2e_metadata
         env:
@@ -560,9 +555,6 @@ jobs:
       - name: router_code_coverage
         run: uv run --no-sync python ./tests/code_coverage_tests/router_code_coverage.py
 
-      - name: router_surface_check
-        run: uv run --no-sync python ./tests/code_coverage_tests/router_surface_check.py
-
       - name: test_chat_completion_imports
         run: uv run --no-sync python ./tests/code_coverage_tests/test_chat_completion_imports.py
 
@@ -629,12 +621,6 @@ jobs:
       - name: check_migrations_lock_timeout
         run: uv run --no-sync python ./tests/code_coverage_tests/check_migrations_lock_timeout.py
 
-      - name: check_no_none_string_error_fields
-        run: uv run --no-sync python ./tests/code_coverage_tests/check_no_none_string_error_fields.py
-
-      - name: test_check_no_none_string_error_fields
-        run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_check_no_none_string_error_fields.py
-
       - name: check_no_publicly_known_master_key
         run: uv run --no-sync python ./tests/code_coverage_tests/check_no_publicly_known_master_key.py
 

.github/workflows/test-mcp-dependency-resolution.yml

@@ -45,7 +45,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
 
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
@@ -132,7 +132,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:

.github/workflows/test-mcp-oauth-e2e.yml

@@ -0,0 +1,180 @@
+name: MCP OAuth happy path
+
+on:
+  pull_request:
+    paths:
+      - '.github/workflows/test-mcp-oauth-e2e.yml'
+      - '.github/e2e-stack/**'
+      - 'tests/e2e/*.py'
+      - 'tests/e2e/pytest.ini'
+      - 'tests/e2e/idp_realm.json'
+      - 'tests/e2e/mcp/**'
+      - 'litellm/experimental_mcp_client/**'
+      - 'litellm/proxy/_experimental/mcp_server/**'
+      - 'litellm/proxy/auth/**'
+      - 'litellm/proxy/management_endpoints/mcp_management_endpoints.py'
+      - 'litellm/proxy/_types.py'
+      - 'litellm/types/mcp_server/mcp_server_manager.py'
+      - 'litellm/proxy/management_endpoints/*sso*.py'
+      - 'litellm/proxy/management_endpoints/sso/**'
+      - 'litellm/proxy/common_utils/encrypt_decrypt_utils.py'
+      - 'litellm/proxy/proxy_server.py'
+      - 'litellm/proxy/schema.prisma'
+      - 'ui/litellm-dashboard/src/app/connect/**'
+      - 'ui/litellm-dashboard/src/app/mcp/oauth/**'
+      - 'pyproject.toml'
+      - 'uv.lock'
+  workflow_dispatch:
+
+permissions: {}
+
+concurrency:
+  group: mcp-oauth-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  oauth:
+    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    environment: e2e-changed
+    timeout-minutes: 45
+    permissions:
+      contents: read
+      id-token: write
+    services:
+      postgres:
+        image: postgres:16.6
+        env:
+          POSTGRES_USER: litellm
+          POSTGRES_PASSWORD: dbpassword9090
+          POSTGRES_DB: litellm
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U litellm"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+    env:
+      DATABASE_HOST: 127.0.0.1
+      DATABASE_PORT: '5432'
+      DATABASE_USER: litellm
+      DATABASE_PASSWORD: dbpassword9090
+      DATABASE_NAME: litellm
+      DATABASE_URL: postgresql://litellm:dbpassword9090@127.0.0.1:5432/litellm
+      E2E_KEYCLOAK_URL: http://127.0.0.1:8081
+      E2E_KEYCLOAK_ADMIN_USER: admin
+      E2E_KEYCLOAK_ADMIN_PASSWORD: e2e-ephemeral-idp-not-a-secret
+      E2E_FIXTURE_MODE: live
+      E2E_PROVIDER_CACHE: '0'
+      E2E_MCP_OAUTH_LIVE: '1'
+      E2E_REQUIRED_TEST_COUNT: '4'
+    steps:
+      - name: Checkout the tested source
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
+
+      - name: Require and materialize the upstream login
+        env:
+          STORAGE_STATE: ${{ secrets.E2E_LINEAR_STORAGE_STATE_B64 }}
+        run: |
+          umask 077
+          python3 - <<'PY'
+          import base64
+          import json
+          import os
+          import secrets
+          from pathlib import Path
+          encoded = os.environ.get("STORAGE_STATE", "")
+          if not encoded:
+              raise SystemExit("E2E_LINEAR_STORAGE_STATE_B64 is required; capture and provision a test-account login")
+          state = json.loads(base64.b64decode(encoded, validate=True))
+          if not isinstance(state, dict) or not state.get("cookies"):
+              raise SystemExit("The captured login must contain browser cookies")
+          directory = Path(os.environ["RUNNER_TEMP"]) / "mcp-oauth-private"
+          directory.mkdir(mode=0o700)
+          path = directory / "linear-state.json"
+          path.write_text(json.dumps(state))
+          with open(os.environ["GITHUB_ENV"], "a") as output:
+              output.write(f"E2E_LINEAR_STORAGE_STATE={path}\n")
+              for name in ("LITELLM_MASTER_KEY", "LITELLM_SALT_KEY"):
+                  value = "sk-e2e-" + secrets.token_hex(24)
+                  print(f"::add-mask::{value}")
+                  output.write(f"{name}={value}\n")
+          PY
+
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: '3.13'
+      - uses: ./.github/actions/setup-uv-with-retries
+        with:
+          version: '0.10.9'
+      - uses: ./.github/actions/cache-cargo-build
+      - name: Install the frozen E2E environment
+        run: |
+          .github/scripts/uv_sync_with_retries.sh --frozen --extra proxy --extra proxy-runtime --extra extra_proxy --group ci --group proxy-dev --group e2e-dev
+          uv run --no-sync python scripts/prisma_generate_if_needed.py
+          uv run --no-sync playwright install --with-deps chromium
+
+      - name: Configure license access
+        id: aws
+        uses: aws-actions/configure-aws-credentials@e7f100cf4c008499ea8adda475de1042d6975c7b # v6.2.0
+        with:
+          role-to-assume: ${{ vars.E2E_AWS_ROLE_TO_ASSUME }}
+          aws-region: us-east-1
+          role-session-name: mcp-oauth-${{ github.run_id }}
+          role-duration-seconds: 900
+          output-env-credentials: false
+          output-credentials: true
+      - name: Load the E2E license
+        env:
+          AWS_ACCESS_KEY_ID: ${{ steps.aws.outputs.aws-access-key-id }}
+          AWS_SECRET_ACCESS_KEY: ${{ steps.aws.outputs.aws-secret-access-key }}
+          AWS_SESSION_TOKEN: ${{ steps.aws.outputs.aws-session-token }}
+          AWS_DEFAULT_REGION: us-east-1
+        run: |
+          license="$(aws secretsmanager get-secret-value --secret-id litellm-e2e-changed-license --query SecretString --output text)"
+          test -n "${license}"
+          echo "::add-mask::${license}"
+          echo "LITELLM_LICENSE=${license}" >> "${GITHUB_ENV}"
+
+      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version-file: ui/litellm-dashboard/.nvmrc
+      - name: Build the gateway consent UI at the tested commit
+        run: |
+          cd ui/litellm-dashboard
+          ../../scripts/with_dashboard_node.sh npm ci
+          ../../scripts/with_dashboard_node.sh npm run build
+          mkdir -p ../../litellm/proxy/_experimental/out
+          cp -r out/. ../../litellm/proxy/_experimental/out/
+          find ../../litellm/proxy/_experimental/out -name '*.html' ! -name index.html | while read -r page; do
+            mkdir -p "${page%.html}"
+            mv "${page}" "${page%.html}/index.html"
+          done
+
+      - name: Prepare the isolated database and IdP
+        run: |
+          umask 077
+          bash .github/e2e-stack/start-idp.sh
+          uv run --no-sync python migrations/run.py > "${RUNNER_TEMP}/mcp-oauth-private/migrations.log" 2>&1
+
+      - name: Run every required OAuth variant without retries
+        run: |
+          umask 077
+          uv run --no-sync pytest -c tests/e2e/pytest.ini tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py \
+            --rootdir=. --reruns 0 --tb=short -o junit_family=xunit1 \
+            --junitxml="${RUNNER_TEMP}/mcp-oauth-private/results.xml" \
+            > "${RUNNER_TEMP}/mcp-oauth-private/pytest.log" 2>&1
+      - name: Report JUnit results and reject skipped or missing cases
+        if: always()
+        run: |
+          uv run --no-sync python .github/e2e-stack/assert_tests_ran.py \
+            "${RUNNER_TEMP}/mcp-oauth-private/results.xml" tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py
+      - name: Remove private login and logs
+        if: always()
+        run: |
+          docker rm -f e2e-keycloak >/dev/null 2>&1 || true
+          rm -rf "${RUNNER_TEMP}/mcp-oauth-private"

.github/workflows/test-merge-smoke.yml

@@ -33,7 +33,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
     env:
       LITELLM_LOCAL_MODEL_COST_MAP: "True"
     steps:
@@ -76,12 +76,6 @@ jobs:
       - name: Import and CLI checks
         run: $IN_NETNS .venv/bin/python .github/scripts/run_merge_smoke.py cli
 
-      - name: Routing and UUID compatibility
-        run: >-
-          $IN_NETNS .venv/bin/python -m pytest
-          tests/unit/router_strategy/test_complexity_router.py::TestCustomDimensions
-          tests/unit/test_uuid_helper.py -q
-
       - name: Proxy startup check
         run: $IN_NETNS .venv/bin/python .github/scripts/run_merge_smoke.py proxy-startup --diagnostics-dir "${RUNNER_TEMP}/smoke-diagnostics"
 
@@ -132,7 +126,7 @@ jobs:
   windows-sdk:
     name: windows-sdk
     runs-on: windows-2025
-    timeout-minutes: 30
+    timeout-minutes: 10
     env:
       LITELLM_LOCAL_MODEL_COST_MAP: "True"
     defaults:
@@ -154,11 +148,8 @@ jobs:
         with:
           version: "0.10.9"
 
-      - name: Set up the Rust bridge
-        uses: ./.github/actions/rust-bridge
-
       - name: Install dependencies
-        run: .github/scripts/uv_sync_with_retries.sh --frozen --group dev --python 3.11
+        run: .github/scripts/uv_sync_with_retries.sh --frozen --no-install-workspace --group dev --python 3.11
 
       - name: Run the Windows tests
         run: uv run --no-sync python -m pytest --tb=short -v tests/windows_tests
@@ -200,36 +191,8 @@ jobs:
       - name: Install the test client dependencies
         run: .github/scripts/uv_sync_with_retries.sh --frozen --no-install-workspace --extra proxy --group dev --python 3.12
 
-      - name: Cache Prisma binaries
+      - name: Build the image from the published package
-        timeout-minutes: 3
+        run: docker build -t litellm-build-from-pip:smoke -f docker/build_from_pip/Dockerfile.build_from_pip .
-        uses: ./.github/actions/cache-prisma-binaries
-
-      - name: Install the published proxy into its own venv
-        run: |
-          uv --no-config venv --python 3.13 /tmp/litellm-pip
-          uv --no-config pip install --python /tmp/litellm-pip/bin/python \
-            "litellm[proxy,proxy-runtime]==1.83.0" \
-            "google-cloud-aiplatform==1.133.0" \
-            "google-genai==1.37.0" \
-            "anthropic[vertex]==0.84.0" \
-            "grpcio==1.78.0" \
-            "prometheus-client==0.20.0" \
-            "langfuse==2.59.7" \
-            "opentelemetry-api==1.28.0" \
-            "opentelemetry-sdk==1.28.0" \
-            "opentelemetry-exporter-otlp==1.28.0" \
-            "ddtrace==4.11.0" \
-            "sentry-sdk==2.21.0" \
-            "mangum==0.17.0" \
-            "azure-ai-contentsafety==1.0.0" \
-            "azure-storage-file-datalake==12.20.0" \
-            "pypdf==6.7.5" \
-            "llm-sandbox==0.3.31" \
-            "detect-secrets==1.5.0" \
-            "prisma==0.11.0" \
-            "openai==2.24.0"
-          export PATH="/tmp/litellm-pip/bin:$PATH"
-          /tmp/litellm-pip/bin/prisma generate --schema=schema.prisma
 
       - name: Start the fake OpenAI endpoint
         run: |
@@ -247,21 +210,23 @@ jobs:
           echo "::add-mask::${key}"
           echo "LITELLM_MASTER_KEY=${key}" >> "${GITHUB_ENV}"
 
-      - name: Start the published proxy
+      - name: Start the proxy container
-        env:
-          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/circle_test
-          LITELLM_LICENSE: bad-license
-          FAKE_OPENAI_API_BASE: http://localhost:8190
-          OTEL_EXPORTER: in_memory
-          USE_DDTRACE: "True"
-          DD_TRACE_OPENAI_ENABLED: "False"
-          GCS_FLUSH_INTERVAL: "1"
-          LITELLM_LOG: ERROR
         run: |
-          cd /tmp/litellm-pip
+          docker run -d \
-          nohup /tmp/litellm-pip/bin/ddtrace-run /tmp/litellm-pip/bin/litellm \
+            -p 4000:4000 \
-            --config "${GITHUB_WORKSPACE}/tests/basic_proxy_startup_tests/build_from_pip_config.yaml" \
+            -e DATABASE_URL=postgresql://postgres:postgres@host.docker.internal:5432/circle_test \
-            --port 4000 > /tmp/litellm-pip.log 2>&1 &
+            -e LITELLM_MASTER_KEY \
+            -e FAKE_OPENAI_API_BASE=http://host.docker.internal:8190 \
+            -e OTEL_EXPORTER=in_memory \
+            -e USE_DDTRACE=True \
+            -e GCS_FLUSH_INTERVAL=1 \
+            -e LITELLM_LOG=ERROR \
+            --add-host host.docker.internal:host-gateway \
+            --name litellm-build-from-pip \
+            -v "${GITHUB_WORKSPACE}/docker/build_from_pip/litellm_config.yaml:/app/config.yaml" \
+            litellm-build-from-pip:smoke \
+            --config /app/config.yaml \
+            --port 4000
           for _ in $(seq 1 150); do
             curl -sf --max-time 5 -o /dev/null http://localhost:4000 && exit 0
             sleep 2
@@ -296,7 +261,7 @@ jobs:
 
       - name: Show the proxy logs
         if: always()
-        run: cat /tmp/litellm-pip.log || true
+        run: docker logs litellm-build-from-pip || true
 
   smoke-passed:
     name: smoke passed

.github/workflows/test-rust-db.yml

@@ -1,116 +0,0 @@
-name: LiteLLM Rust DB
-
-on:
-  push:
-    paths:
-      - "litellm-rust/crates/db/**"
-      - "litellm-rust/Cargo.toml"
-      - "litellm-rust/Cargo.lock"
-      - "litellm-rust/clippy.toml"
-      - "litellm-proxy-extras/litellm_proxy_extras/migrations/**"
-      - "**/schema.prisma"
-      - ".cargo/**"
-      - "rust-toolchain.toml"
-      - "pyproject.toml"
-      - "uv.lock"
-      - ".github/actions/setup-uv-with-retries/**"
-      - ".github/workflows/test-rust-db.yml"
-  pull_request:
-    branches:
-      - main
-      - "litellm_**"
-    paths:
-      - "litellm-rust/crates/db/**"
-      - "litellm-rust/Cargo.toml"
-      - "litellm-rust/Cargo.lock"
-      - "litellm-rust/clippy.toml"
-      - "litellm-proxy-extras/litellm_proxy_extras/migrations/**"
-      - "**/schema.prisma"
-      - ".cargo/**"
-      - "rust-toolchain.toml"
-      - "pyproject.toml"
-      - "uv.lock"
-      - ".github/actions/setup-uv-with-retries/**"
-      - ".github/workflows/test-rust-db.yml"
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-env:
-  CARGO_TERM_COLOR: always
-  RUSTC_WRAPPER: sccache
-  SCCACHE_GHA_ENABLED: "true"
-  SCCACHE_GHA_VERSION: litellm-ci-v1
-  SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
-  CARGO_INCREMENTAL: "0"
-
-jobs:
-  db-queries:
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    services:
-      postgres:
-        image: postgres:16.6@sha256:557fea37a744d5f4c8faab304b0a90858b53ab119735a88c131fd19dab802f36
-        env:
-          POSTGRES_DB: litellm
-          POSTGRES_PASSWORD: postgres
-          POSTGRES_USER: postgres
-        ports:
-          - 5432:5432
-        options: >-
-          --health-cmd "pg_isready -U postgres -d litellm"
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
-    env:
-      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/litellm
-      SQLX_OFFLINE: "false"
-    defaults:
-      run:
-        working-directory: litellm-rust
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - uses: ./.github/actions/setup-uv-with-retries
-        with:
-          version: "0.10.9"
-
-      - run: rustup toolchain install --no-self-update
-
-      - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
-        with:
-          workspaces: litellm-rust
-          cache-on-failure: true
-
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
-      - name: Install Python dependencies
-        working-directory: .
-        run: uv sync --frozen --extra extra-proxy
-
-      - name: Deploy Prisma migrations
-        working-directory: .
-        run: uv run --no-sync prisma migrate deploy --schema litellm-proxy-extras/litellm_proxy_extras/schema.prisma
-
-      - name: Install SQLx CLI
-        run: cargo install sqlx-cli --version 0.9.0 --locked --no-default-features -F postgres
-
-      - name: Check SQLx query metadata
-        working-directory: litellm-rust/crates/db
-        run: cargo sqlx prepare --check
-
-      - name: Test database queries
-        run: cargo test -p litellm-db --locked -- --include-ignored

.github/workflows/test-rust.yml

@@ -4,7 +4,6 @@ on:
   push:
     paths:
       - "litellm-rust/**"
-      - "model_prices_and_context_window.json"
       - "litellm/rust_bridge/**"
       - "scripts/generate_trace_types.py"
       - "scripts/trace_codegen/**"
@@ -17,10 +16,6 @@ on:
       - "litellm/proxy/guardrails/**"
       - "litellm/utils.py"
       - "litellm/ocr/**"
-      - "litellm/chat_completions/dispatch.py"
-      - "litellm/embeddings/dispatch.py"
-      - "litellm/messages/dispatch.py"
-      - "litellm/responses/dispatch.py"
       - "litellm/llms/base_llm/ocr/**"
       - "litellm/llms/custom_httpx/llm_http_handler.py"
       - "Makefile"
@@ -39,7 +34,6 @@ on:
       - "litellm_**"
     paths:
       - "litellm-rust/**"
-      - "model_prices_and_context_window.json"
       - "litellm/rust_bridge/**"
       - "scripts/generate_trace_types.py"
       - "scripts/trace_codegen/**"
@@ -52,10 +46,6 @@ on:
       - "litellm/proxy/guardrails/**"
       - "litellm/utils.py"
       - "litellm/ocr/**"
-      - "litellm/chat_completions/dispatch.py"
-      - "litellm/embeddings/dispatch.py"
-      - "litellm/messages/dispatch.py"
-      - "litellm/responses/dispatch.py"
       - "litellm/llms/base_llm/ocr/**"
       - "litellm/llms/custom_httpx/llm_http_handler.py"
       - "Makefile"
@@ -78,11 +68,6 @@ concurrency:
 
 env:
   CARGO_TERM_COLOR: always
-  RUSTC_WRAPPER: sccache
-  SCCACHE_GHA_ENABLED: "true"
-  SCCACHE_GHA_VERSION: litellm-ci-v1
-  SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
-  CARGO_INCREMENTAL: "0"
 
 jobs:
   rust-lint:
@@ -106,11 +91,6 @@ jobs:
           cache-on-failure: true
           save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - run: cargo clippy --workspace --all-targets --locked -- -D warnings
 
   rust-test:
@@ -153,11 +133,6 @@ jobs:
           cache-on-failure: true
           save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - name: Check generated trace contracts
         working-directory: .
         run: uv run scripts/generate_trace_types.py --check
@@ -204,11 +179,6 @@ jobs:
           cache-on-failure: true
           save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - run: uv build --wheel --out-dir dist
 
       - run: python .github/scripts/verify_linux_native_wheel.py dist/*.whl

.github/workflows/test-terraform-provider.yml

@@ -12,7 +12,6 @@ on:
     paths:
       - "terraform/provider/**"
       - "litellm/proxy/**"
-      - "enterprise/**"
       - ".github/workflows/test-terraform-provider.yml"
 
 permissions:

.github/workflows/test-unit.yml

@@ -5,9 +5,6 @@ on:
     branches:
       - main
       - "litellm_**"
-      - "rc/**"
-      - "stable/**"
-      - "lts/**"
   push:
     branches:
       - main
@@ -45,7 +42,7 @@ jobs:
         id: rust-bridge-key
         if: steps.changes.outputs.decision != 'skip'
         env:
-          RUST_BRIDGE_CACHE_KEY: ${{ runner.os }}-rust-bridge-editable-${{ hashFiles('litellm-rust/Cargo.lock', 'litellm-rust/Cargo.toml', 'litellm-rust/crates/**', 'model_prices_and_context_window.json', 'pyproject.toml', 'packaging/litellm-proxy/pyproject.toml') }}
+          RUST_BRIDGE_CACHE_KEY: ${{ runner.os }}-rust-bridge-editable-${{ hashFiles('litellm-rust/Cargo.lock', 'litellm-rust/Cargo.toml', 'litellm-rust/crates/**', 'pyproject.toml') }}
         run: echo "key=$RUST_BRIDGE_CACHE_KEY" >> "$GITHUB_OUTPUT"
 
       - name: Set up Python
@@ -76,21 +73,9 @@ jobs:
         timeout-minutes: 5
         uses: ./.github/actions/cache-cargo-build
 
-      - name: Set up sccache
-        if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - name: Build editable Rust bridge
         if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
         timeout-minutes: 15
-        env:
-          RUSTC_WRAPPER: sccache
-          SCCACHE_GHA_ENABLED: "true"
-          SCCACHE_GHA_VERSION: litellm-ci-v1
-          SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
-          CARGO_INCREMENTAL: "0"
         run: |
           mkdir -p rust-bridge-dist
           uv run --no-project --with maturin==1.15.0 python -c 'import maturin; maturin.build_editable("rust-bridge-dist")'
@@ -98,18 +83,6 @@ jobs:
           mkdir -p rust-bridge-dist/litellm/rust_bridge
           cp litellm/rust_bridge/_native.abi3.so rust-bridge-dist/litellm/rust_bridge/_native.abi3.so
 
-      - name: Build the gateway wheel once for test shards
-        if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
-        timeout-minutes: 15
-        working-directory: packaging/litellm-proxy
-        env:
-          RUSTC_WRAPPER: sccache
-          SCCACHE_GHA_ENABLED: "true"
-          SCCACHE_GHA_VERSION: litellm-ci-v1
-          SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
-          CARGO_INCREMENTAL: "0"
-        run: uv run --no-project --with maturin==1.15.0 maturin build --profile dev --strip --out ../../rust-bridge-dist
-
       - name: Upload Rust bridge artifact
         if: steps.changes.outputs.decision != 'skip'
         timeout-minutes: 10
@@ -164,10 +137,8 @@ jobs:
         include:
           - shard: core-utils
             test-path: |-
-              tests/unit/accounting
               tests/unit/batch_completion
               tests/unit/decisions
-              tests/unit/extraction
               tests/unit/litellm_core_utils
               tests/unit/systemone
             python-version: "3.12"
@@ -182,7 +153,6 @@ jobs:
               tests/unit/google_genai
               tests/unit/router_strategy
               tests/unit/router_utils
-              tests/unit/redis
               tests/unit/proxy/common_utils/test_cache_aware_routing.py
               tests/unit/enterprise/enterprise_callbacks/send_emails
               tests/unit/enterprise/proxy/test_afile_retrieve_returns_unified_id.py
@@ -245,17 +215,14 @@ jobs:
 
           - shard: llms-providers
             test-path: |-
-              tests/unit/llms/test_api_base_trailing_slash_routes.py
               tests/unit/llms/test_cache_control_and_reasoning.py
               tests/unit/llms/test_custom_llm.py
               tests/unit/llms/test_file_content_block.py
               tests/unit/llms/test_file_search_responses.py
-              tests/unit/llms/test_get_complete_url_trailing_slash.py
               tests/unit/llms/test_lifecycle_fix.py
               tests/unit/llms/test_oss_decision.py
               tests/unit/llms/test_polling_url_origin_match.py
               tests/unit/llms/test_predibase_transformation.py
-              tests/unit/llms/test_subscription_accounts.py
               tests/unit/llms/a2a
               tests/unit/llms/aiml
               tests/unit/llms/aiohttp_openai
@@ -265,7 +232,6 @@ jobs:
               tests/unit/llms/azure_ai
               tests/unit/llms/base_llm
               tests/unit/llms/baseten
-              tests/unit/llms/bigmodel
               tests/unit/llms/black_forest_labs
               tests/unit/llms/brave
               tests/unit/llms/bytez
@@ -274,9 +240,7 @@ jobs:
               tests/unit/llms/chatgpt
               tests/unit/llms/clarifai
               tests/unit/llms/claude_code
-              tests/unit/llms/clf_ai_gateway
               tests/unit/llms/cloudflare
-              tests/unit/llms/codestral
               tests/unit/llms/codex
               tests/unit/llms/cohere
               tests/unit/llms/cometapi
@@ -296,14 +260,12 @@ jobs:
               tests/unit/llms/e2b
               tests/unit/llms/edenai
               tests/unit/llms/elevenlabs
-              tests/unit/llms/empiriolabs
               tests/unit/llms/exa_ai
               tests/unit/llms/fal_ai
               tests/unit/llms/fastcrw
               tests/unit/llms/featherless_ai
               tests/unit/llms/firecrawl
               tests/unit/llms/fireworks_ai
-              tests/unit/llms/friendliai
               tests/unit/llms/gdc
               tests/unit/llms/gemini
               tests/unit/llms/gigachat
@@ -336,7 +298,6 @@ jobs:
               tests/unit/llms/modelscope
               tests/unit/llms/mongodb
               tests/unit/llms/moonshot
-              tests/unit/llms/morph
               tests/unit/llms/nadir
               tests/unit/llms/nebius
               tests/unit/llms/neosantara
@@ -353,7 +314,6 @@ jobs:
               tests/unit/llms/openai_like
               tests/unit/llms/opencode
               tests/unit/llms/opensandbox
-              tests/unit/llms/opensearch
               tests/unit/llms/openrouter
               tests/unit/llms/ovhcloud
               tests/unit/llms/parallel_ai
@@ -361,7 +321,6 @@ jobs:
               tests/unit/llms/pass_through
               tests/unit/llms/perplexity
               tests/unit/llms/pg_vector
-              tests/unit/llms/pi
               tests/unit/llms/publicai
               tests/unit/llms/ragflow
               tests/unit/llms/recraft
@@ -397,7 +356,6 @@ jobs:
               tests/unit/llms/voyage
               tests/unit/llms/wandb
               tests/unit/llms/watsonx
-              tests/unit/llms/webiq
               tests/unit/llms/xai
               tests/unit/llms/xinference
               tests/unit/llms/you_com
@@ -433,7 +391,6 @@ jobs:
 
           - shard: router
             test-path: >-
-              tests/unit/router_rust
               tests/unit/test_router
             python-version: "3.12"
             workers: 4
@@ -453,18 +410,15 @@ jobs:
               tests/unit/embeddings
               tests/unit/endpoints
               tests/unit/files
-              tests/unit/fine_tuning
               tests/unit/images
               tests/unit/interactions
               tests/unit/messages
               tests/unit/ocr
               tests/unit/passthrough
               tests/unit/rag
-              tests/unit/cookbook
               tests/unit/realtime_api
               tests/unit/rerank_api
               tests/unit/search
-              tests/unit/text_completions
               tests/unit/vector_stores
               tests/unit/videos
             python-version: "3.12"
@@ -476,13 +430,6 @@ jobs:
 
           - shard: rust-bridge-harness
             test-path: >-
-              tests/test_litellm_rust/test_dispatch.py
-              tests/test_litellm_rust/test_public_entries.py
-              tests/test_litellm_rust/chat_completions/test_dispatch.py
-              tests/test_litellm_rust/embeddings/test_dispatch.py
-              tests/test_litellm_rust/messages/test_dispatch.py
-              tests/test_litellm_rust/ocr/test_dispatch.py
-              tests/test_litellm_rust/responses/test_dispatch.py
               tests/unit/compression
               tests/unit/harness
               tests/unit/rust_bridge
@@ -597,7 +544,6 @@ jobs:
               tests/unit/proxy/rerank_endpoints
               tests/unit/proxy/realtime_endpoints
               tests/unit/proxy/ui_crud_endpoints
-              tests/unit/proxy/deps
               tests/unit/proxy/config_resolvers
               tests/unit/proxy/utils
             python-version: "3.12"
@@ -619,13 +565,14 @@ jobs:
 
           - shard: mcp-elicitation
             test-path: |-
-              tests/unit/mcp_client/test_mcp_client.py
+              --override-ini=pythonpath=tests
-              tests/unit/proxy/mcp_server/test_capabilities.py
+              tests/unit/experimental_mcp_client/test_mcp_client.py
-              tests/unit/proxy/mcp_server/test_interactions.py
+              tests/unit/proxy/_experimental/mcp_server/test_capabilities.py
-              tests/unit/proxy/mcp_server/test_mcp_elicitation_handler.py
+              tests/unit/proxy/_experimental/mcp_server/test_interactions.py
-              tests/unit/proxy/mcp_server/test_mcp_server_manager.py
+              tests/unit/proxy/_experimental/mcp_server/test_mcp_elicitation_handler.py
-              tests/unit/proxy/mcp_server/test_mcp_server_tool_calls_and_headers.py
+              tests/unit/proxy/_experimental/mcp_server/test_mcp_server_manager.py
-              tests/unit/proxy/mcp_server/test_operations.py
+              tests/unit/proxy/_experimental/mcp_server/test_mcp_server_tool_calls_and_headers.py
+              tests/unit/proxy/_experimental/mcp_server/test_operations.py
             python-version: "3.12"
             workers: 4
             reruns: 0
@@ -633,32 +580,34 @@ jobs:
             job-timeout-minutes: 60
             dist: loadscope
 
-          - shard: proxy-infra-db-spend
+          - shard: mcp-interactions
+            test-path: |-
+              --override-ini=pythonpath=tests
+              tests/integration/mcp/test_interactions.py
+            python-version: "3.12"
+            workers: 4
+            reruns: 0
+            timeout-minutes: 20
+            job-timeout-minutes: 60
+            dist: load
+
+          - shard: proxy-infra
             test-path: |-
               tests/unit/proxy/db
               --ignore=tests/unit/proxy/db/db_transaction_queue/test_e2e_pod_lock_manager.py
               --ignore=tests/unit/proxy/db/test_update_daily_tag_spend.py
+              tests/unit/proxy/middleware
+              --ignore=tests/unit/proxy/middleware/test_request_size_limit_middleware.py
               tests/unit/proxy/spend_tracking
               --ignore=tests/unit/proxy/spend_tracking/test_search_api_logging.py
+              tests/unit/proxy/pass_through_endpoints
+              tests/unit/proxy/experimental
               tests/unit/proxy/common_utils
               --ignore=tests/unit/proxy/common_utils/test_cache_aware_routing.py
               --ignore=tests/unit/proxy/common_utils/test_check_batch_cost.py
               --ignore=tests/unit/proxy/common_utils/test_check_responses_cost.py
               --ignore=tests/unit/proxy/common_utils/test_proxy_encrypt_decrypt.py
               --ignore=tests/unit/proxy/common_utils/test_realtime_cache.py
-            python-version: "3.12"
-            workers: 4
-            reruns: 0
-            timeout-minutes: 20
-            job-timeout-minutes: 60
-            dist: loadscope
-
-          - shard: proxy-infra-routing
-            test-path: |-
-              tests/unit/proxy/middleware
-              --ignore=tests/unit/proxy/middleware/test_request_size_limit_middleware.py
-              tests/unit/proxy/pass_through_endpoints
-              tests/unit/proxy/mcp_server/test_tool_registry.py
               tests/unit/proxy/enterprise_billing
               tests/unit/proxy/types_utils
               tests/unit/proxy/logging_endpoints
@@ -721,7 +670,6 @@ jobs:
           - shard: proxy-extras
             test-path: >-
               tests/unit/litellm_proxy_extras
-              tests/unit/db_scripts
             python-version: "3.12"
             workers: 2
             reruns: 0
@@ -760,6 +708,7 @@ jobs:
               tests/unit/responses/litellm_completion_transformation
               tests/unit/responses/test_additional_tools.py
               tests/unit/responses/test_custom_tool_call.py
+              tests/unit/responses/test_dispatch.py
               tests/unit/responses/test_metadata_codex_callback.py
               tests/unit/responses/test_no_duplicate_spend_logs.py
               tests/unit/responses/test_null_test_fix.py
@@ -860,14 +809,14 @@ jobs:
 
           - shard: mcp-oauth
             test-path: |-
-              tests/unit/proxy/mcp_server/test_discoverable_endpoints.py
+              tests/unit/proxy/_experimental/mcp_server/test_discoverable_endpoints.py
-              tests/unit/proxy/mcp_server/test_mcp_server_manager.py
+              tests/unit/proxy/_experimental/mcp_server/test_mcp_server_manager.py
-              tests/unit/proxy/mcp_server/test_db_credentials.py
+              tests/unit/proxy/_experimental/mcp_server/test_db_credentials.py
-              tests/unit/proxy/mcp_server/test_gateway_dcr_flow.py
+              tests/unit/proxy/_experimental/mcp_server/test_gateway_dcr_flow.py
-              tests/unit/proxy/mcp_server/test_mcp_oauth_passthrough_tools.py
+              tests/unit/proxy/_experimental/mcp_server/test_mcp_oauth_passthrough_tools.py
-              tests/unit/proxy/mcp_server/test_mcp_max_concurrent_requests.py
+              tests/unit/proxy/_experimental/mcp_server/test_mcp_max_concurrent_requests.py
-              tests/unit/proxy/mcp_server/faults/test_list_outcomes.py
+              tests/unit/proxy/_experimental/mcp_server/faults/test_list_outcomes.py
-              tests/unit/proxy/mcp_server/outbound_credentials
+              tests/unit/proxy/_experimental/mcp_server/outbound_credentials
             python-version: "3.12"
             workers: 4
             reruns: 2
@@ -877,21 +826,19 @@ jobs:
 
           - shard: mcp-server
             test-path: |-
-              tests/unit/mcp_client
               tests/unit/experimental_mcp_client
-              tests/unit/proxy/mcp_server
               tests/unit/proxy/_experimental/mcp_server
               tests/unit/responses/mcp
-              --ignore=tests/unit/mcp_client/test_mcp_client.py
+              --ignore=tests/unit/experimental_mcp_client/test_mcp_client.py
-              --ignore=tests/unit/proxy/mcp_server/test_capabilities.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_capabilities.py
-              --ignore=tests/unit/proxy/mcp_server/test_interactions.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_interactions.py
-              --ignore=tests/unit/proxy/mcp_server/test_mcp_elicitation_handler.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_mcp_elicitation_handler.py
-              --ignore=tests/unit/proxy/mcp_server/test_mcp_server_manager.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_mcp_server_manager.py
-              --ignore=tests/unit/proxy/mcp_server/test_mcp_server_tool_calls_and_headers.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_mcp_server_tool_calls_and_headers.py
-              --ignore=tests/unit/proxy/mcp_server/test_operations.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_operations.py
-              --ignore=tests/unit/proxy/mcp_server/test_discoverable_endpoints.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_discoverable_endpoints.py
-              --ignore=tests/unit/proxy/mcp_server/test_db_credentials.py
+              --ignore=tests/unit/proxy/_experimental/mcp_server/test_db_credentials.py
-              --ignore=tests/unit/proxy/mcp_server/outbound_credentials
+              --ignore=tests/unit/proxy/_experimental/mcp_server/outbound_credentials
             python-version: "3.12"
             workers: 4
             reruns: 0
@@ -1004,7 +951,7 @@ jobs:
         timeout-minutes: 3
         with:
           persist-credentials: false
-          fetch-depth: 1
+          fetch-depth: ${{ matrix.shard == 'db-and-spend' && '0' || '1' }}
 
       - name: Detect relevant changes
         id: changes
@@ -1037,37 +984,27 @@ jobs:
         with:
           artifact: ${{ needs.rust-bridge.outputs.artifact }}
 
-      - name: Install dependencies
+      - name: Install dependencies and check schema
         if: steps.changes.outputs.decision != 'skip'
-        timeout-minutes: 5
+        timeout-minutes: 8
         env:
           RUST_BRIDGE_ARTIFACT: ${{ needs.rust-bridge.outputs.artifact }}
+          UNIT_SHARD: ${{ matrix.shard }}
+          SCHEMA_BASE_REF: origin/${{ github.event.repository.default_branch }}
         run: |
           diff -u model_prices_and_context_window.json litellm/model_prices_and_context_window_backup.json
           if [ -z "$RUST_BRIDGE_ARTIFACT" ]; then
-            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils
+            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra caching --extra extra_proxy --extra proxy-runtime --extra utils
           else
-            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project --no-install-package litellm-proxy
+            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra caching --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project
             uv pip install --no-deps --python .venv/bin/python rust-bridge-dist/*.whl
             cp rust-bridge-dist/litellm/rust_bridge/_native.abi3.so litellm/rust_bridge/_native.abi3.so
             uv run --no-sync python -c "import importlib.metadata; import litellm.rust_bridge._native; print(importlib.metadata.version('litellm'))"
           fi
           uv run --no-sync python -c 'import os, sys; print(sys.version); assert f"{sys.version_info.major}.{sys.version_info.minor}" == os.environ["UV_PYTHON"]'
-
+          if [ "$UNIT_SHARD" = "db-and-spend" ]; then
-      - name: Check schema.prisma against the merge base
+            uv run --no-sync python scripts/check_schema_compatibility.py --base-ref "$SCHEMA_BASE_REF"
-        if: steps.changes.outputs.decision != 'skip' && matrix.shard == 'db-and-spend' && github.event_name == 'pull_request'
+          fi
-        timeout-minutes: 3
-        env:
-          GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
-          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
-        run: |
-          base_ref=$(gh api "repos/${GITHUB_REPOSITORY}/branches/${EVENT_BASE_REF}" --jq .name) || base_ref=$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq .base.ref)
-          merge_base=$(gh api "repos/${GITHUB_REPOSITORY}/compare/${base_ref}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
-          test -n "$merge_base"
-          git fetch --no-tags --depth=1 origin "$merge_base"
-          uv run --no-sync python scripts/check_schema_compatibility.py --base-commit "$merge_base"
 
       - name: Cache Prisma binaries
         if: steps.changes.outputs.decision != 'skip'
@@ -1204,20 +1141,18 @@ jobs:
         env:
           CI: "true"
           GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          EVENT_BASE_REF: ${{ github.event.pull_request.base.ref }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
           full_suite() { npm run test -- --run --pool forks --maxWorkers=14; }
 
-          if [ -z "$PR_NUMBER" ]; then
+          if [ -z "$BASE_SHA" ]; then
             echo "Push to $GITHUB_REF_NAME: running the full suite"
             full_suite
             exit 0
           fi
 
-          base_ref=$(gh api "repos/${GITHUB_REPOSITORY}/branches/${EVENT_BASE_REF}" --jq .name) || base_ref=$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq .base.ref)
+          merge_base=$(gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
-          merge_base=$(gh api "repos/${GITHUB_REPOSITORY}/compare/${base_ref}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
           test -n "$merge_base"
           git fetch --no-tags --depth=1 origin "$merge_base" "$HEAD_SHA"
           changed_files=()
@@ -1304,7 +1239,7 @@ jobs:
       - name: Install dependencies
         if: steps.changes.outputs.decision != 'skip'
         run: |
-          .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard
+          .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
 
       - name: Cache Prisma binaries
         if: steps.changes.outputs.decision != 'skip'
@@ -1353,20 +1288,19 @@ jobs:
 
       - name: Run unit tests
         run: |
-          chart=helm/litellm
+          for chart in helm/litellm-helm helm/litellm; do
-          declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
+            declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
-          output="$(mktemp)"
+            output="$(mktemp)"
-          helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
+            helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
-          executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
+            executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
-          if [ "$declared" != "$executed" ]; then
+            if [ "$declared" != "$executed" ]; then
-            echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
+              echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
-            exit 1
+              exit 1
-          fi
+            fi
-          echo "$chart: all $declared declared test suites ran"
+            echo "$chart: all $declared declared test suites ran"
+          done
   openai-sdk-compat:
     name: OpenAI ${{ matrix.openai-version }} compatibility
-    needs: rust-bridge
-    if: ${{ !cancelled() }}
     permissions:
       contents: read
       pull-requests: read
@@ -1403,18 +1337,10 @@ jobs:
         with:
           version: "0.10.9"
 
-      - name: Set up the Rust bridge
-        if: steps.changes.outputs.decision != 'skip'
-        timeout-minutes: 5
-        uses: ./.github/actions/rust-bridge
-        with:
-          artifact: ${{ needs.rust-bridge.outputs.artifact }}
-
       - name: Install dependencies
         if: steps.changes.outputs.decision != 'skip'
         run: |
           .github/scripts/uv_sync_with_retries.sh --frozen --group dev --group proxy-dev --extra proxy --no-install-workspace
-          cp rust-bridge-dist/litellm/rust_bridge/_native.* litellm/rust_bridge/
           uv pip install "openai==${OPENAI_VERSION:?}" --exclude-newer-package openai=2026-10-07T00:00:00Z
           uv run --no-sync python -c 'import os, openai; assert openai.__version__ == os.environ["OPENAI_VERSION"], openai.__version__'
 
@@ -1506,9 +1432,9 @@ jobs:
           RUST_BRIDGE_ARTIFACT: ${{ needs.rust-bridge.outputs.artifact }}
         run: |
           if [ -z "$RUST_BRIDGE_ARTIFACT" ]; then
-            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils
+            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra caching --extra extra_proxy --extra proxy-runtime --extra utils
           else
-            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project --no-install-package litellm-proxy
+            .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra caching --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project
             uv pip install --no-deps --python .venv/bin/python rust-bridge-dist/*.whl
             cp rust-bridge-dist/litellm/rust_bridge/_native.abi3.so litellm/rust_bridge/_native.abi3.so
           fi
@@ -1537,6 +1463,7 @@ jobs:
             tests/unit/caching/test_redis_connection_pool.py \
             tests/unit/caching/test_redis_cluster_cache.py \
             tests/unit/caching/test_evicted_client_closer.py \
+            tests/integration/sdk/test_redis_cluster_iam_auth.py \
             --tb=short -vv -rs -p no:rerunfailures -p no:pytest-retry --timeout=120 --durations=20 \
             --cov=./litellm --cov-report=xml:coverage.xml
 

.github/workflows/weekly_load_anomaly.yml

@@ -0,0 +1,85 @@
+name: "Weekly Load Anomaly Check"
+
+on:
+  schedule:
+    - cron: "0 12 * * 6"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  weekly-load-anomaly:
+    if: github.event_name != 'schedule' || github.repository == 'BerriAI/litellm'
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    services:
+      postgres:
+        image: postgres:16.6
+        env:
+          POSTGRES_USER: llmproxy
+          POSTGRES_PASSWORD: dbpassword9090
+          POSTGRES_DB: litellm
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U llmproxy"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+    env:
+      DATABASE_URL: postgresql://llmproxy:dbpassword9090@localhost:5432/litellm
+      LITELLM_MASTER_KEY: sk-weekly-anomaly-check
+      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+      AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Set up uv
+        uses: ./.github/actions/setup-uv-with-retries
+        with:
+          version: "0.10.9"
+
+      - name: Cache the Rust build
+        uses: ./.github/actions/cache-cargo-build
+
+      - name: Install dependencies
+        run: |
+          .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra proxy
+
+      - name: Cache Prisma binaries
+        uses: ./.github/actions/cache-prisma-binaries
+
+      - name: Generate Prisma client
+        run: |
+          uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
+
+      - name: Start the proxy
+        run: |
+          nohup uv run --no-sync litellm --config tests/e2e/perf/weekly_anomaly_config.yml --port 4000 > proxy.log 2>&1 &
+          for _ in $(seq 1 90); do
+            if curl -fs http://localhost:4000/health/liveliness > /dev/null; then
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "proxy never became live"
+          tail -n 100 proxy.log
+          exit 1
+
+      - name: Run the weekly session anomaly test
+        env:
+          E2E_WEEKLY_ANOMALY: "1"
+        run: |
+          uv run --no-sync pytest tests/e2e/perf/test_weekly_session_anomaly_e2e.py -v --tb=short -rA
+
+      - name: Show proxy log on failure
+        if: failure()
+        run: tail -n 300 proxy.log

.github/workflows/zizmor.yml

@@ -27,15 +27,10 @@ jobs:
         with:
           persist-credentials: false
 
-      - name: Install zizmor
-        run: |
-          curl -fsSL --retry 3 -o "$RUNNER_TEMP/zizmor.tar.gz" \
-            https://github.com/zizmorcore/zizmor/releases/download/v1.31.1/zizmor-x86_64-unknown-linux-gnu.tar.gz
-          echo "d185f726bdaf92df78273217dfdd24ebcd6b0ef48cf3ce776ff5762b6e138c99  $RUNNER_TEMP/zizmor.tar.gz" | sha256sum -c -
-          tar xzf "$RUNNER_TEMP/zizmor.tar.gz" -C "$RUNNER_TEMP" zizmor
-
       - name: Run zizmor
-        env:
+        uses: zizmorcore/zizmor-action@5f14fd08f7cf1cb1609c1e344975f152c7ee938d # v0.5.6
-          GH_TOKEN: ${{ github.token }}
+        with:
-        run: |
+          version: "1.24.1"
-          "$RUNNER_TEMP/zizmor" --no-ignores --no-config --min-severity=medium --format=github .
+          min-severity: medium
+          advanced-security: false
+          annotations: true

.gitignore

@@ -61,7 +61,7 @@ ui/litellm-dashboard/next-env.d.ts
 *.tsbuildinfo
 ui/litellm-dashboard/package.json
 ui/litellm-dashboard/package-lock.json
-helm/litellm/charts/*.tgz
+helm/litellm-helm/*.tgz
 helm/*.tgz
 litellm/proxy/vertex_key.json
 **/.vim/
@@ -105,7 +105,7 @@ litellm_config.yaml
 .cursor
 litellm/proxy/to_delete_loadtest_work/*
 update_model_cost_map.py
-tests/unit/proxy/mcp_server/test_mcp_server_manager.py
+tests/unit/proxy/_experimental/mcp_server/test_mcp_server_manager.py
 scripts/test_vertex_ai_search.py
 LAZY_LOADING_IMPROVEMENTS.md
 STABILIZATION_TODO.md

.grype.yaml

@@ -11,12 +11,3 @@ ignore:
     package:
       name: zlib
       type: apk
-  - vulnerability: GHSA-v4x8-3wr5-3wq8
-    reason: >-
-      filelock ReadWriteLock advisory, and no package in the image takes ReadWriteLock;
-      the fix (4.2.1) is blocked by litellm's own filelock<4.0 cap until #48635 or #48802 lifts it,
-      and the image then moves off 3.32.7, which retires this version-scoped rule
-    package:
-      name: filelock
-      version: 3.32.7
-      type: python

AGENTS.md

@@ -29,9 +29,7 @@ A test must only fail when litellm code changes. Never pin facts we don't own (a
 
 `tests/unit/` mirrors `litellm/` in a parallel path (see `tests/unit/AGENTS.md`). Name tests `test_<filename>.py`, but always match the existing test file in the directory you touch — many provider dirs use longer descriptive names (e.g. `test_anthropic_chat_transformation.py`) to avoid ambiguity across sibling folders. For bug fixes, extend the existing mapped test file rather than creating a new one. Only create a new test file for a new feature (provider, endpoint, or transformation module) that has no mapped test yet, following that directory's naming convention (or `test_<filename>.py` if you're the first test there). One focused regression test beats many shallow ones
 
-End-to-end tests belong in `tests/e2e/`. A rule about one directory's files, like its harness or which CI runs it, goes in that directory's `AGENTS.md`, never only in a README or here. Read a directory's `AGENTS.md` before editing files in it
+End-to-end tests belong in `tests/e2e/` and must follow the harness conventions documented in that directory's `AGENTS.md`
-
-Before you build a new way to move data between the endpoints, the provider transformations, and cost tracking or logging, read the architecture decision records in `docs/adrs/`, which say why those layers are shaped the way they are
 
 When creating PRs, target the repository's current default branch for both internal and external / OSS contributions. Check it with `python3 scripts/default_branch.py --branch` instead of assuming a branch name or relying on cached `origin/HEAD`
 
@@ -41,17 +39,14 @@ Same applies for filing bug reports and feature requests, with .github/ISSUE_TEM
 
 If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just drop the section
 
-Run QA proxies with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`, and swap its `--config` for a copy of `litellm/proxy/dev_config.yaml` that points both `router_settings` (`redis_host`, `redis_port`) and `general_settings.coordination_redis` (`host`, `port`) at one dedicated Redis instance, with credentials when required, so the workers share router state and spend, budget, and rate-limit counters. Check that two worker PIDs are serving, and use the same worker count and Redis setup for the before and after proof. When the PR is about `--run_gunicorn`, `--run_hypercorn`, or `--run_granian`, which `--reload` does not support, drop `--reload`. When it is about behavior without Redis, drop Redis. When the proxy prints `"workers" flag is ignored when reloading is enabled`, that checkout or its uvicorn reloads only one worker, so drop the two-worker check for it. Say in the proof which part you dropped and why
+Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`. The Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into. I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it
-
-Simple PRs may skip before/after e2e when the full diff is mechanical and focused checks completely prove it, such as #49030 restoring an existing helper name. Trace affected callers and run relevant tests/static checks at the tip. For a bug fix, establish that the regression catches the original defect. Size alone never qualifies. New behavior, API/provider semantics, auth, billing, persistence, concurrency or uncertain integration risk need live e2e. In Screenshots / Proof of Fix, record `E2E exception at <tip SHA>`, scope, rationale, commands and results. Tests count as proof only for this exception. Reassess changed diffs, retain CI/bot gates, and honor explicit e2e requests (2026-10-28, unnecessary live QA for #49030)
-
-For other PRs, proof must show the end user's real flow before and after, with commands and output against a live proxy and real provider calls where applicable. The body alone must let a reviewer rerun it, with the proxy command, config, env var names and client code exactly as run, and long blocks collapsed in <details>. Pytest alone is insufficient. For UI or interactive coding-tool flows, drive that surface and embed before/after screenshots, with URLs, clicks and input fields a reviewer can reproduce
 
 If you ever write any human-facing text (pull requests, issues, commit messages, discussion posts, github comments, release notes, docs, etc.), always follow these guidelines to sound less AI-y:
 - don't use emojis
 - don't use "—". Don't use ";" or ":" unless it would be nonsensical not to. Instead, reach for ",", ".", conjunction words, etc. in descending order of preference: vary among them, weighted toward the front of the list, and skip "," where it would cause a comma splice or the sentence is getting long. Overusing any one of them also feels AI-y. A word cap does not penalize you for adding more sentences
 - don't use the pattern "It's not X, it's Y", "You're not X, you're Y", etc.
 - unless explicitly asked, don't use bulleted or numbered lists unless it would be nonsensical not to. Instead, prefer prose
+- don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: if you're adding new line(s) before the next sentence, don't add a ".". Exception is that docs (i.e., litellm-docs) should have trailing "."
 - don't use →. Instead, prefer not to use arrows, and if need be, use -> instead
 - use plain, simple, everyday engineering language: the common phrase engineers actually say over rare compact phrasing, in grammatically complete sentences. When explicitly asked to use bullets or ordered lists and structure legitimately helps the reader, prefer nested bullets (any depth is fine) over dense lines in a flat structure
 
@@ -77,7 +72,7 @@ When referencing or running models (coding, QA'ing, writing docs, writing tests,
 
 Always pull before starting any work. The checkout or worktree may be sitting on a stale branch
 
-Internal contributors branch a new PR off the repository's current default branch, with a branch name prefixed with litellm_
+If you're an internal contributor, when creating a new PR, the typical flow is to branch off the repository's current default branch and create a branch prefixed with litellm_. Do not create a branch prefixed with claude/ and generally do not have / in your branch names
 
 Do not add `Co-Authored-By: Claude` or any Claude attribution to commit messages. Never use a `claude/` prefix or put a `/` in a branch name. Do not add "Generated with Claude Code" (or any similar attribution) to PR descriptions or comments. Do not create a new PR/branch off the existing PR to fix/add something that is related and could've just been committed directly to the existing PR's branch
 
@@ -91,8 +86,6 @@ Do not put names of customers or customer company names in code, PR descriptions
 
 CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
 
-For database operations, connection ownership and proxy dependency wiring, follow [database-access](.agents/skills/database-access/SKILL.md)
-
 Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No `UPDATE`, `DELETE` or `MERGE`, and no `INSERT ... SELECT`: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. `tests/code_coverage_tests/check_migrations_no_data_rewrites.py` enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement `-- data-migration-ok: <what bounds it>`
 
 Follow these coding conventions for new/updated code (a three-line fix in a legacy file shouldn't trigger huge drive-by refactors):
@@ -112,7 +105,6 @@ Follow these coding conventions for new/updated code (a three-line fix in a lega
 - No file sprawl: deliberate file and folder structure
 - Standard over hand-rolled: use the official SDK or a library where one exists. Where none does, follow industry standards instead of inventing local conventions
 - API-fragmentation-aware: when logic must branch on which API surface produced or consumes data (e.g. chat completions vs Anthropic Messages vs Responses API shapes), proactively look for an existing shared helper (e.g. `litellm_core_utils/prompt_templates/factory.py`) before writing per-surface parsing in the new module. If none exists, add one there instead of duplicating the same format-detection logic in every new guardrail/integration
-- Current default for new model names: logic that picks behavior from a model-name table (tokenizer, message framing, supported params) gives a provider's model the table does not list yet that provider's current behavior, never its oldest, since providers ship names faster than any table is updated
 
 Follow conventional commits for commit names and PR titles
 

ARCHITECTURE.md

@@ -2,8 +2,6 @@
 
 This document helps contributors understand where to make changes in LiteLLM.
 
-For why a layer is shaped the way it is, read the architecture decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) (Python SDK and proxy) and [`litellm-rust/docs/adrs/`](litellm-rust/docs/adrs/AGENTS.md) (Rust core)
-
 ---
 
 ## How It Works
@@ -241,8 +239,6 @@ graph LR
 7. `DBSpendUpdateWriter.update_database()` queues spend increments to Redis
 8. Background job `update_spend` flushes queued spend to PostgreSQL every 60s
 
-Cost lookup falls back to the response model when the provider-reported name cannot be priced. Explicitly registered zero rates remain zero, and custom deployment pricing or an explicit `base_model` retain precedence
-
 ### Data Access Layer (Models & Repositories)
 
 Database entities and the operations on them live in two packages at the root of `litellm/` so both the gateway (`proxy/`) and the SDK can use them without importing proxy internals:
@@ -336,13 +332,13 @@ Each translation is isolated in its own file, making it easy to test and modify
 |--------------|----------|------------------|
 | `/v1/chat/completions` | Anthropic | `llms/anthropic/chat/transformation.py` |
 | `/v1/chat/completions` | Bedrock Converse | `llms/bedrock/chat/converse_transformation.py` |
-| `/v1/chat/completions` | Bedrock Invoke | `llms/bedrock/chat/invoke_transformations/anthropic_claude_transformation.py` |
+| `/v1/chat/completions` | Bedrock Invoke | `llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py` |
 | `/v1/chat/completions` | Gemini | `llms/gemini/chat/transformation.py` |
 | `/v1/chat/completions` | Vertex AI | `llms/vertex_ai/gemini/transformation.py` |
 | `/v1/chat/completions` | OpenAI | `llms/openai/chat/gpt_transformation.py` |
-| `/v1/messages` (passthrough) | Anthropic | `llms/anthropic/messages/native/transformation.py` |
+| `/v1/messages` (passthrough) | Anthropic | `llms/anthropic/pass_through/messages/transformation.py` |
-| `/v1/messages` (passthrough) | Bedrock | `llms/bedrock/messages/invoke_transformations/anthropic_claude_transformation.py` |
+| `/v1/messages` (passthrough) | Bedrock | `llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py` |
-| `/v1/messages` (passthrough) | Vertex AI | `llms/vertex_ai/vertex_ai_partner_models/anthropic/messages/transformation.py` |
+| `/v1/messages` (passthrough) | Vertex AI | `llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py` |
 | Passthrough endpoints | All | `proxy/pass_through_endpoints/llm_provider_handlers/` |
 
 ### Example: Debugging prompt caching
@@ -350,7 +346,7 @@ Each translation is isolated in its own file, making it easy to test and modify
 If `/v1/messages` → Bedrock Converse prompt caching isn't working but Bedrock Invoke works:
 
 1. **Bedrock Converse translation**: `llms/bedrock/chat/converse_transformation.py`
-2. **Bedrock Invoke translation**: `llms/bedrock/chat/invoke_transformations/anthropic_claude_transformation.py`
+2. **Bedrock Invoke translation**: `llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py`
 3. Compare how each handles `cache_control` in `transform_request()`
 
 ### How translations work

CONTRIBUTING.md

@@ -8,7 +8,6 @@ Here are the core requirements for any PR submitted to LiteLLM:
 
 - [ ] **Sign the Contributor License Agreement (CLA)** - [see details](#contributor-license-agreement-cla)
 - [ ] **Keep scope isolated** - Your changes should address 1 specific problem at a time
-- [ ] **Prove it works end to end** - If your change affects what the proxy or SDK does, paste the commands you ran against a live proxy and their output in the PR - [see details](#proving-your-change-works-end-to-end)
 
 #### Proxy (Backend) PRs
 
@@ -40,12 +39,12 @@ git clone https://github.com/YOUR_USERNAME/litellm.git
 cd litellm
 
 # Create a new branch for your feature (see "Commit and Branch Conventions" below)
-git checkout -b litellm_your_feature
+git checkout -b feature/your-feature
 
 # Install development dependencies
 make install-dev
 
-# Install git hooks that enforce commit conventions (one-time, opt-in)
+# Install git hooks that enforce commit + branch conventions (one-time, opt-in)
 make install-hooks
 
 # Verify your setup works
@@ -56,7 +55,7 @@ That's it! Your local development environment is ready.
 
 ## Commit and Branch Conventions
 
-Commits follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/). Internal contributors name branches with the `litellm_` prefix and no `/`, and fork branches can use any name. Run `make install-hooks` once per clone to enable the local hook that enforces the commit convention. See the [contributor docs](https://docs.litellm.ai/docs/extras/contributing_code#commit-and-branch-conventions) for details
+Commits follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) and branches follow [Conventional Branches](https://conventional-branch.github.io/). Run `make install-hooks` once per clone to enable the local git hooks that enforce these — see the [contributor docs](https://docs.litellm.ai/docs/extras/contributing_code#commit-and-branch-conventions) for the full type list, examples, the protected-branch bypass list, and how to opt out.
 
 ### 2. Development Workflow
 
@@ -79,8 +78,8 @@ uv run pytest tests/unit/<your_test_file>.py -v
 git add .
 git commit -m "feat(scope): your descriptive commit message"
 
-# Push and create a PR
+# Push and create a PR (branch must follow Conventional Branches — see above)
-git push origin litellm_your_feature
+git push origin feature/your-feature
 ```
 
 ## Adding Testing
@@ -95,8 +94,6 @@ Add your tests to the [`tests/unit/` directory](https://github.com/BerriAI/litel
 - **Only add mocked tests** - no real LLM API calls in this directory
 - For integration tests with real APIs, use the appropriate test directories
 
-Tests that call `litellm.register_model` or change `litellm.model_cost` should use the `local_model_cost_map` fixture from `tests/unit/conftest.py`. It provides a fresh catalog and restores the previous map, so pricing and capability overrides cannot leak into later tests
-
 ### File Naming Convention
 
 The `tests/unit/` directory follows the same structure as `litellm/`:
@@ -249,43 +246,6 @@ If `make test-unit` fails:
 - **Test edge cases**: Don't just test the happy path
 - **Update documentation**: If you change APIs, update docs
 
-## Proving Your Change Works End to End
-
-Tests are required, but they do not prove that your change works for a user. A PR that changes what the proxy or SDK does shows a real run in its "Screenshots / Proof of Fix" section, meaning the exact commands you sent to a live proxy and the output that came back, against real provider APIs and not mocks. `pytest` output does not count as proof, because a passing test only shows that the code does what its mocks expect. A PR that changes only docs or CI shows the rendered page or the CI run instead
-
-A good proof looks like this:
-
-1. Install the proxy dependencies with `make install-proxy-dev`, then start the proxy from your branch with a config that has the model you changed, `uv run litellm --config your_config.yaml --detailed_debug`
-2. Send the request a user would send with `curl`, and paste both the command and the response. When cost or routing is involved, include the response headers, since `x-litellm-response-cost` is what the user sees
-3. For a bug fix, do this twice, once at the merge base with BerriAI/litellm's main (`git merge-base origin/main HEAD`, with your remote for BerriAI/litellm in place of `origin` in a fork clone) to show the bug and once at your branch's latest commit to show the fix. Name both commit hashes
-4. If your change is reachable from more than one endpoint (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`), show each one
-5. If the request can stream, show a `"stream": true` run too, because streaming builds usage and cost on a different path. A stream sends its headers before its usage exists, so show the `usage.cost` on the last chunk, which `/v1/chat/completions` sends when the request has `"stream_options": {"include_usage": true}` and other endpoints send with `include_cost_in_streaming_usage: true` under `litellm_settings`, or, when the proxy has a database, the request's row on the Admin UI's Logs page or `GET /spend/logs`
-6. For UI changes, add before and after screenshots and name the page
-7. If users reach your change through a coding tool such as Claude Code or Codex, drive that tool the way a user does and add before and after screenshots of it, because a `curl` or a headless run alone does not show what the user sees
-
-```bash
-curl -sD - http://localhost:4000/v1/chat/completions \
-  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-  -H "Content-Type: application/json" \
-  -d '{"model": "your-model", "messages": [{"role": "user", "content": "hi"}]}'
-```
-
-When the provider reports its own usage or spend numbers, compare the proxy's numbers against them in the proof
-
-## Contributing an LLM Provider Integration
-
-Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution, and it is where PRs get reworked most often. Follow these rules
-
-**Preserve zero prices.** In [`model_prices_and_context_window.json`](model_prices_and_context_window.json), an explicit `0.0` means that the priced operation is free. A missing field or `null` means no rate is specified, so the relevant calculator may use a fallback. For example, when no batch input rate is configured, batch cache writes use `cache_creation_input_token_cost` if set, otherwise the regular input rate, then apply the batch discount. A zero ordinary input rate does not make separately priced cache reads or writes free. Test `rate is None` before substituting a fallback, since `row.get("cache_creation_input_token_cost") or fallback` would discard a valid zero. Speech keeps character pricing ahead of token pricing even when the character rate is zero. If neither rate is configured, speech cost calculation raises an error. Its model lookup preserves unset rates instead of treating compatibility defaults as explicit zero prices. Perplexity keeps a configured legacy search rate ahead of the current search rate, including zero. When registering custom prices, `None` leaves an existing rate unchanged and `0.0` overrides it
-
-**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. [`ARCHITECTURE.md`](ARCHITECTURE.md) shows where the layers live, and the decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) explain why they are shaped that way. For example, [the ADR on provider usage fields](docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md) covers how provider-specific usage fields reach cost tracking. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
-
-**Bill a new provider usage field from the usage object.** Leave the field on the usage object as an extra and read it in that provider's `cost_calculator.py`. A provider without one gets a new `cost_calculator.py` and an arm for it in `cost_per_token` (`litellm/cost_calculator.py`), which picks the calculator by provider. If the provider's transform builds `Usage` from named fields, as the Anthropic, Gemini, and Bedrock Converse ones do, pass the new field there too. Streamed requests rebuild usage in `ChunkProcessor.calculate_usage` (`litellm/litellm_core_utils/streaming_chunk_builder_utils.py`), which does not copy extras yet the way the Responses-to-chat bridge does (minus a short exclusion list), so until it does, bill the field on streams by naming it there, as Anthropic's `inference_geo` is. A provider that reports its own charge for the request, as OpenRouter does in `usage.cost`, is billed at that charge from `_hidden_params` before any calculator runs, unless the deployment sets its own prices. When the field counts a built-in tool such as web search, add the provider to `get_cost_for_web_search_request` in `litellm/llms/__init__.py`, and add a gate to `litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py` only when no existing gate fires. Then show the field's cost in your proof on every endpoint it reaches, streaming included
-
-**Do not change what a caller sees to make internals easier.** Each endpoint keeps the schema of the API it emulates. A `/v1/responses` caller gets `input_tokens` and a `/v1/chat/completions` caller gets `prompt_tokens`, whatever the provider sent and whatever cost tracking needs. A provider usage field that shows up as an extra key is fine, since it only adds a key, but renaming, removing, or reshaping a key the endpoint defines is not. Reshaping a public response to feed an internal consumer breaks every user of that endpoint
-
-If your change makes an architecture decision, such as a new cross-cutting mechanism or a deliberate difference from a provider's API shape, add an ADR to [`docs/adrs/`](docs/adrs/AGENTS.md) in the same PR
-
 ## Building and Running Locally
 
 ### LiteLLM Proxy Server
@@ -305,8 +265,8 @@ uv run litellm --config your_config.yaml
 If you want to build the Docker image yourself:
 
 ```bash
-# Build the image (runs as a non-root user by default)
+# Build using the non-root Dockerfile
-docker build -t litellm_dev .
+docker build -f docker/Dockerfile.non_root -t litellm_dev .
 
 # Generate a master key. Requests send it as the bearer token
 export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"

Dockerfile

@@ -1,10 +1,10 @@
 # syntax=docker/dockerfile:1.7
 
 # Base image for building
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:360896efb0753a5701c48f860e17130c205eae5a264797aa3e7ea126ee2923a7
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
 
 # Runtime image
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:360896efb0753a5701c48f860e17130c205eae5a264797aa3e7ea126ee2923a7
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 # Pinned by digest like the other base images; bump explicitly on Node upgrades.
 ARG UI_BUILD_IMAGE=node:24.19-alpine3.24@sha256:d32cdf619f63fe0471182d08996dd516c6275bb5fd31ae06e55a570bd9e1ad43
@@ -61,7 +61,6 @@ RUN apk add --no-cache \
     python-3.13-dev \
     rust \
     openssl \
-    libssl3 \
     openssl-dev \
     nodejs \
     npm \
@@ -76,7 +75,6 @@ ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
 COPY pyproject.toml uv.lock ./
 COPY enterprise/pyproject.toml enterprise/
 COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
-COPY packaging/litellm-proxy/pyproject.toml packaging/litellm-proxy/
 
 # Install third-party dependencies (cached unless pyproject.toml/uv.lock change)
 RUN uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
@@ -102,9 +100,7 @@ COPY --from=ui-builder /ui/out/. litellm/proxy/_experimental/out/
 RUN sed -i 's/\r$//' docker/build_admin_ui.sh && chmod +x docker/build_admin_ui.sh && ./docker/build_admin_ui.sh
 
 # Install project and workspace packages (fast - deps already cached)
-ARG MATURIN_PEP517_ARGS=""
+RUN uv sync --frozen --no-default-groups --no-editable \
-RUN uv build --package litellm-proxy --wheel --out-dir /wheels
-RUN uv sync --frozen --no-install-package litellm-proxy --no-default-groups --no-editable \
     --extra proxy \
     --extra proxy-runtime \
     --group admin-mcp \
@@ -114,24 +110,18 @@ RUN uv sync --frozen --no-install-package litellm-proxy --no-default-groups --no
     --extra bedrock-realtime \
     --python python3.13
 
-RUN uv pip install --python /app/.venv/bin/python --no-deps /wheels/litellm_proxy-*.whl
-
 RUN HOME=/opt/prisma XDG_CACHE_HOME=/opt/prisma/.cache PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
     npm_config_cache=/root/.npm \
     prisma generate --schema=./schema.prisma
 
-RUN sed -i 's/\r$//' docker-entrypoint.sh && chmod +x docker-entrypoint.sh
+RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh && \
-
+    sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
-RUN mkdir -p /var/lib/litellm/ui /var/lib/litellm/assets && \
-    cp -r litellm/proxy/_experimental/out/. /var/lib/litellm/ui/ && \
-    cp litellm/proxy/logo.png /var/lib/litellm/assets/logo.png && \
-    touch /var/lib/litellm/ui/.litellm_ui_ready
 
 FROM $LITELLM_BUILD_IMAGE AS liteadmin-builder
 COPY --from=uvbin /uv /usr/local/bin/uv
 RUN apk add --no-cache python-3.13
-ADD --checksum=sha256:4fb1847e28e2be30ba76dac94f02c4903291326516a0f4ba772f6ffb8b25d05c \
+ADD --checksum=sha256:2f7ae5cdd9d91731c0990e74a58239dc3e3fd2bf28dab23b55eafcdc47aaf87e \
-    https://github.com/BerriAI/litellm-admin-agent/archive/5661c805ce479a0cf11b842ea133cf8d22cd7e34.tar.gz /tmp/liteadmin.tar.gz
+    https://github.com/BerriAI/litellm-admin-agent/archive/ef501e94bc9fbacb9233b922abf71427f030408c.tar.gz /tmp/liteadmin.tar.gz
 RUN mkdir /tmp/liteadmin && tar xzf /tmp/liteadmin.tar.gz --strip-components=1 -C /tmp/liteadmin && \
     uv venv /opt/liteadmin --python python3.13 && \
     uv pip install --python /opt/liteadmin/bin/python --require-hashes -r /tmp/liteadmin/requirements.txt && \
@@ -150,35 +140,24 @@ USER root
 # https://github.com/BerriAI/litellm/issues/33518
 RUN echo "https://packages.wolfi.dev/os" >> /etc/apk/repositories
 
-RUN for i in 1 2 3; do \
+# node (without npm) is required by the prisma CLI at runtime
-      apk add --no-cache bash openssl libssl3 tzdata nodejs python-3.13 libsndfile libatomic libevent nginx && break; \
+RUN apk add --no-cache bash openssl tzdata nodejs python-3.13 libsndfile libevent
-      [ "$i" = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
-      sleep 5; \
-    done
 COPY --from=pgbouncer-builder /usr/local/bin/pgbouncer /usr/local/bin/pgbouncer
 
 WORKDIR /app
-
 ENV PATH="/app/.venv/bin:${PATH}" \
-    PYTHONPATH="/app" \
-    PYTHONDONTWRITEBYTECODE=1 \
-    PYTHONUNBUFFERED=1 \
-    HOME=/app \
-    LITELLM_NON_ROOT=true \
     PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
     PRISMA_CLI_PATH=/opt/prisma/binaries/node_modules/.bin/prisma \
     PRISMA_CLI_QUERY_ENGINE_TYPE=binary \
-    PRISMA_SKIP_POSTINSTALL_GENERATE=1 \
-    PRISMA_HIDE_UPDATE_MESSAGE=1 \
-    PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING=1 \
     PRISMA_OFFLINE_MODE=true
 
 # Copy only what runtime needs. The application is installed inside the venv;
 # the rest of the builder's /app is source and build metadata that must not
 # ship (manifest-scanning tools attribute everything in it to this image).
+# entrypoint.sh invokes litellm/proxy/prisma_migration.py by source path.
 COPY --from=builder /app/.venv /app/.venv
 COPY --from=liteadmin-builder /opt/liteadmin /opt/liteadmin
-COPY --from=builder /app/docker-entrypoint.sh /app/docker-entrypoint.sh
+COPY --from=builder /app/docker /app/docker
 COPY --from=builder /app/schema.prisma /app/schema.prisma
 COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/prisma_migration.py
 # enterprise/ is imported by source path at runtime (proxy_cli puts the
@@ -186,33 +165,21 @@ COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/pr
 # enterprise.enterprise_hooks from it)
 COPY --from=builder /app/enterprise /app/enterprise
 COPY --from=builder /app/litellm-proxy-extras /app/litellm-proxy-extras
-COPY --from=builder /app/gateway /app/gateway
+# Prisma CLI + engines are baked under /opt/prisma, a fixed path every
-COPY --from=builder /app/backend /app/backend
+# runtime uid can read and that no cache volume mount shadows. The paths are
-COPY --from=builder /app/migrations /app/migrations
+# pinned via PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH and recorded into the
+# generated client at build time, so `prisma migrate deploy` on a fresh
+# database needs no npm and no network access (#33650, #24554).
 COPY --from=builder /opt/prisma /opt/prisma
-COPY --from=builder /var/lib/litellm /var/lib/litellm
-COPY ui/nginx.conf /etc/nginx/nginx.conf
-COPY ui/docker-entrypoint.d/40-litellm-runtime-config.sh /app/ui/docker-entrypoint.d/40-litellm-runtime-config.sh
 
-RUN find /app/.venv -depth -type d -path "*/tornado/test" -exec rm -rf {} + && \
+RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
-    mkdir -p /app/.cache /var/run/litellm && \
+    find /app/.venv -type d -path "*/tornado/test" -delete && \
-    chown -R 65532:0 /app /var/lib/litellm /var/run/litellm && \
-    chmod -R g=u,g+w /app/.cache /var/lib/litellm /var/run/litellm && \
-    PRISMA_PATH="$(python -c 'import os, prisma; print(os.path.dirname(prisma.__file__))')" && \
-    PROXY_EXTRAS_PATH="$(python -c 'import os, litellm_proxy_extras; print(os.path.dirname(litellm_proxy_extras.__file__))')" && \
-    chmod -R g=u,g+w "$PRISMA_PATH" "$PROXY_EXTRAS_PATH" && \
     chmod -R a+rX /opt/prisma && \
     test -x /opt/prisma/binaries/node_modules/.bin/prisma && \
     test -f /opt/prisma/binaries/node_modules/prisma/build/index.js && \
-    ls /opt/prisma/binaries/node_modules/@prisma/engines/query-engine-* >/dev/null && \
+    python -c "from prisma.client import BINARY_PATHS; paths = list(BINARY_PATHS.query_engine.values()); assert paths and all(p.startswith('/opt/prisma/') for p in paths), paths"
-    python -c "from prisma.client import BINARY_PATHS; paths = list(BINARY_PATHS.query_engine.values()); assert paths and all(p.startswith('/opt/prisma/') for p in paths), paths" && \
-    python -c "from litellm.rust_bridge import _native" && \
-    python -c "import gateway.launch, backend.main"
 
-USER 65532:65532
+EXPOSE 4000/tcp
 
-RUN nginx -t
+ENTRYPOINT ["docker/prod_entrypoint.sh"]
-
+CMD ["--port", "4000"]
-EXPOSE 4000/tcp 4001/tcp 3000/tcp
-
-ENTRYPOINT ["/app/docker-entrypoint.sh"]

Makefile

@@ -22,7 +22,7 @@ help:
 	@echo "  make install-proxy-dev-ci - Install proxy dev dependencies (CI-compatible)"
 	@echo "  make install-test-deps  - Install the full local test environment"
 	@echo "  make install-helm-unittest - Install helm unittest plugin"
-	@echo "  make install-hooks      - Install git hooks (Conventional Commits)"
+	@echo "  make install-hooks      - Install git hooks (Conventional Commits + Branches)"
 	@echo "  make check              - Run CI-equivalent lint on staged files, or on the diff vs the base branch when nothing is staged"
 	@echo "  make pre-commit         - Legacy alias for make check"
 	@echo "  make format             - Apply ruff format code formatting"
@@ -120,7 +120,7 @@ install-helm-unittest:
 		helm plugin install https://github.com/helm-unittest/helm-unittest --version v0.8.2; \
 	}
 
-# Install git hooks that enforce Conventional Commits.
+# Install git hooks that enforce Conventional Commits and Conventional Branches.
 # Opt-in: not chained into install-dev.
 install-hooks:
 	./scripts/install_git_hooks.sh
@@ -153,7 +153,7 @@ lint-install:
 # the same set CI's ':(glob)litellm/**/*.py' selects.
 lint-format-check-changed: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
 	@base_ref=$$($(RESOLVE_BASE)) && \
-	changed=$$(git diff --name-only --diff-filter=ACMR --merge-base "$$base_ref" -- 'litellm/*.py') && \
+	changed=$$(git diff --name-only --diff-filter=ACMR "$$base_ref...HEAD" -- 'litellm/*.py') && \
 	files=$$(printf '%s\n' "$$changed" | grep -v '^litellm/enterprise/' || true) || exit $$?; \
 	if [ -z "$$files" ]; then \
 		echo "No changed litellm Python files to format-check."; \
@@ -313,7 +313,7 @@ test-unit-proxy-core: install-test-deps
 	$(UV_RUN) pytest tests/unit/proxy/auth tests/unit/proxy/client tests/unit/proxy/db tests/unit/proxy/hooks tests/unit/proxy/policy_engine --ignore=tests/unit/proxy/db/db_transaction_queue/test_e2e_pod_lock_manager.py --ignore=tests/unit/proxy/db/test_update_daily_tag_spend.py --tb=short -vv -n 4 --durations=20
 
 test-unit-proxy-misc: install-test-deps
-	$(UV_RUN) pytest tests/unit/proxy/agent_endpoints tests/unit/proxy/anthropic_endpoints tests/unit/proxy/common_utils --ignore=tests/unit/proxy/common_utils/test_cache_aware_routing.py --ignore=tests/unit/proxy/common_utils/test_check_batch_cost.py --ignore=tests/unit/proxy/common_utils/test_check_responses_cost.py --ignore=tests/unit/proxy/common_utils/test_proxy_encrypt_decrypt.py --ignore=tests/unit/proxy/common_utils/test_realtime_cache.py tests/unit/proxy/discovery_endpoints tests/unit/proxy/mcp_server/test_tool_registry.py tests/unit/proxy/google_endpoints tests/unit/proxy/health_endpoints tests/unit/proxy/image_endpoints tests/unit/proxy/middleware --ignore=tests/unit/proxy/middleware/test_request_size_limit_middleware.py tests/unit/proxy/openai_files_endpoint tests/unit/proxy/pass_through_endpoints tests/unit/proxy/prompts tests/unit/proxy/public_endpoints tests/unit/proxy/response_api_endpoints tests/unit/proxy/shutdown tests/unit/proxy/spend_tracking --ignore=tests/unit/proxy/spend_tracking/test_search_api_logging.py tests/unit/proxy/ui_crud_endpoints tests/unit/proxy/vector_store_endpoints tests/unit/proxy/mcp_server/test_mcp_server_tool_calls_and_headers.py --ignore=tests/unit/proxy/google_endpoints/test_gemini_agents_endpoints.py --ignore=tests/unit/proxy/google_endpoints/test_google_endpoint_routing.py --ignore=tests/unit/proxy/google_endpoints/test_google_gemini_proxy_request.py --ignore=tests/unit/proxy/public_endpoints/test_blog_posts_endpoint.py --tb=short -vv -n 4 --durations=20
+	$(UV_RUN) pytest tests/unit/proxy/agent_endpoints tests/unit/proxy/anthropic_endpoints tests/unit/proxy/common_utils --ignore=tests/unit/proxy/common_utils/test_cache_aware_routing.py --ignore=tests/unit/proxy/common_utils/test_check_batch_cost.py --ignore=tests/unit/proxy/common_utils/test_check_responses_cost.py --ignore=tests/unit/proxy/common_utils/test_proxy_encrypt_decrypt.py --ignore=tests/unit/proxy/common_utils/test_realtime_cache.py tests/unit/proxy/discovery_endpoints tests/unit/proxy/experimental tests/unit/proxy/google_endpoints tests/unit/proxy/health_endpoints tests/unit/proxy/image_endpoints tests/unit/proxy/middleware --ignore=tests/unit/proxy/middleware/test_request_size_limit_middleware.py tests/unit/proxy/openai_files_endpoint tests/unit/proxy/pass_through_endpoints tests/unit/proxy/prompts tests/unit/proxy/public_endpoints tests/unit/proxy/response_api_endpoints tests/unit/proxy/shutdown tests/unit/proxy/spend_tracking --ignore=tests/unit/proxy/spend_tracking/test_search_api_logging.py tests/unit/proxy/ui_crud_endpoints tests/unit/proxy/vector_store_endpoints tests/unit/proxy/_experimental/mcp_server/test_mcp_server_tool_calls_and_headers.py --ignore=tests/unit/proxy/google_endpoints/test_gemini_agents_endpoints.py --ignore=tests/unit/proxy/google_endpoints/test_google_endpoint_routing.py --ignore=tests/unit/proxy/google_endpoints/test_google_gemini_proxy_request.py --ignore=tests/unit/proxy/public_endpoints/test_blog_posts_endpoint.py --tb=short -vv -n 4 --durations=20
 
 test-unit-proxy-root: install-test-deps
 	$(UV_RUN) pytest tests/unit/proxy/test_*.py --ignore=tests/unit/proxy/test_aproxy_startup.py --ignore=tests/unit/proxy/test_credential_slot_registry.py --ignore=tests/unit/proxy/test_custom_callback_input.py --ignore=tests/unit/proxy/test_custom_logger_s3_gcs.py --ignore=tests/unit/proxy/test_custom_tokenizer_bug.py --ignore=tests/unit/proxy/test_db_schema_changes.py --ignore=tests/unit/proxy/test_deprecated_key_grace_period.py --ignore=tests/unit/proxy/test_get_favicon.py --ignore=tests/unit/proxy/test_get_image.py --ignore=tests/unit/proxy/test_prisma_client_backoff_retry.py --ignore=tests/unit/proxy/test_prompt_test_endpoint.py --ignore=tests/unit/proxy/test_proxy_config_unit_test.py --ignore=tests/unit/proxy/test_proxy_custom_auth.py --ignore=tests/unit/proxy/test_proxy_reject_logging.py --ignore=tests/unit/proxy/test_proxy_server.py --ignore=tests/unit/proxy/test_proxy_setting_guardrails.py --ignore=tests/unit/proxy/test_proxy_token_counter.py --ignore=tests/unit/proxy/test_proxy_utils.py --ignore=tests/unit/proxy/test_reducto_ocr_route.py --ignore=tests/unit/proxy/test_response_polling_pre_call_checks.py --ignore=tests/unit/proxy/test_server_root_path.py --ignore=tests/unit/proxy/test_ui_path_detection.py --ignore=tests/unit/proxy/test_unit_test_proxy_hooks.py --ignore=tests/unit/proxy/test_update_spend.py --ignore=tests/unit/proxy/test_zero_cost_model_budget_bypass.py --tb=short -vv -n 4 --durations=20
@@ -325,7 +325,7 @@ test-unit-core-utils: install-test-deps
 	$(UV_RUN) pytest tests/unit/litellm_core_utils --tb=short -vv -n 2 --durations=20
 
 test-unit-other: install-test-deps
-	$(UV_RUN) pytest tests/unit/caching tests/unit/responses tests/unit/secret_managers tests/unit/vector_stores tests/unit/a2a_protocol tests/unit/completion_extras tests/unit/containers tests/unit/enterprise tests/unit/mcp_client tests/unit/google_genai tests/unit/images tests/unit/interactions tests/unit/router_strategy tests/unit/router_utils tests/unit/types --tb=short -vv -n 4 --durations=20
+	$(UV_RUN) pytest tests/unit/caching tests/unit/responses tests/unit/secret_managers tests/unit/vector_stores tests/unit/a2a_protocol tests/unit/completion_extras tests/unit/containers tests/unit/enterprise tests/unit/experimental_mcp_client tests/unit/google_genai tests/unit/images tests/unit/interactions tests/unit/router_strategy tests/unit/router_utils tests/unit/types --tb=short -vv -n 4 --durations=20
 
 test-unit-root: install-test-deps
 	$(UV_RUN) pytest tests/unit/test_*.py --tb=short -vv -n 4 --durations=20
@@ -341,7 +341,7 @@ test-integration: install-test-deps
 	$(UV_RUN) pytest tests/
 
 test-unit-helm: install-helm-unittest
-	helm unittest -f 'tests/*.yaml' helm/litellm
+	helm unittest -f 'tests/*.yaml' helm/litellm-helm
 
 # LLM Translation testing targets
 test-llm-translation: install-test-deps

README.md

@@ -141,56 +141,12 @@ response = client.chat.completions.create(
 )
 ```
 
-</details>
+[**Docs: LLM Providers**](https://docs.litellm.ai/docs/providers)
-
-<details>
-<summary><b>Auto Routing</b> - Choose a model for each prompt (Python SDK + AI Gateway)</summary>
-
-### Python SDK
-
-[**Docs: Autorouter SDK**](https://docs.litellm.ai/docs/autorouter_sdk)
-
-Install with `uv add litellm` and set `OPENAI_API_KEY` for the models below.
-
-```python
-import asyncio
-from litellm import Router
-
-router = Router(model_list=[
-    {"model_name": "fast", "litellm_params": {"model": "openai/gpt-6-luna"}},
-    {"model_name": "capable", "litellm_params": {"model": "openai/gpt-6.1-sol"}},
-    {"model_name": "auto", "litellm_params": {
-        "model": "auto_router/complexity_router",
-        "complexity_router_config": {
-            "classifier_type": "heuristic",
-            "tiers": {"SIMPLE": "fast", "MEDIUM": "fast",
-                      "COMPLEX": "capable", "REASONING": "capable"},
-            "return_raw_model_name": True,
-        },
-        "complexity_router_default_model": "capable",
-    }},
-])
-response = asyncio.run(router.acompletion(
-    model="auto", messages=[{"role": "user", "content": "What is 2+2?"}],
-))
-print(response.model, response.choices[0].message.content)
-```
-
-### AI Gateway (Proxy Server)
-
-[**Create an Auto Router**](https://docs.litellm.ai/docs/auto_router/setup), then call its model name. Replace `auto` and the URL below with yours, and set `LITELLM_API_KEY` to your virtual key.
-
-```shell
-curl http://localhost:4000/v1/chat/completions \
-  -H "Authorization: Bearer $LITELLM_API_KEY" \
-  -H "Content-Type: application/json" \
-  -d '{"model":"auto","messages":[{"role":"user","content":"What is 2+2?"}]}'
-```
 
 </details>
 
 <details>
-<summary><b>A2A</b> - Invoke A2A Agents (Python SDK + AI Gateway)</summary>
+<summary><b>Agents</b> - Invoke A2A Agents (Python SDK + AI Gateway)</summary>
 
 [**Supported Providers**](https://docs.litellm.ai/docs/a2a#add-a2a-agents) - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
 
@@ -267,7 +223,7 @@ async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
 ```python
 from mcp import ClientSession, StdioServerParameters
 from mcp.client.stdio import stdio_client
-from litellm import mcp_client
+from litellm import experimental_mcp_client
 import litellm
 
 server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
@@ -277,7 +233,7 @@ async with stdio_client(server_params) as (read, write):
         await session.initialize()
 
         # Load MCP tools in OpenAI format
-        tools = await mcp_client.load_mcp_tools(session=session, format="openai")
+        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
 
         # Use with any LiteLLM model
         response = await litellm.acompletion(
@@ -331,7 +287,7 @@ For MCP OAuth, an upstream may advertise dynamic client registration but refuse
 </details>
 
 <details>
-<summary><b>Agents</b> - Run Claude Code, Codex, OpenCode, pi or Deep Agents on any model (Python SDK)</summary>
+<summary><b>Agents</b> - Run Claude Code, Codex, OpenCode or Deep Agents on any model (Python SDK)</summary>
 
 ### Python SDK - Agents
 
@@ -340,7 +296,7 @@ import litellm
 from litellm import Harness, sandbox
 
 result = litellm.agent(
-    Harness.CLAUDE_CODE,  # or Harness.CODEX, Harness.OPENCODE, Harness.PI, Harness.DEEPAGENTS
+    Harness.CLAUDE_CODE,  # or Harness.CODEX, Harness.OPENCODE, Harness.DEEPAGENTS
     "Find why tests/test_router.py is flaky and fix it.",
     sandbox=sandbox.local("./repo"),
     model="litellm_proxy/claude-sonnet-4-5",  # a model group on your AI Gateway
@@ -349,7 +305,7 @@ result = litellm.agent(
 print(result.text, result.cost, [f.path for f in result.files])
 ```
 
-Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call the agent makes goes through your AI Gateway, tagged `harness,claude_code`. Drop the `litellm_proxy/` prefix to call a provider directly. Install `starlette uvicorn` plus the agent's CLI (`claude`, `codex`, `opencode` or `pi`), or `deepagents langchain-litellm` for Deep Agents.
+Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call the agent makes goes through your AI Gateway, tagged `harness,claude_code`. Drop the `litellm_proxy/` prefix to call a provider directly. Install `starlette uvicorn` plus the agent's CLI (`claude`, `codex` or `opencode`), or `deepagents langchain-litellm` for Deep Agents.
 
 [**Docs: Agent Harnesses**](https://docs.litellm.ai/docs/harness)
 
@@ -379,7 +335,6 @@ Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call th
 | [Bytez (`bytez`)](https://docs.litellm.ai/docs/providers/bytez) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Cerebras (`cerebras`)](https://docs.litellm.ai/docs/providers/cerebras) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Clarifai (`clarifai`)](https://docs.litellm.ai/docs/providers/clarifai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
-| [CLF AI Gateway (`clf_ai_gateway`)](https://docs.litellm.ai/docs/providers/clf_ai_gateway) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Cloudflare AI Workers (`cloudflare`)](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Codestral (`codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Cognition (`cognition`)](https://docs.litellm.ai/docs/providers/cognition) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
@@ -398,9 +353,7 @@ Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call th
 | [Deepseek (`deepseek`)](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Eden AI (`edenai`)](https://docs.litellm.ai/docs/providers/edenai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |  |  |
 | [ElevenLabs (`elevenlabs`)](https://docs.litellm.ai/docs/providers/elevenlabs) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ |  |  |  |
-| [EmpirioLabs AI (`empiriolabs`)](https://docs.litellm.ai/docs/providers/empiriolabs) | ✅ |  | ✅ |  |  |  |  |  |  |  |
 | [Empower (`empower`)](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
-| [eu/jev Decisions API (`eu_jev`)](https://docs.litellm.ai/docs/pass_through/eu_jev) |  |  |  |  |  |  |  |  |  |  |
 | [Fal AI (`fal_ai`)](https://docs.litellm.ai/docs/providers/fal_ai) | ✅ | ✅ | ✅ |  | ✅ |  |  |  |  |  |
 | [Featherless AI (`featherless_ai`)](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Fireworks AI (`fireworks_ai`)](https://docs.litellm.ai/docs/providers/fireworks_ai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
@@ -431,7 +384,6 @@ Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call th
 | [ModelScope (`modelscope`)](https://docs.litellm.ai/docs/providers/modelscope) | ✅ | ✅ | ✅ |  | ✅ |  |  |  |  |  |
 | [Moonshot (`moonshot`)](https://docs.litellm.ai/docs/providers/moonshot) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Morph (`morph`)](https://docs.litellm.ai/docs/providers/morph) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
-| [Nace (`nace`)](https://console.nace.ai/docs/drex) |  |  |  |  |  |  |  |  |  |  |
 | [Nebius AI Studio (`nebius`)](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
 | [NLP Cloud (`nlp_cloud`)](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Novita AI (`novita`)](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
@@ -445,7 +397,6 @@ Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call th
 | [OpenAI-like (`openai_like`)](https://docs.litellm.ai/docs/providers/openai_compatible) |  |  |  | ✅ |  |  |  |  |  |  |
 | [OpenRouter (`openrouter`)](https://docs.litellm.ai/docs/providers/openrouter) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [OVHCloud AI Endpoints (`ovhcloud`)](https://docs.litellm.ai/docs/providers/ovhcloud) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
-| [Parallel AI (`parallel_ai`)](https://docs.litellm.ai/docs/search/parallel_ai) |  |  |  |  |  |  |  |  |  |  |
 | [Perplexity AI (`perplexity`)](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Petals (`petals`)](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Pinstripes (`pinstripes`)](https://docs.litellm.ai/docs/providers/pinstripes) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
@@ -475,7 +426,6 @@ Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call th
 | [Watsonx Text (`watsonx_text`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [xAI (`xai`)](https://docs.litellm.ai/docs/providers/xai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 | [Xinference (`xinference`)](https://docs.litellm.ai/docs/providers/xinference) |  |  |  | ✅ |  |  |  |  |  |  |
-| [ZeroGPU (`zerogpu`)](https://docs.litellm.ai/docs/providers/zerogpu) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
 
 [**Read the Docs**](https://docs.litellm.ai/docs/)
 
@@ -490,7 +440,7 @@ You can use LiteLLM through either the Proxy Server or Python SDK. Both give you
 <tr>
 <th style={{width: '14%'}}></th>
 <th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/simple_proxy">LiteLLM AI Gateway</a></strong></th>
-<th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/python_sdk">LiteLLM Python SDK</a></strong></th>
+<th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/">LiteLLM Python SDK</a></strong></th>
 </tr>
 </thead>
 <tbody>
@@ -665,15 +615,15 @@ The Terraform modules live at [`terraform/litellm/aws/`](./terraform/litellm/aws
 
 ### Verify Docker Image Signatures
 
-All LiteLLM Docker images published to GHCR are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Releases are signed with the key introduced in [commit `e82a3c6`](https://github.com/BerriAI/litellm/commit/e82a3c646908ba0a6ee0beddc4310d7a6cac4065).
+All LiteLLM Docker images published to GHCR are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0).
 
 **Verify using the pinned commit hash (recommended):**
 
-A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the published signing key:
+A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:
 
 ```bash
 cosign verify \
-  --key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
+  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
   ghcr.io/berriai/litellm:<release-tag>
 ```
 
@@ -698,7 +648,7 @@ For companies that need better security, user management and professional suppor
 [Talk to founders](https://enterprise.litellm.ai/demo)
 
 This covers:
-- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/enterprise):**
+- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):**
 - ✅ **Feature Prioritization**
 - ✅ **Custom Integrations**
 - ✅ **Professional Support - Dedicated discord + slack**
@@ -709,8 +659,6 @@ This covers:
 
 We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
 
-If you want to work closely with our team, check out [LiteLLM OSS Champions](https://www.litellm.ai/champions), our program for trusted OSS contributors with their own Slack channel and a voice in the roadmap
-
 ## Quick Start for Contributors
 
 This requires uv to be installed.

SECURITY.md

@@ -1,91 +0,0 @@
-# Data Privacy and Security
-
-## Security Announcements
-
-LiteLLM maintains a security announcements mailing list that is open to anyone. Subscribers receive advance notice, typically one to two days, before we release a fix for a particularly severe vulnerability or for any vulnerability exploitable by an unauthenticated attacker. This notice is provided on a best-effort basis
-
-To subscribe, visit [https://berriai.github.io/security-announce-signup/](https://berriai.github.io/security-announce-signup/)
-
-## Security Vulnerability Reporting Guidelines
-
-> [!WARNING]
-> Reports that do not include a video demonstrating the exploit will be closed without review. See [Reproduction Video Requirement](#reproduction-video-requirement) below.
-
-We value the security community's role in protecting our systems and users. To report a security vulnerability:
-
-- File a private vulnerability report on GitHub: [Report a vulnerability](https://github.com/BerriAI/litellm/security/advisories/new)
-- Include steps to reproduce the issue
-- Include a video or screen recording demonstrating the full exploit against a live LiteLLM instance, from initial access through to impact. A terminal recording (for example asciinema) is fine for CLI-only exploits.
-- Provide any relevant additional information
-
-### Reproduction Video Requirement
-
-A video demonstrating the exploit is required for every report. AI tools have made it easy to produce plausible-sounding vulnerability reports that do not reproduce in practice, and triaging them takes time away from real issues. Reports submitted without a working reproduction video will be closed without review. If you add a video to a closed report, we will reopen and triage it.
-
-### Vulnerability Categories
-
-We classify vulnerabilities into the following categories:
-
-**P0: Supply Chain Attacks**
-
-Attacks that compromise our CI/CD pipeline, allowing a malicious actor to point our PyPI package or Docker images (GHCR or Docker Hub) to vulnerable or tampered artifacts.
-
-**P1: Unauthenticated Proxy Access**
-
-Application-level attacks where an unauthenticated user is able to gain access to protected data on a LiteLLM proxy instance that should be protected (e.g api keys).
-
-**P2: Authenticated Malicious Actions**
-
-Application-level attacks where an authenticated user is able to perform actions beyond their intended permissions, such as privilege escalation or unauthorized data access.
-
-### Bug Bounty Program
-
-We offer bounties for responsibly disclosed vulnerabilities based on severity:
-
-**Note that currently only P0/P1 reports are eligible for a bounty, though submissions for P2 bugs are still encouraged**
-
-| Severity | Bounty Range | Example |
-|----------|-------------|---------|
-| **Critical** | $1,500 - $3,000 | P0 supply chain compromise |
-| **High** | $500 - $1,500 | P1 unauthenticated proxy access |
-| **Medium** | N/A | P2 authenticated privilege escalation |
-| **Low** | N/A | Minor information disclosure, low-impact misconfigurations |
-
-To qualify for a bounty, reports must include clear reproduction steps, a reproduction video as described above, and must not involve systems or accounts you do not own. We review all submissions promptly and will follow up within 5 business days.
-
-### Known Non-Issues
-
-- Attacks that require a misconfiguration on setup (e.g not setting a `master_key` on the proxy configuration), are **explicitly not in scope** and are not considered vulnerable.
-
-## Security Measures
-
-### LiteLLM Github
-
-- All commits run through Github's CodeQL checking
-
-### Self-hosted Instances LiteLLM
-
-- **No data or telemetry is stored on LiteLLM Servers when you self host**
-- For installation and configuration, see: [Self-hosting guided](https://docs.litellm.ai/docs/proxy/deploy)
-- **Telemetry** We run no telemetry when you self host LiteLLM
-
-### LiteLLM Cloud
-
-- We encrypt all data stored using your `LITELLM_MASTER_KEY` and in transit using TLS.
-- Our database and application run on GCP, AWS infrastructure, partly managed by NeonDB.
-    - US data region: Northern California (AWS/GCP `us-west-1`) & Virginia (AWS `us-east-1`)
-    - EU data region Germany/Frankfurt (AWS/GCP `eu-central-1`)
-- All users have access to SSO (Single Sign-On) through OAuth 2.0 with Google, Okta, Microsoft, KeyCloak. 
-- Audit Logs with retention policy
-- Control Allowed IP Addresses that can access your Cloud LiteLLM Instance
-
-For security inquiries, please contact us at security@berri.ai
-
-#### Supported data regions for LiteLLM Cloud
-
-LiteLLM supports the following data regions:
-
-- US, Northern California (AWS/GCP `us-west-1`)
-- Europe, Frankfurt, Germany (AWS/GCP `eu-central-1`)
-
-All data, user accounts, and infrastructure are completely separated between these two regions

backend/Dockerfile

@@ -0,0 +1,110 @@
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
+
+FROM $UV_IMAGE AS uvbin
+
+# ---------- Builder ----------
+FROM $LITELLM_BUILD_IMAGE AS builder
+
+WORKDIR /app
+USER root
+
+COPY --from=uvbin /uv /uvx /usr/local/bin/
+
+# nodejs/npm so `prisma generate` uses Wolfi's Node via PRISMA_USE_GLOBAL_NODE
+# instead of nodeenv downloading one whose dynamic deps may not be in Wolfi
+# (e.g. Node 26.2.0 needs libatomic). Retry for transient apk.cgr.dev flakes.
+RUN for i in 1 2 3; do \
+      apk add --no-cache bash gcc python-3.13 python-3.13-dev openssl openssl-dev libsndfile nodejs npm && break; \
+      [ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
+      sleep 5; \
+    done
+
+# UV_COMPILE_BYTECODE=1   precompiles .pyc at install time → faster cold start.
+# UV_LINK_MODE=copy       avoids hardlink warnings when uv installs from a
+#                         BuildKit cache mount (different filesystem).
+# UV_PYTHON_DOWNLOADS=0   force uv to use the apk-installed CPython instead of
+#                         silently pulling a managed interpreter.
+# PRISMA_USE_GLOBAL_NODE  explicit (matches default) so an env override can't
+#                         silently re-enable nodeenv's Node download.
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
+    UV_LINK_MODE=copy \
+    UV_COMPILE_BYTECODE=1 \
+    UV_PYTHON_DOWNLOADS=0 \
+    PRISMA_USE_GLOBAL_NODE=true \
+    PATH="/app/.venv/bin:${PATH}"
+
+# Stage 1 — install dependencies only.
+RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
+    --mount=type=bind,source=uv.lock,target=uv.lock \
+    --mount=type=bind,source=enterprise/pyproject.toml,target=enterprise/pyproject.toml \
+    --mount=type=bind,source=litellm-proxy-extras/pyproject.toml,target=litellm-proxy-extras/pyproject.toml \
+    uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
+        --extra proxy \
+        --extra proxy-runtime \
+        --group admin-mcp \
+        --extra extra_proxy \
+        --extra semantic-router \
+        --extra saml \
+        --python python3.13
+
+# Stage 2 — copy source and install the project + workspace members.
+COPY . .
+
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-default-groups --no-editable \
+        --extra proxy \
+        --extra proxy-runtime \
+        --group admin-mcp \
+        --extra extra_proxy \
+        --extra semantic-router \
+        --extra saml \
+        --python python3.13
+
+RUN cp "$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"/litellm/rust_bridge/_native*.so litellm/rust_bridge/
+
+RUN HOME=/opt/prisma XDG_CACHE_HOME=/opt/prisma/.cache PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
+    npm_config_cache=/root/.npm \
+    prisma generate --schema=./schema.prisma
+
+RUN sed -i 's/\r$//' docker/component_entrypoint.sh && chmod +x docker/component_entrypoint.sh
+
+# ---------- Runtime ----------
+FROM $LITELLM_RUNTIME_IMAGE AS runtime
+ARG LITELLM_RELEASE_TAG=""
+ENV LITELLM_RELEASE_TAG=${LITELLM_RELEASE_TAG}
+
+USER root
+
+RUN for i in 1 2 3; do \
+      apk add --no-cache bash openssl tzdata python-3.13 libsndfile libatomic && break; \
+      [ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
+      sleep 5; \
+    done
+
+# wolfi-base ships an unprivileged `nonroot` account (UID/GID 65532) with
+# /home/nonroot. We run the backend as that user
+WORKDIR /app
+ENV HOME=/home/nonroot \
+    PATH="/app/.venv/bin:${PATH}" \
+    PYTHONPATH="/app" \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries
+
+COPY --from=builder --chown=nonroot:nonroot /app /app
+COPY --from=builder /opt/prisma /opt/prisma
+
+RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
+    find /app/.venv -type d -path "*/tornado/test" -delete && \
+    chmod -R a+rX /opt/prisma && \
+    python -c "from prisma.client import BINARY_PATHS; paths = list(BINARY_PATHS.query_engine.values()); assert paths and all(p.startswith('/opt/prisma/') for p in paths), paths"
+
+USER nonroot
+
+EXPOSE 4001/tcp
+
+ENTRYPOINT ["/app/docker/component_entrypoint.sh", "uvicorn", "backend.main:app"]
+CMD ["--host", "0.0.0.0", "--port", "4001"]

backend/routes/allowlist.py

@@ -55,7 +55,6 @@ BACKEND_PATH_PREFIXES: tuple[str, ...] = (
     "/cache_settings",
     "/coordination_redis/",
     "/cost_tracking",
-    "/subscription_accounts/",
     "/cost_optimization/",
     "/cost/",
     "/credentials",
@@ -166,7 +165,6 @@ BACKEND_EXACT_PATHS: frozenset[str] = frozenset(
 BACKEND_MOUNT_PATHS: frozenset[str] = frozenset(
     {
         "/admin",
-        "/metrics",
         "/swagger",  # API documentation static assets belong to the backend
         "/mcp",  # lazily-mounted MCP sub-app serves on the backend component
     }

ci_cd/cost_map_guard.py

@@ -1,67 +1,41 @@
 """Guard the cost map on pull requests.
 
-Every pull request whose diff against its merge base touches one of the two cost map files or the pins file gets the
+Every pull request whose diff against its merge base touches one of the three cost map files gets the file
-file checks: the files parse, the backup copy matches the root file, and every capability pinned in
+checks: the files parse, the backup copy matches the root file, and the JSON schema is in sync and validates the
-ci_cd/cost_map_pins.json (a value verified against a live provider call where the provider's model listing says
+map. A pull request that leaves all three untouched skips them, since merging it keeps the base branch's copies
-otherwise) still holds in the tree that merging the pull request into its base branch produces, so a branch cut before
+and its head tree only carries whatever state the branch was cut from. Pull requests from the cost map sync bot
-a pin was added cannot lower it, and a pull request that deletes the pins file fails too. A pull request that leaves
+(branches named litellm_cost_map_sync_*) always get the file checks and additionally may only touch those three
-all three untouched skips them, since merging it keeps the base branch's copies and its head tree only carries
+files and may only add or update models.
-whatever state the branch was cut from. Pull requests from the cost map sync bot (branches named
-litellm_cost_map_sync_*) always get the file checks and additionally may only touch the two cost map files and may
-only add or update models, so a sync can never lower a pinned capability or edit the pins. Price sync pull requests
-(branches named litellm-providers/price-sync*) and pull requests a bot account opened (COST_MAP_GUARD_AUTHOR_TYPE is
-Bot) get the human checks and may not touch the pins file either.
 """
 
 from __future__ import annotations
 
 import argparse
 import json
-import os
 import subprocess
 import sys
-from collections.abc import Iterator, Mapping, Sequence
+from collections.abc import Mapping, Sequence
 from dataclasses import dataclass
 from types import MappingProxyType
 from typing import Final
 
+from generate_model_prices_schema import SPECIAL_ROOT_KEYS, build_schema, render, validation_errors
+
 COST_MAP_PATH: Final = "model_prices_and_context_window.json"
 BACKUP_PATH: Final = "litellm/model_prices_and_context_window_backup.json"
-PINS_PATH: Final = "ci_cd/cost_map_pins.json"
+SCHEMA_PATH: Final = "model_prices_and_context_window.schema.json"
-GUARDED_PATHS: Final = (COST_MAP_PATH, BACKUP_PATH)
+GUARDED_PATHS: Final = (COST_MAP_PATH, BACKUP_PATH, SCHEMA_PATH)
-CHECKED_PATHS: Final = (*GUARDED_PATHS, PINS_PATH)
 BOT_BRANCH_PREFIX: Final = "litellm_cost_map_sync_"
-PRICE_SYNC_BRANCH_PREFIX: Final = "litellm-providers/price-sync"
-TARGET_ENV: Final = "COST_MAP_GUARD_TARGET"
-AUTHOR_TYPE_ENV: Final = "COST_MAP_GUARD_AUTHOR_TYPE"
-BOT_AUTHOR_TYPE: Final = "Bot"
-MERGE_CONFLICT_EXIT_CODE: Final = 1
-SPECIAL_ROOT_KEYS: Final = frozenset({"sample_spec", "fallback_generalizations"})
 EVALUATION_OUTPUT_PRICE_SOURCES: Final[Mapping[str, str]] = MappingProxyType({})
 
 CostMap = dict[str, object]
-_MISSING: Final = object()
 
 
 @dataclass(frozen=True, slots=True)
 class Snapshot:
     cost_map: str
     backup: str
-    pins: str
+    schema: str
-
-
-@dataclass(frozen=True, slots=True)
-class MergeFailed:
-    stderr: str
-
-
-@dataclass(frozen=True, slots=True)
-class Pin:
-    key: str
-    field: str
-    value: object
-    checked: str
-    source: str
 
 
 def _parse_object(text: str, path: str) -> CostMap | str:
@@ -72,6 +46,13 @@ def _parse_object(text: str, path: str) -> CostMap | str:
     return parsed if isinstance(parsed, dict) else f"{path} must be a JSON object at the root"
 
 
+def _rendered_schema(cost_map: CostMap) -> str:
+    try:
+        return render(build_schema(cost_map))
+    except SystemExit as error:
+        return str(error)
+
+
 def _evaluation_output_price_failure(
     key: str, entry: object, evaluation_output_price_sources: Mapping[str, str]
 ) -> str | None:
@@ -97,13 +78,29 @@ def _file_failures(
     *,
     evaluation_output_price_sources: Mapping[str, str] = EVALUATION_OUTPUT_PRICE_SOURCES,
 ) -> tuple[str, ...]:
+    schema_text: Final = _rendered_schema(head_map)
+    if not schema_text.startswith("{"):
+        return (schema_text,)
     backup_failure: Final = (
         ()
         if head.backup == head.cost_map
         else (f"{BACKUP_PATH} differs from {COST_MAP_PATH}; copy the root file over it",)
     )
+    schema_failure: Final = (
+        ()
+        if head.schema == schema_text
+        else (
+            f"{SCHEMA_PATH} is out of sync with {COST_MAP_PATH}; "
+            "run `python ci_cd/generate_model_prices_schema.py` and commit the result",
+        )
+    )
     return (
         *backup_failure,
+        *schema_failure,
+        *(
+            f"{COST_MAP_PATH} does not validate against its schema: {error}"
+            for error in validation_errors(head_map, json.loads(schema_text))[:20]
+        ),
         *(
             failure
             for key, entry in head_map.items()
@@ -121,48 +118,6 @@ def _entries(cost_map: CostMap) -> dict[str, dict[str, object]]:
     return {key: entry for key, entry in cost_map.items() if isinstance(entry, dict)}
 
 
-def _pin(key: str, field: str, spec: object) -> Pin | str:
-    malformed: Final = f"{PINS_PATH}: {key}.{field} needs value, checked, and source"
-    if not isinstance(spec, dict) or "value" not in spec:
-        return malformed
-    checked: Final = spec.get("checked")
-    source: Final = spec.get("source")
-    if not isinstance(checked, str) or not isinstance(source, str) or not checked or not source:
-        return malformed
-    return Pin(key, field, spec["value"], checked, source)
-
-
-def _pins(parsed: CostMap) -> Iterator[Pin | str]:
-    for key, fields in parsed.items():
-        if not isinstance(fields, dict):
-            yield f"{PINS_PATH}: {key} must map field names to pins"
-            continue
-        for field, spec in fields.items():
-            yield _pin(key, field, spec)
-
-
-def _pin_violations(parsed: CostMap, entries: Mapping[str, Mapping[str, object]]) -> Iterator[str]:
-    for pin in _pins(parsed):
-        if isinstance(pin, str):
-            yield pin
-        elif pin.key not in entries:
-            yield f"{PINS_PATH} pins {pin.key}.{pin.field} but {COST_MAP_PATH} has no {pin.key} entry"
-        elif entries[pin.key].get(pin.field, _MISSING) != pin.value:
-            yield (
-                f"{COST_MAP_PATH}: {pin.key}.{pin.field} must stay {json.dumps(pin.value)} "
-                f"(verified {pin.checked}: {pin.source}); change {PINS_PATH} with new evidence first"
-            )
-
-
-def pin_failures(pins_text: str, head_map: CostMap) -> tuple[str, ...]:
-    if not pins_text:
-        return (f"{PINS_PATH} is missing; restore it, its pins were verified against live provider calls",)
-    parsed: Final = _parse_object(pins_text, PINS_PATH)
-    if isinstance(parsed, str):
-        return (parsed,)
-    return tuple(_pin_violations(parsed, _entries(head_map)))
-
-
 def _bot_failures(base: Snapshot, head_map: CostMap, changed_files: Sequence[str]) -> tuple[str, ...]:
     base_map: Final = _parse_object(base.cost_map, COST_MAP_PATH)
     if isinstance(base_map, str):
@@ -192,18 +147,8 @@ def _bot_failures(base: Snapshot, head_map: CostMap, changed_files: Sequence[str
     )
 
 
-def pin_author_failures(head_ref: str, author_type: str, changed_files: Sequence[str]) -> tuple[str, ...]:
-    if PINS_PATH not in changed_files:
-        return ()
-    if head_ref.startswith(PRICE_SYNC_BRANCH_PREFIX):
-        return (f"price sync PRs may not change {PINS_PATH}; a pin moves only in a human PR with new evidence",)
-    if author_type == BOT_AUTHOR_TYPE:
-        return (f"PRs opened by a bot may not change {PINS_PATH}; a pin moves only in a human PR with new evidence",)
-    return ()
-
-
 def touches_cost_map(changed_files: Sequence[str]) -> bool:
-    return any(path in CHECKED_PATHS for path in changed_files)
+    return any(path in GUARDED_PATHS for path in changed_files)
 
 
 def contract_for(bot: bool, changed_files: Sequence[str]) -> str:
@@ -212,21 +157,9 @@ def contract_for(bot: bool, changed_files: Sequence[str]) -> str:
     return "human PR, file checks only" if touches_cost_map(changed_files) else "human PR, cost map untouched"
 
 
-def _merged_pin_failures(base: Snapshot, merged: Snapshot | None) -> tuple[str, ...]:
-    if merged is None:
-        return ("the pull request does not merge cleanly into its base branch; merge the base branch in first",)
-    if not merged.pins and not base.pins:
-        return ()
-    merged_map: Final = _parse_object(merged.cost_map, COST_MAP_PATH)
-    if isinstance(merged_map, str):
-        return (f"after merging into the base branch, {merged_map}",)
-    return pin_failures(merged.pins, merged_map)
-
-
 def guard_failures(
     base: Snapshot,
     head: Snapshot,
-    merged: Snapshot | None,
     changed_files: Sequence[str],
     bot: bool,
     *,
@@ -241,7 +174,6 @@ def guard_failures(
         *_file_failures(
             head, head_map, evaluation_output_price_sources=evaluation_output_price_sources
         ),
-        *_merged_pin_failures(base, merged),
         *(_bot_failures(base, head_map, changed_files) if bot else ()),
     )
 
@@ -252,22 +184,7 @@ def _git(*args: str) -> str | None:
 
 
 def snapshot(revision: str) -> Snapshot:
-    return Snapshot(*(_git("show", f"{revision}:{path}") or "" for path in CHECKED_PATHS))
+    return Snapshot(*(_git("show", f"{revision}:{path}") or "" for path in GUARDED_PATHS))
-
-
-def merged_tree(merge_base: str, target: str, head: str) -> str | MergeFailed | None:
-    result: Final = subprocess.run(
-        ("git", "merge-tree", "--write-tree", f"--merge-base={merge_base}", target, head),
-        check=False,
-        capture_output=True,
-        text=True,
-    )
-    tree: Final = result.stdout.split("\n", 1)[0]
-    if result.returncode == 0:
-        return tree
-    if result.returncode == MERGE_CONFLICT_EXIT_CODE and tree:
-        return None
-    return MergeFailed(result.stderr.strip())
 
 
 def changed_files(base: str, head: str) -> tuple[str, ...] | None:
@@ -279,13 +196,6 @@ def main(argv: Sequence[str]) -> int:
     parser: Final = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--base", required=True, help="merge base of the pull request")
     parser.add_argument("--head", required=True, help="head commit of the pull request")
-    env_target: Final = os.environ.get(TARGET_ENV)
-    parser.add_argument(
-        "--target",
-        default=env_target,
-        required=env_target is None,
-        help=f"tip of the base branch the pull request merges into, defaulting to ${TARGET_ENV}",
-    )
     parser.add_argument("--head-ref", required=True, help="head branch name of the pull request")
     args: Final = parser.parse_args(argv)
     bot: Final = args.head_ref.startswith(BOT_BRANCH_PREFIX)
@@ -293,19 +203,7 @@ def main(argv: Sequence[str]) -> int:
     if changed is None:
         print(f"cost map guard failed: git diff {args.base} {args.head} failed, so the changed files are unknown")
         return 1
-    if not bot and not touches_cost_map(changed):
+    failures: Final = guard_failures(snapshot(args.base), snapshot(args.head), changed, bot)
-        print(f"cost map guard passed ({contract_for(bot, changed)})")
-        return 0
-    merged: Final = merged_tree(args.base, args.target, args.head)
-    if isinstance(merged, MergeFailed):
-        print(f"cost map guard failed: git merge-tree into {args.target} failed: {merged.stderr}")
-        return 1
-    failures: Final = (
-        *guard_failures(
-            snapshot(args.base), snapshot(args.head), None if merged is None else snapshot(merged), changed, bot
-        ),
-        *pin_author_failures(args.head_ref, os.environ.get(AUTHOR_TYPE_ENV, ""), changed),
-    )
     contract: Final = contract_for(bot, changed)
     if failures:
         print(f"cost map guard failed ({contract}):")

ci_cd/cost_map_pins.json

@@ -1,16 +0,0 @@
-{
-    "fireworks_ai/accounts/fireworks/models/minimax-m3": {
-        "supports_vision": {
-            "value": true,
-            "checked": "2026-09-26",
-            "source": "POST https://api.fireworks.ai/inference/v1/chat/completions read a 512x512 PNG back as digit=7, shape=circle, color=red while GET https://api.fireworks.ai/inference/v1/models still said supports_image_input false and GET https://api.fireworks.ai/v1/serverless/models?format=nested carried no supportsImageInput on the row"
-        }
-    },
-    "fireworks_ai/minimax-m3": {
-        "supports_vision": {
-            "value": true,
-            "checked": "2026-09-26",
-            "source": "POST https://api.fireworks.ai/inference/v1/chat/completions read a 512x512 PNG back as digit=7, shape=circle, color=red while GET https://api.fireworks.ai/inference/v1/models still said supports_image_input false and GET https://api.fireworks.ai/v1/serverless/models?format=nested carried no supportsImageInput on the row"
-        }
-    }
-}

ci_cd/generate_model_prices_schema.py

@@ -0,0 +1,409 @@
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+from typing import Final, Optional
+
+import jsonschema
+
+REPO_ROOT = Path(__file__).parent.parent
+PRICES_PATH = REPO_ROOT / "model_prices_and_context_window.json"
+SCHEMA_PATH = REPO_ROOT / "model_prices_and_context_window.schema.json"
+
+SPECIAL_ROOT_KEYS = frozenset({"sample_spec", "fallback_generalizations"})
+
+JsonSchema = dict
+
+NONNEG_NUMBER: JsonSchema = {"type": "number", "minimum": 0}
+NONNEG_INTEGER: JsonSchema = {"type": "integer", "minimum": 0}
+BOOLEAN: JsonSchema = {"type": "boolean"}
+STRING: JsonSchema = {"type": "string"}
+TIME_WINDOW: Final[JsonSchema] = {"type": "string", "pattern": r"^([01]\d|2[0-3]):[0-5]\d-([01]\d|2[0-3]):[0-5]\d$"}
+WEEKDAY_PATTERN: Final = (
+    r"(?i)^(mon|monday|tue|tues|tuesday|wed|wednesday|thu|thur|thurs|thursday|fri|friday|sat|saturday|sun|sunday)$"
+)
+
+EXTRA_BOOLEAN_KEYS = frozenset(
+    {
+        "gemini_native_audio",
+        "gemini_audio_only_live",
+        "uses_embed_content",
+        "use_openai_responses_path",
+        "bedrock_converse_supports_strict_tools",
+        "thinking_always_on",
+    }
+)
+
+HOURS_UTC: Final[JsonSchema] = {
+    "description": 'UTC "HH:MM-HH:MM" window, or a list of them; a window may wrap past midnight.',
+    "oneOf": [TIME_WINDOW, {"type": "array", "items": TIME_WINDOW, "minItems": 1}],
+}
+
+OFF_PEAK_WINDOW: Final[JsonSchema] = {
+    "type": "object",
+    "properties": {
+        "hours_utc": HOURS_UTC,
+        "weekdays": {
+            "type": "array",
+            "description": "ISO-8601 weekday numbers (1 = Monday .. 7 = Sunday) or English day names the window applies on.",
+            "items": {
+                "oneOf": [
+                    {"type": "integer", "minimum": 1, "maximum": 7},
+                    {"type": "string", "pattern": WEEKDAY_PATTERN},
+                ]
+            },
+            "minItems": 1,
+        },
+    },
+    "required": ["hours_utc"],
+    "additionalProperties": False,
+}
+
+OBJECT_KEYS: dict[str, JsonSchema] = {
+    "off_peak_pricing": {
+        "type": "object",
+        "description": "Rates that replace the same-named base fields while the request falls inside the stated UTC windows.",
+        "properties": {
+            "hours_utc": HOURS_UTC,
+            "windows": {"type": "array", "items": OFF_PEAK_WINDOW, "minItems": 1},
+            "weekday_timezone": {
+                "type": "string",
+                "description": "IANA zone the weekdays of each window are read on; defaults to UTC.",
+            },
+            "input_cost_per_token": NONNEG_NUMBER,
+            "output_cost_per_token": NONNEG_NUMBER,
+            "output_cost_per_reasoning_token": NONNEG_NUMBER,
+            "cache_read_input_token_cost": NONNEG_NUMBER,
+            "cache_creation_input_token_cost": NONNEG_NUMBER,
+        },
+        "anyOf": [{"required": ["hours_utc"]}, {"required": ["windows"]}],
+        "additionalProperties": False,
+    },
+    "search_context_cost_per_query": {
+        "type": "object",
+        "description": "USD cost per web search query, keyed by search context size.",
+        "properties": {
+            "search_context_size_low": NONNEG_NUMBER,
+            "search_context_size_medium": NONNEG_NUMBER,
+            "search_context_size_high": NONNEG_NUMBER,
+        },
+        "additionalProperties": False,
+    },
+    "guardrail_cost_per_unit": {
+        "type": "object",
+        "description": "USD cost per billable guardrail unit, keyed by the provider's usage counter name (e.g. Bedrock's contentPolicyUnits).",
+        "additionalProperties": NONNEG_NUMBER,
+    },
+    "metadata": {
+        "type": "object",
+        "description": "Free-form notes about the entry (e.g. pricing derivation).",
+    },
+    "provider_specific_entry": {
+        "type": "object",
+        "description": "Provider-internal routing hints (e.g. bedrock_invocation_schema).",
+    },
+}
+
+ARRAY_KEYS: dict[str, JsonSchema] = {
+    "supported_audio_formats": {
+        "type": "array",
+        "description": "Audio container formats the model can return.",
+        "items": {"type": "string", "enum": ["mp3", "wav"]},
+    },
+    "supported_endpoints": {
+        "type": "array",
+        "description": "OpenAI-style API routes this model can be called through, e.g. /v1/chat/completions.",
+        "items": STRING,
+    },
+    "supported_modalities": {
+        "type": "array",
+        "description": "Input modalities the model accepts.",
+        "items": {"type": "string", "enum": ["text", "image", "audio", "video"]},
+    },
+    "supported_output_modalities": {
+        "type": "array",
+        "description": "Output modalities the model can produce.",
+        "items": {"type": "string", "enum": ["text", "image", "audio", "video", "code"]},
+    },
+    "reasoning_effort_levels": {
+        "type": "array",
+        "description": "Exact reasoning_effort levels this deployment accepts; wins over supports_* flags.",
+        "items": {"type": "string", "enum": ["none", "minimal", "low", "medium", "high", "xhigh", "max"]},
+    },
+    "supported_regions": {
+        "type": "array",
+        "description": "Cloud regions the model is available in ('global' or region ids).",
+        "items": STRING,
+    },
+    "tiered_pricing": {
+        "type": "array",
+        "description": "Context-length or result-count tiered rates; each tier's costs apply within its range.",
+        "items": {
+            "type": "object",
+            "properties": {
+                "range": {
+                    "type": "array",
+                    "description": "[min, max] prompt-token span this tier applies to.",
+                    "items": NONNEG_NUMBER,
+                    "minItems": 2,
+                    "maxItems": 2,
+                },
+                "max_results_range": {
+                    "type": "array",
+                    "description": "[min, max] result-count span this tier applies to (search models).",
+                    "items": NONNEG_NUMBER,
+                    "minItems": 2,
+                    "maxItems": 2,
+                },
+                "input_cost_per_token": NONNEG_NUMBER,
+                "output_cost_per_token": NONNEG_NUMBER,
+                "output_cost_per_reasoning_token": NONNEG_NUMBER,
+                "cache_read_input_token_cost": NONNEG_NUMBER,
+                "cache_creation_input_token_cost": NONNEG_NUMBER,
+                "input_cost_per_query": NONNEG_NUMBER,
+            },
+            "additionalProperties": False,
+        },
+    },
+}
+
+INTEGER_KEYS: dict[str, JsonSchema] = {
+    "max_tokens": {
+        **NONNEG_INTEGER,
+        "description": "Legacy field: max output tokens if the provider specifies it, else max input tokens.",
+    },
+    "max_input_tokens": {
+        **NONNEG_INTEGER,
+        "description": "Maximum prompt/context tokens the model accepts.",
+    },
+    "max_output_tokens": {
+        **NONNEG_INTEGER,
+        "description": "Maximum tokens the model can generate in one response.",
+    },
+    "output_vector_size": {
+        **NONNEG_INTEGER,
+        "description": "Embedding dimension for embedding models.",
+    },
+    "prompt_cache_min_tokens": {
+        **NONNEG_INTEGER,
+        "description": "Smallest prefix the provider will actually cache; absent means the provider default applies.",
+    },
+    "tpm": {**NONNEG_INTEGER, "description": "Provider default tokens-per-minute limit."},
+    "rpm": {**NONNEG_INTEGER, "description": "Provider default requests-per-minute limit."},
+}
+
+NUMBER_KEYS: dict[str, JsonSchema] = {
+    "regional_processing_uplift_multiplier_eu": {
+        "type": "number",
+        "minimum": 1,
+        "description": "Multiplier applied to all token costs for EU data residency (e.g. 1.10 = +10%).",
+    },
+    "regional_processing_uplift_multiplier_us": {
+        "type": "number",
+        "minimum": 1,
+        "description": "Multiplier applied to all token costs for US data residency (e.g. 1.10 = +10%).",
+    },
+    "regional_endpoint_uplift_multiplier": {
+        "type": "number",
+        "minimum": 1,
+        "description": "Multiplier applied to all token costs when served from a non-global Vertex AI endpoint (e.g. 1.10 = +10%).",
+    },
+}
+
+COST_DESCRIPTIONS: dict[str, str] = {
+    "input_cost_per_token": "USD per prompt token.",
+    "output_cost_per_token": "USD per generated token.",
+    "output_cost_per_reasoning_token": "USD per reasoning/thinking token, when billed separately.",
+    "google_maps_grounding_cost_per_query": (
+        "USD per Grounding with Google Maps request; billed per query or per prompt per web_search_billing_unit."
+    ),
+    "cache_creation_input_token_cost": "USD per token written to the provider's prompt cache.",
+    "cache_read_input_token_cost": "USD per prompt token served from the provider's prompt cache.",
+    "input_cost_per_token_batches": "USD per prompt token via the provider's batch API.",
+    "output_cost_per_token_batches": "USD per generated token via the provider's batch API.",
+}
+
+
+def cost_description(key: str) -> Optional[str]:
+    if key in COST_DESCRIPTIONS:
+        return COST_DESCRIPTIONS[key]
+    if key.endswith("_flex"):
+        return "Flex service-tier rate for the same-named base field."
+    if key.endswith("_priority"):
+        return "Priority service-tier rate for the same-named base field."
+    if "_above_" in key:
+        return "Rate applied once the prompt exceeds the token threshold in the field name."
+    return None
+
+
+def cost_schema(key: str) -> JsonSchema:
+    description = cost_description(key)
+    return {**NONNEG_NUMBER, "description": description} if description else dict(NONNEG_NUMBER)
+
+
+def string_key_schemas(modes: tuple) -> dict[str, JsonSchema]:
+    return {
+        "litellm_provider": {
+            "type": "string",
+            "description": "LiteLLM provider slug; one of https://docs.litellm.ai/docs/providers.",
+        },
+        "mode": {
+            "type": "string",
+            "description": "Primary API surface / task type of the model.",
+            "enum": list(modes),
+        },
+        "source": {
+            "type": "string",
+            "description": "URL of the provider pricing/model page this entry was taken from.",
+        },
+        "deprecation_date": {
+            "type": "string",
+            "description": "Date the provider deprecates the model, YYYY-MM-DD.",
+            "format": "date",
+            "pattern": "^\\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\\d|3[01])$",
+        },
+        "web_search_billing_unit": {
+            "type": "string",
+            "description": "Whether web search is billed per query or per prompt.",
+            "enum": ["per_query", "per_prompt"],
+        },
+        "bedrock_output_config_effort_ceiling": {
+            "type": "string",
+            "description": "Highest reasoning effort the Bedrock output_config accepts for this model.",
+            "enum": ["low", "medium", "high", "max", "xhigh"],
+        },
+        "default_reasoning_effort": {
+            "type": "string",
+            "description": (
+                "Reasoning effort the provider applies when the request omits reasoning_effort. "
+                "Gates whether a non-default temperature or the top_p/logprobs sampling params are "
+                "accepted, which hold only when the effort resolves to 'none'."
+            ),
+            "enum": ["none", "minimal", "low", "medium", "high", "xhigh"],
+        },
+        "comment": STRING,
+        "audio_transcription_config": STRING,
+        "vertex_ai_audio_api": {
+            "type": "string",
+            "enum": ["lyria_predict", "lyria_interactions"],
+        },
+    }
+
+
+def classify(key: str, modes: tuple) -> Optional[JsonSchema]:
+    curated = {**OBJECT_KEYS, **ARRAY_KEYS, **string_key_schemas(modes), **INTEGER_KEYS, **NUMBER_KEYS}
+    if key in curated:
+        return curated[key]
+    if key.startswith("supports_") or key in EXTRA_BOOLEAN_KEYS:
+        return BOOLEAN
+    if "cost" in key:
+        return cost_schema(key)
+    return None
+
+
+def build_schema(prices: dict) -> JsonSchema:
+    entries = {name: entry for name, entry in prices.items() if name not in SPECIAL_ROOT_KEYS}
+    all_keys = tuple(sorted({key for entry in entries.values() for key in entry}))
+    modes = tuple(sorted({entry["mode"] for entry in entries.values() if "mode" in entry}))
+    unclassified = tuple(key for key in all_keys if classify(key, modes) is None)
+    if unclassified:
+        raise SystemExit(
+            f"Unclassified keys in {PRICES_PATH.name}: {', '.join(unclassified)}. "
+            f"Add them to the key tables in {Path(__file__).name} and rerun it."
+        )
+    entry_properties = {key: classify(key, modes) for key in all_keys}
+    return {
+        "$schema": "https://json-schema.org/draft/2020-12/schema",
+        "title": "LiteLLM model_prices_and_context_window.json",
+        "description": (
+            "Schema for LiteLLM's model price and context window registry "
+            "(https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json). "
+            "Every top-level key except 'sample_spec' and 'fallback_generalizations' is a model id, "
+            "optionally prefixed with its provider (e.g. 'azure/gpt-5.4'), mapping to a model entry. "
+            "All costs are USD per unit. New optional fields are added regularly, so consumers should "
+            "ignore unknown fields rather than reject them."
+        ),
+        "type": "object",
+        "properties": {
+            "sample_spec": {
+                "type": "object",
+                "description": (
+                    "Documentation placeholder illustrating the entry shape; not a real model and not "
+                    "schema-conformant (several values are prose)."
+                ),
+            },
+            "fallback_generalizations": {
+                "type": "object",
+                "description": "Regex rules that generalize unknown model ids to known families; not a model entry.",
+                "properties": {
+                    "rules": {
+                        "type": "array",
+                        "items": {
+                            "type": "object",
+                            "properties": {
+                                "name": STRING,
+                                "pattern": STRING,
+                                "description": STRING,
+                            },
+                            "required": ["name", "pattern"],
+                            "additionalProperties": True,
+                        },
+                    }
+                },
+                "additionalProperties": False,
+            },
+        },
+        "additionalProperties": {"$ref": "#/$defs/modelEntry"},
+        "$defs": {
+            "modelEntry": {
+                "type": "object",
+                "description": (
+                    "Pricing, limits, and capability flags for one model. Fields other than litellm_provider "
+                    "are optional; boolean capability flags are simply omitted when unknown or false."
+                ),
+                "required": ["litellm_provider"],
+                "properties": entry_properties,
+                "additionalProperties": True,
+            }
+        },
+    }
+
+
+def render(schema: JsonSchema) -> str:
+    return json.dumps(schema, indent=2) + "\n"
+
+
+def validation_errors(prices: dict, schema: JsonSchema) -> tuple:
+    validator = jsonschema.Draft202012Validator(schema, format_checker=jsonschema.Draft202012Validator.FORMAT_CHECKER)
+    return tuple(
+        f"{'.'.join(str(part) for part in error.absolute_path)}: {error.message}"
+        for error in validator.iter_errors(prices)
+    )
+
+
+def main() -> int:
+    check = "--check" in sys.argv[1:]
+    prices = json.loads(PRICES_PATH.read_text())
+    rendered = render(build_schema(prices))
+    errors = validation_errors(prices, json.loads(rendered))
+    if errors:
+        print(f"{PRICES_PATH.name} does not validate against the generated schema:")
+        print("\n".join(errors[:20]))
+        return 1
+    if not check:
+        SCHEMA_PATH.write_text(rendered)
+        print(f"wrote {SCHEMA_PATH}")
+        return 0
+    if not SCHEMA_PATH.exists() or SCHEMA_PATH.read_text() != rendered:
+        print(
+            f"{SCHEMA_PATH.name} is out of sync with {PRICES_PATH.name}. "
+            f"Run `python {Path(__file__).relative_to(REPO_ROOT)}` and commit the result."
+        )
+        return 1
+    print(f"{SCHEMA_PATH.name} is in sync and {PRICES_PATH.name} validates against it")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

ci_cd/security_scans_readme.md

@@ -4,5 +4,6 @@
 
 - Trivy scan on `./docs/` (HIGH/CRITICAL/MEDIUM)
 - Trivy scan on `./ui/` (HIGH/CRITICAL/MEDIUM) 
+- Grype scan on `Dockerfile.database` (fails on CRITICAL)
 - Grype scan on main `Dockerfile` (fails on CRITICAL)
 - Grype CVSS ≥ 4.0 scan on main `Dockerfile` (fails any vulnerabilities with CVSS ≥ 4.0)

ci_cd/spend_logs_guard.py

@@ -1,226 +0,0 @@
-"""Required check: a pull request that touches LiteLLM_SpendLogs in schema.prisma or a migration
-needs an approving review from the spend-log owner before it can merge.
-
-CODEOWNERS can only match file paths, and every schema change edits the same schema.prisma copies,
-so a path rule would put every migration on the owner. This gate reads the diff instead: a schema.prisma
-hunk counts when it adds or removes a line inside the `model LiteLLM_SpendLogs { ... }` block, a migration
-hunk counts when an added line names the table. Without such a hunk it passes at once, otherwise it passes
-only once the owner's latest review on the pull request is an approval that still covers every spend-log line
-in the head (approvals survive pushes, so a later commit may not add spend-log lines the owner never saw).
-Reviews do not trigger the workflow, so the check is re-run by hand after approving.
-"""
-
-from __future__ import annotations
-
-import argparse
-import re
-import subprocess
-import sys
-from collections.abc import Callable, Mapping, Sequence
-from dataclasses import dataclass
-from pathlib import Path
-from types import MappingProxyType
-from typing import Final
-
-from pydantic import BaseModel, ConfigDict, TypeAdapter
-
-WATCHED: Final = (
-    "schema.prisma",
-    "litellm/proxy/schema.prisma",
-    "litellm-proxy-extras/litellm_proxy_extras/schema.prisma",
-)
-MIGRATIONS_PREFIX: Final = "litellm-proxy-extras/litellm_proxy_extras/migrations/"
-TABLE: Final = re.compile(r"(?<![A-Za-z0-9])LiteLLM_SpendLogs(?![A-Za-z0-9])", re.IGNORECASE)
-DECIDING_STATES: Final = frozenset({"APPROVED", "CHANGES_REQUESTED", "DISMISSED"})
-MODEL_START: Final = re.compile(r"^model\s+LiteLLM_SpendLogs\s*\{")
-HUNK: Final = re.compile(r"^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@[^\n]*\n", re.MULTILINE)
-
-
-class Reviewer(BaseModel):
-    model_config = ConfigDict(frozen=True, extra="ignore")
-    login: str
-
-
-class Review(BaseModel):
-    model_config = ConfigDict(frozen=True, extra="ignore")
-    state: str
-    user: Reviewer
-    commit_id: str | None = None
-
-
-_REVIEWS: Final = TypeAdapter(list[Review])
-
-
-@dataclass(frozen=True, slots=True)
-class Block:
-    """Lines of `model LiteLLM_SpendLogs { ... }` in one schema.prisma copy at the merge base and at the head."""
-
-    base: range
-    head: range
-
-
-@dataclass(frozen=True, slots=True)
-class Hunk:
-    removed: tuple[tuple[int, str], ...]
-    added: tuple[tuple[int, str], ...]
-
-
-def watched(path: str) -> bool:
-    return path in WATCHED or (path.startswith(MIGRATIONS_PREFIX) and path.endswith("/migration.sql"))
-
-
-def model_block(schema: str) -> range:
-    """1-based line range of `model LiteLLM_SpendLogs { ... }` including both braces, empty when the schema has none."""
-    lines: Final = schema.splitlines()
-    starts: Final = tuple(number for number, line in enumerate(lines, 1) if MODEL_START.match(line))
-    if not starts:
-        return range(0)
-    ends: Final = tuple(number for number, line in enumerate(lines, 1) if number > starts[0] and line.startswith("}"))
-    return range(starts[0], (ends[0] if ends else len(lines)) + 1)
-
-
-def _hunks(section: str) -> tuple[Hunk, ...]:
-    headers: Final = tuple(HUNK.finditer(section))
-    ends: Final = tuple(header.start() for header in headers[1:]) + (len(section),)
-    return tuple(_hunk(header, section[header.end() : end]) for header, end in zip(headers, ends))
-
-
-def _hunk(header: re.Match[str], body: str) -> Hunk:
-    lines: Final = body.splitlines()
-    return Hunk(
-        removed=tuple(enumerate((line[1:] for line in lines if line.startswith("-")), int(header.group(1)))),
-        added=tuple(enumerate((line[1:] for line in lines if line.startswith("+")), int(header.group(2)))),
-    )
-
-
-def spend_log_hits(diff: str, blocks: Mapping[str, Block]) -> tuple[str, ...]:
-    """`path: +line` or `path: -line` for every changed line that touches the table.
-
-    A schema.prisma line counts when its number falls inside that copy's `blocks` entry (head for added lines,
-    merge base for removed ones), a migration line when it is added and names the table."""
-    sections: Final = re.split(r"^diff --git [^\n]*\n", diff, flags=re.MULTILINE)
-    paths: Final = tuple((_path_of(section), section) for section in sections)
-    return tuple(
-        hit for path, section in paths if watched(path) for hit in _section_hits(path, _hunks(section), blocks)
-    )  # comprehension-ok: path then its hits
-
-
-def _section_hits(path: str, hunks: Sequence[Hunk], blocks: Mapping[str, Block]) -> tuple[str, ...]:
-    if path not in blocks:
-        return tuple(f"{path}: +{line.strip()}" for hunk in hunks for _, line in hunk.added if TABLE.search(line))
-    block: Final = blocks[path]
-    removed: Final = tuple(
-        f"{path}: -{line.strip()}" for hunk in hunks for number, line in hunk.removed if number in block.base
-    )  # comprehension-ok: hunk then its lines
-    added: Final = tuple(
-        f"{path}: +{line.strip()}" for hunk in hunks for number, line in hunk.added if number in block.head
-    )  # comprehension-ok: hunk then its lines
-    return removed + added
-
-
-def _path_of(section: str) -> str:
-    match: Final = re.search(r"^\+\+\+ b/(.+)$", section, flags=re.MULTILINE)
-    return match.group(1) if match else ""
-
-
-def approved_by(reviews: Sequence[Review], login: str) -> bool:
-    """True when `login`'s most recent review that changes GitHub's approval state is an approval.
-
-    COMMENTED and PENDING reviews leave an earlier approval in force on GitHub, so they are skipped here too.
-    """
-    own: Final = tuple(review.state for review in reviews if review.user.login == login)
-    states: Final = tuple(state for state in own if state in DECIDING_STATES)
-    return bool(states) and states[-1] == "APPROVED"
-
-
-def latest_approval(reviews: Sequence[Review], login: str) -> Review | None:
-    """`login`'s most recent APPROVED review on the pull request, the one GitHub keeps in force after pushes."""
-    own: Final = tuple(review for review in reviews if review.user.login == login and review.state == "APPROVED")
-    return own[-1] if own else None
-
-
-def approval_covers(
-    approval: Review, head: str, hits: Sequence[str], hits_at: Callable[[str], Sequence[str] | None]
-) -> bool:
-    """True when the approval is on the head, or the head adds no spend-log line the approved commit did not."""
-    if approval.commit_id is None:
-        return False
-    if approval.commit_id == head:
-        return True
-    reviewed: Final = hits_at(approval.commit_id)
-    return reviewed is not None and set(hits) <= set(reviewed)
-
-
-def added_diff(base: str, head: str) -> str:
-    result: Final = subprocess.run(
-        ("git", "diff", "--unified=0", "--no-color", f"{base}..{head}", "--", *WATCHED, MIGRATIONS_PREFIX),
-        check=True,
-        capture_output=True,
-        text=True,
-    )
-    return result.stdout
-
-
-def file_at(revision: str, path: str) -> str:
-    """Contents of `path` at `revision`, empty when the file does not exist there."""
-    result: Final = subprocess.run(("git", "show", f"{revision}:{path}"), check=False, capture_output=True, text=True)
-    return result.stdout if result.returncode == 0 else ""
-
-
-def schema_blocks(base: str, head: str) -> Mapping[str, Block]:
-    return MappingProxyType(
-        {path: Block(base=model_block(file_at(base, path)), head=model_block(file_at(head, path))) for path in WATCHED}
-    )
-
-
-def fetched(commit: str) -> bool:
-    result: Final = subprocess.run(("git", "fetch", "--no-tags", "--depth=1", "origin", commit), check=False)
-    return result.returncode == 0
-
-
-def reviewed_hits(base: str, commit: str) -> tuple[str, ...] | None:
-    return spend_log_hits(added_diff(base, commit), schema_blocks(base, commit)) if fetched(commit) else None
-
-
-class Arguments(BaseModel):
-    model_config = ConfigDict(frozen=True)
-    base: str
-    head: str
-    approver: str
-    reviews: Path
-
-
-def main(argv: Sequence[str]) -> int:
-    parser: Final = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--base", required=True, help="merge base of the pull request")
-    parser.add_argument("--head", required=True, help="head sha of the pull request")
-    parser.add_argument("--approver", required=True, help="GitHub login whose approval unlocks the merge")
-    parser.add_argument("--reviews", required=True, help="JSON file with the pull request's reviews")
-    args: Final = Arguments.model_validate(vars(parser.parse_args(argv)))
-    hits: Final = spend_log_hits(added_diff(args.base, args.head), schema_blocks(args.base, args.head))
-    if not hits:
-        sys.stdout.write(
-            "No change inside model LiteLLM_SpendLogs in schema.prisma and no added migration line names it; "
-            "nothing to review.\n"
-        )
-        return 0
-    reviews: Final = _REVIEWS.validate_json(args.reviews.read_bytes())
-    approval: Final = latest_approval(reviews, args.approver) if approved_by(reviews, args.approver) else None
-    if approval is not None and approval_covers(
-        approval, args.head, hits, lambda commit: reviewed_hits(args.base, commit)
-    ):
-        sys.stdout.write(f"LiteLLM_SpendLogs change approved by {args.approver}: {', '.join(hits)}\n")
-        return 0
-    listed: Final = "".join(f"  {hit}\n" for hit in hits)
-    reason: Final = (
-        f"This pull request changes LiteLLM_SpendLogs and needs an approving review from @{args.approver}:\n"
-        if approval is None
-        else f"LiteLLM_SpendLogs lines changed after @{args.approver} approved {approval.commit_id}, "
-        f"a new approving review is needed:\n"
-    )
-    sys.stdout.write(f"{reason}{listed}After approving, re-run this job.\n")
-    return 1
-
-
-if __name__ == "__main__":
-    sys.exit(main(sys.argv[1:]))

codecov.yaml

@@ -15,10 +15,9 @@ ignore:
 flag_management:
   default_rules:
     carryforward: true
-  # Dead flags no CI job uploads anymore, and flags whose job is path-gated on
+  # Dead flags no CI job uploads anymore: their carried-forward sessions were
-  # PRs: their carried-forward sessions were measured against old revisions, and
+  # measured against old revisions, and the stale line maps mark comment lines
-  # the stale line maps mark comment lines of since-edited files as missed,
+  # of since-edited files as missed, sinking patch coverage on unrelated PRs.
-  # sinking patch coverage on unrelated PRs.
   individual_flags:
     - name: proxy-mgmt-behavior
       carryforward: false
@@ -100,10 +99,6 @@ flag_management:
       carryforward: false
     - name: proxy-db-endpoints-and-responses
       carryforward: false
-    - name: roi-postgres
-      carryforward: false
-    - name: redis-compat
-      carryforward: false
 
 component_management:
   individual_components:

cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md

@@ -1,114 +0,0 @@
-# Use your ChatGPT subscription through LiteLLM
-
-Sign into ChatGPT in Codex or Pi and send model requests through your existing LiteLLM proxy. The client keeps your login and refreshes it. Each request uses that client's ChatGPT account, while a separate LiteLLM virtual key controls gateway access and usage attribution.
-
-An administrator adds a `chatgpt/` model using the usual model configuration. No ChatGPT login needs to be added to the proxy. This uses the same separate-key convention as provider passthrough, including Claude Code subscription forwarding.
-
-## Add a model to the proxy
-
-Reuse your existing proxy. If you need one, follow the [proxy quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start). Merge this entry into its model list and restart the proxy:
-
-```yaml
-model_list:
-  - model_name: gpt-6-luna
-    litellm_params:
-      model: chatgpt/gpt-6-luna
-    model_info:
-      mode: responses
-```
-
-Use a model available to the caller's subscription. The virtual key must allow the exposed model name. Leave saved ChatGPT logins and API credentials unset for a client-only deployment. Explicit cross-provider fallbacks can incur separate API charges; do not configure them for subscription-only traffic.
-
-## Configure Codex
-
-Use your existing LiteLLM URL and a virtual key assigned to your gateway user. Export that key as `LITELLM_API_KEY` in the environment that launches Codex. Keep the value out of committed configuration.
-
-Add this provider to `~/.codex/config.toml`, preserving your other settings:
-
-```toml
-model = "gpt-6-luna"
-model_provider = "litellm_subscription"
-
-[model_providers.litellm_subscription]
-name = "LiteLLM subscription"
-base_url = "https://litellm.example.com/v1"
-wire_api = "responses"
-supports_websockets = false
-requires_openai_auth = true
-env_http_headers = { "x-litellm-api-key" = "LITELLM_API_KEY" }
-```
-
-Replace `https://litellm.example.com` with your proxy URL. A local proxy normally uses `http://localhost:4000/v1`. Launch Codex, sign into ChatGPT if needed, and send a prompt with a model available to your subscription. The selected model must match the name exposed by the gateway.
-
-The `lite` CLI can set up this provider each time it starts Codex, with no `config.toml` edit. Export `LITELLM_PROXY_URL` and `LITELLM_PROXY_API_KEY`, then run `lite codex --subscription`. Run `lite config set codex_subscription true` to make it the default.
-
-For a managed workspace rollout, use OpenAI's [gateway configuration guide](https://learn.chatgpt.com/docs/enterprise/sign-in-with-chatgpt-through-a-gateway).
-
-## Native compatibility and Pi
-
-For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials. A ChatGPT model keeps its upstream name and needs no deployment of its own.
-
-### Mix ChatGPT and LiteLLM models
-
-On the native route, Codex's model picker shows OpenAI's catalog first and then the other chat and responses models the virtual key can use. Pick one of those and Codex's requests go through LiteLLM's `/v1/responses` with the key's usual routing, budgets, guardrails, and logging. The proxy removes the ChatGPT bearer and account header from that request, so the other provider and your callbacks never receive them. A name is routed this way when another provider serves it as an exact `model_name`, through a router `model_group_alias`, or as a team's public model name. A ChatGPT model, a name that only a wildcard deployment matches, or a name OpenAI's catalog lists for the caller's ChatGPT account keeps the passthrough, so a model OpenAI adds later still uses the subscription even when a deployment of another provider has the same name. The proxy learns the account's list from the catalog Codex fetches, or asks OpenAI with the caller's login. When it cannot get the list, for example because OpenAI is down, the request times out, or the client is not Codex, every request keeps the passthrough for the next minute, as it did before LiteLLM models were added to the picker. When the key can use many models, the picker lists the ones that fit in Codex's catalog size limit, and the proxy logs a warning with how many it left out.
-
-Codex's picker sends a `none` reasoning effort for a LiteLLM model whose catalog entry lists no effort levels. With `litellm_settings: drop_params: true`, the proxy drops it for a model that does not support `none`, and the provider applies its own default. A model whose cost map entry or `model_info` sets `supports_none_reasoning_effort: true` keeps it. Without `drop_params`, the effort reaches the provider unchanged.
-
-Codex also sends its ChatGPT app tools with every turn, which on a LiteLLM model adds a few hundred thousand prompt tokens to each turn. Set `apps = false` under `[features]` in Codex's `config.toml` to turn them off.
-
-Only a key that may call `/v1/responses` gets these models. A key whose `allowed_routes` covers only the native routes sees OpenAI's catalog unchanged, and every request it sends uses the subscription.
-
-Codex sends earlier reasoning items back with each turn. The proxy tags the reasoning that a LiteLLM model returns on this route, OpenAI and Azure deployments included. When a session switches to a ChatGPT model, the proxy drops those items before forwarding the request to OpenAI, which would reject them. Messages, tool calls, and ChatGPT's own reasoning are kept, and a LiteLLM model still gets its own reasoning back.
-
-When the catalog adds LiteLLM models for a key, its `ETag` is a digest of OpenAI's ETag and those models, and that key's responses carry the same value in `X-Models-Etag`. Codex keeps its cached catalog while the two match and fetches it again when OpenAI's catalog or the key's models change. A key whose catalog adds no LiteLLM models keeps OpenAI's ETags.
-
-### Restrict native models with an access group
-
-A key with a model allowlist needs a rule for ChatGPT names that OpenAI adds over time. Put the subscription wildcards in an access group under `chatgpt/` names:
-
-```yaml
-model_list:
-  - model_name: chatgpt/gpt-*
-    litellm_params:
-      model: chatgpt/gpt-*
-    model_info:
-      access_groups: ["chatgpt-native"]
-  - model_name: chatgpt/codex-*
-    litellm_params:
-      model: chatgpt/codex-*
-    model_info:
-      access_groups: ["chatgpt-native"]
-```
-
-Then add `chatgpt-native` to the key's models, next to any other model names it should reach. On the native route, a key that holds only the `chatgpt/` form of a name is checked as that form, so a request for `gpt-5.6-sol` is allowed when the key may call `chatgpt/gpt-5.6-sol`. The key can call every ChatGPT model that matches the wildcards, including ones the proxy configuration does not name.
-
-On `/v1` these wildcards match only a request that asks for a `chatgpt/` name, so a request for a plain `gpt-` or `codex-` name still goes where it went before, such as a `*` catch-all deployment. A `/v1` request for a `chatgpt/` name without a ChatGPT login falls back to the proxy's own saved login if it has one, so keep that login unset as described above.
-
-### Configure Pi
-
-Sign into the built-in `openai-codex` provider in Pi. Merge this override into `~/.pi/agent/models.json` and export your LiteLLM virtual key as `LITELLM_API_KEY` before launching Pi:
-
-```json
-{
-  "providers": {
-    "openai-codex": {
-      "baseUrl": "https://litellm.example.com/backend-api/codex",
-      "headers": {
-        "x-litellm-api-key": "LITELLM_API_KEY"
-      }
-    }
-  }
-}
-```
-
-Keep Pi's subscription login as the provider credential. The header value names an environment variable, not a key to paste into the file.
-
-## Verify account usage
-
-Send a prompt and check the completed request in LiteLLM's Logs page. Its gateway user/key and token counts should match the caller. Give each person their own LiteLLM virtual key for separate attribution. The gateway credential and ChatGPT login are independently authenticated; LiteLLM does not establish that both identities belong to the same person.
-
-The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key needs the `chatgpt-native` access group above or that exact name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
-
-The proxy forwards the ChatGPT bearer and account header to the fixed OpenAI Codex endpoint. It removes gateway credentials and cookies from the upstream request and redacts credential headers in logging callbacks. The client remains responsible for refresh and reauthentication. Native passthrough returns authentication and quota errors without switching subscriptions. Standard model routes retain the caller login across ChatGPT retries; explicitly configured cross-provider fallbacks use their own credentials. An explicit workspace header is preserved for OpenAI to validate.
-
-This flow does not connect accounts through the dashboard or pool subscription allowances. Use a Codex build that supports ChatGPT authentication with a custom model provider. The integration forwards HTTP and SSE requests; it does not add native WebSocket or other Codex backend routes.

cookbook/gpt_realtime_translate.py

@@ -1,245 +0,0 @@
-#!/usr/bin/env python3
-
-import argparse
-import asyncio
-import base64
-import json
-import os
-import sys
-import wave
-from collections.abc import Iterator, Sequence
-from dataclasses import dataclass
-from pathlib import Path
-from urllib.parse import urlencode, urlsplit, urlunsplit
-
-import websockets
-from websockets.asyncio.client import ClientConnection
-
-SAMPLE_RATE = 24_000
-CHANNELS = 1
-SAMPLE_WIDTH = 2
-CHUNK_DURATION_SECONDS = 0.1
-CHUNK_BYTES = int(SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH * CHUNK_DURATION_SECONDS)
-RESPONSE_TIMEOUT_SECONDS = 30.0
-AUDIO_EVENT_TYPES = frozenset(
-    {
-        "session.output_audio.delta",
-        "response.audio.delta",
-        "response.output_audio.delta",
-    }
-)
-TRANSCRIPT_EVENT_TYPES = frozenset(
-    {
-        "session.output_transcript.delta",
-        "response.text.delta",
-        "response.output_audio_transcript.delta",
-    }
-)
-
-
-@dataclass(frozen=True, slots=True)
-class Settings:
-    input_wav: Path
-    output_wav: Path
-    base_url: str
-    model: str
-    target_language: str
-    trailing_silence_seconds: float
-    api_key: str
-
-
-def write_stdout(message: str = "", *, end: str = "\n", flush: bool = False) -> None:
-    sys.stdout.write(f"{message}{end}")
-    if flush:
-        sys.stdout.flush()
-
-
-def write_stderr(message: str) -> None:
-    sys.stderr.write(f"{message}\n")
-
-
-def parse_args(argv: Sequence[str] | None = None) -> Settings | str:
-    parser = argparse.ArgumentParser(
-        description="Stream a 24 kHz PCM16 WAV through gpt-realtime-translate and save the translated audio",
-    )
-    parser.add_argument("input_wav", type=Path)
-    parser.add_argument("--output", type=Path, default=Path("translated.wav"))
-    parser.add_argument("--base-url", default=os.getenv("LITELLM_BASE_URL", "http://localhost:4000"))
-    parser.add_argument("--model", default=os.getenv("REALTIME_TRANSLATE_MODEL", "gpt-realtime-translate"))
-    parser.add_argument("--target-language", default="fr")
-    parser.add_argument("--trailing-silence", type=float, default=1.5)
-    parsed = parser.parse_args(argv)
-    api_key = os.getenv("LITELLM_API_KEY") or os.getenv("OPENAI_API_KEY")
-    if not api_key:
-        return "Set LITELLM_API_KEY or OPENAI_API_KEY before running the script"
-    if parsed.trailing_silence < 0:
-        return "--trailing-silence must be zero or greater"
-    return Settings(
-        input_wav=parsed.input_wav,
-        output_wav=parsed.output,
-        base_url=parsed.base_url,
-        model=parsed.model,
-        target_language=parsed.target_language,
-        trailing_silence_seconds=parsed.trailing_silence,
-        api_key=api_key,
-    )
-
-
-def translation_url(base_url: str, model: str) -> str | None:
-    parsed = urlsplit(base_url.rstrip("/"))
-    scheme = {"http": "ws", "https": "wss", "ws": "ws", "wss": "wss"}.get(parsed.scheme)
-    if not scheme or not parsed.netloc:
-        return None
-    base_path = parsed.path.rstrip("/")
-    realtime_path = (
-        f"{base_path}/realtime/translations" if base_path.endswith("/v1") else f"{base_path}/v1/realtime/translations"
-    )
-    return urlunsplit((scheme, parsed.netloc, realtime_path, urlencode({"model": model}), ""))
-
-
-def read_pcm16_wav(path: Path) -> bytes | str:
-    try:
-        with wave.open(str(path), "rb") as source:
-            actual_format = (
-                source.getnchannels(),
-                source.getsampwidth(),
-                source.getframerate(),
-                source.getcomptype(),
-            )
-            expected_format = (CHANNELS, SAMPLE_WIDTH, SAMPLE_RATE, "NONE")
-            if actual_format != expected_format:
-                return (
-                    f"{path} must be mono, 16-bit PCM, 24 kHz WAV; received "
-                    f"channels={actual_format[0]}, sample_width={actual_format[1]}, "
-                    f"sample_rate={actual_format[2]}, compression={actual_format[3]}"
-                )
-            return source.readframes(source.getnframes())
-    except (OSError, EOFError, wave.Error) as exc:
-        return f"Unable to read {path}: {exc}"
-
-
-def audio_chunks(audio: bytes) -> Iterator[bytes]:
-    return (audio[offset : offset + CHUNK_BYTES] for offset in range(0, len(audio), CHUNK_BYTES))
-
-
-def audio_message(audio: bytes) -> str:
-    return json.dumps(
-        {
-            "type": "session.input_audio_buffer.append",
-            "audio": base64.b64encode(audio).decode("ascii"),
-        }
-    )
-
-
-async def configure_session(connection: ClientConnection, target_language: str) -> str | None:
-    await connection.send(
-        json.dumps(
-            {
-                "type": "session.update",
-                "session": {"audio": {"output": {"language": target_language}}},
-            }
-        )
-    )
-    while True:
-        raw_event = await asyncio.wait_for(connection.recv(), timeout=20)
-        event = json.loads(raw_event)
-        event_type = event.get("type")
-        if event_type == "session.created":
-            write_stdout(f"Session: {event.get('session', {}).get('id', 'created')}")
-        if event_type == "session.updated":
-            return None
-        if event_type == "error":
-            return f"Realtime API error: {json.dumps(event.get('error', event), ensure_ascii=False)}"
-
-
-async def send_audio(
-    connection: ClientConnection, pcm: bytes, trailing_silence_seconds: float, finished: asyncio.Event
-) -> None:
-    silence = bytes(round(trailing_silence_seconds * SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH))
-    try:
-        for chunk in audio_chunks(pcm + silence):
-            await connection.send(audio_message(chunk))
-            await asyncio.sleep(len(chunk) / (SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH))
-        await connection.send(json.dumps({"type": "session.close"}))
-    finally:
-        finished.set()
-
-
-async def receive_translation(
-    connection: ClientConnection, output_path: Path, sender_finished: asyncio.Event
-) -> str | None:
-    audio_received = asyncio.Event()
-    try:
-        with wave.open(str(output_path), "wb") as output:
-            output.setnchannels(CHANNELS)
-            output.setsampwidth(SAMPLE_WIDTH)
-            output.setframerate(SAMPLE_RATE)
-            write_stdout("Translation: ", end="", flush=True)
-            while True:
-                try:
-                    raw_event = await asyncio.wait_for(connection.recv(), timeout=RESPONSE_TIMEOUT_SECONDS)
-                except TimeoutError:
-                    if not sender_finished.is_set():
-                        continue
-                    return (
-                        f"The translation stream went quiet for {RESPONSE_TIMEOUT_SECONDS:g} s without closing the session"
-                    )
-                event = json.loads(raw_event)
-                event_type = event.get("type")
-                if event_type in AUDIO_EVENT_TYPES:
-                    output.writeframes(base64.b64decode(event.get("delta", ""), validate=True))
-                    audio_received.set()
-                elif event_type in TRANSCRIPT_EVENT_TYPES:
-                    write_stdout(event.get("delta", event.get("text", "")), end="", flush=True)
-                elif event_type == "error":
-                    return f"Realtime API error: {json.dumps(event.get('error', event), ensure_ascii=False)}"
-                elif event_type == "session.closed":
-                    write_stdout()
-                    return None if audio_received.is_set() else "The translation stream ended without translated audio"
-    except (OSError, wave.Error) as exc:
-        return f"Unable to write {output_path}: {exc}"
-
-
-async def translate(settings: Settings, pcm: bytes) -> str | None:
-    url = translation_url(settings.base_url, settings.model)
-    if not url:
-        return f"Invalid --base-url: {settings.base_url}"
-    sender_finished = asyncio.Event()
-    try:
-        async with websockets.connect(
-            url,
-            additional_headers={"Authorization": f"Bearer {settings.api_key}"},
-            proxy=None,
-            open_timeout=20,
-            close_timeout=5,
-        ) as connection:
-            configuration_error = await configure_session(connection, settings.target_language)
-            if configuration_error:
-                return configuration_error
-            async with asyncio.TaskGroup() as tasks:
-                receiver = tasks.create_task(receive_translation(connection, settings.output_wav, sender_finished))
-                tasks.create_task(send_audio(connection, pcm, settings.trailing_silence_seconds, sender_finished))
-            return receiver.result()
-    except Exception as exc:
-        return f"Translation failed: {type(exc).__name__}: {exc}"
-
-
-def main(argv: Sequence[str] | None = None) -> int:
-    settings = parse_args(argv)
-    if isinstance(settings, str):
-        write_stderr(settings)
-        return 2
-    pcm = read_pcm16_wav(settings.input_wav)
-    if isinstance(pcm, str):
-        write_stderr(pcm)
-        return 2
-    error = asyncio.run(translate(settings, pcm))
-    if error:
-        write_stderr(error)
-        return 1
-    write_stdout(f"Translated audio: {settings.output_wav}")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

cookbook/litellm-ollama-docker-image/Dockerfile

@@ -0,0 +1,25 @@
+FROM ollama/ollama as ollama
+
+RUN echo "auto installing llama2"
+
+# auto install ollama/llama2
+RUN ollama serve & sleep 2 && ollama pull llama2
+
+RUN echo "installing litellm"
+
+RUN apt-get update
+
+# Install Python
+RUN apt-get install -y python3 python3-pip
+
+# Set the working directory in the container
+WORKDIR /app
+
+# Copy the current directory contents into the container at /app
+COPY . /app
+
+# Install any needed packages specified in requirements.txt
+
+RUN python3 -m pip install litellm
+COPY start.sh /start.sh
+ENTRYPOINT [ "/bin/bash", "/start.sh" ]

cookbook/litellm-ollama-docker-image/requirements.txt

@@ -0,0 +1 @@
+litellm==1.83.14

cookbook/litellm-ollama-docker-image/start.sh

@@ -0,0 +1,2 @@
+ollama serve &
+litellm

cookbook/litellm-ollama-docker-image/test.py

@@ -0,0 +1,35 @@
+import openai
+
+api_base = "http://0.0.0.0:8000"
+
+openai.api_base = api_base
+openai.api_key = "temp-key"
+print(openai.api_base)
+
+
+print("LiteLLM: response from proxy with streaming")
+response = openai.ChatCompletion.create(
+    model="ollama/llama2",
+    messages=[
+        {
+            "role": "user",
+            "content": "this is a test request, acknowledge that you got it",
+        }
+    ],
+    stream=True,
+)
+
+for chunk in response:
+    print(f"LiteLLM: streaming response from proxy {chunk}")
+
+response = openai.ChatCompletion.create(
+    model="ollama/llama2",
+    messages=[
+        {
+            "role": "user",
+            "content": "this is a test request, acknowledge that you got it",
+        }
+    ],
+)
+
+print(f"LiteLLM: response from proxy {response}")

cookbook/litellm_proxy_server/batch_api/bedrock/README.md

@@ -1,41 +0,0 @@
-# Bedrock batch endpoints
-
-Set `aws_bedrock_control_plane_endpoint` on a deployment when batch create, retrieve, and cancel requests must use a custom Bedrock control-plane endpoint. The proxy signs these requests for the configured URL. Without this setting, batch requests use the standard regional Bedrock endpoint
-
-```yaml
-model_list:
-  - model_name: bedrock-batch
-    litellm_params:
-      model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
-      aws_region_name: us-east-1
-      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
-      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
-      aws_batch_role_arn: os.environ/AWS_BATCH_ROLE_ARN
-      s3_output_bucket_name: os.environ/AWS_S3_OUTPUT_BUCKET_NAME
-      aws_bedrock_control_plane_endpoint: os.environ/BEDROCK_CONTROL_PLANE_ENDPOINT
-```
-
-Set `BEDROCK_CONTROL_PLANE_ENDPOINT` to your Bedrock control-plane VPC endpoint or compatible gateway base URL, without `/model-invocation-job`. The proxy adds the operation path. The endpoint must be reachable from the proxy, and the configured AWS region must match the endpoint's signing region
-
-`api_base` and `aws_bedrock_runtime_endpoint` configure runtime inference separately. They do not override the batch control-plane endpoint. `s3_endpoint_url` configures S3 file access separately
-
-The control-plane setting belongs in deployment configuration. Sending `aws_bedrock_control_plane_endpoint` in an API request body is rejected. There is no automatic `AWS_BEDROCK_CONTROL_PLANE_ENDPOINT` environment fallback, so reference the desired environment variable explicitly as shown above
-
-Start the proxy with this configuration and set `LITELLM_API_BASE` to its URL. Set `BEDROCK_BATCH_INPUT_URI` to a valid input JSONL file in S3 that the batch service role can read
-
-```bash
-curl -sS "$LITELLM_API_BASE/v1/batches" \
-  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-  -H 'Content-Type: application/json' \
-  -d "{\"model\":\"bedrock-batch\",\"input_file_id\":\"$BEDROCK_BATCH_INPUT_URI\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\"}"
-```
-
-Use the returned `id` as `BATCH_ID` to retrieve or cancel the same job through the deployment's configured endpoint
-
-```bash
-curl -sS "$LITELLM_API_BASE/v1/batches/$BATCH_ID?model=bedrock-batch" \
-  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
-
-curl -sS -X POST "$LITELLM_API_BASE/v1/batches/$BATCH_ID/cancel?model=bedrock-batch" \
-  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
-```

cookbook/litellm_proxy_server/grafana_dashboard/dashboard_all_metrics/grafana_dashboard.json

@@ -6563,87 +6563,6 @@
       ],
       "title": "litellm_project_model_rate_limit_used_metric",
       "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 447
-      },
-      "id": 115,
-      "panels": [],
-      "title": "Agent tracing export (needs LITELLM_LENS_URL)",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "description": "Request records the Lens exporter delivered to the Lens service against the records it dropped before the service accepted them",
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "showPoints": "never",
-            "spanNulls": false
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 448
-      },
-      "id": 116,
-      "options": {
-        "legend": {
-          "calcs": [],
-          "displayMode": "list",
-          "placement": "bottom",
-          "showLegend": true
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "editorMode": "code",
-          "expr": "sum(rate(litellm_lens_export_rows_written_total[$__rate_interval]))",
-          "legendFormat": "written",
-          "range": true,
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "editorMode": "code",
-          "expr": "sum(rate(litellm_lens_export_rows_dropped_total[$__rate_interval]))",
-          "legendFormat": "dropped",
-          "range": true,
-          "refId": "B"
-        }
-      ],
-      "title": "litellm_lens_export_rows_written and litellm_lens_export_rows_dropped rate",
-      "type": "timeseries"
     }
   ],
   "preload": false,

cookbook/litellm_proxy_server/grafana_dashboard/dashboard_all_metrics/readme.md

@@ -1,10 +1,10 @@
 # LiteLLM All Prometheus Metrics dashboard
 
-Every `litellm_*` metric family the proxy can expose on `/metrics` (143 families across 98 panels), grouped into rows: proxy traffic, latency, spend and tokens, cache, LLM API deployments, key and team rate limits, budgets, guardrails, MCP, managed files and batches, users and teams, the Redis circuit breaker, the spend log cleanup job, the `prometheus_system` service callback metrics (per-service latency, request and failure rates, spend update queue sizes), project model rate limits, and the agent tracing export to Lens. Panel titles are the metric names so you can grep the JSON for the metric you care about
+Every `litellm_*` metric family the proxy can expose on `/metrics` (141 families across 97 panels), grouped into rows: proxy traffic, latency, spend and tokens, cache, LLM API deployments, key and team rate limits, budgets, guardrails, MCP, managed files and batches, users and teams, the Redis circuit breaker, the spend log cleanup job, and the `prometheus_system` service callback metrics (per-service latency, request and failure rates, spend update queue sizes). Panel titles are the metric names so you can grep the JSON for the metric you care about
 
 Import `grafana_dashboard.json` from **Dashboards > New > Import** and pick your Prometheus data source when prompted (the `DS_PROMETHEUS` variable). Counters are plotted as `rate()` over `$__rate_interval`, histograms as p50 / p95 / p99, gauges as the raw value grouped by the most useful label. Every query names the metric exactly as the proxy emits it (counters carry the `_total` suffix the Prometheus client adds), and `tests/unit/integrations/test_prometheus_metric_name_consistency.py` fails if a metric is renamed without updating this dashboard
 
-The first eleven rows and the project model rate limits row need only `callbacks: ["prometheus"]`. The other rows and the `litellm_admission_*` panels are emitted by other subsystems and stay empty until those are on: the service callback row needs `service_callback: ["prometheus_system"]` in `litellm_settings`, the circuit breaker row needs a Redis cache, the cleanup row needs spend log retention, the agent tracing export row needs `LITELLM_LENS_URL`, and admission control needs its middleware enabled. Within the base rows, many panels only fill in once the matching feature is in use: budgets need keys, teams, users or orgs with `max_budget` set, cache panels need caching on, guardrail and MCP panels need those features configured, deployment health needs the router with more than one deployment or a failure to record, and `litellm_in_flight_requests` needs traffic at scrape time. An empty panel for a feature you do not use is expected
+The first eleven rows need only `callbacks: ["prometheus"]`. The last three rows and the `litellm_admission_*` panels are emitted by other subsystems and stay empty until those are on: the service callback row needs `service_callback: ["prometheus_system"]` in `litellm_settings`, the circuit breaker row needs a Redis cache, the cleanup row needs spend log retention, and admission control needs its middleware enabled. Within the base rows, many panels only fill in once the matching feature is in use: budgets need keys, teams, users or orgs with `max_budget` set, cache panels need caching on, guardrail and MCP panels need those features configured, deployment health needs the router with more than one deployment or a failure to record, and `litellm_in_flight_requests` needs traffic at scrape time. An empty panel for a feature you do not use is expected
 
 ## Pre-requisites
 

cookbook/skill_scanner/README.md

@@ -1,60 +0,0 @@
-# Manual skill security scans
-
-This standalone adapter runs NVIDIA SkillSpector on a separate machine. It is not installed in the LiteLLM proxy and does not require a shared Helm or Compose deployment
-
-Start it with Python 3.12 or newer and uv. The script pins SkillSpector and its HTTP dependencies
-
-```sh
-export SKILL_SCANNER_API_KEY="$(openssl rand -hex 32)"
-uv run server.py
-```
-
-For another host, place the loopback service behind an HTTPS reverse proxy that forwards the Authorization header and enforces a 65 MiB request limit. Run it under an unprivileged account with a writable temporary directory and no production files or credentials. The scanner host needs Git and network access to requested sources and optional model providers. Apply network egress restrictions suitable for those sources
-
-Configure the same scanner secret in the gateway's environment, then add this to LiteLLM's configuration
-
-```yaml
-general_settings:
-  skill_scanner:
-    endpoint: https://scanner.example.com/scan
-    api_key_env: SKILL_SCANNER_API_KEY
-    use_llm: false
-    timeout_seconds: 180
-```
-
-The endpoint must be reachable from the gateway. A customer's laptop works while it is reachable through their network, VPN or authenticated HTTPS tunnel. `localhost` refers to the gateway machine, not the admin's browser. HTTP is accepted only for loopback development
-
-Open Skills, Submissions, and Review as a proxy admin. Click **Scan for threats**, inspect the report, then independently approve or reject the proposal. Scanning is optional and never changes approval permissions or blocks publication. The report shows findings, locations, remediation, risk score, scanner version and whether semantic analysis actually ran. Incomplete reports do not establish safety
-
-Uploaded ZIPs are sent privately to the scanner with their exact stored bytes. Metadata-only edits reuse the matching stored archive. If that archive is missing or changed, submit a new ZIP before scanning. The displayed SHA-256 identifies that archive, and the scan examines its files rather than proposed catalog metadata. Remote URL reports describe the source at scan time and do not freeze its contents. GitHub branch/tag refs and explicit-ref subdirectories are supported. Other ref/subdirectory combinations require a ZIP so the adapter does not silently scan the wrong revision
-
-The latest scan result is retained with the submission. Running scans have a deadline. If a gateway restarts or the scanner becomes unavailable, the attempt expires and an admin can retry. Retries replace the latest report. Approving or rejecting during a scan does not cancel it or change the decision
-
-Set `use_llm: true` to request semantic analysis and configure the provider on the scanner machine using [SkillSpector's provider instructions](https://github.com/NVIDIA/skillspector#llm-analysis). Provider credentials stay on that machine. Static analysis can also call OSV for dependency checks
-
-## HTTP contract
-
-The gateway sends an authenticated `POST` to the configured endpoint. The body contains form fields `source` (JSON source descriptor), `use_llm` (boolean), and `expires_at` (timezone-aware ISO timestamp), plus `file` (ZIP) when bytes are available. The service returns HTTP 200 with this JSON shape
-
-```json
-{
-  "scanner_version": "2.12.0",
-  "complete": true,
-  "scan_mode": "static-only",
-  "risk_score": 70,
-  "findings": [{
-    "rule_id": "example-rule",
-    "severity": "HIGH",
-    "message": "Potential data exfiltration",
-    "file": "SKILL.md",
-    "start_line": 8,
-    "explanation": "Instructions request sending local files to an external host",
-    "remediation": "Remove the external upload instruction"
-  }],
-  "limitations": []
-}
-```
-
-Reports are limited to 1 MiB and 500 findings. Scanner uploads allow 64 MiB because the gateway can expand compressed skill uploads into stored compatibility ZIPs. The gateway rejects malformed reports, refuses redirects and verifies TLS. Only the configured scanner credential is sent. A timeout, HTTP error or incomplete report remains visible to the reviewer
-
-The bundled adapter runs at most two child scans concurrently. Each child and its descendants are terminated at the gateway deadline, capped at 570 seconds. Keep the gateway and scanner clocks synchronized. Direct adapter clients may omit `expires_at` to use the cap. Child processes never execute submitted skill scripts. Raise the gateway deadline up to 600 seconds when needed for semantic analysis and configure the reverse proxy timeout to match

cookbook/skill_scanner/server.py

@@ -1,172 +0,0 @@
-# /// script
-# requires-python = ">=3.12"
-# dependencies = [
-#   "fastapi==0.162.0", "uvicorn==0.54.0", "python-multipart==0.0.33",
-#   "skillspector @ git+https://github.com/NVIDIA/skillspector.git@62c2fe8622f580ac292d4078c3b71f790194dbbd",
-# ]
-# ///
-import asyncio
-import hmac
-import json
-import os
-import signal
-import sys
-import tempfile
-from contextlib import suppress
-from datetime import datetime, timezone
-from pathlib import Path
-from typing import Annotated, Final
-from urllib.parse import quote, urlsplit
-
-from fastapi import Depends, FastAPI, File, Form, HTTPException, UploadFile
-from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
-from pydantic import AwareDatetime, TypeAdapter, ValidationError
-
-MAX_UPLOAD: Final = 64 * 1024 * 1024
-MAX_REPORT: Final = 1024 * 1024
-SCAN_TIMEOUT: Final = 570
-SOURCE: Final = TypeAdapter(dict[str, str])
-SLOTS: Final = asyncio.Semaphore(2)
-app: Final = FastAPI(title="SkillSpector adapter", docs_url=None, redoc_url=None, openapi_url=None)
-
-
-def authenticate(credentials: Annotated[HTTPAuthorizationCredentials, Depends(HTTPBearer())]) -> None:
-    token: Final = os.environ.get("SKILL_SCANNER_API_KEY", "")
-    if not token or not hmac.compare_digest(credentials.credentials, token):
-        raise HTTPException(401, "Invalid scanner credentials")
-
-
-def source_target(source: dict[str, str]) -> str:
-    target: Final = (
-        "https://github.com/" + source.get("repo", "") if source.get("source") == "github" else source.get("url", "")
-    )
-    parsed: Final = urlsplit(target)
-    if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password or parsed.fragment:
-        raise HTTPException(400, "Remote sources require an HTTPS URL without credentials or a fragment")
-    ref: Final = source.get("ref")
-    subdir: Final = source.get("path")
-    if ref or subdir:
-        if parsed.hostname != "github.com" or len(parsed.path.strip("/").split("/")) != 2 or not ref:
-            raise HTTPException(400, "Upload a ZIP to scan this source's exact ref and subdirectory")
-        suffix: Final = "/" + quote(subdir, safe="/") if subdir else ""
-        return target.removesuffix(".git").rstrip("/") + "/tree/" + quote(ref, safe="/") + suffix
-    return target
-
-
-async def worker(target: str, output: Path, use_llm: bool) -> None:
-    from skillspector.mcp_server import run_scan
-
-    result: Final = await run_scan(target, use_llm=use_llm, allow_local_targets=not target.startswith("https://"))
-    coverage: Final = result.get("analysis_completeness", {})
-    findings: Final = result["findings"]
-    complete: Final = (
-        result.get("execution_successful") is True
-        and coverage.get("is_complete") is True
-        and (not use_llm or (result.get("llm_available") is True and result.get("llm_used") is True))
-        and len(findings) <= 500
-    )
-    limitations: Final = tuple(str(value)[:2000] for value in coverage.get("limitations", ()))
-    report: Final = {
-        "scanner_version": result["version"],
-        "complete": complete,
-        "scan_mode": result["scan_mode"],
-        "risk_score": result["risk_score"],
-        "findings": [
-            {
-                "rule_id": finding["id"],
-                "severity": finding["severity"],
-                "message": finding.get("pattern") or finding.get("finding") or "Security finding",
-                "file": finding["location"]["file"],
-                "start_line": finding["location"].get("start_line"),
-                "explanation": finding.get("explanation"),
-                "remediation": finding.get("remediation"),
-            }
-            for finding in findings[:500]
-        ],
-        "limitations": (
-            *limitations,
-            *(("Only the first 500 findings are displayed.",) if len(findings) > 500 else ()),
-        ),
-    }
-    encoded: Final = json.dumps(report).encode()
-    if len(encoded) > MAX_REPORT:
-        raise ValueError("Scanner report exceeded the response limit")
-    output.write_bytes(encoded)
-
-
-async def run_worker(target: str, output: Path, use_llm: bool, expires_at: datetime | None = None) -> dict[str, object]:
-    timeout: Final = (
-        SCAN_TIMEOUT
-        if expires_at is None
-        else min(SCAN_TIMEOUT, (expires_at - datetime.now(timezone.utc)).total_seconds())
-    )
-    if timeout <= 0:
-        raise HTTPException(504, "SkillSpector scan deadline expired")
-    process: Final = await asyncio.create_subprocess_exec(
-        sys.executable,
-        str(Path(__file__).resolve()),
-        "--scan",
-        target,
-        str(output),
-        str(use_llm).lower(),
-        stdout=asyncio.subprocess.DEVNULL,
-        stderr=asyncio.subprocess.DEVNULL,
-        start_new_session=True,
-        env={
-            **os.environ,
-            "HOME": str(output.parent),
-            "TMPDIR": str(output.parent),
-            "GIT_CONFIG_GLOBAL": os.devnull,
-            "GIT_CONFIG_NOSYSTEM": "1",
-            "GIT_TERMINAL_PROMPT": "0",
-            "GIT_ALLOW_PROTOCOL": "https",
-        },
-    )
-    try:
-        code: Final = await asyncio.wait_for(process.wait(), timeout)
-        if code != 0 or not output.is_file() or output.stat().st_size > MAX_REPORT:
-            raise HTTPException(502, "SkillSpector could not produce a report for this source")
-        return TypeAdapter(dict[str, object]).validate_json(output.read_bytes())
-    except TimeoutError:
-        raise HTTPException(504, "SkillSpector scan timed out") from None
-    finally:
-        with suppress(ProcessLookupError):
-            os.killpg(process.pid, signal.SIGKILL)
-        await process.wait()
-
-
-@app.post("/scan", dependencies=[Depends(authenticate)])
-async def scan(
-    source: Annotated[str, Form(max_length=16384)],
-    use_llm: Annotated[bool, Form()] = False,
-    expires_at: Annotated[AwareDatetime | None, Form()] = None,
-    file: Annotated[UploadFile | None, File()] = None,
-) -> dict[str, object]:
-    if SLOTS.locked():
-        raise HTTPException(429, "Scanner is busy; try again shortly")
-    try:
-        parsed: Final = SOURCE.validate_json(source)
-    except ValidationError:
-        raise HTTPException(400, "Invalid source metadata") from None
-    async with SLOTS:
-        with tempfile.TemporaryDirectory(prefix="skill-scan-") as directory:
-            root: Final = Path(directory)
-            if file is not None:
-                content: Final = await file.read(MAX_UPLOAD + 1)
-                if len(content) > MAX_UPLOAD:
-                    raise HTTPException(413, "Upload exceeds 64 MiB")
-                archive: Final = root / "skill.zip"
-                archive.write_bytes(content)
-                return await run_worker(str(archive), root / "report.json", use_llm, expires_at)
-            return await run_worker(source_target(parsed), root / "report.json", use_llm, expires_at)
-
-
-if __name__ == "__main__":
-    if len(sys.argv) > 1 and sys.argv[1] == "--scan":
-        asyncio.run(worker(sys.argv[2], Path(sys.argv[3]), sys.argv[4] == "true"))
-    else:
-        import uvicorn
-
-        if not os.environ.get("SKILL_SCANNER_API_KEY"):
-            raise SystemExit("Set SKILL_SCANNER_API_KEY before starting the scanner")
-        uvicorn.run(app, host="127.0.0.1", port=int(os.environ.get("PORT", "8099")))

cosign.pub

@@ -1,4 +1,4 @@
 -----BEGIN PUBLIC KEY-----
-MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEG06dpKM+vEDnmp2zcjyXgVCXxzRg
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKi4ivqGpE231OGH50PKbqy1Y1Kkb
-JA5H2LnmwBKQafZGka6OOrWujCEWATQewiAgyD4WQdJ0pvU9RC4mvNAjvQ==
+POJC8+i2Wko82gBOUCe3M0Vw86H/4rhUhfoYEti4gdJ9wZbYmK0I2EE96g==
 -----END PUBLIC KEY-----

db_scripts/strip_vector_store_internal_metadata.py

@@ -1,201 +0,0 @@
-import argparse
-import os
-import sys
-from collections.abc import Iterator, Sequence
-from dataclasses import dataclass
-from itertools import accumulate, repeat, takewhile
-from typing import Final, Literal, TextIO
-
-import httpx
-from pydantic import BaseModel, ConfigDict, JsonValue, ValidationError
-
-from litellm.vector_stores.utils import is_litellm_written_metadata_key
-
-DEFAULT_API_BASE: Final = "https://api.openai.com/v1"
-LIST_PAGE_SIZE: Final = "100"
-
-
-class VectorStore(BaseModel):
-    model_config = ConfigDict(frozen=True, extra="ignore")
-
-    id: str
-    metadata: dict[str, JsonValue] | None = None
-
-
-class VectorStorePage(BaseModel):
-    model_config = ConfigDict(frozen=True, extra="ignore")
-
-    data: tuple[VectorStore, ...]
-    has_more: bool = False
-
-
-class OpenAIErrorBody(BaseModel):
-    model_config = ConfigDict(frozen=True, extra="ignore")
-
-    message: str = ""
-
-
-class OpenAIErrorResponse(BaseModel):
-    model_config = ConfigDict(frozen=True, extra="ignore")
-
-    error: OpenAIErrorBody
-
-
-@dataclass(frozen=True, slots=True)
-class Options:
-    api_key: str
-    api_base: str
-    vector_store_ids: tuple[str, ...]
-    extra_keys: frozenset[str]
-    apply: bool
-
-
-@dataclass(frozen=True, slots=True)
-class StoreResult:
-    store_id: str
-    matched_keys: tuple[str, ...]
-    status: Literal["dry_run", "updated", "failed"]
-    error: str = ""
-
-
-def is_proxy_written_key(key: str, extra_keys: frozenset[str]) -> bool:
-    return is_litellm_written_metadata_key(key) or key in extra_keys
-
-
-def matching_keys(store: VectorStore, extra_keys: frozenset[str]) -> tuple[str, ...]:
-    return tuple(sorted(key for key in (store.metadata or {}) if is_proxy_written_key(key, extra_keys)))
-
-
-def fetch_page(client: httpx.Client, after: str | None) -> VectorStorePage:
-    params: Final = {"limit": LIST_PAGE_SIZE} if after is None else {"limit": LIST_PAGE_SIZE, "after": after}
-    return VectorStorePage.model_validate_json(client.get("/vector_stores", params=params).raise_for_status().content)
-
-
-def fetch_next_page(client: httpx.Client, page: VectorStorePage | None) -> VectorStorePage | None:
-    if page is None or not page.has_more or not page.data:
-        return None
-    return fetch_page(client, after=page.data[-1].id)
-
-
-def list_all_stores(client: httpx.Client) -> Iterator[VectorStore]:
-    pages: Final = accumulate(
-        repeat(None), lambda page, _: fetch_next_page(client, page), initial=fetch_page(client, after=None)
-    )
-    for page in takewhile(lambda page: page is not None, pages):
-        yield from page.data
-
-
-def retrieve_stores(client: httpx.Client, store_ids: tuple[str, ...]) -> Iterator[VectorStore]:
-    for store_id in store_ids:
-        yield VectorStore.model_validate_json(client.get(f"/vector_stores/{store_id}").raise_for_status().content)
-
-
-def describe_http_error(response: httpx.Response) -> str:
-    try:
-        message: Final = OpenAIErrorResponse.model_validate_json(response.content).error.message
-    except ValidationError:
-        return f"HTTP {response.status_code}"
-    return f"HTTP {response.status_code}: {message}"
-
-
-def delete_keys(client: httpx.Client, store_id: str, keys: tuple[str, ...]) -> StoreResult:
-    try:
-        response: Final = client.post(f"/vector_stores/{store_id}", json={"metadata": dict.fromkeys(keys)})
-    except httpx.TransportError as error:
-        return StoreResult(store_id=store_id, matched_keys=keys, status="failed", error=type(error).__name__)
-    if response.is_success:
-        return StoreResult(store_id=store_id, matched_keys=keys, status="updated")
-    return StoreResult(store_id=store_id, matched_keys=keys, status="failed", error=describe_http_error(response))
-
-
-def clean_store(client: httpx.Client, store: VectorStore, options: Options) -> StoreResult | None:
-    keys: Final = matching_keys(store, options.extra_keys)
-    if not keys:
-        return None
-    if not options.apply:
-        return StoreResult(store_id=store.id, matched_keys=keys, status="dry_run")
-    return delete_keys(client, store.id, keys)
-
-
-def format_result(result: StoreResult) -> str:
-    key_list: Final = ", ".join(result.matched_keys)
-    count: Final = len(result.matched_keys)
-    match result.status:
-        case "dry_run":
-            return f"{result.store_id}: would delete {count} keys: {key_list}"
-        case "updated":
-            return f"{result.store_id}: deleted {count} keys: {key_list}"
-        case "failed":
-            return f"{result.store_id}: failed to delete {count} keys ({result.error}): {key_list}"
-
-
-def clean_and_report(
-    client: httpx.Client, stores: Iterator[VectorStore], options: Options, out: TextIO
-) -> Iterator[StoreResult | None]:
-    for store in stores:
-        result: Final = clean_store(client, store, options)
-        if result is not None:
-            print(format_result(result), file=out)
-        yield result
-
-
-def run(options: Options, client: httpx.Client, out: TextIO) -> int:
-    stores: Final = (
-        retrieve_stores(client, options.vector_store_ids) if options.vector_store_ids else list_all_stores(client)
-    )
-    scanned: Final = tuple(clean_and_report(client, stores, options, out))
-    results: Final = tuple(result for result in scanned if result is not None)
-    updated: Final = sum(1 for result in results if result.status == "updated")
-    failed: Final = sum(1 for result in results if result.status == "failed")
-    mode: Final = "applied" if options.apply else "dry run, pass --apply to delete"
-    print(
-        f"scanned={len(scanned)} matched={len(results)} updated={updated} failed={failed} ({mode})",
-        file=out,
-    )
-    return 1 if failed else 0
-
-
-def parse_options(argv: Sequence[str] | None) -> Options:
-    parser: Final = argparse.ArgumentParser(
-        description="Remove LiteLLM proxy-written keys from OpenAI vector store metadata. Dry run by default."
-    )
-    parser.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY"), help="defaults to $OPENAI_API_KEY")
-    parser.add_argument("--api-base", default=DEFAULT_API_BASE)
-    parser.add_argument("--vector-store-id", action="append", default=[], help="repeatable, defaults to every store")
-    parser.add_argument("--extra-key", action="append", default=[], help="repeatable, another key name to delete")
-    parser.add_argument("--apply", action="store_true", help="delete the matching keys instead of listing them")
-    args: Final = parser.parse_args(argv)
-    if not args.api_key:
-        parser.error("--api-key or OPENAI_API_KEY is required")
-    return Options(
-        api_key=args.api_key,
-        api_base=args.api_base.rstrip("/"),
-        vector_store_ids=tuple(args.vector_store_id),
-        extra_keys=frozenset(args.extra_key),
-        apply=args.apply,
-    )
-
-
-def main(argv: Sequence[str] | None = None, transport: httpx.BaseTransport | None = None) -> int:
-    options: Final = parse_options(argv)
-    with httpx.Client(
-        base_url=options.api_base,
-        headers={"Authorization": f"Bearer {options.api_key}"},
-        timeout=60.0,
-        transport=transport,
-    ) as client:
-        try:
-            return run(options, client, sys.stdout)
-        except httpx.HTTPStatusError as error:
-            sys.stderr.write(f"could not read vector stores: {describe_http_error(error.response)}\n")
-            return 2
-        except httpx.TransportError as error:
-            sys.stderr.write(f"could not reach {options.api_base}: {type(error).__name__}\n")
-            return 2
-        except ValidationError:
-            sys.stderr.write(f"unexpected vector store response shape from {options.api_base}\n")
-            return 2
-
-
-if __name__ == "__main__":
-    sys.exit(main())

docker-compose.hardened.yml

@@ -0,0 +1,44 @@
+services:
+  # Hardened stack: for testing the proxy under non-root, read-only, proxy-enforced constraints.
+  # Keep this file focused on hardening/QA scenarios; leave the main docker-compose.yml for default dev usage.
+  litellm:
+    build:
+      context: .
+      dockerfile: docker/Dockerfile.non_root
+      target: runtime
+    depends_on:
+      - squid
+    user: "101:101"
+    group_add:
+      - "2345"
+    read_only: true
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    tmpfs:
+      - /app/cache:rw,noexec,nosuid,nodev,size=128m,uid=101,gid=101,mode=1777
+      - /app/migrations:rw,noexec,nosuid,nodev,size=64m,uid=101,gid=101,mode=1777
+    volumes:
+      - ./proxy_server_config.yaml:/app/config.yaml:ro
+    environment:
+      LITELLM_NON_ROOT: "true"
+      PRISMA_BINARY_CACHE_DIR: "/app/cache/prisma-python/binaries"
+      XDG_CACHE_HOME: "/app/cache"
+      LITELLM_MIGRATION_DIR: "/app/migrations"
+      HTTP_PROXY: "http://squid:3128"
+      HTTPS_PROXY: "http://squid:3128"
+      NO_PROXY: "localhost,127.0.0.1,db"
+    command:
+      - "--port"
+      - "4000"
+      - "--config"
+      - "/app/config.yaml"
+  squid:
+    image: sameersbn/squid:3.5.27-2
+    restart: unless-stopped
+    ports:
+      - "3128:3128"
+    tmpfs:
+      - /var/spool/squid:rw,noexec,nosuid,nodev,size=64m
+      - /var/log/squid:rw,noexec,nosuid,nodev,size=16m

docker-compose.yml

@@ -1,71 +1,52 @@
-# LiteLLM plus a Postgres database that stores models, virtual keys and spend
-# logs. Used by https://docs.litellm.ai/docs/proxy/docker_quick_start
-#
-#   curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker-compose.yml
-#   printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
-#   docker compose up -d
-#
-# `docker compose up` runs the published image. From a checkout of the repo,
-# `docker compose up --build` builds the Dockerfile next to this file instead.
-# `docker compose --profile monitoring up -d` also starts Prometheus on 9090.
-#
-# Compose reads .env from this directory. Keep it: regenerating LITELLM_SALT_KEY
-# makes credentials already stored in the database unreadable. A database
-# created without LITELLM_SALT_KEY encrypted them with the master key, so when
-# upgrading one, write only LITELLM_MASTER_KEY (the existing one) to .env. For anything
-# beyond local evaluation, pin the image to a specific release tag.
 services:
   litellm:
-    image: docker.litellm.ai/berriai/litellm:main-stable
     build:
       context: .
-      target: runtime
+      args:
-    pull_policy: missing
+        target: runtime
-    ports:
+    image: docker.litellm.ai/berriai/litellm:main-stable
-      - "4000:4000"
+    #########################################
-    # The same image runs every component. The first word of `command` picks
+    ## Uncomment these lines to start proxy with a config.yaml file ##
-    # it: proxy (default), gateway, backend, ui, migrations, metrics,
-    # collector. See docker/README.md
-    # command: ["--config", "/app/config.yaml"]
     # volumes:
-    #   - ./config.yaml:/app/config.yaml:ro
+    #  - ./config.yaml:/app/config.yaml
+    # command:
+    #  - "--config=/app/config.yaml"
+    ##############################################
+    ports:
+      - "4000:4000" # Map the container port to the host, change the host port if necessary
     environment:
-      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:?set it in .env, see the header of this file}
+      DATABASE_URL: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
-      LITELLM_MIGRATION_DIR: /tmp/litellm-migrations
+      # Optional: route read-only queries (find_*, count, group_by, query_raw/_first)
-      DATABASE_URL: postgresql://llmproxy:dbpassword9090@db:5432/litellm
+      # to a separate reader endpoint, e.g. an Aurora reader. Leave unset for
-      STORE_MODEL_IN_DB: "True"
+      # single-DB deployments. With IAM_TOKEN_DB_AUTH enabled, the reader URL
+      # is auto-refreshed alongside the writer.
+      # DATABASE_URL_READ_REPLICA: "postgresql://llmproxy:dbpassword9090@db-reader:5432/litellm"
+      STORE_MODEL_IN_DB: "True" # allows adding models to proxy via UI
     env_file:
-      - path: .env
+      - .env # Load local .env file
-        required: false
-    read_only: true
-    cap_drop:
-      - ALL
-    security_opt:
-      - no-new-privileges:true
-    tmpfs:
-      - /tmp:rw,noexec,nosuid,nodev,size=64m,mode=1777
-      - /app/.cache:rw,noexec,nosuid,nodev,size=128m,mode=1777
     depends_on:
-      db:
+      - db  # Indicates that this service depends on the 'db' service, ensuring 'db' starts first
-        condition: service_healthy
+    healthcheck:  # Defines the health check configuration for the container
-    healthcheck:
       test:
         - CMD-SHELL
-        - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"
+        - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"  # Command to execute for health check
-      interval: 30s
+      interval: 30s  # Perform health check every 30 seconds
-      timeout: 10s
+      timeout: 10s   # Health check command times out after 10 seconds
-      retries: 3
+      retries: 3     # Retry up to 3 times if health check fails
-      start_period: 40s
+      start_period: 40s  # Wait 40 seconds after container start before beginning health checks
 
   db:
     image: postgres:16
     restart: always
+    container_name: litellm_db
     environment:
       POSTGRES_DB: litellm
       POSTGRES_USER: llmproxy
       POSTGRES_PASSWORD: dbpassword9090
+    ports:
+      - "5432:5432"
     volumes:
-      - postgres_data:/var/lib/postgresql/data
+      - postgres_data:/var/lib/postgresql/data # Persists Postgres data across container restarts
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
       interval: 1s
@@ -74,8 +55,6 @@ services:
 
   prometheus:
     image: prom/prometheus
-    profiles:
-      - monitoring
     volumes:
       - prometheus_data:/prometheus
       - ./prometheus.yml:/etc/prometheus/prometheus.yml
@@ -89,5 +68,6 @@ services:
 
 volumes:
   prometheus_data:
+    driver: local
   postgres_data:
-    name: litellm_postgres_data
+    name: litellm_postgres_data # Named volume for Postgres data persistence

docker-entrypoint.sh

@@ -1,140 +0,0 @@
-#!/bin/sh
-# LiteLLM image entrypoint: docker-entrypoint.sh [COMPONENT] [ARGS...]
-#
-# COMPONENT selects the process this container runs; it can also be given as
-# LITELLM_COMPONENT when the command carries no component. Anything after it is
-# passed to that process. A first argument starting with "-" (or no argument at
-# all) keeps the historical behaviour of running the monolithic proxy.
-#
-#   proxy       litellm ARGS                     everything in one process (default)
-#   gateway     python -m gateway.launch ARGS    inference routes, 0.0.0.0:4000
-#   backend     uvicorn backend.main:app ARGS    management routes, 0.0.0.0:4001
-#   ui          nginx serving the admin UI       port 3000
-#   migrations  python migrations/run.py         prisma migrate deploy, then exit
-#   metrics     python -m litellm.proxy.prometheus_metrics_server ARGS
-#   collector   python -m litellm.proxy.collector ARGS
-#   --admin-agent  the bundled LiteAdmin Slack agent (litellm-admin-agent --web ARGS)
-#
-# gateway and backend get their host and port defaults first, so ARGS such as
-# --port 8080 override them. With no DATABASE_URL, proxy assembles DATABASE_URL
-# and DATABASE_URL_READ_REPLICA from DATABASE_HOST and the other discrete
-# variables the way gateway and backend do. An unknown first word is executed
-# as-is (docker run <image> sh), through ddtrace-run like every component when
-# USE_DDTRACE=true. PgBouncer is not a component: LITELLM_PGBOUNCER_ENABLED=true
-# starts it inside proxy and gateway. Components that write Prometheus samples
-# start with an empty PROMETHEUS_MULTIPROC_DIR; metrics, collector and raw
-# readers keep the workers' files, other raw commands clear stale samples.
-set -eu
-
-usage() {
-    awk 'NR == 1 { next } /^#/ { sub(/^# ?/, ""); print; next } { exit }' "$0" >&2
-}
-
-component="${LITELLM_COMPONENT:-proxy}"
-case "${1:-}" in
-    proxy|gateway|backend|ui|migrations|metrics|collector)
-        component="$1"
-        shift
-        ;;
-    --admin-agent)
-        shift
-        export CONNECTION_AUTH_MODE=native
-        exec /opt/liteadmin/bin/litellm-admin-agent --web "$@"
-        ;;
-    ""|-*)
-        ;;
-    help)
-        usage
-        exit 0
-        ;;
-    *)
-        if [ -n "${PROMETHEUS_MULTIPROC_DIR:-}" ]; then
-            mkdir -p "$PROMETHEUS_MULTIPROC_DIR"
-            case "$*" in
-                *litellm.proxy.prometheus_metrics_server*|*litellm.proxy.collector*) ;;
-                *) rm -f "$PROMETHEUS_MULTIPROC_DIR"/*.db "$PROMETHEUS_MULTIPROC_DIR"/litellm_admitted_series_* ;;
-            esac
-        fi
-        component=raw
-        ;;
-esac
-
-case "$component" in
-    raw)
-        ;;
-    proxy)
-        if { [ -z "${DATABASE_URL:-}" ] && [ -n "${DATABASE_HOST:-}" ]; } ||
-            { [ -z "${DATABASE_URL_READ_REPLICA:-}" ] && [ -n "${DATABASE_HOST_READ_REPLICA:-}" ]; }; then
-            set -- python -c 'import os, sys
-from litellm.proxy.db.db_url_settings import (
-    DatabaseURLSettings,
-    add_missing_query_params,
-    connection_params_from_url,
-    idle_lifetime_params,
-    translate_libpq_ssl_params,
-)
-if os.environ.get("USE_AWS_KMS") == "True":
-    from litellm.secret_managers.aws_secret_manager import decrypt_env_var
-    os.environ.update(decrypt_env_var())
-settings = DatabaseURLSettings.from_env()
-settings.apply_writer_url_to_env()
-reader = settings.build_reader_url()
-if reader:
-    os.environ["DATABASE_URL_READ_REPLICA"] = translate_libpq_ssl_params(
-        add_missing_query_params(reader, connection_params_from_url(os.environ.get("DATABASE_URL", "")))
-    )
-if settings.max_idle_connection_lifetime is not None:
-    lifetime = idle_lifetime_params(settings.max_idle_connection_lifetime)
-    for name in ("DATABASE_URL", "DATABASE_URL_READ_REPLICA"):
-        if os.environ.get(name):
-            os.environ[name] = add_missing_query_params(os.environ[name], lifetime)
-os.execvp("litellm", ["litellm", *sys.argv[1:]])' "$@"
-        else
-            set -- litellm "$@"
-        fi
-        ;;
-    gateway)
-        set -- python -m gateway.launch --workers "${NUM_WORKERS:-1}" --host 0.0.0.0 --port 4000 "$@"
-        ;;
-    backend)
-        set -- uvicorn backend.main:app --host 0.0.0.0 --port 4001 "$@"
-        ;;
-    ui)
-        sh "$(dirname "$0")/ui/docker-entrypoint.d/40-litellm-runtime-config.sh"
-        exec nginx -g 'daemon off;' "$@"
-        ;;
-    migrations)
-        set -- python /app/migrations/run.py "$@"
-        ;;
-    metrics)
-        set -- python -m litellm.proxy.prometheus_metrics_server "$@"
-        ;;
-    collector)
-        set -- python -m litellm.proxy.collector "$@"
-        ;;
-    *)
-        echo "docker-entrypoint.sh: unknown LITELLM_COMPONENT '$component'" >&2
-        usage
-        exit 64
-        ;;
-esac
-
-if [ -n "${PROMETHEUS_MULTIPROC_DIR:-}" ]; then
-    case "$component" in
-        raw) ;;
-        metrics|collector) mkdir -p "$PROMETHEUS_MULTIPROC_DIR" ;;
-        *)
-            mkdir -p "$PROMETHEUS_MULTIPROC_DIR"
-            rm -f "$PROMETHEUS_MULTIPROC_DIR"/*.db "$PROMETHEUS_MULTIPROC_DIR"/litellm_admitted_series_*
-            ;;
-    esac
-fi
-
-case "${USE_DDTRACE:-}" in
-    [Tt][Rr][Uu][Ee])
-        export DD_TRACE_OPENAI_ENABLED="False"
-        exec ddtrace-run "$@"
-        ;;
-esac
-
-exec "$@"

docker/Dockerfile.database

@@ -0,0 +1,167 @@
+# syntax=docker/dockerfile:1.7
+
+# Base image for building
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
+
+# Runtime image
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
+# Pinned by digest like the other base images; bump explicitly on Node upgrades.
+ARG UI_BUILD_IMAGE=node:24.19-alpine3.24@sha256:d32cdf619f63fe0471182d08996dd516c6275bb5fd31ae06e55a570bd9e1ad43
+# Checksum from https://www.pgbouncer.org/downloads/ (the Wolfi repo only carries 1.24.x)
+ARG PGBOUNCER_VERSION=1.26.0
+ARG PGBOUNCER_SHA256=afd25dd61ee6775d37b40629b87ce08736b3e6955f3057bb212e410fbf21c71d
+
+FROM $UV_IMAGE AS uvbin
+
+FROM $LITELLM_BUILD_IMAGE AS pgbouncer-builder
+ARG PGBOUNCER_VERSION
+ARG PGBOUNCER_SHA256
+USER root
+RUN apk add --no-cache build-base pkgconf libevent-dev openssl-dev curl
+WORKDIR /build
+RUN curl -fsSL -o pgbouncer.tar.gz "https://www.pgbouncer.org/downloads/files/${PGBOUNCER_VERSION}/pgbouncer-${PGBOUNCER_VERSION}.tar.gz" && \
+    echo "${PGBOUNCER_SHA256}  pgbouncer.tar.gz" | sha256sum -c - && \
+    tar xzf pgbouncer.tar.gz --strip-components=1 && \
+    ./configure --prefix=/usr/local --with-openssl=/usr && \
+    make -j"$(nproc)" pgbouncer && \
+    install -m 0755 pgbouncer /usr/local/bin/pgbouncer
+
+# Admin UI builder. Pinned to the build platform so the architecture-independent
+# Next.js static export compiles once natively even in a multi-arch build,
+# instead of once per target arch under QEMU.
+FROM --platform=$BUILDPLATFORM $UI_BUILD_IMAGE AS ui-builder
+
+ENV NEXT_TELEMETRY_DISABLED=1 \
+    npm_config_fund=false \
+    npm_config_audit=false
+
+WORKDIR /ui
+
+COPY ui/litellm-dashboard/package.json ui/litellm-dashboard/package-lock.json ./
+COPY ui/litellm-dashboard/vendor/ ./vendor/
+RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline
+
+COPY ui/litellm-dashboard/ ./
+RUN npm run build
+
+FROM $LITELLM_BUILD_IMAGE AS builder
+
+WORKDIR /app
+USER root
+
+COPY --from=uvbin /uv /usr/local/bin/uv
+COPY --from=uvbin /uvx /usr/local/bin/uvx
+
+RUN apk add --no-cache \
+    bash \
+    gcc \
+    python-3.13 \
+    python-3.13-dev \
+    openssl \
+    openssl-dev \
+    nodejs \
+    npm \
+    libsndfile
+
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
+    UV_LINK_MODE=copy \
+    UV_PYTHON_DOWNLOADS=0 \
+    PATH="/app/.venv/bin:${PATH}"
+
+# Copy dependency metadata first for layer caching
+COPY pyproject.toml uv.lock ./
+COPY enterprise/pyproject.toml enterprise/
+COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
+
+# Install third-party dependencies (cached unless pyproject.toml/uv.lock change)
+RUN uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
+    --extra proxy \
+    --extra proxy-runtime \
+    --group admin-mcp \
+    --extra extra_proxy \
+    --extra semantic-router \
+    --extra saml \
+    --extra bedrock-realtime \
+    --python python3.13
+
+# Copy full source tree
+COPY . .
+
+# Replace the committed UI bundle with the one built from this exact source.
+# Clearing first drops the committed bundle's content-hashed chunks that COPY
+# would otherwise leave behind alongside the fresh ones.
+RUN rm -rf litellm/proxy/_experimental/out
+COPY --from=ui-builder /ui/out/. litellm/proxy/_experimental/out/
+
+# Build Admin UI before final sync (applies the enterprise color override when present)
+RUN sed -i 's/\r$//' docker/build_admin_ui.sh && chmod +x docker/build_admin_ui.sh && ./docker/build_admin_ui.sh
+
+# Install project and workspace packages (fast - deps already cached)
+RUN uv sync --frozen --no-default-groups --no-editable \
+    --extra proxy \
+    --extra proxy-runtime \
+    --group admin-mcp \
+    --extra extra_proxy \
+    --extra semantic-router \
+    --extra saml \
+    --extra bedrock-realtime \
+    --python python3.13
+
+RUN HOME=/opt/prisma XDG_CACHE_HOME=/opt/prisma/.cache PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
+    npm_config_cache=/root/.npm \
+    prisma generate --schema=./schema.prisma
+
+RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh && \
+    sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
+
+FROM $LITELLM_RUNTIME_IMAGE AS runtime
+ARG LITELLM_RELEASE_TAG=""
+ENV LITELLM_RELEASE_TAG=${LITELLM_RELEASE_TAG}
+
+USER root
+
+# node (without npm) is required by the prisma CLI at runtime
+RUN apk add --no-cache bash openssl tzdata nodejs python-3.13 libsndfile libevent
+COPY --from=pgbouncer-builder /usr/local/bin/pgbouncer /usr/local/bin/pgbouncer
+
+WORKDIR /app
+ENV PATH="/app/.venv/bin:${PATH}" \
+    PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
+    PRISMA_CLI_PATH=/opt/prisma/binaries/node_modules/.bin/prisma \
+    PRISMA_CLI_QUERY_ENGINE_TYPE=binary \
+    PRISMA_OFFLINE_MODE=true
+
+# Copy only what runtime needs. The application is installed inside the venv;
+# the rest of the builder's /app is source and build metadata that must not
+# ship (manifest-scanning tools attribute everything in it to this image).
+# entrypoint.sh invokes litellm/proxy/prisma_migration.py by source path.
+COPY --from=builder /app/.venv /app/.venv
+COPY --from=builder /app/docker /app/docker
+COPY --from=builder /app/schema.prisma /app/schema.prisma
+COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/prisma_migration.py
+# enterprise/ is imported by source path at runtime (proxy_cli puts the
+# working directory on sys.path; litellm/proxy/hooks resolves
+# enterprise.enterprise_hooks from it)
+COPY --from=builder /app/enterprise /app/enterprise
+COPY --from=builder /app/litellm-proxy-extras /app/litellm-proxy-extras
+# Prisma CLI + engines are baked under /opt/prisma, a fixed path every
+# runtime uid can read and that no cache volume mount shadows (unlike
+# /app/.cache or $HOME/.cache in readOnlyRootFilesystem + emptyDir setups).
+# The paths are pinned via PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH and
+# recorded into the generated client at build time, so `prisma migrate
+# deploy` on a fresh database needs no npm and no network access
+# (#33650, #24554).
+COPY --from=builder /opt/prisma /opt/prisma
+
+RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
+    find /app/.venv -type d -path "*/tornado/test" -delete && \
+    chmod -R a+rX /opt/prisma && \
+    test -x /opt/prisma/binaries/node_modules/.bin/prisma && \
+    test -f /opt/prisma/binaries/node_modules/prisma/build/index.js && \
+    python -c "from prisma.client import BINARY_PATHS; paths = list(BINARY_PATHS.query_engine.values()); assert paths and all(p.startswith('/opt/prisma/') for p in paths), paths"
+
+EXPOSE 4000/tcp
+
+ENTRYPOINT ["docker/prod_entrypoint.sh"]
+CMD ["--port", "4000"]

docker/Dockerfile.non_root

@@ -0,0 +1,207 @@
+# syntax=docker/dockerfile:1.7
+
+# Base images
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
+# Pinned by digest like the other base images; bump explicitly on Node upgrades.
+ARG UI_BUILD_IMAGE=node:24.19-alpine3.24@sha256:d32cdf619f63fe0471182d08996dd516c6275bb5fd31ae06e55a570bd9e1ad43
+# Checksum from https://www.pgbouncer.org/downloads/ (the Wolfi repo only carries 1.24.x)
+ARG PGBOUNCER_VERSION=1.26.0
+ARG PGBOUNCER_SHA256=afd25dd61ee6775d37b40629b87ce08736b3e6955f3057bb212e410fbf21c71d
+
+FROM $UV_IMAGE AS uvbin
+
+FROM $LITELLM_BUILD_IMAGE AS pgbouncer-builder
+ARG PGBOUNCER_VERSION
+ARG PGBOUNCER_SHA256
+USER root
+RUN apk add --no-cache build-base pkgconf libevent-dev openssl-dev curl
+WORKDIR /build
+RUN curl -fsSL -o pgbouncer.tar.gz "https://www.pgbouncer.org/downloads/files/${PGBOUNCER_VERSION}/pgbouncer-${PGBOUNCER_VERSION}.tar.gz" && \
+    echo "${PGBOUNCER_SHA256}  pgbouncer.tar.gz" | sha256sum -c - && \
+    tar xzf pgbouncer.tar.gz --strip-components=1 && \
+    ./configure --prefix=/usr/local --with-openssl=/usr && \
+    make -j"$(nproc)" pgbouncer && \
+    install -m 0755 pgbouncer /usr/local/bin/pgbouncer
+
+# Admin UI builder. Pinned to the build platform so the architecture-independent
+# Next.js static export compiles once natively even in a multi-arch build,
+# instead of once per target arch under QEMU.
+FROM --platform=$BUILDPLATFORM $UI_BUILD_IMAGE AS ui-builder
+
+ENV NEXT_TELEMETRY_DISABLED=1 \
+    npm_config_fund=false \
+    npm_config_audit=false
+
+WORKDIR /ui
+
+COPY ui/litellm-dashboard/package.json ui/litellm-dashboard/package-lock.json ./
+COPY ui/litellm-dashboard/vendor/ ./vendor/
+RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline
+
+COPY ui/litellm-dashboard/ ./
+RUN npm run build
+
+FROM $LITELLM_BUILD_IMAGE AS builder
+WORKDIR /app
+USER root
+
+COPY --from=uvbin /uv /usr/local/bin/uv
+COPY --from=uvbin /uvx /usr/local/bin/uvx
+
+RUN for i in 1 2 3; do \
+      apk add --no-cache \
+        python-3.13 \
+        python-3.13-dev \
+        gcc \
+        rust \
+        bash \
+        coreutils \
+        curl \
+        openssl \
+        libsndfile \
+        nodejs \
+        npm && break || sleep 5; \
+    done
+
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
+    UV_LINK_MODE=copy \
+    UV_PYTHON_DOWNLOADS=0 \
+    PATH="/app/.venv/bin:${PATH}" \
+    LITELLM_NON_ROOT=true \
+    XDG_CACHE_HOME=/app/.cache
+
+# Copy dependency metadata first for layer caching
+COPY pyproject.toml uv.lock ./
+COPY enterprise/pyproject.toml enterprise/
+COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
+
+# Install third-party dependencies (cached unless pyproject.toml/uv.lock change)
+RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
+    uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
+    --extra proxy \
+    --extra proxy-runtime \
+    --group admin-mcp \
+    --extra extra_proxy \
+    --extra semantic-router \
+    --extra saml \
+    --extra bedrock-realtime \
+    --python python3.13
+
+# Copy full source tree
+COPY . .
+
+# Replace the committed UI bundle with the one built from this exact source.
+# Clearing first drops the committed bundle's content-hashed chunks that COPY
+# would otherwise leave behind alongside the fresh ones.
+RUN rm -rf litellm/proxy/_experimental/out
+COPY --from=ui-builder /ui/out/. litellm/proxy/_experimental/out/
+
+# Set non-root flag for build time consistency
+ENV LITELLM_NON_ROOT=true
+
+RUN mkdir -p /var/lib/litellm/ui /var/lib/litellm/assets && \
+    cp -r /app/litellm/proxy/_experimental/out/. /var/lib/litellm/ui/ && \
+    cp /app/litellm/proxy/logo.png /var/lib/litellm/assets/logo.png && \
+    touch /var/lib/litellm/ui/.litellm_ui_ready
+
+RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
+    uv sync --frozen --no-default-groups --no-editable \
+      --extra proxy \
+      --extra proxy-runtime \
+      --group admin-mcp \
+      --extra extra_proxy \
+      --extra semantic-router \
+      --extra saml \
+      --extra bedrock-realtime \
+      --python python3.13
+
+RUN HOME=/opt/prisma XDG_CACHE_HOME=/opt/prisma/.cache PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
+    npm_config_cache=/root/.npm \
+    prisma generate --schema=./schema.prisma
+
+RUN sed -i 's/\r$//' docker/entrypoint.sh && chmod +x docker/entrypoint.sh && \
+    sed -i 's/\r$//' docker/prod_entrypoint.sh && chmod +x docker/prod_entrypoint.sh
+
+FROM $LITELLM_RUNTIME_IMAGE AS runtime
+ARG LITELLM_RELEASE_TAG=""
+ENV LITELLM_RELEASE_TAG=${LITELLM_RELEASE_TAG}
+WORKDIR /app
+USER root
+
+RUN for i in 1 2 3; do \
+      apk upgrade --no-cache && break || sleep 5; \
+    done && \
+    for i in 1 2 3; do \
+      apk add --no-cache python-3.13 bash openssl tzdata libsndfile nodejs libevent && break || sleep 5; \
+    done
+COPY --from=pgbouncer-builder /usr/local/bin/pgbouncer /usr/local/bin/pgbouncer
+
+# Copy only what runtime needs. The application is installed inside the venv;
+# the rest of the builder's /app is source and build metadata that must not
+# ship (manifest-scanning tools attribute everything in it to this image).
+# entrypoint.sh invokes litellm/proxy/prisma_migration.py by source path.
+COPY --from=builder /app/.venv /app/.venv
+COPY --from=builder /app/docker /app/docker
+COPY --from=builder /app/schema.prisma /app/schema.prisma
+COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/prisma_migration.py
+# enterprise/ is imported by source path at runtime (proxy_cli puts the
+# working directory on sys.path; litellm/proxy/hooks resolves
+# enterprise.enterprise_hooks from it)
+COPY --from=builder /app/enterprise /app/enterprise
+COPY --from=builder /app/litellm-proxy-extras /app/litellm-proxy-extras
+# Prisma CLI + engines are baked under /opt/prisma, a fixed path every runtime
+# uid can read and that no cache volume mount shadows (unlike /app/.cache or
+# $HOME/.cache under readOnlyRootFilesystem + emptyDir or arbitrary-uid setups).
+# PRISMA_CLI_QUERY_ENGINE_TYPE=binary makes the CLI use the baked binary query
+# engine directly, so `prisma migrate deploy` on a fresh database needs no npm
+# and no network access; without it the CLI looks for the library engine, which
+# prisma stopped baking, and falls back to a download that fails offline or as a
+# non-writable uid (#33650, #24554).
+COPY --from=builder /opt/prisma /opt/prisma
+COPY --from=builder /var/lib/litellm/ui /var/lib/litellm/ui
+COPY --from=builder /var/lib/litellm/assets /var/lib/litellm/assets
+
+# XDG_CACHE_HOME is intentionally left unset so it falls back to $HOME/.cache
+# (/app/.cache, writable by the runtime uid). The prisma bake at the read-only
+# /opt/prisma is anchored by PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH, so
+# nothing needs XDG to point there; pointing it at the read-only bake would
+# deny any XDG-aware library that writes a cache at runtime.
+ENV PATH="/app/.venv/bin:${PATH}" \
+    PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
+    PRISMA_CLI_PATH=/opt/prisma/binaries/node_modules/.bin/prisma \
+    PRISMA_CLI_QUERY_ENGINE_TYPE=binary \
+    HOME=/app \
+    LITELLM_NON_ROOT=true \
+    PRISMA_SKIP_POSTINSTALL_GENERATE=1 \
+    PRISMA_HIDE_UPDATE_MESSAGE=1 \
+    PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING=1 \
+    PRISMA_OFFLINE_MODE=true
+
+RUN mkdir -p /nonexistent /app/.cache /var/lib/litellm/assets /var/lib/litellm/ui && \
+    chown -R nobody:nogroup /app /var/lib/litellm/ui /var/lib/litellm/assets /nonexistent && \
+    PRISMA_PATH=$(python -c "import os, prisma; print(os.path.dirname(prisma.__file__))") && \
+    chown -R nobody:nogroup "$PRISMA_PATH" && \
+    LITELLM_PKG_MIGRATIONS_PATH="$(python -c 'import os, litellm_proxy_extras; print(os.path.dirname(litellm_proxy_extras.__file__))' 2>/dev/null || echo '')/migrations" && \
+    [ -n "$LITELLM_PKG_MIGRATIONS_PATH" ] && chown -R nobody:nogroup "$LITELLM_PKG_MIGRATIONS_PATH" || true && \
+    LITELLM_PROXY_EXTRAS_PATH=$(python -c "import os, litellm_proxy_extras; print(os.path.dirname(litellm_proxy_extras.__file__))" 2>/dev/null || echo "") && \
+    chgrp -R 0 "$PRISMA_PATH" /var/lib/litellm/ui /var/lib/litellm/assets && \
+    [ -n "$LITELLM_PROXY_EXTRAS_PATH" ] && chgrp -R 0 "$LITELLM_PROXY_EXTRAS_PATH" || true && \
+    chmod -R g=u "$PRISMA_PATH" /var/lib/litellm/ui /var/lib/litellm/assets && \
+    [ -n "$LITELLM_PROXY_EXTRAS_PATH" ] && chmod -R g=u "$LITELLM_PROXY_EXTRAS_PATH" || true && \
+    chmod -R g+w "$PRISMA_PATH" /var/lib/litellm/ui /var/lib/litellm/assets && \
+    [ -n "$LITELLM_PROXY_EXTRAS_PATH" ] && chmod -R g+w "$LITELLM_PROXY_EXTRAS_PATH" || true && \
+    chmod -R g+rX "$PRISMA_PATH" /var/lib/litellm/ui /var/lib/litellm/assets && \
+    chmod -R a+rX /opt/prisma && \
+    test -x /opt/prisma/binaries/node_modules/.bin/prisma && \
+    test -f /opt/prisma/binaries/node_modules/prisma/build/index.js && \
+    ls /opt/prisma/binaries/node_modules/@prisma/engines/query-engine-* >/dev/null 2>&1 && \
+    python -c "from prisma.client import BINARY_PATHS; paths = list(BINARY_PATHS.query_engine.values()); assert paths and all(p.startswith('/opt/prisma/') for p in paths), paths"
+
+USER 65534
+
+EXPOSE 4000/tcp
+
+ENTRYPOINT ["/app/docker/prod_entrypoint.sh"]
+CMD ["--port", "4000"]

docker/README.md

@@ -2,15 +2,15 @@
 
 This guide provides instructions for building and running the LiteLLM application using Docker and Docker Compose.
 
-> **Just want to run LiteLLM?** `docker-compose.yml` in the repository root runs
+> **Just want to run LiteLLM?** This guide builds from source. To run the published
-> the published image with a Postgres database, the stack the
+> image instead, use `docker-compose.quickstart.yml` in this directory — the
-> [Docker quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start) documents,
+> two-service stack (gateway + Postgres) that the
-> and it works on its own outside a checkout:
+> [Docker quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start) documents:
 >
 > ```bash
-> curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker-compose.yml
+> curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker/docker-compose.quickstart.yml
 > printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
-> docker compose up -d
+> docker compose -f docker-compose.quickstart.yml up -d
 > ```
 
 ## Prerequisites
@@ -20,52 +20,33 @@ This guide provides instructions for building and running the LiteLLM applicatio
 
 ## Building and Running the Application
 
-The same `docker-compose.yml` builds from source when you pass `--build`: it builds the `Dockerfile` in the repository root, the one image LiteLLM ships. `docker-entrypoint.sh` next to it is the only entrypoint script; every container starts through it
+To build and run the application, you will use the `docker-compose.yml` file located in the root of the project. This file is configured to use the `Dockerfile.non_root` for a secure, non-root container environment.
 
-## One image, many components
+### 1. Set the Master Key
 
-Every LiteLLM container runs the same image. The first word of the container command (or the `LITELLM_COMPONENT` environment variable when the command carries only flags) picks the process the container runs, and everything after it is handed to that process unchanged:
+The application requires a `LITELLM_MASTER_KEY` for signing and validating tokens. You must set this key as an environment variable before running the application.
 
-| Component    | Runs                                                | Port |
+Create a `.env` file in the root of the project and add the following line:
-|--------------|-----------------------------------------------------|------|
-| `proxy`      | `litellm ...` (everything in one process, default)  | 4000 |
-| `gateway`    | `python -m gateway.launch ...` (inference routes)   | 4000 |
-| `backend`    | `uvicorn backend.main:app ...` (management routes)  | 4001 |
-| `ui`         | nginx serving the static admin UI                   | 3000 |
-| `migrations` | `python migrations/run.py`, `prisma migrate deploy` then exit | |
-| `metrics`    | `python -m litellm.proxy.prometheus_metrics_server ...` | `--port` |
-| `collector`  | `python -m litellm.proxy.collector ...`             | |
 
-```bash
+```
-docker run -p 4000:4000 litellm --config /app/config.yaml         # proxy, exactly as before
+LITELLM_MASTER_KEY=your-secret-key
-docker run -p 4000:4000 litellm gateway --port 4000               # componentized data plane
-docker run -p 4001:4001 -e LITELLM_COMPONENT=backend litellm      # same, chosen through the env
-docker run -p 3000:3000 --read-only --tmpfs /tmp litellm ui       # admin UI behind nginx
-docker run -e DATABASE_URL=... litellm migrations                 # one-off schema migration job
-docker run -it litellm sh                                         # anything else runs verbatim
 ```
 
-PgBouncer is not a separate component: `LITELLM_PGBOUNCER_ENABLED=true` starts an in-container PgBouncer in front of `DATABASE_URL` inside `proxy` and `gateway`. `USE_DDTRACE=true` wraps whichever component runs with `ddtrace-run`, and `PROMETHEUS_MULTIPROC_DIR` is emptied of stale samples before any workers fork (the `metrics` and `collector` sidecars only read it, so their restart keeps the live samples)
+Replace `your-secret-key` with a strong, randomly generated secret.
-
-The image runs as uid `65532` (`nonroot` in the Wolfi base) and also works as an arbitrary uid in gid 0, the shape OpenShift `restricted-v2` assigns, because everything it writes at runtime lives under `/app/.cache`, `/var/lib/litellm` and `/tmp`. Mount those (or set `readOnlyRootFilesystem` with emptyDirs there) for a read-only root filesystem. Prisma's CLI and engines are baked under `/opt/prisma`, so migrations need neither network nor a writable home
-
-### 1. Set the Master and Salt Keys
-
-The proxy signs virtual keys with `LITELLM_MASTER_KEY` and encrypts stored provider credentials with `LITELLM_SALT_KEY`. Compose reads both from a `.env` file in the directory you run it from, so generate them once:
-
-```bash
-printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
-```
-
-Keep the file: regenerating `LITELLM_SALT_KEY` makes credentials already stored in the database unreadable. A database created without `LITELLM_SALT_KEY` encrypted them with the master key, so when upgrading one, write only the existing `LITELLM_MASTER_KEY` to `.env` and leave `LITELLM_SALT_KEY` out. Provider keys such as `OPENAI_API_KEY` go in the same file, and the whole file is passed to the container
 
 ### 2. Build and Run the Containers
 
+Once you have set the `LITELLM_MASTER_KEY`, you can build and run the containers using the following command:
+
 ```bash
 docker compose up -d --build
 ```
 
-This command builds the image from the root `Dockerfile` and starts the `litellm` and `db` services in detached mode. Without `--build`, `docker compose up` pulls the published `main-stable` image instead. Add `--profile monitoring` to also start Prometheus on port 9090, scraping the proxy with the root `prometheus.yml`
+This command will:
+
+-   Build the Docker image using `Dockerfile.non_root`.
+-   Start the `litellm`, `litellm_db`, and `prometheus` services in detached mode (`-d`).
+-   The `--build` flag ensures that the image is rebuilt if there are any changes to the Dockerfile or the application code.
 
 ### 3. Verifying the Application is Running
 
@@ -99,13 +80,13 @@ LITELLM_LICENSE="your-enterprise-license"
 PROXY_BASE_URL=https://gateway.example.com
 ```
 
-For the compose deployment described above, put them in its `.env` file and rebuild:
+For the unified source deployment described above, put them in its `.env` file and rebuild:
 
 ```bash
 docker compose up -d --build
 ```
 
-In componentized deployments, set the flag and license on the `backend` component and route `/admin/mcp` to it. The `gateway` component excludes this endpoint. The image bundles the connector for every component
+In componentized deployments, set the flag and license on the backend container and route `/admin/mcp` to the backend service. The gateway component excludes this endpoint. The unified, database, non-root, and backend image builds bundle the connector
 
 Hosting is disabled by default. Opting in requires a valid base Enterprise license; an unlicensed opt-in or invalid flag value prevents startup. Enabling it reserves `/admin`, so rename any MCP server alias called `admin` first
 
@@ -115,18 +96,34 @@ Embedded responses default to `full`; selecting `LITELLM_ADMIN_RESPONSE_VIEW=com
 
 See the [LiteAdmin MCP guide](https://docs.litellm.ai/docs/proxy/liteadmin_mcp#run-liteadmin-mcp-inside-litellm) for client configuration, tool restrictions, and verification
 
-## Hardening
+## Hardened / Offline Testing
 
-The compose file runs the proxy the way a locked-down cluster would: as the image's non-root user with a read-only root filesystem, every capability dropped, `no-new-privileges` set, and tmpfs mounts only at `/tmp` and `/app/.cache`. The image is built for that, so a change that makes the proxy write anywhere else fails here before it fails in Kubernetes. To try an arbitrary uid in gid 0, the shape OpenShift `restricted-v2` assigns, add `user: "101:0"` to the `litellm` service
+To ensure changes are safe for non-root, read-only root filesystems and restricted egress, always validate with the hardened compose file:
-
-Prisma's CLI and engines are baked under `/opt/prisma`, so migrations run without network access. Verify with:
 
 ```bash
-docker run --rm --network none --entrypoint prisma docker.litellm.ai/berriai/litellm:main-stable --version
+docker compose -f docker-compose.yml -f docker-compose.hardened.yml build --no-cache
+docker compose -f docker-compose.yml -f docker-compose.hardened.yml up -d
 ```
 
+This setup:
+- Builds from `docker/Dockerfile.non_root` with Prisma engines and Node toolchain baked into the image.
+- Runs the proxy as a non-root user with a read-only rootfs and only writable tmpfs mounts:
+  - `/app/cache` (Prisma/NPM cache; backing `PRISMA_BINARY_CACHE_DIR`, `NPM_CONFIG_CACHE`, `XDG_CACHE_HOME`)
+  - `/app/migrations` (Prisma migration workspace; backing `LITELLM_MIGRATION_DIR`)
+- Pre-builds and serves the admin UI from read-only paths:
+  - `/var/lib/litellm/ui` (pre-restructured Next.js UI with `.litellm_ui_ready` marker)
+  - `/var/lib/litellm/assets` (UI logos and assets)
+- Routes all outbound traffic through a local Squid proxy that denies egress, so Prisma migrations must use the cached CLI and engines.
+
+You should also verify offline Prisma behaviour with:
+
+```bash
+docker run --rm --network none --entrypoint prisma ghcr.io/berriai/litellm:main-stable --version
+```
+
+This command should succeed (showing engine versions) even with `--network none`, confirming that Prisma binaries are available without network access.
+
 ## Troubleshooting
 
--   **`required variable LITELLM_MASTER_KEY is missing a value`**: Compose did not find a `.env` file with `LITELLM_MASTER_KEY` in the directory you ran it from. Generate one as shown above.
+-   **`build_admin_ui.sh: not found`**: This error can occur if the Docker build context is not set correctly. Ensure that you are running the `docker-compose` command from the root of the project.
--   **`password authentication failed for user "llmproxy"`**: the `litellm_postgres_data` volume was initialised by an older compose file with different credentials. `docker compose down -v` drops it and the next `up` recreates the database.
+-   **`Master key is not initialized`**: This error means the `LITELLM_MASTER_KEY` environment variable is not set. Make sure you have created a `.env` file in the project root with the `LITELLM_MASTER_KEY` defined.
--   **`build_admin_ui.sh: not found`**: the build context is wrong. Run `docker compose` from the root of the repository.

docker/build_from_pip/Dockerfile.build_from_pip

@@ -0,0 +1,64 @@
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea099626a4adb02acbac4c69920fe9d278f82
+FROM $UV_IMAGE AS uvbin
+
+FROM python:3.13-slim@sha256:739e7213785e88c0f702dcdc12c0973afcbd606dbf021a589cab77d6b00b579d
+
+ARG LITELLM_VERSION=1.83.0
+
+WORKDIR /app
+
+COPY --from=uvbin /uv /usr/local/bin/uv
+COPY --from=uvbin /uvx /usr/local/bin/uvx
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends gcc libffi-dev nodejs npm && \
+    rm -rf /var/lib/apt/lists/*
+
+ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
+    UV_LINK_MODE=copy \
+    PATH="/app/.venv/bin:${PATH}"
+
+COPY schema.prisma .
+
+# This image is specifically for validating/installing the published PyPI
+# artifact, not the checked-out source tree.
+# Keep the moved proxy-runtime packages explicit until the published PyPI
+# artifact includes that extra; newer releases will simply dedupe these.
+RUN uv venv --python python && \
+    uv pip install --python /app/.venv/bin/python \
+      "litellm[proxy,proxy-runtime]==${LITELLM_VERSION}" \
+      "google-cloud-aiplatform==1.133.0" \
+      "google-genai==1.37.0" \
+      "anthropic[vertex]==0.84.0" \
+      "grpcio==1.78.0" \
+      "prometheus-client==0.20.0" \
+      "langfuse==2.59.7" \
+      "opentelemetry-api==1.28.0" \
+      "opentelemetry-sdk==1.28.0" \
+      "opentelemetry-exporter-otlp==1.28.0" \
+      "ddtrace==4.11.0" \
+      "sentry-sdk==2.21.0" \
+      "mangum==0.17.0" \
+      "azure-ai-contentsafety==1.0.0" \
+      "azure-storage-file-datalake==12.20.0" \
+      "pypdf==6.7.5" \
+      "llm-sandbox==0.3.31" \
+      "detect-secrets==1.5.0" \
+      "prisma==0.11.0" \
+      "openai==2.24.0"
+
+RUN HOME=/opt/prisma XDG_CACHE_HOME=/opt/prisma/.cache PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
+    npm_config_cache=/root/.npm \
+    prisma generate --schema=./schema.prisma && \
+    chmod -R a+rX /opt/prisma && \
+    python -c "import sys; from prisma.client import BINARY_PATHS; bad = sorted(p for group in BINARY_PATHS.model_dump().values() for p in group.values() if not p.startswith('/opt/prisma/')); sys.exit('prisma engines baked outside /opt/prisma: %r' % bad) if bad else None"
+
+ENV PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries
+
+COPY docker/prod_entrypoint.sh /app/docker/prod_entrypoint.sh
+RUN sed -i 's/\r$//' /app/docker/prod_entrypoint.sh && chmod +x /app/docker/prod_entrypoint.sh
+
+EXPOSE 4000/tcp
+
+ENTRYPOINT ["/app/docker/prod_entrypoint.sh"]
+CMD ["--port", "4000"]

docker/build_from_pip/Readme.md

@@ -0,0 +1,9 @@
+# Docker to build LiteLLM Proxy from litellm pip package
+
+### When to use this ?
+
+If you need to build LiteLLM Proxy from litellm pip package, you can use this Dockerfile as a reference.
+
+### Why build from pip package ?
+
+- If your company has a strict requirement around security / building images you can follow steps outlined here 

docker/build_from_pip/litellm_config.yaml

@@ -0,0 +1,9 @@
+model_list:
+  - model_name: "gpt-4"
+    litellm_params:
+      model: openai/fake
+      api_key: fake-key
+      api_base: os.environ/FAKE_OPENAI_API_BASE
+
+general_settings:    
+    alerting: ["slack"]

docker/component_entrypoint.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+
+# stale samples from a previous container incarnation would be summed into the aggregate
+if [ -n "$PROMETHEUS_MULTIPROC_DIR" ]; then
+    mkdir -p "$PROMETHEUS_MULTIPROC_DIR"
+    rm -f "$PROMETHEUS_MULTIPROC_DIR"/*.db "$PROMETHEUS_MULTIPROC_DIR"/litellm_admitted_series_*
+fi
+
+case "$USE_DDTRACE" in
+    [Tt][Rr][Uu][Ee])
+        export DD_TRACE_OPENAI_ENABLED="False"
+        exec ddtrace-run "$@"
+        ;;
+esac
+
+exec "$@"

docker/entrypoint.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+set -euo pipefail
+
+REPO_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+VENV_PYTHON="$REPO_ROOT/.venv/bin/python"
+MIGRATION_SCRIPT="$REPO_ROOT/litellm/proxy/prisma_migration.py"
+
+if [ -x "$VENV_PYTHON" ]; then
+    "$VENV_PYTHON" "$MIGRATION_SCRIPT"
+elif command -v uv >/dev/null 2>&1; then
+    (cd "$REPO_ROOT" && uv run --no-sync python "$MIGRATION_SCRIPT")
+else
+    python3 "$MIGRATION_SCRIPT"
+fi
+
+echo "Migration script ran successfully!"

docker/install_auto_router.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+set -euo pipefail
+
+# semantic-router dependencies are installed via `uv sync`.

docker/prod_entrypoint.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+
+if [ "$1" = "--admin-agent" ]; then
+    shift
+    export CONNECTION_AUTH_MODE=native
+    exec /opt/liteadmin/bin/litellm-admin-agent --web "$@"
+fi
+
+case "$USE_DDTRACE" in
+    [Tt][Rr][Uu][Ee])
+        export DD_TRACE_OPENAI_ENABLED="False"
+        exec ddtrace-run litellm "$@"
+        ;;
+esac
+
+exec litellm "$@"

docker/tests/nonroot.yaml

@@ -0,0 +1,18 @@
+schemaVersion: 2.0.0
+
+metadataTest:
+  entrypoint: ["docker/prod_entrypoint.sh"]
+  user: "65534"
+  workdir: "/app"
+
+fileExistenceTests:
+  - name: "Prisma Folder"
+    path: "/usr/local/lib/python3.13/site-packages/prisma/"
+    shouldExist: true
+    uid: 65534
+    gid: 65534
+  - name: "Prisma Schema"
+    path: "/usr/local/lib/python3.13/site-packages/prisma/schema.prisma"
+    shouldExist: true
+    uid: 65534
+    gid: 65534

docs/adrs/AGENTS.md

@@ -1,5 +0,0 @@
-# Architecture decision records for the Python SDK and proxy
-
-This folder records decisions about the Python SDK and proxy the way [`litellm-rust/docs/adrs/`](../../litellm-rust/docs/adrs/AGENTS.md) records them for the Rust core, and that file's process applies here unchanged: one short `adr_NNN_short_title.md` per decision that is hard to reverse or that a new contributor would reasonably question, with Context, Decision, Alternatives Considered, and Consequences, superseded by a new ADR instead of edited once accepted. Each folder numbers its own ADRs, so a reference to an ADR in the other folder names its path
-
-Read the ADRs here before you build a new way to move data between the endpoints, the provider transformations, and cost tracking or logging. A PR that makes a decision of that kind adds its ADR in the same PR

docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md

@@ -1,39 +0,0 @@
-# ADR 001: Provider usage fields ride as extras on the normalized usage object
-
-Status: Accepted
-
-## Context
-
-Providers report usage fields that no other provider has, such as xAI's `usage.server_side_tool_usage_details.web_search_calls` or Anthropic's `usage.inference_geo`, and cost tracking has to read them or that spend never reaches `LiteLLM_SpendLogs`. One provider response can be served on `/v1/chat/completions`, `/v1/responses`, and `/v1/messages`, streaming or not, so a field that only one of those paths carries is billed wrong on the others
-
-[#30817](https://github.com/BerriAI/litellm/pull/30817) fixed unbilled xAI web searches. One draft did the copy in the xAI Responses transform to keep provider code out of shared files, and that swapped `/v1/responses` usage to the chat shape, so callers of that endpoint lost the Responses schema. The merged version keeps the schema, copies unknown usage fields in the shared Responses-to-chat bridge for every provider, and gates on the reported field without naming the provider
-
-In October 2026, `Usage` and `ResponseAPIUsage` kept unknown usage keys as pydantic extras, and that bridge copied them. Paths that build usage from named fields kept only the fields they name: the Anthropic, Gemini, and Bedrock Converse transforms, the stream builder, and the bridges from chat usage to `/v1/responses` and `/v1/messages`
-
-## Decision
-
-Provider-only usage fields stay on the usage object as pydantic extras, and cost tracking reads them there, in that provider's cost calculator or in a shared gate that checks the field instead of the provider. An extra reaches callers as an added usage key. No transform renames, removes, or reshapes a usage field the endpoint's schema defines to feed cost tracking, though it may fill one LiteLLM's usage object already defines, such as `server_tool_use`
-
-A shared path that rebuilds usage, such as the stream builder or a bridge between endpoint shapes, copies the extras instead of naming provider fields. A provider transform that builds usage from named fields names the new field. Neither is bypassed with a side channel. Older code that differs, such as the Azure branches in the shared tracker and the Gemini pricing tier read from `_hidden_params`, predates this ADR
-
-## Alternatives Considered
-
-**Reshape usage in the provider transform**, as a draft of #30817 did. It changes the schema that callers of that endpoint rely on, and every provider and endpoint needs its own copy
-
-**A typed field on `ResponseAPIUsage` and `Usage` for each provider's quirk**. The types grow without bound and every bridge needs an update for each one. A field several providers report, like `server_tool_use`, does get one
-
-**A side channel such as `_hidden_params`**. Cost calculators receive `Usage`, so a side channel makes a second source of truth for the same number
-
-**Provider branches or provider field names in shared code**. They put one provider's logic in a file every provider shares, which is why #30817 replaced its xAI-only gate with one that checks the reported field
-
-## Consequences
-
-Benefits:
-
-- Billing a new provider usage field touches that provider's calculator (plus a `cost_per_token` arm when the calculator is new), its transform when that names fields, and for a built-in tool at most one gate and one `get_cost_for_web_search_request` arm
-
-Costs:
-
-- Extras are untyped, so a reader validates them before use, as `_usage_reports_server_side_web_search_calls` does
-- The stream builder and the bridges from chat usage did not copy extras when this was written, so until they do, a field billed on those paths is named in them, and tests and proof runs cover each endpoint and `"stream": true`
-- This covers usage fields LiteLLM prices, not request-side params or a provider's own charge for the request, which OpenRouter and opencode Go report and `response_cost_calculator` reads from `_hidden_params` first

enterprise/enterprise_hooks/aporia_ai.py

@@ -15,10 +15,9 @@ sys.path.insert(
 )  # Adds the parent directory to the system path
 import json
 import sys
-from typing import Final, List, Literal, Optional
+from typing import Any, List, Literal, Optional
 
 from fastapi import HTTPException
-from pydantic import TypeAdapter
 
 import litellm
 from litellm._logging import verbose_proxy_logger
@@ -30,13 +29,11 @@ from litellm.llms.custom_httpx.http_handler import (
     get_async_httpx_client,
     httpxSpecialProvider,
 )
-from litellm.llms.openai.common_utils import decoded_json_object
 from litellm.proxy._types import UserAPIKeyAuth
 from litellm.proxy.guardrails.guardrail_helpers import should_proceed_based_on_metadata
 from litellm.types.guardrails import GuardrailEventHooks
 
 GUARDRAIL_NAME = "aporia"
-_DECODED_JSON: Final = TypeAdapter(object)
 
 
 class AporiaGuardrail(CustomGuardrail):
@@ -71,7 +68,7 @@ class AporiaGuardrail(CustomGuardrail):
     async def prepare_aporia_request(
         self, new_messages: List[dict], response_string: Optional[str] = None
     ) -> dict:
-        data: dict[str, object] = {}
+        data: dict[str, Any] = {}
         if new_messages is not None:
             data["messages"] = new_messages
         if response_string is not None:
@@ -125,10 +122,8 @@ class AporiaGuardrail(CustomGuardrail):
         verbose_proxy_logger.debug("Aporia AI response: %s", response.text)
         if response.status_code == 200:
             # check if the response was flagged
-            _json_response: Final = decoded_json_object(
+            _json_response = response.json()
-                _DECODED_JSON.validate_python(response.json())
+            action: str = _json_response.get(
-            )
-            action: Final = _json_response.get(
                 "action"
             )  # possible values are modify, passthrough, block, rephrase
             if action == "block":

enterprise/enterprise_hooks/blocked_user_list.py

@@ -7,10 +7,9 @@
 ## This accepts a list of user id's for whom calls will be rejected
 
 
-from typing import Final, Literal, Optional
+from typing import Literal, Optional
 
 from fastapi import HTTPException
-from pydantic import TypeAdapter
 
 import litellm
 from litellm._internal_context import with_service_target
@@ -20,9 +19,6 @@ from litellm.integrations.custom_logger import CustomLogger
 from litellm.proxy._types import LiteLLM_EndUserTable, UserAPIKeyAuth
 from litellm.proxy.common_utils.user_api_key_cache import AUTH_OBJECTS_TARGET
 from litellm.proxy.utils import PrismaClient
-from litellm.repositories.table_repositories import EndUserRepository
-
-_REQUEST_BODY_VALUE: Final = TypeAdapter(object)
 
 
 class ENTERPRISE_BlockedUserList(CustomLogger):
@@ -98,14 +94,15 @@ class ENTERPRISE_BlockedUserList(CustomLogger):
                 )
                 if end_user_cache_obj is None and self.prisma_client is not None:
                     # check db
-                    end_user_id: Final = _REQUEST_BODY_VALUE.validate_python(user)
+                    end_user_obj = (
-                    end_user_obj = await EndUserRepository(
+                        await self.prisma_client.db.litellm_endusertable.find_unique(
-                        self.prisma_client
+                            where={"user_id": user}
-                    ).table.find_unique(where={"user_id": end_user_id})
+                        )
+                    )
                     if end_user_obj is None:  # user not in db - assume not blocked
                         end_user_obj = LiteLLM_EndUserTable(user_id=user, blocked=False)
                     cache.set_cache(key=cache_key, value=end_user_obj, ttl=60)
-                    if end_user_obj.blocked is True:
+                    if end_user_obj is not None and end_user_obj.blocked is True:
                         raise HTTPException(
                             status_code=400,
                             detail={

enterprise/enterprise_hooks/google_text_moderation.py

@@ -6,23 +6,15 @@
 # +-----------------------------------------------+
 #  Thank you users! We ❤️ you! - Krrish & Ishaan
 
-from typing import Final
-
 from fastapi import HTTPException
-from pydantic import TypeAdapter
 
 import litellm
 from litellm._logging import verbose_proxy_logger
 from litellm.integrations.custom_logger import CustomLogger
 from litellm.proxy._types import UserAPIKeyAuth
-from litellm.proxy.guardrails._content_utils import (
+from litellm.proxy.guardrails._content_utils import iter_message_text
-    is_non_conversational_request,
-    iter_message_text,
-)
 from litellm.types.utils import CallTypesLiteral
 
-_THRESHOLD_VALUE: Final = TypeAdapter(object)
-
 
 class ENTERPRISE_GoogleTextModeration(CustomLogger):
     user_api_key_cache = None
@@ -77,11 +69,9 @@ class ENTERPRISE_GoogleTextModeration(CustomLogger):
                     f"{category}_confidence_threshold",
                     default_confidence_threshold,
                 )
-            set_confidence_value = _THRESHOLD_VALUE.validate_python(
+            set_confidence_value = getattr(
-                getattr(
+                self,
-                    self,
+                f"{category}_confidence_threshold",
-                    f"{category}_confidence_threshold",
-                )
             )
             verbose_proxy_logger.info(
                 f"Google Text Moderation: {category}_confidence_threshold: {set_confidence_value}"
@@ -97,7 +87,7 @@ class ENTERPRISE_GoogleTextModeration(CustomLogger):
 
     async def async_moderation_hook(
         self,
-        data: dict[str, object],
+        data: dict,
         user_api_key_dict: UserAPIKeyAuth,
         call_type: CallTypesLiteral,
     ):
@@ -105,9 +95,6 @@ class ENTERPRISE_GoogleTextModeration(CustomLogger):
         - Calls Google's Text Moderation API
         - Rejects request if it fails safety check
         """
-        if is_non_conversational_request(call_type, data):
-            verbose_proxy_logger.debug("Google text moderation: skipping non-conversational call type %s", call_type)
-            return
         # Covers multimodal list content + Responses-API input.
         text = "".join(iter_message_text(data))
         if text:

enterprise/litellm_enterprise/enterprise_callbacks/callback_controls.py

@@ -33,8 +33,9 @@ class EnterpriseCallbackControls:
 
             try:
                 disabled_callbacks = EnterpriseCallbackControls.get_disabled_callbacks(litellm_params, standard_callback_dynamic_params)
+                verbose_logger.debug(f"Dynamically disabled callbacks from {X_LITELLM_DISABLE_CALLBACKS}: {disabled_callbacks}")
+                verbose_logger.debug(f"Checking if {callback} is disabled via headers. Disable callbacks from headers: {disabled_callbacks}")
                 if disabled_callbacks is not None:
-                    verbose_logger.debug(f"Checking if {callback} is in dynamically disabled callbacks: {disabled_callbacks}")
                     #########################################################
                     # premium user check
                     #########################################################

enterprise/litellm_enterprise/enterprise_callbacks/example_logging_api.py

@@ -1,20 +1,15 @@
 # this is an example endpoint to receive data from litellm
-from typing import Final
-
 from fastapi import FastAPI, HTTPException, Request
-from pydantic import TypeAdapter
 
 app = FastAPI()
 
-_LOGGED_EVENT: Final = TypeAdapter(object)
-
 
 @app.post("/log-event")
 async def log_event(request: Request):
     try:
         print("Received /log-event request")  # noqa
         # Assuming the incoming request has JSON data
-        data: Final = _LOGGED_EVENT.validate_python(await request.json())
+        data = await request.json()
         print("Received request data:")  # noqa
         print(data)  # noqa
 

enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py

@@ -8,30 +8,25 @@
 ## This provides an LLM Guard Integration for content moderation on the proxy
 
 import asyncio
-from collections.abc import Mapping
+from typing import Final, Optional
-from typing import Final
 
 import aiohttp
 from fastapi import HTTPException
-from pydantic import TypeAdapter
 
 import litellm
 from litellm._logging import verbose_proxy_logger
 from litellm.integrations.custom_logger import CustomLogger
-from litellm.llms.openai.common_utils import decoded_json_object
 from litellm.proxy._types import UserAPIKeyAuth
 from litellm.secret_managers.main import get_secret_str
 from litellm.types.utils import CallTypesLiteral
 
-_DECODED_JSON: Final = TypeAdapter(object)
-
 
 class _ENTERPRISE_LLMGuard(CustomLogger):
     # Class variables or attributes
     def __init__(
         self,
         mock_testing: bool = False,
-        mock_redacted_text: Mapping[str, object] | None = None,
+        mock_redacted_text: Optional[dict] = None,
     ):
         self.mock_redacted_text = mock_redacted_text
         self.llm_guard_mode = litellm.llm_guard_mode
@@ -71,9 +66,7 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
                     async with session.post(
                         analyze_url, json={"prompt": text}
                     ) as response:
-                        redacted_text = _DECODED_JSON.validate_python(
+                        redacted_text = await response.json()
-                            await response.json()
-                        )
             verbose_proxy_logger.debug(
                 f"LLM Guard: Received response - {redacted_text}"
             )
@@ -84,13 +77,12 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
                         "error": f"Invalid content moderation response: {redacted_text}"
                     },
                 )
-            redacted_response: Final = decoded_json_object(redacted_text)
+            if redacted_text.get("is_valid", None) is False:
-            if redacted_response.get("is_valid", None) is False:
                 raise HTTPException(
                     status_code=400,
                     detail={"error": "Violated content safety policy"},
                 )
-            sanitized_prompt: Final = redacted_response.get("sanitized_prompt")
+            sanitized_prompt = redacted_text.get("sanitized_prompt")
             return sanitized_prompt if isinstance(sanitized_prompt, str) else text
         except Exception as e:
             verbose_proxy_logger.exception(

enterprise/litellm_enterprise/enterprise_callbacks/secret_detection.py

@@ -13,28 +13,21 @@ sys.path.insert(
 )  # Adds the parent directory to the system path
 import configparser
 import contextlib
-import hashlib
 import itertools
-import json
 import re
 import tempfile
-from collections import OrderedDict
 from collections.abc import Generator, Iterator, Sequence
 from contextvars import ContextVar
-from functools import partial
 from typing import TYPE_CHECKING, ClassVar, Final, Literal, Optional
 
-from pydantic import JsonValue, TypeAdapter
-
 from litellm._logging import verbose_proxy_logger
 from litellm.caching.caching import DualCache
 from litellm.integrations.custom_guardrail import (
     CustomGuardrail,
     log_guardrail_information,
 )
-from litellm.llms.base_llm.guardrail_translation.utils import effective_skip_tool_message_for_guardrail
 from litellm.proxy._types import UserAPIKeyAuth
-from litellm.proxy.guardrails._content_utils import iter_tool_json_fields, map_tool_json_text, walk_user_text
+from litellm.proxy.guardrails._content_utils import walk_user_text
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.utils import GenericGuardrailAPIInputs
 
@@ -47,7 +40,7 @@ GUARDRAIL_PROVIDER = "hide-secrets"
 
 # Per-invocation tally of redacted secrets by detect-secrets plugin type; None
 # means the guardrail did not run, so _process_response records nothing.
-_masked_entity_count: ContextVar[Optional[dict[str, int]]] = ContextVar(
+_masked_entity_count: ContextVar[Optional[dict]] = ContextVar(
     "hide_secrets_masked_entity_count", default=None
 )
 
@@ -456,19 +449,6 @@ _default_detect_secrets_config = {
 
 _CONFIG_SECTION: Final = "litellm-prompt"
 
-_JSON_STRING: Final = TypeAdapter(str)
-
-_KEY_CONTEXT: Final = 64
-
-# Longest JSON line scanned for a multi-line tool string. Escaping puts the whole string on one line, and
-# detect-secrets regexes backtrack across it, so a longer multi-line string is cut into runs of whole lines,
-# each scanned as its own JSON line beside the key. Every run holds at least two lines, so the high-entropy
-# detectors, which match only a quoted string made wholly of their charset, never judge a lone line.
-_JSON_LINE_LIMIT: Final = 4096
-
-# A detector can stop a match inside an escape of a JSON-escaped line, as at the backslash of \".
-_CUT_ESCAPE: Final = re.compile(r"(?<=.)\\(?:u[0-9A-Fa-f]{0,3})?\Z", re.DOTALL)
-
 _ASSIGNMENT_LINE: Final = re.compile(r"[^\s\[#;:=][^:=]*[:=]")
 
 _SHELL_ASSIGNMENT: Final = re.compile(r"(?P<key>[^\s\[#;:=](?:[^:=]*[^\s:=])?)=(?P<value>\S+)")
@@ -479,10 +459,6 @@ _SHELL_TRAILER: Final = re.compile(r"\\|#.*|-*\w[\w.-]*=\S*")
 
 _SCAN_SUFFIX: Final = ".py"
 
-# Digests of scan inputs that held no secret, so the history an agent resends every turn is scanned once. 65536
-# digests take about 10 MB and hold a few hundred interleaved agent sessions per worker.
-_CLEAN_SCAN_CACHE_SIZE: Final = 65536
-
 
 @contextlib.contextmanager
 def _temp_file(text: str) -> Generator[str, None, None]:
@@ -556,57 +532,6 @@ def _quoted_assignments(text: str) -> tuple[str, ...]:
     )
 
 
-def _json_unescaped(value: str) -> str:
-    for candidate in (value, _CUT_ESCAPE.sub("", value)):
-        try:
-            return _JSON_STRING.validate_json(f'"{candidate}"')
-        except ValueError:
-            continue
-    return value
-
-
-def _key_context(key: str) -> str:
-    return json.dumps(" ".join(key.split()), ensure_ascii=False)[1:-1][-_KEY_CONTEXT:]
-
-
-def _json_line(context: str | None, text: str) -> str:
-    value: Final = json.dumps(text, ensure_ascii=False)
-    return value if context is None else f'"{context}": {value}'
-
-
-def _json_lines(context: str | None, text: str) -> Iterator[str]:
-    """Yield ``text`` as one JSON line, or a long multi-line ``text`` as JSON lines of two or more whole lines."""
-    line: Final = _json_line(context, text)
-    lines: Final = text.splitlines(keepends=True)
-    if len(line) <= _JSON_LINE_LIMIT or len(lines) < 2:
-        yield line
-        return
-    budget: Final = _JSON_LINE_LIMIT - len(_json_line(context, ""))
-    sizes: Final = (len(json.dumps(part, ensure_ascii=False)) - 2 for part in lines)
-    states: Final = itertools.accumulate(sizes, partial(_next_run_state, budget), initial=(0, 0, 0))
-    run_ids: Final = (state[0] for state in itertools.islice(states, 1, None))
-    runs: Final = tuple(
-        tuple(part for part, _ in run) for _, run in itertools.groupby(zip(lines, run_ids), key=lambda pair: pair[1])
-    )
-    whole_runs: Final = (*runs[:-2], runs[-2] + runs[-1]) if len(runs[-1]) < 2 else runs
-    for run in whole_runs:
-        yield _json_line(context, "".join(run[:-1]) + run[-1].splitlines()[0])
-
-
-def _next_run_state(budget: int, state: tuple[int, int, int], size: int) -> tuple[int, int, int]:
-    """Advance ``(run id, lines in run, escaped size of run)`` by one line of escaped ``size``."""
-    run_id, count, total = state
-    if count >= 2 and total + size > budget:
-        return run_id + 1, 1, size
-    return run_id, count + 1, total + size
-
-
-def _value_in_texts(value: str, texts: Sequence[str]) -> str | None:
-    """Return the detected value as it appears in ``texts``, decoding it when it came from a JSON-escaped key line."""
-    candidates: Final = (value, _json_unescaped(value))
-    return next((candidate for candidate in candidates if any(candidate in text for text in texts)), None)
-
-
 class _ENTERPRISE_SecretDetection(CustomGuardrail):
     # Keeps proxied traffic on async_pre_call_hook (the unified apply_guardrail
     # path skips should_run_check and never sees data["prompt"]).
@@ -614,16 +539,11 @@ class _ENTERPRISE_SecretDetection(CustomGuardrail):
 
     def __init__(self, detect_secrets_config: Optional[dict] = None, **kwargs):
         self.user_defined_detect_secrets_config = detect_secrets_config
-        self._clean_scans: OrderedDict[bytes, None] = OrderedDict()
         super().__init__(**kwargs)
 
-    def scan_message_for_secrets(self, message_content: str) -> list[dict[str, str]]:
+    def scan_message_for_secrets(self, message_content: str):
         from detect_secrets.settings import transient_settings
 
-        digest: Final = hashlib.sha256(message_content.encode()).digest()
-        if digest in self._clean_scans:
-            self._clean_scans.move_to_end(digest)
-            return []
         detect_secrets_config = (
             self.user_defined_detect_secrets_config or _default_detect_secrets_config
         )
@@ -632,10 +552,6 @@ class _ENTERPRISE_SecretDetection(CustomGuardrail):
                 (*message_content.splitlines(), *_quoted_assignments(message_content))
             )
 
-        if not found:
-            self._clean_scans[digest] = None
-            if len(self._clean_scans) > _CLEAN_SCAN_CACHE_SIZE:
-                self._clean_scans.popitem(last=False)
         return [
             {"type": secret_type, "value": value}
             for value, secret_type in sorted(
@@ -664,35 +580,6 @@ class _ENTERPRISE_SecretDetection(CustomGuardrail):
         )
         return pattern.sub("[REDACTED]", text)
 
-    def redact_tool_payload(self, payload: JsonValue) -> JsonValue:
-        """Redact secrets in a tool payload's string values, scanning each value alone and as its JSON lines."""
-        fields: Final = tuple(iter_tool_json_fields(payload))
-        texts: Final = tuple(text for _, text in fields)
-        contexts: Final = {key: _key_context(key) for key in {key for key, _ in fields if key is not None}}
-        keyed: Final = tuple(
-            itertools.chain.from_iterable(
-                _json_lines(None if key is None else contexts[key], text) for key, text in fields
-            )
-        )
-        detected: Final = tuple(
-            {**secret, "value": value}
-            for secret in self.scan_message_for_secrets("\n".join((*texts, *keyed)))
-            if (value := _value_in_texts(secret["value"], texts)) is not None
-        )
-        if not detected:
-            return payload
-        counts: Final = _masked_entity_count.get()
-        if counts is not None:
-            for secret in detected:
-                counts[secret["type"]] = counts.get(secret["type"], 0) + 1
-        verbose_proxy_logger.warning(
-            "Detected and redacted secrets in tool payload: %s",
-            sorted(dict.fromkeys(secret["type"] for secret in detected)),
-        )
-        values: Final = sorted({secret["value"] for secret in detected}, key=len, reverse=True)
-        pattern: Final = re.compile("|".join(re.escape(value) for value in values))
-        return map_tool_json_text(payload, lambda text: pattern.sub("[REDACTED]", text))
-
     async def should_run_check(self, user_api_key_dict: UserAPIKeyAuth) -> bool:
         if user_api_key_dict.permissions is not None:
             if GUARDRAIL_NAME in user_api_key_dict.permissions:
@@ -754,12 +641,7 @@ class _ENTERPRISE_SecretDetection(CustomGuardrail):
         _masked_entity_count.set({})
 
         # Covers multimodal list content + Responses-API input.
-        inspected = walk_user_text(
+        inspected = walk_user_text(data, self.redact_text) + self._redact_prompt(data)
-            data,
-            self.redact_text,
-            self.redact_tool_payload,
-            skip_tool_message=effective_skip_tool_message_for_guardrail(self),
-        ) + self._redact_prompt(data)
 
         if inspected == 0:
             # Image-only, empty-text, and unsupported payloads inspected

enterprise/litellm_enterprise/integrations/custom_guardrail.py

@@ -20,15 +20,9 @@ class EnterpriseCustomGuardrailHelper:
           the tag's value (the mode for that tag).
         - If no request tag matches, fall back to default mode(s).
         """
-        from litellm.litellm_core_utils.core_helpers import (
-            get_metadata_variable_name_from_kwargs,
-        )
         from litellm.litellm_core_utils.litellm_logging import (
             StandardLoggingPayloadSetup,
         )
-        from litellm.router_strategy.tag_based_routing import (
-            without_baggage_mapped_tags,
-        )
         from litellm.proxy._types import CommonProxyErrors
         from litellm.proxy.proxy_server import premium_user
 
@@ -42,13 +36,9 @@ class EnterpriseCustomGuardrailHelper:
 
         proxy_server_request = data.get("proxy_server_request", {})
 
-        request_tags = without_baggage_mapped_tags(
+        request_tags = StandardLoggingPayloadSetup.get_request_tags(
-            StandardLoggingPayloadSetup.get_request_tags(
+            litellm_params=data,
-                litellm_params=data,
+            proxy_server_request=proxy_server_request,
-                proxy_server_request=proxy_server_request,
-            ),
-            data.get(get_metadata_variable_name_from_kwargs(data)),
-            keep_serving_deployment_tags=True,
         )
 
         # Check if any request tag matches a Mode tag key

enterprise/litellm_enterprise/proxy/auth/route_checks.py

@@ -2,8 +2,6 @@ import os
 
 from fastapi import HTTPException, status
 
-from litellm.proxy._types import MODEL_LIST_ROUTES
-
 
 class EnterpriseRouteChecks:
     @staticmethod
@@ -45,7 +43,7 @@ class EnterpriseRouteChecks:
 
     # Routes that should remain accessible even when LLM API endpoints are disabled.
     # These are read-only model listing routes needed by the Admin UI.
-    LLM_API_EXEMPT_ROUTES = MODEL_LIST_ROUTES
+    LLM_API_EXEMPT_ROUTES = ["/models", "/v1/models"]
 
     @staticmethod
     def should_call_route(route: str):

enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py

@@ -3,24 +3,20 @@ Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if t
 """
 
 import asyncio
-import json
+from collections.abc import Mapping, Sequence
-import re
-from collections.abc import Callable, Mapping, Sequence
 from dataclasses import replace as dataclasses_replace
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
+from types import MappingProxyType
 from typing import TYPE_CHECKING, Final, List, Literal, Optional, Protocol, Tuple, cast
-from pydantic import TypeAdapter, ValidationError
 
 from litellm_enterprise.proxy.common_utils.managed_object_attribution import (
     ManagedObjectCreatorAttribution,
     ManagedObjectRow,
-    row_metadata,
 )
 
 import litellm
 from litellm._logging import verbose_proxy_logger
 from litellm._uuid import uuid
-from litellm.litellm_core_utils.cloud_storage_security import stored_s3_bucket_kwargs
 from litellm.constants import (
     BATCH_LINE_ITEMS_MAX_PART_BYTES,
     MANAGED_OBJECT_STALENESS_CUTOFF_DAYS,
@@ -29,7 +25,6 @@ from litellm.constants import (
 from litellm.repositories.table_repositories import ManagedObjectRepository
 from litellm.repositories.team_repository import TeamRepository
 from litellm.repositories.verification_token_repository import VerificationTokenRepository
-from litellm.utils import get_utc_datetime
 
 if TYPE_CHECKING:
     from prisma import models as prisma_models
@@ -43,16 +38,7 @@ if TYPE_CHECKING:
     from litellm.types.utils import LiteLLMBatch
 
 
-def _error_names_resource(error: Exception, resource_id: str) -> bool:
-    tokens: Final[list[str]] = re.findall(r"""[^\s"'`<>(]+""", str(error))
-    return any(token.rstrip(".,:;)") == resource_id for token in tokens)
-
-
 CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
-RECENT_BATCH_POLL_WINDOW: Final = timedelta(hours=48)
-GIVE_UP_POLLING_AFTER: Final = timedelta(days=2 * MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
-_FILE_OBJECT_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
-
 
 PROVIDER_TERMINAL_BATCH_STATUSES: Final[Tuple[str, ...]] = (
     "completed",
@@ -97,8 +83,9 @@ def _team_table(prisma_client: "PrismaClient") -> "TableActions[prisma_models.Li
     return TeamRepository(prisma_client).table
 
 
-def _unique_jobs(jobs: Sequence[ManagedObjectRow]) -> tuple[ManagedObjectRow, ...]:
+def _row_metadata(row: object) -> dict[str, object]:
-    return tuple({job.id: job for job in jobs}.values())
+    metadata: Final = getattr(row, "metadata", None)
+    return cast(dict[str, object], metadata) if isinstance(metadata, dict) else {}  # cast-ok: prisma Json object column
 
 
 class CheckBatchCost:
@@ -108,7 +95,6 @@ class CheckBatchCost:
         prisma_client: "PrismaClient",
         llm_router: "Router",
         track_unmanaged_batch_cost: bool = False,
-        clock: Callable[[], datetime] = get_utc_datetime,
     ):
         from litellm.proxy.utils import PrismaClient, ProxyLogging
         from litellm.router import Router
@@ -117,17 +103,11 @@ class CheckBatchCost:
         self.prisma_client: PrismaClient = prisma_client
         self.llm_router: Router = llm_router
         self._track_unmanaged_batch_cost = track_unmanaged_batch_cost
-        self._clock: Final = clock
         self._creator_attribution: Final = ManagedObjectCreatorAttribution(prisma_client, "CheckBatchCost")
         # Cached after the first poll cycle. Once we know the column is absent we skip
         # the guaranteed-failing primary query on every subsequent cycle.
         self._has_batch_processed_column: bool = True
         self.batch_processed_support_confirmed: bool = False
-        self._poll_cursor: tuple[datetime, str] | None = None
-        self._poll_before: datetime = clock()
-        self._arrival_cursor: tuple[datetime, str] | None = None
-        self._arrival_before: datetime = self._poll_before
-        self._poll_arrivals: bool = False
         self._line_item_stores: Final[set[asyncio.Task[None]]] = set()
         self._line_item_store_turn: Final = asyncio.Lock()
         self._poll_idle: Final = asyncio.Event()
@@ -159,9 +139,50 @@ class CheckBatchCost:
             return
         self.batch_processed_support_confirmed = True
 
+    async def _cleanup_stale_managed_objects(self) -> None:
+        """
+        Mark managed objects older than MANAGED_OBJECT_STALENESS_CUTOFF_DAYS days
+        in non-terminal states as 'stale_expired'. These will never complete and
+        should not be polled.
+        """
+        cutoff: Final = datetime.now(timezone.utc) - timedelta(days=MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
+        result: Final = await _managed_object_table(self.prisma_client).update_many(
+            where={
+                "file_purpose": "batch",
+                "status": {"not_in": list(TERMINAL_MANAGED_OBJECT_STATUSES)},
+                "created_at": {"lt": cutoff},
+            },
+            data={"status": "stale_expired"},
+        )
+        if result > 0:
+            verbose_proxy_logger.warning(
+                f"CheckBatchCost: marked {result} stale managed objects "
+                f"(older than {MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days) as stale_expired"
+            )
+
+        if not self._has_batch_processed_column:
+            return
+
+        # A row already in a terminal status is never rewritten by the sweep above, so
+        # without this it keeps a poll-page slot forever and starves newer batches.
+        retired: Final = await _managed_object_table(self.prisma_client).update_many(
+            where={
+                "file_purpose": "batch",
+                "batch_processed": False,
+                "status": {"in": ["complete", "completed"]},
+                "created_at": {"lt": cutoff},
+            },
+            data={"batch_processed": True},
+        )
+        if retired > 0:
+            verbose_proxy_logger.warning(
+                f"CheckBatchCost: gave up on {retired} completed managed objects older than "
+                f"{MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days that were never costed"
+            )
+
     async def _fallback_find_jobs(self) -> "Sequence[ManagedObjectRow]":
         """Query batch jobs without the batch_processed filter (for older schemas)."""
-        return await self._find_jobs(
+        return await _managed_object_table(self.prisma_client).find_many(
             where={
                 "file_purpose": "batch",
                 "status": {
@@ -175,118 +196,10 @@ class CheckBatchCost:
                     ]
                 },
             },
+            take=MAX_OBJECTS_PER_POLL_CYCLE,
+            order={"created_at": "asc"},
         )
 
-    async def _find_jobs(self, where: Mapping[str, object]) -> Sequence[ManagedObjectRow]:
-        lead_arrivals: Final = self._poll_arrivals
-        self._poll_arrivals = not lead_arrivals
-        lead_share: Final = MAX_OBJECTS_PER_POLL_CYCLE - MAX_OBJECTS_PER_POLL_CYCLE // 2
-        lead: Final = await self._find_job_page(where, arrivals=lead_arrivals, take=lead_share)
-        trail: Final = await self._find_job_page(
-            where, arrivals=not lead_arrivals, take=MAX_OBJECTS_PER_POLL_CYCLE - len(lead)
-        )
-        spill: Final = (
-            await self._find_job_page(
-                where, arrivals=lead_arrivals, take=MAX_OBJECTS_PER_POLL_CYCLE - len(lead) - len(trail)
-            )
-            if len(lead) == lead_share
-            else ()
-        )
-        return _unique_jobs((*lead, *trail, *spill))
-
-    async def _find_job_page(
-        self, where: Mapping[str, object], *, arrivals: bool, take: int
-    ) -> Sequence[ManagedObjectRow]:
-        if take == 0:
-            return ()
-        resumed_mid_lane: Final = (self._arrival_cursor if arrivals else self._poll_cursor) is not None
-        page: Final = await self._find_job_page_from_cursor(where, arrivals=arrivals, take=take)
-        if len(page) == take or not resumed_mid_lane:
-            return page
-        wrapped: Final = await self._find_job_page_from_cursor(where, arrivals=arrivals, take=take - len(page))
-        return _unique_jobs((*page, *wrapped))
-
-    async def _find_job_page_from_cursor(
-        self, where: Mapping[str, object], *, arrivals: bool, take: int
-    ) -> Sequence[ManagedObjectRow]:
-        cursor: Final = self._arrival_cursor if arrivals else self._poll_cursor
-        if cursor is None:
-            if arrivals:
-                self._arrival_before = self._clock()
-            else:
-                self._poll_before = self._clock() - RECENT_BATCH_POLL_WINDOW
-        before: Final = self._arrival_before if arrivals else self._poll_before
-        page: Final = await _managed_object_table(self.prisma_client).find_many(
-            where={
-                **where,
-                "created_at": {
-                    "lte": before,
-                    **({"gt": before - RECENT_BATCH_POLL_WINDOW} if arrivals else {}),
-                },
-                **(
-                    {
-                        "OR": [
-                            {"created_at": {"gt": cursor[0]}},
-                            {"created_at": cursor[0], "id": {"gt": cursor[1]}},
-                        ]
-                    }
-                    if cursor is not None
-                    else {}
-                ),
-            },
-            take=take,
-            order=[{"created_at": "asc"}, {"id": "asc"}],
-        )
-        next_cursor: Final = (page[-1].created_at, page[-1].id) if len(page) == take else None
-        if arrivals:
-            self._arrival_cursor = next_cursor
-        else:
-            self._poll_cursor = next_cursor
-        return page
-
-    def _unprocessed_snapshot(self, job: ManagedObjectRow) -> Mapping[str, object]:
-        if self._has_batch_processed_column:
-            return {"id": job.id, "batch_processed": False}
-        return {"id": job.id, "updated_at": job.updated_at}
-
-    def _is_past_give_up_age(self, job: ManagedObjectRow) -> bool:
-        return job.created_at < self._clock() - GIVE_UP_POLLING_AFTER
-
-    async def _expire_if_old_and_unfinished(self, job: ManagedObjectRow) -> None:
-        if job.status in TERMINAL_MANAGED_OBJECT_STATUSES or not self._is_past_give_up_age(job):
-            return
-        await self._expire_unfinished(job)
-
-    @staticmethod
-    def _is_completed_with_output_id_pending(response: "LiteLLMBatch") -> bool:
-        from litellm.proxy.openai_files_endpoints.common_utils import (
-            _completed_batch_safe_to_retire,
-        )
-
-        return (
-            response.status in ("completed", "complete")
-            and response.output_file_id is None
-            and not _completed_batch_safe_to_retire(response)
-        )
-
-    async def _expire_unfinished(self, job: ManagedObjectRow) -> None:
-        try:
-            expired: Final = await _managed_object_table(self.prisma_client).update_many(
-                where={
-                    **self._unprocessed_snapshot(job),
-                    "status": {"not_in": list(TERMINAL_MANAGED_OBJECT_STATUSES)},
-                },
-                data={"status": "stale_expired"},
-            )
-        except Exception as db_err:
-            verbose_proxy_logger.error(f"CheckBatchCost: failed to mark stale job {job.id} as stale_expired: {db_err}")
-            return
-        if expired > 0:
-            verbose_proxy_logger.warning(
-                f"CheckBatchCost: job {job.id} is older than {GIVE_UP_POLLING_AFTER.days} days "
-                "and still not finished, so it is marked stale_expired and no longer polled"
-            )
-
     async def _retire_job(self, job: "ManagedObjectRow", reason: str) -> None:
         """
         Take a row that can never be costed out of the poll page. Leaving it selectable
@@ -294,22 +207,27 @@ class CheckBatchCost:
         once enough such rows accumulate no newer batch is ever reached. Older schemas
         without batch_processed can only be excluded through the status filter.
         """
-        data: Final = {"batch_processed": True} if self._has_batch_processed_column else {"status": "stale_expired"}
+        data: Final = (
+            {"batch_processed": True}
+            if self._has_batch_processed_column
+            else {"status": "stale_expired"}
+        )
         try:
-            retired: Final = await _managed_object_table(self.prisma_client).update_many(
+            await _managed_object_table(self.prisma_client).update(
-                where=self._unprocessed_snapshot(job),
+                where={"id": job.id},
                 data=data,
             )
         except Exception as db_err:
-            verbose_proxy_logger.error(f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}")
+            verbose_proxy_logger.error(
-            return
+                f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}"
-        if retired == 0:
+            )
             return
         verbose_proxy_logger.warning(
-            f"CheckBatchCost: job {job.id} can never be costed ({reason}), so it will no longer be polled"
+            f"CheckBatchCost: job {job.id} can never be costed ({reason}), "
+            "so it will no longer be polled"
         )
 
-    async def _claim_job_for_costing(self, job: "ManagedObjectRow", response: "LiteLLMBatch") -> bool:
+    async def _claim_job_for_costing(self, job: "ManagedObjectRow") -> bool:
         """
         Atomically flip batch_processed from false to true, returning whether this pod won
         the row. Every pod and uvicorn worker schedules its own poller against the shared
@@ -323,16 +241,10 @@ class CheckBatchCost:
         """
         if not self._has_batch_processed_column:
             return True
-        from litellm_enterprise.proxy.hooks.managed_files import postgres_storable_json
-
         try:
             claimed: Final = await _managed_object_table(self.prisma_client).update_many(
                 where={"id": job.id, "batch_processed": False},
-                data={
+                data={"batch_processed": True},
-                    "batch_processed": True,
-                    "status": response.status if response.status != "completed" else "complete",
-                    "file_object": postgres_storable_json(response),
-                },
             )
         except Exception as db_err:
             verbose_proxy_logger.error(
@@ -341,9 +253,7 @@ class CheckBatchCost:
             return False
         return claimed > 0
 
-    async def _release_job_claim(
+    async def _release_job_claim(self, job: "ManagedObjectRow") -> None:
-        self, job: "ManagedObjectRow", original_status: str | None, original_file_object: object
-    ) -> None:
         """Give a claimed row back once billing it failed, so a later poll cycle retries it.
 
         Safe to match on batch_processed=True: while this poller is active the retrieve
@@ -355,11 +265,7 @@ class CheckBatchCost:
         try:
             await _managed_object_table(self.prisma_client).update_many(
                 where={"id": job.id, "batch_processed": True},
-                data={
+                data={"batch_processed": False},
-                    "batch_processed": False,
-                    "status": original_status,
-                    "file_object": json.dumps(original_file_object),
-                },
             )
         except Exception as db_err:
             verbose_proxy_logger.error(
@@ -393,34 +299,28 @@ class CheckBatchCost:
 
         from litellm.exceptions import NotFoundError
 
-        return isinstance(error, (NotFoundError, openai.NotFoundError)) and _error_names_resource(error, batch_id)
+        return isinstance(error, (NotFoundError, openai.NotFoundError)) and batch_id in str(error)
 
     def _batch_deployment_exists(self, model_id: str) -> bool:
         """A 404 only proves the batch is gone when it came from the batch's own
         deployment. Once that deployment leaves the router, default fallbacks can
         silently send the retrieve to a provider that never saw the batch, so its
-        404 must not retire a potentially recoverable row."""
+        404 must not retire the row; the staleness sweep bounds it instead."""
-        return self.llm_router.registry().deployment(model_id) is not None
+        return self.llm_router.get_deployment(model_id=model_id) is not None
 
     @staticmethod
     def _is_output_file_gone_at_provider(error: Exception, output_file_id: Optional[str]) -> bool:
         """A 404 naming the output file means there is nothing to fetch on this or any
         later poll: providers like Vertex AI advertise an output path for every batch,
-        including terminal ones that never wrote it. GCS names a gs:// object without its scheme,
+        including terminal ones that never wrote it. Any other failure may be
-        and S3 names an s3:// object by its key alone. Any other failure keeps retrying."""
+        transient, so it keeps retrying until the staleness sweep bounds it."""
         import openai
 
         from litellm.exceptions import NotFoundError
-        from litellm.llms.base_llm.chat.transformation import BaseLLMException
 
         if not output_file_id:
             return False
-        is_not_found: Final = isinstance(error, (NotFoundError, openai.NotFoundError)) or (
+        return isinstance(error, (NotFoundError, openai.NotFoundError)) and output_file_id in str(error)
-            isinstance(error, BaseLLMException) and error.status_code == 404
-        )
-        object_path: Final = output_file_id.split("://", 1)[-1]
-        names: Final = (output_file_id, object_path, object_path.split("/", 1)[-1])
-        return is_not_found and any(_error_names_resource(error, name) for name in names)
 
     async def _finalize_unbilled_terminal_job(
         self, job: "ManagedObjectRow", response: "LiteLLMBatch"
@@ -442,18 +342,18 @@ class CheckBatchCost:
                 db_batch_object=job,
                 unified_batch_id=_is_base64_encoded_unified_file_id(job.unified_object_id),
             )
-            update_data: Final[dict[str, object]] = {
+            update_data: Final[dict] = {
                 "status": response.status,
                 "file_object": response.model_dump_json(),
                 **({"batch_processed": True} if self._has_batch_processed_column else {}),
             }
-            finalized: Final = await _managed_object_table(self.prisma_client).update_many(
+            await _managed_object_table(self.prisma_client).update(
-                where=self._unprocessed_snapshot(job),
+                where={"id": job.id},
                 data=update_data,
             )
-            if finalized == 0:
+            verbose_proxy_logger.info(
-                return False
+                f"CheckBatchCost: marked job {job.id} as {response.status} in DB"
-            verbose_proxy_logger.info(f"CheckBatchCost: marked job {job.id} as {response.status} in DB")
+            )
             return True
         except Exception as db_err:
             verbose_proxy_logger.error(
@@ -552,10 +452,6 @@ class CheckBatchCost:
         llm_provider: str,
         bare_model_name: str,
     ) -> Optional[Tuple[str, str]]:
-        stored_model_id: Final = self._get_stored_model_id(job)
-        if stored_model_id is not None and self._batch_deployment_exists(stored_model_id):
-            return stored_model_id, job.unified_object_id
-
         deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
         if deployment_id is None:
             verbose_proxy_logger.info(
@@ -570,7 +466,7 @@ class CheckBatchCost:
     def _get_deployment_id_for_bare_model(
         self, bare_model_name: str, llm_provider: str
     ) -> Optional[str]:
-        model_group = self.llm_router.registry().resolve(bare_model_name, kind="from_id")
+        model_group = self.llm_router.resolve_model_name_from_model_id(bare_model_name)
         deployment_id = (
             self._get_deployment_id_for_provider(model_group, llm_provider) if model_group else None
         )
@@ -586,7 +482,7 @@ class CheckBatchCost:
     ) -> Optional[str]:
         from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
 
-        for deployment in self.llm_router.registry().deployments() or []:
+        for deployment in self.llm_router.get_model_list(model_name=None) or []:
             litellm_params = deployment.get("litellm_params") or {}
             actual_model = litellm_params.get("model")
             if not isinstance(actual_model, str):
@@ -629,9 +525,8 @@ class CheckBatchCost:
         """
         from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
 
-        registry = self.llm_router.registry()
+        for deployment_id in self.llm_router.get_model_ids(model_name=model_group):
-        for deployment_id in registry.names(model_group, kind="ids"):
+            deployment_info = self.llm_router.get_deployment(model_id=deployment_id)
-            deployment_info = registry.deployment(deployment_id)
             if deployment_info is None:
                 continue
             try:
@@ -668,36 +563,25 @@ class CheckBatchCost:
         )
 
     @staticmethod
-    def _parse_file_object(job: "ManagedObjectRow") -> Optional[dict[str, object]]:
+    def _get_input_file_id(job: "ManagedObjectRow") -> Optional[str]:
-        file_object: Final = job.file_object
+        import json
-        try:
-            return _FILE_OBJECT_ADAPTER.validate_python(
-                json.loads(file_object) if isinstance(file_object, str) else file_object
-            )
-        except (json.JSONDecodeError, ValueError, ValidationError):
-            return None
 
-    @classmethod
-    def _get_input_file_id(cls, job: "ManagedObjectRow") -> Optional[str]:
         from litellm.types.utils import LiteLLMBatch
 
-        parsed_file_object: Final = cls._parse_file_object(job)
+        file_object: Final = job.file_object
-        if parsed_file_object is None:
+        try:
+            parsed_file_object: Final[object] = (
+                json.loads(file_object) if isinstance(file_object, str) else file_object
+            )
+        except (json.JSONDecodeError, ValueError):
+            return None
+        if not isinstance(parsed_file_object, dict):
             return None
         try:
             return LiteLLMBatch.model_validate(parsed_file_object).input_file_id
         except Exception:
             return None
 
-    @classmethod
-    def _get_stored_model_id(cls, job: "ManagedObjectRow") -> Optional[str]:
-        """The deployment that created the batch, written into file_object by the managed
-        files hook at create time. Older rows have no such key and fall back to matching
-        the bare model name against the router."""
-        parsed_file_object: Final = cls._parse_file_object(job)
-        model_id: Final = parsed_file_object.get("model_id") if parsed_file_object else None
-        return model_id if isinstance(model_id, str) and model_id else None
-
     async def _track_completed_batch_cost(
         self,
         job: "ManagedObjectRow",
@@ -724,11 +608,7 @@ class CheckBatchCost:
         from litellm.files.types import FileContentCallOptions
         from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
         from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-        from litellm.litellm_core_utils.litellm_logging import (
+        from litellm.litellm_core_utils.litellm_logging import deployment_pricing_model_info, mask_api_base_credentials
-            deployment_pricing_model_info,
-            mask_api_base_credentials,
-            resolve_vertex_location_for_cost,
-        )
         from litellm.proxy.openai_files_endpoints.common_utils import (
             _is_base64_encoded_unified_file_id,
         )
@@ -763,10 +643,11 @@ class CheckBatchCost:
             except (IndexError, AttributeError):
                 pass
 
-        credentials: Final = self.llm_router.registry().credentials(model_id, form="with_provider") or {}
+        credentials: Final = self.llm_router.get_deployment_credentials_with_provider(model_id) or {}
         _file_content = await afile_content(
             file_id=cast(str, raw_output_file_id),
-            **cast(FileContentCallOptions, {**credentials, **stored_s3_bucket_kwargs(cast(str, raw_output_file_id))}),
+            _litellm_internal_model_credentials=MappingProxyType(dict(credentials)),
+            **cast(FileContentCallOptions, credentials),
         )
 
         content_bytes: Final = await _file_content_bytes(_file_content)
@@ -785,7 +666,7 @@ class CheckBatchCost:
             except Exception:
                 pass
 
-        deployment_info = self.llm_router.registry().deployment(model_id)
+        deployment_info = self.llm_router.get_deployment(model_id=model_id)
         if deployment_info is None:
             verbose_proxy_logger.info(
                 f"Skipping job {job.unified_object_id} because it is not a valid deployment info"
@@ -837,7 +718,8 @@ class CheckBatchCost:
                         )
                     except Exception as _e:
                         verbose_proxy_logger.warning(
-                            f"CheckBatchCost: failed to create managed file ID for {_file_attr}={_raw_file_id!r}: {_e}"
+                            f"CheckBatchCost: failed to create managed file ID for "
+                            f"{_file_attr}={_raw_file_id!r}: {_e}"
                         )
 
         # Pass the deployment's router-registered pricing (litellm_params custom
@@ -856,17 +738,14 @@ class CheckBatchCost:
             custom_llm_provider=batch_file_provider,
             model_name=model_name,
             model_info=deployment_model_info,
-            vertex_location=resolve_vertex_location_for_cost(
-                custom_llm_provider=llm_provider,
-                litellm_params=deployment_info.litellm_params.model_dump(exclude_none=True),
-                optional_params=None,
-                model=model_name,
-            ),
         )
         error_file_failed_requests: Final = await count_error_file_failed_requests(
             response,
             custom_llm_provider=batch_file_provider,
-            litellm_params=credentials,
+            litellm_params={
+                **credentials,
+                "_litellm_internal_model_credentials": MappingProxyType(dict(credentials)),
+            },
         )
         batch_result: Final = (
             output_file_result
@@ -876,9 +755,8 @@ class CheckBatchCost:
                 failed_requests=output_file_result.failed_requests + error_file_failed_requests,
             )
         )
-        logging_model: Final = batch_result.models[0] if batch_result.models else litellm_model_name or model_id
         logging_obj = LiteLLMLogging(
-            model=logging_model,
+            model=batch_result.models[0],
             messages=[{"role": "user", "content": "<retrieve_batch>"}],
             stream=False,
             call_type="aretrieve_batch",
@@ -909,35 +787,27 @@ class CheckBatchCost:
             custom_llm_provider=str(llm_provider) if llm_provider else None,
         )
 
-        original_status: Final = job.status
+        if not await self._claim_job_for_costing(job):
-        original_file_object: Final = job.file_object
-        if not await self._claim_job_for_costing(job, response.model_copy(update={"usage": batch_result.usage})):
             verbose_proxy_logger.info(
                 f"CheckBatchCost: batch {batch_id} (job {job.id}) was claimed by another pod "
                 "in this window, so its cost is already being tracked there"
             )
             return None
 
-        from litellm.proxy.hooks.proxy_track_cost_callback import track_batch_accounting
+        try:
-
+            await logging_obj.async_success_handler(
-        with track_batch_accounting(logging_obj.litellm_call_id) as accounting:
+                result=response,
-            try:
+                batch_cost=batch_result.cost,
-                await logging_obj.async_success_handler(
+                batch_usage=batch_result.usage,
-                    result=response,
+                batch_models=batch_result.models,
-                    batch_cost=batch_result.cost,
+                batch_successful_requests=batch_result.successful_requests,
-                    batch_usage=batch_result.usage,
+                batch_failed_requests=batch_result.failed_requests,
-                    batch_models=batch_result.models,
+                batch_prompt_cost=batch_result.prompt_cost,
-                    batch_successful_requests=batch_result.successful_requests,
+                batch_completion_cost=batch_result.completion_cost,
-                    batch_failed_requests=batch_result.failed_requests,
+            )
-                    batch_prompt_cost=batch_result.prompt_cost,
+        except Exception:
-                    batch_completion_cost=batch_result.completion_cost,
+            await self._release_job_claim(job)
-                )
+            raise
-            finally:
-                if accounting.retryable_failure:
-                    await self._release_job_claim(job, original_status, original_file_object)
-
-        if accounting.retryable_failure:
-            return None
 
         # Record batch duration (completed_at - created_at)
         if prom_logger and response.completed_at and response.created_at:
@@ -1006,13 +876,10 @@ class CheckBatchCost:
         )
         from litellm.batches.batch_utils import fetch_batch_managed_file_content
         from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
-        from litellm.litellm_core_utils.litellm_logging import (
+        from litellm.litellm_core_utils.litellm_logging import deployment_pricing_model_info
-            deployment_pricing_model_info,
-            resolve_vertex_location_for_cost,
-        )
 
         try:
-            deployment_info: Final = self.llm_router.registry().deployment(model_id)
+            deployment_info: Final = self.llm_router.get_deployment(model_id=model_id)
             if deployment_info is None:
                 verbose_proxy_logger.error(
                     f"CheckBatchCost: deployment {model_id} is no longer in the router, so the line items of batch "
@@ -1025,7 +892,11 @@ class CheckBatchCost:
                 custom_llm_provider=deployment_info.litellm_params.custom_llm_provider,
             )
             cost_provider: Final = cast(BatchFileProvider, llm_provider)
-            file_access: Final = self.llm_router.registry().credentials(model_id, form="with_provider") or {}
+            credentials: Final = self.llm_router.get_deployment_credentials_with_provider(model_id) or {}
+            file_access: Final = {
+                **credentials,
+                "_litellm_internal_model_credentials": MappingProxyType(dict(credentials)),
+            }
             output_content: Final = (
                 await fetch_batch_managed_file_content(
                     response.output_file_id, custom_llm_provider=cost_provider, litellm_params=file_access
@@ -1049,12 +920,6 @@ class CheckBatchCost:
                 cost_provider=cost_provider,
                 model_info=deployment_pricing_model_info(model_id=model_id, deployment_model=litellm_model_name),
                 metadata=line_item_metadata(await self._creator_attribution.batch_metadata(job, batch_id)),
-                vertex_location=resolve_vertex_location_for_cost(
-                    custom_llm_provider=llm_provider,
-                    litellm_params=deployment_info.litellm_params.model_dump(exclude_none=True),
-                    optional_params=None,
-                    model=model_name,
-                ),
             )
             input_content: Final = (
                 b""
@@ -1106,8 +971,8 @@ class CheckBatchCost:
         from litellm.proxy.proxy_server import proxy_config
         from litellm.secret_managers.main import str_to_bool
 
-        api_key: Final = job.api_key if hasattr(job, "api_key") else None
+        api_key: Final = getattr(job, "api_key", None)
-        team_id: Final = job.team_id if hasattr(job, "team_id") else None
+        team_id: Final = getattr(job, "team_id", None)
         has_key_row: Final = api_key is not None and is_valid_sha256_hash(api_key)
         try:
             key_row: Final = (
@@ -1125,9 +990,7 @@ class CheckBatchCost:
         if (has_key_row and key_row is None) or (team_id and team_row is None):
             return True
         settings: Final = get_dynamic_logging_metadata(
-            UserAPIKeyAuth(
+            UserAPIKeyAuth(metadata=_row_metadata(key_row), team_metadata=_row_metadata(team_row), team_id=team_id),
-                metadata=dict(row_metadata(key_row)), team_metadata=dict(row_metadata(team_row)), team_id=team_id
-            ),
             proxy_config,
         )
         callback_vars: Final = (settings.callback_vars if settings is not None else None) or {}
@@ -1152,7 +1015,6 @@ class CheckBatchCost:
         """
         try:
             from litellm.integrations.prometheus import PrometheusLogger
-
             prom_logger = PrometheusLogger.get_instance()
         except Exception as e:
             verbose_proxy_logger.error(f"CheckBatchCost: could not get Prometheus logger: {e}")
@@ -1160,6 +1022,13 @@ class CheckBatchCost:
 
         processed_models: List[Tuple[Optional[str], Optional[str]]] = []
 
+        try:
+            await self._cleanup_stale_managed_objects()
+        except Exception as cleanup_err:
+            verbose_proxy_logger.warning(
+                f"CheckBatchCost: stale cleanup failed (poll will continue): {cleanup_err}"
+            )
+
         # Look for all batches that have not yet been processed by CheckBatchCost.
         # self._has_batch_processed_column is cached after the first probe so that
         # older schemas don't pay a guaranteed-failing primary query + warning on
@@ -1170,7 +1039,7 @@ class CheckBatchCost:
                 # endpoint may transition a batch to "complete" before
                 # CheckBatchCost runs.  The batch_processed=False filter
                 # already prevents reprocessing finished batches.
-                jobs = await self._find_jobs(
+                jobs = await _managed_object_table(self.prisma_client).find_many(
                     where={
                         "file_purpose": "batch",
                         "batch_processed": False,
@@ -1183,6 +1052,8 @@ class CheckBatchCost:
                             ]
                         },
                     },
+                    take=MAX_OBJECTS_PER_POLL_CYCLE,
+                    order={"created_at": "asc"},
                 )
                 self.batch_processed_support_confirmed = True
             except Exception as query_err:
@@ -1201,8 +1072,6 @@ class CheckBatchCost:
             if routing is None:
                 if self._has_unified_id_without_model(job):
                     await self._retire_job(job, "unified object id has no model id")
-                else:
-                    await self._expire_if_old_and_unfinished(job)
                 continue
             model_id, batch_id = routing
 
@@ -1227,18 +1096,13 @@ class CheckBatchCost:
                     prom_logger.record_check_batch_cost_error("provider_retrieval_error")
                 if self._is_batch_gone_at_provider(e, batch_id) and self._batch_deployment_exists(model_id):
                     await self._retire_job(job, f"batch {batch_id} no longer exists at the provider")
-                else:
-                    await self._expire_if_old_and_unfinished(job)
-                continue
-            if self._is_completed_with_output_id_pending(response):
-                verbose_proxy_logger.info(
-                    f"CheckBatchCost: batch {batch_id} is completed but its output file id "
-                    f"has not appeared yet; leaving job {job.id} for the next poll cycle"
-                )
                 continue
 
             ## RETRIEVE THE BATCH JOB OUTPUT FILE
-            if response.status in PROVIDER_TERMINAL_BATCH_STATUSES and response.output_file_id is not None:
+            if (
+                response.status in PROVIDER_TERMINAL_BATCH_STATUSES
+                and response.output_file_id is not None
+            ):
                 try:
                     tracked = await self._track_completed_batch_cost(
                         job=job,
@@ -1259,7 +1123,7 @@ class CheckBatchCost:
                         continue
                     verbose_proxy_logger.error(
                         f"CheckBatchCost: failed to track cost for batch {batch_id} "
-                        f"(job {job.id}): {tracking_err}"
+                        f"(job {job.id}); leaving it unprocessed so the next poll retries: {tracking_err}"
                     )
                     self._record_error(prom_logger, "cost_tracking_error")
                     continue
@@ -1269,26 +1133,39 @@ class CheckBatchCost:
                 # Track this job for the final metrics summary
                 processed_models.append(tracked)
 
-                if not self._has_batch_processed_column:
+                # mark the job as complete
-                    try:
+                try:
-                        await _managed_object_table(self.prisma_client).update(
+                    update_data: dict = {
-                            where={"id": job.id},
+                        "status": response.status if response.status != "completed" else "complete",
-                            data={
+                        "file_object": response.model_dump_json(),
-                                "status": response.status if response.status != "completed" else "complete",
+                    }
-                                "file_object": response.model_dump_json(),
+                    if self._has_batch_processed_column:
-                            },
+                        update_data["batch_processed"] = True
-                        )
+                    await _managed_object_table(self.prisma_client).update(
-                    except Exception as db_err:
+                        where={"id": job.id},
-                        verbose_proxy_logger.error(
+                        data=update_data,
-                            f"CheckBatchCost: failed to mark job {job.id} complete in DB: {db_err}"
+                    )
-                        )
+                except Exception as db_err:
+                    verbose_proxy_logger.error(
+                        f"CheckBatchCost: failed to mark job {job.id} complete in DB: {db_err}"
+                    )
+                    if not self._has_batch_processed_column:
+                        # nothing claimed the row, so it stays in the poll page and its lines wait for that retry
                         continue
                 self._store_batch_line_items_in_background(job, response, model_id, batch_id, billed=True)
 
             elif response.status in PROVIDER_TERMINAL_BATCH_STATUSES:
+                from litellm.proxy.openai_files_endpoints.common_utils import (
+                    _completed_batch_safe_to_retire,
+                )
+
+                if response.status in ("completed", "complete") and not _completed_batch_safe_to_retire(response):
+                    verbose_proxy_logger.info(
+                        f"CheckBatchCost: batch {batch_id} is completed but its output file id "
+                        f"has not appeared yet; leaving job {job.id} for the next poll cycle"
+                    )
+                    continue
                 await self._retire_unbilled_terminal_job(job, response, model_id, batch_id)
-            else:
-                await self._expire_if_old_and_unfinished(job)
 
         # Record polling run metrics (always, even if nothing was processed)
         if prom_logger:

enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py

@@ -7,12 +7,6 @@ attribution, priced against the deployment that served it. The poll's status che
 with BACKGROUND_RESPONSE_STATUS_POLL_CALL_ORIGIN and client reads are marked managed by the
 managed files hook, so both stay free. The spend event carries the id the client holds, the row's
 unified_object_id, so its spend log is found under that id with the _background_cost suffix.
-The spend event reaches the creating key's or team's own logging callbacks the way that creator's
-requests do. Its message content is redacted when the creator's key or team logging turns message
-logging off, else as the proxy is set, the way the batch poll decides it. The row is claimed before
-the creator is read, so reads that lose the race skip the lookups. A creator whose lookup fails, or
-whose logging settings the logger rejects, is still billed, through the proxy's own callbacks only
-and with its message content redacted.
 A read that finds no row, because the create could not store one, stores it under the reading key
 and bills it the same way, so a response the create left untracked is still billed once. A streamed
 background create is free on its own; its stream stores the row under the creating key and bills
@@ -30,12 +24,11 @@ from collections.abc import Awaitable, Callable, Mapping
 from dataclasses import dataclass
 from datetime import datetime, timedelta, timezone
 from types import MappingProxyType
-from typing import TYPE_CHECKING, Final
+from typing import TYPE_CHECKING, Final, cast
 
 from litellm_enterprise.proxy.common_utils.managed_object_attribution import (
     ManagedObjectCreatorAttribution,
     ManagedObjectRow,
-    ResponseCreator,
 )
 from pydantic import BaseModel, ConfigDict, TypeAdapter
 
@@ -61,29 +54,12 @@ from litellm.types.utils import (
 
 if TYPE_CHECKING:
     from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-    from litellm.proxy.proxy_server import ProxyConfig
     from litellm.proxy.utils import PrismaClient, ProxyLogging
     from litellm.repositories.prisma_protocols import TableActions
     from litellm.router import Router
 
 _STORED_RESPONSE_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
-_CALLBACK_LIST_ADAPTER: Final[TypeAdapter[tuple[object, ...]]] = TypeAdapter(tuple[object, ...])
-
-
-class _RoutedModelInfo(BaseModel):
-    model_config = ConfigDict(frozen=True)
-    id: str | None = None
-
-
-class _RoutedDeployment(BaseModel):
-    model_config = ConfigDict(frozen=True)
-    model_info: _RoutedModelInfo = _RoutedModelInfo()
-
-
-_ROUTED_DEPLOYMENTS: Final = TypeAdapter(tuple[_RoutedDeployment, ...])
 _RETRIEVE_RESPONSE_CALL_TYPE: Final = "aget_responses"
-_CALLBACK_LIST_FIELDS: Final = frozenset({"success_callback", "failure_callback", "litellm_disabled_callbacks"})
-_REJECTED_CREATOR_LOGGING: Final[Mapping[str, object]] = MappingProxyType({"turn_off_message_logging": True})
 
 
 @dataclass(frozen=True, slots=True)
@@ -100,7 +76,6 @@ class BilledBackgroundResponse:
     custom_llm_provider: str | None
     litellm_params: Mapping[str, object]
     metadata: Mapping[str, object]
-    creator_logging: Mapping[str, object]
 
 
 @dataclass(frozen=True, slots=True)
@@ -119,51 +94,11 @@ def _without_read_price(response: ResponsesAPIResponse) -> ResponsesAPIResponse:
     return unpriced
 
 
-def _string_callbacks(callbacks: object) -> tuple[str, ...]:
+async def log_billed_background_response(billed: BilledBackgroundResponse) -> BilledCharge:
-    if not isinstance(callbacks, list):
-        return ()
-    return tuple(
-        callback for callback in _CALLBACK_LIST_ADAPTER.validate_python(callbacks) if isinstance(callback, str)
-    )
-
-
-def _running_proxy_config() -> "ProxyConfig":
-    from litellm.proxy.proxy_server import proxy_config
-
-    return proxy_config
-
-
-def creator_logging_fields(creator: ResponseCreator, proxy_config: "ProxyConfig | None") -> Mapping[str, object]:
-    from litellm.proxy._types import UserAPIKeyAuth
-    from litellm.proxy.litellm_pre_call_utils import key_and_team_logging_fields
-
-    if not creator.settings_read:
-        return _REJECTED_CREATOR_LOGGING
-    try:
-        return key_and_team_logging_fields(
-            user_api_key_dict=UserAPIKeyAuth(
-                metadata=dict(creator.key_metadata),
-                team_id=creator.team_id,
-                team_metadata=dict(creator.team_metadata),
-            ),
-            proxy_config=proxy_config if proxy_config is not None else _running_proxy_config(),
-        )
-    except Exception as e:
-        verbose_proxy_logger.error(
-            f"CheckResponsesCost: could not read the creator's logging settings, so only the proxy's "
-            f"callbacks get the billed event: {e}"
-        )
-        return _REJECTED_CREATOR_LOGGING
-
-
-def _billed_logging_obj(billed: BilledBackgroundResponse, creator_logging: Mapping[str, object]) -> "LiteLLMLogging":
     from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-    from litellm.utils import split_dynamic_success_callbacks
+    from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
 
-    sync_callbacks, async_callbacks = split_dynamic_success_callbacks(
+    logging_obj: Final = LiteLLMLogging(
-        _string_callbacks(creator_logging.get("success_callback"))
-    )
-    return LiteLLMLogging(
         model=billed.model,
         messages=[{"role": "user", "content": "<retrieve_response>"}],
         stream=False,
@@ -171,27 +106,7 @@ def _billed_logging_obj(billed: BilledBackgroundResponse, creator_logging: Mappi
         start_time=datetime.now(),
         litellm_call_id=str(uuid.uuid4()),
         function_id=str(uuid.uuid4()),
-        dynamic_success_callbacks=list(sync_callbacks),
-        dynamic_async_success_callbacks=list(async_callbacks) or None,
-        kwargs={key: value for key, value in creator_logging.items() if key not in _CALLBACK_LIST_FIELDS},
     )
-
-
-def _creator_or_proxy_logging_obj(billed: BilledBackgroundResponse) -> "LiteLLMLogging":
-    try:
-        return _billed_logging_obj(billed, billed.creator_logging)
-    except Exception as e:
-        verbose_proxy_logger.error(
-            f"CheckResponsesCost: the creator's logging settings were rejected, so only the proxy's "
-            f"callbacks get the billed event: {e}"
-        )
-        return _billed_logging_obj(billed, _REJECTED_CREATOR_LOGGING)
-
-
-async def log_billed_background_response(billed: BilledBackgroundResponse) -> BilledCharge:
-    from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
-
-    logging_obj: Final = _creator_or_proxy_logging_obj(billed)
     logging_obj.update_environment_variables(
         litellm_params={**billed.litellm_params, "metadata": dict(billed.metadata)},
         optional_params={},
@@ -259,7 +174,6 @@ class CheckResponsesCost:
         prisma_client: "PrismaClient",
         llm_router: "Router",
         log_billed_response: BilledResponseLogger = log_billed_background_response,
-        proxy_config: "ProxyConfig | None" = None,
     ):
         from litellm.proxy.utils import PrismaClient, ProxyLogging
         from litellm.router import Router
@@ -269,11 +183,10 @@ class CheckResponsesCost:
         self.llm_router: Router = llm_router
         self._creator_attribution: Final = ManagedObjectCreatorAttribution(prisma_client, "CheckResponsesCost")
         self._log_billed_response: Final = log_billed_response
-        self._proxy_config: Final = proxy_config
 
     def _resolve_deployment(self, response_id: str) -> Deployment | None:
         model_id: str | None = ResponsesAPIRequestUtils.get_model_id_from_response_id(response_id)
-        return self.llm_router.registry().deployment(model_id) if model_id is not None else None
+        return self.llm_router.get_deployment(model_id=model_id) if model_id is not None else None
 
     async def _get_response(
         self,
@@ -290,14 +203,12 @@ class CheckResponsesCost:
         sees provider env vars, so it fails for every deployment whose credentials
         live in the config; the row then never leaves ``queued``.
         """
-        retrieved: Final[object] = (
+        if not via_router:
-            await self.llm_router.aget_responses(response_id=response_id, litellm_metadata=litellm_metadata)
+            return await litellm.aget_responses(response_id=response_id, litellm_metadata=litellm_metadata)
-            if via_router
+        router_response = await self.llm_router.aget_responses(
-            else await litellm.aget_responses(response_id=response_id, litellm_metadata=litellm_metadata)
+            response_id=response_id, litellm_metadata=litellm_metadata
         )
-        if not isinstance(retrieved, ResponsesAPIResponse):
+        return cast(ResponsesAPIResponse, router_response)
-            raise TypeError(f"Retrieving {response_id} did not return a response object")
-        return retrieved
 
     async def _fetch_unbilled(self, target: _PollTarget) -> ResponsesAPIResponse:
         return await self._get_response(
@@ -383,12 +294,11 @@ class CheckResponsesCost:
             data={"status": job.status or PENDING_RESPONSE_STATUSES[0]},
         )
 
-    def _billed_response(
+    async def _billed_response(
         self,
         job: ManagedObjectRow,
         target: _PollTarget,
         response: ResponsesAPIResponse,
-        creator: ResponseCreator,
     ) -> BilledBackgroundResponse:
         deployment: Final = target.deployment
         return BilledBackgroundResponse(
@@ -400,26 +310,12 @@ class CheckResponsesCost:
             litellm_params=_deployment_litellm_params(deployment) if deployment is not None else MappingProxyType({}),
             metadata=MappingProxyType(
                 {
-                    **creator.metadata,
+                    **(await self._creator_attribution.response_metadata(job, job.unified_object_id)),
                     INTERNAL_CALL_ORIGIN_METADATA_KEY: BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN,
                     **target.model_metadata,
                     **(_deployment_metadata(deployment) if deployment is not None else {}),
-                    **self._requested_model_group(target, job.team_id),
                 }
             ),
-            creator_logging=creator_logging_fields(creator, self._proxy_config),
-        )
-
-    def _requested_model_group(self, target: _PollTarget, team_id: str | None) -> Mapping[str, object]:
-        requested: Final = target.model_metadata.get("model_group")
-        if target.deployment is None or not isinstance(requested, str):
-            return MappingProxyType({})
-        routed: Final = _ROUTED_DEPLOYMENTS.validate_python(
-            self.llm_router.registry().deployments(requested, team_id=team_id) or ()
-        )
-        serving_id: Final = target.deployment.model_info.id
-        return MappingProxyType(
-            {"model_group": requested} if any(row.model_info.id == serving_id for row in routed) else {}
         )
 
     async def _bill_once(
@@ -432,8 +328,7 @@ class CheckResponsesCost:
             verbose_proxy_logger.info(f"Response {job.unified_object_id} was already billed elsewhere, skipping")
             return None
         try:
-            creator: Final = await self._creator_attribution.response_creator(job, job.unified_object_id)
+            charge: Final = await self._log_billed_response(await self._billed_response(job, target, response))
-            charge: Final = await self._log_billed_response(self._billed_response(job, target, response, creator))
         except BaseException:
             await self._release_billing_claim(job)
             raise

enterprise/litellm_enterprise/proxy/common_utils/managed_object_attribution.py

@@ -9,17 +9,12 @@ from __future__ import annotations
 
 from collections.abc import Mapping
 from dataclasses import dataclass
-from datetime import datetime
+from typing import TYPE_CHECKING, Final, Protocol, cast
-from typing import TYPE_CHECKING, Final, Protocol, TypeGuard, cast
 
 from pydantic import TypeAdapter
 
 from litellm._logging import verbose_proxy_logger
 from litellm.constants import CLI_SESSION_KEY_PREFIX
-from litellm.repositories.table_repositories import (
-    DeletedVerificationTokenRepository,
-    DeprecatedVerificationTokenRepository,
-)
 from litellm.repositories.team_repository import TeamRepository
 from litellm.repositories.user_repository import UserRepository
 from litellm.repositories.verification_token_repository import VerificationTokenRepository
@@ -32,12 +27,6 @@ if TYPE_CHECKING:
 
 
 class ManagedObjectRow(Protocol):
-    @property
-    def created_at(self) -> datetime: ...
-
-    @property
-    def updated_at(self) -> datetime: ...
-
     @property
     def id(self) -> str: ...
 
@@ -74,7 +63,6 @@ class ManagedObjectRow(Protocol):
 
 _TAG_LIST_ADAPTER: Final[TypeAdapter[list[object]]] = TypeAdapter(list[object])
 _KEY_OWNER_SCOPE_PREFIX: Final = "key:"
-_MAX_ROTATION_HOPS: Final = 16
 
 
 def _string_tags(request_tags: object) -> list[str] | None:
@@ -101,20 +89,6 @@ def _team_table(prisma_client: PrismaClient) -> TableActions[prisma_models.LiteL
     return TeamRepository(prisma_client).table
 
 
-def _rotated_token_table(
-    prisma_client: PrismaClient,
-) -> TableActions[prisma_models.LiteLLM_DeprecatedVerificationToken]:
-    return DeprecatedVerificationTokenRepository(prisma_client).table
-
-
-def _deleted_token_table(prisma_client: PrismaClient) -> TableActions[prisma_models.LiteLLM_DeletedVerificationToken]:
-    return DeletedVerificationTokenRepository(prisma_client).table
-
-
-def _deleted_team_table(prisma_client: PrismaClient) -> TableActions[prisma_models.LiteLLM_DeletedTeamTable]:
-    return TeamRepository(prisma_client).deleted_table
-
-
 @dataclass(frozen=True, slots=True)
 class _CreatorRows:
     key: prisma_models.LiteLLM_VerificationToken | None
@@ -122,15 +96,6 @@ class _CreatorRows:
     user: prisma_models.LiteLLM_UserTable | None
 
 
-@dataclass(frozen=True, slots=True)
-class ResponseCreator:
-    metadata: Mapping[str, object]
-    key_metadata: Mapping[str, object]
-    team_id: str | None
-    team_metadata: Mapping[str, object]
-    settings_read: bool = True
-
-
 class ManagedObjectCreatorAttribution:
     """Rebuilds the spend-tracking metadata of whoever created a managed object, so the poll
     job's spend row is attributed like the creating request: key hash, team, organization,
@@ -142,15 +107,8 @@ class ManagedObjectCreatorAttribution:
     user's alias, because a resolvable name is more useful on the spend row than a null.
     user_api_key_org_id is resolved too: the spend update writer reads it off this metadata
     to increment organization spend, so leaving it out drops the cost from org accounting.
-    A background response also carries its end user, the creating key's project and its alias,
+    A background response also carries its end user and the key, team, and user budgets, the
-    and the key, team, and user budgets and per-model budgets, the way the read that billed it
+    way the read that billed it before did, and never names a key-scoped creator as its user.
-    before did, and never names a key-scoped creator as its user. A key that sets no budget of
-    its own takes its budget tier's, the same rule auth applies. Its creator's key and team
-    metadata come along too, so the billed event reaches the key's and team's logging callbacks
-    like that read did. A rotated key's metadata is read from the live key its rotations lead
-    to, and a deleted key's or team's from its deleted row. A lookup that fails leaves the
-    creator's settings unread and the attribution to the rows that still load, so the
-    response still bills and still counts toward their budgets.
     """
 
     def __init__(self, prisma_client: PrismaClient, poller_name: str) -> None:
@@ -161,70 +119,17 @@ class ManagedObjectCreatorAttribution:
         rows: Final = await self._rows(job, object_id)
         return {"user_api_key_user_id": job.created_by, **_shared_metadata(job, rows)}
 
-    async def response_creator(self, job: ManagedObjectRow, object_id: str) -> ResponseCreator:
+    async def response_metadata(self, job: ManagedObjectRow, object_id: str) -> Mapping[str, object]:
-        try:
+        rows: Final = await self._rows(job, object_id)
-            return await self._read_response_creator(job, object_id)
+        end_user_id: Final = getattr(job, "end_user_id", None)
-        except Exception as e:
+        return {
-            verbose_proxy_logger.error(
+            "user_api_key_user_id": _creator_user_id(job.created_by),
-                f"{self._poller_name}: could not read the creator of {object_id}, so it bills without their "
+            **_shared_metadata(job, rows),
-                f"settings: {e}"
+            **({"user_api_key_end_user_id": end_user_id} if end_user_id else {}),
-            )
+            **_budget_fields(rows.key, "user_api_key_spend", "user_api_key_max_budget"),
-            return ResponseCreator(
+            **_budget_fields(rows.team, "user_api_key_team_spend", "user_api_key_team_max_budget"),
-                metadata=_response_creator_metadata(job, await self._rows(job, object_id)),
+            **_budget_fields(rows.user, "user_api_key_user_spend", "user_api_key_user_max_budget"),
-                key_metadata={},
+        }
-                team_id=getattr(job, "team_id", None),
-                team_metadata={},
-                settings_read=False,
-            )
-
-    async def _read_response_creator(self, job: ManagedObjectRow, object_id: str) -> ResponseCreator:
-        api_key: Final = getattr(job, "api_key", None)
-        team_id: Final = getattr(job, "team_id", None)
-        rows: Final = _CreatorRows(
-            key=await self._live_key_row(api_key, job.created_by),
-            team=await self._live_team_row(team_id, job.created_at),
-            user=await self._user_row(object_id, _creator_user_id(job.created_by)),
-        )
-        return ResponseCreator(
-            metadata=_response_creator_metadata(job, rows),
-            key_metadata=row_metadata(rows.key) if rows.key is not None else await self._retired_key_metadata(api_key),
-            team_id=team_id,
-            team_metadata=(
-                row_metadata(rows.team) if rows.team is not None else await self._deleted_team_metadata(team_id)
-            ),
-        )
-
-    async def _retired_key_metadata(self, api_key: str | None) -> Mapping[str, object]:
-        if not _names_a_key_row(api_key):
-            return {}
-        current_token: Final = await self._token_rotated_to(api_key, _MAX_ROTATION_HOPS)
-        current_key: Final = (
-            await _token_table(self._prisma_client).find_unique(where={"token": current_token})
-            if current_token != api_key
-            else None
-        )
-        if current_key is not None:
-            return row_metadata(current_key)
-        return row_metadata(
-            await _deleted_token_table(self._prisma_client).find_first(
-                where={"token": current_token}, order={"deleted_at": "desc"}
-            )
-        )
-
-    async def _token_rotated_to(self, token: str, hops_left: int) -> str:
-        rotation: Final = await _rotated_token_table(self._prisma_client).find_unique(where={"token": token})
-        if rotation is None or hops_left == 0:
-            return token
-        return await self._token_rotated_to(rotation.active_token_id, hops_left - 1)
-
-    async def _deleted_team_metadata(self, team_id: str | None) -> Mapping[str, object]:
-        if not team_id:
-            return {}
-        return row_metadata(
-            await _deleted_team_table(self._prisma_client).find_first(
-                where={"team_id": team_id}, order={"deleted_at": "desc"}
-            )
-        )
 
     async def _rows(self, job: ManagedObjectRow, object_id: str) -> _CreatorRows:
         return _CreatorRows(
@@ -247,62 +152,23 @@ class ManagedObjectCreatorAttribution:
     async def _key_row(
         self, object_id: str, api_key: str | None, created_by: str | None
     ) -> prisma_models.LiteLLM_VerificationToken | None:
+        if not api_key or _is_cli_session_key(api_key, created_by):
+            return None
         try:
-            return await self._live_key_row(api_key, created_by)
+            return await _token_table(self._prisma_client).find_unique(where={"token": api_key})
         except Exception as e:
             verbose_proxy_logger.error(f"{self._poller_name}: could not look up the creating key for {object_id}: {e}")
             return None
 
-    async def _live_key_row(
-        self, api_key: str | None, created_by: str | None
-    ) -> prisma_models.LiteLLM_VerificationToken | None:
-        if not api_key or _is_cli_session_key(api_key, created_by):
-            return None
-        return await _token_table(self._prisma_client).find_unique(
-            where={"token": api_key}, include={"litellm_budget_table": True, "litellm_project_table": True}
-        )
-
     async def _team_row(self, object_id: str, team_id: str | None) -> prisma_models.LiteLLM_TeamTable | None:
+        if not team_id:
+            return None
         try:
-            return await self._live_team_row(team_id)
+            return await _team_table(self._prisma_client).find_unique(where={"team_id": team_id})
         except Exception as e:
             verbose_proxy_logger.error(f"{self._poller_name}: could not look up team {team_id} for {object_id}: {e}")
             return None
 
-    async def _live_team_row(
-        self, team_id: str | None, created_before: datetime | None = None
-    ) -> prisma_models.LiteLLM_TeamTable | None:
-        if not team_id:
-            return None
-        team: Final = await _team_table(self._prisma_client).find_unique(where={"team_id": team_id})
-        if team is not None and created_before is not None and team.created_at > created_before:
-            return None
-        return team
-
-
-def _response_creator_metadata(job: ManagedObjectRow, rows: _CreatorRows) -> Mapping[str, object]:
-    end_user_id: Final = getattr(job, "end_user_id", None)
-    return {
-        "user_api_key_user_id": _creator_user_id(job.created_by),
-        **_shared_metadata(job, rows),
-        **({"user_api_key_end_user_id": end_user_id} if end_user_id else {}),
-        "user_api_key_project_id": getattr(rows.key, "project_id", None),
-        "user_api_key_project_alias": getattr(getattr(rows.key, "litellm_project_table", None), "project_alias", None),
-        **_key_budget_fields(rows.key),
-        **_budget_fields(
-            rows.team,
-            "user_api_key_team_spend",
-            "user_api_key_team_max_budget",
-            "user_api_key_team_model_max_budget",
-        ),
-        **_budget_fields(
-            rows.user,
-            "user_api_key_user_spend",
-            "user_api_key_user_max_budget",
-            "user_api_key_user_model_max_budget",
-        ),
-    }
-
 
 def _shared_metadata(job: ManagedObjectRow, rows: _CreatorRows) -> Mapping[str, object]:
     api_key: Final = getattr(job, "api_key", None)
@@ -330,19 +196,6 @@ def _shared_metadata(job: ManagedObjectRow, rows: _CreatorRows) -> Mapping[str,
     }
 
 
-def row_metadata(row: object) -> Mapping[str, object]:
-    metadata: Final = getattr(row, "metadata", None)
-    if not isinstance(metadata, dict):
-        return {}
-    return cast(Mapping[str, object], metadata)  # cast-ok: prisma Json object column
-
-
-def _names_a_key_row(api_key: object) -> TypeGuard[str]:
-    from litellm.litellm_core_utils.litellm_logging import is_valid_sha256_hash
-
-    return isinstance(api_key, str) and is_valid_sha256_hash(api_key)
-
-
 def _is_cli_session_key(api_key: str, created_by: str | None) -> bool:
     return bool(created_by) and api_key == f"{CLI_SESSION_KEY_PREFIX}-{created_by}"
 
@@ -359,28 +212,7 @@ def _organization_id(row: object) -> str | None:
     return cast("str | None", getattr(row, "organization_id", None)) if row is not None else None
 
 
-def _budget_fields(row: object, spend_key: str, max_budget_key: str, model_max_budget_key: str) -> Mapping[str, object]:
+def _budget_fields(row: object, spend_key: str, max_budget_key: str) -> Mapping[str, object]:
     if row is None:
         return {}
-    return {
+    return {spend_key: getattr(row, "spend", None), max_budget_key: getattr(row, "max_budget", None)}
-        spend_key: getattr(row, "spend", None),
-        max_budget_key: getattr(row, "max_budget", None),
-        model_max_budget_key: getattr(row, "model_max_budget", None),
-    }
-
-
-def _key_budget_fields(key: object) -> Mapping[str, object]:
-    if key is None:
-        return {}
-    tier: Final = getattr(key, "litellm_budget_table", None)
-    return {
-        "user_api_key_spend": getattr(key, "spend", None),
-        "user_api_key_max_budget": _own_or_tier_budget(key, tier, "max_budget"),
-        "user_api_key_model_max_budget": _own_or_tier_budget(key, tier, "model_max_budget"),
-    }
-
-
-def _own_or_tier_budget(key: object, tier: object, field: str) -> object:
-    own: Final = getattr(key, field, None)
-    tier_value: Final = getattr(tier, field, None)
-    return own if tier_value is None or own not in (None, {}) else tier_value