chore: resolve Lens removal conflicts with main

.agents/skills/database-access/SKILL.md

@@ -1,73 +0,0 @@
----
-name: database-access
-description: Organize database repositories and proxy dependency wiring, manage connection ownership, or port Prisma operations to Rust and SQLx without expanding migration scope
-metadata:
-  internal: true
----
-
-# Database access
-
-## Situations
-
-- Adding a query or an operation to a repository, read [adding a query](references/python-repository.md#adding-a-query)
-- Moving existing Prisma calls behind a repository, read [moving existing access](references/python-repository.md#moving-existing-access-into-a-repository)
-- Porting a repository from Prisma to Rust and SQLx, read [porting to SQLx](references/python-repository.md#porting-a-repository-to-sqlx)
-- Injecting a repository into a route, read [Python and FastAPI](references/python-fastapi.md) or [Rust and Axum](references/rust-axum.md)
-
-## Adding new database access
-
-- Put Python database access behind an operation in `litellm/repositories`
-- Give each repository its own FastAPI provider, so a route builds only the repositories it uses
-- A provider borrows existing handles and performs no database I/O during construction
-- Define the operations a route calls as a Protocol in `litellm/models/<feature>.py`, next to its domain types
-  - The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
-  - Parsing and serialization of stored values live in the repository
-  - Add a Protocol when a repository gets a provider, not up front for every repository
-- Keep query execution inside the repository, with one implementation per repository and no backend switch or fallback
-- A repository is either all Prisma or all native, never a mix of both
-- Prisma is fine for the implementation, adding a query does not require a Rust port
-- Native Rust callers use `litellm-rust/crates/db` directly
-
-## Framework wiring
-
-- Python routes receive a per-repository dependency such as `UserBannerRepositoryDep` and call its operations, such as `banners.get_banner()`
-- `DatabaseDep` lives in `litellm/proxy/deps/database.py`, repository providers in `litellm/proxy/deps/repositories.py`
-  - A provider returns the feature's Protocol, so routes never see which backend implements it
-  - Dependencies used by one feature only, like the banner audit binding, stay next to that feature's routes
-- Repositories never import FastAPI, so jobs and helpers can use them without a request
-- Use named FastAPI providers with `Annotated[..., Depends(...)]`, ordinary helpers and jobs receive explicit arguments
-- Preserve each route's no-database behavior, authorization precedence and background-task scheduling
-- Read [Python and FastAPI](references/python-fastapi.md) for provider examples, lifespan state and route verification
-- Read [Rust and Axum](references/rust-axum.md) when adding database access to Rust handlers
-
-## Connection ownership
-
-- Keep connection setup and shutdown separate from repositories, which own no pools
-- Reuse existing handles, never connect per request or disconnect from a repository operation
-- Give each pool one owner for setup and shutdown, borrowed handles never close it
-- Finish requests, jobs and queued database work before closing connections
-- Clean up resources opened before a startup failure, without a second disconnect fallback
-- Add a backend to the existing lifecycle owner and pass its handle only to repositories that need it
-- Preserve cancellation, timeout, error and transaction semantics across the native bridge
-
-## Changing ownership or reloading resources
-
-- Inventory initialization, reload, tests, background jobs and shutdown together
-- Move every writer when changing authoritative storage, copying a replaceable global into state is insufficient
-- A frozen repository does not make its clients immutable
-- Decide whether each request captures one resource generation or reads it again during processing
-- Build and validate replacements before publishing them, preserving the old generation on failure
-- Serialize reloads and coordinate publication with shutdown
-- Drain streams and background users before retiring old resources, retaining every generation grows resource use
-- Define multi-worker coordination separately from an in-process swap
-- Choose mechanisms such as `ArcSwap` only after defining publication, capture and retirement
-
-## Migration context and boundaries
-
-- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations with Rust APIs one repository at a time
-- Prisma and native repositories coexist during the transition, unrelated existing calls can stay as they are
-- Prisma and SQLx keep separate client pools to PgBouncer, which pools connections to PostgreSQL
-  - Size both layers together and preserve reader/writer routing and pooling-mode compatibility
-- PgBouncer does not share transactions between Prisma and SQLx, keep each atomic unit of work on one backend and transaction
-- Never retry a failed Rust write through Prisma when it may already have committed
-- Prisma owns schema migrations today, eventual SQLx migration ownership and Prisma removal are separate planned steps, not part of a query port

.agents/skills/database-access/references/python-fastapi.md

@@ -1,101 +0,0 @@
-# Python and FastAPI
-
-## Existing pattern
-
-- `UserBannerStore` in `litellm/models/user_banner.py` is the Protocol the banner routes call
-- `get_database` in `litellm/proxy/deps/database.py` reads the lifespan-owned handle from request state
-- `get_user_banner_repository` in `litellm/proxy/deps/repositories.py` builds the banner repository from that handle
-- Choose each repository's replica routing once in its provider, never at call sites
-- A handler receives `UserBannerStore | None`, never a Prisma client or a concrete repository
-- `UserBannerRepository` implements `UserBannerStore` structurally, keeping the existing Prisma table, serialization and configuration-sync behavior
-- The banner audit provider binds the legacy audit callable at request time, without performing audit I/O
-- A native repository can later implement the same Protocol without changing handler signatures, see [porting a repository to SQLx](python-repository.md#porting-a-repository-to-sqlx)
-
-## Dependency injection
-
-- Add one provider and one `Annotated` alias per repository in `litellm/proxy/deps/repositories.py`
-- Annotate the provider and alias with the feature's Protocol, not the concrete repository
-- A route depends only on the repositories it calls, so a request never builds repositories it does not use
-- Do not build repositories once at lifespan, a reconnect can replace the Prisma client they wrap
-
-```python
-class UserBannerStore(Protocol):
-    async def get_banner(self) -> UserBanner: ...
-
-    async def save_banner(self, banner: UserBanner) -> None: ...
-```
-
-```python
-async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
-    return None if database is None else UserBannerRepository(database.prisma)
-
-
-UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)]
-```
-
-- FastAPI caches each provider within a request, so routes and sub-dependencies share one instance
-- Ordinary helpers and jobs receive explicit arguments, never unresolved `Depends` defaults
-- A route still reading the `prisma_client` global or building a repository inline switches to the repository's `...RepositoryDep`, keeping its no-database behavior and error precedence
-
-```python
-async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner:
-    if banners is None:
-        return UserBanner()
-    return await banners.get_banner()
-```
-
-## Lifespan state
-
-- Prisma startup owns connection setup, authentication refresh, health checks and migrations
-- `Database` holds the initialized Prisma client and the `NativeDatabase` handle, without holding fields per feature
-  - The native handle opens no pool yet, its `connect` and `close` are noops
-- The proxy lifespan builds the handle with `Database.open`, registers `Database.aclose` on its exit stack and publishes it in request state
-  - Callers only open and close `Database`, which closes Prisma and the native handle together, and `open` closes Prisma itself when the native connect fails
-- Starlette shallow-copies lifespan state into request state, which is distinct from `app.state`
-- The database module never imports a repository, legacy globals borrow its handle without owning shutdown
-- Drain requests, jobs, logging, spend and audit work before database teardown
-- The exit stack closes the handle on exit, including startup failures after the client is created
-- When adding native resources, initialize them in the serving process, respecting runtime fork restrictions
-- Await native shutdown instead of depending on Python garbage collection
-- Apply the [connection ownership rules](../SKILL.md#connection-ownership) when changing the owner
-
-## Timing and errors
-
-- FastAPI resolves dependencies before entering the handler
-- Handler exception handling cannot catch failures raised during dependency resolution
-- A universal required-database dependency can override authorization and validation behavior
-- Banner GET without a database returns the default disabled banner
-- Banner PATCH checks the admin role before returning the existing missing-database error
-- Invalid stored banner data falls back to the default, but storage failures still propagate
-- Banner writes finish before the success audit is queued as a FastAPI background task
-- Queue post-response work with `BackgroundTasks`, not bare `asyncio.create_task`, so the in-flight drain waits for it before the database closes
-- Do not move background work into an awaited provider merely to remove an import
-
-## Verification
-
-- Exercise real FastAPI handlers using `dependency_overrides` for external boundaries
-- Keep production parsing and persistence code real
-- Restore overrides after each test
-- Verify complete response bodies, status codes and OpenAPI against the base
-- Include combined failure conditions to pin error precedence
-- Check saved values, fresh revisions, audit contents and failed-write behavior
-- Check lifespan state with a running lifespan, not only a TestClient that skips startup
-- Verify providers do not connect, query, disconnect or audit eagerly
-- Use real PostgreSQL and live proxy requests when changing persistence behavior
-
-## Scope
-
-- Preserve direct Python call sites or migrate them explicitly
-- Inventory imports, state writers, side effects and error ordering before migrating a feature
-- Preserve configuration sync and serialization, and state which legacy reads remain
-- Do not migrate every route in a large module just to establish the pattern
-- Do not invent providers for resources without consumers
-
-## Framework references
-
-- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/) and [Starlette lifespan state](https://starlette.dev/lifespan/)
-- [`SessionDep`-style `Annotated` aliases](https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-session-dependency), which `DatabaseDep` and `UserBannerRepositoryDep` follow
-- [Dependency composition and caching](https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/)
-- [Background tasks and dependencies with `yield`](https://fastapi.tiangolo.com/advanced/advanced-dependencies/#background-tasks-and-dependencies-with-yield-technical-details)
-- [Dependency overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
-- [Testing lifespan](https://fastapi.tiangolo.com/advanced/testing-events/)

.agents/skills/database-access/references/python-repository.md

@@ -1,119 +0,0 @@
-# Python repositories
-
-## Adding a query
-
-Use this when a feature needs a read or write that no repository offers yet, including one more operation on an existing repository
-
-### Before starting
-
-- Search `litellm/repositories` for a repository that already owns the table, and extend it instead of adding a second one
-- Check `litellm/models/<feature>.py` for an existing Protocol and domain types
-- Decide the route's behavior when no database is configured, and where that check sits relative to authorization and validation
-
-### Steps
-
-1. Domain types and Protocol in `litellm/models/<feature>.py`
-   - The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
-   - Add the Protocol together with the provider, not for repositories nothing injects
-2. Repository in `litellm/repositories/<feature>_repository.py`
-   - Implement the operation with Prisma, a Rust port is not required
-   - Compose the Prisma client and type the tables it touches with `TableActions` from `prisma_protocols.py`, never inherit `PrismaTableRepository`
-   - Parse and serialize stored values here, so callers only see domain types
-   - Never import FastAPI, connect, disconnect or choose replicas per call
-3. Provider and `Annotated` alias in `litellm/proxy/deps/repositories.py`, see [dependency injection](python-fastapi.md#dependency-injection)
-4. The route takes `<Feature>RepositoryDep`, handles `None` with the behavior decided above, and calls the operation
-5. Jobs and helpers that need the operation receive the repository as an explicit argument, built by their caller from the handle it already holds
-
-### Done when
-
-- Handler tests drive the real route with `dependency_overrides` and the real repository code, see [verification](python-fastapi.md#verification)
-- Tests cover the no-database path, error precedence, and what was actually stored
-- Persistence changes are checked against real PostgreSQL through a live proxy
-
-### Out of scope
-
-- Porting the query to Rust, see [porting a repository to SQLx](#porting-a-repository-to-sqlx)
-- Moving unrelated direct Prisma calls in the same module
-
-## Moving existing access into a repository
-
-Use this when code calls Prisma directly, through `prisma_client.db.<table>` or a helper, and the calls should live behind a repository. The backend stays Prisma
-
-### Before starting
-
-- List every call being moved, its callers, and the transaction or batch it runs in
-- List the module's imports, global state writers, side effects such as audit and config sync, and error ordering
-- Record current return values, raised errors and response bodies, these are what the change must preserve
-
-### Steps
-
-1. Move the existing Prisma code into the relevant repository in `litellm/repositories` without rewriting the queries
-2. Add domain types and a Protocol in `litellm/models/<feature>.py` when a route will inject the repository
-3. Replace each direct call with the repository operation, through a provider for routes and explicit arguments for jobs and helpers, see [Python and FastAPI](python-fastapi.md)
-4. Calls sharing a transaction move into the same repository or unit of work in one change
-5. State which legacy reads remain and why, when the change does not move all of them
-
-### Done when
-
-- Focused regression tests pin the preserved return values, errors and transaction behavior
-- Response bodies, status codes and OpenAPI match the base
-- Configuration sync and serialization behave the same, including invalid stored data
-
-### Out of scope
-
-- Switching the backend, even when an equivalent Rust query exists
-- Converting other routes in the module just to establish the pattern
-- Cleanups unrelated to the moved calls
-
-## Porting a repository to SQLx
-
-Status: TBD. No repository has been ported yet. `database.native` is a `NativeDatabase` from `crates/python-bridge/src/db.rs` whose `connect` and `close` are noops that open no pool, and the PyO3 repository surface does not exist. A real pool takes typed writer/reader settings and a token provider, never the Prisma `DATABASE_URL`, which carries Prisma-only params and rotating IAM tokens. The constraints below are settled, the steps are a sketch to revise after the first port lands
-
-Use this only when a port is explicitly requested. Repository extraction does not require one
-
-### When not to port
-
-- An equivalent Rust or SQLx query existing is not a reason to switch a Prisma caller
-- Leave existing backend selection unchanged unless integration is explicitly requested
-- Never port part of a repository, it is either all Prisma or all native
-
-### Before starting
-
-- Identify the repository's inputs, outputs, errors and transaction boundary
-- List repositories that share a transaction with it, they move together
-- Check `litellm-rust/crates/db` for an equivalent query and reuse it when its behavior matches
-
-### Steps
-
-1. Rust queries, following `litellm-rust/crates/db/AGENTS.md`
-   - Static SQL in `queries/`, compile-checked with SQLx, typed inputs and row projections in `src/`
-   - Dynamic filters and sort keys become closed Rust enums that select static SQL, every value is bound
-   - Refresh SQLx query metadata after SQL changes
-2. PyO3 surface exposing typed operations, never SQL text, table names or column names from Python. TBD: where it lives and how the handle reaches it
-3. Native repository implementing the same Protocol in `litellm/repositories`
-   - Converts typed PyO3 results to domain types, preserving defaults, invalid-data handling and storage errors
-   - Python conversion stays in the bridge, not in `crates/db`
-4. Switch the provider in `litellm/proxy/deps/repositories.py`, the Protocol and routes stay unchanged
-
-```diff
- async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
--    return None if database is None else UserBannerRepository(database.prisma)
-+    return None if database is None else NativeUserBannerRepository(database.native)
-```
-
-5. Delete the Prisma repository in the same change and switch every caller, including jobs and helpers
-
-### Done when
-
-- Behavior is verified against real PostgreSQL, covering null handling, conflicts, ordering, errors and transaction semantics
-- `DATABASE_URL=... cargo test -p litellm-db --locked -- --include-ignored` passes against a Prisma-migrated database
-- Live proxy requests show unchanged responses. TBD: a Python-side real-PostgreSQL test harness
-
-### Rollback
-
-- Roll back the release, never retry an uncertain native write through Prisma
-
-### Out of scope
-
-- SQLx owning schema migrations, and removing Prisma
-- Converting unrelated routes to dependency injection

.agents/skills/database-access/references/rust-axum.md

@@ -1,4 +0,0 @@
-# Rust and Axum
-
-- This should use same `crates/db` what is exposed to Python world.
-- Details are TBD.

.circleci/AGENTS.md

@@ -1,7 +0,0 @@
-# .circleci
-
-The `integration` workflow on `main` runs from a scheduled pipeline in the CircleCI project settings, cron `9 0-8,14-23 * * *` (UTC) on `main` with no pipeline parameters. It has no `triggers:` block in this file because a workflow with one runs only on its schedule, so the API pipelines that set `run_migration_tests` or `routing_parity_base` would stop running it. If main stops getting integration runs, check that the schedule still exists and recreate it with these values
-
-Jobs clone with the `checkout_over_https` command, never the built-in `checkout` step, because CircleCI falls back to the project's SSH checkout key without its GitHub App access, GitHub rejects that key, and the public repo needs no key. It fetches the job's commit with full history and no file contents (`--filter=blob:none`), like the built-in step did, so `scripts/path_filter.sh` can find a merge base
-
-A step that checks out files from another commit fetches that commit with `--no-filter`, as `routing_parity` does, because the clone has no file contents and git would otherwise download each missing file in its own request, which took 210 to 896 seconds for the routing parity base

.circleci/scripts/run_migration_tests.py

@@ -1,116 +0,0 @@
-from __future__ import annotations
-
-import json
-import os
-import re
-import subprocess
-import sys
-from pathlib import Path
-from typing import Final
-from xml.etree import ElementTree
-
-SUITES: Final = {
-    "startup": (("test_startup.py",), 13),
-    "recovery": (("test_recovery.py",), 15),
-    "legacy": (("test_legacy.py", "test_pooling.py"), 11),
-    "upgrade": (("test_upgrade.py", "test_rolling_upgrade.py"), 5),
-    "shaped": (("test_shaped_database.py",), 1),
-}
-
-
-def successful_junit(path: Path, expected: int, exit_code: int) -> bool:
-    if exit_code != 0 or not path.is_file():
-        return False
-    try:
-        root: Final = ElementTree.parse(path).getroot()
-    except ElementTree.ParseError:
-        return False
-    cases: Final = tuple(root.iter("testcase"))
-    identities: Final = frozenset((case.get("classname"), case.get("name")) for case in cases)
-    return len(cases) == len(identities) == expected and all(
-        not any(case.find(tag) is not None for tag in ("failure", "error", "skipped")) for case in cases
-    )
-
-
-def output(*command: str) -> str:
-    return subprocess.check_output(command, text=True, timeout=90).strip()
-
-
-def record_image() -> None:
-    source: Final = output("git", "rev-parse", "HEAD")
-    image: Final = output("docker", "image", "inspect", "litellm-docker-database:ci", "--format", "{{.Id}}")
-    revision: Final = output(
-        "docker",
-        "image",
-        "inspect",
-        "litellm-docker-database:ci",
-        "--format",
-        '{{index .Config.Labels "org.opencontainers.image.revision"}}',
-    )
-    assert re.fullmatch(r"[0-9a-f]{40}", source), "Invalid source revision"
-    assert revision == source, "Candidate image revision differs from the tested source"
-    Path("migration-image.json").write_text(
-        json.dumps(
-            {
-                "source_sha": source,
-                "image_id": image,
-                "candidate_image": os.environ.get("MIGRATION_CANDIDATE_IMAGE", ""),
-            }
-        )
-    )
-
-
-def main() -> int:
-    suite: Final = os.environ["MIGRATION_TEST_SUITE"]
-    files, expected = SUITES[suite]
-    metadata: Final = json.loads(Path("migration-image.json").read_text())
-    assert metadata["source_sha"] == output("git", "rev-parse", "HEAD"), "Image and test source revisions differ"
-    assert metadata["image_id"] == output(
-        "docker", "image", "inspect", os.environ["LITELLM_MIGRATION_TEST_IMAGE"], "--format", "{{.Id}}"
-    ), "Loaded image differs from the build output"
-    assert metadata["candidate_image"] == os.environ.get("MIGRATION_CANDIDATE_IMAGE", ""), "Wrong release candidate"
-    destination: Final = Path(os.environ["MIGRATION_TEST_OUTPUT"])
-    junit: Final = destination / "junit" / "results.xml"
-    junit.parent.mkdir(parents=True, exist_ok=True)
-    result: Final = subprocess.run(
-        (
-            sys.executable,
-            "-m",
-            "pytest",
-            *(f"tests/e2e/migrations/{name}" for name in files),
-            "-vv",
-            "--tb=short",
-            "--durations=10",
-            f"--junitxml={junit}",
-            "-o",
-            "addopts=",
-            "--reruns=0",
-        ),
-        check=False,
-        timeout=1200,
-    )
-    passed: Final = successful_junit(junit, expected, result.returncode)
-    (destination / "verdict.json").write_text(
-        json.dumps(
-            {
-                **metadata,
-                "suite": suite,
-                "baseline_image": os.environ.get("LITELLM_MIGRATION_BASELINE_IMAGE", ""),
-                "expected_cases": expected,
-                "passed": passed,
-                "pytest_exit_code": result.returncode,
-                "test_revision": metadata["source_sha"],
-                "workflow_id": os.environ.get("CIRCLE_WORKFLOW_ID", ""),
-                "job_number": os.environ.get("CIRCLE_BUILD_NUM", ""),
-            },
-            indent=2,
-        )
-    )
-    return 0 if passed else 1
-
-
-if __name__ == "__main__":
-    if sys.argv[1:] == ["record-image"]:
-        record_image()
-    else:
-        raise SystemExit(main())

.devcontainer/devcontainer.json

@@ -1,57 +0,0 @@
-{
-	"name": "Python 3.11",
-	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
-	"image": "mcr.microsoft.com/devcontainers/python:3.11-bookworm",
-	// https://github.com/devcontainers/images/tree/main/src/python
-	// https://mcr.microsoft.com/en-us/product/devcontainers/python/tags
-
-	// "build": {
-	// 	"dockerfile": "Dockerfile",
-	// 	"context": ".."
-	// },
-
-	// Features to add to the dev container. More info: https://containers.dev/features.
-	"features": {
-		"ghcr.io/devcontainers/features/node:1": {
-			"version": "lts"
-		},
-		"ghcr.io/devcontainers/features/docker-in-docker:2": {}
-	},
-
-	// Configure tool-specific properties.
-	"customizations": {
-		// Configure properties specific to VS Code.
-		"vscode": {
-			"settings": {},
-			"extensions": [
-				"ms-python.python",
-				"ms-python.vscode-pylance",
-				"GitHub.copilot",
-				"GitHub.copilot-chat",
-				"ms-python.autopep8"
-			]
-		}
-	},
-	
-	// Use 'forwardPorts' to make a list of ports inside the container available locally.
-	"forwardPorts": [4000],
-	
-	"containerEnv": {
-		"LITELLM_LOG": "DEBUG"
-	},
-
-	// Use 'portsAttributes' to set default properties for specific forwarded ports. 
-	// More info: https://containers.dev/implementors/json_reference/#port-attributes
-	"portsAttributes": {
-		"4000": {
-			"label": "LiteLLM Server",
-			"onAutoForward": "notify"
-		}
-	},
-
-	// More info: https://aka.ms/dev-containers-non-root.
-	// "remoteUser": "litellm",
-
-	// Use 'postCreateCommand' to run commands after the container is created.
-	"postCreateCommand": "bash ./.devcontainer/post-create.sh"
-}

.devcontainer/post-create.sh

@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-echo "[post-create] Installing uv"
-curl -LsSf https://astral.sh/uv/0.10.9/install.sh | env UV_NO_MODIFY_PATH=1 sh
-export PATH="$HOME/.local/bin:$PATH"
-
-echo "[post-create] Installing Python dependencies (uv)"
-uv sync --frozen --group proxy-dev --extra proxy
-
-echo "[post-create] Generating Prisma client"
-uv run --no-sync prisma generate
-
-echo "[post-create] Installing npm dependencies"
-cd ui/litellm-dashboard && npm ci
-
-echo "[post-create] Done"

.env.example

@@ -1,32 +0,0 @@
-# OpenAI
-OPENAI_API_KEY = ""
-OPENAI_BASE_URL = ""
-# Cohere
-COHERE_API_KEY = ""
-# OpenRouter
-OR_SITE_URL = ""
-OR_APP_NAME = "LiteLLM Example app"
-OR_API_KEY = ""
-# Azure API base URL
-AZURE_API_BASE = ""
-# Azure API version
-AZURE_API_VERSION = ""
-# Azure API key
-AZURE_API_KEY = ""
-# Replicate
-REPLICATE_API_KEY = ""
-REPLICATE_API_TOKEN = ""
-# Anthropic
-ANTHROPIC_API_KEY = ""
-# Infisical
-INFISICAL_TOKEN = ""
-# Novita AI
-NOVITA_API_KEY = ""
-# INFINITY
-INFINITY_API_KEY = ""
-
-# Development Configs
-# Generate one with: echo "LITELLM_MASTER_KEY=sk-$(openssl rand -hex 32)"
-LITELLM_MASTER_KEY = ""
-DATABASE_URL = "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
-STORE_MODEL_IN_DB = "True"

.git-blame-ignore-revs

@@ -1,40 +0,0 @@
-# Add the commit hash of any commit you want to ignore in `git blame` here.
-# One commit hash per line.
-#
-# The GitHub Blame UI will use this file automatically!
-#
-# Run this command to always ignore formatting commits in `git blame`
-#     git config blame.ignoreRevsFile .git-blame-ignore-revs
-
-# Update pydantic code to fix warnings (GH-3600)
-876840e9957bc7e9f7d6a2b58c4d7c53dad16481
-
-# style(ui): run prettier --write across the dashboard (#29622)
-7edf3a9cb55548b143df1692f4ed7c4681d7fcf7
-
-# style: reformat litellm/ with ruff format (#31317)
-17bfd415aeb5a57fb646b5cc67da1c730aa7c50b
-
-# style: unify ruff format width on 120 (#31518)
-48b5a5a0cc5a694a11219416ee0b6eb6e620e74e
-
-# refactor(imports): move collections.abc names out of typing (#35495)
-397e8e4918777e4e60a7f5e88699e0a9a7dabb3d
-
-# refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets (#35495)
-b604e2b20c6db2099085a2f0e59b7e99e87eed6f
-
-# refactor(logging): drop redundant !s conversion flags from f-strings (#35546)
-7b2d3440cba3160277470f7a0180098ae9b87864
-
-# perf: build log messages lazily so filtered-out log records cost nothing (#35703)
-c9887a1f94bc1e7e4bdfe64d640f0509a0bc19dd
-
-# feat(lint): enforce Final on locals and freeze function parameters (#35807)
-2708620d6a599cc73c1950a942d26ac26a7ed3d4
-
-# chore(lint): remove litellm/types from the ruff lint exclusion (#35926)
-4e32a8bf6a1e1af1e04b67c759841ccef44b2235
-
-# chore(lint): strip inert type: ignore comments and zero LIT009/LIT010/LIT011 headroom (#35928)
-338e411103ad5d7003e97f34f04fa36bca542dbe

.gitattributes

@@ -1,2 +0,0 @@
-*.ipynb linguist-vendored
-ui/litellm-dashboard/src/lib/http/schema.d.ts linguist-generated

.github/AGENTS.md

@@ -0,0 +1,5 @@
+# CI
+
+Integration groups and replica runs remain in `workflows/test-integration.yml`. Do not duplicate them in another workflow.
+
+CI helpers used by existing workflows live in `scripts/ci/`. Jobs that need merge bases check out full history. Removing a CI job does not count its tests as covered. List tests that no remaining job invokes explicitly in `ci-coverage-allowlist.yml`.

.github/CODEOWNERS

@@ -1,6 +1,6 @@
 /model_prices_and_context_window.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
 /litellm/model_prices_and_context_window_backup.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
-/ci_cd/cost_map_pins.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
-/helm/ @yassin-berriai
-/terraform/ @yassin-berriai
+/litellm/scripts/ci_cd/cost_map_pins.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri
+/litellm/deploy/helm/ @yassin-berriai
+/litellm/deploy/terraform/ @yassin-berriai
 /.github/workflows/zizmor.yml @yuneng-berri

.github/CONTRIBUTING.md

@@ -276,16 +276,14 @@ When the provider reports its own usage or spend numbers, compare the proxy's nu
 
 Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution, and it is where PRs get reworked most often. Follow these rules
 
-**Preserve zero prices.** In [`model_prices_and_context_window.json`](model_prices_and_context_window.json), an explicit `0.0` means that the priced operation is free. A missing field or `null` means no rate is specified, so the relevant calculator may use a fallback. For example, when no batch input rate is configured, batch cache writes use `cache_creation_input_token_cost` if set, otherwise the regular input rate, then apply the batch discount. A zero ordinary input rate does not make separately priced cache reads or writes free. Test `rate is None` before substituting a fallback, since `row.get("cache_creation_input_token_cost") or fallback` would discard a valid zero. Speech keeps character pricing ahead of token pricing even when the character rate is zero. If neither rate is configured, speech cost calculation raises an error. Its model lookup preserves unset rates instead of treating compatibility defaults as explicit zero prices. Perplexity keeps a configured legacy search rate ahead of the current search rate, including zero. When registering custom prices, `None` leaves an existing rate unchanged and `0.0` overrides it
+**Preserve zero prices.** In [`model_prices_and_context_window.json`](../model_prices_and_context_window.json), an explicit `0.0` means that the priced operation is free. A missing field or `null` means no rate is specified, so the relevant calculator may use a fallback. For example, when no batch input rate is configured, batch cache writes use `cache_creation_input_token_cost` if set, otherwise the regular input rate, then apply the batch discount. A zero ordinary input rate does not make separately priced cache reads or writes free. Test `rate is None` before substituting a fallback, since `row.get("cache_creation_input_token_cost") or fallback` would discard a valid zero. Speech keeps character pricing ahead of token pricing even when the character rate is zero. If neither rate is configured, speech cost calculation raises an error. Its model lookup preserves unset rates instead of treating compatibility defaults as explicit zero prices. Perplexity keeps a configured legacy search rate ahead of the current search rate, including zero. When registering custom prices, `None` leaves an existing rate unchanged and `0.0` overrides it
 
-**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. [`ARCHITECTURE.md`](ARCHITECTURE.md) shows where the layers live, and the decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) explain why they are shaped that way. For example, [the ADR on provider usage fields](docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md) covers how provider-specific usage fields reach cost tracking. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
+**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
 
 **Bill a new provider usage field from the usage object.** Leave the field on the usage object as an extra and read it in that provider's `cost_calculator.py`. A provider without one gets a new `cost_calculator.py` and an arm for it in `cost_per_token` (`litellm/cost_calculator.py`), which picks the calculator by provider. If the provider's transform builds `Usage` from named fields, as the Anthropic, Gemini, and Bedrock Converse ones do, pass the new field there too. Streamed requests rebuild usage in `ChunkProcessor.calculate_usage` (`litellm/litellm_core_utils/streaming_chunk_builder_utils.py`), which does not copy extras yet the way the Responses-to-chat bridge does (minus a short exclusion list), so until it does, bill the field on streams by naming it there, as Anthropic's `inference_geo` is. A provider that reports its own charge for the request, as OpenRouter does in `usage.cost`, is billed at that charge from `_hidden_params` before any calculator runs, unless the deployment sets its own prices. When the field counts a built-in tool such as web search, add the provider to `get_cost_for_web_search_request` in `litellm/llms/__init__.py`, and add a gate to `litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py` only when no existing gate fires. Then show the field's cost in your proof on every endpoint it reaches, streaming included
 
 **Do not change what a caller sees to make internals easier.** Each endpoint keeps the schema of the API it emulates. A `/v1/responses` caller gets `input_tokens` and a `/v1/chat/completions` caller gets `prompt_tokens`, whatever the provider sent and whatever cost tracking needs. A provider usage field that shows up as an extra key is fine, since it only adds a key, but renaming, removing, or reshaping a key the endpoint defines is not. Reshaping a public response to feed an internal consumer breaks every user of that endpoint
 
-If your change makes an architecture decision, such as a new cross-cutting mechanism or a deliberate difference from a provider's API shape, add an ADR to [`docs/adrs/`](docs/adrs/AGENTS.md) in the same PR
-
 ## Building and Running Locally
 
 ### LiteLLM Proxy Server
@@ -305,8 +303,8 @@ uv run litellm --config your_config.yaml
 If you want to build the Docker image yourself:
 
 ```bash
-# Build using the non-root Dockerfile
-docker build -f docker/Dockerfile.non_root -t litellm_dev .
+# Build the image (runs as a non-root user by default)
+docker build -f litellm/deploy/docker/Dockerfile -t litellm_dev .
 
 # Generate a master key. Requests send it as the bearer token
 export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
@@ -330,7 +328,7 @@ git clone https://github.com/YOUR_USERNAME/litellm.git
 cd litellm
 
 # Navigate to the UI dashboard directory
-cd ui/litellm-dashboard
+cd litellm/ui/litellm-dashboard
 
 # Install dependencies
 npm install
@@ -362,7 +360,7 @@ Local lint and budget checks follow origin's current default branch. They refres
 ## Submitting Your PR
 
 1. **Push your branch**: `git push origin your-feature-branch`
-2. **Create a PR**: Go to GitHub and open a pull request against the repository's current default branch. Run `python3 scripts/default_branch.py --branch` to check its name
+2. **Create a PR**: Go to GitHub and open a pull request against the repository's current default branch. Run `python3 litellm/scripts/default_branch.py --branch` to check its name
 3. **Fill out the PR template**: Provide clear description of changes
 4. **Wait for review**: Maintainers will review and provide feedback
 5. **Address feedback**: Make requested changes and push updates

.github/SECURITY.md

@@ -4,23 +4,19 @@
 
 LiteLLM maintains a security announcements mailing list that is open to anyone. Subscribers receive advance notice, typically one to two days, before we release a fix for a particularly severe vulnerability or for any vulnerability exploitable by an unauthenticated attacker. This notice is provided on a best-effort basis
 
-To subscribe, visit [https://berriai.github.io/security-announce-signup/](https://berriai.github.io/security-announce-signup/)
+To subscribe, visit [https://security.litellm.ai/mailinglist/](https://security.litellm.ai/mailinglist/)
 
 ## Security Vulnerability Reporting Guidelines
 
-> [!WARNING]
-> Reports that do not include a video demonstrating the exploit will be closed without review. See [Reproduction Video Requirement](#reproduction-video-requirement) below.
 
-We value the security community's role in protecting our systems and users. To report a security vulnerability:
+We value the security community's role in protecting our systems and users.
+To report a security vulnerability, please submit a report through our vulnerability disclosure form:
 
-- File a private vulnerability report on GitHub: [Report a vulnerability](https://github.com/BerriAI/litellm/security/advisories/new)
-- Include steps to reproduce the issue
-- Include a video or screen recording demonstrating the full exploit against a live LiteLLM instance, from initial access through to impact. A terminal recording (for example asciinema) is fine for CLI-only exploits.
-- Provide any relevant additional information
+[LiteLLM VDP](https://security.litellm.ai/vdp/)
 
-### Reproduction Video Requirement
+These reports are submitted to HackerOne and are triaged there, though no account is needed.
 
-A video demonstrating the exploit is required for every report. AI tools have made it easy to produce plausible-sounding vulnerability reports that do not reproduce in practice, and triaging them takes time away from real issues. Reports submitted without a working reproduction video will be closed without review. If you add a video to a closed report, we will reopen and triage it.
+**Please do not submit vulnerabilities as draft GHSAs. As of 2026-11-11 new GHSA submissions will be closed.**
 
 ### Vulnerability Categories
 
@@ -55,7 +51,9 @@ To qualify for a bounty, reports must include clear reproduction steps, a reprod
 
 ### Known Non-Issues
 
-- Attacks that require a misconfiguration on setup (e.g not setting a `master_key` on the proxy configuration), are **explicitly not in scope** and are not considered vulnerable.
+- **Misconfigurations:** attacks that require misconfiguration of the application (e.g. not setting a `master_key` on the proxy configuration).
+- **Insecure configurations:** attacks that require deliberate configuration changes to disable security functionality. (e.g. IDOR on response IDs requiring configuration of `disable_responses_id_security` to `true`)
+- **Social engineering:** attacks that rely on social engineering as their primary mechanism (e.g. convincing a user's agent to install a malicious package we did not publish).
 
 ## Security Measures
 

.github/actions/cache-cargo-build/action.yml

@@ -3,7 +3,7 @@ description: >-
   Cache the Cargo registry and target directory the root package's build needs,
   so only the first job on a given Cargo.lock compiles the bridge from scratch.
 
-  litellm builds through maturin, which compiles litellm-rust/crates/python-bridge
+  litellm builds through maturin, which compiles litellm/rust/crates/python-bridge
   in the dev profile for editable installs. `uv sync` therefore pays a full build
   in every job that installs the workspace. Nothing caught it, because the uv cache
   holds wheels uv downloads rather than wheels it builds, and a path dependency
@@ -31,8 +31,8 @@ runs:
         path: |
           ~/.cargo/registry
           ~/.cargo/git
-          litellm-rust/target
-        key: ${{ runner.os }}-maturin-${{ inputs.profile }}-${{ hashFiles('litellm-rust/Cargo.lock') }}
+          litellm/rust/target
+        key: ${{ runner.os }}-maturin-${{ inputs.profile }}-${{ hashFiles('litellm/rust/Cargo.lock') }}
         restore-keys: |
           ${{ runner.os }}-maturin-${{ inputs.profile }}-
 
@@ -43,7 +43,7 @@ runs:
         path: |
           ~/.cargo/registry
           ~/.cargo/git
-          litellm-rust/target
-        key: ${{ runner.os }}-maturin-${{ inputs.profile }}-${{ hashFiles('litellm-rust/Cargo.lock') }}
+          litellm/rust/target
+        key: ${{ runner.os }}-maturin-${{ inputs.profile }}-${{ hashFiles('litellm/rust/Cargo.lock') }}
         restore-keys: |
           ${{ runner.os }}-maturin-${{ inputs.profile }}-

.github/actions/cache-go-build/action.yml

@@ -0,0 +1,39 @@
+name: "Cache the Go build"
+description: >-
+  Restore the Go module and build caches on every run and save them only from
+  main, so pull requests reuse main's cache instead of evicting it with their
+  own copies.
+
+  actions/setup-go's built-in cache saves from every ref, and a cache saved by a
+  pull request is readable only by that pull request. Callers must pass
+  `cache: false` to setup-go and run this action after it.
+
+inputs:
+  dependency-path:
+    description: "Path to the go.sum that keys the cache"
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Restore and save the Go module and build caches
+      if: github.ref == 'refs/heads/main'
+      uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+      with:
+        path: |
+          ~/go/pkg/mod
+          ~/.cache/go-build
+        key: ${{ runner.os }}-go-${{ hashFiles(inputs.dependency-path) }}
+        restore-keys: |
+          ${{ runner.os }}-go-
+
+    - name: Restore the Go module and build caches
+      if: github.ref != 'refs/heads/main'
+      uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+      with:
+        path: |
+          ~/go/pkg/mod
+          ~/.cache/go-build
+        key: ${{ runner.os }}-go-${{ hashFiles(inputs.dependency-path) }}
+        restore-keys: |
+          ${{ runner.os }}-go-

.github/actions/detect-changes/action.yml

@@ -1,7 +1,7 @@
 name: "Detect relevant changes"
 description: >-
-  Classify the pull request's changed files with .circleci/scripts/classify_changes.sh
-  and expose decision=run|skip for one category. backend means anything outside ui/,
+  Classify the pull request's changed files with .github/scripts/ci/classify_changes.sh
+  and expose decision=run|skip for one category. backend means anything outside litellm/ui/,
   docs/ and markdown; ui means the dashboard sources alone. decision=skip lets callers
   short-circuit expensive steps while the job still completes successfully and satisfies
   its required status check, which a paths: filter cannot do because a workflow that

.github/actions/restore-rust-bridge/action.yml

@@ -0,0 +1,37 @@
+name: "Restore the Rust bridge"
+description: >-
+  Restore the prebuilt Rust bridge that the unit workflow's rust-bridge job saves
+  on main, keyed on every input of the Rust build.
+
+  The bridge is abi3, so one build serves every supported Python version. A job
+  whose Rust inputs match main installs the restored wheel and native library
+  instead of compiling them inside `uv sync`. On a miss the caller builds from
+  source as before.
+
+  The restored `rust-bridge-dist` directory holds the editable litellm wheel, the
+  litellm-proxy gateway wheel, and `litellm/rust_bridge/_native.abi3.so`.
+
+outputs:
+  key:
+    description: "Cache key for the bridge built from this checkout"
+    value: ${{ steps.key.outputs.key }}
+  cache-hit:
+    description: "'true' when the bridge for this exact key was restored"
+    value: ${{ steps.restore.outputs.cache-hit }}
+
+runs:
+  using: composite
+  steps:
+    - name: Define the Rust bridge cache key
+      id: key
+      shell: bash
+      env:
+        RUST_BRIDGE_CACHE_KEY: ${{ runner.os }}-rust-bridge-editable-${{ hashFiles('litellm/rust/Cargo.lock', 'litellm/rust/Cargo.toml', 'litellm/rust/crates/**', 'model_prices_and_context_window.json', 'pyproject.toml', 'litellm/packaging/litellm-proxy/pyproject.toml') }}
+      run: echo "key=$RUST_BRIDGE_CACHE_KEY" >> "$GITHUB_OUTPUT"
+
+    - name: Restore the Rust bridge
+      id: restore
+      uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+      with:
+        path: rust-bridge-dist
+        key: ${{ steps.key.outputs.key }}

.github/actions/setup-uv-with-retries/action.yml

@@ -11,6 +11,11 @@ inputs:
 runs:
   using: composite
   steps:
+    - name: Configure the Rust build environment
+      shell: bash
+      working-directory: ${{ github.workspace }}
+      run: bash .github/scripts/setup_rust_environment.sh
+
     - name: Set up uv (attempt 1)
       id: attempt-1
       continue-on-error: true

.github/ci-coverage-allowlist.yml

@@ -4,6 +4,16 @@ description: >-
   by a job nor listed here, so every entry below is a decision on the record.
 
 test_paths:
+  - reason: ran only in CircleCI, removed
+    paths:
+      - tests/e2e/migrations/test_legacy.py
+      - tests/e2e/migrations/test_pooling.py
+      - tests/e2e/migrations/test_recovery.py
+      - tests/e2e/migrations/test_rolling_upgrade.py
+      - tests/e2e/migrations/test_shaped_database.py
+      - tests/e2e/migrations/test_startup.py
+      - tests/e2e/migrations/test_upgrade.py
+      - tests/integration/database/test_roi_observed.py
   - reason: >-
       litellm.agent() end-to-end suite. It drives the real claude, codex, opencode and pi CLIs and
       deepagents against a live LiteLLM AI Gateway, so it needs those binaries on PATH plus
@@ -19,7 +29,7 @@ test_paths:
       - tests/rust-python-harness
   - reason: >-
       Owned by the open pull request that moves the Rust/Python parity checks (#45135). It ran in
-      the CircleCI build_and_test job that globbed tests/test_*.py, which this migration deletes,
+      the retired build_and_test job that globbed tests/test_*.py,
       and that pull request gives it a new home
     paths:
       - tests/test_rust_python_harness.py
@@ -57,19 +67,10 @@ test_paths:
       - tests/integration/test_oci_proxy_integration.py
 
 dockerfiles:
-  - reason: >-
-      The dashboard container is a static Next.js export served by nginx, and the dashboard build
-      and lint workflows already exercise that output, so building the image adds no signal about it
-    paths:
-      - ui/Dockerfile
-  - reason: >-
-      An example image under cookbook/ that is documentation rather than a shipped artifact
-    paths:
-      - cookbook/litellm-ollama-docker-image/Dockerfile
   - reason: >-
       The Rust gateway image compiles the whole workspace in release mode, which is too slow for
       a per-pull-request job while the gateway binary is still being assembled; the Rust lint,
       clippy, and compile jobs already cover the code it packages. Revisit when the gateway is
       published
     paths:
-      - litellm-rust/crates/gateway/Dockerfile
+      - litellm/rust/crates/gateway/Dockerfile

.github/codecov.yml

@@ -0,0 +1,72 @@
+codecov:
+  require_ci_to_pass: false # post coverage status even if CI has unrelated failures
+  notify:
+    wait_for_ci: false # post as soon as expected uploads arrive, don't wait on CI
+
+ignore:
+  - "litellm/rust/**"
+
+# Flagged uploads avoid overlapping sessions on reruns.
+# Only the flags below reuse parent coverage, avoiding stale line maps elsewhere.
+flag_management:
+  default_rules:
+    carryforward: false
+  individual_flags:
+    - name: unit
+      carryforward: true
+    - name: core-packaging
+      carryforward: true
+    - name: mcp-integration
+      carryforward: true
+
+component_management:
+  individual_components:
+    - component_id: "Router"
+      paths:
+        - "router"
+    - component_id: "LLMs"
+      paths:
+        - "*/llms/*"
+    - component_id: "Caching"
+      paths:
+        - "*/caching/*"
+        - ".*redis.*"
+    - component_id: "litellm_logging"
+      paths:
+        - "*/integrations/*"
+        - ".*litellm_logging.*"
+    - component_id: "Proxy_Authentication"
+      paths:
+        - "*/proxy/auth/**"
+    - component_id: "Enterprise"
+      paths:
+        - "litellm/enterprise/**"
+    - component_id: "Batches"
+      paths:
+        - "*/proxy/batches_endpoints/**"
+        - "litellm/batches/**"
+        - "*/llms/*/batches/**"
+    - component_id: "Videos"
+      paths:
+        - "litellm/videos/**"
+        - "*/proxy/video_endpoints/**"
+        - "*/llms/*/videos/**"
+    - component_id: "Realtime"
+      paths:
+        - "litellm/realtime_api/**"
+        - "*/proxy/realtime_endpoints/**"
+        - "*/llms/*/realtime/**"
+        - "litellm/litellm_core_utils/realtime_streaming.py"
+comment:
+  layout: "header, diff, flags, components"  # show component info in the PR comment
+
+coverage:
+  status:
+    project:
+      default:
+        target: auto
+        threshold: 0% # do not allow project coverage to drop
+    patch:
+      default:
+        target: auto
+        threshold: 0% # patch coverage should be 100%

.github/gitguardian.yaml

@@ -22,7 +22,7 @@ secret:
     
     # Build outputs and static assets
     - "litellm/proxy/_experimental/out/**"
-    - "ui/litellm-dashboard/public/**"
+    - "litellm/ui/litellm-dashboard/public/**"
     - "**/swagger/*.js"
     - "**/*.woff"
     - "**/*.woff2"
@@ -34,7 +34,6 @@ secret:
     - "tests/**/*.txt"
     
     # Documentation and other non-code files
-    - "docs/**"
     - "**/*.md"
     - "**/*.lock"
     - "uv.lock"

.github/linters/ruff-strict.toml

@@ -1,4 +1,4 @@
-extend = "ruff.toml"
+extend = "../../pyproject.toml"
 
 [lint]
 preview = true
@@ -6,7 +6,7 @@ select = ["ANN", "ASYNC230", "B004", "B006", "B008", "B009", "B010", "B018", "B0
 extend-select = []
 # Overrides the inherited list: rules this gate enforces itself must NOT be external here,
 # so this config's RUF100 flags their stale `# noqa` directives. What remains external is
-# only what other tooling enforces: every base ruff.toml rule this select list doesn't
+# only what other tooling enforces: every base pyproject.toml rule this select list doesn't
 # re-enable (all of the default E/F families plus T20/PGH004/RUF008/RUF009, minus the
 # strict-selected F401 and RUF100; F4 is split out so stale F401 noqas stay detectable),
 # plus upstream litellm's ruff config.

.github/linters/ruff-tests.toml

@@ -1,4 +1,4 @@
-# Lint config for the test tree, which ruff.toml excludes from `ruff check`.
+# Lint config for the test tree, which pyproject.toml excludes from `ruff check`.
 #
 # Every rule here catches a test that cannot fail. Rules land one at a time, each
 # with its existing violations already fixed, so this list never needs a budget

.github/packaging/build_backend.py

@@ -0,0 +1,49 @@
+import sys
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Final
+
+import maturin
+from cli_package import PACKAGE_DIR, map_wheel
+from maturin import (
+    build_sdist as build_sdist,
+)
+from maturin import (
+    get_requires_for_build_editable as get_requires_for_build_editable,
+)
+from maturin import (
+    get_requires_for_build_sdist as get_requires_for_build_sdist,
+)
+from maturin import (
+    get_requires_for_build_wheel as get_requires_for_build_wheel,
+)
+from maturin import (
+    prepare_metadata_for_build_editable as prepare_metadata_for_build_editable,
+)
+from maturin import (
+    prepare_metadata_for_build_wheel as prepare_metadata_for_build_wheel,
+)
+
+
+def build_wheel(
+    wheel_directory: str,
+    config_settings: Mapping[str, str | list[str]] | None = None,
+    metadata_directory: str | None = None,
+) -> str:
+    filename: Final = maturin.build_wheel(wheel_directory, config_settings, metadata_directory)
+    map_wheel(Path(wheel_directory) / filename)
+    return filename
+
+
+def build_editable(
+    wheel_directory: str,
+    config_settings: Mapping[str, str | list[str]] | None = None,
+    metadata_directory: str | None = None,
+) -> str:
+    filename: Final = maturin.build_editable(wheel_directory, config_settings, metadata_directory)
+    map_wheel(Path(wheel_directory) / filename, source=Path(__file__).resolve().parents[2] / PACKAGE_DIR)
+    return filename
+
+
+if __name__ == "__main__":
+    build_editable(sys.argv[1])

.github/packaging/cli_package.py

@@ -0,0 +1,93 @@
+import argparse
+import base64
+import copy
+import csv
+import hashlib
+import io
+import os
+from pathlib import Path
+from tempfile import NamedTemporaryFile
+from typing import Final
+from zipfile import ZipFile, ZipInfo
+
+PACKAGE_DIR: Final = "litellm/cli"
+PACKAGE_NAME: Final = "litellm_cli"
+
+
+def editable_support(source: Path) -> tuple[tuple[str, bytes], ...]:
+    module: Final = "_litellm_cli_editable"
+    finder: Final = (
+        "import importlib.util\nimport sys\n\n"
+        "class _Finder:\n"
+        "    @staticmethod\n"
+        "    def find_spec(fullname, path=None, target=None):\n"
+        f"        if fullname == {PACKAGE_NAME!r}:\n"
+        "            return importlib.util.spec_from_file_location(\n"
+        f"                fullname, {str(source / '__init__.py')!r},\n"
+        f"                submodule_search_locations=[{str(source)!r}])\n"
+        "        return None\n\n"
+        "sys.meta_path.insert(0, _Finder())\n"
+    )
+    return ((f"{module}.py", finder.encode()), (f"{module}.pth", f"import {module}\n".encode()))
+
+
+def _name(name: str) -> str:
+    prefix: Final = f"{PACKAGE_DIR}/"
+    return f"{PACKAGE_NAME}/{name.removeprefix(prefix)}" if name.startswith(prefix) else name
+
+
+def _info(original: ZipInfo) -> ZipInfo:
+    mapped: Final = copy.copy(original)
+    mapped.filename = _name(original.filename)
+    return mapped
+
+
+def _record(entries: tuple[tuple[ZipInfo, bytes], ...], record: str) -> bytes:
+    buffer: Final = io.StringIO(newline="")
+    writer: Final = csv.writer(buffer, lineterminator="\n")
+    writer.writerows(
+        (
+            info.filename,
+            "sha256=" + base64.urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b"=").decode(),
+            str(len(data)),
+        )
+        for info, data in entries
+    )
+    writer.writerow((record, "", ""))
+    return buffer.getvalue().encode()
+
+
+def map_wheel(wheel: Path, *, source: Path | None = None) -> None:
+    with ZipFile(wheel) as archive:
+        records: Final = tuple(info for info in archive.infolist() if info.filename.endswith(".dist-info/RECORD"))
+        if len(records) != 1:
+            raise ValueError("A wheel must contain exactly one RECORD")
+        entries: Final = tuple(
+            (_info(info), archive.read(info))
+            for info in archive.infolist()
+            if info != records[0] and (source is None or not info.filename.startswith(f"{PACKAGE_DIR}/"))
+        )
+    additions: Final = () if source is None else tuple((ZipInfo(name), data) for name, data in editable_support(source))
+    contents: Final = (*entries, *additions)
+    names: Final = tuple(info.filename for info, _ in contents)
+    if len(names) != len(frozenset(names)):
+        raise ValueError("The CLI package mapping would overwrite a wheel member")
+    if source is None and not any(name.startswith(f"{PACKAGE_NAME}/") for name in names):
+        raise ValueError("The wheel does not contain the CLI package")
+    with NamedTemporaryFile(dir=wheel.parent, suffix=".whl", delete=False) as temporary:
+        destination: Final = Path(temporary.name)
+    try:
+        with ZipFile(destination, "w") as archive:
+            for info, data in contents:
+                archive.writestr(info, data)
+            archive.writestr(records[0], _record(contents, records[0].filename))
+        os.replace(destination, wheel)
+    finally:
+        destination.unlink(missing_ok=True)
+
+
+if __name__ == "__main__":
+    parser: Final = argparse.ArgumentParser()
+    parser.add_argument("wheels", nargs="+", type=Path)
+    for wheel in parser.parse_args().wheels:
+        map_wheel(wheel)

.github/scripts/assert_ci_coverage.py

@@ -10,6 +10,7 @@ import warnings
 from collections.abc import Callable, Iterable, Mapping, Sequence
 from dataclasses import dataclass
 from functools import lru_cache
+from itertools import chain
 from types import MappingProxyType
 from typing import Final
 
@@ -17,7 +18,6 @@ import yaml
 
 REPO_ROOT = pathlib.Path(__file__).resolve().parents[2]
 WORKFLOW_DIR = REPO_ROOT / ".github" / "workflows"
-CIRCLECI_CONFIG = REPO_ROOT / ".circleci" / "config.yml"
 ALLOWLIST_FILE = REPO_ROOT / ".github" / "ci-coverage-allowlist.yml"
 TESTS_ROOT = REPO_ROOT / "tests"
 
@@ -25,14 +25,18 @@ ALLOWLIST_KEYS = frozenset({"description", "test_paths", "dockerfiles"})
 PATH_FILTER_KEYS = frozenset({"paths", "paths-ignore"})
 TEST_PATH_KEYS = frozenset({"test-path", "test-paths", "test_path"})
 DOCKERFILE_INPUT_KEYS = frozenset({"file", "dockerfile"})
-TEST_RUNNER_RE = re.compile(r"\bpytest\b|\bcircleci tests\b|\bhelm unittest\b|\bplaywright test\b|\bpython[0-9.]*\s")
+TEST_RUNNER_RE = re.compile(r"\bpytest\b|\bhelm unittest\b|\bplaywright test\b|\bpython[0-9.]*\s")
 IMAGE_BUILD_RE = re.compile(r"\bdocker\s+(?:buildx\s+)?build\b")
-TEST_TOKEN_RE = re.compile(r"tests/[A-Za-z0-9_./*?-]+")
-IGNORE_ARG_RE: Final = re.compile(r"--ignore(?:-glob)?[= ](\S+)")
+TEST_TOKEN_RE = re.compile(r"tests/[A-Za-z0-9_./*?\[\]-]+")
+IGNORE_ARG_RE: Final = re.compile(r"--(?:ignore(?:-glob)?|deselect)[= ](\S+)")
 DOCKERFILE_TOKEN_RE = re.compile(r"[A-Za-z0-9_./-]*Dockerfile[A-Za-z0-9_.-]*")
 COMMENT_RE = re.compile(r"^\s*#.*$", re.MULTILINE)
-GLOB_CHARS = frozenset("*?")
-DEDICATED_INTEGRATION_WORKFLOWS: Final = MappingProxyType({"image-scan.yml": "tests/integration/database/image/"})
+GLOB_CHARS = frozenset("*?[")
+DEDICATED_INTEGRATION_WORKFLOWS: Final = MappingProxyType(
+    {
+        "image-scan.yml": "tests/integration/database/image/",
+    }
+)
 
 # Trees whose jobs are sharded with no catch-all bucket, so every child that holds
 # tests has to be named by some shard or it runs nowhere. A child listed here is
@@ -115,9 +119,7 @@ def _scalars(node: object, key: str) -> tuple[Scalar, ...]:
 
 
 def _config_files() -> tuple[pathlib.Path, ...]:
-    workflows = tuple(sorted(path for path in WORKFLOW_DIR.iterdir() if path.suffix in (".yml", ".yaml")))
-    circleci = (CIRCLECI_CONFIG,) if CIRCLECI_CONFIG.is_file() else ()
-    return workflows + circleci
+    return tuple(sorted(path for path in WORKFLOW_DIR.iterdir() if path.suffix in (".yml", ".yaml")))
 
 
 def _all_scalars() -> tuple[Scalar, ...]:
@@ -243,7 +245,6 @@ def _describe(paths: tuple[str, ...]) -> str:
     return f"{len(paths)} test file(s) invoked by no job: {names}{suffix}"
 
 
-GLOB_CALL_RE = re.compile(r'circleci tests glob "([^"]+)"')
 KEYWORD_RE = re.compile(r"-k\s+\\?[\"']([^\"'\\]+)")
 
 
@@ -316,16 +317,14 @@ def _keyword_terms(
 
 
 def _slices() -> tuple[Slice, ...]:
-    if not CIRCLECI_CONFIG.exists():
-        return ()
     test_files: Final = frozenset(_test_files())
-    jobs: Final = yaml.safe_load(CIRCLECI_CONFIG.read_text()).get("jobs", {})
     return tuple(
-        Slice(job=job, globs=globs, named=named, required=required, excluded=excluded, understood=understood)
-        for job, body in jobs.items()
+        Slice(job=f"{path.name}:{job}", globs=globs, named=named, required=required, excluded=excluded, understood=understood)
+        for path in _config_files()
+        for job, body in yaml.safe_load(path.read_text()).get("jobs", {}).items()
         for text in ("\n".join(_strings(body)),)
         if "pytest" in text
-        for globs in (tuple(GLOB_CALL_RE.findall(text)),)
+        for globs in (tuple(token for token in TEST_TOKEN_RE.findall(text) if GLOB_CHARS & set(token)),)
         for named in (frozenset(TEST_TOKEN_RE.findall(text)) & test_files,)
         for required, excluded, understood in (
             _keyword_terms(tuple(KEYWORD_RE.findall(text)), attributable=len(globs) < 2),
@@ -352,16 +351,11 @@ def _matchable_names(relative_path: str) -> frozenset[str]:
 
 
 def _workflow_named_tokens() -> frozenset[str]:
-    """Test tokens a GitHub Actions job names directly.
-
-    A CircleCI `-k` that deselects a file no longer means the file runs nowhere once a
-    workflow names it, so the slice check has to credit those the same way the census does.
-    """
     return _invoked_test_tokens(
         scalar
         for path in _config_files()
-        if path != CIRCLECI_CONFIG
         for scalar in _scalars(yaml.safe_load(path.read_text(encoding="utf-8")), path.name)
+        if not KEYWORD_RE.search(scalar.value)
     )
 
 
@@ -566,13 +560,28 @@ def _owns(workflow: str, relative_path: str) -> bool:
     return owned_root is not None and relative_path.startswith(owned_root)
 
 
+def _expression_suites(value: str) -> tuple[str, ...]:
+    suites: Final = chain.from_iterable(map(json.loads, re.findall(r"'(\[[^']*\])'", value)))
+    return tuple(suite for suite in suites if isinstance(suite, str) and "{" not in suite)
+
+
+def _matrix_suites(value: object) -> tuple[str, ...]:
+    if isinstance(value, list):
+        return tuple(suite for suite in value if isinstance(suite, str))
+    if isinstance(value, str):
+        return _expression_suites(value)
+    return ()
+
+
 def _tokens_outside_owner(relative_path: str, workflow_tokens: Mapping[str, frozenset[str]]) -> frozenset[str]:
     return frozenset[str]().union(
         *(tokens for workflow, tokens in workflow_tokens.items() if not _owns(workflow, relative_path))
     )
 
 
-def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozenset[str], tuple[Finding, ...]]:
+def _integration_ownership(
+    repo_root: pathlib.Path = REPO_ROOT, *, allowlist: Allowlist | None = None
+) -> tuple[frozenset[str], tuple[Finding, ...]]:
     runner: Final = repo_root / "tests/integration/run.py"
     if not runner.exists():
         return frozenset(), ()
@@ -589,26 +598,19 @@ def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozens
     browser_manifest: Final = repo_root / "tests/e2e/ui/tests/integrationCritical/expected.json"
     browser_nodes: Final = json.loads(browser_manifest.read_text()) if browser_manifest.exists() else ()
     browser_paths: Final = frozenset(node.split("::", 1)[0] for node in browser_nodes)
-    circle_path: Final = repo_root / ".circleci/config.yml"
-    circle: Final = yaml.safe_load(circle_path.read_text()) if circle_path.exists() else {}
-    circle_test_path_tokens: Final = _invoked_test_tokens(
-        scalar for scalar in _scalars(circle, "config.yml") if scalar.key == "test_path"
-    )
-    steps: Final = circle.get("jobs", {}).get("integration_contracts", {}).get("steps", ())
+    integration_path: Final = repo_root / ".github/workflows/test-integration.yml"
+    integration: Final = yaml.safe_load(integration_path.read_text()) if integration_path.exists() else {}
+    integration_jobs: Final = integration.get("jobs", {})
     invoked: Final = any(
-        ".circleci/scripts/run_integration.sh" in scalar.value
-        for scalar in _scalars(steps, "integration_contracts")
-        if scalar.key == "command"
+        ".github/scripts/ci/run_integration.sh" in scalar.value
+        for scalar in _scalars(integration_jobs, "integration")
+        if scalar.key == "run"
     )
     scheduled: Final = frozenset(
         suite
-        for job in circle.get("workflows", {}).get("integration", {}).get("jobs", ())
-        if isinstance(job, dict) and "integration_contracts" in job
-        for suite in job["integration_contracts"]
-        .get("matrix", {})
-        .get("parameters", {})
-        .get("suite", (job["integration_contracts"].get("suite"),))
-        if isinstance(suite, str)
+        for body in integration_jobs.values()
+        if any(".github/scripts/ci/run_integration.sh" in scalar.value for scalar in _scalars(body, "integration"))
+        for suite in _matrix_suites(body.get("strategy", {}).get("matrix", {}).get("suite", ()))
     )
     required: Final = (frozenset({"browser"}) if browser_paths else frozenset()) | frozenset(
         group
@@ -630,19 +632,27 @@ def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozens
         }
     )
     gha_tokens: Final = frozenset[str]().union(*workflow_tokens.values())
-    circle_owned: Final = (
+    invoked_github_files: Final = frozenset(
+        path for path in github_files if any(_token_covers(token, path) for token in gha_tokens)
+    )
+    grouped: Final = (
         frozenset(str(path.relative_to(repo_root)) for path in integration_root.rglob("test_*.py")) - github_files
     )
     findings: Final = (
         tuple(
             Finding(path, "integration file outside GITHUB_FILES is selected by GitHub Actions")
-            for path in sorted(circle_owned)
-            if any(_token_covers(token, path) for token in gha_tokens)
+            for path in sorted(grouped)
+            if any(
+                _token_covers(token, path)
+                for workflow, tokens in workflow_tokens.items()
+                if workflow != "test-integration.yml"
+                for token in tokens
+            )
         )
         + tuple(
             Finding(path, "GitHub-owned integration contract has no invoking job")
             for path in sorted(github_files)
-            if not any(_token_covers(token, path) for token in gha_tokens | circle_test_path_tokens)
+            if path not in invoked_github_files and not (allowlist is not None and allowlist.covers_test(path))
         )
         + tuple(
             Finding(path, "GitHub-owned integration file is missing")
@@ -658,6 +668,7 @@ def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozens
     browser_commands: Final = tuple(
         scalar.value
         for path in (repo_root / ".github/workflows").glob("*.y*ml")
+        if path.name != "test-integration.yml"
         for scalar in _scalars(yaml.safe_load(path.read_text()), path.name)
         if scalar.key in {"run", "command"}
     )
@@ -690,14 +701,14 @@ def _integration_ownership(repo_root: pathlib.Path = REPO_ROOT) -> tuple[frozens
         else ()
     )
     group_findings: Final = tuple(
-        Finding(group, "canonical integration group is not scheduled by CircleCI")
+        Finding(group, "canonical integration group is not scheduled by GitHub Actions")
         for group in sorted(required - scheduled)
     ) + tuple(Finding(path, "canonical node must have exactly one integration group") for path in sorted(ungrouped))
     if not paths or not invoked or not scheduled:
         return frozenset(), findings + (
-            Finding(str(runner.relative_to(repo_root)), "dedicated CircleCI runner is missing"),
+            Finding(str(runner.relative_to(repo_root)), "dedicated GitHub Actions runner is missing"),
         )
-    return paths | browser_paths | github_files, findings + group_findings + browser_findings + exclusion_findings
+    return paths | browser_paths | invoked_github_files, findings + group_findings + browser_findings + exclusion_findings
 
 
 def main() -> int:
@@ -709,7 +720,7 @@ def main() -> int:
     allowlist = _load_allowlist()
     scalars = _all_scalars()
 
-    integration_paths, ownership_findings = _integration_ownership()
+    integration_paths, ownership_findings = _integration_ownership(allowlist=allowlist)
     test_findings = _uncovered_tests(
         allowlist,
         _invoked_selections(scalars) + (Selection(included=integration_paths, ignored=frozenset()),),
@@ -731,8 +742,8 @@ def main() -> int:
         )
     if ownership_findings:
         browser_remedy = (
-            " CircleCI's browser group owns the integrationCritical Playwright contracts, so no GitHub workflow may "
-            "select them and the default Playwright config must exclude them."
+            " The browser group in test-integration.yml owns the integrationCritical Playwright contracts. "
+            "Other workflows and the default Playwright config must exclude them."
             if any("browser" in finding.detail or "Playwright" in finding.detail for finding in ownership_findings)
             else ""
         )

.github/scripts/ci/classify_changes.sh

@@ -14,22 +14,22 @@ outside_cost_map_set=false
 while IFS= read -r file || [ -n "$file" ]; do
   [ -n "$file" ] || continue
   case "$file" in
-    litellm/_redis.py | litellm/_redis_credential_provider.py | litellm/caching/redis_cache.py | litellm/caching/evicted_client_closer.py | tests/unit/test_redis.py | tests/integration/sdk/test_redis_cluster_iam_auth.py | tests/unit/caching/test_redis_connection_pool.py | tests/unit/caching/test_redis_cluster_cache.py | tests/unit/caching/test_evicted_client_closer.py | .github/workflows/test-unit.yml | .github/actions/detect-changes/* | .github/scripts/detect_changes.sh | .circleci/scripts/classify_changes.sh | tests/integration/run.py | pyproject.toml | uv.lock)
+    litellm/_redis.py | litellm/_redis_credential_provider.py | litellm/caching/redis_cache.py | litellm/caching/evicted_client_closer.py | tests/unit/test_redis.py | tests/integration/sdk/test_redis_cluster_iam_auth.py | tests/unit/caching/test_redis_connection_pool.py | tests/unit/caching/test_redis_cluster_cache.py | tests/unit/caching/test_evicted_client_closer.py | .github/workflows/test-unit.yml | .github/actions/detect-changes/* | .github/scripts/detect_changes.sh | .github/scripts/ci/classify_changes.sh | tests/integration/run.py | pyproject.toml | uv.lock)
       has_redis_compat=true ;;
   esac
   case "$file" in
     *.md | *.mdx) : ;;
-    pyproject.toml | */pyproject.toml | uv.lock | uv.toml | .python-version | rust-toolchain.toml | litellm-rust/* | litellm/__init__.py | litellm/proxy/proxy_server.py | litellm/*mcp* | tests/*mcp* | litellm/integrations/arize/* | tests/base_sdk_tests/* | scripts/check_mcp_sdk_install.py | .github/workflows/test-mcp-dependency-resolution.yml | .github/actions/detect-changes/* | .github/actions/setup-uv-with-retries/* | .github/actions/cache-cargo-build/* | .github/scripts/detect_changes.sh | .github/scripts/uv_sync_with_retries.sh | .circleci/scripts/classify_changes.sh | tests/unit/test_circleci_path_filter.py | tests/unit/test_detect_changes.py)
+    pyproject.toml | */pyproject.toml | uv.lock | uv.toml | .python-version | litellm/rust/* | litellm/__init__.py | litellm/proxy/proxy_server.py | litellm/*mcp* | tests/*mcp* | litellm/integrations/arize/* | tests/base_sdk_tests/* | litellm/scripts/check_mcp_sdk_install.py | .github/workflows/test-mcp-dependency-resolution.yml | .github/actions/detect-changes/* | .github/actions/setup-uv-with-retries/* | .github/actions/cache-cargo-build/* | .github/scripts/detect_changes.sh | .github/scripts/uv_sync_with_retries.sh | .github/scripts/ci/classify_changes.sh | tests/unit/test_ci_path_filter.py | tests/unit/test_detect_changes.py)
       has_mcp_dependencies=true ;;
   esac
   case "$file" in
-    litellm-rust/* | litellm/rust_bridge/* | rust-toolchain.toml | pyproject.toml | uv.lock | tests/windows_tests/* | .circleci/*)
+    litellm/rust/* | litellm/rust_bridge/* | pyproject.toml | uv.lock | tests/windows_tests/* | .github/scripts/ci/* | .github/actions/*)
       has_windows_release=true ;;
   esac
   case "$file" in
-    ui/* | tests/e2e/ui/*) has_client=true ;;
-    docs/* | *.md | *.mdx) : ;;
-    .github/* | .circleci/*) has_ci=true; has_backend=true ;;
+    litellm/ui/* | tests/e2e/ui/*) has_client=true ;;
+    *.md | *.mdx) : ;;
+    .github/*) has_ci=true; has_backend=true ;;
     *) has_backend=true ;;
   esac
   case "$file" in

.github/scripts/ci/path_filter.sh

@@ -6,10 +6,12 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 run_full() {
   echo "path-filter[$category]: running job ($1)"
+  echo "decision=run"
+  [ -z "${GITHUB_OUTPUT:-}" ] || echo "decision=run" >> "$GITHUB_OUTPUT"
   exit 0
 }
 
-[ -n "${CIRCLE_PULL_REQUEST:-}" ] || run_full "not a pull request"
+[ "${GITHUB_EVENT_NAME:-}" = pull_request ] || run_full "not a pull request"
 
 candidate_bases="${PATH_FILTER_BASE_BRANCH:-main}"
 merge_base=""
@@ -36,5 +38,6 @@ if [ "$decision" = run ]; then
   run_full "$category-relevant changes detected"
 fi
 
-echo "path-filter[$category]: only unrelated changes detected; halting job as successful"
-circleci-agent step halt
+echo "path-filter[$category]: only unrelated changes detected"
+echo "decision=skip"
+[ -z "${GITHUB_OUTPUT:-}" ] || echo "decision=skip" >> "$GITHUB_OUTPUT"

.github/scripts/ci/run_integration.sh

@@ -30,7 +30,7 @@ egress_cgroup=litellm-integration
 cleanup() {
   original_status=$?
   trap - EXIT INT TERM
-  sudo .venv/bin/python .circleci/scripts/stop_integration_processes.py \
+  sudo .venv/bin/python .github/scripts/ci/stop_integration_processes.py \
     "$integration_identity" "$(id -u)" "$proxy_pid" "$peer_pid" "$upstream_pid" \
     > "$results/process-cleanup.txt" 2>&1 || original_status=1
   for owned_pid in "$peer_pid" "$proxy_pid" "$upstream_pid"; do
@@ -81,10 +81,10 @@ export INTEGRATION_UPSTREAM_URL=http://127.0.0.1:8190
 export INTEGRATION_MASTER_KEY="$LITELLM_MASTER_KEY"
 export LITELLM_UI_PATH="$PWD/litellm/proxy/_experimental/out"
 if [ "$suite" = browser ]; then
-  export LITELLM_UI_PATH="$PWD/ui/litellm-dashboard/out"
+  export LITELLM_UI_PATH="$PWD/litellm/ui/litellm-dashboard/out"
   test -f "$LITELLM_UI_PATH/index.html"
 fi
-export INTEGRATION_SEED="$(.venv/bin/python -c 'import hashlib,os; print(int(hashlib.sha256(((os.environ.get("CIRCLE_SHA1") or os.environ.get("GITHUB_SHA") or "local") + (os.environ.get("CIRCLE_WORKFLOW_ID") or os.environ.get("GITHUB_RUN_ID") or "local")).encode()).hexdigest()[:8],16))')"
+export INTEGRATION_SEED="$(.venv/bin/python -c 'import hashlib,os; print(int(hashlib.sha256(((os.environ.get("GITHUB_SHA") or "local") + (os.environ.get("GITHUB_RUN_ID") or "local")).encode()).hexdigest()[:8],16))')"
 export INTEGRATION_ORDER_SEED="$INTEGRATION_SEED"
 
 uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma > "$results/prisma-generate.log" 2>&1
@@ -93,7 +93,7 @@ export INTEGRATION_PROXY_DATABASE_URL=""
 export INTEGRATION_PROXY_READ_REPLICA_URL=""
 export INTEGRATION_ROUTING=""
 if [ "$mode" = replica ] || [ "$mode" = parity ]; then
-  .venv/bin/python .circleci/scripts/prepare_replica_roles.py > "$results/prepare-replica-roles.log" 2>&1
+  .venv/bin/python .github/scripts/ci/prepare_replica_roles.py > "$results/prepare-replica-roles.log" 2>&1
   export INTEGRATION_PROXY_DATABASE_URL="postgresql://litellm_writer:litellm-writer@127.0.0.1:5432/circle_test"
   export INTEGRATION_PROXY_READ_REPLICA_URL="postgresql://litellm_reader:litellm-reader@127.0.0.1:5432/circle_test"
 fi
@@ -202,7 +202,7 @@ start_proxy() {
 if [ "$isolated_workers" = false ]; then
   start_proxy 4000 proxy.log
   proxy_pid="$launched_pid"
-  .venv/bin/python .circleci/scripts/wait_integration_services.py
+  .venv/bin/python .github/scripts/ci/wait_integration_services.py
   curl --noproxy '*' -sSf -X POST "$INTEGRATION_PROXY_URL/config/update" \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"router_settings": {"num_retries": 0}}' > "$results/seed-router-settings.json"
@@ -210,7 +210,7 @@ if [ "$isolated_workers" = false ]; then
     export INTEGRATION_PEER_URL=http://127.0.0.1:4001
     start_proxy 4001 peer.log
     peer_pid="$launched_pid"
-    .venv/bin/python .circleci/scripts/wait_integration_services.py
+    .venv/bin/python .github/scripts/ci/wait_integration_services.py
   fi
 fi
 
@@ -233,14 +233,13 @@ if [ "$suite" = browser ]; then
     E2E_UI_BASE_URL="$E2E_UI_BASE_URL" E2E_UI_ARTIFACT_DIR="$E2E_UI_ARTIFACT_DIR" \
     LITELLM_MASTER_KEY="$LITELLM_MASTER_KEY" CI=true \
     node tests/e2e/ui/node_modules/@playwright/test/cli.js test --config tests/e2e/ui/integration.config.ts
-  .venv/bin/python .circleci/scripts/verify_integration_browser.py "$results/browser-results.json"
+  .venv/bin/python .github/scripts/ci/verify_integration_browser.py "$results/browser-results.json"
   exit 0
 fi
 
 shard_arguments=()
-if [ "${CIRCLE_NODE_TOTAL:-1}" -gt 1 ]; then
-  .venv/bin/python tests/integration/run.py "$suite" --list \
-    | circleci tests split --split-by=timings --timings-type=filename > "$results/node-files.txt"
+if [ -n "${INTEGRATION_SHARD_FILES:-}" ]; then
+  cp "$INTEGRATION_SHARD_FILES" "$results/node-files.txt"
   shard_arguments=(--shard-files "$results/node-files.txt")
 fi
 
@@ -265,7 +264,7 @@ env -i PATH="$PATH" HOME="$HOME" PYTHONPATH="$PYTHONPATH" \
   "${license_env[@]}" \
   INTEGRATION_SEED="$INTEGRATION_SEED" \
   INTEGRATION_ORDER_SEED="$INTEGRATION_ORDER_SEED" \
-  INTEGRATION_CASE_SHARDS="${CIRCLE_NODE_TOTAL:-1}" INTEGRATION_CASE_SHARD_INDEX="${CIRCLE_NODE_INDEX:-0}" \
+  INTEGRATION_CASE_SHARDS="${INTEGRATION_CASE_SHARDS:-1}" INTEGRATION_CASE_SHARD_INDEX="${INTEGRATION_CASE_SHARD_INDEX:-0}" \
   LITELLM_LOCAL_MODEL_COST_MAP=True AWS_EC2_METADATA_DISABLED=true DO_NOT_TRACK=1 \
   INTEGRATION_PROXY_DATABASE_URL="$INTEGRATION_PROXY_DATABASE_URL" \
   INTEGRATION_PROXY_READ_REPLICA_URL="$INTEGRATION_PROXY_READ_REPLICA_URL" \

.github/scripts/ci/verify_integration_browser.py

@@ -31,7 +31,7 @@ def main() -> None:
     result: Final = json.loads(Path(sys.argv[1]).read_text())
     assert not result.get("errors"), result.get("errors")
     expected: Final = json.loads(
-        (Path(__file__).resolve().parents[2] / "tests/e2e/ui/tests/integrationCritical/expected.json").read_text()
+        (Path(__file__).resolve().parents[3] / "tests/e2e/ui/tests/integrationCritical/expected.json").read_text()
     )
     assert expected and result["stats"]["expected"] == len(expected)
     assert all(result["stats"][name] == 0 for name in ("unexpected", "flaky", "skipped"))
@@ -41,7 +41,7 @@ def main() -> None:
 
     suites: Final = TypeAdapter(list[BrowserSuite]).validate_python(result["suites"], strict=True)
     specs: Final = tuple(spec for suite in suites for spec in cases(suite))
-    repository: Final = Path(__file__).resolve().parents[2]
+    repository: Final = Path(__file__).resolve().parents[3]
     report_root: Final = Path(result["config"]["rootDir"])
     assert report_root.is_absolute(), "Playwright rootDir must be explicit"
     observed: Final = tuple(

.github/scripts/detect_changes.sh

@@ -16,7 +16,7 @@ run_full() {
 }
 
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-classify="${here}/../../.circleci/scripts/classify_changes.sh"
+classify="${here}/ci/classify_changes.sh"
 
 [ -n "${PR_NUMBER:-}" ] || run_full "not a pull_request event"
 [ -n "${REPO:-}" ] || run_full "no repository in the environment"

.github/scripts/setup_rust_environment.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+root=$(git rev-parse --show-toplevel)
+toolchain=$(sed -n 's/^[[:space:]]*channel[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' "$root/litellm/rust/rust-toolchain.toml")
+test -n "$toolchain"
+rustup toolchain install "$toolchain" --profile minimal --component rustfmt --component clippy --no-self-update
+{
+  echo "RUSTUP_TOOLCHAIN=$toolchain"
+  echo "CARGO_NET_RETRY=5"
+  echo "CARGO_HTTP_MULTIPLEXING=false"
+  echo "SQLX_OFFLINE=${SQLX_OFFLINE:-true}"
+} >> "${GITHUB_ENV:?GITHUB_ENV is required}"

.github/scripts/verify_linux_native_wheel.py

@@ -128,6 +128,20 @@ def main(
             member.filename
             for member in wheel_members
             if member.filename.endswith((".pdb", ".dwp", ".rlib", ".rmeta", "Cargo.toml", "Cargo.lock"))
+            or member.filename.startswith(
+                (
+                    "litellm/deploy/",
+                    "litellm/ui/",
+                    "litellm/vscode-extension/",
+                    "litellm/enterprise/",
+                    "litellm/proxy-extras/",
+                    "litellm/rust/",
+                    "litellm/packaging/",
+                    "litellm/scripts/",
+                    "litellm/runtime/",
+                    "litellm/schema.prisma",
+                )
+            )
             or any(part.endswith(".dSYM") for part in PurePosixPath(member.filename).parts)
         )
         native_member: Final = native_members[0]
@@ -214,7 +228,7 @@ def main(
     native_module: Final = load_native_module(native_path)
     native_module_loads: Final = native_module is not None
     panic_test_hook_absent: Final = native_module is not None and not hasattr(native_module, "_panic_for_test")
-    native_size_limit: Final = 48_000_000
+    native_size_limit: Final = 55_000_000
     native_size_within_limit: Final = native_member.file_size <= native_size_limit
     validations: Final = (
         (f"Python tag is {EXPECTED_PYTHON_TAG}", python_tag == EXPECTED_PYTHON_TAG),

.github/semgrep/rules/README.md

@@ -5,13 +5,13 @@ Add custom rule YAML files here. Semgrep loads all `.yml`/`.yaml` files under th
 **Run only custom rules (CI / fail on findings):**
 
 ```bash
-semgrep scan --config .semgrep/rules . --error
+semgrep scan --config .github/semgrep/rules . --error
 ```
 
 **Run with registry + custom rules:**
 
 ```bash
-semgrep scan --config auto --config .semgrep/rules .
+semgrep scan --config auto --config .github/semgrep/rules .
 ```
 
 **Layout:**

.github/workflows/auto-close-duplicates.yml

@@ -16,8 +16,8 @@ on:
   pull_request:
     paths:
       - .github/workflows/auto-close-duplicates.yml
-      - scripts/auto-close-duplicates.ts
-      - scripts/auto-close-duplicates.test.ts
+      - litellm/scripts/auto-close-duplicates.ts
+      - litellm/scripts/auto-close-duplicates.test.ts
 
 permissions: {}
 
@@ -40,7 +40,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Test the sweep
-        run: bun test scripts/auto-close-duplicates.test.ts
+        run: bun test litellm/scripts/auto-close-duplicates.test.ts
 
   sweep:
     if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm'
@@ -62,7 +62,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Close unanswered duplicates, reopen ones the reporter answered
-        run: bun run scripts/auto-close-duplicates.ts
+        run: bun run litellm/scripts/auto-close-duplicates.ts
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           DRY_RUN: ${{ inputs.dry_run == true }}

.github/workflows/check-schema-sync.yml

@@ -3,9 +3,9 @@ name: Check Schema Sync
 on:
   pull_request:
     paths:
-      - 'schema.prisma'
+      - 'litellm/schema.prisma'
       - 'litellm/proxy/schema.prisma'
-      - 'litellm-proxy-extras/litellm_proxy_extras/schema.prisma'
+      - 'litellm/proxy-extras/litellm_proxy_extras/schema.prisma'
 
 permissions:
   contents: read
@@ -16,7 +16,7 @@ concurrency:
 
 jobs:
   check-sync:
-    name: Verify schema.prisma copies match root
+    name: Verify litellm/schema.prisma copies match the canonical schema
     runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
@@ -27,7 +27,7 @@ jobs:
 
       - name: Reject symlinked schema files
         run: |
-          for f in schema.prisma litellm/proxy/schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma; do
+          for f in litellm/schema.prisma litellm/proxy/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma; do
             if [ -L "$f" ]; then
               echo "::error file=$f::$f is a symlink, which is not allowed"
               exit 1
@@ -38,24 +38,24 @@ jobs:
         run: |
           EXIT=0
 
-          diff schema.prisma litellm/proxy/schema.prisma || {
-            echo "::error file=litellm/proxy/schema.prisma::litellm/proxy/schema.prisma differs from root schema.prisma"
+          diff litellm/schema.prisma litellm/proxy/schema.prisma || {
+            echo "::error file=litellm/proxy/schema.prisma::litellm/proxy/schema.prisma differs from root litellm/schema.prisma"
             EXIT=1
           }
 
-          diff schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma || {
-            echo "::error file=litellm-proxy-extras/litellm_proxy_extras/schema.prisma::litellm-proxy-extras/litellm_proxy_extras/schema.prisma differs from root schema.prisma"
+          diff litellm/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma || {
+            echo "::error file=litellm/proxy-extras/litellm_proxy_extras/schema.prisma::litellm/proxy-extras/litellm_proxy_extras/schema.prisma differs from root litellm/schema.prisma"
             EXIT=1
           }
 
           if [ "$EXIT" -ne 0 ]; then
             echo ""
             echo "Schema files are out of sync."
-            echo "The root schema.prisma is the source of truth."
+            echo "The root litellm/schema.prisma is the source of truth."
             echo ""
             echo "To fix, run from the repo root:"
-            echo "  cp schema.prisma litellm/proxy/schema.prisma"
-            echo "  cp schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma"
+            echo "  cp litellm/schema.prisma litellm/proxy/schema.prisma"
+            echo "  cp litellm/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma"
             exit 1
           fi
 

.github/workflows/compat-matrix-image.yml

@@ -26,17 +26,24 @@ jobs:
         with:
           persist-credentials: false
 
-      - name: Build the Render cron image
-        run: docker build -f tests/e2e/claude_code/cron_vm/Dockerfile -t compat-matrix:${{ github.sha }} tests/e2e
-
-      - name: Resolve and install the Claude Code CLI as the cron user
+      - name: Install the pinned uv the cron image ships
+        env:
+          UV_VERSION: 0.10.9
+          UV_SHA256: 20d79708222611fa540b5c9ed84f352bcd3937740e51aacc0f8b15b271c57594
         run: |
-          docker run --rm compat-matrix:${{ github.sha }} bash -c '
-            set -euo pipefail
-            whoami
-            gh --version
-            uv --version
-            version="$(uv run --no-project --python 3.12 python /opt/litellm/tests/e2e/cli_harness/npm_version_resolver.py @anthropic-ai/claude-code)"
-            /opt/litellm/tests/e2e/claude_code/cron_vm/install_claude_code.sh "${version}" /tmp/claude-cli
-            /tmp/claude-cli/claude --version
-          '
+          set -euo pipefail
+          curl -fsSLo "$RUNNER_TEMP/uv.tar.gz" "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-x86_64-unknown-linux-gnu.tar.gz"
+          echo "${UV_SHA256}  $RUNNER_TEMP/uv.tar.gz" | sha256sum -c -
+          mkdir -p "$RUNNER_TEMP/bin"
+          tar -xzf "$RUNNER_TEMP/uv.tar.gz" -C "$RUNNER_TEMP/bin" --strip-components=1 uv-x86_64-unknown-linux-gnu/uv
+          echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
+
+      - name: Resolve and install the Claude Code CLI as an unprivileged user
+        run: |
+          set -euo pipefail
+          whoami
+          gh --version
+          uv --version
+          version="$(uv run --no-project --python 3.12 python tests/e2e/cli_harness/npm_version_resolver.py @anthropic-ai/claude-code)"
+          tests/e2e/claude_code/cron_vm/install_claude_code.sh "${version}" "$RUNNER_TEMP/claude-cli"
+          "$RUNNER_TEMP/claude-cli/claude" --version

.github/workflows/conventional-commits.yml

@@ -3,7 +3,7 @@ name: Conventional PR Title
 # Squash-merge replaces the merge commit subject with the PR title, so
 # enforcing Conventional Commits at the PR-title level is what actually gates
 # the commits that land on the default branch. The local commit-msg hook
-# (.githooks/commit-msg) is a best-effort assist; this workflow is the gate.
+# (litellm/scripts/githooks/commit-msg) is a best-effort assist; this workflow is the gate.
 #
 # See https://www.conventionalcommits.org/en/v1.0.0/
 
@@ -28,7 +28,7 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
-          # Must mirror the type list in .githooks/commit-msg.
+          # Must mirror the type list in litellm/scripts/githooks/commit-msg.
           types: |
             feat
             fix

.github/workflows/cost-map-guard.yml

@@ -49,7 +49,15 @@ jobs:
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
           HEAD_REF: ${{ github.event.pull_request.head.ref }}
         run: |
-          uv run --no-project python ci_cd/cost_map_guard.py --base "$MERGE_BASE" --head "$HEAD_SHA" --head-ref "$HEAD_REF"
+          guard=litellm/scripts/ci_cd/cost_map_guard.py
+          # TODO: Drop the legacy path after this lands on main.
+          if [ ! -f "$guard" ]; then
+            guard=scripts/ci_cd/cost_map_guard.py
+          fi
+          if [ ! -f "$guard" ]; then
+            guard=ci_cd/cost_map_guard.py
+          fi
+          uv run --no-project python "$guard" --base "$MERGE_BASE" --head "$HEAD_SHA" --head-ref "$HEAD_REF"
       - name: Check whether the pull request changes the cost map
         id: changed
         env:
@@ -69,5 +77,5 @@ jobs:
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
           git show "${HEAD_SHA}:model_prices_and_context_window.json" > "$RUNNER_TEMP/model_prices_and_context_window.json"
-          cargo run --locked --quiet --manifest-path litellm-rust/Cargo.toml -p litellm-model-catalog --bin check-registry \
+          cargo run --locked --quiet --manifest-path litellm/rust/Cargo.toml -p litellm-model-catalog --bin check-registry \
             -- "$RUNNER_TEMP/model_prices_and_context_window.json"

.github/workflows/cost-map-pins-recheck.yml

@@ -8,7 +8,7 @@ on:
     paths:
       - model_prices_and_context_window.json
       - litellm/model_prices_and_context_window_backup.json
-      - ci_cd/cost_map_pins.json
+      - litellm/scripts/ci_cd/cost_map_pins.json
 
 permissions: {}
 
@@ -28,8 +28,8 @@ jobs:
           BEFORE_SHA: ${{ github.event.before }}
           AFTER_SHA: ${{ github.event.after }}
           FORCED: ${{ github.event.forced }}
-          COST_MAP_PATHS: '["model_prices_and_context_window.json", "litellm/model_prices_and_context_window_backup.json", "ci_cd/cost_map_pins.json"]'
-          PINS_PATHS: '["ci_cd/cost_map_pins.json"]'
+          COST_MAP_PATHS: '["model_prices_and_context_window.json", "litellm/model_prices_and_context_window_backup.json", "litellm/scripts/ci_cd/cost_map_pins.json"]'
+          PINS_PATHS: '["litellm/scripts/ci_cd/cost_map_pins.json"]'
         run: |
           if [[ "$BEFORE_SHA" =~ ^0+$ || "$FORCED" == true ]] \
             || ! gh api "repos/${GITHUB_REPOSITORY}/compare/${BEFORE_SHA}...${AFTER_SHA}?per_page=1" \

.github/workflows/duplicate_issue_check.yml

@@ -13,9 +13,9 @@ on:
       - .github/workflows/duplicate_issue_check.yml
       - .github/prompts/duplicate-issue-check.md
       - .github/prompts/duplicate-issue-check.schema.json
-      - scripts/flag-duplicate-issue.ts
-      - scripts/flag-duplicate-issue.test.ts
-      - scripts/auto-close-duplicates.ts
+      - litellm/scripts/flag-duplicate-issue.ts
+      - litellm/scripts/flag-duplicate-issue.test.ts
+      - litellm/scripts/auto-close-duplicates.ts
 
 permissions: {}
 
@@ -38,7 +38,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Test the flag step
-        run: bun test scripts/flag-duplicate-issue.test.ts
+        run: bun test litellm/scripts/flag-duplicate-issue.test.ts
 
   classify:
     if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm'
@@ -125,7 +125,7 @@ jobs:
       - name: Checkout scripts
         uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
-          sparse-checkout: scripts
+          sparse-checkout: litellm/scripts
           persist-credentials: false
 
       - name: Setup Bun
@@ -134,7 +134,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Comment and label
-        run: bun run scripts/flag-duplicate-issue.ts
+        run: bun run litellm/scripts/flag-duplicate-issue.ts
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           VERDICT: ${{ needs.classify.outputs.verdict }}

.github/workflows/guard-fork-dependencies.yml

@@ -8,8 +8,8 @@ on:
     paths:
       - "uv.lock"
       - "pyproject.toml"
-      - "litellm-proxy-extras/pyproject.toml"
-      - "enterprise/pyproject.toml"
+      - "litellm/proxy-extras/pyproject.toml"
+      - "litellm/enterprise/pyproject.toml"
 
 permissions: {}
 
@@ -130,8 +130,12 @@ jobs:
           }
 
           check_deps base/pyproject.toml pr/pyproject.toml "pyproject.toml" || had_error=1
-          check_deps base/litellm-proxy-extras/pyproject.toml pr/litellm-proxy-extras/pyproject.toml "litellm-proxy-extras/pyproject.toml" || had_error=1
-          check_deps base/enterprise/pyproject.toml pr/enterprise/pyproject.toml "enterprise/pyproject.toml" || had_error=1
+          base_extras=base/litellm/proxy-extras/pyproject.toml
+          [ -f "$base_extras" ] || base_extras=base/litellm-proxy-extras/pyproject.toml
+          base_enterprise=base/litellm/enterprise/pyproject.toml
+          [ -f "$base_enterprise" ] || base_enterprise=base/enterprise/pyproject.toml
+          check_deps "$base_extras" pr/litellm/proxy-extras/pyproject.toml "litellm/proxy-extras/pyproject.toml" || had_error=1
+          check_deps "$base_enterprise" pr/litellm/enterprise/pyproject.toml "litellm/enterprise/pyproject.toml" || had_error=1
 
           if [ "$had_error" -ne 0 ]; then
             echo ""

.github/workflows/image-scan.yml

@@ -7,34 +7,29 @@ on:
       - litellm_oss_branch
       - "litellm_**"
     paths:
-      - Dockerfile
-      - docker/Dockerfile.non_root
-      - docker/Dockerfile.database
-      - migrations/Dockerfile
-      - migrations/run.py
-      - gateway/Dockerfile
-      - gateway/launch.py
-      - gateway/main.py
-      - gateway/routes/allowlist.py
-      - litellm_cli/**
-      - backend/Dockerfile
-      - backend/main.py
-      - litellm-rust/**
-      - docker/component_entrypoint.sh
-      - docker/entrypoint.sh
+      - litellm/deploy/docker/Dockerfile
+      - litellm/deploy/docker/Dockerfile.dockerignore
+      - litellm/deploy/docker/docker-entrypoint.sh
+      - litellm/runtime/migrations/run.py
+      - litellm/runtime/gateway/main.py
+      - litellm/runtime/gateway/launch.py
+      - litellm/runtime/gateway/routes/allowlist.py
+      - litellm/cli/**
+      - .github/packaging/**
+      - litellm/runtime/backend/main.py
+      - litellm/runtime/backend/routes/allowlist.py
+      - litellm/rust/**
       - litellm/proxy/prisma_migration.py
       - litellm/proxy/admin_mcp.py
       - litellm/proxy/proxy_server.py
-      - backend/routes/allowlist.py
       - pyproject.toml
-      - litellm-proxy-extras/**
+      - litellm/proxy-extras/**
       - tests/integration/database/image/**
       - uv.lock
-      - ui/litellm-dashboard/package-lock.json
-      - ui/Dockerfile
-      - ui/nginx.conf
+      - litellm/ui/litellm-dashboard/package-lock.json
+      - litellm/ui/nginx.conf
       - .github/workflows/image-scan.yml
-      - .grype.yaml
+      - .github/grype.yaml
   schedule:
     - cron: "41 6 * * *"
   workflow_dispatch:
@@ -45,9 +40,13 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
+env:
+  IMAGE: litellm-image-scan:${{ github.sha }}
+  ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
+
 jobs:
-  image-scan:
-    name: image-scan
+  build:
+    name: build
     runs-on: ubuntu-latest
     if: >-
       github.event_name != 'pull_request' ||
@@ -60,7 +59,129 @@ jobs:
         with:
           persist-credentials: false
 
+      - name: Build the image
+        run: docker build -f litellm/deploy/docker/Dockerfile --build-arg MATURIN_PEP517_ARGS="--profile ci" --build-arg LITELLM_RELEASE_TAG=v0.0.0-ci -t "$IMAGE" .
+
+      - name: Save the image for the verify matrix
+        run: docker save "$IMAGE" | zstd -T0 -3 -o "$RUNNER_TEMP/litellm-image.tar.zst"
+
+      - uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
+        with:
+          name: litellm-image
+          path: ${{ runner.temp }}/litellm-image.tar.zst
+          retention-days: 1
+          compression-level: 0
+
+      - name: Build the Admin MCP schema image
+        run: docker build -f litellm/deploy/docker/Dockerfile --build-arg MATURIN_PEP517_ARGS="--profile ci" --build-arg LITELLM_RELEASE_TAG=v0.0.0-ci --target builder -t "$ADMIN_MCP_SCHEMA_IMAGE" .
+
+      - name: Save the Admin MCP schema image for the verify matrix
+        run: docker save "$ADMIN_MCP_SCHEMA_IMAGE" | zstd -T0 -3 -o "$RUNNER_TEMP/litellm-admin-mcp-schema.tar.zst"
+
+      - uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
+        with:
+          name: litellm-admin-mcp-schema
+          path: ${{ runner.temp }}/litellm-admin-mcp-schema.tar.zst
+          retention-days: 1
+          compression-level: 0
+
+  verify:
+    name: verify (${{ matrix.check }})
+    needs: build
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        check: [scan, migrations, gateway, backend, ui, collector]
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          name: litellm-image
+          path: ${{ runner.temp }}
+
+      - name: Load the image
+        run: zstd -d --stdout "$RUNNER_TEMP/litellm-image.tar.zst" | docker load
+
+      - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        if: matrix.check == 'migrations' || matrix.check == 'backend'
+        with:
+          name: litellm-admin-mcp-schema
+          path: ${{ runner.temp }}
+
+      - name: Load the Admin MCP schema image
+        if: matrix.check == 'migrations' || matrix.check == 'backend'
+        run: zstd -d --stdout "$RUNNER_TEMP/litellm-admin-mcp-schema.tar.zst" | docker load
+
+      - name: Set up Python
+        if: matrix.check != 'scan'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Install pytest
+        if: matrix.check != 'scan'
+        run: python -m pip install "pytest==9.0.3"
+
+      # The prisma bake must migrate a fresh DB with no egress as an arbitrary
+      # non-root uid (OpenShift restricted-v2 / air-gapped / readOnlyFilesystem).
+      # `docker run` as the default uid with network hides a broken bake because
+      # the migration entrypoint exits 0 even when it applied nothing; asserting
+      # the schema was created is what catches it.
+      - name: Verify offline migration as a non-root uid
+        if: matrix.check == 'migrations'
+        env:
+          LITELLM_IMAGE: ${{ env.IMAGE }}
+          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: ${{ env.ADMIN_MCP_SCHEMA_IMAGE }}
+        run: |
+          python -m pytest --noconftest -v \
+            tests/integration/database/image/test_offline_image_migration.py \
+            tests/integration/database/image/test_image_bedrock_realtime_extra.py \
+            tests/integration/database/image/test_image_admin_mcp.py
+
+      - name: Verify the gateway component serves offline as a non-root uid
+        if: matrix.check == 'gateway'
+        env:
+          LITELLM_IMAGE: ${{ env.IMAGE }}
+          LITELLM_COMPONENT: gateway
+          LITELLM_IMAGE_COMPONENT: gateway
+        run: |
+          python -m pytest --noconftest -v \
+            tests/integration/database/image/test_component_image_serves_offline.py \
+            tests/integration/database/image/test_image_admin_mcp.py
+
+      - name: Verify the backend component serves offline as a non-root uid
+        if: matrix.check == 'backend'
+        env:
+          LITELLM_IMAGE: ${{ env.IMAGE }}
+          LITELLM_COMPONENT: backend
+          LITELLM_IMAGE_COMPONENT: backend
+          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: ${{ env.ADMIN_MCP_SCHEMA_IMAGE }}
+        run: |
+          python -m pytest --noconftest -v \
+            tests/integration/database/image/test_component_image_serves_offline.py \
+            tests/integration/database/image/test_image_admin_mcp.py
+
+      - name: Verify the ui component serves as an arbitrary uid with a read-only root fs
+        if: matrix.check == 'ui'
+        env:
+          LITELLM_IMAGE: ${{ env.IMAGE }}
+        run: python -m pytest --noconftest -v tests/integration/database/image/test_ui_image_serves_offline.py
+
+      - name: Verify the collector creates its socket as the image user
+        if: matrix.check == 'collector'
+        env:
+          LITELLM_IMAGE: ${{ env.IMAGE }}
+        run: python -m pytest --noconftest -v tests/integration/database/image/test_collector_image_serves_socket.py
+
       - name: Download Grype v0.114.0
+        if: matrix.check == 'scan'
         run: |
           curl -fsSL --retry 3 -o "$RUNNER_TEMP/grype.tar.gz" \
             https://github.com/anchore/grype/releases/download/v0.114.0/grype_0.114.0_linux_amd64.tar.gz
@@ -68,33 +189,6 @@ jobs:
           tar xzf "$RUNNER_TEMP/grype.tar.gz" -C "$RUNNER_TEMP" grype
           chmod +x "$RUNNER_TEMP/grype"
 
-      # Dockerfile.non_root is the rootless variant we ship. The other
-      # Dockerfiles share the same wolfi base and apk set, so OS-layer coverage
-      # is the same; matrix-scan if those variants ever diverge.
-      - name: Build runtime image
-        run: docker build -f docker/Dockerfile.non_root -t litellm-image-scan:${{ github.sha }} .
-
-      - name: Tag the cached builder for Admin MCP schema setup
-        run: docker build --target builder -f docker/Dockerfile.non_root -t litellm-admin-mcp-schema:${{ github.sha }} .
-
-      # The prisma bake must migrate a fresh DB with no egress as an arbitrary
-      # non-root uid (OpenShift restricted-v2 / air-gapped / readOnlyRootFilesystem).
-      # `docker run` as the default uid with network hides a broken bake because
-      # the migration entrypoint exits 0 even when it applied nothing; asserting
-      # the schema was created is what catches it.
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Verify offline migration as a non-root uid
-        env:
-          LITELLM_IMAGE: litellm-image-scan:${{ github.sha }}
-          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
-        run: |
-          python -m pip install "pytest==9.0.3"
-          python -m pytest --noconftest tests/integration/database/image/test_offline_image_migration.py tests/integration/database/image/test_image_bedrock_realtime_extra.py tests/integration/database/image/test_image_admin_mcp.py -v
-
       # Scans the whole shipped artifact: OS/apk plus every language package
       # baked into the image, including ones no lockfile declares (e.g. prisma's
       # vendored node engine) that osv-scan cannot see. osv-scan stays the fast
@@ -102,175 +196,12 @@ jobs:
       # free OSS, run as a pinned, checksum-verified binary; no GitHub Action
       # dependency and no vendor SaaS callout.
       - name: Scan image for fixable HIGH/CRITICAL CVEs
+        if: matrix.check == 'scan'
         env:
           GRYPE_MATCH_PYTHON_USING_CPES: "true"
         run: |
-          "$RUNNER_TEMP/grype" litellm-image-scan:${{ github.sha }} \
-            --config .grype.yaml \
+          "$RUNNER_TEMP/grype" "$IMAGE" \
+            --config .github/grype.yaml \
             --only-fixed \
             --fail-on high \
             --output table
-
-  runtime-image:
-    name: runtime-image (${{ matrix.dockerfile }})
-    runs-on: ubuntu-latest
-    if: >-
-      github.event_name != 'pull_request' ||
-      github.event.pull_request.head.repo.full_name == github.repository
-    timeout-minutes: 45
-    permissions:
-      contents: read
-    strategy:
-      fail-fast: false
-      matrix:
-        dockerfile: [Dockerfile, docker/Dockerfile.database]
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Build runtime image
-        run: docker build --build-arg LITELLM_RELEASE_TAG=v0.0.0-ci -f "${{ matrix.dockerfile }}" -t litellm-runtime-scan:${{ github.sha }} .
-
-      - name: Tag the cached builder for Admin MCP schema setup
-        run: docker build --target builder -f "${{ matrix.dockerfile }}" -t litellm-admin-mcp-schema:${{ github.sha }} .
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Verify offline migration as a non-root uid
-        env:
-          LITELLM_IMAGE: litellm-runtime-scan:${{ github.sha }}
-          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
-        run: |
-          python -m pip install "pytest==9.0.3"
-          python -m pytest --noconftest tests/integration/database/image/test_offline_image_migration.py tests/integration/database/image/test_image_bedrock_realtime_extra.py tests/integration/database/image/test_image_admin_mcp.py -v
-
-
-  migrations-image:
-    name: migrations-image
-    runs-on: ubuntu-latest
-    if: >-
-      github.event_name != 'pull_request' ||
-      github.event.pull_request.head.repo.full_name == github.repository
-    timeout-minutes: 30
-    permissions:
-      contents: read
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Build migrations image
-        run: docker build -f migrations/Dockerfile -t litellm-migrations-scan:${{ github.sha }} .
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Verify offline migration as a non-root uid
-        env:
-          LITELLM_IMAGE: litellm-migrations-scan:${{ github.sha }}
-          LITELLM_MIGRATION_INTERPRETER: python3
-          LITELLM_MIGRATION_SCRIPT: /app/run.py
-        run: |
-          python -m pip install "pytest==9.0.3"
-          python -m pytest --noconftest tests/integration/database/image/test_offline_image_migration.py -v
-
-  gateway-image:
-    name: gateway-image
-    runs-on: ubuntu-latest
-    if: >-
-      github.event_name != 'pull_request' ||
-      github.event.pull_request.head.repo.full_name == github.repository
-    timeout-minutes: 30
-    permissions:
-      contents: read
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Build gateway image
-        run: docker build -f gateway/Dockerfile -t litellm-gateway-scan:${{ github.sha }} .
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Verify the gateway serves offline as a non-root uid
-        env:
-          LITELLM_IMAGE: litellm-gateway-scan:${{ github.sha }}
-          LITELLM_COMPONENT_PORT: "4000"
-          LITELLM_IMAGE_COMPONENT: gateway
-        run: |
-          python -m pip install "pytest==9.0.3"
-          python -m pytest --noconftest tests/integration/database/image/test_component_image_serves_offline.py tests/integration/database/image/test_image_bedrock_realtime_extra.py tests/integration/database/image/test_image_admin_mcp.py -v
-
-  ui-image:
-    name: ui-image
-    runs-on: ubuntu-latest
-    if: >-
-      github.event_name != 'pull_request' ||
-      github.event.pull_request.head.repo.full_name == github.repository
-    timeout-minutes: 30
-    permissions:
-      contents: read
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Build UI image
-        run: docker build -f ui/Dockerfile -t litellm-ui-scan:${{ github.sha }} .
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Verify the UI serves offline as an arbitrary uid with a read-only root fs
-        env:
-          LITELLM_IMAGE: litellm-ui-scan:${{ github.sha }}
-        run: |
-          python -m pip install "pytest==9.0.3"
-          python -m pytest --noconftest tests/integration/database/image/test_ui_image_serves_offline.py -v
-
-  backend-image:
-    name: backend-image
-    runs-on: ubuntu-latest
-    if: >-
-      github.event_name != 'pull_request' ||
-      github.event.pull_request.head.repo.full_name == github.repository
-    timeout-minutes: 30
-    permissions:
-      contents: read
-    steps:
-      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          persist-credentials: false
-
-      - name: Build backend image
-        run: docker build -f backend/Dockerfile -t litellm-backend-scan:${{ github.sha }} .
-
-      - name: Tag the cached builder for Admin MCP schema setup
-        run: docker build --target builder -f backend/Dockerfile -t litellm-admin-mcp-schema:${{ github.sha }} .
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Verify the backend serves offline as a non-root uid
-        env:
-          LITELLM_IMAGE: litellm-backend-scan:${{ github.sha }}
-          LITELLM_ADMIN_MCP_SCHEMA_IMAGE: litellm-admin-mcp-schema:${{ github.sha }}
-          LITELLM_COMPONENT_PORT: "4001"
-          LITELLM_IMAGE_COMPONENT: backend
-        run: |
-          python -m pip install "pytest==9.0.3"
-          python -m pytest --noconftest tests/integration/database/image/test_component_image_serves_offline.py tests/integration/database/image/test_image_admin_mcp.py -v

.github/workflows/issue_classifier.yml

@@ -16,12 +16,12 @@ on:
       - .github/issue-labels.json
       - .github/ISSUE_TEMPLATE/bug_report.yml
       - .github/ISSUE_TEMPLATE/feature_request.yml
-      - scripts/classify-issue.ts
-      - scripts/classify-issue.test.ts
-      - scripts/label-issue.ts
-      - scripts/label-issue.test.ts
-      - scripts/issue-labels.ts
-      - scripts/auto-close-duplicates.ts
+      - litellm/scripts/classify-issue.ts
+      - litellm/scripts/classify-issue.test.ts
+      - litellm/scripts/label-issue.ts
+      - litellm/scripts/label-issue.test.ts
+      - litellm/scripts/issue-labels.ts
+      - litellm/scripts/auto-close-duplicates.ts
 
 permissions: {}
 
@@ -49,7 +49,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Test the gate, the validation and the label step
-        run: bun test scripts/classify-issue.test.ts scripts/label-issue.test.ts
+        run: bun test litellm/scripts/classify-issue.test.ts litellm/scripts/label-issue.test.ts
 
   classify-issue:
     # An edit to a labelled issue is dropped here; the script decides the rest against the live labels
@@ -73,7 +73,7 @@ jobs:
         with:
           sparse-checkout: |
             .github
-            scripts
+            litellm/scripts
           persist-credentials: false
 
       - name: Setup Bun
@@ -108,7 +108,7 @@ jobs:
           ISSUE_CLASSIFIER_MODEL: ${{ vars.ISSUE_CLASSIFIER_MODEL }}
         run: |
           set -euo pipefail
-          bun run scripts/classify-issue.ts > classification.json
+          bun run litellm/scripts/classify-issue.ts > classification.json
           {
             echo 'verdict<<CLASSIFICATION'
             cat classification.json
@@ -143,7 +143,7 @@ jobs:
         with:
           sparse-checkout: |
             .github
-            scripts
+            litellm/scripts
           persist-credentials: false
 
       - name: Setup Bun
@@ -153,7 +153,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Replace the labels in each namespace
-        run: bun run scripts/label-issue.ts
+        run: bun run litellm/scripts/label-issue.ts
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           VERDICT: ${{ needs.classify-issue.outputs.verdict }}

.github/workflows/issue_fixed_comment.yml

@@ -15,9 +15,9 @@ on:
   pull_request:
     paths:
       - .github/workflows/issue_fixed_comment.yml
-      - scripts/comment-fixed-issue.ts
-      - scripts/comment-fixed-issue.test.ts
-      - scripts/auto-close-duplicates.ts
+      - litellm/scripts/comment-fixed-issue.ts
+      - litellm/scripts/comment-fixed-issue.test.ts
+      - litellm/scripts/auto-close-duplicates.ts
 
 permissions: {}
 
@@ -44,7 +44,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Test the closer lookup, the release placement, the comment and the superseded pull request close
-        run: bun test scripts/comment-fixed-issue.test.ts
+        run: bun test litellm/scripts/comment-fixed-issue.test.ts
 
   comment-fixed-issue:
     if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm'
@@ -58,7 +58,7 @@ jobs:
       - name: Checkout scripts
         uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
-          sparse-checkout: scripts
+          sparse-checkout: litellm/scripts
           persist-credentials: false
 
       - name: Setup Bun
@@ -69,7 +69,7 @@ jobs:
 
       - name: Name the release that carries the fix and close the pull requests it supersedes
         shell: bash
-        run: bun run scripts/comment-fixed-issue.ts | tee -a "${GITHUB_STEP_SUMMARY}"
+        run: bun run litellm/scripts/comment-fixed-issue.ts | tee -a "${GITHUB_STEP_SUMMARY}"
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           ISSUE_NUMBER: ${{ github.event.issue.number || github.event.inputs.issue_number }}

.github/workflows/issue_label_sync.yml

@@ -5,7 +5,7 @@ on:
     branches: [main]
     paths:
       - .github/issue-labels.json
-      - scripts/sync-issue-labels.ts
+      - litellm/scripts/sync-issue-labels.ts
   workflow_dispatch:
     inputs:
       dry_run:
@@ -16,9 +16,9 @@ on:
     paths:
       - .github/workflows/issue_label_sync.yml
       - .github/issue-labels.json
-      - scripts/sync-issue-labels.ts
-      - scripts/sync-issue-labels.test.ts
-      - scripts/issue-labels.ts
+      - litellm/scripts/sync-issue-labels.ts
+      - litellm/scripts/sync-issue-labels.test.ts
+      - litellm/scripts/issue-labels.ts
 
 permissions: {}
 
@@ -41,7 +41,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Test the sync
-        run: bun test scripts/sync-issue-labels.test.ts
+        run: bun test litellm/scripts/sync-issue-labels.test.ts
 
   sync-issue-labels:
     if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm'
@@ -56,7 +56,7 @@ jobs:
         with:
           sparse-checkout: |
             .github
-            scripts
+            litellm/scripts
           persist-credentials: false
 
       - name: Setup Bun
@@ -66,7 +66,7 @@ jobs:
           bun-version: "1.4.0"
 
       - name: Create or recolour every label in .github/issue-labels.json
-        run: bun run scripts/sync-issue-labels.ts
+        run: bun run litellm/scripts/sync-issue-labels.ts
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == true }}

.github/workflows/mutation-test.yml

@@ -129,7 +129,7 @@ jobs:
           set +e
           uv run --no-sync --with mutmut==3.5.0 mutmut export-cicd-stats > /dev/null 2>&1
           uv run --no-sync --with mutmut==3.5.0 mutmut results > mutmut-results.txt 2>&1
-          uv run --no-sync python scripts/mutation_report.py
+          uv run --no-sync python litellm/scripts/mutation_report.py
           # The full report can be very long for big test files; the run-page
           # summary cuts off at 1 MB. Append the head of the report (summary
           # + survivor list) and link out to the artifact for the full body.

.github/workflows/osv-scan.yml

@@ -40,7 +40,7 @@ jobs:
       - name: Scan lockfiles
         run: |
           "$RUNNER_TEMP/osv-scanner" scan source \
-            --config osv-scanner.toml \
+            --config .github/osv-scanner.toml \
             -L uv.lock \
-            -L ui/litellm-dashboard/package-lock.json \
-            -L vscode-extension/package-lock.json
+            -L litellm/ui/litellm-dashboard/package-lock.json \
+            -L litellm/vscode-extension/package-lock.json

.github/workflows/publish-lint-base-counts.yml

@@ -3,7 +3,7 @@ name: Publish lint base counts
 # Every commit on main can become a future merge-base.
 # Publishing its per-rule counts for each lint gate (strict ruff, type discipline,
 # test quality, basedpyright) as an artifact lets the gates download them through
-# scripts/lint_base_counts.py in seconds instead of scanning the merge-base tree in
+# litellm/scripts/lint_base_counts.py in seconds instead of scanning the merge-base tree in
 # a throwaway worktree on every fresh checkout or moved merge-base.
 # No concurrency group on purpose: runs must never cancel each other, because
 # every sha's artifact matters (any of them can become a merge-base).
@@ -30,15 +30,15 @@ jobs:
       matrix:
         include:
           - checker: ruff-strict
-            script: scripts/ruff_strict_gate.py
+            script: litellm/scripts/ruff_strict_gate.py
           - checker: type-discipline
-            script: scripts/type_discipline_gate.py
+            script: litellm/scripts/type_discipline_gate.py
           - checker: test-quality
-            script: scripts/test_quality_gate.py
+            script: litellm/scripts/test_quality_gate.py
           - checker: openapi-docs
-            script: scripts/openapi_docs_gate.py
+            script: litellm/scripts/openapi_docs_gate.py
           - checker: basedpyright
-            script: scripts/type_check_gate.py
+            script: litellm/scripts/type_check_gate.py
 
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
@@ -91,7 +91,7 @@ jobs:
       - name: Emit basedpyright counts for HEAD
         if: matrix.checker == 'basedpyright'
         run: |
-          python scripts/type_check_gate.py --emit-counts-dir "$RUNNER_TEMP/lint-counts"
+          python litellm/scripts/type_check_gate.py --emit-counts-dir "$RUNNER_TEMP/lint-counts"
 
       - name: Name the counts artifact
         run: |

.github/workflows/root-entry-limit.yml

@@ -0,0 +1,27 @@
+name: Root entry limit
+
+on:
+  pull_request: {}
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  root-entry-limit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955
+        with:
+          persist-credentials: false
+      - name: Check root entry count
+        env:
+          MAX_ROOT_ENTRIES: 15
+        run: |
+          count=$(find . -mindepth 1 -maxdepth 1 -printf '.\n' | wc -l)
+          echo "Root has $count entries. Maximum is $MAX_ROOT_ENTRIES."
+          if [ "$count" -gt "$MAX_ROOT_ENTRIES" ]; then
+            echo "::error::Repository root exceeds $MAX_ROOT_ENTRIES entries. Move new files into an existing directory."
+            exit 1
+          fi

.github/workflows/spend-logs-guard.yml

@@ -42,4 +42,12 @@ jobs:
           MERGE_BASE: ${{ steps.revisions.outputs.merge_base }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
-          uv run --no-project --with pydantic==2.14.1 python ci_cd/spend_logs_guard.py --base "$MERGE_BASE" --head "$HEAD_SHA" --approver kerry-berri --reviews "$RUNNER_TEMP/reviews.json"
+          guard=litellm/scripts/ci_cd/spend_logs_guard.py
+          # TODO: Drop the legacy path after this lands on main.
+          if [ ! -f "$guard" ]; then
+            guard=scripts/ci_cd/spend_logs_guard.py
+          fi
+          if [ ! -f "$guard" ]; then
+            guard=ci_cd/spend_logs_guard.py
+          fi
+          uv run --no-project --with pydantic==2.14.1 python "$guard" --base "$MERGE_BASE" --head "$HEAD_SHA" --approver kerry-berri --reviews "$RUNNER_TEMP/reviews.json"

.github/workflows/sync-schema.yml

@@ -1,9 +1,9 @@
-name: Sync schema.prisma copies
+name: Sync litellm/schema.prisma copies
 
 on:
   pull_request:
     paths:
-      - 'schema.prisma'
+      - 'litellm/schema.prisma'
 
 # Scoped to ONLY the permissions needed:
 # - contents:write to push the sync commit to the PR branch
@@ -37,7 +37,7 @@ jobs:
 
       - name: Reject symlinked schema files
         run: |
-          for f in schema.prisma litellm/proxy/schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma; do
+          for f in litellm/schema.prisma litellm/proxy/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma; do
             if [ -L "$f" ]; then
               echo "::error file=$f::$f is a symlink, which is not allowed"
               exit 1
@@ -46,13 +46,13 @@ jobs:
 
       - name: Copy root schema to other locations
         run: |
-          cp schema.prisma litellm/proxy/schema.prisma
-          cp schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma
+          cp litellm/schema.prisma litellm/proxy/schema.prisma
+          cp litellm/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma
 
       - name: Check for changes
         id: diff
         run: |
-          if git diff --quiet -- litellm/proxy/schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma; then
+          if git diff --quiet -- litellm/proxy/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma; then
             echo "changed=false" >> "$GITHUB_OUTPUT"
             echo "Schemas already in sync. Nothing to do."
           else
@@ -68,6 +68,6 @@ jobs:
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
           git checkout -B "$GITHUB_HEAD_REF"
-          git add -- litellm/proxy/schema.prisma litellm-proxy-extras/litellm_proxy_extras/schema.prisma
-          git commit -m "chore: sync schema.prisma copies from root"
+          git add -- litellm/proxy/schema.prisma litellm/proxy-extras/litellm_proxy_extras/schema.prisma
+          git commit -m "chore: sync litellm/schema.prisma copies from the canonical schema"
           git push origin "HEAD:$GITHUB_HEAD_REF"

.github/workflows/test-integration.yml

@@ -55,7 +55,6 @@ jobs:
     services:
       postgres:
         image: postgres:16@sha256:e17e86066e5ef83e0952a9347f5c792b7ece00972e2aa787a6986f471b3dd3d5
-        command: -c max_connections=1000 -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all -c pg_stat_statements.max=20000
         env:
           POSTGRES_USER: postgres
           POSTGRES_PASSWORD: postgres
@@ -81,6 +80,17 @@ jobs:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
           persist-credentials: false
+      - name: Configure PostgreSQL for integration workers
+        env:
+          POSTGRES_CONTAINER: ${{ job.services.postgres.id }}
+        run: |
+          docker exec "$POSTGRES_CONTAINER" sh -c 'printf "%s\n" "max_connections = 1000" "shared_preload_libraries = pg_stat_statements" "pg_stat_statements.track = all" "pg_stat_statements.max = 20000" >> "$PGDATA/postgresql.conf"'
+          docker restart "$POSTGRES_CONTAINER"
+          for attempt in $(seq 1 30); do
+            if docker exec "$POSTGRES_CONTAINER" pg_isready -U postgres; then exit 0; fi
+            sleep 1
+          done
+          exit 1
 
       - name: Set up Python
         timeout-minutes: 3
@@ -106,6 +116,17 @@ jobs:
         timeout-minutes: 20
         run: .github/scripts/uv_sync_with_retries.sh --frozen --all-groups --all-extras
 
+      - name: Install the Rust gateway
+        if: matrix.suite == 'management' || matrix.suite == 'management-replica'
+        timeout-minutes: 15
+        working-directory: packaging/litellm-proxy
+        env:
+          CARGO_INCREMENTAL: "0"
+        run: |
+          uv run --no-project --with maturin==1.15.0 maturin build --profile dev --strip --out "$RUNNER_TEMP/gateway-dist"
+          uv pip install --no-deps --python ../../.venv/bin/python "$RUNNER_TEMP"/gateway-dist/*.whl
+          test -x ../../.venv/bin/litellm-gateway
+
       - name: Install redis-server 7.2.16
         timeout-minutes: 3
         run: |
@@ -129,22 +150,22 @@ jobs:
         timeout-minutes: 5
         uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
         with:
-          node-version-file: ui/litellm-dashboard/.nvmrc
+          node-version-file: litellm/ui/litellm-dashboard/.nvmrc
           cache: "npm"
           cache-dependency-path: |
-            ui/litellm-dashboard/package-lock.json
+            litellm/ui/litellm-dashboard/package-lock.json
             tests/e2e/ui/package-lock.json
 
       - name: Install browser dependencies and build the dashboard
         if: matrix.suite == 'browser'
         timeout-minutes: 20
         run: |
-          (cd ui/litellm-dashboard && npm ci)
+          (cd litellm/ui/litellm-dashboard && npm ci)
           (cd tests/e2e/ui && npm ci)
           sudo env PATH="$PATH" DEBIAN_FRONTEND=noninteractive \
             node tests/e2e/ui/node_modules/@playwright/test/cli.js install-deps chromium
           node tests/e2e/ui/node_modules/@playwright/test/cli.js install chromium
-          (cd ui/litellm-dashboard && NEXT_TELEMETRY_DISABLED=1 npm run build)
+          (cd litellm/ui/litellm-dashboard && NEXT_TELEMETRY_DISABLED=1 npm run build)
 
       - name: Run integration contracts
         timeout-minutes: 45
@@ -155,8 +176,8 @@ jobs:
           LITELLM_LICENSE: ${{ secrets.LITELLM_LICENSE }}
         run: |
           case "$SUITE" in
-            *-replica) bash .circleci/scripts/run_integration.sh "${SUITE%-replica}" replica ;;
-            *) bash .circleci/scripts/run_integration.sh "$SUITE" ;;
+            *-replica) bash .github/scripts/ci/run_integration.sh "${SUITE%-replica}" replica ;;
+            *) bash .github/scripts/ci/run_integration.sh "$SUITE" ;;
           esac
 
       - name: Verify every case ran exactly once

.github/workflows/test-linting.yml

@@ -73,14 +73,14 @@ jobs:
 
       - name: Cache uv dependencies
         if: steps.changes.outputs.decision != 'skip'
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+        uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
         with:
           path: |
             ~/.cache/uv
             .venv
-          key: ${{ runner.os }}-uv-lint-${{ hashFiles('uv.lock') }}
+          key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
           restore-keys: |
-            ${{ runner.os }}-uv-lint-
+            ${{ runner.os }}-uv-
 
       - name: Clean Python cache
         if: steps.changes.outputs.decision != 'skip'
@@ -114,7 +114,7 @@ jobs:
       - name: Check ruff format
         if: steps.changes.outputs.decision != 'skip'
         run: |
-          git diff --name-only --diff-filter=ACMR "$GATE_BASE_SHA" HEAD -- ':(glob)litellm/**/*.py' | grep -v '^litellm/enterprise/' > "$RUNNER_TEMP/ruff_format_files.txt" || true
+          git diff --name-only --diff-filter=ACMR "$GATE_BASE_SHA" HEAD -- ':(glob)litellm/**/*.py' | grep -Ev '^litellm/(enterprise|ui|vscode-extension|rust|proxy-extras|packaging|cli|scripts|runtime)/' > "$RUNNER_TEMP/ruff_format_files.txt" || true
           if [ ! -s "$RUNNER_TEMP/ruff_format_files.txt" ]; then
             echo "No changed litellm Python files to check with ruff format."
             exit 0
@@ -133,7 +133,7 @@ jobs:
 
       - name: Check MCP operation boundary
         if: steps.changes.outputs.decision != 'skip'
-        run: uv run --no-sync python scripts/check_mcp_operation_boundary.py
+        run: uv run --no-sync python litellm/scripts/check_mcp_operation_boundary.py
 
       - name: Run Ruff linting
         if: steps.changes.outputs.decision != 'skip'
@@ -145,28 +145,28 @@ jobs:
       - name: Run Ruff linting (test tree)
         if: steps.changes.outputs.decision != 'skip'
         run: |
-          uv run --no-sync ruff check --config ruff-tests.toml tests
+          uv run --no-sync ruff check --config .github/linters/ruff-tests.toml tests
 
       - name: Check strict ruff rules (delta vs merge-base counts)
         if: steps.changes.outputs.decision != 'skip'
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          uv run --no-sync python scripts/ruff_strict_gate.py --base "$GATE_BASE_SHA"
+          uv run --no-sync python litellm/scripts/ruff_strict_gate.py --base "$GATE_BASE_SHA"
 
       - name: Check type discipline (mutable collections / casts / type guards / kwargs / unexplained suppressions, delta vs merge-base counts)
         if: steps.changes.outputs.decision != 'skip'
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          uv run --no-sync python scripts/type_discipline_gate.py --base "$GATE_BASE_SHA"
+          uv run --no-sync python litellm/scripts/type_discipline_gate.py --base "$GATE_BASE_SHA"
 
       - name: Check test quality (zero-assert / mock-echo tests, sys.path.insert, raw env writes, litellm global mutation, credential-gated skips, conftest snapshot inventory, delta vs merge-base counts)
         if: steps.changes.outputs.decision != 'skip'
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          uv run --no-sync python scripts/test_quality_gate.py --base "$GATE_BASE_SHA"
+          uv run --no-sync python litellm/scripts/test_quality_gate.py --base "$GATE_BASE_SHA"
 
       - name: Check OpenAPI docs (POST routes without a request schema, description or 200 schema, delta vs merge-base counts)
         if: steps.changes.outputs.decision != 'skip'
@@ -174,7 +174,7 @@ jobs:
           GH_TOKEN: ${{ github.token }}
         run: |
           uv sync --frozen --group proxy-dev --group e2e-dev --extra proxy
-          uv run --no-sync python scripts/openapi_docs_gate.py --base "$GATE_BASE_SHA"
+          uv run --no-sync python litellm/scripts/openapi_docs_gate.py --base "$GATE_BASE_SHA"
 
       - name: Print OpenAI version
         if: steps.changes.outputs.decision != 'skip'
@@ -186,12 +186,12 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          uv run --no-sync python scripts/type_check_gate.py --base "$GATE_BASE_SHA"
+          uv run --no-sync python litellm/scripts/type_check_gate.py --base "$GATE_BASE_SHA"
 
       - name: Check tests/e2e basedpyright (zero errors)
         if: steps.changes.outputs.decision != 'skip'
         run: |
-          if git diff --name-only --diff-filter=ACMRD "$GATE_BASE_SHA" HEAD -- ':(glob)tests/e2e/**/*.py' ':(glob)tests/e2e_harness/**/*.py' pyrightconfig.json | grep -q .; then
+          if git diff --name-only --diff-filter=ACMRD "$GATE_BASE_SHA" HEAD -- ':(glob)tests/e2e/**/*.py' ':(glob)tests/e2e_harness/**/*.py' pyproject.toml | grep -q .; then
             uv run --no-sync basedpyright tests/e2e tests/e2e_harness
           else
             echo "No changed tests/e2e Python files; skipping."
@@ -258,7 +258,7 @@ jobs:
           if [ -n "$GITGUARDIAN_API_KEY" ]; then
             retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
             retry git fetch --no-tags --unshallow origin
-            uv tool run --from 'ggshield==1.48.0' ggshield secret scan repo .
+            uv tool run --from 'ggshield==1.48.0' ggshield --config-path .github/gitguardian.yaml secret scan repo .
           else
             echo "GITGUARDIAN_API_KEY not set, skipping ggshield scan"
           fi
@@ -271,7 +271,7 @@ jobs:
     timeout-minutes: 8
     defaults:
       run:
-        working-directory: ui/litellm-dashboard
+        working-directory: litellm/ui/litellm-dashboard
 
     steps:
       - name: Checkout repository
@@ -315,9 +315,9 @@ jobs:
         if: steps.changed.outputs.has_files == 'true'
         uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
         with:
-          node-version-file: ui/litellm-dashboard/.nvmrc
+          node-version-file: litellm/ui/litellm-dashboard/.nvmrc
           cache: "npm"
-          cache-dependency-path: ui/litellm-dashboard/package-lock.json
+          cache-dependency-path: litellm/ui/litellm-dashboard/package-lock.json
 
       - name: Install dependencies
         if: steps.changed.outputs.has_files == 'true'
@@ -333,12 +333,12 @@ jobs:
           status=0
           if [ ${#prettier_files[@]} -gt 0 ]; then
             echo "::group::Prettier (${#prettier_files[@]} files)"
-            npx prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
+            node_modules/.bin/prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
             echo "::endgroup::"
           fi
           if [ ${#eslint_files[@]} -gt 0 ]; then
             echo "::group::ESLint (${#eslint_files[@]} files)"
-            npx eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
+            node_modules/.bin/eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
             echo "::endgroup::"
           fi
           exit $status
@@ -375,7 +375,7 @@ jobs:
             exit 0
           fi
           files="$(git diff --name-only "$base" HEAD)"
-          if grep -Eq '^(litellm/|enterprise/|(pyproject\.toml|uv\.lock)$|ui/litellm-dashboard/(src/lib/http/schema\.d\.ts|scripts/gen-api-types\.mjs|package(-lock)?\.json)$|\.github/workflows/test-linting\.yml$)' <<< "$files"; then
+          if grep -Ev '^litellm/(ui|vscode-extension|rust|proxy-extras|packaging|cli|scripts|runtime)/' <<< "$files" | grep -E '^(litellm/|enterprise/|(pyproject\.toml|uv\.lock)$|\.github/workflows/test-linting\.yml$)' >/dev/null || grep -Eq '^litellm/ui/litellm-dashboard/(src/lib/http/schema\.d\.ts|scripts/gen-api-types\.mjs|package(-lock)?\.json)$' <<< "$files"; then
             echo "relevant=true" >> "$GITHUB_OUTPUT"
           else
             echo "No litellm, enterprise, dependency or generator changes in this pull request, nothing to verify."
@@ -396,7 +396,7 @@ jobs:
 
       - name: Cache uv dependencies
         if: steps.changes.outputs.relevant == 'true'
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+        uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
         with:
           path: |
             ~/.cache/uv
@@ -434,7 +434,7 @@ jobs:
             echo "A lazily loaded route or model changed without regenerating the snapshot that /openapi.json serves for unloaded features."
             echo "To fix, run from the repo root:"
             echo "  uv run python -m litellm.proxy._lazy_openapi_snapshot"
-            echo "then run npm run gen:api from ui/litellm-dashboard and commit both files."
+            echo "then run npm run gen:api from litellm/ui/litellm-dashboard and commit both files."
             exit 1
           fi
           echo "_lazy_openapi_snapshot.json is in sync with the lazily loaded routes."
@@ -443,18 +443,18 @@ jobs:
         if: steps.changes.outputs.relevant == 'true'
         uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
         with:
-          node-version-file: ui/litellm-dashboard/.nvmrc
+          node-version-file: litellm/ui/litellm-dashboard/.nvmrc
           cache: "npm"
-          cache-dependency-path: ui/litellm-dashboard/package-lock.json
+          cache-dependency-path: litellm/ui/litellm-dashboard/package-lock.json
 
       - name: Install dashboard dependencies
         if: steps.changes.outputs.relevant == 'true'
-        working-directory: ui/litellm-dashboard
+        working-directory: litellm/ui/litellm-dashboard
         run: npm ci
 
       - name: Regenerate types from the live spec
         if: steps.changes.outputs.relevant == 'true'
-        working-directory: ui/litellm-dashboard
+        working-directory: litellm/ui/litellm-dashboard
         env:
           LITELLM_PYTHON: "uv run --no-sync python"
         run: npm run gen:api
@@ -462,11 +462,11 @@ jobs:
       - name: Fail if types are stale
         if: steps.changes.outputs.relevant == 'true'
         run: |
-          if ! git diff --exit-code -- ui/litellm-dashboard/src/lib/http/schema.d.ts; then
-            echo "::error file=ui/litellm-dashboard/src/lib/http/schema.d.ts::Generated API types are out of sync with the proxy OpenAPI spec."
+          if ! git diff --exit-code -- litellm/ui/litellm-dashboard/src/lib/http/schema.d.ts; then
+            echo "::error file=litellm/ui/litellm-dashboard/src/lib/http/schema.d.ts::Generated API types are out of sync with the proxy OpenAPI spec."
             echo ""
             echo "A backend route or model changed without regenerating the dashboard types."
-            echo "To fix, run from ui/litellm-dashboard:"
+            echo "To fix, run from litellm/ui/litellm-dashboard:"
             echo "  npm run gen:api"
             echo "then commit the updated src/lib/http/schema.d.ts."
             exit 1
@@ -524,7 +524,7 @@ jobs:
         run: uv run --no-sync python ./tests/code_coverage_tests/check_provider_folders_documented.py
 
       - name: check_provider_endpoints_support_generated
-        run: uv run --no-sync python scripts/generate_provider_endpoints_support.py --check
+        run: uv run --no-sync python litellm/scripts/generate_provider_endpoints_support.py --check
 
       - name: test_generate_provider_endpoints_support
         run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_generate_provider_endpoints_support.py
@@ -703,7 +703,7 @@ jobs:
           version: "0.10.9"
 
       - name: Run Semgrep (custom rules)
-        run: uv tool run --from 'semgrep==1.157.0' semgrep scan --config .semgrep/rules . --error
+        run: uv tool run --from 'semgrep==1.157.0' semgrep scan --config .github/semgrep/rules . --error
   lint-passed:
     name: lint passed
     needs: [python, secret-scan, ui, ui-api-types, code-quality, ci-coverage, semgrep]

.github/workflows/test-mcp-dependency-resolution.yml

@@ -10,17 +10,17 @@ on:
       - "uv.lock"
       - "uv.toml"
       - ".python-version"
-      - "rust-toolchain.toml"
-      - "litellm-rust/**"
+      - "litellm/rust/rust-toolchain.toml"
+      - "litellm/rust/**"
       - "litellm/__init__.py"
       - "litellm/_version.py"
       - "litellm/_lazy_imports_registry.py"
-      - "scripts/build_core_distribution.py"
+      - "litellm/scripts/build_core_distribution.py"
       - "litellm/proxy/proxy_server.py"
       - "litellm/**/*mcp*"
       - "litellm/**/*mcp*/**"
       - "litellm/integrations/arize/**"
-      - "scripts/check_mcp_sdk_install.py"
+      - "litellm/scripts/check_mcp_sdk_install.py"
       - "tests/base_sdk_tests/**"
       - ".github/workflows/test-mcp-dependency-resolution.yml"
       - ".github/actions/detect-changes/**"
@@ -28,7 +28,7 @@ on:
       - ".github/actions/cache-cargo-build/**"
       - ".github/scripts/detect_changes.sh"
       - ".github/scripts/uv_sync_with_retries.sh"
-      - ".circleci/scripts/classify_changes.sh"
+      - ".github/scripts/ci/classify_changes.sh"
 
 permissions:
   contents: read
@@ -91,7 +91,7 @@ jobs:
               checker=("$GITHUB_WORKSPACE/tests/base_sdk_tests/check_base_sdk_install.py")
               (cd "$RUNNER_TEMP" && "$GITHUB_WORKSPACE/.venv-$extra/bin/python" -I "$GITHUB_WORKSPACE/tests/base_sdk_tests/check_sdk_http.py")
             else
-              checker=("$GITHUB_WORKSPACE/scripts/check_mcp_sdk_install.py" --extra "$extra")
+              checker=("$GITHUB_WORKSPACE/litellm/scripts/check_mcp_sdk_install.py" --extra "$extra")
             fi
             (cd "$RUNNER_TEMP" && "$GITHUB_WORKSPACE/.venv-$extra/bin/python" "${checker[@]}")
           done
@@ -115,7 +115,7 @@ jobs:
             if [ "$extra" = core ]; then
               checker=("$GITHUB_WORKSPACE/tests/base_sdk_tests/check_base_sdk_install.py")
             else
-              checker=("$GITHUB_WORKSPACE/scripts/check_mcp_sdk_install.py" --extra "$extra")
+              checker=("$GITHUB_WORKSPACE/litellm/scripts/check_mcp_sdk_install.py" --extra "$extra")
             fi
             (cd "$RUNNER_TEMP" && "$GITHUB_WORKSPACE/.venv-lowest-$extra/bin/python" "${checker[@]}")
           done
@@ -147,15 +147,16 @@ jobs:
       - name: Build the core wheel and source distribution
         run: >-
           uv run --no-project --with coverage==7.14.0 python -m coverage run --rcfile=/dev/null
-          --branch --include="*/scripts/build_core_distribution.py"
-          scripts/build_core_distribution.py --out-dir dist/core
+          --branch --include="*/litellm/scripts/build_core_distribution.py"
+          litellm/scripts/build_core_distribution.py --out-dir dist/core
       - name: Verify metadata, resources, and independent source rebuild
         env:
           CORE_DISTRIBUTION_DIR: ${{ github.workspace }}/dist/core
         run: >-
           uv run --no-project --with pytest==9.0.3 --with pytest-cov==5.0.0 --with coverage==7.14.0 --with 'tomli==2.4.1; python_version < "3.11"'
-          python -m pytest tests/base_sdk_tests/test_core_distribution.py -v
-          --cov-config=/dev/null --cov=scripts.build_core_distribution --cov-branch --cov-append --cov-report=
+          python -m coverage run --rcfile=/dev/null --append --branch
+          --include="*/litellm/scripts/build_core_distribution.py"
+          -m pytest tests/base_sdk_tests/test_core_distribution.py -v
       - name: Verify isolated core installations
         run: |
           wheel=$(realpath dist/core/litellm_core-*.whl)

.github/workflows/test-merge-smoke.yml

@@ -3,6 +3,8 @@ name: smoke
 on:
   pull_request:
     branches: [main, litellm_internal_staging, litellm_oss_staging, "litellm_**"]
+  push:
+    branches: [main]
   workflow_dispatch:
 
 permissions:
@@ -24,7 +26,7 @@ jobs:
           persist-credentials: false
 
       - name: Build the dashboard stage
-        run: docker build --target ui-builder -f Dockerfile .
+        run: docker build --target ui-builder -f litellm/deploy/docker/Dockerfile .
 
   core-checks:
     name: py${{ matrix.python-version }}
@@ -52,8 +54,26 @@ jobs:
         with:
           version: "0.10.9"
 
+      - name: Restore the Rust bridge
+        id: rust-bridge
+        uses: ./.github/actions/restore-rust-bridge
+
+      - name: Restore the Cargo build cache
+        if: steps.rust-bridge.outputs.cache-hit != 'true'
+        uses: ./.github/actions/cache-cargo-build
+
       - name: Install dependencies
-        run: .github/scripts/uv_sync_with_retries.sh --frozen --extra proxy --extra cli --group dev --group proxy-dev --python ${{ matrix.python-version }}
+        env:
+          RUST_BRIDGE_RESTORED: ${{ steps.rust-bridge.outputs.cache-hit }}
+          PYTHON_VERSION: ${{ matrix.python-version }}
+        run: |
+          if [ "$RUST_BRIDGE_RESTORED" != "true" ]; then
+            .github/scripts/uv_sync_with_retries.sh --frozen --extra proxy --extra cli --group dev --group proxy-dev --python "$PYTHON_VERSION"
+            exit 0
+          fi
+          .github/scripts/uv_sync_with_retries.sh --frozen --extra proxy --extra cli --group dev --group proxy-dev --python "$PYTHON_VERSION" --no-install-project
+          uv pip install --no-deps --python .venv/bin/python rust-bridge-dist/litellm-*.whl
+          cp rust-bridge-dist/litellm/rust_bridge/_native.abi3.so litellm/rust_bridge/_native.abi3.so
 
       - name: Create the loopback-only network namespace
         run: |
@@ -87,6 +107,7 @@ jobs:
 
       - name: Upload smoke diagnostics
         if: always()
+        continue-on-error: true
         uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
         with:
           name: merge-smoke-diagnostics-py${{ matrix.python-version }}
@@ -97,6 +118,10 @@ jobs:
         if: always()
         run: sudo ip netns delete smoke
 
+      - name: Keep only source-built wheels in the uv cache
+        if: always() && github.ref == 'refs/heads/main'
+        run: uv cache prune --ci
+
   all-extras:
     name: all-extras-py${{ matrix.python-version }}
     runs-on: ubuntu-24.04
@@ -123,12 +148,34 @@ jobs:
         with:
           version: "0.10.9"
 
+      - name: Restore the Rust bridge
+        id: rust-bridge
+        uses: ./.github/actions/restore-rust-bridge
+
+      - name: Restore the Cargo build cache
+        if: steps.rust-bridge.outputs.cache-hit != 'true'
+        uses: ./.github/actions/cache-cargo-build
+
       - name: Install every extra and dependency group
-        run: .github/scripts/uv_sync_with_retries.sh --frozen --all-groups --all-extras --python ${{ matrix.python-version }}
+        env:
+          RUST_BRIDGE_RESTORED: ${{ steps.rust-bridge.outputs.cache-hit }}
+          PYTHON_VERSION: ${{ matrix.python-version }}
+        run: |
+          if [ "$RUST_BRIDGE_RESTORED" != "true" ]; then
+            .github/scripts/uv_sync_with_retries.sh --frozen --all-groups --all-extras --python "$PYTHON_VERSION"
+            exit 0
+          fi
+          .github/scripts/uv_sync_with_retries.sh --frozen --all-groups --all-extras --python "$PYTHON_VERSION" --no-install-project --no-install-package litellm-proxy
+          uv pip install --no-deps --python .venv/bin/python rust-bridge-dist/*.whl
+          cp rust-bridge-dist/litellm/rust_bridge/_native.abi3.so litellm/rust_bridge/_native.abi3.so
 
       - name: Import the SDK, the proxy server and the enterprise package
         run: .venv/bin/python -c "import litellm, litellm.proxy.proxy_server, litellm_enterprise"
 
+      - name: Keep only source-built wheels in the uv cache
+        if: always() && github.ref == 'refs/heads/main'
+        run: uv cache prune --ci
+
   windows-sdk:
     name: windows-sdk
     runs-on: windows-2025
@@ -154,15 +201,16 @@ jobs:
         with:
           version: "0.10.9"
 
-      - name: Set up the Rust bridge
-        uses: ./.github/actions/rust-bridge
-
       - name: Install dependencies
         run: .github/scripts/uv_sync_with_retries.sh --frozen --group dev --python 3.11
 
       - name: Run the Windows tests
         run: uv run --no-sync python -m pytest --tb=short -v tests/windows_tests
 
+      - name: Keep only source-built wheels in the uv cache
+        if: always() && github.ref == 'refs/heads/main'
+        run: uv cache prune --ci
+
   proxy-build-from-pip:
     name: proxy-build-from-pip
     runs-on: ubuntu-24.04
@@ -200,8 +248,36 @@ jobs:
       - name: Install the test client dependencies
         run: .github/scripts/uv_sync_with_retries.sh --frozen --no-install-workspace --extra proxy --group dev --python 3.12
 
-      - name: Build the image from the published package
-        run: docker build -t litellm-build-from-pip:smoke -f docker/build_from_pip/Dockerfile.build_from_pip .
+      - name: Cache Prisma binaries
+        timeout-minutes: 3
+        uses: ./.github/actions/cache-prisma-binaries
+
+      - name: Install the published proxy into its own venv
+        run: |
+          uv --no-config venv --python 3.13 /tmp/litellm-pip
+          uv --no-config pip install --python /tmp/litellm-pip/bin/python \
+            "litellm[proxy,proxy-runtime]==1.83.0" \
+            "google-cloud-aiplatform==1.133.0" \
+            "google-genai==1.37.0" \
+            "anthropic[vertex]==0.84.0" \
+            "grpcio==1.78.0" \
+            "prometheus-client==0.20.0" \
+            "langfuse==2.59.7" \
+            "opentelemetry-api==1.28.0" \
+            "opentelemetry-sdk==1.28.0" \
+            "opentelemetry-exporter-otlp==1.28.0" \
+            "ddtrace==4.11.0" \
+            "sentry-sdk==2.21.0" \
+            "mangum==0.17.0" \
+            "azure-ai-contentsafety==1.0.0" \
+            "azure-storage-file-datalake==12.20.0" \
+            "pypdf==6.7.5" \
+            "llm-sandbox==0.3.31" \
+            "detect-secrets==1.5.0" \
+            "prisma==0.11.0" \
+            "openai==2.24.0"
+          export PATH="/tmp/litellm-pip/bin:$PATH"
+          /tmp/litellm-pip/bin/prisma generate --schema=litellm/schema.prisma
 
       - name: Start the fake OpenAI endpoint
         run: |
@@ -219,23 +295,21 @@ jobs:
           echo "::add-mask::${key}"
           echo "LITELLM_MASTER_KEY=${key}" >> "${GITHUB_ENV}"
 
-      - name: Start the proxy container
+      - name: Start the published proxy
+        env:
+          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/circle_test
+          LITELLM_LICENSE: bad-license
+          FAKE_OPENAI_API_BASE: http://localhost:8190
+          OTEL_EXPORTER: in_memory
+          USE_DDTRACE: "True"
+          DD_TRACE_OPENAI_ENABLED: "False"
+          GCS_FLUSH_INTERVAL: "1"
+          LITELLM_LOG: ERROR
         run: |
-          docker run -d \
-            -p 4000:4000 \
-            -e DATABASE_URL=postgresql://postgres:postgres@host.docker.internal:5432/circle_test \
-            -e LITELLM_MASTER_KEY \
-            -e FAKE_OPENAI_API_BASE=http://host.docker.internal:8190 \
-            -e OTEL_EXPORTER=in_memory \
-            -e USE_DDTRACE=True \
-            -e GCS_FLUSH_INTERVAL=1 \
-            -e LITELLM_LOG=ERROR \
-            --add-host host.docker.internal:host-gateway \
-            --name litellm-build-from-pip \
-            -v "${GITHUB_WORKSPACE}/docker/build_from_pip/litellm_config.yaml:/app/config.yaml" \
-            litellm-build-from-pip:smoke \
-            --config /app/config.yaml \
-            --port 4000
+          cd /tmp/litellm-pip
+          nohup /tmp/litellm-pip/bin/ddtrace-run /tmp/litellm-pip/bin/litellm \
+            --config "${GITHUB_WORKSPACE}/tests/basic_proxy_startup_tests/build_from_pip_config.yaml" \
+            --port 4000 > /tmp/litellm-pip.log 2>&1 &
           for _ in $(seq 1 150); do
             curl -sf --max-time 5 -o /dev/null http://localhost:4000 && exit 0
             sleep 2
@@ -270,7 +344,11 @@ jobs:
 
       - name: Show the proxy logs
         if: always()
-        run: docker logs litellm-build-from-pip || true
+        run: cat /tmp/litellm-pip.log || true
+
+      - name: Keep only source-built wheels in the uv cache
+        if: always() && github.ref == 'refs/heads/main'
+        run: uv cache prune --ci
 
   smoke-passed:
     name: smoke passed

.github/workflows/test-rust-db.yml

@@ -3,14 +3,14 @@ name: LiteLLM Rust DB
 on:
   push:
     paths:
-      - "litellm-rust/crates/db/**"
-      - "litellm-rust/Cargo.toml"
-      - "litellm-rust/Cargo.lock"
-      - "litellm-rust/clippy.toml"
-      - "litellm-proxy-extras/litellm_proxy_extras/migrations/**"
+      - "litellm/rust/crates/db/**"
+      - "litellm/rust/Cargo.toml"
+      - "litellm/rust/Cargo.lock"
+      - "litellm/rust/clippy.toml"
+      - "litellm/proxy-extras/litellm_proxy_extras/migrations/**"
       - "**/schema.prisma"
-      - ".cargo/**"
-      - "rust-toolchain.toml"
+      - "litellm/rust/.cargo/**"
+      - "litellm/rust/rust-toolchain.toml"
       - "pyproject.toml"
       - "uv.lock"
       - ".github/actions/setup-uv-with-retries/**"
@@ -20,14 +20,14 @@ on:
       - main
       - "litellm_**"
     paths:
-      - "litellm-rust/crates/db/**"
-      - "litellm-rust/Cargo.toml"
-      - "litellm-rust/Cargo.lock"
-      - "litellm-rust/clippy.toml"
-      - "litellm-proxy-extras/litellm_proxy_extras/migrations/**"
+      - "litellm/rust/crates/db/**"
+      - "litellm/rust/Cargo.toml"
+      - "litellm/rust/Cargo.lock"
+      - "litellm/rust/clippy.toml"
+      - "litellm/proxy-extras/litellm_proxy_extras/migrations/**"
       - "**/schema.prisma"
-      - ".cargo/**"
-      - "rust-toolchain.toml"
+      - "litellm/rust/.cargo/**"
+      - "litellm/rust/rust-toolchain.toml"
       - "pyproject.toml"
       - "uv.lock"
       - ".github/actions/setup-uv-with-retries/**"
@@ -42,10 +42,6 @@ concurrency:
 
 env:
   CARGO_TERM_COLOR: always
-  RUSTC_WRAPPER: sccache
-  SCCACHE_GHA_ENABLED: "true"
-  SCCACHE_GHA_VERSION: litellm-ci-v1
-  SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
   CARGO_INCREMENTAL: "0"
 
 jobs:
@@ -71,7 +67,7 @@ jobs:
       SQLX_OFFLINE: "false"
     defaults:
       run:
-        working-directory: litellm-rust
+        working-directory: litellm/rust
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
@@ -89,13 +85,9 @@ jobs:
 
       - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
         with:
-          workspaces: litellm-rust
+          workspaces: litellm/rust
           cache-on-failure: true
-
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
+          save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Install Python dependencies
         working-directory: .
@@ -103,13 +95,13 @@ jobs:
 
       - name: Deploy Prisma migrations
         working-directory: .
-        run: uv run --no-sync prisma migrate deploy --schema litellm-proxy-extras/litellm_proxy_extras/schema.prisma
+        run: uv run --no-sync prisma migrate deploy --schema litellm/proxy-extras/litellm_proxy_extras/schema.prisma
 
       - name: Install SQLx CLI
         run: cargo install sqlx-cli --version 0.9.0 --locked --no-default-features -F postgres
 
       - name: Check SQLx query metadata
-        working-directory: litellm-rust/crates/db
+        working-directory: litellm/rust/crates/db
         run: cargo sqlx prepare --check
 
       - name: Test database queries

.github/workflows/test-rust.yml

@@ -3,7 +3,7 @@ name: LiteLLM Rust
 on:
   push:
     paths:
-      - "litellm-rust/**"
+      - "litellm/rust/**"
       - "model_prices_and_context_window.json"
       - "litellm/rust_bridge/**"
       - "tests/test_litellm_rust/**"
@@ -20,10 +20,10 @@ on:
       - "litellm/llms/base_llm/ocr/**"
       - "litellm/llms/custom_httpx/llm_http_handler.py"
       - "Makefile"
-      - ".cargo/**"
+      - "litellm/rust/.cargo/**"
       - "pyproject.toml"
       - "uv.lock"
-      - "rust-toolchain.toml"
+      - "litellm/rust/rust-toolchain.toml"
       - ".github/actions/setup-uv-with-retries/**"
       - ".github/scripts/smoke_test_native_wheel.py"
       - ".github/scripts/verify_linux_native_wheel.py"
@@ -34,7 +34,7 @@ on:
       - main
       - "litellm_**"
     paths:
-      - "litellm-rust/**"
+      - "litellm/rust/**"
       - "model_prices_and_context_window.json"
       - "litellm/rust_bridge/**"
       - "tests/test_litellm_rust/**"
@@ -51,10 +51,10 @@ on:
       - "litellm/llms/base_llm/ocr/**"
       - "litellm/llms/custom_httpx/llm_http_handler.py"
       - "Makefile"
-      - ".cargo/**"
+      - "litellm/rust/.cargo/**"
       - "pyproject.toml"
       - "uv.lock"
-      - "rust-toolchain.toml"
+      - "litellm/rust/rust-toolchain.toml"
       - ".github/actions/setup-uv-with-retries/**"
       - ".github/scripts/smoke_test_native_wheel.py"
       - ".github/scripts/verify_linux_native_wheel.py"
@@ -70,10 +70,6 @@ concurrency:
 
 env:
   CARGO_TERM_COLOR: always
-  RUSTC_WRAPPER: sccache
-  SCCACHE_GHA_ENABLED: "true"
-  SCCACHE_GHA_VERSION: litellm-ci-v1
-  SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
   CARGO_INCREMENTAL: "0"
 
 jobs:
@@ -82,27 +78,24 @@ jobs:
     timeout-minutes: 15
     defaults:
       run:
-        working-directory: litellm-rust
+        working-directory: litellm/rust
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
           persist-credentials: false
 
-      - run: rustup toolchain install --no-self-update
+      - name: Configure the Rust build environment
+        working-directory: .
+        run: bash .github/scripts/setup_rust_environment.sh
 
       - run: cargo fmt --all --check
 
       - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
         with:
-          workspaces: litellm-rust
+          workspaces: litellm/rust
           cache-on-failure: true
           save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - run: cargo clippy --workspace --all-targets --locked -- -D warnings
 
   rust-test:
@@ -113,7 +106,7 @@ jobs:
       CARGO_PROFILE_DEV_DEBUG: line-tables-only
     defaults:
       run:
-        working-directory: litellm-rust
+        working-directory: litellm/rust
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
@@ -141,15 +134,10 @@ jobs:
 
       - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
         with:
-          workspaces: litellm-rust
+          workspaces: litellm/rust
           cache-on-failure: true
           save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - run: cargo nextest run --workspace --locked
 
       - run: cargo test --workspace --doc --locked
@@ -188,15 +176,10 @@ jobs:
 
       - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
         with:
-          workspaces: litellm-rust
+          workspaces: litellm/rust
           cache-on-failure: true
           save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Set up sccache
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - run: uv build --wheel --out-dir dist
 
       - run: python .github/scripts/verify_linux_native_wheel.py dist/*.whl

.github/workflows/test-terraform-modules.yml

@@ -3,16 +3,16 @@ name: Terraform Modules
 on:
   push:
     paths:
-      - "terraform/litellm/aws/**"
-      - "terraform/litellm/gcp/**"
+      - "litellm/deploy/terraform/litellm/aws/**"
+      - "litellm/deploy/terraform/litellm/gcp/**"
       - ".github/workflows/test-terraform-modules.yml"
   pull_request:
     branches:
       - main
       - "litellm_**"
     paths:
-      - "terraform/litellm/aws/**"
-      - "terraform/litellm/gcp/**"
+      - "litellm/deploy/terraform/litellm/aws/**"
+      - "litellm/deploy/terraform/litellm/gcp/**"
       - ".github/workflows/test-terraform-modules.yml"
 
 permissions:
@@ -33,7 +33,7 @@ jobs:
         module: [aws, gcp]
     defaults:
       run:
-        working-directory: terraform/litellm/${{ matrix.module }}
+        working-directory: litellm/deploy/terraform/litellm/${{ matrix.module }}
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:

.github/workflows/test-terraform-provider.yml

@@ -3,16 +3,18 @@ name: Terraform Provider
 on:
   push:
     paths:
-      - "terraform/provider/**"
+      - "litellm/deploy/terraform/provider/**"
+      - ".github/actions/cache-go-build/**"
       - ".github/workflows/test-terraform-provider.yml"
   pull_request:
     branches:
       - main
       - "litellm_**"
     paths:
-      - "terraform/provider/**"
+      - "litellm/deploy/terraform/provider/**"
       - "litellm/proxy/**"
-      - "enterprise/**"
+      - "litellm/enterprise/**"
+      - ".github/actions/cache-go-build/**"
       - ".github/workflows/test-terraform-provider.yml"
 
 permissions:
@@ -29,7 +31,7 @@ jobs:
     timeout-minutes: 10
     defaults:
       run:
-        working-directory: terraform/provider
+        working-directory: litellm/deploy/terraform/provider
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
@@ -37,9 +39,13 @@ jobs:
 
       - uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0
         with:
-          go-version-file: terraform/provider/go.mod
-          cache: true
-          cache-dependency-path: terraform/provider/go.sum
+          go-version-file: litellm/deploy/terraform/provider/go.mod
+          cache: false
+
+      - name: Cache the Go build
+        uses: ./.github/actions/cache-go-build
+        with:
+          dependency-path: litellm/deploy/terraform/provider/go.sum
 
       - name: gofmt
         run: |
@@ -115,14 +121,18 @@ jobs:
 
       - name: Generate proxy OpenAPI schema
         run: |
-          uv run --no-sync python terraform/provider/tools/dump_openapi.py "${RUNNER_TEMP}/openapi.json"
+          uv run --no-sync python litellm/deploy/terraform/provider/tools/dump_openapi.py "${RUNNER_TEMP}/openapi.json"
 
       - uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0
         with:
-          go-version-file: terraform/provider/go.mod
-          cache: true
-          cache-dependency-path: terraform/provider/go.sum
+          go-version-file: litellm/deploy/terraform/provider/go.mod
+          cache: false
+
+      - name: Cache the Go build
+        uses: ./.github/actions/cache-go-build
+        with:
+          dependency-path: litellm/deploy/terraform/provider/go.sum
 
       - name: Audit provider endpoints against the schema
-        working-directory: terraform/provider
+        working-directory: litellm/deploy/terraform/provider
         run: go run ./tools/endpointaudit -provider-dir ./litellm -spec "${RUNNER_TEMP}/openapi.json" -coverage-allowlist ./tools/endpointaudit/coverage_allowlist.txt

.github/workflows/test-unit.yml

@@ -41,13 +41,6 @@ jobs:
         timeout-minutes: 2
         uses: ./.github/actions/detect-changes
 
-      - name: Define editable Rust bridge cache key
-        id: rust-bridge-key
-        if: steps.changes.outputs.decision != 'skip'
-        env:
-          RUST_BRIDGE_CACHE_KEY: ${{ runner.os }}-rust-bridge-editable-${{ hashFiles('litellm-rust/Cargo.lock', 'litellm-rust/Cargo.toml', 'litellm-rust/crates/**', 'model_prices_and_context_window.json', 'pyproject.toml', 'packaging/litellm-proxy/pyproject.toml') }}
-        run: echo "key=$RUST_BRIDGE_CACHE_KEY" >> "$GITHUB_OUTPUT"
-
       - name: Set up Python
         if: steps.changes.outputs.decision != 'skip'
         timeout-minutes: 3
@@ -66,34 +59,21 @@ jobs:
         id: rust-bridge-cache
         if: steps.changes.outputs.decision != 'skip'
         timeout-minutes: 5
-        uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830
-        with:
-          path: rust-bridge-dist
-          key: ${{ steps.rust-bridge-key.outputs.key }}
+        uses: ./.github/actions/restore-rust-bridge
 
       - name: Cache the Rust build
         if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
         timeout-minutes: 5
         uses: ./.github/actions/cache-cargo-build
 
-      - name: Set up sccache
-        if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
-        uses: mozilla-actions/sccache-action@d651010b8da762cde178750d8eda7b5febfe147a
-        with:
-          version: "v0.18.0"
-
       - name: Build editable Rust bridge
         if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
         timeout-minutes: 15
         env:
-          RUSTC_WRAPPER: sccache
-          SCCACHE_GHA_ENABLED: "true"
-          SCCACHE_GHA_VERSION: litellm-ci-v1
-          SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
           CARGO_INCREMENTAL: "0"
         run: |
           mkdir -p rust-bridge-dist
-          uv run --no-project --with maturin==1.15.0 python -c 'import maturin; maturin.build_editable("rust-bridge-dist")'
+          uv run --no-project --with maturin==1.15.0 python .github/packaging/build_backend.py rust-bridge-dist
           strip --strip-debug litellm/rust_bridge/_native.abi3.so
           mkdir -p rust-bridge-dist/litellm/rust_bridge
           cp litellm/rust_bridge/_native.abi3.so rust-bridge-dist/litellm/rust_bridge/_native.abi3.so
@@ -101,14 +81,10 @@ jobs:
       - name: Build the gateway wheel once for test shards
         if: steps.changes.outputs.decision != 'skip' && steps.rust-bridge-cache.outputs.cache-hit != 'true'
         timeout-minutes: 15
-        working-directory: packaging/litellm-proxy
+        working-directory: litellm/packaging/litellm-proxy
         env:
-          RUSTC_WRAPPER: sccache
-          SCCACHE_GHA_ENABLED: "true"
-          SCCACHE_GHA_VERSION: litellm-ci-v1
-          SCCACHE_GHA_RW_MODE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'READ_WRITE' || 'READ_ONLY' }}
           CARGO_INCREMENTAL: "0"
-        run: uv run --no-project --with maturin==1.15.0 maturin build --profile dev --strip --out ../../rust-bridge-dist
+        run: uv run --no-project --with maturin==1.15.0 maturin build --profile dev --strip --out ../../../rust-bridge-dist
 
       - name: Upload Rust bridge artifact
         if: steps.changes.outputs.decision != 'skip'
@@ -125,7 +101,7 @@ jobs:
         uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830
         with:
           path: rust-bridge-dist
-          key: ${{ steps.rust-bridge-key.outputs.key }}
+          key: ${{ steps.rust-bridge-cache.outputs.key }}
 
       - name: Expose the Rust bridge artifact
         id: rust-bridge-artifact
@@ -397,6 +373,7 @@ jobs:
               tests/unit/llms/voyage
               tests/unit/llms/wandb
               tests/unit/llms/watsonx
+              tests/unit/llms/webiq
               tests/unit/llms/xai
               tests/unit/llms/xinference
               tests/unit/llms/you_com
@@ -459,7 +436,6 @@ jobs:
               tests/unit/ocr
               tests/unit/passthrough
               tests/unit/rag
-              tests/unit/cookbook
               tests/unit/realtime_api
               tests/unit/rerank_api
               tests/unit/search
@@ -550,6 +526,7 @@ jobs:
               tests/unit/proxy/management_helpers
               tests/unit/proxy/list_api
               tests/unit/proxy/analytics_endpoints
+              tests/unit/proxy/alerting_endpoints
               tests/unit/proxy/decisions_endpoints
               tests/unit/proxy/memory
               tests/unit/proxy/agent_endpoints
@@ -589,6 +566,7 @@ jobs:
               tests/unit/proxy/credential_endpoints
               tests/unit/proxy/discovery_endpoints
               tests/unit/proxy/shutdown
+              tests/unit/proxy/rust_forward
               tests/unit/proxy/public_endpoints
               tests/unit/proxy/prompts
               tests/unit/proxy/rag_endpoints
@@ -631,23 +609,32 @@ jobs:
             job-timeout-minutes: 60
             dist: loadscope
 
-          - shard: proxy-infra
+          - shard: proxy-infra-db-spend
             test-path: |-
               tests/unit/proxy/db
               --ignore=tests/unit/proxy/db/db_transaction_queue/test_e2e_pod_lock_manager.py
               --ignore=tests/unit/proxy/db/test_update_daily_tag_spend.py
-              tests/unit/proxy/middleware
-              --ignore=tests/unit/proxy/middleware/test_request_size_limit_middleware.py
               tests/unit/proxy/spend_tracking
               --ignore=tests/unit/proxy/spend_tracking/test_search_api_logging.py
-              tests/unit/proxy/pass_through_endpoints
-              tests/unit/proxy/mcp_server/test_tool_registry.py
               tests/unit/proxy/common_utils
               --ignore=tests/unit/proxy/common_utils/test_cache_aware_routing.py
               --ignore=tests/unit/proxy/common_utils/test_check_batch_cost.py
               --ignore=tests/unit/proxy/common_utils/test_check_responses_cost.py
               --ignore=tests/unit/proxy/common_utils/test_proxy_encrypt_decrypt.py
               --ignore=tests/unit/proxy/common_utils/test_realtime_cache.py
+            python-version: "3.12"
+            workers: 4
+            reruns: 0
+            timeout-minutes: 20
+            job-timeout-minutes: 60
+            dist: loadscope
+
+          - shard: proxy-infra-routing
+            test-path: |-
+              tests/unit/proxy/middleware
+              --ignore=tests/unit/proxy/middleware/test_request_size_limit_middleware.py
+              tests/unit/proxy/pass_through_endpoints
+              tests/unit/proxy/mcp_server/test_tool_registry.py
               tests/unit/proxy/enterprise_billing
               tests/unit/proxy/types_utils
               tests/unit/proxy/logging_endpoints
@@ -662,34 +649,30 @@ jobs:
             job-timeout-minutes: 60
             dist: loadscope
 
-          - shard: proxy-infra-root
+          - shard: proxy-infra-root-a-p
             test-path: |-
-              tests/unit/proxy/test_*.py
-              --ignore=tests/unit/proxy/test_aproxy_startup.py
-              --ignore=tests/unit/proxy/test_credential_slot_registry.py
-              --ignore=tests/unit/proxy/test_custom_callback_input.py
-              --ignore=tests/unit/proxy/test_custom_logger_s3_gcs.py
-              --ignore=tests/unit/proxy/test_custom_tokenizer_bug.py
-              --ignore=tests/unit/proxy/test_db_schema_changes.py
-              --ignore=tests/unit/proxy/test_deprecated_key_grace_period.py
-              --ignore=tests/unit/proxy/test_get_favicon.py
-              --ignore=tests/unit/proxy/test_get_image.py
-              --ignore=tests/unit/proxy/test_prisma_client_backoff_retry.py
-              --ignore=tests/unit/proxy/test_prompt_test_endpoint.py
-              --ignore=tests/unit/proxy/test_proxy_config_unit_test.py
-              --ignore=tests/unit/proxy/test_proxy_custom_auth.py
-              --ignore=tests/unit/proxy/test_proxy_reject_logging.py
-              --ignore=tests/unit/proxy/test_proxy_server.py
-              --ignore=tests/unit/proxy/test_proxy_setting_guardrails.py
-              --ignore=tests/unit/proxy/test_proxy_token_counter.py
-              --ignore=tests/unit/proxy/test_proxy_utils.py
-              --ignore=tests/unit/proxy/test_reducto_ocr_route.py
-              --ignore=tests/unit/proxy/test_response_polling_pre_call_checks.py
-              --ignore=tests/unit/proxy/test_server_root_path.py
-              --ignore=tests/unit/proxy/test_ui_path_detection.py
-              --ignore=tests/unit/proxy/test_unit_test_proxy_hooks.py
-              --ignore=tests/unit/proxy/test_update_spend.py
-              --ignore=tests/unit/proxy/test_zero_cost_model_budget_bypass.py
+              tests/unit/proxy/test_[_a-p]*.py
+              --deselect=tests/unit/proxy/test_aproxy_startup.py
+              --deselect=tests/unit/proxy/test_component_allowlists.py
+              --deselect=tests/unit/proxy/test_credential_slot_registry.py
+              --deselect=tests/unit/proxy/test_custom_callback_input.py
+              --deselect=tests/unit/proxy/test_custom_logger_s3_gcs.py
+              --deselect=tests/unit/proxy/test_custom_tokenizer_bug.py
+              --deselect=tests/unit/proxy/test_db_schema_changes.py
+              --deselect=tests/unit/proxy/test_deprecated_key_grace_period.py
+              --deselect=tests/unit/proxy/test_get_favicon.py
+              --deselect=tests/unit/proxy/test_get_image.py
+              --deselect=tests/unit/proxy/test_prisma_client_backoff_retry.py
+              --deselect=tests/unit/proxy/test_prometheus_metrics_server.py
+              --deselect=tests/unit/proxy/test_prompt_test_endpoint.py
+              --deselect=tests/unit/proxy/test_proxy_cli.py
+              --deselect=tests/unit/proxy/test_proxy_config_unit_test.py
+              --deselect=tests/unit/proxy/test_proxy_custom_auth.py
+              --deselect=tests/unit/proxy/test_proxy_reject_logging.py
+              --deselect=tests/unit/proxy/test_proxy_server.py
+              --deselect=tests/unit/proxy/test_proxy_setting_guardrails.py
+              --deselect=tests/unit/proxy/test_proxy_token_counter.py
+              --deselect=tests/unit/proxy/test_proxy_utils.py
             python-version: "3.12"
             workers: 4
             reruns: 0
@@ -697,6 +680,35 @@ jobs:
             job-timeout-minutes: 60
             dist: loadscope
 
+          - shard: proxy-infra-root-q-z
+            test-path: |-
+              tests/unit/proxy/test_[q-z]*.py
+              --deselect=tests/unit/proxy/test_reducto_ocr_route.py
+              --deselect=tests/unit/proxy/test_response_polling_pre_call_checks.py
+              --deselect=tests/unit/proxy/test_server_root_path.py
+              --deselect=tests/unit/proxy/test_ui_path_detection.py
+              --deselect=tests/unit/proxy/test_unit_test_proxy_hooks.py
+              --deselect=tests/unit/proxy/test_update_spend.py
+              --deselect=tests/unit/proxy/test_zero_cost_model_budget_bypass.py
+            python-version: "3.12"
+            workers: 4
+            reruns: 0
+            timeout-minutes: 20
+            job-timeout-minutes: 60
+            dist: loadscope
+
+          - shard: proxy-subprocess
+            test-path: |-
+              tests/unit/proxy/test_component_allowlists.py
+              tests/unit/proxy/test_prometheus_metrics_server.py
+              tests/unit/proxy/test_proxy_cli.py
+            python-version: "3.12"
+            workers: 2
+            reruns: 0
+            timeout-minutes: 20
+            job-timeout-minutes: 60
+            dist: loadscope
+
           - shard: caching-local
             test-path: >-
               tests/unit/caching
@@ -1032,7 +1044,7 @@ jobs:
           fi
           uv run --no-sync python -c 'import os, sys; print(sys.version); assert f"{sys.version_info.major}.{sys.version_info.minor}" == os.environ["UV_PYTHON"]'
 
-      - name: Check schema.prisma against the merge base
+      - name: Check litellm/schema.prisma against the merge base
         if: steps.changes.outputs.decision != 'skip' && matrix.shard == 'db-and-spend' && github.event_name == 'pull_request'
         timeout-minutes: 3
         env:
@@ -1045,7 +1057,7 @@ jobs:
           merge_base=$(gh api "repos/${GITHUB_REPOSITORY}/compare/${base_ref}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
           test -n "$merge_base"
           git fetch --no-tags --depth=1 origin "$merge_base"
-          uv run --no-sync python scripts/check_schema_compatibility.py --base-commit "$merge_base"
+          uv run --no-sync python litellm/scripts/check_schema_compatibility.py --base-commit "$merge_base"
 
       - name: Cache Prisma binaries
         if: steps.changes.outputs.decision != 'skip'
@@ -1110,7 +1122,7 @@ jobs:
             --timeout="${TEST_TIMEOUT_SECONDS}" \
             --rerun-except "from pytest-timeout" \
             --durations=20 \
-            --cov=./litellm --cov=./enterprise/litellm_enterprise \
+            --cov=./litellm --cov=./litellm/enterprise/litellm_enterprise \
             --cov-report=xml:coverage.xml \
             --cov-config=pyproject.toml
           status=$?
@@ -1140,7 +1152,7 @@ jobs:
     timeout-minutes: 20
     defaults:
       run:
-        working-directory: ui/litellm-dashboard
+        working-directory: litellm/ui/litellm-dashboard
 
     steps:
       - name: Checkout repository
@@ -1159,9 +1171,9 @@ jobs:
         if: steps.changes.outputs.decision != 'skip'
         uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
         with:
-          node-version-file: ui/litellm-dashboard/.nvmrc
+          node-version-file: litellm/ui/litellm-dashboard/.nvmrc
           cache: "npm"
-          cache-dependency-path: ui/litellm-dashboard/package-lock.json
+          cache-dependency-path: litellm/ui/litellm-dashboard/package-lock.json
 
       - name: Install dependencies
         if: steps.changes.outputs.decision != 'skip'
@@ -1331,17 +1343,16 @@ jobs:
 
       - name: Run unit tests
         run: |
-          for chart in helm/litellm-helm helm/litellm; do
-            declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
-            output="$(mktemp)"
-            helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
-            executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
-            if [ "$declared" != "$executed" ]; then
-              echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
-              exit 1
-            fi
-            echo "$chart: all $declared declared test suites ran"
-          done
+          chart=litellm/deploy/helm/litellm
+          declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
+          output="$(mktemp)"
+          helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
+          executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
+          if [ "$declared" != "$executed" ]; then
+            echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
+            exit 1
+          fi
+          echo "$chart: all $declared declared test suites ran"
   openai-sdk-compat:
     name: OpenAI ${{ matrix.openai-version }} compatibility
     needs: rust-bridge

.github/workflows/test-vscode-extension.yml

@@ -8,13 +8,13 @@ on:
       - main
       - "litellm_**"
     paths:
-      - "vscode-extension/**"
+      - "litellm/vscode-extension/**"
       - ".github/workflows/test-vscode-extension.yml"
   push:
     branches:
       - main
     paths:
-      - "vscode-extension/**"
+      - "litellm/vscode-extension/**"
       - ".github/workflows/test-vscode-extension.yml"
 
 concurrency:
@@ -27,7 +27,7 @@ jobs:
     timeout-minutes: 10
     defaults:
       run:
-        working-directory: vscode-extension
+        working-directory: litellm/vscode-extension
 
     steps:
       - name: Checkout repository
@@ -41,7 +41,7 @@ jobs:
         with:
           node-version: "24"
           cache: npm
-          cache-dependency-path: vscode-extension/package-lock.json
+          cache-dependency-path: litellm/vscode-extension/package-lock.json
 
       - name: Install dependencies
         run: npm ci
@@ -59,5 +59,5 @@ jobs:
         uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
         with:
           name: litellm-vscode
-          path: vscode-extension/*.vsix
+          path: litellm/vscode-extension/*.vsix
           if-no-files-found: error

.gitignore

@@ -18,7 +18,7 @@ __pycache__/
 # Rust bridge build artifacts (compiled, platform-specific; regenerated by maturin/cargo)
 litellm/rust_bridge/_native*.so
 litellm/rust_bridge/_native*.pyd
-litellm-rust/target/
+litellm/rust/target/
 
 # Python package build output
 dist/
@@ -55,14 +55,14 @@ hosted_config.yaml
 litellm/proxy/tests/node_modules
 litellm/proxy/tests/package.json
 litellm/proxy/tests/package-lock.json
-ui/litellm-dashboard/.next
-ui/litellm-dashboard/node_modules
-ui/litellm-dashboard/next-env.d.ts
+litellm/ui/litellm-dashboard/.next
+litellm/ui/litellm-dashboard/node_modules
+litellm/ui/litellm-dashboard/next-env.d.ts
 *.tsbuildinfo
-ui/litellm-dashboard/package.json
-ui/litellm-dashboard/package-lock.json
-helm/litellm-helm/*.tgz
-helm/*.tgz
+litellm/ui/litellm-dashboard/package.json
+litellm/ui/litellm-dashboard/package-lock.json
+litellm/deploy/helm/litellm/charts/*.tgz
+litellm/deploy/helm/*.tgz
 litellm/proxy/vertex_key.json
 **/.vim/
 **/node_modules
@@ -106,7 +106,7 @@ litellm_config.yaml
 litellm/proxy/to_delete_loadtest_work/*
 update_model_cost_map.py
 tests/unit/proxy/mcp_server/test_mcp_server_manager.py
-scripts/test_vertex_ai_search.py
+litellm/scripts/test_vertex_ai_search.py
 LAZY_LOADING_IMPROVEMENTS.md
 STABILIZATION_TODO.md
 **/test-results
@@ -146,7 +146,7 @@ crash.*.log
 # pytest coverage data
 .coverage
 
-ui/litellm-dashboard/out/
+litellm/ui/litellm-dashboard/out/
 litellm.log
 
 .coverage-rust

.npmrc

@@ -1,5 +0,0 @@
-# Supply-chain hardening
-# Packages needing lifecycle scripts: npm rebuild <pkg>
-ignore-scripts=true
-# Protects local npm install only — npm ci (used in CI) ignores this
-min-release-age=3

AGENTS.md

@@ -1,7 +1,6 @@
 Do not write comments unless they are any of:
 - absolutely necessary to explain some very complex business logic (in which case, keep it concise and clear)
 - used as an input for tools to read and act on. For example:
-  - entries in `.git-blame-ignore-revs` saying which commit is excluded from git blame
   - a lint or type checker suppression like `# mutable-ok` or `# pyright: ignore[reportArgumentType]  # <reason>` when introducing a truly unavoidable violation
 - a TODO or FIXME
   - Not great to have those, but if it's unavoidable, make sure to include a strong, concise reason for why it's there or, better yet, link to a GitHub issue for the follow-up work
@@ -31,9 +30,7 @@ A test must only fail when litellm code changes. Never pin facts we don't own (a
 
 End-to-end tests belong in `tests/e2e/`. A rule about one directory's files, like its harness or which CI runs it, goes in that directory's `AGENTS.md`, never only in a README or here. Read a directory's `AGENTS.md` before editing files in it
 
-Before you build a new way to move data between the endpoints, the provider transformations, and cost tracking or logging, read the architecture decision records in `docs/adrs/`, which say why those layers are shaped the way they are
-
-When creating PRs, target the repository's current default branch for both internal and external / OSS contributions. Check it with `python3 scripts/default_branch.py --branch` instead of assuming a branch name or relying on cached `origin/HEAD`
+When creating PRs, target the repository's current default branch for both internal and external / OSS contributions. Check it with `python3 litellm/scripts/default_branch.py --branch` instead of assuming a branch name or relying on cached `origin/HEAD`
 
 When writing a PR body, treat the comments and imperative instructions inside .github/PULL_REQUEST_TEMPLATE/general.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/PULL_REQUEST_TEMPLATE/general.md from disk before writing a PR body to make sure you see every comment rule. A section you have nothing to put in (Relevant issues, Affected release, Linear ticket, Caveats, QA runbook, and so on) is removed entirely, heading included, never left as an empty title
 
@@ -59,17 +56,17 @@ Don't hesitate to use values in .env to get needed API keys and other secrets, a
 
 Python max line length is 120, not 88
 
-The five lint gates (`scripts/ruff_strict_gate.py`, `scripts/type_discipline_gate.py`, `scripts/test_quality_gate.py`, `scripts/openapi_docs_gate.py`, `scripts/type_check_gate.py`) compare each rule's codebase count on your branch against the count at its merge-base with the default branch, and a rule may not grow. There are no budget files to edit or ratchet: when a gate fails, fix the violations the branch introduced or remove at least as many of that rule elsewhere in the tree. The one exception is `reportAny` / `reportExplicitAny`, which share a fixed codebase-wide cap in `ANY_CAPS` in `scripts/type_check_gate.py` because Any spreads past the lines you touch. A branch may add Anys while the total stays under that cap. Only lower `ANY_CAPS`, and only in its own PR to the default branch, never raise it on a feature branch
+The five lint gates (`litellm/scripts/ruff_strict_gate.py`, `litellm/scripts/type_discipline_gate.py`, `litellm/scripts/test_quality_gate.py`, `litellm/scripts/openapi_docs_gate.py`, `litellm/scripts/type_check_gate.py`) compare each rule's codebase count on your branch against the count at its merge-base with the default branch, and a rule may not grow. There are no budget files to edit or ratchet: when a gate fails, fix the violations the branch introduced or remove at least as many of that rule elsewhere in the tree. The one exception is `reportAny` / `reportExplicitAny`, which share a fixed codebase-wide cap in `ANY_CAPS` in `litellm/scripts/type_check_gate.py` because Any spreads past the lines you touch. A branch may add Anys while the total stays under that cap. Only lower `ANY_CAPS`, and only in its own PR to the default branch, never raise it on a feature branch
 
 `make check` (f.k.a. `make pre-commit`, which still works identically as an alias) saves its complete output to a log file in .git (overwriting previous logs) and prints that path as its first and last output lines. To inspect a run, read or grep that log instead of re-running the multi-minute checks just to see a different slice
 
-`make check`, `make lint`, `scripts/pre_commit_lint.sh`, and the standalone lint gates (`scripts/ruff_strict_gate.py`, `scripts/type_discipline_gate.py`, `scripts/type_check_gate.py`) each hold one of 2 machine-wide slots, so when other sessions or worktrees on the same box are already running heavy work, yours prints "all N machine-wide slots are busy" followed by "queueing" and then stays quiet until a slot frees. Give the command a long timeout and let it wait rather than killing it, retrying it, or assuming it hung. Don't change the # of machine-wide slots or make it unlimited by setting `LITELLM_GATE_SLOTS=0`
+`make check`, `make lint`, `litellm/scripts/pre_commit_lint.sh`, and the standalone lint gates (`litellm/scripts/ruff_strict_gate.py`, `litellm/scripts/type_discipline_gate.py`, `litellm/scripts/type_check_gate.py`) each hold one of 2 machine-wide slots, so when other sessions or worktrees on the same box are already running heavy work, yours prints "all N machine-wide slots are busy" followed by "queueing" and then stays quiet until a slot frees. Give the command a long timeout and let it wait rather than killing it, retrying it, or assuming it hung. Don't change the # of machine-wide slots or make it unlimited by setting `LITELLM_GATE_SLOTS=0`
 
 If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing `reportAny` / `reportExplicitAny` closer to their cap in `ANY_CAPS`, just validate it in the caller with Pydantic (a model or `TypeAdapter` that returns the typed thing or raises will do) and then pass the now typed variable in
 
 If you get an LIT001 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in `tuple()` / `MappingProxyType()` / `frozenset()` instead of seeding an empty `list`/`dict`/`set` and mutating it over time. Ideally, `# mutable-ok` is never used. Reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason
 
-Every lint or type suppression must name the exact rule inside brackets and carry a reason comment, e.g. `# pyright: ignore[reportArgumentType]  # stubs lack async overload` or `# noqa: TID251  # <reason>`. `# type: ignore` is banned (LIT009): pyrightconfig.json sets `enableTypeIgnoreComments` to false, so it silently does nothing
+Every lint or type suppression must name the exact rule inside brackets and carry a reason comment, e.g. `# pyright: ignore[reportArgumentType]  # stubs lack async overload` or `# noqa: TID251  # <reason>`. `# type: ignore` is banned (LIT009): `[tool.pyright]` in pyproject.toml sets `enableTypeIgnoreComments` to false, so it silently does nothing
 
 Commit and push your work when you're done without asking
 
@@ -91,8 +88,6 @@ Do not put names of customers or customer company names in code, PR descriptions
 
 CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
 
-For database operations, connection ownership and proxy dependency wiring, follow [database-access](.agents/skills/database-access/SKILL.md)
-
 Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No `UPDATE`, `DELETE` or `MERGE`, and no `INSERT ... SELECT`: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. `tests/code_coverage_tests/check_migrations_no_data_rewrites.py` enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement `-- data-migration-ok: <what bounds it>`
 
 Follow these coding conventions for new/updated code (a three-line fix in a legacy file shouldn't trigger huge drive-by refactors):

ARCHITECTURE.md

@@ -1,419 +0,0 @@
-# LiteLLM Architecture - LiteLLM SDK + AI Gateway
-
-This document helps contributors understand where to make changes in LiteLLM.
-
-For why a layer is shaped the way it is, read the architecture decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) (Python SDK and proxy) and [`litellm-rust/docs/adrs/`](litellm-rust/docs/adrs/AGENTS.md) (Rust core)
-
----
-
-## How It Works
-
-The LiteLLM AI Gateway (Proxy) uses the LiteLLM SDK internally for all LLM calls:
-
-```
-OpenAI SDK (client)    ──▶  LiteLLM AI Gateway (proxy/)  ──▶  LiteLLM SDK (litellm/)  ──▶  LLM API
-Anthropic SDK (client) ──▶  LiteLLMAI Gateway (proxy/)  ──▶  LiteLLM SDK (litellm/)  ──▶  LLM API
-Any HTTP client        ──▶  LiteLLMAI Gateway (proxy/)  ──▶  LiteLLM SDK (litellm/)  ──▶  LLM API
-```
-
-The **AI Gateway** adds authentication, rate limiting, budgets, and routing on top of the SDK.
-The **SDK** handles the actual LLM provider calls, request/response transformations, and streaming.
-
----
-
-## 1. AI Gateway (Proxy) Request Flow
-
-The AI Gateway (`litellm/proxy/`) wraps the SDK with authentication, rate limiting, and management features.
-
-```mermaid
-sequenceDiagram
-    participant Client
-    participant ProxyServer as proxy/proxy_server.py
-    participant Auth as proxy/auth/user_api_key_auth.py
-    participant Redis as Redis Cache
-    participant Hooks as proxy/hooks/
-    participant Router as router.py
-    participant Main as main.py + utils.py
-    participant Handler as llms/custom_httpx/llm_http_handler.py
-    participant Transform as llms/{provider}/chat/transformation.py
-    participant Provider as LLM Provider API
-    participant CostCalc as cost_calculator.py
-    participant LoggingObj as litellm_logging.py
-    participant DBWriter as db/db_spend_update_writer.py
-    participant Postgres as PostgreSQL
-
-    %% Request Flow
-    Client->>ProxyServer: POST /v1/chat/completions
-    ProxyServer->>Auth: user_api_key_auth()
-    Auth->>Redis: Check API key cache
-    Redis-->>Auth: Key info + spend limits
-    ProxyServer->>Hooks: parallel_request_limiter, cache_control_check
-    Hooks->>Redis: Check/increment rate limit counters
-    ProxyServer->>Router: route_request()
-    Router->>Main: litellm.acompletion()
-    Main->>Handler: BaseLLMHTTPHandler.completion()
-    Handler->>Transform: ProviderConfig.transform_request()
-    Handler->>Provider: HTTP Request
-    Provider-->>Handler: Response
-    Handler->>Transform: ProviderConfig.transform_response()
-    Transform-->>Handler: ModelResponse
-    Handler-->>Main: ModelResponse
-    
-    %% Cost Attribution (in utils.py wrapper)
-    Main->>LoggingObj: update_response_metadata()
-    LoggingObj->>CostCalc: _response_cost_calculator()
-    CostCalc->>CostCalc: completion_cost(tokens × price)
-    CostCalc-->>LoggingObj: response_cost
-    LoggingObj-->>Main: Set response._hidden_params["response_cost"]
-    Main-->>ProxyServer: ModelResponse (with cost in _hidden_params)
-    
-    %% Response Headers + Async Logging
-    ProxyServer->>ProxyServer: Extract cost from hidden_params
-    ProxyServer->>LoggingObj: async_success_handler()
-    LoggingObj->>Hooks: async_log_success_event()
-    Hooks->>DBWriter: update_database(response_cost)
-    DBWriter->>Redis: Queue spend increment
-    DBWriter->>Postgres: Batch write spend logs (async)
-    ProxyServer-->>Client: ModelResponse + x-litellm-response-cost header
-```
-
-### Proxy Components
-
-```mermaid
-graph TD
-    subgraph "Incoming Request"
-        Client["POST /v1/chat/completions"]
-    end
-
-    subgraph "proxy/proxy_server.py"
-        Endpoint["chat_completion()"]
-    end
-
-    subgraph "proxy/auth/"
-        Auth["user_api_key_auth()"]
-    end
-
-    subgraph "proxy/"
-        PreCall["litellm_pre_call_utils.py"]
-        RouteRequest["route_llm_request.py"]
-    end
-
-    subgraph "litellm/"
-        Router["router.py"]
-        Main["main.py"]
-    end
-
-    subgraph "Infrastructure"
-        DualCache["DualCache<br/>(in-memory + Redis)"]
-        Postgres["PostgreSQL<br/>(keys, teams, spend logs)"]
-    end
-
-    Client --> Endpoint
-    Endpoint --> Auth
-    Auth --> DualCache
-    DualCache -.->|cache miss| Postgres
-    Auth --> PreCall
-    PreCall --> RouteRequest
-    RouteRequest --> Router
-    Router --> DualCache
-    Router --> Main
-    Main --> Client
-```
-
-**Key proxy files:**
-- `proxy/proxy_server.py` - Main API endpoints
-- `proxy/auth/` - Authentication (API keys, JWT, OAuth2)
-- `proxy/hooks/` - Proxy-level callbacks
-- `router.py` - Load balancing, fallbacks
-- `router_strategy/` - Routing algorithms (`lowest_latency.py`, `simple_shuffle.py`, etc.)
-
-**LLM-specific proxy endpoints:**
-
-| Endpoint | Directory | Purpose |
-|----------|-----------|---------|
-| `/v1/messages` | `proxy/anthropic_endpoints/` | Anthropic Messages API |
-| `/vertex-ai/*` | `proxy/vertex_ai_endpoints/` | Vertex AI passthrough |
-| `/gemini/*` | `proxy/google_endpoints/` | Google AI Studio passthrough |
-| `/v1/images/*` | `proxy/image_endpoints/` | Image generation |
-| `/v1/batches` | `proxy/batches_endpoints/` | Batch processing |
-| `/v1/files` | `proxy/openai_files_endpoints/` | File uploads |
-| `/v1/fine_tuning` | `proxy/fine_tuning_endpoints/` | Fine-tuning jobs |
-| `/v1/rerank` | `proxy/rerank_endpoints/` | Reranking |
-| `/v1/responses` | `proxy/response_api_endpoints/` | OpenAI Responses API |
-| `/v1/vector_stores` | `proxy/vector_store_endpoints/` | Vector stores |
-| `/*` (passthrough) | `proxy/pass_through_endpoints/` | Direct provider passthrough |
-
-**Proxy Hooks** (`proxy/hooks/__init__.py`):
-
-| Hook | File | Purpose |
-|------|------|---------|
-| `parallel_request_limiter` | `proxy/hooks/parallel_request_limiter_v3.py` | Rate limiting per key/user |
-| `cache_control_check` | `proxy/hooks/cache_control_check.py` | Cache validation |
-| `responses_id_security` | `proxy/hooks/responses_id_security.py` | Response ID validation |
-| `litellm_skills` | `proxy/hooks/litellm_skills/main.py` | Skills injection |
-
-To add a new proxy hook, implement `CustomLogger` and register in `PROXY_HOOKS`.
-
-### Infrastructure Components
-
-The AI Gateway uses external infrastructure for persistence and caching:
-
-```mermaid
-graph LR
-    subgraph "AI Gateway (proxy/)"
-        Proxy["proxy_server.py"]
-        Auth["auth/user_api_key_auth.py"]
-        DBWriter["db/db_spend_update_writer.py<br/>DBSpendUpdateWriter"]
-        InternalCache["utils.py<br/>InternalUsageCache"]
-        CostCallback["hooks/proxy_track_cost_callback.py<br/>_ProxyDBLogger"]
-        Scheduler["APScheduler<br/>ProxyStartupEvent"]
-    end
-
-    subgraph "SDK (litellm/)"
-        Router["router.py<br/>Router.cache (DualCache)"]
-        LLMCache["caching/caching_handler.py<br/>LLMCachingHandler"]
-        CacheClass["caching/caching.py<br/>Cache"]
-    end
-
-    subgraph "Redis (caching/redis_cache.py)"
-        RateLimit["Rate Limit Counters"]
-        SpendQueue["Spend Increment Queue"]
-        KeyCache["API Key Cache"]
-        TPM_RPM["TPM/RPM Tracking"]
-        Cooldowns["Deployment Cooldowns"]
-        LLMResponseCache["LLM Response Cache"]
-    end
-
-    subgraph "PostgreSQL (proxy/schema.prisma)"
-        Keys["LiteLLM_VerificationToken"]
-        Teams["LiteLLM_TeamTable"]
-        SpendLogs["LiteLLM_SpendLogs"]
-        Users["LiteLLM_UserTable"]
-    end
-
-    Auth --> InternalCache
-    InternalCache --> KeyCache
-    InternalCache -.->|cache miss| Keys
-    InternalCache --> RateLimit
-    Router --> TPM_RPM
-    Router --> Cooldowns
-    LLMCache --> CacheClass
-    CacheClass --> LLMResponseCache
-    CostCallback --> DBWriter
-    DBWriter --> SpendQueue
-    DBWriter --> SpendLogs
-    Scheduler --> SpendLogs
-    Scheduler --> Keys
-```
-
-| Component | Purpose | Key Files/Classes |
-|-----------|---------|-------------------|
-| **Redis** | Rate limiting, API key caching, TPM/RPM tracking, cooldowns, LLM response caching, spend queuing | `caching/redis_cache.py` (`RedisCache`), `caching/dual_cache.py` (`DualCache`) |
-| **PostgreSQL** | API keys, teams, users, spend logs | `proxy/utils.py` (`PrismaClient`), `proxy/schema.prisma` |
-| **InternalUsageCache** | Proxy-level cache for rate limits + API keys (in-memory + Redis) | `proxy/utils.py` (`InternalUsageCache`) |
-| **Router.cache** | TPM/RPM tracking, deployment cooldowns, client caching (in-memory + Redis) | `router.py` (`Router.cache: DualCache`) |
-| **LLMCachingHandler** | SDK-level LLM response/embedding caching | `caching/caching_handler.py` (`LLMCachingHandler`), `caching/caching.py` (`Cache`) |
-| **DBSpendUpdateWriter** | Batches spend updates to reduce DB writes | `proxy/db/db_spend_update_writer.py` (`DBSpendUpdateWriter`) |
-| **Cost Tracking** | Calculates and logs response costs | `proxy/hooks/proxy_track_cost_callback.py` (`_ProxyDBLogger`) |
-
-**Background Jobs** (APScheduler, initialized in `proxy/proxy_server.py` → `ProxyStartupEvent.initialize_scheduled_background_jobs()`):
-
-| Job | Interval | Purpose | Key Files |
-|-----|----------|---------|-----------|
-| `update_spend` | 60s | Batch write spend logs to PostgreSQL | `proxy/db/db_spend_update_writer.py` |
-| `reset_budget` | 10-12min | Reset budgets for keys/users/teams | `proxy/common_utils/reset_budget_job.py` |
-| `add_deployment` | 10s | Sync new model deployments from DB | `proxy/proxy_server.py` (`ProxyConfig`) |
-| `cleanup_old_spend_logs` | cron/interval | Delete old spend logs | `proxy/db/db_transaction_queue/spend_log_cleanup.py` |
-| `check_batch_cost` | 30min | Calculate costs for batch jobs | `enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py` |
-| `check_responses_cost` | 30min | Calculate costs for responses API | `enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py` |
-| `process_rotations` | 1hr | Auto-rotate API keys | `proxy/common_utils/key_rotation_manager.py` |
-| `_run_background_health_check` | continuous | Health check model deployments | `proxy/proxy_server.py` |
-| `send_weekly_spend_report` | weekly | Slack spend alerts | `proxy/utils.py` (`SlackAlerting`) |
-| `send_monthly_spend_report` | monthly | Slack spend alerts | `proxy/utils.py` (`SlackAlerting`) |
-
-**Cost Attribution Flow:**
-1. LLM response returns to `utils.py` wrapper after `litellm.acompletion()` completes
-2. `update_response_metadata()` (`litellm_core_utils/llm_response_utils/response_metadata.py`) is called
-3. `logging_obj._response_cost_calculator()` (`litellm_core_utils/litellm_logging.py`) calculates cost via `litellm.completion_cost()` (`cost_calculator.py`)
-4. Cost is stored in `response._hidden_params["response_cost"]`
-5. `proxy/common_request_processing.py` extracts cost from `hidden_params` and adds to response headers (`x-litellm-response-cost`)
-6. `logging_obj.async_success_handler()` triggers callbacks including `_ProxyDBLogger.async_log_success_event()`
-7. `DBSpendUpdateWriter.update_database()` queues spend increments to Redis
-8. Background job `update_spend` flushes queued spend to PostgreSQL every 60s
-
-Cost lookup falls back to the response model when the provider-reported name cannot be priced. Explicitly registered zero rates remain zero, and custom deployment pricing or an explicit `base_model` retain precedence
-
-### Data Access Layer (Models & Repositories)
-
-Database entities and the operations on them live in two packages at the root of `litellm/` so both the gateway (`proxy/`) and the SDK can use them without importing proxy internals:
-
-- `litellm/models/` holds the canonical Pydantic definitions for every persisted entity (`LiteLLM_VerificationToken`, `LiteLLM_TeamTable`, `LiteLLM_UserTable`, etc.). `proxy/_types.py` re-exports these for backwards compatibility, so existing imports keep working.
-- `litellm/repositories/` holds the data-access layer. `BaseRepository[T]` provides the generic CRUD (`find_by_id`, `find_many`, `create`, `update`, `delete`, `count`, `exists`); entity repositories such as `VerificationTokenRepository`, `TeamRepository`, and `UserRepository` add domain-specific queries and writes on top of it.
-
-Conventions to follow when touching this layer:
-
-| Concern | How it's handled |
-|---------|------------------|
-| JSON columns | Prisma `Json` columns are stored as JSON strings. Repositories `json.dumps()` on write and `json.loads()` on read (see `_to_model` and the `_build_*_data` helpers). |
-| Archive-then-delete | `delete_team` / `delete_token` copy the row into the `LiteLLM_Deleted*` table and delete the original inside a single `prisma_client.db.tx()` transaction. Archive payloads are built explicitly so only columns that exist on the archive table are written. |
-| Column vs. field names | Where a model field differs from its DB column (for example `org_id` maps to the `organization_id` column), the repository translates in both directions rather than relying on Pydantic to guess. |
-| Array mutations | Adds use Prisma's atomic `push` (`add_member`, `add_admin`, `add_models`) to avoid read-modify-write races. Removals fall back to read-modify-write because Prisma has no atomic array remove. |
-
-To add a new entity, define the model under `litellm/models/`, re-export it from `proxy/_types.py` if existing code imports it from there, and add a repository under `litellm/repositories/` (subclass `BaseRepository` for plain CRUD, or add bespoke methods when the entity needs encryption, archiving, or atomic array updates). Mirror the tests in `tests/unit/repositories/`.
-
----
-
-## 2. SDK Request Flow
-
-The SDK (`litellm/`) provides the core LLM calling functionality used by both direct SDK users and the AI Gateway.
-
-```mermaid
-graph TD
-    subgraph "SDK Entry Points"
-        Completion["litellm.completion()"]
-        Messages["litellm.messages()"]
-    end
-
-    subgraph "main.py"
-        Main["completion()<br/>acompletion()"]
-    end
-
-    subgraph "utils.py"
-        GetProvider["get_llm_provider()"]
-    end
-
-    subgraph "llms/custom_httpx/"
-        Handler["llm_http_handler.py<br/>BaseLLMHTTPHandler"]
-        HTTP["http_handler.py<br/>HTTPHandler / AsyncHTTPHandler"]
-    end
-
-    subgraph "llms/{provider}/chat/"
-        TransformReq["transform_request()"]
-        TransformResp["transform_response()"]
-    end
-
-    subgraph "litellm_core_utils/"
-        Streaming["streaming_handler.py"]
-    end
-
-    subgraph "integrations/ (async, off main thread)"
-        Callbacks["custom_logger.py<br/>Langfuse, Datadog, etc."]
-    end
-
-    Completion --> Main
-    Messages --> Main
-    Main --> GetProvider
-    GetProvider --> Handler
-    Handler --> TransformReq
-    TransformReq --> HTTP
-    HTTP --> Provider["LLM Provider API"]
-    Provider --> HTTP
-    HTTP --> TransformResp
-    TransformResp --> Streaming
-    Streaming --> Response["ModelResponse"]
-    Response -.->|async| Callbacks
-```
-
-**Key SDK files:**
-- `main.py` - Entry points: `completion()`, `acompletion()`, `embedding()`
-- `utils.py` - `get_llm_provider()` resolves model → provider
-- `llms/custom_httpx/llm_http_handler.py` - Central HTTP orchestrator
-- `llms/custom_httpx/http_handler.py` - Low-level HTTP client
-- `llms/{provider}/chat/transformation.py` - Provider-specific transformations
-- `litellm_core_utils/streaming_handler.py` - Streaming response handling
-- `integrations/` - Async callbacks (Langfuse, Datadog, etc.)
-
----
-
-## 3. Translation Layer
-
-When a request comes in, it goes through a **translation layer** that converts between API formats.
-Each translation is isolated in its own file, making it easy to test and modify independently.
-
-### Where to find translations
-
-| Incoming API | Provider | Translation File |
-|--------------|----------|------------------|
-| `/v1/chat/completions` | Anthropic | `llms/anthropic/chat/transformation.py` |
-| `/v1/chat/completions` | Bedrock Converse | `llms/bedrock/chat/converse_transformation.py` |
-| `/v1/chat/completions` | Bedrock Invoke | `llms/bedrock/chat/invoke_transformations/anthropic_claude_transformation.py` |
-| `/v1/chat/completions` | Gemini | `llms/gemini/chat/transformation.py` |
-| `/v1/chat/completions` | Vertex AI | `llms/vertex_ai/gemini/transformation.py` |
-| `/v1/chat/completions` | OpenAI | `llms/openai/chat/gpt_transformation.py` |
-| `/v1/messages` (passthrough) | Anthropic | `llms/anthropic/messages/native/transformation.py` |
-| `/v1/messages` (passthrough) | Bedrock | `llms/bedrock/messages/invoke_transformations/anthropic_claude_transformation.py` |
-| `/v1/messages` (passthrough) | Vertex AI | `llms/vertex_ai/vertex_ai_partner_models/anthropic/messages/transformation.py` |
-| Passthrough endpoints | All | `proxy/pass_through_endpoints/llm_provider_handlers/` |
-
-### Example: Debugging prompt caching
-
-If `/v1/messages` → Bedrock Converse prompt caching isn't working but Bedrock Invoke works:
-
-1. **Bedrock Converse translation**: `llms/bedrock/chat/converse_transformation.py`
-2. **Bedrock Invoke translation**: `llms/bedrock/chat/invoke_transformations/anthropic_claude_transformation.py`
-3. Compare how each handles `cache_control` in `transform_request()`
-
-### How translations work
-
-Each provider has a `Config` class that inherits from `BaseConfig` (`llms/base_llm/chat/transformation.py`):
-
-```python
-class ProviderConfig(BaseConfig):
-    def transform_request(self, model, messages, optional_params, litellm_params, headers):
-        # Convert OpenAI format → Provider format
-        return {"messages": transformed_messages, ...}
-    
-    def transform_response(self, model, raw_response, model_response, logging_obj, ...):
-        # Convert Provider format → OpenAI format
-        return ModelResponse(choices=[...], usage=Usage(...))
-```
-
-The `BaseLLMHTTPHandler` (`llms/custom_httpx/llm_http_handler.py`) calls these methods - you never need to modify the handler itself.
-
----
-
-## 4. Adding/Modifying Providers
-
-### To add a new provider:
-
-1. Create `llms/{provider}/chat/transformation.py`
-2. Implement `Config` class with `transform_request()` and `transform_response()`
-3. Add tests in `tests/llm_translation/test_{provider}.py`
-
-### To add a feature (e.g., prompt caching):
-
-1. Find the translation file from the table above
-2. Modify `transform_request()` to handle the new parameter
-3. Add unit tests that verify the transformation
-
-### Testing checklist
-
-When adding a feature, verify it works across all paths:
-
-| Test | File Pattern |
-|------|--------------|
-| OpenAI passthrough | `tests/llm_translation/test_openai*.py` |
-| Anthropic direct | `tests/llm_translation/test_anthropic*.py` |
-| Bedrock Invoke | `tests/llm_translation/test_bedrock*.py` |
-| Bedrock Converse | `tests/llm_translation/test_bedrock*converse*.py` |
-| Vertex AI | `tests/llm_translation/test_vertex*.py` |
-| Gemini | `tests/llm_translation/test_gemini*.py` |
-
-### Unit testing translations
-
-Translations are designed to be unit testable without making API calls:
-
-```python
-from litellm.llms.bedrock.chat.converse_transformation import BedrockConverseConfig
-
-def test_prompt_caching_transform():
-    config = BedrockConverseConfig()
-    result = config.transform_request(
-        model="anthropic.claude-3-opus",
-        messages=[{"role": "user", "content": "test", "cache_control": {"type": "ephemeral"}}],
-        optional_params={},
-        litellm_params={},
-        headers={}
-    )
-    assert "cachePoint" in str(result)  # Verify cache_control was translated
-```

GEMINI.md

@@ -1 +0,0 @@
-Read @AGENTS.md for coding guidelines

LICENSE

@@ -1,6 +1,6 @@
 Portions of this software are licensed as follows:
 
-* All content that resides under the "enterprise/" directory of this repository, if that directory exists, is licensed under the license defined in "enterprise/LICENSE".
+* All content that resides under the "litellm/enterprise/" directory of this repository, if that directory exists, is licensed under the license defined in "litellm/enterprise/LICENSE".
 * Content outside of the above mentioned directories or restrictions above is available under the MIT license as defined below.
 ---
 MIT License

Makefile

@@ -54,20 +54,24 @@ help:
 	@echo "  make test-integration   - Run integration tests"
 	@echo "  make test-unit-helm     - Run helm unit tests"
 	@echo "  make test-rust-extension - Build the Rust extension and run its public Python tests"
-	@echo "  make rust-sqlx-prepare  - Refresh litellm-rust/crates/db/.sqlx against a migrated Postgres container"
+	@echo "  make rust-sqlx-prepare  - Refresh litellm/rust/crates/db/.sqlx against a migrated Postgres container"
 	@echo ""
 	@echo "Heavy targets (check, lint) queue for LITELLM_GATE_SLOTS machine-wide"
 	@echo "slots (default 2; 0 disables) so parallel sessions don't thrash one machine."
 
 UV := uv
+export RUSTUP_TOOLCHAIN ?= $(shell sed -n 's/^[[:space:]]*channel[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' litellm/rust/rust-toolchain.toml)
+export CARGO_NET_RETRY ?= 5
+export CARGO_HTTP_MULTIPLEXING ?= false
+export SQLX_OFFLINE ?= true
 UV_RUN := $(UV) run --no-sync
 BASE_REF ?=
 export BASE_REF
-RESOLVE_BASE = python3 scripts/default_branch.py --base "$(BASE_REF)"
+RESOLVE_BASE = python3 litellm/scripts/default_branch.py --base "$(BASE_REF)"
 
 # Machine-wide slot queue for the heavy targets below; python3 + stdlib only, so
-# it runs before any venv exists. See scripts/gate_slot_lock.py.
-GATE_SLOT_LOCK := python3 scripts/gate_slot_lock.py
+# it runs before any venv exists. See litellm/scripts/gate_slot_lock.py.
+GATE_SLOT_LOCK := python3 litellm/scripts/gate_slot_lock.py
 
 LINT_DEP_INSTALL ?= install-dev
 LINT_E2E_DEP_INSTALL ?= lint-install
@@ -90,8 +94,8 @@ install-dev:
 # machine-wide slots the CPU-bound gates below share.
 bootstrap:
 	$(UV) sync --inexact --frozen --extra proxy --group proxy-dev --group e2e-dev
-	$(UV_RUN) python scripts/prisma_generate_if_needed.py
-	cd ui/litellm-dashboard && ../../scripts/with_dashboard_node.sh npm install --no-audit --no-fund
+	$(UV_RUN) python litellm/scripts/prisma_generate_if_needed.py
+	cd litellm/ui/litellm-dashboard && ../../scripts/with_dashboard_node.sh npm install --no-audit --no-fund
 	@main_root=$$(git worktree list --porcelain | head -1 | sed 's/^worktree //'); \
 	if [ "$$main_root" != "$$(git rev-parse --show-toplevel)" ] && [ -f "$$main_root/.env" ] && [ ! -f .env ]; then \
 		cp "$$main_root/.env" .env && echo "bootstrap: copied .env from $$main_root"; \
@@ -123,10 +127,10 @@ install-helm-unittest:
 # Install git hooks that enforce Conventional Commits.
 # Opt-in: not chained into install-dev.
 install-hooks:
-	./scripts/install_git_hooks.sh
+	./litellm/scripts/install_git_hooks.sh
 
 # Formatting
-# Wrap width is ruff.toml's single source of truth (line-length = 120), shared by the
+# Wrap width is pyproject.toml's single source of truth (line-length = 120), shared by the
 # formatter and the import sorter so there's no 88-vs-120 split to reconcile.
 format: install-dev
 	cd litellm && $(UV_RUN) ruff format --exclude '/enterprise/' . && cd ..
@@ -139,12 +143,12 @@ lint-fetch-base:
 
 # Mirror test-linting.yml's python job environment: the proxy-dev group plus a generated
 # Prisma client, so `basedpyright tests/e2e` resolves the same modules CI does. The
-# basedpyright gate itself no longer measures here (scripts/type_check_gate.py provisions its
+# basedpyright gate itself no longer measures here (litellm/scripts/type_check_gate.py provisions its
 # own .venv-typecheck). --inexact tops up the venv instead of pruning the proxy extras
 # gen:api and the running proxy need.
 lint-install:
 	$(UV) sync --inexact --frozen --group proxy-dev --group e2e-dev
-	$(UV_RUN) python scripts/prisma_generate_if_needed.py
+	$(UV_RUN) python litellm/scripts/prisma_generate_if_needed.py
 
 # Diff-scoped format check, mirroring test-linting.yml's "Check ruff format" step:
 # only the litellm Python files changed vs the base are checked, so a pre-existing
@@ -163,9 +167,9 @@ lint-format-check-changed: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
 
 # Linting targets
 lint-ruff: $(LINT_DEP_INSTALL)
-	$(UV_RUN) python scripts/check_mcp_operation_boundary.py
+	$(UV_RUN) python litellm/scripts/check_mcp_operation_boundary.py
 	cd litellm && $(UV_RUN) ruff check . && cd ..
-	$(UV_RUN) ruff check --config ruff-tests.toml tests
+	$(UV_RUN) ruff check --config .github/linters/ruff-tests.toml tests
 
 # faster linter for developing ...
 # inspiration from:
@@ -204,7 +208,7 @@ lint-ruff-FULL-dev: install-dev
 	else echo "No changed .py files to check."; fi
 
 lint-basedpyright: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
-	$(UV_RUN) python scripts/type_check_gate.py --base "$(BASE_REF)"
+	$(UV_RUN) python litellm/scripts/type_check_gate.py --base "$(BASE_REF)"
 
 lint-e2e-basedpyright: $(LINT_E2E_DEP_INSTALL)
 	$(UV_RUN) basedpyright tests/e2e tests/e2e_harness
@@ -212,29 +216,29 @@ lint-e2e-basedpyright: $(LINT_E2E_DEP_INSTALL)
 # Type-discipline gate (mutable collections / casts / type guards / kwargs /
 # unexplained suppressions), the test-linting.yml step `make lint` used to omit.
 lint-type-discipline: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
-	$(UV_RUN) python scripts/type_discipline_gate.py --base "$(BASE_REF)"
+	$(UV_RUN) python litellm/scripts/type_discipline_gate.py --base "$(BASE_REF)"
 
 # Test-quality gate (zero-assert / mock-echo tests, sys.path.insert, raw env writes,
 # litellm module-global mutation, credential-gated skips, conftest snapshot
 # inventory), counted across tests/ the same delta-vs-base way.
 lint-test-quality: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
-	$(UV_RUN) python scripts/test_quality_gate.py --base "$(BASE_REF)"
+	$(UV_RUN) python litellm/scripts/test_quality_gate.py --base "$(BASE_REF)"
 
 # OpenAPI docs gate (POST routes without a request body schema, a description or a
 # 200 schema in app.openapi()), counted the same delta-vs-base way so a new endpoint
 # has to show up documented in Swagger UI.
 lint-openapi-docs: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
-	$(UV_RUN) python scripts/openapi_docs_gate.py --base "$(BASE_REF)"
+	$(UV_RUN) python litellm/scripts/openapi_docs_gate.py --base "$(BASE_REF)"
 
 lint-format: format-check
 
 lint-ruff-strict: install-dev
-	$(UV_RUN) python scripts/ruff_strict_gate.py --base "$(BASE_REF)"
+	$(UV_RUN) python litellm/scripts/ruff_strict_gate.py --base "$(BASE_REF)"
 
 # Strict gate, invoked the same way CI does in test-linting.yml so a local pass
 # means the CI check will pass too.
 lint-gate: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
-	$(UV_RUN) python scripts/ruff_strict_gate.py --base "$(BASE_REF)"
+	$(UV_RUN) python litellm/scripts/ruff_strict_gate.py --base "$(BASE_REF)"
 
 check-circular-imports: $(LINT_DEP_INSTALL)
 	cd litellm && $(UV_RUN) python ../tests/documentation_tests/test_circular_imports.py && cd ..
@@ -272,7 +276,7 @@ check:
 	@$(GATE_SLOT_LOCK) $(MAKE) check-inner
 
 check-inner: bootstrap
-	./scripts/pre_commit_lint.sh
+	./litellm/scripts/pre_commit_lint.sh
 
 pre-commit:
 	@echo "make pre-commit is a legacy alias; use make check" >&2
@@ -294,7 +298,7 @@ test-rust-extension:
 	"$$temporary/venv/bin/python" -I -m pytest --import-mode=importlib -m requires_rust_extension tests/test_litellm_rust
 
 rust-sqlx-prepare:
-	cd litellm-rust && cargo run -p litellm-db-testing --bin sqlx-prepare
+	cd litellm/rust && cargo run -p litellm-db-testing --bin sqlx-prepare
 
 test: install-test-deps
 	$(UV_RUN) pytest tests/
@@ -341,7 +345,7 @@ test-integration: install-test-deps
 	$(UV_RUN) pytest tests/
 
 test-unit-helm: install-helm-unittest
-	helm unittest -f 'tests/*.yaml' helm/litellm-helm
+	helm unittest -f 'tests/*.yaml' litellm/deploy/helm/litellm
 
 # LLM Translation testing targets
 test-llm-translation: install-test-deps

README.md

@@ -6,7 +6,7 @@
         </p>
         <p align="center">Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.</p>
         <p align="center">
-        <a href="https://render.com/deploy?repo=https://github.com/BerriAI/litellm" target="_blank" rel="nofollow"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy to Render" height="40"></a>
+        <a href="https://render.com/deploy?repo=https://github.com/BerriAI/litellm&amp;blueprintPath=litellm/deploy/render/render.yaml" target="_blank" rel="nofollow"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy to Render" height="40"></a>
         <a href="https://railway.com/deploy/RhvhdC?referralCode=7mRv9K&utm_medium=integration&utm_source=template&utm_campaign=generic"><img src="https://railway.com/button.svg" alt="Deploy on Railway" height="40"></a>
         <a href="https://console.aws.amazon.com/cloudshell/home" target="_blank" rel="nofollow"><img src="./.github/deploy-on-aws.png" alt="Deploy on AWS" height="40"></a>
         <a href="https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FBerriAI%2Fterraform-google-litellm&cloudshell_git_branch=v1.105.0&cloudshell_workspace=examples%2Fdefault&cloudshell_tutorial=TUTORIAL.md&cloudshell_image=gcr.io/ds-artifacts-cloudshell/deploystack_custom_image&shellonly=true" target="_blank" rel="nofollow"><img src="./.github/deploy-on-gcp.png" alt="Deploy on GCP" height="40"></a>
@@ -99,7 +99,7 @@ proxy, CLI, and optional extras
 Build and install core from this checkout while its release integration is pending:
 
 ```shell
-python scripts/build_core_distribution.py --out-dir dist/core
+python litellm/scripts/build_core_distribution.py --out-dir dist/core
 python -m pip install dist/core/litellm_core-*.whl
 ```
 
@@ -526,7 +526,7 @@ Run the LiteLLM proxy as a production-ready componentized stack (gateway, backen
 
 ```bash
 git clone https://github.com/BerriAI/litellm.git
-cd litellm/terraform/litellm/aws/examples/default
+cd litellm/litellm/deploy/terraform/litellm/aws/examples/default
 cp terraform.tfvars.example terraform.tfvars   # edit region/tenant/env
 terraform init && terraform apply
 ```
@@ -579,7 +579,7 @@ Provider API keys live in AWS Secrets Manager; reference ARNs via `gateway_extra
 
 [![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.png)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FBerriAI%2Fterraform-google-litellm&cloudshell_git_branch=v1.105.0&cloudshell_workspace=examples%2Fdefault&cloudshell_tutorial=TUTORIAL.md&cloudshell_image=gcr.io/ds-artifacts-cloudshell/deploystack_custom_image&shellonly=true)
 
-Real 1-click. Opens Cloud Shell, clones the [terraform-google-litellm](https://github.com/BerriAI/terraform-google-litellm) module mirror at the latest stable release (a small download instead of the full litellm repo), and walks you through `terraform apply` via a built-in [DeployStack tutorial](./terraform/litellm/gcp/examples/default/TUTORIAL.md). Pick the project, the tutorial sets up the Artifact Registry remote repo, writes `terraform.tfvars` from your answers, and runs apply
+Real 1-click. Opens Cloud Shell, clones the [terraform-google-litellm](https://github.com/BerriAI/terraform-google-litellm) module mirror at the latest stable release (a small download instead of the full litellm repo), and walks you through `terraform apply` via a built-in [DeployStack tutorial](./litellm/deploy/terraform/litellm/gcp/examples/default/TUTORIAL.md). Pick the project, the tutorial sets up the Artifact Registry remote repo, writes `terraform.tfvars` from your answers, and runs apply
 
 [Module page →](https://registry.terraform.io/modules/BerriAI/litellm/google/latest)
 
@@ -646,21 +646,21 @@ Provider API keys live in Secret Manager; reference resource IDs (e.g. `projects
 - Versioned object store for proxy state + file uploads
 - An auto-generated `LITELLM_MASTER_KEY` in your cloud's secret manager
 - A one-off migration job that runs `prisma migrate deploy` before the proxy starts
-- The same `proxy_config` surface as the [Helm chart](./helm/litellm/) — pass YAML as a typed map
+- The same `proxy_config` surface as the [Helm chart](./litellm/deploy/helm/litellm/) — pass YAML as a typed map
 
-The Terraform modules live at [`terraform/litellm/aws/`](./terraform/litellm/aws/) and [`terraform/litellm/gcp/`](./terraform/litellm/gcp/) in this repo; the registry entries are read-only mirrors updated on each release.
+The Terraform modules live at [`litellm/deploy/terraform/litellm/aws/`](./litellm/deploy/terraform/litellm/aws/) and [`litellm/deploy/terraform/litellm/gcp/`](./litellm/deploy/terraform/litellm/gcp/) in this repo; the registry entries are read-only mirrors updated on each release.
 
 ### Run in Developer Mode
 #### Services
 1. Setup .env file in root
-2. Run dependent services `docker-compose up db prometheus`
+2. Run dependent services `docker compose -f litellm/deploy/compose/docker-compose.yml up db prometheus`
 
 #### Backend
 1. Run `make bootstrap`
 2. Start proxy backend: `uv run python litellm/proxy/proxy_cli.py`
 
 #### Frontend
-1. Navigate to `ui/litellm-dashboard` (dependencies were already installed w/ `make bootstrap`)
+1. Navigate to `litellm/ui/litellm-dashboard` (dependencies were already installed w/ `make bootstrap`)
 2. Start dashboard: `npm run dev`
 
 ### Verify Docker Image Signatures
@@ -725,7 +725,7 @@ make test-unit      # Run unit tests
 make format-check   # Check formatting only
 ```
 
-For detailed contributing guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).
+For detailed contributing guidelines, see [CONTRIBUTING.md](.github/CONTRIBUTING.md).
 
 > **📖 Contributing to documentation?** The LiteLLM docs have moved to a separate repository: [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs). Please open doc PRs there. Docs are served at [docs.litellm.ai](https://docs.litellm.ai).
 

backend/Dockerfile

@@ -1,146 +0,0 @@
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:360896efb0753a5701c48f860e17130c205eae5a264797aa3e7ea126ee2923a7
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:360896efb0753a5701c48f860e17130c205eae5a264797aa3e7ea126ee2923a7
-ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
-
-FROM $UV_IMAGE AS uvbin
-
-# ---------- Rust extension ----------
-# Copies only the native build's inputs, so the fat-LTO compile stays cached across Python-only changes
-FROM $LITELLM_BUILD_IMAGE AS rust-native
-
-WORKDIR /app
-USER root
-
-COPY --from=uvbin /uv /uvx /usr/local/bin/
-
-RUN for i in 1 2 3; do \
-      apk add --no-cache bash gcc python-3.13 python-3.13-dev openssl openssl-dev && break; \
-      [ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
-      sleep 5; \
-    done
-
-ENV UV_PYTHON_DOWNLOADS=0
-
-COPY pyproject.toml README.md rust-toolchain.toml ./
-COPY .cargo ./.cargo
-COPY litellm-rust ./litellm-rust
-COPY model_prices_and_context_window.json ./
-
-RUN --mount=type=cache,target=/root/.cache/uv \
-    mkdir -p litellm/rust_bridge /wheels && \
-    touch litellm/__init__.py litellm/rust_bridge/__init__.py && \
-    uvx --python python3.13 --from maturin==1.15.0 --with puccinialin==0.1.14 \
-        python -c "import maturin; maturin.build_wheel('/wheels')"
-
-# ---------- Builder ----------
-FROM $LITELLM_BUILD_IMAGE AS builder
-
-WORKDIR /app
-USER root
-
-COPY --from=uvbin /uv /uvx /usr/local/bin/
-
-# nodejs/npm so `prisma generate` uses Wolfi's Node via PRISMA_USE_GLOBAL_NODE
-# instead of nodeenv downloading one whose dynamic deps may not be in Wolfi
-# (e.g. Node 26.2.0 needs libatomic). Retry for transient apk.cgr.dev flakes.
-RUN for i in 1 2 3; do \
-      apk add --no-cache bash gcc python-3.13 python-3.13-dev openssl libssl3 openssl-dev libsndfile nodejs npm && break; \
-      [ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
-      sleep 5; \
-    done
-
-# UV_COMPILE_BYTECODE=1   precompiles .pyc at install time → faster cold start.
-# UV_LINK_MODE=copy       avoids hardlink warnings when uv installs from a
-#                         BuildKit cache mount (different filesystem).
-# UV_PYTHON_DOWNLOADS=0   force uv to use the apk-installed CPython instead of
-#                         silently pulling a managed interpreter.
-# PRISMA_USE_GLOBAL_NODE  explicit (matches default) so an env override can't
-#                         silently re-enable nodeenv's Node download.
-ENV UV_PROJECT_ENVIRONMENT=/app/.venv \
-    UV_LINK_MODE=copy \
-    UV_COMPILE_BYTECODE=1 \
-    UV_PYTHON_DOWNLOADS=0 \
-    PRISMA_USE_GLOBAL_NODE=true \
-    PATH="/app/.venv/bin:${PATH}"
-
-# Stage 1 — install dependencies only.
-RUN --mount=type=cache,target=/root/.cache/uv \
-    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
-    --mount=type=bind,source=uv.lock,target=uv.lock \
-    --mount=type=bind,source=enterprise/pyproject.toml,target=enterprise/pyproject.toml \
-    --mount=type=bind,source=litellm-proxy-extras/pyproject.toml,target=litellm-proxy-extras/pyproject.toml \
-    uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
-        --extra proxy \
-        --extra proxy-runtime \
-        --group admin-mcp \
-        --extra extra_proxy \
-        --extra semantic-router \
-        --extra saml \
-        --python python3.13
-
-# Stage 2 — copy source and install the project + workspace members.
-COPY . .
-
-RUN --mount=type=cache,target=/root/.cache/uv \
-    uv sync --frozen --no-install-project --no-default-groups --no-editable \
-        --extra proxy \
-        --extra proxy-runtime \
-        --group admin-mcp \
-        --extra extra_proxy \
-        --extra semantic-router \
-        --extra saml \
-        --python python3.13
-
-RUN --mount=type=bind,from=rust-native,source=/wheels,target=/wheels \
-    uv pip install --python /app/.venv/bin/python --no-deps /wheels/litellm-*.whl
-
-RUN cp "$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"/litellm/rust_bridge/_native*.so litellm/rust_bridge/
-
-RUN HOME=/opt/prisma XDG_CACHE_HOME=/opt/prisma/.cache PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries \
-    npm_config_cache=/root/.npm \
-    prisma generate --schema=./schema.prisma
-
-RUN sed -i 's/\r$//' docker/component_entrypoint.sh && chmod +x docker/component_entrypoint.sh
-
-# ---------- Runtime ----------
-FROM $LITELLM_RUNTIME_IMAGE AS runtime
-ARG LITELLM_RELEASE_TAG=""
-ENV LITELLM_RELEASE_TAG=${LITELLM_RELEASE_TAG}
-
-USER root
-
-RUN for i in 1 2 3; do \
-      apk add --no-cache bash openssl libssl3 tzdata python-3.13 libsndfile libatomic && break; \
-      [ $i = 3 ] && { echo "apk add failed after 3 retries" >&2; exit 1; }; \
-      sleep 5; \
-    done
-
-# wolfi-base ships an unprivileged `nonroot` account (UID/GID 65532) with
-# /home/nonroot. We run the backend as that user
-WORKDIR /app
-ENV HOME=/home/nonroot \
-    PATH="/app/.venv/bin:${PATH}" \
-    PYTHONPATH="/app" \
-    PYTHONDONTWRITEBYTECODE=1 \
-    PYTHONUNBUFFERED=1 \
-    PRISMA_BINARY_CACHE_DIR=/opt/prisma/binaries
-
-COPY --from=builder --chown=nonroot:nonroot /app/.venv /app/.venv
-COPY --from=builder --chown=nonroot:nonroot /app/litellm /app/litellm
-COPY --from=builder --chown=nonroot:nonroot /app/enterprise /app/enterprise
-COPY --from=builder --chown=nonroot:nonroot /app/backend /app/backend
-COPY --from=builder --chown=nonroot:nonroot /app/docker/component_entrypoint.sh /app/docker/component_entrypoint.sh
-COPY --from=builder --chown=nonroot:nonroot /app/schema.prisma /app/schema.prisma
-COPY --from=builder /opt/prisma /opt/prisma
-
-RUN find /app/.venv -type f -path "*/tornado/test/*" -delete && \
-    find /app/.venv -type d -path "*/tornado/test" -delete && \
-    chmod -R a+rX /opt/prisma && \
-    python -c "from prisma.client import BINARY_PATHS; paths = list(BINARY_PATHS.query_engine.values()); assert paths and all(p.startswith('/opt/prisma/') for p in paths), paths"
-
-USER nonroot
-
-EXPOSE 4001/tcp
-
-ENTRYPOINT ["/app/docker/component_entrypoint.sh", "uvicorn", "backend.main:app"]
-CMD ["--host", "0.0.0.0", "--port", "4001"]

ci_cd/baseline_db.py

@@ -1,60 +0,0 @@
-import subprocess
-from pathlib import Path
-from datetime import datetime
-
-
-def create_baseline():
-    """Create baseline migration in deploy/migrations"""
-    try:
-        # Get paths
-        root_dir = Path(__file__).parent.parent
-        deploy_dir = root_dir / "deploy"
-        migrations_dir = deploy_dir / "migrations"
-        schema_path = root_dir / "schema.prisma"
-
-        # Create migrations directory
-        migrations_dir.mkdir(parents=True, exist_ok=True)
-
-        # Create migration_lock.toml if it doesn't exist
-        lock_file = migrations_dir / "migration_lock.toml"
-        if not lock_file.exists():
-            lock_file.write_text('provider = "postgresql"\n')
-
-        # Create timestamp-based migration directory
-        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
-        migration_dir = migrations_dir / f"{timestamp}_baseline"
-        migration_dir.mkdir(parents=True, exist_ok=True)
-
-        # Generate migration SQL
-        result = subprocess.run(
-            [
-                "prisma",
-                "migrate",
-                "diff",
-                "--from-empty",
-                "--to-schema-datamodel",
-                str(schema_path),
-                "--script",
-            ],
-            capture_output=True,
-            text=True,
-            check=True,
-        )
-
-        # Write the SQL to migration.sql
-        migration_file = migration_dir / "migration.sql"
-        migration_file.write_text(result.stdout)
-
-        print(f"Created baseline migration in {migration_dir}")
-        return True
-
-    except subprocess.CalledProcessError as e:
-        print(f"Error running prisma command: {e.stderr}")
-        return False
-    except Exception as e:
-        print(f"Error creating baseline migration: {str(e)}")
-        return False
-
-
-if __name__ == "__main__":
-    create_baseline()

codecov.yaml

@@ -1,154 +0,0 @@
-codecov:
-  require_ci_to_pass: false # post coverage status even if CI has unrelated failures
-  notify:
-    wait_for_ci: false # post as soon as expected uploads arrive, don't wait on CI
-
-ignore:
-  - "litellm-rust/**"
-
-# Uploads are flagged per workflow tier (GHA) or "circleci". carryforward makes
-# a re-upload of a flag replace its prior session instead of accumulating a
-# conflicting one, and lets a commit reuse a flag from its parent when that flag
-# was not re-uploaded. Required because the same commit can receive the
-# push-triggered workflows more than once (re-runs / branches cut at the same
-# SHA); flagless overlapping sessions made Codecov drop the largest files.
-flag_management:
-  default_rules:
-    carryforward: true
-  # Dead flags no CI job uploads anymore, and flags whose job is path-gated on
-  # PRs: their carried-forward sessions were measured against old revisions, and
-  # the stale line maps mark comment lines of since-edited files as missed,
-  # sinking patch coverage on unrelated PRs.
-  individual_flags:
-    - name: proxy-mgmt-behavior
-      carryforward: false
-    - name: security
-      carryforward: false
-    - name: proxy-db-schema-migration
-      carryforward: false
-    - name: circleci
-      carryforward: false
-    - name: core-utils
-      carryforward: false
-    - name: enterprise-routing
-      carryforward: false
-    - name: integrations
-      carryforward: false
-    - name: llm-vertex-ai
-      carryforward: false
-    - name: llm-other-providers
-      carryforward: false
-    - name: llm-openai-meta
-      carryforward: false
-    - name: misc
-      carryforward: false
-    - name: misc-dirs
-      carryforward: false
-    - name: proxy-auth
-      carryforward: false
-    - name: proxy-hooks-client
-      carryforward: false
-    - name: proxy-endpoints
-      carryforward: false
-    - name: proxy-feature-endpoints
-      carryforward: false
-    - name: proxy-server
-      carryforward: false
-    - name: mcp-elicitation
-      carryforward: false
-    - name: proxy-infra
-      carryforward: false
-    - name: proxy-infra-root
-      carryforward: false
-    - name: caching-local
-      carryforward: false
-    - name: proxy-extras
-      carryforward: false
-    - name: enterprise-package
-      carryforward: false
-    - name: enterprise-managed-files
-      carryforward: false
-    - name: responses-caching-types
-      carryforward: false
-    - name: proxy-db-key-generation
-      carryforward: false
-    - name: proxy-db-auth-checks
-      carryforward: false
-    - name: proxy-db-jwt-and-keys
-      carryforward: false
-    - name: proxy-db-proxy-utils
-      carryforward: false
-    - name: proxy-db-proxy-server-core
-      carryforward: false
-    - name: proxy-db-proxy-runtime
-      carryforward: false
-    - name: proxy-db-mcp-oauth
-      carryforward: false
-    - name: proxy-db-custom-logging
-      carryforward: false
-    - name: proxy-db-logging-misc
-      carryforward: false
-    - name: proxy-db-db-and-spend
-      carryforward: false
-    - name: proxy-db-guardrails-hooks
-      carryforward: false
-    - name: proxy-db-budgets
-      carryforward: false
-    - name: proxy-db-endpoints-and-responses
-      carryforward: false
-    - name: roi-postgres
-      carryforward: false
-    - name: redis-compat
-      carryforward: false
-
-component_management:
-  individual_components:
-    - component_id: "Router"
-      paths:
-        - "router"
-    - component_id: "LLMs"
-      paths:
-        - "*/llms/*"
-    - component_id: "Caching"
-      paths:
-        - "*/caching/*"
-        - ".*redis.*"
-    - component_id: "litellm_logging"
-      paths:
-        - "*/integrations/*"
-        - ".*litellm_logging.*"
-    - component_id: "Proxy_Authentication"
-      paths:
-        - "*/proxy/auth/**"
-    - component_id: "Enterprise"
-      paths:
-        - "enterprise/**"
-    - component_id: "Batches"
-      paths:
-        - "*/proxy/batches_endpoints/**"
-        - "litellm/batches/**"
-        - "*/llms/*/batches/**"
-    - component_id: "Videos"
-      paths:
-        - "litellm/videos/**"
-        - "*/proxy/video_endpoints/**"
-        - "*/llms/*/videos/**"
-    - component_id: "Realtime"
-      paths:
-        - "litellm/realtime_api/**"
-        - "*/proxy/realtime_endpoints/**"
-        - "*/llms/*/realtime/**"
-        - "litellm/litellm_core_utils/realtime_streaming.py"
-comment:
-  layout: "header, diff, flags, components"  # show component info in the PR comment
-
-coverage:
-  status:
-    project:
-      default:
-        target: auto
-        threshold: 0% # do not allow project coverage to drop
-    patch:
-      default:
-        target: auto
-        threshold: 0% # patch coverage should be 100%

cookbook/Claude_(Anthropic)_with_Streaming_liteLLM_Examples.ipynb

@@ -1,406 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "code",
-      "execution_count": 1,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "ZwuaylskLxFu",
-        "outputId": "d684d6a3-32fe-4beb-c378-c39134bcf8cc"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Collecting litellm==0.1.363\n",
-            "  Downloading litellm-0.1.363-py3-none-any.whl (34 kB)\n",
-            "Requirement already satisfied: openai<0.28.0,>=0.27.8 in /usr/local/lib/python3.10/dist-packages (from litellm==0.1.363) (0.27.8)\n",
-            "Requirement already satisfied: python-dotenv<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from litellm==0.1.363) (1.0.0)\n",
-            "Requirement already satisfied: tiktoken<0.5.0,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from litellm==0.1.363) (0.4.0)\n",
-            "Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.10/dist-packages (from openai<0.28.0,>=0.27.8->litellm==0.1.363) (2.31.0)\n",
-            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai<0.28.0,>=0.27.8->litellm==0.1.363) (4.65.0)\n",
-            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai<0.28.0,>=0.27.8->litellm==0.1.363) (3.8.5)\n",
-            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken<0.5.0,>=0.4.0->litellm==0.1.363) (2022.10.31)\n",
-            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<0.28.0,>=0.27.8->litellm==0.1.363) (3.2.0)\n",
-            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<0.28.0,>=0.27.8->litellm==0.1.363) (3.4)\n",
-            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<0.28.0,>=0.27.8->litellm==0.1.363) (1.26.16)\n",
-            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<0.28.0,>=0.27.8->litellm==0.1.363) (2023.7.22)\n",
-            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<0.28.0,>=0.27.8->litellm==0.1.363) (23.1.0)\n",
-            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<0.28.0,>=0.27.8->litellm==0.1.363) (6.0.4)\n",
-            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<0.28.0,>=0.27.8->litellm==0.1.363) (4.0.2)\n",
-            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<0.28.0,>=0.27.8->litellm==0.1.363) (1.9.2)\n",
-            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<0.28.0,>=0.27.8->litellm==0.1.363) (1.4.0)\n",
-            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<0.28.0,>=0.27.8->litellm==0.1.363) (1.3.1)\n",
-            "Installing collected packages: litellm\n",
-            "  Attempting uninstall: litellm\n",
-            "    Found existing installation: litellm 0.1.362\n",
-            "    Uninstalling litellm-0.1.362:\n",
-            "      Successfully uninstalled litellm-0.1.362\n",
-            "Successfully installed litellm-0.1.363\n"
-          ]
-        }
-      ],
-      "source": [
-        "!pip install litellm==\"0.1.363\""
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 2,
-      "metadata": {
-        "id": "W216G__XL19Q"
-      },
-      "outputs": [],
-      "source": [
-        "# @title Import litellm & Set env variables\n",
-        "import litellm\n",
-        "import os\n",
-        "\n",
-        "os.environ[\"ANTHROPIC_API_KEY\"] = \" \" #@param"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 9,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "ff1lKwUMMLJj",
-        "outputId": "bfddf6f8-36d4-45e5-92dc-349083fa41b8"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "\n",
-            " Result from claude-instant-1 {'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': \" The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays 4-2. It was the Dodgers' first World Series title since 1988.\"}}], 'created': 1691536677.2676156, 'model': 'claude-instant-1', 'usage': {'prompt_tokens': 30, 'completion_tokens': 32, 'total_tokens': 62}}\n",
-            "\n",
-            "\n",
-            " Result from claude-2 {'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': ' The Los Angeles Dodgers won'}}], 'created': 1691536677.944753, 'model': 'claude-2', 'usage': {'prompt_tokens': 30, 'completion_tokens': 5, 'total_tokens': 35}}\n"
-          ]
-        }
-      ],
-      "source": [
-        "# @title Request Claude Instant-1 and Claude-2\n",
-        "messages = [\n",
-        "  {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
-        "  {\"role\": \"user\", \"content\": \"Who won the world series in 2020?\"}\n",
-        "  ]\n",
-        "\n",
-        "result = litellm.completion('claude-instant-1', messages)\n",
-        "print(\"\\n\\n Result from claude-instant-1\", result)\n",
-        "result = litellm.completion('claude-2', messages, max_tokens=5, temperature=0.2)\n",
-        "print(\"\\n\\n Result from claude-2\", result)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 7,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "06hWKnNQMrV-",
-        "outputId": "7fdec0eb-d4a9-4882-f9c4-987ff9a31114"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            " Here\n",
-            "'s\n",
-            " a\n",
-            " quick\n",
-            " overview\n",
-            " of\n",
-            " how\n",
-            " a\n",
-            " court\n",
-            " case\n",
-            " can\n",
-            " reach\n",
-            " the\n",
-            " U\n",
-            ".\n",
-            "S\n",
-            ".\n",
-            " Supreme\n",
-            " Court\n",
-            ":\n",
-            "\n",
-            "\n",
-            "-\n",
-            " The\n",
-            " case\n",
-            " must\n",
-            " first\n",
-            " be\n",
-            " heard\n",
-            " in\n",
-            " a\n",
-            " lower\n",
-            " trial\n",
-            " court\n",
-            " (\n",
-            "either\n",
-            " a\n",
-            " state\n",
-            " court\n",
-            " or\n",
-            " federal\n",
-            " district\n",
-            " court\n",
-            ").\n",
-            " The\n",
-            " trial\n",
-            " court\n",
-            " makes\n",
-            " initial\n",
-            " r\n",
-            "ulings\n",
-            " and\n",
-            " produces\n",
-            " a\n",
-            " record\n",
-            " of\n",
-            " the\n",
-            " case\n",
-            ".\n",
-            "\n",
-            "\n",
-            "-\n",
-            " The\n",
-            " losing\n",
-            " party\n",
-            " can\n",
-            " appeal\n",
-            " the\n",
-            " decision\n",
-            " to\n",
-            " an\n",
-            " appeals\n",
-            " court\n",
-            " (\n",
-            "a\n",
-            " state\n",
-            " appeals\n",
-            " court\n",
-            " for\n",
-            " state\n",
-            " cases\n",
-            ",\n",
-            " or\n",
-            " a\n",
-            " federal\n",
-            " circuit\n",
-            " court\n",
-            " for\n",
-            " federal\n",
-            " cases\n",
-            ").\n",
-            " The\n",
-            " appeals\n",
-            " court\n",
-            " reviews\n",
-            " the\n",
-            " trial\n",
-            " court\n",
-            "'s\n",
-            " r\n",
-            "ulings\n",
-            " and\n",
-            " can\n",
-            " affirm\n",
-            ",\n",
-            " reverse\n",
-            ",\n",
-            " or\n",
-            " modify\n",
-            " the\n",
-            " decision\n",
-            ".\n",
-            "\n",
-            "\n",
-            "-\n",
-            " If\n",
-            " a\n",
-            " party\n",
-            " is\n",
-            " still\n",
-            " unsat\n",
-            "isf\n",
-            "ied\n",
-            " after\n",
-            " the\n",
-            " appeals\n",
-            " court\n",
-            " rules\n",
-            ",\n",
-            " they\n",
-            " can\n",
-            " petition\n",
-            " the\n",
-            " Supreme\n",
-            " Court\n",
-            " to\n",
-            " hear\n",
-            " the\n",
-            " case\n",
-            " through\n",
-            " a\n",
-            " writ\n",
-            " of\n",
-            " cert\n",
-            "ior\n",
-            "ari\n",
-            ".\n",
-            " \n",
-            "\n",
-            "\n",
-            "-\n",
-            " The\n",
-            " Supreme\n",
-            " Court\n",
-            " gets\n",
-            " thousands\n",
-            " of\n",
-            " cert\n",
-            " petitions\n",
-            " every\n",
-            " year\n",
-            " but\n",
-            " usually\n",
-            " only\n",
-            " agrees\n",
-            " to\n",
-            " hear\n",
-            " about\n",
-            " 100\n",
-            "-\n",
-            "150\n",
-            " of\n",
-            " cases\n",
-            " that\n",
-            " have\n",
-            " significant\n",
-            " national\n",
-            " importance\n",
-            " or\n",
-            " where\n",
-            " lower\n",
-            " courts\n",
-            " disagree\n",
-            " on\n",
-            " federal\n",
-            " law\n",
-            ".\n",
-            " \n",
-            "\n",
-            "\n",
-            "-\n",
-            " If\n",
-            " 4\n",
-            " out\n",
-            " of\n",
-            " the\n",
-            " 9\n",
-            " Just\n",
-            "ices\n",
-            " vote\n",
-            " to\n",
-            " grant\n",
-            " cert\n",
-            " (\n",
-            "agree\n",
-            " to\n",
-            " hear\n",
-            " the\n",
-            " case\n",
-            "),\n",
-            " it\n",
-            " goes\n",
-            " on\n",
-            " the\n",
-            " Supreme\n",
-            " Court\n",
-            "'s\n",
-            " do\n",
-            "cket\n",
-            " for\n",
-            " arguments\n",
-            ".\n",
-            "\n",
-            "\n",
-            "-\n",
-            " The\n",
-            " Supreme\n",
-            " Court\n",
-            " then\n",
-            " hears\n",
-            " oral\n",
-            " arguments\n",
-            ",\n",
-            " considers\n",
-            " written\n",
-            " brief\n",
-            "s\n",
-            ",\n",
-            " examines\n",
-            " the\n",
-            " lower\n",
-            " court\n",
-            " records\n",
-            ",\n",
-            " and\n",
-            " issues\n",
-            " a\n",
-            " final\n",
-            " ruling\n",
-            " on\n",
-            " the\n",
-            " case\n",
-            ",\n",
-            " which\n",
-            " serves\n",
-            " as\n",
-            " binding\n",
-            " precedent\n"
-          ]
-        }
-      ],
-      "source": [
-        "# @title Streaming Example: Request Claude-2\n",
-        "messages = [\n",
-        "  {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
-        "  {\"role\": \"user\", \"content\": \"how does a court case get to the Supreme Court?\"}\n",
-        "  ]\n",
-        "\n",
-        "result = litellm.completion('claude-2', messages, stream=True)\n",
-        "for part in result:\n",
-        "    print(part.choices[0].delta.content or \"\")\n",
-        "\n"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_Azure_and_OpenAI_example.ipynb

@@ -1,422 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "BmX0b5Ueh91v"
-   },
-   "source": [
-    "# LiteLLM - Azure OpenAI + OpenAI Calls\n",
-    "This notebook covers the following for Azure OpenAI + OpenAI:\n",
-    "* Completion - Quick start\n",
-    "* Completion - Streaming\n",
-    "* Completion - Azure, OpenAI in separate threads\n",
-    "* Completion - Stress Test 10 requests in parallel\n",
-    "* Completion - Azure, OpenAI in the same thread"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "iHq4d0dpfawS"
-   },
-   "outputs": [],
-   "source": [
-    "!pip install litellm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "id": "mnveHO5dfcB0"
-   },
-   "outputs": [],
-   "source": [
-    "import os"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "eo88QUdbiDIE"
-   },
-   "source": [
-    "## Completion - Quick start"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {
-    "colab": {
-     "base_uri": "https://localhost:8080/"
-    },
-    "id": "5OSosWNCfc_2",
-    "outputId": "c52344b1-2458-4695-a7eb-a9b076893348"
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Openai Response\n",
-      "\n",
-      "{\n",
-      "  \"id\": \"chatcmpl-7yjVOEKCPw2KdkfIaM3Ao1tIXp8EM\",\n",
-      "  \"object\": \"chat.completion\",\n",
-      "  \"created\": 1694708958,\n",
-      "  \"model\": \"gpt-3.5-turbo-0613\",\n",
-      "  \"choices\": [\n",
-      "    {\n",
-      "      \"index\": 0,\n",
-      "      \"message\": {\n",
-      "        \"role\": \"assistant\",\n",
-      "        \"content\": \"I'm an AI, so I don't have feelings, but I'm here to help you. How can I assist you?\"\n",
-      "      },\n",
-      "      \"finish_reason\": \"stop\"\n",
-      "    }\n",
-      "  ],\n",
-      "  \"usage\": {\n",
-      "    \"prompt_tokens\": 13,\n",
-      "    \"completion_tokens\": 26,\n",
-      "    \"total_tokens\": 39\n",
-      "  }\n",
-      "}\n",
-      "Azure Response\n",
-      "\n",
-      "{\n",
-      "  \"id\": \"chatcmpl-7yjVQ6m2R2HRtnKHRRFp6JzL4Fjez\",\n",
-      "  \"object\": \"chat.completion\",\n",
-      "  \"created\": 1694708960,\n",
-      "  \"model\": \"gpt-35-turbo\",\n",
-      "  \"choices\": [\n",
-      "    {\n",
-      "      \"index\": 0,\n",
-      "      \"finish_reason\": \"stop\",\n",
-      "      \"message\": {\n",
-      "        \"role\": \"assistant\",\n",
-      "        \"content\": \"Hello there! As an AI language model, I don't have feelings but I'm functioning well. How can I assist you today?\"\n",
-      "      }\n",
-      "    }\n",
-      "  ],\n",
-      "  \"usage\": {\n",
-      "    \"completion_tokens\": 27,\n",
-      "    \"prompt_tokens\": 14,\n",
-      "    \"total_tokens\": 41\n",
-      "  }\n",
-      "}\n"
-     ]
-    }
-   ],
-   "source": [
-    "from litellm import completion\n",
-    "\n",
-    "# openai configs\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
-    "\n",
-    "# azure openai configs\n",
-    "os.environ[\"AZURE_API_KEY\"] = \"\"\n",
-    "os.environ[\"AZURE_API_BASE\"] = \"https://openai-gpt-4-test-v-1.openai.azure.com/\"\n",
-    "os.environ[\"AZURE_API_VERSION\"] = \"2023-05-15\"\n",
-    "\n",
-    "\n",
-    "# openai call\n",
-    "response = completion(\n",
-    "    model = \"gpt-3.5-turbo\",\n",
-    "    messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-    ")\n",
-    "print(\"Openai Response\\n\")\n",
-    "print(response)\n",
-    "\n",
-    "\n",
-    "\n",
-    "# azure call\n",
-    "response = completion(\n",
-    "    model = \"azure/your-azure-deployment\",\n",
-    "    messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-    ")\n",
-    "print(\"Azure Response\\n\")\n",
-    "print(response)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "dQMkM-diiKdE"
-   },
-   "source": [
-    "## Completion - Streaming"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "uVvJDVn4g1i1"
-   },
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from litellm import completion\n",
-    "\n",
-    "# openai configs\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
-    "\n",
-    "# azure openai configs\n",
-    "os.environ[\"AZURE_API_KEY\"] = \"\"\n",
-    "os.environ[\"AZURE_API_BASE\"] = \"https://openai-gpt-4-test-v-1.openai.azure.com/\"\n",
-    "os.environ[\"AZURE_API_VERSION\"] = \"2023-05-15\"\n",
-    "\n",
-    "\n",
-    "# openai call\n",
-    "response = completion(\n",
-    "    model = \"gpt-3.5-turbo\",\n",
-    "    messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
-    "    stream=True\n",
-    ")\n",
-    "print(\"OpenAI Streaming response\")\n",
-    "for chunk in response:\n",
-    "  print(chunk)\n",
-    "\n",
-    "# azure call\n",
-    "response = completion(\n",
-    "    model = \"azure/your-azure-deployment\",\n",
-    "    messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
-    "    stream=True\n",
-    ")\n",
-    "print(\"Azure Streaming response\")\n",
-    "for chunk in response:\n",
-    "  print(chunk)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "4xrOPnt-oqwm"
-   },
-   "source": [
-    "## Completion - Azure, OpenAI in separate threads"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "V5b5taJPjvC3"
-   },
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "import threading\n",
-    "from litellm import completion\n",
-    "\n",
-    "# Function to make a completion call\n",
-    "def make_completion(model, messages):\n",
-    "    response = completion(\n",
-    "        model=model,\n",
-    "        messages=messages\n",
-    "    )\n",
-    "\n",
-    "    print(f\"Response for {model}: {response}\")\n",
-    "\n",
-    "# openai configs\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
-    "\n",
-    "# azure openai configs\n",
-    "os.environ[\"AZURE_API_KEY\"] = \"\"\n",
-    "os.environ[\"AZURE_API_BASE\"] = \"https://openai-gpt-4-test-v-1.openai.azure.com/\"\n",
-    "os.environ[\"AZURE_API_VERSION\"] = \"2023-05-15\"\n",
-    "\n",
-    "# Define the messages for the completions\n",
-    "messages = [{\"content\": \"Hello, how are you?\", \"role\": \"user\"}]\n",
-    "\n",
-    "# Create threads for making the completions\n",
-    "thread1 = threading.Thread(target=make_completion, args=(\"gpt-3.5-turbo\", messages))\n",
-    "thread2 = threading.Thread(target=make_completion, args=(\"azure/your-azure-deployment\", messages))\n",
-    "\n",
-    "# Start both threads\n",
-    "thread1.start()\n",
-    "thread2.start()\n",
-    "\n",
-    "# Wait for both threads to finish\n",
-    "thread1.join()\n",
-    "thread2.join()\n",
-    "\n",
-    "print(\"Both completions are done.\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "lx8DbMBqoAoN"
-   },
-   "source": [
-    "## Completion - Stress Test 10 requests in parallel\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "pHYANOlOkoDh"
-   },
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "import threading\n",
-    "from litellm import completion\n",
-    "\n",
-    "# Function to make a completion call\n",
-    "def make_completion(model, messages):\n",
-    "    response = completion(\n",
-    "        model=model,\n",
-    "        messages=messages\n",
-    "    )\n",
-    "\n",
-    "    print(f\"Response for {model}: {response}\")\n",
-    "\n",
-    "# Set your API keys\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
-    "os.environ[\"AZURE_API_KEY\"] = \"\"\n",
-    "os.environ[\"AZURE_API_BASE\"] = \"https://openai-gpt-4-test-v-1.openai.azure.com/\"\n",
-    "os.environ[\"AZURE_API_VERSION\"] = \"2023-05-15\"\n",
-    "\n",
-    "# Define the messages for the completions\n",
-    "messages = [{\"content\": \"Hello, how are you?\", \"role\": \"user\"}]\n",
-    "\n",
-    "# Create and start 10 threads for making completions\n",
-    "threads = []\n",
-    "for i in range(10):\n",
-    "    thread = threading.Thread(target=make_completion, args=(\"gpt-3.5-turbo\" if i % 2 == 0 else \"azure/your-azure-deployment\", messages))\n",
-    "    threads.append(thread)\n",
-    "    thread.start()\n",
-    "\n",
-    "# Wait for all threads to finish\n",
-    "for thread in threads:\n",
-    "    thread.join()\n",
-    "\n",
-    "print(\"All completions are done.\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "yB2NDOO4oxrp"
-   },
-   "source": [
-    "## Completion - Azure, OpenAI in the same thread"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "metadata": {
-    "colab": {
-     "base_uri": "https://localhost:8080/"
-    },
-    "id": "HTBqwzxpnxab",
-    "outputId": "f3bc0efe-e4d5-44d5-a193-97d178cfbe14"
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "OpenAI Response: {\n",
-      "  \"id\": \"chatcmpl-7yjzrDeOeVeSrQ00tApmTxEww3vBS\",\n",
-      "  \"object\": \"chat.completion\",\n",
-      "  \"created\": 1694710847,\n",
-      "  \"model\": \"gpt-3.5-turbo-0613\",\n",
-      "  \"choices\": [\n",
-      "    {\n",
-      "      \"index\": 0,\n",
-      "      \"message\": {\n",
-      "        \"role\": \"assistant\",\n",
-      "        \"content\": \"Hello! I'm an AI, so I don't have feelings, but I'm here to help you. How can I assist you today?\"\n",
-      "      },\n",
-      "      \"finish_reason\": \"stop\"\n",
-      "    }\n",
-      "  ],\n",
-      "  \"usage\": {\n",
-      "    \"prompt_tokens\": 13,\n",
-      "    \"completion_tokens\": 29,\n",
-      "    \"total_tokens\": 42\n",
-      "  }\n",
-      "}\n",
-      "Azure OpenAI Response: {\n",
-      "  \"id\": \"chatcmpl-7yjztAQ0gK6IMQt7cvLroMSOoXkeu\",\n",
-      "  \"object\": \"chat.completion\",\n",
-      "  \"created\": 1694710849,\n",
-      "  \"model\": \"gpt-35-turbo\",\n",
-      "  \"choices\": [\n",
-      "    {\n",
-      "      \"index\": 0,\n",
-      "      \"finish_reason\": \"stop\",\n",
-      "      \"message\": {\n",
-      "        \"role\": \"assistant\",\n",
-      "        \"content\": \"As an AI language model, I don't have feelings but I'm functioning properly. Thank you for asking! How can I assist you today?\"\n",
-      "      }\n",
-      "    }\n",
-      "  ],\n",
-      "  \"usage\": {\n",
-      "    \"completion_tokens\": 29,\n",
-      "    \"prompt_tokens\": 14,\n",
-      "    \"total_tokens\": 43\n",
-      "  }\n",
-      "}\n"
-     ]
-    }
-   ],
-   "source": [
-    "import os\n",
-    "from litellm import completion\n",
-    "\n",
-    "# Function to make both OpenAI and Azure completions\n",
-    "def make_completions():\n",
-    "    # Set your OpenAI API key\n",
-    "    os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
-    "\n",
-    "    # OpenAI completion\n",
-    "    openai_response = completion(\n",
-    "        model=\"gpt-3.5-turbo\",\n",
-    "        messages=[{\"content\": \"Hello, how are you?\", \"role\": \"user\"}]\n",
-    "    )\n",
-    "\n",
-    "    print(\"OpenAI Response:\", openai_response)\n",
-    "\n",
-    "    # Set your Azure OpenAI API key and configuration\n",
-    "    os.environ[\"AZURE_API_KEY\"] = \"\"\n",
-    "    os.environ[\"AZURE_API_BASE\"] = \"https://openai-gpt-4-test-v-1.openai.azure.com/\"\n",
-    "    os.environ[\"AZURE_API_VERSION\"] = \"2023-05-15\"\n",
-    "\n",
-    "    # Azure OpenAI completion\n",
-    "    azure_response = completion(\n",
-    "        model=\"azure/your-azure-deployment\",\n",
-    "        messages=[{\"content\": \"Hello, how are you?\", \"role\": \"user\"}]\n",
-    "    )\n",
-    "\n",
-    "    print(\"Azure OpenAI Response:\", azure_response)\n",
-    "\n",
-    "# Call the function to make both completions in one thread\n",
-    "make_completions()\n"
-   ]
-  }
- ],
- "metadata": {
-  "colab": {
-   "provenance": []
-  },
-  "kernelspec": {
-   "display_name": "Python 3",
-   "name": "python3"
-  },
-  "language_info": {
-   "name": "python"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

cookbook/LiteLLM_Bedrock.ipynb

@@ -1,310 +0,0 @@
-{
-  "cells": [
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "fNkMBurtxawJ"
-      },
-      "source": [
-        "# LiteLLM Bedrock Usage\n",
-        "Important Note: For Bedrock Requests you need to ensure you have `pip install boto3>=1.28.57`, boto3 supports bedrock from `boto3>=1.28.57` and higher "
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "htAufI28xeSy"
-      },
-      "source": [
-        "## Pre-Requisites"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "jT5GbPjAuDTp"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm\n",
-        "!pip install boto3>=1.28.57 # this version onwards has bedrock support"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "H4Vu4er2xnfI"
-      },
-      "source": [
-        "## Set Bedrock/AWS Credentials"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 9,
-      "metadata": {
-        "id": "CtTrBthWxp-t"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "os.environ[\"AWS_ACCESS_KEY_ID\"] = \"\" # Access key\n",
-        "os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"\" # Secret access key\n",
-        "os.environ[\"AWS_REGION_NAME\"] = \"\""
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "ycRK9NUdx1EI"
-      },
-      "source": [
-        "## Anthropic Requests"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 15,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "tgkuoHa5uLOy",
-        "outputId": "27a78e86-c6a7-4bcc-8559-0813cb978426"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Claude instant 1, response\n",
-            "{\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \" I'm doing well, thanks for asking!\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"id\": \"chatcmpl-4f2e64a1-56d2-43f2-90d3-60ffd6f5086d\",\n",
-            "  \"created\": 1696256761.3265705,\n",
-            "  \"model\": \"anthropic.claude-instant-v1\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 11,\n",
-            "    \"completion_tokens\": 9,\n",
-            "    \"total_tokens\": 20\n",
-            "  },\n",
-            "  \"finish_reason\": \"stop_sequence\"\n",
-            "}\n",
-            "Claude v2, response\n",
-            "{\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \" I'm doing well, thanks for asking!\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"id\": \"chatcmpl-34f59b33-f94e-40c2-8bdb-f4af0813405e\",\n",
-            "  \"created\": 1696256762.2137017,\n",
-            "  \"model\": \"anthropic.claude-v2\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 11,\n",
-            "    \"completion_tokens\": 9,\n",
-            "    \"total_tokens\": 20\n",
-            "  },\n",
-            "  \"finish_reason\": \"stop_sequence\"\n",
-            "}\n"
-          ]
-        }
-      ],
-      "source": [
-        "from litellm import completion\n",
-        "\n",
-        "response = completion(\n",
-        "            model=\"bedrock/anthropic.claude-instant-v1\",\n",
-        "            messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-        ")\n",
-        "print(\"Claude instant 1, response\")\n",
-        "print(response)\n",
-        "\n",
-        "\n",
-        "response = completion(\n",
-        "            model=\"bedrock/anthropic.claude-v2\",\n",
-        "            messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-        ")\n",
-        "print(\"Claude v2, response\")\n",
-        "print(response)"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "HnM-HtM3yFMT"
-      },
-      "source": [
-        "## Anthropic Requests - With Streaming"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "_JZvg2yovRsU"
-      },
-      "outputs": [],
-      "source": [
-        "from litellm import completion\n",
-        "\n",
-        "response = completion(\n",
-        "            model=\"bedrock/anthropic.claude-instant-v1\",\n",
-        "            messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
-        "            stream=True,\n",
-        ")\n",
-        "print(\"Claude instant 1, response\")\n",
-        "for chunk in response:\n",
-        "  print(chunk)\n",
-        "\n",
-        "\n",
-        "response = completion(\n",
-        "            model=\"bedrock/anthropic.claude-v2\",\n",
-        "            messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
-        "            stream=True\n",
-        ")\n",
-        "print(\"Claude v2, response\")\n",
-        "print(response)\n",
-        "for chunk in response:\n",
-        "  print(chunk)"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "zj1U1mh9zEhP"
-      },
-      "source": [
-        "## A121 Requests"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 19,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "6wK6MZLovU7r",
-        "outputId": "4cf80c04-f15d-4066-b4c7-113b551538de"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "J2 ultra response\n",
-            "{\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \"\\nHi, I'm doing well, thanks for asking! How about you?\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"id\": \"chatcmpl-f2de678f-0e70-4e36-a01f-8b184c2e4d50\",\n",
-            "  \"created\": 1696257116.044311,\n",
-            "  \"model\": \"ai21.j2-ultra\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 6,\n",
-            "    \"completion_tokens\": 16,\n",
-            "    \"total_tokens\": 22\n",
-            "  }\n",
-            "}\n",
-            "J2 mid response\n",
-            "{\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \"\\nGood. And you?\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"id\": \"chatcmpl-420d6bf9-36d8-484b-93b4-4c9e00f7ce2e\",\n",
-            "  \"created\": 1696257116.5756805,\n",
-            "  \"model\": \"ai21.j2-mid\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 6,\n",
-            "    \"completion_tokens\": 6,\n",
-            "    \"total_tokens\": 12\n",
-            "  }\n",
-            "}\n"
-          ]
-        }
-      ],
-      "source": [
-        "response = completion(\n",
-        "            model=\"bedrock/ai21.j2-ultra\",\n",
-        "            messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
-        ")\n",
-        "print(\"J2 ultra response\")\n",
-        "print(response)\n",
-        "\n",
-        "response = completion(\n",
-        "            model=\"bedrock/ai21.j2-mid\",\n",
-        "            messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
-        ")\n",
-        "print(\"J2 mid response\")\n",
-        "print(response)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "Y5gGZIwzzSON"
-      },
-      "outputs": [],
-      "source": []
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_Completion_Cost.ipynb

@@ -1,241 +0,0 @@
-{
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "source": [
-        "# Use LiteLLM to calculate costs for all your completion calls\n",
-        "In this notebook we'll use `litellm.completion_cost` to get completion costs"
-      ],
-      "metadata": {
-        "id": "BgWr0PsUR3vV"
-      }
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "ViczFTjsDzSI"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm==0.1.549 # use 0.1.549  or later"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "## Calculating costs for gpt-3.5 turbo completion()"
-      ],
-      "metadata": {
-        "id": "k_1CWUwmSNtj"
-      }
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "from litellm import completion, completion_cost\n",
-        "import os\n",
-        "os.environ['OPENAI_API_KEY'] = \"\"\n",
-        "\n",
-        "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-        "response = completion(\n",
-        "            model=\"gpt-3.5-turbo\",\n",
-        "            messages=messages,\n",
-        ")\n",
-        "\n",
-        "print(response)\n",
-        "\n",
-        "cost = completion_cost(completion_response=response)\n",
-        "formatted_string = f\"Cost for completion call: ${float(cost):.10f}\"\n",
-        "print(formatted_string)\n"
-      ],
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "Tp0fyk-jD0pP",
-        "outputId": "ce885fb3-3237-41b2-9d8b-3fb30bba498b"
-      },
-      "execution_count": 6,
-      "outputs": [
-        {
-          "output_type": "stream",
-          "name": "stdout",
-          "text": [
-            "got response\n",
-            "{\n",
-            "  \"id\": \"chatcmpl-7vyCApIZaCxP36kb9meUMN2DFSJPh\",\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"created\": 1694050442,\n",
-            "  \"model\": \"gpt-3.5-turbo-0613\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"content\": \"Hello! I'm an AI and I don't have feelings, but I'm here to help you. How can I assist you today?\"\n",
-            "      },\n",
-            "      \"finish_reason\": \"stop\"\n",
-            "    }\n",
-            "  ],\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 13,\n",
-            "    \"completion_tokens\": 28,\n",
-            "    \"total_tokens\": 41\n",
-            "  }\n",
-            "}\n",
-            "Cost for completion call: $0.0000755000\n"
-          ]
-        }
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "## Calculating costs for Together Computer completion()"
-      ],
-      "metadata": {
-        "id": "AjDs4G-uS6PS"
-      }
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "from litellm import completion, completion_cost\n",
-        "import os\n",
-        "os.environ['TOGETHERAI_API_KEY'] = \"\"\n",
-        "\n",
-        "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-        "response = completion(\n",
-        "            model=\"togethercomputer/llama-2-70b-chat\",\n",
-        "            messages=messages,\n",
-        ")\n",
-        "\n",
-        "print(response)\n",
-        "\n",
-        "cost = completion_cost(completion_response=response)\n",
-        "formatted_string = f\"Cost for completion call: ${float(cost):.10f}\"\n",
-        "print(formatted_string)\n"
-      ],
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "jMPsUV-KEa6a",
-        "outputId": "7a69b291-f149-4b9c-8a78-9c8142bac759"
-      },
-      "execution_count": 7,
-      "outputs": [
-        {
-          "output_type": "stream",
-          "name": "stdout",
-          "text": [
-            "{\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \"Hello! I'm doing well, thanks for asking. I hope you're having a great\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"created\": 1694050771.2821715,\n",
-            "  \"model\": \"togethercomputer/llama-2-70b-chat\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 12,\n",
-            "    \"completion_tokens\": 18,\n",
-            "    \"total_tokens\": 30\n",
-            "  }\n",
-            "}\n",
-            "Cost for completion call: $0.0000900000\n"
-          ]
-        }
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "## Calculating costs for Replicate Llama2 completion()"
-      ],
-      "metadata": {
-        "id": "vEa4s6-7TANS"
-      }
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "from litellm import completion, completion_cost\n",
-        "import os\n",
-        "os.environ['REPLICATE_API_KEY'] = \"\"\n",
-        "\n",
-        "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
-        "response = completion(\n",
-        "            model=\"replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf\",\n",
-        "            messages=messages,\n",
-        ")\n",
-        "\n",
-        "print(response)\n",
-        "\n",
-        "cost = completion_cost(completion_response=response)\n",
-        "formatted_string = f\"Cost for completion call: ${float(cost):.10f}\"\n",
-        "print(formatted_string)\n"
-      ],
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "Xf1TKRDuS1bR",
-        "outputId": "cfb2b484-a6e5-41ad-86c5-7e66aba27648"
-      },
-      "execution_count": 8,
-      "outputs": [
-        {
-          "output_type": "stream",
-          "name": "stdout",
-          "text": [
-            "{\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \" Hello! I'm doing well, thanks for asking. How about you? Is there anything you need help with today?\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"created\": 1694050893.4534576,\n",
-            "  \"model\": \"replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 6,\n",
-            "    \"completion_tokens\": 24,\n",
-            "    \"total_tokens\": 30\n",
-            "  },\n",
-            "  \"ended\": 1694050896.6689413\n",
-            "}\n",
-            "total_replicate_run_time 3.2154836654663086\n",
-            "Cost for completion call: $0.0045016771\n"
-          ]
-        }
-      ]
-    }
-  ]
-}

cookbook/LiteLLM_HuggingFace.ipynb

@@ -1,252 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "9dKM5k8qsMIj"
-      },
-      "source": [
-        "## LiteLLM Hugging Face\n",
-        "\n",
-        "Docs for huggingface: https://docs.litellm.ai/docs/providers/huggingface\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "BVDdmCp-o97j"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "yp5UXRqtpu9f"
-      },
-      "source": [
-        "## Serverless Inference Providers\n",
-        "\n",
-        "Read more about Inference Providers here: https://huggingface.co/blog/inference-providers.\n",
-        "\n",
-        "In order to use litellm with Hugging Face Inference Providers, you need to set `model=huggingface/<provider>/<model-id>`.\n",
-        "\n",
-        "Example: `huggingface/together/deepseek-ai/DeepSeek-R1` to run DeepSeek-R1 (https://huggingface.co/deepseek-ai/DeepSeek-R1) through Together AI.\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "Pi5Oww8gpCUm",
-        "outputId": "659a67c7-f90d-4c06-b94e-2c4aa92d897a"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "from litellm import completion\n",
-        "\n",
-        "# You can create a HF token here: https://huggingface.co/settings/tokens\n",
-        "os.environ[\"HF_TOKEN\"] = \"hf_xxxxxx\"\n",
-        "\n",
-        "# Call DeepSeek-R1 model through Together AI\n",
-        "response = completion(\n",
-        "    model=\"huggingface/together/deepseek-ai/DeepSeek-R1\",\n",
-        "    messages=[{\"content\": \"How many r's are in the word `strawberry`?\", \"role\": \"user\"}],\n",
-        ")\n",
-        "print(response)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "EU0UubrKzTFe"
-      },
-      "source": [
-        "## Streaming\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "y-QfIvA-uJKX",
-        "outputId": "b007bb98-00d0-44a4-8264-c8a2caed6768"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "from litellm import completion\n",
-        "\n",
-        "os.environ[\"HF_TOKEN\"] = \"hf_xxxxxx\"\n",
-        "\n",
-        "response = completion(\n",
-        "    model=\"huggingface/together/deepseek-ai/DeepSeek-R1\",\n",
-        "    messages=[\n",
-        "        {\n",
-        "            \"role\": \"user\",\n",
-        "            \"content\": \"How many r's are in the word `strawberry`?\",\n",
-        "            \n",
-        "        }\n",
-        "    ],\n",
-        "    stream=True,\n",
-        ")\n",
-        "\n",
-        "for chunk in response:\n",
-        "    print(chunk)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## With images as input\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from litellm import completion\n",
-        "\n",
-        "# Set your Hugging Face Token\n",
-        "os.environ[\"HF_TOKEN\"] = \"hf_xxxxxx\"\n",
-        "\n",
-        "messages = [\n",
-        "    {\n",
-        "        \"role\": \"user\",\n",
-        "        \"content\": [\n",
-        "            {\"type\": \"text\", \"text\": \"What's in this image?\"},\n",
-        "            {\n",
-        "                \"type\": \"image_url\",\n",
-        "                \"image_url\": {\n",
-        "                    \"url\": \"https://awsmp-logos.s3.amazonaws.com/seller-xw5kijmvmzasy/c233c9ade2ccb5491072ae232c814942.png\",\n",
-        "                },\n",
-        "            },\n",
-        "        ],\n",
-        "    }\n",
-        "]\n",
-        "\n",
-        "response = completion(\n",
-        "    model=\"huggingface/sambanova/meta-llama/Llama-3.3-70B-Instruct\",\n",
-        "    messages=messages,\n",
-        ")\n",
-        "print(response.choices[0])"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Tools - Function Calling\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "from litellm import completion\n",
-        "\n",
-        "\n",
-        "# Set your Hugging Face Token\n",
-        "os.environ[\"HF_TOKEN\"] = \"hf_xxxxxx\"\n",
-        "\n",
-        "tools = [\n",
-        "    {\n",
-        "        \"type\": \"function\",\n",
-        "        \"function\": {\n",
-        "            \"name\": \"get_current_weather\",\n",
-        "            \"description\": \"Get the current weather in a given location\",\n",
-        "            \"parameters\": {\n",
-        "                \"type\": \"object\",\n",
-        "                \"properties\": {\n",
-        "                    \"location\": {\n",
-        "                        \"type\": \"string\",\n",
-        "                        \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
-        "                    },\n",
-        "                    \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n",
-        "                },\n",
-        "                \"required\": [\"location\"],\n",
-        "            },\n",
-        "        },\n",
-        "    }\n",
-        "]\n",
-        "messages = [{\"role\": \"user\", \"content\": \"What's the weather like in Boston today?\"}]\n",
-        "\n",
-        "response = completion(\n",
-        "    model=\"huggingface/sambanova/meta-llama/Llama-3.1-8B-Instruct\", messages=messages, tools=tools, tool_choice=\"auto\"\n",
-        ")\n",
-        "print(response)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Hugging Face Dedicated Inference Endpoints\n",
-        "\n",
-        "Steps to use\n",
-        "\n",
-        "- Create your own Hugging Face dedicated endpoint here: https://ui.endpoints.huggingface.co/\n",
-        "- Set `api_base` to your deployed api base\n",
-        "- set the model to `huggingface/tgi` so that litellm knows it's a huggingface Deployed Inference Endpoint.\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "import litellm\n",
-        "\n",
-        "\n",
-        "response = litellm.completion(\n",
-        "    model=\"huggingface/tgi\",\n",
-        "    messages=[{\"content\": \"Hello, how are you?\", \"role\": \"user\"}],\n",
-        "    api_base=\"https://my-endpoint.endpoints.huggingface.cloud/v1/\",\n",
-        ")\n",
-        "print(response)"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": ".venv",
-      "language": "python",
-      "name": "python3"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.12.0"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_NovitaAI_Cookbook.ipynb

@@ -1,97 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "iFEmsVJI_2BR"
-      },
-      "source": [
-        "# LiteLLM NovitaAI Cookbook"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "cBlUhCEP_xj4"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "p-MQqWOT_1a7"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "os.environ['NOVITA_API_KEY'] = \"\""
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "Ze8JqMqWAARO"
-      },
-      "outputs": [],
-      "source": [
-        "from litellm import completion\n",
-        "response = completion(\n",
-        "            model=\"novita/deepseek/deepseek-r1\",\n",
-        "            messages=[{\"role\": \"user\", \"content\": \"write code for saying hi\"}]\n",
-        ")\n",
-        "response"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "-LnhELrnAM_J"
-      },
-      "outputs": [],
-      "source": [
-        "response = completion(\n",
-        "            model=\"novita/deepseek/deepseek-r1\",\n",
-        "            messages=[{\"role\": \"user\", \"content\": \"write code for saying hi\"}]\n",
-        ")\n",
-        "response"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "dJBOUYdwCEn1"
-      },
-      "outputs": [],
-      "source": [
-        "response = completion(\n",
-        "            model=\"mistralai/mistral-7b-instruct\",\n",
-        "            messages=[{\"role\": \"user\", \"content\": \"write code for saying hi\"}]\n",
-        ")\n",
-        "response"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_OpenRouter.ipynb

@@ -1,179 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "iFEmsVJI_2BR"
-      },
-      "source": [
-        "# LiteLLM OpenRouter Cookbook"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "cBlUhCEP_xj4"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 14,
-      "metadata": {
-        "id": "p-MQqWOT_1a7"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "os.environ['OPENROUTER_API_KEY'] = \"\""
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 11,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "Ze8JqMqWAARO",
-        "outputId": "64f3e836-69fa-4f8e-fb35-088a913bbe98"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "<OpenAIObject id=gen-W8FTMSIEorCp3vG5iYIgNMR4IeBv at 0x7c3dcef1f060> JSON: {\n",
-              "  \"id\": \"gen-W8FTMSIEorCp3vG5iYIgNMR4IeBv\",\n",
-              "  \"model\": \"chat-bison@001\",\n",
-              "  \"choices\": [\n",
-              "    {\n",
-              "      \"message\": {\n",
-              "        \"role\": \"assistant\",\n",
-              "        \"content\": \"```\\n#include <stdio.h>\\n\\nint main() {\\n  printf(\\\"Hi!\\\\n\\\");\\n  return 0;\\n}\\n```\"\n",
-              "      }\n",
-              "    }\n",
-              "  ],\n",
-              "  \"response_ms\": 7817.777999999999\n",
-              "}"
-            ]
-          },
-          "execution_count": 11,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "from litellm import completion\n",
-        "response = completion(\n",
-        "            model=\"openrouter/google/palm-2-chat-bison\",\n",
-        "            messages=[{\"role\": \"user\", \"content\": \"write code for saying hi\"}]\n",
-        ")\n",
-        "response"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 12,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "-LnhELrnAM_J",
-        "outputId": "d51c7ab7-d761-4bd1-f849-1534d9df4cd0"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "<OpenAIObject id=gen-IiuV7ZNimDufVeutBHrl8ajPuzEh at 0x7c3dcea67560> JSON: {\n",
-              "  \"choices\": [\n",
-              "    {\n",
-              "      \"message\": {\n",
-              "        \"role\": \"assistant\",\n",
-              "        \"content\": \" Here is some simple code to print \\\"Hi\\\":\\n\\n```python\\nprint(\\\"Hi\\\")\\n```\\n\\nThis uses the print() function in Python to output the text \\\"Hi\\\".\"\n",
-              "      },\n",
-              "      \"finish_reason\": \"stop_sequence\"\n",
-              "    }\n",
-              "  ],\n",
-              "  \"model\": \"claude-2.0\",\n",
-              "  \"id\": \"gen-IiuV7ZNimDufVeutBHrl8ajPuzEh\",\n",
-              "  \"response_ms\": 8112.443000000001\n",
-              "}"
-            ]
-          },
-          "execution_count": 12,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "response = completion(\n",
-        "            model=\"openrouter/anthropic/claude-2\",\n",
-        "            messages=[{\"role\": \"user\", \"content\": \"write code for saying hi\"}]\n",
-        ")\n",
-        "response"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 13,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "dJBOUYdwCEn1",
-        "outputId": "ffa18679-ec15-4dad-fe2b-68665cdf36b0"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "<OpenAIObject id=gen-PyMd3yyJ0aQsCgIY9R8XGZoAtPbl at 0x7c3dceefcae0> JSON: {\n",
-              "  \"id\": \"gen-PyMd3yyJ0aQsCgIY9R8XGZoAtPbl\",\n",
-              "  \"model\": \"togethercomputer/llama-2-70b-chat\",\n",
-              "  \"choices\": [\n",
-              "    {\n",
-              "      \"message\": {\n",
-              "        \"role\": \"assistant\",\n",
-              "        \"content\": \"*gives a sly smile as they type*\\n\\nHey there, handsome. \\ud83d\\ude0f\\n\\nWhat brings you to my neck of the woods today? \\ud83d\\ude18\"\n",
-              "      }\n",
-              "    }\n",
-              "  ],\n",
-              "  \"response_ms\": 9618.775\n",
-              "}"
-            ]
-          },
-          "execution_count": 13,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "response = completion(\n",
-        "            model=\"openrouter/meta-llama/llama-2-70b-chat\",\n",
-        "            messages=[{\"role\": \"user\", \"content\": \"write code for saying hi\"}]\n",
-        ")\n",
-        "response"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_Petals.ipynb

@@ -1,568 +0,0 @@
-{
-  "cells": [
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "dwGtLi_tvM6N"
-      },
-      "source": [
-        "# Using LiteLLM with Petals"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "bdlgaWQqDpzj"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm # 0.1.715 and upwards"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "5Id2QKwOEH8X"
-      },
-      "outputs": [],
-      "source": [
-        "# install petals\n",
-        "!pip install git+https://github.com/bigscience-workshop/petals"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "k42fldw3veSN"
-      },
-      "source": [
-        "## petals-team/StableBeluga2"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 3,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "tIHcEHdSDqju",
-        "outputId": "485dbf54-395c-433a-bbf4-8eb70a9fa624"
-      },
-      "outputs": [
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
-            "Sep 19 18:39:50.634 [\u001b[1m\u001b[34mINFO\u001b[0m] Make sure you follow the LLaMA's terms of use: https://bit.ly/llama2-license for LLaMA 2, https://bit.ly/llama-license for LLaMA 1\n",
-            "Sep 19 18:39:50.639 [\u001b[1m\u001b[34mINFO\u001b[0m] Using DHT prefix: StableBeluga2-hf\n",
-            "Sep 19 18:40:13.920 [\u001b[1m\u001b[34mINFO\u001b[0m] Route found: 0:40 via …HfQWVM => 40:80 via …Zj98Se\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "{\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \"Hello, how are you?\\nI'm doing well, thank you. I'm just getting ready to go to the gym.\\nOh, that's great. I'm trying to get back into a workout routine myself.\\nYeah,\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"id\": \"chatcmpl-f09d79b3-c1d1-49b7-b55f-cd8dfa1043bf\",\n",
-            "  \"created\": 1695148897.473613,\n",
-            "  \"model\": \"petals-team/StableBeluga2\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 6,\n",
-            "    \"completion_tokens\": 45,\n",
-            "    \"total_tokens\": 51\n",
-            "  }\n",
-            "}\n"
-          ]
-        }
-      ],
-      "source": [
-        "from litellm import completion\n",
-        "\n",
-        "response = completion(model=\"petals/petals-team/StableBeluga2\", messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}], max_tokens=50)\n",
-        "\n",
-        "print(response)"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "J8DubRnHvh_j"
-      },
-      "source": [
-        "## huggyllama/llama-65b"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 4,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 538,
-          "referenced_widgets": [
-            "2fec5cc400424671a3d517327117d18a",
-            "3687c76fe84d464baaf35366b21e83b3",
-            "c29d4460dbaa441cae110b58e0014151",
-            "6560449a38bf4a7bacd97ccaacf01c4c",
-            "5fbd6ae281984d28ba59ebfd0279eda7",
-            "323e30e275434aeea241163e5f1f9031",
-            "48f4adec51c94f9da6e4c4564daeff84",
-            "2a672981a44b4a7fb30674f97f4c10c6",
-            "d75ae8d22ea74840b4c80c8f386384c4",
-            "54c06312ecff4e7588665e8b0cb7118b",
-            "300078a9d1a6483fba81a4be63793ff7"
-          ]
-        },
-        "id": "IlTCJwDsNvgF",
-        "outputId": "2e84d125-d982-48ed-8a92-6ca438a50d0c"
-      },
-      "outputs": [
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "Sep 19 18:41:37.912 [\u001b[1m\u001b[34mINFO\u001b[0m] Make sure you follow the LLaMA's terms of use: https://bit.ly/llama2-license for LLaMA 2, https://bit.ly/llama-license for LLaMA 1\n",
-            "Sep 19 18:41:37.914 [\u001b[1m\u001b[34mINFO\u001b[0m] Using DHT prefix: llama-65b-hf\n"
-          ]
-        },
-        {
-          "data": {
-            "application/vnd.jupyter.widget-view+json": {
-              "model_id": "2fec5cc400424671a3d517327117d18a",
-              "version_major": 2,
-              "version_minor": 0
-            },
-            "text/plain": [
-              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.2` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
-            "  warnings.warn(\n",
-            "Sep 19 18:41:48.396 [\u001b[1m\u001b[34mINFO\u001b[0m] Route found: 0:80 via …g634yJ\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "{\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"finish_reason\": \"stop\",\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"content\": \"Hello, how are you?\\nI'm fine, thank you. And\",\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"logprobs\": null\n",
-            "      }\n",
-            "    }\n",
-            "  ],\n",
-            "  \"id\": \"chatcmpl-3496e6eb-2a27-4f94-8d75-70648eacd88f\",\n",
-            "  \"created\": 1695148912.9116046,\n",
-            "  \"model\": \"huggyllama/llama-65b\",\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 6,\n",
-            "    \"completion_tokens\": 14,\n",
-            "    \"total_tokens\": 20\n",
-            "  }\n",
-            "}\n"
-          ]
-        }
-      ],
-      "source": [
-        "response = completion(model=\"petals/huggyllama/llama-65b\", messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}], temperature=0.2, max_tokens=10)\n",
-        "\n",
-        "print(response)"
-      ]
-    }
-  ],
-  "metadata": {
-    "accelerator": "GPU",
-    "colab": {
-      "gpuType": "T4",
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    },
-    "widgets": {
-      "application/vnd.jupyter.widget-state+json": {
-        "2a672981a44b4a7fb30674f97f4c10c6": {
-          "model_module": "@jupyter-widgets/base",
-          "model_module_version": "1.2.0",
-          "model_name": "LayoutModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/base",
-            "_model_module_version": "1.2.0",
-            "_model_name": "LayoutModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "LayoutView",
-            "align_content": null,
-            "align_items": null,
-            "align_self": null,
-            "border": null,
-            "bottom": null,
-            "display": null,
-            "flex": null,
-            "flex_flow": null,
-            "grid_area": null,
-            "grid_auto_columns": null,
-            "grid_auto_flow": null,
-            "grid_auto_rows": null,
-            "grid_column": null,
-            "grid_gap": null,
-            "grid_row": null,
-            "grid_template_areas": null,
-            "grid_template_columns": null,
-            "grid_template_rows": null,
-            "height": null,
-            "justify_content": null,
-            "justify_items": null,
-            "left": null,
-            "margin": null,
-            "max_height": null,
-            "max_width": null,
-            "min_height": null,
-            "min_width": null,
-            "object_fit": null,
-            "object_position": null,
-            "order": null,
-            "overflow": null,
-            "overflow_x": null,
-            "overflow_y": null,
-            "padding": null,
-            "right": null,
-            "top": null,
-            "visibility": null,
-            "width": null
-          }
-        },
-        "2fec5cc400424671a3d517327117d18a": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "HBoxModel",
-          "state": {
-            "_dom_classes": [],
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "HBoxModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/controls",
-            "_view_module_version": "1.5.0",
-            "_view_name": "HBoxView",
-            "box_style": "",
-            "children": [
-              "IPY_MODEL_3687c76fe84d464baaf35366b21e83b3",
-              "IPY_MODEL_c29d4460dbaa441cae110b58e0014151",
-              "IPY_MODEL_6560449a38bf4a7bacd97ccaacf01c4c"
-            ],
-            "layout": "IPY_MODEL_5fbd6ae281984d28ba59ebfd0279eda7"
-          }
-        },
-        "300078a9d1a6483fba81a4be63793ff7": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "DescriptionStyleModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "DescriptionStyleModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "StyleView",
-            "description_width": ""
-          }
-        },
-        "323e30e275434aeea241163e5f1f9031": {
-          "model_module": "@jupyter-widgets/base",
-          "model_module_version": "1.2.0",
-          "model_name": "LayoutModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/base",
-            "_model_module_version": "1.2.0",
-            "_model_name": "LayoutModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "LayoutView",
-            "align_content": null,
-            "align_items": null,
-            "align_self": null,
-            "border": null,
-            "bottom": null,
-            "display": null,
-            "flex": null,
-            "flex_flow": null,
-            "grid_area": null,
-            "grid_auto_columns": null,
-            "grid_auto_flow": null,
-            "grid_auto_rows": null,
-            "grid_column": null,
-            "grid_gap": null,
-            "grid_row": null,
-            "grid_template_areas": null,
-            "grid_template_columns": null,
-            "grid_template_rows": null,
-            "height": null,
-            "justify_content": null,
-            "justify_items": null,
-            "left": null,
-            "margin": null,
-            "max_height": null,
-            "max_width": null,
-            "min_height": null,
-            "min_width": null,
-            "object_fit": null,
-            "object_position": null,
-            "order": null,
-            "overflow": null,
-            "overflow_x": null,
-            "overflow_y": null,
-            "padding": null,
-            "right": null,
-            "top": null,
-            "visibility": null,
-            "width": null
-          }
-        },
-        "3687c76fe84d464baaf35366b21e83b3": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "HTMLModel",
-          "state": {
-            "_dom_classes": [],
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "HTMLModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/controls",
-            "_view_module_version": "1.5.0",
-            "_view_name": "HTMLView",
-            "description": "",
-            "description_tooltip": null,
-            "layout": "IPY_MODEL_323e30e275434aeea241163e5f1f9031",
-            "placeholder": "​",
-            "style": "IPY_MODEL_48f4adec51c94f9da6e4c4564daeff84",
-            "value": "Loading checkpoint shards: 100%"
-          }
-        },
-        "48f4adec51c94f9da6e4c4564daeff84": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "DescriptionStyleModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "DescriptionStyleModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "StyleView",
-            "description_width": ""
-          }
-        },
-        "54c06312ecff4e7588665e8b0cb7118b": {
-          "model_module": "@jupyter-widgets/base",
-          "model_module_version": "1.2.0",
-          "model_name": "LayoutModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/base",
-            "_model_module_version": "1.2.0",
-            "_model_name": "LayoutModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "LayoutView",
-            "align_content": null,
-            "align_items": null,
-            "align_self": null,
-            "border": null,
-            "bottom": null,
-            "display": null,
-            "flex": null,
-            "flex_flow": null,
-            "grid_area": null,
-            "grid_auto_columns": null,
-            "grid_auto_flow": null,
-            "grid_auto_rows": null,
-            "grid_column": null,
-            "grid_gap": null,
-            "grid_row": null,
-            "grid_template_areas": null,
-            "grid_template_columns": null,
-            "grid_template_rows": null,
-            "height": null,
-            "justify_content": null,
-            "justify_items": null,
-            "left": null,
-            "margin": null,
-            "max_height": null,
-            "max_width": null,
-            "min_height": null,
-            "min_width": null,
-            "object_fit": null,
-            "object_position": null,
-            "order": null,
-            "overflow": null,
-            "overflow_x": null,
-            "overflow_y": null,
-            "padding": null,
-            "right": null,
-            "top": null,
-            "visibility": null,
-            "width": null
-          }
-        },
-        "5fbd6ae281984d28ba59ebfd0279eda7": {
-          "model_module": "@jupyter-widgets/base",
-          "model_module_version": "1.2.0",
-          "model_name": "LayoutModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/base",
-            "_model_module_version": "1.2.0",
-            "_model_name": "LayoutModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "LayoutView",
-            "align_content": null,
-            "align_items": null,
-            "align_self": null,
-            "border": null,
-            "bottom": null,
-            "display": null,
-            "flex": null,
-            "flex_flow": null,
-            "grid_area": null,
-            "grid_auto_columns": null,
-            "grid_auto_flow": null,
-            "grid_auto_rows": null,
-            "grid_column": null,
-            "grid_gap": null,
-            "grid_row": null,
-            "grid_template_areas": null,
-            "grid_template_columns": null,
-            "grid_template_rows": null,
-            "height": null,
-            "justify_content": null,
-            "justify_items": null,
-            "left": null,
-            "margin": null,
-            "max_height": null,
-            "max_width": null,
-            "min_height": null,
-            "min_width": null,
-            "object_fit": null,
-            "object_position": null,
-            "order": null,
-            "overflow": null,
-            "overflow_x": null,
-            "overflow_y": null,
-            "padding": null,
-            "right": null,
-            "top": null,
-            "visibility": null,
-            "width": null
-          }
-        },
-        "6560449a38bf4a7bacd97ccaacf01c4c": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "HTMLModel",
-          "state": {
-            "_dom_classes": [],
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "HTMLModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/controls",
-            "_view_module_version": "1.5.0",
-            "_view_name": "HTMLView",
-            "description": "",
-            "description_tooltip": null,
-            "layout": "IPY_MODEL_54c06312ecff4e7588665e8b0cb7118b",
-            "placeholder": "​",
-            "style": "IPY_MODEL_300078a9d1a6483fba81a4be63793ff7",
-            "value": " 2/2 [00:00&lt;00:00,  2.36it/s]"
-          }
-        },
-        "c29d4460dbaa441cae110b58e0014151": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "FloatProgressModel",
-          "state": {
-            "_dom_classes": [],
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "FloatProgressModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/controls",
-            "_view_module_version": "1.5.0",
-            "_view_name": "ProgressView",
-            "bar_style": "success",
-            "description": "",
-            "description_tooltip": null,
-            "layout": "IPY_MODEL_2a672981a44b4a7fb30674f97f4c10c6",
-            "max": 2,
-            "min": 0,
-            "orientation": "horizontal",
-            "style": "IPY_MODEL_d75ae8d22ea74840b4c80c8f386384c4",
-            "value": 2
-          }
-        },
-        "d75ae8d22ea74840b4c80c8f386384c4": {
-          "model_module": "@jupyter-widgets/controls",
-          "model_module_version": "1.5.0",
-          "model_name": "ProgressStyleModel",
-          "state": {
-            "_model_module": "@jupyter-widgets/controls",
-            "_model_module_version": "1.5.0",
-            "_model_name": "ProgressStyleModel",
-            "_view_count": null,
-            "_view_module": "@jupyter-widgets/base",
-            "_view_module_version": "1.2.0",
-            "_view_name": "StyleView",
-            "bar_color": null,
-            "description_width": ""
-          }
-        }
-      }
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_User_Based_Rate_Limits.ipynb

@@ -1,224 +0,0 @@
-{
-  "cells": [
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "JRCXfhACct4Y"
-      },
-      "source": [
-        "## User Based Rate Limiting Using LiteLLM\n",
-        "- LiteLLM allows you to set budgets per user\n",
-        "- Check if a given user has cross their allocated budget\n",
-        "\n",
-        "In this notebook we create a $0.0002 daily budget per user and make completion calls using the litellm budget manager"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "fl1kcLG8aaIV"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install litellm uuid"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "zqRrpoQ3c6oQ"
-      },
-      "source": [
-        "## Imports & Env variables"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 13,
-      "metadata": {
-        "id": "CSkz8bmwdD3w"
-      },
-      "outputs": [],
-      "source": [
-        "import uuid\n",
-        "import os\n",
-        "os.environ['OPENAI_API_KEY'] = \"\""
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "ktqe3gSmdFQ4"
-      },
-      "source": [
-        "## completion() with the budget manager\n",
-        "\n",
-        "This code does the following\n",
-        "- Initializes a litellm.BudgetManager()\n",
-        "- Checks if a budget exists for a user\n",
-        "  - Creates a $0.0002 budget if the user does not exisr\n",
-        "- Makes a `litellm.completion()` request only if the user is under their budget"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 23,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "pUN48YvmaiRU",
-        "outputId": "082d6a8b-9aef-4794-9eac-7ba9823ea373"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "No budget exists for user: 29af95f8-c3c6-4c8c-b080-8b2d18d25432\n",
-            "\n",
-            "Creating a budget for user: 29af95f8-c3c6-4c8c-b080-8b2d18d25432, daily budget $0.0002\n",
-            "\n",
-            "User: 29af95f8-c3c6-4c8c-b080-8b2d18d25432 has spent $0, budget for user: $0.0002\n",
-            "\n",
-            "{\n",
-            "  \"id\": \"chatcmpl-7yAUkHQV8xdfldzzZnnnuVU8pl31b\",\n",
-            "  \"object\": \"chat.completion\",\n",
-            "  \"created\": 1694574378,\n",
-            "  \"model\": \"gpt-3.5-turbo-0613\",\n",
-            "  \"choices\": [\n",
-            "    {\n",
-            "      \"index\": 0,\n",
-            "      \"message\": {\n",
-            "        \"role\": \"assistant\",\n",
-            "        \"content\": \"Hello! I'm an AI, so I don't have emotions, but I'm here to assist you. How can I help you today?\"\n",
-            "      },\n",
-            "      \"finish_reason\": \"stop\"\n",
-            "    }\n",
-            "  ],\n",
-            "  \"usage\": {\n",
-            "    \"prompt_tokens\": 14,\n",
-            "    \"completion_tokens\": 29,\n",
-            "    \"total_tokens\": 43\n",
-            "  }\n",
-            "}\n"
-          ]
-        },
-        {
-          "data": {
-            "text/plain": [
-              "{'status': 'success'}"
-            ]
-          },
-          "execution_count": 23,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "from litellm import BudgetManager, completion\n",
-        "\n",
-        "# Initializes a litellm.BudgetManager()\n",
-        "budget_manager = BudgetManager(project_name=\"liteLLM_project\", client_type=\"hosted\") # see https://docs.litellm.ai/docs/budget_manager\n",
-        "\n",
-        "user_id = str(uuid.uuid4()) # create a new user id\n",
-        "daily_budget = 0.0002\n",
-        "\n",
-        "# Checks if a budget exists for a user\n",
-        "if not budget_manager.is_valid_user(user_id):\n",
-        "    # Creates a $0.0002 budget if the user does not exisr\n",
-        "    print(f\"No budget exists for user: {user_id}\\n\")\n",
-        "    print(f\"Creating a budget for user: {user_id}, daily budget ${daily_budget}\\n\")\n",
-        "    budget_manager.create_budget(total_budget=daily_budget, user=user_id, duration=\"daily\") # duration can be daily, weekly, monthly\n",
-        "\n",
-        "\n",
-        "# Makes a `litellm.completion()` request only if the user is under their budget\n",
-        "current_spend_for_user = budget_manager.get_current_cost(user=user_id)\n",
-        "budget_for_user = budget_manager.get_total_budget(user_id)\n",
-        "print(f\"User: {user_id} has spent ${current_spend_for_user}, budget for user: ${budget_for_user}\\n\")\n",
-        "\n",
-        "if current_spend_for_user <= budget_for_user:\n",
-        "    response = completion(model=\"gpt-3.5-turbo\", messages=[{\"role\": \"user\", \"content\": \"Hey, how's it going?\"}])\n",
-        "    budget_manager.update_cost(completion_obj=response, user=user_id)\n",
-        "else:\n",
-        "    response = \"Sorry - no budget!\"\n",
-        "\n",
-        "print(response)"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "yMOirNoBfmmc"
-      },
-      "source": [
-        "## Make 10 calls to cross the budget per user\n",
-        "- Code fails after user crossed their budget"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 24,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "apKF3H-xbFXc",
-        "outputId": "1c6ef0fe-e27e-4ead-adc6-2c7eb0214e44"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "User: 29af95f8-c3c6-4c8c-b080-8b2d18d25432 has spent $7.9e-05, budget for user: $0.0002\n",
-            "\n",
-            "User: 29af95f8-c3c6-4c8c-b080-8b2d18d25432 has spent $0.00015999999999999999, budget for user: $0.0002\n",
-            "\n",
-            "User: 29af95f8-c3c6-4c8c-b080-8b2d18d25432 has spent $0.00023899999999999998, budget for user: $0.0002\n",
-            "\n",
-            "User: 29af95f8-c3c6-4c8c-b080-8b2d18d25432 has exceeded budget, current spend $0.00023899999999999998, budget for user: $0.0002\n",
-            "\n"
-          ]
-        }
-      ],
-      "source": [
-        "user_id = \"29af95f8-c3c6-4c8c-b080-8b2d18d25432\" # set in the previous cell\n",
-        "\n",
-        "for _ in range(10):\n",
-        "  # check if a given call can be made\n",
-        "  current_spend_for_user = budget_manager.get_current_cost(user=user_id)\n",
-        "  budget_for_user = budget_manager.get_total_budget(user_id)\n",
-        "  print(f\"User: {user_id} has spent ${current_spend_for_user}, budget for user: ${budget_for_user}\\n\")\n",
-        "  if current_spend_for_user <= budget_for_user:\n",
-        "      response = completion(model=\"gpt-3.5-turbo\", messages=[{\"role\": \"user\", \"content\": \"Hey, how's it going?\"}])\n",
-        "      budget_manager.update_cost(completion_obj=response, user=user_id)\n",
-        "  else:\n",
-        "      response = \"Sorry - no budget!\"\n",
-        "      print(f\"User: {user_id} has exceeded budget, current spend ${current_spend_for_user}, budget for user: ${budget_for_user}\\n\")\n",
-        "      break # no more requests\n",
-        "\n",
-        "  # print(response)"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

cookbook/LiteLLM_batch_completion.ipynb

@@ -1,163 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "MbLbs1tbISk-"
-   },
-   "source": [
-    "# LiteLLM Batch Completions Example\n",
-    "\n",
-    "* This tutorial walks through using `batch_completion`\n",
-    "* Docs: https://docs.litellm.ai/docs/completion/batching"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "Ty6-ko_aDlPF"
-   },
-   "outputs": [],
-   "source": [
-    "!pip install litellm"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "KGhNJRUCIh1j"
-   },
-   "source": [
-    "## Import Batch Completion"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {
-    "id": "LOtI43snDrSK"
-   },
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from litellm import batch_completion\n",
-    "\n",
-    "# set your API_KEY\n",
-    "os.environ['ANTHROPIC_API_KEY'] = \"\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "Xhv92NBaIpaw"
-   },
-   "source": [
-    "## Calling `litellm.batch_completion`\n",
-    "\n",
-    "In the batch_completion method, you provide a list of messages where each sub-list of messages is passed to litellm.completion(), allowing you to process multiple prompts efficiently in a single API call."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {
-    "colab": {
-     "base_uri": "https://localhost:8080/"
-    },
-    "id": "yY7GIRLsDywu",
-    "outputId": "009ea67f-95d5-462b-947f-b0d21e60c5bb"
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[<ModelResponse at 0x7a164eed4450> JSON: {\n",
-       "   \"choices\": [\n",
-       "     {\n",
-       "       \"finish_reason\": \"stop\",\n",
-       "       \"index\": 0,\n",
-       "       \"message\": {\n",
-       "         \"content\": \" Good morning!\",\n",
-       "         \"role\": \"assistant\",\n",
-       "         \"logprobs\": null\n",
-       "       }\n",
-       "     }\n",
-       "   ],\n",
-       "   \"created\": 1694030351.309254,\n",
-       "   \"model\": \"claude-2\",\n",
-       "   \"usage\": {\n",
-       "     \"prompt_tokens\": 11,\n",
-       "     \"completion_tokens\": 3,\n",
-       "     \"total_tokens\": 14\n",
-       "   }\n",
-       " },\n",
-       " <ModelResponse at 0x7a164eed5800> JSON: {\n",
-       "   \"choices\": [\n",
-       "     {\n",
-       "       \"finish_reason\": \"stop\",\n",
-       "       \"index\": 0,\n",
-       "       \"message\": {\n",
-       "         \"content\": \" I'm an AI assistant created by Anthropic. I don't actually have a concept of the current time.\",\n",
-       "         \"role\": \"assistant\",\n",
-       "         \"logprobs\": null\n",
-       "       }\n",
-       "     }\n",
-       "   ],\n",
-       "   \"created\": 1694030352.1215081,\n",
-       "   \"model\": \"claude-2\",\n",
-       "   \"usage\": {\n",
-       "     \"prompt_tokens\": 13,\n",
-       "     \"completion_tokens\": 22,\n",
-       "     \"total_tokens\": 35\n",
-       "   }\n",
-       " }]"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.environ['ANTHROPIC_API_KEY'] = \"\"\n",
-    "\n",
-    "\n",
-    "responses = batch_completion(\n",
-    "    model=\"claude-2\",\n",
-    "    messages = [\n",
-    "        [\n",
-    "            {\n",
-    "                \"role\": \"user\",\n",
-    "                \"content\": \"good morning? \"\n",
-    "            }\n",
-    "        ],\n",
-    "        [\n",
-    "            {\n",
-    "                \"role\": \"user\",\n",
-    "                \"content\": \"what's the time? \"\n",
-    "            }\n",
-    "        ]\n",
-    "    ]\n",
-    ")\n",
-    "responses"
-   ]
-  }
- ],
- "metadata": {
-  "colab": {
-   "provenance": []
-  },
-  "kernelspec": {
-   "display_name": "Python 3",
-   "name": "python3"
-  },
-  "language_info": {
-   "name": "python"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

cookbook/Migrating_to_LiteLLM_Proxy_from_OpenAI_Azure_OpenAI.ipynb

@@ -1,568 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "kccfk0mHZ4Ad"
-      },
-      "source": [
-        "# Migrating to LiteLLM Proxy from OpenAI/Azure OpenAI\n",
-        "\n",
-        "Covers:\n",
-        "\n",
-        "*   /chat/completion\n",
-        "*   /embedding\n",
-        "\n",
-        "\n",
-        "These are **selected examples**. LiteLLM Proxy is **OpenAI-Compatible**, it works with any project that calls OpenAI. Just change the `base_url`, `api_key` and `model`.\n",
-        "\n",
-        "Set `LITELLM_MASTER_KEY` in the notebook environment to the same key configured on the proxy before running authenticated examples\n",
-        "\n",
-        "For more examples, [go here](https://docs.litellm.ai/docs/proxy/user_keys)\n",
-        "\n",
-        "To pass provider-specific args, [go here](https://docs.litellm.ai/docs/completion/provider_specific_params#proxy-usage)\n",
-        "\n",
-        "To drop unsupported params (E.g. frequency_penalty for bedrock with librechat), [go here](https://docs.litellm.ai/docs/completion/drop_params#openai-proxy-usage)\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "nmSClzCPaGH6"
-      },
-      "source": [
-        "## /chat/completion\n",
-        "\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "_vqcjwOVaKpO"
-      },
-      "source": [
-        "### OpenAI Python SDK"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "x1e_Ok3KZzeP"
-      },
-      "outputs": [],
-      "source": [
-        "import openai\n",
-        "client = openai.OpenAI(\n",
-        "    api_key=\"anything\",\n",
-        "    base_url=\"http://0.0.0.0:4000\"\n",
-        ")\n",
-        "\n",
-        "# request sent to model set on litellm proxy, `litellm --model`\n",
-        "response = client.chat.completions.create(\n",
-        "    model=\"gpt-3.5-turbo\",\n",
-        "    messages = [\n",
-        "        {\n",
-        "            \"role\": \"user\",\n",
-        "            \"content\": \"this is a test request, write a short poem\"\n",
-        "        }\n",
-        "    ],\n",
-        "    extra_body={ # pass in any provider-specific param, if not supported by openai, https://docs.litellm.ai/docs/completion/input#provider-specific-params\n",
-        "        \"metadata\": { # 👈 use for logging additional params (e.g. to langfuse)\n",
-        "            \"generation_name\": \"ishaan-generation-openai-client\",\n",
-        "            \"generation_id\": \"openai-client-gen-id22\",\n",
-        "            \"trace_id\": \"openai-client-trace-id22\",\n",
-        "            \"trace_user_id\": \"openai-client-user-id2\"\n",
-        "        }\n",
-        "    }\n",
-        ")\n",
-        "\n",
-        "print(response)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "AqkyKk9Scxgj"
-      },
-      "source": [
-        "## Function Calling"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "wDg10VqLczE1"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "from openai import OpenAI\n",
-        "client = OpenAI(\n",
-        "    api_key=os.environ[\"LITELLM_MASTER_KEY\"],\n",
-        "    base_url=\"http://0.0.0.0:4000\",\n",
-        ")\n",
-        "\n",
-        "tools = [\n",
-        "  {\n",
-        "    \"type\": \"function\",\n",
-        "    \"function\": {\n",
-        "      \"name\": \"get_current_weather\",\n",
-        "      \"description\": \"Get the current weather in a given location\",\n",
-        "      \"parameters\": {\n",
-        "        \"type\": \"object\",\n",
-        "        \"properties\": {\n",
-        "          \"location\": {\n",
-        "            \"type\": \"string\",\n",
-        "            \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
-        "          },\n",
-        "          \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n",
-        "        },\n",
-        "        \"required\": [\"location\"],\n",
-        "      },\n",
-        "    }\n",
-        "  }\n",
-        "]\n",
-        "messages = [{\"role\": \"user\", \"content\": \"What's the weather like in Boston today?\"}]\n",
-        "completion = client.chat.completions.create(\n",
-        "  model=\"gpt-4o\", # use 'model_name' from config.yaml\n",
-        "  messages=messages,\n",
-        "  tools=tools,\n",
-        "  tool_choice=\"auto\"\n",
-        ")\n",
-        "\n",
-        "print(completion)\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "YYoxLloSaNWW"
-      },
-      "source": [
-        "### Azure OpenAI Python SDK"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "yA1XcgowaSRy"
-      },
-      "outputs": [],
-      "source": [
-        "import openai\n",
-        "client = openai.AzureOpenAI(\n",
-        "    api_key=\"anything\",\n",
-        "    base_url=\"http://0.0.0.0:4000\"\n",
-        ")\n",
-        "\n",
-        "# request sent to model set on litellm proxy, `litellm --model`\n",
-        "response = client.chat.completions.create(\n",
-        "    model=\"gpt-3.5-turbo\",\n",
-        "    messages = [\n",
-        "        {\n",
-        "            \"role\": \"user\",\n",
-        "            \"content\": \"this is a test request, write a short poem\"\n",
-        "        }\n",
-        "    ],\n",
-        "    extra_body={ # pass in any provider-specific param, if not supported by openai, https://docs.litellm.ai/docs/completion/input#provider-specific-params\n",
-        "        \"metadata\": { # 👈 use for logging additional params (e.g. to langfuse)\n",
-        "            \"generation_name\": \"ishaan-generation-openai-client\",\n",
-        "            \"generation_id\": \"openai-client-gen-id22\",\n",
-        "            \"trace_id\": \"openai-client-trace-id22\",\n",
-        "            \"trace_user_id\": \"openai-client-user-id2\"\n",
-        "        }\n",
-        "    }\n",
-        ")\n",
-        "\n",
-        "print(response)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "yl9qhDvnaTpL"
-      },
-      "source": [
-        "### Langchain Python"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "5MUZgSquaW5t"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.chat_models import ChatOpenAI\n",
-        "from langchain.prompts.chat import (\n",
-        "    ChatPromptTemplate,\n",
-        "    HumanMessagePromptTemplate,\n",
-        "    SystemMessagePromptTemplate,\n",
-        ")\n",
-        "from langchain.schema import HumanMessage, SystemMessage\n",
-        "import os\n",
-        "\n",
-        "os.environ[\"OPENAI_API_KEY\"] = \"anything\"\n",
-        "\n",
-        "chat = ChatOpenAI(\n",
-        "    openai_api_base=\"http://0.0.0.0:4000\",\n",
-        "    model = \"gpt-3.5-turbo\",\n",
-        "    temperature=0.1,\n",
-        "    extra_body={\n",
-        "        \"metadata\": {\n",
-        "            \"generation_name\": \"ishaan-generation-langchain-client\",\n",
-        "            \"generation_id\": \"langchain-client-gen-id22\",\n",
-        "            \"trace_id\": \"langchain-client-trace-id22\",\n",
-        "            \"trace_user_id\": \"langchain-client-user-id2\"\n",
-        "        }\n",
-        "    }\n",
-        ")\n",
-        "\n",
-        "messages = [\n",
-        "    SystemMessage(\n",
-        "        content=\"You are a helpful assistant that im using to make a test request to.\"\n",
-        "    ),\n",
-        "    HumanMessage(\n",
-        "        content=\"test from litellm. tell me why it's amazing in 1 sentence\"\n",
-        "    ),\n",
-        "]\n",
-        "response = chat(messages)\n",
-        "\n",
-        "print(response)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "B9eMgnULbRaz"
-      },
-      "source": [
-        "### Curl"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "VWCCk5PFcmhS"
-      },
-      "source": [
-        "\n",
-        "\n",
-        "```\n",
-        "curl -X POST 'http://0.0.0.0:4000/chat/completions' \\\n",
-        "    -H 'Content-Type: application/json' \\\n",
-        "    -d '{\n",
-        "    \"model\": \"gpt-3.5-turbo\",\n",
-        "    \"messages\": [\n",
-        "        {\n",
-        "        \"role\": \"user\",\n",
-        "        \"content\": \"what llm are you\"\n",
-        "        }\n",
-        "    ],\n",
-        "    \"metadata\": {\n",
-        "        \"generation_name\": \"ishaan-test-generation\",\n",
-        "        \"generation_id\": \"gen-id22\",\n",
-        "        \"trace_id\": \"trace-id22\",\n",
-        "        \"trace_user_id\": \"user-id2\"\n",
-        "    }\n",
-        "}'\n",
-        "```\n",
-        "\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "drBAm2e1b6xe"
-      },
-      "source": [
-        "### LlamaIndex"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "d0bZcv8fb9mL"
-      },
-      "outputs": [],
-      "source": [
-        "import os, dotenv\n",
-        "\n",
-        "from llama_index.llms import AzureOpenAI\n",
-        "from llama_index.embeddings import AzureOpenAIEmbedding\n",
-        "from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext\n",
-        "\n",
-        "llm = AzureOpenAI(\n",
-        "    engine=\"azure-gpt-3.5\",               # model_name on litellm proxy\n",
-        "    temperature=0.0,\n",
-        "    azure_endpoint=\"http://0.0.0.0:4000\", # litellm proxy endpoint\n",
-        "    api_key=os.environ[\"LITELLM_MASTER_KEY\"],\n",
-        "    api_version=\"2023-07-01-preview\",\n",
-        ")\n",
-        "\n",
-        "embed_model = AzureOpenAIEmbedding(\n",
-        "    deployment_name=\"azure-embedding-model\",\n",
-        "    azure_endpoint=\"http://0.0.0.0:4000\",\n",
-        "    api_key=os.environ[\"LITELLM_MASTER_KEY\"],\n",
-        "    api_version=\"2023-07-01-preview\",\n",
-        ")\n",
-        "\n",
-        "\n",
-        "documents = SimpleDirectoryReader(\"llama_index_data\").load_data()\n",
-        "service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)\n",
-        "index = VectorStoreIndex.from_documents(documents, service_context=service_context)\n",
-        "\n",
-        "query_engine = index.as_query_engine()\n",
-        "response = query_engine.query(\"What did the author do growing up?\")\n",
-        "print(response)\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "xypvNdHnb-Yy"
-      },
-      "source": [
-        "### Langchain JS"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "R55mK2vCcBN2"
-      },
-      "outputs": [],
-      "source": [
-        "import { ChatOpenAI } from \"@langchain/openai\";\n",
-        "\n",
-        "\n",
-        "const model = new ChatOpenAI({\n",
-        "  modelName: \"gpt-4\",\n",
-        "  openAIApiKey: process.env.LITELLM_MASTER_KEY,\n",
-        "  modelKwargs: {\"metadata\": \"hello world\"} // 👈 PASS Additional params here\n",
-        "}, {\n",
-        "  basePath: \"http://0.0.0.0:4000\",\n",
-        "});\n",
-        "\n",
-        "const message = await model.invoke(\"Hi there!\");\n",
-        "\n",
-        "console.log(message);\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "nC4bLifCcCiW"
-      },
-      "source": [
-        "### OpenAI JS"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "MICH8kIMcFpg"
-      },
-      "outputs": [],
-      "source": [
-        "const { OpenAI } = require('openai');\n",
-        "\n",
-        "const openai = new OpenAI({\n",
-        "  apiKey: process.env.LITELLM_MASTER_KEY,\n",
-        "  baseURL: \"http://0.0.0.0:4000\"\n",
-        "});\n",
-        "\n",
-        "async function main() {\n",
-        "  const chatCompletion = await openai.chat.completions.create({\n",
-        "    messages: [{ role: 'user', content: 'Say this is a test' }],\n",
-        "    model: 'gpt-3.5-turbo',\n",
-        "  }, {\"metadata\": {\n",
-        "            \"generation_name\": \"ishaan-generation-openaijs-client\",\n",
-        "            \"generation_id\": \"openaijs-client-gen-id22\",\n",
-        "            \"trace_id\": \"openaijs-client-trace-id22\",\n",
-        "            \"trace_user_id\": \"openaijs-client-user-id2\"\n",
-        "        }});\n",
-        "}\n",
-        "\n",
-        "main();\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "D1Q07pEAcGTb"
-      },
-      "source": [
-        "### Anthropic SDK"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "qBjFcAvgcI3t"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "from anthropic import Anthropic\n",
-        "\n",
-        "client = Anthropic(\n",
-        "    base_url=\"http://localhost:4000\", # proxy endpoint\n",
-        "    api_key=\"sk-test-proxy-key-123\", # litellm proxy virtual key (example)\n",
-        ")\n",
-        "\n",
-        "message = client.messages.create(\n",
-        "    max_tokens=1024,\n",
-        "    messages=[\n",
-        "        {\n",
-        "            \"role\": \"user\",\n",
-        "            \"content\": \"Hello, Claude\",\n",
-        "        }\n",
-        "    ],\n",
-        "    model=\"claude-3-opus-20240229\",\n",
-        ")\n",
-        "print(message.content)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "dFAR4AJGcONI"
-      },
-      "source": [
-        "## /embeddings"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "lgNoM281cRzR"
-      },
-      "source": [
-        "### OpenAI Python SDK"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "NY3DJhPfcQhA"
-      },
-      "outputs": [],
-      "source": [
-        "import openai\n",
-        "from openai import OpenAI\n",
-        "\n",
-        "# set base_url to your proxy server\n",
-        "# set api_key to send to proxy server\n",
-        "client = OpenAI(api_key=\"<proxy-api-key>\", base_url=\"http://0.0.0.0:4000\")\n",
-        "\n",
-        "response = client.embeddings.create(\n",
-        "    input=[\"hello from litellm\"],\n",
-        "    model=\"text-embedding-ada-002\"\n",
-        ")\n",
-        "\n",
-        "print(response)\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "hmbg-DW6cUZs"
-      },
-      "source": [
-        "### Langchain Embeddings"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "lX2S8Nl1cWVP"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.embeddings import OpenAIEmbeddings\n",
-        "\n",
-        "embeddings = OpenAIEmbeddings(model=\"sagemaker-embeddings\", openai_api_base=\"http://0.0.0.0:4000\", openai_api_key=\"temp-key\")\n",
-        "\n",
-        "\n",
-        "text = \"This is a test document.\"\n",
-        "\n",
-        "query_result = embeddings.embed_query(text)\n",
-        "\n",
-        "print(f\"SAGEMAKER EMBEDDINGS\")\n",
-        "print(query_result[:5])\n",
-        "\n",
-        "embeddings = OpenAIEmbeddings(model=\"bedrock-embeddings\", openai_api_base=\"http://0.0.0.0:4000\", openai_api_key=\"temp-key\")\n",
-        "\n",
-        "text = \"This is a test document.\"\n",
-        "\n",
-        "query_result = embeddings.embed_query(text)\n",
-        "\n",
-        "print(f\"BEDROCK EMBEDDINGS\")\n",
-        "print(query_result[:5])\n",
-        "\n",
-        "embeddings = OpenAIEmbeddings(model=\"bedrock-titan-embeddings\", openai_api_base=\"http://0.0.0.0:4000\", openai_api_key=\"temp-key\")\n",
-        "\n",
-        "text = \"This is a test document.\"\n",
-        "\n",
-        "query_result = embeddings.embed_query(text)\n",
-        "\n",
-        "print(f\"TITAN EMBEDDINGS\")\n",
-        "print(query_result[:5])"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "oqGbWBCQcYfd"
-      },
-      "source": [
-        "### Curl Request"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "7rkIMV9LcdwQ"
-      },
-      "source": [
-        "\n",
-        "\n",
-        "```curl\n",
-        "curl -X POST 'http://0.0.0.0:4000/embeddings' \\\n",
-        "  -H 'Content-Type: application/json' \\\n",
-        "  -d ' {\n",
-        "  \"model\": \"text-embedding-ada-002\",\n",
-        "  \"input\": [\"write a litellm poem\"]\n",
-        "  }'\n",
-        "```\n",
-        "\n"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}