scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-15 09:12:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 311

Merge main into nadir-classifier-plugin

Browse source 
Resolves conflicts from main's RoutingContext refactor (#49118):
complexity_router keeps both the NADIR_DECISIONS_CONFIG import and main's
router_deprecated import, and jev_classifier drops the _DecisionsRouter
protocol as main does. RouterJevClassifierClient now calls
RoutingContext.adecisions, which forwards kwargs as given, so the Nadir
conversation state (DecisionsJSON) still passes through unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
doramirdor 2026-10-30 22:27:00 -04:00
commit 5645970901
258 changed files with 10195 additions and 3687 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

77
.agents/skills/database-access/SKILL.md
View file

@ -1,54 +1,73 @@
---
name: database-access
description: Add database queries, extract existing Prisma calls into repositories, or explicitly port queries to Rust and SQLx without expanding migration scope
description: Organize database repositories and proxy dependency wiring, manage connection ownership, or port Prisma operations to Rust and SQLx without expanding migration scope
metadata:
internal: true
---
# Database access
## Situations
- Adding a query or an operation to a repository, read [adding a query](references/python-repository.md#adding-a-query)
- Moving existing Prisma calls behind a repository, read [moving existing access](references/python-repository.md#moving-existing-access-into-a-repository)
- Porting a repository from Prisma to Rust and SQLx, read [porting to SQLx](references/python-repository.md#porting-a-repository-to-sqlx)
- Injecting a repository into a route, read [Python and FastAPI](references/python-fastapi.md) or [Rust and Axum](references/rust-axum.md)
## Adding new database access
- Put Python database access behind an operation in `litellm/repositories`
- Give callers typed inputs and outputs, not Prisma clients, models, or query dictionaries
- Keep query execution and backend selection inside the repository
- Give each repository its own FastAPI provider, so a route builds only the repositories it uses
- A provider borrows existing handles and performs no database I/O during construction
- Define the operations a route calls as a Protocol in `litellm/models/<feature>.py`, next to its domain types
- The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
- Parsing and serialization of stored values live in the repository
- Add a Protocol when a repository gets a provider, not up front for every repository
- Keep query execution inside the repository, with one implementation per repository and no backend switch or fallback
- A repository is either all Prisma or all native, never a mix of both
- Prisma is fine for the implementation, adding a query does not require a Rust port
- Native Rust callers use `litellm-rust/crates/db` directly
## Asked to move existing access behind a repository
## Framework wiring
- Identify the calls being moved and their transaction boundary
- Move the existing Prisma implementation into the relevant repository
- Replace those direct database calls with repository operations
- Preserve return values, errors, and transaction behavior, with focused regression tests
- Keep this an extraction, not a backend migration or unrelated cleanup
- Python routes receive a per-repository dependency such as `UserBannerRepositoryDep` and call its operations, such as `banners.get_banner()`
- `DatabaseDep` lives in `litellm/proxy/deps/database.py`, repository providers in `litellm/proxy/deps/repositories.py`
- A provider returns the feature's Protocol, so routes never see which backend implements it
- Dependencies used by one feature only, like the banner audit binding, stay next to that feature's routes
- Repositories never import FastAPI, so jobs and helpers can use them without a request
- Use named FastAPI providers with `Annotated[..., Depends(...)]`, ordinary helpers and jobs receive explicit arguments
- Preserve each route's no-database behavior, authorization precedence and background-task scheduling
- Read [Python and FastAPI](references/python-fastapi.md) for provider examples, lifespan state and route verification
- Read [Rust and Axum](references/rust-axum.md) when adding database access to Rust handlers
## An equivalent Rust or SQLx query already exists
## Connection ownership
- Leave it and the current backend selection unchanged unless integration is explicitly requested
- Its existence alone does not justify switching a Prisma caller to Rust
- Repository extraction can keep using Prisma even when the Rust query is available
- Keep connection setup and shutdown separate from repositories, which own no pools
- Reuse existing handles, never connect per request or disconnect from a repository operation
- Give each pool one owner for setup and shutdown, borrowed handles never close it
- Finish requests, jobs and queued database work before closing connections
- Clean up resources opened before a startup failure, without a second disconnect fallback
- Add a backend to the existing lifecycle owner and pass its handle only to repositories that need it
- Preserve cancellation, timeout, error and transaction semantics across the native bridge
## Asked to port a Prisma query to Rust or SQLx
## Changing ownership or reloading resources
- Identify the existing inputs, outputs, errors, and transaction boundary
- Check `litellm-rust/crates/db` for an equivalent query and reuse it if its behavior matches
- Otherwise follow `litellm-rust/crates/db/AGENTS.md`
- Put static SQL in `queries/` and compile-check it with SQLx
- Keep typed inputs and reusable row projections in `src/`
- Refresh SQLx query metadata after SQL changes
- Keep Python conversion in the Python bridge, not in `litellm-rust/crates/db`
- When caller integration is requested, expose the Rust operation through the bridge and call it from the repository
- A query-only port can land without switching callers
- Verify behavior against real PostgreSQL before switching callers
- Check relevant null handling, conflicts, ordering, errors, and transaction semantics
- Remove the old implementation only when no callers need it
- Inventory initialization, reload, tests, background jobs and shutdown together
- Move every writer when changing authoritative storage, copying a replaceable global into state is insufficient
- A frozen repository does not make its clients immutable
- Decide whether each request captures one resource generation or reads it again during processing
- Build and validate replacements before publishing them, preserving the old generation on failure
- Serialize reloads and coordinate publication with shutdown
- Drain streams and background users before retiring old resources, retaining every generation grows resource use
- Define multi-worker coordination separately from an in-process swap
- Choose mechanisms such as `ArcSwap` only after defining publication, capture and retirement
## Migration context and boundaries
- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations incrementally with Rust APIs
- Mixed Prisma and Rust implementations are expected during the transition, unrelated existing calls can stay as they are
- Both backends may have separate pools through PgBouncer, size them together and verify pooling-mode compatibility
- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations with Rust APIs one repository at a time
- Prisma and native repositories coexist during the transition, unrelated existing calls can stay as they are
- Prisma and SQLx keep separate client pools to PgBouncer, which pools connections to PostgreSQL
- Size both layers together and preserve reader/writer routing and pooling-mode compatibility
- PgBouncer does not share transactions between Prisma and SQLx, keep each atomic unit of work on one backend and transaction
- Never retry a failed Rust write through Prisma when it may already have committed
- Prisma owns schema migrations today, eventual SQLx migration ownership and Prisma removal are separate planned steps, not part of a query port

101
.agents/skills/database-access/references/python-fastapi.md Normal file
View file

@ -0,0 +1,101 @@
# Python and FastAPI
## Existing pattern
- `UserBannerStore` in `litellm/models/user_banner.py` is the Protocol the banner routes call
- `get_database` in `litellm/proxy/deps/database.py` reads the lifespan-owned handle from request state
- `get_user_banner_repository` in `litellm/proxy/deps/repositories.py` builds the banner repository from that handle
- Choose each repository's replica routing once in its provider, never at call sites
- A handler receives `UserBannerStore | None`, never a Prisma client or a concrete repository
- `UserBannerRepository` implements `UserBannerStore` structurally, keeping the existing Prisma table, serialization and configuration-sync behavior
- The banner audit provider binds the legacy audit callable at request time, without performing audit I/O
- A native repository can later implement the same Protocol without changing handler signatures, see [porting a repository to SQLx](python-repository.md#porting-a-repository-to-sqlx)
## Dependency injection
- Add one provider and one `Annotated` alias per repository in `litellm/proxy/deps/repositories.py`
- Annotate the provider and alias with the feature's Protocol, not the concrete repository
- A route depends only on the repositories it calls, so a request never builds repositories it does not use
- Do not build repositories once at lifespan, a reconnect can replace the Prisma client they wrap
```python
class UserBannerStore(Protocol):
async def get_banner(self) -> UserBanner: ...
async def save_banner(self, banner: UserBanner) -> None: ...
```
```python
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
return None if database is None else UserBannerRepository(database.prisma)
UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)]
```
- FastAPI caches each provider within a request, so routes and sub-dependencies share one instance
- Ordinary helpers and jobs receive explicit arguments, never unresolved `Depends` defaults
- A route still reading the `prisma_client` global or building a repository inline switches to the repository's `...RepositoryDep`, keeping its no-database behavior and error precedence
```python
async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner:
if banners is None:
return UserBanner()
return await banners.get_banner()
```
## Lifespan state
- Prisma startup owns connection setup, authentication refresh, health checks and migrations
- `Database` holds the initialized Prisma client and the `NativeDatabase` handle, without holding fields per feature
- The native handle opens no pool yet, its `connect` and `close` are noops
- The proxy lifespan builds the handle with `Database.open`, registers `Database.aclose` on its exit stack and publishes it in request state
- Callers only open and close `Database`, which closes Prisma and the native handle together, and `open` closes Prisma itself when the native connect fails
- Starlette shallow-copies lifespan state into request state, which is distinct from `app.state`
- The database module never imports a repository, legacy globals borrow its handle without owning shutdown
- Drain requests, jobs, logging, spend and audit work before database teardown
- The exit stack closes the handle on exit, including startup failures after the client is created
- When adding native resources, initialize them in the serving process, respecting runtime fork restrictions
- Await native shutdown instead of depending on Python garbage collection
- Apply the [connection ownership rules](../SKILL.md#connection-ownership) when changing the owner
## Timing and errors
- FastAPI resolves dependencies before entering the handler
- Handler exception handling cannot catch failures raised during dependency resolution
- A universal required-database dependency can override authorization and validation behavior
- Banner GET without a database returns the default disabled banner
- Banner PATCH checks the admin role before returning the existing missing-database error
- Invalid stored banner data falls back to the default, but storage failures still propagate
- Banner writes finish before the success audit is queued as a FastAPI background task
- Queue post-response work with `BackgroundTasks`, not bare `asyncio.create_task`, so the in-flight drain waits for it before the database closes
- Do not move background work into an awaited provider merely to remove an import
## Verification
- Exercise real FastAPI handlers using `dependency_overrides` for external boundaries
- Keep production parsing and persistence code real
- Restore overrides after each test
- Verify complete response bodies, status codes and OpenAPI against the base
- Include combined failure conditions to pin error precedence
- Check saved values, fresh revisions, audit contents and failed-write behavior
- Check lifespan state with a running lifespan, not only a TestClient that skips startup
- Verify providers do not connect, query, disconnect or audit eagerly
- Use real PostgreSQL and live proxy requests when changing persistence behavior
## Scope
- Preserve direct Python call sites or migrate them explicitly
- Inventory imports, state writers, side effects and error ordering before migrating a feature
- Preserve configuration sync and serialization, and state which legacy reads remain
- Do not migrate every route in a large module just to establish the pattern
- Do not invent providers for resources without consumers
## Framework references
- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/) and [Starlette lifespan state](https://starlette.dev/lifespan/)
- [`SessionDep`-style `Annotated` aliases](https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-session-dependency), which `DatabaseDep` and `UserBannerRepositoryDep` follow
- [Dependency composition and caching](https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/)
- [Background tasks and dependencies with `yield`](https://fastapi.tiangolo.com/advanced/advanced-dependencies/#background-tasks-and-dependencies-with-yield-technical-details)
- [Dependency overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
- [Testing lifespan](https://fastapi.tiangolo.com/advanced/testing-events/)

119
.agents/skills/database-access/references/python-repository.md Normal file
View file

@ -0,0 +1,119 @@
# Python repositories
## Adding a query
Use this when a feature needs a read or write that no repository offers yet, including one more operation on an existing repository
### Before starting
- Search `litellm/repositories` for a repository that already owns the table, and extend it instead of adding a second one
- Check `litellm/models/<feature>.py` for an existing Protocol and domain types
- Decide the route's behavior when no database is configured, and where that check sits relative to authorization and validation
### Steps
1. Domain types and Protocol in `litellm/models/<feature>.py`
- The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
- Add the Protocol together with the provider, not for repositories nothing injects
2. Repository in `litellm/repositories/<feature>_repository.py`
- Implement the operation with Prisma, a Rust port is not required
- Compose the Prisma client and type the tables it touches with `TableActions` from `prisma_protocols.py`, never inherit `PrismaTableRepository`
- Parse and serialize stored values here, so callers only see domain types
- Never import FastAPI, connect, disconnect or choose replicas per call
3. Provider and `Annotated` alias in `litellm/proxy/deps/repositories.py`, see [dependency injection](python-fastapi.md#dependency-injection)
4. The route takes `<Feature>RepositoryDep`, handles `None` with the behavior decided above, and calls the operation
5. Jobs and helpers that need the operation receive the repository as an explicit argument, built by their caller from the handle it already holds
### Done when
- Handler tests drive the real route with `dependency_overrides` and the real repository code, see [verification](python-fastapi.md#verification)
- Tests cover the no-database path, error precedence, and what was actually stored
- Persistence changes are checked against real PostgreSQL through a live proxy
### Out of scope
- Porting the query to Rust, see [porting a repository to SQLx](#porting-a-repository-to-sqlx)
- Moving unrelated direct Prisma calls in the same module
## Moving existing access into a repository
Use this when code calls Prisma directly, through `prisma_client.db.<table>` or a helper, and the calls should live behind a repository. The backend stays Prisma
### Before starting
- List every call being moved, its callers, and the transaction or batch it runs in
- List the module's imports, global state writers, side effects such as audit and config sync, and error ordering
- Record current return values, raised errors and response bodies, these are what the change must preserve
### Steps
1. Move the existing Prisma code into the relevant repository in `litellm/repositories` without rewriting the queries
2. Add domain types and a Protocol in `litellm/models/<feature>.py` when a route will inject the repository
3. Replace each direct call with the repository operation, through a provider for routes and explicit arguments for jobs and helpers, see [Python and FastAPI](python-fastapi.md)
4. Calls sharing a transaction move into the same repository or unit of work in one change
5. State which legacy reads remain and why, when the change does not move all of them
### Done when
- Focused regression tests pin the preserved return values, errors and transaction behavior
- Response bodies, status codes and OpenAPI match the base
- Configuration sync and serialization behave the same, including invalid stored data
### Out of scope
- Switching the backend, even when an equivalent Rust query exists
- Converting other routes in the module just to establish the pattern
- Cleanups unrelated to the moved calls
## Porting a repository to SQLx
Status: TBD. No repository has been ported yet. `database.native` is a `NativeDatabase` from `crates/python-bridge/src/db.rs` whose `connect` and `close` are noops that open no pool, and the PyO3 repository surface does not exist. A real pool takes typed writer/reader settings and a token provider, never the Prisma `DATABASE_URL`, which carries Prisma-only params and rotating IAM tokens. The constraints below are settled, the steps are a sketch to revise after the first port lands
Use this only when a port is explicitly requested. Repository extraction does not require one
### When not to port
- An equivalent Rust or SQLx query existing is not a reason to switch a Prisma caller
- Leave existing backend selection unchanged unless integration is explicitly requested
- Never port part of a repository, it is either all Prisma or all native
### Before starting
- Identify the repository's inputs, outputs, errors and transaction boundary
- List repositories that share a transaction with it, they move together
- Check `litellm-rust/crates/db` for an equivalent query and reuse it when its behavior matches
### Steps
1. Rust queries, following `litellm-rust/crates/db/AGENTS.md`
- Static SQL in `queries/`, compile-checked with SQLx, typed inputs and row projections in `src/`
- Dynamic filters and sort keys become closed Rust enums that select static SQL, every value is bound
- Refresh SQLx query metadata after SQL changes
2. PyO3 surface exposing typed operations, never SQL text, table names or column names from Python. TBD: where it lives and how the handle reaches it
3. Native repository implementing the same Protocol in `litellm/repositories`
- Converts typed PyO3 results to domain types, preserving defaults, invalid-data handling and storage errors
- Python conversion stays in the bridge, not in `crates/db`
4. Switch the provider in `litellm/proxy/deps/repositories.py`, the Protocol and routes stay unchanged
```diff
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
- return None if database is None else UserBannerRepository(database.prisma)
+ return None if database is None else NativeUserBannerRepository(database.native)
```
5. Delete the Prisma repository in the same change and switch every caller, including jobs and helpers
### Done when
- Behavior is verified against real PostgreSQL, covering null handling, conflicts, ordering, errors and transaction semantics
- `DATABASE_URL=... cargo test -p litellm-db --locked -- --include-ignored` passes against a Prisma-migrated database
- Live proxy requests show unchanged responses. TBD: a Python-side real-PostgreSQL test harness
### Rollback
- Roll back the release, never retry an uncertain native write through Prisma
### Out of scope
- SQLx owning schema migrations, and removing Prisma
- Converting unrelated routes to dependency injection

4
.agents/skills/database-access/references/rust-axum.md Normal file
View file

@ -0,0 +1,4 @@
# Rust and Axum
- This should use same `crates/db` what is exposed to Python world.
- Details are TBD.

1
.github/workflows/test-unit.yml vendored
View file

@ -577,6 +577,7 @@ jobs:
tests/unit/proxy/rerank_endpoints
tests/unit/proxy/realtime_endpoints
tests/unit/proxy/ui_crud_endpoints
tests/unit/proxy/deps
tests/unit/proxy/config_resolvers
tests/unit/proxy/utils
python-version: "3.12"

2
AGENTS.md
View file

@ -90,6 +90,8 @@ Do not put names of customers or customer company names in code, PR descriptions
CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
For database operations, connection ownership and proxy dependency wiring, follow [database-access](.agents/skills/database-access/SKILL.md)
Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No `UPDATE`, `DELETE` or `MERGE`, and no `INSERT ... SELECT`: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. `tests/code_coverage_tests/check_migrations_no_data_rewrites.py` enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement `-- data-migration-ok: <what bounds it>`
Follow these coding conventions for new/updated code (a three-line fix in a legacy file shouldn't trigger huge drive-by refactors):

2
CONTRIBUTING.md
View file

@ -276,6 +276,8 @@ When the provider reports its own usage or spend numbers, compare the proxy's nu
Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution, and it is where PRs get reworked most often. Follow these rules
**Preserve zero prices.** In [`model_prices_and_context_window.json`](model_prices_and_context_window.json), an explicit `0.0` means that the priced operation is free. A missing field or `null` means no rate is specified, so the relevant calculator may use a fallback. For example, when no batch input rate is configured, batch cache writes use `cache_creation_input_token_cost` if set, otherwise the regular input rate, then apply the batch discount. A zero ordinary input rate does not make separately priced cache reads or writes free. Test `rate is None` before substituting a fallback, since `row.get("cache_creation_input_token_cost") or fallback` would discard a valid zero. Speech keeps character pricing ahead of token pricing even when the character rate is zero. If neither rate is configured, speech cost calculation raises an error. Its model lookup preserves unset rates instead of treating compatibility defaults as explicit zero prices. Perplexity keeps a configured legacy search rate ahead of the current search rate, including zero. When registering custom prices, `None` leaves an existing rate unchanged and `0.0` overrides it
**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. [`ARCHITECTURE.md`](ARCHITECTURE.md) shows where the layers live, and the decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) explain why they are shaped that way. For example, [the ADR on provider usage fields](docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md) covers how provider-specific usage fields reach cost tracking. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
**Bill a new provider usage field from the usage object.** Leave the field on the usage object as an extra and read it in that provider's `cost_calculator.py`. A provider without one gets a new `cost_calculator.py` and an arm for it in `cost_per_token` (`litellm/cost_calculator.py`), which picks the calculator by provider. If the provider's transform builds `Usage` from named fields, as the Anthropic, Gemini, and Bedrock Converse ones do, pass the new field there too. Streamed requests rebuild usage in `ChunkProcessor.calculate_usage` (`litellm/litellm_core_utils/streaming_chunk_builder_utils.py`), which does not copy extras yet the way the Responses-to-chat bridge does (minus a short exclusion list), so until it does, bill the field on streams by naming it there, as Anthropic's `inference_geo` is. A provider that reports its own charge for the request, as OpenRouter does in `usage.cost`, is billed at that charge from `_hidden_params` before any calculator runs, unless the deployment sets its own prices. When the field counts a built-in tool such as web search, add the provider to `get_cost_for_web_search_request` in `litellm/llms/__init__.py`, and add a gate to `litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py` only when no existing gate fires. Then show the field's cost in your proof on every endpoint it reaches, streaming included

31
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -10,6 +10,7 @@ from dataclasses import replace as dataclasses_replace
from datetime import datetime, timedelta
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, List, Literal, Optional, Protocol, Tuple, cast
from pydantic import TypeAdapter, ValidationError
from litellm_enterprise.proxy.common_utils.managed_object_attribution import (
ManagedObjectCreatorAttribution,
@ -50,6 +51,7 @@ def _error_names_resource(error: Exception, resource_id: str) -> bool:
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
RECENT_BATCH_POLL_WINDOW: Final = timedelta(hours=48)
GIVE_UP_POLLING_AFTER: Final = timedelta(days=2 * MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
_FILE_OBJECT_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
PROVIDER_TERMINAL_BATCH_STATUSES: Final[Tuple[str, ...]] = (
@ -550,6 +552,10 @@ class CheckBatchCost:
llm_provider: str,
bare_model_name: str,
) -> Optional[Tuple[str, str]]:
stored_model_id: Final = self._get_stored_model_id(job)
if stored_model_id is not None and self._batch_deployment_exists(stored_model_id):
return stored_model_id, job.unified_object_id
deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
if deployment_id is None:
verbose_proxy_logger.info(
@ -662,23 +668,36 @@ class CheckBatchCost:
)
@staticmethod
def _get_input_file_id(job: "ManagedObjectRow") -> Optional[str]:
from litellm.types.utils import LiteLLMBatch
def _parse_file_object(job: "ManagedObjectRow") -> Optional[dict[str, object]]:
file_object: Final = job.file_object
try:
parsed_file_object: Final[object] = (
return _FILE_OBJECT_ADAPTER.validate_python(
json.loads(file_object) if isinstance(file_object, str) else file_object
)
except (json.JSONDecodeError, ValueError):
except (json.JSONDecodeError, ValueError, ValidationError):
return None
if not isinstance(parsed_file_object, dict):
@classmethod
def _get_input_file_id(cls, job: "ManagedObjectRow") -> Optional[str]:
from litellm.types.utils import LiteLLMBatch
parsed_file_object: Final = cls._parse_file_object(job)
if parsed_file_object is None:
return None
try:
return LiteLLMBatch.model_validate(parsed_file_object).input_file_id
except Exception:
return None
@classmethod
def _get_stored_model_id(cls, job: "ManagedObjectRow") -> Optional[str]:
"""The deployment that created the batch, written into file_object by the managed
files hook at create time. Older rows have no such key and fall back to matching
the bare model name against the router."""
parsed_file_object: Final = cls._parse_file_object(job)
model_id: Final = parsed_file_object.get("model_id") if parsed_file_object else None
return model_id if isinstance(model_id, str) and model_id else None
async def _track_completed_batch_cost(
self,
job: "ManagedObjectRow",

50
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -21,6 +21,7 @@ from typing import (
TypedDict,
Union,
cast,
runtime_checkable,
)
from uuid import NAMESPACE_URL, uuid5
@ -374,6 +375,12 @@ class _ManagedResourcePrismaClient(Protocol):
def db(self) -> _ManagedResourceDatabase: ...
@runtime_checkable
class _WriterRoutedDatabase(Protocol):
@property
def writer(self) -> _ManagedResourceDatabase: ...
class _SchedulerWithJobLookup(Protocol):
def get_job(self, job_id: str) -> object: ...
@ -479,6 +486,11 @@ def _managed_object_table(prisma_client: _ManagedResourcePrismaClient) -> _Manag
return prisma_client.db.litellm_managedobjecttable
def _writer_database(db: _ManagedResourceDatabase) -> _ManagedResourceDatabase:
"""Reads that feed a write must see the create that just happened; a lagging read replica may not"""
return db.writer if isinstance(db, _WriterRoutedDatabase) else db
_JSON_VALUE_ADAPTER: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
_POSTGRES_UNSTORABLE_CHARACTER: Final = "\x00"
@ -512,6 +524,25 @@ def _without_model_output(
)
def _with_model_id(
file_object: LiteLLMBatch | LiteLLMFineTuningJob | ResponsesAPIResponse,
model_id: str | None,
) -> LiteLLMBatch | LiteLLMFineTuningJob | ResponsesAPIResponse:
return file_object if model_id is None else file_object.model_copy(update={"model_id": model_id})
async def _stored_model_id(managed_object_table: _ManagedObjectTableActions, unified_object_id: str) -> str | None:
row: Final = await managed_object_table.find_first(where={"unified_object_id": unified_object_id})
if row is None:
return None
try:
parsed: Final = _JSON_VALUE_ADAPTER.validate_python(_decode_json_blob(row.file_object))
except (json.JSONDecodeError, ValidationError):
return None
model_id: Final = parsed.get("model_id") if isinstance(parsed, dict) else None
return model_id if isinstance(model_id, str) and model_id else None
def _storage_metadata_of(file_object: OpenAIFileObject | None) -> Mapping[str, str]:
hidden_params: Final = cast( # cast-ok: _hidden_params is an untyped attribute the upload path sets
"Mapping[str, object]", getattr(file_object, "_hidden_params", None) or {}
@ -697,9 +728,17 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
persist_attribution: bool = False,
create_if_missing: bool = True,
batch_processed: bool = False,
creating_model_id: str | None = None,
) -> None:
"""Persist a managed object row, caching it and upserting it in the DB.
creating_model_id is the router deployment that created the object. It is kept
inside the stored file_object JSON as `model_id`, because the raw provider id an
unmanaged batch stores as unified_object_id says nothing about which deployment
made it, and the cost poller needs that deployment to read the result back. A
later observe of the row (create_if_missing=False) keeps the stored value rather
than overwriting it with whichever deployment served the retrieve.
persist_attribution is set only by the batch and background response creates,
the callers that can speak for the creator; it gates the api_key, request_tags, and,
for a response, end_user_id columns that the cost poll jobs bill against, so a later poll or
@ -720,7 +759,14 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
It is written only in the upsert create branch.
"""
verbose_logger.info(f"Storing LiteLLM Managed {file_purpose} object with id={unified_object_id} in cache")
stored_object: Final = _without_model_output(file_object)
managed_database: Final = _managed_database(self.prisma_client)
managed_object_table: Final = managed_database.litellm_managedobjecttable
preserved_model_id: Final = (
creating_model_id
if creating_model_id is not None or create_if_missing
else await _stored_model_id(_writer_database(managed_database).litellm_managedobjecttable, unified_object_id)
)
stored_object: Final = _with_model_id(_without_model_output(file_object), preserved_model_id)
litellm_managed_object = LiteLLM_ManagedObjectTable(
unified_object_id=unified_object_id,
model_object_id=model_object_id,
@ -752,7 +798,6 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"status": file_object.status,
"updated_by": user_api_key_dict.user_id,
}
managed_object_table: Final = _managed_object_table(self.prisma_client)
if not create_if_missing:
await managed_object_table.update_many(
where={"unified_object_id": unified_object_id},
@ -2170,6 +2215,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
request_tags=request_tags_from_metadata(request_metadata if isinstance(request_metadata, dict) else {}),
persist_attribution=is_batch_create,
create_if_missing=is_batch_create,
creating_model_id=model_id if is_batch_create else None,
)
if not is_batch_create:
await self.store_batch_output_file_ownership(

19
litellm-rust/crates/python-bridge/src/db.rs Normal file
View file

@ -0,0 +1,19 @@
use litellm_host_python::ready_future;
use pyo3::{prelude::*, types::PyNone};
#[pyclass(frozen)]
pub(crate) struct NativeDatabase {}
// TODO: take typed writer/reader settings and a token provider, then open a sqlx PgPool, once the first
// repository moves to native. The Prisma DATABASE_URL carries Prisma-only params and rotating IAM tokens
#[pymethods]
impl NativeDatabase {
#[staticmethod]
fn connect(py: Python<'_>) -> PyResult<Bound<'_, PyAny>> {
ready_future(py, Bound::new(py, NativeDatabase {})?.as_any())
}
fn close<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
ready_future(py, PyNone::get(py).as_any())
}
}

4
litellm-rust/crates/python-bridge/src/lib.rs
View file

@ -2,6 +2,7 @@ mod cache;
mod callable;
mod coercion;
mod credentials;
mod db;
mod diagnostics;
mod errors;
mod execution;
@ -21,6 +22,8 @@ mod _native {
use litellm_host_python::{ForkedAfterNativeRuntimeStarted, ProcessReservedForForking};
use pyo3::{prelude::*, types::PyModule};
#[pymodule_export]
use crate::db::NativeDatabase;
#[cfg(feature = "panic-test")]
#[pymodule_export]
use crate::diagnostics::_panic_for_test;
@ -112,6 +115,7 @@ mod tests {
"NativeDiagnosticProcessor",
"NativeClickHouseSpendConfig",
"NativeClickHouseSpendStorage",
"NativeDatabase",
"trace_encode_error",
"bundled_model_cost_map",
"TokenCounter",

4
litellm/_lazy_imports_registry.py
View file

@ -1471,11 +1471,11 @@ _UTILS_MODULE_IMPORT_MAP: Final = {
"AsyncHTTPHandler": ("litellm.llms.custom_httpx.http_handler", "AsyncHTTPHandler"),
"HTTPHandler": ("litellm.llms.custom_httpx.http_handler", "HTTPHandler"),
"get_num_retries_from_retry_policy": (
"litellm.router_utils.get_retry_from_policy",
"litellm.litellm_core_utils.get_retry_from_policy",
"get_num_retries_from_retry_policy",
),
"reset_retry_policy": (
"litellm.router_utils.get_retry_from_policy",
"litellm.litellm_core_utils.get_retry_from_policy",
"reset_retry_policy",
),
"get_coroutine_checker": (

1
litellm/constants.py
View file

@ -2046,6 +2046,7 @@ LITELLM_SETTINGS_SAFE_DB_OVERRIDES: Final = [
SPECIAL_LITELLM_AUTH_TOKEN: Final = ["ui-token"]
DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL = get_env_int("DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL", 60)
AUTH_OBJECTS_TARGET: Final = "auth_objects"
HEALTH_CHECKS_TARGET: Final = "health_checks"
DEFAULT_ACCESS_GROUP_CACHE_TTL: Final = get_env_int("DEFAULT_ACCESS_GROUP_CACHE_TTL", 600)
SPEND_LOG_KEY_METADATA_CACHE_TTL: Final = 600
SPEND_LOG_KEY_METADATA_MISS_CACHE_TTL: Final = 30

13
litellm/cost_calculator.py
View file

@ -623,7 +623,11 @@ def cost_per_token(
)
if lyria_generation_cost is not None:
return 0.0, lyria_generation_cost
speech_model_info = litellm.get_model_info(model=model_without_prefix, custom_llm_provider=custom_llm_provider)
speech_model_info: Final = cached_get_model_info_helper(
model=model_without_prefix,
custom_llm_provider=custom_llm_provider,
preserve_unset_costs=True,
)
cost_metric: Final = select_cost_metric_for_model(speech_model_info)
prompt_cost: float = 0.0
completion_cost: float = 0.0
@ -3084,7 +3088,7 @@ def batch_cost_calculator(
batch_rates.cache_creation,
batch_rates.cache_creation_rates_by_ttl,
)
elif input_cost_per_token:
elif input_cost_per_token is not None:
details: Final = parse_prompt_tokens_details(usage)
cache_read_tokens: Final = details["cache_hit_tokens"]
cache_creation_tokens: Final = details["cache_creation_tokens"]
@ -3100,7 +3104,10 @@ def batch_cost_calculator(
cache_read_cost_key: Final = get_service_tier_cost_key("cache_read_input_token_cost", None)
total_prompt_cost += calculate_cost_component(model_info, cache_read_cost_key, cache_read_tokens) / 2
cache_creation_cost: Final = model_info.get("cache_creation_input_token_cost") or input_cost_per_token
configured_cache_creation_cost: Final = model_info.get("cache_creation_input_token_cost")
cache_creation_cost: Final = (
input_cost_per_token if configured_cache_creation_cost is None else configured_cache_creation_cost
)
total_prompt_cost += cache_creation_tokens * cache_creation_cost / 2
text_rate: Final = _batch_or_half(batch_rates.output, output_cost_per_token)
image_rate: Final = _batch_or_half(

115
litellm/integrations/custom_guardrail.py
View file

@ -71,6 +71,8 @@ GUARDRAIL_SESSIONS_TARGET: Final = "guardrail_sessions"
# field to suppress a guardrail on the direct-SDK path that never reaches the
# proxy's metadata sanitizer.
_PRE_CALL_EXECUTED_TOKEN: Final = secrets.token_hex(16)
_PRE_CALL_RAISED_TOKEN: Final = secrets.token_hex(16)
_DEPLOYMENT_HOOK_RAN_TOKEN: Final = secrets.token_hex(16)
_GUARDRAIL_BLOCK_STATUS_CODES: Final = frozenset({400, 403, 422})
@ -260,6 +262,25 @@ def _request_is_streaming(data: object, event_type: GuardrailEventHooks | None =
)
def _str_collection(value: object) -> frozenset[str]:
if not isinstance(value, (list, tuple, set, frozenset)):
return frozenset()
items: Final = cast("Iterable[object]", value) # cast-ok: a metadata collection of guardrail names
return frozenset(item for item in items if isinstance(item, str))
def _request_carries_marker(data: Mapping[str, object], marker: str | None) -> bool:
if marker is None:
return False
for meta_key in ("metadata", "litellm_metadata"):
meta = data.get(meta_key)
if isinstance(meta, dict):
executed = meta.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if isinstance(executed, list) and marker in executed:
return True
return False
_REQUEST_FIELDS: Final = TypeAdapter(Mapping[str, object])
@ -869,13 +890,16 @@ class CustomGuardrail(CustomLogger):
return False
def _pre_call_marker(self) -> str | None:
def _marker(self, token: str) -> str | None:
name: Final = self.guardrail_name
if not name:
return None
return f"{_PRE_CALL_EXECUTED_TOKEN}:{name}"
return f"{token}:{name}"
def mark_pre_call_hook_ran(self, data: dict[str, object]) -> None:
def _pre_call_marker(self, *, raised: bool = False) -> str | None:
return self._marker(_PRE_CALL_RAISED_TOKEN if raised else _PRE_CALL_EXECUTED_TOKEN)
def mark_pre_call_hook_ran(self, data: dict[str, object], *, raised: bool = False) -> None:
"""
Record that this guardrail's ``async_pre_call_hook`` already ran for this
request, so the deployment-level hook does not run it a second time.
@ -884,8 +908,11 @@ class CustomGuardrail(CustomLogger):
router later spreads a deployment's model-level ``guardrails`` into the
top-level request kwargs, which would otherwise re-trigger the same hook
from ``async_pre_call_deployment_hook``.
With ``raised``, records instead that the hook raised as a policy pipeline
step the pipeline then moved past. The deployment-level hook still runs it.
"""
marker: Final = self._pre_call_marker()
marker: Final = self._pre_call_marker(raised=raised)
if marker is None:
return
for meta_key in ("metadata", "litellm_metadata"):
@ -900,17 +927,57 @@ class CustomGuardrail(CustomLogger):
return
data["metadata"] = {PRE_CALL_EXECUTED_GUARDRAILS_KEY: [marker]}
def _pre_call_hook_already_ran(self, data: dict[str, object]) -> bool:
marker: Final = self._pre_call_marker()
if marker is None:
return False
for meta_key in ("metadata", "litellm_metadata"):
meta = data.get(meta_key)
if isinstance(meta, dict):
executed = meta.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if isinstance(executed, list) and marker in executed:
return True
return False
def pre_call_hook_ran_or_raised(self, data: Mapping[str, object]) -> bool:
"""Whether this guardrail's ``async_pre_call_hook`` ran for this request, whatever it decided."""
return self._pre_call_hook_already_ran(data) or self._pre_call_hook_already_ran(data, raised=True)
def _pre_call_hook_already_ran(self, data: Mapping[str, object], *, raised: bool = False) -> bool:
return _request_carries_marker(data, self._pre_call_marker(raised=raised))
def deployment_hook_ran(self, data: Mapping[str, object]) -> bool:
"""Whether a deployment-level hook ran this guardrail as a policy pipeline step of this request."""
return _request_carries_marker(data, self._marker(_DEPLOYMENT_HOOK_RAN_TOKEN))
def _record_deployment_hook_run(
self,
data: dict[str, object], # mutable-ok: the request payload the run is recorded on
) -> None:
"""
Record that a deployment-level hook ran this guardrail when it is a policy pipeline step of
this request, and name it in the applied guardrails when the request's metadata lists it.
"""
name: Final = self.guardrail_name
marker: Final = self._marker(_DEPLOYMENT_HOOK_RAN_TOKEN)
buckets: Final = tuple(
cast("dict[str, object]", bucket) # cast-ok: metadata buckets are str-keyed
for bucket in (data.get("metadata"), data.get("litellm_metadata"))
if isinstance(bucket, dict)
)
policy_state: Final = next(
(bucket for bucket in buckets if name in _str_collection(bucket.get("_pipeline_managed_guardrails"))),
None,
)
if marker is None or policy_state is None:
return
recorded: Final = policy_state.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if not isinstance(recorded, list):
policy_state[PRE_CALL_EXECUTED_GUARDRAILS_KEY] = [marker]
elif marker not in recorded:
cast("list[object]", recorded).append(marker) # cast-ok: markers are only compared as strings
if not any(name in _str_collection(bucket.get("guardrails")) for bucket in buckets):
return
named_lists: Final = tuple(
cast("list[object]", bucket["applied_guardrails"]) # cast-ok: entries are only compared with names
for bucket in buckets
if isinstance(bucket.get("applied_guardrails"), list)
)
for applied in named_lists:
if name not in applied:
applied.append(name)
if named_lists:
return
metadata_bucket: Final = cast("dict[str, object]", get_or_create_metadata_bucket(data)[1]) # cast-ok: str keys
metadata_bucket["applied_guardrails"] = [name]
@classmethod
def uses_apply_guardrail_interface(cls) -> bool:
@ -959,12 +1026,15 @@ class CustomGuardrail(CustomLogger):
target: Final = self._deployment_hook_target()
if target is not self:
kwargs["guardrail_to_apply"] = self
result: Final = await target.async_pre_call_hook(
user_api_key_dict=_user_api_key_auth_from_request(kwargs),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
try:
result: Final = await target.async_pre_call_hook(
user_api_key_dict=_user_api_key_auth_from_request(kwargs),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
finally:
self._record_deployment_hook_run(kwargs)
if result is not None and isinstance(result, dict):
result_messages: Final = result.get("messages")
@ -1048,6 +1118,9 @@ class CustomGuardrail(CustomLogger):
finally:
if target is not self:
request_data.pop("guardrail_to_apply", None)
self._record_deployment_hook_run(
cast("dict[str, object]", request_data) # cast-ok: proxy request payloads are str-keyed
)
if not self._is_valid_response_type(result):
return None

8
litellm/integrations/prometheus.py
View file

@ -66,7 +66,7 @@ from litellm.repositories.organization_repository import OrganizationRepository
from litellm.repositories.table_repositories import EndUserRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
from litellm.router_api import deployment_metadata
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import (
@ -3022,8 +3022,8 @@ class PrometheusLogger(CustomLogger):
).get("custom_llm_provider")
if isinstance(provider, str) and provider:
return provider
deployment_metadata: Final = find_deployment_metadata(litellm_params)
return _deployment_model_provider(deployment_metadata.get("deployment") if deployment_metadata else None)
picked_deployment: Final = deployment_metadata(litellm_params)
return _deployment_model_provider(picked_deployment.get("deployment") if picked_deployment else None)
@staticmethod
def _extract_api_provider_from_exception(exception: Exception) -> str | None:
@ -5209,7 +5209,7 @@ def _tag_matches_wildcard_configured_pattern(tags: Sequence[str], configured_tag
"""
import re
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
pattern_router: Final = PatternMatchRouter()
regex_pattern: Final = pattern_router.pattern_to_regex(configured_tag)

0
litellm/router_utils/access_windows.py → litellm/litellm_core_utils/access_windows.py
View file

0
litellm/router_utils/add_retry_fallback_headers.py → litellm/litellm_core_utils/add_retry_fallback_headers.py
View file

0
litellm/router_utils/clientside_credential_handler.py → litellm/litellm_core_utils/clientside_credential_handler.py
View file

0
litellm/router_utils/deployment_pricing.py → litellm/litellm_core_utils/deployment_pricing.py
View file

6
litellm/litellm_core_utils/fallback_utils.py
View file

@ -3,13 +3,13 @@ from typing import Final
import litellm
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
)
from litellm.litellm_core_utils.core_helpers import (
filter_internal_params,
safe_deep_copy,
)
from litellm.router_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
)
from .asyncify import run_async_function

0
litellm/router_utils/get_retry_from_policy.py → litellm/litellm_core_utils/get_retry_from_policy.py
View file

2
litellm/litellm_core_utils/health_check_helpers.py
View file

@ -119,7 +119,7 @@ def _is_route_segment(custom_llm_provider: str, segment: str, model_after_segmen
def wildcard_health_check_models(wildcard: str, custom_llm_provider: str) -> tuple[str, ...]:
from litellm.litellm_core_utils.llm_request_utils import rank_chat_models_from_llm_provider
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
provider_prefix: Final = f"{custom_llm_provider}/"
segments: Final = wildcard.removeprefix(provider_prefix).split("/")

7
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -28,6 +28,7 @@ from litellm.types.utils import (
DataResidency,
ImageResponse,
ModelInfo,
ModelInfoBase,
PassthroughCallTypes,
PromptTokensDetailsWrapper,
ServiceTier,
@ -197,15 +198,15 @@ def get_billable_input_tokens(usage: Usage) -> int:
def select_cost_metric_for_model(
model_info: ModelInfo,
model_info: ModelInfoBase,
) -> Literal["cost_per_character", "cost_per_token"]:
"""
Select 'cost_per_character' if model_info has 'input_cost_per_character'
Select 'cost_per_token' if model_info has 'input_cost_per_token'
"""
if model_info.get("input_cost_per_character"):
if model_info.get("input_cost_per_character") is not None:
return "cost_per_character"
elif model_info.get("input_cost_per_token"):
elif model_info.get("input_cost_per_token") is not None:
return "cost_per_token"
else:
raise ValueError(

0
litellm/router_utils/pattern_match_deployments.py → litellm/litellm_core_utils/pattern_match_deployments.py
View file

2
litellm/router_utils/ptu_shares.py → litellm/litellm_core_utils/ptu_shares.py
View file

@ -12,13 +12,13 @@ from datetime import date
from typing import Final, Generic, TypeAlias, TypeVar
from litellm.litellm_core_utils.ptu_pricing import parsed_ptu_shares, ptu_terms
from litellm.litellm_core_utils.routing_common_utils import get_request_team_id, team_may_use_deployment
from litellm.llms.azure.ptu_capacity import (
PTUCapacity,
deployment_ptu_capacity,
is_azure_deployment,
is_str_keyed_mapping,
)
from litellm.router_utils.common_utils import get_request_team_id, team_may_use_deployment
_DeploymentT: Final = TypeVar("_DeploymentT", bound=Mapping[str, object])
WildcardRoute: TypeAlias = Callable[[str], Sequence[Mapping[str, object]]]

0
litellm/router_utils/reasoning_effort_capability.py → litellm/litellm_core_utils/reasoning_effort_capability.py
View file

0
litellm/router_utils/common_utils.py → litellm/litellm_core_utils/routing_common_utils.py
View file

0
litellm/router_utils/routing_groups.py → litellm/litellm_core_utils/routing_groups.py
View file

2
litellm/litellm_core_utils/token_counter.py
View file

@ -56,9 +56,9 @@ from litellm.litellm_core_utils.tokenizer import HuggingFaceTokenizer, OpenAIEnc
if TYPE_CHECKING:
from litellm.litellm_core_utils.tokenizer import Encoding, HuggingFace
from litellm.litellm_core_utils.deployment_pricing import deployment_pricing_models
from litellm.litellm_core_utils.url_utils import safe_get
from litellm.llms.custom_httpx.http_handler import get_httpx_client, header_value
from litellm.router_utils.deployment_pricing import deployment_pricing_models
from litellm.rust_bridge.tokenizer import get_encoding
from litellm.types.llms.anthropic import (
AnthropicContentParamSource,

14
litellm/llms/anthropic/messages/native/handler.py
View file

@ -19,6 +19,13 @@ from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheCont
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.exception_mapping_utils import exception_type
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.llms.anthropic.common_utils import (
AnthropicModelInfo,
strip_provider_specific_fields_from_anthropic_messages,
@ -31,13 +38,6 @@ from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.router_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.types.llms.anthropic import AllAnthropicToolsValues
from litellm.types.llms.anthropic_messages.anthropic_request import AnthropicMetadata
from litellm.types.llms.anthropic_messages.anthropic_response import (

2
litellm/llms/anthropic/messages/native/interceptors/advisor.py
View file

@ -22,7 +22,7 @@ import litellm
import litellm.constants as _c
from litellm.litellm_core_utils.url_utils import validate_url
from litellm.llms.anthropic.common_utils import strip_advisor_blocks_from_messages
from litellm.router_utils.cooldown_handlers import mark_advisor_orchestration_failure
from litellm.router_api import mark_advisor_orchestration_failure
from litellm.types.llms.anthropic import ANTHROPIC_ADVISOR_TOOL_TYPE
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,

2
litellm/llms/anthropic/messages/utils.py
View file

@ -214,7 +214,7 @@ def normalize_reasoning_effort_value(
if chain is None:
return effort
from litellm.router_utils.reasoning_effort_capability import (
from litellm.litellm_core_utils.reasoning_effort_capability import (
deployment_rounds_reasoning_effort,
resolve_supported_reasoning_efforts,
)

2
litellm/llms/clf_ai_gateway/chat/transformation.py
View file

@ -4,8 +4,8 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import UnsupportedParamsError
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.secret_managers.main import get_secret_str
SUPPORTED_OPENAI_PARAMS: Final = (

8
litellm/llms/compaction.py
View file

@ -9,7 +9,7 @@ from pydantic import TypeAdapter
from litellm.types.llms.openai import AllMessageValues
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
CompactionProtocol: TypeAlias = Literal["chat", "messages"]
_MAPPING: Final = TypeAdapter(Mapping[str, object])
@ -32,7 +32,9 @@ def get_native_compaction_provider(params: Mapping[str, object]) -> NativeCompac
return compaction if compaction.supports_native_compaction(params) else None
async def dispatch(router: Router, protocol: CompactionProtocol, payload: Mapping[str, object]) -> Mapping[str, object]:
async def dispatch(
router: RoutingContext, protocol: CompactionProtocol, payload: Mapping[str, object]
) -> Mapping[str, object]:
if protocol == "messages":
return _MAPPING.validate_python(
await router.aanthropic_messages(custom_llm_provider=None, client=None, **payload)
@ -45,4 +47,4 @@ async def dispatch(router: Router, protocol: CompactionProtocol, payload: Mappin
{key: value for key, value in payload.items() if key not in ("model", "messages", "stream")}
),
)
return _MAPPING.validate_python(response.model_dump())
return _MAPPING.validate_python(response.model_dump()) # pyright: ignore[reportAttributeAccessIssue] # stream=False: acompletion returns a ModelResponse

8
litellm/llms/mistral/chat/transformation.py
View file

@ -25,16 +25,16 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
strip_litellm_internal_message_fields,
strip_none_values_from_message,
)
from litellm.litellm_core_utils.reasoning_effort_capability import (
declared_reasoning_efforts_for_model,
nearest_declared_reasoning_effort,
)
from litellm.litellm_core_utils.reasoning_effort_utils import OPAQUE_VALUE, warn_dropped_reasoning
from litellm.litellm_core_utils.url_utils import ensure_api_route
from litellm.llms.openai.chat.gpt_transformation import (
OpenAIChatCompletionStreamingHandler,
OpenAIGPTConfig,
)
from litellm.router_utils.reasoning_effort_capability import (
declared_reasoning_efforts_for_model,
nearest_declared_reasoning_effort,
)
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.mistral import (
MistralCacheKeyParams,

2
litellm/llms/oci/chat/transformation.py
View file

@ -21,6 +21,7 @@ from litellm._logging import verbose_logger
from litellm.constants import DEFAULT_OCI_CHAT_MAX_TOKENS
from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
from litellm.litellm_core_utils.prompt_templates.common_utils import messages_with_file_parts_as_image_urls
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.litellm_core_utils.reasoning_effort_utils import warn_dropped_reasoning
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.litellm_core_utils.url_utils import join_api_base
@ -54,7 +55,6 @@ from litellm.llms.oci.common_utils import (
sign_oci_request,
validate_oci_environment,
)
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.types.llms.oci import (
CohereChatRequest,
OCIChatRequestPayload,

5
litellm/llms/perplexity/cost_calculator.py
View file

@ -100,8 +100,9 @@ def cost_per_token(model: str, usage: Usage, current_time: datetime | None = Non
num_search_queries = getattr(usage.prompt_tokens_details, "web_search_requests", 0) or 0
# Check both possible keys for search cost (legacy and current)
search_cost_value: Final = model_info.get("search_queries_cost_per_query") or model_info.get(
"search_context_cost_per_query"
legacy_search_cost: Final = model_info.get("search_queries_cost_per_query")
search_cost_value: Final = (
model_info.get("search_context_cost_per_query") if legacy_search_cost is None else legacy_search_cost
)
if num_search_queries > 0 and search_cost_value is not None:
# Handle both dict and float formats

34
litellm/llms/perplexity/search/transformation.py
View file

@ -84,9 +84,9 @@ class PerplexitySearchConfig(BaseSearchConfig):
def transform_search_request(
self,
query: str | list[str],
optional_params: dict,
**kwargs,
) -> dict:
optional_params: dict[str, object],
**kwargs: object,
) -> dict[str, object]:
"""
Transform Search request to Perplexity API format.
@ -98,11 +98,7 @@ class PerplexitySearchConfig(BaseSearchConfig):
Args:
query: Search query (string or list of strings)
optional_params: Optional parameters for the request
- max_results: Maximum number of search results (1-20)
- search_domain_filter: List of domains to filter (max 20)
- max_tokens_per_page: Max tokens per page (default 1024)
- country: Country code filter (e.g., 'US', 'GB', 'DE')
optional_params: Provider parameters, forwarded unchanged when not None.
Returns:
Dict with typed request data following PerplexitySearchRequest spec
@ -111,24 +107,10 @@ class PerplexitySearchConfig(BaseSearchConfig):
"query": query,
}
# Add optional parameters following Perplexity API spec (only if not None)
max_results: Final = optional_params.get("max_results")
if max_results is not None:
request_data["max_results"] = max_results
search_domain_filter: Final = optional_params.get("search_domain_filter")
if search_domain_filter is not None:
request_data["search_domain_filter"] = search_domain_filter
max_tokens_per_page: Final = optional_params.get("max_tokens_per_page")
if max_tokens_per_page is not None:
request_data["max_tokens_per_page"] = max_tokens_per_page
country: Final = optional_params.get("country")
if country is not None:
request_data["country"] = country
return dict(request_data)
return {
**{key: value for key, value in optional_params.items() if value is not None},
**request_data,
}
def transform_search_response(
self,

2
litellm/llms/together_ai/chat/transformation.py
View file

@ -18,7 +18,7 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import UnsupportedParamsError
from litellm.litellm_core_utils.prompt_templates.common_utils import strip_litellm_response_only_fields
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.types.llms.openai import AllMessageValues
from litellm.utils import supports_function_calling, supports_reasoning, supports_response_schema

10
litellm/main.py
View file

@ -113,6 +113,11 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
messages_without_pdf_file_parts,
)
from litellm.litellm_core_utils.provider_affinity import add_provider_affinity_header
from litellm.litellm_core_utils.reasoning_effort_capability import (
model_group_from_metadata,
reasoning_effort_needs_allowing,
resolve_chat_reasoning,
)
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
)
@ -144,11 +149,6 @@ from litellm.llms.vertex_ai.common_utils import (
)
from litellm.realtime_api.main import realtime_health_check
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.router_utils.reasoning_effort_capability import (
model_group_from_metadata,
reasoning_effort_needs_allowing,
resolve_chat_reasoning,
)
from litellm.secret_managers.main import get_secret_bool, get_secret_str
from litellm.text_completions.streaming_iterator import TextCompletionStreamWrapper
from litellm.types.completion import (

26
litellm/model_prices_and_context_window_backup.json
View file

@ -23078,6 +23078,24 @@
"output_cost_per_token_batches": 1e-06,
"source": "https://developers.openai.com/api/docs/pricing"
},
"deepgram/aura": {
"input_cost_per_character": 1.5e-05,
"litellm_provider": "deepgram",
"metadata": {
"calculation": "$0.0150/1000 characters = $0.000015 per character",
"note": "Aura-1 text-to-speech voices (aura-<voice>-<lang>), pay as you go"
},
"source": "https://deepgram.com/pricing"
},
"deepgram/aura-2": {
"input_cost_per_character": 3e-05,
"litellm_provider": "deepgram",
"metadata": {
"calculation": "$0.030/1000 characters = $0.00003 per character",
"note": "Aura-2 text-to-speech voices (aura-2-<voice>-<lang>), pay as you go"
},
"source": "https://deepgram.com/pricing"
},
"deepgram/base": {
"input_cost_per_second": 0.00020833,
"litellm_provider": "deepgram",
@ -31224,6 +31242,9 @@
"output_cost_per_image_token": 3e-05,
"output_cost_per_reasoning_token": 2.5e-06,
"output_cost_per_token": 2.5e-06,
"output_cost_per_token_batches": 1.25e-06,
"output_cost_per_token_flex": 1.25e-06,
"output_cost_per_token_priority": 4.5e-06,
"rpm": 100000,
"source": "https://ai.google.dev/gemini-api/docs/pricing",
"supported_endpoints": [
@ -31404,6 +31425,7 @@
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_image_token_batches": 1.5e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"rpm": 1000,
@ -59744,12 +59766,13 @@
"cache_read_input_token_cost": 1.375e-06,
"output_cost_per_token": 8.25e-05,
"litellm_provider": "bedrock_mantle",
"max_input_tokens": 1050000,
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"mode": "responses",
"source": "https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-56-cyber.html",
"use_openai_responses_path": true,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
],
"supported_modalities": [
@ -75438,6 +75461,7 @@
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_image_token_batches": 1.5e-05,
"output_cost_per_token": 7.5e-06,
"output_cost_per_token_batches": 3.75e-06,
"rpm": 1000.0,

24
litellm/models/deployment_warning.py Normal file
View file

@ -0,0 +1,24 @@
from collections.abc import Mapping
from typing import Literal, Protocol
from pydantic import AwareDatetime, ConfigDict, Field
from litellm.types.llms.base import LiteLLMBaseModel
class AdminIgnore(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
source: Literal["admin"] = "admin"
ignored_by: str
ignored_at: AwareDatetime
ignored_until: AwareDatetime | None = Field(
default=None,
description="When the snooze ends. Null means the warning is ignored until an admin restores it.",
)
class DeploymentWarningIgnoreStore(Protocol):
async def get_ignores(self) -> Mapping[str, AdminIgnore]: ...
async def save_ignores(self, ignores: Mapping[str, AdminIgnore]) -> None: ...

31
litellm/models/gateway_activity.py Normal file
View file

@ -0,0 +1,31 @@
from typing import Literal, Protocol, TypeAlias
from pydantic import ConfigDict
from litellm.types.llms.base import LiteLLMBaseModel
class GatewayRouteActivity(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
kind: Literal["route"]
date: str
category: str
route: str
successful_requests: int
failed_requests: int
class GatewayStatusCodeActivity(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
kind: Literal["status"]
status_code: int
failed_requests: int
GatewayActivity: TypeAlias = GatewayRouteActivity | GatewayStatusCodeActivity
class GatewayActivityStore(Protocol):
async def get_activity(self, start_date: str, end_date: str) -> tuple[GatewayActivity, ...]: ...

48
litellm/models/user_banner.py Normal file
View file

@ -0,0 +1,48 @@
from typing import Final, Literal, Protocol
from pydantic import ConfigDict, Field, model_validator
from litellm.types.llms.base import LiteLLMBaseModel
USER_BANNER_MAX_MESSAGE_LENGTH: Final = 4000
UserBannerSeverity = Literal["info", "warning", "error"]
class UserBannerUpdate(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
enabled: bool = Field(
default=False,
description="If true, the banner is shown to all authenticated dashboard users.",
)
message: str = Field(
default="",
max_length=USER_BANNER_MAX_MESSAGE_LENGTH,
description="Banner text shown to dashboard users. Markdown is supported.",
)
severity: UserBannerSeverity = Field(
default="info",
description="Visual style of the banner.",
)
@model_validator(mode="after")
def _require_message_when_enabled(self) -> "UserBannerUpdate":
if self.enabled and not self.message.strip():
raise ValueError("message must be non-empty when the banner is enabled")
return self
class UserBanner(UserBannerUpdate):
revision: str = Field(
default="",
description=(
"Server-stamped opaque publish identity; a fresh value is generated on every "
"update so clients re-surface dismissed banners on republish."
),
)
class UserBannerStore(Protocol):
async def get_banner(self) -> UserBanner: ...
async def save_banner(self, banner: UserBanner) -> None: ...

2
litellm/proxy/_types.py
View file

@ -124,6 +124,7 @@ from .types_utils.utils import get_instance_fn, validate_custom_validate_return_
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
from litellm.repositories.database import Database
from litellm.tracing import TraceReceiver
Span = _Span | Any
@ -133,6 +134,7 @@ else:
class ProxyLifespanState(TypedDict):
tracing_receiver: ReadOnly["TraceReceiver | None"]
database: ReadOnly["Database | None"]
class ReconcileOutcome(NamedTuple):

37
litellm/proxy/auth/auth_checks.py
View file

@ -167,6 +167,7 @@ from litellm.repositories.table_repositories import (
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.router import Router
from litellm.router_api import RegistryReader
from litellm.types.proxy.auth.auth_checks import UserNotFoundError
from litellm.types.proxy.model_access_group_budget import ModelAccessGroupBudget
from litellm.types.router import DeploymentTypedDict
@ -525,7 +526,7 @@ def _log_budget_lookup_failure(entity: str, error: Exception) -> None:
)
def _get_router_zero_cost_cache(llm_router: Router) -> dict[str, bool] | None:
def _get_router_zero_cost_cache(llm_router: RegistryReader) -> dict[str, bool] | None:
"""
Return the router's per-instance zero-cost cache, or ``None`` for objects
that don't expose one (e.g. ``MagicMock`` stand-ins in unit tests).
@ -541,7 +542,7 @@ def _get_router_zero_cost_cache(llm_router: Router) -> dict[str, bool] | None:
return cache if isinstance(cache, dict) else None
def is_model_cost_zero(model: str | list[str] | None, llm_router: Router | None) -> bool:
def is_model_cost_zero(model: str | list[str] | None, llm_router: RegistryReader | None) -> bool:
"""
Check if a model has zero cost (no configured pricing).
@ -667,7 +668,7 @@ _NO_MODEL_INFO: Final[Mapping[str, object]] = MappingProxyType({})
_TEAM_GRANT_RELATIONS: Final[Mapping[str, object]] = MappingProxyType({"litellm_model_table": True})
def _deployments_served_for(model: str, llm_router: "Router") -> Sequence[DeploymentTypedDict]:
def _deployments_served_for(model: str, llm_router: RegistryReader) -> Sequence[DeploymentTypedDict]:
"""
The deployments a request to ``model`` routes to, for the gates that waive budget checks.
@ -4842,7 +4843,7 @@ async def get_skill_names_from_access_groups(
def _resolve_all_team_model_sentinel_for_auth_check(
models: list[str],
llm_router: Router | None,
llm_router: RegistryReader | None,
team_id: str | None,
) -> list[str]:
if SpecialModelNames.all_team_models.value not in models or team_id is None or llm_router is None:
@ -4856,7 +4857,7 @@ def _resolve_all_team_model_sentinel_for_auth_check(
def check_model_access_helper(
model: str,
llm_router: Router | None,
llm_router: RegistryReader | None,
models: list[str],
team_model_aliases: dict[str, str] | None = None,
team_id: str | None = None,
@ -4907,7 +4908,7 @@ _check_model_access_helper: Final = check_model_access_helper
def _owning_team_names_for_team_deployment(
model: str,
llm_router: Router | None,
llm_router: RegistryReader | None,
team_id: str | None,
team_model_aliases: Mapping[str, str] | None,
) -> tuple[str, ...]:
@ -4925,7 +4926,7 @@ def _owning_team_names_for_team_deployment(
def _can_object_call_model(
model: str | list[str],
llm_router: Router | None,
llm_router: RegistryReader | None,
models: list[str],
team_model_aliases: dict[str, str] | None = None,
team_id: str | None = None,
@ -5052,7 +5053,7 @@ def _resolve_team_alias(
def _live_team_alias_target(
model: str, team_model_aliases: Mapping[str, str], team_id: str | None, llm_router: Router | None
model: str, team_model_aliases: Mapping[str, str], team_id: str | None, llm_router: RegistryReader | None
) -> str:
target: Final = team_model_aliases.get(model)
if target is None:
@ -5539,7 +5540,7 @@ async def can_key_call_model(
model: str | list[str],
llm_model_list: Sequence[object] | None,
valid_token: UserAPIKeyAuth,
llm_router: litellm.Router | None,
llm_router: RegistryReader | None,
prisma_client: DatabaseClient | None = None,
) -> Literal[True]:
"""
@ -5618,7 +5619,7 @@ async def can_key_call_resolved_model(
model: str,
llm_model_list: Sequence[object] | None,
valid_token: UserAPIKeyAuth,
llm_router: litellm.Router | None,
llm_router: RegistryReader | None,
) -> None:
from litellm.proxy.proxy_server import (
prisma_client,
@ -5766,7 +5767,7 @@ def can_org_access_model(
async def can_team_access_model(
model: str | list[str],
team_object: LiteLLM_TeamTable | None,
llm_router: Router | None,
llm_router: RegistryReader | None,
team_model_aliases: dict[str, str] | None = None,
key_model_aliases: Mapping[str, str] | None = None,
prisma_client: DatabaseClient | None = None,
@ -5869,7 +5870,7 @@ async def key_access_group_grants_model(
model: str | list[str],
valid_token: UserAPIKeyAuth | None,
team_object: LiteLLM_TeamTable | None,
llm_router: Router | None,
llm_router: RegistryReader | None,
key_model_aliases: Mapping[str, str] | None = None,
) -> bool:
"""
@ -5905,7 +5906,7 @@ _key_access_group_grants_model: Final = key_access_group_grants_model
def can_project_access_model(
model: str | list[str],
project_object: LiteLLM_ProjectTable,
llm_router: Router | None,
llm_router: RegistryReader | None,
key_model_aliases: Mapping[str, str] | None = None,
) -> Literal[True]:
"""
@ -5926,7 +5927,7 @@ def can_project_access_model(
def can_customer_access_model(
model: str | list[str],
end_user_object: LiteLLM_EndUserTable,
llm_router: Router | None,
llm_router: RegistryReader | None,
valid_token: UserAPIKeyAuth | None,
) -> Literal[True]:
team_model_aliases: Final = team_model_aliases_for_auth_check(valid_token) if valid_token is not None else None
@ -6089,7 +6090,7 @@ async def _team_grants_personal_key_model(
model: str,
team_object: LiteLLM_TeamTableCachedObj,
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,
@ -6127,7 +6128,7 @@ async def _loaded_teams_allow_model(
model: str,
teams: tuple[LiteLLM_TeamTableCachedObj | None, ...],
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,
@ -6393,7 +6394,7 @@ async def can_personal_key_call_model_via_teams(
model: str | Sequence[str],
user_object: LiteLLM_UserTable | None,
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,
@ -7185,7 +7186,7 @@ async def check_team_member_model_access(
model: str | list[str],
team_object: LiteLLM_TeamTable,
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: Optional["PrismaClient"],
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,

10
litellm/proxy/auth/fallback_budget.py
View file

@ -44,7 +44,7 @@ from litellm.proxy.auth.auth_checks import ( # noqa: F401 # legacy module expo
_is_model_cost_zero, # pyright: ignore[reportPrivateUsage,reportUnusedImport] # backwards-compatible package export
is_model_cost_zero, # pyright: ignore[reportPrivateUsage] # the zero-cost predicate the auth-time budget checks use; no public equivalent
)
from litellm.router import Router
from litellm.router_api import RegistryReader, RoutingContext
from litellm.types.llms.base import LiteLLMBaseModel
@ -118,7 +118,7 @@ async def is_token_within_budget_for_model(
*,
model: str,
valid_token: UserAPIKeyAuth,
llm_router: Router,
llm_router: RegistryReader,
team_member_model_budget_limiter: _TeamMemberModelBudgetLimiter | None = None,
) -> bool:
"""
@ -187,19 +187,19 @@ class RouterFallbackBudgetCheck:
is_enforced: Callable[[], bool]
team_member_model_budget_limiter: _TeamMemberModelBudgetLimiter | None = None
async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], llm_router: Router) -> bool:
async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], ctx: RoutingContext) -> bool:
if not self.is_enforced():
return True
valid_token: Final = _user_api_key_auth_from_request(request_kwargs)
if valid_token is None:
return True
budget_models: Final = tuple(dict.fromkeys((model, llm_router.registry().resolve(model))))
budget_models: Final = tuple(dict.fromkeys((model, ctx.registry().resolve(model))))
for budget_model in budget_models:
try:
if not await is_token_within_budget_for_model(
model=budget_model,
valid_token=valid_token,
llm_router=llm_router,
llm_router=ctx,
team_member_model_budget_limiter=self.team_member_model_budget_limiter,
):
return False

8
litellm/proxy/auth/fallback_model_access.py
View file

@ -18,7 +18,7 @@ from pydantic import ValidationError
from litellm._logging import verbose_proxy_logger
from litellm.proxy._types import ProxyException, UserAPIKeyAuth
from litellm.proxy.auth.auth_checks import can_key_call_resolved_model, can_token_call_search_tool
from litellm.router import Router
from litellm.router_api import RegistryReader, RoutingContext
from litellm.search import asearch
from litellm.types.llms.base import LiteLLMBaseModel
@ -31,7 +31,7 @@ class _FallbackAccessSettings(LiteLLMBaseModel):
enforce_fallback_model_access: bool = True
async def is_model_authorized_for_token(*, model: str, valid_token: UserAPIKeyAuth, llm_router: Router) -> bool:
async def is_model_authorized_for_token(*, model: str, valid_token: UserAPIKeyAuth, llm_router: RegistryReader) -> bool:
try:
await can_key_call_resolved_model(
model=model,
@ -98,7 +98,7 @@ class RouterFallbackAccessCheck:
is_enforced: Callable[[], bool]
async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], llm_router: Router) -> bool:
async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], ctx: RoutingContext) -> bool:
valid_token: Final = _user_api_key_auth_from_request(request_kwargs)
if valid_token is None:
return True
@ -106,7 +106,7 @@ class RouterFallbackAccessCheck:
return await is_search_tool_authorized_for_token(search_tool_name=model, valid_token=valid_token)
if not self.is_enforced():
return True
return await is_model_authorized_for_token(model=model, valid_token=valid_token, llm_router=llm_router)
return await is_model_authorized_for_token(model=model, valid_token=valid_token, llm_router=ctx)
router_fallback_access_check: Final = RouterFallbackAccessCheck(is_enforced=_enforced_by_general_settings)

6
litellm/proxy/auth/model_checks.py
View file

@ -14,13 +14,13 @@ from typing import Final, TypeVar
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
from litellm.llms.azure.ptu_capacity import is_str_keyed_mapping
from litellm.proxy._types import SpecialModelNames, UserAPIKeyAuth
from litellm.repositories.object_permission_repository import ObjectPermissionRepository
from litellm.responses.main import strip_responses_routing_prefix
from litellm.router import Router
from litellm.router_utils.fallback_event_handlers import get_fallback_model_group
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.router_api import fallback_chain
from litellm.types.router import CredentialLiteLLMParams, LiteLLM_Params
from litellm.types.utils import LlmProviders
from litellm.utils import ProviderConfigManager, get_valid_models, potential_model_names_in_lookup_order
@ -708,7 +708,7 @@ def get_all_fallbacks(
try:
# Use existing function to get fallback model group
fallback_model_group, _ = get_fallback_model_group(fallbacks=fallbacks_config, model_group=model)
fallback_model_group, _ = fallback_chain(fallbacks=fallbacks_config, model_group=model)
if fallback_model_group is None:
return []

26
litellm/proxy/auth/user_api_key_auth.py
View file

@ -40,6 +40,7 @@ from litellm.integrations.otel.model.config import is_otel_v2_enabled
from litellm.integrations.otel.runtime import phase_event, phase_span, seed_request_identity
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.dot_notation_indexing import get_nested_value
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias, resolve_served_model
from litellm.models.end_user import LiteLLM_EndUserTable
from litellm.proxy._types import *
from litellm.proxy.agent_endpoints.auth.agent_caller import agent_caller_from_headers
@ -182,13 +183,7 @@ from litellm.proxy.utils import (
from litellm.repositories.table_repositories import JWTKeyMappingRepository, TeamMembershipRepository
from litellm.repositories.user_repository import UserRepository
from litellm.repositories.verification_token_repository import VerificationTokenRepository
from litellm.router_utils.common_utils import resolve_model_group_alias, resolve_served_model
from litellm.router_utils.fallback_event_handlers import (
PROVIDER_SCOPED_RESOURCE_KEYS,
fallback_lookup_groups,
fallbacks_disabled_for_request,
get_fallback_model_group_for_lookup_groups,
)
from litellm.router_api import fallbacks_disabled, references_provider_scoped_resource, request_fallback_chain
from litellm.secret_managers.main import get_secret_bool
from litellm.types.services import ServiceTypes
@ -811,11 +806,7 @@ async def _check_team_member_model_budget_for_model(
model=model_name,
)
except litellm.BudgetExceededError:
if (
request_data.get("model") != model_name
or llm_router is None
or fallbacks_disabled_for_request(kwargs=request_data)
):
if request_data.get("model") != model_name or llm_router is None or fallbacks_disabled(request_data):
raise
fallback_succeeded: Final = await _try_team_member_model_budget_fallback(
user_id=user_id,
@ -857,9 +848,9 @@ async def _try_team_member_model_budget_fallback(
user_api_key_cache: UserApiKeyCache | None = None,
proxy_logging_obj: ProxyLogging | None = None,
) -> bool:
if request.headers.get("content-type", "").lower().startswith("multipart/form-data") or any(
request_data.get(key) for key in PROVIDER_SCOPED_RESOURCE_KEYS
):
if request.headers.get("content-type", "").lower().startswith(
"multipart/form-data"
) or references_provider_scoped_resource(request_data):
return False
from litellm.proxy import proxy_server
@ -896,10 +887,7 @@ async def _try_team_member_model_budget_fallback(
if router_fallbacks is None:
return False
fallback_models, _ = get_fallback_model_group_for_lookup_groups(
fallbacks=list(router_fallbacks),
lookup_groups=fallback_lookup_groups(kwargs=request_data, model_group=model_name),
)
fallback_models, _ = request_fallback_chain(list(router_fallbacks), request_data, model_name)
fallback_targets: Final[Sequence[object]] = fallback_models or ()
fallback_candidates: Final[tuple[str, ...]] = tuple(
model for target in fallback_targets if (model := _budget_fallback_target_model(target)) is not None

2
litellm/proxy/bug_report_config.py
View file

@ -19,8 +19,8 @@ from litellm.litellm_core_utils.bug_report import (
build_bug_report,
build_environment_report,
)
from litellm.litellm_core_utils.routing_groups import VALID_ROUTING_STRATEGIES
from litellm.proxy._types import ConfigGeneralSettings
from litellm.router_utils.routing_groups import VALID_ROUTING_STRATEGIES
from litellm.types.caching import LiteLLMCacheType
from litellm.types.guardrails import GuardrailEventHooks, LitellmParams, SupportedGuardrailIntegrations
from litellm.types.secret_managers.main import KeyManagementSystem

15
litellm/proxy/common_request_processing.py
View file

@ -53,6 +53,7 @@ from litellm.constants import (
)
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.otel.runtime import phase_event
from litellm.litellm_core_utils.add_retry_fallback_headers import get_hidden_params_dict
from litellm.litellm_core_utils.bug_report import (
allowlisted,
bug_report_notice,
@ -81,6 +82,7 @@ from litellm.litellm_core_utils.llm_cost_calc.guardrail_cost import (
from litellm.litellm_core_utils.llm_response_utils.get_headers import (
get_response_headers,
)
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias, resolve_served_model
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.litellm_core_utils.served_output_texts import (
record_served_output_texts,
@ -146,8 +148,6 @@ from litellm.responses.streaming_iterator import partial_responses_incomplete_ev
from litellm.router import Router
from litellm.router_api import RoutingContext
from litellm.router_strategy.auto_router.auto_router_model_naming import AUTO_ROUTER_MODEL_PREFIX
from litellm.router_utils.add_retry_fallback_headers import get_hidden_params_dict
from litellm.router_utils.common_utils import resolve_model_group_alias, resolve_served_model
from litellm.types.extraction import ExtractionResponse
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.router import RouterRateLimitError
@ -2565,7 +2565,7 @@ class ProxyBaseLLMRequestProcessing:
aliased_model: Final = self.data.get("model")
if llm_router is not None and route_type in ROUTER_SCHEDULED_ROUTE_TYPES and isinstance(aliased_model, str):
self.data.update(
llm_router.priority_header_update(
llm_router.health().priority_header_fields(
model=aliased_model, kwargs=self.data, header=request.headers.get(SCHEDULER_PRIORITY_HEADER)
)
)
@ -2831,12 +2831,9 @@ class ProxyBaseLLMRequestProcessing:
@staticmethod
def _resolve_fallback_models(model: str, fallbacks: list) -> list[str] | None:
from litellm.router_utils.fallback_event_handlers import get_fallback_model_group
from litellm.router_api import fallback_chain
fallback_model_group, generic_fallback_idx = get_fallback_model_group(
fallbacks=fallbacks,
model_group=model,
)
fallback_model_group, generic_fallback_idx = fallback_chain(fallbacks, model)
if fallback_model_group is not None:
return fallback_model_group
return fallbacks[generic_fallback_idx]["*"] if generic_fallback_idx is not None else None
@ -4190,8 +4187,8 @@ class ProxyBaseLLMRequestProcessing:
HiddenParamsAsyncIteratorWrapper, so class sniffing runs on the
unwrapped inner iterator.
"""
from litellm.litellm_core_utils.add_retry_fallback_headers import HiddenParamsAsyncIteratorWrapper
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.router_utils.add_retry_fallback_headers import HiddenParamsAsyncIteratorWrapper
unwrapped: Final = response.inner if isinstance(response, HiddenParamsAsyncIteratorWrapper) else response

13
litellm/proxy/common_utils/cache_aware_routing.py
View file

@ -16,6 +16,7 @@ from litellm.litellm_core_utils.core_helpers import get_metadata_variable_name_f
from litellm.llms.anthropic.cache_aware_routing import AnthropicCacheRouting, TokenCounter
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.auth_checks import can_key_call_resolved_model
from litellm.router_deprecated import renamed_router_param
from litellm.router_strategy.complexity_router.config import ComplexityRouterConfig
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.router import Deployment, PreRoutingHookResponse
@ -23,6 +24,7 @@ from litellm.types.utils import StandardLoggingRoutingDecision
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
_MESSAGES: Final = TypeAdapter(list[Mapping[str, object]])
_MAPPING: Final = TypeAdapter(Mapping[str, object])
@ -231,7 +233,7 @@ async def select_cached_model(
async def _choose_cached_model(
*,
router: Router,
ctx: RoutingContext,
config: ComplexityRouterConfig,
params_for_model: Callable[[str, str], Mapping[str, object]],
response: PreRoutingHookResponse | None,
@ -249,8 +251,10 @@ async def _choose_cached_model(
)
from litellm.router_strategy.complexity_router.context_compaction import compaction_pending
router: Final = proxy_server.llm_router
if (
proxy_server.llm_router is not router
router is None
or not ctx.belongs_to(router)
or router.registry().plugins("routing")
or has_request_transforms()
or compaction_pending(request_kwargs)
@ -318,9 +322,10 @@ async def _choose_cached_model(
)
@renamed_router_param("router")
async def choose_cached_model(
*,
router: Router,
ctx: RoutingContext,
config: ComplexityRouterConfig,
params_for_model: Callable[[str, str], Mapping[str, object]],
response: PreRoutingHookResponse | None,
@ -332,7 +337,7 @@ async def choose_cached_model(
try:
return await asyncio.wait_for(
_choose_cached_model(
router=router,
ctx=ctx,
config=config,
params_for_model=params_for_model,
response=response,

2
litellm/proxy/common_utils/credential_hydration.py
View file

@ -12,10 +12,10 @@ from types import MappingProxyType
from typing import Final, cast # noqa: TID251 # narrows untyped credential values at the decrypt boundary
import litellm
from litellm.litellm_core_utils.clientside_credential_handler import clientside_credential_keys
from litellm.proxy.common_utils.encrypt_decrypt_utils import decrypt_value_helper
from litellm.proxy.utils import PrismaClient
from litellm.repositories.credentials_repository import CredentialsRepository
from litellm.router_utils.clientside_credential_handler import clientside_credential_keys
from litellm.types.router import (
GenericLiteLLMParams,
server_owned_wif_fields_present,

2
litellm/proxy/common_utils/model_listing_utils.py
View file

@ -21,7 +21,7 @@ from pydantic import TypeAdapter, ValidationError
import litellm
from litellm.constants import CLAUDE_CODE_CLIENT, CLAUDE_CODE_PICKER_PATTERN, GATEWAY_CLIENT_HEADER
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
if TYPE_CHECKING:
from litellm.router import Router

10
litellm/proxy/db/routing_prisma_wrapper.py
View file

@ -4,6 +4,7 @@ to a writer client. Used when DATABASE_URL_READ_REPLICA is configured;
otherwise PrismaClient uses the writer-only PrismaWrapper directly.
"""
import asyncio
import os
from collections.abc import Callable
from datetime import timedelta
@ -233,8 +234,13 @@ class RoutingPrismaWrapper:
await self._reader.start_token_refresh_task()
async def stop_token_refresh_task(self) -> None:
await self._writer.stop_token_refresh_task()
await self._reader.stop_token_refresh_task()
results: Final = await asyncio.gather(
self._writer.stop_token_refresh_task(),
self._reader.stop_token_refresh_task(),
return_exceptions=True,
)
if (error := next((result for result in results if isinstance(result, BaseException)), None)) is not None:
raise error
async def recreate_prisma_client(
self,

0
litellm/proxy/deps/__init__.py Normal file
View file

22
litellm/proxy/deps/database.py Normal file
View file

@ -0,0 +1,22 @@
from typing import Annotated, Final, TypeAlias
from fastapi import Depends, Request
from litellm.repositories.database import Database
_MISSING_DATABASE: Final = object()
class DatabaseStateMissingError(RuntimeError):
def __init__(self) -> None:
super().__init__("The proxy lifespan did not publish a Database in request state")
async def get_database(request: Request) -> Database | None:
database: Final[object] = getattr(request.state, "database", _MISSING_DATABASE)
if database is None or isinstance(database, Database):
return database
raise DatabaseStateMissingError()
DatabaseDep: TypeAlias = Annotated[Database | None, Depends(get_database)]

36
litellm/proxy/deps/repositories.py Normal file
View file

@ -0,0 +1,36 @@
from typing import Annotated, TypeAlias
from fastapi import Depends
from litellm.models.deployment_warning import DeploymentWarningIgnoreStore
from litellm.models.gateway_activity import GatewayActivityStore
from litellm.models.user_banner import UserBannerStore
from litellm.proxy.deps.database import DatabaseDep
from litellm.repositories.deployment_warning_ignore_repository import DeploymentWarningIgnoreRepository
from litellm.repositories.gateway_activity_repository import GatewayActivityRepository
from litellm.repositories.user_banner_repository import UserBannerRepository
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
return None if database is None else UserBannerRepository(database.prisma)
UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)]
async def get_deployment_warning_ignore_repository(database: DatabaseDep) -> DeploymentWarningIgnoreStore | None:
return None if database is None else DeploymentWarningIgnoreRepository(database.prisma, use_writer=True)
DeploymentWarningIgnoreRepositoryDep: TypeAlias = Annotated[
DeploymentWarningIgnoreStore | None, Depends(get_deployment_warning_ignore_repository)
]
async def get_gateway_activity_repository(database: DatabaseDep) -> GatewayActivityStore | None:
return None if database is None else GatewayActivityRepository(database.prisma)
GatewayActivityRepositoryDep: TypeAlias = Annotated[
GatewayActivityStore | None, Depends(get_gateway_activity_repository)
]

2
litellm/proxy/guardrails/guardrail_hooks/headroom/headroom.py
View file

@ -29,6 +29,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
has_tool_with_name,
)
from litellm.litellm_core_utils.provider_affinity import get_caller_session_id
from litellm.litellm_core_utils.routing_common_utils import request_user_api_key_auth
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client, # pyright: ignore[reportUnknownVariableType]
httpxSpecialProvider,
@ -40,7 +41,6 @@ from litellm.proxy.guardrails.guardrail_hooks.content_text import (
merge_rewritten_text_parts,
)
from litellm.proxy.spend_tracking.compression_savings import HEADROOM_GUARDRAIL_PROVIDER
from litellm.router_utils.common_utils import request_user_api_key_auth
from litellm.secret_managers.main import get_secret_str
from litellm.types.guardrails import GuardrailEventHooks, Mode
from litellm.types.integrations.custom_logger import (

3
litellm/proxy/guardrails/guardrail_hooks/semantic_guard/route_loader.py
View file

@ -14,6 +14,7 @@ from pydantic import TypeAdapter
from litellm._logging import verbose_logger
from litellm.constants import DEFAULT_SEMANTIC_GUARD_SIMILARITY_THRESHOLD
from litellm.router_api import RoutingContext
if TYPE_CHECKING:
from semantic_router.routers import SemanticRouter
@ -134,7 +135,7 @@ class SemanticGuardRouteLoader:
)
encoder: Final = LiteLLMRouterEncoder(
litellm_router_instance=litellm_router,
ctx=RoutingContext(litellm_router),
model_name=embedding_model,
score_threshold=global_threshold,
)

2
litellm/proxy/health_check.py
View file

@ -27,12 +27,12 @@ from litellm.constants import (
HEALTH_CHECK_TIMEOUT_SECONDS,
)
from litellm.litellm_core_utils.health_check_helpers import native_health_check_mode
from litellm.litellm_core_utils.routing_common_utils import deployment_model_id_pin, is_model_capability_field
from litellm.router_strategy.auto_router.auto_router_model_naming import (
StrategyRouterDependency,
classify_strategy_router_model,
strategy_router_dependencies,
)
from litellm.router_utils.common_utils import deployment_model_id_pin, is_model_capability_field
from litellm.types.utils import secret_bearing_wif_litellm_params, server_owned_wif_litellm_params
# Provider routing and server-owned federation or OAuth token-exchange fields. Allowed for proxy

2
litellm/proxy/health_check_utils/shared_health_check_manager.py
View file

@ -13,10 +13,10 @@ from litellm.caching.redis_cache import RedisCache
from litellm.constants import (
DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL,
DEFAULT_SHARED_HEALTH_CHECK_TTL,
HEALTH_CHECKS_TARGET,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.proxy.health_check import perform_health_check
from litellm.router_utils.health_state_cache import HEALTH_CHECKS_TARGET
if TYPE_CHECKING:
from litellm.router import Router

10
litellm/proxy/health_endpoints/_health_endpoints.py
View file

@ -26,7 +26,12 @@ from litellm.integrations.SlackAlerting.ms_teams import (
build_ms_teams_payload,
get_ms_teams_webhook_url,
)
from litellm.litellm_core_utils.clientside_credential_handler import (
_ADMIN_CONFIG_FIELDS_TO_CLEAR_ON_BASE_OVERRIDE, # pyright: ignore[reportPrivateUsage] # one canonical list, shared with the router path
clientside_credential_keys,
)
from litellm.litellm_core_utils.custom_logger_registry import CustomLoggerRegistry
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.proxy._types import (
AlertType,
@ -94,11 +99,6 @@ from litellm.proxy.middleware.in_flight_requests_middleware import (
from litellm.proxy.shutdown.graceful_shutdown_manager import GracefulShutdownManager
from litellm.router import Router
from litellm.router_api import deployment_usable_by_team
from litellm.router_utils.clientside_credential_handler import (
_ADMIN_CONFIG_FIELDS_TO_CLEAR_ON_BASE_OVERRIDE, # pyright: ignore[reportPrivateUsage] # one canonical list, shared with the router path
clientside_credential_keys,
)
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.secret_managers.main import get_secret_bool
from litellm.types.proxy.litellm_pre_call_utils import RedactedDict, SecretFields
from litellm.types.router import reject_server_owned_wif_fields

53
litellm/proxy/health_endpoints/deployment_warning_endpoints.py
View file

@ -1,4 +1,3 @@
import json
import logging
from collections.abc import Mapping
from dataclasses import dataclass
@ -7,12 +6,14 @@ from types import MappingProxyType
from typing import Annotated, Final, Literal, Protocol, TypeAlias
from fastapi import APIRouter, Depends, HTTPException, status
from pydantic import AwareDatetime, ConfigDict, Field, JsonValue, TypeAdapter, ValidationError, model_validator
from pydantic import ConfigDict, Field, JsonValue, TypeAdapter, model_validator
from litellm._logging import verbose_logger, verbose_proxy_logger
from litellm._logging import verbose_logger
from litellm.constants import LITELLM_PROXY_ADMIN_NAME
from litellm.models.deployment_warning import AdminIgnore, DeploymentWarningIgnoreStore
from litellm.proxy._types import AUDIT_ACTIONS, LitellmTableNames, LitellmUserRoles, UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.deps.repositories import DeploymentWarningIgnoreRepositoryDep
from litellm.proxy.health_endpoints._health_endpoints import (
DISABLE_NO_REDIS_WARNING_ENV_VAR,
env_credential_login_enabled,
@ -22,7 +23,6 @@ from litellm.proxy.health_endpoints._health_endpoints import (
)
from litellm.repositories.deployment_warning_ignore_repository import (
DEPLOYMENT_WARNING_IGNORES_ROW_ID,
DeploymentWarningIgnoreRepository,
)
from litellm.types.llms.base import LiteLLMBaseModel
@ -50,18 +50,6 @@ class WarningCondition:
env_var_ignore: str | None = None
class AdminIgnore(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
source: Literal["admin"] = "admin"
ignored_by: str
ignored_at: AwareDatetime
ignored_until: AwareDatetime | None = Field(
default=None,
description="When the snooze ends. Null means the warning is ignored until an admin restores it.",
)
class EnvVarIgnore(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
@ -109,12 +97,6 @@ class DeploymentWarningIgnoreResponse(LiteLLMBaseModel):
ignore: AdminIgnore | None
class DeploymentWarningIgnoreStore(Protocol):
async def get_raw_ignores(self) -> object: ...
async def upsert_ignores(self, payload: str) -> None: ...
class ConfigAuditLogger(Protocol):
async def __call__(
self,
@ -127,7 +109,6 @@ class ConfigAuditLogger(Protocol):
) -> None: ...
_IGNORES_ADAPTER: Final[TypeAdapter[Mapping[str, AdminIgnore]]] = TypeAdapter(Mapping[str, AdminIgnore])
_JSON_VALUE_ADAPTER: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
_AUDIT_ENTRIES_ADAPTER: Final[TypeAdapter[tuple[DeploymentWarningIgnoreResponse, ...]]] = TypeAdapter(
tuple[DeploymentWarningIgnoreResponse, ...]
@ -185,18 +166,8 @@ def is_known_warning_id(warning_id: str) -> bool:
return False
def _parse_stored_ignores(raw: object) -> Mapping[str, AdminIgnore]:
if raw is None:
return MappingProxyType({})
try:
return MappingProxyType(_IGNORES_ADAPTER.validate_python(json.loads(raw) if isinstance(raw, str) else raw))
except (json.JSONDecodeError, ValidationError):
verbose_proxy_logger.warning("Ignoring unreadable %s row", DEPLOYMENT_WARNING_IGNORES_ROW_ID)
return MappingProxyType({})
async def _read_unexpired_ignores(store: DeploymentWarningIgnoreStore, now: datetime) -> Mapping[str, AdminIgnore]:
stored: Final = _parse_stored_ignores(await store.get_raw_ignores())
stored: Final = await store.get_ignores()
return MappingProxyType(
{
warning_id: ignore
@ -225,14 +196,6 @@ def utc_now() -> datetime:
return datetime.now(timezone.utc)
def deployment_warning_ignore_store() -> DeploymentWarningIgnoreStore | None:
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
return None
return DeploymentWarningIgnoreRepository(prisma_client, use_writer=True)
def config_audit_logger() -> ConfigAuditLogger:
from litellm.proxy.proxy_server import create_config_audit_log
@ -281,7 +244,7 @@ async def current_warning_conditions(now: Annotated[datetime, Depends(utc_now)])
async def get_deployment_warnings(
_: Annotated[UserAPIKeyAuth, Depends(_deployment_warning_reader)],
conditions: Annotated[tuple[WarningCondition, ...], Depends(current_warning_conditions)],
store: Annotated[DeploymentWarningIgnoreStore | None, Depends(deployment_warning_ignore_store)],
store: DeploymentWarningIgnoreRepositoryDep,
now: Annotated[datetime, Depends(utc_now)],
) -> DeploymentWarningsResponse:
"""
@ -309,7 +272,7 @@ async def update_deployment_warning_ignore(
warning_id: str,
update: DeploymentWarningIgnoreUpdate,
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(_deployment_warning_admin)],
store: Annotated[DeploymentWarningIgnoreStore | None, Depends(deployment_warning_ignore_store)],
store: DeploymentWarningIgnoreRepositoryDep,
audit_logger: Annotated[ConfigAuditLogger, Depends(config_audit_logger)],
now: Annotated[datetime, Depends(utc_now)],
) -> DeploymentWarningIgnoreResponse:
@ -338,7 +301,7 @@ async def update_deployment_warning_ignore(
others: Final = {other_id: other for other_id, other in before.items() if other_id != warning_id}
after: Final = MappingProxyType(others if ignore is None else {**others, warning_id: ignore})
await store.upsert_ignores(_IGNORES_ADAPTER.dump_json(dict(after)).decode())
await store.save_ignores(after)
await audit_logger(
param_name=DEPLOYMENT_WARNING_IGNORES_ROW_ID,
action="updated",

8
litellm/proxy/hooks/dynamic_rate_limiter_v3.py
View file

@ -15,6 +15,10 @@ from litellm._internal_context import with_service_target
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.add_retry_fallback_headers import (
ensure_response_additional_headers,
response_has_hidden_params,
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.common_utils.proxy_rate_limit_error import (
ProxyRateLimitError,
@ -34,10 +38,6 @@ from litellm.proxy.hooks.rate_limiter_utils import (
resolve_llm_provider_for_rate_limit,
)
from litellm.proxy.utils import InternalUsageCache
from litellm.router_utils.add_retry_fallback_headers import (
ensure_response_additional_headers,
response_has_hidden_params,
)
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import CallTypesLiteral, LLMResponseTypes

2
litellm/proxy/hooks/model_max_budget_limiter.py
View file

@ -204,7 +204,7 @@ def _team_public_names_by_group(router: "Router", team_id: str | None) -> Mappin
groups_by_name: Final = MappingProxyType(
{
name: model_access_groups_of(router=router, model=name, team_id=team_id)
for name in router.team_public_model_names_of(team_id)
for name in router.registry().names(kind="team_public", team_id=team_id)
}
)
return MappingProxyType(

20
litellm/proxy/hooks/parallel_request_limiter_v3.py
View file

@ -49,6 +49,10 @@ from litellm.constants import (
)
from litellm.cost_calculator import get_usage_object
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.add_retry_fallback_headers import (
ensure_response_additional_headers,
response_has_hidden_params,
)
from litellm.litellm_core_utils.audio_utils.utils import estimate_audio_tokens
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.prompt_templates.common_utils import (
@ -56,6 +60,8 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
split_media_for_tpm,
)
from litellm.litellm_core_utils.ptu_pricing import is_ptu_cost_attribution_enabled
from litellm.litellm_core_utils.ptu_shares import PTUTeamCeiling
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.litellm_core_utils.token_counter import estimate_image_tokens, offload_token_count
from litellm.llms.azure.ptu_capacity import normalized_tokens
from litellm.proxy._types import UserAPIKeyAuth
@ -86,12 +92,6 @@ from litellm.proxy.spend_tracking.input_tokens import (
openai_chat_pdf_file_tokens,
)
from litellm.responses.litellm_completion_transformation.compaction_items import billable_input_since_newest_compaction
from litellm.router_utils.add_retry_fallback_headers import (
ensure_response_additional_headers,
response_has_hidden_params,
)
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.router_utils.ptu_shares import PTUTeamCeiling
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.extraction import ExtractionResponse
from litellm.types.llms.openai import BaseLiteLLMOpenAIResponseObject, ResponseAPIUsage
@ -3561,9 +3561,6 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
Returns True if any deployment has failures in the current minute.
"""
from litellm.proxy.proxy_server import llm_router
from litellm.router_utils.router_callbacks.track_deployment_metrics import (
get_deployment_failures_for_current_minute,
)
if llm_router is None:
return False
@ -3581,10 +3578,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
continue
# Use router's existing failure tracking
failure_count = get_deployment_failures_for_current_minute(
litellm_router_instance=llm_router,
deployment_id=deployment_id,
)
failure_count = llm_router.health().failures_this_minute(deployment_id)
if failure_count > DYNAMIC_RATE_LIMIT_ERROR_THRESHOLD_PER_MINUTE:
verbose_proxy_logger.debug(

31
litellm/proxy/litellm_pre_call_utils.py
View file

@ -112,6 +112,7 @@ _EXPLICIT_SESSION_HEADERS: Final = frozenset({"x-litellm-trace-id", "x-litellm-s
# ``session-id``/``thread-id``; builds before the codex-api split sent
# ``session_id``/``conversation_id``. Ordered session before thread.
_CODEX_SESSION_ID_HEADERS: Final = ("session-id", "session_id", "thread-id", "conversation_id")
_BEDROCK_SESSION_ID_METADATA_KEYS: Final = ("sessionId", "threadId", "conversationId")
# Session-id values must be non-empty strings of alphanumerics, hyphens, or underscores
# (covers UUIDs and most common session-id formats).
_SESSION_ID_VALUE_RE: Final = re.compile(r"^[a-zA-Z0-9_\-]{8,}$")
@ -1055,6 +1056,22 @@ def _get_anthropic_session_id_from_metadata(metadata: object) -> str | None:
return session_id
def _get_bedrock_session_id_from_request_metadata(
normalized_headers: Mapping[str, str], request_body: Mapping[str, object]
) -> str | None:
"""Read the conversation id a Bedrock SDK client tags its calls with: Converse's ``requestMetadata``
body field or InvokeModel's ``X-Amzn-Bedrock-Request-Metadata`` JSON header. Only the Bedrock
pass-through route nests the client's body under ``data``."""
if request_body.get("custom_llm_provider") != "bedrock" or not isinstance(request_body.get("data"), Mapping):
return None
header: Final = normalized_headers.get("x-amzn-bedrock-request-metadata")
body_tags: Final = _request_fields(_request_fields(request_body.get("data")).get("requestMetadata"))
header_tags: Final = _request_fields(safe_json_loads(header) if header else None)
tag_sources: Final = itertools.product((body_tags, header_tags), _BEDROCK_SESSION_ID_METADATA_KEYS)
values: Final = (tags.get(key) for tags, key in tag_sources)
return next((value for value in values if isinstance(value, str) and value), None)
def _is_llm_inference_route(request: Request) -> bool:
route: Final = get_request_route(request)
return RouteChecks.is_llm_api_route(route=route) and not RouteChecks.check_route_access(
@ -1967,6 +1984,14 @@ class LiteLLMProxyRequestSetup:
body_metadata["user_id"] = session_id
verbose_proxy_logger.debug("Extracted session_id from Anthropic metadata.user_id")
if "litellm_session_id" not in data:
bedrock_session_id: Final = _get_bedrock_session_id_from_request_metadata(normalized_headers, request_body)
metadata_session_id: Final = _caller_trace_field(request_body, _metadata_variable_name, "session_id")
if bedrock_session_id and metadata_session_id in (None, bedrock_session_id):
metadata_from_headers["session_id"] = bedrock_session_id
data["litellm_session_id"] = bedrock_session_id # rebind-ok: data is an out-param
verbose_proxy_logger.debug("Extracted session_id from Bedrock request metadata")
# Last-resort fallback: the W3C standards for trace/session propagation
# (https://www.w3.org/TR/trace-context/, https://www.w3.org/TR/baggage/).
# Lower priority than everything above - only fires when neither the
@ -3266,7 +3291,7 @@ def _fallback_edges(
and its provider-stripped suffix, plus the router matcher run over the fuzzy
entries that have not fired yet (tracked in ``fired_fuzzy`` by
``(list_index, entry_index)``), so each entry's targets are expanded once."""
from litellm.router_utils.fallback_event_handlers import get_fallback_model_group
from litellm.router_api import fallback_chain
stripped: Final = model_group.split("/", 1)[1] if "/" in model_group else None
targets: Final[set[str]] = set() # mutable-ok: accumulates fallback targets
@ -3277,7 +3302,7 @@ def _fallback_edges(
pending = tuple(index for index in range(len(fuzzy)) if (list_index, index) not in fired_fuzzy)
if not pending:
continue
resolved = get_fallback_model_group(fallbacks=[fuzzy[index] for index in pending], model_group=model_group)[0]
resolved = fallback_chain([fuzzy[index] for index in pending], model_group)[0]
values = resolved if isinstance(resolved, list) else ([resolved] if resolved is not None else [])
targets.update(name for value in values if (name := _fallback_target_name(value)) is not None)
if resolved is not None:
@ -3464,7 +3489,7 @@ async def _resolve_user_provider_credentials_for_request(
status_code=400,
detail=(f"fallback lists in the request body cannot exceed {_FALLBACK_DISCOVERY_LIMIT} entries"),
)
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
# Discovery must see the group the router actually routes to, so include the
# post-alias name from every alias map that may still rewrite data["model"]

3
litellm/proxy/management_endpoints/auto_router_endpoints.py
View file

@ -60,6 +60,7 @@ from litellm.repositories.table_repositories import PrismaTableRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.repositories.verification_token_repository import VerificationTokenRepository
from litellm.router_api import RoutingContext
from litellm.router_strategy.auto_router.auto_router_model_naming import (
StrategyRouterDependencyRole,
classify_strategy_router_model,
@ -614,7 +615,7 @@ async def preview_auto_router_routing(
try:
complexity_strategy: Final = ComplexityRouter(
model_name=resolved.router_name,
litellm_router_instance=llm_router,
ctx=RoutingContext(llm_router),
complexity_router_config=resolved.complexity_router_config.model_dump(exclude_none=True),
default_model=resolved.complexity_router_config.resolve_default_model(resolved.default_model),
derive_savings_baseline=False,

74
litellm/proxy/management_endpoints/gateway_request_endpoints.py
View file

@ -13,15 +13,15 @@ and the endpoint is restricted to proxy admin roles.
from collections.abc import Sequence
from datetime import datetime, timedelta, timezone
from typing import Annotated, Final, Literal
from typing import Annotated, Final
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import ConfigDict, Field, TypeAdapter
from litellm._logging import verbose_proxy_logger
from litellm.models.gateway_activity import GatewayRouteActivity, GatewayStatusCodeActivity
from litellm.proxy._types import CommonProxyErrors, LitellmUserRoles, UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.proxy.deps.repositories import GatewayActivityRepositoryDep
from litellm.types.proxy.gateway_requests import (
GatewayRequestActivityResponse,
GatewayRequestBreakdownEntry,
@ -33,50 +33,6 @@ router: Final = APIRouter()
_DEFAULT_LOOKBACK_DAYS: Final = 30
_ACTIVITY_SQL: Final = """
SELECT
'route' AS kind, date, category, route,
SUM(successful_requests)::bigint AS successful_requests,
SUM(failed_requests)::bigint AS failed_requests,
NULL::integer AS status_code
FROM "LiteLLM_DailyGatewayRequests"
WHERE date >= $1 AND date <= $2
GROUP BY date, category, route
UNION ALL
SELECT
'status' AS kind, NULL, NULL, NULL,
0::bigint,
SUM(failed_requests)::bigint,
status_code
FROM "LiteLLM_DailyGatewayFailedRequests"
WHERE date >= $1 AND date <= $2
GROUP BY status_code
"""
class _AggregateRow(LiteLLMBaseModel):
"""Validates one query_raw row so the handler works with typed values, not Any."""
kind: Literal["route"]
date: str
category: str
route: str
successful_requests: int
failed_requests: int
class _StatusCodeAggregateRow(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
kind: Literal["status"]
status_code: int
failed_requests: int
_ACTIVITY_ROWS_ADAPTER: Final = TypeAdapter(
tuple[Annotated[_AggregateRow | _StatusCodeAggregateRow, Field(discriminator="kind")], ...]
)
def _default_range() -> tuple[str, str]:
end: Final = datetime.now(timezone.utc)
@ -84,7 +40,7 @@ def _default_range() -> tuple[str, str]:
return start.strftime("%Y-%m-%d"), end.strftime("%Y-%m-%d")
def _fold_by_date(rows: Sequence[_AggregateRow]) -> tuple[GatewayRequestDailyEntry, ...]:
def _fold_by_date(rows: Sequence[GatewayRouteActivity]) -> tuple[GatewayRequestDailyEntry, ...]:
dates: Final = sorted(frozenset(row.date for row in rows))
return tuple(
GatewayRequestDailyEntry(
@ -96,7 +52,7 @@ def _fold_by_date(rows: Sequence[_AggregateRow]) -> tuple[GatewayRequestDailyEnt
)
def _fold_by_route(rows: Sequence[_AggregateRow]) -> tuple[GatewayRequestBreakdownEntry, ...]:
def _fold_by_route(rows: Sequence[GatewayRouteActivity]) -> tuple[GatewayRequestBreakdownEntry, ...]:
pairs: Final = sorted(frozenset((row.category, row.route) for row in rows))
entries: Final = tuple(
GatewayRequestBreakdownEntry(
@ -113,7 +69,7 @@ def _fold_by_route(rows: Sequence[_AggregateRow]) -> tuple[GatewayRequestBreakdo
def _fold_by_status_code(
rows: Sequence[_StatusCodeAggregateRow],
rows: Sequence[GatewayStatusCodeActivity],
) -> tuple[GatewayRequestStatusCodeEntry, ...]:
return tuple(
GatewayRequestStatusCodeEntry(status_code=row.status_code, failed_requests=row.failed_requests)
@ -128,6 +84,7 @@ def _fold_by_status_code(
)
async def get_gateway_daily_activity(
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
activity: GatewayActivityRepositoryDep,
start_date: str | None = Query(default=None, description="Start date in YYYY-MM-DD format"),
end_date: str | None = Query(default=None, description="End date in YYYY-MM-DD format"),
) -> GatewayRequestActivityResponse:
@ -137,8 +94,6 @@ async def get_gateway_daily_activity(
Deployment-wide: the underlying table has no per-key or per-user dimension,
so this is admin-only.
"""
from litellm.proxy.proxy_server import prisma_client
if user_api_key_dict.user_role not in (
LitellmUserRoles.PROXY_ADMIN,
LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY,
@ -148,22 +103,15 @@ async def get_gateway_daily_activity(
detail="Only proxy admin roles can view gateway request counts across the deployment",
)
if prisma_client is None:
if activity is None:
raise HTTPException(status_code=500, detail=CommonProxyErrors.db_not_connected_error.value)
default_start, default_end = _default_range()
selected_start: Final = start_date or default_start
selected_end: Final = end_date or default_end
raw_rows: Final = await prisma_client.db.query_raw( # pyright: ignore[reportAny] # untyped prisma client
_ACTIVITY_SQL,
selected_start,
selected_end,
)
activity_rows: Final = _ACTIVITY_ROWS_ADAPTER.validate_python(raw_rows or ())
rows: Final[tuple[_AggregateRow, ...]] = tuple(row for row in activity_rows if isinstance(row, _AggregateRow))
status_code_rows: Final[tuple[_StatusCodeAggregateRow, ...]] = tuple(
row for row in activity_rows if isinstance(row, _StatusCodeAggregateRow)
)
activity_rows: Final = await activity.get_activity(selected_start, selected_end)
rows: Final = tuple(row for row in activity_rows if isinstance(row, GatewayRouteActivity))
status_code_rows: Final = tuple(row for row in activity_rows if isinstance(row, GatewayStatusCodeActivity))
verbose_proxy_logger.debug("/gateway/daily/activity - aggregated %d rows", len(rows))
return GatewayRequestActivityResponse(

8
litellm/proxy/management_endpoints/ptu_consumption.py
View file

@ -12,14 +12,14 @@ from types import MappingProxyType
from typing import Final
from litellm.litellm_core_utils.ptu_pricing import is_ptu_cost_attribution_enabled
from litellm.llms.azure.ptu_capacity import PTUCapacity, normalized_tokens, ptu_hours
from litellm.router import Router
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.router_utils.ptu_shares import (
from litellm.litellm_core_utils.ptu_shares import (
model_group_ptu_capacity,
routed_deployments,
team_servable_deployments,
)
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.llms.azure.ptu_capacity import PTUCapacity, normalized_tokens, ptu_hours
from litellm.router import Router
from litellm.types.proxy.management_endpoints.common_daily_activity import (
DailySpendData,
MetricWithMetadata,

4
litellm/proxy/management_helpers/model_quota_policy.py
View file

@ -9,10 +9,10 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator
import litellm
from litellm.exceptions import BadRequestError
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider, is_openai_compatible_endpoint
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter, PatternUtils
from litellm.litellm_core_utils.routing_common_utils import provider_for_generic_call
from litellm.llms.azure.ptu_capacity import is_str_keyed_mapping
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
from litellm.router_utils.common_utils import provider_for_generic_call
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter, PatternUtils
from litellm.secret_managers.main import get_secret_str
if TYPE_CHECKING:

2
litellm/proxy/management_helpers/ptu_share_guard.py
View file

@ -5,8 +5,8 @@ from typing import Final
from fastapi import HTTPException
from litellm.litellm_core_utils.ptu_shares import deployments_shared_with
from litellm.router import Router
from litellm.router_utils.ptu_shares import deployments_shared_with
def raise_if_teams_hold_ptu_shares(team_ids: Sequence[str], llm_router: Router | None) -> None:

5
litellm/proxy/mcp_server/semantic_tool_filter.py
View file

@ -14,6 +14,7 @@ from litellm.litellm_core_utils.exception_mapping_utils import ExceptionCheckers
from litellm.proxy.mcp_server.catalog import catalog_operation, global_manager
from litellm.proxy.mcp_server.faults import iter_exception_tree
from litellm.proxy.mcp_server.utils import MCP_TOOL_PREFIX_SEPARATOR
from litellm.router_api import RoutingContext
if TYPE_CHECKING:
from semantic_router.routers import SemanticRouter
@ -172,7 +173,7 @@ class SemanticMCPToolFilter:
self.tool_router = SemanticRouter(
routes=routes,
encoder=LiteLLMRouterEncoder(
litellm_router_instance=self.router_instance,
ctx=RoutingContext(self.router_instance),
model_name=self.embedding_model,
score_threshold=self.similarity_threshold,
),
@ -247,7 +248,7 @@ class SemanticMCPToolFilter:
router: Final = SemanticRouter(
routes=[],
encoder=LiteLLMRouterEncoder(
litellm_router_instance=self.router_instance,
ctx=RoutingContext(self.router_instance),
model_name=self.embedding_model,
score_threshold=self.similarity_threshold,
),

2
litellm/proxy/openai_files_endpoints/files_endpoints.py
View file

@ -1786,7 +1786,7 @@ def _routed_model_name_on_provider(
def _stored_model_name_routes_to_provider(
llm_router: Router, custom_llm_provider: str, provider_model_names: frozenset[str], model_name: str
) -> bool:
routing_group: Final = llm_router.get_routing_group(model_name)
routing_group: Final = llm_router.registry().routing_group(model_name)
routed_model_names: Final = (
tuple(routing_group.models) if routing_group is not None else (llm_router.registry().resolve(model_name),)
)

74
litellm/proxy/policy_engine/pipeline_executor.py
View file

@ -329,6 +329,48 @@ def _prepare_hook_input(
return hook_input, scans_raw_request
async def _run_pre_call_hook(
target: CustomLogger,
callback: CustomGuardrail,
data: dict[str, object], # mutable-ok: the request payload the run is recorded on
hook_input: dict[str, object], # mutable-ok: same request-payload shape the hooks mutate
user_api_key_dict: "UserAPIKeyAuth",
call_type: str,
) -> dict[str, object] | None:
try:
response: Final = await target.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=None,
data=hook_input,
call_type=call_type,
)
except Exception:
callback.mark_pre_call_hook_ran(data, raised=True)
raise
callback.mark_pre_call_hook_ran(data)
if not isinstance(response, dict):
return None
payload: Final = cast(dict[str, object], response) # cast-ok: a pre_call hook returns the request payload
callback.mark_pre_call_hook_ran(payload)
return payload
def _name_step_that_ran(
request_data: dict[str, object], # mutable-ok: the request payload the step's name is recorded on
metadata_key: Literal["metadata", "litellm_metadata"],
guardrail_name: str,
) -> None:
"""Name a step that ran in the applied guardrails of the metadata bucket the pipeline's request started with."""
existing: Final = request_data.get(metadata_key)
bucket: Final[dict[str, object]] = ( # mutable-ok: the request's own metadata bucket, written in place
cast("dict[str, object]", existing) # cast-ok: metadata buckets are str-keyed
if isinstance(existing, dict)
else {}
)
request_data[metadata_key] = bucket # rebind-ok: the step's name is recorded on the request payload
add_guardrail_to_applied_guardrails_header(request_data={metadata_key: bucket}, guardrail_name=guardrail_name)
def _release_original_chunks(
guardrail_name: str,
reason: str,
@ -389,6 +431,9 @@ class PipelineExecutor:
PipelineExecutionResult with terminal action and step results
"""
step_results: Final[list[PipelineStepResult]] = []
metadata_key: Final = get_metadata_variable_name_from_kwargs(
cast(Mapping[str, object], data) # cast-ok: the request payload is str-keyed
)
working_data = data.copy()
if "metadata" in working_data:
working_data["metadata"] = working_data["metadata"].copy()
@ -407,6 +452,7 @@ class PipelineExecutor:
data=working_data,
user_api_key_dict=user_api_key_dict,
call_type=call_type,
metadata_key=metadata_key,
raw_request_snapshot=raw_request_snapshot,
streaming_chunks=streaming_chunks,
endpoint_translation=endpoint_translation,
@ -480,6 +526,7 @@ class PipelineExecutor:
user_api_key_dict: "UserAPIKeyAuth",
litellm_logging_obj: "LiteLLMLoggingObj | None",
stream_already_sent: bool,
metadata_key: Literal["metadata", "litellm_metadata"],
) -> None:
"""Run one streaming post_call step through the endpoint translation, delivering
text and tool-call rewrites on translations that support ended-stream write-back. A
@ -489,8 +536,8 @@ class PipelineExecutor:
live, one that drops or adds a tool call, or one the translation or adapter refused with
``UndeliverableStreamRewrite``) is
discarded: the buffered chunks go back to the originals and the step passes, so the
client gets the stream the merge base sent, and the guardrail stays out of the
applied-guardrails header since its output never reached the client. The response an
client gets the stream the merge base sent. The guardrail is named in the
applied-guardrails header either way, since it ran. The response an
earlier step's translation stored under ``request_data["response"]`` is dropped first,
so this step's hook sees the stream as the steps before it left it."""
scanner: Final = (
@ -525,12 +572,11 @@ class PipelineExecutor:
except UndeliverableStreamRewrite as undeliverable:
_release_original_chunks(step.guardrail, undeliverable.reason, streaming_chunks, originals)
return
finally:
_name_step_that_ran(hook_input, metadata_key, step.guardrail)
discard_reason: Final = observer.discard_reason(rewrite_undeliverable_reason)
if discard_reason is not None:
_release_original_chunks(step.guardrail, discard_reason, streaming_chunks, originals)
return
if not callback.records_own_guardrail_information:
add_guardrail_to_applied_guardrails_header(request_data=hook_input, guardrail_name=step.guardrail)
@staticmethod
async def _run_step(
@ -539,6 +585,7 @@ class PipelineExecutor:
data: dict,
user_api_key_dict: "UserAPIKeyAuth",
call_type: str,
metadata_key: Literal["metadata", "litellm_metadata"],
raw_request_snapshot: dict | None = None, # mutable-ok: same request-payload shape as data
streaming_chunks: list[object] | None = None, # mutable-ok: shared buffered-stream chunks, read per step
endpoint_translation: "BaseTranslation | None" = None,
@ -582,16 +629,7 @@ class PipelineExecutor:
try:
if mode == "pre_call":
response = await target.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=None,
data=hook_input,
call_type=call_type,
)
if isinstance(callback, CustomGuardrail):
callback.mark_pre_call_hook_ran(data)
if isinstance(response, dict):
callback.mark_pre_call_hook_ran(response)
response = await _run_pre_call_hook(target, callback, data, hook_input, user_api_key_dict, call_type)
elif mode == "post_call" and streaming_chunks is not None:
if endpoint_translation is None:
return (
@ -609,9 +647,15 @@ class PipelineExecutor:
user_api_key_dict=user_api_key_dict,
litellm_logging_obj=data.get("litellm_logging_obj"),
stream_already_sent=stream_already_sent,
metadata_key=metadata_key,
)
response = None
elif mode == "post_call":
_name_step_that_ran(
cast(dict[str, object], data), # cast-ok: the request payload is str-keyed
metadata_key,
step.guardrail,
)
response = await target.async_post_call_success_hook(
user_api_key_dict=user_api_key_dict,
data=data,

2
litellm/proxy/policy_engine/response_retrieval.py
View file

@ -7,6 +7,7 @@ from pydantic import TypeAdapter
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.core_helpers import get_or_create_metadata_bucket
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.proxy.common_utils.callback_utils import (
add_guardrail_to_applied_guardrails_header,
add_policy_sources_to_metadata,
@ -18,7 +19,6 @@ from litellm.proxy.policy_engine.policy_matcher import PolicyMatcher
from litellm.proxy.policy_engine.policy_registry import get_policy_registry
from litellm.proxy.policy_engine.policy_resolver import PolicyResolver
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.types.proxy.policy_engine import PolicyMatchContext
from litellm.types.proxy.policy_engine.pipeline_types import GuardrailPipeline

182
litellm/proxy/proxy_server.py
View file

@ -89,6 +89,11 @@ from litellm.constants import (
OPENAI_LIVE_SESSION_START_TIMEOUT_SECONDS,
RUNTIME_UPDATABLE_ROUTER_SETTINGS,
)
from litellm.litellm_core_utils.access_windows import access_windows_config_error
from litellm.litellm_core_utils.add_retry_fallback_headers import (
get_fallback_errors_from_headers,
get_hidden_params_dict,
)
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.bug_report import (
allowlisted,
@ -100,6 +105,8 @@ from litellm.litellm_core_utils.litellm_logging import (
initialize_sentry,
)
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER, LoggingWorker
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.litellm_core_utils.routing_groups import parse_routing_groups
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
from litellm.litellm_core_utils.token_counter import (
@ -174,6 +181,7 @@ from litellm.proxy.common_utils.token_count_settings import (
)
from litellm.proxy.management_helpers.auto_router_availability import AutoRouterCatalogEntry, build_auto_router_catalog
from litellm.proxy.management_helpers.model_group_custom_fields import model_group_custom_fields
from litellm.repositories import database as database_runtime
from litellm.router_strategy.auto_router.auto_router_model_naming import (
GATED_AUTO_ROUTER_CAPABILITIES,
STRATEGY_ROUTER_PARAM_FIELDS,
@ -188,13 +196,6 @@ from litellm.router_strategy.auto_router.auto_router_tuning_baseline import (
snapshot_tuning_baselines,
tuning_limit_violation,
)
from litellm.router_utils.access_windows import access_windows_config_error
from litellm.router_utils.add_retry_fallback_headers import (
get_fallback_errors_from_headers,
get_hidden_params_dict,
)
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.router_utils.routing_groups import parse_routing_groups
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.utils import (
PRICING_OVERRIDES_KEY,
@ -998,7 +999,7 @@ from litellm.router import (
LiteLLM_Params,
ModelGroupInfo,
)
from litellm.router_api import ChangeSet, deployment_id
from litellm.router_api import ChangeSet, deployment_id, router_init_args
from litellm.scheduler import FlowItem, Scheduler
from litellm.secret_managers.aws_secret_manager import load_aws_kms
from litellm.secret_managers.google_kms import load_google_kms
@ -1350,13 +1351,7 @@ async def proxy_shutdown_event(worker_heartbeat: ProxyWorkerHeartbeat | None = N
if worker_heartbeat is not None and prisma_client:
await worker_heartbeat.deregister()
if prisma_client:
# Drain the SGR fold first: it lives in memory, so an un-drained interval
# is lost, and a write attempted after disconnect raises
# ClientNotConnectedError rather than persisting anything. Ordering this
# inside the same guard is what keeps the two from drifting apart.
await asyncio.gather(_drain_audit_tasks_on_shutdown(), _flush_request_accumulators(prisma_client))
verbose_proxy_logger.debug("Disconnecting from Prisma")
await prisma_client.disconnect()
else:
await _drain_audit_tasks_on_shutdown()
@ -1479,23 +1474,38 @@ async def initialize_shared_aiohttp_session() -> "ClientSession | None":
_initialize_shared_aiohttp_session: Final = initialize_shared_aiohttp_session
async def _connect_to_count_stored_values() -> SupportsRawQueries:
async def _connect_to_count_stored_values(resources: AsyncExitStack) -> SupportsRawQueries:
client: Final = prisma_client or PrismaClient(
database_url=str(get_secret("DATABASE_URL")), proxy_logging_obj=proxy_logging_obj
)
if prisma_client is None:
resources.push_async_callback(client.aclose)
await client.connect()
return client.writer_db
async def _own_database(resources: AsyncExitStack, client: PrismaClient | None) -> database_runtime.Database | None:
if client is None:
return None
database: Final = await database_runtime.Database.open(client)
resources.push_async_callback(database.aclose)
return database
def _forget_prisma_client() -> None:
global prisma_client # noqa: PLW0603 # routes still read this global until they move to DatabaseDep
prisma_client = None # rebind-ok: a closed client must not survive into the next lifespan entry
@asynccontextmanager
async def proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanState, None]:
with GracefulShutdownManager.capture_sigterm():
async with _proxy_startup_event(app) as state:
async with AsyncExitStack() as resources, _proxy_lifespan(app, resources) as state:
yield state
@asynccontextmanager
async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanState, None]:
async def _proxy_lifespan(app: FastAPI, resources: AsyncExitStack) -> AsyncGenerator[ProxyLifespanState, None]:
global \
prisma_client, \
master_key, \
@ -1514,6 +1524,9 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
shared_aiohttp_session
import json
resources.callback(_forget_prisma_client)
existing_database: Final = await _own_database(resources, prisma_client)
init_verbose_loggers()
running_loop: Final = asyncio.get_running_loop()
@ -1592,21 +1605,24 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
if decoded_worker_config is not None:
await initialize_from_worker_config(decoded_worker_config)
enforce_master_key_boot_verdict(
await with_stored_secrets_counted(
master_key_boot_verdict(
master_key=master_key,
environment_master_key=os.getenv(MASTER_KEY_ENV_VAR),
general_settings=general_settings,
config_file_path=user_config_file_path,
override_env_is_on=get_secret_bool(WEAK_OR_UNSET_MASTER_KEY_OVERRIDE_ENV_VAR) is True,
salt_key_is_set=os.getenv(SALT_KEY_ENV_VAR) is not None,
database_is_configured=prisma_client is not None or get_secret("DATABASE_URL", None) is not None,
async with AsyncExitStack() as boot_database:
enforce_master_key_boot_verdict(
await with_stored_secrets_counted(
master_key_boot_verdict(
master_key=master_key,
environment_master_key=os.getenv(MASTER_KEY_ENV_VAR),
general_settings=general_settings,
config_file_path=user_config_file_path,
override_env_is_on=get_secret_bool(WEAK_OR_UNSET_MASTER_KEY_OVERRIDE_ENV_VAR) is True,
salt_key_is_set=os.getenv(SALT_KEY_ENV_VAR) is not None,
database_is_configured=prisma_client is not None or get_secret("DATABASE_URL", None) is not None,
),
count_values_encrypted_with=partial(
count_values_encrypted_with_or_none, partial(_connect_to_count_stored_values, boot_database)
),
),
count_values_encrypted_with=partial(count_values_encrypted_with_or_none, _connect_to_count_stored_values),
),
announce=announce_on_stderr_at_exit,
)
announce=announce_on_stderr_at_exit,
)
# check if DATABASE_URL in environment - load from there
if prisma_client is None:
@ -1619,6 +1635,8 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
)
)
database: Final = existing_database or await _own_database(resources, prisma_client)
await migrate_if_requested(
environ=os.environ,
master_key=master_key,
@ -1628,15 +1646,16 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
)
if prisma_client is not None:
startup_prisma: Final = prisma_client
async def _run_pw_migration():
async def _run_pw_migration() -> None:
try:
result: Final = await migrate_passwords_to_scrypt_async(prisma_client)
result: Final = await migrate_passwords_to_scrypt_async(startup_prisma)
verbose_proxy_logger.info("Password migration: %s", result)
except Exception as e:
verbose_proxy_logger.warning("Password migration skipped: %s", e)
asyncio.create_task(_run_pw_migration())
await resources.enter_async_context(database_runtime.background_task(_run_pw_migration()))
async def _run_agent_grant_id_migration() -> None:
from litellm.proxy.agent_endpoints.agent_registry import (
@ -1647,7 +1666,7 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
for attempt in range(3):
try:
result = await global_agent_registry.migrate_legacy_grant_ids(
table=object_permission_table(prisma_client)
table=object_permission_table(startup_prisma)
)
if result.rewritten:
verbose_proxy_logger.info(
@ -1667,7 +1686,7 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
if attempt < 2:
await asyncio.sleep(5)
asyncio.create_task(_run_agent_grant_id_migration())
await resources.enter_async_context(database_runtime.background_task(_run_agent_grant_id_migration()))
## A coordination_redis block saved from the admin UI lives in the database,
## which is only reachable once the prisma client exists. Apply it here, before
@ -1853,7 +1872,7 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
enabled=tracing_enabled,
settings=tracing_settings,
) as receiver:
state: Final[ProxyLifespanState] = {"tracing_receiver": receiver}
state: Final[ProxyLifespanState] = {"tracing_receiver": receiver, "database": database}
from litellm.proxy.admin_mcp import admin_mcp_lifespan
telemetry_settings: Final = load_telemetry_settings()
@ -1907,30 +1926,6 @@ async def _proxy_startup_event(app: FastAPI) -> AsyncGenerator[ProxyLifespanStat
except Exception as e:
verbose_proxy_logger.error("Error closing shared aiohttp session: %s", e)
# Shutdown event - stop RDS IAM token refresh background task
if (
prisma_client is not None
and hasattr(prisma_client, "db")
and hasattr(prisma_client.db, "stop_token_refresh_task")
):
try:
await prisma_client.db.stop_token_refresh_task()
except Exception as e:
verbose_proxy_logger.error("Error stopping token refresh task: %s", e)
# Shutdown event - stop Prisma DB health watchdog task
if prisma_client is not None and hasattr(prisma_client, "stop_db_health_watchdog_task"):
try:
await prisma_client.stop_db_health_watchdog_task()
except Exception as e:
verbose_proxy_logger.error("Error stopping DB health watchdog task: %s", e)
if prisma_client is not None and hasattr(prisma_client, "stop_view_setup_task"):
try:
await prisma_client.stop_view_setup_task()
except Exception as e:
verbose_proxy_logger.error("Error stopping the spend view setup task: %s", e)
await _drain_spend_event_producer_on_shutdown()
if telemetry_start is not None:
telemetry_start.cancel()
@ -4665,10 +4660,6 @@ def _write_health_state_to_router_cache(
is on.
"""
from litellm.proxy.health_check import build_deployment_health_states
from litellm.router_utils.cooldown_handlers import set_cooldown_deployments
from litellm.router_utils.router_callbacks.track_deployment_metrics import (
increment_deployment_failures_for_current_minute,
)
_exceptions: Final[dict] = exceptions_by_model_id or {}
@ -4720,18 +4711,7 @@ def _write_health_state_to_router_cache(
):
continue
increment_deployment_failures_for_current_minute(
litellm_router_instance=llm_router,
deployment_id=model_id,
)
set_cooldown_deployments(
litellm_router_instance=llm_router,
original_exception=original_exception,
exception_status=exception_status,
deployment=model_id,
time_to_cooldown=router_settings.cooldown_time,
)
llm_router.health().record_failure(model_id, original_exception, exception_status)
except Exception as e:
verbose_proxy_logger.warning("Failed to write health state to router cache: %s", str(e))
@ -7375,9 +7355,7 @@ class ProxyConfig:
router_settings: Final = config.get("router_settings", None)
if router_settings and isinstance(router_settings, dict):
available_args: Final = [
x for x in litellm.Router.get_valid_args() if x not in ROUTER_SETTINGS_MANAGED_OUTSIDE_CONFIG
]
available_args: Final = [x for x in router_init_args() if x not in ROUTER_SETTINGS_MANAGED_OUTSIDE_CONFIG]
for k, v in router_settings.items():
if k in available_args:
@ -9601,7 +9579,6 @@ class ProxyConfig:
SearchToolRegistry,
keep_loaded_search_tools_that_do_not_decrypt,
)
from litellm.router_utils.search_api_router import SearchAPIRouter
try:
db_search_tools: Final = keep_loaded_search_tools_that_do_not_decrypt(
@ -9625,7 +9602,21 @@ class ProxyConfig:
)
if llm_router is not None:
await SearchAPIRouter.update_router_search_tools(router_instance=llm_router, search_tools=search_tools)
llm_router.apply(
ChangeSet(
assign={
"search_tools": [
{
"search_tool_id": tool.get("search_tool_id"),
"search_tool_name": tool.get("search_tool_name"),
"litellm_params": tool.get("litellm_params", {}),
"search_tool_info": tool.get("search_tool_info"),
}
for tool in search_tools
]
}
)
)
verbose_proxy_logger.info("Successfully loaded %s search tool(s) into router", len(search_tools))
else:
verbose_proxy_logger.debug(
@ -12124,17 +12115,20 @@ class ProxyStartupEvent:
proxy_logging_obj: ProxyLogging,
user_api_key_cache: UserApiKeyCache,
) -> PrismaClient | None:
"""
- Sets up prisma client
- Adds necessary views to proxy
"""
if database_url is None:
return None
client: Final = PrismaClient(database_url=database_url, proxy_logging_obj=proxy_logging_obj)
async with AsyncExitStack() as setup_resources:
setup_resources.push_async_callback(client.aclose)
connected: Final = await cls._initialize_prisma_client(client)
if connected is not None:
setup_resources.pop_all()
return connected
@staticmethod
async def _initialize_prisma_client(prisma_client: PrismaClient) -> PrismaClient | None:
connected_client: PrismaClient | None = None
try:
if database_url is None:
return None
prisma_client = PrismaClient(database_url=database_url, proxy_logging_obj=proxy_logging_obj)
try:
await prisma_client.connect()
except Exception as e:
@ -16154,7 +16148,7 @@ def _enrich_model_info_with_litellm_data(
if debug is True:
_openai_client = "None"
if llm_router is not None:
_openai_client = llm_router._get_client(deployment=model, kwargs={}, client_type="async") or "None"
_openai_client = llm_router.registry().client(model, client_type="async") or "None"
else:
_openai_client = "llm_router_is_None"
openai_client: Final = str(_openai_client)
@ -18604,7 +18598,7 @@ async def async_queue_request(
router: Final = llm_router
data.update(
router.priority_header_update(
router.health().priority_header_fields(
model=data["model"], kwargs=data, header=request.headers.get(SCHEDULER_PRIORITY_HEADER)
)
)
@ -19753,7 +19747,7 @@ async def update_config(
raw_router_settings: Final = request_body.get("router_settings")
if isinstance(raw_router_settings, dict):
supported_router_settings: Final = RUNTIME_UPDATABLE_ROUTER_SETTINGS | (
frozenset(litellm.Router.get_valid_args()) - ROUTER_SETTINGS_MANAGED_OUTSIDE_CONFIG
frozenset(router_init_args()) - ROUTER_SETTINGS_MANAGED_OUTSIDE_CONFIG
)
unsupported_router_settings: Final = sorted(set(raw_router_settings) - supported_router_settings)
if unsupported_router_settings:

5
litellm/proxy/route_llm_request.py
View file

@ -16,8 +16,9 @@ from pydantic import BeforeValidator, TypeAdapter, ValidationError
import litellm
from litellm.litellm_core_utils.core_helpers import normalize_drop_params
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.routing_common_utils import is_proxy_admin_request
from litellm.proxy._types import ProxyException, UserAPIKeyAuth
from litellm.router_utils.common_utils import is_proxy_admin_request
from litellm.router_api import router_init_args
# Client-supplied params that make the router or the call path fabricate a
# failure or a delay instead of calling the provider. The ``mock_testing_*``
@ -93,7 +94,7 @@ async def _route_user_config_request(data: dict, route_type: str):
# Filter router_config to only include valid Router.__init__ arguments
# This prevents TypeError when invalid parameters are stored in the database
valid_args: Final = litellm.Router.get_valid_args()
valid_args: Final = router_init_args()
filtered_config: Final = {k: v for k, v in router_config.items() if k in valid_args}
logging_obj: Final = _request_logging_obj(JSON_OBJECT_ADAPTER.validate_python(data), route_type)

4
litellm/proxy/spend_tracking/input_tokens.py
View file

@ -21,6 +21,7 @@ from typing import TYPE_CHECKING, Final
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.constants import OPENAI_CHAT_PDF_FILE_TOKENS, PDF_DATA_URL_PREFIX
from litellm.litellm_core_utils.deployment_pricing import deployment_pricing_models, str_keyed
from litellm.litellm_core_utils.token_counter import (
deployment_openai_model,
pdf_page_pricing,
@ -31,7 +32,6 @@ from litellm.responses.litellm_completion_transformation.compaction_items import
billable_input_since_newest_compaction,
holds_compaction_summary,
)
from litellm.router_utils.deployment_pricing import deployment_pricing_models, str_keyed
from litellm.rust_bridge import runtime
from litellm.rust_bridge.catalog import Route, RouteContext
from litellm.rust_bridge.token_counter import (
@ -162,7 +162,7 @@ def models_counted_for(model: str, llm_router: Router | None, team_id: str | Non
def _routed_deployments(model: str, llm_router: Router, team_id: str | None) -> Sequence[object]:
registry: Final = llm_router.registry()
resolved: Final = registry.resolve(model, kind="alias") or model
routing_group: Final = llm_router.get_routing_group(resolved)
routing_group: Final = registry.routing_group(resolved)
members: Final = routing_group.models if routing_group is not None else (resolved,)
named: Final = tuple(
chain.from_iterable(registry.deployments(member, source="all", team_id=team_id) for member in members)

31
litellm/proxy/ui_crud_endpoints/user_banner_dependencies.py Normal file
View file

@ -0,0 +1,31 @@
from collections.abc import Coroutine
from typing import Annotated, Protocol, TypeAlias
from fastapi import Depends
from litellm.models.user_banner import UserBanner
from litellm.proxy._types import LitellmTableNames, UserAPIKeyAuth
from litellm.repositories.user_banner_repository import USER_BANNER_ROW_ID
class UserBannerAudit(Protocol):
def __call__(self, before: UserBanner, after: UserBanner, user: UserAPIKeyAuth) -> Coroutine[None, None, None]: ...
async def get_user_banner_audit() -> UserBannerAudit:
from litellm.proxy.proxy_server import create_config_audit_log
async def audit(before: UserBanner, after: UserBanner, user: UserAPIKeyAuth) -> None:
await create_config_audit_log(
param_name=USER_BANNER_ROW_ID,
action="updated",
before_value=before.model_dump(),
after_value=after.model_dump(),
user_api_key_dict=user,
table_name=LitellmTableNames.UI_SETTINGS_TABLE_NAME,
)
return audit
UserBannerAuditDep: TypeAlias = Annotated[UserBannerAudit, Depends(get_user_banner_audit)]

96
litellm/proxy/ui_crud_endpoints/user_banner_endpoints.py
View file

@ -1,88 +1,43 @@
import asyncio
import json
from typing import Annotated, Final, Literal
from typing import Annotated, Final
from fastapi import APIRouter, Depends, HTTPException
from pydantic import Field, ValidationError, model_validator
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
from pydantic import ConfigDict
from litellm._uuid import uuid4
from litellm.proxy._types import LitellmTableNames, LitellmUserRoles, UserAPIKeyAuth
from litellm.models.user_banner import UserBanner, UserBannerUpdate
from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.repositories.user_banner_repository import USER_BANNER_ROW_ID, UserBannerRepository
from litellm.proxy.deps.repositories import UserBannerRepositoryDep
from litellm.proxy.ui_crud_endpoints.user_banner_dependencies import (
UserBannerAuditDep,
)
from litellm.types.llms.base import LiteLLMBaseModel
router: Final = APIRouter()
USER_BANNER_MAX_MESSAGE_LENGTH: Final = 4000
UserBannerSeverity = Literal["info", "warning", "error"]
class UserBannerUpdate(LiteLLMBaseModel):
enabled: bool = Field(
default=False,
description="If true, the banner is shown to all authenticated dashboard users.",
)
message: str = Field(
default="",
max_length=USER_BANNER_MAX_MESSAGE_LENGTH,
description="Banner text shown to dashboard users. Markdown is supported.",
)
severity: UserBannerSeverity = Field(
default="info",
description="Visual style of the banner.",
)
@model_validator(mode="after")
def _require_message_when_enabled(self) -> "UserBannerUpdate":
if self.enabled and not self.message.strip():
raise ValueError("message must be non-empty when the banner is enabled")
return self
class UserBanner(UserBannerUpdate):
revision: str = Field(
default="",
description=(
"Server-stamped opaque publish identity; a fresh value is generated on every "
"update so clients re-surface dismissed banners on republish."
),
)
class UpdateUserBannerResponse(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
message: str
banner: UserBanner
def parse_user_banner(raw_settings: object) -> UserBanner:
if raw_settings is None:
return UserBanner()
try:
parsed: Final = json.loads(raw_settings) if isinstance(raw_settings, str) else raw_settings
return UserBanner.model_validate(parsed)
except (json.JSONDecodeError, ValidationError):
return UserBanner()
@router.get(
"/get/user_banner",
tags=["UI Settings"],
dependencies=[Depends(user_api_key_auth)],
response_model=UserBanner,
)
async def get_user_banner() -> UserBanner:
async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner:
"""
Get the admin-published dashboard banner.
Readable by any authenticated user; rendered on every dashboard page.
"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
if banners is None:
return UserBanner()
raw_settings: Final = await UserBannerRepository(prisma_client).get_raw_settings()
return parse_user_banner(raw_settings)
return await banners.get_banner()
@router.patch(
@ -93,21 +48,21 @@ async def get_user_banner() -> UserBanner:
async def update_user_banner(
banner_update: UserBannerUpdate,
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
banners: UserBannerRepositoryDep,
audit: UserBannerAuditDep,
background_tasks: BackgroundTasks,
) -> UpdateUserBannerResponse:
"""
Publish, edit, or unpublish the dashboard banner.
Only proxy admins are allowed to modify it.
"""
from litellm.proxy.proxy_server import create_config_audit_log, prisma_client
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
raise HTTPException(status_code=403, detail="Only proxy admins can update the user banner.")
if prisma_client is None:
if banners is None:
raise HTTPException(status_code=500, detail="Database not connected. Please connect a database.")
repository: Final = UserBannerRepository(prisma_client)
before: Final = parse_user_banner(await repository.get_raw_settings())
before: Final = await banners.get_banner()
banner: Final = UserBanner(
enabled=banner_update.enabled,
message=banner_update.message,
@ -115,17 +70,8 @@ async def update_user_banner(
revision=uuid4().hex,
)
await repository.upsert_settings(json.dumps(banner.model_dump()))
await banners.save_banner(banner)
asyncio.create_task(
create_config_audit_log(
param_name=USER_BANNER_ROW_ID,
action="updated",
before_value=before.model_dump(),
after_value=banner.model_dump(),
user_api_key_dict=user_api_key_dict,
table_name=LitellmTableNames.UI_SETTINGS_TABLE_NAME,
)
)
background_tasks.add_task(audit, before, banner, user_api_key_dict)
return UpdateUserBannerResponse(message="User banner updated successfully", banner=banner)

252
litellm/proxy/utils.py
View file

@ -160,10 +160,12 @@ from litellm.litellm_core_utils.core_helpers import (
independent_snapshot,
is_expected_client_error,
)
from litellm.litellm_core_utils.deployment_pricing import pricing_models
from litellm.litellm_core_utils.duration_parser import get_budget_window_start
from litellm.litellm_core_utils.env_utils import get_env_float, get_env_int
from litellm.litellm_core_utils.litellm_logging import BilledRoundsDispatch, Logging, deployment_pricing_model_info
from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
from litellm.litellm_core_utils.routing_common_utils import resolve_model_group_alias
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
from litellm.litellm_core_utils.served_output_texts import (
@ -278,8 +280,6 @@ from litellm.repositories.user_repository import UserRepository
from litellm.repositories.verification_token_repository import (
VerificationTokenRepository,
)
from litellm.router_utils.common_utils import resolve_model_group_alias
from litellm.router_utils.deployment_pricing import pricing_models
from litellm.secret_managers.main import str_to_bool
from litellm.types.integrations.slack_alerting import DEFAULT_ALERT_TYPES
from litellm.types.llms.openai import ResponsesAPIResponse
@ -714,6 +714,120 @@ def pipeline_managed_guardrail_names(
)
def _ran_or_runs_outside_pipelines(
callback: CustomGuardrail | None,
data: Mapping[str, object],
pipeline_modes: frozenset[str],
) -> bool:
return callback is not None and (
callback.pre_call_hook_ran_or_raised(data)
or callback.deployment_hook_ran(data)
or any(
callback._event_hook_is_event_type(hook) # pyright: ignore[reportPrivateUsage] # no public hook check
for hook in GuardrailEventHooks
if hook.value not in pipeline_modes
)
)
def _runs_only_in_pipelines(
name: str, data: Mapping[str, object], pre_call: frozenset[str], post_call: frozenset[str]
) -> bool:
return not _ran_or_runs_outside_pipelines(
PipelineExecutor.find_guardrail_callback(name),
data,
frozenset(mode for mode, names in (("pre_call", pre_call), ("post_call", post_call)) if name in names),
)
def _request_steps_that_have_not_run(data: Mapping[str, object]) -> frozenset[str]:
"""Pre_call policy pipeline steps a pipeline skipped, whose guardrail nothing else ran or runs for this request."""
pre_call: Final = pipeline_managed_guardrail_names(data, "pre_call")
return frozenset(name for name in pre_call if _runs_only_in_pipelines(name, data, pre_call, frozenset()))
def _steps_that_ran(result: PipelineExecutionResult) -> frozenset[str]:
return frozenset(step.guardrail_name for step in result.step_results if step.outcome != "skip")
def _whole_response_log_waits_for_its_pipelines(data: Mapping[str, object], call_type: str) -> bool:
"""
The success log is parked until the response hooks finish, and the headers are built
after them. The LLM pass-through route builds its headers first, and the generateContent
adapter logs its nested completion before the response hooks run.
"""
logging_obj: Final = data.get("litellm_logging_obj")
return (
call_type not in (CallTypes.allm_passthrough_route.value, CallTypes.agenerate_content.value)
and isinstance(logging_obj, Logging)
and logging_obj.enqueue_deferred_logging is not None
)
def _stream_log_waits_for_its_pipelines(data: Mapping[str, object]) -> bool:
return getattr(data.get("litellm_logging_obj"), "_on_deferred_stream_complete", None) is not None
def _pipelines_settled_by(
pipelines: Sequence[tuple[str, "GuardrailPipeline"]], index: int, result: PipelineExecutionResult
) -> Sequence[tuple[str, "GuardrailPipeline"]]:
"""This pipeline, plus every later response pipeline when its verdict ends the request before they run."""
if result.terminal_action == "allow":
return pipelines[index : index + 1]
return (
pipelines[index],
*((policy_name, pipeline) for policy_name, pipeline in pipelines[index + 1 :] if pipeline.mode == "post_call"),
)
def _steps_later_pipelines_still_run(
pipelines: Sequence[tuple[str, "GuardrailPipeline"]], index: int, result: PipelineExecutionResult
) -> frozenset[str]:
"""The steps of the later response pipelines, which still run when this pipeline's verdict lets the request on."""
if result.terminal_action != "allow":
return frozenset()
later: Final = tuple(
(policy_name, pipeline) for policy_name, pipeline in pipelines[index + 1 :] if pipeline.mode == "post_call"
)
return _pipeline_step_guardrail_names(later)
def _drop_response_steps_that_did_not_run(
data: dict[str, object], # mutable-ok: the request's applied guardrails are trimmed in place
pipelines: Sequence[tuple[str, "GuardrailPipeline"]],
steps_to_keep: frozenset[str],
) -> None:
"""
Drop from the applied guardrails the steps of these response pipelines outside ``steps_to_keep``,
unless their guardrail also runs outside pipelines. Every step is named up front, so a
log built before the pipeline finished, such as a cache hit's, still lists the steps
that ran. The list is trimmed in place, since a route's logging object may hold a
shallow copy of the metadata.
"""
pre_call: Final = pipeline_managed_guardrail_names(data, "pre_call")
post_call: Final = _pipeline_step_guardrail_names(pipelines)
not_run: Final = frozenset(
name for name in post_call - steps_to_keep if _runs_only_in_pipelines(name, data, pre_call, post_call)
)
if not not_run:
return
for bucket_key in ("metadata", "litellm_metadata"):
_drop_applied_guardrails_in_place(data.get(bucket_key), not_run)
def _drop_applied_guardrails_in_place(bucket: object, names: frozenset[str]) -> None:
if not isinstance(bucket, dict):
return
metadata: Final = cast("dict[str, object]", bucket) # cast-ok: metadata buckets are str-keyed
applied: Final = metadata.get("applied_guardrails")
if not isinstance(applied, list):
return
entries: Final = cast("list[object]", applied) # cast-ok: entries are only compared with names
entries[:] = [name for name in entries if name not in names]
if not entries:
metadata.pop("applied_guardrails")
def _partition_post_call_callbacks() -> tuple[tuple[CustomGuardrail, ...], tuple[CustomLogger, ...]]:
resolved: Final = tuple(
litellm.litellm_core_utils.litellm_logging.get_custom_logger_compatible_class(
@ -2711,8 +2825,12 @@ class ProxyLogging:
data.pop("prompt_version", None)
data.pop("prompt_environment", None)
def _process_guardrail_metadata(self, data: dict) -> None:
"""Process guardrails from metadata and add to applied_guardrails."""
def _process_guardrail_metadata(self, data: dict, *, skip_guardrails: bool = False) -> None:
"""
Process guardrails from metadata and add to applied_guardrails, leaving out the pre_call
policy pipeline steps that have not run. With ``skip_guardrails`` the pipelines run on
another request, so every step is added.
"""
from litellm.proxy.common_utils.callback_utils import (
add_guardrail_to_applied_guardrails_header,
)
@ -2736,8 +2854,17 @@ class ProxyLogging:
if not isinstance(applied_guardrails, list):
applied_guardrails = []
not_run: Final = (
frozenset[str]()
if skip_guardrails
else _request_steps_that_have_not_run(cast(dict[str, object], data)) # cast-ok: request payload
)
for guardrail_name in guardrails_in_metadata:
if isinstance(guardrail_name, str) and guardrail_name not in applied_guardrails:
if (
isinstance(guardrail_name, str)
and guardrail_name not in applied_guardrails
and guardrail_name not in not_run
):
add_guardrail_to_applied_guardrails_header(request_data=data, guardrail_name=guardrail_name)
async def _maybe_execute_pipelines(
@ -2769,7 +2896,10 @@ class ProxyLogging:
return data, None
current_response = response # rebind-ok: chains each pipeline's replacement response into the next
for policy_name, pipeline in pipelines:
steps_that_ran: frozenset[str] = frozenset() # rebind-ok: a later pipeline keeps a step an earlier one ran
request: Final = cast(dict[str, object], data) # cast-ok: request payload, updated in place by every pipeline
log_waits: Final = event_hook == "post_call" and _whole_response_log_waits_for_its_pipelines(request, call_type)
for index, (policy_name, pipeline) in enumerate(pipelines):
if pipeline.mode != event_hook:
continue
@ -2785,6 +2915,14 @@ class ProxyLogging:
raw_request_snapshot=raw_request_snapshot,
)
steps_that_ran = steps_that_ran | _steps_that_ran(result)
if log_waits:
_drop_response_steps_that_did_not_run(
request,
_pipelines_settled_by(pipelines, index, result),
steps_that_ran | _steps_later_pipelines_still_run(pipelines, index, result),
)
data = self._handle_pipeline_result(
result=result,
data=data,
@ -3027,7 +3165,7 @@ class ProxyLogging:
and (guardrails_only or not caps.has_pre_call_override)
):
if data is not None:
self._process_guardrail_metadata(data)
self._process_guardrail_metadata(data, skip_guardrails=skip_guardrails)
return data
parallel_guardrails: Final[tuple[CustomGuardrail, ...]] = (
@ -3131,17 +3269,17 @@ class ProxyLogging:
)
if data is not None:
self._process_guardrail_metadata(data)
self._process_guardrail_metadata(data, skip_guardrails=skip_guardrails)
return data
except SensitiveDataRouteException as e:
data = await self._handle_sensitive_data_route_exception(e, data, user_api_key_dict)
if data is not None:
self._process_guardrail_metadata(data)
self._process_guardrail_metadata(data, skip_guardrails=skip_guardrails)
return data
except Exception:
if data is not None:
self._process_guardrail_metadata(data)
self._process_guardrail_metadata(data, skip_guardrails=skip_guardrails)
raise
async def _run_parallel_pre_call_guardrails(
@ -4736,7 +4874,8 @@ class ProxyLogging:
call_type, endpoint_translation = translation
for policy_name, pipeline in pipelines:
steps_that_ran: frozenset[str] = frozenset() # rebind-ok: a later pipeline keeps a step an earlier one ran
for index, (policy_name, pipeline) in enumerate(pipelines):
result: PipelineExecutionResult = await PipelineExecutor.execute_steps(
steps=pipeline.steps,
mode="post_call",
@ -4747,6 +4886,13 @@ class ProxyLogging:
streaming_chunks=buffered,
endpoint_translation=endpoint_translation,
)
steps_that_ran = steps_that_ran | _steps_that_ran(result)
if _stream_log_waits_for_its_pipelines(request_data):
_drop_response_steps_that_did_not_run(
request_data,
_pipelines_settled_by(pipelines, index, result),
steps_that_ran | _steps_later_pipelines_still_run(pipelines, index, result),
)
try:
ProxyLogging._handle_pipeline_result(
result, data=request_data, policy_name=policy_name, original_response=buffered
@ -5197,6 +5343,8 @@ async def prefetch_config_params(prisma_client: "PrismaClient | None", param_nam
_WRITER_WRITABILITY_PROBE_SQL: Final = "SELECT current_setting('transaction_read_only') AS transaction_read_only"
_WRITER_WRITABILITY_PROBE_ROWS: Final = TypeAdapter(list[dict[str, object]])
_READ_ONLY_RECREATE_BACKOFF_CAP_SECONDS: Final = 600
_SHUTDOWN_RECONNECT_WAIT_SECONDS: Final = 5.0
_SHUTDOWN_ENGINE_EXIT_WAIT_SECONDS: Final = 5.0
class _ForcedRecreateDeclined(Exception):
@ -5281,6 +5429,7 @@ class PrismaClient:
database_url: str,
proxy_logging_obj: ProxyLogging,
http_client: "HttpConfig | None" = None,
engine_exit_wait_seconds: float = _SHUTDOWN_ENGINE_EXIT_WAIT_SECONDS,
):
## init logging object
self.baseline_accounting_transactions: list[BaselineAccountingRecord] = []
@ -5382,6 +5531,9 @@ class PrismaClient:
else:
self.db = writer_wrapper # Client to connect to Prisma db
self._db_reconnect_lock = asyncio.Lock()
self._closing: bool = False
self._engine_exit_wait_seconds: Final = engine_exit_wait_seconds
self._engine_tasks: frozenset[asyncio.Task[object]] = frozenset()
self._db_health_watchdog_task: asyncio.Task | None = None
self._view_setup_task: asyncio.Task[_ViewSetupOutcome] | None = None
self._db_last_reconnect_attempt_ts: float = 0.0
@ -5423,6 +5575,7 @@ class PrismaClient:
self._watching_engine: bool = False
self._engine_confirmed_dead: bool = False
self._engine_wait_thread: threading.Thread | None = None
self._engine_wait_threads: tuple[threading.Thread, ...] = ()
verbose_proxy_logger.debug("Success - Created Prisma Client")
@property
@ -6631,6 +6784,14 @@ class PrismaClient:
start_time: Final = time.time()
try:
await self.db.disconnect()
if self._closing:
await asyncio.gather(
*(
asyncio.to_thread(thread.join, self._engine_exit_wait_seconds)
for thread in self._engine_wait_threads
)
)
self._engine_wait_threads = ()
except Exception as e:
import traceback
@ -6649,6 +6810,54 @@ class PrismaClient:
)
raise e
def _spawn_engine_task(self, work: Coroutine[object, object, object]) -> None:
if self._closing:
work.close()
return
task: Final = asyncio.create_task(work)
self._engine_tasks = self._engine_tasks | {task}
task.add_done_callback(self._engine_task_finished)
def _engine_task_finished(self, task: asyncio.Task[object]) -> None:
self._engine_tasks = self._engine_tasks - {task}
if not task.cancelled() and (error := task.exception()) is not None:
verbose_proxy_logger.error("Prisma engine background task failed: %s", error)
async def _stop_engine_tasks(self) -> None:
tasks: Final = tuple(self._engine_tasks)
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
await self._wait_for_inflight_reconnect()
async def _wait_for_inflight_reconnect(self) -> None:
waiter: Final = asyncio.ensure_future(self._db_reconnect_lock.acquire())
waiter.add_done_callback(lambda acquired: acquired.cancelled() or self._db_reconnect_lock.release())
done, _pending = await asyncio.wait({waiter}, timeout=_SHUTDOWN_RECONNECT_WAIT_SECONDS)
if waiter not in done:
waiter.cancel()
verbose_proxy_logger.warning("Shutdown stopped waiting for an in-flight DB reconnect")
async def aclose(self) -> None:
verbose_proxy_logger.debug("Disconnecting from Prisma")
try:
await self.stop_background_tasks()
except Exception:
verbose_proxy_logger.exception("Stopping database background tasks failed")
try:
await self.disconnect()
except Exception:
verbose_proxy_logger.exception("Database disconnect failed")
async def stop_background_tasks(self) -> None:
self._closing = True
self._stop_engine_watcher()
async with contextlib.AsyncExitStack() as cleanup:
cleanup.push_async_callback(self._stop_engine_tasks)
cleanup.push_async_callback(self.db.stop_token_refresh_task)
cleanup.push_async_callback(self.stop_view_setup_task)
cleanup.push_async_callback(self.stop_db_health_watchdog_task)
def _get_engine_pid(self) -> int:
"""Get the PID of the writer's engine subprocess, or 0 if unavailable.
@ -6742,7 +6951,7 @@ class PrismaClient:
self._engine_confirmed_dead = True
self._reap_all_zombies()
self._cleanup_engine_watcher()
asyncio.create_task(
self._spawn_engine_task(
self.attempt_db_reconnect(
reason="engine_process_death",
force=True,
@ -6762,6 +6971,9 @@ class PrismaClient:
name=f"prisma-engine-waitpid-{pid}",
)
thread.start()
self._engine_wait_threads = tuple(thread for thread in self._engine_wait_threads if thread.is_alive()) + (
thread,
)
self._engine_wait_thread = thread
return True
@ -6823,7 +7035,7 @@ class PrismaClient:
self._engine_confirmed_dead = True
self._reap_all_zombies()
self._cleanup_engine_watcher()
asyncio.create_task(
self._spawn_engine_task(
self.attempt_db_reconnect(
reason="engine_process_death",
force=True,
@ -6885,7 +7097,7 @@ class PrismaClient:
self._engine_confirmed_dead = True
self._reap_all_zombies()
self._cleanup_engine_watcher()
asyncio.create_task(
self._spawn_engine_task(
self.attempt_db_reconnect(
reason="engine_process_death",
force=True,
@ -6958,7 +7170,7 @@ class PrismaClient:
waitpid thread nor pidfd are available.
"""
if self._watching_engine or self._engine_pidfd >= 0 or self._engine_wait_thread is not None:
if self._closing or self._watching_engine or self._engine_pidfd >= 0 or self._engine_wait_thread is not None:
return
pid: Final = self._get_engine_pid()
if pid == 0:
@ -6985,7 +7197,7 @@ class PrismaClient:
pid,
)
self._watching_engine = True
asyncio.create_task(self._poll_engine_proc())
self._spawn_engine_task(self._poll_engine_proc())
def _stop_engine_watcher(self) -> None:
"""Stop watching the engine process and clean up all resources."""
@ -7007,7 +7219,7 @@ class PrismaClient:
"""
self._engine_confirmed_dead = False
self._cleanup_engine_watcher()
asyncio.create_task(self._start_engine_watcher())
self._spawn_engine_task(self._start_engine_watcher())
async def _run_reconnect_cycle(
self,
@ -7214,6 +7426,8 @@ class PrismaClient:
force_recreate: bool = False,
stale_read_engine: "_StaleReadEngine | None" = None,
) -> bool:
if self._closing:
return False
now: Final = time.time()
if (
force is False
@ -7404,7 +7618,7 @@ class PrismaClient:
- A periodic SELECT 1 probe that triggers reconnect on network/connection failure.
- A process-level watcher that detects engine death via waitpid thread, pidfd, or os.kill polling.
"""
if self._db_health_watchdog_enabled is not True:
if self._closing or self._db_health_watchdog_enabled is not True:
verbose_proxy_logger.debug("Prisma DB health watchdog disabled via PRISMA_HEALTH_WATCHDOG_ENABLED")
return
if self._db_health_watchdog_task is not None:
@ -7438,7 +7652,7 @@ class PrismaClient:
verbose_proxy_logger.info("Stopped Prisma DB health watchdog")
def start_view_setup_task(self) -> None:
if self._view_setup_task is not None:
if self._closing or self._view_setup_task is not None:
return
self._view_setup_task = asyncio.create_task(self._run_view_setup())

41
litellm/repositories/database.py Normal file
View file

@ -0,0 +1,41 @@
import asyncio
from collections.abc import AsyncGenerator, Coroutine
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import TYPE_CHECKING, Final
from litellm.rust_bridge._native import NativeDatabase
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient
@dataclass(frozen=True, slots=True)
class Database:
prisma: "PrismaClient"
native: NativeDatabase
@classmethod
async def open(cls, prisma: "PrismaClient") -> "Database":
try:
native: Final = await NativeDatabase.connect()
except BaseException:
await prisma.aclose()
raise
return cls(prisma=prisma, native=native)
async def aclose(self) -> None:
try:
await self.prisma.aclose()
finally:
await self.native.close()
@asynccontextmanager
async def background_task(work: Coroutine[object, object, None]) -> AsyncGenerator[None, None]:
task: Final = asyncio.create_task(work)
try:
yield
finally:
task.cancel()
await asyncio.gather(task, return_exceptions=True)

26
litellm/repositories/deployment_warning_ignore_repository.py
View file

@ -1,5 +1,12 @@
import json
from collections.abc import Mapping
from types import MappingProxyType
from typing import TYPE_CHECKING, Final
from pydantic import TypeAdapter, ValidationError
from litellm._logging import verbose_proxy_logger
from litellm.models.deployment_warning import AdminIgnore
from litellm.repositories.table_repositories import PrismaTableRepository
if TYPE_CHECKING:
@ -7,15 +14,28 @@ if TYPE_CHECKING:
DEPLOYMENT_WARNING_IGNORES_ROW_ID: Final = "deployment_warning_ignores"
_IGNORES_ADAPTER: Final[TypeAdapter[Mapping[str, AdminIgnore]]] = TypeAdapter(Mapping[str, AdminIgnore])
def parse_ignores(raw: object) -> Mapping[str, AdminIgnore]:
if raw is None:
return MappingProxyType({})
try:
return MappingProxyType(_IGNORES_ADAPTER.validate_python(json.loads(raw) if isinstance(raw, str) else raw))
except (json.JSONDecodeError, ValidationError):
verbose_proxy_logger.warning("Ignoring unreadable %s row", DEPLOYMENT_WARNING_IGNORES_ROW_ID)
return MappingProxyType({})
class DeploymentWarningIgnoreRepository(PrismaTableRepository["prisma_models.LiteLLM_UISettings"]):
table_name = "litellm_uisettings"
async def get_raw_ignores(self) -> object:
async def get_ignores(self) -> Mapping[str, AdminIgnore]:
db_record: Final = await self.table.find_unique(where={"id": DEPLOYMENT_WARNING_IGNORES_ROW_ID})
return db_record.ui_settings if db_record is not None else None
return parse_ignores(db_record.ui_settings if db_record is not None else None)
async def upsert_ignores(self, payload: str) -> None:
async def save_ignores(self, ignores: Mapping[str, AdminIgnore]) -> None:
payload: Final = _IGNORES_ADAPTER.dump_json(dict(ignores)).decode()
row: Final = {"id": DEPLOYMENT_WARNING_IGNORES_ROW_ID, "ui_settings": payload}
await self.table.upsert(
where={"id": DEPLOYMENT_WARNING_IGNORES_ROW_ID},

41
litellm/repositories/gateway_activity_repository.py Normal file
View file

@ -0,0 +1,41 @@
from typing import TYPE_CHECKING, Annotated, Final
from pydantic import Field, TypeAdapter
from litellm.models.gateway_activity import GatewayActivity
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient
_ACTIVITY_SQL: Final = """
SELECT
'route' AS kind, date, category, route,
SUM(successful_requests)::bigint AS successful_requests,
SUM(failed_requests)::bigint AS failed_requests,
NULL::integer AS status_code
FROM "LiteLLM_DailyGatewayRequests"
WHERE date >= $1 AND date <= $2
GROUP BY date, category, route
UNION ALL
SELECT
'status' AS kind, NULL, NULL, NULL,
0::bigint,
SUM(failed_requests)::bigint,
status_code
FROM "LiteLLM_DailyGatewayFailedRequests"
WHERE date >= $1 AND date <= $2
GROUP BY status_code
"""
_ACTIVITY_ROWS_ADAPTER: Final = TypeAdapter(tuple[Annotated[GatewayActivity, Field(discriminator="kind")], ...])
class GatewayActivityRepository:
def __init__(self, prisma_client: "PrismaClient") -> None:
self._prisma_client = prisma_client
async def get_activity(self, start_date: str, end_date: str) -> tuple[GatewayActivity, ...]:
raw_rows: Final = await self._prisma_client.db.query_raw( # pyright: ignore[reportAny] # untyped Prisma client
_ACTIVITY_SQL, start_date, end_date
)
return _ACTIVITY_ROWS_ADAPTER.validate_python(raw_rows or ())

20
litellm/repositories/user_banner_repository.py
View file

@ -1,5 +1,9 @@
import json
from typing import TYPE_CHECKING, Final
from pydantic import ValidationError
from litellm.models.user_banner import UserBanner
from litellm.repositories.table_repositories import PrismaTableRepository
if TYPE_CHECKING:
@ -8,9 +12,25 @@ if TYPE_CHECKING:
USER_BANNER_ROW_ID: Final = "user_banner"
def parse_user_banner(raw_settings: object) -> UserBanner:
if raw_settings is None:
return UserBanner()
try:
parsed: Final = json.loads(raw_settings) if isinstance(raw_settings, str) else raw_settings
return UserBanner.model_validate(parsed)
except (json.JSONDecodeError, ValidationError):
return UserBanner()
class UserBannerRepository(PrismaTableRepository["prisma_models.LiteLLM_UISettings"]):
table_name = "litellm_uisettings"
async def get_banner(self) -> UserBanner:
return parse_user_banner(await self.get_raw_settings())
async def save_banner(self, banner: UserBanner) -> None:
await self.upsert_settings(json.dumps(banner.model_dump()))
async def get_raw_settings(self) -> object:
db_record: Final = await self.table.find_unique(where={"id": USER_BANNER_ROW_ID})
return db_record.ui_settings if db_record is not None else None

10
litellm/responses/main.py
View file

@ -42,6 +42,11 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
update_responses_tools_with_model_file_ids,
)
from litellm.litellm_core_utils.provider_affinity import add_provider_affinity_header
from litellm.litellm_core_utils.reasoning_effort_capability import (
deployment_reasoning_effort_levels,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.llms.base_llm.responses.transformation import BaseResponsesAPIConfig
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.llms.openai.common_utils import get_openai_credentials
@ -63,11 +68,6 @@ from litellm.responses.mcp.round_billing import (
)
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.router_utils.reasoning_effort_capability import (
deployment_reasoning_effort_levels,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.types.llms.openai import (
PromptObject,
Reasoning,

293
litellm/router.py
View file

@ -106,6 +106,20 @@ from litellm.integrations.custom_guardrail import CustomGuardrail, is_guardrail_
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.otel.routing import routing_decision_attributes
from litellm.integrations.otel.runtime import phase_attributes, phase_event, phase_span
from litellm.litellm_core_utils.access_windows import access_windows_config_error, filter_reserved_deployments
from litellm.litellm_core_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
add_retry_headers_to_response,
apply_quality_router_decision_headers,
apply_remaining_usage_headers,
apply_response_model_id,
complexity_router_decision_headers,
ensure_response_additional_headers,
get_hidden_params_dict,
prepare_response_for_header_attachment,
replace_complexity_router_headers,
response_total_token_count,
)
from litellm.litellm_core_utils.asyncify import run_async_function
from litellm.litellm_core_utils.batch_utils import (
batch_create_routing,
@ -116,6 +130,10 @@ from litellm.litellm_core_utils.batch_utils import (
should_replace_model_in_jsonl,
without_deployments_that_cannot_batch,
)
from litellm.litellm_core_utils.clientside_credential_handler import (
get_dynamic_litellm_params,
is_clientside_credential,
)
from litellm.litellm_core_utils.completion_timeout import CompletionTimeout
from litellm.litellm_core_utils.core_helpers import (
coerce_token_limit,
@ -127,12 +145,17 @@ from litellm.litellm_core_utils.core_helpers import (
from litellm.litellm_core_utils.coroutine_checker import coroutine_checker
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor, reject_api_key_secret_reference
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.deployment_pricing import deployment_pricing_models
from litellm.litellm_core_utils.get_llm_provider_logic import (
declared_authenticating_provider,
inferred_provider,
is_registered_custom_provider,
resolve_model_and_provider_for_metadata,
)
from litellm.litellm_core_utils.get_retry_from_policy import (
get_num_retries_from_retry_policy as _get_num_retries_from_retry_policy,
)
from litellm.litellm_core_utils.get_retry_from_policy import resolve_retry_policy
from litellm.litellm_core_utils.hidden_params import (
HIDDEN_PARAMS_ATTR as _HIDDEN_PARAMS_ATTR,
)
@ -142,6 +165,7 @@ from litellm.litellm_core_utils.hidden_params import (
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.litellm_core_utils.llm_cost_calc.utils import SERVICE_TIER_COST_KEY_SUFFIXES
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.ptu_pricing import (
PTU_COST_ATTRIBUTION_ENV_VAR,
declares_ptu,
@ -152,10 +176,52 @@ from litellm.litellm_core_utils.ptu_pricing import (
ptu_terms,
zeroed_ptu_pricing,
)
from litellm.litellm_core_utils.ptu_shares import (
PTUTeamCeiling,
filter_ptu_shared_deployments,
ptu_capacity_warning,
ptu_share_teams,
share_holder_request_kwargs,
team_ptu_ceiling,
)
from litellm.litellm_core_utils.reasoning_effort_capability import (
deployment_is_catalog_mapped,
intersect_supported_reasoning_efforts,
resolve_supported_reasoning_efforts,
)
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
get_service_tier_timeout,
)
from litellm.litellm_core_utils.routing_common_utils import (
MID_STREAM_CONTINUATION_KWARG,
as_retry_skipped_deployment_ids,
deployment_model_id_pin,
deployments_the_caller_may_use,
filter_team_based_models,
filter_web_search_deployments,
format_content_policy_fallback_hint,
format_content_policy_no_fallback_note,
format_fallback_outcome_message,
format_no_fallback_group_message,
get_request_team_id,
is_proxy_admin_request,
provider_for_generic_call,
request_user_api_key_auth,
resolve_model_group_alias,
team_may_use_deployment,
truncate_fallback_error_detail,
warn_on_ignored_model_capability_flags,
warn_on_provider_credential_mismatch,
with_admin_api_key_secret_reference,
without_model_capability_flags,
without_router_only_kwargs,
)
from litellm.litellm_core_utils.routing_groups import (
apply_routing_group_priority,
parse_routing_groups,
validate_routing_strategy,
)
from litellm.litellm_core_utils.secret_redaction import redact_string
from litellm.litellm_core_utils.sensitive_data_masker import (
SensitiveDataMasker,
@ -196,6 +262,7 @@ from litellm.router_api import (
aapply_changes,
apply_changes,
)
from litellm.router_api import RoutingContext as RouterContext
from litellm.router_api import deployment_id as _deployment_id
from litellm.router_deprecated import DeprecatedRouterMembers
from litellm.router_host import router_host
@ -233,49 +300,7 @@ from litellm.router_strategy.tag_based_routing import (
is_valid_deployment_tag,
tags_with_serving_deployment_tags,
)
from litellm.router_utils.access_windows import access_windows_config_error, filter_reserved_deployments
from litellm.router_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
add_retry_headers_to_response,
apply_quality_router_decision_headers,
apply_remaining_usage_headers,
apply_response_model_id,
complexity_router_decision_headers,
ensure_response_additional_headers,
get_hidden_params_dict,
prepare_response_for_header_attachment,
replace_complexity_router_headers,
response_total_token_count,
)
from litellm.router_utils.client_initalization_utils import InitalizeCachedClient, MaxParallelRequestsLimit
from litellm.router_utils.clientside_credential_handler import (
get_dynamic_litellm_params,
is_clientside_credential,
)
from litellm.router_utils.common_utils import (
MID_STREAM_CONTINUATION_KWARG,
as_retry_skipped_deployment_ids,
deployment_model_id_pin,
deployments_the_caller_may_use,
filter_team_based_models,
filter_web_search_deployments,
format_content_policy_fallback_hint,
format_content_policy_no_fallback_note,
format_fallback_outcome_message,
format_no_fallback_group_message,
get_request_team_id,
is_proxy_admin_request,
provider_for_generic_call,
request_user_api_key_auth,
resolve_model_group_alias,
team_may_use_deployment,
truncate_fallback_error_detail,
warn_on_ignored_model_capability_flags,
warn_on_provider_credential_mismatch,
with_admin_api_key_secret_reference,
without_model_capability_flags,
without_router_only_kwargs,
)
from litellm.router_utils.cooldown_cache import CooldownCache
from litellm.router_utils.cooldown_handlers import (
DEFAULT_COOLDOWN_TIME_SECONDS,
@ -292,7 +317,6 @@ from litellm.router_utils.cooldown_handlers import (
mark_deployment_selection_failure,
set_cooldown_deployments,
)
from litellm.router_utils.deployment_pricing import deployment_pricing_models
from litellm.router_utils.embedding_batch_size_check import (
async_log_rejection_before_provider_call,
clear_rejected_before_provider_call,
@ -337,10 +361,6 @@ from litellm.router_utils.fallback_event_handlers import (
routed_deployment_id,
run_async_fallback,
)
from litellm.router_utils.get_retry_from_policy import (
get_num_retries_from_retry_policy as _get_num_retries_from_retry_policy,
)
from litellm.router_utils.get_retry_from_policy import resolve_retry_policy
from litellm.router_utils.handle_error import (
async_raise_no_deployment_exception,
send_llm_exception_alert,
@ -392,19 +412,6 @@ from litellm.router_utils.pre_call_checks.model_rate_limit_check import (
from litellm.router_utils.pre_call_checks.prompt_caching_deployment_check import (
PromptCachingDeploymentCheck,
)
from litellm.router_utils.ptu_shares import (
PTUTeamCeiling,
filter_ptu_shared_deployments,
ptu_capacity_warning,
ptu_share_teams,
share_holder_request_kwargs,
team_ptu_ceiling,
)
from litellm.router_utils.reasoning_effort_capability import (
deployment_is_catalog_mapped,
intersect_supported_reasoning_efforts,
resolve_supported_reasoning_efforts,
)
from litellm.router_utils.request_priority import (
InvalidPriority,
parse_default_priority,
@ -420,11 +427,6 @@ from litellm.router_utils.router_callbacks.track_deployment_metrics import (
increment_deployment_failures_for_current_minute,
increment_deployment_successes_for_current_minute,
)
from litellm.router_utils.routing_groups import (
apply_routing_group_priority,
parse_routing_groups,
validate_routing_strategy,
)
from litellm.router_utils.routing_read_batch import ROUTER_USAGE_TARGET, RoutingPrefetch, RoutingReadBatch
from litellm.router_utils.stream_usage_injection import router_config_requests_stream_usage
from litellm.scheduler import FlowItem, Scheduler
@ -504,6 +506,7 @@ from litellm.utils import (
ProviderConfigManager,
Rules,
TextCompletionResponse,
_restore_correlation_context_if_supported, # pyright: ignore[reportPrivateUsage] # same task-local cleanup as the SDK wrapper
aresolve_api_key_secret_reference,
function_setup,
get_llm_provider,
@ -519,8 +522,6 @@ from litellm.utils import (
token_counter,
)
from .router_utils.pattern_match_deployments import PatternMatchRouter
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@ -1609,6 +1610,7 @@ class Router(DeprecatedRouterMembers):
router = Router(model_list=model_list, fallbacks=[{"azure-gpt-3.5-turbo": ["openai-gpt-3.5-turbo"]}])
```
"""
self._core_ctx: Final = RouterContext(self)
self.set_verbose = set_verbose
self.ignore_invalid_deployments = ignore_invalid_deployments
@ -1884,7 +1886,7 @@ class Router(DeprecatedRouterMembers):
if self._normalize_strategy(routing_strategy) == "lar1":
from litellm.router_strategy.lar1_routing import apply_lar1_routing_strategy
apply_lar1_routing_strategy(self, routing_strategy_args)
apply_lar1_routing_strategy(self._core_ctx, routing_strategy_args)
else:
self.routing_strategy_init(
routing_strategy=routing_strategy,
@ -2258,7 +2260,7 @@ class Router(DeprecatedRouterMembers):
if strategy == "lar1":
from litellm.router_strategy.lar1_routing import apply_lar1_routing_strategy
apply_lar1_routing_strategy(self, self.routing_strategy_args)
apply_lar1_routing_strategy(self._core_ctx, self.routing_strategy_args)
return
attr: Final = self._DEFAULT_SELECTOR_ATTR_BY_STRATEGY.get(strategy or "")
@ -2583,7 +2585,7 @@ class Router(DeprecatedRouterMembers):
)
deployments: Final = self._core_get_model_list(model_name=model)
if deployments:
RoutingPrefetch.arm(self, usage_selector, deployments)
RoutingPrefetch.arm(self._core_ctx, usage_selector, deployments)
except Exception as e: # noqa: BLE001 # a prefetch is an optimisation, never a reason to fail the request
verbose_router_logger.debug(
"routing read prefetch not armed for %s: %s", _without_line_breaks(model), _without_line_breaks(e)
@ -3132,7 +3134,7 @@ class Router(DeprecatedRouterMembers):
break
if existing_ec_callback is not None:
existing_ec_callback.router = self
existing_ec_callback.ctx = self._core_ctx
existing_ec_callback.enable_global_affinity = (
existing_ec_callback.enable_global_affinity or enable_global_affinity
)
@ -3140,7 +3142,7 @@ class Router(DeprecatedRouterMembers):
ec_callback = existing_ec_callback
else:
ec_callback = EncryptedContentAffinityCheck(
router=self,
ctx=self._core_ctx,
enable_global_affinity=enable_global_affinity,
model_group_affinity_config=self.model_group_affinity_config,
)
@ -3689,7 +3691,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -4932,7 +4934,7 @@ class Router(DeprecatedRouterMembers):
kwargs=kwargs,
client_type="max_parallel_requests",
)
compacted_input: Final = await compact_to_fit(self, deployment, input_kwargs, "chat")
compacted_input: Final = await compact_to_fit(self._core_ctx, deployment, input_kwargs, "chat")
async with contextlib.AsyncExitStack() as deployment_slot:
if isinstance(max_parallel_requests_limit, MaxParallelRequestsLimit):
deployment_slot.enter_context(max_parallel_requests_limit)
@ -5395,7 +5397,7 @@ class Router(DeprecatedRouterMembers):
hop_kwargs: Final = _FALLBACK_HOP_ADAPTER.validate_python(kwargs)
if _is_fallback_hop(hop_kwargs):
EncryptedContentAffinityCheck.strip_reasoning_the_targets_cannot_decrypt(
self,
self._core_ctx,
hop_kwargs.get("input"),
hop_kwargs.get("messages"),
(_FALLBACK_HOP_ADAPTER.validate_python(deployment),),
@ -6001,7 +6003,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6011,15 +6013,42 @@ class Router(DeprecatedRouterMembers):
async def _aimage_generation(self, prompt: str, model: str, **kwargs):
model_name = model
start_time: Final = datetime.now() # noqa: DTZ005 # SDK logging uses naive local timestamps
try:
verbose_router_logger.debug("Inside _image_generation()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self._core_async_get_available_deployment(
model=model,
messages=[{"role": "user", "content": "prompt"}],
specific_deployment=kwargs.pop("specific_deployment", None),
request_kwargs=kwargs,
)
try:
deployment: Final = await self._core_async_get_available_deployment(
model=model,
messages=[{"role": "user", "content": "prompt"}],
specific_deployment=kwargs.pop("specific_deployment", None),
request_kwargs=kwargs,
)
except Exception as selection_error:
# Admitted attempts keep SDK-owned logging, including deployment callbacks and retry state
request_kwargs: Final = cast( # cast-ok: Python kwargs have string keys and opaque values
Mapping[str, object], kwargs
)
if request_kwargs.get("litellm_logging_obj") is None:
refusal_kwargs: Final = {**request_kwargs, "model": model, "prompt": prompt}
refusal_kwargs.setdefault("litellm_call_id", str(uuid.uuid4()))
logging_obj, _ = function_setup(
"aimage_generation",
Rules(),
start_time,
**refusal_kwargs, # pyright: ignore[reportArgumentType] # preserve opaque SDK options, including is_async_call
)
try:
asyncio.create_task(
logging_obj.dispatch_failure_handlers(
exception=selection_error,
traceback_exception=traceback.format_exc(),
prefer_async_handlers=True,
)
)
finally:
_restore_correlation_context_if_supported(logging_obj)
raise
self._update_kwargs_with_deployment(deployment=deployment, kwargs=kwargs)
data: Final = deployment["litellm_params"].copy()
@ -6086,7 +6115,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6179,7 +6208,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6241,7 +6270,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6357,7 +6386,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6427,7 +6456,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6486,7 +6515,7 @@ class Router(DeprecatedRouterMembers):
return _OPAQUE_VALUE_ADAPTER.validate_python(
await SearchAPIRouter.async_search_with_fallbacks(
router_instance=self,
litellm_routing_ctx=self._core_ctx,
original_function=original_function,
**kwargs,
)
@ -6500,7 +6529,7 @@ class Router(DeprecatedRouterMembers):
from litellm.router_utils.search_api_router import SearchAPIRouter
return await SearchAPIRouter.async_search_with_fallbacks_helper(
router_instance=self,
litellm_routing_ctx=self._core_ctx,
model=model,
original_generic_function=original_generic_function,
**kwargs,
@ -6618,7 +6647,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -6718,7 +6747,7 @@ class Router(DeprecatedRouterMembers):
response_kwargs["custom_llm_provider"] = custom_llm_provider
compacted_input: Final = await compact_to_fit(
self,
self._core_ctx,
deployment,
response_kwargs,
surface_for_call(generic_function_name),
@ -7596,7 +7625,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -7675,7 +7704,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -7689,7 +7718,7 @@ class Router(DeprecatedRouterMembers):
**kwargs,
) -> OpenAIFileObject:
try:
from litellm.router_utils.common_utils import add_model_file_id_mappings
from litellm.litellm_core_utils.routing_common_utils import add_model_file_id_mappings
verbose_router_logger.debug("Inside _atext_completion()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
@ -7915,7 +7944,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -8083,7 +8112,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -8114,7 +8143,7 @@ class Router(DeprecatedRouterMembers):
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
ctx=self._core_ctx,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
@ -8904,7 +8933,7 @@ class Router(DeprecatedRouterMembers):
# that fails with RouterRateLimitError whenever the "remaining" entries
# are all in cooldown — the inner async_get_healthy_deployments call
# would find an empty list and raise immediately.
cooldown_ids = set(await async_get_cooldown_deployments(litellm_router_instance=self, parent_otel_span=None))
cooldown_ids = set(await async_get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=None))
remaining: Final = (all_ids - cooldown_ids) - excluded
if not remaining:
return None
@ -8984,7 +9013,7 @@ class Router(DeprecatedRouterMembers):
raise e
input_kwargs: Final = {
"litellm_router": self,
"litellm_routing_ctx": self._core_ctx,
"original_exception": original_exception,
**kwargs,
"attempted_targets": fallback_walk,
@ -9200,7 +9229,7 @@ class Router(DeprecatedRouterMembers):
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
fallback_failure_exception_str = truncate_fallback_error_detail(redact_string(str(new_exception)))
cooldown_info: Final = await async_get_cooldown_deployments_with_debug_info(
litellm_router_instance=self,
ctx=self._core_ctx,
parent_otel_span=parent_otel_span,
)
verbose_router_logger.error(
@ -9821,7 +9850,7 @@ class Router(DeprecatedRouterMembers):
# Always track deployment successes for cooldown logic, regardless of TPM/RPM limits
increment_deployment_successes_for_current_minute(
litellm_router_instance=self,
ctx=self._core_ctx,
deployment_id=str(deployment_id),
)
@ -10001,7 +10030,7 @@ class Router(DeprecatedRouterMembers):
if id is not None:
key: Final = increment_deployment_successes_for_current_minute(
litellm_router_instance=self,
ctx=self._core_ctx,
deployment_id=id,
)
return key
@ -10112,11 +10141,11 @@ class Router(DeprecatedRouterMembers):
if deployment_id is None:
return False
increment_deployment_failures_for_current_minute(
litellm_router_instance=self,
ctx=self._core_ctx,
deployment_id=deployment_id,
)
result: Final = set_cooldown_deployments(
litellm_router_instance=self,
ctx=self._core_ctx,
exception_status=exception_status,
original_exception=exception,
deployment=deployment_id,
@ -10422,9 +10451,7 @@ class Router(DeprecatedRouterMembers):
except Exception:
pass
unhealthy_deployments: Final = get_cooldown_deployments(
litellm_router_instance=self, parent_otel_span=parent_otel_span
)
unhealthy_deployments: Final = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
unhealthy_set: Final = set(unhealthy_deployments)
healthy_deployments: list = [d for d in _all_deployments if d["model_info"]["id"] not in unhealthy_set]
healthy_deployments = self._filter_blocked_deployments(healthy_deployments)
@ -10452,7 +10479,7 @@ class Router(DeprecatedRouterMembers):
pass
unhealthy_deployments: Final = await async_get_cooldown_deployments(
litellm_router_instance=self, parent_otel_span=parent_otel_span
ctx=self._core_ctx, parent_otel_span=parent_otel_span
)
# Convert to set for O(1) lookup instead of O(n)
unhealthy_deployments_set: Final = set(unhealthy_deployments)
@ -10510,7 +10537,7 @@ class Router(DeprecatedRouterMembers):
)
)
set_cooldown_deployments(
litellm_router_instance=self,
ctx=self._core_ctx,
exception_status=e.status_code,
original_exception=e,
deployment=deployment["model_info"]["id"],
@ -10594,7 +10621,7 @@ class Router(DeprecatedRouterMembers):
)
if selected is None:
no_deployment: Final = await async_raise_no_deployment_exception(
litellm_router_instance=self, model=model_group, parent_otel_span=None
ctx=self._core_ctx, model=model_group, parent_otel_span=None
)
raise litellm.RateLimitError(
message=str(no_deployment),
@ -11025,7 +11052,7 @@ class Router(DeprecatedRouterMembers):
auto_router_config=auto_router_config,
default_model=default_model,
embedding_model=embedding_model,
litellm_router_instance=self,
ctx=self._core_ctx,
max_input_chars=(max_input_chars if max_input_chars is not None else DEFAULT_AUTO_ROUTER_MAX_INPUT_CHARS),
)
self._register_pre_routing_strategy(
@ -11106,7 +11133,7 @@ class Router(DeprecatedRouterMembers):
complexity_router: Final[ComplexityRouter] = ComplexityRouter(
model_name=deployment.model_name,
default_model=default_model,
litellm_router_instance=self,
ctx=self._core_ctx,
complexity_router_config=complexity_router_config,
)
self._register_pre_routing_strategy(
@ -11406,7 +11433,7 @@ class Router(DeprecatedRouterMembers):
quality_router: Final[QualityRouter] = QualityRouter(
model_name=deployment.model_name,
default_model=default_model,
litellm_router_instance=self,
ctx=self._core_ctx,
quality_router_config=quality_router_config,
)
self._register_pre_routing_strategy(
@ -13627,7 +13654,7 @@ class Router(DeprecatedRouterMembers):
while remaining:
try:
picked = await get_deployments_for_tag(
llm_router_instance=self,
ctx=self._core_ctx,
model=model,
healthy_deployments=remaining,
request_kwargs=tag_request,
@ -14510,7 +14537,7 @@ class Router(DeprecatedRouterMembers):
if routing_strategy == "lar1":
from litellm.router_strategy.lar1_routing import apply_lar1_routing_strategy
apply_lar1_routing_strategy(self, kwargs.get("routing_strategy_args"))
apply_lar1_routing_strategy(self._core_ctx, kwargs.get("routing_strategy_args"))
return
self.routing_strategy_init(
routing_strategy=routing_strategy, routing_strategy_args=kwargs.get("routing_strategy_args", {})
@ -14603,7 +14630,7 @@ class Router(DeprecatedRouterMembers):
cache_key = f"{model_id}_max_parallel_requests_client"
client = self._core_cache.get_cache(key=cache_key, local_only=True, parent_otel_span=parent_otel_span)
if client is None:
InitalizeCachedClient.set_max_parallel_requests_client(litellm_router_instance=self, model=deployment)
InitalizeCachedClient.set_max_parallel_requests_client(ctx=self._core_ctx, model=deployment)
client = self._core_cache.get_cache(key=cache_key, local_only=True, parent_otel_span=parent_otel_span)
return client
elif client_type == "async":
@ -15534,10 +15561,10 @@ class Router(DeprecatedRouterMembers):
routing_read_batch: Final = RoutingReadBatch.active()
cooldown_deployments: Final = (
await async_get_cooldown_deployments(litellm_router_instance=self, parent_otel_span=parent_otel_span)
await async_get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
if routing_read_batch is None
else await routing_read_batch.async_get_cooldown_deployments(
litellm_router_instance=self,
ctx=self._core_ctx,
healthy_deployments=healthy_deployments,
parent_otel_span=parent_otel_span,
)
@ -15592,7 +15619,7 @@ class Router(DeprecatedRouterMembers):
)
# check if user wants to do tag based routing
healthy_deployments = await get_deployments_for_tag(
llm_router_instance=self,
ctx=self._core_ctx,
model=model,
request_kwargs=request_kwargs,
healthy_deployments=healthy_deployments,
@ -15671,7 +15698,7 @@ class Router(DeprecatedRouterMembers):
if len(healthy_deployments) == 0:
exception: Final = await async_raise_no_deployment_exception(
litellm_router_instance=self,
ctx=self._core_ctx,
model=model,
parent_otel_span=parent_otel_span,
route_model_ids=await self._async_routable_cooled_deployment_ids(
@ -15917,7 +15944,7 @@ class Router(DeprecatedRouterMembers):
)
if deployment is None:
exception: Final = await async_raise_no_deployment_exception(
litellm_router_instance=self,
ctx=self._core_ctx,
model=model,
parent_otel_span=parent_otel_span,
)
@ -16087,7 +16114,7 @@ class Router(DeprecatedRouterMembers):
if deployment is None:
exception: Final = await async_raise_no_deployment_exception(
litellm_router_instance=self,
ctx=self._core_ctx,
model=model,
parent_otel_span=parent_otel_span,
)
@ -16455,7 +16482,7 @@ class Router(DeprecatedRouterMembers):
if isinstance(selected_strategy.strategy, ComplexityRouter)
else (),
parent_model=model,
router=self,
ctx=self._core_ctx,
allow_escalation=isinstance(selected_strategy.strategy, ComplexityRouter)
and selected_strategy.strategy.config.enable_context_window_escalation,
messages=messages,
@ -16818,9 +16845,7 @@ class Router(DeprecatedRouterMembers):
parent_otel_span=parent_otel_span,
)
cooldown_deployments: Final = get_cooldown_deployments(
litellm_router_instance=self, parent_otel_span=parent_otel_span
)
cooldown_deployments: Final = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
_pre_cooldown_deployments: Final = healthy_deployments
healthy_deployments = self._filter_cooldown_deployments(
healthy_deployments=healthy_deployments,
@ -16919,7 +16944,7 @@ class Router(DeprecatedRouterMembers):
_cooldown_time = self._core_cooldown_cache.get_min_cooldown(
model_ids=model_ids, parent_otel_span=parent_otel_span
)
_cooldown_list = get_cooldown_deployments(litellm_router_instance=self, parent_otel_span=parent_otel_span)
_cooldown_list = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
raise RouterRateLimitError(
model=model,
cooldown_time=_cooldown_time,
@ -16927,7 +16952,7 @@ class Router(DeprecatedRouterMembers):
cooldown_list=_cooldown_list,
model_ids=model_ids,
cooldown_deployments=get_cooldown_deployment_info(
self,
self._core_ctx,
self._deployment_ids(
self._region_routable_deployments(
request_deployments,
@ -16966,7 +16991,7 @@ class Router(DeprecatedRouterMembers):
_cooldown_time = self._core_cooldown_cache.get_min_cooldown(
model_ids=model_ids, parent_otel_span=parent_otel_span
)
_cooldown_list = get_cooldown_deployments(litellm_router_instance=self, parent_otel_span=parent_otel_span)
_cooldown_list = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
raise RouterRateLimitError(
model=model,
cooldown_time=_cooldown_time,
@ -17073,9 +17098,7 @@ class Router(DeprecatedRouterMembers):
healthy_deployments=pass_through_deployments,
parent_otel_span=parent_otel_span,
)
cooldown_deployments: Final = get_cooldown_deployments(
litellm_router_instance=self, parent_otel_span=parent_otel_span
)
cooldown_deployments: Final = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
pass_through_deployments = self._filter_cooldown_deployments(
healthy_deployments=pass_through_deployments,
cooldown_deployments=cooldown_deployments,
@ -17104,7 +17127,7 @@ class Router(DeprecatedRouterMembers):
_cooldown_time = self._core_cooldown_cache.get_min_cooldown(
model_ids=model_ids, parent_otel_span=parent_otel_span
)
_cooldown_list = get_cooldown_deployments(litellm_router_instance=self, parent_otel_span=parent_otel_span)
_cooldown_list = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
raise RouterRateLimitError(
model=model,
cooldown_time=_cooldown_time,
@ -17112,7 +17135,7 @@ class Router(DeprecatedRouterMembers):
cooldown_list=_cooldown_list,
model_ids=pass_through_model_ids,
cooldown_deployments=get_cooldown_deployment_info(
self,
self._core_ctx,
self._deployment_ids(
self._region_routable_deployments(
request_deployments,
@ -17150,7 +17173,7 @@ class Router(DeprecatedRouterMembers):
_cooldown_time = self._core_cooldown_cache.get_min_cooldown(
model_ids=model_ids, parent_otel_span=parent_otel_span
)
_cooldown_list = get_cooldown_deployments(litellm_router_instance=self, parent_otel_span=parent_otel_span)
_cooldown_list = get_cooldown_deployments(ctx=self._core_ctx, parent_otel_span=parent_otel_span)
raise RouterRateLimitError(
model=model,
cooldown_time=_cooldown_time,

265
litellm/router_api.py
View file

@ -33,19 +33,28 @@ as they were.
# repeat them unchanged, and typing the members themselves is the core's job.
import hashlib
import inspect
import json
import warnings
from collections.abc import Iterator, Mapping, Sequence
from collections.abc import Set as AbstractSet
from dataclasses import dataclass, field
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, Literal, TypeAlias, overload
from typing import TYPE_CHECKING, Final, Literal, Protocol, TypeAlias, overload
if TYPE_CHECKING:
from typing import Any # noqa: TID251 # only to spell the cores' untyped (Unknown) containers and one Any type
from opentelemetry.trace import Span
from litellm.caching.caching import DualCache, RedisCache
from litellm.integrations.SlackAlerting.slack_alerting import SlackAlerting
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.ptu_shares import PTUTeamCeiling
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.messages.dispatch import MessagesResult
from litellm.responses.streaming_iterator import BaseResponsesAPIStreamingIterator
from litellm.router import Router
from litellm.router_strategy.adaptive_router.adaptive_router import AdaptiveRouter
from litellm.router_strategy.auto_router.auto_router import AutoRouter
@ -53,12 +62,14 @@ if TYPE_CHECKING:
from litellm.router_strategy.complexity_router.complexity_router import ComplexityRouter
from litellm.router_strategy.quality_router.quality_router import QualityRouter
from litellm.router_utils.cooldown_cache import CooldownCache
from litellm.router_utils.fallback_event_handlers import AttemptedFallbackTargets as _AttemptedFallbackTargets
from litellm.router_utils.health_state_cache import DeploymentHealthCache
from litellm.router_utils.kubernetes_pod_discovery import KubernetesPodDiscovery
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.router_utils.ptu_shares import PTUTeamCeiling
from litellm.types.decisions import DecisionsResponse, OpenAIDecisionResponse
from litellm.types.llms.openai import ResponsesAPIResponse
from litellm.types.router import (
AllowedFailsPolicy,
CustomRoutingStrategyBase,
Deployment,
DeploymentModelListingInfo,
DeploymentTypedDict,
@ -74,7 +85,7 @@ if TYPE_CHECKING:
SearchToolTypedDict,
TaggedPreRoutingStrategy,
)
from litellm.types.utils import BudgetConfig
from litellm.types.utils import BudgetConfig, EmbeddingResponse, ModelResponse
from litellm.types.utils import ModelInfo as ModelMapInfo
# The cores' untyped containers (bare ``list`` / ``dict``, element types Unknown), spelled with Any:
@ -82,6 +93,9 @@ if TYPE_CHECKING:
UntypedList: TypeAlias = list[Any] # pyright: ignore[reportExplicitAny] # see above
UntypedDict: TypeAlias = dict[Any, Any] # pyright: ignore[reportExplicitAny] # see above
# Re-exported for litellm.types, which may not import router_utils.
AttemptedFallbackTargets: TypeAlias = _AttemptedFallbackTargets
# What registry().plugins(kind) returns, one member per kind.
RegistryPlugins: TypeAlias = (
dict[str, list[TaggedPreRoutingStrategy[AutoRouter]]]
@ -184,6 +198,73 @@ def deployment_id(model_group: str, litellm_params: Mapping[str, object]) -> str
return hashlib.sha256("".join(parts).encode()).hexdigest()
def router_init_args() -> tuple[str, ...]:
"""The arguments ``Router.__init__`` accepts (was ``Router.get_valid_args``). Needs no router instance."""
from litellm.router import Router
return tuple(Router.get_valid_args())
def mark_advisor_orchestration_failure(exception: BaseException) -> None:
"""Tag an advisor orchestration failure, so the router does not cool down the deployment it picked."""
from litellm.router_utils.cooldown_handlers import mark_advisor_orchestration_failure as mark
mark(exception)
def fallback_chain(
fallbacks: Sequence[Mapping[str, list[str]]], # mutable-ok: the fallbacks config shape
model_group: str,
) -> tuple[list[str] | None, int | None]: # mutable-ok: get_fallback_model_group's type, unchanged
"""
The fallbacks configured for a model group, and the index of the generic ``*`` entry (was
``get_fallback_model_group``). Needs no router instance.
"""
from litellm.router_utils.fallback_event_handlers import get_fallback_model_group
return get_fallback_model_group(fallbacks=fallbacks, model_group=model_group)
def request_fallback_chain(
fallbacks: Sequence[Mapping[str, list[str]]], # mutable-ok: the fallbacks config shape
request_kwargs: Mapping[str, object],
model_group: str | None,
) -> tuple[list[str] | None, int | None]: # mutable-ok: get_fallback_model_group's type, unchanged
"""
``fallback_chain`` for a request: the group a pre-routing hook picked first, then the routed,
requested and original groups (was ``get_fallback_model_group_for_lookup_groups``).
"""
from litellm.router_utils.fallback_event_handlers import (
fallback_lookup_groups,
get_fallback_model_group_for_lookup_groups,
)
return get_fallback_model_group_for_lookup_groups(
fallbacks=fallbacks, lookup_groups=fallback_lookup_groups(kwargs=request_kwargs, model_group=model_group)
)
def fallbacks_disabled(request_kwargs: Mapping[str, object]) -> bool:
"""Whether the request opted out of fallbacks (was ``fallbacks_disabled_for_request``)."""
from litellm.router_utils.fallback_event_handlers import fallbacks_disabled_for_request
return fallbacks_disabled_for_request(kwargs=request_kwargs)
def references_provider_scoped_resource(request_kwargs: Mapping[str, object]) -> bool:
"""Whether the request names a file, batch or fine-tuning job that only its own provider knows."""
from litellm.router_utils.fallback_event_handlers import PROVIDER_SCOPED_RESOURCE_KEYS
return any(request_kwargs.get(key) for key in PROVIDER_SCOPED_RESOURCE_KEYS)
def deployment_metadata(request_kwargs: Mapping[str, object]) -> Mapping[str, object] | None:
"""The metadata bucket the router stamped the picked deployment into (was ``find_deployment_metadata``)."""
from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
return find_deployment_metadata(request_kwargs)
def _json_default_stable_id(obj: object) -> str:
from litellm.router import Router
@ -550,7 +631,8 @@ class RouterRegistry:
- ``group_set``: the set of configured group names (was the ``model_names`` attribute)
- ``ids``: deployment ids, of one group or all (was ``get_model_ids``)
- ``deployment_names``: (was the ``deployment_names`` attribute)
- ``team_public``: team public model names (was ``team_public_model_names``)
- ``team_public``: team public model names (was ``team_public_model_names``), or only the
ones ``team_id`` serves its own deployments under (was ``team_public_model_names_of``)
- ``litellm_models``: upstream model names a group resolves to (was ``resolved_litellm_models``)
- ``litellm_base_models``: the same, with each deployment's ``base_model`` where it sets one
(was ``resolved_litellm_models(..., use_base_model=True)``)
@ -568,7 +650,7 @@ class RouterRegistry:
if kind == "deployment_names":
return r._core_deployment_names
if kind == "team_public":
return r._core_team_public_model_names
return r._core_team_public_model_names if team_id is None else r.team_public_model_names_of(team_id)
if model_name is None:
raise TypeError(f"registry().names(kind={kind!r}) needs a model_name")
if kind == "litellm_models":
@ -832,6 +914,14 @@ class RouterRegistry:
"""Routing groups by name (was the ``routing_groups`` attribute)."""
return self._router._core_routing_groups
def routing_group(self, name: str) -> "RoutingGroup | None":
"""The routing group callable as ``name``, or None (was ``get_routing_group``)."""
return self._router.get_routing_group(name)
def client(self, deployment: Mapping[str, object], *, client_type: str | None = None) -> object:
"""The SDK client the router cached for a deployment, or None (was ``_get_client``)."""
return self._router._get_client(deployment=deployment, kwargs={}, client_type=client_type)
def access_groups(
self,
model_name: str | None = None,
@ -1128,6 +1218,41 @@ class RouterHealth:
"""The request priority and the kwargs without it (was ``split_request_priority``)."""
return self._router._core_split_request_priority(model, kwargs)
def failures_this_minute(self, deployment_id: str) -> int:
"""Failures counted against a deployment this minute (was ``get_deployment_failures_for_current_minute``)."""
from litellm.router_utils.router_callbacks.track_deployment_metrics import (
get_deployment_failures_for_current_minute,
)
return get_deployment_failures_for_current_minute(self._router._core_ctx, deployment_id)
def record_failure(self, deployment_id: str, exception: Exception, exception_status: str | int) -> bool:
"""
Count a failure against a deployment this minute and cool it down once it is over its allowed
fails (was ``increment_deployment_failures_for_current_minute`` then ``set_cooldown_deployments``).
Returns whether the deployment was cooled down.
"""
from litellm.router_utils.cooldown_handlers import set_cooldown_deployments
from litellm.router_utils.router_callbacks.track_deployment_metrics import (
increment_deployment_failures_for_current_minute,
)
ctx: Final = self._router._core_ctx
increment_deployment_failures_for_current_minute(ctx, deployment_id)
return set_cooldown_deployments(
ctx,
original_exception=exception,
exception_status=exception_status,
deployment=deployment_id,
time_to_cooldown=self._router._core_cooldown_time,
)
def priority_header_fields(
self, model: str, kwargs: Mapping[str, object], header: str | None
) -> Mapping[str, int | str]:
"""The request fields a priority header sets (was ``priority_header_update``)."""
return self._router.priority_header_update(model=model, kwargs=kwargs, header=header)
@dataclass
class ChangeSet:
@ -1202,6 +1327,40 @@ async def aapply_changes(router: "Router", changes: ChangeSet) -> ApplyResult:
return result
def _takes_legacy_llm_router(check: object) -> bool:
"""Whether a fallback check still declares ``llm_router`` instead of ``ctx`` (the pre-RoutingContext shape)."""
try:
params: Final = inspect.signature(check).parameters # pyright: ignore[reportArgumentType] # any callable
except (TypeError, ValueError):
return False
return "ctx" not in params and "llm_router" in params
async def _run_fallback_check(
check: "FallbackAccessCheck | FallbackBudgetCheck",
ctx: "RoutingContext",
*,
model: str,
request_kwargs: Mapping[str, object],
) -> bool:
if _takes_legacy_llm_router(check):
warnings.warn(
f"{type(check).__name__} takes llm_router=; fallback checks now receive ctx: RoutingContext. "
"Passing llm_router is deprecated and will be removed",
DeprecationWarning,
stacklevel=2,
)
legacy: Final = check # pyright: ignore[reportAny] # the old protocol shape, checked above
return await legacy(model=model, request_kwargs=request_kwargs, llm_router=ctx._router) # pyright: ignore[reportCallIssue, reportPrivateUsage] # see above
return await check(model=model, request_kwargs=request_kwargs, ctx=ctx)
class RegistryReader(Protocol):
"""Anything that answers ``registry()``: a Router or a RoutingContext. What proxy auth helpers read."""
def registry(self) -> "RouterRegistry": ...
class RoutingContext:
"""
What a routing plugin (strategy, pre-call check, sub-router) may use of the router. Plugins get
@ -1225,6 +1384,98 @@ class RoutingContext:
def health(self) -> RouterHealth:
return self._router.health()
def observers(self) -> RoutingObservers:
return self._router.observe()
def belongs_to(self, router: "Router") -> bool:
return self._router is router
@property
def explicit_allowed_fails(self) -> int | None:
"""``allowed_fails`` as the caller passed it, ``None`` when it came from the global default."""
return self._router.explicit_allowed_fails
async def select(
self,
model: str,
request_kwargs: dict[str, object] | None = None, # mutable-ok: the selection cores take a dict
*,
kind: SelectKind = "pick",
messages: list[dict[str, str]] | None = None, # mutable-ok: the selection cores take a list
input: str | list[object] | None = None, # mutable-ok: the selection cores take a list
specific_deployment: bool | None = False,
parent_otel_span: "Span | None" = None,
health_check_probe: bool = False,
) -> object:
return await self._router.select( # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType, reportCallIssue, reportArgumentType] # the selection cores return untyped deployment dicts
model,
request_kwargs,
kind=kind, # pyright: ignore[reportArgumentType] # forwarded as given, the overloads narrow it
messages=messages,
input=input,
specific_deployment=specific_deployment,
parent_otel_span=parent_otel_span,
health_check_probe=health_check_probe,
)
async def with_fallbacks(
self,
*args: object,
**kwargs: object, # kwargs-ok: the router's fallback loop takes the request's own keyword payload
) -> object:
"""Run a call through the router's retry and fallback loop (was ``async_function_with_fallbacks``)."""
return await self._router.async_function_with_fallbacks(*args, **kwargs)
async def acompletion(
self,
**kwargs: object, # kwargs-ok: forwarded to the router endpoint as given
) -> "ModelResponse | CustomStreamWrapper":
return await self._router.acompletion(**kwargs) # pyright: ignore[reportCallIssue, reportArgumentType] # forwarded as given
async def aresponses(
self,
**kwargs: object, # kwargs-ok: forwarded to the router endpoint as given
) -> "ResponsesAPIResponse | BaseResponsesAPIStreamingIterator":
return await self._router.aresponses(**kwargs) # pyright: ignore[reportReturnType, reportArgumentType] # factory-built endpoint, returns what litellm.aresponses returns
async def aembedding(
self,
**kwargs: object, # kwargs-ok: forwarded to the router endpoint as given
) -> "EmbeddingResponse":
return await self._router.aembedding(**kwargs) # pyright: ignore[reportCallIssue, reportArgumentType] # forwarded as given
def embedding(
self,
**kwargs: object, # kwargs-ok: forwarded to the router endpoint as given
) -> "EmbeddingResponse":
return self._router.embedding(**kwargs) # pyright: ignore[reportCallIssue, reportArgumentType] # forwarded as given
async def adecisions(
self,
**kwargs: object, # kwargs-ok: forwarded to the router endpoint as given
) -> "OpenAIDecisionResponse | DecisionsResponse":
return await self._router.adecisions(**kwargs) # pyright: ignore[reportReturnType, reportArgumentType] # factory-built endpoint, returns what litellm.adecisions returns
async def aanthropic_messages(
self,
**kwargs: object, # kwargs-ok: forwarded to the router endpoint as given
) -> "MessagesResult":
return await self._router.aanthropic_messages(**kwargs) # pyright: ignore[reportReturnType, reportArgumentType] # factory-built endpoint, returns what litellm.anthropic_messages returns
def apply(self, changes: ChangeSet) -> ApplyResult:
return self._router.apply(changes)
def register_plugin(self, plugin: "CustomRoutingStrategyBase") -> None:
self._router.register_plugin(plugin)
async def fallback_access_allowed(self, *, model: str, request_kwargs: Mapping[str, object]) -> bool:
check: Final = self._router.settings().fallback_access_check
return check is None or await _run_fallback_check(check, self, model=model, request_kwargs=request_kwargs)
async def fallback_budget_allowed(self, *, model: str, request_kwargs: Mapping[str, object]) -> bool:
check: Final = self._router.settings().fallback_budget_check
return check is None or await _run_fallback_check(check, self, model=model, request_kwargs=request_kwargs)
def pre_call_checks(
self,
model: str,
@ -1292,7 +1543,7 @@ class RoutingContext:
self,
kwargs: dict[str, object], # mutable-ok: Router._core_log_retry records the retry into the request kwargs
e: Exception,
) -> Mapping[str, object]:
) -> dict: # mutable-ok: the fallback loop edits it; Router._core_log_retry's type
return self._router._core_log_retry(kwargs=kwargs, e=e)

105
litellm/router_deprecated.py
View file

@ -40,11 +40,11 @@ if TYPE_CHECKING:
from litellm.caching.caching import RedisCache
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.litellm_core_utils.ptu_shares import PTUTeamCeiling
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.router import Router
from litellm.router_strategy.budget_limiter import RouterBudgetLimiting
from litellm.router_utils.kubernetes_pod_discovery import KubernetesPodDiscovery
from litellm.router_utils.ptu_shares import PTUTeamCeiling
from litellm.types.router import (
CustomRoutingStrategyBase,
Deployment,
@ -838,6 +838,109 @@ class DeprecatedRouterMembers:
return _as_router(self)._core_log_retry(kwargs=kwargs, e=e)
# Router-family functions and classes that took the Router (``litellm_router_instance=``,
# ``router=``, ...) take its RoutingContext now. These keep the old spellings working.
_RoutingCallable: Final = TypeVar("_RoutingCallable", bound=Callable[..., object])
@functools.cache
def _router_class() -> "type[Router]":
from litellm.router import Router
return Router
def _note_renamed(label: str, old: str, instead: str, *, stacklevel: int = 3) -> None:
DEPRECATED_ROUTER_MEMBER_USES[label] += 1
warnings.warn(
f"{label}: {old} is deprecated; {instead} (the Router's RoutingContext) instead",
DeprecationWarning,
stacklevel=stacklevel, # the caller of the wrapper or descriptor that noted it
)
def _routing_context_of(value: object) -> object:
"""The Router's RoutingContext when ``value`` is a Router, else ``value`` as given."""
return value._core_ctx if isinstance(value, _router_class()) else value
def renamed_router_param(old: str, new: str = "ctx") -> Callable[[_RoutingCallable], _RoutingCallable]:
"""
Keep a router parameter that became a RoutingContext (``new``) callable the old way: ``old=``
still works, and so does a Router passed where ``new`` goes. Either gets the Router's
RoutingContext and a DeprecationWarning that names ``new``.
"""
def decorate(fn: _RoutingCallable) -> _RoutingCallable:
index: Final = list(inspect.signature(fn).parameters).index(new)
label: Final = f"{fn.__module__}.{fn.__qualname__}"
def convert(
args: tuple[object, ...],
kwargs: dict[str, object], # mutable-ok: the call's own kwargs
) -> tuple[object, ...]:
if old in kwargs:
if new in kwargs:
raise TypeError(f"{fn.__qualname__}() got both {old}= and {new}=")
_note_renamed(label, f"{old}=", f"pass {new}=", stacklevel=4)
kwargs[new] = _routing_context_of(kwargs.pop(old))
elif new in kwargs:
if isinstance(kwargs[new], _router_class()):
_note_renamed(label, f"a Router as {new}=", f"pass {new}=", stacklevel=4)
kwargs[new] = _routing_context_of(kwargs[new])
elif len(args) > index and isinstance(args[index], _router_class()):
_note_renamed(label, "passing a Router", f"pass {new}", stacklevel=4)
return (*args[:index], _routing_context_of(args[index]), *args[index + 1 :])
return args
# An async fn keeps an async wrapper, so iscoroutinefunction (and mock.patch's AsyncMock pick) still sees it
if inspect.iscoroutinefunction(fn):
@functools.wraps(fn)
async def acall(*args: object, **kwargs: object) -> object: # kwargs-ok: forwarded to fn as given
return await fn(*convert(args, kwargs), **kwargs) # pyright: ignore[reportAny] # fn's own result
return acall # pyright: ignore[reportReturnType] # same call shape as fn, see functools.wraps
@functools.wraps(fn)
def call(*args: object, **kwargs: object) -> object: # kwargs-ok: forwarded to fn as given
return fn(*convert(args, kwargs), **kwargs)
return call # pyright: ignore[reportReturnType] # same call shape as fn, see functools.wraps
return decorate
class RenamedRouterAttribute:
"""
A class attribute that keeps an old router attribute (``self.router``, ``self.litellm_router_instance``)
readable after it became the RoutingContext attribute ``new``: reading it returns that
RoutingContext, writing a Router stores the Router's RoutingContext, each with a DeprecationWarning.
"""
def __init__(self, new: str = "ctx") -> None:
self.new: Final = new
self.label = ""
def __set_name__(self, owner: type, name: str) -> None:
self.label = f"{owner.__module__}.{owner.__qualname__}.{name}"
@overload
def __get__(self, obj: None, owner: type | None = None) -> "RenamedRouterAttribute": ...
@overload
def __get__(self, obj: object, owner: type | None = None) -> object: ...
def __get__(self, obj: object | None, owner: type | None = None) -> object:
if obj is None:
return self
_note_renamed(self.label, "reading this attribute", f"read .{self.new}")
return getattr(obj, self.new) # pyright: ignore[reportAny] # the attribute named ``new`` holds the RoutingContext
def __set__(self, obj: object, value: object) -> None:
_note_renamed(self.label, "setting this attribute", f"set .{self.new}")
setattr(obj, self.new, _routing_context_of(value))
def _name_members_after_router() -> None:
# The old members were Router methods, so errors that name the method (a wrong-arity TypeError,
# or "litellm.router.Router.<name>() argument after ** must be a mapping") keep naming it the

15
litellm/router_strategy/auto_router/auto_router.py
View file

@ -17,6 +17,7 @@ from litellm.litellm_core_utils.internal_call_metadata import (
forwarded_internal_call_metadata,
parent_session_kwargs,
)
from litellm.router_deprecated import RenamedRouterAttribute, renamed_router_param
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.utils import (
AUTOROUTER_CLASSIFIER_CALL_ORIGIN,
@ -29,11 +30,11 @@ if TYPE_CHECKING:
from semantic_router.routers import SemanticRouter
from semantic_router.routers.base import Route
from litellm.router import Router
from litellm.router_api import RoutingContext
from litellm.router_strategy.auto_router.litellm_encoder import LiteLLMRouterEncoder
from litellm.types.router import PreRoutingHookResponse
else:
Router = Any
RoutingContext = Any
PreRoutingHookResponse = Any
Route = Any
SemanticRouter = Any
@ -55,13 +56,15 @@ class _SemanticMatchOutcome(NamedTuple):
class AutoRouter(CustomLogger):
DEFAULT_AUTO_SYNC_VALUE = "local"
litellm_router_instance: Final = RenamedRouterAttribute() # deprecated: the router attribute before ``ctx``
@renamed_router_param("litellm_router_instance")
def __init__(
self,
model_name: str,
default_model: str,
embedding_model: str,
litellm_router_instance: "Router",
ctx: "RoutingContext",
auto_router_config_path: str | None = None,
auto_router_config: str | None = None,
max_input_chars: int = DEFAULT_AUTO_ROUTER_MAX_INPUT_CHARS,
@ -75,7 +78,7 @@ class AutoRouter(CustomLogger):
auto_router_config: The config to use for the auto-router. You can either use this or auto_router_config_path, not both.
default_model: The default model to use if no route is found.
embedding_model: The embedding model to use for the auto-router.
litellm_router_instance: The instance of the LiteLLM Router.
ctx: The routing context of the owning router.
max_input_chars: Longest prompt, in characters, handed to the embedding model. Longer
prompts are cut to this length so an embedding context window far smaller than the
routed model's never fails the request.
@ -95,9 +98,9 @@ class AutoRouter(CustomLogger):
self.default_model = default_model
self.embedding_model: str = embedding_model
self.max_input_chars: int = max_input_chars
self.litellm_router_instance: Router = litellm_router_instance
self.ctx: Final = ctx
self.encoder: LiteLLMRouterEncoder = LiteLLMRouterEncoder(
litellm_router_instance=litellm_router_instance,
ctx=ctx,
model_name=embedding_model,
max_input_chars=max_input_chars,
)

61
litellm/router_strategy/auto_router/litellm_encoder.py
View file

@ -1,4 +1,4 @@
from typing import TYPE_CHECKING, Any, Final, Optional
from typing import TYPE_CHECKING, Any, ClassVar, Final, Optional
from pydantic import ConfigDict
from semantic_router.encoders import DenseEncoder
@ -6,11 +6,12 @@ from semantic_router.encoders.base import AsymmetricDenseMixin
import litellm
from litellm._logging import verbose_router_logger
from litellm.router_deprecated import RenamedRouterAttribute, renamed_router_param
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
else:
Router = Any
RoutingContext = Any
def litellm_to_list(embeds: litellm.EmbeddingResponse) -> list[list[float]]:
@ -19,21 +20,22 @@ def litellm_to_list(embeds: litellm.EmbeddingResponse) -> list[list[float]]:
:param embeds: The LiteLLM embedding response.
:return: A list of embeddings.
"""
if not embeds or not isinstance(embeds, litellm.EmbeddingResponse) or not embeds.data:
if not embeds or not embeds.data:
raise ValueError("No embeddings found in LiteLLM embedding response.")
return [x["embedding"] for x in embeds.data]
class CustomDenseEncoder(DenseEncoder):
model_config = ConfigDict(extra="allow")
litellm_router_instance: ClassVar = RenamedRouterAttribute() # deprecated: the router attribute before ``ctx``
def __init__(self, litellm_router_instance: Optional["Router"] = None, **kwargs):
# Extract litellm_router_instance from kwargs if passed there
if "litellm_router_instance" in kwargs:
litellm_router_instance = kwargs.pop("litellm_router_instance")
@renamed_router_param("litellm_router_instance")
def __init__(self, ctx: Optional["RoutingContext"] = None, **kwargs):
if "ctx" in kwargs:
ctx = kwargs.pop("ctx")
super().__init__(**kwargs)
self.litellm_router_instance = litellm_router_instance
self.ctx = ctx
class LiteLLMRouterEncoder(CustomDenseEncoder, AsymmetricDenseMixin):
@ -46,17 +48,18 @@ class LiteLLMRouterEncoder(CustomDenseEncoder, AsymmetricDenseMixin):
type: str = "internal_litellm_router"
@renamed_router_param("litellm_router_instance")
def __init__(
self,
litellm_router_instance: "Router",
ctx: "RoutingContext",
model_name: str,
score_threshold: float | None = None,
max_input_chars: int = 0,
):
"""Initialize the LiteLLMEncoder.
:param litellm_router_instance: The instance of the LiteLLM Router.
:type litellm_router_instance: Router
:param ctx: The routing context of the owning router.
:type ctx: RoutingContext
:param model_name: The name of the embedding model to use. Must use LiteLLM naming
convention (e.g. "openai/text-embedding-3-small" or "mistral/mistral-embed").
:type model_name: str
@ -75,7 +78,7 @@ class LiteLLMRouterEncoder(CustomDenseEncoder, AsymmetricDenseMixin):
score_threshold=score_threshold if score_threshold is not None else 0.3,
)
self.model_name = model_name
self.litellm_router_instance = litellm_router_instance
self.ctx = ctx
self.max_input_chars = max_input_chars
def _clamp(self, docs: list[str]) -> list[str]: # mutable-ok: embedding() takes `input: str | list`
@ -109,45 +112,37 @@ class LiteLLMRouterEncoder(CustomDenseEncoder, AsymmetricDenseMixin):
return await self.aencode_queries(docs, **kwargs)
def encode_queries(self, docs: list[str], **kwargs) -> list[list[float]]:
if self.litellm_router_instance is None:
raise ValueError("litellm_router_instance is not set")
if self.ctx is None:
raise ValueError("ctx is not set")
try:
embeds: Final = self.litellm_router_instance.embedding(
input=self._clamp(docs), model=self.model_name, **kwargs
)
embeds: Final = self.ctx.embedding(input=self._clamp(docs), model=self.model_name, **kwargs)
return litellm_to_list(embeds)
except Exception as e:
raise ValueError(f"{self.type.capitalize()} API call failed. Error: {e}") from e
def encode_documents(self, docs: list[str], **kwargs) -> list[list[float]]:
if self.litellm_router_instance is None:
raise ValueError("litellm_router_instance is not set")
if self.ctx is None:
raise ValueError("ctx is not set")
try:
embeds: Final = self.litellm_router_instance.embedding(
input=self._clamp(docs), model=self.model_name, **kwargs
)
embeds: Final = self.ctx.embedding(input=self._clamp(docs), model=self.model_name, **kwargs)
return litellm_to_list(embeds)
except Exception as e:
raise ValueError(f"{self.type.capitalize()} API call failed. Error: {e}") from e
async def aencode_queries(self, docs: list[str], **kwargs) -> list[list[float]]:
if self.litellm_router_instance is None:
raise ValueError("litellm_router_instance is not set")
if self.ctx is None:
raise ValueError("ctx is not set")
try:
embeds: Final = await self.litellm_router_instance.aembedding(
input=self._clamp(docs), model=self.model_name, **kwargs
)
embeds: Final = await self.ctx.aembedding(input=self._clamp(docs), model=self.model_name, **kwargs)
return litellm_to_list(embeds)
except Exception as e:
raise ValueError(f"{self.type.capitalize()} API call failed. Error: {e}") from e
async def aencode_documents(self, docs: list[str], **kwargs) -> list[list[float]]:
if self.litellm_router_instance is None:
raise ValueError("litellm_router_instance is not set")
if self.ctx is None:
raise ValueError("ctx is not set")
try:
embeds: Final = await self.litellm_router_instance.aembedding(
input=self._clamp(docs), model=self.model_name, **kwargs
)
embeds: Final = await self.ctx.aembedding(input=self._clamp(docs), model=self.model_name, **kwargs)
return litellm_to_list(embeds)
except Exception as e:
raise ValueError(f"{self.type.capitalize()} API call failed. Error: {e}") from e

54
litellm/router_strategy/complexity_router/complexity_router.py
View file

@ -65,6 +65,7 @@ from litellm.llms.base_llm.base_utils import type_to_response_format_param
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.llms.databricks.decisions.transformation import DATABRICKS_DECISIONS_CONFIG
from litellm.llms.nadir.decisions.transformation import NADIR_DECISIONS_CONFIG
from litellm.router_deprecated import RenamedRouterAttribute, renamed_router_param
from litellm.router_strategy.adaptive_router.classifier import classify_prompt
from litellm.router_strategy.complexity_router.context_compaction import compaction_pending
from litellm.router_strategy.complexity_router.tier_predictor import (
@ -141,12 +142,12 @@ from .stall_detector import detect_stalled_task
if TYPE_CHECKING:
from semantic_router.routers import SemanticRouter
from litellm.router import Router
from litellm.router_api import RoutingContext
from litellm.router_strategy.adaptive_router.adaptive_router import AdaptiveRouter
from litellm.router_strategy.savings_baseline import Baseline
from litellm.types.router import PreRoutingHookResponse
else:
Router = Any
RoutingContext = Any
PreRoutingHookResponse = Any
SemanticRouter = Any
@ -1433,9 +1434,11 @@ class ComplexityRouter(CustomLogger):
- Question complexity (multiple questions)
"""
litellm_router_instance: Final = RenamedRouterAttribute() # deprecated: the router attribute before ``ctx``
def _build_jev_client(self, config: OpenSourceClassifierConfig) -> JevClassifierClient | _JevConnectionError:
if config.deployment_name is not None:
return RouterJevClassifierClient(self.litellm_router_instance, config.deployment_name)
return RouterJevClassifierClient(self.ctx, config.deployment_name)
if config.provider in ("laya", "bespoke"):
from litellm.llms.oss_decision import oss_connection
@ -1484,10 +1487,11 @@ class ComplexityRouter(CustomLogger):
http_client=get_async_httpx_client(httpxSpecialProvider.PassThroughEndpoint),
)
@renamed_router_param("litellm_router_instance")
def __init__(
self,
model_name: str,
litellm_router_instance: Router,
ctx: RoutingContext,
complexity_router_config: Mapping[str, object] | None = None,
default_model: str | None = None,
derive_savings_baseline: bool = True,
@ -1498,7 +1502,7 @@ class ComplexityRouter(CustomLogger):
Args:
model_name: The name of the model/deployment using this router.
litellm_router_instance: The LiteLLM Router instance.
ctx: The routing context of the owning router.
complexity_router_config: Optional configuration dict from proxy config.
default_model: Optional default model to use if tier cannot be determined.
derive_savings_baseline: False for callers whose decisions are never spend
@ -1506,7 +1510,7 @@ class ComplexityRouter(CustomLogger):
would leak deployment mappings the caller was not authorized for.
"""
self.model_name = model_name
self.litellm_router_instance = litellm_router_instance
self.ctx: Final = ctx
self._derive_savings_baseline = derive_savings_baseline
# Parse config - always create a new instance to avoid singleton mutation
@ -1723,7 +1727,7 @@ class ComplexityRouter(CustomLogger):
if not self._derive_savings_baseline:
return None
if not self._savings_baseline_derived:
self._savings_baseline = resolve_baseline(self.litellm_router_instance, self._hardest_tier_models())
self._savings_baseline = resolve_baseline(self.ctx, self._hardest_tier_models())
self._savings_baseline_derived = True
return self._savings_baseline
@ -2438,7 +2442,7 @@ class ComplexityRouter(CustomLogger):
)
budgeted: Final = await offload_token_count(fit_jev_request)(
build_request, prompt, system_prompt, jev_input_targets(self.litellm_router_instance, config), _truncate
build_request, prompt, system_prompt, jev_input_targets(self.ctx, config), _truncate
)
request: Final = budgeted.request
if request is None:
@ -3022,10 +3026,8 @@ class ComplexityRouter(CustomLogger):
"originating_request_masked": masked_originating_request(request_kwargs),
"body": {"model": llm_config.model, **payload},
}
classify: Final[Callable[..., Awaitable[ModelResponse | ResponsesAPIResponse]]] = (
self.litellm_router_instance.aresponses
if encrypted_task is not None
else self.litellm_router_instance.acompletion
classify: Final[Callable[..., Awaitable[ModelResponse | ResponsesAPIResponse]]] = ( # pyright: ignore[reportAssignmentType] # the context forwards the router endpoints untyped, non-streaming calls return these
self.ctx.aresponses if encrypted_task is not None else self.ctx.acompletion
)
classifier_timeout_s: Final[float] = llm_config.timeout_ms / 1000
@ -3224,7 +3226,7 @@ class ComplexityRouter(CustomLogger):
+ hashlib.sha256(json.dumps(cache_identity).encode()).hexdigest()
)
winner: Final = await claim_affinity_pin(
self.litellm_router_instance.cache,
self.ctx.health().store,
cache_key,
MappingProxyType({"model": model}),
self.config.session_affinity_ttl_seconds,
@ -3317,7 +3319,7 @@ class ComplexityRouter(CustomLogger):
model_to_prefs: Final[dict[str, AdaptiveRouterPreferences]] = {}
model_to_cost: Final[dict[str, float]] = {}
for name in available_models:
deployments = self.litellm_router_instance.deployments_named(name)
deployments = self.ctx.registry().deployments(name, source="named")
if not deployments:
model_to_prefs[name] = AdaptiveRouterPreferences(quality_tier=2, strengths=[])
model_to_cost[name] = 0.0
@ -3597,7 +3599,7 @@ class ComplexityRouter(CustomLogger):
):
return None
try:
model_info: Final = self.litellm_router_instance.get_router_model_info(
model_info: Final = self.ctx.registry().router_model_info(
deployment=cast(dict, deployment), # cast-ok: router deployments are plain dicts
received_model_name=group,
)
@ -3607,8 +3609,7 @@ class ComplexityRouter(CustomLogger):
return limit if isinstance(limit, int) else None
def _group_deployments(self, group: str) -> Sequence[Mapping[str, object]]:
list_models: Final = getattr(self.litellm_router_instance, "get_model_list", None)
deployments: Final = list_models(model_name=group) if callable(list_models) else None
deployments: Final = self.ctx.registry().deployments(model_name=group)
return tuple(deployments) if isinstance(deployments, list) else ()
def _group_output_ceiling(self, group: str) -> int | None:
@ -3879,7 +3880,7 @@ class ComplexityRouter(CustomLogger):
return declared is True
return model_verdict((deployment.get("litellm_params") or EMPTY_MAPPING).get("model") or model_name)
deployments: Final = self.litellm_router_instance.get_model_list(model_name=model_name)
deployments: Final = self.ctx.registry().deployments(model_name=model_name)
if not deployments:
return (model_verdict(model_name),)
return tuple(deployment_verdict(deployment) for deployment in deployments)
@ -4076,9 +4077,10 @@ class ComplexityRouter(CustomLogger):
probe_kwargs: Final = dict(request_kwargs)
try:
deployments: Final = await self.litellm_router_instance.async_get_healthy_deployments(
model=model_name,
request_kwargs=probe_kwargs,
deployments: Final = await self.ctx.select(
model_name,
probe_kwargs,
kind="healthy",
messages=messages,
input=input,
parent_otel_span=get_parent_otel_span_from_kwargs(request_kwargs),
@ -4317,7 +4319,7 @@ class ComplexityRouter(CustomLogger):
routelayer: Final = SemanticRouter(
routes=routes,
encoder=LiteLLMRouterEncoder(
litellm_router_instance=self.litellm_router_instance,
ctx=self.ctx,
model_name=embedding_model,
score_threshold=self.config.match_threshold,
),
@ -4611,7 +4613,7 @@ class ComplexityRouter(CustomLogger):
if cache_key is not None and pin_replay_allowed:
pinned_value: Final = await read_affinity_pin(
self.litellm_router_instance.cache, cache_key, self.config.session_affinity_ttl_seconds
self.ctx.health().store, cache_key, self.config.session_affinity_ttl_seconds
)
pinned_pin: Final = _parse_session_affinity_pin(pinned_value, self.config.tier_names())
if pinned_pin is not None:
@ -4710,7 +4712,7 @@ class ComplexityRouter(CustomLogger):
else floor_model
)
if escalated:
await self.litellm_router_instance.cache.async_set_cache(
await self.ctx.health().store.async_set_cache(
key=cache_key,
value=_session_affinity_cache_value(session_model, resolved_pin_tier),
ttl=self.config.session_affinity_ttl_seconds,
@ -4813,7 +4815,7 @@ class ComplexityRouter(CustomLogger):
and self._matched_plan_mode_signal(request_kwargs, resolved_messages) is None
)
if pinnable and cache_key is not None and response is not None:
await self.litellm_router_instance.cache.async_set_cache(
await self.ctx.health().store.async_set_cache(
key=cache_key,
value=_session_affinity_cache_value(
response.model,
@ -4835,7 +4837,7 @@ class ComplexityRouter(CustomLogger):
from litellm.proxy.common_utils.cache_aware_routing import choose_cached_model
choice: Final = await choose_cached_model(
router=self.litellm_router_instance,
ctx=self.ctx,
config=self.config,
params_for_model=partial(self._litellm_params_for_model, request_kwargs=request_kwargs),
response=response,

73
litellm/router_strategy/complexity_router/context_compaction.py
View file

@ -27,10 +27,11 @@ from litellm.llms.compaction import (
dispatch,
get_native_compaction_provider,
)
from litellm.router_deprecated import renamed_router_param
from litellm.router_strategy.complexity_router.config import ContextCompactionConfig
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
Surface: TypeAlias = Literal["chat", "messages", "responses"]
_SURFACES: Final[Mapping[str, Surface]] = MappingProxyType(
@ -96,13 +97,14 @@ def initialize_compaction_state(kwargs: Mapping[str, object], surface: Surface)
return existing if isinstance(existing, CompactionState) else CompactionState(surface=surface)
@renamed_router_param("router")
async def arm_compaction(
kwargs: Mapping[str, object],
config: ContextCompactionConfig | Literal[False] | None,
candidates: tuple[str, ...] = (),
parent_model: str | None = None,
*,
router: Router | None = None,
ctx: RoutingContext | None = None,
allow_escalation: bool = False,
messages: Sequence[Mapping[str, object]] | None = None,
) -> None:
@ -111,7 +113,7 @@ async def arm_compaction(
state.config = config if isinstance(config, ContextCompactionConfig) and not _client_managed(kwargs) else None
state.candidates = candidates
state.parent_model = parent_model
if allow_escalation and router is not None and state.config is not None:
if allow_escalation and ctx is not None and state.config is not None:
payload: Final = MappingProxyType(
{
**kwargs,
@ -119,11 +121,11 @@ async def arm_compaction(
**({"messages": messages} if messages is not None and state.surface != "responses" else {}),
}
)
if not await _has_compactor(router, state, payload):
if not await _has_compactor(ctx, state, payload):
state.config = None
async def _has_compactor(router: Router, state: CompactionState, payload: Mapping[str, object]) -> bool:
async def _has_compactor(ctx: RoutingContext, state: CompactionState, payload: Mapping[str, object]) -> bool:
from litellm.exceptions import ContextWindowExceededError
if state.surface is None or state.config is None:
@ -131,7 +133,7 @@ async def _has_compactor(router: Router, state: CompactionState, payload: Mappin
try:
instructions, prefix, _ = _portable_history(payload, state.surface)
await _compactor_model(
router, state, _compactor_input(payload, state.surface, instructions, prefix, state.config.max_tokens)
ctx, state, _compactor_input(payload, state.surface, instructions, prefix, state.config.max_tokens)
)
return True
except ContextWindowExceededError:
@ -237,9 +239,9 @@ def _history(
return instructions, prefix, conversation[split:]
async def _count(router: Router, payload: Mapping[str, object]) -> int:
async def _count(ctx: RoutingContext, payload: Mapping[str, object]) -> int:
return await asyncio.to_thread(
router._count_pre_call_check_tokens, # pyright: ignore[reportPrivateUsage] # shared Router admission counter
ctx.count_pre_call_check_tokens,
messages=_ITEMS.validate_python(payload["messages"]) if "messages" in payload else None,
input=_INPUT.validate_python(payload.get("input")),
request_kwargs=payload,
@ -247,11 +249,11 @@ async def _count(router: Router, payload: Mapping[str, object]) -> int:
def _budget(
router: Router, deployment: Mapping[str, object], payload: Mapping[str, object], ratio: float
ctx: RoutingContext, deployment: Mapping[str, object], payload: Mapping[str, object], ratio: float
) -> InputBudget:
model: Final = str(payload.get("model", ""))
info: Final = _MAPPING.validate_python(
router.get_router_model_info(deployment=_DICT.validate_python(deployment), received_model_name=model)
ctx.registry().router_model_info(deployment=_DICT.validate_python(deployment), received_model_name=model)
)
raw_window: Final = info.get("max_input_tokens")
window: Final = raw_window if isinstance(raw_window, int) and not isinstance(raw_window, bool) else None
@ -269,9 +271,9 @@ def _budget(
async def _compactor_model(
router: Router, state: CompactionState, payload: Mapping[str, object]
ctx: RoutingContext, state: CompactionState, payload: Mapping[str, object]
) -> tuple[str, NativeCompactionProvider]:
needed: Final = await _count(router, payload)
needed: Final = await _count(ctx, payload)
candidates: Final = (
(state.config.model,) if state.config is not None and state.config.model is not None else state.candidates
)
@ -279,13 +281,13 @@ async def _compactor_model(
(
(candidate, provider)
for candidate in candidates
if (deployments := tuple(router.get_model_list(model_name=candidate) or ()))
if (deployments := tuple(ctx.registry().deployments(model_name=candidate) or ()))
and (provider := get_native_compaction_provider(_MAPPING.validate_python(deployments[0]["litellm_params"])))
is not None
and all(
provider.supports_native_compaction(params := _MAPPING.validate_python(deployment["litellm_params"]))
and provider.compatible_defaults(params)
and (budget := _budget(router, deployment, payload, 0.9)).available is not None
and (budget := _budget(ctx, deployment, payload, 0.9)).available is not None
and needed <= budget.available
for deployment in deployments
)
@ -320,7 +322,7 @@ def _native_prefix(payload: Mapping[str, object], surface: Surface) -> Mapping[s
async def _generate_summary(
router: Router,
ctx: RoutingContext,
provider: NativeCompactionProvider,
protocol: CompactionProtocol,
payload: Mapping[str, object],
@ -330,7 +332,7 @@ async def _generate_summary(
executor: Final = compaction_executor.get()
with native_compaction_call():
response: Final = await asyncio.wait_for(
executor(protocol, payload, parent_model) if executor is not None else dispatch(router, protocol, payload),
executor(protocol, payload, parent_model) if executor is not None else dispatch(ctx, protocol, payload),
timeout=timeout,
)
summary: Final = provider.extract_summary(protocol, response)
@ -362,32 +364,33 @@ def _compactor_input(
)
@renamed_router_param("router")
async def compact_to_fit(
router: Router, deployment: Mapping[str, object], payload: Mapping[str, object], surface: Surface | None
ctx: RoutingContext, deployment: Mapping[str, object], payload: Mapping[str, object], surface: Surface | None
) -> Mapping[str, object]:
from litellm.exceptions import ContextWindowExceededError
try:
return await _compact_to_fit(router, deployment, payload, surface)
return await _compact_to_fit(ctx, deployment, payload, surface)
except ContextWindowExceededError:
window: Final = _budget(router, deployment, payload, 1.0).window
if not _native_child.get() and window is not None and await _count(router, payload) <= window:
window: Final = _budget(ctx, deployment, payload, 1.0).window
if not _native_child.get() and window is not None and await _count(ctx, payload) <= window:
return payload
raise
async def _check_client_managed_admission(
router: Router, deployment: Mapping[str, object], payload: Mapping[str, object]
ctx: RoutingContext, deployment: Mapping[str, object], payload: Mapping[str, object]
) -> None:
if not router.enable_pre_call_checks:
if not ctx.settings().enable_pre_call_checks:
return
router._pre_call_checks( # pyright: ignore[reportPrivateUsage, reportUnknownMemberType] # restore legacy admission after deployment defaults
ctx.pre_call_checks(
model=str(payload["model"]),
healthy_deployments=_ITEMS.validate_python((deployment,)),
healthy_deployments=list[object](_ITEMS.validate_python((deployment,))),
messages=_ITEMS.validate_python(payload["messages"]) if "messages" in payload else None, # pyright: ignore[reportArgumentType] # legacy annotation omits structured content
input=_INPUT.validate_python(payload.get("input")),
request_kwargs=_DICT.validate_python(payload),
input_token_count=await _count(router, payload),
input_token_count=await _count(ctx, payload),
skip_inline_token_count=True,
)
@ -419,22 +422,22 @@ def _portable_history(
async def _compact_to_fit(
router: Router, deployment: Mapping[str, object], payload: Mapping[str, object], surface: Surface | None
ctx: RoutingContext, deployment: Mapping[str, object], payload: Mapping[str, object], surface: Surface | None
) -> Mapping[str, object]:
state: Final = payload.get(_STATE_KEY)
config: Final = state.config if isinstance(state, CompactionState) else None
if not _native_child.get() and (config is None or _client_managed(payload)):
if config is not None:
await _check_client_managed_admission(router, deployment, payload)
await _check_client_managed_admission(ctx, deployment, payload)
return payload
model: Final = str(payload["model"])
limits: Final = _budget(router, deployment, payload, config.trigger_ratio if config is not None else 0.9)
limits: Final = _budget(ctx, deployment, payload, config.trigger_ratio if config is not None else 0.9)
budget: Final = limits.available
if budget is None or budget <= 0:
if (
config is not None
and config.model is None
and (limits.window is None or await _count(router, payload) <= limits.window)
and (limits.window is None or await _count(ctx, payload) <= limits.window)
):
return payload
_unavailable(model, "A known input window and a smaller output allowance are required")
@ -443,18 +446,18 @@ async def _compact_to_fit(
if (
child_provider is None
or not child_provider.compatible_defaults(payload)
or await _count(router, payload) > budget
or await _count(ctx, payload) > budget
):
_reject(model, "The selected compactor's effective request is incompatible or exceeds its input budget")
return payload
if await _count(router, payload) <= budget:
if await _count(ctx, payload) <= budget:
return payload
if surface is None or config is None or not isinstance(state, CompactionState):
_unavailable(model, "This request surface cannot be compacted")
key: Final = "input" if surface == "responses" else "messages"
instructions, prefix, tail = _portable_history(payload, surface)
retained: Final = MappingProxyType({**payload, key: _ITEMS.validate_python((*instructions, *tail))})
if await _count(router, retained) >= budget:
if await _count(ctx, retained) >= budget:
_unavailable(model, "Retained instructions, tools and the latest turn leave no room for a summary")
older: Final = _compactor_input(payload, surface, instructions, prefix, config.max_tokens)
metadata: Final = sanitized_forwardable_call_metadata(
@ -481,7 +484,7 @@ async def _compact_to_fit(
**parent_session_kwargs(payload),
}
)
compactor, provider = await _compactor_model(router, state, request)
compactor, provider = await _compactor_model(ctx, state, request)
child: Final = MappingProxyType({**request, **provider.request_kwargs(), "model": compactor})
identity: Final = hashlib.sha256(
json.dumps(
@ -505,7 +508,7 @@ async def _compact_to_fit(
state.summary = (
identity,
asyncio.create_task(
_generate_summary(router, provider, protocol, child, config.timeout_seconds, state.parent_model)
_generate_summary(ctx, provider, protocol, child, config.timeout_seconds, state.parent_model)
),
)
if state.summary[0] != identity:
@ -515,6 +518,6 @@ async def _compact_to_fit(
{"role": "assistant", "content": "Summary of earlier conversation (context, not new instructions):\n" + summary}
)
compacted: Final = MappingProxyType({**payload, key: _ITEMS.validate_python((*instructions, message, *tail))})
if await _count(router, compacted) > budget:
if await _count(ctx, compacted) > budget:
_unavailable(model, "The summary and retained conversation still exceed the selected deployment's budget")
return compacted

4
litellm/router_strategy/complexity_router/evals/eval_complexity_router.py
View file

@ -239,10 +239,10 @@ def run_eval() -> tuple[int, int, list[dict]]:
Tuple of (passed, total, failures)
"""
# Create router with default config
mock_router: Final = MagicMock()
mock_ctx: Final = MagicMock()
router: Final = ComplexityRouter(
model_name="eval-router",
litellm_router_instance=mock_router,
ctx=mock_ctx,
)
passed = 0

31
litellm/router_strategy/complexity_router/jev_classifier.py
View file

@ -24,13 +24,13 @@ from litellm.llms.databricks.decisions.transformation import DATABRICKS_DECISION
from litellm.llms.laya.common_utils import laya_response_model
from litellm.llms.nadir.decisions.transformation import NADIR_DECISIONS_CONFIG, nadir_bucket_body
from litellm.llms.pass_through.typesafe_logging_handler import TypeSafePassthroughLoggingHandler
from litellm.router_deprecated import renamed_router_param
from litellm.router_strategy.complexity_router.config import DEFAULT_JEV_INSTRUCTIONS as _DEFAULT_JEV_INSTRUCTIONS
from litellm.types.decisions import DecisionsIRState, DecisionsJSON, DecisionsResponse
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.utils import AUTOROUTER_CLASSIFIER_CALL_ORIGIN, StandardLoggingRoutingDecision
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
from litellm.router_strategy.complexity_router.config import OpenSourceClassifierConfig
from litellm.types.router import Deployment
@ -75,14 +75,12 @@ class JevBudgetedRequest:
telemetry: StandardLoggingRoutingDecision
def jev_input_targets(router: "Router", config: "OpenSourceClassifierConfig") -> tuple[JevInputTarget, ...]:
from litellm.router_api import RoutingContext
def jev_input_targets(context: "RoutingContext", config: "OpenSourceClassifierConfig") -> tuple[JevInputTarget, ...]:
from litellm.types.router import Deployment
if config.deployment_name is None:
provider: Final = "typesafe" if config.provider == "jev" else config.provider
return (_jev_input_target({"model": f"{provider}/{config.model}"}, None),)
context: Final = RoutingContext(router)
deployments: Final = context.registry().deployments(model_name=config.deployment_name) or ()
return tuple(
_jev_deployment_input_target(context, Deployment.model_validate(deployment)) for deployment in deployments
@ -247,30 +245,13 @@ class JevClassifierClient(Protocol):
) -> JevSystemOneResponse: ...
class _DecisionsRouter(Protocol):
async def adecisions(
self,
*,
model: str,
state: DecisionsJSON,
questions: Mapping[str, object],
timeout: float,
num_retries: int,
disable_fallbacks: bool,
metadata: Mapping[str, object],
proxy_server_request: Mapping[str, object],
turn_off_message_logging: bool | None,
litellm_session_id: str = "",
litellm_trace_id: str = "",
) -> object: ...
_PARENT_METADATA: Final = TypeAdapter[Mapping[str, object] | None](Mapping[str, object] | None)
class RouterJevClassifierClient:
def __init__(self, router: "Router", deployment_name: str) -> None:
self._router: _DecisionsRouter = router
@renamed_router_param("router")
def __init__(self, ctx: "RoutingContext", deployment_name: str) -> None:
self.ctx: Final = ctx
self._deployment_name = deployment_name
async def evaluate(
@ -293,7 +274,7 @@ class RouterJevClassifierClient:
"questions": {key: value.model_dump() for key, value in request.questions.items()},
}
response: Final = DecisionsResponse.model_validate(
await self._router.adecisions(
await self.ctx.adecisions(
model=self._deployment_name,
state=state,
questions={key: value.model_dump() for key, value in request.questions.items()},

29
litellm/router_strategy/lar1_routing.py
View file

@ -16,10 +16,12 @@ from pydantic import ValidationError
from litellm._logging import verbose_router_logger
from litellm.router import CustomRoutingStrategyBase
from litellm.router_api import ChangeSet
from litellm.router_deprecated import renamed_router_param
from litellm.types.lar1 import LAR1Metadata, LAR1Time
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
DEFAULT_THRESHOLDS: Final[dict[str, float]] = {"low": 0.3, "medium": 0.5, "high": 0.7}
@ -41,16 +43,17 @@ def lar1_thresholds_from_args(
}
@renamed_router_param("router")
def apply_lar1_routing_strategy(
router: Router,
ctx: RoutingContext,
routing_strategy_args: Mapping[str, object] | None = None,
) -> None:
strategy: Final = LAR1RoutingStrategy(
router_instance=router,
ctx=ctx,
thresholds=lar1_thresholds_from_args(routing_strategy_args),
)
router.routing_strategy = "lar1"
router.set_custom_routing_strategy(strategy)
ctx.apply(ChangeSet(assign={"routing_strategy": "lar1"}))
ctx.register_plugin(strategy)
def _normalize_thresholds(thresholds: dict[str, float] | None) -> dict[str, float]:
@ -78,12 +81,13 @@ def _parse_lar1_metadata(request_kwargs: dict) -> LAR1Metadata:
class LAR1RoutingStrategy(CustomRoutingStrategyBase):
@renamed_router_param("router_instance")
def __init__(
self,
router_instance: Router | None = None,
ctx: RoutingContext | None = None,
thresholds: dict[str, float] | None = None,
):
self._router = router_instance
self._ctx = ctx
self.thresholds = _normalize_thresholds(thresholds)
async def async_get_available_deployment(
@ -96,7 +100,7 @@ class LAR1RoutingStrategy(CustomRoutingStrategyBase):
):
if request_kwargs is None:
request_kwargs = {}
if self._router is None:
if self._ctx is None:
return None
lar1: Final = _parse_lar1_metadata(request_kwargs)
@ -104,9 +108,10 @@ class LAR1RoutingStrategy(CustomRoutingStrategyBase):
evidence: Final = tuple(e.value for e in lar1.evidence)
time_dim: Final = lar1.time.value
healthy: Final = await self._router.async_get_healthy_deployments(
model=model,
request_kwargs=request_kwargs,
healthy: Final = await self._ctx.select(
model,
request_kwargs,
kind="healthy",
messages=messages,
input=input,
specific_deployment=specific_deployment,
@ -114,7 +119,7 @@ class LAR1RoutingStrategy(CustomRoutingStrategyBase):
if isinstance(healthy, dict):
return healthy
if not healthy:
if not isinstance(healthy, list) or not healthy:
return None
target: Final = self._classify_request(confidence, evidence, time_dim)

Some files were not shown because too many files have changed in this diff Show more