scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-02 07:13:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 49

Merge remote-tracking branch 'origin/main' into litellm_passthrough_auth_guardrails

Browse source
This commit is contained in:
yucheng 2026-10-31 01:16:50 -07:00
commit d90609236e
455 changed files with 25772 additions and 5012 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

77
.agents/skills/database-access/SKILL.md
View file

@ -1,54 +1,73 @@
---
name: database-access
description: Add database queries, extract existing Prisma calls into repositories, or explicitly port queries to Rust and SQLx without expanding migration scope
description: Organize database repositories and proxy dependency wiring, manage connection ownership, or port Prisma operations to Rust and SQLx without expanding migration scope
metadata:
internal: true
---
# Database access
## Situations
- Adding a query or an operation to a repository, read [adding a query](references/python-repository.md#adding-a-query)
- Moving existing Prisma calls behind a repository, read [moving existing access](references/python-repository.md#moving-existing-access-into-a-repository)
- Porting a repository from Prisma to Rust and SQLx, read [porting to SQLx](references/python-repository.md#porting-a-repository-to-sqlx)
- Injecting a repository into a route, read [Python and FastAPI](references/python-fastapi.md) or [Rust and Axum](references/rust-axum.md)
## Adding new database access
- Put Python database access behind an operation in `litellm/repositories`
- Give callers typed inputs and outputs, not Prisma clients, models, or query dictionaries
- Keep query execution and backend selection inside the repository
- Give each repository its own FastAPI provider, so a route builds only the repositories it uses
- A provider borrows existing handles and performs no database I/O during construction
- Define the operations a route calls as a Protocol in `litellm/models/<feature>.py`, next to its domain types
- The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
- Parsing and serialization of stored values live in the repository
- Add a Protocol when a repository gets a provider, not up front for every repository
- Keep query execution inside the repository, with one implementation per repository and no backend switch or fallback
- A repository is either all Prisma or all native, never a mix of both
- Prisma is fine for the implementation, adding a query does not require a Rust port
- Native Rust callers use `litellm-rust/crates/db` directly
## Asked to move existing access behind a repository
## Framework wiring
- Identify the calls being moved and their transaction boundary
- Move the existing Prisma implementation into the relevant repository
- Replace those direct database calls with repository operations
- Preserve return values, errors, and transaction behavior, with focused regression tests
- Keep this an extraction, not a backend migration or unrelated cleanup
- Python routes receive a per-repository dependency such as `UserBannerRepositoryDep` and call its operations, such as `banners.get_banner()`
- `DatabaseDep` lives in `litellm/proxy/deps/database.py`, repository providers in `litellm/proxy/deps/repositories.py`
- A provider returns the feature's Protocol, so routes never see which backend implements it
- Dependencies used by one feature only, like the banner audit binding, stay next to that feature's routes
- Repositories never import FastAPI, so jobs and helpers can use them without a request
- Use named FastAPI providers with `Annotated[..., Depends(...)]`, ordinary helpers and jobs receive explicit arguments
- Preserve each route's no-database behavior, authorization precedence and background-task scheduling
- Read [Python and FastAPI](references/python-fastapi.md) for provider examples, lifespan state and route verification
- Read [Rust and Axum](references/rust-axum.md) when adding database access to Rust handlers
## An equivalent Rust or SQLx query already exists
## Connection ownership
- Leave it and the current backend selection unchanged unless integration is explicitly requested
- Its existence alone does not justify switching a Prisma caller to Rust
- Repository extraction can keep using Prisma even when the Rust query is available
- Keep connection setup and shutdown separate from repositories, which own no pools
- Reuse existing handles, never connect per request or disconnect from a repository operation
- Give each pool one owner for setup and shutdown, borrowed handles never close it
- Finish requests, jobs and queued database work before closing connections
- Clean up resources opened before a startup failure, without a second disconnect fallback
- Add a backend to the existing lifecycle owner and pass its handle only to repositories that need it
- Preserve cancellation, timeout, error and transaction semantics across the native bridge
## Asked to port a Prisma query to Rust or SQLx
## Changing ownership or reloading resources
- Identify the existing inputs, outputs, errors, and transaction boundary
- Check `litellm-rust/crates/db` for an equivalent query and reuse it if its behavior matches
- Otherwise follow `litellm-rust/crates/db/AGENTS.md`
- Put static SQL in `queries/` and compile-check it with SQLx
- Keep typed inputs and reusable row projections in `src/`
- Refresh SQLx query metadata after SQL changes
- Keep Python conversion in the Python bridge, not in `litellm-rust/crates/db`
- When caller integration is requested, expose the Rust operation through the bridge and call it from the repository
- A query-only port can land without switching callers
- Verify behavior against real PostgreSQL before switching callers
- Check relevant null handling, conflicts, ordering, errors, and transaction semantics
- Remove the old implementation only when no callers need it
- Inventory initialization, reload, tests, background jobs and shutdown together
- Move every writer when changing authoritative storage, copying a replaceable global into state is insufficient
- A frozen repository does not make its clients immutable
- Decide whether each request captures one resource generation or reads it again during processing
- Build and validate replacements before publishing them, preserving the old generation on failure
- Serialize reloads and coordinate publication with shutdown
- Drain streams and background users before retiring old resources, retaining every generation grows resource use
- Define multi-worker coordination separately from an in-process swap
- Choose mechanisms such as `ArcSwap` only after defining publication, capture and retirement
## Migration context and boundaries
- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations incrementally with Rust APIs
- Mixed Prisma and Rust implementations are expected during the transition, unrelated existing calls can stay as they are
- Both backends may have separate pools through PgBouncer, size them together and verify pooling-mode compatibility
- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations with Rust APIs one repository at a time
- Prisma and native repositories coexist during the transition, unrelated existing calls can stay as they are
- Prisma and SQLx keep separate client pools to PgBouncer, which pools connections to PostgreSQL
- Size both layers together and preserve reader/writer routing and pooling-mode compatibility
- PgBouncer does not share transactions between Prisma and SQLx, keep each atomic unit of work on one backend and transaction
- Never retry a failed Rust write through Prisma when it may already have committed
- Prisma owns schema migrations today, eventual SQLx migration ownership and Prisma removal are separate planned steps, not part of a query port

101
.agents/skills/database-access/references/python-fastapi.md Normal file
View file

@ -0,0 +1,101 @@
# Python and FastAPI
## Existing pattern
- `UserBannerStore` in `litellm/models/user_banner.py` is the Protocol the banner routes call
- `get_database` in `litellm/proxy/deps/database.py` reads the lifespan-owned handle from request state
- `get_user_banner_repository` in `litellm/proxy/deps/repositories.py` builds the banner repository from that handle
- Choose each repository's replica routing once in its provider, never at call sites
- A handler receives `UserBannerStore | None`, never a Prisma client or a concrete repository
- `UserBannerRepository` implements `UserBannerStore` structurally, keeping the existing Prisma table, serialization and configuration-sync behavior
- The banner audit provider binds the legacy audit callable at request time, without performing audit I/O
- A native repository can later implement the same Protocol without changing handler signatures, see [porting a repository to SQLx](python-repository.md#porting-a-repository-to-sqlx)
## Dependency injection
- Add one provider and one `Annotated` alias per repository in `litellm/proxy/deps/repositories.py`
- Annotate the provider and alias with the feature's Protocol, not the concrete repository
- A route depends only on the repositories it calls, so a request never builds repositories it does not use
- Do not build repositories once at lifespan, a reconnect can replace the Prisma client they wrap
```python
class UserBannerStore(Protocol):
async def get_banner(self) -> UserBanner: ...
async def save_banner(self, banner: UserBanner) -> None: ...
```
```python
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
return None if database is None else UserBannerRepository(database.prisma)
UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)]
```
- FastAPI caches each provider within a request, so routes and sub-dependencies share one instance
- Ordinary helpers and jobs receive explicit arguments, never unresolved `Depends` defaults
- A route still reading the `prisma_client` global or building a repository inline switches to the repository's `...RepositoryDep`, keeping its no-database behavior and error precedence
```python
async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner:
if banners is None:
return UserBanner()
return await banners.get_banner()
```
## Lifespan state
- Prisma startup owns connection setup, authentication refresh, health checks and migrations
- `Database` holds the initialized Prisma client and the `NativeDatabase` handle, without holding fields per feature
- The native handle opens no pool yet, its `connect` and `close` are noops
- The proxy lifespan builds the handle with `Database.open`, registers `Database.aclose` on its exit stack and publishes it in request state
- Callers only open and close `Database`, which closes Prisma and the native handle together, and `open` closes Prisma itself when the native connect fails
- Starlette shallow-copies lifespan state into request state, which is distinct from `app.state`
- The database module never imports a repository, legacy globals borrow its handle without owning shutdown
- Drain requests, jobs, logging, spend and audit work before database teardown
- The exit stack closes the handle on exit, including startup failures after the client is created
- When adding native resources, initialize them in the serving process, respecting runtime fork restrictions
- Await native shutdown instead of depending on Python garbage collection
- Apply the [connection ownership rules](../SKILL.md#connection-ownership) when changing the owner
## Timing and errors
- FastAPI resolves dependencies before entering the handler
- Handler exception handling cannot catch failures raised during dependency resolution
- A universal required-database dependency can override authorization and validation behavior
- Banner GET without a database returns the default disabled banner
- Banner PATCH checks the admin role before returning the existing missing-database error
- Invalid stored banner data falls back to the default, but storage failures still propagate
- Banner writes finish before the success audit is queued as a FastAPI background task
- Queue post-response work with `BackgroundTasks`, not bare `asyncio.create_task`, so the in-flight drain waits for it before the database closes
- Do not move background work into an awaited provider merely to remove an import
## Verification
- Exercise real FastAPI handlers using `dependency_overrides` for external boundaries
- Keep production parsing and persistence code real
- Restore overrides after each test
- Verify complete response bodies, status codes and OpenAPI against the base
- Include combined failure conditions to pin error precedence
- Check saved values, fresh revisions, audit contents and failed-write behavior
- Check lifespan state with a running lifespan, not only a TestClient that skips startup
- Verify providers do not connect, query, disconnect or audit eagerly
- Use real PostgreSQL and live proxy requests when changing persistence behavior
## Scope
- Preserve direct Python call sites or migrate them explicitly
- Inventory imports, state writers, side effects and error ordering before migrating a feature
- Preserve configuration sync and serialization, and state which legacy reads remain
- Do not migrate every route in a large module just to establish the pattern
- Do not invent providers for resources without consumers
## Framework references
- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/) and [Starlette lifespan state](https://starlette.dev/lifespan/)
- [`SessionDep`-style `Annotated` aliases](https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-session-dependency), which `DatabaseDep` and `UserBannerRepositoryDep` follow
- [Dependency composition and caching](https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/)
- [Background tasks and dependencies with `yield`](https://fastapi.tiangolo.com/advanced/advanced-dependencies/#background-tasks-and-dependencies-with-yield-technical-details)
- [Dependency overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
- [Testing lifespan](https://fastapi.tiangolo.com/advanced/testing-events/)

119
.agents/skills/database-access/references/python-repository.md Normal file
View file

@ -0,0 +1,119 @@
# Python repositories
## Adding a query
Use this when a feature needs a read or write that no repository offers yet, including one more operation on an existing repository
### Before starting
- Search `litellm/repositories` for a repository that already owns the table, and extend it instead of adding a second one
- Check `litellm/models/<feature>.py` for an existing Protocol and domain types
- Decide the route's behavior when no database is configured, and where that check sits relative to authorization and validation
### Steps
1. Domain types and Protocol in `litellm/models/<feature>.py`
- The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
- Add the Protocol together with the provider, not for repositories nothing injects
2. Repository in `litellm/repositories/<feature>_repository.py`
- Implement the operation with Prisma, a Rust port is not required
- Compose the Prisma client and type the tables it touches with `TableActions` from `prisma_protocols.py`, never inherit `PrismaTableRepository`
- Parse and serialize stored values here, so callers only see domain types
- Never import FastAPI, connect, disconnect or choose replicas per call
3. Provider and `Annotated` alias in `litellm/proxy/deps/repositories.py`, see [dependency injection](python-fastapi.md#dependency-injection)
4. The route takes `<Feature>RepositoryDep`, handles `None` with the behavior decided above, and calls the operation
5. Jobs and helpers that need the operation receive the repository as an explicit argument, built by their caller from the handle it already holds
### Done when
- Handler tests drive the real route with `dependency_overrides` and the real repository code, see [verification](python-fastapi.md#verification)
- Tests cover the no-database path, error precedence, and what was actually stored
- Persistence changes are checked against real PostgreSQL through a live proxy
### Out of scope
- Porting the query to Rust, see [porting a repository to SQLx](#porting-a-repository-to-sqlx)
- Moving unrelated direct Prisma calls in the same module
## Moving existing access into a repository
Use this when code calls Prisma directly, through `prisma_client.db.<table>` or a helper, and the calls should live behind a repository. The backend stays Prisma
### Before starting
- List every call being moved, its callers, and the transaction or batch it runs in
- List the module's imports, global state writers, side effects such as audit and config sync, and error ordering
- Record current return values, raised errors and response bodies, these are what the change must preserve
### Steps
1. Move the existing Prisma code into the relevant repository in `litellm/repositories` without rewriting the queries
2. Add domain types and a Protocol in `litellm/models/<feature>.py` when a route will inject the repository
3. Replace each direct call with the repository operation, through a provider for routes and explicit arguments for jobs and helpers, see [Python and FastAPI](python-fastapi.md)
4. Calls sharing a transaction move into the same repository or unit of work in one change
5. State which legacy reads remain and why, when the change does not move all of them
### Done when
- Focused regression tests pin the preserved return values, errors and transaction behavior
- Response bodies, status codes and OpenAPI match the base
- Configuration sync and serialization behave the same, including invalid stored data
### Out of scope
- Switching the backend, even when an equivalent Rust query exists
- Converting other routes in the module just to establish the pattern
- Cleanups unrelated to the moved calls
## Porting a repository to SQLx
Status: TBD. No repository has been ported yet. `database.native` is a `NativeDatabase` from `crates/python-bridge/src/db.rs` whose `connect` and `close` are noops that open no pool, and the PyO3 repository surface does not exist. A real pool takes typed writer/reader settings and a token provider, never the Prisma `DATABASE_URL`, which carries Prisma-only params and rotating IAM tokens. The constraints below are settled, the steps are a sketch to revise after the first port lands
Use this only when a port is explicitly requested. Repository extraction does not require one
### When not to port
- An equivalent Rust or SQLx query existing is not a reason to switch a Prisma caller
- Leave existing backend selection unchanged unless integration is explicitly requested
- Never port part of a repository, it is either all Prisma or all native
### Before starting
- Identify the repository's inputs, outputs, errors and transaction boundary
- List repositories that share a transaction with it, they move together
- Check `litellm-rust/crates/db` for an equivalent query and reuse it when its behavior matches
### Steps
1. Rust queries, following `litellm-rust/crates/db/AGENTS.md`
- Static SQL in `queries/`, compile-checked with SQLx, typed inputs and row projections in `src/`
- Dynamic filters and sort keys become closed Rust enums that select static SQL, every value is bound
- Refresh SQLx query metadata after SQL changes
2. PyO3 surface exposing typed operations, never SQL text, table names or column names from Python. TBD: where it lives and how the handle reaches it
3. Native repository implementing the same Protocol in `litellm/repositories`
- Converts typed PyO3 results to domain types, preserving defaults, invalid-data handling and storage errors
- Python conversion stays in the bridge, not in `crates/db`
4. Switch the provider in `litellm/proxy/deps/repositories.py`, the Protocol and routes stay unchanged
```diff
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
- return None if database is None else UserBannerRepository(database.prisma)
+ return None if database is None else NativeUserBannerRepository(database.native)
```
5. Delete the Prisma repository in the same change and switch every caller, including jobs and helpers
### Done when
- Behavior is verified against real PostgreSQL, covering null handling, conflicts, ordering, errors and transaction semantics
- `DATABASE_URL=... cargo test -p litellm-db --locked -- --include-ignored` passes against a Prisma-migrated database
- Live proxy requests show unchanged responses. TBD: a Python-side real-PostgreSQL test harness
### Rollback
- Roll back the release, never retry an uncertain native write through Prisma
### Out of scope
- SQLx owning schema migrations, and removing Prisma
- Converting unrelated routes to dependency injection

4
.agents/skills/database-access/references/rust-axum.md Normal file
View file

@ -0,0 +1,4 @@
# Rust and Axum
- This should use same `crates/db` what is exposed to Python world.
- Details are TBD.

1
.github/workflows/test-unit.yml vendored
View file

@ -577,6 +577,7 @@ jobs:
tests/unit/proxy/rerank_endpoints
tests/unit/proxy/realtime_endpoints
tests/unit/proxy/ui_crud_endpoints
tests/unit/proxy/deps
tests/unit/proxy/config_resolvers
tests/unit/proxy/utils
python-version: "3.12"

2
AGENTS.md
View file

@ -90,6 +90,8 @@ Do not put names of customers or customer company names in code, PR descriptions
CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
For database operations, connection ownership and proxy dependency wiring, follow [database-access](.agents/skills/database-access/SKILL.md)
Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No `UPDATE`, `DELETE` or `MERGE`, and no `INSERT ... SELECT`: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. `tests/code_coverage_tests/check_migrations_no_data_rewrites.py` enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement `-- data-migration-ok: <what bounds it>`
Follow these coding conventions for new/updated code (a three-line fix in a legacy file shouldn't trigger huge drive-by refactors):

2
CONTRIBUTING.md
View file

@ -276,6 +276,8 @@ When the provider reports its own usage or spend numbers, compare the proxy's nu
Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution, and it is where PRs get reworked most often. Follow these rules
**Preserve zero prices.** In [`model_prices_and_context_window.json`](model_prices_and_context_window.json), an explicit `0.0` means that the priced operation is free. A missing field or `null` means no rate is specified, so the relevant calculator may use a fallback. For example, when no batch input rate is configured, batch cache writes use `cache_creation_input_token_cost` if set, otherwise the regular input rate, then apply the batch discount. A zero ordinary input rate does not make separately priced cache reads or writes free. Test `rate is None` before substituting a fallback, since `row.get("cache_creation_input_token_cost") or fallback` would discard a valid zero. Speech keeps character pricing ahead of token pricing even when the character rate is zero. If neither rate is configured, speech cost calculation raises an error. Its model lookup preserves unset rates instead of treating compatibility defaults as explicit zero prices. Perplexity keeps a configured legacy search rate ahead of the current search rate, including zero. When registering custom prices, `None` leaves an existing rate unchanged and `0.0` overrides it
**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. [`ARCHITECTURE.md`](ARCHITECTURE.md) shows where the layers live, and the decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) explain why they are shaped that way. For example, [the ADR on provider usage fields](docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md) covers how provider-specific usage fields reach cost tracking. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
**Bill a new provider usage field from the usage object.** Leave the field on the usage object as an extra and read it in that provider's `cost_calculator.py`. A provider without one gets a new `cost_calculator.py` and an arm for it in `cost_per_token` (`litellm/cost_calculator.py`), which picks the calculator by provider. If the provider's transform builds `Usage` from named fields, as the Anthropic, Gemini, and Bedrock Converse ones do, pass the new field there too. Streamed requests rebuild usage in `ChunkProcessor.calculate_usage` (`litellm/litellm_core_utils/streaming_chunk_builder_utils.py`), which does not copy extras yet the way the Responses-to-chat bridge does (minus a short exclusion list), so until it does, bill the field on streams by naming it there, as Anthropic's `inference_geo` is. A provider that reports its own charge for the request, as OpenRouter does in `usage.cost`, is billed at that charge from `_hidden_params` before any calculator runs, unless the deployment sets its own prices. When the field counts a built-in tool such as web search, add the provider to `get_cost_for_web_search_request` in `litellm/llms/__init__.py`, and add a gate to `litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py` only when no existing gate fires. Then show the field's cost in your proof on every endpoint it reaches, streaming included

31
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -10,6 +10,7 @@ from dataclasses import replace as dataclasses_replace
from datetime import datetime, timedelta
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, List, Literal, Optional, Protocol, Tuple, cast
from pydantic import TypeAdapter, ValidationError
from litellm_enterprise.proxy.common_utils.managed_object_attribution import (
ManagedObjectCreatorAttribution,
@ -50,6 +51,7 @@ def _error_names_resource(error: Exception, resource_id: str) -> bool:
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
RECENT_BATCH_POLL_WINDOW: Final = timedelta(hours=48)
GIVE_UP_POLLING_AFTER: Final = timedelta(days=2 * MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
_FILE_OBJECT_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
PROVIDER_TERMINAL_BATCH_STATUSES: Final[Tuple[str, ...]] = (
@ -550,6 +552,10 @@ class CheckBatchCost:
llm_provider: str,
bare_model_name: str,
) -> Optional[Tuple[str, str]]:
stored_model_id: Final = self._get_stored_model_id(job)
if stored_model_id is not None and self._batch_deployment_exists(stored_model_id):
return stored_model_id, job.unified_object_id
deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
if deployment_id is None:
verbose_proxy_logger.info(
@ -662,23 +668,36 @@ class CheckBatchCost:
)
@staticmethod
def _get_input_file_id(job: "ManagedObjectRow") -> Optional[str]:
from litellm.types.utils import LiteLLMBatch
def _parse_file_object(job: "ManagedObjectRow") -> Optional[dict[str, object]]:
file_object: Final = job.file_object
try:
parsed_file_object: Final[object] = (
return _FILE_OBJECT_ADAPTER.validate_python(
json.loads(file_object) if isinstance(file_object, str) else file_object
)
except (json.JSONDecodeError, ValueError):
except (json.JSONDecodeError, ValueError, ValidationError):
return None
if not isinstance(parsed_file_object, dict):
@classmethod
def _get_input_file_id(cls, job: "ManagedObjectRow") -> Optional[str]:
from litellm.types.utils import LiteLLMBatch
parsed_file_object: Final = cls._parse_file_object(job)
if parsed_file_object is None:
return None
try:
return LiteLLMBatch.model_validate(parsed_file_object).input_file_id
except Exception:
return None
@classmethod
def _get_stored_model_id(cls, job: "ManagedObjectRow") -> Optional[str]:
"""The deployment that created the batch, written into file_object by the managed
files hook at create time. Older rows have no such key and fall back to matching
the bare model name against the router."""
parsed_file_object: Final = cls._parse_file_object(job)
model_id: Final = parsed_file_object.get("model_id") if parsed_file_object else None
return model_id if isinstance(model_id, str) and model_id else None
async def _track_completed_batch_cost(
self,
job: "ManagedObjectRow",

50
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -21,6 +21,7 @@ from typing import (
TypedDict,
Union,
cast,
runtime_checkable,
)
from uuid import NAMESPACE_URL, uuid5
@ -374,6 +375,12 @@ class _ManagedResourcePrismaClient(Protocol):
def db(self) -> _ManagedResourceDatabase: ...
@runtime_checkable
class _WriterRoutedDatabase(Protocol):
@property
def writer(self) -> _ManagedResourceDatabase: ...
class _SchedulerWithJobLookup(Protocol):
def get_job(self, job_id: str) -> object: ...
@ -479,6 +486,11 @@ def _managed_object_table(prisma_client: _ManagedResourcePrismaClient) -> _Manag
return prisma_client.db.litellm_managedobjecttable
def _writer_database(db: _ManagedResourceDatabase) -> _ManagedResourceDatabase:
"""Reads that feed a write must see the create that just happened; a lagging read replica may not"""
return db.writer if isinstance(db, _WriterRoutedDatabase) else db
_JSON_VALUE_ADAPTER: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
_POSTGRES_UNSTORABLE_CHARACTER: Final = "\x00"
@ -512,6 +524,25 @@ def _without_model_output(
)
def _with_model_id(
file_object: LiteLLMBatch | LiteLLMFineTuningJob | ResponsesAPIResponse,
model_id: str | None,
) -> LiteLLMBatch | LiteLLMFineTuningJob | ResponsesAPIResponse:
return file_object if model_id is None else file_object.model_copy(update={"model_id": model_id})
async def _stored_model_id(managed_object_table: _ManagedObjectTableActions, unified_object_id: str) -> str | None:
row: Final = await managed_object_table.find_first(where={"unified_object_id": unified_object_id})
if row is None:
return None
try:
parsed: Final = _JSON_VALUE_ADAPTER.validate_python(_decode_json_blob(row.file_object))
except (json.JSONDecodeError, ValidationError):
return None
model_id: Final = parsed.get("model_id") if isinstance(parsed, dict) else None
return model_id if isinstance(model_id, str) and model_id else None
def _storage_metadata_of(file_object: OpenAIFileObject | None) -> Mapping[str, str]:
hidden_params: Final = cast( # cast-ok: _hidden_params is an untyped attribute the upload path sets
"Mapping[str, object]", getattr(file_object, "_hidden_params", None) or {}
@ -697,9 +728,17 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
persist_attribution: bool = False,
create_if_missing: bool = True,
batch_processed: bool = False,
creating_model_id: str | None = None,
) -> None:
"""Persist a managed object row, caching it and upserting it in the DB.
creating_model_id is the router deployment that created the object. It is kept
inside the stored file_object JSON as `model_id`, because the raw provider id an
unmanaged batch stores as unified_object_id says nothing about which deployment
made it, and the cost poller needs that deployment to read the result back. A
later observe of the row (create_if_missing=False) keeps the stored value rather
than overwriting it with whichever deployment served the retrieve.
persist_attribution is set only by the batch and background response creates,
the callers that can speak for the creator; it gates the api_key, request_tags, and,
for a response, end_user_id columns that the cost poll jobs bill against, so a later poll or
@ -720,7 +759,14 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
It is written only in the upsert create branch.
"""
verbose_logger.info(f"Storing LiteLLM Managed {file_purpose} object with id={unified_object_id} in cache")
stored_object: Final = _without_model_output(file_object)
managed_database: Final = _managed_database(self.prisma_client)
managed_object_table: Final = managed_database.litellm_managedobjecttable
preserved_model_id: Final = (
creating_model_id
if creating_model_id is not None or create_if_missing
else await _stored_model_id(_writer_database(managed_database).litellm_managedobjecttable, unified_object_id)
)
stored_object: Final = _with_model_id(_without_model_output(file_object), preserved_model_id)
litellm_managed_object = LiteLLM_ManagedObjectTable(
unified_object_id=unified_object_id,
model_object_id=model_object_id,
@ -752,7 +798,6 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"status": file_object.status,
"updated_by": user_api_key_dict.user_id,
}
managed_object_table: Final = _managed_object_table(self.prisma_client)
if not create_if_missing:
await managed_object_table.update_many(
where={"unified_object_id": unified_object_id},
@ -2170,6 +2215,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
request_tags=request_tags_from_metadata(request_metadata if isinstance(request_metadata, dict) else {}),
persist_attribution=is_batch_create,
create_if_missing=is_batch_create,
creating_model_id=model_id if is_batch_create else None,
)
if not is_batch_create:
await self.store_batch_output_file_ownership(

19
litellm-rust/crates/python-bridge/src/db.rs Normal file
View file

@ -0,0 +1,19 @@
use litellm_host_python::ready_future;
use pyo3::{prelude::*, types::PyNone};
#[pyclass(frozen)]
pub(crate) struct NativeDatabase {}
// TODO: take typed writer/reader settings and a token provider, then open a sqlx PgPool, once the first
// repository moves to native. The Prisma DATABASE_URL carries Prisma-only params and rotating IAM tokens
#[pymethods]
impl NativeDatabase {
#[staticmethod]
fn connect(py: Python<'_>) -> PyResult<Bound<'_, PyAny>> {
ready_future(py, Bound::new(py, NativeDatabase {})?.as_any())
}
fn close<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
ready_future(py, PyNone::get(py).as_any())
}
}

4
litellm-rust/crates/python-bridge/src/lib.rs
View file

@ -2,6 +2,7 @@ mod cache;
mod callable;
mod coercion;
mod credentials;
mod db;
mod diagnostics;
mod errors;
mod execution;
@ -21,6 +22,8 @@ mod _native {
use litellm_host_python::{ForkedAfterNativeRuntimeStarted, ProcessReservedForForking};
use pyo3::{prelude::*, types::PyModule};
#[pymodule_export]
use crate::db::NativeDatabase;
#[cfg(feature = "panic-test")]
#[pymodule_export]
use crate::diagnostics::_panic_for_test;
@ -112,6 +115,7 @@ mod tests {
"NativeDiagnosticProcessor",
"NativeClickHouseSpendConfig",
"NativeClickHouseSpendStorage",
"NativeDatabase",
"trace_encode_error",
"bundled_model_cost_map",
"TokenCounter",

4
litellm/_lazy_imports_registry.py
View file

@ -1469,11 +1469,11 @@ _UTILS_MODULE_IMPORT_MAP: Final = {
"AsyncHTTPHandler": ("litellm.llms.custom_httpx.http_handler", "AsyncHTTPHandler"),
"HTTPHandler": ("litellm.llms.custom_httpx.http_handler", "HTTPHandler"),
"get_num_retries_from_retry_policy": (
"litellm.router_utils.get_retry_from_policy",
"litellm.litellm_core_utils.get_retry_from_policy",
"get_num_retries_from_retry_policy",
),
"reset_retry_policy": (
"litellm.router_utils.get_retry_from_policy",
"litellm.litellm_core_utils.get_retry_from_policy",
"reset_retry_policy",
),
"get_coroutine_checker": (

4
litellm/a2a_protocol/litellm_completion_bridge/handler.py
View file

@ -22,7 +22,7 @@ from litellm.a2a_protocol.litellm_completion_bridge.transformation import (
A2AStreamingContext,
)
from litellm.a2a_protocol.providers.config_manager import A2AProviderConfigManager
from litellm.constants import A2A_CHAT_COMPLETION_BRIDGE_PROVIDERS
from litellm.constants import A2A_CHAT_COMPLETION_BRIDGE_PROVIDERS, A2A_COMPLETION_BRIDGE_WORKING_MESSAGE
from litellm.interactions.agents.utils import merge_agent_headers
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.types.utils import ModelResponse
@ -268,7 +268,7 @@ class A2ACompletionBridgeHandler:
ctx=ctx,
state="working",
final=False,
message_text="Processing request...",
message_text=A2A_COMPLETION_BRIDGE_WORKING_MESSAGE,
)
yield working_event

112
litellm/a2a_protocol/main.py
View file

@ -191,6 +191,23 @@ def _set_litellm_params_on_logging_obj(
}
async def _record_send_message_billing(
kwargs: Mapping[str, object],
request: "SendMessageRequest",
response_dict: Mapping[str, object],
litellm_params: Mapping[str, object],
agent_id: str | None,
) -> None:
prompt_tokens, completion_tokens, _ = await asyncify(A2ARequestUtils.calculate_usage_from_request_response)(
request=request,
response_dict=response_dict,
)
_set_usage_on_logging_obj(kwargs=kwargs, prompt_tokens=prompt_tokens, completion_tokens=completion_tokens)
if "error" not in response_dict:
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
def _get_a2a_model_info(a2a_client: "A2AClientType", kwargs: Mapping[str, object]) -> str:
"""
Extract agent info and set model/custom_llm_provider for cost tracking.
@ -483,13 +500,21 @@ async def asend_message(
if custom_llm_provider:
if request is None:
raise ValueError("request is required for completion bridge")
return await _send_message_via_completion_bridge(
bridge_response: Final = await _send_message_via_completion_bridge(
request=request,
custom_llm_provider=custom_llm_provider,
api_base=api_base,
litellm_params=litellm_params,
agent_extra_headers=agent_extra_headers,
)
await _record_send_message_billing(
kwargs=kwargs,
request=request,
response_dict=bridge_response.model_dump(mode="json", exclude_none=True),
litellm_params=litellm_params,
agent_id=agent_id,
)
return bridge_response
# Standard A2A client flow
if request is None:
@ -538,30 +563,14 @@ async def asend_message(
# Wrap in LiteLLM response type for _hidden_params support
response: Final = LiteLLMSendMessageResponse.from_a2a_response(a2a_response, request_id=str(request.id))
# Calculate token usage from request and response
response_dict: Final[dict[str, object]] = a2a_response.root.model_dump(mode="json", exclude_none=True)
(
prompt_tokens,
completion_tokens,
_,
) = await asyncify(A2ARequestUtils.calculate_usage_from_request_response)(
request=request,
response_dict=response_dict,
)
# Set usage on logging obj for standard logging payload
_set_usage_on_logging_obj(
await _record_send_message_billing(
kwargs=kwargs,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
request=request,
response_dict=a2a_response.root.model_dump(mode="json", exclude_none=True),
litellm_params=litellm_params,
agent_id=agent_id,
)
# Merge agent pricing params into the logging obj so cost is calculated
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
# Set agent_id on logging obj for SpendLogs tracking
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
return response
@ -646,6 +655,7 @@ async def asend_message_streaming(
metadata: dict[str, object] | None = None,
proxy_server_request: dict[str, object] | None = None,
agent_extra_headers: dict[str, str] | None = None,
agent_name: str | None = None,
**kwargs: object,
) -> AsyncIterator["SendStreamingMessageResponse | Mapping[str, object]"]:
"""
@ -661,6 +671,7 @@ async def asend_message_streaming(
agent_id: Optional agent ID for tracking in SpendLogs
metadata: Optional metadata dict (contains user_api_key, user_id, team_id, etc.)
proxy_server_request: Optional proxy server request data
agent_name: Agent name logged as a2a_agent/<agent_name> for completion bridge agents
Yields:
SendStreamingMessageResponse chunks from the agent
@ -685,6 +696,7 @@ async def asend_message_streaming(
print(chunk)
```
"""
called_at: Final = datetime.datetime.now()
litellm_params = litellm_params or {}
custom_llm_provider: Final = litellm_params.get("custom_llm_provider")
@ -705,20 +717,47 @@ async def asend_message_streaming(
request.params.model_dump(mode="json") if hasattr(request.params, "model_dump") else dict(request.params)
)
async for chunk in A2ACompletionBridgeHandler.handle_streaming(
bridge_stream: Final = A2ACompletionBridgeHandler.handle_streaming(
request_id=str(request.id),
params=params,
litellm_params=litellm_params,
api_base=api_base,
agent_extra_headers=agent_extra_headers,
):
yield chunk
)
caller_logging_obj: Final = kwargs.get("litellm_logging_obj")
bridge_logging_obj: Final = (
caller_logging_obj
if isinstance(caller_logging_obj, Logging)
else _build_streaming_logging_obj(
request=request,
agent_name=agent_name or "unknown",
agent_id=agent_id,
litellm_params=litellm_params,
metadata=metadata,
proxy_server_request=proxy_server_request,
start_time=called_at,
)
)
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
bridge_iterator: Final = A2AStreamingIterator(
stream=bridge_stream,
request=request,
logging_obj=bridge_logging_obj,
agent_name=agent_name or "unknown",
)
try:
async for chunk in bridge_iterator:
yield chunk
except (GeneratorExit, asyncio.CancelledError):
await bridge_iterator.abill_on_disconnect()
raise
return
if request is None:
raise ValueError("request is required")
called_at: Final = datetime.datetime.now()
_raw_logging_obj: Final = kwargs.get("litellm_logging_obj")
logging_obj: Logging | None = _raw_logging_obj if isinstance(_raw_logging_obj, Logging) else None
@ -742,12 +781,12 @@ async def asend_message_streaming(
assert a2a_client is not None
agent_name: Final = _get_a2a_model_info(a2a_client, kwargs)
card_agent_name: Final = _get_a2a_model_info(a2a_client, kwargs)
if logging_obj is None:
logging_obj = _build_streaming_logging_obj(
request=request,
agent_name=agent_name,
agent_name=card_agent_name,
agent_id=agent_id,
litellm_params=litellm_params,
metadata=metadata,
@ -755,7 +794,7 @@ async def asend_message_streaming(
start_time=called_at,
)
verbose_logger.info("A2A send_message_streaming request_id=%s, agent=%s", request.id, agent_name)
verbose_logger.info("A2A send_message_streaming request_id=%s, agent=%s", request.id, card_agent_name)
agent_card: Final = _get_a2a_client_agent_card(a2a_client)
card_url: Final = get_agent_card_url(agent_card) if agent_card else None
@ -766,18 +805,23 @@ async def asend_message_streaming(
agent_card=agent_card,
card_url=card_url,
api_base=api_base,
agent_name=agent_name,
agent_name=card_agent_name,
)
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
async for chunk in A2AStreamingIterator(
url_agent_iterator: Final = A2AStreamingIterator(
stream=stream,
request=request,
logging_obj=logging_obj,
agent_name=agent_name,
):
yield chunk
agent_name=card_agent_name,
)
try:
async for chunk in url_agent_iterator:
yield chunk
except (GeneratorExit, asyncio.CancelledError):
await url_agent_iterator.abill_on_disconnect()
raise
async def create_a2a_client(

2
litellm/a2a_protocol/providers/pydantic_ai_agents/transformation.py
View file

@ -515,6 +515,8 @@ class PydanticAITransformation:
"contextId": context_id,
"kind": "artifact-update",
"taskId": task_id,
"append": i > 0,
"lastChunk": is_last_chunk,
"artifact": {
"artifactId": artifact_id,
"parts": [

2
litellm/a2a_protocol/providers/watsonx_orchestrate/transformation.py
View file

@ -206,6 +206,8 @@ class WatsonxOrchestrateTransformation:
"contextId": context_id,
"kind": "artifact-update",
"taskId": task_id,
"append": i > 0,
"lastChunk": is_last,
"artifact": {
"artifactId": artifact_id,
"parts": [{"kind": "text", "text": chunk_text}],

89
litellm/a2a_protocol/streaming_iterator.py
View file

@ -3,16 +3,18 @@ A2A Streaming Iterator with token tracking and logging support.
"""
import asyncio
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Mapping
from datetime import datetime
from typing import TYPE_CHECKING, Final
import anyio
from pydantic import TypeAdapter
import litellm
from litellm._logging import verbose_logger
from litellm.a2a_protocol.cost_calculator import A2ACostCalculator
from litellm.a2a_protocol.utils import A2ARequestUtils
from litellm.constants import A2A_COMPLETION_BRIDGE_WORKING_MESSAGE
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -23,6 +25,23 @@ if TYPE_CHECKING:
_OPAQUE_PAYLOAD: Final = TypeAdapter(object)
def _chunk_payload(chunk: "SendStreamingMessageResponse | Mapping[str, object]") -> Mapping[str, object]:
if isinstance(chunk, Mapping):
return chunk
return chunk.model_dump(mode="json", exclude_none=True) if hasattr(chunk, "model_dump") else {}
def _artifact_update(payload: Mapping[str, object]) -> tuple[str, bool] | None:
result: Final = as_str_mapping(payload.get("result"))
if result is None or result.get("kind") != "artifact-update":
return None
artifact: Final = as_str_mapping(result.get("artifact"))
artifact_id: Final = artifact.get("artifactId") if artifact is not None else None
if not isinstance(artifact_id, str):
return None
return artifact_id, result.get("append") is True
class A2AStreamingIterator:
"""
Async iterator for A2A streaming responses with token tracking.
@ -32,7 +51,7 @@ class A2AStreamingIterator:
def __init__(
self,
stream: AsyncIterator["SendStreamingMessageResponse"],
stream: AsyncIterator["SendStreamingMessageResponse | Mapping[str, object]"],
request: "SendStreamingMessageRequest",
logging_obj: LiteLLMLoggingObj,
agent_name: str = "unknown",
@ -44,14 +63,16 @@ class A2AStreamingIterator:
self.start_time = logging_obj.start_time
# Collect chunks for token counting
self.chunks: list[SendStreamingMessageResponse] = []
self.collected_text_parts: list[str] = []
self.final_chunk: SendStreamingMessageResponse | None = None
self.chunks: list[SendStreamingMessageResponse | Mapping[str, object]] = []
self.latest_message_text = ""
self.artifact_texts: Mapping[str, str] = {}
self.saw_error = False
self.final_chunk: SendStreamingMessageResponse | Mapping[str, object] | None = None
def __aiter__(self):
return self
async def __anext__(self) -> "SendStreamingMessageResponse":
async def __anext__(self) -> "SendStreamingMessageResponse | Mapping[str, object]":
try:
chunk: Final = await self.stream.__anext__()
if self.logging_obj.completion_start_time is None:
@ -59,38 +80,54 @@ class A2AStreamingIterator:
# Store chunk
self.chunks.append(chunk)
payload: Final = _chunk_payload(chunk)
self.saw_error = self.saw_error or payload.get("error") is not None
# Extract text from chunk for token counting
self._collect_text_from_chunk(chunk)
self._collect_text_from_chunk(payload)
# Check if this is the final chunk (completed status)
if self._is_completed_chunk(chunk):
if self._is_completed_chunk(payload):
self.final_chunk = chunk
return chunk
except StopAsyncIteration:
# Stream ended - handle logging
if self.final_chunk is None and self.chunks:
self.final_chunk = self.chunks[-1]
await self._handle_stream_complete()
raise
def _collect_text_from_chunk(self, chunk: "SendStreamingMessageResponse") -> None:
"""Extract text from a streaming chunk and add to collected parts."""
async def abill_on_disconnect(self) -> None:
if not self.chunks:
return
with anyio.CancelScope(shield=True):
await self._handle_stream_complete()
def _collect_text_from_chunk(self, payload: Mapping[str, object]) -> None:
"""Record the reply text a streaming chunk carries."""
try:
chunk_dict: Final = chunk.model_dump(mode="json", exclude_none=True) if hasattr(chunk, "model_dump") else {}
text: Final = A2ARequestUtils.extract_text_from_response(chunk_dict)
if text:
self.collected_text_parts.append(text)
text: Final = A2ARequestUtils.extract_text_from_stream_event(payload)
if not text:
return
artifact_update: Final = _artifact_update(payload)
if artifact_update is None and text == A2A_COMPLETION_BRIDGE_WORKING_MESSAGE:
return
if artifact_update is None:
self.latest_message_text = text
return
artifact_id, append = artifact_update
artifact_text: Final = self.artifact_texts.get(artifact_id, "") + text if append else text
self.artifact_texts = {**self.artifact_texts, artifact_id: artifact_text}
except Exception:
verbose_logger.debug("Failed to extract text from A2A streaming chunk")
def _is_completed_chunk(self, chunk: "SendStreamingMessageResponse") -> bool:
def _reply_text(self) -> str:
artifact_text: Final = " ".join(text for text in self.artifact_texts.values() if text)
return artifact_text or self.latest_message_text
def _is_completed_chunk(self, payload: Mapping[str, object]) -> bool:
"""Check if chunk indicates stream completion."""
try:
chunk_dict: Final = chunk.model_dump(mode="json", exclude_none=True) if hasattr(chunk, "model_dump") else {}
result: Final = chunk_dict.get("result", {})
result: Final = payload.get("result", {})
if isinstance(result, dict):
status: Final = result.get("status", {})
if isinstance(status, dict):
@ -101,6 +138,8 @@ class A2AStreamingIterator:
async def _handle_stream_complete(self) -> None:
"""Handle logging and token counting when stream completes."""
if self.final_chunk is None and self.chunks:
self.final_chunk = self.chunks[-1]
try:
end_time: Final = datetime.now()
@ -109,9 +148,7 @@ class A2AStreamingIterator:
input_text: Final = A2ARequestUtils.extract_text_from_message(input_message)
prompt_tokens: Final = await asyncify(A2ARequestUtils.count_tokens)(input_text)
# Use the last (most complete) text from chunks
output_text: Final = self.collected_text_parts[-1] if self.collected_text_parts else ""
completion_tokens: Final = await asyncify(A2ARequestUtils.count_tokens)(output_text)
completion_tokens: Final = await asyncify(A2ARequestUtils.count_tokens)(self._reply_text())
total_tokens: Final = prompt_tokens + completion_tokens
@ -128,7 +165,7 @@ class A2AStreamingIterator:
self.logging_obj.model_call_details["stream"] = False
# Calculate cost using A2ACostCalculator
response_cost: Final = A2ACostCalculator.calculate_a2a_cost(self.logging_obj)
response_cost: Final = 0.0 if self.saw_error else A2ACostCalculator.calculate_a2a_cost(self.logging_obj)
self.logging_obj.model_call_details["response_cost"] = response_cost
# Build result for logging
@ -167,9 +204,7 @@ class A2AStreamingIterator:
# Add final chunk result if available
if self.final_chunk:
try:
chunk_dict: Final = as_str_mapping(
_OPAQUE_PAYLOAD.validate_python(self.final_chunk.model_dump(mode="json", exclude_none=True))
)
chunk_dict: Final = as_str_mapping(_OPAQUE_PAYLOAD.validate_python(_chunk_payload(self.final_chunk)))
if chunk_dict is not None:
result["result"] = chunk_dict.get("result", {})
except Exception:

50
litellm/a2a_protocol/utils.py
View file

@ -6,13 +6,17 @@ import hashlib
from collections.abc import Mapping
from typing import TYPE_CHECKING, Final, Protocol
from pydantic import TypeAdapter
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.core_helpers import has_attributes
from litellm.litellm_core_utils.core_helpers import as_str_mapping, has_attributes
if TYPE_CHECKING:
from a2a.types import SendMessageRequest, SendStreamingMessageRequest
_OBJECT_ITEMS: Final = TypeAdapter(tuple[object, ...])
class _SupportsModelDump(Protocol):
def model_dump(self, *, mode: str) -> object: ...
@ -34,6 +38,9 @@ class A2ARequestUtils:
"""
if message is None:
return ""
# a2a-sdk Part RootModels keep kind/text on part.root, so read pydantic messages as dicts
if has_attributes(message, _SupportsModelDump, "model_dump"):
return A2ARequestUtils.extract_text_from_message(message.model_dump(mode="json"))
# Handle both dict and object access
if isinstance(message, dict):
@ -63,8 +70,8 @@ class A2ARequestUtils:
Returns:
Text from response message parts
"""
result: Final = response_dict.get("result", {})
if not isinstance(result, dict):
result: Final = as_str_mapping(response_dict.get("result"))
if result is None:
return ""
# Direct message format (A2A spec): detect by explicit kind tag only.
@ -72,10 +79,36 @@ class A2ARequestUtils:
# type that happens to include a "parts" field.
if result.get("kind") == "message":
return A2ARequestUtils.extract_text_from_message(result)
if result.get("kind") == "task":
return A2ARequestUtils.extract_text_from_task(result)
message: Final = result.get("message", {})
return A2ARequestUtils.extract_text_from_message(message)
@staticmethod
def extract_text_from_task(task: Mapping[str, object]) -> str:
artifacts: Final = task.get("artifacts")
artifact_items: Final = _OBJECT_ITEMS.validate_python(artifacts) if isinstance(artifacts, list) else ()
artifact_texts: Final = tuple(
A2ARequestUtils.extract_text_from_message(artifact) for artifact in artifact_items
)
if any(artifact_texts):
return " ".join(text for text in artifact_texts if text)
status: Final = as_str_mapping(task.get("status"))
return A2ARequestUtils.extract_text_from_message(status.get("message")) if status is not None else ""
@staticmethod
def extract_text_from_stream_event(event: Mapping[str, object]) -> str:
result: Final = as_str_mapping(event.get("result"))
if result is None:
return ""
if result.get("kind") == "artifact-update":
return A2ARequestUtils.extract_text_from_message(result.get("artifact"))
if result.get("kind") == "status-update":
status: Final = as_str_mapping(result.get("status"))
return A2ARequestUtils.extract_text_from_message(status.get("message")) if status is not None else ""
return A2ARequestUtils.extract_text_from_response(event)
@staticmethod
def get_input_message_from_request(
request: "SendMessageRequest | SendStreamingMessageRequest",
@ -128,17 +161,8 @@ class A2ARequestUtils:
Returns:
Tuple of (prompt_tokens, completion_tokens, total_tokens)
"""
# Count input tokens. Dump the message to a dict first so extraction hits
# the dict branch — request-side parts are a2a-sdk Part RootModels whose
# kind/text live on part.root, which the object branch cannot read. This
# mirrors how the response side already works (it operates on model_dump).
input_message: Final = A2ARequestUtils.get_input_message_from_request(request)
input_text: Final = A2ARequestUtils.extract_text_from_message(
input_message.model_dump(mode="json")
if has_attributes(input_message, _SupportsModelDump, "model_dump")
else input_message
)
prompt_tokens: Final = A2ARequestUtils.count_tokens(input_text)
prompt_tokens: Final = A2ARequestUtils.count_tokens(A2ARequestUtils.extract_text_from_message(input_message))
# Count output tokens
output_text: Final = A2ARequestUtils.extract_text_from_response(response_dict)

7
litellm/constants.py
View file

@ -177,6 +177,8 @@ MID_STREAM_FAILURE_LOG_WAIT_SECONDS: Final = 1.0
# Set to 0 to disable truncation.
MAX_BASE64_LENGTH_FOR_LOGGING: Final = get_env_int("MAX_BASE64_LENGTH_FOR_LOGGING", 64)
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS: Final = 256 * 1024
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_STRINGS: Final = 256
BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS: Final = 8 * 1024 * 1024
REDACTED_BY_LITELLM: Final = "redacted-by-litellm"
# in-memory stand-in handed to provider converters for redacted arguments; never stored
REDACTED_TOOL_CALL_ARGUMENTS_PLACEHOLDER: Final = "{}"
@ -634,6 +636,8 @@ STS_CREDENTIAL_EXPIRY_SAFETY_MARGIN_SECONDS: Final = 60
BEDROCK_MIN_THINKING_BUDGET_TOKENS: Final = get_env_int("BEDROCK_MIN_THINKING_BUDGET_TOKENS", 1024)
# Anthropic's Messages API rejects thinking.budget_tokens < 1024.
ANTHROPIC_MIN_THINKING_BUDGET_TOKENS: Final = 1024
ANTHROPIC_MAX_USER_ID_LENGTH: Final = 512
ANTHROPIC_COMPATIBLE_MAX_USER_ID_LENGTH: Final = 255
ANTHROPIC_BILLING_METADATA_PREFIX: Final = "x-anthropic-billing-header:"
REPLICATE_POLLING_DELAY_SECONDS: Final = get_env_float("REPLICATE_POLLING_DELAY_SECONDS", 0.5)
DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS: Final = get_env_int("DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS", 128000)
@ -902,6 +906,7 @@ PROVIDERS_THAT_AUTHENTICATE_ON_PROVIDER_INFO: Final = frozenset(
SUBSCRIPTION_BACKED_PROVIDERS: Final = frozenset({"chatgpt"})
A2A_CHAT_COMPLETION_BRIDGE_PROVIDERS: Final = frozenset({"databricks_agent"})
A2A_COMPLETION_BRIDGE_WORKING_MESSAGE: Final = "Processing request..."
LITELLM_EMBEDDING_PROVIDERS_SUPPORTING_INPUT_ARRAY_OF_TOKENS: Final = [
"openai",
@ -1162,6 +1167,7 @@ openai_compatible_providers: Final[list[str]] = [
"cortecs",
"scx-ai",
"prism",
"hostinger_router",
"sail",
"reka",
"streamlake",
@ -2046,6 +2052,7 @@ LITELLM_SETTINGS_SAFE_DB_OVERRIDES: Final = [
SPECIAL_LITELLM_AUTH_TOKEN: Final = ["ui-token"]
DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL = get_env_int("DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL", 60)
AUTH_OBJECTS_TARGET: Final = "auth_objects"
HEALTH_CHECKS_TARGET: Final = "health_checks"
DEFAULT_ACCESS_GROUP_CACHE_TTL: Final = get_env_int("DEFAULT_ACCESS_GROUP_CACHE_TTL", 600)
SPEND_LOG_KEY_METADATA_CACHE_TTL: Final = 600
SPEND_LOG_KEY_METADATA_MISS_CACHE_TTL: Final = 30

13
litellm/cost_calculator.py
View file

@ -623,7 +623,11 @@ def cost_per_token(
)
if lyria_generation_cost is not None:
return 0.0, lyria_generation_cost
speech_model_info = litellm.get_model_info(model=model_without_prefix, custom_llm_provider=custom_llm_provider)
speech_model_info: Final = cached_get_model_info_helper(
model=model_without_prefix,
custom_llm_provider=custom_llm_provider,
preserve_unset_costs=True,
)
cost_metric: Final = select_cost_metric_for_model(speech_model_info)
prompt_cost: float = 0.0
completion_cost: float = 0.0
@ -3084,7 +3088,7 @@ def batch_cost_calculator(
batch_rates.cache_creation,
batch_rates.cache_creation_rates_by_ttl,
)
elif input_cost_per_token:
elif input_cost_per_token is not None:
details: Final = parse_prompt_tokens_details(usage)
cache_read_tokens: Final = details["cache_hit_tokens"]
cache_creation_tokens: Final = details["cache_creation_tokens"]
@ -3100,7 +3104,10 @@ def batch_cost_calculator(
cache_read_cost_key: Final = get_service_tier_cost_key("cache_read_input_token_cost", None)
total_prompt_cost += calculate_cost_component(model_info, cache_read_cost_key, cache_read_tokens) / 2
cache_creation_cost: Final = model_info.get("cache_creation_input_token_cost") or input_cost_per_token
configured_cache_creation_cost: Final = model_info.get("cache_creation_input_token_cost")
cache_creation_cost: Final = (
input_cost_per_token if configured_cache_creation_cost is None else configured_cache_creation_cost
)
total_prompt_cost += cache_creation_tokens * cache_creation_cost / 2
text_rate: Final = _batch_or_half(batch_rates.output, output_cost_per_token)
image_rate: Final = _batch_or_half(

168
litellm/decisions/call.py
View file

@ -3,10 +3,10 @@ from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field, replace
from functools import partial
from types import MappingProxyType
from typing import Final, Literal, TypeAlias
from typing import Annotated, Final, Literal, TypeAlias
import httpx
from pydantic import BaseModel, ConfigDict, StrictBool, TypeAdapter, ValidationError
from pydantic import BaseModel, BeforeValidator, ConfigDict, TypeAdapter, ValidationError
from pydantic_core import to_jsonable_python
from typing_extensions import assert_never
@ -23,6 +23,7 @@ from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.decisions.transformation import (
BaseDecisionsConfig,
DecisionsResponseValidationError,
catalog_mode,
image_media_type,
image_param,
input_images,
@ -61,7 +62,7 @@ from litellm.types.decisions import (
UnsupportedDecisionsRequest,
)
from litellm.types.llms.openai import ChatCompletionUserMessage
from litellm.types.utils import LlmProviders, ProviderSpecificHeader
from litellm.types.utils import LlmProviders, ProviderSpecificHeader, is_decisions_model_mode
from litellm.utils import ProviderConfigManager, get_model_cost_key
SystemOneQuestions: TypeAlias = Mapping[str, DecisionQuestion | Mapping[str, object]]
@ -218,7 +219,7 @@ def supported_decisions_providers() -> tuple[str, ...]:
)
def _provider_config(model: str, custom_llm_provider: str) -> BaseDecisionsConfig:
def get_decisions_provider_config(model: str, custom_llm_provider: str) -> BaseDecisionsConfig:
provider: Final = next((member for member in LlmProviders if member.value == custom_llm_provider), None)
provider_config: Final = (
None if provider is None else ProviderConfigManager.get_provider_decisions_config(model, provider)
@ -257,19 +258,105 @@ def _request_safety_identifier(
return None
def _fields_without_unsupported_media(
fields: RequestFields,
def _bool_or_none(value: object) -> bool | None:
return value if isinstance(value, bool) else None
def _str_or_none(value: object) -> str | None:
return value if isinstance(value, str) else None
_DeploymentFlag: TypeAlias = Annotated[bool | None, BeforeValidator(_bool_or_none)]
class _DeploymentCapabilities(BaseModel):
model_config = ConfigDict(frozen=True, from_attributes=True)
mode: Annotated[str | None, BeforeValidator(_str_or_none)] = None
supports_box_questions: _DeploymentFlag = None
supports_audio_input: _DeploymentFlag = None
supports_video_input: _DeploymentFlag = None
def decisions_media_flag(self, flag: bool | None, model: str, custom_llm_provider: str) -> bool | None:
if flag is None:
return None
mode: Final = self.mode if self.mode is not None else catalog_mode(model, custom_llm_provider)
return flag if mode is None or is_decisions_model_mode(mode) else None
@dataclass(frozen=True, slots=True)
class _ResolvedSupport:
box_questions: bool
audio: bool
videos: bool
def _takes_audio(
provider_config: BaseDecisionsConfig, model: str, custom_llm_provider: str, deployment: _DeploymentCapabilities
) -> bool:
return provider_config.supports_audio_input(
model,
custom_llm_provider,
deployment.decisions_media_flag(deployment.supports_audio_input, model, custom_llm_provider),
)
def _takes_videos(
provider_config: BaseDecisionsConfig, model: str, custom_llm_provider: str, deployment: _DeploymentCapabilities
) -> bool:
return provider_config.supports_video_input(
model,
custom_llm_provider,
deployment.decisions_media_flag(deployment.supports_video_input, model, custom_llm_provider),
)
def deployment_refuses_media(
model: str,
custom_llm_provider: str,
model_info: Mapping[str, object],
call_kwargs: Mapping[str, object],
*,
audio: bool,
videos: bool,
) -> bool:
if _drops_params(call_kwargs):
return False
provider_config: Final = get_decisions_provider_config(model, custom_llm_provider)
canonical_model: Final = provider_config.canonical_model(model)
deployment: Final = _DeploymentCapabilities.model_validate(model_info)
return (audio and not _takes_audio(provider_config, canonical_model, custom_llm_provider, deployment)) or (
videos and not _takes_videos(provider_config, canonical_model, custom_llm_provider, deployment)
)
def _resolved_support(
provider_config: BaseDecisionsConfig,
model: str,
custom_llm_provider: str,
headers: Mapping[str, str],
kwargs: Mapping[str, object],
fields: RequestFields,
) -> _ResolvedSupport:
deployment: Final = _DeploymentCapabilities.model_validate(kwargs.get("model_info") or {})
return _ResolvedSupport(
box_questions=provider_config.supports_box_questions(
model, custom_llm_provider, headers, deployment.supports_box_questions
),
audio=bool(fields.audio) and _takes_audio(provider_config, model, custom_llm_provider, deployment),
videos=bool(fields.videos) and _takes_videos(provider_config, model, custom_llm_provider, deployment),
)
def _fields_without_unsupported_media(
fields: RequestFields, support: _ResolvedSupport, kwargs: Mapping[str, object]
) -> RequestFields:
if not _drops_params(kwargs):
return fields
return replace(
fields,
audio=fields.audio if provider_config.supports_audio_input(model, custom_llm_provider) else None,
videos=fields.videos if provider_config.supports_video_input(model, custom_llm_provider) else None,
audio=fields.audio if support.audio else None,
videos=fields.videos if support.videos else None,
)
@ -292,10 +379,8 @@ class _MalformedImages:
def _unsupported_params(
ir_request: DecisionsIRRequest,
provider_config: BaseDecisionsConfig,
model: str,
custom_llm_provider: str,
headers: Mapping[str, str],
supports_box_questions: bool,
support: _ResolvedSupport,
) -> tuple[str, ...]:
refuses_safety_identifier: Final = (
ir_request.safety_identifier is not None and not provider_config.supports_safety_identifier
@ -304,13 +389,9 @@ def _unsupported_params(
None if provider_config.supports_images_in(ir_request.input, headers) else image_param(ir_request.input)
)
refuses_assets: Final = bool(ir_request.assets) and not provider_config.supports_assets(headers)
refuses_box_questions: Final = _has_box_question(ir_request.questions) and not supports_box_questions
refuses_audio: Final = bool(ir_request.audio) and not provider_config.supports_audio_input(
model, custom_llm_provider
)
refuses_videos: Final = bool(ir_request.videos) and not provider_config.supports_video_input(
model, custom_llm_provider
)
refuses_box_questions: Final = _has_box_question(ir_request.questions) and not support.box_questions
refuses_audio: Final = bool(ir_request.audio) and not support.audio
refuses_videos: Final = bool(ir_request.videos) and not support.videos
return (
*(("safety_identifier",) if refuses_safety_identifier else ()),
*(() if refused_image_param is None else (refused_image_param,)),
@ -333,7 +414,7 @@ def _without_unsupported_params(
ir_request: DecisionsIRRequest,
provider_config: BaseDecisionsConfig,
headers: Mapping[str, str],
supports_box_questions: bool,
support: _ResolvedSupport,
) -> DecisionsIRRequest:
return replace(
ir_request,
@ -345,9 +426,7 @@ def _without_unsupported_params(
safety_identifier=ir_request.safety_identifier if provider_config.supports_safety_identifier else None,
assets=ir_request.assets if provider_config.supports_assets(headers) else MappingProxyType({}),
questions=(
ir_request.questions
if supports_box_questions
else tuple(filter(_is_not_box_question, ir_request.questions))
ir_request.questions if support.box_questions else tuple(filter(_is_not_box_question, ir_request.questions))
),
)
@ -368,39 +447,20 @@ def _without_malformed_images(
return well_formed
class _DeploymentBoxQuestions(BaseModel):
model_config = ConfigDict(frozen=True, from_attributes=True)
supports_box_questions: StrictBool | None = None
def _deployment_supports_box_questions(kwargs: Mapping[str, object]) -> bool | None:
try:
return _DeploymentBoxQuestions.model_validate(kwargs.get("model_info") or {}).supports_box_questions
except ValidationError:
return None
def _provider_ir_request(
ir_request: DecisionsIRRequest,
*,
provider_config: BaseDecisionsConfig,
model: str,
custom_llm_provider: str,
headers: Mapping[str, str],
support: _ResolvedSupport,
kwargs: Mapping[str, object],
) -> DecisionsIRRequest | _UnsupportedParams | _NothingLeftAfterDrop:
supports_box_questions: Final = provider_config.supports_box_questions(
model, custom_llm_provider, headers, _deployment_supports_box_questions(kwargs)
)
unsupported: Final = _unsupported_params(
ir_request, provider_config, model, custom_llm_provider, headers, supports_box_questions
)
unsupported: Final = _unsupported_params(ir_request, provider_config, headers, support)
if not unsupported:
return ir_request
if not _drops_params(kwargs):
return _UnsupportedParams(names=unsupported)
supported: Final = _without_unsupported_params(ir_request, provider_config, headers, supports_box_questions)
supported: Final = _without_unsupported_params(ir_request, provider_config, headers, support)
if _leaves_no_input(supported.input):
return _NothingLeftAfterDrop(names=unsupported, missing="input to decide on")
if not supported.questions:
@ -408,7 +468,7 @@ def _provider_ir_request(
return supported
def _outbound_headers(
def decisions_outbound_headers(
extra_headers: Mapping[str, str] | None, custom_llm_provider: str, kwargs: Mapping[str, object]
) -> Mapping[str, str]:
forwarded: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(
@ -466,7 +526,7 @@ def prepare_call(
api_base=call_api_base,
api_key=api_key,
)
provider_config: Final = _provider_config(upstream_model, provider)
provider_config: Final = get_decisions_provider_config(upstream_model, provider)
if not upstream_model:
raise litellm.BadRequestError(
message="A model name is required for the Decisions API",
@ -478,11 +538,12 @@ def prepare_call(
except ValueError as error:
raise litellm.BadRequestError(message=str(error), model=model, llm_provider=provider) from error
try:
headers: Final = _outbound_headers(extra_headers, provider, kwargs)
headers: Final = decisions_outbound_headers(extra_headers, provider, kwargs)
support: Final = _resolved_support(provider_config, canonical_model, provider, headers, kwargs, fields)
request_safety_identifier: Final = _request_safety_identifier(safety_identifier, provider_config, kwargs)
request: Final = _fields_without_unsupported_media(
fields, provider_config, canonical_model, provider, kwargs
).validate(safety_identifier=request_safety_identifier, drops_params=_drops_params(kwargs))
request: Final = _fields_without_unsupported_media(fields, support, kwargs).validate(
safety_identifier=request_safety_identifier, drops_params=_drops_params(kwargs)
)
except ValidationError as error:
raise litellm.BadRequestError(
message=f"Invalid Decisions request: {error}",
@ -523,12 +584,7 @@ def prepare_call(
llm_provider=provider,
)
ir_request: Final = _provider_ir_request(
caller_ir_request,
provider_config=provider_config,
model=canonical_model,
custom_llm_provider=provider,
headers=headers,
kwargs=kwargs,
caller_ir_request, provider_config=provider_config, headers=headers, support=support, kwargs=kwargs
)
if isinstance(ir_request, _UnsupportedParams):
raise litellm.UnsupportedParamsError(

210
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -14,7 +14,7 @@ import itertools
import os
import re
from collections.abc import Callable, Container, Iterable, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, cast
from typing import TYPE_CHECKING, Any, Final, TypeAlias, TypeVar, cast
from urllib.parse import urlparse
from pydantic import TypeAdapter, ValidationError
@ -102,6 +102,192 @@ def _validated_object_list(value: object) -> list[object] | None:
return None
_MessageT: Final = TypeVar("_MessageT")
_CacheControlRewrite: TypeAlias = Callable[[object], ChatCompletionCachedContent | None]
_MARK_CARRYING_MESSAGE_FIELDS: Final = ("content", "tool_calls")
def _mark_carrying_blocks(fields: Mapping[str, object]) -> Sequence[object]:
lists: Final = tuple(fields.get(key) for key in _MARK_CARRYING_MESSAGE_FIELDS)
return tuple(
itertools.chain.from_iterable(
_OBJECT_LIST_ADAPTER.validate_python(value) for value in lists if isinstance(value, list)
)
)
def _block_carries_cache_control(block: object) -> bool:
return isinstance(block, dict) and _CONTENT_PART_ADAPTER.validate_python(block).get("cache_control") is not None
_ROLES_SENDING_MESSAGE_MARK_WITH_LIST_CONTENT: Final = frozenset({"tool", "function"})
def _message_mark_reaches_provider(fields: Mapping[str, object]) -> bool:
if fields.get("cache_control") is None:
return False
return (
not isinstance(fields.get("content"), list)
or fields.get("role") in _ROLES_SENDING_MESSAGE_MARK_WITH_LIST_CONTENT
)
def cache_control_count(message: object) -> int:
if not isinstance(message, dict):
return 0
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
block_marks: Final = sum(1 for block in _mark_carrying_blocks(fields) if _block_carries_cache_control(block))
return int(_message_mark_reaches_provider(fields)) + block_marks
def _fields_with_cache_control(fields: Mapping[str, object], rewrite: _CacheControlRewrite) -> Mapping[str, object]:
others: Final = {key: value for key, value in fields.items() if key != "cache_control"}
if "cache_control" not in fields:
return others
cache_control: Final = rewrite(fields["cache_control"])
return others if cache_control is None else {**others, "cache_control": cache_control}
def block_with_cache_control(block: object, rewrite: _CacheControlRewrite) -> object:
if not isinstance(block, dict):
return block
return _fields_with_cache_control(_CONTENT_PART_ADAPTER.validate_python(block), rewrite)
def with_cache_control(message: _MessageT, rewrite: _CacheControlRewrite) -> _MessageT:
if not isinstance(message, dict):
return message
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
blocks_rewritten: Final = {
key: (
[block_with_cache_control(block, rewrite) for block in _OBJECT_LIST_ADAPTER.validate_python(value)]
if key in _MARK_CARRYING_MESSAGE_FIELDS and isinstance(value, list)
else value
)
for key, value in fields.items()
}
return cast( # cast-ok: the same message shape with each cache marker rewritten or dropped
_MessageT, _fields_with_cache_control(blocks_rewritten, rewrite)
)
def dropped_cache_control(_: object) -> None:
return None
def _first_marked_index(value: object) -> int | None:
if not isinstance(value, list):
return None
blocks: Final = _OBJECT_LIST_ADAPTER.validate_python(value)
return next((index for index, block in enumerate(blocks) if _block_carries_cache_control(block)), None)
_MarkSlot: TypeAlias = tuple[str, int]
_MESSAGE_MARK_SLOT: Final[_MarkSlot] = ("cache_control", -1)
def _block_mark_slots(fields: Mapping[str, object], key: str) -> tuple[_MarkSlot, ...]:
value: Final = fields.get(key)
if not isinstance(value, list):
return ()
blocks: Final = _OBJECT_LIST_ADAPTER.validate_python(value)
return tuple((key, index) for index, block in enumerate(blocks) if _block_carries_cache_control(block))
def _mark_slots_in_wire_order(fields: Mapping[str, object]) -> tuple[_MarkSlot, ...]:
content_slots: Final = _block_mark_slots(fields, "content")
tool_call_slots: Final = _block_mark_slots(fields, "tool_calls")
if not _message_mark_reaches_provider(fields):
return content_slots + tool_call_slots
if isinstance(fields.get("content"), list):
return content_slots + (_MESSAGE_MARK_SLOT,) + tool_call_slots
return (_MESSAGE_MARK_SLOT,) + content_slots + tool_call_slots
def _blocks_without_marks(key: str, value: object, dropped_slots: frozenset[_MarkSlot]) -> object:
if key not in _MARK_CARRYING_MESSAGE_FIELDS or not isinstance(value, list):
return value
return [
block_with_cache_control(block, dropped_cache_control) if (key, index) in dropped_slots else block
for index, block in enumerate(_OBJECT_LIST_ADAPTER.validate_python(value))
]
def _without_oldest_marks(message: _MessageT, dropped: int) -> _MessageT:
if dropped == 0 or not isinstance(message, dict):
return message
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
dropped_slots: Final = frozenset(_mark_slots_in_wire_order(fields)[:dropped])
blocks_unmarked: Final = {key: _blocks_without_marks(key, value, dropped_slots) for key, value in fields.items()}
return cast( # cast-ok: the same message with its oldest wire-order marks dropped
_MessageT,
_fields_with_cache_control(blocks_unmarked, dropped_cache_control)
if _MESSAGE_MARK_SLOT in dropped_slots
else blocks_unmarked,
)
def tool_message_with_single_cache_mark(message: _MessageT) -> _MessageT:
if not isinstance(message, dict):
return message
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
content: Final = fields.get("content")
if fields.get("role") != "tool" or not isinstance(content, list):
return message
kept_index: Final = None if fields.get("cache_control") is not None else _first_marked_index(content)
return cast( # cast-ok: the same tool message keeping only the mark its single cache point reads
_MessageT,
{
**fields,
"content": [
block if index == kept_index else block_with_cache_control(block, dropped_cache_control)
for index, block in enumerate(_OBJECT_LIST_ADAPTER.validate_python(content))
],
},
)
_BudgetSpend: TypeAlias = Callable[[tuple[int, int], int], tuple[int, int]]
def _newest_marks_kept(state: tuple[int, int], marks: int) -> tuple[int, int]:
remaining, _ = state
kept: Final = min(marks, remaining)
return remaining - kept, marks - kept
def _whole_message_kept(state: tuple[int, int], marks: int) -> tuple[int, int]:
remaining, _ = state
if marks > remaining:
return remaining, marks
return remaining - marks, 0
def _messages_within_budget(
messages: Sequence[_MessageT], reserved_marks: int, spend: _BudgetSpend
) -> Sequence[_MessageT]:
budget: Final = max(MAX_CACHE_CONTROL_BLOCKS - reserved_marks, 0)
newest_first: Final = tuple(reversed(messages))
budget_after_each: Final = itertools.accumulate(
(cache_control_count(message) for message in newest_first), spend, initial=(budget, 0)
)
dropped_counts: Final = tuple(dropped for _, dropped in itertools.islice(budget_after_each, 1, None))
kept_newest_first: Final = tuple(
_without_oldest_marks(message, dropped) for message, dropped in zip(newest_first, dropped_counts)
)
return tuple(reversed(kept_newest_first))
def newest_marks_within_cache_control_limit(messages: Sequence[_MessageT], reserved_marks: int) -> Sequence[_MessageT]:
return _messages_within_budget(messages, reserved_marks, _newest_marks_kept)
def whole_messages_within_cache_control_limit(
messages: Sequence[_MessageT], reserved_marks: int
) -> Sequence[_MessageT]:
return _messages_within_budget(messages, reserved_marks, _whole_message_kept)
def _part_with_cache_control_renamed(part: object) -> object:
if not _accepts_prompt_cache_breakpoint(part):
return part
@ -1439,3 +1625,25 @@ class AnthropicCacheControlHook(CustomPromptManagement):
llm_router=None,
)
return None
def messages_within_cache_control_cap(
messages: Sequence[_MessageT], tools: object, cache_control: object = None, request_kwargs: object = None
) -> Sequence[_MessageT]:
"""Under drop_params, drop the oldest message hints the provider's cache_control cap has no room for.
Tool hints and a top-level ``cache_control`` stay on the wire, so they are reserved first.
"""
reserved_marks: Final = AnthropicCacheControlHook.count_external_cache_breakpoints(
_validated_object_list(tools), cache_control, request_kwargs
)
kept: Final = newest_marks_within_cache_control_limit(messages, reserved_marks)
dropped: Final = sum(map(cache_control_count, messages)) - sum(map(cache_control_count, kept))
if dropped > 0:
verbose_logger.warning(
"Dropped %d cache_control hint(s) from the oldest messages because drop_params is on and the "
"provider accepts at most %d cache_control blocks per request",
dropped,
MAX_CACHE_CONTROL_BLOCKS,
)
return kept

158
litellm/integrations/custom_guardrail.py
View file

@ -3,8 +3,9 @@ import copy
import hashlib
import os
import secrets
from collections.abc import Mapping, Sequence
from collections.abc import Iterable, Mapping, Sequence
from datetime import datetime
from itertools import chain
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, ClassVar, Final, Literal, Optional, cast, get_args
@ -70,6 +71,8 @@ GUARDRAIL_SESSIONS_TARGET: Final = "guardrail_sessions"
# field to suppress a guardrail on the direct-SDK path that never reaches the
# proxy's metadata sanitizer.
_PRE_CALL_EXECUTED_TOKEN: Final = secrets.token_hex(16)
_PRE_CALL_RAISED_TOKEN: Final = secrets.token_hex(16)
_DEPLOYMENT_HOOK_RAN_TOKEN: Final = secrets.token_hex(16)
_GUARDRAIL_BLOCK_STATUS_CODES: Final = frozenset({400, 403, 422})
@ -208,6 +211,46 @@ def guardrail_request_data_with_streaming(
}
_GUARDRAIL_ENTRY_WITH_SETTINGS: Final = TypeAdapter(Mapping[str, object])
def merged_guardrail_entries(*entry_lists: Iterable[object]) -> list[object]:
"""Merges guardrail lists so each guardrail is named once: the first ``{name: settings}`` entry naming it wins over
any plain name in either order, else its first plain name stays"""
entries: Final = tuple(chain.from_iterable(entry_lists))
settings_names: Final = tuple(_settings_entry_names(entry) for entry in entries)
named_with_settings: Final = frozenset[str]().union(*settings_names)
return [
entry
for index, entry in enumerate(entries)
if not _superseded_guardrail_entry(entries, settings_names, named_with_settings, index)
]
def guardrail_entry_names(entry: object) -> tuple[str, ...]:
if isinstance(entry, str):
return (entry,)
if isinstance(entry, dict):
return tuple(_GUARDRAIL_ENTRY_WITH_SETTINGS.validate_python(entry))
return ()
def _settings_entry_names(entry: object) -> frozenset[str]:
return frozenset(guardrail_entry_names(entry)) if isinstance(entry, dict) else frozenset()
def _superseded_guardrail_entry(
entries: tuple[object, ...],
settings_names: tuple[frozenset[str], ...],
named_with_settings: frozenset[str],
index: int,
) -> bool:
entry: Final = entries[index]
if isinstance(entry, str):
return entry in named_with_settings or entry in entries[:index]
return bool(settings_names[index]) and settings_names[index] <= frozenset[str]().union(*settings_names[:index])
def _request_is_streaming(data: object, event_type: GuardrailEventHooks | None = None) -> bool:
if event_type in _REALTIME_STREAMING_HOOKS:
return True
@ -219,6 +262,25 @@ def _request_is_streaming(data: object, event_type: GuardrailEventHooks | None =
)
def _str_collection(value: object) -> frozenset[str]:
if not isinstance(value, (list, tuple, set, frozenset)):
return frozenset()
items: Final = cast("Iterable[object]", value) # cast-ok: a metadata collection of guardrail names
return frozenset(item for item in items if isinstance(item, str))
def _request_carries_marker(data: Mapping[str, object], marker: str | None) -> bool:
if marker is None:
return False
for meta_key in ("metadata", "litellm_metadata"):
meta = data.get(meta_key)
if isinstance(meta, dict):
executed = meta.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if isinstance(executed, list) and marker in executed:
return True
return False
_REQUEST_FIELDS: Final = TypeAdapter(Mapping[str, object])
@ -828,13 +890,16 @@ class CustomGuardrail(CustomLogger):
return False
def _pre_call_marker(self) -> str | None:
def _marker(self, token: str) -> str | None:
name: Final = self.guardrail_name
if not name:
return None
return f"{_PRE_CALL_EXECUTED_TOKEN}:{name}"
return f"{token}:{name}"
def mark_pre_call_hook_ran(self, data: dict[str, object]) -> None:
def _pre_call_marker(self, *, raised: bool = False) -> str | None:
return self._marker(_PRE_CALL_RAISED_TOKEN if raised else _PRE_CALL_EXECUTED_TOKEN)
def mark_pre_call_hook_ran(self, data: dict[str, object], *, raised: bool = False) -> None:
"""
Record that this guardrail's ``async_pre_call_hook`` already ran for this
request, so the deployment-level hook does not run it a second time.
@ -843,8 +908,11 @@ class CustomGuardrail(CustomLogger):
router later spreads a deployment's model-level ``guardrails`` into the
top-level request kwargs, which would otherwise re-trigger the same hook
from ``async_pre_call_deployment_hook``.
With ``raised``, records instead that the hook raised as a policy pipeline
step the pipeline then moved past. The deployment-level hook still runs it.
"""
marker: Final = self._pre_call_marker()
marker: Final = self._pre_call_marker(raised=raised)
if marker is None:
return
for meta_key in ("metadata", "litellm_metadata"):
@ -859,17 +927,57 @@ class CustomGuardrail(CustomLogger):
return
data["metadata"] = {PRE_CALL_EXECUTED_GUARDRAILS_KEY: [marker]}
def _pre_call_hook_already_ran(self, data: dict[str, object]) -> bool:
marker: Final = self._pre_call_marker()
if marker is None:
return False
for meta_key in ("metadata", "litellm_metadata"):
meta = data.get(meta_key)
if isinstance(meta, dict):
executed = meta.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if isinstance(executed, list) and marker in executed:
return True
return False
def pre_call_hook_ran_or_raised(self, data: Mapping[str, object]) -> bool:
"""Whether this guardrail's ``async_pre_call_hook`` ran for this request, whatever it decided."""
return self._pre_call_hook_already_ran(data) or self._pre_call_hook_already_ran(data, raised=True)
def _pre_call_hook_already_ran(self, data: Mapping[str, object], *, raised: bool = False) -> bool:
return _request_carries_marker(data, self._pre_call_marker(raised=raised))
def deployment_hook_ran(self, data: Mapping[str, object]) -> bool:
"""Whether a deployment-level hook ran this guardrail as a policy pipeline step of this request."""
return _request_carries_marker(data, self._marker(_DEPLOYMENT_HOOK_RAN_TOKEN))
def _record_deployment_hook_run(
self,
data: dict[str, object], # mutable-ok: the request payload the run is recorded on
) -> None:
"""
Record that a deployment-level hook ran this guardrail when it is a policy pipeline step of
this request, and name it in the applied guardrails when the request's metadata lists it.
"""
name: Final = self.guardrail_name
marker: Final = self._marker(_DEPLOYMENT_HOOK_RAN_TOKEN)
buckets: Final = tuple(
cast("dict[str, object]", bucket) # cast-ok: metadata buckets are str-keyed
for bucket in (data.get("metadata"), data.get("litellm_metadata"))
if isinstance(bucket, dict)
)
policy_state: Final = next(
(bucket for bucket in buckets if name in _str_collection(bucket.get("_pipeline_managed_guardrails"))),
None,
)
if marker is None or policy_state is None:
return
recorded: Final = policy_state.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if not isinstance(recorded, list):
policy_state[PRE_CALL_EXECUTED_GUARDRAILS_KEY] = [marker]
elif marker not in recorded:
cast("list[object]", recorded).append(marker) # cast-ok: markers are only compared as strings
if not any(name in _str_collection(bucket.get("guardrails")) for bucket in buckets):
return
named_lists: Final = tuple(
cast("list[object]", bucket["applied_guardrails"]) # cast-ok: entries are only compared with names
for bucket in buckets
if isinstance(bucket.get("applied_guardrails"), list)
)
for applied in named_lists:
if name not in applied:
applied.append(name)
if named_lists:
return
metadata_bucket: Final = cast("dict[str, object]", get_or_create_metadata_bucket(data)[1]) # cast-ok: str keys
metadata_bucket["applied_guardrails"] = [name]
@classmethod
def uses_apply_guardrail_interface(cls) -> bool:
@ -918,12 +1026,15 @@ class CustomGuardrail(CustomLogger):
target: Final = self._deployment_hook_target()
if target is not self:
kwargs["guardrail_to_apply"] = self
result: Final = await target.async_pre_call_hook(
user_api_key_dict=_user_api_key_auth_from_request(kwargs),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
try:
result: Final = await target.async_pre_call_hook(
user_api_key_dict=_user_api_key_auth_from_request(kwargs),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
finally:
self._record_deployment_hook_run(kwargs)
if result is not None and isinstance(result, dict):
result_messages: Final = result.get("messages")
@ -1007,6 +1118,9 @@ class CustomGuardrail(CustomLogger):
finally:
if target is not self:
request_data.pop("guardrail_to_apply", None)
self._record_deployment_hook_run(
cast("dict[str, object]", request_data) # cast-ok: proxy request payloads are str-keyed
)
if not self._is_valid_response_type(result):
return None

5
litellm/integrations/otel/routing.py
View file

@ -21,6 +21,11 @@ _ROUTING_SCALAR_FIELDS: Final = frozenset(
"classifier_cost",
"classifier_failure_reason",
"classifier_error_type",
"classifier_input_budget_tokens",
"classifier_input_tokens",
"classifier_input_count_method",
"classifier_input_truncated",
"classifier_input_budget_status",
"classifier_confidence",
"classifier_primary_rule",
"classifier_capability_boundary",

8
litellm/integrations/prometheus.py
View file

@ -66,7 +66,7 @@ from litellm.repositories.organization_repository import OrganizationRepository
from litellm.repositories.table_repositories import EndUserRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
from litellm.router_api import deployment_metadata
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import (
@ -3022,8 +3022,8 @@ class PrometheusLogger(CustomLogger):
).get("custom_llm_provider")
if isinstance(provider, str) and provider:
return provider
deployment_metadata: Final = find_deployment_metadata(litellm_params)
return _deployment_model_provider(deployment_metadata.get("deployment") if deployment_metadata else None)
picked_deployment: Final = deployment_metadata(litellm_params)
return _deployment_model_provider(picked_deployment.get("deployment") if picked_deployment else None)
@staticmethod
def _extract_api_provider_from_exception(exception: Exception) -> str | None:
@ -5209,7 +5209,7 @@ def _tag_matches_wildcard_configured_pattern(tags: Sequence[str], configured_tag
"""
import re
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
pattern_router: Final = PatternMatchRouter()
regex_pattern: Final = pattern_router.pattern_to_regex(configured_tag)

0
litellm/router_utils/access_windows.py → litellm/litellm_core_utils/access_windows.py
View file

0
litellm/router_utils/add_retry_fallback_headers.py → litellm/litellm_core_utils/add_retry_fallback_headers.py
View file

0
litellm/router_utils/clientside_credential_handler.py → litellm/litellm_core_utils/clientside_credential_handler.py
View file

0
litellm/router_utils/deployment_pricing.py → litellm/litellm_core_utils/deployment_pricing.py
View file

1
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -98,6 +98,7 @@ class ExceptionCheckers:
known_exception_substrings: Final = [
"exceed context limit",
"this model's maximum context length is",
"requested token count exceeds the model's maximum context length",
"string too long. expected a string with maximum length",
"model's maximum context limit",
"is longer than the model's context length",

6
litellm/litellm_core_utils/fallback_utils.py
View file

@ -3,13 +3,13 @@ from typing import Final
import litellm
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
)
from litellm.litellm_core_utils.core_helpers import (
filter_internal_params,
safe_deep_copy,
)
from litellm.router_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
)
from .asyncify import run_async_function

0
litellm/router_utils/get_retry_from_policy.py → litellm/litellm_core_utils/get_retry_from_policy.py
View file

2
litellm/litellm_core_utils/health_check_helpers.py
View file

@ -119,7 +119,7 @@ def _is_route_segment(custom_llm_provider: str, segment: str, model_after_segmen
def wildcard_health_check_models(wildcard: str, custom_llm_provider: str) -> tuple[str, ...]:
from litellm.litellm_core_utils.llm_request_utils import rank_chat_models_from_llm_provider
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
provider_prefix: Final = f"{custom_llm_provider}/"
segments: Final = wildcard.removeprefix(provider_prefix).split("/")

11
litellm/litellm_core_utils/litellm_logging.py
View file

@ -165,7 +165,7 @@ from litellm.types.llms.openai import (
)
from litellm.types.mcp import MCPPostCallResponseObject
from litellm.types.prompts.init_prompts import PromptSpec
from litellm.types.rerank import RerankResponse
from litellm.types.rerank import RerankResponse, get_rerank_usage
from litellm.types.router import RouterRateLimitError, SilentExperimentFlag
from litellm.types.utils import (
DEPLOYMENT_SCOPED_PRICING_FIELDS,
@ -355,6 +355,7 @@ def _get_provider_request_id(original_exception: Exception) -> str | None:
# Cache custom pricing keys as frozenset for O(1) lookups instead of looping through 49 keys
_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
_METADATA_MAPPING: Final = TypeAdapter(Mapping[str, object])
_RESPONSE_MAPPING: Final = TypeAdapter(Mapping[str, object])
_MODEL_INFO_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = _CUSTOM_PRICING_KEYS | DEPLOYMENT_SCOPED_PRICING_FIELDS
_UNSERIALIZABLE_METADATA_KEYS: Final[frozenset[str]] = frozenset(
("user_api_key_auth", "user_api_key_budget_reservation")
@ -6817,7 +6818,11 @@ class StandardLoggingPayloadSetup:
total_tokens=0,
)
usage: Final = response_obj.get("usage", None) or {}
usage: Final = response_obj.get("usage", None)
if usage is None:
rerank_usage: Final = get_rerank_usage(_RESPONSE_MAPPING.validate_python(response_obj))
if rerank_usage is not None:
return Usage(**rerank_usage)
if usage is None or (not isinstance(usage, dict) and not isinstance(usage, Usage)):
return Usage(
prompt_tokens=0,
@ -6853,7 +6858,7 @@ class StandardLoggingPayloadSetup:
return _empty
_raw: Final = response_obj.get("usage", None)
if _raw is None:
return _empty
return dict(get_rerank_usage(_RESPONSE_MAPPING.validate_python(response_obj)) or _empty)
if isinstance(_raw, ResponseAPIUsage):
return ResponseAPILoggingUtils.transform_response_api_usage_to_chat_usage(_raw).model_dump()
if isinstance(_raw, dict):

3
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
View file

@ -208,11 +208,14 @@ class StandardBuiltInToolCostTracking:
usage=usage, response_object=response_object
)
response_model: Final = _output_item_field(response_object, "model")
if model_info is not None and resolved_usage is not None and resolved_provider is not None:
result: Final = get_cost_for_web_search_request(
custom_llm_provider=resolved_provider,
usage=resolved_usage,
model_info=model_info,
response_model=response_model if isinstance(response_model, str) else None,
lookup_model_info=StandardBuiltInToolCostTracking._safe_get_model_info,
)
if result is not None:
return result

7
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -28,6 +28,7 @@ from litellm.types.utils import (
DataResidency,
ImageResponse,
ModelInfo,
ModelInfoBase,
PassthroughCallTypes,
PromptTokensDetailsWrapper,
ServiceTier,
@ -197,15 +198,15 @@ def get_billable_input_tokens(usage: Usage) -> int:
def select_cost_metric_for_model(
model_info: ModelInfo,
model_info: ModelInfoBase,
) -> Literal["cost_per_character", "cost_per_token"]:
"""
Select 'cost_per_character' if model_info has 'input_cost_per_character'
Select 'cost_per_token' if model_info has 'input_cost_per_token'
"""
if model_info.get("input_cost_per_character"):
if model_info.get("input_cost_per_character") is not None:
return "cost_per_character"
elif model_info.get("input_cost_per_token"):
elif model_info.get("input_cost_per_token") is not None:
return "cost_per_token"
else:
raise ValueError(

52
litellm/litellm_core_utils/logging_utils.py
View file

@ -3,13 +3,15 @@ import functools
import inspect
import re
import time
from collections.abc import Iterator, Mapping, Sequence
from collections.abc import Callable, Iterator, Mapping, Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, TypeAlias, cast
from litellm._logging import format_base64_size, verbose_logger
from litellm.constants import (
BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS,
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS,
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_STRINGS,
MAX_BASE64_LENGTH_FOR_LOGGING,
)
from litellm.litellm_core_utils.core_helpers import get_litellm_metadata_from_kwargs
@ -44,6 +46,8 @@ Helper utils used for logging callbacks
# Regex matching data-URI base64 content: "data:<mime>;base64,<payload>"
# Captures: group(1)=mime_type, group(2)=base64_payload
_DATA_URI_RE: Final = re.compile(r"data:([^;,\s]{1,255});base64,([A-Za-z0-9+/=]+)")
_DATA_URI_PREFIX: Final = "data:"
_DATA_URI_BASE64_MARKER: Final = ";base64,"
# Maximum nesting depth for _truncate_base64_in_value to guard against
# pathological payloads. OpenAI message format is typically 3-4 levels deep.
@ -60,15 +64,20 @@ def _base64_data_uri_replacer(match: re.Match[str]) -> str:
return f"data:{mime_type};base64,[base64_data truncated: {size_str}]"
def _may_contain_base64_data_uri(value: str) -> bool:
return ";" in value and _DATA_URI_PREFIX in value and _DATA_URI_BASE64_MARKER in value
def _truncate_base64_in_string(value: str) -> str:
"""Replace long base64 data-URI payloads in a string with a size placeholder."""
if MAX_BASE64_LENGTH_FOR_LOGGING <= 0:
if MAX_BASE64_LENGTH_FOR_LOGGING <= 0 or not _may_contain_base64_data_uri(value):
return value
return _DATA_URI_RE.sub(_base64_data_uri_replacer, value)
def _truncate_base64_in_value(
value: str | dict[str, object] | list[object] | None,
truncate_string: Callable[[str], str],
) -> str | dict[str, object] | list[object] | None:
"""Iteratively truncate base64 data URIs in a JSON-like value (str/list/dict).
@ -78,13 +87,13 @@ def _truncate_base64_in_value(
# Stack entries: (source_value, depth, parent_container, key_or_index)
# We mutate *copies* of dicts/lists in-place via parent references.
if isinstance(value, str):
return _truncate_base64_in_string(value)
return truncate_string(value)
if not isinstance(value, (dict, list)):
return value
# Shallow-copy the root so we don't mutate the caller's data.
root: Final = {k: v for k, v in value.items()} if isinstance(value, dict) else list(value)
stack: Final[list] = [(root, 0)]
stack: Final[list[tuple[dict[str, object] | list[object], int]]] = [(root, 0)] # mutable-ok: explicit stack
while stack:
container, depth = stack.pop()
@ -93,19 +102,19 @@ def _truncate_base64_in_value(
if isinstance(container, dict):
for k, v in container.items():
if isinstance(v, str):
container[k] = _truncate_base64_in_string(v)
container[k] = truncate_string(v)
elif isinstance(v, dict):
copy: dict | list = {ck: cv for ck, cv in v.items()}
copy: dict[str, object] | list[object] = {ck: cv for ck, cv in v.items()}
container[k] = copy
stack.append((copy, depth + 1))
elif isinstance(v, list):
copy = list(v)
container[k] = copy
stack.append((copy, depth + 1))
elif isinstance(container, list):
else:
for i, v in enumerate(container):
if isinstance(v, str):
container[i] = _truncate_base64_in_string(v)
container[i] = truncate_string(v)
elif isinstance(v, dict):
copy = {ck: cv for ck, cv in v.items()}
container[i] = copy
@ -127,8 +136,15 @@ def truncate_base64_in_messages(
"""
if messages is None or MAX_BASE64_LENGTH_FOR_LOGGING <= 0:
return messages
return _copy_messages(messages, _truncate_base64_in_string)
def _copy_messages(
messages: str | dict[str, object] | list[object], # mutable-ok: same contract as truncate_base64_in_messages
truncate_string: Callable[[str], str],
) -> str | dict[str, object] | list[object] | None: # mutable-ok: same contract as truncate_base64_in_messages
try:
return _truncate_base64_in_value(messages)
return _truncate_base64_in_value(messages, truncate_string)
except Exception as e:
verbose_logger.debug("Failed to truncate base64 in messages: %s", e)
return messages
@ -151,19 +167,31 @@ def _iter_string_leaves(value: _StringTree) -> Iterator[str]:
pass
def _is_few_plain_strings(leaves: tuple[str, ...]) -> bool:
return len(leaves) < BASE64_TRUNCATION_OFFLOAD_THRESHOLD_STRINGS and not any(
_may_contain_base64_data_uri(leaf) for leaf in leaves
)
async def truncate_base64_in_messages_async(
messages: str | list | dict | None, # mutable-ok: same contract as truncate_base64_in_messages
) -> str | list | dict | None: # mutable-ok: same contract as truncate_base64_in_messages
"""
Same result as truncate_base64_in_messages, but payloads whose string content
reaches BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS are scanned in a worker
thread so the regex pass over multi-MB base64 images does not block the event loop.
reaches BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS are copied in a worker thread,
so neither the regex pass over multi-MB base64 images nor the copy of a long
history blocks the event loop. A few strings with no data URI, under
BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS in total, are copied inline, since
checking them for data URIs and copying them costs less than the thread hop.
"""
if messages is None or MAX_BASE64_LENGTH_FOR_LOGGING <= 0:
return messages
total_chars: Final = sum(len(leaf) for leaf in _iter_string_leaves(messages))
leaves: Final = tuple(_iter_string_leaves(messages))
total_chars: Final = sum(len(leaf) for leaf in leaves)
if total_chars < BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS:
return truncate_base64_in_messages(messages)
if total_chars < BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS and _is_few_plain_strings(leaves):
return _copy_messages(messages, lambda text: text)
return await asyncio.to_thread(truncate_base64_in_messages, messages)

4
litellm/router_utils/pattern_match_deployments.py → litellm/litellm_core_utils/pattern_match_deployments.py
View file

@ -279,7 +279,9 @@ class PatternMatchRouter:
PatternMatchRouter.set_deployment_model_name(
matched_pattern=pattern_match, litellm_deployment_litellm_model=deployment["litellm_params"]["model"]
)
for deployment in self.patterns[pattern]
for deployment in self.patterns.get( # pyright: ignore[reportUnknownMemberType] # untyped deployment dicts
pattern, ()
)
)

17
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -88,17 +88,22 @@ def handle_any_messages_to_chat_completion_str_messages_conversion(
return [{"input": str(messages)}]
def _message_with_str_content(message: AllMessageValues) -> AllMessageValues:
texts: Final = convert_content_list_to_str(message=message)
if not texts:
return message
return cast( # cast-ok: the same message with its content flattened to one string
AllMessageValues, {**message, "content": texts}
)
def handle_messages_with_content_list_to_str_conversion(
messages: list[AllMessageValues],
) -> list[AllMessageValues]:
"""
Handles messages with content list conversion
Returns new messages with each content list flattened to a string, leaving the caller's messages untouched
"""
for message in messages:
texts = convert_content_list_to_str(message=message)
if texts:
message["content"] = texts
return messages
return [_message_with_str_content(message) for message in messages]
def strip_name_from_message(message: AllMessageValues, allowed_name_roles: list[str] = ["user"]) -> AllMessageValues:

42
litellm/litellm_core_utils/provider_affinity.py
View file

@ -2,7 +2,12 @@ import re
from collections.abc import Mapping
from typing import Final
from litellm.constants import SESSION_ID_FROM_TRACE_ID_PARAM_KEY, SESSION_ID_GENERATED_METADATA_KEY
from litellm.constants import (
CALLER_SESSION_ID_METADATA_KEY,
SESSION_ID_FROM_TRACE_ID_PARAM_KEY,
SESSION_ID_GENERATED_METADATA_KEY,
)
from litellm.litellm_core_utils.core_helpers import as_str_mapping
_HTTP_HEADER_NAME_PATTERN: Final = re.compile(r"^[!#$%&'*+\-.^_`|~0-9A-Za-z]+$")
_FORBIDDEN_AFFINITY_HEADERS: Final = frozenset(
@ -98,6 +103,41 @@ def get_stable_session_id(litellm_params: object | None) -> str | None:
return _non_generated_litellm_session_id(litellm_params) or _metadata_session_id(litellm_params)
def get_caller_session_id(litellm_params: object | None) -> str | None:
"""Resolve a caller conversation without accepting a generated session or a promoted trace id."""
if litellm_params is None:
return None
metadata_values: Final = _metadata_values(litellm_params)
preserved_session: Final = next(
(
value
for metadata in metadata_values
if isinstance(value := _get_value(metadata, CALLER_SESSION_ID_METADATA_KEY), str) and value
),
None,
)
if preserved_session:
return preserved_session
direct_session: Final = _get_value(litellm_params, "session_id")
if isinstance(direct_session, str) and direct_session:
return direct_session
proxy_request: Final = _get_value(litellm_params, "proxy_server_request")
headers: Final = as_str_mapping(_get_value(proxy_request, "headers"))
trace_header: Final = next(
(value for key, value in (headers or {}).items() if key.lower() == "x-litellm-trace-id"), None
)
candidates: Final = (
get_caller_litellm_session_id(litellm_params),
*(
_get_value(metadata, "session_id")
for metadata in metadata_values
if not _get_value(metadata, SESSION_ID_GENERATED_METADATA_KEY)
),
)
return next((value for value in candidates if isinstance(value, str) and value and value != trace_header), None)
def add_provider_affinity_header(
headers: Mapping[str, object], litellm_params: object | None, default_header: str | None = None
) -> dict[str, object]: # mutable-ok: downstream handlers add auth and signing headers

2
litellm/router_utils/ptu_shares.py → litellm/litellm_core_utils/ptu_shares.py
View file

@ -12,13 +12,13 @@ from datetime import date
from typing import Final, Generic, TypeAlias, TypeVar
from litellm.litellm_core_utils.ptu_pricing import parsed_ptu_shares, ptu_terms
from litellm.litellm_core_utils.routing_common_utils import get_request_team_id, team_may_use_deployment
from litellm.llms.azure.ptu_capacity import (
PTUCapacity,
deployment_ptu_capacity,
is_azure_deployment,
is_str_keyed_mapping,
)
from litellm.router_utils.common_utils import get_request_team_id, team_may_use_deployment
_DeploymentT: Final = TypeVar("_DeploymentT", bound=Mapping[str, object])
WildcardRoute: TypeAlias = Callable[[str], Sequence[Mapping[str, object]]]

0
litellm/router_utils/reasoning_effort_capability.py → litellm/litellm_core_utils/reasoning_effort_capability.py
View file

0
litellm/router_utils/common_utils.py → litellm/litellm_core_utils/routing_common_utils.py
View file

0
litellm/router_utils/routing_groups.py → litellm/litellm_core_utils/routing_groups.py
View file

34
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -329,6 +329,20 @@ class ChunkProcessor:
models[0] if models else None,
)
@staticmethod
def _get_upstream_provider(chunks: Sequence[object]) -> str | None:
def _provider_from_chunk(chunk: object) -> str | None:
hidden_params: Final = get_hidden_params(chunk)
if hidden_params is None:
return None
provider: Final[object] = hidden_params.get("upstream_provider")
return provider if isinstance(provider, str) and provider else None
return next(
(provider for chunk in chunks if (provider := _provider_from_chunk(chunk)) is not None),
None,
)
@staticmethod
def apply_provider_assembled_streaming_metadata(
response: ModelResponse,
@ -506,10 +520,22 @@ class ChunkProcessor:
chunks,
first_chunk_model,
)
if provider_response_model is not None:
response._hidden_params = dict( # pyright: ignore[reportPrivateUsage] # ModelResponse exposes no public hidden-params setter
response._hidden_params, # pyright: ignore[reportPrivateUsage] # ModelResponse exposes no public hidden-params getter
provider_response_model=provider_response_model,
upstream_provider: Final = self._get_upstream_provider(chunks)
provider_hidden_params: Final[Mapping[str, str]] = MappingProxyType(
{
key: value
for key, value in (
("provider_response_model", provider_response_model),
("upstream_provider", upstream_provider),
)
if value is not None
}
)
if provider_hidden_params:
response_hidden_params: Final[Mapping[str, builtins.object]] = get_hidden_params(response) or {}
set_hidden_params(
response,
dict(response_hidden_params, **provider_hidden_params),
)
return response

23
litellm/litellm_core_utils/streaming_handler.py
View file

@ -277,6 +277,7 @@ class _VertexChunkLike(Protocol):
class _ParsedChunkHiddenParams(LiteLLMBaseModel):
provider_specific_fields: Mapping[str, object] | None = None
upstream_provider: str | None = None
def _provider_response_model(chunk: object) -> str | None:
@ -293,9 +294,17 @@ def _parsed_provider_hidden_params(hidden: object) -> _ParsedChunkHiddenParams |
return None
def _provider_upstream_provider(chunk: object) -> str | None:
parsed: Final = _parsed_provider_hidden_params(get_hidden_params(chunk))
if parsed is None or not parsed.upstream_provider:
return None
return parsed.upstream_provider
def _provider_hidden_params(
chunk: object,
provider_response_model: str | None,
upstream_provider: str | None,
) -> Mapping[str, object] | None:
hidden: Final = get_hidden_params(chunk)
parsed: Final = _parsed_provider_hidden_params(hidden)
@ -308,6 +317,10 @@ def _provider_hidden_params(
for key, value in (
("provider_response_model", provider_response_model),
("provider_specific_fields", provider_specific_fields),
(
"upstream_provider",
parsed.upstream_provider if parsed is not None and parsed.upstream_provider else upstream_provider,
),
)
if value is not None
}
@ -360,6 +373,7 @@ class CustomStreamWrapper:
self.system_fingerprint: str | None = None
self._provider_response_model: str | None = None
self._upstream_provider: str | None = None
self.received_finish_reason: str | None = None
self.intermittent_finish_reason: str | None = None # finish reasons that show up mid-stream
self.special_tokens = [
@ -1653,10 +1667,17 @@ class CustomStreamWrapper:
provider_response_model: Final = _provider_response_model(chunk)
if provider_response_model is not None:
self._provider_response_model = provider_response_model
upstream_provider: Final = _provider_upstream_provider(chunk)
if self._upstream_provider is None and upstream_provider is not None:
self._upstream_provider = upstream_provider
prebuilt_delta: Final = self._prebuilt_plain_delta(chunk)
plain_delta_created: bool = prebuilt_delta is not None # rebind-ok: the first chunk builds its delta below
model_response = self.model_response_creator(
hidden_params=_provider_hidden_params(chunk, self._provider_response_model),
hidden_params=_provider_hidden_params(
chunk,
self._provider_response_model,
self._upstream_provider,
),
delta=prebuilt_delta,
)
response_obj: dict[str, Any] = {}

2
litellm/litellm_core_utils/token_counter.py
View file

@ -56,9 +56,9 @@ from litellm.litellm_core_utils.tokenizer import HuggingFaceTokenizer, OpenAIEnc
if TYPE_CHECKING:
from litellm.litellm_core_utils.tokenizer import Encoding, HuggingFace
from litellm.litellm_core_utils.deployment_pricing import deployment_pricing_models
from litellm.litellm_core_utils.url_utils import safe_get
from litellm.llms.custom_httpx.http_handler import get_httpx_client, header_value
from litellm.router_utils.deployment_pricing import deployment_pricing_models
from litellm.rust_bridge.tokenizer import get_encoding
from litellm.types.llms.anthropic import (
AnthropicContentParamSource,

47
litellm/llms/__init__.py
View file

@ -9,6 +9,8 @@ from litellm.types.utils import CallTypes
from . import *
if TYPE_CHECKING:
from collections.abc import Callable
from litellm.llms.base_llm.guardrail_translation.base_translation import (
BaseTranslation,
)
@ -30,7 +32,29 @@ def get_cost_for_google_maps_grounding_request(
return cost_per_google_maps_grounding_request(usage=usage, model_info=model_info)
def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", model_info: "ModelInfo") -> float | None:
def _with_served_model_search_pricing(
model_info: "ModelInfo",
response_model: str | None,
served_model_provider: str,
lookup_model_info: "Callable[[str, str | None], ModelInfo | None] | None",
) -> "ModelInfo":
if model_info.get("search_context_cost_per_query") is not None or response_model is None:
return model_info
served_model_info: Final = (
lookup_model_info(response_model, served_model_provider) if lookup_model_info is not None else None
)
if served_model_info is None:
return model_info
return {**model_info, "search_context_cost_per_query": served_model_info.get("search_context_cost_per_query")}
def get_cost_for_web_search_request(
custom_llm_provider: str,
usage: "Usage",
model_info: "ModelInfo",
response_model: str | None = None,
lookup_model_info: "Callable[[str, str | None], ModelInfo | None] | None" = None,
) -> float | None:
"""
Get the cost for a web search request for a given model.
@ -38,6 +62,8 @@ def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", mo
custom_llm_provider: The custom LLM provider.
usage: The usage object.
model_info: The model info.
response_model: The model the provider response names.
lookup_model_info: Looks up a model's cost map entry, returning None when it is not mapped.
"""
if custom_llm_provider == "gemini":
from .gemini.cost_calculator import cost_per_web_search_request
@ -46,17 +72,25 @@ def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", mo
elif custom_llm_provider == "anthropic":
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
return get_cost_for_anthropic_web_search(
model_info=_with_served_model_search_pricing(model_info, response_model, "anthropic", lookup_model_info),
usage=usage,
)
elif custom_llm_provider.startswith("vertex_ai"):
# Anthropic Claude models on Vertex AI populate server_tool_use.web_search_requests
# (same as the direct Anthropic API), not prompt_tokens_details.web_search_requests
# (which is the Gemini field). Route claude-* models to the Anthropic calculator.
model_key: Final[str] = model_info.get("key", "") if model_info else ""
if "claude" in model_key.lower():
if "claude" in (response_model or model_key).lower():
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
verbose_logger.debug("vertex_ai/claude model detected — routing web search cost to Anthropic calculator")
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
return get_cost_for_anthropic_web_search(
model_info=_with_served_model_search_pricing(
model_info, response_model, "vertex_ai", lookup_model_info
),
usage=usage,
)
from .vertex_ai.gemini.cost_calculator import (
cost_per_web_search_request as cost_per_web_search_request_vertex_ai,
@ -71,7 +105,10 @@ def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", mo
return None
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
return get_cost_for_anthropic_web_search(
model_info=_with_served_model_search_pricing(model_info, response_model, "azure_ai", lookup_model_info),
usage=usage,
)
elif custom_llm_provider == "perplexity":
# Perplexity handles search costs internally in its own cost calculator
# Return 0.0 to indicate costs are already accounted for

40
litellm/llms/anthropic/chat/transformation.py
View file

@ -167,6 +167,7 @@ _DEFER_LOADING_ADAPTER: Final = TypeAdapter(bool)
_TOOL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_TOOLS_ADAPTER: Final = TypeAdapter(list[object])
_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_REQUEST_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_NATIVE_OUTPUT_STRING_FORMATS: Final = frozenset(
("date-time", "time", "date", "duration", "email", "hostname", "uri", "ipv4", "ipv6", "uuid")
)
@ -2254,6 +2255,16 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
headers=headers,
)
@staticmethod
def _messages_within_cache_control_cap(
messages: Sequence[AllMessageValues], optional_params: Mapping[str, object]
) -> Sequence[AllMessageValues]:
from litellm.integrations.anthropic_cache_control_hook import messages_within_cache_control_cap
return messages_within_cache_control_cap(
messages, optional_params.get("tools"), optional_params.get("cache_control"), optional_params
)
def transform_request(
self,
model: str,
@ -2271,9 +2282,18 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
anthropic_messages_pt,
)
drop_params: Final = litellm.drop_params is True or litellm_params.get("drop_params") is True
drop_params: Final = drops_unsupported_params(litellm_params.get("drop_params"))
capped_messages: Final = (
list(
self._messages_within_cache_control_cap(
messages, _REQUEST_PARAMS_ADAPTER.validate_python(optional_params)
)
)
if drop_params
else messages
)
if "tools" not in optional_params and messages is not None and has_tool_call_blocks(messages):
if "tools" not in optional_params and has_tool_call_blocks(capped_messages):
optional_params["tools"], _ = self._map_tools(add_dummy_tool(custom_llm_provider="anthropic"))
# Drop thinking param if thinking is enabled but thinking_blocks are missing
@ -2285,9 +2305,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
# Related issue: https://github.com/BerriAI/litellm/issues/18926
if (
optional_params.get("thinking") is not None
and messages is not None
and last_assistant_with_tool_calls_has_no_thinking_blocks(messages)
and not any_assistant_message_has_thinking_blocks(messages)
and last_assistant_with_tool_calls_has_no_thinking_blocks(capped_messages)
and not any_assistant_message_has_thinking_blocks(capped_messages)
):
if litellm.modify_params:
optional_params.pop("thinking", None)
@ -2332,15 +2351,18 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
_name_forward_map, _name_reverse_map = self._sanitize_tool_names_in_request(
optional_params=optional_params,
)
if _name_forward_map:
messages = self._rewrite_tool_names_in_messages(messages, _name_forward_map)
request_messages: Final = (
self._rewrite_tool_names_in_messages(capped_messages, _name_forward_map)
if _name_forward_map
else capped_messages
)
if _name_reverse_map and isinstance(litellm_params, dict):
litellm_params[ANTHROPIC_TOOL_NAME_REVERSE_MAP_KEY] = _name_reverse_map
# Only the leading system run becomes the top-level system prompt. A later
# system message stays in the conversation: hoisting it rewrites the cached
# prefix and re-bills the whole history at cache-write pricing (#36559).
leading_system_run, later_messages = split_leading_system_run(messages)
leading_system_run, later_messages = split_leading_system_run(request_messages)
anthropic_system_message_list: Final = self.translate_system_message(messages=list(leading_system_run))
# Handling anthropic API Prompt Caching
if len(anthropic_system_message_list) > 0:
@ -2362,7 +2384,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
except Exception as e:
raise AnthropicError(
status_code=400,
message=f"{e}\nReceived Messages={messages}",
message=f"{e}\nReceived Messages={request_messages}",
) # don't use verbose_logger.exception, if exception is raised
self.update_headers_with_optional_anthropic_beta(

37
litellm/llms/anthropic/common_utils.py
View file

@ -4,6 +4,7 @@ This file contains common utils for anthropic calls.
import copy
import json
import os
import re
from collections.abc import Container, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
@ -19,6 +20,8 @@ from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm.constants import (
ANTHROPIC_COMPATIBLE_MAX_USER_ID_LENGTH,
ANTHROPIC_MAX_USER_ID_LENGTH,
DEFAULT_MODEL_CREATED_AT_TIME,
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
@ -178,6 +181,40 @@ def tools_without_custom_field(tools: Sequence[object], hoist_defer_loading: boo
return [_tool_without_custom_field(tool, hoist_defer_loading, drop_custom) for tool in tools]
def anthropic_pass_through_api_base() -> str:
return os.getenv("ANTHROPIC_API_BASE") or os.getenv("ANTHROPIC_BASE_URL") or "https://api.anthropic.com"
def _same_host_and_port(url: httpx.URL, other: str) -> bool:
try:
other_url: Final = httpx.URL(other)
except httpx.InvalidURL:
return False
return (url.host, url.port) == (other_url.host, other_url.port)
def anthropic_upstream_user_id_limit(url: str) -> int | None:
upstream: Final = httpx.URL(url)
if not upstream.path.removesuffix("/").endswith("/v1/messages"):
return None
if upstream.host == "api.anthropic.com":
return ANTHROPIC_MAX_USER_ID_LENGTH
return (
ANTHROPIC_COMPATIBLE_MAX_USER_ID_LENGTH
if _same_host_and_port(upstream, anthropic_pass_through_api_base())
else None
)
def _has_email_shape(token: str) -> bool:
at_sign: Final = token.find("@", 1)
return at_sign != -1 and token.find(".", at_sign + 2, -1) != -1
def anthropic_accepts_user_id(user_id: str, max_length: int) -> bool:
return len(user_id) <= max_length and user_id.isprintable() and not any(map(_has_email_shape, user_id.split()))
def is_anthropic_messages_url(url: str) -> bool:
"""Check whether a URL addresses Anthropic's Messages API."""
parsed_url: Final = urlparse(url)

14
litellm/llms/anthropic/messages/native/handler.py
View file

@ -19,6 +19,13 @@ from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheCont
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.exception_mapping_utils import exception_type
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.llms.anthropic.common_utils import (
AnthropicModelInfo,
strip_provider_specific_fields_from_anthropic_messages,
@ -31,13 +38,6 @@ from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.router_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.types.llms.anthropic import AllAnthropicToolsValues
from litellm.types.llms.anthropic_messages.anthropic_request import AnthropicMetadata
from litellm.types.llms.anthropic_messages.anthropic_response import (

2
litellm/llms/anthropic/messages/native/interceptors/advisor.py
View file

@ -22,7 +22,7 @@ import litellm
import litellm.constants as _c
from litellm.litellm_core_utils.url_utils import validate_url
from litellm.llms.anthropic.common_utils import strip_advisor_blocks_from_messages
from litellm.router_utils.cooldown_handlers import mark_advisor_orchestration_failure
from litellm.router_api import mark_advisor_orchestration_failure
from litellm.types.llms.anthropic import ANTHROPIC_ADVISOR_TOOL_TYPE
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,

2
litellm/llms/anthropic/messages/utils.py
View file

@ -214,7 +214,7 @@ def normalize_reasoning_effort_value(
if chain is None:
return effort
from litellm.router_utils.reasoning_effort_capability import (
from litellm.litellm_core_utils.reasoning_effort_capability import (
deployment_rounds_reasoning_effort,
resolve_supported_reasoning_efforts,
)

12
litellm/llms/azure_ai/decisions/transformation.py
View file

@ -12,7 +12,7 @@ from litellm.llms.azure_ai.common_utils import (
AZURE_ENTRA_LITELLM_PARAM_KEYS,
get_azure_ai_auth_headers,
)
from litellm.llms.base_llm.decisions.transformation import BaseDecisionsConfig
from litellm.llms.base_llm.decisions.transformation import BaseDecisionsConfig, DecisionsInputLimits
_FOUNDRY_ROUTE_SUFFIX: Final = re.compile(r"(/api/projects/[^/]+)?(/openai/v1|/openai|/models|/v1)?$")
_AZURE_AI_CREDENTIAL_PARAM_KEYS: Final = AZURE_ENTRA_LITELLM_PARAM_KEYS | frozenset({"azure_ad_token_provider"})
@ -26,6 +26,16 @@ class AzureAIDecisionsConfig(BaseDecisionsConfig):
api_base_env = ("AZURE_AI_API_BASE",)
api_key_required = False
def get_input_limits(
self, model: str, custom_llm_provider: str, headers: Mapping[str, str]
) -> DecisionsInputLimits:
limits: Final = super().get_input_limits(model, custom_llm_provider, headers)
return (
limits
if limits.max_request_tokens is not None
else super().get_input_limits("Microsoft-Decision-1", custom_llm_provider, headers)
)
def missing_api_base_message(self, custom_llm_provider: str) -> str:
return "Missing AZURE_AI_API_BASE - set AZURE_AI_API_BASE or pass api_base, e.g. https://<resource>.services.ai.azure.com"

55
litellm/llms/base_llm/decisions/transformation.py
View file

@ -1,7 +1,7 @@
import json
from abc import ABC
from collections.abc import Callable, Mapping, Sequence
from dataclasses import replace
from dataclasses import dataclass, replace
from itertools import chain
from types import MappingProxyType
from typing import Final
@ -12,6 +12,7 @@ from pydantic_core import ErrorDetails
from typing_extensions import assert_never
import litellm
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
from litellm.litellm_core_utils.url_utils import join_api_base
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.secret_managers.main import get_secret_str
@ -297,13 +298,7 @@ def _systemone_question(question: DecisionsIRQuestion) -> Mapping[str, object]:
assert_never(question)
def ir_to_systemone_request(
model: str, request: DecisionsIRRequest, *, supports_audio: bool = False, supports_video: bool = False
) -> Mapping[str, object] | UnsupportedDecisionsRequest:
if request.audio and not supports_audio:
return UnsupportedDecisionsRequest(reason="audio input is not supported by this Decisions model")
if request.videos and not supports_video:
return UnsupportedDecisionsRequest(reason="video input is not supported by this Decisions model")
def ir_to_systemone_request(model: str, request: DecisionsIRRequest) -> Mapping[str, object]:
state: Final = _systemone_state(request.input)
keyed_questions: Final = zip(systemone_keys(request.questions), request.questions, strict=True)
images: Final = input_images(request.input)
@ -503,6 +498,16 @@ def ir_to_systemone_response(response: DecisionsIRResponse, request: DecisionsIR
)
def catalog_mode(model: str, custom_llm_provider: str) -> str | None:
try:
key: Final = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider)["key"]
except Exception: # noqa: BLE001 # get_model_info raises bare Exception for an unmapped model, which the catalog does not list
return None
entry: Final = GetModelCostMap.loaded_model_cost_map().get(key)
mode: Final = entry.get("mode") if entry is not None else None
return mode if isinstance(mode, str) else None
def _is_decisions_model(model: str, custom_llm_provider: str) -> bool:
try:
model_mode: Final = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider).get("mode")
@ -511,6 +516,12 @@ def _is_decisions_model(model: str, custom_llm_provider: str) -> bool:
return is_decisions_model_mode(model_mode)
@dataclass(frozen=True, slots=True)
class DecisionsInputLimits:
max_request_tokens: int | None = None
max_question_tokens: int | None = None
class BaseDecisionsConfig(ABC):
path: str = "/v1/systemone"
api_key_env: tuple[str, ...] = ()
@ -533,6 +544,19 @@ class BaseDecisionsConfig(ABC):
def provider_reported_cost(self, response: DecisionsIRResponse) -> float | None:
return None
def get_input_limits(
self, model: str, custom_llm_provider: str, headers: Mapping[str, str]
) -> DecisionsInputLimits:
try:
limit: Final = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider).get(
"max_input_tokens"
)
except Exception: # noqa: BLE001 # get_model_info raises bare Exception for an unmapped model
return DecisionsInputLimits()
return DecisionsInputLimits(
max_request_tokens=limit if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0 else None
)
def missing_api_base_message(self, custom_llm_provider: str) -> str:
return f"api_base is required for Decisions provider '{custom_llm_provider}'"
@ -621,12 +645,16 @@ class BaseDecisionsConfig(ABC):
model=model, custom_llm_provider=custom_llm_provider
)
def supports_audio_input(self, model: str, custom_llm_provider: str) -> bool:
def supports_audio_input(self, model: str, custom_llm_provider: str, deployment_flag: bool | None) -> bool:
if deployment_flag is not None:
return deployment_flag
return _is_decisions_model(model, custom_llm_provider) and litellm.supports_audio_input(
model=model, custom_llm_provider=custom_llm_provider
)
def supports_video_input(self, model: str, custom_llm_provider: str) -> bool:
def supports_video_input(self, model: str, custom_llm_provider: str, deployment_flag: bool | None) -> bool:
if deployment_flag is not None:
return deployment_flag
return _is_decisions_model(model, custom_llm_provider) and litellm.supports_video_input(
model=model, custom_llm_provider=custom_llm_provider
)
@ -637,12 +665,7 @@ class BaseDecisionsConfig(ABC):
request: DecisionsIRRequest,
custom_llm_provider: str,
) -> Mapping[str, object] | UnsupportedDecisionsRequest:
return ir_to_systemone_request(
self.request_model(model),
request,
supports_audio=self.supports_audio_input(model, custom_llm_provider),
supports_video=self.supports_video_input(model, custom_llm_provider),
)
return ir_to_systemone_request(self.request_model(model), request)
def unwrap_response(self, payload: object) -> object:
return payload

35
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -25,6 +25,7 @@ from litellm.constants import (
)
from litellm.litellm_core_utils.core_helpers import (
as_str_mapping,
drops_unsupported_params,
filter_exceptions_from_params,
filter_internal_params,
map_finish_reason,
@ -2129,6 +2130,24 @@ class AmazonConverseConfig(BaseConfig):
return data
@staticmethod
def _claude_messages_within_cache_control_cap(
messages: Sequence[AllMessageValues],
capability_model: str,
optional_params: Mapping[str, object],
litellm_params: Mapping[str, object],
) -> Sequence[AllMessageValues]:
if not drops_unsupported_params(litellm_params.get("drop_params")) or "claude" not in capability_model.lower():
return messages
from litellm.integrations.anthropic_cache_control_hook import (
messages_within_cache_control_cap,
tool_message_with_single_cache_mark,
)
return messages_within_cache_control_cap(
tuple(map(tool_message_with_single_cache_mark, messages)), optional_params.get("tools")
)
async def _async_transform_request(
self,
model: str,
@ -2140,7 +2159,13 @@ class AmazonConverseConfig(BaseConfig):
base_model: Final = litellm_params.get("base_model")
capability_model: Final = bedrock_capability_model(model, base_model)
messages, system_content_blocks = self._transform_system_message(
messages, model=capability_model, deployment_model=model
list(
self._claude_messages_within_cache_control_cap(
messages, capability_model, _JSON_OBJECT.validate_python(optional_params), litellm_params
)
),
model=capability_model,
deployment_model=model,
)
# Convert last user message to guarded_text if guardrailConfig is present
@ -2221,7 +2246,13 @@ class AmazonConverseConfig(BaseConfig):
base_model: Final = litellm_params.get("base_model")
capability_model: Final = bedrock_capability_model(model, base_model)
messages, system_content_blocks = self._transform_system_message(
messages, model=capability_model, deployment_model=model
list(
self._claude_messages_within_cache_control_cap(
messages, capability_model, _JSON_OBJECT.validate_python(optional_params), litellm_params
)
),
model=capability_model,
deployment_model=model,
)
# Convert last user message to guarded_text if guardrailConfig is present

6
litellm/llms/bedrock/rerank/transformation.py
View file

@ -6,6 +6,8 @@ Why separate file? Make it easy to see how transformation works
from typing import Final
from pydantic import TypeAdapter
from litellm._uuid import uuid
from litellm.types.llms.bedrock import (
BedrockRerankBedrockRerankingConfiguration,
@ -90,7 +92,9 @@ class BedrockRerankConfig:
example input:
{"results":[{"index":0,"relevanceScore":0.6847912669181824},{"index":1,"relevanceScore":0.5980774760246277}]}
"""
_billed_units = RerankBilledUnits(**response.get("usage", {"search_units": 1})) # by default 1 search unit
_billed_units: Final = TypeAdapter(RerankBilledUnits).validate_python(
response.get("usage", {"search_units": 1})
)
_tokens: Final = RerankTokens(**response.get("usage", {}))
rerank_meta: Final = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens)

2
litellm/llms/clf_ai_gateway/chat/transformation.py
View file

@ -4,8 +4,8 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import UnsupportedParamsError
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.secret_managers.main import get_secret_str
SUPPORTED_OPENAI_PARAMS: Final = (

8
litellm/llms/compaction.py
View file

@ -9,7 +9,7 @@ from pydantic import TypeAdapter
from litellm.types.llms.openai import AllMessageValues
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
CompactionProtocol: TypeAlias = Literal["chat", "messages"]
_MAPPING: Final = TypeAdapter(Mapping[str, object])
@ -32,7 +32,9 @@ def get_native_compaction_provider(params: Mapping[str, object]) -> NativeCompac
return compaction if compaction.supports_native_compaction(params) else None
async def dispatch(router: Router, protocol: CompactionProtocol, payload: Mapping[str, object]) -> Mapping[str, object]:
async def dispatch(
router: RoutingContext, protocol: CompactionProtocol, payload: Mapping[str, object]
) -> Mapping[str, object]:
if protocol == "messages":
return _MAPPING.validate_python(
await router.aanthropic_messages(custom_llm_provider=None, client=None, **payload)
@ -45,4 +47,4 @@ async def dispatch(router: Router, protocol: CompactionProtocol, payload: Mappin
{key: value for key, value in payload.items() if key not in ("model", "messages", "stream")}
),
)
return _MAPPING.validate_python(response.model_dump())
return _MAPPING.validate_python(response.model_dump()) # pyright: ignore[reportAttributeAccessIssue] # stream=False: acompletion returns a ModelResponse

15
litellm/llms/custom_httpx/aiohttp_transport.py
View file

@ -110,6 +110,15 @@ def map_aiohttp_exceptions() -> Generator[None, None, None]:
raise mapped_exc(message) from exc
def _unread_payload(content: object) -> bytes:
"""Bytes the upstream sent before the connection dropped that aiohttp buffered but never handed out, since its
reads raise the stored payload error ahead of the buffer."""
if not isinstance(content, aiohttp.StreamReader):
return b""
buffered: Final = b"".join(content._buffer) # pyright: ignore[reportPrivateUsage] # no public read drains it
return buffered[content._buffer_offset :] # pyright: ignore[reportPrivateUsage] # see above
def _release_transport_anchor(_transport: object) -> None:
"""Release a transport anchored to a response stream. See ``_anchor_transport_to``."""
@ -144,6 +153,12 @@ class AiohttpResponseStream(httpx.AsyncByteStream):
raise httpx.ReadError(str(e)) from e
except aiohttp.http_exceptions.TransferEncodingError as e:
raise httpx.ReadError(str(e)) from e
except aiohttp.ClientPayloadError:
unread: Final = _unread_payload(self._aiohttp_response.content)
if unread:
yield unread
with map_aiohttp_exceptions():
raise
except Exception:
# For other exceptions, use the normal mapping
with map_aiohttp_exceptions():

2
litellm/llms/databricks/embed/handler.py
View file

@ -26,6 +26,7 @@ class DatabricksEmbeddingHandler(OpenAILikeEmbeddingHandler, DatabricksBase):
aembedding=None,
custom_endpoint: bool | None = None,
headers: dict | None = None,
drop_params: bool = False,
) -> EmbeddingResponse:
# Check for custom user agent in optional_params or environment
# This allows partners building on LiteLLM to set their own telemetry
@ -58,4 +59,5 @@ class DatabricksEmbeddingHandler(OpenAILikeEmbeddingHandler, DatabricksBase):
aembedding=aembedding,
custom_endpoint=True,
headers=headers,
drop_params=drop_params,
)

5
litellm/llms/deepinfra/chat/transformation.py
View file

@ -130,7 +130,8 @@ class DeepInfraConfig(OpenAIGPTConfig):
- Input: {"role": "tool", "content": [{"type": "text", "text": "result"}]}
- Output: {"role": "tool", "content": "[{\"type\": \"text\", \"text\": \"result\"}]"}
"""
for message in messages:
copied_messages: Final = [message.copy() for message in messages]
for message in copied_messages:
if message.get("role") == "tool":
content = message.get("content")
@ -149,7 +150,7 @@ class DeepInfraConfig(OpenAIGPTConfig):
# For complex content, serialize the entire array as JSON string
message["content"] = json.dumps(content)
return messages
return copied_messages
@overload
def _transform_messages(

38
litellm/llms/fireworks_ai/chat/transformation.py
View file

@ -23,6 +23,7 @@ from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionToolParam,
ChatCompletionToolParamFunctionChunk,
OpenAIChatCompletionToolParam,
)
from litellm.types.utils import (
@ -45,6 +46,7 @@ from ...openai.chat.gpt_transformation import (
OpenAIChatCompletionStreamingHandler,
OpenAIGPTConfig,
)
from ...openai.common_utils import without_cache_control
from ..common_utils import (
FIREROUTER,
FireworksAIException,
@ -480,15 +482,23 @@ class FireworksAIConfig(FireworksAIMixin, OpenAIGPTConfig):
return (("response_format", _json_schema_response_format(choice_schema, "choice")),)
def _transform_tools(self, tools: list[OpenAIChatCompletionToolParam]) -> list[OpenAIChatCompletionToolParam]:
for tool in tools:
if tool.get("type") != "function":
continue
function = tool["function"]
function.pop("strict", None)
params = function.get("parameters")
if isinstance(params, dict):
unpack_legacy_defs(params)
return tools
return [self._fireworks_tool(tool) for tool in tools]
@staticmethod
def _fireworks_tool(tool: OpenAIChatCompletionToolParam) -> OpenAIChatCompletionToolParam:
if tool.get("type") != "function":
return tool
function: Final = tool["function"]
params: Final = function.get("parameters")
unstrict_function: Final = cast( # cast-ok: the same function definition minus `strict`
ChatCompletionToolParamFunctionChunk, {key: value for key, value in function.items() if key != "strict"}
)
new_function: Final[ChatCompletionToolParamFunctionChunk] = (
{**unstrict_function, "parameters": unpack_legacy_defs(params, copy=True)}
if isinstance(params, dict)
else unstrict_function
)
return {**tool, "function": new_function}
def _transform_messages_helper(
self, messages: list[AllMessageValues], model: str, litellm_params: dict
@ -496,10 +506,6 @@ class FireworksAIConfig(FireworksAIMixin, OpenAIGPTConfig):
"""
Strip fields not permitted by FireworksAI from messages.
"""
from litellm.litellm_core_utils.prompt_templates.common_utils import (
filter_value_from_dict,
)
supports_vision_value: Final = self._get_model_cost_capability_exact(model=model, capability="supports_vision")
for message in messages:
if message["role"] == "user":
@ -530,9 +536,11 @@ class FireworksAIConfig(FireworksAIMixin, OpenAIGPTConfig):
model=model,
llm_provider="fireworks_ai",
)
filter_value_from_dict(cast(dict, message), "cache_control")
return [strip_litellm_response_only_fields(message) for message in messages]
return [
strip_litellm_response_only_fields(cast(AllMessageValues, without_cache_control(message)))
for message in messages
]
# Cached index of fireworks_ai/* entries from litellm.model_cost. Building
# this index requires a full scan of model_cost (tens of thousands of

9
litellm/llms/groq/chat/transformation.py
View file

@ -140,7 +140,8 @@ class GroqChatConfig(OpenAILikeChatConfig):
def _transform_messages(
self, messages: list[AllMessageValues], model: str, is_async: bool = False
) -> list[AllMessageValues] | Coroutine[object, object, list[AllMessageValues]]:
for idx, message in enumerate(messages):
transformed_messages: Final = list(messages)
for idx, message in enumerate(transformed_messages):
"""
1. Don't pass 'null' function_call assistant message to groq - https://github.com/BerriAI/litellm/issues/5839
"""
@ -154,12 +155,12 @@ class GroqChatConfig(OpenAILikeChatConfig):
for k, v in _message.items():
if v is not None:
new_message[k] = v
messages[idx] = strip_litellm_internal_message_fields(new_message)
transformed_messages[idx] = strip_litellm_internal_message_fields(new_message)
if is_async:
return super()._transform_messages(messages=messages, model=model, is_async=True)
return super()._transform_messages(messages=transformed_messages, model=model, is_async=True)
else:
return super()._transform_messages(messages=messages, model=model, is_async=False)
return super()._transform_messages(messages=transformed_messages, model=model, is_async=False)
def transform_messages(
self,

3
litellm/llms/hosted_vllm/chat/transformation.py
View file

@ -2,6 +2,7 @@
Translate from OpenAI's `/v1/chat/completions` to VLLM's `/v1/chat/completions`
"""
import copy
import json
from collections.abc import Coroutine
from typing import Final, Literal, cast, overload
@ -88,7 +89,7 @@ class HostedVLLMChatConfig(OpenAIGPTConfig):
) -> dict:
_tools = non_default_params.pop("tools", None)
if _tools is not None:
_tools = remove_additional_properties(_tools)
_tools = remove_additional_properties(copy.deepcopy(_tools)) # pyright: ignore[reportUnknownArgumentType] # tools arrive untyped through the bare-dict params
_tools = remove_strict_from_schema(_tools)
if isinstance(_tools, list):
_tools = self._convert_custom_tools_to_function_tools(_tools)

4
litellm/llms/hosted_vllm/rerank/transformation.py
View file

@ -244,5 +244,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
return RerankResponse(
id=response.get("id") or str(uuid.uuid4()),
results=rerank_results,
meta=rerank_meta,
meta=TypeAdapter(RerankResponseMeta).validate_python(response["meta"])
if "meta" in response and response["meta"] is not None
else rerank_meta,
)

11
litellm/llms/mistral/chat/transformation.py
View file

@ -25,16 +25,16 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
strip_litellm_internal_message_fields,
strip_none_values_from_message,
)
from litellm.litellm_core_utils.reasoning_effort_capability import (
declared_reasoning_efforts_for_model,
nearest_declared_reasoning_effort,
)
from litellm.litellm_core_utils.reasoning_effort_utils import OPAQUE_VALUE, warn_dropped_reasoning
from litellm.litellm_core_utils.url_utils import ensure_api_route
from litellm.llms.openai.chat.gpt_transformation import (
OpenAIChatCompletionStreamingHandler,
OpenAIGPTConfig,
)
from litellm.router_utils.reasoning_effort_capability import (
declared_reasoning_efforts_for_model,
nearest_declared_reasoning_effort,
)
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.mistral import (
MistralCacheKeyParams,
@ -755,7 +755,8 @@ class MistralConfig(OpenAIGPTConfig):
if _name is not None:
# Remove name if not a tool message
if message["role"] != "tool" or isinstance(_name, str) and len(_name.strip()) == 0:
message.pop("name", None)
message_without_name: Final = {k: v for k, v in message.items() if k != "name"}
return cast(AllMessageValues, message_without_name) # cast-ok: the same message minus name
return message

2
litellm/llms/oci/chat/transformation.py
View file

@ -21,6 +21,7 @@ from litellm._logging import verbose_logger
from litellm.constants import DEFAULT_OCI_CHAT_MAX_TOKENS
from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
from litellm.litellm_core_utils.prompt_templates.common_utils import messages_with_file_parts_as_image_urls
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.litellm_core_utils.reasoning_effort_utils import warn_dropped_reasoning
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.litellm_core_utils.url_utils import join_api_base
@ -54,7 +55,6 @@ from litellm.llms.oci.common_utils import (
sign_oci_request,
validate_oci_environment,
)
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.types.llms.oci import (
CohereChatRequest,
OCIChatRequestPayload,

24
litellm/llms/ollama/chat/transformation.py
View file

@ -235,6 +235,19 @@ class OllamaChatConfig(BaseConfig):
litellm_params["function_name"] = function_name
tools: Final = optional_params.pop("tools", None)
def transform_tool_call(typed_tool: ChatCompletionAssistantToolCall) -> OllamaToolCall:
function: Final = OllamaToolCallFunction(
name=typed_tool["function"].get("name") or "",
arguments=json.loads(typed_tool["function"]["arguments"])
if "arguments" in typed_tool["function"]
else {},
)
tool_id: Final = typed_tool.get("id")
ollama_tool_call: Final = (
OllamaToolCall(function=function) if tool_id is None else OllamaToolCall(id=tool_id, function=function)
)
return ollama_tool_call
new_messages: Final = []
for m in messages:
if isinstance(
@ -248,16 +261,7 @@ class OllamaChatConfig(BaseConfig):
for tool in tool_calls:
typed_tool = ChatCompletionAssistantToolCall(**tool)
if typed_tool["type"] == "function":
arguments = {}
if "arguments" in typed_tool["function"]:
arguments = json.loads(typed_tool["function"]["arguments"])
ollama_tool_call = OllamaToolCall(
function=OllamaToolCallFunction(
name=typed_tool["function"].get("name") or "",
arguments=arguments,
)
)
new_tools.append(ollama_tool_call)
new_tools.append(transform_tool_call(typed_tool))
reasoning_content, parsed_content = extract_reasoning_content(cast(dict, m))
content_str = convert_content_list_to_str(cast(AllMessageValues, m))
images = extract_images_from_message(cast(AllMessageValues, m))

186
litellm/llms/openai/chat/gpt_transformation.py
View file

@ -14,7 +14,7 @@ from pydantic import TypeAdapter, ValidationError
import litellm
from litellm.constants import OPENAI_SYSTEM_MESSAGES_FIRST_PROVIDERS
from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params, map_finish_reason
from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import (
extract_reasoning_content,
handle_invalid_parallel_tool_calls,
@ -64,7 +64,7 @@ from litellm.types.utils import (
)
from litellm.utils import convert_to_model_response_object, declared_mid_conversation_system_support
from ..common_utils import OpenAIError
from ..common_utils import OpenAIError, without_cache_control
from ..workload_identity import get_workload_identity_bearer_token, resolve_openai_workload_identity_config
if TYPE_CHECKING:
@ -128,6 +128,29 @@ def _litellm_params_str(litellm_params: Mapping[str, object] | None, key: str) -
return value if isinstance(value, str) else None
def _drops_params(litellm_params: Mapping[str, object]) -> bool:
return drops_unsupported_params(litellm_params.get("drop_params"))
def _names_claude(model: str, litellm_params: Mapping[str, object]) -> bool:
base_model: Final = litellm_params.get("base_model")
return "claude" in model.lower() or (isinstance(base_model, str) and "claude" in base_model.lower())
def _claude_messages_within_cache_control_cap(
model: str,
messages: Sequence[AllMessageValues],
tools: Sequence["ChatCompletionToolParam"] | None,
litellm_params: Mapping[str, object],
request_params: Mapping[str, object],
) -> Sequence[AllMessageValues]:
if not _names_claude(model, litellm_params) or not _drops_params(litellm_params):
return messages
from litellm.integrations.anthropic_cache_control_hook import messages_within_cache_control_cap
return list(messages_within_cache_control_cap(messages, tools, request_params.get("cache_control"), request_params))
def _prompt_cache_options(optional_params: Mapping[str, object]) -> object | None:
try:
extra_body: Final = _OBJECT_MAPPING.validate_python(optional_params.get("extra_body"))
@ -311,52 +334,58 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
return content_copy
async def _async_handle_pdf_url(self, content_item: ChatCompletionFileObjectFile) -> ChatCompletionFileObjectFile:
file_id: Final = content_item.get("file_id")
content_copy: Final = content_item.copy()
file_id: Final = content_copy.get("file_id")
if file_id is not None: # check for file id being url done in _handle_pdf_url
base64_data: Final = await async_convert_url_to_base64(file_id)
content_item["file_data"] = base64_data
content_item["filename"] = "my_file.pdf"
content_item.pop("file_id")
return content_item
content_copy["file_data"] = base64_data
content_copy["filename"] = "my_file.pdf"
content_copy.pop("file_id")
return content_copy
def _common_file_data_check(self, content_item: ChatCompletionFileObjectFile) -> ChatCompletionFileObjectFile:
file_data: Final = content_item.get("file_data")
filename: Final = content_item.get("filename")
if file_data is not None and filename is None:
content_item["filename"] = "my_file.pdf"
return {**content_item, "filename": "my_file.pdf"}
return content_item
@staticmethod
def _image_item_for_openai(content_item: ChatCompletionImageObject) -> ChatCompletionImageObject:
image_url: Final = content_item["image_url"]
if isinstance(image_url, str):
return {**content_item, "image_url": {"url": image_url}}
if not isinstance(image_url, dict):
return content_item
new_image_url_obj: Final = cast(
ChatCompletionImageUrlObject,
{k: v for k, v in image_url.items() if k not in {"format", "video_metadata"}},
)
return {**content_item, "image_url": new_image_url_obj}
@staticmethod
def _file_item_for_openai(content_item: ChatCompletionFileObject) -> ChatCompletionFileObject:
litellm_specific_params: Final = {"format", "detail", "video_metadata"}
file_obj: Final = content_item.get("file")
if file_obj is None:
raise litellm.BadRequestError(
message="Content block has type='file' but is missing the required 'file' field",
model=None,
llm_provider="openai",
)
new_file_obj: Final = ChatCompletionFileObjectFile(
**{k: v for k, v in file_obj.items() if k not in litellm_specific_params}
)
return {**content_item, "file": new_file_obj}
def _apply_common_transform_content_item(
self,
content_item: OpenAIMessageContentListBlock,
) -> OpenAIMessageContentListBlock:
litellm_specific_params: Final = {"format", "detail", "video_metadata"}
if content_item.get("type") == "image_url":
content_item = cast(ChatCompletionImageObject, content_item)
if isinstance(content_item["image_url"], str):
content_item["image_url"] = {
"url": content_item["image_url"],
}
elif isinstance(content_item["image_url"], dict):
new_image_url_obj: Final = cast(
ChatCompletionImageUrlObject,
{k: v for k, v in content_item["image_url"].items() if k not in {"format", "video_metadata"}},
)
content_item["image_url"] = new_image_url_obj
elif content_item.get("type") == "file":
content_item = cast(ChatCompletionFileObject, content_item)
file_obj: Final = content_item.get("file")
if file_obj is None:
raise litellm.BadRequestError(
message="Content block has type='file' but is missing the required 'file' field",
model=None,
llm_provider="openai",
)
new_file_obj: Final = ChatCompletionFileObjectFile(
**{k: v for k, v in file_obj.items() if k not in litellm_specific_params}
)
content_item["file"] = new_file_obj
return self._image_item_for_openai(cast(ChatCompletionImageObject, content_item))
if content_item.get("type") == "file":
return self._file_item_for_openai(cast(ChatCompletionFileObject, content_item))
return content_item
def _transform_content_item(
@ -419,31 +448,34 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
hoisted_messages: Final = hoist_images_from_tool_messages(stripped_messages)
async def _async_transform():
for message in hoisted_messages:
message_content = message.get("content")
message_role = message.get("role")
if message_role == "user" and message_content and isinstance(message_content, list):
message_content_types = cast(list[OpenAIMessageContentListBlock], message_content)
for i, content_item in enumerate(message_content_types):
message_content_types[i] = await self._async_transform_content_item(
cast(OpenAIMessageContentListBlock, content_item),
)
return hoisted_messages
return [await self._async_user_message_for_openai(message) for message in hoisted_messages]
if is_async:
return _async_transform()
else:
for message in hoisted_messages:
message_content = message.get("content")
message_role = message.get("role")
if message_role == "user" and message_content and isinstance(message_content, list):
message_content_types = cast(list[OpenAIMessageContentListBlock], message_content)
for i, content_item in enumerate(message_content):
message_content_types[i] = self._transform_content_item(
cast(OpenAIMessageContentListBlock, content_item)
)
return hoisted_messages
return [self._user_message_for_openai(message) for message in hoisted_messages]
@staticmethod
def _user_content_parts(message: AllMessageValues) -> Sequence[OpenAIMessageContentListBlock] | None:
message_content: Final = message.get("content")
if message.get("role") != "user" or not message_content or not isinstance(message_content, list):
return None
return cast(Sequence[OpenAIMessageContentListBlock], message_content)
def _user_message_for_openai(self, message: AllMessageValues) -> AllMessageValues:
parts: Final = self._user_content_parts(message)
if parts is None:
return message
return cast( # cast-ok: the same user message with each content part rebuilt for OpenAI
AllMessageValues, {**message, "content": [self._transform_content_item(part) for part in parts]}
)
async def _async_user_message_for_openai(self, message: AllMessageValues) -> AllMessageValues:
parts: Final = self._user_content_parts(message)
if parts is None:
return message
return cast( # cast-ok: the same user message with each content part rebuilt for OpenAI
AllMessageValues, {**message, "content": [await self._async_transform_content_item(part) for part in parts]}
)
def transform_messages(
self,
@ -462,23 +494,15 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
messages: list[AllMessageValues],
tools: list["ChatCompletionToolParam"] | None = None,
) -> tuple[list[AllMessageValues], list["ChatCompletionToolParam"] | None]:
from litellm.litellm_core_utils.prompt_templates.common_utils import (
filter_value_from_dict,
)
from litellm.types.llms.openai import ChatCompletionToolParam
for i, message in enumerate(messages):
messages[i] = cast(
AllMessageValues,
filter_value_from_dict(message, "cache_control"),
)
if tools is not None:
for i, tool in enumerate(tools):
tools[i] = cast(
ChatCompletionToolParam,
filter_value_from_dict(tool, "cache_control"),
)
return messages, tools
new_messages: Final = [cast(AllMessageValues, without_cache_control(message)) for message in messages]
new_tools: Final = (
[cast(ChatCompletionToolParam, without_cache_control(tool)) for tool in tools]
if tools is not None
else None
)
return new_messages, new_tools
def translate_developer_role_to_system_role(
self,
@ -527,6 +551,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
tools: Sequence["ChatCompletionToolParam"] | None,
prompt_cache_options: object,
litellm_params: Mapping[str, object],
request_params: Mapping[str, object],
) -> tuple[Sequence[AllMessageValues], Sequence["ChatCompletionToolParam"] | None]:
"""
A deployment that reads OpenAI's ``prompt_cache_breakpoint`` (gpt-5.6+ on
@ -543,10 +568,11 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
model, custom_llm_provider, api_base, prompt_cache_options
)
if not reads_breakpoints and self._should_preserve_cache_control_for_endpoint(custom_llm_provider, api_base):
return messages, tools
drops_breakpoints: Final = not reads_breakpoints and (
litellm.drop_params is True or litellm_params.get("drop_params") is True
)
return (
_claude_messages_within_cache_control_cap(model, messages, tools, litellm_params, request_params),
tools,
)
drops_breakpoints: Final = not reads_breakpoints and _drops_params(litellm_params)
renamed: Final = (
[AnthropicCacheControlHook.cache_control_as_prompt_cache_breakpoints(message) for message in messages]
if reads_breakpoints
@ -561,7 +587,13 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
if drops_breakpoints
else list(tools)
)
return self.remove_cache_control_flag_from_messages_and_tools(model=model, messages=renamed, tools=kept_tools)
sent_messages, sent_tools = self.remove_cache_control_flag_from_messages_and_tools(
model=model, messages=renamed, tools=kept_tools
)
return (
_claude_messages_within_cache_control_cap(model, sent_messages, sent_tools, litellm_params, request_params),
sent_tools,
)
def _sanitized_tools_update_for_openai(
self,
@ -623,6 +655,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
tools=optional_params.get("tools", []),
prompt_cache_options=_prompt_cache_options(optional_params),
litellm_params=litellm_params,
request_params=optional_params,
)
if tools is not None and len(tools) > 0:
optional_params["tools"] = tools
@ -654,6 +687,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
tools=optional_params.get("tools", []),
prompt_cache_options=_prompt_cache_options(optional_params),
litellm_params=litellm_params,
request_params=optional_params,
)
if tools is not None and len(tools) > 0:
optional_params["tools"] = tools

9
litellm/llms/openai/chat/o_series_transformation.py
View file

@ -153,12 +153,13 @@ class OpenAIOSeriesConfig(OpenAIGPTConfig):
- role: system ==> translate to role 'user'
"""
_supports_system_messages: Final = supports_system_messages(model, "openai")
for i, message in enumerate(messages):
transformed_messages: Final = list(messages)
for i, message in enumerate(transformed_messages):
if message["role"] == "system" and not _supports_system_messages:
new_message = ChatCompletionUserMessage(content=message["content"], role="user")
messages[i] = new_message # Replace the old message with the new one
transformed_messages[i] = new_message # Replace the old message with the new one
if is_async:
return super()._transform_messages(messages, model, is_async=cast(Literal[True], True))
return super()._transform_messages(transformed_messages, model, is_async=cast(Literal[True], True))
else:
return super()._transform_messages(messages, model, is_async=cast(Literal[False], False))
return super()._transform_messages(transformed_messages, model, is_async=cast(Literal[False], False))

44
litellm/llms/openai/common_utils.py
View file

@ -3,6 +3,7 @@ Common helpers / utils across al OpenAI endpoints
"""
import asyncio
import copy
import hashlib
import inspect
import json
@ -31,7 +32,7 @@ if TYPE_CHECKING:
import litellm
from litellm.exceptions import error_response_without_credentials
from litellm.litellm_core_utils.completion_timeout import CompletionTimeout
from litellm.litellm_core_utils.core_helpers import as_str_mapping, decoded_json_object
from litellm.litellm_core_utils.core_helpers import as_str_mapping, decoded_json_object, iter_items
from litellm.litellm_core_utils.reasoning_effort_utils import warn_dropped_reasoning_params
from litellm.litellm_core_utils.token_counter import token_counter, usage_tokenizer_model
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@ -43,6 +44,47 @@ from litellm.llms.custom_httpx.http_handler import (
)
def _is_plain_dict(value: object) -> bool:
return isinstance(value, dict)
def _is_list(value: object) -> bool:
return isinstance(value, list)
def _contains_cache_control(value: object) -> bool:
pending: Final[list[object]] = [value] # mutable-ok: a local worklist, never returned or stored
while pending:
node = pending.pop()
mapping = as_str_mapping(node)
if mapping is not None:
if "cache_control" in mapping:
return True
pending.extend(mapping.values())
elif _is_list(node):
pending.extend(iter_items(node))
return False
def without_cache_control(item: object) -> object:
"""Return `item` with every `cache_control` key removed, without touching the original.
`filter_value_from_dict` deletes the key in place and recurses into nested dicts and
lists, so the caller's own message, tool or input object has to be copied first: it
may reuse the same list on a provider that does support prompt caching. The copy is
skipped when there is nothing to strip, which keeps the object identity the
pass-through paths rely on and avoids a deep copy on every request.
"""
from litellm.litellm_core_utils.prompt_templates.common_utils import filter_value_from_dict
if not _is_plain_dict(item) or not _contains_cache_control(item):
return item
copied: Final = copy.deepcopy(item)
if isinstance(copied, dict):
filter_value_from_dict(copied, "cache_control")
return copied
def _get_client_init_params(cls: type) -> tuple[str, ...]:
"""Extract __init__ parameter names (excluding 'self') from a class."""
return tuple(p for p in inspect.signature(cls.__init__).parameters if p != "self")

4
litellm/llms/openai/decisions/transformation.py
View file

@ -354,10 +354,10 @@ class OpenAIDecisionsConfig(BaseDecisionsConfig):
) -> str | None:
return resolve_openai_bearer_token(api_key=api_key, api_base=api_base, litellm_params=litellm_params)
def supports_audio_input(self, model: str, custom_llm_provider: str) -> bool:
def supports_audio_input(self, model: str, custom_llm_provider: str, deployment_flag: bool | None) -> bool:
return False
def supports_video_input(self, model: str, custom_llm_provider: str) -> bool:
def supports_video_input(self, model: str, custom_llm_provider: str, deployment_flag: bool | None) -> bool:
return False
def transform_decisions_request(

22
litellm/llms/openai/openai.py
View file

@ -16,6 +16,7 @@ from openai._base_client import make_request_options
from openai._constants import RAW_RESPONSE_HEADER
from openai._legacy_response import LegacyAPIResponse
from openai._types import RequestOptions
from openai.pagination import SyncCursorPage
from openai.types import CreateEmbeddingResponse
from openai.types.beta.assistant_deleted import AssistantDeleted
from openai.types.chat import ChatCompletion, ChatCompletionChunk
@ -40,6 +41,7 @@ from litellm.types.utils import (
LiteLLMBatch,
ModelResponse,
ModelResponseStream,
openai_batch_status,
)
from litellm.utils import (
CustomStreamWrapper,
@ -2182,6 +2184,18 @@ class OpenAIFilesAPI(BaseLLM):
return response
def _with_openai_batch_fields(batches: Sequence[Batch]) -> Sequence[Batch]:
return [
batch.model_copy(
update={
"object": batch.object or "batch",
"status": openai_batch_status(batch.status),
}
)
for batch in batches
]
class OpenAIBatchesAPI(BaseLLM):
"""
OpenAI methods to support for batches
@ -2372,7 +2386,7 @@ class OpenAIBatchesAPI(BaseLLM):
):
verbose_logger.debug("listing batches, after= %s, limit= %s", after, limit)
response: Final = await openai_client.batches.list(after=after, limit=limit)
return response
return response.model_copy(update={"data": _with_openai_batch_fields(response.data)})
def list_batches(
self,
@ -2384,7 +2398,7 @@ class OpenAIBatchesAPI(BaseLLM):
organization: str | None,
after: str | None = None,
limit: int | None = None,
client: OpenAI | None = None,
client: OpenAI | AsyncOpenAI | None = None,
litellm_params: Mapping[str, object] | None = None,
):
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
@ -2409,7 +2423,9 @@ class OpenAIBatchesAPI(BaseLLM):
)
return self.alist_batches(openai_client=openai_client, after=after, limit=limit)
response: Final = openai_client.batches.list(after=after, limit=limit)
return response
if not isinstance(response, SyncCursorPage):
return response
return response.model_copy(update={"data": _with_openai_batch_fields(response.data)})
class OpenAIAssistantsAPI(BaseLLM):

53
litellm/llms/openai/responses/transformation.py
View file

@ -52,7 +52,7 @@ from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
from litellm.types.workload_identity import OPENAI_WIF_KWARGS_KEYS
from ..common_utils import OpenAIError
from ..common_utils import OpenAIError, without_cache_control
from ..workload_identity import get_workload_identity_bearer_token, resolve_openai_workload_identity_config
OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS: Final = 16
@ -658,13 +658,6 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
input=self._replayable_input(marker_safe_input, reads_breakpoints, maps_anthropic_tool_search),
tools=mapped_tools,
)
if marker_safe_input is not validated_input:
self._strip_caller_input_as_if_unmarked(
model=model,
caller_input=validated_input,
reads_breakpoints=reads_breakpoints,
maps_anthropic_tool_search=maps_anthropic_tool_search,
)
object_schema_tools: Final = self._tools_with_object_parameters(model=model, tools=stripped_tools)
sanitized_tools: Final = self._sanitized_tool_schemas_for_openai(
model=model, tools=object_schema_tools, litellm_params=litellm_params
@ -679,17 +672,6 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
return marked_input
return input_without_unanswered_anthropic_tool_search_calls(marked_input)
def _strip_caller_input_as_if_unmarked(
self,
model: str,
caller_input: str | ResponseInputParam,
reads_breakpoints: bool,
maps_anthropic_tool_search: bool,
) -> None:
self.remove_cache_control_flag_from_input_and_tools(
model=model, input=self._replayable_input(caller_input, reads_breakpoints, maps_anthropic_tool_search)
)
def _input_with_wire_safe_prompt_cache_breakpoints(
self,
model: str,
@ -800,24 +782,29 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
the chat path. Strips Anthropic-only `cache_control` markers from
Responses API input content blocks and tools.
`filter_value_from_dict` mutates each dict in place, so the same
objects are returned.
`filter_value_from_dict` deletes the key in place and recurses, so each
item is copied first: the caller keeps its own input list and may reuse
it on a provider that does support prompt caching.
"""
from litellm.litellm_core_utils.prompt_templates.common_utils import (
filter_value_from_dict,
new_input: Final = (
cast( # cast-ok: the comprehension rebuilds the input item for item
ResponseInputParam,
[without_cache_control(item) for item in input],
)
if isinstance(input, list)
else input
)
if isinstance(input, list):
for item in input:
if isinstance(item, dict):
filter_value_from_dict(cast(dict, item), "cache_control")
new_tools: Final = (
cast( # cast-ok: the comprehension rebuilds the tool list item for item
Sequence[ALL_RESPONSES_API_TOOL_PARAMS],
[without_cache_control(tool) for tool in tools],
)
if tools is not None
else None
)
if tools is not None:
for tool in tools:
if isinstance(tool, dict):
filter_value_from_dict(cast(dict, tool), "cache_control")
return input, tools
return new_input, new_tools
def _drop_bridge_minted_reasoning_items(self, input: str | ResponseInputParam) -> str | ResponseInputParam:
if self.custom_llm_provider not in _PROVIDERS_REPLAYING_ONLY_THEIR_OWN_REASONING or not isinstance(input, list):

14
litellm/llms/openai_like/README.md
View file

@ -125,6 +125,20 @@ The provider inherits all request/response handling from OpenAI's Responses API
If `supported_endpoints` is omitted, it defaults to `[]` (only chat completions, which is always enabled for JSON providers).
When only some models serve an endpoint natively, list them under `model_supported_endpoints`. Every other model reaches `/v1/responses` and `/v1/messages` through chat completions:
```json
{
"your_provider": {
"supported_endpoints": ["/v1/chat/completions"],
"model_supported_endpoints": {
"model-a": ["/v1/messages"],
"model-b": ["/v1/responses"]
}
}
}
```
### How It Works
1. `json_loader.py` checks `supported_endpoints` for `/v1/responses`

77
litellm/llms/openai_like/embedding/handler.py
View file

@ -3,11 +3,16 @@
## Allows jina ai embedding calls - which don't allow 'encoding_format' in payload.
import json
import re
from collections.abc import Mapping
from types import MappingProxyType
from typing import Final
import httpx
from pydantic import TypeAdapter
import litellm
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,
@ -17,6 +22,57 @@ from litellm.types.utils import EmbeddingResponse
from ..common_utils import OpenAILikeBase, OpenAILikeError
_KEY_HEADERS_ADAPTER: Final = TypeAdapter(Mapping[str, str])
_EXTRA_HEADERS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_TRANSPORT_HEADER_NAMES: Final = frozenset(
{
"host",
"content-length",
"transfer-encoding",
"connection",
"keep-alive",
"proxy-connection",
"te",
"trailer",
"upgrade",
"expect",
}
)
_HEADER_NAME: Final = re.compile(r"[!#$%&'*+\-.^_`|~0-9A-Za-z]+")
_HEADER_VALUE: Final = re.compile(r"[\t\x20-\x7e]*")
def _header_text(name: str, value: object) -> str | None:
if isinstance(value, bool) or not isinstance(value, str | int | float):
return None
if _HEADER_NAME.fullmatch(name) is None:
return None
return str(value).strip(" \t\r\n")
def _extra_headers_as_text(extra_headers: Mapping[str, object], model: str, drop_params: bool) -> Mapping[str, str]:
caller_set: Final = {
name: value for name, value in extra_headers.items() if name.lower() not in _TRANSPORT_HEADER_NAMES
}
texts: Final = MappingProxyType({name: _header_text(name, value) for name, value in caller_set.items()})
refused: Final = sorted(name for name, text in texts.items() if text is None)
if refused and not drop_params:
raise litellm.UnsupportedParamsError(
message=f"extra_headers {', '.join(map(repr, refused))} cannot be sent. A header needs a name made of "
"letters, digits or !#$%&'*+-.^_`|~ and a text or number value. "
"To drop them from the call, set `litellm.drop_params = True`.",
model=model,
)
named: Final = MappingProxyType({name: text for name, text in texts.items() if text is not None})
unsendable: Final = sorted(name for name, text in named.items() if _HEADER_VALUE.fullmatch(text) is None)
if unsendable:
verbose_logger.warning(
"extra_headers %s dropped for model %r, since a header value must be printable ASCII on one line",
unsendable,
model,
)
return MappingProxyType({name: text for name, text in named.items() if name not in unsendable})
class OpenAILikeEmbeddingHandler(OpenAILikeBase):
def __init__(self, **kwargs):
@ -94,6 +150,7 @@ class OpenAILikeEmbeddingHandler(OpenAILikeBase):
aembedding=None,
custom_endpoint: bool | None = None,
headers: dict | None = None,
drop_params: bool = False,
) -> EmbeddingResponse:
api_base, headers = self._validate_environment(
api_base=api_base,
@ -103,8 +160,20 @@ class OpenAILikeEmbeddingHandler(OpenAILikeBase):
custom_endpoint=custom_endpoint,
)
model = model
filtered_optional_params: Final = {k: v for k, v in optional_params.items() if v not in (None, "")}
data: Final = {"model": model, "input": input, **filtered_optional_params}
data: Final = {
"model": model,
"input": input,
**{k: v for k, v in optional_params.items() if k != "extra_headers" and v not in (None, "")},
}
extra_headers: Final = _extra_headers_as_text(
_EXTRA_HEADERS_ADAPTER.validate_python(optional_params.get("extra_headers") or {}), model, drop_params
)
key_headers: Final = _KEY_HEADERS_ADAPTER.validate_python(headers)
extra_header_names: Final = frozenset(name.lower() for name in extra_headers)
request_headers: Final = {
**{name: value for name, value in key_headers.items() if name.lower() not in extra_header_names},
**extra_headers,
}
## LOGGING
logging_obj.pre_call(
@ -123,7 +192,7 @@ class OpenAILikeEmbeddingHandler(OpenAILikeBase):
api_key=api_key,
timeout=timeout,
client=client,
headers=headers,
headers=request_headers,
)
if client is None or isinstance(client, AsyncHTTPHandler):
self.client = HTTPHandler(timeout=timeout)
@ -134,7 +203,7 @@ class OpenAILikeEmbeddingHandler(OpenAILikeBase):
try:
response: Final = self.client.post(
api_base,
headers=headers,
headers=request_headers,
data=json.dumps(data),
)

25
litellm/llms/openai_like/json_loader.py
View file

@ -3,7 +3,7 @@ JSON-based provider configuration loader for OpenAI-compatible providers.
"""
import json
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import Final
@ -14,6 +14,7 @@ from litellm.types.utils import ProviderSpecificModelInfo
_PROVIDER_INFO_ADAPTER: Final = TypeAdapter(ProviderSpecificModelInfo)
_SPECIAL_HANDLING_ADAPTER: Final = TypeAdapter(Mapping[str, bool])
_MODEL_ENDPOINTS_ADAPTER: Final = TypeAdapter(Mapping[str, tuple[str, ...]])
class SimpleProviderConfig:
@ -30,7 +31,10 @@ class SimpleProviderConfig:
self.special_handling: Mapping[str, bool] = _SPECIAL_HANDLING_ADAPTER.validate_python(
data.get("special_handling", {})
)
self.supported_endpoints = data.get("supported_endpoints", [])
self.supported_endpoints: Sequence[str] = data.get("supported_endpoints", [])
self.model_supported_endpoints: Mapping[str, tuple[str, ...]] = _MODEL_ENDPOINTS_ADAPTER.validate_python(
data.get("model_supported_endpoints", {})
)
self.provider_info: ProviderSpecificModelInfo = (
_PROVIDER_INFO_ADAPTER.validate_python(data["provider_info"]) if "provider_info" in data else {}
)
@ -90,6 +94,23 @@ class JSONProviderRegistry:
return False
return "/v1/responses" in provider.supported_endpoints
@classmethod
def supports_embeddings(cls, slug: str) -> bool:
provider: Final = cls._providers.get(slug)
if provider is None:
return False
return "/v1/embeddings" in provider.supported_endpoints
@classmethod
def serves_endpoint(cls, slug: str, model: str | None, endpoint: str) -> bool:
provider: Final = cls._providers.get(slug)
if provider is None:
return False
model_endpoints: Final = (
provider.model_supported_endpoints.get(model.removeprefix(f"{slug}/"), ()) if model else ()
)
return endpoint in provider.supported_endpoints or endpoint in model_endpoints
@classmethod
def list_providers(cls) -> list:
"""List all registered provider slugs"""

12
litellm/llms/openai_like/providers.json
View file

@ -211,7 +211,7 @@
"param_mappings": {
"max_completion_tokens": "max_tokens"
},
"supported_endpoints": ["/v1/chat/completions", "/v1/responses", "/v1/embeddings"]
"supported_endpoints": ["/v1/chat/completions", "/v1/responses"]
},
"scx-ai": {
"base_url": "https://api.scx.ai/v1",
@ -231,6 +231,16 @@
"api_base_env": "PRISM_API_BASE",
"supported_endpoints": ["/v1/chat/completions", "/v1/responses", "/v1/messages"]
},
"hostinger_router": {
"base_url": "https://router.hostinger.com/v1",
"api_key_env": "HOSTINGER_ROUTER_API_KEY",
"api_base_env": "HOSTINGER_ROUTER_API_BASE",
"supported_endpoints": ["/v1/chat/completions", "/v1/embeddings"],
"model_supported_endpoints": {
"claude-sonnet-5": ["/v1/messages"],
"gpt-5.2": ["/v1/responses"]
}
},
"sail": {
"base_url": "https://api.sailresearch.com/v1",
"api_key_env": "SAIL_API_KEY",

30
litellm/llms/openrouter/chat/transformation.py
View file

@ -14,7 +14,7 @@ import httpx
from pydantic import ConfigDict, TypeAdapter
from litellm.litellm_core_utils.core_helpers import finite_cost
from litellm.litellm_core_utils.hidden_params import HIDDEN_PARAMS_ATTR, set_hidden_params
from litellm.litellm_core_utils.hidden_params import HIDDEN_PARAMS_ATTR, get_hidden_params, set_hidden_params
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
@ -212,19 +212,27 @@ class OpenrouterConfig(OpenAIGPTConfig):
# OpenRouter returns cost information in the usage object when usage.include=true
try:
response_json: Final = _JSON_OBJECT.validate_python(raw_response.json())
response_provider: Final = response_json.get("provider")
if isinstance(response_provider, str) and response_provider:
set_hidden_params(
model_response,
{**(get_hidden_params(model_response) or {}), "upstream_provider": response_provider},
)
if "usage" in response_json and response_json["usage"]:
response_cost: Final = _JSON_OBJECT.validate_python(response_json["usage"]).get("cost")
if response_cost is not None:
# Store cost in hidden params for the cost calculator to use
if not hasattr(model_response, HIDDEN_PARAMS_ATTR):
set_hidden_params(model_response, {})
hidden_params: Final = cast( # cast-ok: preserve mapping operations on dynamic response metadata
dict[str, object], getattr(model_response, HIDDEN_PARAMS_ATTR)
cost_hidden_params: Final = (
cast( # cast-ok: preserve mapping operations on dynamic response metadata
dict[str, object], getattr(model_response, HIDDEN_PARAMS_ATTR)
)
)
if "additional_headers" not in hidden_params:
hidden_params["additional_headers"] = {}
if "additional_headers" not in cost_hidden_params:
cost_hidden_params["additional_headers"] = {}
additional_headers: Final = cast( # cast-ok: preserve mapping operations on response metadata
dict[str, object], hidden_params["additional_headers"]
dict[str, object], cost_hidden_params["additional_headers"]
)
reported_cost: Final = (
finite_cost(float(response_cost)) if isinstance(response_cost, str | int | float) else None
@ -282,7 +290,7 @@ class OpenRouterChatCompletionStreamingHandler(BaseModelResponseIterator):
for choice in chunk["choices"]:
choice["delta"]["reasoning_content"] = choice["delta"].get("reasoning")
new_choices.append(choice)
return ModelResponseStream(
model_response: Final = ModelResponseStream(
id=chunk["id"],
object="chat.completion.chunk",
created=chunk["created"],
@ -290,6 +298,12 @@ class OpenRouterChatCompletionStreamingHandler(BaseModelResponseIterator):
model=chunk["model"],
choices=new_choices,
)
typed_chunk: Final[Mapping[str, object]] = cast( # cast-ok: stream chunks are decoded JSON objects
Mapping[str, object], chunk
)
upstream_provider: Final[object] = typed_chunk.get("provider")
if isinstance(upstream_provider, str) and upstream_provider:
set_hidden_params(model_response, {"upstream_provider": upstream_provider})
except KeyError as e:
raise OpenRouterException(
message=f"KeyError: {e}, Got unexpected response from OpenRouter: {chunk}",
@ -298,3 +312,5 @@ class OpenRouterChatCompletionStreamingHandler(BaseModelResponseIterator):
)
except Exception as e:
raise e
else:
return model_response

87
litellm/llms/openrouter/common_utils.py
View file

@ -1,5 +1,92 @@
from collections.abc import Mapping, Sequence
from typing import Final
import httpx
from pydantic import BaseModel, ConfigDict
import litellm
from litellm.litellm_core_utils.url_utils import join_api_base
from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
_DEFAULT_API_BASE: Final = "https://openrouter.ai/api/v1"
_CHAT_CATALOG_ROUTE: Final = "models/user"
_CHAT_COMPLETIONS_PATH: Final = "/chat/completions"
_BATCH_ONLY_SUFFIX: Final = ":batch"
_CATALOG_TIMEOUT: Final = httpx.Timeout(10.0)
class OpenRouterException(BaseLLMException):
pass
class _CatalogModel(BaseModel):
model_config = ConfigDict(frozen=True)
id: str
class _Catalog(BaseModel):
model_config = ConfigDict(frozen=True)
data: tuple[_CatalogModel, ...]
def _chat_catalog_url(api_base: str) -> str:
return join_api_base(api_base.rstrip("/").removesuffix(_CHAT_COMPLETIONS_PATH), _CHAT_CATALOG_ROUTE)
class OpenRouterModelInfo(BaseLLMModelInfo):
@staticmethod
def get_api_key(api_key: str | None = None) -> str | None:
return (
api_key
or litellm.api_key
or litellm.openrouter_key
or get_secret_str("OPENROUTER_API_KEY")
or get_secret_str("OR_API_KEY")
)
@staticmethod
def get_api_base(api_base: str | None = None) -> str:
return api_base or litellm.api_base or get_secret_str("OPENROUTER_API_BASE") or _DEFAULT_API_BASE
@staticmethod
def get_base_model(model: str) -> str:
return model.removeprefix("openrouter/")
def validate_environment(
self,
headers: Mapping[str, str],
model: str,
messages: Sequence[AllMessageValues],
optional_params: Mapping[str, object],
litellm_params: Mapping[str, object],
api_key: str | None = None,
api_base: str | None = None,
) -> dict[str, str]: # mutable-ok: BaseLLMModelInfo.validate_environment returns a dict
resolved_key: Final = self.get_api_key(api_key)
if resolved_key is None:
return {**headers}
return {**headers, "Authorization": f"Bearer {resolved_key}"}
def get_models(
self, api_key: str | None = None, api_base: str | None = None
) -> list[str]: # mutable-ok: matches get_models' list[str] contract shared by every provider override
resolved_key: Final = self.get_api_key(api_key)
if resolved_key is None:
raise ValueError("Set OPENROUTER_API_KEY or the deployment's api_key to list OpenRouter models")
headers: Final = self.validate_environment(
headers={}, model="", messages=(), optional_params={}, litellm_params={}, api_key=resolved_key
)
response: Final = litellm.module_level_client.get( # pyright: ignore[reportUnknownMemberType] # legacy HTTP wrapper has untyped query parameters
url=_chat_catalog_url(self.get_api_base(api_base)), headers=headers, timeout=_CATALOG_TIMEOUT
)
response.raise_for_status()
return [
f"openrouter/{model.id}"
for model in _Catalog.model_validate_json(response.content).data
if not model.id.endswith(_BATCH_ONLY_SUFFIX)
]

5
litellm/llms/perplexity/cost_calculator.py
View file

@ -100,8 +100,9 @@ def cost_per_token(model: str, usage: Usage, current_time: datetime | None = Non
num_search_queries = getattr(usage.prompt_tokens_details, "web_search_requests", 0) or 0
# Check both possible keys for search cost (legacy and current)
search_cost_value: Final = model_info.get("search_queries_cost_per_query") or model_info.get(
"search_context_cost_per_query"
legacy_search_cost: Final = model_info.get("search_queries_cost_per_query")
search_cost_value: Final = (
model_info.get("search_context_cost_per_query") if legacy_search_cost is None else legacy_search_cost
)
if num_search_queries > 0 and search_cost_value is not None:
# Handle both dict and float formats

34
litellm/llms/perplexity/search/transformation.py
View file

@ -84,9 +84,9 @@ class PerplexitySearchConfig(BaseSearchConfig):
def transform_search_request(
self,
query: str | list[str],
optional_params: dict,
**kwargs,
) -> dict:
optional_params: dict[str, object],
**kwargs: object,
) -> dict[str, object]:
"""
Transform Search request to Perplexity API format.
@ -98,11 +98,7 @@ class PerplexitySearchConfig(BaseSearchConfig):
Args:
query: Search query (string or list of strings)
optional_params: Optional parameters for the request
- max_results: Maximum number of search results (1-20)
- search_domain_filter: List of domains to filter (max 20)
- max_tokens_per_page: Max tokens per page (default 1024)
- country: Country code filter (e.g., 'US', 'GB', 'DE')
optional_params: Provider parameters, forwarded unchanged when not None.
Returns:
Dict with typed request data following PerplexitySearchRequest spec
@ -111,24 +107,10 @@ class PerplexitySearchConfig(BaseSearchConfig):
"query": query,
}
# Add optional parameters following Perplexity API spec (only if not None)
max_results: Final = optional_params.get("max_results")
if max_results is not None:
request_data["max_results"] = max_results
search_domain_filter: Final = optional_params.get("search_domain_filter")
if search_domain_filter is not None:
request_data["search_domain_filter"] = search_domain_filter
max_tokens_per_page: Final = optional_params.get("max_tokens_per_page")
if max_tokens_per_page is not None:
request_data["max_tokens_per_page"] = max_tokens_per_page
country: Final = optional_params.get("country")
if country is not None:
request_data["country"] = country
return dict(request_data)
return {
**{key: value for key, value in optional_params.items() if value is not None},
**request_data,
}
def transform_search_response(
self,

2
litellm/llms/together_ai/chat/transformation.py
View file

@ -18,7 +18,7 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import UnsupportedParamsError
from litellm.litellm_core_utils.prompt_templates.common_utils import strip_litellm_response_only_fields
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.types.llms.openai import AllMessageValues
from litellm.utils import supports_function_calling, supports_reasoning, supports_response_schema

4
litellm/llms/together_ai/rerank/transformation.py
View file

@ -6,6 +6,8 @@ Why separate file? Make it easy to see how transformation works
from typing import Final
from pydantic import TypeAdapter
from litellm._uuid import uuid
from litellm.types.rerank import (
RerankBilledUnits,
@ -19,7 +21,7 @@ from litellm.types.rerank import (
class TogetherAIRerankConfig:
def _transform_response(self, response: dict) -> RerankResponse:
_billed_units: Final = RerankBilledUnits(**response.get("usage", {}))
_billed_units: Final = TypeAdapter(RerankBilledUnits).validate_python(response.get("usage", {}))
_tokens: Final = RerankTokens(**response.get("usage", {}))
rerank_meta: Final = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens)

13
litellm/llms/typesafe/decisions/transformation.py
View file

@ -6,7 +6,7 @@ from types import MappingProxyType
from typing import Final, Literal
from litellm.constants import TYPESAFE_DANGER_ZONE_HEADER_PREFIX
from litellm.llms.base_llm.decisions.transformation import BaseDecisionsConfig, image_media_type
from litellm.llms.base_llm.decisions.transformation import BaseDecisionsConfig, DecisionsInputLimits, image_media_type
from litellm.types.decisions import (
DecisionsAsset,
DecisionsIRMessages,
@ -19,6 +19,7 @@ from litellm.types.decisions import (
)
_IMAGES_HEADER: Final = f"{TYPESAFE_DANGER_ZONE_HEADER_PREFIX}images"
_LONG_CONTEXT_HEADER: Final = f"{TYPESAFE_DANGER_ZONE_HEADER_PREFIX}long-context"
_ASSET_FORMATS: Final[Mapping[str, Literal["png", "jpg"]]] = MappingProxyType(
{"image/png": "png", "image/jpeg": "jpg", "image/jpg": "jpg"}
)
@ -84,6 +85,16 @@ class TypeSafeDecisionsConfig(BaseDecisionsConfig):
def get_default_api_base(self) -> str | None:
return "https://api.typesafe.ai"
def get_input_limits(
self, model: str, custom_llm_provider: str, headers: Mapping[str, str]
) -> DecisionsInputLimits:
long_context_values: Final = tuple(
value.strip().lower() for name, value in headers.items() if name.lower() == _LONG_CONTEXT_HEADER
)
if long_context_values and all(value == "true" for value in long_context_values):
return DecisionsInputLimits(max_request_tokens=262144, max_question_tokens=262144)
return DecisionsInputLimits(max_request_tokens=65536, max_question_tokens=32768)
def supports_images_in(
self, decision_input: DecisionsIRState | DecisionsIRMessages, headers: Mapping[str, str]
) -> bool:

43
litellm/llms/vertex_ai/context_caching/transformation.py
View file

@ -110,38 +110,23 @@ def separate_cached_messages(
messages: list[AllMessageValues],
) -> tuple[list[AllMessageValues], list[AllMessageValues]]:
"""
Returns separated cached and non-cached messages.
Args:
messages: List of messages to be separated.
Returns:
Tuple containing:
- cached_messages: List of cached messages.
- non_cached_messages: List of non-cached messages.
Returns the first continuous block of cached messages, plus every system message once
any message is cached, since generateContent rejects a system_instruction beside a
cachedContent, and the remaining messages.
"""
cached_messages: list[AllMessageValues] = []
non_cached_messages: list[AllMessageValues] = []
filtered_messages: Final = [(idx, message) for idx, message in enumerate(messages) if is_cached_message(message)]
if not filtered_messages:
return [], messages
# Extract cached messages and their indices
filtered_messages: Final[list[tuple[int, AllMessageValues]]] = []
for idx, message in enumerate(messages):
if is_cached_message(message=message):
filtered_messages.append((idx, message))
# Validate only one block of continuous cached messages
last_continuous_block_idx: Final = get_first_continuous_block_idx(filtered_messages)
# Separate messages based on the block of cached messages
if filtered_messages and last_continuous_block_idx is not None:
first_cached_idx: Final = filtered_messages[0][0]
last_cached_idx: Final = filtered_messages[last_continuous_block_idx][0]
cached_messages = messages[first_cached_idx : last_cached_idx + 1]
non_cached_messages = messages[:first_cached_idx] + messages[last_cached_idx + 1 :]
else:
non_cached_messages = messages
return cached_messages, non_cached_messages
cached_block: Final = range(filtered_messages[0][0], filtered_messages[last_continuous_block_idx][0] + 1)
goes_in_cache: Final = tuple(
idx in cached_block or message.get("role") == "system" for idx, message in enumerate(messages)
)
return (
[message for message, cached in zip(messages, goes_in_cache) if cached],
[message for message, cached in zip(messages, goes_in_cache) if not cached],
)
def cached_messages_end_on_supported_turn(cached_messages: Sequence[AllMessageValues]) -> bool:

3
litellm/llms/watsonx/chat/transformation.py
View file

@ -4,6 +4,7 @@ Translation from OpenAI's `/chat/completions` endpoint to IBM WatsonX's `/text/c
Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-chat
"""
import copy
from typing import Final
from litellm import verbose_logger
@ -57,7 +58,7 @@ class IBMWatsonXChatConfig(IBMWatsonXMixin, OpenAIGPTConfig):
_tools = non_default_params.pop("tools", None)
if _tools is not None:
# remove 'additionalProperties' from tools
_tools = remove_additional_properties(_tools)
_tools = remove_additional_properties(copy.deepcopy(_tools)) # pyright: ignore[reportUnknownArgumentType] # tools arrive untyped through the bare-dict params
# remove 'strict' from tools
_tools = remove_strict_from_schema(_tools)
if _tools is not None:

4
litellm/llms/xai/chat/transformation.py
View file

@ -1,3 +1,4 @@
import copy
from collections.abc import AsyncIterator, Iterator, Mapping
from types import MappingProxyType
from typing import Any, Final
@ -208,7 +209,8 @@ class XAIChatConfig(OpenAIGPTConfig):
tools = [
filtered_tool
for filtered_tool in (
_FILTERED_TOOL.validate_python(filter_value_from_dict(tool, "strict")) for tool in value
_FILTERED_TOOL.validate_python(filter_value_from_dict(copy.deepcopy(tool), "strict")) # pyright: ignore[reportUnknownArgumentType] # tools arrive untyped through the bare-dict params
for tool in value
)
if filtered_tool is not None
]

53
litellm/main.py
View file

@ -113,6 +113,11 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
messages_without_pdf_file_parts,
)
from litellm.litellm_core_utils.provider_affinity import add_provider_affinity_header
from litellm.litellm_core_utils.reasoning_effort_capability import (
model_group_from_metadata,
reasoning_effort_needs_allowing,
resolve_chat_reasoning,
)
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
)
@ -144,11 +149,6 @@ from litellm.llms.vertex_ai.common_utils import (
)
from litellm.realtime_api.main import realtime_health_check
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.router_utils.reasoning_effort_capability import (
model_group_from_metadata,
reasoning_effort_needs_allowing,
resolve_chat_reasoning,
)
from litellm.secret_managers.main import get_secret_bool, get_secret_str
from litellm.text_completions.streaming_iterator import TextCompletionStreamWrapper
from litellm.types.completion import (
@ -6809,6 +6809,23 @@ async def aembedding(*args, **kwargs) -> EmbeddingResponse:
)
def _embedding_headers(param: str, configured: object, model: str, drop_params: bool) -> Mapping[str, object]:
if not configured:
return {}
if isinstance(configured, Mapping):
return dict(cast(Mapping[str, object], configured)) # cast-ok: header names from untyped request JSON
try:
return dict(cast(Iterable[tuple[str, object]], configured)) # cast-ok: name and value pairs from request JSON
except (TypeError, ValueError):
if drop_params:
return {}
raise litellm.UnsupportedParamsError(
message=f"{param} must be a JSON object of header names to values. "
"To drop it from the call, set `litellm.drop_params = True`.",
model=model,
)
# fmt: off
# Overload for when aembedding=True (returns coroutine)
@ -6923,11 +6940,10 @@ def embedding(
azure_ad_token_provider: Final = kwargs.get("azure_ad_token_provider", None)
aembedding: Final[bool | None] = kwargs.get("aembedding", None)
extra_headers: Final = kwargs.get("extra_headers", None)
headers = kwargs.get("headers", None) or extra_headers
if headers is None:
headers = {}
if extra_headers is not None:
headers.update(extra_headers)
headers = {
**_embedding_headers("headers", kwargs.get("headers"), model, drop_unsupported_params),
**_embedding_headers("extra_headers", extra_headers, model, drop_unsupported_params),
}
# Inject proxy auth headers if configured
if litellm.proxy_auth is not None:
try:
@ -7091,7 +7107,7 @@ def embedding(
# set API KEY
api_key = api_key or litellm.api_key or litellm.openai_key or get_secret_str("OPENAI_API_KEY")
if headers is not None and headers != {}:
if headers:
optional_params["extra_headers"] = headers
requested_encoding_format: Final = (
@ -7183,14 +7199,15 @@ def embedding(
custom_llm_provider == "openai_like"
or custom_llm_provider == "llamafile"
or custom_llm_provider == "lm_studio"
or JSONProviderRegistry.supports_embeddings(custom_llm_provider)
):
api_base = api_base or litellm.api_base or get_secret_str("OPENAI_LIKE_API_BASE")
# set API KEY
if api_key is None:
api_key = api_key or litellm.api_key or litellm.openai_like_key or get_secret_str("OPENAI_LIKE_API_KEY")
if api_key is None and not JSONProviderRegistry.supports_embeddings(custom_llm_provider):
api_key = litellm.api_key or litellm.openai_like_key or get_secret_str("OPENAI_LIKE_API_KEY")
if headers is not None and headers != {}:
if headers:
optional_params["extra_headers"] = headers
## EMBEDDING CALL
@ -7205,10 +7222,9 @@ def embedding(
optional_params=optional_params,
client=client,
aembedding=aembedding,
drop_params=drop_unsupported_params,
)
elif custom_llm_provider == "oci":
if headers is None:
headers = {}
response = base_llm_http_handler.embedding(
model=model,
input=input,
@ -7233,11 +7249,6 @@ def embedding(
or litellm.api_key
)
# Use the merged headers variable (already merged at the top of the function)
# Don't overwrite it with just extra_headers
if headers is None:
headers = {}
response = base_llm_http_handler.embedding(
model=model,
input=input,

162
litellm/model_prices_and_context_window_backup.json
View file

@ -23078,6 +23078,24 @@
"output_cost_per_token_batches": 1e-06,
"source": "https://developers.openai.com/api/docs/pricing"
},
"deepgram/aura": {
"input_cost_per_character": 1.5e-05,
"litellm_provider": "deepgram",
"metadata": {
"calculation": "$0.0150/1000 characters = $0.000015 per character",
"note": "Aura-1 text-to-speech voices (aura-<voice>-<lang>), pay as you go"
},
"source": "https://deepgram.com/pricing"
},
"deepgram/aura-2": {
"input_cost_per_character": 3e-05,
"litellm_provider": "deepgram",
"metadata": {
"calculation": "$0.030/1000 characters = $0.00003 per character",
"note": "Aura-2 text-to-speech voices (aura-2-<voice>-<lang>), pay as you go"
},
"source": "https://deepgram.com/pricing"
},
"deepgram/base": {
"input_cost_per_second": 0.00020833,
"litellm_provider": "deepgram",
@ -25284,6 +25302,38 @@
"cache_creation_input_token_cost": 1e-06,
"prompt_cache_min_tokens": 2048
},
"eu.anthropic.claude-haiku-4-5-20251001-v1:0": {
"cache_creation_input_token_cost": 1.375e-06,
"cache_creation_input_token_cost_above_1hr": 2.2e-06,
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"deprecation_date": "2026-10-15",
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 5.5e-06,
"source": "https://aws.amazon.com/bedrock/pricing/",
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_native_structured_output": true,
"supports_parallel_tool_use_config": true,
"prompt_cache_min_tokens": 4096,
"input_cost_per_token_batches": 5.5e-07,
"output_cost_per_token_batches": 2.75e-06,
"token_count_model": {
"model": "anthropic.claude-haiku-4-5-20251001-v1:0",
"litellm_provider": "bedrock"
}
},
"eu.anthropic.claude-3-5-sonnet-20240620-v1:0": {
"input_cost_per_token": 3e-06,
"litellm_provider": "bedrock",
@ -29561,6 +29611,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -30167,6 +30218,7 @@
"text"
],
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_pdf_input": true,
@ -30210,6 +30262,7 @@
"text"
],
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -30274,6 +30327,7 @@
"text"
],
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -30334,6 +30388,7 @@
"text"
],
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -30401,6 +30456,7 @@
"text"
],
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
@ -31218,6 +31274,9 @@
"output_cost_per_image_token": 3e-05,
"output_cost_per_reasoning_token": 2.5e-06,
"output_cost_per_token": 2.5e-06,
"output_cost_per_token_batches": 1.25e-06,
"output_cost_per_token_flex": 1.25e-06,
"output_cost_per_token_priority": 4.5e-06,
"rpm": 100000,
"source": "https://ai.google.dev/gemini-api/docs/pricing",
"supported_endpoints": [
@ -31398,6 +31457,7 @@
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_image_token_batches": 1.5e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"rpm": 1000,
@ -32405,6 +32465,7 @@
"text"
],
"supports_audio_output": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -32479,6 +32540,7 @@
],
"supports_audio_output": false,
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -32543,6 +32605,7 @@
],
"supports_audio_output": false,
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -32603,6 +32666,7 @@
],
"supports_audio_output": false,
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -32670,6 +32734,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,
@ -39622,6 +39687,57 @@
"/v1/images/generations"
]
},
"hostinger_router/gpt-5.2": {
"cache_read_input_token_cost": 0.00000021,
"input_cost_per_token": 0.0000021,
"litellm_provider": "hostinger_router",
"max_input_tokens": 272000,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 0.0000168,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_response_schema": true,
"supports_vision": true,
"supported_endpoints": ["/v1/chat/completions", "/v1/responses"],
"source": "https://router.hostinger.com/v1/models"
},
"hostinger_router/claude-sonnet-5": {
"cache_creation_input_token_cost": 3.3e-06,
"cache_creation_input_token_cost_above_1hr": 5.28e-06,
"cache_read_input_token_cost": 2.64e-07,
"input_cost_per_token": 2.64e-06,
"litellm_provider": "hostinger_router",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.32e-05,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": true,
"supports_response_schema": true,
"supported_endpoints": ["/v1/chat/completions", "/v1/messages"],
"source": "https://router.hostinger.com/v1/models"
},
"hostinger_router/text-embedding-3-large": {
"input_cost_per_token": 1.716e-07,
"litellm_provider": "hostinger_router",
"max_input_tokens": 8191,
"mode": "embedding",
"output_cost_per_token": 0.0,
"source": "https://router.hostinger.com/v1/models"
},
"hostinger_router/text-embedding-3-small": {
"input_cost_per_token": 3e-08,
"litellm_provider": "hostinger_router",
"max_input_tokens": 8191,
"mode": "embedding",
"output_cost_per_token": 0.0,
"source": "https://router.hostinger.com/v1/models"
},
"hyperbolic/NousResearch/Hermes-3-Llama-3.1-70B": {
"input_cost_per_token": 1.2e-07,
"litellm_provider": "hyperbolic",
@ -47624,6 +47740,24 @@
"output_cost_per_token": 0.0,
"source": "https://cohere.com/pricing"
},
"rerank-v5.0-fast": {
"input_cost_per_token": 2e-08,
"litellm_provider": "cohere",
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"mode": "rerank",
"output_cost_per_token": 0.0,
"source": "https://cohere.com/blog/rerank-5"
},
"rerank-v5.0-pro": {
"input_cost_per_token": 5e-08,
"litellm_provider": "cohere",
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"mode": "rerank",
"output_cost_per_token": 0.0,
"source": "https://cohere.com/blog/rerank-5"
},
"nvidia_nim/nvidia/nv-rerankqa-mistral-4b-v3": {
"input_cost_per_query": 0.0,
"input_cost_per_token": 0.0,
@ -51304,6 +51438,11 @@
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"mode": "chat",
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"output_cost_per_token": 5e-06,
"output_cost_per_token_batches": 2.5e-06,
"regional_endpoint_uplift_multiplier": 1.1,
@ -51332,6 +51471,11 @@
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"mode": "chat",
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"output_cost_per_token": 5e-06,
"output_cost_per_token_batches": 2.5e-06,
"regional_endpoint_uplift_multiplier": 1.1,
@ -52078,6 +52222,11 @@
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"mode": "chat",
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_batches": 7.5e-06,
"regional_endpoint_uplift_multiplier": 1.1,
@ -52183,6 +52332,11 @@
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"mode": "chat",
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_batches": 7.5e-06,
"regional_endpoint_uplift_multiplier": 1.1,
@ -52713,6 +52867,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -59714,12 +59869,13 @@
"cache_read_input_token_cost": 1.375e-06,
"output_cost_per_token": 8.25e-05,
"litellm_provider": "bedrock_mantle",
"max_input_tokens": 1050000,
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"mode": "responses",
"source": "https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-56-cyber.html",
"use_openai_responses_path": true,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
],
"supported_modalities": [
@ -75398,6 +75554,7 @@
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_image_token_batches": 1.5e-05,
"output_cost_per_token": 7.5e-06,
"output_cost_per_token_batches": 3.75e-06,
"rpm": 1000.0,
@ -76498,6 +76655,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -76543,6 +76701,7 @@
],
"supports_audio_input": true,
"supports_audio_output": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
@ -76654,6 +76813,7 @@
"text"
],
"supports_audio_input": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_minimal_reasoning_effort": false,
"supports_native_streaming": true,

24
litellm/models/deployment_warning.py Normal file
View file

@ -0,0 +1,24 @@
from collections.abc import Mapping
from typing import Literal, Protocol
from pydantic import AwareDatetime, ConfigDict, Field
from litellm.types.llms.base import LiteLLMBaseModel
class AdminIgnore(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
source: Literal["admin"] = "admin"
ignored_by: str
ignored_at: AwareDatetime
ignored_until: AwareDatetime | None = Field(
default=None,
description="When the snooze ends. Null means the warning is ignored until an admin restores it.",
)
class DeploymentWarningIgnoreStore(Protocol):
async def get_ignores(self) -> Mapping[str, AdminIgnore]: ...
async def save_ignores(self, ignores: Mapping[str, AdminIgnore]) -> None: ...

31
litellm/models/gateway_activity.py Normal file
View file

@ -0,0 +1,31 @@
from typing import Literal, Protocol, TypeAlias
from pydantic import ConfigDict
from litellm.types.llms.base import LiteLLMBaseModel
class GatewayRouteActivity(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
kind: Literal["route"]
date: str
category: str
route: str
successful_requests: int
failed_requests: int
class GatewayStatusCodeActivity(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
kind: Literal["status"]
status_code: int
failed_requests: int
GatewayActivity: TypeAlias = GatewayRouteActivity | GatewayStatusCodeActivity
class GatewayActivityStore(Protocol):
async def get_activity(self, start_date: str, end_date: str) -> tuple[GatewayActivity, ...]: ...

48
litellm/models/user_banner.py Normal file
View file

@ -0,0 +1,48 @@
from typing import Final, Literal, Protocol
from pydantic import ConfigDict, Field, model_validator
from litellm.types.llms.base import LiteLLMBaseModel
USER_BANNER_MAX_MESSAGE_LENGTH: Final = 4000
UserBannerSeverity = Literal["info", "warning", "error"]
class UserBannerUpdate(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
enabled: bool = Field(
default=False,
description="If true, the banner is shown to all authenticated dashboard users.",
)
message: str = Field(
default="",
max_length=USER_BANNER_MAX_MESSAGE_LENGTH,
description="Banner text shown to dashboard users. Markdown is supported.",
)
severity: UserBannerSeverity = Field(
default="info",
description="Visual style of the banner.",
)
@model_validator(mode="after")
def _require_message_when_enabled(self) -> "UserBannerUpdate":
if self.enabled and not self.message.strip():
raise ValueError("message must be non-empty when the banner is enabled")
return self
class UserBanner(UserBannerUpdate):
revision: str = Field(
default="",
description=(
"Server-stamped opaque publish identity; a fresh value is generated on every "
"update so clients re-surface dismissed banners on republish."
),
)
class UserBannerStore(Protocol):
async def get_banner(self) -> UserBanner: ...
async def save_banner(self, banner: UserBanner) -> None: ...

26
litellm/provider_endpoints_support_backup.json
View file

@ -1087,6 +1087,32 @@
"generateContent": false
}
},
"hostinger_router": {
"display_name": "Hostinger Router (`hostinger_router`)",
"url": "https://docs.litellm.ai/docs/providers/hostinger_router",
"endpoints": {
"chat_completions": true,
"responses": true,
"text_completion": true,
"embeddings": true,
"image_generations": false,
"image_edits": false,
"audio_transcriptions": false,
"audio_speech": false,
"rerank": false,
"ocr": false,
"search": false,
"moderations": false,
"realtime": false,
"video_generations": false,
"batches": false,
"messages": true,
"interactions": true,
"systemone": false,
"a2a": true,
"generateContent": true
}
},
"crusoe": {
"display_name": "Crusoe (`crusoe`)",
"url": "https://docs.litellm.ai/docs/providers/crusoe",

14
litellm/proxy/_lazy_openapi_snapshot.json
View file

@ -58015,7 +58015,7 @@
"paths": {
"/toolset/{toolset_name}/mcp": {
"delete": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_delete",
"parameters": [
{
@ -58054,7 +58054,7 @@
]
},
"get": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_get",
"parameters": [
{
@ -58093,7 +58093,7 @@
]
},
"head": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_head",
"parameters": [
{
@ -58132,7 +58132,7 @@
]
},
"options": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_options",
"parameters": [
{
@ -58171,7 +58171,7 @@
]
},
"patch": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_patch",
"parameters": [
{
@ -58210,7 +58210,7 @@
]
},
"post": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_post",
"parameters": [
{
@ -58249,7 +58249,7 @@
]
},
"put": {
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: non-admin API keys must have the toolset\nlisted in their object_permission.mcp_toolsets grant list, or the request\nwill be rejected with a 403.",
"description": "Namespace a toolset as its own MCP endpoint.\n\nConnecting to /toolset/<name>/mcp exposes exactly the tools defined in\nthe toolset. Access is enforced: API keys other than an admin's key with no\nteam must hold the toolset through one of their grant sources, or the\nrequest will be rejected with a 403.",
"operationId": "toolset_mcp_route_toolset__toolset_name__mcp_put",
"parameters": [
{

3
litellm/proxy/_types.py
View file

@ -124,6 +124,7 @@ from .types_utils.utils import get_instance_fn, validate_custom_validate_return_
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
from litellm.repositories.database import Database
from litellm.tracing import TraceReceiver
Span = _Span | Any
@ -133,6 +134,7 @@ else:
class ProxyLifespanState(TypedDict):
tracing_receiver: ReadOnly["TraceReceiver | None"]
database: ReadOnly["Database | None"]
class ReconcileOutcome(NamedTuple):
@ -3758,6 +3760,7 @@ class SpendLogsMetadata(TypedDict):
kubernetes_pod_routing: ReadOnly[StandardLoggingKubernetesPodRouting | None]
internal_call_origin: InternalCallOrigin | None
billed_retry_attempt: ReadOnly[NotRequired[bool | None]]
upstream_provider: ReadOnly[NotRequired[str | None]]
litellm_roi_estimator: ReadOnly[NotRequired[bool | None]]
is_silent_experiment: ReadOnly[NotRequired[bool | None]]
guardrail_information: list[StandardLoggingGuardrailInformation] | None

3
litellm/proxy/agent_endpoints/a2a_endpoints.py
View file

@ -431,6 +431,7 @@ async def _handle_stream_message(
proxy_server_request: dict[str, object] | None = None,
*,
agent_extra_headers: dict[str, str] | None = None,
agent_name: str | None = None,
user_api_key_dict: UserAPIKeyAuth | None = None,
request_data: dict[str, object] | None = None,
proxy_logging_obj: ProxyLogging | None = None,
@ -504,6 +505,7 @@ async def _handle_stream_message(
metadata=metadata,
proxy_server_request=proxy_server_request,
agent_extra_headers=agent_extra_headers,
agent_name=agent_name,
)
if (
@ -932,6 +934,7 @@ async def invoke_agent_a2a(
metadata=data.get("metadata", {}),
proxy_server_request=data.get("proxy_server_request"),
agent_extra_headers=agent_extra_headers,
agent_name=str(agent_name),
user_api_key_dict=user_api_key_auth_for_call,
request_data=data,
proxy_logging_obj=proxy_logging_obj,

118
litellm/proxy/auth/auth_checks.py
View file

@ -167,6 +167,7 @@ from litellm.repositories.table_repositories import (
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.router import Router
from litellm.router_api import RegistryReader
from litellm.types.proxy.auth.auth_checks import UserNotFoundError
from litellm.types.proxy.model_access_group_budget import ModelAccessGroupBudget
from litellm.types.router import DeploymentTypedDict
@ -525,7 +526,7 @@ def _log_budget_lookup_failure(entity: str, error: Exception) -> None:
)
def _get_router_zero_cost_cache(llm_router: Router) -> dict[str, bool] | None:
def _get_router_zero_cost_cache(llm_router: RegistryReader) -> dict[str, bool] | None:
"""
Return the router's per-instance zero-cost cache, or ``None`` for objects
that don't expose one (e.g. ``MagicMock`` stand-ins in unit tests).
@ -541,7 +542,7 @@ def _get_router_zero_cost_cache(llm_router: Router) -> dict[str, bool] | None:
return cache if isinstance(cache, dict) else None
def is_model_cost_zero(model: str | list[str] | None, llm_router: Router | None) -> bool:
def is_model_cost_zero(model: str | list[str] | None, llm_router: RegistryReader | None) -> bool:
"""
Check if a model has zero cost (no configured pricing).
@ -667,7 +668,7 @@ _NO_MODEL_INFO: Final[Mapping[str, object]] = MappingProxyType({})
_TEAM_GRANT_RELATIONS: Final[Mapping[str, object]] = MappingProxyType({"litellm_model_table": True})
def _deployments_served_for(model: str, llm_router: "Router") -> Sequence[DeploymentTypedDict]:
def _deployments_served_for(model: str, llm_router: RegistryReader) -> Sequence[DeploymentTypedDict]:
"""
The deployments a request to ``model`` routes to, for the gates that waive budget checks.
@ -937,53 +938,58 @@ _GUARDRAIL_MODIFICATION_KEYS: Final[tuple] = (
)
def _guardrail_modification_check(request_body: Mapping[str, object], team_object: LiteLLM_TeamTable | None) -> None:
def requests_guardrail_modification(request_body: Mapping[str, object]) -> bool:
"""Whether a request body sets the plural ``guardrails`` list or a per-request toggle that influences whether
default-on guardrails run (``disable_global_guardrails``, ``disable_global_guardrail`` singular, and
``opted_out_global_guardrails``), at the top level or in either metadata key"""
return (
_requests_guardrail_modification_in(request_body.get("metadata"))
or _requests_guardrail_modification_in(request_body.get("litellm_metadata"))
or _requests_guardrail_modification_in(request_body)
)
def _requests_guardrail_modification_in(container: object) -> bool:
coerced: Final = _guardrail_container_as_dict(container)
if coerced is None:
return False
return any(key in coerced for key in _GUARDRAIL_MODIFICATION_KEYS)
def _guardrail_container_as_dict(container: object) -> Mapping[str, object] | None:
"""Accept dict or JSON-string (from multipart/form-data or extra_body).
Without this, an attacker can smuggle guardrail keys past the check by
sending ``{"metadata": "{\\"disable_global_guardrails\\": true}"}`` —
``isinstance(dict)`` on the string returns False, the check returns
no-modification, and ``add_litellm_data_to_request`` parses the string
to a dict downstream.
"""
if isinstance(container, Mapping):
return container
if isinstance(container, str):
parsed: Final = _safe_json_loads_obj(container)
return parsed if isinstance(parsed, dict) else None
return None
def guardrail_modification_check(request_body: Mapping[str, object], team_object: LiteLLM_TeamTable | None) -> None:
"""
Reject user-supplied metadata flags that would modify guardrail behavior
unless the team has explicit permission. Checked keys include the plural
``guardrails`` list plus the per-request toggles that influence whether
default-on guardrails run (``disable_global_guardrails``,
``disable_global_guardrail`` singular, and ``opted_out_global_guardrails``).
unless the team has explicit permission.
User-supplied values for the bypass toggles are also silently ignored by
``_get_admin_metadata`` at read time; this check adds defense in depth by
failing loudly at the auth layer so operators see an explicit 403 instead
of a confusing silent-ignore.
"""
if requests_guardrail_modification(request_body):
require_guardrail_modification_permission(team_object)
def require_guardrail_modification_permission(team_object: LiteLLM_TeamTable | None) -> None:
from litellm.proxy.guardrails.guardrail_helpers import can_modify_guardrails
def _coerce_to_dict(container: object) -> dict | None:
"""Accept dict or JSON-string (from multipart/form-data or extra_body).
Without this, an attacker can smuggle guardrail keys past the check by
sending ``{"metadata": "{\\"disable_global_guardrails\\": true}"}`` —
``isinstance(dict)`` on the string returns False, the check returns
no-modification, and ``add_litellm_data_to_request`` parses the string
to a dict downstream.
"""
if isinstance(container, dict):
return container
if isinstance(container, str):
parsed: Final = _safe_json_loads_obj(container)
return parsed if isinstance(parsed, dict) else None
return None
def _user_requested_modification(container: object) -> bool:
coerced: Final = _coerce_to_dict(container)
if coerced is None:
return False
return any(key in coerced for key in _GUARDRAIL_MODIFICATION_KEYS)
# Check both metadata keys — callers can populate either depending on the
# endpoint. Cover the top-level too so root-level injection is rejected.
modifies: Final = (
_user_requested_modification(request_body.get("metadata"))
or _user_requested_modification(request_body.get("litellm_metadata"))
or _user_requested_modification(request_body)
)
if not modifies:
return
if not can_modify_guardrails(team_object):
raise HTTPException(
status_code=403,
@ -1485,7 +1491,7 @@ async def common_checks(
_enforce_user_param_check(general_settings, request, request_body, route)
_global_proxy_budget_check(global_proxy_spend, skip_all_budget_checks, route)
_guardrail_modification_check(_typed_request_body(request_body), team_object)
guardrail_modification_check(_typed_request_body(request_body), team_object)
# 10 [OPTIONAL] Organization RBAC checks
organization_role_based_access_check(user_object=user_object, route=route, request_body=request_body)
@ -4837,7 +4843,7 @@ async def get_skill_names_from_access_groups(
def _resolve_all_team_model_sentinel_for_auth_check(
models: list[str],
llm_router: Router | None,
llm_router: RegistryReader | None,
team_id: str | None,
) -> list[str]:
if SpecialModelNames.all_team_models.value not in models or team_id is None or llm_router is None:
@ -4851,7 +4857,7 @@ def _resolve_all_team_model_sentinel_for_auth_check(
def check_model_access_helper(
model: str,
llm_router: Router | None,
llm_router: RegistryReader | None,
models: list[str],
team_model_aliases: dict[str, str] | None = None,
team_id: str | None = None,
@ -4902,7 +4908,7 @@ _check_model_access_helper: Final = check_model_access_helper
def _owning_team_names_for_team_deployment(
model: str,
llm_router: Router | None,
llm_router: RegistryReader | None,
team_id: str | None,
team_model_aliases: Mapping[str, str] | None,
) -> tuple[str, ...]:
@ -4920,7 +4926,7 @@ def _owning_team_names_for_team_deployment(
def _can_object_call_model(
model: str | list[str],
llm_router: Router | None,
llm_router: RegistryReader | None,
models: list[str],
team_model_aliases: dict[str, str] | None = None,
team_id: str | None = None,
@ -5047,7 +5053,7 @@ def _resolve_team_alias(
def _live_team_alias_target(
model: str, team_model_aliases: Mapping[str, str], team_id: str | None, llm_router: Router | None
model: str, team_model_aliases: Mapping[str, str], team_id: str | None, llm_router: RegistryReader | None
) -> str:
target: Final = team_model_aliases.get(model)
if target is None:
@ -5534,7 +5540,7 @@ async def can_key_call_model(
model: str | list[str],
llm_model_list: Sequence[object] | None,
valid_token: UserAPIKeyAuth,
llm_router: litellm.Router | None,
llm_router: RegistryReader | None,
prisma_client: DatabaseClient | None = None,
) -> Literal[True]:
"""
@ -5613,7 +5619,7 @@ async def can_key_call_resolved_model(
model: str,
llm_model_list: Sequence[object] | None,
valid_token: UserAPIKeyAuth,
llm_router: litellm.Router | None,
llm_router: RegistryReader | None,
) -> None:
from litellm.proxy.proxy_server import (
prisma_client,
@ -5761,7 +5767,7 @@ def can_org_access_model(
async def can_team_access_model(
model: str | list[str],
team_object: LiteLLM_TeamTable | None,
llm_router: Router | None,
llm_router: RegistryReader | None,
team_model_aliases: dict[str, str] | None = None,
key_model_aliases: Mapping[str, str] | None = None,
prisma_client: DatabaseClient | None = None,
@ -5864,7 +5870,7 @@ async def key_access_group_grants_model(
model: str | list[str],
valid_token: UserAPIKeyAuth | None,
team_object: LiteLLM_TeamTable | None,
llm_router: Router | None,
llm_router: RegistryReader | None,
key_model_aliases: Mapping[str, str] | None = None,
) -> bool:
"""
@ -5900,7 +5906,7 @@ _key_access_group_grants_model: Final = key_access_group_grants_model
def can_project_access_model(
model: str | list[str],
project_object: LiteLLM_ProjectTable,
llm_router: Router | None,
llm_router: RegistryReader | None,
key_model_aliases: Mapping[str, str] | None = None,
) -> Literal[True]:
"""
@ -5921,7 +5927,7 @@ def can_project_access_model(
def can_customer_access_model(
model: str | list[str],
end_user_object: LiteLLM_EndUserTable,
llm_router: Router | None,
llm_router: RegistryReader | None,
valid_token: UserAPIKeyAuth | None,
) -> Literal[True]:
team_model_aliases: Final = team_model_aliases_for_auth_check(valid_token) if valid_token is not None else None
@ -6084,7 +6090,7 @@ async def _team_grants_personal_key_model(
model: str,
team_object: LiteLLM_TeamTableCachedObj,
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,
@ -6122,7 +6128,7 @@ async def _loaded_teams_allow_model(
model: str,
teams: tuple[LiteLLM_TeamTableCachedObj | None, ...],
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,
@ -6388,7 +6394,7 @@ async def can_personal_key_call_model_via_teams(
model: str | Sequence[str],
user_object: LiteLLM_UserTable | None,
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,
@ -7180,7 +7186,7 @@ async def check_team_member_model_access(
model: str | list[str],
team_object: LiteLLM_TeamTable,
valid_token: UserAPIKeyAuth,
llm_router: Router | None,
llm_router: RegistryReader | None,
prisma_client: Optional["PrismaClient"],
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging,

10
litellm/proxy/auth/fallback_budget.py
View file

@ -44,7 +44,7 @@ from litellm.proxy.auth.auth_checks import ( # noqa: F401 # legacy module expo
_is_model_cost_zero, # pyright: ignore[reportPrivateUsage,reportUnusedImport] # backwards-compatible package export
is_model_cost_zero, # pyright: ignore[reportPrivateUsage] # the zero-cost predicate the auth-time budget checks use; no public equivalent
)
from litellm.router import Router
from litellm.router_api import RegistryReader, RoutingContext
from litellm.types.llms.base import LiteLLMBaseModel
@ -118,7 +118,7 @@ async def is_token_within_budget_for_model(
*,
model: str,
valid_token: UserAPIKeyAuth,
llm_router: Router,
llm_router: RegistryReader,
team_member_model_budget_limiter: _TeamMemberModelBudgetLimiter | None = None,
) -> bool:
"""
@ -187,19 +187,19 @@ class RouterFallbackBudgetCheck:
is_enforced: Callable[[], bool]
team_member_model_budget_limiter: _TeamMemberModelBudgetLimiter | None = None
async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], llm_router: Router) -> bool:
async def __call__(self, *, model: str, request_kwargs: Mapping[str, object], ctx: RoutingContext) -> bool:
if not self.is_enforced():
return True
valid_token: Final = _user_api_key_auth_from_request(request_kwargs)
if valid_token is None:
return True
budget_models: Final = tuple(dict.fromkeys((model, llm_router.registry().resolve(model))))
budget_models: Final = tuple(dict.fromkeys((model, ctx.registry().resolve(model))))
for budget_model in budget_models:
try:
if not await is_token_within_budget_for_model(
model=budget_model,
valid_token=valid_token,
llm_router=llm_router,
llm_router=ctx,
team_member_model_budget_limiter=self.team_member_model_budget_limiter,
):
return False

Some files were not shown because too many files have changed in this diff Show more