scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-12 08:42:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 455

chore: merge main and move the OpenRouter discovery code into the router core
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Details
LiteLLM Rust DB / db-queries (push) Has been cancelled
Details
LiteLLM Rust / rust-lint (push) Has been cancelled
Details
LiteLLM Rust / rust-test (push) Has been cancelled
Details
LiteLLM Rust / rust-wheel (push) Has been cancelled
Details

Browse source
This commit is contained in:
mateo-berri 2026-11-01 09:07:36 -08:00
commit 0d27e0ef74
662 changed files with 68255 additions and 25603 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

77
.agents/skills/database-access/SKILL.md
View file

@ -1,54 +1,73 @@
---
name: database-access
description: Add database queries, extract existing Prisma calls into repositories, or explicitly port queries to Rust and SQLx without expanding migration scope
description: Organize database repositories and proxy dependency wiring, manage connection ownership, or port Prisma operations to Rust and SQLx without expanding migration scope
metadata:
internal: true
---
# Database access
## Situations
- Adding a query or an operation to a repository, read [adding a query](references/python-repository.md#adding-a-query)
- Moving existing Prisma calls behind a repository, read [moving existing access](references/python-repository.md#moving-existing-access-into-a-repository)
- Porting a repository from Prisma to Rust and SQLx, read [porting to SQLx](references/python-repository.md#porting-a-repository-to-sqlx)
- Injecting a repository into a route, read [Python and FastAPI](references/python-fastapi.md) or [Rust and Axum](references/rust-axum.md)
## Adding new database access
- Put Python database access behind an operation in `litellm/repositories`
- Give callers typed inputs and outputs, not Prisma clients, models, or query dictionaries
- Keep query execution and backend selection inside the repository
- Give each repository its own FastAPI provider, so a route builds only the repositories it uses
- A provider borrows existing handles and performs no database I/O during construction
- Define the operations a route calls as a Protocol in `litellm/models/<feature>.py`, next to its domain types
- The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
- Parsing and serialization of stored values live in the repository
- Add a Protocol when a repository gets a provider, not up front for every repository
- Keep query execution inside the repository, with one implementation per repository and no backend switch or fallback
- A repository is either all Prisma or all native, never a mix of both
- Prisma is fine for the implementation, adding a query does not require a Rust port
- Native Rust callers use `litellm-rust/crates/db` directly
## Asked to move existing access behind a repository
## Framework wiring
- Identify the calls being moved and their transaction boundary
- Move the existing Prisma implementation into the relevant repository
- Replace those direct database calls with repository operations
- Preserve return values, errors, and transaction behavior, with focused regression tests
- Keep this an extraction, not a backend migration or unrelated cleanup
- Python routes receive a per-repository dependency such as `UserBannerRepositoryDep` and call its operations, such as `banners.get_banner()`
- `DatabaseDep` lives in `litellm/proxy/deps/database.py`, repository providers in `litellm/proxy/deps/repositories.py`
- A provider returns the feature's Protocol, so routes never see which backend implements it
- Dependencies used by one feature only, like the banner audit binding, stay next to that feature's routes
- Repositories never import FastAPI, so jobs and helpers can use them without a request
- Use named FastAPI providers with `Annotated[..., Depends(...)]`, ordinary helpers and jobs receive explicit arguments
- Preserve each route's no-database behavior, authorization precedence and background-task scheduling
- Read [Python and FastAPI](references/python-fastapi.md) for provider examples, lifespan state and route verification
- Read [Rust and Axum](references/rust-axum.md) when adding database access to Rust handlers
## An equivalent Rust or SQLx query already exists
## Connection ownership
- Leave it and the current backend selection unchanged unless integration is explicitly requested
- Its existence alone does not justify switching a Prisma caller to Rust
- Repository extraction can keep using Prisma even when the Rust query is available
- Keep connection setup and shutdown separate from repositories, which own no pools
- Reuse existing handles, never connect per request or disconnect from a repository operation
- Give each pool one owner for setup and shutdown, borrowed handles never close it
- Finish requests, jobs and queued database work before closing connections
- Clean up resources opened before a startup failure, without a second disconnect fallback
- Add a backend to the existing lifecycle owner and pass its handle only to repositories that need it
- Preserve cancellation, timeout, error and transaction semantics across the native bridge
## Asked to port a Prisma query to Rust or SQLx
## Changing ownership or reloading resources
- Identify the existing inputs, outputs, errors, and transaction boundary
- Check `litellm-rust/crates/db` for an equivalent query and reuse it if its behavior matches
- Otherwise follow `litellm-rust/crates/db/AGENTS.md`
- Put static SQL in `queries/` and compile-check it with SQLx
- Keep typed inputs and reusable row projections in `src/`
- Refresh SQLx query metadata after SQL changes
- Keep Python conversion in the Python bridge, not in `litellm-rust/crates/db`
- When caller integration is requested, expose the Rust operation through the bridge and call it from the repository
- A query-only port can land without switching callers
- Verify behavior against real PostgreSQL before switching callers
- Check relevant null handling, conflicts, ordering, errors, and transaction semantics
- Remove the old implementation only when no callers need it
- Inventory initialization, reload, tests, background jobs and shutdown together
- Move every writer when changing authoritative storage, copying a replaceable global into state is insufficient
- A frozen repository does not make its clients immutable
- Decide whether each request captures one resource generation or reads it again during processing
- Build and validate replacements before publishing them, preserving the old generation on failure
- Serialize reloads and coordinate publication with shutdown
- Drain streams and background users before retiring old resources, retaining every generation grows resource use
- Define multi-worker coordination separately from an in-process swap
- Choose mechanisms such as `ArcSwap` only after defining publication, capture and retirement
## Migration context and boundaries
- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations incrementally with Rust APIs
- Mixed Prisma and Rust implementations are expected during the transition, unrelated existing calls can stay as they are
- Both backends may have separate pools through PgBouncer, size them together and verify pooling-mode compatibility
- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations with Rust APIs one repository at a time
- Prisma and native repositories coexist during the transition, unrelated existing calls can stay as they are
- Prisma and SQLx keep separate client pools to PgBouncer, which pools connections to PostgreSQL
- Size both layers together and preserve reader/writer routing and pooling-mode compatibility
- PgBouncer does not share transactions between Prisma and SQLx, keep each atomic unit of work on one backend and transaction
- Never retry a failed Rust write through Prisma when it may already have committed
- Prisma owns schema migrations today, eventual SQLx migration ownership and Prisma removal are separate planned steps, not part of a query port

101
.agents/skills/database-access/references/python-fastapi.md Normal file
View file

@ -0,0 +1,101 @@
# Python and FastAPI
## Existing pattern
- `UserBannerStore` in `litellm/models/user_banner.py` is the Protocol the banner routes call
- `get_database` in `litellm/proxy/deps/database.py` reads the lifespan-owned handle from request state
- `get_user_banner_repository` in `litellm/proxy/deps/repositories.py` builds the banner repository from that handle
- Choose each repository's replica routing once in its provider, never at call sites
- A handler receives `UserBannerStore | None`, never a Prisma client or a concrete repository
- `UserBannerRepository` implements `UserBannerStore` structurally, keeping the existing Prisma table, serialization and configuration-sync behavior
- The banner audit provider binds the legacy audit callable at request time, without performing audit I/O
- A native repository can later implement the same Protocol without changing handler signatures, see [porting a repository to SQLx](python-repository.md#porting-a-repository-to-sqlx)
## Dependency injection
- Add one provider and one `Annotated` alias per repository in `litellm/proxy/deps/repositories.py`
- Annotate the provider and alias with the feature's Protocol, not the concrete repository
- A route depends only on the repositories it calls, so a request never builds repositories it does not use
- Do not build repositories once at lifespan, a reconnect can replace the Prisma client they wrap
```python
class UserBannerStore(Protocol):
async def get_banner(self) -> UserBanner: ...
async def save_banner(self, banner: UserBanner) -> None: ...
```
```python
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
return None if database is None else UserBannerRepository(database.prisma)
UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)]
```
- FastAPI caches each provider within a request, so routes and sub-dependencies share one instance
- Ordinary helpers and jobs receive explicit arguments, never unresolved `Depends` defaults
- A route still reading the `prisma_client` global or building a repository inline switches to the repository's `...RepositoryDep`, keeping its no-database behavior and error precedence
```python
async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner:
if banners is None:
return UserBanner()
return await banners.get_banner()
```
## Lifespan state
- Prisma startup owns connection setup, authentication refresh, health checks and migrations
- `Database` holds the initialized Prisma client and the `NativeDatabase` handle, without holding fields per feature
- The native handle opens no pool yet, its `connect` and `close` are noops
- The proxy lifespan builds the handle with `Database.open`, registers `Database.aclose` on its exit stack and publishes it in request state
- Callers only open and close `Database`, which closes Prisma and the native handle together, and `open` closes Prisma itself when the native connect fails
- Starlette shallow-copies lifespan state into request state, which is distinct from `app.state`
- The database module never imports a repository, legacy globals borrow its handle without owning shutdown
- Drain requests, jobs, logging, spend and audit work before database teardown
- The exit stack closes the handle on exit, including startup failures after the client is created
- When adding native resources, initialize them in the serving process, respecting runtime fork restrictions
- Await native shutdown instead of depending on Python garbage collection
- Apply the [connection ownership rules](../SKILL.md#connection-ownership) when changing the owner
## Timing and errors
- FastAPI resolves dependencies before entering the handler
- Handler exception handling cannot catch failures raised during dependency resolution
- A universal required-database dependency can override authorization and validation behavior
- Banner GET without a database returns the default disabled banner
- Banner PATCH checks the admin role before returning the existing missing-database error
- Invalid stored banner data falls back to the default, but storage failures still propagate
- Banner writes finish before the success audit is queued as a FastAPI background task
- Queue post-response work with `BackgroundTasks`, not bare `asyncio.create_task`, so the in-flight drain waits for it before the database closes
- Do not move background work into an awaited provider merely to remove an import
## Verification
- Exercise real FastAPI handlers using `dependency_overrides` for external boundaries
- Keep production parsing and persistence code real
- Restore overrides after each test
- Verify complete response bodies, status codes and OpenAPI against the base
- Include combined failure conditions to pin error precedence
- Check saved values, fresh revisions, audit contents and failed-write behavior
- Check lifespan state with a running lifespan, not only a TestClient that skips startup
- Verify providers do not connect, query, disconnect or audit eagerly
- Use real PostgreSQL and live proxy requests when changing persistence behavior
## Scope
- Preserve direct Python call sites or migrate them explicitly
- Inventory imports, state writers, side effects and error ordering before migrating a feature
- Preserve configuration sync and serialization, and state which legacy reads remain
- Do not migrate every route in a large module just to establish the pattern
- Do not invent providers for resources without consumers
## Framework references
- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/) and [Starlette lifespan state](https://starlette.dev/lifespan/)
- [`SessionDep`-style `Annotated` aliases](https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-session-dependency), which `DatabaseDep` and `UserBannerRepositoryDep` follow
- [Dependency composition and caching](https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/)
- [Background tasks and dependencies with `yield`](https://fastapi.tiangolo.com/advanced/advanced-dependencies/#background-tasks-and-dependencies-with-yield-technical-details)
- [Dependency overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
- [Testing lifespan](https://fastapi.tiangolo.com/advanced/testing-events/)

119
.agents/skills/database-access/references/python-repository.md Normal file
View file

@ -0,0 +1,119 @@
# Python repositories
## Adding a query
Use this when a feature needs a read or write that no repository offers yet, including one more operation on an existing repository
### Before starting
- Search `litellm/repositories` for a repository that already owns the table, and extend it instead of adding a second one
- Check `litellm/models/<feature>.py` for an existing Protocol and domain types
- Decide the route's behavior when no database is configured, and where that check sits relative to authorization and validation
### Steps
1. Domain types and Protocol in `litellm/models/<feature>.py`
- The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads
- Add the Protocol together with the provider, not for repositories nothing injects
2. Repository in `litellm/repositories/<feature>_repository.py`
- Implement the operation with Prisma, a Rust port is not required
- Compose the Prisma client and type the tables it touches with `TableActions` from `prisma_protocols.py`, never inherit `PrismaTableRepository`
- Parse and serialize stored values here, so callers only see domain types
- Never import FastAPI, connect, disconnect or choose replicas per call
3. Provider and `Annotated` alias in `litellm/proxy/deps/repositories.py`, see [dependency injection](python-fastapi.md#dependency-injection)
4. The route takes `<Feature>RepositoryDep`, handles `None` with the behavior decided above, and calls the operation
5. Jobs and helpers that need the operation receive the repository as an explicit argument, built by their caller from the handle it already holds
### Done when
- Handler tests drive the real route with `dependency_overrides` and the real repository code, see [verification](python-fastapi.md#verification)
- Tests cover the no-database path, error precedence, and what was actually stored
- Persistence changes are checked against real PostgreSQL through a live proxy
### Out of scope
- Porting the query to Rust, see [porting a repository to SQLx](#porting-a-repository-to-sqlx)
- Moving unrelated direct Prisma calls in the same module
## Moving existing access into a repository
Use this when code calls Prisma directly, through `prisma_client.db.<table>` or a helper, and the calls should live behind a repository. The backend stays Prisma
### Before starting
- List every call being moved, its callers, and the transaction or batch it runs in
- List the module's imports, global state writers, side effects such as audit and config sync, and error ordering
- Record current return values, raised errors and response bodies, these are what the change must preserve
### Steps
1. Move the existing Prisma code into the relevant repository in `litellm/repositories` without rewriting the queries
2. Add domain types and a Protocol in `litellm/models/<feature>.py` when a route will inject the repository
3. Replace each direct call with the repository operation, through a provider for routes and explicit arguments for jobs and helpers, see [Python and FastAPI](python-fastapi.md)
4. Calls sharing a transaction move into the same repository or unit of work in one change
5. State which legacy reads remain and why, when the change does not move all of them
### Done when
- Focused regression tests pin the preserved return values, errors and transaction behavior
- Response bodies, status codes and OpenAPI match the base
- Configuration sync and serialization behave the same, including invalid stored data
### Out of scope
- Switching the backend, even when an equivalent Rust query exists
- Converting other routes in the module just to establish the pattern
- Cleanups unrelated to the moved calls
## Porting a repository to SQLx
Status: TBD. No repository has been ported yet. `database.native` is a `NativeDatabase` from `crates/python-bridge/src/db.rs` whose `connect` and `close` are noops that open no pool, and the PyO3 repository surface does not exist. A real pool takes typed writer/reader settings and a token provider, never the Prisma `DATABASE_URL`, which carries Prisma-only params and rotating IAM tokens. The constraints below are settled, the steps are a sketch to revise after the first port lands
Use this only when a port is explicitly requested. Repository extraction does not require one
### When not to port
- An equivalent Rust or SQLx query existing is not a reason to switch a Prisma caller
- Leave existing backend selection unchanged unless integration is explicitly requested
- Never port part of a repository, it is either all Prisma or all native
### Before starting
- Identify the repository's inputs, outputs, errors and transaction boundary
- List repositories that share a transaction with it, they move together
- Check `litellm-rust/crates/db` for an equivalent query and reuse it when its behavior matches
### Steps
1. Rust queries, following `litellm-rust/crates/db/AGENTS.md`
- Static SQL in `queries/`, compile-checked with SQLx, typed inputs and row projections in `src/`
- Dynamic filters and sort keys become closed Rust enums that select static SQL, every value is bound
- Refresh SQLx query metadata after SQL changes
2. PyO3 surface exposing typed operations, never SQL text, table names or column names from Python. TBD: where it lives and how the handle reaches it
3. Native repository implementing the same Protocol in `litellm/repositories`
- Converts typed PyO3 results to domain types, preserving defaults, invalid-data handling and storage errors
- Python conversion stays in the bridge, not in `crates/db`
4. Switch the provider in `litellm/proxy/deps/repositories.py`, the Protocol and routes stay unchanged
```diff
async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None:
- return None if database is None else UserBannerRepository(database.prisma)
+ return None if database is None else NativeUserBannerRepository(database.native)
```
5. Delete the Prisma repository in the same change and switch every caller, including jobs and helpers
### Done when
- Behavior is verified against real PostgreSQL, covering null handling, conflicts, ordering, errors and transaction semantics
- `DATABASE_URL=... cargo test -p litellm-db --locked -- --include-ignored` passes against a Prisma-migrated database
- Live proxy requests show unchanged responses. TBD: a Python-side real-PostgreSQL test harness
### Rollback
- Roll back the release, never retry an uncertain native write through Prisma
### Out of scope
- SQLx owning schema migrations, and removing Prisma
- Converting unrelated routes to dependency injection

4
.agents/skills/database-access/references/rust-axum.md Normal file
View file

@ -0,0 +1,4 @@
# Rust and Axum
- This should use same `crates/db` what is exposed to Python world.
- Details are TBD.

7
.circleci/AGENTS.md Normal file
View file

@ -0,0 +1,7 @@
# .circleci
The `integration` workflow on `main` runs from a scheduled pipeline in the CircleCI project settings, cron `9 0-8,14-23 * * *` (UTC) on `main` with no pipeline parameters. It has no `triggers:` block in this file because a workflow with one runs only on its schedule, so the API pipelines that set `run_migration_tests` or `routing_parity_base` would stop running it. If main stops getting integration runs, check that the schedule still exists and recreate it with these values
Jobs clone with the `checkout_over_https` command, never the built-in `checkout` step, because CircleCI falls back to the project's SSH checkout key without its GitHub App access, GitHub rejects that key, and the public repo needs no key. It fetches the job's commit with full history and no file contents (`--filter=blob:none`), like the built-in step did, so `scripts/path_filter.sh` can find a merge base
A step that checks out files from another commit fetches that commit with `--no-filter`, as `routing_parity` does, because the clone has no file contents and git would otherwise download each missing file in its own request, which took 210 to 896 seconds for the routing parity base

58
.circleci/config.yml
View file

@ -21,19 +21,30 @@ orbs:
win: circleci/windows@5.0 # Add Windows orb
commands:
checkout_migration_source:
checkout_over_https:
parameters:
revision:
type: string
default: ""
shell:
type: string
default: /bin/bash -eo pipefail
steps:
- run:
name: Select the requested migration test revision
name: Check out the public repository over HTTPS
shell: << parameters.shell >>
environment:
MIGRATION_SOURCE_SHA: << pipeline.parameters.migration_source_sha >>
REQUESTED_REVISION: << parameters.revision >>
command: |
revision="${MIGRATION_SOURCE_SHA:-$CIRCLE_SHA1}"
revision="${REQUESTED_REVISION:-$CIRCLE_SHA1}"
[[ "$revision" =~ ^[0-9a-f]{40}$ ]] || exit 1
git init
git remote add origin https://github.com/BerriAI/litellm.git
git fetch --depth 1 origin "$revision"
git checkout --detach FETCH_HEAD
for attempt in 1 2 3; do
git fetch --filter=blob:none origin "$revision" && git checkout --detach FETCH_HEAD && break
[[ $attempt -lt 3 ]] || exit 1
sleep 5
done
skip_if_unrelated_changes:
parameters:
category:
@ -347,7 +358,7 @@ commands:
- when:
condition: << parameters.checkout >>
steps:
- checkout
- checkout_over_https
- setup_google_dns
- install_uv
- install_rust
@ -377,7 +388,8 @@ jobs:
CARGO_HTTP_MULTIPLEXING: "false"
CARGO_NET_RETRY: "5"
steps:
- checkout
- checkout_over_https:
shell: bash.exe
- run:
name: Skip job when no windows-release-relevant files changed
shell: bash.exe
@ -451,7 +463,7 @@ jobs:
password: ${DOCKERHUB_PASSWORD}
working_directory: ~/project
steps:
- checkout
- checkout_over_https
- setup_google_dns
- install_uv
- install_rust
@ -482,7 +494,7 @@ jobs:
working_directory: ~/project
steps:
- checkout
- checkout_over_https
- skip_if_unrelated_changes
- attach_workspace:
at: ~/project
@ -559,7 +571,7 @@ jobs:
resource_class: medium
working_directory: ~/project
steps:
- checkout
- checkout_over_https
- run:
name: Generate LiteLLM master key
command: |
@ -659,7 +671,7 @@ jobs:
resource_class: large
working_directory: ~/project
steps:
- checkout
- checkout_over_https
- run:
name: Generate LiteLLM master key
command: |
@ -740,7 +752,7 @@ jobs:
resource_class: large
working_directory: ~/project
steps:
- checkout
- checkout_over_https
- skip_if_unrelated_changes:
category: client
- setup_google_dns
@ -794,7 +806,7 @@ jobs:
# the rest of the suite: nothing else performs a logout.
PROXY_LOGOUT_URL: "https://www.example.com"
steps:
- checkout
- checkout_over_https
- run:
name: Generate LiteLLM master key
command: |
@ -940,7 +952,7 @@ jobs:
# and by migration.serverRootPath.config.ts, which refuses to run without it.
SERVER_ROOT_PATH: "/litellm"
steps:
- checkout
- checkout_over_https
- run:
name: Generate LiteLLM master key
command: |
@ -1066,11 +1078,12 @@ jobs:
- when:
condition: << parameters.migration_qualification >>
steps:
- checkout_migration_source
- checkout_over_https:
revision: << pipeline.parameters.migration_source_sha >>
- unless:
condition: << parameters.migration_qualification >>
steps:
- checkout
- checkout_over_https
- skip_if_unrelated_changes
- run:
@ -1122,7 +1135,8 @@ jobs:
MIGRATION_TEST_OUTPUT: /tmp/migration-results
PYTHONPATH: tests/e2e
steps:
- checkout_migration_source
- checkout_over_https:
revision: << pipeline.parameters.migration_source_sha >>
- run:
name: Generate LiteLLM master key
command: |
@ -1197,7 +1211,7 @@ jobs:
resource_class: medium
working_directory: ~/project
steps:
- checkout
- checkout_over_https
- skip_if_unrelated_changes
- attach_workspace:
at: ~/project
@ -1371,7 +1385,7 @@ jobs:
environment:
UV_PYTHON_DOWNLOADS: never
steps:
- checkout
- checkout_over_https
- install_uv
- run:
name: Install the locked shard verifier and coverage dependencies
@ -1419,7 +1433,7 @@ jobs:
ROUTING_PARITY_BASE: << pipeline.parameters.routing_parity_base >>
command: |
[[ "$ROUTING_PARITY_BASE" =~ ^[0-9a-f]{40}$ ]] || exit 1
git fetch --depth 1 origin "$ROUTING_PARITY_BASE"
git fetch --depth 1 --no-filter origin "$ROUTING_PARITY_BASE"
git rm -r -f --quiet litellm enterprise litellm-proxy-extras
git checkout "$ROUTING_PARITY_BASE" -- litellm enterprise litellm-proxy-extras
git reset --quiet
@ -1494,7 +1508,7 @@ jobs:
environment:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/litellm_test
steps:
- checkout
- checkout_over_https
- skip_if_unrelated_changes
- setup_litellm_test_deps:
checkout: false

15
.github/PULL_REQUEST_TEMPLATE/general.md vendored
View file

@ -4,9 +4,20 @@
Drop every section you have nothing to put in, heading included: a bare "## Relevant issues" or
"## Affected release" with nothing under it must not appear in the final description -->
<!-- Two or three plain sentences a teammate can read in 10 seconds, written the way you'd explain it in Slack
## Handwritten 1-3 sentence description
<!-- The PR author rewrites this in their own words before asking for review. Being able to sum up the PR in a few
plain sentences proves the author understands what it does, and human-written technical English is far easier
for a reviewer to parse than AI-written English
An AI agent may draft a starting point here, but always wraps its draft in the make-me-human-readable tags below,
backticks included so the tags stay visible on GitHub, so nobody mistakes it for the author's words. The author rewrites it in their own words and deletes the tags.
Tags still present at review time mean the summary is AI-written and has not been rewritten yet
Say what the PR does for users and why, then anything that keeps it working over time (a sync job, a config someone owns)
No headings, bullets, tables, or code names here. Everything else goes inside the Details block below -->
No bullets, tables, or code names here. Everything else goes inside the Details block below -->
`<make-me-human-readable>`
<blah>
`</make-me-human-readable>`
<details>
<summary>Details</summary>

2
.github/workflows/create-rc-branch.yml vendored
View file

@ -2,7 +2,7 @@ name: Create RC Branch
on:
schedule:
- cron: "0 3 * * 5"
- cron: "0 3 * * 1"
timezone: "America/Los_Angeles"
workflow_dispatch:

1
.github/workflows/test-unit.yml vendored
View file

@ -577,6 +577,7 @@ jobs:
tests/unit/proxy/rerank_endpoints
tests/unit/proxy/realtime_endpoints
tests/unit/proxy/ui_crud_endpoints
tests/unit/proxy/deps
tests/unit/proxy/config_resolvers
tests/unit/proxy/utils
python-version: "3.12"

4
AGENTS.md
View file

@ -41,6 +41,8 @@ Same applies for filing bug reports and feature requests, with .github/ISSUE_TEM
If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just drop the section
Run QA proxies with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`, and swap its `--config` for a copy of `litellm/proxy/dev_config.yaml` that points both `router_settings` (`redis_host`, `redis_port`) and `general_settings.coordination_redis` (`host`, `port`) at one dedicated Redis instance, with credentials when required, so the workers share router state and spend, budget, and rate-limit counters. Check that two worker PIDs are serving, and use the same worker count and Redis setup for the before and after proof. When the PR is about `--run_gunicorn`, `--run_hypercorn`, or `--run_granian`, which `--reload` does not support, drop `--reload`. When it is about behavior without Redis, drop Redis. When the proxy prints `"workers" flag is ignored when reloading is enabled`, that checkout or its uvicorn reloads only one worker, so drop the two-worker check for it. Say in the proof which part you dropped and why
Simple PRs may skip before/after e2e when the full diff is mechanical and focused checks completely prove it, such as #49030 restoring an existing helper name. Trace affected callers and run relevant tests/static checks at the tip. For a bug fix, establish that the regression catches the original defect. Size alone never qualifies. New behavior, API/provider semantics, auth, billing, persistence, concurrency or uncertain integration risk need live e2e. In Screenshots / Proof of Fix, record `E2E exception at <tip SHA>`, scope, rationale, commands and results. Tests count as proof only for this exception. Reassess changed diffs, retain CI/bot gates, and honor explicit e2e requests (2026-10-28, unnecessary live QA for #49030)
For other PRs, proof must show the end user's real flow before and after, with commands and output against a live proxy and real provider calls where applicable. Pytest alone is insufficient. For UI or interactive coding-tool flows, drive that surface and embed before/after screenshots, with URLs, clicks and input fields a reviewer can reproduce
@ -90,6 +92,8 @@ Do not put names of customers or customer company names in code, PR descriptions
CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
For database operations, connection ownership and proxy dependency wiring, follow [database-access](.agents/skills/database-access/SKILL.md)
Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No `UPDATE`, `DELETE` or `MERGE`, and no `INSERT ... SELECT`: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. `tests/code_coverage_tests/check_migrations_no_data_rewrites.py` enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement `-- data-migration-ok: <what bounds it>`
Follow these coding conventions for new/updated code (a three-line fix in a legacy file shouldn't trigger huge drive-by refactors):

2
CONTRIBUTING.md
View file

@ -276,6 +276,8 @@ When the provider reports its own usage or spend numbers, compare the proxy's nu
Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution, and it is where PRs get reworked most often. Follow these rules
**Preserve zero prices.** In [`model_prices_and_context_window.json`](model_prices_and_context_window.json), an explicit `0.0` means that the priced operation is free. A missing field or `null` means no rate is specified, so the relevant calculator may use a fallback. For example, when no batch input rate is configured, batch cache writes use `cache_creation_input_token_cost` if set, otherwise the regular input rate, then apply the batch discount. A zero ordinary input rate does not make separately priced cache reads or writes free. Test `rate is None` before substituting a fallback, since `row.get("cache_creation_input_token_cost") or fallback` would discard a valid zero. Speech keeps character pricing ahead of token pricing even when the character rate is zero. If neither rate is configured, speech cost calculation raises an error. Its model lookup preserves unset rates instead of treating compatibility defaults as explicit zero prices. Perplexity keeps a configured legacy search rate ahead of the current search rate, including zero. When registering custom prices, `None` leaves an existing rate unchanged and `0.0` overrides it
**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. [`ARCHITECTURE.md`](ARCHITECTURE.md) shows where the layers live, and the decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) explain why they are shaped that way. For example, [the ADR on provider usage fields](docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md) covers how provider-specific usage fields reach cost tracking. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
**Bill a new provider usage field from the usage object.** Leave the field on the usage object as an extra and read it in that provider's `cost_calculator.py`. A provider without one gets a new `cost_calculator.py` and an arm for it in `cost_per_token` (`litellm/cost_calculator.py`), which picks the calculator by provider. If the provider's transform builds `Usage` from named fields, as the Anthropic, Gemini, and Bedrock Converse ones do, pass the new field there too. Streamed requests rebuild usage in `ChunkProcessor.calculate_usage` (`litellm/litellm_core_utils/streaming_chunk_builder_utils.py`), which does not copy extras yet the way the Responses-to-chat bridge does (minus a short exclusion list), so until it does, bill the field on streams by naming it there, as Anthropic's `inference_geo` is. A provider that reports its own charge for the request, as OpenRouter does in `usage.cost`, is billed at that charge from `_hidden_params` before any calculator runs, unless the deployment sets its own prices. When the field counts a built-in tool such as web search, add the provider to `get_cost_for_web_search_request` in `litellm/llms/__init__.py`, and add a gate to `litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py` only when no existing gate fires. Then show the field's cost in your proof on every endpoint it reaches, streaming included

2
README.md
View file

@ -696,7 +696,7 @@ For companies that need better security, user management and professional suppor
[Talk to founders](https://enterprise.litellm.ai/demo)
This covers:
- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):**
- ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/enterprise):**
- ✅ **Feature Prioritization**
- ✅ **Custom Integrations**
- ✅ **Professional Support - Dedicated discord + slack**

31
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -10,6 +10,7 @@ from dataclasses import replace as dataclasses_replace
from datetime import datetime, timedelta
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, List, Literal, Optional, Protocol, Tuple, cast
from pydantic import TypeAdapter, ValidationError
from litellm_enterprise.proxy.common_utils.managed_object_attribution import (
ManagedObjectCreatorAttribution,
@ -50,6 +51,7 @@ def _error_names_resource(error: Exception, resource_id: str) -> bool:
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
RECENT_BATCH_POLL_WINDOW: Final = timedelta(hours=48)
GIVE_UP_POLLING_AFTER: Final = timedelta(days=2 * MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
_FILE_OBJECT_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
PROVIDER_TERMINAL_BATCH_STATUSES: Final[Tuple[str, ...]] = (
@ -550,6 +552,10 @@ class CheckBatchCost:
llm_provider: str,
bare_model_name: str,
) -> Optional[Tuple[str, str]]:
stored_model_id: Final = self._get_stored_model_id(job)
if stored_model_id is not None and self._batch_deployment_exists(stored_model_id):
return stored_model_id, job.unified_object_id
deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
if deployment_id is None:
verbose_proxy_logger.info(
@ -662,23 +668,36 @@ class CheckBatchCost:
)
@staticmethod
def _get_input_file_id(job: "ManagedObjectRow") -> Optional[str]:
from litellm.types.utils import LiteLLMBatch
def _parse_file_object(job: "ManagedObjectRow") -> Optional[dict[str, object]]:
file_object: Final = job.file_object
try:
parsed_file_object: Final[object] = (
return _FILE_OBJECT_ADAPTER.validate_python(
json.loads(file_object) if isinstance(file_object, str) else file_object
)
except (json.JSONDecodeError, ValueError):
except (json.JSONDecodeError, ValueError, ValidationError):
return None
if not isinstance(parsed_file_object, dict):
@classmethod
def _get_input_file_id(cls, job: "ManagedObjectRow") -> Optional[str]:
from litellm.types.utils import LiteLLMBatch
parsed_file_object: Final = cls._parse_file_object(job)
if parsed_file_object is None:
return None
try:
return LiteLLMBatch.model_validate(parsed_file_object).input_file_id
except Exception:
return None
@classmethod
def _get_stored_model_id(cls, job: "ManagedObjectRow") -> Optional[str]:
"""The deployment that created the batch, written into file_object by the managed
files hook at create time. Older rows have no such key and fall back to matching
the bare model name against the router."""
parsed_file_object: Final = cls._parse_file_object(job)
model_id: Final = parsed_file_object.get("model_id") if parsed_file_object else None
return model_id if isinstance(model_id, str) and model_id else None
async def _track_completed_batch_cost(
self,
job: "ManagedObjectRow",

1
enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py
View file

@ -200,6 +200,7 @@ async def log_billed_background_response(billed: BilledBackgroundResponse) -> Bi
unpriced: Final = _without_read_price(billed.response)
cost: Final = logging_obj.response_cost_calculator(result=unpriced)
end_time: Final = datetime.now()
logging_obj.capture_usage_before_logging(unpriced, logging_obj.start_time, end_time)
GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue(
async_coroutine=logging_obj.async_success_handler(
result=unpriced, start_time=logging_obj.start_time, end_time=end_time

50
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -21,6 +21,7 @@ from typing import (
TypedDict,
Union,
cast,
runtime_checkable,
)
from uuid import NAMESPACE_URL, uuid5
@ -374,6 +375,12 @@ class _ManagedResourcePrismaClient(Protocol):
def db(self) -> _ManagedResourceDatabase: ...
@runtime_checkable
class _WriterRoutedDatabase(Protocol):
@property
def writer(self) -> _ManagedResourceDatabase: ...
class _SchedulerWithJobLookup(Protocol):
def get_job(self, job_id: str) -> object: ...
@ -479,6 +486,11 @@ def _managed_object_table(prisma_client: _ManagedResourcePrismaClient) -> _Manag
return prisma_client.db.litellm_managedobjecttable
def _writer_database(db: _ManagedResourceDatabase) -> _ManagedResourceDatabase:
"""Reads that feed a write must see the create that just happened; a lagging read replica may not"""
return db.writer if isinstance(db, _WriterRoutedDatabase) else db
_JSON_VALUE_ADAPTER: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
_POSTGRES_UNSTORABLE_CHARACTER: Final = "\x00"
@ -512,6 +524,25 @@ def _without_model_output(
)
def _with_model_id(
file_object: LiteLLMBatch | LiteLLMFineTuningJob | ResponsesAPIResponse,
model_id: str | None,
) -> LiteLLMBatch | LiteLLMFineTuningJob | ResponsesAPIResponse:
return file_object if model_id is None else file_object.model_copy(update={"model_id": model_id})
async def _stored_model_id(managed_object_table: _ManagedObjectTableActions, unified_object_id: str) -> str | None:
row: Final = await managed_object_table.find_first(where={"unified_object_id": unified_object_id})
if row is None:
return None
try:
parsed: Final = _JSON_VALUE_ADAPTER.validate_python(_decode_json_blob(row.file_object))
except (json.JSONDecodeError, ValidationError):
return None
model_id: Final = parsed.get("model_id") if isinstance(parsed, dict) else None
return model_id if isinstance(model_id, str) and model_id else None
def _storage_metadata_of(file_object: OpenAIFileObject | None) -> Mapping[str, str]:
hidden_params: Final = cast( # cast-ok: _hidden_params is an untyped attribute the upload path sets
"Mapping[str, object]", getattr(file_object, "_hidden_params", None) or {}
@ -697,9 +728,17 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
persist_attribution: bool = False,
create_if_missing: bool = True,
batch_processed: bool = False,
creating_model_id: str | None = None,
) -> None:
"""Persist a managed object row, caching it and upserting it in the DB.
creating_model_id is the router deployment that created the object. It is kept
inside the stored file_object JSON as `model_id`, because the raw provider id an
unmanaged batch stores as unified_object_id says nothing about which deployment
made it, and the cost poller needs that deployment to read the result back. A
later observe of the row (create_if_missing=False) keeps the stored value rather
than overwriting it with whichever deployment served the retrieve.
persist_attribution is set only by the batch and background response creates,
the callers that can speak for the creator; it gates the api_key, request_tags, and,
for a response, end_user_id columns that the cost poll jobs bill against, so a later poll or
@ -720,7 +759,14 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
It is written only in the upsert create branch.
"""
verbose_logger.info(f"Storing LiteLLM Managed {file_purpose} object with id={unified_object_id} in cache")
stored_object: Final = _without_model_output(file_object)
managed_database: Final = _managed_database(self.prisma_client)
managed_object_table: Final = managed_database.litellm_managedobjecttable
preserved_model_id: Final = (
creating_model_id
if creating_model_id is not None or create_if_missing
else await _stored_model_id(_writer_database(managed_database).litellm_managedobjecttable, unified_object_id)
)
stored_object: Final = _with_model_id(_without_model_output(file_object), preserved_model_id)
litellm_managed_object = LiteLLM_ManagedObjectTable(
unified_object_id=unified_object_id,
model_object_id=model_object_id,
@ -752,7 +798,6 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"status": file_object.status,
"updated_by": user_api_key_dict.user_id,
}
managed_object_table: Final = _managed_object_table(self.prisma_client)
if not create_if_missing:
await managed_object_table.update_many(
where={"unified_object_id": unified_object_id},
@ -2170,6 +2215,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
request_tags=request_tags_from_metadata(request_metadata if isinstance(request_metadata, dict) else {}),
persist_attribution=is_batch_create,
create_if_missing=is_batch_create,
creating_model_id=model_id if is_batch_create else None,
)
if not is_batch_create:
await self.store_batch_output_file_ownership(

2
litellm-rust/crates/core-utils/src/core_helpers.rs
View file

@ -13,6 +13,7 @@ const FINISH_REASONS: &[(&str, &str)] = &[
("max_tokens", "length"),
("refusal", "content_filter"),
("compaction", "length"),
("pause_turn", "length"),
("model_context_window_exceeded", "length"),
("guardrail_intervened", "content_filter"),
("content_filtered", "content_filter"),
@ -75,6 +76,7 @@ mod tests {
#[case::end_turn("end_turn", "stop")]
#[case::stop_sequence("stop_sequence", "stop")]
#[case::max_tokens("max_tokens", "length")]
#[case::pause_turn("pause_turn", "length")]
#[case::model_context_window_exceeded("model_context_window_exceeded", "length")]
#[case::refusal("refusal", "content_filter")]
#[case::guardrail_intervened("guardrail_intervened", "content_filter")]

1
litellm-rust/crates/cost/AGENTS.md Normal file
View file

@ -0,0 +1 @@
The price vocabulary (meter, band, channel, `When`, `Billing`) is explained in [../model-catalog/AGENTS.md](../model-catalog/AGENTS.md#concepts), the only crate that maps it to JSON keys

8
litellm-rust/crates/cost/src/price.rs
View file

@ -19,6 +19,7 @@ pub enum ServiceTier {
Balanced,
}
/// A prompt-size range. `OverPromptTokens(n)` is spelled `_above_<n>_tokens` or `_above_<n / 1000>k_tokens`
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Band {
#[default]
@ -26,12 +27,15 @@ pub enum Band {
OverPromptTokens(NonZeroU64),
}
/// The request condition a price applies under, its service tier and prompt band
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct When {
pub tier: ServiceTier,
pub band: Band,
}
/// The route whose price list a call reads, from the `_batches` and `_decisions` suffixes. Not part of `When`,
/// since a missing batch price is half the online rate and a missing decisions price is the full online rate
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Channel {
#[default]
@ -40,6 +44,7 @@ pub enum Channel {
Decisions,
}
/// The key of one rate, what is counted, through which channel and under which condition
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Price {
pub meter: Meter,
@ -47,6 +52,7 @@ pub struct Price {
pub when: When,
}
/// Every rate a row lists, its off-peak overlay and the row attributes that say how the rates apply
#[derive(Clone, Debug, Default, PartialEq)]
pub struct Prices {
rates: BTreeMap<Price, UnitPrice>,
@ -117,6 +123,7 @@ pub enum TierScope {
WholeCall,
}
/// Rates that replace the row's token rates while a UTC window in the schedule is open
#[derive(Clone, Debug, PartialEq)]
pub struct OffPeak {
schedule: Schedule,
@ -133,6 +140,7 @@ pub struct OffPeakRates {
pub reasoning: Option<UnitPrice>,
}
/// Multipliers applied on top of a priced call, never prices themselves
#[derive(Clone, Debug, Default, PartialEq)]
pub struct Adjustments(BTreeMap<Uplift, Multiplier>);

3
litellm-rust/crates/cost/src/price/meter.rs
View file

@ -1,5 +1,6 @@
use std::num::NonZeroU64;
/// What a price counts. A variation only one meter has, such as cache TTL or image size, lives inside its variant
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Meter {
Input(InputUnit),
@ -51,12 +52,14 @@ pub enum OutputUnit {
VideoSecond,
}
/// `Explicit` is a cache the caller created ahead of time, spelled `explicit_cache_read_*`. `Implicit` is every other read
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum CacheReadMode {
Implicit,
Explicit,
}
/// How long a cache write lives. `OneHour` and `TwelveHours` are spelled `_above_1hr` and `_above_12hr`
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum CacheTtl {
Default,

42
litellm-rust/crates/model-catalog/AGENTS.md
View file

@ -1,3 +1,41 @@
## Concepts
Every price in a row lowers to one `Price { meter, channel, when }` and its `UnitPrice`. These keys come from the `FlatKey::parse` tests
- `input_cost_per_token`
- meter `Input(Token(Text))`, channel `Online`, tier `Standard`, band `Base`
- `output_cost_per_token_above_272k_tokens_priority`
- meter `Output(Token(Text))`, channel `Online`
- `_priority` sets tier `Priority`
- `_above_272k_tokens` sets band `OverPromptTokens(272000)`
- `cache_creation_input_token_cost_above_1hr_above_200k_tokens`
- meter `CacheWrite { Text, OneHour }`, where `_above_1hr` is the cache TTL, not a band
- `_above_200k_tokens` sets band `OverPromptTokens(200000)`
- channel `Online`, tier `Standard`
- `ocr_cost_per_page_batches`
- meter `Feature(OcrPage)`, tier `Standard`, band `Base`
- `_batches` sets channel `Batch`
- Meter is what gets counted, such as text input tokens, reasoning tokens, an image of a given size or a web search
- A variation only one meter has stays inside it (cache TTL, image size, resolution, `_above_Ns_interval`, `max_results_range`)
- `UnitPrice` is USD for one unit of the meter, such as one token, image, page or second
- Band is a range of prompt sizes, starting where `_above_<N>_tokens` or `_above_<N>k_tokens` says
- `Prices::band` picks one band from prompt tokens alone, the highest edge passed wins, and `Band::Base` is below every edge
- The whole call moves, so every token meter, output included, bills all its tokens at that band's rate
- `Billing::band_edge` says whether a prompt exactly at the edge is in the band (exclusive by default, inclusive for xAI)
- A band only another tier or the batch channel prices never moves the call
- `When` is the request's condition, its service tier and band
- A missing price falls back from `(tier, band)` to `(standard, band)`, `(tier, base)`, then `(standard, base)`
- `Channel` is the route whose price list the call reads, and each falls back differently
- `Online` falls back across `When` as above
- `Batch` (`_batches`) uses bands but no service tier, and a missing price is half the online rate
- `Decisions` (`_decisions`, OpenAI `/v1/decisions`) overrides single online fields, and a missing price is the full online rate
- Off-peak and `Adjustments` are never prices
- Off-peak swaps in other token rates while a UTC window is open
- `Adjustments` multiplies a priced call for region, residency, geo or `speed: fast`
- `Billing` on `Prices` holds row attributes that change how its prices apply (text unit, band edge, cache-read fallback, wall clock, web search unit, tier scope)
- `lower.rs` derives it from the row and its `litellm_provider`, so `litellm-cost` holds no provider list
## Layout
`wire/` reads one entry as written. `EntryWire::read` makes one pass over the entry's keys and puts every non-null key in exactly one place, which is an attribute, a flat price parsed by `FlatKey::parse`, a price block, a `supports_*` capability or `unknown`. A known key with a bad value becomes an `Error` that names the key, and the rest of the entry is still read
@ -6,8 +44,6 @@
`catalog.rs` reads the document, indexes aliases and runs the checks across entries. A bad entry keeps its raw fields and its errors are listed in `Loaded.errors`. Deciding what to do with them is the caller's job. `Catalog::written` gives back every top-level key and value as the document wrote them, in document order. `Catalog::served` gives back every model and alias as Python serves it, without `sample_spec` and `fallback_generalizations`
`Billing` on `Prices` holds the row attributes pricing reads (text unit, band edge, cache-read fallback, wall clock, web search unit, tier scope). `lower.rs` derives it from the row and its `litellm_provider`, so `litellm-cost` holds no provider list
`store/` is the catalog's in-memory store and the only place that indexes model names. Its rules are in [src/store/AGENTS.md](src/store/AGENTS.md)
`layers.rs`, `resolve/`, `model_info.rs` and `register.rs` hold types only. The functions over them land with their first caller
@ -16,7 +52,7 @@ Catalog errors use one opaque `Error` with typed causes and JSON Pointer accesso
## Prices
The price vocabulary lives in `litellm-cost`, and only this crate knows JSON key names. Every price is keyed by `Meter`, `Channel` and `When`. A variation only one meter has stays in the meter (cache TTL, image size, resolution, `_above_Ns_interval`, `max_results_range`). Service tier and prompt band make up `When`. Batch and decisions are a `Channel` because Python bills them with their own rules. Off-peak is an overlay on `Prices`, and regional uplifts are `Adjustments`, never prices
The price vocabulary lives in `litellm-cost`, and only this crate knows JSON key names
A new `N`, image size or resolution in an existing key shape parses with no Rust change. A new unit, token kind, feature or tier needs a variant in `litellm-cost` plus its spelling in `wire/flat_key.rs` and `FLAT_KEY_PATTERN` in `schema.rs`

19
litellm-rust/crates/python-bridge/src/db.rs Normal file
View file

@ -0,0 +1,19 @@
use litellm_host_python::ready_future;
use pyo3::{prelude::*, types::PyNone};
#[pyclass(frozen)]
pub(crate) struct NativeDatabase {}
// TODO: take typed writer/reader settings and a token provider, then open a sqlx PgPool, once the first
// repository moves to native. The Prisma DATABASE_URL carries Prisma-only params and rotating IAM tokens
#[pymethods]
impl NativeDatabase {
#[staticmethod]
fn connect(py: Python<'_>) -> PyResult<Bound<'_, PyAny>> {
ready_future(py, Bound::new(py, NativeDatabase {})?.as_any())
}
fn close<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
ready_future(py, PyNone::get(py).as_any())
}
}

4
litellm-rust/crates/python-bridge/src/lib.rs
View file

@ -2,6 +2,7 @@ mod cache;
mod callable;
mod coercion;
mod credentials;
mod db;
mod diagnostics;
mod errors;
mod execution;
@ -21,6 +22,8 @@ mod _native {
use litellm_host_python::{ForkedAfterNativeRuntimeStarted, ProcessReservedForForking};
use pyo3::{prelude::*, types::PyModule};
#[pymodule_export]
use crate::db::NativeDatabase;
#[cfg(feature = "panic-test")]
#[pymodule_export]
use crate::diagnostics::_panic_for_test;
@ -112,6 +115,7 @@ mod tests {
"NativeDiagnosticProcessor",
"NativeClickHouseSpendConfig",
"NativeClickHouseSpendStorage",
"NativeDatabase",
"trace_encode_error",
"bundled_model_cost_map",
"TokenCounter",

5
litellm/__init__.py
View file

@ -1647,6 +1647,7 @@ def set_global_gitlab_config(config: Mapping[str, object]) -> None:
# Lazy loading system for heavy modules to reduce initial import time and memory usage
if TYPE_CHECKING:
from . import guardrails as guardrails
from litellm.types.utils import ModelInfo as _ModelInfoType
from litellm.types.utils import PriorityReservationSettings
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
@ -2499,10 +2500,10 @@ def __getattr__(name: str) -> Any:
handler_func: Final = registry[name]
return handler_func(name)
if name == "proxy":
if name in ("proxy", "guardrails"):
import importlib
return importlib.import_module("litellm.proxy")
return importlib.import_module(f"litellm.{name}")
if name == "harness" or name in _AGENT_EXPORTS:
import importlib

5
litellm/_internal_context.py
View file

@ -21,6 +21,7 @@ _T = TypeVar("_T")
# When True, suppresses async logging and billing for internal sub-calls
# (e.g., emulated file-search steps that make nested LLM calls).
is_internal_call: Final[ContextVar[bool]] = ContextVar("is_internal_call", default=False)
internal_parent_request_id: Final[ContextVar[str | None]] = ContextVar("internal_parent_request_id", default=None)
# One request prices its totals, its per-token-type lines and the rates it reports on
# separate code paths. Each reads the clock for off-peak pricing, so without a pinned
@ -40,13 +41,15 @@ _mirrored_call: Final[ContextVar[bool]] = ContextVar("mirrored_call", default=Fa
@contextmanager
def internal_sub_call() -> Generator[None]:
def internal_sub_call(parent_request_id: str | None = None) -> Generator[None]:
"""A nested call the parent call logs and bills, so the parent must fold this call's cost into its own."""
token: Final = is_internal_call.set(True)
parent_token: Final = internal_parent_request_id.set(parent_request_id or internal_parent_request_id.get())
try:
yield
finally:
is_internal_call.reset(token)
internal_parent_request_id.reset(parent_token)
@contextmanager

4
litellm/_lazy_imports_registry.py
View file

@ -1469,11 +1469,11 @@ _UTILS_MODULE_IMPORT_MAP: Final = {
"AsyncHTTPHandler": ("litellm.llms.custom_httpx.http_handler", "AsyncHTTPHandler"),
"HTTPHandler": ("litellm.llms.custom_httpx.http_handler", "HTTPHandler"),
"get_num_retries_from_retry_policy": (
"litellm.router_utils.get_retry_from_policy",
"litellm.litellm_core_utils.get_retry_from_policy",
"get_num_retries_from_retry_policy",
),
"reset_retry_policy": (
"litellm.router_utils.get_retry_from_policy",
"litellm.litellm_core_utils.get_retry_from_policy",
"reset_retry_policy",
),
"get_coroutine_checker": (

4
litellm/a2a_protocol/litellm_completion_bridge/handler.py
View file

@ -22,7 +22,7 @@ from litellm.a2a_protocol.litellm_completion_bridge.transformation import (
A2AStreamingContext,
)
from litellm.a2a_protocol.providers.config_manager import A2AProviderConfigManager
from litellm.constants import A2A_CHAT_COMPLETION_BRIDGE_PROVIDERS
from litellm.constants import A2A_CHAT_COMPLETION_BRIDGE_PROVIDERS, A2A_COMPLETION_BRIDGE_WORKING_MESSAGE
from litellm.interactions.agents.utils import merge_agent_headers
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.types.utils import ModelResponse
@ -268,7 +268,7 @@ class A2ACompletionBridgeHandler:
ctx=ctx,
state="working",
final=False,
message_text="Processing request...",
message_text=A2A_COMPLETION_BRIDGE_WORKING_MESSAGE,
)
yield working_event

112
litellm/a2a_protocol/main.py
View file

@ -191,6 +191,23 @@ def _set_litellm_params_on_logging_obj(
}
async def _record_send_message_billing(
kwargs: Mapping[str, object],
request: "SendMessageRequest",
response_dict: Mapping[str, object],
litellm_params: Mapping[str, object],
agent_id: str | None,
) -> None:
prompt_tokens, completion_tokens, _ = await asyncify(A2ARequestUtils.calculate_usage_from_request_response)(
request=request,
response_dict=response_dict,
)
_set_usage_on_logging_obj(kwargs=kwargs, prompt_tokens=prompt_tokens, completion_tokens=completion_tokens)
if "error" not in response_dict:
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
def _get_a2a_model_info(a2a_client: "A2AClientType", kwargs: Mapping[str, object]) -> str:
"""
Extract agent info and set model/custom_llm_provider for cost tracking.
@ -483,13 +500,21 @@ async def asend_message(
if custom_llm_provider:
if request is None:
raise ValueError("request is required for completion bridge")
return await _send_message_via_completion_bridge(
bridge_response: Final = await _send_message_via_completion_bridge(
request=request,
custom_llm_provider=custom_llm_provider,
api_base=api_base,
litellm_params=litellm_params,
agent_extra_headers=agent_extra_headers,
)
await _record_send_message_billing(
kwargs=kwargs,
request=request,
response_dict=bridge_response.model_dump(mode="json", exclude_none=True),
litellm_params=litellm_params,
agent_id=agent_id,
)
return bridge_response
# Standard A2A client flow
if request is None:
@ -538,30 +563,14 @@ async def asend_message(
# Wrap in LiteLLM response type for _hidden_params support
response: Final = LiteLLMSendMessageResponse.from_a2a_response(a2a_response, request_id=str(request.id))
# Calculate token usage from request and response
response_dict: Final[dict[str, object]] = a2a_response.root.model_dump(mode="json", exclude_none=True)
(
prompt_tokens,
completion_tokens,
_,
) = await asyncify(A2ARequestUtils.calculate_usage_from_request_response)(
request=request,
response_dict=response_dict,
)
# Set usage on logging obj for standard logging payload
_set_usage_on_logging_obj(
await _record_send_message_billing(
kwargs=kwargs,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
request=request,
response_dict=a2a_response.root.model_dump(mode="json", exclude_none=True),
litellm_params=litellm_params,
agent_id=agent_id,
)
# Merge agent pricing params into the logging obj so cost is calculated
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
# Set agent_id on logging obj for SpendLogs tracking
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
return response
@ -646,6 +655,7 @@ async def asend_message_streaming(
metadata: dict[str, object] | None = None,
proxy_server_request: dict[str, object] | None = None,
agent_extra_headers: dict[str, str] | None = None,
agent_name: str | None = None,
**kwargs: object,
) -> AsyncIterator["SendStreamingMessageResponse | Mapping[str, object]"]:
"""
@ -661,6 +671,7 @@ async def asend_message_streaming(
agent_id: Optional agent ID for tracking in SpendLogs
metadata: Optional metadata dict (contains user_api_key, user_id, team_id, etc.)
proxy_server_request: Optional proxy server request data
agent_name: Agent name logged as a2a_agent/<agent_name> for completion bridge agents
Yields:
SendStreamingMessageResponse chunks from the agent
@ -685,6 +696,7 @@ async def asend_message_streaming(
print(chunk)
```
"""
called_at: Final = datetime.datetime.now()
litellm_params = litellm_params or {}
custom_llm_provider: Final = litellm_params.get("custom_llm_provider")
@ -705,20 +717,47 @@ async def asend_message_streaming(
request.params.model_dump(mode="json") if hasattr(request.params, "model_dump") else dict(request.params)
)
async for chunk in A2ACompletionBridgeHandler.handle_streaming(
bridge_stream: Final = A2ACompletionBridgeHandler.handle_streaming(
request_id=str(request.id),
params=params,
litellm_params=litellm_params,
api_base=api_base,
agent_extra_headers=agent_extra_headers,
):
yield chunk
)
caller_logging_obj: Final = kwargs.get("litellm_logging_obj")
bridge_logging_obj: Final = (
caller_logging_obj
if isinstance(caller_logging_obj, Logging)
else _build_streaming_logging_obj(
request=request,
agent_name=agent_name or "unknown",
agent_id=agent_id,
litellm_params=litellm_params,
metadata=metadata,
proxy_server_request=proxy_server_request,
start_time=called_at,
)
)
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
bridge_iterator: Final = A2AStreamingIterator(
stream=bridge_stream,
request=request,
logging_obj=bridge_logging_obj,
agent_name=agent_name or "unknown",
)
try:
async for chunk in bridge_iterator:
yield chunk
except (GeneratorExit, asyncio.CancelledError):
await bridge_iterator.abill_on_disconnect()
raise
return
if request is None:
raise ValueError("request is required")
called_at: Final = datetime.datetime.now()
_raw_logging_obj: Final = kwargs.get("litellm_logging_obj")
logging_obj: Logging | None = _raw_logging_obj if isinstance(_raw_logging_obj, Logging) else None
@ -742,12 +781,12 @@ async def asend_message_streaming(
assert a2a_client is not None
agent_name: Final = _get_a2a_model_info(a2a_client, kwargs)
card_agent_name: Final = _get_a2a_model_info(a2a_client, kwargs)
if logging_obj is None:
logging_obj = _build_streaming_logging_obj(
request=request,
agent_name=agent_name,
agent_name=card_agent_name,
agent_id=agent_id,
litellm_params=litellm_params,
metadata=metadata,
@ -755,7 +794,7 @@ async def asend_message_streaming(
start_time=called_at,
)
verbose_logger.info("A2A send_message_streaming request_id=%s, agent=%s", request.id, agent_name)
verbose_logger.info("A2A send_message_streaming request_id=%s, agent=%s", request.id, card_agent_name)
agent_card: Final = _get_a2a_client_agent_card(a2a_client)
card_url: Final = get_agent_card_url(agent_card) if agent_card else None
@ -766,18 +805,23 @@ async def asend_message_streaming(
agent_card=agent_card,
card_url=card_url,
api_base=api_base,
agent_name=agent_name,
agent_name=card_agent_name,
)
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)
async for chunk in A2AStreamingIterator(
url_agent_iterator: Final = A2AStreamingIterator(
stream=stream,
request=request,
logging_obj=logging_obj,
agent_name=agent_name,
):
yield chunk
agent_name=card_agent_name,
)
try:
async for chunk in url_agent_iterator:
yield chunk
except (GeneratorExit, asyncio.CancelledError):
await url_agent_iterator.abill_on_disconnect()
raise
async def create_a2a_client(

2
litellm/a2a_protocol/providers/pydantic_ai_agents/transformation.py
View file

@ -515,6 +515,8 @@ class PydanticAITransformation:
"contextId": context_id,
"kind": "artifact-update",
"taskId": task_id,
"append": i > 0,
"lastChunk": is_last_chunk,
"artifact": {
"artifactId": artifact_id,
"parts": [

2
litellm/a2a_protocol/providers/watsonx_orchestrate/transformation.py
View file

@ -206,6 +206,8 @@ class WatsonxOrchestrateTransformation:
"contextId": context_id,
"kind": "artifact-update",
"taskId": task_id,
"append": i > 0,
"lastChunk": is_last,
"artifact": {
"artifactId": artifact_id,
"parts": [{"kind": "text", "text": chunk_text}],

89
litellm/a2a_protocol/streaming_iterator.py
View file

@ -3,16 +3,18 @@ A2A Streaming Iterator with token tracking and logging support.
"""
import asyncio
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Mapping
from datetime import datetime
from typing import TYPE_CHECKING, Final
import anyio
from pydantic import TypeAdapter
import litellm
from litellm._logging import verbose_logger
from litellm.a2a_protocol.cost_calculator import A2ACostCalculator
from litellm.a2a_protocol.utils import A2ARequestUtils
from litellm.constants import A2A_COMPLETION_BRIDGE_WORKING_MESSAGE
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -23,6 +25,23 @@ if TYPE_CHECKING:
_OPAQUE_PAYLOAD: Final = TypeAdapter(object)
def _chunk_payload(chunk: "SendStreamingMessageResponse | Mapping[str, object]") -> Mapping[str, object]:
if isinstance(chunk, Mapping):
return chunk
return chunk.model_dump(mode="json", exclude_none=True) if hasattr(chunk, "model_dump") else {}
def _artifact_update(payload: Mapping[str, object]) -> tuple[str, bool] | None:
result: Final = as_str_mapping(payload.get("result"))
if result is None or result.get("kind") != "artifact-update":
return None
artifact: Final = as_str_mapping(result.get("artifact"))
artifact_id: Final = artifact.get("artifactId") if artifact is not None else None
if not isinstance(artifact_id, str):
return None
return artifact_id, result.get("append") is True
class A2AStreamingIterator:
"""
Async iterator for A2A streaming responses with token tracking.
@ -32,7 +51,7 @@ class A2AStreamingIterator:
def __init__(
self,
stream: AsyncIterator["SendStreamingMessageResponse"],
stream: AsyncIterator["SendStreamingMessageResponse | Mapping[str, object]"],
request: "SendStreamingMessageRequest",
logging_obj: LiteLLMLoggingObj,
agent_name: str = "unknown",
@ -44,14 +63,16 @@ class A2AStreamingIterator:
self.start_time = logging_obj.start_time
# Collect chunks for token counting
self.chunks: list[SendStreamingMessageResponse] = []
self.collected_text_parts: list[str] = []
self.final_chunk: SendStreamingMessageResponse | None = None
self.chunks: list[SendStreamingMessageResponse | Mapping[str, object]] = []
self.latest_message_text = ""
self.artifact_texts: Mapping[str, str] = {}
self.saw_error = False
self.final_chunk: SendStreamingMessageResponse | Mapping[str, object] | None = None
def __aiter__(self):
return self
async def __anext__(self) -> "SendStreamingMessageResponse":
async def __anext__(self) -> "SendStreamingMessageResponse | Mapping[str, object]":
try:
chunk: Final = await self.stream.__anext__()
if self.logging_obj.completion_start_time is None:
@ -59,38 +80,54 @@ class A2AStreamingIterator:
# Store chunk
self.chunks.append(chunk)
payload: Final = _chunk_payload(chunk)
self.saw_error = self.saw_error or payload.get("error") is not None
# Extract text from chunk for token counting
self._collect_text_from_chunk(chunk)
self._collect_text_from_chunk(payload)
# Check if this is the final chunk (completed status)
if self._is_completed_chunk(chunk):
if self._is_completed_chunk(payload):
self.final_chunk = chunk
return chunk
except StopAsyncIteration:
# Stream ended - handle logging
if self.final_chunk is None and self.chunks:
self.final_chunk = self.chunks[-1]
await self._handle_stream_complete()
raise
def _collect_text_from_chunk(self, chunk: "SendStreamingMessageResponse") -> None:
"""Extract text from a streaming chunk and add to collected parts."""
async def abill_on_disconnect(self) -> None:
if not self.chunks:
return
with anyio.CancelScope(shield=True):
await self._handle_stream_complete()
def _collect_text_from_chunk(self, payload: Mapping[str, object]) -> None:
"""Record the reply text a streaming chunk carries."""
try:
chunk_dict: Final = chunk.model_dump(mode="json", exclude_none=True) if hasattr(chunk, "model_dump") else {}
text: Final = A2ARequestUtils.extract_text_from_response(chunk_dict)
if text:
self.collected_text_parts.append(text)
text: Final = A2ARequestUtils.extract_text_from_stream_event(payload)
if not text:
return
artifact_update: Final = _artifact_update(payload)
if artifact_update is None and text == A2A_COMPLETION_BRIDGE_WORKING_MESSAGE:
return
if artifact_update is None:
self.latest_message_text = text
return
artifact_id, append = artifact_update
artifact_text: Final = self.artifact_texts.get(artifact_id, "") + text if append else text
self.artifact_texts = {**self.artifact_texts, artifact_id: artifact_text}
except Exception:
verbose_logger.debug("Failed to extract text from A2A streaming chunk")
def _is_completed_chunk(self, chunk: "SendStreamingMessageResponse") -> bool:
def _reply_text(self) -> str:
artifact_text: Final = " ".join(text for text in self.artifact_texts.values() if text)
return artifact_text or self.latest_message_text
def _is_completed_chunk(self, payload: Mapping[str, object]) -> bool:
"""Check if chunk indicates stream completion."""
try:
chunk_dict: Final = chunk.model_dump(mode="json", exclude_none=True) if hasattr(chunk, "model_dump") else {}
result: Final = chunk_dict.get("result", {})
result: Final = payload.get("result", {})
if isinstance(result, dict):
status: Final = result.get("status", {})
if isinstance(status, dict):
@ -101,6 +138,8 @@ class A2AStreamingIterator:
async def _handle_stream_complete(self) -> None:
"""Handle logging and token counting when stream completes."""
if self.final_chunk is None and self.chunks:
self.final_chunk = self.chunks[-1]
try:
end_time: Final = datetime.now()
@ -109,9 +148,7 @@ class A2AStreamingIterator:
input_text: Final = A2ARequestUtils.extract_text_from_message(input_message)
prompt_tokens: Final = await asyncify(A2ARequestUtils.count_tokens)(input_text)
# Use the last (most complete) text from chunks
output_text: Final = self.collected_text_parts[-1] if self.collected_text_parts else ""
completion_tokens: Final = await asyncify(A2ARequestUtils.count_tokens)(output_text)
completion_tokens: Final = await asyncify(A2ARequestUtils.count_tokens)(self._reply_text())
total_tokens: Final = prompt_tokens + completion_tokens
@ -128,7 +165,7 @@ class A2AStreamingIterator:
self.logging_obj.model_call_details["stream"] = False
# Calculate cost using A2ACostCalculator
response_cost: Final = A2ACostCalculator.calculate_a2a_cost(self.logging_obj)
response_cost: Final = 0.0 if self.saw_error else A2ACostCalculator.calculate_a2a_cost(self.logging_obj)
self.logging_obj.model_call_details["response_cost"] = response_cost
# Build result for logging
@ -167,9 +204,7 @@ class A2AStreamingIterator:
# Add final chunk result if available
if self.final_chunk:
try:
chunk_dict: Final = as_str_mapping(
_OPAQUE_PAYLOAD.validate_python(self.final_chunk.model_dump(mode="json", exclude_none=True))
)
chunk_dict: Final = as_str_mapping(_OPAQUE_PAYLOAD.validate_python(_chunk_payload(self.final_chunk)))
if chunk_dict is not None:
result["result"] = chunk_dict.get("result", {})
except Exception:

50
litellm/a2a_protocol/utils.py
View file

@ -6,13 +6,17 @@ import hashlib
from collections.abc import Mapping
from typing import TYPE_CHECKING, Final, Protocol
from pydantic import TypeAdapter
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.core_helpers import has_attributes
from litellm.litellm_core_utils.core_helpers import as_str_mapping, has_attributes
if TYPE_CHECKING:
from a2a.types import SendMessageRequest, SendStreamingMessageRequest
_OBJECT_ITEMS: Final = TypeAdapter(tuple[object, ...])
class _SupportsModelDump(Protocol):
def model_dump(self, *, mode: str) -> object: ...
@ -34,6 +38,9 @@ class A2ARequestUtils:
"""
if message is None:
return ""
# a2a-sdk Part RootModels keep kind/text on part.root, so read pydantic messages as dicts
if has_attributes(message, _SupportsModelDump, "model_dump"):
return A2ARequestUtils.extract_text_from_message(message.model_dump(mode="json"))
# Handle both dict and object access
if isinstance(message, dict):
@ -63,8 +70,8 @@ class A2ARequestUtils:
Returns:
Text from response message parts
"""
result: Final = response_dict.get("result", {})
if not isinstance(result, dict):
result: Final = as_str_mapping(response_dict.get("result"))
if result is None:
return ""
# Direct message format (A2A spec): detect by explicit kind tag only.
@ -72,10 +79,36 @@ class A2ARequestUtils:
# type that happens to include a "parts" field.
if result.get("kind") == "message":
return A2ARequestUtils.extract_text_from_message(result)
if result.get("kind") == "task":
return A2ARequestUtils.extract_text_from_task(result)
message: Final = result.get("message", {})
return A2ARequestUtils.extract_text_from_message(message)
@staticmethod
def extract_text_from_task(task: Mapping[str, object]) -> str:
artifacts: Final = task.get("artifacts")
artifact_items: Final = _OBJECT_ITEMS.validate_python(artifacts) if isinstance(artifacts, list) else ()
artifact_texts: Final = tuple(
A2ARequestUtils.extract_text_from_message(artifact) for artifact in artifact_items
)
if any(artifact_texts):
return " ".join(text for text in artifact_texts if text)
status: Final = as_str_mapping(task.get("status"))
return A2ARequestUtils.extract_text_from_message(status.get("message")) if status is not None else ""
@staticmethod
def extract_text_from_stream_event(event: Mapping[str, object]) -> str:
result: Final = as_str_mapping(event.get("result"))
if result is None:
return ""
if result.get("kind") == "artifact-update":
return A2ARequestUtils.extract_text_from_message(result.get("artifact"))
if result.get("kind") == "status-update":
status: Final = as_str_mapping(result.get("status"))
return A2ARequestUtils.extract_text_from_message(status.get("message")) if status is not None else ""
return A2ARequestUtils.extract_text_from_response(event)
@staticmethod
def get_input_message_from_request(
request: "SendMessageRequest | SendStreamingMessageRequest",
@ -128,17 +161,8 @@ class A2ARequestUtils:
Returns:
Tuple of (prompt_tokens, completion_tokens, total_tokens)
"""
# Count input tokens. Dump the message to a dict first so extraction hits
# the dict branch — request-side parts are a2a-sdk Part RootModels whose
# kind/text live on part.root, which the object branch cannot read. This
# mirrors how the response side already works (it operates on model_dump).
input_message: Final = A2ARequestUtils.get_input_message_from_request(request)
input_text: Final = A2ARequestUtils.extract_text_from_message(
input_message.model_dump(mode="json")
if has_attributes(input_message, _SupportsModelDump, "model_dump")
else input_message
)
prompt_tokens: Final = A2ARequestUtils.count_tokens(input_text)
prompt_tokens: Final = A2ARequestUtils.count_tokens(A2ARequestUtils.extract_text_from_message(input_message))
# Count output tokens
output_text: Final = A2ARequestUtils.extract_text_from_response(response_dict)

6
litellm/accounting/protocols.py
View file

@ -3,7 +3,7 @@
The event row's ``payload`` is a private envelope versioned by the row's ``schema_version``.
Usage envelopes retain the public event and request-scoped pricing context: quantities with units,
model/provider, requested and billed tier/region when known, and per-request/deployment overrides.
These inputs must survive request teardown. Cost-map rates are resolved at compilation, not capture.
These inputs, including the effective cost-map rates, must survive request teardown.
Cost envelopes retain the public CostEvent, including its required finalized pricing snapshot.
T5/T6 own envelope serialization; this public API does not introduce a second persisted schema.
"""
@ -38,8 +38,8 @@ class CostCompiler(Protocol):
async def compile(self, event: OutboundUsageEvent) -> CostEvent | Unpriced:
"""Reuse the committed charge or finalize one charge and its delivery intent.
Resolve captured request-scoped context by event identity and cost-map rates
at compilation. Persist the resolved inputs in the required snapshot. Replay
Resolve captured request-scoped context and rates by event identity.
Persist the resolved inputs in the required snapshot. Replay
reuses the committed result and never consumes free allowance again.
Missing pricing is Unpriced, never an inferred zero.
"""

4
litellm/accounting/types.py
View file

@ -276,8 +276,8 @@ class NamedCost(BaseModel):
class PricingSnapshot(BaseModel):
"""Retained inputs and result of authoritative compilation, reproducible after cost-map rates change.
Request-scoped inputs and overrides come from the usage row's versioned payload. Cost-map rates are
resolved at compilation, not frozen at capture. Retries reuse the committed charge and snapshot.
Request-scoped inputs, overrides and effective rates come from the usage row's versioned payload.
Retries reuse the committed charge and snapshot.
Component names match ``CostBreakdown``: reasoning_cost is part of output_cost, not an additional charge.
Component sums and float tolerances belong to reconciliation, not this structural contract.
"""

95
litellm/batches/batch_utils.py
View file

@ -3,9 +3,10 @@ from collections.abc import Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from dataclasses import replace as dataclasses_replace
from enum import Enum
from itertools import groupby
from typing import Any, Final, Literal, cast
from pydantic import TypeAdapter
from pydantic import JsonValue, TypeAdapter
import litellm
from litellm._logging import verbose_logger
@ -15,6 +16,7 @@ from litellm.llms.base_llm.ocr.transformation import OCRUsageInfo
from litellm.llms.bedrock.batches.transformation import titan_embedding_usage_from_batch_output
from litellm.llms.vertex_ai.batches.transformation import (
is_native_vertex_batch_output_row,
native_vertex_batch_row_model,
native_vertex_batch_row_stats,
)
from litellm.types.llms.openai import Batch
@ -210,6 +212,7 @@ class _BatchOutputLineStats:
cache_creation_tokens: int
reasoning_tokens: int
model: str | None
usage: Usage | None
def _classify_output_line_stats(
@ -268,7 +271,7 @@ def _compute_output_line_stats(
raw_model: Final = response_body.get("model")
response_model: Final = raw_model if isinstance(raw_model, str) and raw_model else None
completion_details: Final = usage.completion_tokens_details
line_prompt_cost, line_completion_cost = _output_line_cost(
line_prompt_cost, line_completion_cost = _output_line_cost_or_pending(
response_body=response_body,
usage=usage,
custom_llm_provider=custom_llm_provider,
@ -286,7 +289,15 @@ def _compute_output_line_stats(
cache_read_tokens=prompt_details["cache_hit_tokens"],
cache_creation_tokens=prompt_details["cache_creation_tokens"],
reasoning_tokens=(completion_details.reasoning_tokens if completion_details else None) or 0,
model=response_model,
model=response_model or model_name,
usage=(
usage.model_copy(update=ocr_usage.model_dump(exclude_none=True))
if (ocr_usage := _ocr_usage_info_from_response_body(response_body)) is not None
and ocr_usage.model_dump(exclude_none=True)
else usage
if response_body.get("usage") is not None or usage.total_tokens > 0
else None
),
)
@ -298,6 +309,54 @@ def _ocr_usage_info_from_response_body(response_body: Mapping[str, object]) -> O
return OCRUsageInfo.model_validate(raw_usage_info)
def _output_line_cost_or_pending(
response_body: Mapping[str, object],
usage: Usage,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "bedrock", "mistral"],
model_name: str | None,
response_model: str | None,
model_info: ModelInfo | None,
vertex_location: str | None,
) -> tuple[float, float]:
try:
return _output_line_cost(
response_body, usage, custom_llm_provider, model_name, response_model, model_info, vertex_location
)
except Exception: # noqa: BLE001 # retain measured usage when existing batch pricing fails
verbose_logger.warning("Batch line pricing unavailable; retaining usage with the existing zero-cost fallback")
return 0.0, 0.0
def _combined_reported_batch_usage(usages: tuple[Usage | None, ...]) -> Mapping[str, object] | None:
from litellm.cost_calculator import BaseTokenUsageProcessor
reported: Final = [usage for usage in usages if usage is not None]
if not reported:
return None
fields: Final = tuple(TypeAdapter(dict[str, JsonValue]).validate_python(usage.model_dump()) for usage in reported)
quantities: Final = {
key: sum(value for usage in fields if isinstance(value := usage.get(key), (int, float)))
for key in OCRUsageInfo.model_fields
if any(isinstance(usage.get(key), (int, float)) for usage in fields)
}
return {**BaseTokenUsageProcessor.combine_usage_objects(usage_objects=reported).model_dump(), **quantities}
def _batch_usage_by_request(usages: tuple[tuple[str | None, Usage | None], ...]) -> tuple[Mapping[str, object], ...]:
rows: Final = tuple(
{"model": model, "usage": usage.model_dump() if usage is not None else None} for model, usage in usages
)
return tuple(sorted(rows, key=lambda row: json.dumps(row, sort_keys=True)))
def _batch_usage_by_model(usages: tuple[tuple[str | None, Usage | None], ...]) -> Mapping[str, object]:
ordered: Final = sorted(usages, key=lambda item: item[0] or "")
return {
model: _combined_reported_batch_usage(tuple(item[1] for item in group))
for model, group in groupby(ordered, key=lambda item: item[0] or "")
}
def _output_line_cost(
response_body: Mapping[str, object],
usage: Usage,
@ -361,6 +420,9 @@ def _aggregate_batch_cost_usage_models(
completion_tokens=sum(stats.completion_tokens for stats in line_stats),
reasoning_tokens=sum(stats.reasoning_tokens for stats in line_stats),
**cache_token_params,
batch_usage_by_model=_batch_usage_by_model(tuple((stats.model, stats.usage) for stats in line_stats)),
batch_usage_by_request=_batch_usage_by_request(tuple((stats.model, stats.usage) for stats in line_stats)),
batch_unknown_usage_requests=sum(stats.usage is None for stats in line_stats),
)
batch_models: Final = [model_name] if model_name else [stats.model for stats in line_stats if stats.model]
total_prompt_cost: Final = sum((stats.prompt_cost for stats in line_stats), 0.0)
@ -401,17 +463,21 @@ def calculate_vertex_ai_batch_cost_and_usage(
from litellm.cost_calculator import batch_cost_calculator
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
row_stats: Final = tuple(
native_vertex_batch_row_stats(
row,
model_name,
model_info=model_info,
calculate_usage=VertexGeminiConfig.calculate_usage,
cost_calculator=batch_cost_calculator,
vertex_location=vertex_location,
observations: Final = tuple(
(
native_vertex_batch_row_model(row, model_name),
native_vertex_batch_row_stats(
row,
model_name,
model_info=model_info,
calculate_usage=VertexGeminiConfig.calculate_usage,
cost_calculator=batch_cost_calculator,
vertex_location=vertex_location,
),
)
for row in vertex_ai_batch_responses
)
row_stats: Final = tuple(stats for _, stats in observations)
priced: Final = tuple(stats for stats in row_stats if stats is not None)
total_prompt_cost: Final = sum(stats.prompt_cost for stats in priced)
total_completion_cost: Final = sum(stats.completion_cost for stats in priced)
@ -435,6 +501,13 @@ def calculate_vertex_ai_batch_cost_and_usage(
total_tokens=total_tokens,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
batch_usage_by_model=_batch_usage_by_model(
tuple((model, stats.usage if stats is not None else None) for model, stats in observations)
),
batch_usage_by_request=_batch_usage_by_request(
tuple((model, stats.usage if stats is not None else None) for model, stats in observations)
),
batch_unknown_usage_requests=sum(stats is None for stats in row_stats),
),
models=(
[model_name]

13
litellm/caching/caching.py
View file

@ -25,7 +25,11 @@ from litellm._internal_context import current_service_target, service_target
from litellm._logging import verbose_logger
from litellm.constants import CACHED_STREAMING_CHUNK_DELAY
from litellm.integrations.otel.runtime import phase_span
from litellm.litellm_core_utils.model_param_helper import ModelParamHelper, peek_reasoning_summary_aliases
from litellm.litellm_core_utils.model_param_helper import (
ModelParamHelper,
embeds_thought_signature_in_tool_call_id,
peek_reasoning_summary_aliases,
)
from litellm.types.caching import *
from litellm.types.utils import EmbeddingResponse, is_litellm_owned_kwarg
@ -421,7 +425,12 @@ class Cache:
reasoning_summary_material: Final = (
"" if reasoning_summary is None else f"reasoning_summary: {reasoning_summary}"
)
hashed_cache_key = Cache._get_hashed_cache_key(cache_key + reasoning_summary_material)
tool_call_id_shape_material: Final = (
"" if embeds_thought_signature_in_tool_call_id(kwargs) else "thought_signature_in_tool_call_id: False"
)
hashed_cache_key = Cache._get_hashed_cache_key(
cache_key + reasoning_summary_material + tool_call_id_shape_material
)
hashed_cache_key = self._add_namespace_to_cache_key(hashed_cache_key, **kwargs)
verbose_logger.debug(
"\nCreated cache key: %s (source material length: %d)",

11
litellm/constants.py
View file

@ -177,6 +177,8 @@ MID_STREAM_FAILURE_LOG_WAIT_SECONDS: Final = 1.0
# Set to 0 to disable truncation.
MAX_BASE64_LENGTH_FOR_LOGGING: Final = get_env_int("MAX_BASE64_LENGTH_FOR_LOGGING", 64)
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS: Final = 256 * 1024
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_STRINGS: Final = 256
BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS: Final = 8 * 1024 * 1024
REDACTED_BY_LITELLM: Final = "redacted-by-litellm"
# in-memory stand-in handed to provider converters for redacted arguments; never stored
REDACTED_TOOL_CALL_ARGUMENTS_PLACEHOLDER: Final = "{}"
@ -430,6 +432,7 @@ WEBSOCKET_CLOSE_REASON_MAX_BYTES: Final = 123
DEEPGRAM_DEFAULT_API_BASE: Final = "https://api.deepgram.com/v1"
NADIR_DEFAULT_API_BASE: Final = "https://api.getnadir.com/v1"
DEEPGRAM_LISTEN_DEFAULT_MODEL: Final = "nova-3"
DEEPGRAM_SPEAK_DEFAULT_MODEL: Final = "aura-asteria-en"
BEDROCK_REALTIME_PENDING_OPENING_MESSAGE_SCOPE_KEY: Final = "litellm.bedrock_realtime.pending_opening_message"
BEDROCK_REALTIME_SESSION_COMMITTED_SCOPE_KEY: Final = "litellm.bedrock_realtime.session_committed"
@ -634,6 +637,8 @@ STS_CREDENTIAL_EXPIRY_SAFETY_MARGIN_SECONDS: Final = 60
BEDROCK_MIN_THINKING_BUDGET_TOKENS: Final = get_env_int("BEDROCK_MIN_THINKING_BUDGET_TOKENS", 1024)
# Anthropic's Messages API rejects thinking.budget_tokens < 1024.
ANTHROPIC_MIN_THINKING_BUDGET_TOKENS: Final = 1024
ANTHROPIC_MAX_USER_ID_LENGTH: Final = 512
ANTHROPIC_COMPATIBLE_MAX_USER_ID_LENGTH: Final = 255
ANTHROPIC_BILLING_METADATA_PREFIX: Final = "x-anthropic-billing-header:"
REPLICATE_POLLING_DELAY_SECONDS: Final = get_env_float("REPLICATE_POLLING_DELAY_SECONDS", 0.5)
DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS: Final = get_env_int("DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS", 128000)
@ -671,7 +676,7 @@ request_timeout: float = (
DEFAULT_REQUEST_TIMEOUT_SECONDS if _request_timeout_from_env is None else _request_timeout_from_env
)
request_timeout_explicitly_set: bool = _request_timeout_from_env is not None
DEFAULT_A2A_AGENT_TIMEOUT: Final[float] = get_env_float("DEFAULT_A2A_AGENT_TIMEOUT", 6000.0) # 10 minutes
DEFAULT_A2A_AGENT_TIMEOUT: Final[float] = get_env_float("DEFAULT_A2A_AGENT_TIMEOUT", 6000.0)
AGENT_KILL_SWITCH_TIMEOUT_SECONDS: Final = 10.0
AGENT_KILL_SWITCH_RESPONSE_BODY_MAX_CHARS: Final = 2000
# Patterns that indicate a localhost/internal URL in A2A agent cards that should be
@ -902,6 +907,7 @@ PROVIDERS_THAT_AUTHENTICATE_ON_PROVIDER_INFO: Final = frozenset(
SUBSCRIPTION_BACKED_PROVIDERS: Final = frozenset({"chatgpt"})
A2A_CHAT_COMPLETION_BRIDGE_PROVIDERS: Final = frozenset({"databricks_agent"})
A2A_COMPLETION_BRIDGE_WORKING_MESSAGE: Final = "Processing request..."
LITELLM_EMBEDDING_PROVIDERS_SUPPORTING_INPUT_ARRAY_OF_TOKENS: Final = [
"openai",
@ -1162,6 +1168,7 @@ openai_compatible_providers: Final[list[str]] = [
"cortecs",
"scx-ai",
"prism",
"hostinger_router",
"sail",
"reka",
"streamlake",
@ -1685,6 +1692,7 @@ BAGGAGE_MAPPED_TAGS_METADATA_KEY: Final = "_baggage_mapped_tags"
VECTOR_STORE_CALLER_METADATA_KWARG: Final = "litellm_vector_store_caller_metadata"
ROUTER_USAGE_COUNTED_TOKENS_METADATA_KEY: Final = "_litellm_router_usage_counted_tokens"
INTERNAL_CALL_ORIGIN_METADATA_KEY: Final = "internal_call_origin"
RESPONSES_WEBSOCKET_CALL_ID_METADATA_KEY: Final = "responses_websocket_call_id"
BILLED_RETRY_ATTEMPT_METADATA_KEY: Final = "billed_retry_attempt"
MANAGED_BACKGROUND_RESPONSE_METADATA_KEY: Final = "_managed_background_response"
SESSION_ID_GENERATED_METADATA_KEY: Final = "litellm_session_id_generated"
@ -2046,6 +2054,7 @@ LITELLM_SETTINGS_SAFE_DB_OVERRIDES: Final = [
SPECIAL_LITELLM_AUTH_TOKEN: Final = ["ui-token"]
DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL = get_env_int("DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL", 60)
AUTH_OBJECTS_TARGET: Final = "auth_objects"
HEALTH_CHECKS_TARGET: Final = "health_checks"
DEFAULT_ACCESS_GROUP_CACHE_TTL: Final = get_env_int("DEFAULT_ACCESS_GROUP_CACHE_TTL", 600)
SPEND_LOG_KEY_METADATA_CACHE_TTL: Final = 600
SPEND_LOG_KEY_METADATA_MISS_CACHE_TTL: Final = 30

111
litellm/cost_calculator.py
View file

@ -623,7 +623,11 @@ def cost_per_token(
)
if lyria_generation_cost is not None:
return 0.0, lyria_generation_cost
speech_model_info = litellm.get_model_info(model=model_without_prefix, custom_llm_provider=custom_llm_provider)
speech_model_info: Final = cached_get_model_info_helper(
model=model_without_prefix,
custom_llm_provider=custom_llm_provider,
preserve_unset_costs=True,
)
cost_metric: Final = select_cost_metric_for_model(speech_model_info)
prompt_cost: float = 0.0
completion_cost: float = 0.0
@ -1550,12 +1554,66 @@ def _breakdown_total(litellm_logging_obj: LitellmLoggingObject | None) -> float
return float(total) if isinstance(total, (int, float)) and not isinstance(total, bool) else None
def _cost_map_priced_input_output(
model: str | None,
custom_llm_provider: str | None,
usage: Usage | None,
call_type: CallTypesLiteral | None,
service_tier: str | None,
data_residency: str | None,
) -> tuple[float, float]:
if not model or usage is None:
return 0.0, 0.0
try:
return cost_per_token(
model=model,
custom_llm_provider=custom_llm_provider,
usage_object=usage,
call_type=call_type or "completion",
service_tier=service_tier,
data_residency=data_residency,
)
except Exception as pricing_error: # noqa: BLE001 # cost_per_token raises plain Exception for an unmapped model
verbose_logger.debug("No cost map price for reported-cost split of %s: %s", model, pricing_error)
return 0.0, 0.0
def split_reported_cost_by_cost_map_ratio(
reported_total: float,
model: str | None,
custom_llm_provider: str | None,
usage: Usage | None,
call_type: CallTypesLiteral | None,
service_tier: str | None,
data_residency: str | None,
) -> tuple[float, float]:
"""Apportions a provider-reported total between input and output in the ratio litellm's own cost map would
price the same usage, so a per-input-token model books the whole total on input. Unmapped or zero-priced
usage keeps the whole total on output."""
priced_input, priced_output = _cost_map_priced_input_output(
model=model,
custom_llm_provider=custom_llm_provider,
usage=usage,
call_type=call_type,
service_tier=service_tier,
data_residency=data_residency,
)
priced_total: Final = priced_input + priced_output
if priced_total <= 0:
return 0.0, reported_total
input_cost: Final = reported_total * (priced_input / priced_total)
return input_cost, reported_total - input_cost
def _adjusted_provider_billed_cost(
reported_cost: float,
custom_llm_provider: str | None,
pricing_provider: str | None,
model: str | None,
region_name: str | None,
litellm_logging_obj: LitellmLoggingObject | None,
usage: Usage | None,
call_type: CallTypesLiteral | None,
service_tier: str | None,
data_residency: str | None,
) -> float:
@ -1565,10 +1623,19 @@ def _adjusted_provider_billed_cost(
adjusted_cost, margin_percent, margin_fixed_amount, margin_total_amount = _apply_cost_margin(
base_cost=discounted_cost, custom_llm_provider=custom_llm_provider
)
input_cost, output_cost = split_reported_cost_by_cost_map_ratio(
reported_total=adjusted_cost,
model=model,
custom_llm_provider=pricing_provider,
usage=usage,
call_type=call_type,
service_tier=service_tier,
data_residency=data_residency,
)
_store_cost_breakdown_in_logging_obj(
litellm_logging_obj=litellm_logging_obj,
prompt_tokens_cost_usd_dollar=0.0,
completion_tokens_cost_usd_dollar=adjusted_cost,
prompt_tokens_cost_usd_dollar=input_cost,
completion_tokens_cost_usd_dollar=output_cost,
cost_for_built_in_tools_cost_usd_dollar=0.0,
total_cost_usd_dollar=adjusted_cost,
original_cost=reported_cost,
@ -2076,9 +2143,16 @@ def completion_cost(
return _adjusted_provider_billed_cost(
reported_cost=reported_cost,
custom_llm_provider=custom_llm_provider,
pricing_provider=_pricing_provider_for_cost_candidate(
candidate_provider=candidate_pricing_provider,
request_provider=request_pricing_provider,
current_provider=custom_llm_provider,
),
model=model_for_discount,
region_name=region_name if isinstance(region_name, str) else None,
litellm_logging_obj=litellm_logging_obj,
usage=cost_per_token_usage_object,
call_type=call_type,
service_tier=service_tier,
data_residency=data_residency,
)
@ -2383,10 +2457,28 @@ def response_cost_calculator(
provider_response_cost: Final = get_response_cost_from_hidden_params(hidden_params)
if provider_response_cost is not None and not custom_pricing:
if _breakdown_total(litellm_logging_obj) != provider_response_cost:
header_priced_model: Final = select_model_name_for_cost_calc(
model=model,
completion_response=response_object,
custom_llm_provider=custom_llm_provider,
custom_pricing=custom_pricing,
base_model=base_model,
router_model_id=router_model_id,
region_name=region_name,
)
header_input_cost, header_output_cost = split_reported_cost_by_cost_map_ratio(
reported_total=provider_response_cost,
model=header_priced_model or model,
custom_llm_provider=pricing_provider or custom_llm_provider,
usage=get_usage_object(response_object),
call_type=call_type,
service_tier=service_tier,
data_residency=data_residency,
)
_store_cost_breakdown_in_logging_obj(
litellm_logging_obj=litellm_logging_obj,
prompt_tokens_cost_usd_dollar=0.0,
completion_tokens_cost_usd_dollar=provider_response_cost,
prompt_tokens_cost_usd_dollar=header_input_cost,
completion_tokens_cost_usd_dollar=header_output_cost,
cost_for_built_in_tools_cost_usd_dollar=0.0,
total_cost_usd_dollar=provider_response_cost,
original_cost=provider_response_cost,
@ -3084,7 +3176,7 @@ def batch_cost_calculator(
batch_rates.cache_creation,
batch_rates.cache_creation_rates_by_ttl,
)
elif input_cost_per_token:
elif input_cost_per_token is not None:
details: Final = parse_prompt_tokens_details(usage)
cache_read_tokens: Final = details["cache_hit_tokens"]
cache_creation_tokens: Final = details["cache_creation_tokens"]
@ -3100,7 +3192,10 @@ def batch_cost_calculator(
cache_read_cost_key: Final = get_service_tier_cost_key("cache_read_input_token_cost", None)
total_prompt_cost += calculate_cost_component(model_info, cache_read_cost_key, cache_read_tokens) / 2
cache_creation_cost: Final = model_info.get("cache_creation_input_token_cost") or input_cost_per_token
configured_cache_creation_cost: Final = model_info.get("cache_creation_input_token_cost")
cache_creation_cost: Final = (
input_cost_per_token if configured_cache_creation_cost is None else configured_cache_creation_cost
)
total_prompt_cost += cache_creation_tokens * cache_creation_cost / 2
text_rate: Final = _batch_or_half(batch_rates.output, output_cost_per_token)
image_rate: Final = _batch_or_half(
@ -3312,7 +3407,7 @@ class RealtimeAPITokenUsageProcessor(BaseTokenUsageProcessor):
)
_RESPONSES_WS_BILLABLE_EVENT_TYPES: Final = frozenset({"response.completed", "response.incomplete"})
_RESPONSES_WS_BILLABLE_EVENT_TYPES: Final = frozenset({"response.completed", "response.incomplete", "response.failed"})
class _ResponsesWsEventResponse(LiteLLMBaseModel):

168
litellm/decisions/call.py
View file

@ -3,10 +3,10 @@ from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field, replace
from functools import partial
from types import MappingProxyType
from typing import Final, Literal, TypeAlias
from typing import Annotated, Final, Literal, TypeAlias
import httpx
from pydantic import BaseModel, ConfigDict, StrictBool, TypeAdapter, ValidationError
from pydantic import BaseModel, BeforeValidator, ConfigDict, TypeAdapter, ValidationError
from pydantic_core import to_jsonable_python
from typing_extensions import assert_never
@ -23,6 +23,7 @@ from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.decisions.transformation import (
BaseDecisionsConfig,
DecisionsResponseValidationError,
catalog_mode,
image_media_type,
image_param,
input_images,
@ -61,7 +62,7 @@ from litellm.types.decisions import (
UnsupportedDecisionsRequest,
)
from litellm.types.llms.openai import ChatCompletionUserMessage
from litellm.types.utils import LlmProviders, ProviderSpecificHeader
from litellm.types.utils import LlmProviders, ProviderSpecificHeader, is_decisions_model_mode
from litellm.utils import ProviderConfigManager, get_model_cost_key
SystemOneQuestions: TypeAlias = Mapping[str, DecisionQuestion | Mapping[str, object]]
@ -218,7 +219,7 @@ def supported_decisions_providers() -> tuple[str, ...]:
)
def _provider_config(model: str, custom_llm_provider: str) -> BaseDecisionsConfig:
def get_decisions_provider_config(model: str, custom_llm_provider: str) -> BaseDecisionsConfig:
provider: Final = next((member for member in LlmProviders if member.value == custom_llm_provider), None)
provider_config: Final = (
None if provider is None else ProviderConfigManager.get_provider_decisions_config(model, provider)
@ -257,19 +258,105 @@ def _request_safety_identifier(
return None
def _fields_without_unsupported_media(
fields: RequestFields,
def _bool_or_none(value: object) -> bool | None:
return value if isinstance(value, bool) else None
def _str_or_none(value: object) -> str | None:
return value if isinstance(value, str) else None
_DeploymentFlag: TypeAlias = Annotated[bool | None, BeforeValidator(_bool_or_none)]
class _DeploymentCapabilities(BaseModel):
model_config = ConfigDict(frozen=True, from_attributes=True)
mode: Annotated[str | None, BeforeValidator(_str_or_none)] = None
supports_box_questions: _DeploymentFlag = None
supports_audio_input: _DeploymentFlag = None
supports_video_input: _DeploymentFlag = None
def decisions_media_flag(self, flag: bool | None, model: str, custom_llm_provider: str) -> bool | None:
if flag is None:
return None
mode: Final = self.mode if self.mode is not None else catalog_mode(model, custom_llm_provider)
return flag if mode is None or is_decisions_model_mode(mode) else None
@dataclass(frozen=True, slots=True)
class _ResolvedSupport:
box_questions: bool
audio: bool
videos: bool
def _takes_audio(
provider_config: BaseDecisionsConfig, model: str, custom_llm_provider: str, deployment: _DeploymentCapabilities
) -> bool:
return provider_config.supports_audio_input(
model,
custom_llm_provider,
deployment.decisions_media_flag(deployment.supports_audio_input, model, custom_llm_provider),
)
def _takes_videos(
provider_config: BaseDecisionsConfig, model: str, custom_llm_provider: str, deployment: _DeploymentCapabilities
) -> bool:
return provider_config.supports_video_input(
model,
custom_llm_provider,
deployment.decisions_media_flag(deployment.supports_video_input, model, custom_llm_provider),
)
def deployment_refuses_media(
model: str,
custom_llm_provider: str,
model_info: Mapping[str, object],
call_kwargs: Mapping[str, object],
*,
audio: bool,
videos: bool,
) -> bool:
if _drops_params(call_kwargs):
return False
provider_config: Final = get_decisions_provider_config(model, custom_llm_provider)
canonical_model: Final = provider_config.canonical_model(model)
deployment: Final = _DeploymentCapabilities.model_validate(model_info)
return (audio and not _takes_audio(provider_config, canonical_model, custom_llm_provider, deployment)) or (
videos and not _takes_videos(provider_config, canonical_model, custom_llm_provider, deployment)
)
def _resolved_support(
provider_config: BaseDecisionsConfig,
model: str,
custom_llm_provider: str,
headers: Mapping[str, str],
kwargs: Mapping[str, object],
fields: RequestFields,
) -> _ResolvedSupport:
deployment: Final = _DeploymentCapabilities.model_validate(kwargs.get("model_info") or {})
return _ResolvedSupport(
box_questions=provider_config.supports_box_questions(
model, custom_llm_provider, headers, deployment.supports_box_questions
),
audio=bool(fields.audio) and _takes_audio(provider_config, model, custom_llm_provider, deployment),
videos=bool(fields.videos) and _takes_videos(provider_config, model, custom_llm_provider, deployment),
)
def _fields_without_unsupported_media(
fields: RequestFields, support: _ResolvedSupport, kwargs: Mapping[str, object]
) -> RequestFields:
if not _drops_params(kwargs):
return fields
return replace(
fields,
audio=fields.audio if provider_config.supports_audio_input(model, custom_llm_provider) else None,
videos=fields.videos if provider_config.supports_video_input(model, custom_llm_provider) else None,
audio=fields.audio if support.audio else None,
videos=fields.videos if support.videos else None,
)
@ -292,10 +379,8 @@ class _MalformedImages:
def _unsupported_params(
ir_request: DecisionsIRRequest,
provider_config: BaseDecisionsConfig,
model: str,
custom_llm_provider: str,
headers: Mapping[str, str],
supports_box_questions: bool,
support: _ResolvedSupport,
) -> tuple[str, ...]:
refuses_safety_identifier: Final = (
ir_request.safety_identifier is not None and not provider_config.supports_safety_identifier
@ -304,13 +389,9 @@ def _unsupported_params(
None if provider_config.supports_images_in(ir_request.input, headers) else image_param(ir_request.input)
)
refuses_assets: Final = bool(ir_request.assets) and not provider_config.supports_assets(headers)
refuses_box_questions: Final = _has_box_question(ir_request.questions) and not supports_box_questions
refuses_audio: Final = bool(ir_request.audio) and not provider_config.supports_audio_input(
model, custom_llm_provider
)
refuses_videos: Final = bool(ir_request.videos) and not provider_config.supports_video_input(
model, custom_llm_provider
)
refuses_box_questions: Final = _has_box_question(ir_request.questions) and not support.box_questions
refuses_audio: Final = bool(ir_request.audio) and not support.audio
refuses_videos: Final = bool(ir_request.videos) and not support.videos
return (
*(("safety_identifier",) if refuses_safety_identifier else ()),
*(() if refused_image_param is None else (refused_image_param,)),
@ -333,7 +414,7 @@ def _without_unsupported_params(
ir_request: DecisionsIRRequest,
provider_config: BaseDecisionsConfig,
headers: Mapping[str, str],
supports_box_questions: bool,
support: _ResolvedSupport,
) -> DecisionsIRRequest:
return replace(
ir_request,
@ -345,9 +426,7 @@ def _without_unsupported_params(
safety_identifier=ir_request.safety_identifier if provider_config.supports_safety_identifier else None,
assets=ir_request.assets if provider_config.supports_assets(headers) else MappingProxyType({}),
questions=(
ir_request.questions
if supports_box_questions
else tuple(filter(_is_not_box_question, ir_request.questions))
ir_request.questions if support.box_questions else tuple(filter(_is_not_box_question, ir_request.questions))
),
)
@ -368,39 +447,20 @@ def _without_malformed_images(
return well_formed
class _DeploymentBoxQuestions(BaseModel):
model_config = ConfigDict(frozen=True, from_attributes=True)
supports_box_questions: StrictBool | None = None
def _deployment_supports_box_questions(kwargs: Mapping[str, object]) -> bool | None:
try:
return _DeploymentBoxQuestions.model_validate(kwargs.get("model_info") or {}).supports_box_questions
except ValidationError:
return None
def _provider_ir_request(
ir_request: DecisionsIRRequest,
*,
provider_config: BaseDecisionsConfig,
model: str,
custom_llm_provider: str,
headers: Mapping[str, str],
support: _ResolvedSupport,
kwargs: Mapping[str, object],
) -> DecisionsIRRequest | _UnsupportedParams | _NothingLeftAfterDrop:
supports_box_questions: Final = provider_config.supports_box_questions(
model, custom_llm_provider, headers, _deployment_supports_box_questions(kwargs)
)
unsupported: Final = _unsupported_params(
ir_request, provider_config, model, custom_llm_provider, headers, supports_box_questions
)
unsupported: Final = _unsupported_params(ir_request, provider_config, headers, support)
if not unsupported:
return ir_request
if not _drops_params(kwargs):
return _UnsupportedParams(names=unsupported)
supported: Final = _without_unsupported_params(ir_request, provider_config, headers, supports_box_questions)
supported: Final = _without_unsupported_params(ir_request, provider_config, headers, support)
if _leaves_no_input(supported.input):
return _NothingLeftAfterDrop(names=unsupported, missing="input to decide on")
if not supported.questions:
@ -408,7 +468,7 @@ def _provider_ir_request(
return supported
def _outbound_headers(
def decisions_outbound_headers(
extra_headers: Mapping[str, str] | None, custom_llm_provider: str, kwargs: Mapping[str, object]
) -> Mapping[str, str]:
forwarded: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(
@ -466,7 +526,7 @@ def prepare_call(
api_base=call_api_base,
api_key=api_key,
)
provider_config: Final = _provider_config(upstream_model, provider)
provider_config: Final = get_decisions_provider_config(upstream_model, provider)
if not upstream_model:
raise litellm.BadRequestError(
message="A model name is required for the Decisions API",
@ -478,11 +538,12 @@ def prepare_call(
except ValueError as error:
raise litellm.BadRequestError(message=str(error), model=model, llm_provider=provider) from error
try:
headers: Final = _outbound_headers(extra_headers, provider, kwargs)
headers: Final = decisions_outbound_headers(extra_headers, provider, kwargs)
support: Final = _resolved_support(provider_config, canonical_model, provider, headers, kwargs, fields)
request_safety_identifier: Final = _request_safety_identifier(safety_identifier, provider_config, kwargs)
request: Final = _fields_without_unsupported_media(
fields, provider_config, canonical_model, provider, kwargs
).validate(safety_identifier=request_safety_identifier, drops_params=_drops_params(kwargs))
request: Final = _fields_without_unsupported_media(fields, support, kwargs).validate(
safety_identifier=request_safety_identifier, drops_params=_drops_params(kwargs)
)
except ValidationError as error:
raise litellm.BadRequestError(
message=f"Invalid Decisions request: {error}",
@ -523,12 +584,7 @@ def prepare_call(
llm_provider=provider,
)
ir_request: Final = _provider_ir_request(
caller_ir_request,
provider_config=provider_config,
model=canonical_model,
custom_llm_provider=provider,
headers=headers,
kwargs=kwargs,
caller_ir_request, provider_config=provider_config, headers=headers, support=support, kwargs=kwargs
)
if isinstance(ir_request, _UnsupportedParams):
raise litellm.UnsupportedParamsError(

2
litellm/endpoints/speech/speech_to_completion_bridge/handler.py
View file

@ -113,7 +113,7 @@ class SpeechToCompletionBridgeHandler:
voice=voice,
)
with internal_sub_call():
with internal_sub_call(logging_obj.litellm_call_id):
result: Final = completion(
**request_data,
)

66
litellm/guardrails.py Normal file
View file

@ -0,0 +1,66 @@
"""Invoke configured gateway guardrails without generating a completion."""
from collections.abc import Mapping, Sequence
from enum import Enum
from typing import Final
import httpx
from pydantic import TypeAdapter
import litellm
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client, # pyright: ignore[reportUnknownVariableType] # shared factory has untyped optional parameters
)
from litellm.types.guardrails import NativeGuardrailResponse
_NATIVE_RESPONSE: Final[TypeAdapter[NativeGuardrailResponse]] = TypeAdapter(NativeGuardrailResponse)
class _InputType(Enum):
REQUEST = "request"
RESPONSE = "response"
async def apply(
guardrail_name: str,
*,
input: str | None = None,
response: str | None = None,
messages: Sequence[Mapping[str, object]] | None = None,
metadata: Mapping[str, object] | None = None,
api_base: str | None = None,
api_key: str | None = None,
timeout: float | httpx.Timeout | None = None,
) -> NativeGuardrailResponse:
"""Return a configured guardrail's native structured findings.
Supply exactly one of ``input`` or ``response``. Grounding guardrails can
use ``messages`` as evidence. Connection settings default to
``litellm.api_base`` and ``litellm.api_key``.
"""
if (input is None) == (response is None):
raise ValueError("Supply exactly one of input or response.")
resolved_base: Final = api_base if api_base is not None else litellm.api_base
if not resolved_base:
raise ValueError("Set api_base or litellm.api_base to your LiteLLM gateway URL.")
resolved_key: Final = api_key if api_key is not None else litellm.api_key
input_type: Final = _InputType.RESPONSE if response is not None else _InputType.REQUEST
payload: Final = dict[str, object](
guardrail_name=guardrail_name,
text=response if response is not None else input,
input_type=input_type.value,
response_format="native",
)
if messages is not None:
payload["messages"] = [dict(message) for message in messages]
if metadata is not None:
payload["metadata"] = dict(metadata)
gateway_root: Final = resolved_base.rstrip("/").removesuffix("/v1")
client: Final = get_async_httpx_client(llm_provider="litellm_guardrails")
result: Final = await client.post( # pyright: ignore[reportUnknownMemberType] # shared handler has untyped optional parameters
url=f"{gateway_root}/guardrails/apply_guardrail",
json=payload,
headers={"Authorization": f"Bearer {resolved_key}"} if resolved_key else {},
timeout=timeout,
)
return _NATIVE_RESPONSE.validate_json(result.content)

261
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -13,8 +13,8 @@ import copy
import itertools
import os
import re
from collections.abc import Callable, Container, Iterable, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, cast
from collections.abc import Callable, Container, Iterable, Iterator, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, TypeAlias, TypeVar, cast
from urllib.parse import urlparse
from pydantic import TypeAdapter, ValidationError
@ -32,6 +32,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
tool_call_ids_answered_by_client,
tool_call_ids_answered_in,
)
from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import string_field
from litellm.llms.anthropic.common_utils import (
is_claude_code_one_shot_subagent_request,
supports_anthropic_cache_control,
@ -102,6 +103,192 @@ def _validated_object_list(value: object) -> list[object] | None:
return None
_MessageT: Final = TypeVar("_MessageT")
_CacheControlRewrite: TypeAlias = Callable[[object], ChatCompletionCachedContent | None]
_MARK_CARRYING_MESSAGE_FIELDS: Final = ("content", "tool_calls")
def _mark_carrying_blocks(fields: Mapping[str, object]) -> Sequence[object]:
lists: Final = tuple(fields.get(key) for key in _MARK_CARRYING_MESSAGE_FIELDS)
return tuple(
itertools.chain.from_iterable(
_OBJECT_LIST_ADAPTER.validate_python(value) for value in lists if isinstance(value, list)
)
)
def _block_carries_cache_control(block: object) -> bool:
return isinstance(block, dict) and _CONTENT_PART_ADAPTER.validate_python(block).get("cache_control") is not None
_ROLES_SENDING_MESSAGE_MARK_WITH_LIST_CONTENT: Final = frozenset({"tool", "function"})
def _message_mark_reaches_provider(fields: Mapping[str, object]) -> bool:
if fields.get("cache_control") is None:
return False
return (
not isinstance(fields.get("content"), list)
or string_field(fields, "role") in _ROLES_SENDING_MESSAGE_MARK_WITH_LIST_CONTENT
)
def cache_control_count(message: object) -> int:
if not isinstance(message, dict):
return 0
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
block_marks: Final = sum(1 for block in _mark_carrying_blocks(fields) if _block_carries_cache_control(block))
return int(_message_mark_reaches_provider(fields)) + block_marks
def _fields_with_cache_control(fields: Mapping[str, object], rewrite: _CacheControlRewrite) -> Mapping[str, object]:
others: Final = {key: value for key, value in fields.items() if key != "cache_control"}
if "cache_control" not in fields:
return others
cache_control: Final = rewrite(fields["cache_control"])
return others if cache_control is None else {**others, "cache_control": cache_control}
def block_with_cache_control(block: object, rewrite: _CacheControlRewrite) -> object:
if not isinstance(block, dict):
return block
return _fields_with_cache_control(_CONTENT_PART_ADAPTER.validate_python(block), rewrite)
def with_cache_control(message: _MessageT, rewrite: _CacheControlRewrite) -> _MessageT:
if not isinstance(message, dict):
return message
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
blocks_rewritten: Final = {
key: (
[block_with_cache_control(block, rewrite) for block in _OBJECT_LIST_ADAPTER.validate_python(value)]
if key in _MARK_CARRYING_MESSAGE_FIELDS and isinstance(value, list)
else value
)
for key, value in fields.items()
}
return cast( # cast-ok: the same message shape with each cache marker rewritten or dropped
_MessageT, _fields_with_cache_control(blocks_rewritten, rewrite)
)
def dropped_cache_control(_: object) -> None:
return None
def _first_marked_index(value: object) -> int | None:
if not isinstance(value, list):
return None
blocks: Final = _OBJECT_LIST_ADAPTER.validate_python(value)
return next((index for index, block in enumerate(blocks) if _block_carries_cache_control(block)), None)
_MarkSlot: TypeAlias = tuple[str, int]
_MESSAGE_MARK_SLOT: Final[_MarkSlot] = ("cache_control", -1)
def _block_mark_slots(fields: Mapping[str, object], key: str) -> tuple[_MarkSlot, ...]:
value: Final = fields.get(key)
if not isinstance(value, list):
return ()
blocks: Final = _OBJECT_LIST_ADAPTER.validate_python(value)
return tuple((key, index) for index, block in enumerate(blocks) if _block_carries_cache_control(block))
def _mark_slots_in_wire_order(fields: Mapping[str, object]) -> tuple[_MarkSlot, ...]:
content_slots: Final = _block_mark_slots(fields, "content")
tool_call_slots: Final = _block_mark_slots(fields, "tool_calls")
if not _message_mark_reaches_provider(fields):
return content_slots + tool_call_slots
if isinstance(fields.get("content"), list):
return content_slots + (_MESSAGE_MARK_SLOT,) + tool_call_slots
return (_MESSAGE_MARK_SLOT,) + content_slots + tool_call_slots
def _blocks_without_marks(key: str, value: object, dropped_slots: frozenset[_MarkSlot]) -> object:
if key not in _MARK_CARRYING_MESSAGE_FIELDS or not isinstance(value, list):
return value
return [
block_with_cache_control(block, dropped_cache_control) if (key, index) in dropped_slots else block
for index, block in enumerate(_OBJECT_LIST_ADAPTER.validate_python(value))
]
def _without_oldest_marks(message: _MessageT, dropped: int) -> _MessageT:
if dropped == 0 or not isinstance(message, dict):
return message
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
dropped_slots: Final = frozenset(_mark_slots_in_wire_order(fields)[:dropped])
blocks_unmarked: Final = {key: _blocks_without_marks(key, value, dropped_slots) for key, value in fields.items()}
return cast( # cast-ok: the same message with its oldest wire-order marks dropped
_MessageT,
_fields_with_cache_control(blocks_unmarked, dropped_cache_control)
if _MESSAGE_MARK_SLOT in dropped_slots
else blocks_unmarked,
)
def tool_message_with_single_cache_mark(message: _MessageT) -> _MessageT:
if not isinstance(message, dict):
return message
fields: Final = _CONTENT_PART_ADAPTER.validate_python(message)
content: Final = fields.get("content")
if fields.get("role") != "tool" or not isinstance(content, list):
return message
kept_index: Final = None if fields.get("cache_control") is not None else _first_marked_index(content)
return cast( # cast-ok: the same tool message keeping only the mark its single cache point reads
_MessageT,
{
**fields,
"content": [
block if index == kept_index else block_with_cache_control(block, dropped_cache_control)
for index, block in enumerate(_OBJECT_LIST_ADAPTER.validate_python(content))
],
},
)
_BudgetSpend: TypeAlias = Callable[[tuple[int, int], int], tuple[int, int]]
def _newest_marks_kept(state: tuple[int, int], marks: int) -> tuple[int, int]:
remaining, _ = state
kept: Final = min(marks, remaining)
return remaining - kept, marks - kept
def _whole_message_kept(state: tuple[int, int], marks: int) -> tuple[int, int]:
remaining, _ = state
if marks > remaining:
return remaining, marks
return remaining - marks, 0
def _messages_within_budget(
messages: Sequence[_MessageT], reserved_marks: int, spend: _BudgetSpend
) -> Sequence[_MessageT]:
budget: Final = max(MAX_CACHE_CONTROL_BLOCKS - reserved_marks, 0)
newest_first: Final = tuple(reversed(messages))
budget_after_each: Final = itertools.accumulate(
(cache_control_count(message) for message in newest_first), spend, initial=(budget, 0)
)
dropped_counts: Final = tuple(dropped for _, dropped in itertools.islice(budget_after_each, 1, None))
kept_newest_first: Final = tuple(
_without_oldest_marks(message, dropped) for message, dropped in zip(newest_first, dropped_counts)
)
return tuple(reversed(kept_newest_first))
def newest_marks_within_cache_control_limit(messages: Sequence[_MessageT], reserved_marks: int) -> Sequence[_MessageT]:
return _messages_within_budget(messages, reserved_marks, _newest_marks_kept)
def whole_messages_within_cache_control_limit(
messages: Sequence[_MessageT], reserved_marks: int
) -> Sequence[_MessageT]:
return _messages_within_budget(messages, reserved_marks, _whole_message_kept)
def _part_with_cache_control_renamed(part: object) -> object:
if not _accepts_prompt_cache_breakpoint(part):
return part
@ -259,22 +446,41 @@ def _as_object_list(value: object | None) -> list[object] | None:
return _validated_object_list(value)
def _tool_call_carries_cache_breakpoint(tool_call: object, message: object, answered_tool_ids: Container[str]) -> bool:
def _server_tool_call_id(tool_call: object) -> str | None:
tool_call_id: Final = _attribute_or_key(tool_call, "id")
if isinstance(tool_call_id, str) and tool_call_id.startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX):
return tool_call_id
return None
def _tool_call_carries_cache_breakpoint(tool_call: object, client_tool_call_ids: Container[str]) -> bool:
if _attribute_or_key(tool_call, "cache_control") is None:
return False
server_tool_call_id: Final = _server_tool_call_id(tool_call)
return server_tool_call_id is None or server_tool_call_id in client_tool_call_ids
return not tool_call_is_rebuilt_as_server_tool_use(
_attribute_or_key(tool_call, "id"), _attribute_or_key(message, "provider_specific_fields"), answered_tool_ids
def _server_tool_call_ids_without_a_kept_result(message: object) -> Iterator[str]:
provider_specific_fields: Final = _attribute_or_key(message, "provider_specific_fields")
tool_calls: Final = _as_object_list(_attribute_or_key(message, "tool_calls")) or ()
return (
tool_call_id
for tool_call in tool_calls
if (tool_call_id := _server_tool_call_id(tool_call)) is not None
and not tool_call_is_rebuilt_as_server_tool_use(tool_call_id, provider_specific_fields)
)
def _tool_call_ids_answered_by_client_when_a_server_tool_call_is_marked(
def _client_tool_call_ids_when_a_server_tool_call_is_marked(
messages: Sequence[object], model: str | None = None, custom_llm_provider: str | None = None
) -> frozenset[str]:
if not _marks_a_server_tool_call(messages):
return frozenset()
if model is not None and not _builds_anthropic_messages(model, custom_llm_provider):
return tool_call_ids_answered_in(messages)
server_calls_sent_as_client_calls: Final = itertools.chain.from_iterable(
map(_server_tool_call_ids_without_a_kept_result, messages)
)
return tool_call_ids_answered_in(messages) | frozenset(server_calls_sent_as_client_calls)
return tool_call_ids_answered_by_client(messages, _keeps_mid_conversation_system(model, custom_llm_provider))
@ -331,12 +537,7 @@ def _model_info_flags_mid_conversation_system(model: str, custom_llm_provider: s
def _is_marked_server_tool_call(tool_call: object) -> bool:
tool_call_id: Final = _attribute_or_key(tool_call, "id")
return (
isinstance(tool_call_id, str)
and tool_call_id.startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX)
and _attribute_or_key(tool_call, "cache_control") is not None
)
return _server_tool_call_id(tool_call) is not None and _attribute_or_key(tool_call, "cache_control") is not None
def _tool_carries_cache_breakpoint(tool: object) -> bool:
@ -482,7 +683,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
reserved_blocks: Final = AnthropicCacheControlHook._blocks_reserved_outside_messages(
remaining_points, external_breakpoints, openai_dialect
)
answered_tool_ids: Final = _tool_call_ids_answered_by_client_when_a_server_tool_call_is_marked(
answered_tool_ids: Final = _client_tool_call_ids_when_a_server_tool_call_is_marked(
processed_messages, model, custom_llm_provider
)
breakpoints_before: Final = AnthropicCacheControlHook.count_request_cache_breakpoints(
@ -571,7 +772,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
)
message_list: Final = tuple(messages)
client_answers: Final = (
_tool_call_ids_answered_by_client_when_a_server_tool_call_is_marked(message_list)
_client_tool_call_ids_when_a_server_tool_call_is_marked(message_list)
if answered_tool_ids is None
else answered_tool_ids
)
@ -663,7 +864,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
so earlier points win when slots are scarce.
"""
client_answers: Final = (
_tool_call_ids_answered_by_client_when_a_server_tool_call_is_marked(messages)
_client_tool_call_ids_when_a_server_tool_call_is_marked(messages)
if answered_tool_ids is None
else answered_tool_ids
)
@ -753,11 +954,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
content_count: Final = sum(1 for block in content if _carries_cache_breakpoint(block)) if content else 0
tool_calls: Final = _as_object_list(_attribute_or_key(message, "tool_calls"))
tool_call_count: Final = (
sum(
1
for tool_call in tool_calls
if _tool_call_carries_cache_breakpoint(tool_call, message, answered_tool_ids)
)
sum(1 for tool_call in tool_calls if _tool_call_carries_cache_breakpoint(tool_call, answered_tool_ids))
if tool_calls
else 0
)
@ -1439,3 +1636,25 @@ class AnthropicCacheControlHook(CustomPromptManagement):
llm_router=None,
)
return None
def messages_within_cache_control_cap(
messages: Sequence[_MessageT], tools: object, cache_control: object = None, request_kwargs: object = None
) -> Sequence[_MessageT]:
"""Under drop_params, drop the oldest message hints the provider's cache_control cap has no room for.
Tool hints and a top-level ``cache_control`` stay on the wire, so they are reserved first.
"""
reserved_marks: Final = AnthropicCacheControlHook.count_external_cache_breakpoints(
_validated_object_list(tools), cache_control, request_kwargs
)
kept: Final = newest_marks_within_cache_control_limit(messages, reserved_marks)
dropped: Final = sum(map(cache_control_count, messages)) - sum(map(cache_control_count, kept))
if dropped > 0:
verbose_logger.warning(
"Dropped %d cache_control hint(s) from the oldest messages because drop_params is on and the "
"provider accepts at most %d cache_control blocks per request",
dropped,
MAX_CACHE_CONTROL_BLOCKS,
)
return kept

115
litellm/integrations/custom_guardrail.py
View file

@ -71,6 +71,8 @@ GUARDRAIL_SESSIONS_TARGET: Final = "guardrail_sessions"
# field to suppress a guardrail on the direct-SDK path that never reaches the
# proxy's metadata sanitizer.
_PRE_CALL_EXECUTED_TOKEN: Final = secrets.token_hex(16)
_PRE_CALL_RAISED_TOKEN: Final = secrets.token_hex(16)
_DEPLOYMENT_HOOK_RAN_TOKEN: Final = secrets.token_hex(16)
_GUARDRAIL_BLOCK_STATUS_CODES: Final = frozenset({400, 403, 422})
@ -260,6 +262,25 @@ def _request_is_streaming(data: object, event_type: GuardrailEventHooks | None =
)
def _str_collection(value: object) -> frozenset[str]:
if not isinstance(value, (list, tuple, set, frozenset)):
return frozenset()
items: Final = cast("Iterable[object]", value) # cast-ok: a metadata collection of guardrail names
return frozenset(item for item in items if isinstance(item, str))
def _request_carries_marker(data: Mapping[str, object], marker: str | None) -> bool:
if marker is None:
return False
for meta_key in ("metadata", "litellm_metadata"):
meta = data.get(meta_key)
if isinstance(meta, dict):
executed = meta.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if isinstance(executed, list) and marker in executed:
return True
return False
_REQUEST_FIELDS: Final = TypeAdapter(Mapping[str, object])
@ -869,13 +890,16 @@ class CustomGuardrail(CustomLogger):
return False
def _pre_call_marker(self) -> str | None:
def _marker(self, token: str) -> str | None:
name: Final = self.guardrail_name
if not name:
return None
return f"{_PRE_CALL_EXECUTED_TOKEN}:{name}"
return f"{token}:{name}"
def mark_pre_call_hook_ran(self, data: dict[str, object]) -> None:
def _pre_call_marker(self, *, raised: bool = False) -> str | None:
return self._marker(_PRE_CALL_RAISED_TOKEN if raised else _PRE_CALL_EXECUTED_TOKEN)
def mark_pre_call_hook_ran(self, data: dict[str, object], *, raised: bool = False) -> None:
"""
Record that this guardrail's ``async_pre_call_hook`` already ran for this
request, so the deployment-level hook does not run it a second time.
@ -884,8 +908,11 @@ class CustomGuardrail(CustomLogger):
router later spreads a deployment's model-level ``guardrails`` into the
top-level request kwargs, which would otherwise re-trigger the same hook
from ``async_pre_call_deployment_hook``.
With ``raised``, records instead that the hook raised as a policy pipeline
step the pipeline then moved past. The deployment-level hook still runs it.
"""
marker: Final = self._pre_call_marker()
marker: Final = self._pre_call_marker(raised=raised)
if marker is None:
return
for meta_key in ("metadata", "litellm_metadata"):
@ -900,17 +927,57 @@ class CustomGuardrail(CustomLogger):
return
data["metadata"] = {PRE_CALL_EXECUTED_GUARDRAILS_KEY: [marker]}
def _pre_call_hook_already_ran(self, data: dict[str, object]) -> bool:
marker: Final = self._pre_call_marker()
if marker is None:
return False
for meta_key in ("metadata", "litellm_metadata"):
meta = data.get(meta_key)
if isinstance(meta, dict):
executed = meta.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if isinstance(executed, list) and marker in executed:
return True
return False
def pre_call_hook_ran_or_raised(self, data: Mapping[str, object]) -> bool:
"""Whether this guardrail's ``async_pre_call_hook`` ran for this request, whatever it decided."""
return self._pre_call_hook_already_ran(data) or self._pre_call_hook_already_ran(data, raised=True)
def _pre_call_hook_already_ran(self, data: Mapping[str, object], *, raised: bool = False) -> bool:
return _request_carries_marker(data, self._pre_call_marker(raised=raised))
def deployment_hook_ran(self, data: Mapping[str, object]) -> bool:
"""Whether a deployment-level hook ran this guardrail as a policy pipeline step of this request."""
return _request_carries_marker(data, self._marker(_DEPLOYMENT_HOOK_RAN_TOKEN))
def _record_deployment_hook_run(
self,
data: dict[str, object], # mutable-ok: the request payload the run is recorded on
) -> None:
"""
Record that a deployment-level hook ran this guardrail when it is a policy pipeline step of
this request, and name it in the applied guardrails when the request's metadata lists it.
"""
name: Final = self.guardrail_name
marker: Final = self._marker(_DEPLOYMENT_HOOK_RAN_TOKEN)
buckets: Final = tuple(
cast("dict[str, object]", bucket) # cast-ok: metadata buckets are str-keyed
for bucket in (data.get("metadata"), data.get("litellm_metadata"))
if isinstance(bucket, dict)
)
policy_state: Final = next(
(bucket for bucket in buckets if name in _str_collection(bucket.get("_pipeline_managed_guardrails"))),
None,
)
if marker is None or policy_state is None:
return
recorded: Final = policy_state.get(PRE_CALL_EXECUTED_GUARDRAILS_KEY)
if not isinstance(recorded, list):
policy_state[PRE_CALL_EXECUTED_GUARDRAILS_KEY] = [marker]
elif marker not in recorded:
cast("list[object]", recorded).append(marker) # cast-ok: markers are only compared as strings
if not any(name in _str_collection(bucket.get("guardrails")) for bucket in buckets):
return
named_lists: Final = tuple(
cast("list[object]", bucket["applied_guardrails"]) # cast-ok: entries are only compared with names
for bucket in buckets
if isinstance(bucket.get("applied_guardrails"), list)
)
for applied in named_lists:
if name not in applied:
applied.append(name)
if named_lists:
return
metadata_bucket: Final = cast("dict[str, object]", get_or_create_metadata_bucket(data)[1]) # cast-ok: str keys
metadata_bucket["applied_guardrails"] = [name]
@classmethod
def uses_apply_guardrail_interface(cls) -> bool:
@ -959,12 +1026,15 @@ class CustomGuardrail(CustomLogger):
target: Final = self._deployment_hook_target()
if target is not self:
kwargs["guardrail_to_apply"] = self
result: Final = await target.async_pre_call_hook(
user_api_key_dict=_user_api_key_auth_from_request(kwargs),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
try:
result: Final = await target.async_pre_call_hook(
user_api_key_dict=_user_api_key_auth_from_request(kwargs),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
)
finally:
self._record_deployment_hook_run(kwargs)
if result is not None and isinstance(result, dict):
result_messages: Final = result.get("messages")
@ -1048,6 +1118,9 @@ class CustomGuardrail(CustomLogger):
finally:
if target is not self:
request_data.pop("guardrail_to_apply", None)
self._record_deployment_hook_run(
cast("dict[str, object]", request_data) # cast-ok: proxy request payloads are str-keyed
)
if not self._is_valid_response_type(result):
return None

5
litellm/integrations/otel/routing.py
View file

@ -21,6 +21,11 @@ _ROUTING_SCALAR_FIELDS: Final = frozenset(
"classifier_cost",
"classifier_failure_reason",
"classifier_error_type",
"classifier_input_budget_tokens",
"classifier_input_tokens",
"classifier_input_count_method",
"classifier_input_truncated",
"classifier_input_budget_status",
"classifier_confidence",
"classifier_primary_rule",
"classifier_capability_boundary",

8
litellm/integrations/prometheus.py
View file

@ -66,7 +66,7 @@ from litellm.repositories.organization_repository import OrganizationRepository
from litellm.repositories.table_repositories import EndUserRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
from litellm.router_api import deployment_metadata
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import (
@ -3022,8 +3022,8 @@ class PrometheusLogger(CustomLogger):
).get("custom_llm_provider")
if isinstance(provider, str) and provider:
return provider
deployment_metadata: Final = find_deployment_metadata(litellm_params)
return _deployment_model_provider(deployment_metadata.get("deployment") if deployment_metadata else None)
picked_deployment: Final = deployment_metadata(litellm_params)
return _deployment_model_provider(picked_deployment.get("deployment") if picked_deployment else None)
@staticmethod
def _extract_api_provider_from_exception(exception: Exception) -> str | None:
@ -5209,7 +5209,7 @@ def _tag_matches_wildcard_configured_pattern(tags: Sequence[str], configured_tag
"""
import re
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
pattern_router: Final = PatternMatchRouter()
regex_pattern: Final = pattern_router.pattern_to_regex(configured_tag)

0
litellm/router_utils/access_windows.py → litellm/litellm_core_utils/access_windows.py
View file

0
litellm/router_utils/add_retry_fallback_headers.py → litellm/litellm_core_utils/add_retry_fallback_headers.py
View file

0
litellm/router_utils/clientside_credential_handler.py → litellm/litellm_core_utils/clientside_credential_handler.py
View file

10
litellm/litellm_core_utils/core_helpers.py
View file

@ -311,9 +311,12 @@ _FINISH_REASON_MAP: Final[dict[str, OpenAIChatCompletionFinishReason]] = {
"tool_use": "tool_calls",
"refusal": "content_filter",
"compaction": "length",
"pause_turn": "length",
"model_context_window_exceeded": "length",
# Cohere
"COMPLETE": "stop",
"STOP_SEQUENCE": "stop",
"TOOL_CALL": "tool_calls",
"ERROR_TOXIC": "content_filter",
"ERROR": "stop",
# HuggingFace / Together AI
@ -1073,9 +1076,14 @@ def billed_usage_cost(response: object, custom_llm_provider: str | None) -> floa
_MESSAGES_REPORTED_COST_PROVIDERS: Final = frozenset({"opencode_go"})
def reports_native_messages_cost(custom_llm_provider: object) -> bool:
"""Whether a Messages upstream is one whose own reported cost litellm bills from instead of the price map."""
return isinstance(custom_llm_provider, str) and custom_llm_provider in _MESSAGES_REPORTED_COST_PROVIDERS
def native_messages_reported_cost(usage: object, custom_llm_provider: object) -> float | None:
"""Only a Messages upstream known to price its own responses is trusted, every other one keeps the price map."""
if not isinstance(custom_llm_provider, str) or custom_llm_provider not in _MESSAGES_REPORTED_COST_PROVIDERS:
if not isinstance(custom_llm_provider, str) or not reports_native_messages_cost(custom_llm_provider):
return None
return provider_reported_usage_cost(usage, custom_llm_provider)

0
litellm/router_utils/deployment_pricing.py → litellm/litellm_core_utils/deployment_pricing.py
View file

6
litellm/litellm_core_utils/fallback_utils.py
View file

@ -3,13 +3,13 @@ from typing import Final
import litellm
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
)
from litellm.litellm_core_utils.core_helpers import (
filter_internal_params,
safe_deep_copy,
)
from litellm.router_utils.add_retry_fallback_headers import (
add_fallback_headers_to_response,
)
from .asyncify import run_async_function

3
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -84,6 +84,7 @@ OPTIONAL_KWARGS_KEYS: Final = (
"github_copilot_host",
"github_copilot_user_session",
"fireworks_forward_user_id",
"thought_signature_in_tool_call_id",
"responses_api_version",
"allowed_openai_params",
PROVIDER_AFFINITY_HEADER_KWARG_KEY,
@ -99,6 +100,8 @@ OPTIONAL_KWARGS_KEYS: Final = (
# Backward-compatible alias for existing imports/tests.
_OPTIONAL_KWARGS_KEYS: Final = OPTIONAL_KWARGS_KEYS
SUPPLEMENTAL_KWARGS_KEYS: Final = OPTIONAL_KWARGS_KEYS | {"provider_specific_header", "extra_headers"}
_CONTROL_OPTIONS: Final = TypeAdapter(ControlOptions)
_CONTROL_OPTION_NAMES: Final = tuple(field.name for field in fields(ControlOptions))
_MAX_SHOWN_INT_BITS: Final = 64

0
litellm/router_utils/get_retry_from_policy.py → litellm/litellm_core_utils/get_retry_from_policy.py
View file

2
litellm/litellm_core_utils/health_check_helpers.py
View file

@ -119,7 +119,7 @@ def _is_route_segment(custom_llm_provider: str, segment: str, model_after_segmen
def wildcard_health_check_models(wildcard: str, custom_llm_provider: str) -> tuple[str, ...]:
from litellm.litellm_core_utils.llm_request_utils import rank_chat_models_from_llm_provider
from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.litellm_core_utils.pattern_match_deployments import PatternMatchRouter
provider_prefix: Final = f"{custom_llm_provider}/"
segments: Final = wildcard.removeprefix(provider_prefix).split("/")

318
litellm/litellm_core_utils/litellm_logging.py
View file

@ -16,9 +16,10 @@ import traceback
from collections.abc import Awaitable, Callable, Coroutine, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from datetime import datetime as dt_object
from enum import Enum
from functools import lru_cache
from types import MappingProxyType, TracebackType
from typing import TYPE_CHECKING, Any, Final, Literal, NamedTuple, cast
from typing import TYPE_CHECKING, Any, ClassVar, Final, Literal, NamedTuple, cast
from httpx import Response, Timeout
from pydantic import BaseModel, JsonValue, TypeAdapter
@ -47,6 +48,7 @@ from litellm.constants import (
KUBERNETES_POD_ROUTING_KEY,
LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL,
PROVIDER_REQUEST_ID_HEADERS,
REALTIME_METERED_USAGE_KEY,
REDACTED_BY_LITELLM,
)
from litellm.cost_calculator import (
@ -84,6 +86,7 @@ from litellm.litellm_core_utils.core_helpers import (
proxy_stamped_kubernetes_pod_routing,
proxy_stamped_used_client_oauth_token,
reconstruct_model_name,
reports_native_messages_cost,
set_response_cost_in_hidden_params,
single_log_line,
)
@ -163,6 +166,7 @@ from litellm.types.llms.openai import (
ResponseIncompleteEvent,
ResponsesAPIResponse,
)
from litellm.types.llms.vertex_ai import GenerateContentResponseBody, UsageMetadata
from litellm.types.mcp import MCPPostCallResponseObject
from litellm.types.prompts.init_prompts import PromptSpec
from litellm.types.rerank import RerankResponse, get_rerank_usage
@ -266,6 +270,9 @@ if TYPE_CHECKING:
from litellm.litellm_core_utils.llm_cost_calc.utils import BilledTokenRates
from litellm.llms.base_llm.passthrough.transformation import PassthroughStreamCollector
from litellm.proxy.hooks.autorouter_baseline_cache import BaselineCacheContext, CapturedBaselineObservation
from litellm.proxy.hooks.search_free_tier import SearchAllowanceAllocation
from litellm.proxy.spend_tracking.usage_capture import CapturedUsage
from litellm.proxy.spend_tracking.usage_meter import UsageCaptureError
try:
from litellm_enterprise.enterprise_callbacks.callback_controls import (
EnterpriseCallbackControls,
@ -571,6 +578,7 @@ def _resolve_mantle_region_for_cost(
_PER_CALL_LITELLM_PARAM_DICTS: Final = frozenset({"metadata", "litellm_metadata"})
_LITELLM_PARAMS: Final = TypeAdapter(dict[str, object])
_SERVED_ANTHROPIC_BODY: Final = TypeAdapter(dict[str, object])
_GOOGLE_USAGE_FIELDS: Final = frozenset(UsageMetadata.__annotations__)
_REDACTED_RESULT: Final = TypeAdapter(object)
_GUARDRAIL_INFORMATION_KEY: Final = "standard_logging_guardrail_information"
@ -630,7 +638,60 @@ def same_attempt_marker(first: object, second: object) -> bool:
return first is second or first == second
class UsageObservation(Enum):
KNOWN = "known"
UNKNOWN = "unknown"
ESTIMATED = "estimated"
@dataclass(frozen=True, slots=True)
class UsageCaptureObservation:
usage_json: str | None
provenance: UsageObservation
response_cost: float | None = None
@classmethod
def from_usage(
cls, usage: Usage | None, provenance: UsageObservation, response_cost: float | None = None
) -> "UsageCaptureObservation | UsageCaptureError":
from litellm.proxy.spend_tracking.usage_meter import UsageCaptureError
try:
serialized: Final = (
None if usage is None or provenance is UsageObservation.UNKNOWN else usage.model_dump_json()
)
except Exception as error: # noqa: BLE001 # observation failures must not interrupt legacy retry handling
return UsageCaptureError(str(error))
else:
return cls(serialized, provenance, response_cost)
class Logging(LiteLLMLoggingBaseClass):
usage_capture_callback: ClassVar[
Callable[
["Logging", object, datetime.datetime, datetime.datetime, bool], "CapturedUsage | UsageCaptureError | None"
]
| None
] = None
def capture_usage_before_logging(
self,
response: object,
start_time: datetime.datetime | None = None,
end_time: datetime.datetime | None = None,
*,
freeze_only: bool = False,
) -> None:
capture: Final = type(self).usage_capture_callback
if capture is not None and (not self.stream or self._is_assembled_stream_success(response)):
capture(
self,
response,
start_time or self.start_time,
end_time or datetime.datetime.now(datetime.timezone.utc),
freeze_only,
)
global \
supabaseClient, \
promptLayerLogger, \
@ -828,6 +889,7 @@ class Logging(LiteLLMLoggingBaseClass):
self.finalized_cost: FinalizedCost | None = None
self.prepared_cost: PreparedCost | None = None
self.cost_allowance_discount: float | None = None
self.cost_allowance_allocation: SearchAllowanceAllocation | None = None
self.cost_finalization_lock = asyncio.Lock()
self.deferred_success_result: tuple[object, datetime.datetime, datetime.datetime] | None = None
self.billed_token_rates: BilledTokenRates | None = None
@ -884,7 +946,7 @@ class Logging(LiteLLMLoggingBaseClass):
new_call.start_time = start_time
new_call.litellm_params = litellm_params
new_call.model_call_details = {
**self.model_call_details,
**{key: value for key, value in self.model_call_details.items() if not key.startswith("_accounting_")},
"litellm_call_id": litellm_call_id,
"start_time": start_time,
"first_api_call_start_time": None,
@ -894,6 +956,7 @@ class Logging(LiteLLMLoggingBaseClass):
new_call.finalized_cost = None
new_call.prepared_cost = None
new_call.cost_allowance_discount = None
new_call.cost_allowance_allocation = None
new_call.cost_finalization_lock = asyncio.Lock()
new_call.deferred_success_result = None
new_call.streaming_chunks = []
@ -2424,7 +2487,7 @@ class Logging(LiteLLMLoggingBaseClass):
_is_sync_litellm_request = is_sync_litellm_request
def _is_assembled_stream_success(self, result=None) -> bool:
def _is_assembled_stream_success(self, result: object = None) -> bool:
"""Final assembled stream export (not a per-chunk success call).
Per-chunk callers pass a ``ModelResponseStream`` (or ``None``); the
@ -2459,7 +2522,7 @@ class Logging(LiteLLMLoggingBaseClass):
"""
from litellm.litellm_core_utils.thread_pool_executor import executor
if self._is_assembled_stream_success(result):
if self._is_assembled_stream_success(TypeAdapter(object).validate_python(result)):
if self.model_call_details.get("has_dispatched_final_stream_success"):
return
self.model_call_details["has_dispatched_final_stream_success"] = True
@ -2477,6 +2540,11 @@ class Logging(LiteLLMLoggingBaseClass):
)
return
self.capture_usage_before_logging(
TypeAdapter(object).validate_python(result),
TypeAdapter[datetime.datetime | None](datetime.datetime | None).validate_python(start_time),
TypeAdapter[datetime.datetime | None](datetime.datetime | None).validate_python(end_time),
)
await self.async_success_handler(
result,
start_time=start_time,
@ -2503,6 +2571,17 @@ class Logging(LiteLLMLoggingBaseClass):
return self.prepared_cost
recorded_cache_hit: Final[object] = self.model_call_details.get("cache_hit")
deferred: Final = self.deferred_success_result
observed_response: Final = _LITELLM_PARAMS.validate_python(
_extract_response_obj_and_hidden_params(deferred[0] if deferred else response, None)[0]
)
self.model_call_details["_accounting_usage_observed"] = (
UsageObservation.KNOWN
if (
self.model_call_details.get("combined_usage_object")
or StandardLoggingPayloadSetup.get_usage_as_dict(observed_response)
)
else UsageObservation.UNKNOWN
)
prepared: Final = self.prepare_success_logging(
result=deferred[0] if deferred is not None else response,
start_time=deferred[1] if deferred is not None else self.start_time,
@ -2540,17 +2619,29 @@ class Logging(LiteLLMLoggingBaseClass):
},
usage,
)
self.record_partial_usage_for_failure(usage, raw_cost)
self.record_partial_usage_for_failure(
usage, raw_cost, observation=self.model_call_details["_accounting_usage_observed"]
)
return self.prepared_cost
def record_finalized_cost(self, finalized: FinalizedCost) -> None:
self.finalized_cost = finalized
self.model_call_details["response_cost"] = finalized.amount
def record_partial_usage_for_failure(self, usage: Usage, response_cost: float) -> None:
def record_usage_capture(self, captured: "CapturedUsage") -> None:
self.model_call_details["_accounting_capture"] = captured
def record_usage_pricing(self, pricing: object) -> None:
self.model_call_details["_accounting_pricing_snapshot"] = pricing
def record_partial_usage_for_failure(
self, usage: Usage, response_cost: float, *, observation: UsageObservation = UsageObservation.KNOWN
) -> None:
"""Stash what an interrupted stream already consumed so the failure log bills it instead of zero."""
self.model_call_details["combined_usage_object"] = usage
self.model_call_details["response_cost"] = response_cost
if type(self).usage_capture_callback is not None:
self.model_call_details["_accounting_usage_observed"] = observation
def record_failure_after_earlier_rounds(
self, earlier: BilledRoundsSpend, failed_usage: Usage, failed_cost: float
@ -2630,6 +2721,11 @@ class Logging(LiteLLMLoggingBaseClass):
own usage while the request's failure log still bills every attempt."""
usage: Final = self._model_call_details_view.get("combined_usage_object")
response_cost: Final = self._model_call_details_view.get("response_cost")
if isinstance(usage, Usage) and type(self).usage_capture_callback is not None:
self.capture_failed_attempt_usage(usage)
self.model_call_details.pop("_accounting_capture", None)
self.model_call_details.pop("_accounting_pricing_snapshot", None)
self.model_call_details.pop("_accounting_usage_observed", None)
self.model_call_details.pop("combined_usage_object", None)
self.model_call_details.pop("response_cost", None)
if isinstance(usage, Usage):
@ -2639,6 +2735,53 @@ class Logging(LiteLLMLoggingBaseClass):
(usage, float(response_cost) if isinstance(response_cost, (int, float)) else 0.0),
)
def capture_failed_attempt_usage(self, usage: Usage) -> None:
from litellm.constants import BILLED_RETRY_ATTEMPT_METADATA_KEY
from litellm.proxy.spend_tracking.usage_capture import CapturedUsage
capture: Final = type(self).usage_capture_callback
if capture is None:
return
attempt: Final = self.attempt_scoped_copy()
attempt.prepared_cost = None
attempt.finalized_cost = None
identity: Final = (
f"{self.litellm_call_id}-attempt-{len(self.model_call_details.get('failed_attempts_usage', ()))}"
)
params: Final = _LITELLM_PARAMS.validate_python(self.model_call_details.get("litellm_params") or {})
metadata: Final = StandardLoggingPayloadSetup.merge_litellm_metadata(params)
observed: Final[object] = self.model_call_details.get("_accounting_usage_observed")
cost: Final[object] = self.model_call_details.get("response_cost")
attempt.model_call_details = {
**attempt.model_call_details,
"litellm_call_id": identity,
"parent_request_id": self.litellm_call_id,
"_accounting_capture": None,
"_accounting_prior_captures": (),
"_accounting_usage_snapshot": UsageCaptureObservation.from_usage(
usage,
observed if isinstance(observed, UsageObservation) else UsageObservation.KNOWN,
float(cost) if isinstance(cost, (int, float)) else None,
),
"litellm_params": {
**params,
"metadata": {**metadata, BILLED_RETRY_ATTEMPT_METADATA_KEY: True, "status": "failure"},
"litellm_metadata": {**metadata, BILLED_RETRY_ATTEMPT_METADATA_KEY: True, "status": "failure"},
},
}
captured: Final = capture(
attempt,
{"id": identity, "usage": usage.model_dump(), "status": "failed"},
self.start_time,
datetime.datetime.now(datetime.timezone.utc),
False,
)
if isinstance(captured, CapturedUsage):
self.model_call_details["_accounting_prior_captures"] = (
*self.model_call_details.get("_accounting_prior_captures", ()),
captured,
)
def record_continued_partial(self, usage: Usage, response_cost: float, response_id: str) -> None:
"""Bill what a cut stream already consumed on this continuation's success log, under the id the caller saw."""
self.model_call_details["continued_partial_usage"] = usage
@ -2796,6 +2939,55 @@ class Logging(LiteLLMLoggingBaseClass):
_update_completion_start_time = update_completion_start_time
def normalize_endpoint_logging_result(self, result: object) -> object:
call_type: Final[object] = self.call_type
if call_type == CallTypes.anthropic_messages.value:
return self._anthropic_messages_logged_response(result=result)
if call_type in (CallTypes.generate_content.value, CallTypes.agenerate_content.value):
return self._handle_non_streaming_google_genai_generate_content_response_logging(result=result)
if call_type in (CallTypes.asend_message.value, CallTypes.send_message.value):
return self._handle_a2a_response_logging(result=result)
return result
def usage_capture_response(self, result: object) -> object:
if self.stream and not self._is_assembled_stream_success(result):
return None
copied: Final = result.model_copy(deep=True) if isinstance(result, BaseModel) else result
normalized: Final = self.normalize_logging_result(self.normalize_endpoint_logging_result(copied))
if self.model_call_details.get("call_type") == CallTypes.arealtime.value and isinstance(result, list):
frames: Final = TypeAdapter(tuple[dict[str, object], ...]).validate_python(result)
measured: Final = any(
isinstance(_LITELLM_PARAMS.validate_python(frame.get("response") or {}).get("usage"), Mapping)
for frame in frames
if frame.get("type") == "response.done"
)
if not measured:
metered: Final = self.model_call_details.get(REALTIME_METERED_USAGE_KEY)
return {
"id": self.litellm_call_id,
"usage": {"metered_usage": TypeAdapter(object).dump_python(metered, mode="json")}
if metered is not None
else None,
}
assembled: Final = self._get_assembled_streaming_response(
normalized, self.start_time, self.start_time, True, []
)
return normalized if assembled is None else assembled
def usage_capture_pricing(self, result: object) -> Mapping[str, JsonValue]:
from litellm.utils import count_characters
return TypeAdapter(dict[str, JsonValue]).validate_python(
{
"custom_pricing": self.custom_pricing_for(result),
"custom_cost_per_token": self.custom_cost_per_token,
"custom_cost_per_second": self.custom_cost_per_second,
"prompt_characters": count_characters(self._prompt_for_cost_calculation())
if self.model_call_details.get("call_type") in (CallTypes.speech.value, CallTypes.aspeech.value)
else None,
}
)
def normalize_logging_result(self, result: object) -> object:
"""
Some endpoints return a different type of result than what is expected by the logging system.
@ -2806,8 +2998,20 @@ class Logging(LiteLLMLoggingBaseClass):
combined_usage_object: Final = (
RealtimeAPITokenUsageProcessor.collect_and_combine_usage_from_realtime_stream_results(results=result)
)
metered: Final[object] = self.model_call_details.get(REALTIME_METERED_USAGE_KEY)
captured_usage: Final = (
combined_usage_object.model_copy(
update={
"metered_usage": TypeAdapter(dict[str, JsonValue]).validate_python(
TypeAdapter(object).dump_python(metered, mode="json")
)
}
)
if metered is not None
else combined_usage_object
)
logging_result = RealtimeAPITokenUsageProcessor.create_logging_realtime_object(
usage=combined_usage_object,
usage=captured_usage,
results=result,
)
@ -3068,7 +3272,14 @@ class Logging(LiteLLMLoggingBaseClass):
start_time = self.start_time
if end_time is None:
end_time = datetime.datetime.now()
if self.completion_start_time is None:
# A request re-dispatched to a fallback reuses this logging object but runs in
# its own window, so a first-token time from the previous attempt cannot belong
# to it. Keeping it makes the attempt report a negative TTFT.
first_token_seconds: Final = _timestamp_seconds(self.completion_start_time)
attempt_start_seconds: Final = _timestamp_seconds(start_time)
if first_token_seconds is None or (
attempt_start_seconds is not None and first_token_seconds < attempt_start_seconds
):
self.completion_start_time = end_time
self.model_call_details["completion_start_time"] = self.completion_start_time
@ -3080,15 +3291,7 @@ class Logging(LiteLLMLoggingBaseClass):
else cache_hit
)
if self.call_type == CallTypes.anthropic_messages.value:
result = self._anthropic_messages_logged_response(result=result)
elif (
self.call_type == CallTypes.generate_content.value
or self.call_type == CallTypes.agenerate_content.value
):
result = self._handle_non_streaming_google_genai_generate_content_response_logging(result=result)
elif self.call_type == CallTypes.asend_message.value or self.call_type == CallTypes.send_message.value:
result = self._handle_a2a_response_logging(result=result)
result = self.normalize_endpoint_logging_result(result)
logging_result: Final = self.normalize_logging_result(result=result)
if cache_hit is None and _served_from_cache_at_no_charge(logging_result):
@ -3441,6 +3644,7 @@ class Logging(LiteLLMLoggingBaseClass):
self._with_client_facing_stream_model(complete_streaming_response)
)
self.model_call_details["complete_streaming_response"] = logged_streaming_response
self.capture_usage_before_logging(logged_streaming_response, start_time, end_time)
## STANDARDIZED LOGGING PAYLOAD
if not for_async_request:
self.model_call_details["standard_logging_object"] = self._build_standard_logging_payload(
@ -3964,6 +4168,8 @@ class Logging(LiteLLMLoggingBaseClass):
cost_for_built_in_tools_cost_usd_dollar=0.0,
)
self.capture_usage_before_logging(result, start_time, end_time)
self.truncated_messages_for_logging = await truncate_base64_in_messages_async(
StandardLoggingPayloadSetup.append_system_prompt_messages(
kwargs=self.model_call_details, messages=self.model_call_details.get("messages")
@ -4990,16 +5196,34 @@ class Logging(LiteLLMLoggingBaseClass):
def billable_result(self, result: object) -> object:
"""The response the cost calculator prices: a /v1/messages body served through the chat
bridge is replaced by the ModelResponse its spend row is built from, anything else is
bridge is replaced by the ModelResponse its spend row is built from, a native one is only
converted when its provider reports the cost the bill is read from, anything else is
priced as is."""
served: Final[object] = result
call_type: Final[object] = getattr(self, "call_type", None)
if not isinstance(result, dict):
return served
if self.bridged_model_response is None and call_type != CallTypes.anthropic_messages.value:
if self.bridged_model_response is not None:
return self._anthropic_messages_logged_response(result=served)
if call_type != CallTypes.anthropic_messages.value:
return served
if not reports_native_messages_cost(self.model_call_details.get("custom_llm_provider")):
return self._served_with_requested_speed(result=served)
return self._anthropic_messages_logged_response(result=served)
def _served_with_requested_speed(self, result: object) -> object:
"""A native /v1/messages body priced as served bills the speed the client asked for when its
usage block reports none, the same speed its spend row is priced at."""
body: Final = _SERVED_ANTHROPIC_BODY.validate_python(result)
usage: Final = body.get("usage")
requested: Final = _LITELLM_PARAMS.validate_python(getattr(self, "optional_params", None) or {}).get("speed")
if not isinstance(requested, str) or not isinstance(usage, dict):
return result
reported: Final = _SERVED_ANTHROPIC_BODY.validate_python(usage)
if isinstance(reported.get("speed"), str):
return result
return {**body, "usage": {**reported, "speed": requested}}
def _anthropic_messages_logged_response(self, result: object) -> ModelResponse:
"""
The ModelResponse a /v1/messages spend_logs row is built from.
@ -6808,45 +7032,19 @@ class StandardLoggingPayloadSetup:
@staticmethod
def get_usage_from_response_obj(response_obj: dict | None, combined_usage_object: Usage | None = None) -> Usage:
## BASE CASE ##
if combined_usage_object is not None:
return combined_usage_object
if response_obj is None:
return Usage(
prompt_tokens=0,
completion_tokens=0,
total_tokens=0,
)
usage: Final = response_obj.get("usage", None)
if usage is None:
rerank_usage: Final = get_rerank_usage(_RESPONSE_MAPPING.validate_python(response_obj))
if rerank_usage is not None:
return Usage(**rerank_usage)
if usage is None or (not isinstance(usage, dict) and not isinstance(usage, Usage)):
return Usage(
prompt_tokens=0,
completion_tokens=0,
total_tokens=0,
)
elif isinstance(usage, Usage):
return usage
elif isinstance(usage, ResponseAPIUsage):
return ResponseAPILoggingUtils.transform_response_api_usage_to_chat_usage(usage)
elif isinstance(usage, dict):
if ResponseAPILoggingUtils.is_response_api_usage(usage):
return ResponseAPILoggingUtils.transform_response_api_usage_to_chat_usage(usage)
if InteractionsUsageObjectTransformation.is_interactions_usage_object(usage):
return InteractionsUsageObjectTransformation.transform_interactions_usage_object(usage)
return Usage(**usage)
raise ValueError(f"usage is required, got={usage} of type {type(usage)}")
reported: Final = response_obj.get("usage") if response_obj else None
if isinstance(reported, Usage):
return reported
normalized: Final = StandardLoggingPayloadSetup.get_usage_as_dict(response_obj)
return Usage(**normalized) if normalized else Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0)
@staticmethod
def get_usage_as_dict(
response_obj: dict | None,
response_obj: dict[str, Any] | None,
combined_usage_object: Usage | None = None,
) -> dict:
) -> dict[str, Any]:
"""
Like get_usage_from_response_obj but returns a plain dict, skipping
the Pydantic Usage construction on the hot path.
@ -6858,7 +7056,21 @@ class StandardLoggingPayloadSetup:
return _empty
_raw: Final = response_obj.get("usage", None)
if _raw is None:
return dict(get_rerank_usage(_RESPONSE_MAPPING.validate_python(response_obj)) or _empty)
google_usage: Final[object] = response_obj.get("usageMetadata")
if isinstance(google_usage, dict) and _GOOGLE_USAGE_FIELDS.intersection(google_usage):
return litellm.VertexGeminiConfig.calculate_usage(
cast( # cast-ok: retain converter-supported modalities and token_count aliases
GenerateContentResponseBody, response_obj
)
).model_dump()
reported: Final = response_obj.get("usage_info")
if isinstance(reported, BaseModel):
return reported.model_dump(exclude_none=True)
return (
{key: value for key, value in reported.items() if value is not None}
if isinstance(reported, dict)
else dict(get_rerank_usage(_RESPONSE_MAPPING.validate_python(response_obj)) or _empty)
)
if isinstance(_raw, ResponseAPIUsage):
return ResponseAPILoggingUtils.transform_response_api_usage_to_chat_usage(_raw).model_dump()
if isinstance(_raw, dict):

3
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
View file

@ -208,11 +208,14 @@ class StandardBuiltInToolCostTracking:
usage=usage, response_object=response_object
)
response_model: Final = _output_item_field(response_object, "model")
if model_info is not None and resolved_usage is not None and resolved_provider is not None:
result: Final = get_cost_for_web_search_request(
custom_llm_provider=resolved_provider,
usage=resolved_usage,
model_info=model_info,
response_model=response_model if isinstance(response_model, str) else None,
lookup_model_info=StandardBuiltInToolCostTracking._safe_get_model_info,
)
if result is not None:
return result

7
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -28,6 +28,7 @@ from litellm.types.utils import (
DataResidency,
ImageResponse,
ModelInfo,
ModelInfoBase,
PassthroughCallTypes,
PromptTokensDetailsWrapper,
ServiceTier,
@ -197,15 +198,15 @@ def get_billable_input_tokens(usage: Usage) -> int:
def select_cost_metric_for_model(
model_info: ModelInfo,
model_info: ModelInfoBase,
) -> Literal["cost_per_character", "cost_per_token"]:
"""
Select 'cost_per_character' if model_info has 'input_cost_per_character'
Select 'cost_per_token' if model_info has 'input_cost_per_token'
"""
if model_info.get("input_cost_per_character"):
if model_info.get("input_cost_per_character") is not None:
return "cost_per_character"
elif model_info.get("input_cost_per_token"):
elif model_info.get("input_cost_per_token") is not None:
return "cost_per_token"
else:
raise ValueError(

52
litellm/litellm_core_utils/logging_utils.py
View file

@ -3,13 +3,15 @@ import functools
import inspect
import re
import time
from collections.abc import Iterator, Mapping, Sequence
from collections.abc import Callable, Iterator, Mapping, Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, TypeAlias, cast
from litellm._logging import format_base64_size, verbose_logger
from litellm.constants import (
BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS,
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS,
BASE64_TRUNCATION_OFFLOAD_THRESHOLD_STRINGS,
MAX_BASE64_LENGTH_FOR_LOGGING,
)
from litellm.litellm_core_utils.core_helpers import get_litellm_metadata_from_kwargs
@ -44,6 +46,8 @@ Helper utils used for logging callbacks
# Regex matching data-URI base64 content: "data:<mime>;base64,<payload>"
# Captures: group(1)=mime_type, group(2)=base64_payload
_DATA_URI_RE: Final = re.compile(r"data:([^;,\s]{1,255});base64,([A-Za-z0-9+/=]+)")
_DATA_URI_PREFIX: Final = "data:"
_DATA_URI_BASE64_MARKER: Final = ";base64,"
# Maximum nesting depth for _truncate_base64_in_value to guard against
# pathological payloads. OpenAI message format is typically 3-4 levels deep.
@ -60,15 +64,20 @@ def _base64_data_uri_replacer(match: re.Match[str]) -> str:
return f"data:{mime_type};base64,[base64_data truncated: {size_str}]"
def _may_contain_base64_data_uri(value: str) -> bool:
return ";" in value and _DATA_URI_PREFIX in value and _DATA_URI_BASE64_MARKER in value
def _truncate_base64_in_string(value: str) -> str:
"""Replace long base64 data-URI payloads in a string with a size placeholder."""
if MAX_BASE64_LENGTH_FOR_LOGGING <= 0:
if MAX_BASE64_LENGTH_FOR_LOGGING <= 0 or not _may_contain_base64_data_uri(value):
return value
return _DATA_URI_RE.sub(_base64_data_uri_replacer, value)
def _truncate_base64_in_value(
value: str | dict[str, object] | list[object] | None,
truncate_string: Callable[[str], str],
) -> str | dict[str, object] | list[object] | None:
"""Iteratively truncate base64 data URIs in a JSON-like value (str/list/dict).
@ -78,13 +87,13 @@ def _truncate_base64_in_value(
# Stack entries: (source_value, depth, parent_container, key_or_index)
# We mutate *copies* of dicts/lists in-place via parent references.
if isinstance(value, str):
return _truncate_base64_in_string(value)
return truncate_string(value)
if not isinstance(value, (dict, list)):
return value
# Shallow-copy the root so we don't mutate the caller's data.
root: Final = {k: v for k, v in value.items()} if isinstance(value, dict) else list(value)
stack: Final[list] = [(root, 0)]
stack: Final[list[tuple[dict[str, object] | list[object], int]]] = [(root, 0)] # mutable-ok: explicit stack
while stack:
container, depth = stack.pop()
@ -93,19 +102,19 @@ def _truncate_base64_in_value(
if isinstance(container, dict):
for k, v in container.items():
if isinstance(v, str):
container[k] = _truncate_base64_in_string(v)
container[k] = truncate_string(v)
elif isinstance(v, dict):
copy: dict | list = {ck: cv for ck, cv in v.items()}
copy: dict[str, object] | list[object] = {ck: cv for ck, cv in v.items()}
container[k] = copy
stack.append((copy, depth + 1))
elif isinstance(v, list):
copy = list(v)
container[k] = copy
stack.append((copy, depth + 1))
elif isinstance(container, list):
else:
for i, v in enumerate(container):
if isinstance(v, str):
container[i] = _truncate_base64_in_string(v)
container[i] = truncate_string(v)
elif isinstance(v, dict):
copy = {ck: cv for ck, cv in v.items()}
container[i] = copy
@ -127,8 +136,15 @@ def truncate_base64_in_messages(
"""
if messages is None or MAX_BASE64_LENGTH_FOR_LOGGING <= 0:
return messages
return _copy_messages(messages, _truncate_base64_in_string)
def _copy_messages(
messages: str | dict[str, object] | list[object], # mutable-ok: same contract as truncate_base64_in_messages
truncate_string: Callable[[str], str],
) -> str | dict[str, object] | list[object] | None: # mutable-ok: same contract as truncate_base64_in_messages
try:
return _truncate_base64_in_value(messages)
return _truncate_base64_in_value(messages, truncate_string)
except Exception as e:
verbose_logger.debug("Failed to truncate base64 in messages: %s", e)
return messages
@ -151,19 +167,31 @@ def _iter_string_leaves(value: _StringTree) -> Iterator[str]:
pass
def _is_few_plain_strings(leaves: tuple[str, ...]) -> bool:
return len(leaves) < BASE64_TRUNCATION_OFFLOAD_THRESHOLD_STRINGS and not any(
_may_contain_base64_data_uri(leaf) for leaf in leaves
)
async def truncate_base64_in_messages_async(
messages: str | list | dict | None, # mutable-ok: same contract as truncate_base64_in_messages
) -> str | list | dict | None: # mutable-ok: same contract as truncate_base64_in_messages
"""
Same result as truncate_base64_in_messages, but payloads whose string content
reaches BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS are scanned in a worker
thread so the regex pass over multi-MB base64 images does not block the event loop.
reaches BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS are copied in a worker thread,
so neither the regex pass over multi-MB base64 images nor the copy of a long
history blocks the event loop. A few strings with no data URI, under
BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS in total, are copied inline, since
checking them for data URIs and copying them costs less than the thread hop.
"""
if messages is None or MAX_BASE64_LENGTH_FOR_LOGGING <= 0:
return messages
total_chars: Final = sum(len(leaf) for leaf in _iter_string_leaves(messages))
leaves: Final = tuple(_iter_string_leaves(messages))
total_chars: Final = sum(len(leaf) for leaf in leaves)
if total_chars < BASE64_TRUNCATION_OFFLOAD_THRESHOLD_CHARS:
return truncate_base64_in_messages(messages)
if total_chars < BASE64_TRUNCATION_INLINE_COPY_MAX_CHARS and _is_few_plain_strings(leaves):
return _copy_messages(messages, lambda text: text)
return await asyncio.to_thread(truncate_base64_in_messages, messages)

11
litellm/litellm_core_utils/model_param_helper.py
View file

@ -17,6 +17,7 @@ from openai.types.responses.response_create_params import (
ResponseCreateParamsNonStreaming,
ResponseCreateParamsStreaming,
)
from pydantic import TypeAdapter, ValidationError
from litellm._logging import verbose_logger
from litellm.types.llms.anthropic import AnthropicMessagesRequest
@ -41,6 +42,16 @@ def peek_reasoning_summary_aliases(optional_params: Mapping[str, object]) -> obj
return None
_OPTIONAL_BOOL: Final = TypeAdapter[bool | None](bool | None)
def embeds_thought_signature_in_tool_call_id(litellm_params: Mapping[str, object]) -> bool:
try:
return _OPTIONAL_BOOL.validate_python(litellm_params.get("thought_signature_in_tool_call_id")) is not False
except ValidationError:
return True
class ModelParamHelper:
# Cached at class level — deterministic set built from static OpenAI type annotations
relevant_logging_args: ClassVar[frozenset[str]] = frozenset()

0
litellm/router_utils/pattern_match_deployments.py → litellm/litellm_core_utils/pattern_match_deployments.py
View file

60
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -8,7 +8,7 @@ import io
import json
import mimetypes
import re
from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from collections.abc import Callable, Hashable, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from graphlib import CycleError, TopologicalSorter
from itertools import chain, groupby, islice
@ -88,17 +88,22 @@ def handle_any_messages_to_chat_completion_str_messages_conversion(
return [{"input": str(messages)}]
def _message_with_str_content(message: AllMessageValues) -> AllMessageValues:
texts: Final = convert_content_list_to_str(message=message)
if not texts:
return message
return cast( # cast-ok: the same message with its content flattened to one string
AllMessageValues, {**message, "content": texts}
)
def handle_messages_with_content_list_to_str_conversion(
messages: list[AllMessageValues],
) -> list[AllMessageValues]:
"""
Handles messages with content list conversion
Returns new messages with each content list flattened to a string, leaving the caller's messages untouched
"""
for message in messages:
texts = convert_content_list_to_str(message=message)
if texts:
message["content"] = texts
return messages
return [_message_with_str_content(message) for message in messages]
def strip_name_from_message(message: AllMessageValues, allowed_name_roles: list[str] = ["user"]) -> AllMessageValues:
@ -233,7 +238,7 @@ def is_non_content_values_set(message: AllMessageValues) -> bool:
return any(message.get(key, None) is not None for key in message if key not in ignore_keys)
_IMAGE_CONTENT_PART_TYPES: Final = frozenset({"image_url", "input_image", "image"})
_IMAGE_CONTENT_PART_TYPES: Final = ("image_url", "input_image", "image", "computer_screenshot")
_IMAGE_SCAN_MAX_DEPTH: Final = 4
@ -552,15 +557,24 @@ def as_openai_image_part(part: Mapping[str, object]) -> ChatCompletionImageObjec
return {"type": "image_url", "image_url": {"url": url}} if url else None
def request_contains_image_content(messages: Sequence[Mapping[str, object]]) -> bool:
"""Whether any message carries an image content part, across the dialects that reach
pre-routing hooks untranslated: chat-completions ``image_url``, Responses ``input_image``,
and Anthropic Messages ``image``, including images nested inside ``tool_result`` blocks."""
return any(
isinstance(content, list) and _content_parts_contain_image(content)
for message in messages
for content in (message.get("content"),)
)
def _message_contains_image_content(message: object) -> bool:
match message:
case {"content": list()}:
return _content_parts_contain_image(content_parts(_CONTENT_PART.validate_python(message)))
case {"type": "function_call_output" | "custom_tool_call_output" | "tool_result", "output": list()}:
output: Final = _CONTENT_PART.validate_python(message)["output"]
return _content_parts_contain_image(_CONTENT_PARTS.validate_python(output))
case {"type": "computer_call_output", "output": dict()}:
return _content_parts_contain_image((_CONTENT_PART.validate_python(message)["output"],))
case _:
return False
def request_contains_image_content(messages: object) -> bool:
"""Inspect native message content without translating or validating the request."""
if not isinstance(messages, (list, tuple)):
return False
return any(_message_contains_image_content(message) for message in _CONTENT_PARTS.validate_python(messages))
def audio_or_image_in_message_content(message: AllMessageValues) -> bool:
@ -3221,3 +3235,15 @@ def text_completion_prompt_to_messages(prompt: object) -> tuple[AllMessageValues
if string_entries and len(string_entries) == len(entries):
return tuple(ChatCompletionUserMessage(role="user", content=entry) for entry in string_entries)
raise ValueError(f"`prompt` must be a non-empty string or a non-empty list of strings. Got: {prompt_type_name}.")
def is_present_non_string(value: object) -> bool:
return value is not None and not isinstance(value, str)
def invalid_type_message(field_path: str, value: object) -> str:
return f"Invalid type for '{field_path}': expected a string, got {type(value).__name__}"
def is_unhashable_value_behind(error: TypeError, value: object) -> bool:
return not isinstance(value, Hashable) and f"unhashable type: '{type(value).__name__}'" in str(error)

182
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -26,11 +26,13 @@ from litellm.litellm_core_utils.core_helpers import as_str_mapping, attribute_or
from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import (
anthropic_system_messages,
find_anthropic_server_tool_result,
is_anthropic_server_tool_use,
is_anthropic_tool_search_use,
message_field,
parts_of,
place_mid_conversation_system,
split_leading_system_run,
string_field,
)
from litellm.litellm_core_utils.url_utils import async_safe_get, safe_get
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_async_httpx_client, header_value
@ -1252,6 +1254,22 @@ def encode_tool_call_id_with_signature(tool_call_id: str, thought_signature: str
_encode_tool_call_id_with_signature = encode_tool_call_id_with_signature
def _tool_call_id_prefix(tool_call_id: object) -> str | None:
return tool_call_id.split(THOUGHT_SIGNATURE_SEPARATOR, 1)[0] if isinstance(tool_call_id, str) else None
def _called_tool_name(tool_call_id: str, tool_calls: Sequence[Mapping[str, object]]) -> str | None:
exact_matches: Final = tuple(call for call in tool_calls if call.get("id") == tool_call_id)
prefix: Final = _tool_call_id_prefix(tool_call_id)
prefix_matches: Final = tuple(
call for call in tool_calls if prefix and _tool_call_id_prefix(call.get("id")) == prefix
)
matches: Final = exact_matches or (prefix_matches if len(prefix_matches) == 1 else ())
function: Final = matches[0].get("function") if matches else None
name: Final = function.get("name") if isinstance(function, Mapping) else None
return name if isinstance(name, str) else None
def get_thought_signature_from_tool(tool: Mapping[str, object]) -> str | None:
"""Extract thought signature from tool call's provider_specific_fields.
@ -1691,16 +1709,14 @@ def convert_to_gemini_tool_call_result(
)
) is not None:
media_parts.append(media_part)
name: str | None = message.get("name", "")
# Recover name from last message with tool calls
if last_message_with_tool_calls:
tools: Final = last_message_with_tool_calls.get("tool_calls", [])
msg_tool_call_id: Final = message.get("tool_call_id", None)
for tool in tools:
prev_tool_call_id = tool.get("id", None)
if msg_tool_call_id and prev_tool_call_id and msg_tool_call_id == prev_tool_call_id:
name = tool.get("function", {}).get("name", "")
msg_tool_call_id: Final = message.get("tool_call_id", None)
called_tool_name: Final = (
_called_tool_name(msg_tool_call_id, last_message_with_tool_calls.get("tool_calls") or ())
if last_message_with_tool_calls and isinstance(msg_tool_call_id, str) and msg_tool_call_id
else None
)
name: Final[str | None] = called_tool_name if called_tool_name is not None else message.get("name", "")
# Echo the OpenAI tool_call_id on functionResponse (strip thought-signature suffix).
gemini_call_id: str | None = None
@ -2008,7 +2024,7 @@ def _server_tool_answers_replaced_after(messages: Sequence[object], index: int)
following: Final = _messages_until_the_next_assistant(messages, index)
a_user_turn_remains: Final = any(
message_field(message, "role") in ("user", "tool", "function")
and message_field(message, "tool_call_id") not in kept_server_result_ids
and string_field(message, "tool_call_id") not in kept_server_result_ids
for message in following
)
return _tool_call_ids_in_tool_messages(following) & kept_server_result_ids if a_user_turn_remains else frozenset()
@ -2059,6 +2075,54 @@ def _placed_and_paired(
_AnthropicToolInvokeItem: TypeAlias = AnthropicMessagesToolUseParam | dict[str, Any]
def _is_server_tool_result(block: Mapping[str, object]) -> bool:
block_type: Final = block.get("type")
return isinstance(block_type, str) and block_type.endswith("_tool_result")
def _split_unrun_server_tool_uses(
groups: Sequence[Sequence[_AnthropicToolInvokeItem]],
) -> tuple[tuple[Sequence[_AnthropicToolInvokeItem], ...], tuple[_AnthropicToolInvokeItem, ...]]:
completed: Final = tuple(group for group in groups if any(_is_server_tool_result(block) for block in group))
unrun: Final = tuple(group[0] for group in groups if not any(_is_server_tool_result(block) for block in group))
return completed, unrun
def _block_id(block: Mapping[str, object]) -> object:
return block.get("id")
def _unrun_server_tool_use_ids(blocks: Sequence[Mapping[str, object]]) -> frozenset[object]:
result_ids: Final = frozenset(block.get("tool_use_id") for block in blocks if _is_server_tool_result(block))
return frozenset(
_block_id(block)
for block in blocks
if block.get("type") == "server_tool_use" and _block_id(block) not in result_ids
)
def _last_block_id_among(blocks: Sequence[Mapping[str, object]], ids: frozenset[object]) -> object:
return next(block_id for block in reversed(blocks) if (block_id := _block_id(block)) in ids)
def _paused_turn_replay(
tool_invokes: Sequence[_AnthropicToolInvokeItem],
thinking_blocks: Sequence[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock],
*,
is_final_message: bool,
) -> tuple[
tuple[_AnthropicToolInvokeItem, ...],
tuple[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, ...],
]:
unrun_ids: Final = _unrun_server_tool_use_ids(tool_invokes)
if not unrun_ids:
return tuple(tool_invokes), tuple(thinking_blocks)
dropped_ids: Final = (
unrun_ids - frozenset((_last_block_id_among(tool_invokes, unrun_ids),)) if is_final_message else unrun_ids
)
return tuple(block for block in tool_invokes if _block_id(block) not in dropped_ids), ()
def convert_to_anthropic_tool_invoke(
tool_calls: list[ChatCompletionAssistantToolCall],
web_search_results: Sequence[object] | None = None,
@ -2141,6 +2205,15 @@ def convert_to_anthropic_tool_invoke(
anthropic_tool_invoke.append(server_tool_result)
elif is_anthropic_tool_search_use(tool_id, tool_name) and tool_id not in answered_tool_ids:
continue
elif is_anthropic_server_tool_use(tool_id) and tool_id not in answered_tool_ids:
anthropic_tool_invoke.append(
{
"type": "server_tool_use",
"id": tool_id,
"name": tool_name,
"input": tool_input,
}
)
else:
sanitized_tool_id = sanitize_anthropic_tool_use_id(tool_id)
_anthropic_tool_use_param = AnthropicMessagesToolUseParam(
@ -3109,6 +3182,30 @@ def anthropic_messages_pt(
_has_server_tool_calls = True
break
# Get web_search_results and tool_results from provider_specific_fields
# for server_tool_use reconstruction.
# Fixes: https://github.com/BerriAI/litellm/issues/17737
_provider_specific_fields_raw = assistant_content_block.get("provider_specific_fields")
_provider_specific_fields: dict[str, Any] = (
cast(dict[str, Any], _provider_specific_fields_raw)
if isinstance(_provider_specific_fields_raw, dict)
else {}
)
tool_invoke_results = (
convert_to_anthropic_tool_invoke(
assistant_tool_calls,
web_search_results=_provider_specific_fields.get("web_search_results"),
tool_results=_provider_specific_fields.get("tool_results"),
answered_tool_ids=client_answered_tool_ids,
)
if assistant_tool_calls is not None
else ()
)
turn_paused = bool(_unrun_server_tool_use_ids(tool_invoke_results))
replayed_tool_invokes, replayed_thinking_blocks = _paused_turn_replay(
tool_invoke_results, thinking_blocks or (), is_final_message=msg_i == len(messages) - 1
)
if (
thinking_blocks is not None
and _has_server_tool_calls
@ -3121,26 +3218,12 @@ def anthropic_messages_pt(
# We must preserve this interleaved order because Anthropic
# verifies thinking block signatures based on position.
# Build the tool call groups (server_tool_use + its result)
_provider_specific_fields_raw_tc = assistant_content_block.get("provider_specific_fields")
_provider_specific_fields_tc: dict[str, Any] = {}
if isinstance(_provider_specific_fields_raw_tc, dict):
_provider_specific_fields_tc = cast(dict[str, Any], _provider_specific_fields_raw_tc)
_web_search_results_tc = _provider_specific_fields_tc.get("web_search_results")
_tool_results_tc = _provider_specific_fields_tc.get("tool_results")
tool_invoke_results = convert_to_anthropic_tool_invoke(
assistant_tool_calls,
web_search_results=_web_search_results_tc,
tool_results=_tool_results_tc,
answered_tool_ids=client_answered_tool_ids,
)
# Group tool invoke results into (server_tool_use, result) pairs
# and separate regular tool_use blocks
server_tool_groups: list[list[_AnthropicToolInvokeItem]] = []
regular_tool_uses: list[_AnthropicToolInvokeItem] = []
_current_group: list[_AnthropicToolInvokeItem] = []
for item in tool_invoke_results:
for item in replayed_tool_invokes:
item_type = item.get("type", "") if isinstance(item, dict) else getattr(item, "type", "")
if item_type == "server_tool_use":
if _current_group:
@ -3154,6 +3237,9 @@ def anthropic_messages_pt(
_current_group.append(item)
if _current_group:
server_tool_groups.append(_current_group)
completed_server_tool_groups, pending_server_tool_uses = _split_unrun_server_tool_uses(
server_tool_groups
)
# Build the text block if content is a non-empty string
text_element = None
@ -3177,15 +3263,15 @@ def anthropic_messages_pt(
# Any remaining groups (after all thinking blocks) go after.
tb_idx = 0
grp_idx = 0
num_tb = len(thinking_blocks) if thinking_blocks else 0
num_grp = len(server_tool_groups)
num_tb = len(replayed_thinking_blocks)
num_grp = len(completed_server_tool_groups)
while tb_idx < num_tb or grp_idx < num_grp:
if tb_idx < num_tb and grp_idx < num_grp:
# Emit thinking block then its tool group
assistant_content.append(thinking_blocks[tb_idx])
assistant_content.append(replayed_thinking_blocks[tb_idx])
tb_idx += 1
for block in server_tool_groups[grp_idx]:
for block in completed_server_tool_groups[grp_idx]:
item_id = block.get("id") if isinstance(block, dict) else getattr(block, "id", None)
if item_id and item_id in unique_tool_ids:
continue
@ -3195,11 +3281,11 @@ def anthropic_messages_pt(
grp_idx += 1
elif tb_idx < num_tb:
# More thinking blocks than tool groups - emit before text
assistant_content.append(thinking_blocks[tb_idx])
assistant_content.append(replayed_thinking_blocks[tb_idx])
tb_idx += 1
else:
# More tool groups than thinking blocks - emit remaining
for block in server_tool_groups[grp_idx]:
for block in completed_server_tool_groups[grp_idx]:
item_id = block.get("id") if isinstance(block, dict) else getattr(block, "id", None)
if item_id and item_id in unique_tool_ids:
continue
@ -3212,8 +3298,8 @@ def anthropic_messages_pt(
if text_element is not None:
assistant_content.append(text_element)
# Add regular (non-server) tool calls at the end
for item in regular_tool_uses:
# Add regular tool calls, then a paused turn's unrun server tool call, at the end
for item in (*regular_tool_uses, *pending_server_tool_uses):
item_id = item.get("id") if isinstance(item, dict) else getattr(item, "id", None)
if item_id and item_id in unique_tool_ids:
continue
@ -3233,18 +3319,18 @@ def anthropic_messages_pt(
)
_content_list = assistant_content_block.get("content") if _content_is_list else None
if (
thinking_blocks is not None and not _content_is_list
): # IMPORTANT: ADD THIS FIRST, ELSE ANTHROPIC WILL RAISE AN ERROR
assistant_content.extend(thinking_blocks)
if not _content_is_list: # IMPORTANT: ADD THIS FIRST, ELSE ANTHROPIC WILL RAISE AN ERROR
assistant_content.extend(replayed_thinking_blocks)
if _content_is_list and _content_list is not None:
for m in _with_missing_thinking_blocks(_content_list, thinking_blocks or ()):
for m in _with_missing_thinking_blocks(_content_list, replayed_thinking_blocks):
if not isinstance(m, dict):
continue
# handle thinking blocks
if (thinking_block := _as_thinking_block(m)) is not None:
assistant_content.extend(
_replayable_thinking_blocks([thinking_block], drop_params=drop_params)
()
if turn_paused
else _replayable_thinking_blocks([thinking_block], drop_params=drop_params)
)
continue
text_block = cast(str, m.get("text", ""))
@ -3284,25 +3370,9 @@ def anthropic_messages_pt(
assistant_content.append(_anthropic_text_content_element)
if assistant_tool_calls is not None: # support assistant tool invoke conversion
# Get web_search_results and tool_results from provider_specific_fields
# for server_tool_use reconstruction.
# Fixes: https://github.com/BerriAI/litellm/issues/17737
_provider_specific_fields_raw = assistant_content_block.get("provider_specific_fields")
_provider_specific_fields: dict[str, Any] = {}
if isinstance(_provider_specific_fields_raw, dict):
_provider_specific_fields = cast(dict[str, Any], _provider_specific_fields_raw)
_web_search_results = _provider_specific_fields.get("web_search_results")
_tool_results = _provider_specific_fields.get("tool_results")
tool_invoke_results = convert_to_anthropic_tool_invoke(
assistant_tool_calls,
web_search_results=_web_search_results,
tool_results=_tool_results,
answered_tool_ids=client_answered_tool_ids,
)
# Prevent "tool_use ids must be unique" errors by filtering duplicates
# This can happen when merging history that already contains the tool calls
for item in tool_invoke_results:
for item in replayed_tool_invokes:
# tool_use items are typically dicts, but handle objects just in case
item_id = item.get("id") if isinstance(item, dict) else getattr(item, "id", None)

9
litellm/litellm_core_utils/prompt_templates/mid_conversation_system.py
View file

@ -85,6 +85,11 @@ def message_field(message: object, key: str) -> object:
return mapping.get(key) if mapping is not None else getattr(message, key, None)
def string_field(message: object, key: str) -> str | None:
value: Final = message_field(message, key)
return value if isinstance(value, str) else None
def is_system_message(message: object) -> bool:
return message_field(message, "role") == "system"
@ -286,6 +291,10 @@ def _separate_thinking_blocks_render(message: object) -> bool:
return any(_thinking_block_renders(block) for block in parts_of(message_field(message, "thinking_blocks")))
def is_anthropic_server_tool_use(tool_id: str) -> bool:
return tool_id.startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX)
def is_anthropic_tool_search_use(tool_id: str, tool_name: str) -> bool:
return tool_id.startswith(ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX) and tool_name in ANTHROPIC_TOOL_SEARCH_TOOL_NAMES

2
litellm/router_utils/ptu_shares.py → litellm/litellm_core_utils/ptu_shares.py
View file

@ -12,13 +12,13 @@ from datetime import date
from typing import Final, Generic, TypeAlias, TypeVar
from litellm.litellm_core_utils.ptu_pricing import parsed_ptu_shares, ptu_terms
from litellm.litellm_core_utils.routing_common_utils import get_request_team_id, team_may_use_deployment
from litellm.llms.azure.ptu_capacity import (
PTUCapacity,
deployment_ptu_capacity,
is_azure_deployment,
is_str_keyed_mapping,
)
from litellm.router_utils.common_utils import get_request_team_id, team_may_use_deployment
_DeploymentT: Final = TypeVar("_DeploymentT", bound=Mapping[str, object])
WildcardRoute: TypeAlias = Callable[[str], Sequence[Mapping[str, object]]]

2
litellm/litellm_core_utils/realtime_streaming.py
View file

@ -631,6 +631,8 @@ class RealTimeStreaming:
if self.session_tools or self.tool_calls:
self.logging_obj.model_call_details["realtime_tools"] = self.session_tools
self.logging_obj.model_call_details["realtime_tool_calls"] = self.tool_calls
if LiteLLMLogging.usage_capture_callback is not None:
self.logging_obj.capture_usage_before_logging(self.messages)
# Route through the bounded logging worker (per-coroutine timeout +
# concurrency cap) instead of a bare create_task, so a slow callback
# can't leave suspended tasks pinning each call's response in memory.

0
litellm/router_utils/reasoning_effort_capability.py → litellm/litellm_core_utils/reasoning_effort_capability.py
View file

0
litellm/router_utils/common_utils.py → litellm/litellm_core_utils/routing_common_utils.py
View file

0
litellm/router_utils/routing_groups.py → litellm/litellm_core_utils/routing_groups.py
View file

19
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -1509,11 +1509,20 @@ def _delta_reasoning_items(delta: object) -> Sequence[object]:
return cast(Sequence[object], items) if isinstance(items, list) else () # cast-ok: items forwarded verbatim
def _chunk_reasoning_items(chunk: object) -> Sequence[object]:
def _chunk_first_delta(chunk: object) -> object:
choices: Final = _field_of(chunk, "choices")
if not isinstance(choices, list) or not choices:
return ()
return _delta_reasoning_items(_field_of(cast(Sequence[object], choices)[0], "delta")) # cast-ok: untyped chunk
return None
return _field_of(cast(Sequence[object], choices)[0], "delta") # cast-ok: untyped chunk
def _chunk_reasoning_items(chunk: object) -> Sequence[object]:
return _delta_reasoning_items(_chunk_first_delta(chunk))
def _chunk_refusal(chunk: object) -> str:
refusal: Final = _field_of(_chunk_first_delta(chunk), "refusal")
return refusal if isinstance(refusal, str) else ""
_NON_TEXT_DELTA_FIELDS: Final = (
@ -1526,6 +1535,7 @@ _NON_TEXT_DELTA_FIELDS: Final = (
"audio",
"images",
"provider_specific_fields",
"refusal",
)
@ -1674,6 +1684,9 @@ def _assembled_message_fields(processor: ChunkProcessor, chunks: Sequence) -> It
yield "reasoning_content", processor.get_combined_reasoning_content(reasoning_chunks)
typed_chunks: Final = cast(Sequence[object], chunks) # cast-ok: chunks are untyped upstream
combined_refusal: Final = "".join(_chunk_refusal(chunk) for chunk in typed_chunks)
if combined_refusal:
yield "refusal", combined_refusal
reasoning_items: Final = tuple(chain.from_iterable(_chunk_reasoning_items(chunk) for chunk in typed_chunks))
if reasoning_items:
yield "reasoning_items", list(reasoning_items)

70
litellm/litellm_core_utils/streaming_handler.py
View file

@ -349,11 +349,14 @@ class CustomStreamWrapper:
make_call: Callable | None = None,
_response_headers: dict | httpx.Headers | None = None,
count_prompt_tokens: Callable[[], int] | None = None,
is_mock: bool = False,
):
self.model = model
self.make_call = make_call
self.count_prompt_tokens = count_prompt_tokens
self.custom_llm_provider = custom_llm_provider
self.is_mock = is_mock
self.uses_openai_chat_stream = False
self.logging_obj: LiteLLMLoggingObject = logging_obj
self.completion_stream = completion_stream
self.sent_first_chunk = False
@ -376,6 +379,7 @@ class CustomStreamWrapper:
self._upstream_provider: str | None = None
self.received_finish_reason: str | None = None
self.intermittent_finish_reason: str | None = None # finish reasons that show up mid-stream
self.received_any_finish_reason = False # a later chunk can reset intermittent_finish_reason
self.special_tokens = [
"<|assistant|>",
"<|system|>",
@ -901,13 +905,21 @@ class CustomStreamWrapper:
except Exception as e:
raise e
def _served_openrouter_model(self) -> str | None:
if self.custom_llm_provider == LlmProviders.OPENROUTER.value or (
self.custom_llm_provider == "cached_response"
and self.logging_obj.model_call_details.get("custom_llm_provider") == LlmProviders.OPENROUTER.value
):
return self._provider_response_model
return None
def model_response_creator(
self,
chunk: dict | None = None,
hidden_params: Mapping[str, object] | None = None,
delta: Delta | None = None,
) -> ModelResponseStream:
_model: Final = self._cached_model_name
_model: Final = self._served_openrouter_model() or self._cached_model_name
_logging_obj_llm_provider: Final = self._cached_logging_llm_provider
if chunk is None:
@ -1302,8 +1314,8 @@ class CustomStreamWrapper:
if _is_delta_empty:
model_response.choices[0].delta = Delta(content=self._close_open_think_block())
# get any function call arguments
model_response.choices[0].finish_reason = map_finish_reason(
finish_reason=self.received_finish_reason
model_response.choices[0].finish_reason = self._normalize_tool_call_finish_reason(
map_finish_reason(finish_reason=self.received_finish_reason)
) # ensure consistent output to openai
self.sent_last_chunk = True
@ -1597,6 +1609,7 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
else: # openai / azure chat model
self.uses_openai_chat_stream = True
if isinstance(chunk, ChatCompletionChunk):
self._openai_handler_receives_chat_chunks = True
if any(str(choice.finish_reason) == "abort" for choice in chunk.choices or ()):
@ -1617,6 +1630,7 @@ class CustomStreamWrapper:
return _ProviderChunkEarlyReturn(None)
completion_obj["content"] = response_obj["text"]
self.intermittent_finish_reason = response_obj.get("finish_reason", None)
self.received_any_finish_reason |= response_obj["finish_reason"] is not None
if response_obj["is_finished"]:
if response_obj["finish_reason"] == "error":
raise Exception(
@ -1693,7 +1707,7 @@ class CustomStreamWrapper:
return dispatch_result.value
response_obj = dispatch_result.response_obj
model_response.model = self.model
model_response.model = self._served_openrouter_model() or self.model
## FUNCTION CALL PARSING
original_chunk: Final[_UpstreamChunkLike | None] = (
response_obj.get("original_chunk") if response_obj is not None else None
@ -1737,6 +1751,8 @@ class CustomStreamWrapper:
# if function returned but type set to None - mistral's api returns type: None
tool["type"] = "function"
model_response.choices[0].delta = Delta(**_json_delta)
if model_response.choices[0].delta.tool_calls:
self.tool_call = True
except Exception as e:
verbose_logger.exception(
"litellm.CustomStreamWrapper.chunk_creator(): Exception occured - %s", e
@ -1966,6 +1982,11 @@ class CustomStreamWrapper:
if self.logging_obj.is_sync_litellm_request(litellm_params):
self.logging_obj.success_handler(processed_chunk, None, None, cache_hit)
def _normalize_tool_call_finish_reason(self, finish_reason: str) -> str:
if finish_reason == "stop" and self.tool_call:
return "tool_calls"
return finish_reason
def finish_reason_handler(self):
model_response: Final = self.model_response_creator(
hidden_params=None if self.received_model_id is None else {"received_model_id": self.received_model_id}
@ -1981,21 +2002,30 @@ class CustomStreamWrapper:
if served_tier is not None:
setattr(model_response, "service_tier", served_tier) # noqa: B010 # pydantic extra, not a declared field
_finish_reason: Final = self.received_finish_reason or self.intermittent_finish_reason
if _finish_reason is not None:
model_response.choices[0].finish_reason = _finish_reason
else:
model_response.choices[0].finish_reason = "stop"
## if tool use
if (
model_response.choices[0].finish_reason == "stop" and self.tool_call
): # don't overwrite for other - potential error finish reasons
model_response.choices[0].finish_reason = "tool_calls"
model_response.choices[0].finish_reason = self._normalize_tool_call_finish_reason(
_finish_reason if _finish_reason is not None else "stop"
)
close_tag: Final = self._close_open_think_block()
if close_tag is not None:
model_response.choices[0].delta.content = close_tag
return model_response
def _should_fail_on_clean_eof(self) -> bool:
return (
not self.is_mock
and self.uses_openai_chat_stream
and self.received_finish_reason is None
and self.intermittent_finish_reason is None
and not self.received_any_finish_reason
)
@staticmethod
def _stream_eof_error() -> httpx.RemoteProtocolError:
return httpx.RemoteProtocolError(
"Stream ended before a finish_reason was received",
request=httpx.Request(method="POST", url="https://docs.litellm.ai/docs"),
)
def _final_chunk_for_logging(self, processed_chunk: "ModelResponseStream") -> "ModelResponseStream | None":
if self.received_finish_reason is not None or self.intermittent_finish_reason is not None:
return processed_chunk
@ -2181,6 +2211,13 @@ class CustomStreamWrapper:
return response
except StopIteration:
if self._should_fail_on_clean_eof():
error: Final = self._stream_eof_error()
eof_traceback: Final = traceback.format_exc()
self._record_partial_usage_for_failure()
threading.Thread(target=self.logging_obj.failure_handler, args=(error, eof_traceback)).start()
self.close()
self._handle_stream_fallback_error(error)
if self.sent_last_chunk is True:
complete_streaming_response: Final = self._build_complete_streaming_response()
response = self._new_usage_chunk()
@ -2384,6 +2421,9 @@ class CustomStreamWrapper:
continue
return usage_free_chunk
except (StopAsyncIteration, StopIteration):
if self._should_fail_on_clean_eof():
await self.aclose()
self._log_stream_failure_and_raise(self._stream_eof_error())
return await self._finalize_completed_stream(cache_hit=cache_hit)
except httpx.TimeoutException as e: # if httpx read timeout error occues
traceback_exception = traceback.format_exc()
@ -2529,7 +2569,7 @@ class CustomStreamWrapper:
if partial is None or not isinstance(usage, Usage):
return None
if self.model:
partial.model = self.model
partial.model = self._served_openrouter_model() or self.model
backfill_missing_cache_usage_fields(usage)
return partial

2
litellm/litellm_core_utils/token_counter.py
View file

@ -56,9 +56,9 @@ from litellm.litellm_core_utils.tokenizer import HuggingFaceTokenizer, OpenAIEnc
if TYPE_CHECKING:
from litellm.litellm_core_utils.tokenizer import Encoding, HuggingFace
from litellm.litellm_core_utils.deployment_pricing import deployment_pricing_models
from litellm.litellm_core_utils.url_utils import safe_get
from litellm.llms.custom_httpx.http_handler import get_httpx_client, header_value
from litellm.router_utils.deployment_pricing import deployment_pricing_models
from litellm.rust_bridge.tokenizer import get_encoding
from litellm.types.llms.anthropic import (
AnthropicContentParamSource,

47
litellm/llms/__init__.py
View file

@ -9,6 +9,8 @@ from litellm.types.utils import CallTypes
from . import *
if TYPE_CHECKING:
from collections.abc import Callable
from litellm.llms.base_llm.guardrail_translation.base_translation import (
BaseTranslation,
)
@ -30,7 +32,29 @@ def get_cost_for_google_maps_grounding_request(
return cost_per_google_maps_grounding_request(usage=usage, model_info=model_info)
def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", model_info: "ModelInfo") -> float | None:
def _with_served_model_search_pricing(
model_info: "ModelInfo",
response_model: str | None,
served_model_provider: str,
lookup_model_info: "Callable[[str, str | None], ModelInfo | None] | None",
) -> "ModelInfo":
if model_info.get("search_context_cost_per_query") is not None or response_model is None:
return model_info
served_model_info: Final = (
lookup_model_info(response_model, served_model_provider) if lookup_model_info is not None else None
)
if served_model_info is None:
return model_info
return {**model_info, "search_context_cost_per_query": served_model_info.get("search_context_cost_per_query")}
def get_cost_for_web_search_request(
custom_llm_provider: str,
usage: "Usage",
model_info: "ModelInfo",
response_model: str | None = None,
lookup_model_info: "Callable[[str, str | None], ModelInfo | None] | None" = None,
) -> float | None:
"""
Get the cost for a web search request for a given model.
@ -38,6 +62,8 @@ def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", mo
custom_llm_provider: The custom LLM provider.
usage: The usage object.
model_info: The model info.
response_model: The model the provider response names.
lookup_model_info: Looks up a model's cost map entry, returning None when it is not mapped.
"""
if custom_llm_provider == "gemini":
from .gemini.cost_calculator import cost_per_web_search_request
@ -46,17 +72,25 @@ def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", mo
elif custom_llm_provider == "anthropic":
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
return get_cost_for_anthropic_web_search(
model_info=_with_served_model_search_pricing(model_info, response_model, "anthropic", lookup_model_info),
usage=usage,
)
elif custom_llm_provider.startswith("vertex_ai"):
# Anthropic Claude models on Vertex AI populate server_tool_use.web_search_requests
# (same as the direct Anthropic API), not prompt_tokens_details.web_search_requests
# (which is the Gemini field). Route claude-* models to the Anthropic calculator.
model_key: Final[str] = model_info.get("key", "") if model_info else ""
if "claude" in model_key.lower():
if "claude" in (response_model or model_key).lower():
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
verbose_logger.debug("vertex_ai/claude model detected — routing web search cost to Anthropic calculator")
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
return get_cost_for_anthropic_web_search(
model_info=_with_served_model_search_pricing(
model_info, response_model, "vertex_ai", lookup_model_info
),
usage=usage,
)
from .vertex_ai.gemini.cost_calculator import (
cost_per_web_search_request as cost_per_web_search_request_vertex_ai,
@ -71,7 +105,10 @@ def get_cost_for_web_search_request(custom_llm_provider: str, usage: "Usage", mo
return None
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
return get_cost_for_anthropic_web_search(
model_info=_with_served_model_search_pricing(model_info, response_model, "azure_ai", lookup_model_info),
usage=usage,
)
elif custom_llm_provider == "perplexity":
# Perplexity handles search costs internally in its own cost calculator
# Return 0.0 to indicate costs are already accounted for

9
litellm/llms/a2a/common_utils.py
View file

@ -160,8 +160,11 @@ async def resolve_a2a_hop_auth_header(
litellm_params: Mapping[str, object],
custom_llm_provider: object,
resolve_entra_header: AgentAuthHeaderResolver = resolve_azure_ai_agent_auth_header,
agent_extra_headers: Mapping[str, str] | None = None,
) -> Mapping[str, str] | None:
"""Entra credentials authenticate the A2A hop only; a completion-bridge agent hands them to the model provider it bridges to."""
if not a2a_hop_uses_entra(litellm_params, custom_llm_provider):
if a2a_hop_uses_entra(litellm_params, custom_llm_provider):
return await resolve_entra_header(litellm_params)
if custom_llm_provider or any(name.lower() == "authorization" for name in (agent_extra_headers or {})):
return None
return await resolve_entra_header(litellm_params)
api_key: Final = litellm_params.get("api_key")
return {"Authorization": f"Bearer {api_key}"} if isinstance(api_key, str) and api_key else None

28
litellm/llms/anthropic/chat/handler.py
View file

@ -70,7 +70,12 @@ from litellm.types.utils import (
)
from ...base import BaseLLM
from ..common_utils import AnthropicError, process_anthropic_headers, refusal_provider_specific_fields
from ..common_utils import (
AnthropicError,
process_anthropic_headers,
refusal_params,
stop_reason_provider_specific_fields,
)
from .transformation import ANTHROPIC_TOOL_NAME_REVERSE_MAP_KEY, AnthropicConfig, JsonModeRequest
if TYPE_CHECKING:
@ -926,6 +931,17 @@ class ModelResponseIterator:
def _web_search_call_snapshot(self) -> dict[str, object]:
return dict(self._web_search_calls)
def _refresh_pending_web_search_call(self, tool_use_id: str) -> bool:
if tool_use_id not in self._web_search_calls:
return False
self._web_search_calls[tool_use_id] = build_web_search_call(
tool_use_id,
self._server_tool_inputs.get(tool_use_id, {}),
{"content": []},
status="in_progress",
)
return True
def _complete_web_search_call(self, result: dict[str, object]) -> None:
tool_use_id: Final = result.get("tool_use_id")
if not isinstance(tool_use_id, str) or tool_use_id not in self._web_search_calls:
@ -1128,6 +1144,8 @@ class ModelResponseIterator:
self._server_tool_inputs[self._current_server_tool_id] = json.loads(args)
except (json.JSONDecodeError, TypeError):
pass
if self._refresh_pending_web_search_call(self._current_server_tool_id):
provider_specific_fields["web_search_calls"] = self._web_search_call_snapshot()
self._current_server_tool_id = None
if self.is_response_format_tool:
text = self._flush_response_format_buffer()
@ -1140,8 +1158,8 @@ class ModelResponseIterator:
# These are automatically handled by Anthropic API, we just pass them through
pass
elif type_chunk == "message_delta":
finish_reason, usage, container, refusal_fields = self._handle_message_delta(chunk)
provider_specific_fields.update(refusal_fields)
finish_reason, usage, container, stop_reason_fields = self._handle_message_delta(chunk)
provider_specific_fields.update(stop_reason_fields)
text = self._release_answer_close() + self._drain_split_answer_blocks()
if container:
provider_specific_fields["container"] = container
@ -1190,6 +1208,7 @@ class ModelResponseIterator:
)
text, tool_use = self._handle_json_mode_chunk(text=text, tool_use=tool_use)
refusal_fields: Final = refusal_params(provider_specific_fields)
returned_chunk: Final = ModelResponseStream(
choices=[
@ -1201,6 +1220,7 @@ class ModelResponseIterator:
provider_specific_fields=(provider_specific_fields if provider_specific_fields else None),
thinking_blocks=(thinking_blocks if thinking_blocks else None),
reasoning_content=reasoning_content,
**refusal_fields,
),
finish_reason=finish_reason,
)
@ -1371,7 +1391,7 @@ class ModelResponseIterator:
self._handle_usage(anthropic_usage_chunk=message_delta["usage"]) if "usage" in message_delta else None
)
container: Final = message_delta["delta"].get("container")
return finish_reason, usage, container, refusal_provider_specific_fields(message_delta["delta"])
return finish_reason, usage, container, stop_reason_provider_specific_fields(message_delta["delta"])
def _handle_accumulated_json_chunk(self, data_str: str, is_final: bool = False) -> ModelResponseStream | None:
"""

205
litellm/llms/anthropic/chat/transformation.py
View file

@ -2,7 +2,9 @@ import copy
import json
import re
import time
from collections.abc import Callable, Mapping, Sequence
from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from functools import partial
from itertools import chain
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, ClassVar, Final, NoReturn, cast
@ -27,7 +29,10 @@ from litellm.constants import (
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params, map_finish_reason
from litellm.litellm_core_utils.prompt_templates.common_utils import (
DEFS_MAX_INLINED_BYTES,
invalid_type_message,
is_object_schema,
is_present_non_string,
is_unhashable_value_behind,
sanitize_input_schema_for_anthropic,
schema_fragment_ref,
schema_grammar_branch_count,
@ -45,6 +50,8 @@ from litellm.litellm_core_utils.prompt_templates.image_handling import (
inline_remote_image_urls,
)
from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import (
message_field,
parts_of,
place_mid_conversation_system,
split_leading_system_run,
)
@ -120,13 +127,14 @@ from litellm.utils import (
)
from ..common_utils import (
REFUSAL_STOP_FIELDS,
STOP_REASON_FIELDS,
AnthropicError,
AnthropicModelInfo,
eager_input_streaming_flag,
process_anthropic_headers,
refusal_provider_specific_fields,
refusal_params,
requires_native_compaction_beta,
stop_reason_provider_specific_fields,
strip_advisor_blocks_from_messages,
tools_marked_for_a_tool_config_point,
)
@ -167,6 +175,9 @@ _DEFER_LOADING_ADAPTER: Final = TypeAdapter(bool)
_TOOL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_TOOLS_ADAPTER: Final = TypeAdapter(list[object])
_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_RESPONSE_BLOCKS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
_PENDING_WEB_SEARCH_RESULT: Final = MappingProxyType({"content": ()})
_REQUEST_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_NATIVE_OUTPUT_STRING_FORMATS: Final = frozenset(
("date-time", "time", "date", "duration", "email", "hostname", "uri", "ipv4", "ipv6", "uuid")
)
@ -175,6 +186,76 @@ NATIVE_OUTPUT_MAX_OPTIONAL_PARAMETERS: Final = 24
NATIVE_OUTPUT_MAX_GRAMMAR_BRANCHES: Final = 2048
def _tool_call_ids_failing(
index: int, tool_calls: object, fails: Callable[[object], bool]
) -> Iterator[tuple[str, object]]:
ids: Final = ((position, message_field(tool_call, "id")) for position, tool_call in enumerate(parts_of(tool_calls)))
return (
(f"messages[{index}].tool_calls[{position}].id", tool_call_id)
for position, tool_call_id in ids
if fails(tool_call_id)
)
def _roles_failing(messages: Sequence[object], fails: Callable[[object], bool]) -> Iterator[tuple[str, object]]:
roles: Final = ((index, message_field(message, "role")) for index, message in enumerate(messages))
return ((f"messages[{index}].role", role) for index, role in roles if fails(role))
def _is_non_empty_non_string(value: object) -> bool:
return bool(value) and is_present_non_string(value)
def _assistant_tool_call_ids_failing(
messages: Sequence[object], fails: Callable[[object], bool]
) -> Iterator[tuple[str, object]]:
assistant_turns: Final = (
(index, message) for index, message in enumerate(messages) if message_field(message, "role") == "assistant"
)
return chain.from_iterable(
_tool_call_ids_failing(index, message_field(message, "tool_calls"), fails) for index, message in assistant_turns
)
def _tool_answer_ids_failing(
messages: Sequence[object], fails: Callable[[object], bool]
) -> Iterator[tuple[str, object]]:
answers: Final = (
(index, message_field(message, "tool_call_id"))
for index, message in enumerate(messages)
if message_field(message, "role") in ("tool", "function")
)
return ((f"messages[{index}].tool_call_id", tool_call_id) for index, tool_call_id in answers if fails(tool_call_id))
def _is_non_string_value_behind(error: AttributeError, value: object) -> bool:
return is_present_non_string(value) and value is error.obj
def _fields_behind(messages: Sequence[object], error: Exception) -> Iterable[tuple[str, object]]:
match error:
case TypeError():
caused_error: Final = partial(is_unhashable_value_behind, error)
return chain(
_assistant_tool_call_ids_failing(messages, caused_error),
_tool_answer_ids_failing(messages, caused_error),
_roles_failing(messages, caused_error),
)
case AttributeError():
return _assistant_tool_call_ids_failing(messages, partial(_is_non_string_value_behind, error))
case _:
return ()
def non_string_message_field_error(messages: Sequence[object], error: Exception) -> AnthropicError | None:
malformed_field: Final = next(
chain(_roles_failing(messages, _is_non_empty_non_string), _fields_behind(messages, error)), None
)
if malformed_field is None:
return None
return AnthropicError(status_code=400, message=invalid_type_message(*malformed_field))
def _tool_type(tool: object) -> str | None:
match tool:
case {"type": str() as tool_type}:
@ -2254,6 +2335,16 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
headers=headers,
)
@staticmethod
def _messages_within_cache_control_cap(
messages: Sequence[AllMessageValues], optional_params: Mapping[str, object]
) -> Sequence[AllMessageValues]:
from litellm.integrations.anthropic_cache_control_hook import messages_within_cache_control_cap
return messages_within_cache_control_cap(
messages, optional_params.get("tools"), optional_params.get("cache_control"), optional_params
)
def transform_request(
self,
model: str,
@ -2271,9 +2362,18 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
anthropic_messages_pt,
)
drop_params: Final = litellm.drop_params is True or litellm_params.get("drop_params") is True
drop_params: Final = drops_unsupported_params(litellm_params.get("drop_params"))
capped_messages: Final = (
list(
self._messages_within_cache_control_cap(
messages, _REQUEST_PARAMS_ADAPTER.validate_python(optional_params)
)
)
if drop_params
else messages
)
if "tools" not in optional_params and messages is not None and has_tool_call_blocks(messages):
if "tools" not in optional_params and has_tool_call_blocks(capped_messages):
optional_params["tools"], _ = self._map_tools(add_dummy_tool(custom_llm_provider="anthropic"))
# Drop thinking param if thinking is enabled but thinking_blocks are missing
@ -2285,9 +2385,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
# Related issue: https://github.com/BerriAI/litellm/issues/18926
if (
optional_params.get("thinking") is not None
and messages is not None
and last_assistant_with_tool_calls_has_no_thinking_blocks(messages)
and not any_assistant_message_has_thinking_blocks(messages)
and last_assistant_with_tool_calls_has_no_thinking_blocks(capped_messages)
and not any_assistant_message_has_thinking_blocks(capped_messages)
):
if litellm.modify_params:
optional_params.pop("thinking", None)
@ -2332,25 +2431,34 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
_name_forward_map, _name_reverse_map = self._sanitize_tool_names_in_request(
optional_params=optional_params,
)
if _name_forward_map:
messages = self._rewrite_tool_names_in_messages(messages, _name_forward_map)
request_messages: Final = (
self._rewrite_tool_names_in_messages(capped_messages, _name_forward_map)
if _name_forward_map
else capped_messages
)
if _name_reverse_map and isinstance(litellm_params, dict):
litellm_params[ANTHROPIC_TOOL_NAME_REVERSE_MAP_KEY] = _name_reverse_map
# Only the leading system run becomes the top-level system prompt. A later
# system message stays in the conversation: hoisting it rewrites the cached
# prefix and re-bills the whole history at cache-write pricing (#36559).
leading_system_run, later_messages = split_leading_system_run(messages)
leading_system_run, later_messages = split_leading_system_run(request_messages)
anthropic_system_message_list: Final = self.translate_system_message(messages=list(leading_system_run))
# Handling anthropic API Prompt Caching
if len(anthropic_system_message_list) > 0:
optional_params["system"] = anthropic_system_message_list
conversation: Final = place_mid_conversation_system(
later_messages,
supports_mid_conversation_system=supports_mid_conversation_system(
model=model, custom_llm_provider=self.custom_llm_provider
),
)
try:
conversation: Final = place_mid_conversation_system(
later_messages,
supports_mid_conversation_system=supports_mid_conversation_system(
model=model, custom_llm_provider=self.custom_llm_provider
),
)
except TypeError as error:
field_error: Final = non_string_message_field_error(messages, error)
if field_error is None:
raise
raise field_error from error
# Format rest of message according to anthropic guidelines
try:
anthropic_messages = anthropic_messages_pt(
@ -2360,9 +2468,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
drop_params=drop_params,
)
except Exception as e:
raise AnthropicError(
conversion_field_error: Final = non_string_message_field_error(messages, e)
raise conversion_field_error or AnthropicError(
status_code=400,
message=f"{e}\nReceived Messages={messages}",
message=f"{e}\nReceived Messages={request_messages}",
) # don't use verbose_logger.exception, if exception is raised
self.update_headers_with_optional_anthropic_beta(
@ -2897,28 +3006,24 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
web_search_results: Sequence[object],
completion_response: Mapping[str, object],
) -> list[object]:
content: Final = completion_response.get("content")
blocks: Final = content if isinstance(content, Sequence) else ()
inputs: Final = {
call_id: tool_input
for block in blocks
if isinstance(block, Mapping)
and block.get("type") == "server_tool_use"
and block.get("name") == "web_search"
and isinstance((call_id := block.get("id")), str)
and isinstance((tool_input := block.get("input")), Mapping)
blocks: Final = _RESPONSE_BLOCKS.validate_python(completion_response.get("content") or ())
results_by_id: Final = {
tool_use_id: result
for result in _RESPONSE_BLOCKS.validate_python(web_search_results)
if result.get("type") == "web_search_tool_result"
and isinstance((tool_use_id := result.get("tool_use_id")), str)
}
return [
build_web_search_call(
tool_id=tool_use_id,
tool_input=inputs.get(tool_use_id, {}),
result=result,
tool_id=call_id,
tool_input=block.get("input"),
result=results_by_id.get(call_id, _PENDING_WEB_SEARCH_RESULT),
status=None if call_id in results_by_id else "in_progress",
)
for result in web_search_results
if isinstance(result, dict)
and result.get("type") == "web_search_tool_result"
and isinstance((tool_use_id := result.get("tool_use_id")), str)
and tool_use_id in inputs
for block in blocks
if block.get("type") == "server_tool_use"
and block.get("name") == "web_search"
and isinstance((call_id := block.get("id")), str)
]
def _build_provider_specific_fields(
@ -2942,10 +3047,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if web_search_results is not None:
provider_specific_fields["web_search_results"] = web_search_results
provider_specific_fields["web_search_calls"] = self._build_web_search_calls(
web_search_results,
completion_response,
)
web_search_calls: Final = self._build_web_search_calls(web_search_results or (), completion_response)
if web_search_results is not None or web_search_calls:
provider_specific_fields["web_search_calls"] = web_search_calls
if tool_results is not None:
provider_specific_fields["tool_results"] = tool_results
@ -2967,10 +3072,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if compaction_blocks is not None:
provider_specific_fields["compaction_blocks"] = compaction_blocks
refusal_fields: Final = refusal_provider_specific_fields(
REFUSAL_STOP_FIELDS.validate_python(completion_response)
stop_reason_fields: Final = stop_reason_provider_specific_fields(
STOP_REASON_FIELDS.validate_python(completion_response)
)
return {**provider_specific_fields, **refusal_fields}
return {**provider_specific_fields, **stop_reason_fields}
def transform_parsed_response(
self,
@ -3045,22 +3150,26 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if json_extra_content:
merged_text = merged_text + json_extra_content if merged_text else json_extra_content
refusal_fields: Final = refusal_params(STOP_REASON_FIELDS.validate_python(completion_response))
_message = litellm.Message(
tool_calls=tool_calls_for_message,
content=merged_text or None,
provider_specific_fields=provider_specific_fields,
thinking_blocks=thinking_blocks,
reasoning_content=reasoning_content,
**refusal_fields,
)
_message.provider_specific_fields = provider_specific_fields
if json_mode_message is not None:
refusal_fields: Final = refusal_provider_specific_fields(
REFUSAL_STOP_FIELDS.validate_python(completion_response)
stop_reason_fields: Final = stop_reason_provider_specific_fields(
STOP_REASON_FIELDS.validate_python(completion_response)
)
completion_response["stop_reason"] = "stop"
if refusal_fields:
json_mode_message.provider_specific_fields = dict(refusal_fields)
if stop_reason_fields:
json_mode_message.provider_specific_fields = dict(stop_reason_fields)
if "refusal" in refusal_fields:
json_mode_message.refusal = refusal_fields["refusal"]
if thinking_blocks:
json_mode_message.thinking_blocks = thinking_blocks
json_mode_message.reasoning_content = reasoning_content

69
litellm/llms/anthropic/common_utils.py
View file

@ -4,6 +4,7 @@ This file contains common utils for anthropic calls.
import copy
import json
import os
import re
from collections.abc import Container, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
@ -19,6 +20,8 @@ from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm.constants import (
ANTHROPIC_COMPATIBLE_MAX_USER_ID_LENGTH,
ANTHROPIC_MAX_USER_ID_LENGTH,
DEFAULT_MODEL_CREATED_AT_TIME,
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
@ -178,6 +181,40 @@ def tools_without_custom_field(tools: Sequence[object], hoist_defer_loading: boo
return [_tool_without_custom_field(tool, hoist_defer_loading, drop_custom) for tool in tools]
def anthropic_pass_through_api_base() -> str:
return os.getenv("ANTHROPIC_API_BASE") or os.getenv("ANTHROPIC_BASE_URL") or "https://api.anthropic.com"
def _same_host_and_port(url: httpx.URL, other: str) -> bool:
try:
other_url: Final = httpx.URL(other)
except httpx.InvalidURL:
return False
return (url.host, url.port) == (other_url.host, other_url.port)
def anthropic_upstream_user_id_limit(url: str) -> int | None:
upstream: Final = httpx.URL(url)
if not upstream.path.removesuffix("/").endswith("/v1/messages"):
return None
if upstream.host == "api.anthropic.com":
return ANTHROPIC_MAX_USER_ID_LENGTH
return (
ANTHROPIC_COMPATIBLE_MAX_USER_ID_LENGTH
if _same_host_and_port(upstream, anthropic_pass_through_api_base())
else None
)
def _has_email_shape(token: str) -> bool:
at_sign: Final = token.find("@", 1)
return at_sign != -1 and token.find(".", at_sign + 2, -1) != -1
def anthropic_accepts_user_id(user_id: str, max_length: int) -> bool:
return len(user_id) <= max_length and user_id.isprintable() and not any(map(_has_email_shape, user_id.split()))
def is_anthropic_messages_url(url: str) -> bool:
"""Check whether a URL addresses Anthropic's Messages API."""
parsed_url: Final = urlparse(url)
@ -2481,7 +2518,7 @@ def create_anthropic_model_list_response(
}
class RefusalStopFields(TypedDict, total=False):
class StopReasonFields(TypedDict, total=False):
stop_reason: ReadOnly[object]
stop_details: ReadOnly[object]
@ -2498,16 +2535,38 @@ def _dict_or_empty(value: object) -> object:
return value if _is_dict(value) else _no_stop_fields()
REFUSAL_STOP_FIELDS: Final[TypeAdapter[RefusalStopFields]] = TypeAdapter(
Annotated[RefusalStopFields, BeforeValidator(_dict_or_empty)]
STOP_REASON_FIELDS: Final[TypeAdapter[StopReasonFields]] = TypeAdapter(
Annotated[StopReasonFields, BeforeValidator(_dict_or_empty)]
)
_STOP_REASONS_KEPT_RAW: Final = frozenset({"refusal", "pause_turn"})
def refusal_provider_specific_fields(message: Mapping[str, object]) -> Mapping[str, object]:
def stop_reason_provider_specific_fields(message: Mapping[str, object]) -> Mapping[str, object]:
stop_reason: Final = message.get("stop_reason")
stop_details: Final = message.get("stop_details")
if stop_details is not None:
return {"stop_reason": stop_reason, "stop_details": stop_details}
if stop_reason == "refusal":
if stop_reason in _STOP_REASONS_KEPT_RAW:
return {"stop_reason": stop_reason}
return {}
class _RefusalStopDetails(TypedDict, total=False):
explanation: ReadOnly[object]
_REFUSAL_STOP_DETAILS: Final[TypeAdapter[_RefusalStopDetails]] = TypeAdapter(
Annotated[_RefusalStopDetails, BeforeValidator(_dict_or_empty)]
)
class RefusalParams(TypedDict, total=False):
refusal: ReadOnly[str]
def refusal_params(message: Mapping[str, object]) -> RefusalParams:
if message.get("stop_reason") != "refusal":
return {}
explanation: Final = _REFUSAL_STOP_DETAILS.validate_python(message.get("stop_details")).get("explanation")
return {"refusal": explanation} if isinstance(explanation, str) and explanation else {}

3
litellm/llms/anthropic/messages/adapters/handler.py
View file

@ -118,9 +118,10 @@ def _extract_proxy_litellm_metadata(
def _requested_router(kwargs: Mapping[str, object]) -> "Router | None":
from litellm.router import Router
from litellm.router_python_core import PythonCore
requested: Final = kwargs.get("litellm_router")
return requested if isinstance(requested, Router) else None
return requested if isinstance(requested, (Router, PythonCore)) else None # pyright: ignore[reportReturnType] # the router passes its core, which carries Router's members
def request_safeguards_evaluator(

52
litellm/llms/anthropic/messages/adapters/streaming_iterator.py
View file

@ -33,6 +33,7 @@ from litellm.llms.anthropic.messages.safeguards import StreamedSafeguardResults
from litellm.llms.anthropic.messages.utils import anthropic_sse_frame
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.llms.anthropic import (
AnthropicStopDetails,
AppliedEdit,
CompactionBlock,
ContentBlockDelta,
@ -284,9 +285,7 @@ class _CombinedChunkSplitter:
@staticmethod
def _is_combined(chunk: "ModelResponseStream") -> bool:
"""True if ``chunk`` carries response content AND a finish_reason."""
from litellm.llms.anthropic.messages.native.utils import (
openai_chat_refusal_text,
)
from litellm.llms.anthropic.messages.native.utils import refusal_reply_text
choices: Final = _optional_attr_sequence(chunk, "choices")
if not choices:
@ -302,7 +301,7 @@ class _CombinedChunkSplitter:
or _optional_attr(delta, "tool_calls")
or _optional_attr(delta, "reasoning_content")
or _optional_attr(delta, "thinking_blocks")
or openai_chat_refusal_text(delta)
or refusal_reply_text(delta)
)
_PAYLOAD_FIELD_GROUPS: "tuple[tuple[str, ...], ...]" = (
@ -659,6 +658,8 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
self.compaction_block = compaction_block
self.iterations_usage = iterations_usage
self._refusal_text: str = ""
self._paused_turn: bool = False
self._claude_refusal_stop_details: AnthropicStopDetails | None = None
self.sent_compaction_block: bool = False
# Per-phase flags so the compaction block's start/delta/stop events
# are emitted (and the public state machine is advanced) in
@ -953,7 +954,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
current_content_block_index=self.current_content_block_index,
applied_edits=(self.applied_edits if is_final_chunk and not will_merge_into_held else None),
)
processed_chunk = self._with_refusal_stop_details(processed_chunk)
processed_chunk = self._with_stop_reason_fields(processed_chunk)
# Check if this is a usage chunk and we have a held stop_reason chunk
if will_merge_into_held:
@ -1188,7 +1189,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
current_content_block_index=self.current_content_block_index,
applied_edits=(self.applied_edits if is_final_chunk and not will_merge_into_held else None),
)
processed_chunk = self._with_refusal_stop_details(processed_chunk)
processed_chunk = self._with_stop_reason_fields(processed_chunk)
# Check if this is a usage chunk and we have a held stop_reason chunk
if will_merge_into_held:
@ -1406,14 +1407,22 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
def _increment_content_block_index(self):
self.current_content_block_index += 1
def _with_refusal_stop_details(
def _with_stop_reason_fields(
self,
processed_chunk: ContentBlockDelta | MessageBlockDelta,
) -> ContentBlockDelta | MessageBlockDelta:
if processed_chunk["type"] != "message_delta" or not self._refusal_text:
if processed_chunk["type"] != "message_delta":
return processed_chunk
delta: Final = processed_chunk["delta"]
if delta.get("stop_reason") == "max_tokens":
if self._paused_turn:
paused_delta: Final[MessageDelta] = {**delta, "stop_reason": "pause_turn"}
paused_chunk: Final[MessageBlockDelta] = {**processed_chunk, "delta": paused_delta}
return paused_chunk
if delta.get("stop_reason") == "refusal" and self._claude_refusal_stop_details is not None:
claude_refusal_delta: Final[MessageDelta] = {**delta, "stop_details": self._claude_refusal_stop_details}
claude_refusal_chunk: Final[MessageBlockDelta] = {**processed_chunk, "delta": claude_refusal_delta}
return claude_refusal_chunk
if not self._refusal_text or delta.get("stop_reason") == "max_tokens":
return processed_chunk
from litellm.llms.anthropic.messages.native.utils import (
refusal_stop_details,
@ -1470,9 +1479,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
@staticmethod
def _is_blank_delta(chunk: "ModelResponseStream") -> bool:
from litellm.llms.anthropic.common_utils import is_empty_unsigned_thinking_block
from litellm.llms.anthropic.messages.native.utils import (
openai_chat_refusal_text,
)
from litellm.llms.anthropic.messages.native.utils import refusal_reply_text
choice: Final = chunk.choices[0]
if choice.finish_reason is not None:
@ -1482,7 +1489,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
return False
if getattr(delta, "content", None):
return False
if openai_chat_refusal_text(delta):
if refusal_reply_text(delta):
return False
if getattr(delta, "reasoning_content", None):
return False
@ -1497,6 +1504,18 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
return False
return True
def _record_anthropic_stop_fields(self, delta: object) -> None:
from litellm.llms.anthropic.messages.native.utils import (
anthropic_refusal_stop_details,
anthropic_stop_reason,
)
if anthropic_stop_reason(delta) == "pause_turn":
self._paused_turn = True
claude_refusal_stop_details: Final = anthropic_refusal_stop_details(delta)
if claude_refusal_stop_details is not None:
self._claude_refusal_stop_details = claude_refusal_stop_details
def _should_start_new_content_block(self, chunk: "ModelResponseStream") -> bool:
"""
Determine if we should start a new content block based on the processed chunk.
@ -1507,16 +1526,15 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
- Different content types in the response
- Specific markers in the content
"""
from litellm.llms.anthropic.messages.native.utils import (
openai_chat_refusal_text,
)
from litellm.llms.anthropic.messages.native.utils import refusal_reply_text
from .transformation import LiteLLMAnthropicMessagesAdapter
self._record_anthropic_stop_fields(chunk.choices[0].delta)
if chunk.choices[0].finish_reason is not None:
return False
refusal_text: Final = openai_chat_refusal_text(chunk.choices[0].delta)
refusal_text: Final = refusal_reply_text(chunk.choices[0].delta)
if refusal_text is not None:
self._refusal_text = self._refusal_text + refusal_text

41
litellm/llms/anthropic/messages/adapters/transformation.py
View file

@ -150,7 +150,9 @@ from litellm.llms.anthropic.messages.native.mid_conversation_system import (
is_system_role_message,
)
from litellm.llms.anthropic.messages.native.utils import (
openai_chat_refusal_text,
anthropic_refusal_stop_details,
anthropic_stop_reason,
refusal_reply_text,
refusal_stop_details,
)
from litellm.types.llms.anthropic import (
@ -398,6 +400,10 @@ class LiteLLMAnthropicMessagesAdapter:
return None
def _streamed_tool_call_signature(self, tool_call_id: str, tool_call: object) -> str | None:
id_parts: Final = tool_call_id.split(THOUGHT_SIGNATURE_SEPARATOR, 1)
return id_parts[1] if len(id_parts) > 1 else self._extract_signature_from_tool_call(tool_call)
def _extract_signature_from_tool_use_content(self, content: Mapping[str, object]) -> str | None:
"""
Extract signature from a tool_use content block's provider_specific_fields.
@ -1467,7 +1473,7 @@ class LiteLLMAnthropicMessagesAdapter:
new_content.append(
AnthropicResponseContentBlockText(type="text", text=choice.message.content).model_dump()
)
if (refusal_text := openai_chat_refusal_text(choice.message)) is not None:
if (refusal_text := refusal_reply_text(choice.message)) is not None:
new_content.append(AnthropicResponseContentBlockText(type="text", text=refusal_text).model_dump())
# Handle tool calls (in parallel to text content)
if choice.message.tool_calls is not None and len(choice.message.tool_calls) > 0:
@ -1634,7 +1640,15 @@ class LiteLLMAnthropicMessagesAdapter:
tool_name_mapping=tool_name_mapping,
)
refusal_text: Final = next(
(text for choice in response.choices if (text := openai_chat_refusal_text(choice.message)) is not None),
(text for choice in response.choices if (text := refusal_reply_text(choice.message)) is not None),
None,
)
claude_refusal_stop_details: Final = next(
(
details
for choice in response.choices
if (details := anthropic_refusal_stop_details(choice.message)) is not None
),
None,
)
@ -1643,8 +1657,10 @@ class LiteLLMAnthropicMessagesAdapter:
## extract finish reason
openai_finish_reason: Final = response.choices[0].finish_reason if response.choices else "stop"
translated_finish_reason: Final = self._translate_openai_finish_reason_to_anthropic(
openai_finish_reason=openai_finish_reason
translated_finish_reason: Final = (
"pause_turn"
if response.choices and anthropic_stop_reason(response.choices[0].message) == "pause_turn"
else self._translate_openai_finish_reason_to_anthropic(openai_finish_reason=openai_finish_reason)
)
anthropic_finish_reason: Final = (
"compaction"
@ -1686,7 +1702,11 @@ class LiteLLMAnthropicMessagesAdapter:
usage=anthropic_usage,
content=anthropic_content,
stop_reason=anthropic_finish_reason,
stop_details=(refusal_stop_details(refusal_text) if anthropic_finish_reason == "refusal" else None),
stop_details=(
(claude_refusal_stop_details or refusal_stop_details(refusal_text))
if anthropic_finish_reason == "refusal"
else None
),
)
applied_edits: Final = polyfill_result.applied_edits_for_response() if polyfill_result else None
@ -1712,10 +1732,7 @@ class LiteLLMAnthropicMessagesAdapter:
):
raw_id = choice.delta.tool_calls[0].id or str(uuid.uuid4())
tool_name = choice.delta.tool_calls[0].function.name or ""
thought_sig: str | None = None
if THOUGHT_SIGNATURE_SEPARATOR in raw_id:
parts = raw_id.split(THOUGHT_SIGNATURE_SEPARATOR, 1)
thought_sig = parts[1] if len(parts) > 1 else None
thought_sig = self._streamed_tool_call_signature(raw_id, choice.delta.tool_calls[0])
tool_block: dict[str, object] = {
"type": "tool_use",
"id": normalize_anthropic_tool_use_id(raw_id),
@ -1727,7 +1744,7 @@ class LiteLLMAnthropicMessagesAdapter:
"signature": thought_sig,
}
return "tool_use", cast("ContentBlockContentBlockDict", tool_block)
elif (choice.delta.content is not None and len(choice.delta.content) > 0) or openai_chat_refusal_text(
elif (choice.delta.content is not None and len(choice.delta.content) > 0) or refusal_reply_text(
choice.delta
) is not None:
return "text", TextBlock(type="text", text="")
@ -1810,7 +1827,7 @@ class LiteLLMAnthropicMessagesAdapter:
return "thinking_delta", ContentThinkingBlockDelta(type="thinking_delta", thinking=reasoning_content)
else:
refusal_text: Final = "".join(
refusal for choice in choices if (refusal := openai_chat_refusal_text(choice.delta)) is not None
refusal for choice in choices if (refusal := refusal_reply_text(choice.delta)) is not None
)
return "text_delta", ContentTextBlockDelta(type="text_delta", text=text + refusal_text)

11
litellm/llms/anthropic/messages/context_management/editors/clear_tool_uses.py
View file

@ -74,6 +74,7 @@ class _ClearToolUsesFields(BaseModel):
clear_tool_inputs: bool | tuple[str, ...] | None = None
_BLOCK: Final = TypeAdapter(Mapping[str, object])
_BLOCKS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
# api.anthropic.com, measured 2026-10-19 and 2026-10-28: its wording for the shapes pydantic words differently
_API_MESSAGES: Final = MappingProxyType(
@ -317,12 +318,10 @@ def _clear_tool_results(
new_blocks: list[object] = []
mutated = False
for block in content:
if (
isinstance(block, dict)
and block.get("type") == "tool_result"
and block.get("tool_use_id") in ids_to_clear
):
new_blocks.append(_cleared_result(block))
if (parsed := _parse(_BLOCK, block)) is not None and _block_id(
parsed, "tool_result", "tool_use_id"
) in ids_to_clear:
new_blocks.append(_cleared_result(parsed))
mutated = True
cleared += 1
else:

14
litellm/llms/anthropic/messages/native/handler.py
View file

@ -19,6 +19,13 @@ from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheCont
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.exception_mapping_utils import exception_type
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.llms.anthropic.common_utils import (
AnthropicModelInfo,
strip_provider_specific_fields_from_anthropic_messages,
@ -31,13 +38,6 @@ from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.router_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
)
from litellm.types.llms.anthropic import AllAnthropicToolsValues
from litellm.types.llms.anthropic_messages.anthropic_request import AnthropicMetadata
from litellm.types.llms.anthropic_messages.anthropic_response import (

2
litellm/llms/anthropic/messages/native/interceptors/advisor.py
View file

@ -22,7 +22,7 @@ import litellm
import litellm.constants as _c
from litellm.litellm_core_utils.url_utils import validate_url
from litellm.llms.anthropic.common_utils import strip_advisor_blocks_from_messages
from litellm.router_utils.cooldown_handlers import mark_advisor_orchestration_failure
from litellm.router_api import mark_advisor_orchestration_failure
from litellm.types.llms.anthropic import ANTHROPIC_ADVISOR_TOOL_TYPE
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,

30
litellm/llms/anthropic/messages/native/utils.py
View file

@ -2,7 +2,7 @@ from collections.abc import Iterable, Mapping, Sequence
from functools import lru_cache
from typing import TYPE_CHECKING, Final, cast, get_type_hints
from pydantic import JsonValue, TypeAdapter, ValidationError
from pydantic import ConfigDict, JsonValue, TypeAdapter, ValidationError, with_config
from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheControlHook
from litellm.litellm_core_utils.core_helpers import as_str_mapping
@ -162,6 +162,34 @@ def openai_chat_refusal_text(message_or_delta: object) -> str | None:
return _mapping_str_field(getattr(message_or_delta, "provider_specific_fields", None), "refusal")
def anthropic_stop_reason(message_or_delta: object) -> str | None:
return _mapping_str_field(getattr(message_or_delta, "provider_specific_fields", None), "stop_reason")
def refusal_reply_text(message_or_delta: object) -> str | None:
if anthropic_stop_reason(message_or_delta) == "refusal":
return None
return openai_chat_refusal_text(message_or_delta)
@with_config(ConfigDict(extra="allow"))
class _AnthropicRefusalStopDetails(AnthropicStopDetails, total=False):
pass
_ANTHROPIC_REFUSAL_STOP_DETAILS: Final = TypeAdapter(_AnthropicRefusalStopDetails)
def anthropic_refusal_stop_details(message_or_delta: object) -> AnthropicStopDetails | None:
provider_specific_fields: Final = getattr(message_or_delta, "provider_specific_fields", None)
if _mapping_str_field(provider_specific_fields, "stop_reason") != "refusal":
return None
try:
return _ANTHROPIC_REFUSAL_STOP_DETAILS.validate_python(_mapping_field(provider_specific_fields, "stop_details"))
except ValidationError:
return refusal_stop_details(openai_chat_refusal_text(message_or_delta))
def _responses_message_refusal_text(item: object) -> str | None:
from openai.types.responses import ResponseOutputMessage, ResponseOutputRefusal

272
litellm/llms/anthropic/messages/stream_assembly.py
View file

@ -155,6 +155,8 @@ def collapse_pure_text_chunks(
continue
json_str = ev[idx + len("data:") :].strip()
if not json_str.startswith("{"):
continue
try:
data = json.loads(json_str)
except (json.JSONDecodeError, ValueError):
@ -206,6 +208,11 @@ def collapse_pure_text_chunks(
return out
def _has_non_object_data(event_str: str) -> bool:
data_start: Final = event_str.find("data:")
return data_start != -1 and not event_str[data_start + len("data:") :].lstrip().startswith("{")
def build_complete_streaming_response_legacy(
all_chunks: Sequence[str | bytes],
litellm_logging_obj: LiteLLMLoggingObj,
@ -243,6 +250,8 @@ def build_complete_streaming_response_legacy(
# text payload happens to contain "[DONE]" is not dropped.
if any(line.strip() == "data: [DONE]" for line in event_str.split("\n")):
continue
if _has_non_object_data(event_str):
continue
transformed_openai_chunk = anthropic_model_response_iterator.convert_str_chunk_to_generic_chunk(
chunk=event_str
)
@ -268,21 +277,25 @@ def build_complete_streaming_response_legacy(
return complete_streaming_response
def extract_sse_data(event_str: str) -> dict | None:
"""Parse the JSON object from the ``data:`` line of an Anthropic SSE event."""
def _sse_data_payload(event_str: str) -> str | None:
for line in split_sse_lines(event_str):
stripped = line.strip()
if stripped.startswith("data:"):
payload = stripped[len("data:") :].strip()
if not payload or payload == "[DONE]":
return None
try:
return json.loads(payload)
except (ValueError, TypeError):
return None
return stripped[len("data:") :].strip()
return None
def extract_sse_data(event_str: str) -> dict | None:
"""Parse the JSON object from the ``data:`` line of an Anthropic SSE event."""
payload: Final = _sse_data_payload(event_str)
if payload is None or not payload.startswith("{"):
return None
try:
return json.loads(payload)
except (ValueError, TypeError):
return None
def build_usage_only_response_from_chunks(
all_chunks: Sequence[str | bytes],
model: str,
@ -415,27 +428,81 @@ _ANTHROPIC_EVENT_TYPES: Final = frozenset(
)
_ERROR_PAYLOAD_KEYS: Final = frozenset({"type", "error", "request_id"})
def _is_anthropic_event_type(event_type: object) -> bool:
return isinstance(event_type, str) and event_type in _ANTHROPIC_EVENT_TYPES
def is_raw_sse_stream(all_chunks: Sequence[object]) -> bool:
return any(isinstance(chunk, (str, bytes)) for chunk in all_chunks)
def _joined_sse_stream(all_chunks: Sequence[object]) -> str | None:
raw: Final = b"".join(
def _joined_sse_bytes(all_chunks: Sequence[object]) -> bytes:
return b"".join(
chunk if isinstance(chunk, bytes) else chunk.encode("utf-8")
for chunk in all_chunks
if isinstance(chunk, (str, bytes))
)
def _joined_sse_stream(all_chunks: Sequence[object]) -> str | None:
try:
return codecs.getincrementaldecoder("utf-8")().decode(raw, final=False)
return codecs.getincrementaldecoder("utf-8")().decode(_joined_sse_bytes(all_chunks), final=False)
except UnicodeDecodeError:
return None
def _sse_event_data(event: str) -> Mapping[str, object] | None:
return as_str_mapping(extract_sse_data(event))
_JSON_ESCAPED_LINE_BREAKS: Final = MappingProxyType(
str.maketrans({"\x85": "\\u0085", "\u2028": "\\u2028", "\u2029": "\\u2029"})
)
def _sse_frames(sse_stream: str) -> Sequence[str]:
return "\n".join(split_sse_lines(sse_stream)).split("\n\n")
def _frames_of_the_widest_reader(sse_stream: str) -> Sequence[str]:
"""The stream's frames as httpx's ``str.splitlines`` reads them, past a form feed or U+2028 in any line."""
return "\n".join(sse_stream.splitlines()).split("\n\n")
def _sse_field(line: str) -> tuple[str, str]:
name, _, value = line.partition(":")
return name, value.removeprefix(" ")
def _sse_fields(frame: str) -> tuple[tuple[str, str], ...]:
"""The frame's fields as a client reads them: a name matches exactly and a comment is no field."""
return tuple(_sse_field(line) for line in frame.split("\n") if line and not line.startswith(":"))
def _carries_no_data(fields: Sequence[tuple[str, str]]) -> bool:
match fields:
case () | (("event", _),) | (("data", "[DONE]"),):
return True
case _:
return False
def _json_mapping(payload: str) -> Mapping[str, object] | None:
try:
value: Final[object] = json.loads(payload)
except ValueError:
return None
return as_str_mapping(value)
def _parsed_sse_events(sse_stream: str) -> tuple[Mapping[str, object], ...]:
return tuple(
event_data
for event in split_sse_chunk_into_events(sse_stream)
if (event_data := extract_sse_data(event)) is not None
if (event_data := _sse_event_data(event)) is not None
)
@ -444,7 +511,8 @@ def _anthropic_message_start(sse_stream: str) -> Mapping[str, object] | None:
(
message
for event_data in _parsed_sse_events(sse_stream)
if event_data.get("type") == "message_start" and isinstance(message := event_data.get("message"), dict)
if event_data.get("type") == "message_start"
and (message := as_str_mapping(event_data.get("message"))) is not None
),
None,
)
@ -482,11 +550,151 @@ def is_anthropic_sse_stream(all_chunks: Sequence[object]) -> bool:
sse_stream: Final = _joined_sse_stream(all_chunks)
if sse_stream is None:
return False
return any(event.get("type") in _ANTHROPIC_EVENT_TYPES for event in _parsed_sse_events(sse_stream))
return any(_is_anthropic_event_type(event.get("type")) for event in _parsed_sse_events(sse_stream))
def _is_json_no_reader_can_take(payload: str) -> bool:
try:
value: Final[object] = json.loads(payload)
except ValueError:
return False
if value is None:
return False
event: Final = as_str_mapping(value)
return event is None or isinstance(event.get("type"), (list, dict))
def has_unreadable_json_event(all_chunks: Sequence[object]) -> bool:
"""Whether a frame's data is JSON that no reader of these streams can take as an event."""
sse_stream: Final = codecs.getincrementaldecoder("utf-8")(errors="replace").decode(
_joined_sse_bytes(all_chunks), final=False
)
return any(_is_json_no_reader_can_take(_sse_data_payload(event) or "") for event in _sse_frames(sse_stream))
def anthropic_sse_event_type(event: str) -> object:
return next((event_data.get("type") for event_data in _parsed_sse_events(event)), None)
def anthropic_sse_event_types(all_chunks: Sequence[object]) -> frozenset[str]:
"""The string ``type`` of every event the stream sent."""
event_types: Final = (event.get("type") for event in _parsed_sse_events(_joined_sse_stream(all_chunks) or ""))
return frozenset(event_type for event_type in event_types if isinstance(event_type, str))
def _is_named_error_event(event: str) -> bool:
names: Final = tuple(value for name, value in _sse_fields(event) if name == "event")
return names[-1:] == ("error",)
def _is_error_event(event: str) -> bool:
event_type: Final = anthropic_sse_event_type(event)
return event_type == "error" or (_is_named_error_event(event) and not _is_anthropic_event_type(event_type))
def _frames_a_client_dispatches(sse_stream: str) -> Sequence[str]:
return "\n".join((*split_sse_lines(sse_stream), "")).split("\n\n")[:-1]
def _error_events(sse_stream: str) -> tuple[str, ...]:
return tuple(event for event in _frames_a_client_dispatches(sse_stream) if _is_error_event(event))
def _plain_error_event(event: str) -> str:
payload: Final = _sse_data_payload(event) or ""
one_line: Final = payload.translate(_JSON_ESCAPED_LINE_BREAKS) if _json_mapping(payload) is not None else payload
return "\n".join(("event: error", *(f"data: {line}" for line in one_line.splitlines()[:1])))
def anthropic_sse_error_events(all_chunks: Sequence[object]) -> tuple[str, ...]:
"""The ``error`` events of the stream, each as an ``event:`` line and the ``data:`` line assembly read.
A stream whose upstream fails after it started ends with an ``error`` event in place of
``message_stop``. That frame is the outcome the client is owed, not content, so assembly through
the upstream error leaves it out and a caller re-emitting a rewritten response puts it back,
without any other line of the frame, since that is content no guardrail read.
A frame counts by its data ``type``, or by its SSE ``event:`` name unless its data is another
Anthropic event, since a client SDK raises on the name alone and an upstream that is not
Anthropic may send anything as the data.
"""
return tuple(_plain_error_event(event) for event in _error_events(_joined_sse_stream(all_chunks) or ""))
def anthropic_sse_stopped_block_index(event: str) -> object:
return next(
(
event_data.get("index")
for event_data in _parsed_sse_events(event)
if event_data.get("type") == "content_block_stop"
),
None,
)
def _block_index(event_data: Mapping[str, object]) -> int | None:
index: Final = event_data.get("index")
return index if isinstance(index, int) else None
def _started_blocks(events: Sequence[Mapping[str, object]]) -> Mapping[int, Mapping[str, object]]:
starts: Final = (event for event in events if event.get("type") == "content_block_start")
return MappingProxyType(
{
index: block
for event in starts
if (index := _block_index(event)) is not None
and (block := as_str_mapping(event.get("content_block"))) is not None
}
)
def anthropic_sse_started_blocks(all_chunks: Sequence[object]) -> Mapping[int, Mapping[str, object]]:
"""The ``content_block`` each ``content_block_start`` of the stream opened, by block index."""
return _started_blocks(_parsed_sse_events(_joined_sse_stream(all_chunks) or ""))
def _unfinished_blocks(events: Sequence[Mapping[str, object]]) -> Mapping[int, Mapping[str, object]]:
stopped: Final = frozenset(_block_index(event) for event in events if event.get("type") == "content_block_stop")
return MappingProxyType({index: block for index, block in _started_blocks(events).items() if index not in stopped})
def anthropic_sse_unfinished_blocks(all_chunks: Sequence[object]) -> Mapping[int, Mapping[str, object]]:
"""The blocks the stream opened and never stopped, which only a failed upstream leaves."""
return _unfinished_blocks(_parsed_sse_events(_joined_sse_stream(all_chunks) or ""))
def _sse_events_with_data(sse_stream: str) -> tuple[tuple[str, Mapping[str, object]], ...]:
return tuple(
(event, event_data)
for event in split_sse_chunk_into_events(sse_stream)
if (event_data := _sse_event_data(event)) is not None
)
def _content_events_before_upstream_error(sse_stream: str) -> tuple[str, ...]:
"""The stream's events minus its upstream ``error`` and every cut-off block assembly cannot read.
A cut-off text block is still text the client was sent, so it stays. A cut-off tool call's
input is a JSON prefix, and a cut-off thinking block has no signature, so neither is a block the
client can act on, and both are left out.
"""
if not _error_events(sse_stream):
return (sse_stream,)
events: Final = _sse_events_with_data(sse_stream)
cut_off: Final = frozenset(
index
for index, block in _unfinished_blocks(tuple(event_data for _, event_data in events)).items()
if block.get("type") != "text"
)
return tuple(
event
for event, event_data in events
if event_data.get("type") != "error" and _block_index(event_data) not in cut_off
)
def assemble_anthropic_sse_stream(
all_chunks: Sequence[object], *, restore_identity: bool = False
all_chunks: Sequence[object], *, restore_identity: bool = False, through_upstream_error: bool = False
) -> ModelResponse | None:
"""Assemble raw Anthropic SSE frames into a ModelResponse.
@ -494,6 +702,11 @@ def assemble_anthropic_sse_stream(
assembler does not carry through. It is off by default so callers that re-emit the assembled
response keep the wire shape they had before this helper was shared. The writes land on a
freshly built object that is unreachable from caller state until returned.
A stream whose upstream failed after it started ends with an ``error`` event in place of
``message_stop`` and assembles to None, unless ``through_upstream_error`` asks for the content
the client was sent before the failure. That is for a guardrail that scans the stream and then
passes the upstream error on; a caller recording what was served keeps None.
"""
sse_stream: Final = _joined_sse_stream(all_chunks)
if sse_stream is None:
@ -502,9 +715,12 @@ def assemble_anthropic_sse_stream(
if message_start is None:
return None
model: Final = message_start.get("model") if restore_identity else None
content_events: Final = (
_content_events_before_upstream_error(sse_stream) if through_upstream_error else (sse_stream,)
)
try:
assembled: Final = build_complete_streaming_response(
all_chunks=(sse_stream,),
all_chunks=content_events,
litellm_logging_obj=None, # pyright: ignore[reportArgumentType] # only forwarded to stream_chunk_builder, which accepts None
model=model if isinstance(model, str) else "",
)
@ -522,6 +738,21 @@ def assemble_anthropic_sse_stream(
return assembled
def _is_error_payload(payload: str) -> bool:
event: Final = _json_mapping(payload)
if event is None or not _ERROR_PAYLOAD_KEYS.issuperset(event):
return False
return event.get("type") == "error" or ("type" not in event and isinstance(event.get("error"), Mapping))
def _is_error_frame(fields: Sequence[tuple[str, str]]) -> bool:
match fields:
case (("data", payload),) | (("event", "error"), ("data", payload)):
return _is_error_payload(payload)
case _:
return False
def is_sse_error_stream(all_chunks: Sequence[object]) -> bool:
"""Whether the buffered stream carries nothing but error frames.
@ -537,7 +768,6 @@ def is_sse_error_stream(all_chunks: Sequence[object]) -> bool:
sse_stream: Final = _joined_sse_stream(all_chunks)
if sse_stream is None:
return False
events: Final = _parsed_sse_events(sse_stream)
return len(events) > 0 and all(
event.get("type") == "error" or isinstance(event.get("error"), Mapping) for event in events
)
frames: Final = tuple(_sse_fields(frame) for frame in _frames_of_the_widest_reader(sse_stream))
with_data: Final = tuple(fields for fields in frames if not _carries_no_data(fields))
return len(with_data) > 0 and all(_is_error_frame(fields) for fields in with_data)

2
litellm/llms/anthropic/messages/utils.py
View file

@ -214,7 +214,7 @@ def normalize_reasoning_effort_value(
if chain is None:
return effort
from litellm.router_utils.reasoning_effort_capability import (
from litellm.litellm_core_utils.reasoning_effort_capability import (
deployment_rounds_reasoning_effort,
resolve_supported_reasoning_efforts,
)

12
litellm/llms/azure_ai/decisions/transformation.py
View file

@ -12,7 +12,7 @@ from litellm.llms.azure_ai.common_utils import (
AZURE_ENTRA_LITELLM_PARAM_KEYS,
get_azure_ai_auth_headers,
)
from litellm.llms.base_llm.decisions.transformation import BaseDecisionsConfig
from litellm.llms.base_llm.decisions.transformation import BaseDecisionsConfig, DecisionsInputLimits
_FOUNDRY_ROUTE_SUFFIX: Final = re.compile(r"(/api/projects/[^/]+)?(/openai/v1|/openai|/models|/v1)?$")
_AZURE_AI_CREDENTIAL_PARAM_KEYS: Final = AZURE_ENTRA_LITELLM_PARAM_KEYS | frozenset({"azure_ad_token_provider"})
@ -26,6 +26,16 @@ class AzureAIDecisionsConfig(BaseDecisionsConfig):
api_base_env = ("AZURE_AI_API_BASE",)
api_key_required = False
def get_input_limits(
self, model: str, custom_llm_provider: str, headers: Mapping[str, str]
) -> DecisionsInputLimits:
limits: Final = super().get_input_limits(model, custom_llm_provider, headers)
return (
limits
if limits.max_request_tokens is not None
else super().get_input_limits("Microsoft-Decision-1", custom_llm_provider, headers)
)
def missing_api_base_message(self, custom_llm_provider: str) -> str:
return "Missing AZURE_AI_API_BASE - set AZURE_AI_API_BASE or pass api_base, e.g. https://<resource>.services.ai.azure.com"

55
litellm/llms/base_llm/decisions/transformation.py
View file

@ -1,7 +1,7 @@
import json
from abc import ABC
from collections.abc import Callable, Mapping, Sequence
from dataclasses import replace
from dataclasses import dataclass, replace
from itertools import chain
from types import MappingProxyType
from typing import Final
@ -12,6 +12,7 @@ from pydantic_core import ErrorDetails
from typing_extensions import assert_never
import litellm
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
from litellm.litellm_core_utils.url_utils import join_api_base
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.secret_managers.main import get_secret_str
@ -297,13 +298,7 @@ def _systemone_question(question: DecisionsIRQuestion) -> Mapping[str, object]:
assert_never(question)
def ir_to_systemone_request(
model: str, request: DecisionsIRRequest, *, supports_audio: bool = False, supports_video: bool = False
) -> Mapping[str, object] | UnsupportedDecisionsRequest:
if request.audio and not supports_audio:
return UnsupportedDecisionsRequest(reason="audio input is not supported by this Decisions model")
if request.videos and not supports_video:
return UnsupportedDecisionsRequest(reason="video input is not supported by this Decisions model")
def ir_to_systemone_request(model: str, request: DecisionsIRRequest) -> Mapping[str, object]:
state: Final = _systemone_state(request.input)
keyed_questions: Final = zip(systemone_keys(request.questions), request.questions, strict=True)
images: Final = input_images(request.input)
@ -503,6 +498,16 @@ def ir_to_systemone_response(response: DecisionsIRResponse, request: DecisionsIR
)
def catalog_mode(model: str, custom_llm_provider: str) -> str | None:
try:
key: Final = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider)["key"]
except Exception: # noqa: BLE001 # get_model_info raises bare Exception for an unmapped model, which the catalog does not list
return None
entry: Final = GetModelCostMap.loaded_model_cost_map().get(key)
mode: Final = entry.get("mode") if entry is not None else None
return mode if isinstance(mode, str) else None
def _is_decisions_model(model: str, custom_llm_provider: str) -> bool:
try:
model_mode: Final = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider).get("mode")
@ -511,6 +516,12 @@ def _is_decisions_model(model: str, custom_llm_provider: str) -> bool:
return is_decisions_model_mode(model_mode)
@dataclass(frozen=True, slots=True)
class DecisionsInputLimits:
max_request_tokens: int | None = None
max_question_tokens: int | None = None
class BaseDecisionsConfig(ABC):
path: str = "/v1/systemone"
api_key_env: tuple[str, ...] = ()
@ -533,6 +544,19 @@ class BaseDecisionsConfig(ABC):
def provider_reported_cost(self, response: DecisionsIRResponse) -> float | None:
return None
def get_input_limits(
self, model: str, custom_llm_provider: str, headers: Mapping[str, str]
) -> DecisionsInputLimits:
try:
limit: Final = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider).get(
"max_input_tokens"
)
except Exception: # noqa: BLE001 # get_model_info raises bare Exception for an unmapped model
return DecisionsInputLimits()
return DecisionsInputLimits(
max_request_tokens=limit if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0 else None
)
def missing_api_base_message(self, custom_llm_provider: str) -> str:
return f"api_base is required for Decisions provider '{custom_llm_provider}'"
@ -621,12 +645,16 @@ class BaseDecisionsConfig(ABC):
model=model, custom_llm_provider=custom_llm_provider
)
def supports_audio_input(self, model: str, custom_llm_provider: str) -> bool:
def supports_audio_input(self, model: str, custom_llm_provider: str, deployment_flag: bool | None) -> bool:
if deployment_flag is not None:
return deployment_flag
return _is_decisions_model(model, custom_llm_provider) and litellm.supports_audio_input(
model=model, custom_llm_provider=custom_llm_provider
)
def supports_video_input(self, model: str, custom_llm_provider: str) -> bool:
def supports_video_input(self, model: str, custom_llm_provider: str, deployment_flag: bool | None) -> bool:
if deployment_flag is not None:
return deployment_flag
return _is_decisions_model(model, custom_llm_provider) and litellm.supports_video_input(
model=model, custom_llm_provider=custom_llm_provider
)
@ -637,12 +665,7 @@ class BaseDecisionsConfig(ABC):
request: DecisionsIRRequest,
custom_llm_provider: str,
) -> Mapping[str, object] | UnsupportedDecisionsRequest:
return ir_to_systemone_request(
self.request_model(model),
request,
supports_audio=self.supports_audio_input(model, custom_llm_provider),
supports_video=self.supports_video_input(model, custom_llm_provider),
)
return ir_to_systemone_request(self.request_model(model), request)
def unwrap_response(self, payload: object) -> object:
return payload

3
litellm/llms/base_llm/files/batch_records.py
View file

@ -19,6 +19,8 @@ def responses_batch_body_to_chat_body(
openai_request_body: Mapping[str, object],
custom_llm_provider: str | None = None,
drop_params: bool | None = None,
*,
caller_input: str | ResponseInputParam | None = None,
) -> dict[str, object]: # mutable-ok: provider transforms take the bridged chat body as a plain dict
"""
Rewrite the body of an OpenAI `/v1/responses` batch record as a Chat Completions body.
@ -52,5 +54,6 @@ def responses_batch_body_to_chat_body(
responses_api_request=responses_request,
custom_llm_provider=custom_llm_provider,
drop_params=drop_params,
caller_input=caller_input,
metadata=openai_request_body.get("metadata"),
)

27
litellm/llms/bedrock/chat/agentcore/transformation.py
View file

@ -544,6 +544,17 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
return AgentCoreParsedResponse(content=content, usage=usage_data, final_message=final_message)
@staticmethod
def _message_stop_chunk(model: str) -> ModelResponseStream:
chunk: Final = ModelResponseStream(
id=f"chatcmpl-{uuid.uuid4()}",
created=0,
model=model,
object="chat.completion.chunk",
)
chunk.choices = [StreamingChoices(finish_reason="stop", index=0, delta=Delta())]
return chunk
def _stream_agentcore_response_sync(
self,
response: httpx.Response,
@ -553,6 +564,8 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
Internal sync generator that parses SSE and yields ModelResponse chunks.
"""
buffer = ""
sent_finish = False # rebind-ok: flips when a finish chunk goes out
message_stopped = False # rebind-ok: flips on messageStop, the only end some runtimes send
for text_chunk in response.iter_text():
buffer += text_chunk
@ -624,7 +637,9 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
total_tokens=usage_data.get("totalTokens", 0),
),
)
sent_finish = True
yield chunk
message_stopped |= "messageStop" in event_payload
# Process final message
if "message" in data_obj and isinstance(data_obj["message"], dict):
@ -641,12 +656,16 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
delta=Delta(),
)
]
sent_finish = True
yield chunk
except json.JSONDecodeError:
verbose_logger.debug("Skipping non-JSON SSE line: %s", line[:100])
continue
if message_stopped and not sent_finish:
yield self._message_stop_chunk(model)
def get_sync_custom_stream_wrapper(
self,
model: str,
@ -775,6 +794,8 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
Internal async generator that parses SSE and yields ModelResponse chunks.
"""
buffer = ""
sent_finish = False # rebind-ok: flips when a finish chunk goes out
message_stopped = False # rebind-ok: flips on messageStop, the only end some runtimes send
async for text_chunk in response.aiter_text():
buffer += text_chunk
@ -846,7 +867,9 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
total_tokens=usage_data.get("totalTokens", 0),
),
)
sent_finish = True
yield chunk
message_stopped |= "messageStop" in event_payload
# Process final message
if "message" in data_obj and isinstance(data_obj["message"], dict):
@ -863,12 +886,16 @@ class AmazonAgentCoreConfig(BaseConfig, BaseAWSLLM):
delta=Delta(),
)
]
sent_finish = True
yield chunk
except json.JSONDecodeError:
verbose_logger.debug("Skipping non-JSON SSE line: %s", line[:100])
continue
if message_stopped and not sent_finish:
yield self._message_stop_chunk(model)
async def get_async_custom_stream_wrapper(
self,
model: str,

35
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -25,6 +25,7 @@ from litellm.constants import (
)
from litellm.litellm_core_utils.core_helpers import (
as_str_mapping,
drops_unsupported_params,
filter_exceptions_from_params,
filter_internal_params,
map_finish_reason,
@ -2129,6 +2130,24 @@ class AmazonConverseConfig(BaseConfig):
return data
@staticmethod
def _claude_messages_within_cache_control_cap(
messages: Sequence[AllMessageValues],
capability_model: str,
optional_params: Mapping[str, object],
litellm_params: Mapping[str, object],
) -> Sequence[AllMessageValues]:
if not drops_unsupported_params(litellm_params.get("drop_params")) or "claude" not in capability_model.lower():
return messages
from litellm.integrations.anthropic_cache_control_hook import (
messages_within_cache_control_cap,
tool_message_with_single_cache_mark,
)
return messages_within_cache_control_cap(
tuple(map(tool_message_with_single_cache_mark, messages)), optional_params.get("tools")
)
async def _async_transform_request(
self,
model: str,
@ -2140,7 +2159,13 @@ class AmazonConverseConfig(BaseConfig):
base_model: Final = litellm_params.get("base_model")
capability_model: Final = bedrock_capability_model(model, base_model)
messages, system_content_blocks = self._transform_system_message(
messages, model=capability_model, deployment_model=model
list(
self._claude_messages_within_cache_control_cap(
messages, capability_model, _JSON_OBJECT.validate_python(optional_params), litellm_params
)
),
model=capability_model,
deployment_model=model,
)
# Convert last user message to guarded_text if guardrailConfig is present
@ -2221,7 +2246,13 @@ class AmazonConverseConfig(BaseConfig):
base_model: Final = litellm_params.get("base_model")
capability_model: Final = bedrock_capability_model(model, base_model)
messages, system_content_blocks = self._transform_system_message(
messages, model=capability_model, deployment_model=model
list(
self._claude_messages_within_cache_control_cap(
messages, capability_model, _JSON_OBJECT.validate_python(optional_params), litellm_params
)
),
model=capability_model,
deployment_model=model,
)
# Convert last user message to guarded_text if guardrailConfig is present

10
litellm/llms/bedrock/files/transformation.py
View file

@ -42,6 +42,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
extract_file_data,
text_completion_prompt_to_messages,
)
from litellm.llms.anthropic.common_utils import AnthropicError
from litellm.llms.base_llm.base_utils import map_developer_role_to_system_role
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.files.batch_records import responses_batch_body_to_chat_body
@ -347,7 +348,14 @@ class _BatchInputLine:
record: _OpenAIBatchRecord
_BATCH_RECORD_CONVERSION_ERRORS: Final = (AttributeError, LookupError, TypeError, ValueError, BadRequestError)
_BATCH_RECORD_CONVERSION_ERRORS: Final = (
AttributeError,
LookupError,
TypeError,
ValueError,
BadRequestError,
AnthropicError,
)
def _batch_line_error(line_number: int, problem: str) -> BedrockError:

2
litellm/llms/clf_ai_gateway/chat/transformation.py
View file

@ -4,8 +4,8 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import UnsupportedParamsError
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.router_utils.reasoning_effort_capability import declared_reasoning_efforts_for_model
from litellm.secret_managers.main import get_secret_str
SUPPORTED_OPENAI_PARAMS: Final = (

44
litellm/llms/cohere/common_utils.py
View file

@ -3,6 +3,7 @@ from typing import Final, Literal
from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.llms.cohere import CohereV2StreamedToolCall
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import (
ChatCompletionToolCallChunk,
@ -222,6 +223,7 @@ class CohereV2ModelResponseIterator:
self.content_blocks: list = []
self.tool_index = -1
self.json_mode = json_mode
self.tool_call_indexes_with_arguments: frozenset[int] = frozenset()
def _parse_content_delta(self, chunk: dict) -> str:
"""Parse content-delta chunks to extract text."""
@ -234,20 +236,23 @@ class CohereV2ModelResponseIterator:
return content
return ""
def _parse_tool_call_delta(self, chunk: dict) -> ChatCompletionToolCallChunk | None:
"""Parse tool-call-delta chunks to extract tool calls."""
delta: Final = chunk.get("delta", {})
tool_calls: Final = delta.get("tool_calls", [])
if tool_calls:
return {
"id": tool_calls[0].get("id", ""),
"type": "function",
"function": {
"name": tool_calls[0].get("name", ""),
"arguments": tool_calls[0].get("arguments", ""),
},
}
return None
def _parse_tool_call_delta(self, chunk: dict, index: int) -> ChatCompletionToolCallChunk:
"""Parse tool-call-start and tool-call-delta chunks to extract tool calls."""
tool_call: Final = CohereV2StreamedToolCall.model_validate(chunk)
if tool_call.arguments:
self.tool_call_indexes_with_arguments = self.tool_call_indexes_with_arguments | {index}
return {
"id": tool_call.id,
"type": "function",
"function": {"name": tool_call.name, "arguments": tool_call.arguments},
"index": index,
}
def _parse_tool_call_end(self, index: int) -> ChatCompletionToolCallChunk | None:
"""Close a tool call that streamed no arguments with an empty JSON object, as OpenAI streams one."""
if index in self.tool_call_indexes_with_arguments:
return None
return {"id": None, "type": "function", "function": {"name": None, "arguments": "{}"}, "index": index}
def _parse_tool_plan_delta(self, chunk: dict) -> dict | None:
"""Parse tool-plan-delta events to extract tool plan."""
@ -301,7 +306,8 @@ class CohereV2ModelResponseIterator:
v2 format:
- Content: chunk.type == "content-delta" -> chunk.delta.message.content.text
- Tool calls: chunk.type == "tool-call-delta" -> chunk.delta.tool_calls
- Tool calls: chunk.type in ("tool-call-start", "tool-call-delta") -> chunk.delta.message.tool_calls
- Tool call end: chunk.type == "tool-call-end" -> "{}" arguments when the call streamed none
- Tool plan: chunk.event == "tool-plan-delta" -> chunk.data.delta.message.tool_plan
- Citations: chunk.event == "citation-start" -> chunk.data.delta.message.citations
- Finish: chunk.event == "message-end" -> chunk.data.delta.finish_reason
@ -321,8 +327,12 @@ class CohereV2ModelResponseIterator:
# Handle different chunk types
if chunk_type == "content-delta":
text = self._parse_content_delta(chunk)
elif chunk_type == "tool-call-delta":
tool_use = self._parse_tool_call_delta(chunk)
elif chunk_type in ("tool-call-start", "tool-call-delta", "tool-call-end"):
tool_use = (
self._parse_tool_call_end(index)
if chunk_type == "tool-call-end"
else self._parse_tool_call_delta(chunk, index)
)
elif event_type == "tool-plan-delta":
provider_specific_fields = self._parse_tool_plan_delta(chunk)
elif event_type == "citation-start":

8
litellm/llms/compaction.py
View file

@ -9,7 +9,7 @@ from pydantic import TypeAdapter
from litellm.types.llms.openai import AllMessageValues
if TYPE_CHECKING:
from litellm.router import Router
from litellm.router_api import RoutingContext
CompactionProtocol: TypeAlias = Literal["chat", "messages"]
_MAPPING: Final = TypeAdapter(Mapping[str, object])
@ -32,7 +32,9 @@ def get_native_compaction_provider(params: Mapping[str, object]) -> NativeCompac
return compaction if compaction.supports_native_compaction(params) else None
async def dispatch(router: Router, protocol: CompactionProtocol, payload: Mapping[str, object]) -> Mapping[str, object]:
async def dispatch(
router: RoutingContext, protocol: CompactionProtocol, payload: Mapping[str, object]
) -> Mapping[str, object]:
if protocol == "messages":
return _MAPPING.validate_python(
await router.aanthropic_messages(custom_llm_provider=None, client=None, **payload)
@ -45,4 +47,4 @@ async def dispatch(router: Router, protocol: CompactionProtocol, payload: Mappin
{key: value for key, value in payload.items() if key not in ("model", "messages", "stream")}
),
)
return _MAPPING.validate_python(response.model_dump())
return _MAPPING.validate_python(response.model_dump()) # pyright: ignore[reportAttributeAccessIssue] # stream=False: acompletion returns a ModelResponse

5
litellm/llms/custom_httpx/aiohttp_handler.py
View file

@ -14,6 +14,7 @@ from litellm.llms.base_llm.chat.transformation import BaseConfig
from litellm.llms.base_llm.image_variations.transformation import (
BaseImageVariationConfig,
)
from litellm.llms.custom_httpx.aiohttp_pool_diagnostics import build_pool_wait_trace_config
from litellm.llms.custom_httpx.aiohttp_transport import LiteLLMAiohttpTransport
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
@ -90,7 +91,7 @@ class BaseLLMAIOHTTPHandler:
if self.transport is None:
connector: Final = self._get_connector()
if connector:
return aiohttp.ClientSession(connector=connector)
return aiohttp.ClientSession(connector=connector, trace_configs=[build_pool_wait_trace_config()])
transport: Final = self.transport or self._get_or_create_transport()
if transport is not None and hasattr(transport, "_get_valid_client_session"):
@ -99,7 +100,7 @@ class BaseLLMAIOHTTPHandler:
except RuntimeError:
pass
return aiohttp.ClientSession()
return aiohttp.ClientSession(trace_configs=[build_pool_wait_trace_config()])
def _get_async_client_session(self, dynamic_client_session: ClientSession | None = None) -> ClientSession:
if dynamic_client_session:

130
litellm/llms/custom_httpx/aiohttp_pool_diagnostics.py Normal file
View file

@ -0,0 +1,130 @@
import asyncio
from types import SimpleNamespace
from typing import Final
from aiohttp import (
ClientSession,
TraceConfig,
TraceConnectionQueuedEndParams,
TraceConnectionQueuedStartParams,
TraceRequestExceptionParams,
TraceRequestRedirectParams,
TraceRequestStartParams,
)
from yarl import URL
from litellm._logging import verbose_logger
_POOL_WAIT_WARNING_SECONDS: Final = 1.0
class _PoolWaitContext(SimpleNamespace):
def __init__(self, trace_request_ctx: object = None) -> None:
super().__init__()
self.host: str | None = None
self.port: int | None = None
self.started_at: float | None = None
self.waited: float = 0.0
self.warning: asyncio.TimerHandle | None = None
self.warned: bool = False
def set_destination(self, params: TraceRequestStartParams) -> None:
self.host = params.url.host
self.port = params.url.port
def redirect(self, params: TraceRequestRedirectParams) -> None:
location: Final = params.response.headers.get("Location") or params.response.headers.get("URI")
if location is None:
return
try:
destination: Final = params.url.join(URL(location))
host: Final = destination.host
port: Final = destination.port
except ValueError:
return
self.host = host
self.port = port
def start(self, session: ClientSession) -> None:
loop: Final = asyncio.get_running_loop()
self.started_at = loop.time()
if not self.warned:
self.warning = loop.call_later(max(0.0, _POOL_WAIT_WARNING_SECONDS - self.waited), self.warn, session)
def elapsed(self) -> float:
return self.waited + (
asyncio.get_running_loop().time() - self.started_at if self.started_at is not None else 0.0
)
def warn(self, session: ClientSession) -> None:
if self.warned or self.started_at is None:
return
self.warned = True
connector: Final = session.connector
if connector is None:
return
verbose_logger.warning(
"aiohttp connection pool saturated: host=%s port=%s waited=%.3fs for a connection slot "
"(acquired_total=%s, limit=%s, limit_per_host=%s). "
"Wait occurred before sending the request upstream.",
self.host,
self.port,
self.elapsed(),
len(connector._acquired), # pyright: ignore[reportPrivateUsage] # aiohttp exposes no public acquired count
connector.limit,
connector.limit_per_host,
)
def finish(self, session: ClientSession, failed: bool = False) -> None:
if self.warning is not None:
self.warning.cancel()
self.warning = None
if self.started_at is None:
return
if failed or self.elapsed() >= _POOL_WAIT_WARNING_SECONDS:
self.warn(session)
self.waited = self.elapsed()
self.started_at = None
async def _on_request_start(session: ClientSession, context: SimpleNamespace, params: TraceRequestStartParams) -> None:
if isinstance(context, _PoolWaitContext):
context.set_destination(params)
async def _on_queued_start(
session: ClientSession, context: SimpleNamespace, params: TraceConnectionQueuedStartParams
) -> None:
if isinstance(context, _PoolWaitContext):
context.start(session)
async def _on_queued_end(
session: ClientSession, context: SimpleNamespace, params: TraceConnectionQueuedEndParams
) -> None:
if isinstance(context, _PoolWaitContext):
context.finish(session)
async def _on_request_exception(
session: ClientSession, context: SimpleNamespace, params: TraceRequestExceptionParams
) -> None:
if isinstance(context, _PoolWaitContext):
context.finish(session, failed=True)
async def _on_request_redirect(
session: ClientSession, context: SimpleNamespace, params: TraceRequestRedirectParams
) -> None:
if isinstance(context, _PoolWaitContext):
context.redirect(params)
def build_pool_wait_trace_config() -> TraceConfig:
trace: Final = TraceConfig(trace_config_ctx_factory=_PoolWaitContext)
trace.on_request_start.append(_on_request_start)
trace.on_connection_queued_start.append(_on_queued_start)
trace.on_connection_queued_end.append(_on_queued_end)
trace.on_request_exception.append(_on_request_exception)
trace.on_request_redirect.append(_on_request_redirect)
return trace

15
litellm/llms/custom_httpx/aiohttp_transport.py
View file

@ -110,6 +110,15 @@ def map_aiohttp_exceptions() -> Generator[None, None, None]:
raise mapped_exc(message) from exc
def _unread_payload(content: object) -> bytes:
"""Bytes the upstream sent before the connection dropped that aiohttp buffered but never handed out, since its
reads raise the stored payload error ahead of the buffer."""
if not isinstance(content, aiohttp.StreamReader):
return b""
buffered: Final = b"".join(content._buffer) # pyright: ignore[reportPrivateUsage] # no public read drains it
return buffered[content._buffer_offset :] # pyright: ignore[reportPrivateUsage] # see above
def _release_transport_anchor(_transport: object) -> None:
"""Release a transport anchored to a response stream. See ``_anchor_transport_to``."""
@ -144,6 +153,12 @@ class AiohttpResponseStream(httpx.AsyncByteStream):
raise httpx.ReadError(str(e)) from e
except aiohttp.http_exceptions.TransferEncodingError as e:
raise httpx.ReadError(str(e)) from e
except aiohttp.ClientPayloadError:
unread: Final = _unread_payload(self._aiohttp_response.content)
if unread:
yield unread
with map_aiohttp_exceptions():
raise
except Exception:
# For other exceptions, use the normal mapping
with map_aiohttp_exceptions():

2
litellm/llms/custom_httpx/http_handler.py
View file

@ -44,6 +44,7 @@ from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
)
from litellm.llms.custom_httpx.aiohttp_pool_diagnostics import build_pool_wait_trace_config
from litellm.llms.custom_httpx.http2_connection_spread import AsyncHTTP2ConnectionSpread
from litellm.types.llms.custom_http import *
@ -1456,6 +1457,7 @@ class AsyncHTTPHandler:
connector=TCPConnector(**transport_connector_kwargs),
cookie_jar=DummyCookieJar(),
trust_env=trust_env,
trace_configs=[build_pool_wait_trace_config()],
)
# Use shared session if provided and valid

39
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -131,6 +131,8 @@ from litellm.responses.streaming_iterator import (
ResponsesAPIStreamingIterator,
ResponsesWebSocketStreaming,
SyncResponsesAPIStreamingIterator,
TokenLimitCallback,
with_token_limit_checks,
)
from litellm.types.containers.main import (
ContainerFileListResponse,
@ -598,10 +600,10 @@ def _decoded_body_headers(response: httpx.Response) -> httpx.Headers:
)
def _collect_ws_project_quota_callbacks() -> tuple[ProjectQuotaCallback, ...]:
"""Duck-type discover proxy hooks exposing per-frame project ITPM/OTPM
enforcement, so the Responses WebSocket loop can charge every
``response.create`` frame, not just the connection's first one.
def _ws_callbacks_exposing(method_name: str) -> tuple[object, ...]:
"""Duck-type discover proxy hooks exposing a per-frame check, so the
Responses WebSocket loop can run it on every ``response.create`` frame,
not just the connection's first one.
Uses duck-typing on ``litellm.callbacks`` (rather than importing the
proxy hook directly) to avoid a layering violation (SDK importing from
@ -612,13 +614,28 @@ def _collect_ws_project_quota_callbacks() -> tuple[ProjectQuotaCallback, ...]:
callbacks: Final = cast( # cast-ok: callback registry is inspected before protocol use
Sequence[object], _litellm.callbacks
)
return tuple(callback for callback in callbacks if callable(getattr(callback, method_name, None)))
def _collect_ws_project_quota_callbacks() -> tuple[ProjectQuotaCallback, ...]:
return tuple(
cast(ProjectQuotaCallback, callback) # cast-ok: required callback method is callable
for callback in callbacks
if callable(getattr(callback, "enforce_project_io_token_quota_for_frame", None))
for callback in _ws_callbacks_exposing("enforce_project_io_token_quota_for_frame")
)
def _collect_ws_token_limit_callbacks() -> tuple[TokenLimitCallback, ...]:
return tuple(
cast(TokenLimitCallback, callback) # cast-ok: required callback method is callable
for callback in _ws_callbacks_exposing("enforce_token_limits_for_frame")
)
def _ws_model_group(litellm_metadata: Mapping[str, object] | None, model: str) -> str:
model_group: Final = (litellm_metadata or {}).get("model_group")
return model_group if isinstance(model_group, str) and model_group else model
def _unwrap_provider_error(error: Exception) -> Exception:
if not isinstance(error, httpx.TransportError):
return error
@ -7285,6 +7302,12 @@ class BaseLLMHTTPHandler:
- Forwards events over the websocket connection
"""
_ws_quota_callbacks: Final = _collect_ws_project_quota_callbacks()
_ws_turn_admission: Final = with_token_limit_checks(
turn_admission,
_collect_ws_token_limit_callbacks(),
user_api_key_dict,
_ws_model_group(litellm_metadata, model),
)
_ws_guardrail_callbacks: list = []
_ws_output_guardrail_callbacks: list = []
@ -7332,7 +7355,7 @@ class BaseLLMHTTPHandler:
custom_llm_provider=custom_llm_provider,
first_message=first_message,
quota_callbacks=_ws_quota_callbacks,
turn_admission=turn_admission,
turn_admission=_ws_turn_admission,
guardrail_callbacks=_ws_guardrail_callbacks, # pyright: ignore[reportUnknownArgumentType] # duck-typed callbacks from litellm.callbacks
frame_guard=frame_guard,
**kwargs,
@ -7429,7 +7452,7 @@ class BaseLLMHTTPHandler:
authorized_model=model,
custom_llm_provider=custom_llm_provider,
request_defaults=request_defaults,
turn_admission=turn_admission,
turn_admission=_ws_turn_admission,
frame_guard=frame_guard,
)
return await streaming.bidirectional_forward()

2
litellm/llms/databricks/embed/handler.py
View file

@ -26,6 +26,7 @@ class DatabricksEmbeddingHandler(OpenAILikeEmbeddingHandler, DatabricksBase):
aembedding=None,
custom_endpoint: bool | None = None,
headers: dict | None = None,
drop_params: bool = False,
) -> EmbeddingResponse:
# Check for custom user agent in optional_params or environment
# This allows partners building on LiteLLM to set their own telemetry
@ -58,4 +59,5 @@ class DatabricksEmbeddingHandler(OpenAILikeEmbeddingHandler, DatabricksBase):
aembedding=aembedding,
custom_endpoint=True,
headers=headers,
drop_params=drop_params,
)

77
litellm/llms/deepgram/common_utils.py
View file

@ -1,16 +1,21 @@
import json
import math
import re
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import Final
from urllib.parse import parse_qs, urlparse
from urllib.parse import parse_qs, urlparse, urlsplit, urlunsplit
import httpx
import litellm
from litellm.constants import DEEPGRAM_DEFAULT_API_BASE, DEEPGRAM_LISTEN_DEFAULT_MODEL
from litellm.constants import DEEPGRAM_DEFAULT_API_BASE, DEEPGRAM_LISTEN_DEFAULT_MODEL, DEEPGRAM_SPEAK_DEFAULT_MODEL
from litellm.litellm_core_utils.url_utils import join_api_base
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.passthrough_endpoints.pass_through_endpoints import WebSocketSessionUsage
from litellm.types.utils import LlmProviders
from litellm.utils import ModelInfoLookupMiss, lookup_model_info_for_registration
_WEBSOCKET_SCHEMES: Final = MappingProxyType({"https": "wss", "http": "ws", "wss": "wss", "ws": "ws"})
DEEPGRAM_LISTEN_CALLBACK_PARAMS: Final = frozenset({"callback", "callback_method"})
@ -199,3 +204,71 @@ def deepgram_listen_transcript(websocket_messages: Sequence[Mapping[str, object]
if frame.get("type") == "Results"
if (transcript := _final_transcript(frame)) is not None
)
_AURA_GENERATION: Final = re.compile(r"aura(?:-\d+)?(?=-[a-z])")
def deepgram_speak_websocket_target(api_base: str | None, query_string: str) -> str:
speak_url: Final = urlsplit(join_api_base(api_base or DEEPGRAM_DEFAULT_API_BASE, "speak"))
scheme: Final = _WEBSOCKET_SCHEMES.get(speak_url.scheme, speak_url.scheme)
params: Final = _first_occurrences(query_string)
query: Final = params if params.get("model") else params.remove("model").add("model", DEEPGRAM_SPEAK_DEFAULT_MODEL)
return urlunsplit((scheme, speak_url.netloc, speak_url.path, str(query), ""))
def deepgram_speak_model(upstream_url: str) -> str:
models: Final = parse_qs(urlparse(upstream_url).query).get("model")
return models[0] if models else DEEPGRAM_SPEAK_DEFAULT_MODEL
def deepgram_speak_requested_model(query_string: str) -> str:
"""Read off the upstream target, so the voice a key is authorized for is the voice Deepgram receives"""
return deepgram_speak_model(deepgram_speak_websocket_target(None, query_string))
def deepgram_speak_registry_key(upstream_url: str) -> str:
"""Deepgram bills every voice of an Aura generation at one rate, so ``aura-2-thalia-en`` is priced at the
``deepgram/aura-2`` row"""
model: Final = deepgram_speak_model(upstream_url)
generation: Final = _AURA_GENERATION.match(model)
return f"{LlmProviders.DEEPGRAM.value}/{generation.group() if generation else model}"
def deepgram_speak_character_rate(upstream_url: str) -> float | None:
"""Only an exact registry hit counts, so a voice is never billed at a neighbouring row's rate"""
registry_key: Final = deepgram_speak_registry_key(upstream_url)
model_info: Final = lookup_model_info_for_registration(registry_key, LlmProviders.DEEPGRAM.value)
if isinstance(model_info, ModelInfoLookupMiss) or model_info["key"] != registry_key:
return None
return model_info.get("input_cost_per_character")
@dataclass(frozen=True, slots=True)
class DeepgramSpeakMeter:
"""Holds the rate checked when the socket opened, so a cost-map reload mid-session can't change or drop the price"""
upstream_url: str
character_rate: float
def measure(self, client_text: str) -> int:
"""Counts a Speak message's text as it is sent. Deepgram also speaks, and bills, unflushed text on its own (a
long buffer, Close, a close frame), so counting only flushed text would let a client get audio free, while
counting on send can only over-bill text Deepgram discards, such as a Clear or an oversized Speak. Never
raises, so a malformed frame can't end the session"""
try:
message: Final[object] = json.loads(client_text)
except (ValueError, RecursionError):
return 0
match message:
case {"type": "Speak", "text": str() as text}:
return len(text)
case _:
return 0
def price(self, total: int) -> WebSocketSessionUsage:
return WebSocketSessionUsage(
model=deepgram_speak_model(self.upstream_url),
custom_llm_provider=LlmProviders.DEEPGRAM.value,
response_cost=total * self.character_rate,
)

Some files were not shown because too many files have changed in this diff Show more