scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-27 06:13:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 72

Merge remote-tracking branch 'origin/main' into litellm_azure_entra_token_followups
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
Details
LiteLLM Rust / rust-test (push) Has been cancelled
Details
LiteLLM Rust / rust-wheel (push) Has been cancelled
Details

Browse source 
# Conflicts:
#	litellm/llms/custom_httpx/llm_http_handler.py
This commit is contained in:
mateo-berri 2026-10-20 14:40:34 -07:00
commit 33f6e80aec
1319 changed files with 78883 additions and 13770 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

90
.githooks/pre-push
View file

@ -1,90 +0,0 @@
#!/usr/bin/env bash
#
# pre-push — enforce Conventional Branches
# https://conventional-branch.github.io/
#
# Branch format: <type>/<description>
# <type> must be one of: feature, bugfix, hotfix, release, chore
#
# Protected branches (always allowed):
# - main
# - dependabot/*
# - gh-readonly-queue/*
#
# Tag pushes and branch deletions are skipped.
# Bypass: git push --no-verify.
set -eu
ZERO_OID="0000000000000000000000000000000000000000"
ZERO_OID_SHA256="0000000000000000000000000000000000000000000000000000000000000000"
ALLOWED_TYPES="feature|bugfix|hotfix|release|chore"
BRANCH_PATTERN="^(${ALLOWED_TYPES})/.+"
PROTECTED_NAMES="main"
PROTECTED_PREFIXES="dependabot/ gh-readonly-queue/"
is_protected() {
branch="$1"
for name in $PROTECTED_NAMES; do
if [ "$branch" = "$name" ]; then
return 0
fi
done
for prefix in $PROTECTED_PREFIXES; do
case "$branch" in "$prefix"*) return 0 ;; esac
done
return 1
}
invalid=""
while read -r local_ref local_oid remote_ref remote_oid; do
# Branch deletion (no local commit being pushed).
if [ "$local_oid" = "$ZERO_OID" ] || [ "$local_oid" = "$ZERO_OID_SHA256" ]; then
continue
fi
# Only validate branch pushes; ignore tags and other ref namespaces.
case "$remote_ref" in
refs/heads/*) ;;
*) continue ;;
esac
branch="${remote_ref#refs/heads/}"
if is_protected "$branch"; then
continue
fi
if ! printf '%s' "$branch" | grep -Eq "$BRANCH_PATTERN"; then
invalid="$invalid $branch"
fi
done
if [ -n "$invalid" ]; then
cat >&2 <<EOF
✗ Branch name does not follow Conventional Branches.
Invalid:$invalid
Expected: <type>/<description>
Allowed types: feature, bugfix, hotfix, release, chore
Examples:
feature/weighted-round-robin
bugfix/streaming-empty-chunks
chore/bump-deps
hotfix/auth-bypass
Protected (always allowed): main, dependabot/*, gh-readonly-queue/*.
See https://conventional-branch.github.io/
Rename with: git branch -m <new-name>
To bypass (use sparingly): git push --no-verify
EOF
exit 1
fi
exit 0

2
.github/CODEOWNERS vendored
View file

@ -1,2 +1,4 @@
/model_prices_and_context_window.json @mateo-berri @ryan-crabbe-berri @kerry-berri
/litellm/model_prices_and_context_window_backup.json @mateo-berri @ryan-crabbe-berri @kerry-berri
/helm/ @yassin-berriai
/terraform/ @yassin-berriai

9
.github/PULL_REQUEST_TEMPLATE/general.md vendored
View file

@ -4,6 +4,13 @@
Drop every section you have nothing to put in, heading included: a bare "## Relevant issues" or
"## Affected release" with nothing under it must not appear in the final description -->
<!-- Two or three plain sentences a teammate can read in 10 seconds, written the way you'd explain it in Slack
Say what the PR does for users and why, then anything that keeps it working over time (a sync job, a config someone owns)
No headings, bullets, tables, or code names here. Everything else goes inside the Details block below -->
<details>
<summary>Details</summary>
## TLDR
<!-- Fill in the bullets below and keep each one short and concrete: one line per bullet, roughly 10 words max
@ -164,3 +171,5 @@ Example checklists:
## Final Attestation
- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
</details>

2
.github/ci-coverage-allowlist.yml vendored
View file

@ -5,7 +5,7 @@ description: >-
test_paths:
- reason: >-
litellm.agent() end-to-end suite. It drives the real claude, codex and opencode CLIs and
litellm.agent() end-to-end suite. It drives the real claude, codex, opencode and pi CLIs and
deepagents against a live LiteLLM AI Gateway, so it needs those binaries on PATH plus
LITELLM_PROXY_API_BASE / LITELLM_PROXY_API_KEY, and skips without them. Run manually
before changing litellm/harness; the mocked coverage runs in tests/unit/harness and

3
.github/workflows/test-unit.yml vendored
View file

@ -266,6 +266,7 @@ jobs:
tests/unit/llms/featherless_ai
tests/unit/llms/firecrawl
tests/unit/llms/fireworks_ai
tests/unit/llms/friendliai
tests/unit/llms/gdc
tests/unit/llms/gemini
tests/unit/llms/gigachat
@ -321,6 +322,7 @@ jobs:
tests/unit/llms/pass_through
tests/unit/llms/perplexity
tests/unit/llms/pg_vector
tests/unit/llms/pi
tests/unit/llms/publicai
tests/unit/llms/ragflow
tests/unit/llms/recraft
@ -410,6 +412,7 @@ jobs:
tests/unit/embeddings
tests/unit/endpoints
tests/unit/files
tests/unit/fine_tuning
tests/unit/images
tests/unit/interactions
tests/unit/messages

10
CONTRIBUTING.md
View file

@ -39,12 +39,12 @@ git clone https://github.com/YOUR_USERNAME/litellm.git
cd litellm
# Create a new branch for your feature (see "Commit and Branch Conventions" below)
git checkout -b feature/your-feature
git checkout -b litellm_your_feature
# Install development dependencies
make install-dev
# Install git hooks that enforce commit + branch conventions (one-time, opt-in)
# Install git hooks that enforce commit conventions (one-time, opt-in)
make install-hooks
# Verify your setup works
@ -55,7 +55,7 @@ That's it! Your local development environment is ready.
## Commit and Branch Conventions
Commits follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) and branches follow [Conventional Branches](https://conventional-branch.github.io/). Run `make install-hooks` once per clone to enable the local git hooks that enforce these — see the [contributor docs](https://docs.litellm.ai/docs/extras/contributing_code#commit-and-branch-conventions) for the full type list, examples, the protected-branch bypass list, and how to opt out.
Commits follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/). Internal contributors name branches with the `litellm_` prefix and no `/`, and fork branches can use any name. Run `make install-hooks` once per clone to enable the local hook that enforces the commit convention. See the [contributor docs](https://docs.litellm.ai/docs/extras/contributing_code#commit-and-branch-conventions) for details
### 2. Development Workflow
@ -78,8 +78,8 @@ uv run pytest tests/unit/<your_test_file>.py -v
git add .
git commit -m "feat(scope): your descriptive commit message"
# Push and create a PR (branch must follow Conventional Branches — see above)
git push origin feature/your-feature
# Push and create a PR
git push origin litellm_your_feature
```
## Adding Testing

4
Makefile
View file

@ -22,7 +22,7 @@ help:
@echo " make install-proxy-dev-ci - Install proxy dev dependencies (CI-compatible)"
@echo " make install-test-deps - Install the full local test environment"
@echo " make install-helm-unittest - Install helm unittest plugin"
@echo " make install-hooks - Install git hooks (Conventional Commits + Branches)"
@echo " make install-hooks - Install git hooks (Conventional Commits)"
@echo " make check - Run CI-equivalent lint on staged files, or on the diff vs the base branch when nothing is staged"
@echo " make pre-commit - Legacy alias for make check"
@echo " make format - Apply ruff format code formatting"
@ -120,7 +120,7 @@ install-helm-unittest:
helm plugin install https://github.com/helm-unittest/helm-unittest --version v0.8.2; \
}
# Install git hooks that enforce Conventional Commits and Conventional Branches.
# Install git hooks that enforce Conventional Commits.
# Opt-in: not chained into install-dev.
install-hooks:
./scripts/install_git_hooks.sh

6
README.md
View file

@ -287,7 +287,7 @@ For MCP OAuth, an upstream may advertise dynamic client registration but refuse
</details>
<details>
<summary><b>Agents</b> - Run Claude Code, Codex, OpenCode or Deep Agents on any model (Python SDK)</summary>
<summary><b>Agents</b> - Run Claude Code, Codex, OpenCode, pi or Deep Agents on any model (Python SDK)</summary>
### Python SDK - Agents
@ -296,7 +296,7 @@ import litellm
from litellm import Harness, sandbox
result = litellm.agent(
Harness.CLAUDE_CODE, # or Harness.CODEX, Harness.OPENCODE, Harness.DEEPAGENTS
Harness.CLAUDE_CODE, # or Harness.CODEX, Harness.OPENCODE, Harness.PI, Harness.DEEPAGENTS
"Find why tests/test_router.py is flaky and fix it.",
sandbox=sandbox.local("./repo"),
model="litellm_proxy/claude-sonnet-4-5", # a model group on your AI Gateway
@ -305,7 +305,7 @@ result = litellm.agent(
print(result.text, result.cost, [f.path for f in result.files])
```
Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call the agent makes goes through your AI Gateway, tagged `harness,claude_code`. Drop the `litellm_proxy/` prefix to call a provider directly. Install `starlette uvicorn` plus the agent's CLI (`claude`, `codex` or `opencode`), or `deepagents langchain-litellm` for Deep Agents.
Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call the agent makes goes through your AI Gateway, tagged `harness,claude_code`. Drop the `litellm_proxy/` prefix to call a provider directly. Install `starlette uvicorn` plus the agent's CLI (`claude`, `codex`, `opencode` or `pi`), or `deepagents langchain-litellm` for Deep Agents.
[**Docs: Agent Harnesses**](https://docs.litellm.ai/docs/harness)

41
cookbook/litellm_proxy_server/batch_api/bedrock/README.md Normal file
View file

@ -0,0 +1,41 @@
# Bedrock batch endpoints
Set `aws_bedrock_control_plane_endpoint` on a deployment when batch create, retrieve, and cancel requests must use a custom Bedrock control-plane endpoint. The proxy signs these requests for the configured URL. Without this setting, batch requests use the standard regional Bedrock endpoint
```yaml
model_list:
- model_name: bedrock-batch
litellm_params:
model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
aws_region_name: us-east-1
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_batch_role_arn: os.environ/AWS_BATCH_ROLE_ARN
s3_output_bucket_name: os.environ/AWS_S3_OUTPUT_BUCKET_NAME
aws_bedrock_control_plane_endpoint: os.environ/BEDROCK_CONTROL_PLANE_ENDPOINT
```
Set `BEDROCK_CONTROL_PLANE_ENDPOINT` to your Bedrock control-plane VPC endpoint or compatible gateway base URL, without `/model-invocation-job`. The proxy adds the operation path. The endpoint must be reachable from the proxy, and the configured AWS region must match the endpoint's signing region
`api_base` and `aws_bedrock_runtime_endpoint` configure runtime inference separately. They do not override the batch control-plane endpoint. `s3_endpoint_url` configures S3 file access separately
The control-plane setting belongs in deployment configuration. Sending `aws_bedrock_control_plane_endpoint` in an API request body is rejected. There is no automatic `AWS_BEDROCK_CONTROL_PLANE_ENDPOINT` environment fallback, so reference the desired environment variable explicitly as shown above
Start the proxy with this configuration and set `LITELLM_API_BASE` to its URL. Set `BEDROCK_BATCH_INPUT_URI` to a valid input JSONL file in S3 that the batch service role can read
```bash
curl -sS "$LITELLM_API_BASE/v1/batches" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"bedrock-batch\",\"input_file_id\":\"$BEDROCK_BATCH_INPUT_URI\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\"}"
```
Use the returned `id` as `BATCH_ID` to retrieve or cancel the same job through the deployment's configured endpoint
```bash
curl -sS "$LITELLM_API_BASE/v1/batches/$BATCH_ID?model=bedrock-batch" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
curl -sS -X POST "$LITELLM_API_BASE/v1/batches/$BATCH_ID/cancel?model=bedrock-batch" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
```

16
enterprise/litellm_enterprise/integrations/custom_guardrail.py
View file

@ -20,9 +20,15 @@ class EnterpriseCustomGuardrailHelper:
the tag's value (the mode for that tag).
- If no request tag matches, fall back to default mode(s).
"""
from litellm.litellm_core_utils.core_helpers import (
get_metadata_variable_name_from_kwargs,
)
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
)
from litellm.router_strategy.tag_based_routing import (
without_baggage_mapped_tags,
)
from litellm.proxy._types import CommonProxyErrors
from litellm.proxy.proxy_server import premium_user
@ -36,9 +42,13 @@ class EnterpriseCustomGuardrailHelper:
proxy_server_request = data.get("proxy_server_request", {})
request_tags = StandardLoggingPayloadSetup.get_request_tags(
litellm_params=data,
proxy_server_request=proxy_server_request,
request_tags = without_baggage_mapped_tags(
StandardLoggingPayloadSetup.get_request_tags(
litellm_params=data,
proxy_server_request=proxy_server_request,
),
data.get(get_metadata_variable_name_from_kwargs(data)),
keep_serving_deployment_tags=True,
)
# Check if any request tag matches a Mode tag key

151
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -4,6 +4,7 @@ Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if t
import asyncio
import json
import re
from collections.abc import Mapping, Sequence
from dataclasses import replace as dataclasses_replace
from datetime import datetime, timedelta, timezone
@ -39,7 +40,12 @@ if TYPE_CHECKING:
from litellm.types.utils import LiteLLMBatch
def _error_names_resource(error: Exception, resource_id: str) -> bool:
return resource_id in re.findall(r"""[^\s"'`]+""", str(error))
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
RECENT_BATCH_POLL_WINDOW: Final = timedelta(hours=48)
PROVIDER_TERMINAL_BATCH_STATUSES: Final[Tuple[str, ...]] = (
"completed",
@ -109,6 +115,11 @@ class CheckBatchCost:
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
self.batch_processed_support_confirmed: bool = False
self._poll_cursor: tuple[datetime, str] | None = None
self._poll_before: datetime = datetime.now(timezone.utc)
self._arrival_cursor: tuple[datetime, str] | None = None
self._arrival_before: datetime = self._poll_before
self._poll_arrivals: bool = False
self._line_item_stores: Final[set[asyncio.Task[None]]] = set()
self._line_item_store_turn: Final = asyncio.Lock()
self._poll_idle: Final = asyncio.Event()
@ -161,29 +172,9 @@ class CheckBatchCost:
f"(older than {MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days) as stale_expired"
)
if not self._has_batch_processed_column:
return
# A row already in a terminal status is never rewritten by the sweep above, so
# without this it keeps a poll-page slot forever and starves newer batches.
retired: Final = await _managed_object_table(self.prisma_client).update_many(
where={
"file_purpose": "batch",
"batch_processed": False,
"status": {"in": ["complete", "completed"]},
"created_at": {"lt": cutoff},
},
data={"batch_processed": True},
)
if retired > 0:
verbose_proxy_logger.warning(
f"CheckBatchCost: gave up on {retired} completed managed objects older than "
f"{MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days that were never costed"
)
async def _fallback_find_jobs(self) -> "Sequence[ManagedObjectRow]":
"""Query batch jobs without the batch_processed filter (for older schemas)."""
return await _managed_object_table(self.prisma_client).find_many(
return await self._find_jobs(
where={
"file_purpose": "batch",
"status": {
@ -197,10 +188,59 @@ class CheckBatchCost:
]
},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
async def _find_jobs(self, where: Mapping[str, object]) -> Sequence[ManagedObjectRow]:
arrivals: Final = self._poll_arrivals
page: Final = await self._find_job_page(where, arrivals=arrivals)
self._poll_arrivals = not arrivals if page else arrivals
if page:
return page
return await self._find_job_page(where, arrivals=not arrivals)
async def _find_job_page(self, where: Mapping[str, object], *, arrivals: bool) -> Sequence[ManagedObjectRow]:
cursor: Final = self._arrival_cursor if arrivals else self._poll_cursor
if cursor is None:
if arrivals:
self._arrival_before = datetime.now(timezone.utc)
else:
self._poll_before = datetime.now(timezone.utc) - RECENT_BATCH_POLL_WINDOW
before: Final = self._arrival_before if arrivals else self._poll_before
page: Final = await _managed_object_table(self.prisma_client).find_many(
where={
**where,
"created_at": {
"lte": before,
**({"gt": before - RECENT_BATCH_POLL_WINDOW} if arrivals else {}),
},
**(
{
"OR": [
{"created_at": {"gt": cursor[0]}},
{"created_at": cursor[0], "id": {"gt": cursor[1]}},
]
}
if cursor is not None
else {}
),
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order=[{"created_at": "asc"}, {"id": "asc"}],
)
next_cursor: Final = (page[-1].created_at, page[-1].id) if len(page) == MAX_OBJECTS_PER_POLL_CYCLE else None
if arrivals:
self._arrival_cursor = next_cursor
else:
self._poll_cursor = next_cursor
return page
def _unprocessed_snapshot(self, job: ManagedObjectRow) -> Mapping[str, object]:
return {
"id": job.id,
"updated_at": job.updated_at,
**({"batch_processed": False} if self._has_batch_processed_column else {}),
}
async def _retire_job(self, job: "ManagedObjectRow", reason: str) -> None:
"""
Take a row that can never be costed out of the poll page. Leaving it selectable
@ -208,24 +248,19 @@ class CheckBatchCost:
once enough such rows accumulate no newer batch is ever reached. Older schemas
without batch_processed can only be excluded through the status filter.
"""
data: Final = (
{"batch_processed": True}
if self._has_batch_processed_column
else {"status": "stale_expired"}
)
data: Final = {"batch_processed": True} if self._has_batch_processed_column else {"status": "stale_expired"}
try:
await _managed_object_table(self.prisma_client).update(
where={"id": job.id},
retired: Final = await _managed_object_table(self.prisma_client).update_many(
where=self._unprocessed_snapshot(job),
data=data,
)
except Exception as db_err:
verbose_proxy_logger.error(
f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}"
)
verbose_proxy_logger.error(f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}")
return
if retired == 0:
return
verbose_proxy_logger.warning(
f"CheckBatchCost: job {job.id} can never be costed ({reason}), "
"so it will no longer be polled"
f"CheckBatchCost: job {job.id} can never be costed ({reason}), so it will no longer be polled"
)
async def _claim_job_for_costing(self, job: "ManagedObjectRow", response: "LiteLLMBatch") -> bool:
@ -312,13 +347,13 @@ class CheckBatchCost:
from litellm.exceptions import NotFoundError
return isinstance(error, (NotFoundError, openai.NotFoundError)) and batch_id in str(error)
return isinstance(error, (NotFoundError, openai.NotFoundError)) and _error_names_resource(error, batch_id)
def _batch_deployment_exists(self, model_id: str) -> bool:
"""A 404 only proves the batch is gone when it came from the batch's own
deployment. Once that deployment leaves the router, default fallbacks can
silently send the retrieve to a provider that never saw the batch, so its
404 must not retire the row; the staleness sweep bounds it instead."""
404 must not retire a potentially recoverable row."""
return self.llm_router.get_deployment(model_id=model_id) is not None
@staticmethod
@ -333,7 +368,7 @@ class CheckBatchCost:
if not output_file_id:
return False
return isinstance(error, (NotFoundError, openai.NotFoundError)) and output_file_id in str(error)
return isinstance(error, (NotFoundError, openai.NotFoundError)) and _error_names_resource(error, output_file_id)
async def _finalize_unbilled_terminal_job(
self, job: "ManagedObjectRow", response: "LiteLLMBatch"
@ -355,18 +390,18 @@ class CheckBatchCost:
db_batch_object=job,
unified_batch_id=_is_base64_encoded_unified_file_id(job.unified_object_id),
)
update_data: Final[dict] = {
update_data: Final[dict[str, object]] = {
"status": response.status,
"file_object": response.model_dump_json(),
**({"batch_processed": True} if self._has_batch_processed_column else {}),
}
await _managed_object_table(self.prisma_client).update(
where={"id": job.id},
finalized: Final = await _managed_object_table(self.prisma_client).update_many(
where=self._unprocessed_snapshot(job),
data=update_data,
)
verbose_proxy_logger.info(
f"CheckBatchCost: marked job {job.id} as {response.status} in DB"
)
if finalized == 0:
return False
verbose_proxy_logger.info(f"CheckBatchCost: marked job {job.id} as {response.status} in DB")
return True
except Exception as db_err:
verbose_proxy_logger.error(
@ -619,7 +654,11 @@ class CheckBatchCost:
from litellm.files.types import FileContentCallOptions
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.litellm_core_utils.litellm_logging import deployment_pricing_model_info, mask_api_base_credentials
from litellm.litellm_core_utils.litellm_logging import (
deployment_pricing_model_info,
mask_api_base_credentials,
resolve_vertex_location_for_cost,
)
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
)
@ -748,6 +787,12 @@ class CheckBatchCost:
custom_llm_provider=batch_file_provider,
model_name=model_name,
model_info=deployment_model_info,
vertex_location=resolve_vertex_location_for_cost(
custom_llm_provider=llm_provider,
litellm_params=deployment_info.litellm_params.model_dump(exclude_none=True),
optional_params=None,
model=model_name,
),
)
error_file_failed_requests: Final = await count_error_file_failed_requests(
response,
@ -889,7 +934,10 @@ class CheckBatchCost:
)
from litellm.batches.batch_utils import fetch_batch_managed_file_content
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import deployment_pricing_model_info
from litellm.litellm_core_utils.litellm_logging import (
deployment_pricing_model_info,
resolve_vertex_location_for_cost,
)
try:
deployment_info: Final = self.llm_router.get_deployment(model_id=model_id)
@ -933,6 +981,12 @@ class CheckBatchCost:
cost_provider=cost_provider,
model_info=deployment_pricing_model_info(model_id=model_id, deployment_model=litellm_model_name),
metadata=line_item_metadata(await self._creator_attribution.batch_metadata(job, batch_id)),
vertex_location=resolve_vertex_location_for_cost(
custom_llm_provider=llm_provider,
litellm_params=deployment_info.litellm_params.model_dump(exclude_none=True),
optional_params=None,
model=model_name,
),
)
input_content: Final = (
b""
@ -1028,6 +1082,7 @@ class CheckBatchCost:
"""
try:
from litellm.integrations.prometheus import PrometheusLogger
prom_logger = PrometheusLogger.get_instance()
except Exception as e:
verbose_proxy_logger.error(f"CheckBatchCost: could not get Prometheus logger: {e}")
@ -1052,7 +1107,7 @@ class CheckBatchCost:
# endpoint may transition a batch to "complete" before
# CheckBatchCost runs. The batch_processed=False filter
# already prevents reprocessing finished batches.
jobs = await _managed_object_table(self.prisma_client).find_many(
jobs = await self._find_jobs(
where={
"file_purpose": "batch",
"batch_processed": False,
@ -1065,8 +1120,6 @@ class CheckBatchCost:
]
},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
self.batch_processed_support_confirmed = True
except Exception as query_err:

26
enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py
View file

@ -59,6 +59,19 @@ if TYPE_CHECKING:
from litellm.router import Router
_STORED_RESPONSE_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
class _RoutedModelInfo(BaseModel):
model_config = ConfigDict(frozen=True)
id: str | None = None
class _RoutedDeployment(BaseModel):
model_config = ConfigDict(frozen=True)
model_info: _RoutedModelInfo = _RoutedModelInfo()
_ROUTED_DEPLOYMENTS: Final = TypeAdapter(tuple[_RoutedDeployment, ...])
_RETRIEVE_RESPONSE_CALL_TYPE: Final = "aget_responses"
@ -314,10 +327,23 @@ class CheckResponsesCost:
INTERNAL_CALL_ORIGIN_METADATA_KEY: BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN,
**target.model_metadata,
**(_deployment_metadata(deployment) if deployment is not None else {}),
**self._requested_model_group(target, job.team_id),
}
),
)
def _requested_model_group(self, target: _PollTarget, team_id: str | None) -> Mapping[str, object]:
requested: Final = target.model_metadata.get("model_group")
if target.deployment is None or not isinstance(requested, str):
return MappingProxyType({})
routed: Final = _ROUTED_DEPLOYMENTS.validate_python(
self.llm_router.get_model_list(model_name=requested, team_id=team_id) or ()
)
serving_id: Final = target.deployment.model_info.id
return MappingProxyType(
{"model_group": requested} if any(row.model_info.id == serving_id for row in routed) else {}
)
async def _bill_once(
self,
job: ManagedObjectRow,

62
enterprise/litellm_enterprise/proxy/common_utils/managed_object_attribution.py
View file

@ -9,6 +9,7 @@ from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime
from typing import TYPE_CHECKING, Final, Protocol, cast
from pydantic import TypeAdapter
@ -27,6 +28,12 @@ if TYPE_CHECKING:
class ManagedObjectRow(Protocol):
@property
def created_at(self) -> datetime: ...
@property
def updated_at(self) -> datetime: ...
@property
def id(self) -> str: ...
@ -107,8 +114,10 @@ class ManagedObjectCreatorAttribution:
user's alias, because a resolvable name is more useful on the spend row than a null.
user_api_key_org_id is resolved too: the spend update writer reads it off this metadata
to increment organization spend, so leaving it out drops the cost from org accounting.
A background response also carries its end user and the key, team, and user budgets, the
way the read that billed it before did, and never names a key-scoped creator as its user.
A background response also carries its end user, the creating key's project and its alias,
and the key, team, and user budgets and per-model budgets, the way the read that billed it
before did, and never names a key-scoped creator as its user. A key that sets no budget of
its own takes its budget tier's, the same rule auth applies.
"""
def __init__(self, prisma_client: PrismaClient, poller_name: str) -> None:
@ -126,9 +135,23 @@ class ManagedObjectCreatorAttribution:
"user_api_key_user_id": _creator_user_id(job.created_by),
**_shared_metadata(job, rows),
**({"user_api_key_end_user_id": end_user_id} if end_user_id else {}),
**_budget_fields(rows.key, "user_api_key_spend", "user_api_key_max_budget"),
**_budget_fields(rows.team, "user_api_key_team_spend", "user_api_key_team_max_budget"),
**_budget_fields(rows.user, "user_api_key_user_spend", "user_api_key_user_max_budget"),
"user_api_key_project_id": getattr(rows.key, "project_id", None),
"user_api_key_project_alias": getattr(
getattr(rows.key, "litellm_project_table", None), "project_alias", None
),
**_key_budget_fields(rows.key),
**_budget_fields(
rows.team,
"user_api_key_team_spend",
"user_api_key_team_max_budget",
"user_api_key_team_model_max_budget",
),
**_budget_fields(
rows.user,
"user_api_key_user_spend",
"user_api_key_user_max_budget",
"user_api_key_user_model_max_budget",
),
}
async def _rows(self, job: ManagedObjectRow, object_id: str) -> _CreatorRows:
@ -155,7 +178,9 @@ class ManagedObjectCreatorAttribution:
if not api_key or _is_cli_session_key(api_key, created_by):
return None
try:
return await _token_table(self._prisma_client).find_unique(where={"token": api_key})
return await _token_table(self._prisma_client).find_unique(
where={"token": api_key}, include={"litellm_budget_table": True, "litellm_project_table": True}
)
except Exception as e:
verbose_proxy_logger.error(f"{self._poller_name}: could not look up the creating key for {object_id}: {e}")
return None
@ -212,7 +237,28 @@ def _organization_id(row: object) -> str | None:
return cast("str | None", getattr(row, "organization_id", None)) if row is not None else None
def _budget_fields(row: object, spend_key: str, max_budget_key: str) -> Mapping[str, object]:
def _budget_fields(row: object, spend_key: str, max_budget_key: str, model_max_budget_key: str) -> Mapping[str, object]:
if row is None:
return {}
return {spend_key: getattr(row, "spend", None), max_budget_key: getattr(row, "max_budget", None)}
return {
spend_key: getattr(row, "spend", None),
max_budget_key: getattr(row, "max_budget", None),
model_max_budget_key: getattr(row, "model_max_budget", None),
}
def _key_budget_fields(key: object) -> Mapping[str, object]:
if key is None:
return {}
tier: Final = getattr(key, "litellm_budget_table", None)
return {
"user_api_key_spend": getattr(key, "spend", None),
"user_api_key_max_budget": _own_or_tier_budget(key, tier, "max_budget"),
"user_api_key_model_max_budget": _own_or_tier_budget(key, tier, "model_max_budget"),
}
def _own_or_tier_budget(key: object, tier: object, field: str) -> object:
own: Final = getattr(key, field, None)
tier_value: Final = getattr(tier, field, None)
return own if tier_value is None or own not in (None, {}) else tier_value

63
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -6,6 +6,7 @@ import base64
import json
import time
from collections.abc import Awaitable, Callable, Iterator, Mapping, Sequence
from itertools import chain
from types import MappingProxyType
from typing import (
TYPE_CHECKING,
@ -479,6 +480,13 @@ def _with_managed_background_marker(request: Mapping[str, object]) -> dict[str,
_MANAGED_FILES_TARGET: Final = "managed_files"
_PROVIDER_FILE_ID_KWARGS: Final = MappingProxyType(
{
CallTypes.acreate_batch: ("input_file_id",),
CallTypes.acreate_fine_tuning_job: ("training_file", "validation_file"),
}
)
_PROVIDER_FILE_ID_KEYS: Final = tuple(dict.fromkeys(chain.from_iterable(_PROVIDER_FILE_ID_KWARGS.values())))
_PROVIDER_FILE_RETRIEVE_RETRY_DELAYS_SECONDS: Final = (0.5, 1.0, 2.0)
@ -1562,6 +1570,10 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
if isinstance(file_id, str) and file_id
)
await self.check_file_ids_access(fine_tuning_file_ids, user_api_key_dict)
if fine_tuning_file_ids:
data["model_file_id_mapping"] = await self.get_model_file_id_mapping(
fine_tuning_file_ids, user_api_key_dict.parent_otel_span
)
return data
@ -1576,24 +1588,27 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
if request_kwargs is None:
return healthy_deployments
input_file_id = cast(Optional[str], request_kwargs.get("input_file_id"))
model_file_id_mapping = cast(
Optional[Dict[str, Dict[str, str]]],
request_kwargs.get("model_file_id_mapping"),
request: Final = cast(Mapping[str, object], request_kwargs)
model_file_id_mapping: Final = cast(Mapping[str, Mapping[str, str]], request.get("model_file_id_mapping") or {})
requested_file_ids: Final = (request.get(key) for key in _PROVIDER_FILE_ID_KEYS)
model_ids_per_file: Final = tuple(
frozenset(model_file_id_mapping[file_id])
for file_id in requested_file_ids
if isinstance(file_id, str) and file_id in model_file_id_mapping
)
allowed_model_ids: Final[frozenset[str]] = (
model_ids_per_file[0].intersection(*model_ids_per_file[1:]) if model_ids_per_file else frozenset()
)
allowed_model_ids = []
if input_file_id and model_file_id_mapping:
model_id_dict = model_file_id_mapping.get(input_file_id, {})
allowed_model_ids = list(model_id_dict.keys())
if len(allowed_model_ids) == 0:
return healthy_deployments
return [
holders: Final[list[dict[str, object]]] = [
deployment
for deployment in healthy_deployments
if deployment.get("model_info", {}).get("id") in allowed_model_ids
]
return holders if holders or "input_file_id" in request else healthy_deployments
async def async_pre_call_deployment_hook(
self, kwargs: Dict[str, Any], call_type: Optional[CallTypes]
@ -1601,25 +1616,21 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"""
Allow modifying the request just before it's sent to the deployment.
"""
accessor_key: Optional[str] = None
if call_type and call_type == CallTypes.acreate_batch:
accessor_key = "input_file_id"
elif call_type and call_type == CallTypes.acreate_fine_tuning_job:
accessor_key = "training_file"
else:
accessor_keys: Final = _PROVIDER_FILE_ID_KWARGS.get(call_type) if call_type else None
if accessor_keys is None:
return kwargs
if accessor_key:
model_file_id_mapping = cast(Optional[Dict[str, Dict[str, str]]], kwargs.get("model_file_id_mapping"))
# model_info may be at top-level or nested under litellm_metadata
# (batch/file operations use litellm_metadata)
model_id = cast(Optional[str], kwargs.get("model_info", {}).get("id", None))
if model_id is None:
model_id = cast(
Optional[str],
kwargs.get("litellm_metadata", {}).get("model_info", {}).get("id", None),
)
for accessor_key in accessor_keys:
input_file_id = cast(Optional[str], kwargs.get(accessor_key))
model_file_id_mapping = cast(Optional[Dict[str, Dict[str, str]]], kwargs.get("model_file_id_mapping"))
# model_info may be at top-level or nested under litellm_metadata
# (batch/file operations use litellm_metadata)
model_id = cast(Optional[str], kwargs.get("model_info", {}).get("id", None))
if model_id is None:
model_id = cast(
Optional[str],
kwargs.get("litellm_metadata", {}).get("model_info", {}).get("id", None),
)
mapped_file_id: Optional[str] = None
if input_file_id and model_file_id_mapping and model_id:
mapped_file_id = model_file_id_mapping.get(input_file_id, {}).get(model_id, None)
@ -1813,7 +1824,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
),
)
async def get_model_file_id_mapping(self, file_ids: List[str], litellm_parent_otel_span: Span) -> dict:
async def get_model_file_id_mapping(self, file_ids: Sequence[str], litellm_parent_otel_span: Span) -> dict:
"""
Get model-specific file IDs for a list of proxy file IDs.
Returns a dictionary mapping litellm_proxy/ file_id -> model_id -> model_file_id

3
litellm/__init__.py
View file

@ -354,6 +354,7 @@ llm_guard_mode: Literal["all", "key-specific", "request-specific"] = "all"
guardrail_name_config_map: Dict[str, GuardrailItem] = {}
include_cost_in_streaming_usage: bool = False
reasoning_auto_summary: bool = False
reasoning_effort_rounding: Literal["up", "down"] = "up"
### PROMPTS ####
from litellm.types.prompts.init_prompts import PromptSpec
@ -550,6 +551,7 @@ from litellm.litellm_core_utils.get_model_cost_map import get_model_cost_map, ma
model_cost = get_model_cost_map(url=model_cost_map_url)
cost_discount_config: Dict[str, float] = {} # Provider-specific cost discounts {"vertex_ai": 0.05} = 5% discount
apply_search_free_tier: bool = False
cost_margin_config: Dict[
str, Union[float, Dict[str, float]]
] = {} # Provider-specific or global cost margins. Examples:
@ -2384,6 +2386,7 @@ _AGENT_EXPORTS: Final = frozenset(
"ClaudeCodeOptions",
"CodexOptions",
"OpenCodeOptions",
"PiOptions",
"DeepAgentsOptions",
"ToolLoopOptions",
}

2
litellm/a2a_protocol/litellm_completion_bridge/handler.py
View file

@ -80,7 +80,7 @@ class A2ACompletionBridgeHandler:
@staticmethod
def _build_completion_params(
params: dict[str, Any],
litellm_params: Mapping[str, Any],
litellm_params: Mapping[str, object],
api_base: str | None,
agent_extra_headers: Mapping[str, str] | None,
*,

4
litellm/a2a_protocol/main.py
View file

@ -602,9 +602,9 @@ def _build_streaming_logging_obj(
litellm_params: dict[str, object] | None,
metadata: dict[str, object] | None,
proxy_server_request: dict[str, object] | None,
start_time: datetime.datetime,
) -> Logging:
"""Build logging object for streaming A2A requests."""
start_time: Final = datetime.datetime.now()
model: Final = f"a2a_agent/{agent_name}"
logging_obj: Final = Logging(
@ -718,6 +718,7 @@ async def asend_message_streaming(
if request is None:
raise ValueError("request is required")
called_at: Final = datetime.datetime.now()
_raw_logging_obj: Final = kwargs.get("litellm_logging_obj")
logging_obj: Logging | None = _raw_logging_obj if isinstance(_raw_logging_obj, Logging) else None
@ -751,6 +752,7 @@ async def asend_message_streaming(
litellm_params=litellm_params,
metadata=metadata,
proxy_server_request=proxy_server_request,
start_time=called_at,
)
verbose_logger.info("A2A send_message_streaming request_id=%s, agent=%s", request.id, agent_name)

4
litellm/a2a_protocol/streaming_iterator.py
View file

@ -41,7 +41,7 @@ class A2AStreamingIterator:
self.request = request
self.logging_obj = logging_obj
self.agent_name = agent_name
self.start_time = datetime.now()
self.start_time = logging_obj.start_time
# Collect chunks for token counting
self.chunks: list[SendStreamingMessageResponse] = []
@ -54,6 +54,8 @@ class A2AStreamingIterator:
async def __anext__(self) -> "SendStreamingMessageResponse":
try:
chunk: Final = await self.stream.__anext__()
if self.logging_obj.completion_start_time is None:
self.logging_obj.update_completion_start_time(completion_start_time=datetime.now())
# Store chunk
self.chunks.append(chunk)

7
litellm/batches/batch_line_items.py
View file

@ -38,6 +38,7 @@ class BatchLineItemSource:
cost_provider: BatchFileProvider
model_info: ModelInfo | None
metadata: StandardBatchLineItemMetadata
vertex_location: str | None = None
def line_item_metadata(attribution: Mapping[str, object]) -> StandardBatchLineItemMetadata:
@ -140,7 +141,11 @@ def _line_item(
source: BatchLineItemSource, custom_id: str | None, request: Mapping[str, object] | None, entry: dict[str, object]
) -> StandardBatchLineItem:
priced: Final = batch_cost_and_usage(
(entry,), source.cost_provider, model_name=source.model, model_info=source.model_info
(entry,),
source.cost_provider,
model_name=source.model,
model_info=source.model_info,
vertex_location=source.vertex_location,
)
succeeded: Final = priced.successful_requests > 0
return StandardBatchLineItem(

37
litellm/batches/batch_utils.py
View file

@ -77,9 +77,10 @@ async def calculate_batch_cost_and_usage(
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "mistral"],
model_name: str | None = None,
model_info: ModelInfo | None = None,
vertex_location: str | None = None,
) -> BatchCostUsageResult:
"""Calculate the cost and usage of a batch, see batch_cost_and_usage."""
return batch_cost_and_usage(file_content_dictionary, custom_llm_provider, model_name, model_info)
return batch_cost_and_usage(file_content_dictionary, custom_llm_provider, model_name, model_info, vertex_location)
def batch_cost_and_usage(
@ -87,6 +88,7 @@ def batch_cost_and_usage(
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "mistral"],
model_name: str | None = None,
model_info: ModelInfo | None = None,
vertex_location: str | None = None,
) -> BatchCostUsageResult:
"""
Calculate the cost and usage of a batch.
@ -96,16 +98,21 @@ def batch_cost_and_usage(
pricing. Threaded through to batch_cost_calculator so that
deployment-specific pricing (e.g. input_cost_per_token_batches)
is used instead of the global cost map.
vertex_location: The Vertex AI location the batch ran on; non-global
locations apply the model's regional-endpoint uplift multiplier.
"""
first_row: Final = file_content_dictionary[0] if file_content_dictionary else None
if _uses_native_vertex_output(custom_llm_provider, model_name, first_row):
return calculate_vertex_ai_batch_cost_and_usage(file_content_dictionary, model_name, model_info=model_info)
return calculate_vertex_ai_batch_cost_and_usage(
file_content_dictionary, model_name, model_info=model_info, vertex_location=vertex_location
)
return _aggregate_batch_cost_usage_models(
entries=file_content_dictionary,
custom_llm_provider=custom_llm_provider,
model_name=model_name,
model_info=model_info,
vertex_location=vertex_location,
)
@ -115,6 +122,7 @@ async def handle_completed_batch(
model_name: str | None = None,
litellm_params: dict[str, object] | None = None,
model_info: ModelInfo | None = None,
vertex_location: str | None = None,
) -> BatchCostUsageResult:
"""Fetch a completed batch's output file and aggregate its cost, usage, and
models in a single pass over the JSONL lines, so the parsed file content is
@ -128,6 +136,8 @@ async def handle_completed_batch(
model_info: Optional deployment-level model info with custom pricing,
threaded through so a deployment's configured rates win over the
global cost map.
vertex_location: The Vertex AI location the batch ran on; non-global
locations apply the model's regional-endpoint uplift multiplier.
"""
# A completed batch whose request lines all failed has no output file - the
# results are written to a separate error_file_id and output_file_id is None.
@ -155,7 +165,10 @@ async def handle_completed_batch(
output_file_result: Final = (
calculate_vertex_ai_batch_cost_and_usage(
_iter_batch_output_entries(file_content), model_name, model_info=model_info
_iter_batch_output_entries(file_content),
model_name,
model_info=model_info,
vertex_location=vertex_location,
)
if _uses_native_vertex_output(
custom_llm_provider, model_name, next(_iter_batch_output_entries(file_content), None)
@ -165,6 +178,7 @@ async def handle_completed_batch(
custom_llm_provider=custom_llm_provider,
model_name=model_name,
model_info=model_info,
vertex_location=vertex_location,
)
)
@ -203,6 +217,7 @@ def _classify_output_line_stats(
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "bedrock", "mistral"],
model_name: str | None,
model_info: ModelInfo | None,
vertex_location: str | None = None,
) -> Iterator[_BatchOutputLineStats | _LineOutcome]:
"""Classify every output line in a single pass, so counting failures never needs
a second read of a potentially huge output file. A line the provider reported as
@ -213,7 +228,7 @@ def _classify_output_line_stats(
if not _batch_response_was_successful(entry, custom_llm_provider):
yield _LineOutcome.PROVIDER_FAILED
continue
stats = _safe_output_line_stats(entry, custom_llm_provider, model_name, model_info)
stats = _safe_output_line_stats(entry, custom_llm_provider, model_name, model_info, vertex_location)
yield stats if stats is not None else _LineOutcome.UNCOSTABLE
@ -222,13 +237,14 @@ def _safe_output_line_stats(
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "bedrock", "mistral"],
model_name: str | None,
model_info: ModelInfo | None,
vertex_location: str | None = None,
) -> _BatchOutputLineStats | None:
"""Return the stats for one provider-successful batch output line, or None when
it cannot be costed, so a single bad line never aborts the whole batch's cost
accounting."""
custom_id: Final = entry.get("custom_id") if isinstance(entry, dict) else None
try:
return _compute_output_line_stats(entry, custom_llm_provider, model_name, model_info)
return _compute_output_line_stats(entry, custom_llm_provider, model_name, model_info, vertex_location)
except Exception as e: # noqa: BLE001 # any single line's costing failure must not abort the whole batch
verbose_logger.warning(
"batch output line could not be costed, so it is billed at $0 and the rest of the batch "
@ -244,6 +260,7 @@ def _compute_output_line_stats(
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "bedrock", "mistral"],
model_name: str | None,
model_info: ModelInfo | None,
vertex_location: str | None = None,
) -> _BatchOutputLineStats:
response_body: Final = _get_response_from_batch_job_output_file(entry, custom_llm_provider)
usage: Final = _get_batch_job_usage_from_response_body(response_body, custom_llm_provider)
@ -258,6 +275,7 @@ def _compute_output_line_stats(
model_name=model_name,
response_model=response_model,
model_info=model_info,
vertex_location=vertex_location,
)
return _BatchOutputLineStats(
prompt_cost=line_prompt_cost,
@ -287,6 +305,7 @@ def _output_line_cost(
model_name: str | None,
response_model: str | None,
model_info: ModelInfo | None,
vertex_location: str | None = None,
) -> tuple[float, float]:
"""(prompt_cost, completion_cost) for one output line, priced at batch rates."""
from litellm.cost_calculator import batch_cost_calculator, ocr_batch_cost
@ -307,6 +326,7 @@ def _output_line_cost(
model=cost_model,
custom_llm_provider=custom_llm_provider,
model_info=model_info,
vertex_location=vertex_location,
)
@ -315,11 +335,14 @@ def _aggregate_batch_cost_usage_models(
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic", "bedrock", "mistral"],
model_name: str | None = None,
model_info: ModelInfo | None = None,
vertex_location: str | None = None,
) -> BatchCostUsageResult:
"""Aggregate cost, usage, models, and pass/fail counts from batch output
entries in a single pass, holding one small stats record per line instead
of the parsed file."""
all_results: Final = tuple(_classify_output_line_stats(entries, custom_llm_provider, model_name, model_info))
all_results: Final = tuple(
_classify_output_line_stats(entries, custom_llm_provider, model_name, model_info, vertex_location)
)
line_stats: Final = tuple(result for result in all_results if isinstance(result, _BatchOutputLineStats))
failed_requests: Final = sum(1 for result in all_results if result is _LineOutcome.PROVIDER_FAILED)
successful_requests: Final = len(all_results) - failed_requests
@ -366,6 +389,7 @@ def calculate_vertex_ai_batch_cost_and_usage(
vertex_ai_batch_responses: Iterable[dict[str, object]],
model_name: str | None = None,
model_info: ModelInfo | None = None,
vertex_location: str | None = None,
) -> BatchCostUsageResult:
"""
Cost and usage of a native Vertex predictions.jsonl, one
@ -384,6 +408,7 @@ def calculate_vertex_ai_batch_cost_and_usage(
model_info=model_info,
calculate_usage=VertexGeminiConfig.calculate_usage,
cost_calculator=batch_cost_calculator,
vertex_location=vertex_location,
)
for row in vertex_ai_batch_responses
)

26
litellm/caching/dual_cache.py
View file

@ -59,16 +59,6 @@ class PendingBatchRead:
previous_access_times: dict[str, float | None]
@dataclass(frozen=True, slots=True)
class DeclaredBatchRead:
"""A ``async_batch_get_cache`` split in two: the memory half done, the Redis half declared on a ``RedisBatch``
so it rides that batch's next round trip, resolved later with ``async_resolve_batch_get``."""
keys: tuple[str, ...]
pending: PendingBatchRead
result: BatchResult[Mapping[str, object]] | None
def _log_deferred_increment_failure(future: asyncio.Future[float]) -> None:
if future.cancelled():
return
@ -382,22 +372,6 @@ class DualCache(BaseCache):
await self.in_memory_cache.async_set_cache(key, value, **self._backfill_kwargs(kwargs))
return merged
async def declare_batch_get(self, keys: Sequence[str], batch: RedisBatch) -> DeclaredBatchRead:
pending: Final = await self._prepare_batch_get(
list(keys),
local_only=False,
throttle_redis=False,
)
return DeclaredBatchRead(
keys=tuple(keys),
pending=pending,
result=batch.mget(pending.redis_keys) if pending.redis_keys else None,
)
async def async_resolve_batch_get(self, declared: DeclaredBatchRead) -> list[object | None]:
redis_result: Final = None if declared.result is None else await declared.result
return await self._apply_batch_get(declared.pending, redis_result)
async def async_batch_get_cache(
self,
keys: list,

47
litellm/caching/redis_cache.py
View file

@ -134,7 +134,12 @@ _INCREMENT_WITH_FLOOR_LUA: Final = (
"return count"
)
_INCREMENT_IF_PRESENT_LUA: Final = (
"if redis.call('EXISTS', KEYS[1]) == 0 then return false end return redis.call('INCRBYFLOAT', KEYS[1], ARGV[1])"
)
_LUA_COUNT: Final = TypeAdapter(int)
_LUA_OPTIONAL_TOTAL: Final[TypeAdapter[float | None]] = TypeAdapter(float | None)
_OPTIONAL_COUNTS: Final = TypeAdapter(tuple[int | None, ...])
@ -675,7 +680,7 @@ class RedisCache(BaseCache):
# redis namespaces
self.namespace = namespace
# for high traffic, we store the redis results in memory and then batch write to redis
self.redis_batch_writing_buffer: list = []
self.redis_batch_writing_buffer: list[tuple[str, object]] = []
if redis_flush_size is None:
self.redis_flush_size: int = 100
else:
@ -1568,6 +1573,46 @@ class RedisCache(BaseCache):
count: Final = await _redis_client.eval(_INCREMENT_WITH_FLOOR_LUA, 1, namespaced_key, value, ttl)
return _LUA_COUNT.validate_python(count)
@_redis_circuit_breaker_guard
async def async_increment_if_present(
self, key: str, value: float, parent_otel_span: Span | None = None
) -> float | None:
_redis_client: Final = self._async_commands()
namespaced_key: Final = self.check_and_fix_namespace(key=key)
start_time: Final = time.time()
try:
total: Final = _LUA_OPTIONAL_TOTAL.validate_python(
await _redis_client.eval(_INCREMENT_IF_PRESENT_LUA, 1, namespaced_key, value)
)
except Exception as e:
failed_at: Final = time.time()
asyncio.create_task(
self.service_logger_obj.async_service_failure_hook(
service=ServiceTypes.REDIS,
duration=failed_at - start_time,
error=e,
call_type="async_increment_if_present",
caller=_get_call_stack_info(),
start_time=start_time,
end_time=failed_at,
parent_otel_span=parent_otel_span,
)
)
raise
end_time: Final = time.time()
asyncio.create_task(
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=end_time - start_time,
call_type="async_increment_if_present",
caller=_get_call_stack_info(),
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
)
)
return total
async def flush_cache_buffer(self):
print_verbose(f"flushing to redis....reached size of buffer {len(self.redis_batch_writing_buffer)}")
await self.async_set_cache_pipeline(self.redis_batch_writing_buffer)

20
litellm/constants.py
View file

@ -191,6 +191,11 @@ DEFAULT_IMAGE_HEIGHT: Final = get_env_int("DEFAULT_IMAGE_HEIGHT", 300)
# Maps to OpenAI's 50 MB payload limit - requests with images exceeding this size will be rejected
# Set MAX_IMAGE_URL_DOWNLOAD_SIZE_MB=0 to disable image URL handling entirely
MAX_IMAGE_URL_DOWNLOAD_SIZE_MB: Final = get_env_float("MAX_IMAGE_URL_DOWNLOAD_SIZE_MB", 50.0)
MAX_DECOMPRESSED_REQUEST_BODY_SIZE_MB: Final = get_env_float("MAX_DECOMPRESSED_REQUEST_BODY_SIZE_MB", 50.0)
MAX_REQUEST_BODY_DECOMPRESSION_RATIO: Final = get_env_int("MAX_REQUEST_BODY_DECOMPRESSION_RATIO", 200)
MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB: Final = get_env_float(
"MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB", 200.0
)
MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB: Final = get_env_int(
"MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB", 1024
) # 1MB = 1024KB
@ -252,6 +257,13 @@ MCP_PER_USER_TOKEN_EXPIRY_BUFFER_SECONDS: Final = get_env_int("MCP_PER_USER_TOKE
# MCP timeout defaults (seconds). Override via env vars for slow/custom MCP servers.
MCP_CLIENT_TIMEOUT: Final = get_env_float("LITELLM_MCP_CLIENT_TIMEOUT", 60.0)
MCP_TOOL_LISTING_TIMEOUT: Final = get_env_float("LITELLM_MCP_TOOL_LISTING_TIMEOUT", 30.0)
MCP_UPSTREAM_RATE_LIMIT_MAX_RETRIES: Final = get_env_int("LITELLM_MCP_UPSTREAM_RATE_LIMIT_MAX_RETRIES", 3)
MCP_UPSTREAM_RATE_LIMIT_BASE_DELAY_SECONDS: Final = get_env_float(
"LITELLM_MCP_UPSTREAM_RATE_LIMIT_BASE_DELAY_SECONDS", 0.5
)
MCP_UPSTREAM_RATE_LIMIT_MAX_DELAY_SECONDS: Final = get_env_float(
"LITELLM_MCP_UPSTREAM_RATE_LIMIT_MAX_DELAY_SECONDS", 5.0
)
MCP_METADATA_TIMEOUT: Final = get_env_float("LITELLM_MCP_METADATA_TIMEOUT", 10.0)
MCP_HEALTH_CHECK_TIMEOUT: Final = get_env_float("LITELLM_MCP_HEALTH_CHECK_TIMEOUT", 10.0)
AGENT_HEALTH_CHECK_TIMEOUT_SECONDS: Final = get_env_float("LITELLM_AGENT_HEALTH_CHECK_TIMEOUT", 5.0)
@ -1042,7 +1054,7 @@ openai_compatible_endpoints: Final[list[str]] = [
]
openai_compatible_providers: Final[list] = [
openai_compatible_providers: Final[list[str]] = [
"anyscale",
"groq",
"nvidia_nim",
@ -1633,6 +1645,7 @@ OUTPUT_TOKEN_CEILING_PARAMS: Final = frozenset({"max_tokens", "max_completion_to
CLIENT_OUTPUT_CEILING_METADATA_KEY: Final = "_client_output_ceiling"
CONSUMED_REQUEST_TAGS_METADATA_KEY: Final = "_consumed_request_tags"
ROUTING_REQUEST_TAGS_METADATA_KEY: Final = "_routing_request_tags"
BAGGAGE_MAPPED_TAGS_METADATA_KEY: Final = "_baggage_mapped_tags"
VECTOR_STORE_CALLER_METADATA_KWARG: Final = "litellm_vector_store_caller_metadata"
ROUTER_USAGE_COUNTED_TOKENS_METADATA_KEY: Final = "_litellm_router_usage_counted_tokens"
INTERNAL_CALL_ORIGIN_METADATA_KEY: Final = "internal_call_origin"
@ -1842,6 +1855,9 @@ SPEND_LOG_CLEANUP_BATCH_FAILURE_BACKOFF_SECONDS: Final = get_env_float(
SPEND_LOG_CLEANUP_RUN_BUDGET_SECONDS: Final = get_env_float("SPEND_LOG_CLEANUP_RUN_BUDGET_SECONDS", 300.0)
SPEND_LOG_CLEANUP_BATCH_TIMEOUT_SECONDS: Final = get_env_float("SPEND_LOG_CLEANUP_BATCH_TIMEOUT_SECONDS", 30.0)
SPEND_LOG_CLEANUP_REMAINING_COUNT_CAP: Final = get_env_int("SPEND_LOG_CLEANUP_REMAINING_COUNT_CAP", 100000)
SPEND_LOG_SIZE_CHECK_JOB_NAME: Final = "spend_log_size_check"
SPEND_LOG_SIZE_CHECK_INTERVAL: Final = os.getenv("SPEND_LOG_SIZE_CHECK_INTERVAL", "1d")
SPEND_LOG_SIZE_CLEANUP_RECHECK_BATCHES: Final = get_env_int("SPEND_LOG_SIZE_CLEANUP_RECHECK_BATCHES", 10)
SCHEDULED_JOB_SHUTDOWN_FINISH_TIMEOUT_SECONDS: Final = get_env_float(
"SCHEDULED_JOB_SHUTDOWN_FINISH_TIMEOUT_SECONDS", 5.0
)
@ -1963,6 +1979,7 @@ LITELLM_SETTINGS_SAFE_DB_OVERRIDES: Final = [
"public_model_groups",
"public_model_groups_links",
"cost_discount_config",
"apply_search_free_tier",
"cost_margin_config",
"block_requests_for_models_without_pricing",
"budget_exceeded_throttle_percentage",
@ -2303,6 +2320,7 @@ HARNESS_STDERR_TAIL_LINES: Final = 40
HARNESS_STREAM_READ_CHUNK_BYTES: Final = 64 * 1024
HARNESS_EVENT_QUEUE_MAX_SIZE: Final = 1024
HARNESS_PROCESS_KILL_GRACE_SECONDS: Final = 5.0
HARNESS_RESUME_COMMAND_TIMEOUT_SECONDS: Final = 30.0
HARNESS_SNAPSHOT_SKIP_DIRS: Final = frozenset(
{
".git",

123
litellm/cost_calculator.py
View file

@ -45,6 +45,7 @@ from litellm.litellm_core_utils.llm_cost_calc.utils import (
get_regional_uplift_multiplier,
get_service_tier_cost_key,
get_token_type_cost_breakdown,
get_vertex_regional_endpoint_uplift,
parse_prompt_tokens_details,
select_cost_metric_for_model,
)
@ -137,6 +138,7 @@ from litellm.types.utils import (
TranscriptionUsageTokensObject,
Usage,
VectorStoreSearchResponse,
is_server_derived_pricing_key,
)
from litellm.utils import (
CallTypes,
@ -149,6 +151,8 @@ from litellm.utils import (
TextCompletionResponse,
TranscriptionResponse,
cached_get_model_info_helper,
model_info_with_pricing_overrides,
runtime_model_cost_registrations,
token_counter,
unmapped_model_generalization,
)
@ -437,6 +441,7 @@ def cost_per_token(
### REQUEST MODEL ###
request_model: str | None = None, # original request model for router detection
custom_model_info: OCRPricing | None = None,
bedrock_model_info: ModelInfo | None = None,
) -> tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@ -676,6 +681,7 @@ def cost_per_token(
model=model,
custom_llm_provider=custom_llm_provider,
data_residency=data_residency,
vertex_location=vertex_location,
)
elif call_type == "atranscription" or call_type == "transcription":
if _transcription_usage_has_token_details(usage_block):
@ -738,6 +744,15 @@ def cost_per_token(
elif custom_llm_provider == "anthropic":
return anthropic_cost_per_token(model=model, usage=usage_block, service_tier=service_tier)
elif custom_llm_provider == "bedrock":
resolved_bedrock_info: Final = bedrock_model_info or _bedrock_pricing_info(model, None, "bedrock", False)
if resolved_bedrock_info is not None:
return generic_cost_per_token(
model=model,
usage=usage_block,
custom_llm_provider="bedrock",
service_tier=service_tier,
model_info=resolved_bedrock_info,
)
return bedrock_cost_per_token(model=model, usage=usage_block, service_tier=service_tier)
elif custom_llm_provider == "openai":
return openai_cost_per_token(
@ -860,11 +875,46 @@ def _cost_map_entry_prices_anything(entry: Mapping[str, object]) -> bool:
)
def _declares_rates(model_info: ModelInfo | None) -> bool:
entry: Final = _raw_cost_map_entry(model_info["key"]) if model_info is not None else None
def _bedrock_registration_has_prices(entry: Mapping[str, object]) -> bool:
return _cost_map_entry_prices_anything(entry) or any(
value is not None and field.startswith("cache_") and is_server_derived_pricing_key(field)
for field, value in entry.items()
)
def _bedrock_pricing_info(
model: str | None,
invoked_model_id: str | None,
custom_llm_provider: str | None,
explicit_pricing: bool,
) -> ModelInfo | None:
if explicit_pricing or custom_llm_provider != "bedrock" or not model:
return None
registration: Final = next(
(
(key, entry)
for key, entry in runtime_model_cost_registrations(model, "bedrock")
if _bedrock_registration_has_prices(entry)
),
None,
)
profile: Final = _cost_map_model_info(invoked_model_id, "bedrock") if invoked_model_id else None
if profile is None or not _declares_rates(profile):
return _cost_map_model_info(registration[0], None) if registration is not None else None
if registration is None or registration[0] == profile["key"]:
return profile
return model_info_with_pricing_overrides(profile, registration[1])
def cost_map_key_prices_anything(key: str) -> bool:
entry: Final = _raw_cost_map_entry(key)
return entry is not None and _cost_map_entry_prices_anything(entry)
def _declares_rates(model_info: ModelInfo | None) -> bool:
return model_info is not None and cost_map_key_prices_anything(model_info["key"])
def _base_model_pricing_provider(base_model: str, custom_llm_provider: str | None) -> str | None:
"""The provider to price an explicit base_model under, when the deployment's own provider has
no rates for it (e.g. chatgpt/ subscription rows) but the base model's own row does"""
@ -886,6 +936,16 @@ def _pricing_provider_for_cost_candidate(
return current_provider if isinstance(current_provider, str) else None
def _invoked_model_id_for_cost(
optional_params: object,
logging_obj: LitellmLoggingObject | None,
) -> str | None:
recorded: Final[object] = getattr(logging_obj, "model_call_details", None)
return _get_hidden_str_for_cost_calc(optional_params, "model_id") or _get_hidden_str_for_cost_calc(
recorded, "model_id"
)
def select_model_name_for_cost_calc(
model: str | None,
completion_response: object | None,
@ -894,6 +954,7 @@ def select_model_name_for_cost_calc(
custom_llm_provider: str | None = None,
router_model_id: str | None = None,
region_name: str | None = None,
invoked_model_id: str | None = None,
) -> str | None:
"""
1. If custom pricing is true, return received model name
@ -914,6 +975,13 @@ def select_model_name_for_cost_calc(
hidden_params: Final[dict | None] = getattr(completion_response, "_hidden_params", None)
provider_response_model: Final = _get_hidden_str_for_cost_calc(hidden_params, "provider_response_model")
explicit_pricing: Final = custom_pricing is True or base_model is not None
configured_model: Final = model or (
completion_response_model if isinstance(completion_response_model, str) else None
)
bedrock_model_info: Final = _bedrock_pricing_info(
configured_model, invoked_model_id, custom_llm_provider, explicit_pricing
)
bedrock_pricing_model: Final = bedrock_model_info["key"] if bedrock_model_info is not None else None
priced_from_response: Final = provider_response_model is not None or completion_response_model is not None
priced_region: Final = (
_get_hidden_str_for_cost_calc(hidden_params, "region_name") or region_name
@ -931,8 +999,8 @@ def select_model_name_for_cost_calc(
else:
return_model = model
elif base_model is not None or provider_response_model is not None:
return_model = base_model if base_model is not None else provider_response_model
elif base_model is not None or bedrock_pricing_model is not None or provider_response_model is not None:
return_model = base_model if base_model is not None else bedrock_pricing_model or provider_response_model
elif completion_response_model is None and hidden_params is not None:
if hidden_params.get("model", None) is not None and len(hidden_params["model"]) > 0:
@ -1103,6 +1171,8 @@ def get_usage_object(
if isinstance(completion_response, (DecisionsResponse, OpenAIDecisionResponse)):
return None if completion_response.usage is None else _decisions_usage(completion_response.usage)
if isinstance(completion_response, HttpxBinaryResponseContent):
return completion_response.usage
if usage_obj is None:
return None
if isinstance(usage_obj, Usage):
@ -1441,6 +1511,7 @@ def completion_cost(
litellm_model_name: str | None = None,
router_model_id: str | None = None,
litellm_logging_obj: LitellmLoggingObject | None = None,
pricing_provider: str | None = None,
### SERVICE TIER ###
service_tier: str | None = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
@ -1461,6 +1532,8 @@ def completion_cost(
"""
try:
call_type = infer_call_type(call_type, completion_response) or "completion"
if pricing_provider is not None:
custom_llm_provider = pricing_provider # rebind-ok: model_info.pricing_provider overrides the routing provider for cost-map lookups
if call_type == CallTypes.aresponses_websocket.value and isinstance(
completion_response, LiteLLMRealtimeStreamLoggingObject
@ -1486,6 +1559,7 @@ def completion_cost(
optional_params=optional_params,
custom_pricing=custom_pricing,
base_model=base_model,
pricing_provider=pricing_provider,
standard_built_in_tools_params=standard_built_in_tools_params,
litellm_model_name=litellm_model_name,
router_model_id=router_model_id,
@ -1544,6 +1618,10 @@ def completion_cost(
if base_model is not None and custom_pricing is not True
else None
)
invoked_model_id: Final = _invoked_model_id_for_cost(
_LOGGED_LITELLM_PARAMS.validate_python(optional_params) if optional_params is not None else None,
litellm_logging_obj,
)
selected_model: Final = select_model_name_for_cost_calc(
model=model,
completion_response=completion_response,
@ -1552,8 +1630,15 @@ def completion_cost(
base_model=base_model,
router_model_id=router_model_id,
region_name=region_name,
invoked_model_id=invoked_model_id,
)
bedrock_model_info: Final = _bedrock_pricing_info(
model or _get_response_model(completion_response),
invoked_model_id,
_get_provider_for_cost_calc(model, custom_llm_provider),
explicit_pricing,
)
requested_model: Final = model
request_pricing_provider: Final = custom_llm_provider if isinstance(custom_llm_provider, str) else None
raw_candidates: Final = (
@ -1565,10 +1650,10 @@ def completion_cost(
*(((model, request_pricing_provider),) if model is not None else ()),
)
potential_model_names: Final = (
raw_candidates if explicit_pricing else _priced_candidates_before_unpriced_rules(raw_candidates)
raw_candidates if custom_pricing is True else _priced_candidates_before_unpriced_rules(raw_candidates)
)
for idx, (model, pricing_provider) in enumerate(potential_model_names):
for idx, (model, candidate_pricing_provider) in enumerate(potential_model_names):
try:
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug("selected model name for cost calculation: %s", model)
@ -1635,7 +1720,9 @@ def completion_cost(
hidden_params = getattr(completion_response, "_hidden_params", None)
if hidden_params is not None:
custom_llm_provider = hidden_params.get("custom_llm_provider", custom_llm_provider or None)
custom_llm_provider = pricing_provider or hidden_params.get(
"custom_llm_provider", custom_llm_provider or None
)
region_name = hidden_params.get("region_name", region_name)
else:
@ -1921,7 +2008,7 @@ def completion_cost(
prompt_tokens=prompt_tokens or 0,
completion_tokens=completion_tokens or 0,
custom_llm_provider=_pricing_provider_for_cost_candidate(
candidate_provider=pricing_provider,
candidate_provider=candidate_pricing_provider,
request_provider=request_pricing_provider,
current_provider=custom_llm_provider,
),
@ -1942,6 +2029,7 @@ def completion_cost(
vertex_location=vertex_location,
response=completion_response,
custom_model_info=_ocr_model_info(litellm_logging_obj, custom_pricing, router_model_id),
bedrock_model_info=bedrock_model_info,
)
# Get additional costs from provider (e.g., routing fees, infrastructure costs)
@ -2024,7 +2112,7 @@ def completion_cost(
_token_type_breakdown = get_token_type_cost_breakdown(
model=model,
custom_llm_provider=_pricing_provider_for_cost_candidate(
candidate_provider=pricing_provider,
candidate_provider=candidate_pricing_provider,
request_provider=request_pricing_provider,
current_provider=custom_llm_provider,
),
@ -2033,6 +2121,7 @@ def completion_cost(
data_residency=data_residency,
vertex_location=vertex_location,
custom_cost_per_token=custom_cost_per_token,
model_info=bedrock_model_info,
)
_reasoning_cost = _token_type_breakdown.reasoning_cost
_cache_read_cost = _token_type_breakdown.cache_read_cost
@ -2146,6 +2235,7 @@ def response_cost_calculator(
litellm_model_name: str | None = None,
router_model_id: str | None = None,
litellm_logging_obj: LitellmLoggingObject | None = None,
pricing_provider: str | None = None,
### SERVICE TIER ###
service_tier: str | None = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
@ -2186,6 +2276,7 @@ def response_cost_calculator(
litellm_model_name=litellm_model_name,
router_model_id=router_model_id,
litellm_logging_obj=litellm_logging_obj,
pricing_provider=pricing_provider,
service_tier=service_tier,
data_residency=data_residency,
vertex_location=vertex_location,
@ -2335,7 +2426,16 @@ def pricing_entry_for_cost_calc(
custom_llm_provider=custom_llm_provider,
router_model_id=router_model_id,
region_name=region_name,
invoked_model_id=_invoked_model_id_for_cost(None, litellm_logging_obj),
)
bedrock_model_info: Final = _bedrock_pricing_info(
model or _get_response_model(completion_response),
_invoked_model_id_for_cost(None, litellm_logging_obj),
_get_provider_for_cost_calc(model, custom_llm_provider),
custom_pricing is True or base_model is not None,
)
if bedrock_model_info is not None:
return bedrock_model_info["key"], bedrock_model_info
candidates: Final = (selected_model, _get_response_model(completion_response), model)
resolved: Final = next(
(info for info in (_cost_map_model_info(name, custom_llm_provider) for name in candidates if name) if info),
@ -2783,6 +2883,8 @@ def batch_cost_calculator(
custom_llm_provider: str | None = None,
model_info: ModelInfo | None = None,
data_residency: str | None = None,
*,
vertex_location: str | None = None,
) -> tuple[float, float]:
"""
Calculate the cost of a batch job.
@ -2872,7 +2974,8 @@ def batch_cost_calculator(
total_prompt_cost *= uplift
total_completion_cost *= uplift
return total_prompt_cost, total_completion_cost
vertex_uplift: Final = get_vertex_regional_endpoint_uplift(model_info, vertex_location)
return total_prompt_cost * vertex_uplift, total_completion_cost * vertex_uplift
def _batch_prompt_cost(

68
litellm/decisions/call.py
View file

@ -1,3 +1,4 @@
import json
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field, replace
from functools import partial
@ -6,9 +7,11 @@ from typing import Final, Literal, TypeAlias
import httpx
from pydantic import BaseModel, ConfigDict, StrictBool, TypeAdapter, ValidationError
from pydantic_core import to_jsonable_python
from typing_extensions import assert_never
import litellm
from litellm.cost_calculator import cost_map_key_prices_anything
from litellm.litellm_core_utils.core_helpers import RESPONSE_COST_HEADER, normalize_drop_params
from litellm.litellm_core_utils.get_provider_specific_headers import ProviderSpecificHeaderUtils
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -19,6 +22,7 @@ from litellm.litellm_core_utils.prompt_templates.image_handling import (
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.decisions.transformation import (
BaseDecisionsConfig,
DecisionsResponseValidationError,
image_media_type,
image_param,
input_images,
@ -56,6 +60,7 @@ from litellm.types.decisions import (
OpenAIDecisionResponse,
UnsupportedDecisionsRequest,
)
from litellm.types.llms.openai import ChatCompletionUserMessage
from litellm.types.utils import LlmProviders, ProviderSpecificHeader
from litellm.utils import ProviderConfigManager, get_model_cost_key
@ -415,6 +420,30 @@ def _outbound_headers(
return MappingProxyType({**(extra_headers or {}), **forwarded})
_LOGGED_FIELDS_ADAPTER: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(Mapping[str, object])
def _jsonable_fallback(value: object) -> object:
return dict(_LOGGED_FIELDS_ADAPTER.validate_python(value)) if isinstance(value, Mapping) else str(value)
def _logged_request_message(caller_request: object) -> ChatCompletionUserMessage:
caller_fields: Final = _LOGGED_FIELDS_ADAPTER.validate_python(
to_jsonable_python(caller_request, fallback=_jsonable_fallback, bytes_mode="base64")
)
sent_fields: Final = json.dumps(
dict((name, value) for name, value in caller_fields.items() if value not in (None, [], {}))
)
return ChatCompletionUserMessage(role="user", content=sent_fields)
def log_caller_request(kwargs: Mapping[str, object], caller_request: object) -> None:
"""Record every request field the caller sent as the logged `messages`, before any validation."""
logging_obj: Final = kwargs.get("litellm_logging_obj")
if isinstance(logging_obj, LiteLLMLoggingObj):
logging_obj.update_messages([_logged_request_message(caller_request)])
def prepare_call(
*,
model: str,
@ -660,6 +689,13 @@ def _request_body(call: DecisionsCall) -> Mapping[str, object]:
return body
def _pricing_model(response: DecisionsIRResponse, call: DecisionsCall) -> str:
returned: Final = get_model_cost_key(f"{call.custom_llm_provider}/{response.model}")
if returned is not None and cost_map_key_prices_anything(returned):
return returned
return f"{call.custom_llm_provider}/{call.model}"
def _format_response(response: DecisionsIRResponse, call: DecisionsCall) -> DecisionsResponse | OpenAIDecisionResponse:
formatted: Final = _formatted_response(response, call)
provider_cost: Final = call.provider_config.provider_reported_cost(response)
@ -667,10 +703,7 @@ def _format_response(response: DecisionsIRResponse, call: DecisionsCall) -> Deci
{
"model": f"{call.custom_llm_provider}/{call.model}",
"custom_llm_provider": call.custom_llm_provider,
"provider_response_model": (
get_model_cost_key(f"{call.custom_llm_provider}/{response.model}")
or f"{call.custom_llm_provider}/{call.model}"
),
"provider_response_model": _pricing_model(response, call),
**({"additional_headers": {RESPONSE_COST_HEADER: provider_cost}} if provider_cost is not None else {}),
}
)
@ -700,18 +733,21 @@ def _formatted_response(
def _map_upstream_exception(error: Exception, call: DecisionsCall) -> Exception:
if isinstance(error, BaseLLMException) and error.status_code_is_synthesized:
provider_label: Final = f"{call.custom_llm_provider[0].upper()}{call.custom_llm_provider[1:]}Exception"
return litellm.APIConnectionError(
message=f"{provider_label} - {error.message}",
llm_provider=call.custom_llm_provider,
model=f"{call.custom_llm_provider}/{call.model}",
)
return litellm.exception_type(
model=f"{call.custom_llm_provider}/{call.model}",
custom_llm_provider=call.custom_llm_provider,
original_exception=error,
)
provider_label: Final = f"{call.custom_llm_provider[0].upper()}{call.custom_llm_provider[1:]}Exception"
model: Final = f"{call.custom_llm_provider}/{call.model}"
match error:
case DecisionsResponseValidationError():
return litellm.InternalServerError(
message=f"{provider_label} - {error.message}", llm_provider=call.custom_llm_provider, model=model
)
case BaseLLMException(status_code_is_synthesized=True):
return litellm.APIConnectionError(
message=f"{provider_label} - {error.message}", llm_provider=call.custom_llm_provider, model=model
)
case _:
return litellm.exception_type(
model=model, custom_llm_provider=call.custom_llm_provider, original_exception=error
)
async def asend(call: DecisionsCall) -> DecisionsResponse | OpenAIDecisionResponse:

11
litellm/decisions/main.py
View file

@ -17,6 +17,7 @@ from litellm.decisions.call import (
SystemOneQuestions,
SystemOneRequestFields,
asend,
log_caller_request,
prepare_call,
reject_images_next_to_input,
send,
@ -78,6 +79,16 @@ def _prepare_decisions_call(
extra_headers: Mapping[str, str] | None,
kwargs: Mapping[str, object],
) -> DecisionsCall:
log_caller_request(
kwargs,
{
"input": decision_input,
"questions": questions,
"audio": audio,
"videos": videos,
**{key: kwargs[key] for key in _SYSTEMONE_FIELDS if key in kwargs},
},
)
return prepare_call(
model=model,
fields=_request_fields(model, decision_input, questions, audio, videos, kwargs),

9
litellm/endpoints/speech/speech_to_completion_bridge/handler.py
View file

@ -6,6 +6,8 @@ from typing import TYPE_CHECKING, Final
from typing_extensions import TypedDict
from litellm._internal_context import internal_sub_call
if TYPE_CHECKING:
from litellm import LiteLLMLoggingObj
from litellm.types.llms.openai import HttpxBinaryResponseContent
@ -111,9 +113,10 @@ class SpeechToCompletionBridgeHandler:
voice=voice,
)
result: Final = completion(
**request_data,
)
with internal_sub_call():
result: Final = completion(
**request_data,
)
requested_response_format: Final = optional_params.get("response_format")
if isinstance(result, ModelResponse):

15
litellm/endpoints/speech/speech_to_completion_bridge/transformation.py
View file

@ -5,11 +5,12 @@ from typing import TYPE_CHECKING, Final, cast
from typing_extensions import NotRequired, ReadOnly, TypedDict
from litellm.constants import OPENAI_CHAT_COMPLETION_PARAMS
from litellm.litellm_core_utils.hidden_params import get_hidden_params, served_from_cache, set_hidden_param
if TYPE_CHECKING:
from litellm import Logging as LiteLLMLoggingObj
from litellm.types.llms.openai import ChatCompletionUserMessage, HttpxBinaryResponseContent
from litellm.types.utils import ModelResponse
from litellm.types.utils import ModelResponse, Usage
def _completion_response_cost(model_response: "ModelResponse") -> float | None:
@ -20,6 +21,13 @@ def _completion_response_cost(model_response: "ModelResponse") -> float | None:
return response_cost if isinstance(response_cost, float) else None
def _completion_usage(model_response: "ModelResponse") -> "Usage | None":
from litellm.types.utils import Usage
usage: Final = getattr(model_response, "usage", None)
return usage if isinstance(usage, Usage) else None
GEMINI_TTS_CHAT_AUDIO_FORMAT: Final = "pcm16"
GEMINI_TTS_RAW_RESPONSE_FORMAT: Final = "pcm"
GEMINI_TTS_SUPPORTED_RESPONSE_FORMATS: Final = frozenset({"wav", GEMINI_TTS_RAW_RESPONSE_FORMAT})
@ -179,4 +187,9 @@ class SpeechToCompletionBridgeTransformationHandler:
)
binary_response: Final = HttpxBinaryResponseContent(response)
binary_response.set_response_cost(_completion_response_cost(model_response))
binary_response.usage = _completion_usage(model_response)
cached_hidden_params: Final = get_hidden_params(model_response) if served_from_cache(model_response) else None
if cached_hidden_params is not None:
set_hidden_param(binary_response, "cache_hit", True)
set_hidden_param(binary_response, "cache_key", cached_hidden_params.get("cache_key"))
return binary_response

7
litellm/fine_tuning/main.py
View file

@ -173,6 +173,7 @@ def create_fine_tuning_job(
try:
_is_async: Final = kwargs.pop("acreate_fine_tuning_job", False) is True
optional_params: Final = GenericLiteLLMParams(**kwargs)
provider_model: Final = model.removeprefix(f"{custom_llm_provider}/")
# handle hyperparameters
hyperparameters = hyperparameters or {} # original hyperparameters
@ -218,7 +219,7 @@ def create_fine_tuning_job(
)
create_fine_tuning_job_data_dict = _build_fine_tuning_job_data(
model,
provider_model,
training_file,
_oai_hyperparameters,
suffix,
@ -263,7 +264,7 @@ def create_fine_tuning_job(
extra_body = _prepare_azure_extra_body(extra_body, kwargs, azure_specific_hyperparams)
create_fine_tuning_job_data_dict = _build_fine_tuning_job_data(
model,
provider_model,
training_file,
_oai_hyperparameters,
suffix,
@ -298,7 +299,7 @@ def create_fine_tuning_job(
response = vertex_fine_tuning_apis_instance.create_fine_tuning_job(
_is_async=_is_async,
create_fine_tuning_job_data=_build_fine_tuning_job_data(
model,
provider_model,
training_file,
_oai_hyperparameters,
suffix,

4
litellm/google_genai/streaming_iterator.py
View file

@ -71,7 +71,7 @@ class BaseGoogleGenAIGenerateContentStreamingIterator:
):
self.litellm_logging_obj = litellm_logging_obj
self.request_body = request_body
self.start_time = datetime.now()
self.start_time = litellm_logging_obj.start_time
self.collected_chunks: list[bytes] = []
self.model = model
self.custom_llm_provider = custom_llm_provider
@ -192,6 +192,8 @@ class AsyncGoogleGenAIGenerateContentStreamingIterator(BaseGoogleGenAIGenerateCo
async def __anext__(self):
try:
chunk: Final = await _anext_google_genai_sse_chunk(self.stream_iterator)
if self.litellm_logging_obj.completion_start_time is None:
self.litellm_logging_obj.update_completion_start_time(completion_start_time=datetime.now())
self.collected_chunks.append(chunk)
return chunk
except StopAsyncIteration:

4
litellm/harness/__init__.py
View file

@ -1,4 +1,4 @@
"""Agent harnesses: run Claude Code, Codex, OpenCode, Deep Agents or Tool Loop on any LiteLLM model.
"""Agent harnesses: run Claude Code, Codex, OpenCode, pi, Deep Agents or Tool Loop on any LiteLLM model.
The entrypoints live on the top-level package:
@ -30,6 +30,7 @@ from litellm.harness.options import (
CodexOptions,
DeepAgentsOptions,
OpenCodeOptions,
PiOptions,
ToolLoopOptions,
)
from litellm.harness.runtime import (
@ -78,6 +79,7 @@ __all__ = (
"OpenCodeOptions",
"OptionsMismatch",
"OutputInvalid",
"PiOptions",
"Reasoning",
"Result",
"SandboxError",

119
litellm/harness/endpoint.py
View file

@ -22,7 +22,7 @@ from typing import TYPE_CHECKING, Any, Final, Protocol
import httpx
import openai
from pydantic import ConfigDict, TypeAdapter
from pydantic import ConfigDict, TypeAdapter, ValidationError
import litellm
from litellm.constants import (
@ -172,16 +172,57 @@ def usage_from_mapping(usage: object) -> tuple[int, int]:
return _as_int(input_tokens), _as_int(output_tokens)
def usage_from_body(body: object) -> tuple[int, int]:
"""Usage from a non-streaming JSON response body."""
if not isinstance(body, Mapping):
return 0, 0
if isinstance(body.get("usage"), Mapping):
return usage_from_mapping(body["usage"])
response = body.get("response")
if isinstance(response, Mapping):
return usage_from_mapping(response.get("usage"))
return 0, 0
@dataclass(frozen=True, slots=True)
class PromptCache:
"""Prompt tokens read from or written to the provider's cache. Anthropic's usage counts them outside input_tokens,
while OpenAI's prompt_tokens and the Responses API's input_tokens already include them."""
read_tokens: int = 0
creation_tokens: int = 0
outside_input: bool = False
def prompt_tokens(self, input_tokens: int) -> int:
if self.outside_input:
return input_tokens + self.read_tokens + self.creation_tokens
return input_tokens
NO_PROMPT_CACHE: Final = PromptCache()
def json_object(value: object) -> Mapping[str, object]:
try:
return _JSON_OBJECT_ADAPTER.validate_python(value)
except ValidationError:
return MappingProxyType({})
def prompt_cache_from_mapping(usage: object) -> PromptCache:
"""Cache token counts from a usage dict using OpenAI, Anthropic, or Responses field names."""
fields: Final = json_object(usage)
details: Final = json_object(fields.get("prompt_tokens_details", fields.get("input_tokens_details")))
return PromptCache(
read_tokens=_as_int(fields.get("cache_read_input_tokens") or details.get("cached_tokens")),
creation_tokens=_as_int(
fields.get("cache_creation_input_tokens")
or details.get("cache_write_tokens")
or details.get("cache_creation_tokens")
),
outside_input="prompt_tokens" not in fields and "input_tokens_details" not in fields,
)
def usage_mapping_of_body(body: object) -> Mapping[str, object]:
fields: Final = json_object(body)
usage: Final = json_object(fields.get("usage"))
return usage if usage else json_object(json_object(fields.get("response")).get("usage"))
def usage_mapping_of_json(raw: bytes) -> Mapping[str, object]:
try:
return usage_mapping_of_body(_JSON_VALUE_ADAPTER.validate_python(json.loads(raw)))
except ValueError:
return MappingProxyType({})
class SSEUsageParser:
@ -190,6 +231,7 @@ class SSEUsageParser:
def __init__(self) -> None:
self.input_tokens = 0
self.output_tokens = 0
self.prompt_cache = NO_PROMPT_CACHE
self._buffer = b""
def feed(self, chunk: bytes) -> None:
@ -226,21 +268,23 @@ class SSEUsageParser:
elif event_type == "response.completed":
self._absorb_response_completed(event)
elif isinstance(event.get("usage"), Mapping):
self._set(*usage_from_mapping(event["usage"]))
self._absorb_usage(event["usage"])
def _absorb_message_start(self, event: Mapping[str, object]) -> None:
message = event.get("message")
if isinstance(message, Mapping):
self._set(*usage_from_mapping(message.get("usage")))
self._absorb_usage(json_object(event.get("message")).get("usage"))
def _absorb_message_delta(self, event: Mapping[str, object]) -> None:
# message_delta output_tokens is cumulative for the whole message.
self._set(*usage_from_mapping(event.get("usage")))
self._absorb_usage(event.get("usage"))
def _absorb_response_completed(self, event: Mapping[str, object]) -> None:
response = event.get("response")
if isinstance(response, Mapping):
self._set(*usage_from_mapping(response.get("usage")))
self._absorb_usage(json_object(event.get("response")).get("usage"))
def _absorb_usage(self, usage: object) -> None:
self._set(*usage_from_mapping(usage))
prompt_cache: Final = prompt_cache_from_mapping(usage)
if prompt_cache.read_tokens or prompt_cache.creation_tokens:
self.prompt_cache = prompt_cache
def _set(self, input_tokens: int, output_tokens: int) -> None:
if input_tokens:
@ -249,13 +293,19 @@ class SSEUsageParser:
self.output_tokens = output_tokens
def compute_cost(model: str | None, input_tokens: int, output_tokens: int) -> float:
"""Cost from LiteLLM's price map. Never raises; unknown models cost 0.0."""
def compute_cost(
model: str | None, input_tokens: int, output_tokens: int, prompt_cache: PromptCache = NO_PROMPT_CACHE
) -> float:
"""Cost from LiteLLM's price map, cached prompt tokens at their cache prices. Never raises; unknown models cost 0.0."""
if not model or not (input_tokens or output_tokens):
return 0.0
try:
prompt_cost, completion_cost = litellm.cost_per_token(
model=model, prompt_tokens=input_tokens, completion_tokens=output_tokens
model=model,
prompt_tokens=prompt_cache.prompt_tokens(input_tokens),
completion_tokens=output_tokens,
cache_read_input_tokens=prompt_cache.read_tokens,
cache_creation_input_tokens=prompt_cache.creation_tokens,
)
return float(prompt_cost) + float(completion_cost)
except Exception: # accounting must never break a call; the price-map lookup raises bare Exception
@ -565,9 +615,10 @@ class ModelEndpoint:
input_tokens: int,
output_tokens: int,
cost: float | None,
prompt_cache: PromptCache,
) -> None:
if cost is None:
cost = compute_cost(model, input_tokens, output_tokens)
cost = compute_cost(model, input_tokens, output_tokens, prompt_cache)
self.usage.add(input_tokens, output_tokens, cost)
async def _forward(self, request: Request, route: str, body: Mapping[str, object]) -> Response:
@ -617,14 +668,12 @@ class ModelEndpoint:
) -> None:
if is_sse:
parser.close()
tokens = (parser.input_tokens, parser.output_tokens)
else:
try:
body: Final[object] = _JSON_VALUE_ADAPTER.validate_python(json.loads(collected))
tokens = usage_from_body(body)
except ValueError:
tokens = (0, 0)
self._record(model, tokens[0], tokens[1], header_cost(upstream.headers))
self._record(
model, parser.input_tokens, parser.output_tokens, header_cost(upstream.headers), parser.prompt_cache
)
return
usage: Final = usage_mapping_of_json(collected)
self._record(model, *usage_from_mapping(usage), header_cost(upstream.headers), prompt_cache_from_mapping(usage))
def _sdk_kwargs(
self, body: Mapping[str, object]
@ -667,9 +716,9 @@ class ModelEndpoint:
return self._responses.StreamingResponse(
self._sdk_stream(route, response, model), media_type=SSE_MEDIA_TYPE
)
data = to_jsonable(response)
input_tokens, output_tokens = usage_from_body(data)
self._record(model, input_tokens, output_tokens, hidden_cost(response))
data: Final = to_jsonable(response)
usage: Final = usage_mapping_of_body(data)
self._record(model, *usage_from_mapping(usage), hidden_cost(response), prompt_cache_from_mapping(usage))
return self._json(data)
async def _sdk_stream(self, route: str, iterator: AsyncIterable[object], model: str | None) -> AsyncIterator[bytes]:
@ -688,4 +737,4 @@ class ModelEndpoint:
yield f"event: error\ndata: {json.dumps(error_body(e, message))}\n\n".encode()
finally:
parser.close()
self._record(model, parser.input_tokens, parser.output_tokens, None)
self._record(model, parser.input_tokens, parser.output_tokens, None, parser.prompt_cache)

86
litellm/harness/handlers/cli_handler.py
View file

@ -12,17 +12,27 @@ from __future__ import annotations
import asyncio
import os
from collections import deque
from collections.abc import AsyncIterator, Sequence
from collections.abc import AsyncIterator, Mapping, Sequence
from dataclasses import dataclass
from typing import Final
from litellm._logging import verbose_logger
from litellm.constants import HARNESS_STDERR_TAIL_LINES, HARNESS_STREAM_READ_CHUNK_BYTES
from litellm.constants import (
HARNESS_RESUME_COMMAND_TIMEOUT_SECONDS,
HARNESS_STDERR_TAIL_LINES,
HARNESS_STREAM_READ_CHUNK_BYTES,
)
from litellm.harness.context import SessionContext
from litellm.harness.errors import HarnessInstallFailed, SandboxError
from litellm.harness.errors import HarnessError, HarnessInstallFailed, SandboxError
from litellm.harness.handlers.base import BaseHarnessHandler
from litellm.harness.sandbox.base import Process, Sandbox
from litellm.harness.sandbox.base import CompletedRun, Process, Sandbox
from litellm.harness.types import Event
from litellm.llms.base_llm.harness.transformation import BaseCLIHarnessConfig, HarnessSessionSetup
from litellm.llms.base_llm.harness.transformation import (
BaseCLIHarnessConfig,
HarnessSessionCopy,
HarnessSessionSetup,
HarnessTurnError,
)
from litellm.llms.base_llm.harness.utils import decode_json_line, read_skill_files
# Link <private_dir>/<dir> to a LiteLLM-owned cache dir so a later session can resume.
@ -81,6 +91,52 @@ async def persist_dir(sandbox: Sandbox, link_path: str, cache_subpath: str) -> N
)
@dataclass(frozen=True, slots=True)
class ResumeFailed:
reason: str
async def bounded_resume_run(
sandbox: Sandbox, cmd: Sequence[str], env: Mapping[str, str]
) -> CompletedRun | ResumeFailed:
try:
run: Final = await sandbox.run(cmd, env=env, timeout=HARNESS_RESUME_COMMAND_TIMEOUT_SECONDS)
except SandboxError as exc:
return ResumeFailed(str(exc))
return run if run.exit_code == 0 else ResumeFailed(run.stderr.strip() or f"exit code {run.exit_code}")
async def import_session_copy(sandbox: Sandbox, copy: HarnessSessionCopy, env: Mapping[str, str]) -> str | ResumeFailed:
try:
await sandbox.write(copy.path, copy.data)
except SandboxError as exc:
return ResumeFailed(str(exc))
imported: Final = await bounded_resume_run(sandbox, copy.import_command, env)
return imported if isinstance(imported, ResumeFailed) else copy.session_id
async def resumable_session_id(
config: BaseCLIHarnessConfig[object, object],
sandbox: Sandbox,
setup_env: Mapping[str, str],
private_dir: str,
native_session_id: str,
) -> str | ResumeFailed:
export_cmd: Final = config.transform_resume_export_command(native_session_id)
if export_cmd is None:
return native_session_id
exported: Final = await bounded_resume_run(sandbox, export_cmd, setup_env)
if isinstance(exported, ResumeFailed):
return exported
try:
copy: Final = config.transform_resume_copy(exported.stdout, private_dir, sandbox.workdir)
except HarnessError as exc:
return ResumeFailed(str(exc))
if copy is None:
return native_session_id
return await import_session_copy(sandbox, copy, setup_env)
async def copy_skills(sandbox: Sandbox, skills: Sequence[str], skills_root: str) -> None:
for skill in skills:
name = os.path.basename(os.path.realpath(os.fspath(skill)))
@ -89,13 +145,14 @@ async def copy_skills(sandbox: Sandbox, skills: Sequence[str], skills_root: str)
class CLIHarnessHandler(BaseHarnessHandler):
config: BaseCLIHarnessConfig
config: BaseCLIHarnessConfig[object, object]
def __init__(self, config: BaseCLIHarnessConfig) -> None:
super().__init__(config)
self._private_dir: str | None = None
self._setup: HarnessSessionSetup | None = None
self._native_id: str | None = None
self._pending_resume: str | None = None
self._proc: Process | None = None
async def start(self, ctx: SessionContext) -> None:
@ -119,6 +176,9 @@ class CLIHarnessHandler(BaseHarnessHandler):
async def turn(self, ctx: SessionContext, prompt: str) -> AsyncIterator[Event]:
if self._setup is None or self._private_dir is None:
raise RuntimeError("CLIHarnessHandler.turn() called before start()")
if self._pending_resume is not None:
self._native_id = await self._prepared_session_id(ctx, self._setup, self._private_dir, self._pending_resume)
self._pending_resume = None
request = self.config.transform_turn_request(ctx, self._setup, self._private_dir, prompt, self._native_id)
argv: Final = list(request.argv) # mutable-ok: Sandbox.exec takes list[str]
proc = await ctx.sandbox.exec(argv, env=request.env, cwd=request.cwd)
@ -158,4 +218,16 @@ class CLIHarnessHandler(BaseHarnessHandler):
return self._native_id
async def resume(self, ctx: SessionContext, native_session_id: str) -> None:
self._native_id = native_session_id
if native_session_id != self._native_id:
self._native_id = native_session_id
self._pending_resume = native_session_id
async def _prepared_session_id(
self, ctx: SessionContext, setup: HarnessSessionSetup, private_dir: str, saved_id: str
) -> str:
target: Final = await resumable_session_id(self.config, ctx.sandbox, setup.env, private_dir, saved_id)
match target:
case ResumeFailed(reason=reason):
raise HarnessTurnError(f"could not prepare session {saved_id} for resume: {reason}")
case session_id:
return session_id

2
litellm/harness/handlers/deepagents_handler.py
View file

@ -183,7 +183,7 @@ class DeepAgentsHandler(BaseHarnessHandler):
break
resume: dict[str, object] = {} # mutable-ok: Command(resume=) payload, filled per answered approval
for interrupt in state.interrupts:
decisions: list[dict[str, Any]] = [] # mutable-ok: HITL decisions collected across awaited approvals
decisions: list[dict[str, str]] = [] # mutable-ok: HITL decisions collected across awaited approvals
for request in approval_requests(getattr(interrupt, "value", None)):
approval = Approval(
tool=normalized_tool_name(str(request.get("name") or "")),

29
litellm/harness/options.py
View file

@ -4,28 +4,37 @@ from __future__ import annotations
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from typing import Literal
from typing import Literal, TypeAlias
PiThinkingLevel: TypeAlias = Literal["off", "minimal", "low", "medium", "high", "xhigh", "max"]
@dataclass(frozen=True)
class ClaudeCodeOptions:
config: Mapping[str, object] = field(default_factory=dict)
env: Mapping[str, str] = field(default_factory=dict)
config: Mapping[str, object] = field(default_factory=dict[str, object])
env: Mapping[str, str] = field(default_factory=dict[str, str])
@dataclass(frozen=True)
class CodexOptions:
reasoning_effort: Literal["low", "medium", "high", "xhigh"] | None = None
web_search: bool = False
config: Mapping[str, object] = field(default_factory=dict)
env: Mapping[str, str] = field(default_factory=dict)
config: Mapping[str, object] = field(default_factory=dict[str, object])
env: Mapping[str, str] = field(default_factory=dict[str, str])
@dataclass(frozen=True)
class OpenCodeOptions:
agent: str = "build"
config: Mapping[str, object] = field(default_factory=dict)
env: Mapping[str, str] = field(default_factory=dict)
config: Mapping[str, object] = field(default_factory=dict[str, object])
env: Mapping[str, str] = field(default_factory=dict[str, str])
@dataclass(frozen=True)
class PiOptions:
thinking: PiThinkingLevel | None = None
config: Mapping[str, object] = field(default_factory=dict[str, object])
env: Mapping[str, str] = field(default_factory=dict[str, str])
@dataclass(frozen=True)
@ -36,7 +45,9 @@ class DeepAgentsOptions:
@dataclass(frozen=True)
class ToolLoopOptions:
completion_kwargs: Mapping[str, object] = field(default_factory=dict)
completion_kwargs: Mapping[str, object] = field(default_factory=dict[str, object])
HarnessOptions = ClaudeCodeOptions | CodexOptions | OpenCodeOptions | DeepAgentsOptions | ToolLoopOptions
HarnessOptions: TypeAlias = (
ClaudeCodeOptions | CodexOptions | OpenCodeOptions | PiOptions | DeepAgentsOptions | ToolLoopOptions
)

16
litellm/harness/sandbox/local.py
View file

@ -14,7 +14,7 @@ from typing import Final
from litellm.constants import HARNESS_PROCESS_KILL_GRACE_SECONDS
from litellm.harness.errors import SandboxError
from litellm.harness.sandbox.base import CompletedRun
from litellm.harness.sandbox.base import CompletedRun, Process
from litellm.harness.sandbox.snapshot import snapshot_local
_SECRET_PREFIXES: Final = (
@ -103,7 +103,7 @@ class SubprocessHandle:
await self._proc.wait()
async def _read_all(handle: SubprocessHandle) -> tuple[bytes, bytes, int]:
async def _read_all(handle: Process) -> tuple[bytes, bytes, int]:
if handle.stdin is not None:
handle.stdin.close()
stdout, stderr = await asyncio.gather(handle.stdout.read(), handle.stderr.read())
@ -111,13 +111,16 @@ async def _read_all(handle: SubprocessHandle) -> tuple[bytes, bytes, int]:
return stdout, stderr, exit_code
async def collect_output(handle: SubprocessHandle, cmd: Sequence[str], timeout: float | None) -> CompletedRun:
"""Close stdin, read stdout/stderr to EOF; kill and raise SandboxError on timeout."""
async def collect_output(handle: Process, cmd: Sequence[str], timeout: float | None) -> CompletedRun:
"""Close stdin, read stdout/stderr to EOF; kill on cancel, and kill and raise SandboxError on timeout."""
try:
stdout, stderr, code = await asyncio.wait_for(_read_all(handle), timeout)
except asyncio.TimeoutError:
await handle.kill()
raise SandboxError(f"command timed out after {timeout}s: {cmd[0]}")
except asyncio.CancelledError:
await handle.kill()
raise
return CompletedRun(
stdout=stdout.decode("utf-8", errors="replace"),
stderr=stderr.decode("utf-8", errors="replace"),
@ -180,14 +183,15 @@ class LocalSandbox:
self._check_open()
if not cmd:
raise SandboxError("exec() needs a non-empty command")
work_dir: Final = self._resolve_cwd(cwd)
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
cwd=self._resolve_cwd(cwd),
env=self.child_env(env),
cwd=work_dir,
env=self.child_env(MappingProxyType({**(env or {}), "PWD": work_dir})),
start_new_session=True,
)
except (FileNotFoundError, PermissionError) as exc:

2
litellm/harness/sync.py
View file

@ -411,7 +411,7 @@ def agent(
options: HarnessOptions | None = None,
install: bool = False,
) -> Result | EventStream:
"""Run an agent harness (Claude Code, Codex, OpenCode, Deep Agents, Tool Loop) on one prompt.
"""Run an agent harness (Claude Code, Codex, OpenCode, pi, Deep Agents, Tool Loop) on one prompt.
Returns a Result. With stream=True it returns an iterator of events instead.
Prefix the model with `litellm_proxy/` to route every model call through your

1
litellm/harness/types.py
View file

@ -26,6 +26,7 @@ class Harness(Enum):
CLAUDE_CODE = "claude_code"
CODEX = "codex"
OPENCODE = "opencode"
PI = "pi"
DEEPAGENTS = "deepagents"
TOOL_LOOP = "tool_loop"

21
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -342,6 +342,20 @@ EXTERNAL_BREAKPOINTS_STAMP: Final = "_litellm_external_breakpoints"
CUSTOM_LLM_PROVIDER_STAMP: Final = "_litellm_custom_llm_provider"
def _accepts_cache_control(part: object) -> bool:
match part:
case {"type": "thinking" | "redacted_thinking"}:
return False
case dict():
return True
case _:
return False
def _last_index_accepting_cache_control(parts: Sequence[object]) -> int | None:
return next((index for index in reversed(range(len(parts))) if _accepts_cache_control(parts[index])), None)
class AnthropicCacheControlHook(CustomPromptManagement):
@staticmethod
def _request_value(request_kwargs: object, key: str) -> object:
@ -740,10 +754,11 @@ class AnthropicCacheControlHook(CustomPromptManagement):
# 1. if string, insert cache control in the message
if isinstance(message_content, str):
message["cache_control"] = control
# 2. list of objects - only apply to last item per Anthropic spec
# 2. list of objects - only apply to the last item Claude accepts a breakpoint on
elif isinstance(message_content, list):
if len(message_content) > 0 and isinstance(message_content[-1], dict):
message_content[-1]["cache_control"] = control # pyright: ignore[reportGeneralTypeIssues] # loose runtime dict
markable_index: Final = _last_index_accepting_cache_control(message_content)
if markable_index is not None:
message_content[markable_index]["cache_control"] = control # pyright: ignore[reportGeneralTypeIssues] # loose runtime dict
return message
@staticmethod

13
litellm/integrations/arize/_utils.py
View file

@ -884,7 +884,7 @@ def _set_session_and_user_attrs(span: "Span", kwargs: dict, standard_logging_pay
safe_set_attribute(span, "litellm.key_alias", str(key_alias))
def _set_response_cost_attr(span: "Span", standard_logging_payload) -> None:
def _set_response_cost_attr(span: "Span", standard_logging_payload: Mapping[str, object] | None) -> None:
"""Emit cost attributes from the StandardLoggingPayload when present.
Uses the OpenInference `llm.cost.total` key so Arize / Phoenix can
@ -892,15 +892,22 @@ def _set_response_cost_attr(span: "Span", standard_logging_payload) -> None:
single total in `StandardLoggingPayload.response_cost`, so we cannot
split it into prompt/completion. We also keep the legacy
`llm.response.cost` key for back-compat with any consumer querying it.
When the price lookup failed, the call's total is unknown, so no cost is
emitted and the backend does not record the call as free. A guardrail fee
in `response_cost` stays on the guardrail span, as in OTel v2.
"""
if not isinstance(standard_logging_payload, dict):
return
cost: Final = standard_logging_payload.get("response_cost")
if cost is None:
cost_lookup_failed: Final = (
standard_logging_payload.get("response_cost_failure_debug_info") is not None
and standard_logging_payload.get("cache_hit") is not True
)
if not isinstance(cost, (int, float, str)) or cost_lookup_failed:
return
try:
cost_value: Final = float(cost)
except (TypeError, ValueError):
except ValueError:
return
safe_set_attribute(span, "llm.cost.total", cost_value)
safe_set_attribute(span, "llm.response.cost", cost_value)

14
litellm/integrations/arize/arize_phoenix.py
View file

@ -354,17 +354,16 @@ class ArizePhoenixLogger(OpenTelemetry):
_project_name, tracer = self._resolve_tracer_for_kwargs(kwargs)
ctx, parent_span = self._get_phoenix_context(kwargs, tracer=tracer)
status: Final = Status(StatusCode.OK if success else StatusCode.ERROR)
status_code: Final = StatusCode.OK if success else StatusCode.ERROR
span: Final = tracer.start_span(
name=self._get_span_name(kwargs),
start_time=self._to_ns(start_time),
context=ctx,
)
span.set_status(status)
self.set_attributes(span, kwargs, response_obj)
if not success:
self._record_exception_on_span(span=span, kwargs=kwargs)
span_error_message: Final = None if success else self._record_exception_on_span(span=span, kwargs=kwargs)
span.set_status(Status(status_code, span_error_message))
if success:
self._maybe_log_raw_request(kwargs, response_obj, start_time, end_time, span)
@ -373,10 +372,11 @@ class ArizePhoenixLogger(OpenTelemetry):
self._create_guardrail_span(kwargs=kwargs, context=ctx)
if parent_span is not None:
parent_span.set_status(status)
self.set_attributes(parent_span, kwargs, response_obj)
if not success:
self._record_exception_on_span(span=parent_span, kwargs=kwargs)
parent_error_message: Final = (
None if success else self._record_exception_on_span(span=parent_span, kwargs=kwargs)
)
parent_span.set_status(Status(status_code, parent_error_message))
parent_span.end(end_time=self._to_ns(end_time))
if success:

24
litellm/integrations/datadog/datadog_llm_obs.py
View file

@ -12,7 +12,7 @@ import os
from collections.abc import Mapping, Sequence
from datetime import datetime
from types import MappingProxyType
from typing import Any, Final, Literal
from typing import Final, Literal
import httpx
from pydantic import ConfigDict, TypeAdapter, ValidationError
@ -54,6 +54,7 @@ from litellm.types.utils import (
PROMPT_CARRYING_GUARDRAIL_FIELDS,
PROMPT_QUOTING_ROUTING_DECISION_FIELDS,
CallTypes,
StandardLoggingMetadata,
StandardLoggingPayload,
StandardLoggingPayloadErrorInformation,
)
@ -93,8 +94,12 @@ _ROUTER_DIMENSIONS: Final[tuple[str, ...]] = ("router_tier", "router_cause", "ro
_COST_DIMENSIONS: Final[tuple[str, ...]] = ("team", "user", "key_alias", "model_group", *_ROUTER_DIMENSIONS)
def _metadata_of(standard_logging_payload: StandardLoggingPayload) -> Mapping[str, Any]:
metadata: Final = standard_logging_payload.get("metadata")
def _logging_metadata_of(standard_logging_payload: StandardLoggingPayload) -> StandardLoggingMetadata | None:
return standard_logging_payload.get("metadata")
def _metadata_of(standard_logging_payload: StandardLoggingPayload) -> Mapping[str, object]:
metadata: Final = _logging_metadata_of(standard_logging_payload)
return metadata or _EMPTY_MAPPING
@ -1057,14 +1062,23 @@ class DataDogLLMObsLogger(CustomBatchLogger):
# Get budget information from metadata
metadata: Final = _metadata_of(standard_logging_payload)
logging_metadata: Final = _logging_metadata_of(standard_logging_payload)
# API key max budget
user_api_key_max_budget: Final = metadata.get("user_api_key_max_budget")
user_api_key_max_budget: Final = (
logging_metadata["user_api_key_max_budget"]
if logging_metadata and "user_api_key_max_budget" in logging_metadata
else None
)
if user_api_key_max_budget is not None:
spend_metrics["user_api_key_max_budget"] = float(user_api_key_max_budget)
# API key spend
user_api_key_spend: Final = metadata.get("user_api_key_spend")
user_api_key_spend: Final = (
logging_metadata["user_api_key_spend"]
if logging_metadata and "user_api_key_spend" in logging_metadata
else None
)
if user_api_key_spend is not None:
try:
spend_metrics["user_api_key_spend"] = float(user_api_key_spend)

15
litellm/integrations/langfuse/langfuse_otel.py
View file

@ -1,7 +1,7 @@
import base64
import json
import os
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, Optional
@ -9,6 +9,7 @@ from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
from litellm.integrations.langfuse.langfuse_otel_attributes import (
LangfuseLLMObsOTELAttributes,
serialize_langfuse_datetime,
)
from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
@ -55,7 +56,7 @@ class LangfuseOtelLogger(OpenTelemetry):
LangfuseOtelLogger._set_langfuse_specific_attributes(span=span, kwargs=kwargs, response_obj=response_obj)
@staticmethod
def _extract_langfuse_metadata(kwargs: dict) -> dict:
def _extract_langfuse_metadata(kwargs: Mapping[str, Any]) -> dict:
"""
Extracts Langfuse metadata from the standard LiteLLM kwargs structure.
@ -221,7 +222,7 @@ class LangfuseOtelLogger(OpenTelemetry):
)
@staticmethod
def _set_langfuse_specific_attributes(span: Span, kwargs, response_obj):
def _set_langfuse_specific_attributes(span: Span, kwargs: Mapping[str, Any], response_obj):
"""
Sets Langfuse specific metadata attributes onto the OTEL span.
@ -255,6 +256,14 @@ class LangfuseOtelLogger(OpenTelemetry):
safe_dumps(messages),
)
completion_start_time: Final = kwargs.get("completion_start_time")
if isinstance(completion_start_time, datetime):
safe_set_attribute(
span,
LangfuseSpanAttributes.OBSERVATION_COMPLETION_START_TIME.value,
serialize_langfuse_datetime(completion_start_time),
)
LangfuseOtelLogger._set_observation_output(span=span, response_obj=response_obj)
@staticmethod

5
litellm/integrations/langfuse/langfuse_otel_attributes.py
View file

@ -6,6 +6,7 @@ Relevant Issue: https://github.com/BerriAI/litellm/issues/13764
import json
from collections.abc import Mapping
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final
from pydantic import BaseModel
@ -29,6 +30,10 @@ if TYPE_CHECKING:
from opentelemetry.trace import Span
def serialize_langfuse_datetime(value: datetime) -> str:
return json.dumps(value.astimezone().isoformat())
def get_output_content_by_type(
response_obj: None
| dict

3
litellm/integrations/langfuse/langfuse_sdk.py
View file

@ -39,6 +39,7 @@ from pydantic import ConfigDict
import litellm
from litellm._logging import verbose_logger
from litellm.integrations.langfuse.langfuse import PROMPT_CACHE_TTL_ENV, parse_langfuse_debug, whole_number
from litellm.integrations.langfuse.langfuse_otel_attributes import serialize_langfuse_datetime
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_httpx_client
from litellm.types.llms.base import LiteLLMBaseModel
@ -168,7 +169,7 @@ def _string_or_none(value: object) -> str | None:
def _serialize_datetime(value: object) -> str | None:
"""A datetime the way the SDK's ``EventSerializer`` sends one: a JSON string, naive values read as local time."""
if isinstance(value, datetime):
return safe_dumps(value.astimezone().isoformat())
return serialize_langfuse_datetime(value)
return _serialize(value)

115
litellm/integrations/opentelemetry.py
View file

@ -53,6 +53,7 @@ from litellm.types.utils import (
StandardCallbackDynamicParams,
StandardLoggingMetadata,
StandardLoggingPayload,
StandardLoggingPayloadErrorInformation,
)
# OpenTelemetry imports moved to individual functions to avoid import errors when not installed
@ -104,8 +105,26 @@ class _ResponseWithUsageView(TypedDict, total=False):
usage: "_UsageCompletionTokensView | None"
class _FailurePayloadView(TypedDict, total=False):
error_information: ReadOnly[StandardLoggingPayloadErrorInformation | None]
error_str: ReadOnly[str | None]
class _FailureKwargsView(TypedDict, total=False):
exception: ReadOnly[BaseException | None]
standard_logging_object: ReadOnly[_FailurePayloadView | None]
_JSON_OBJECT: Final = TypeAdapter(Mapping[str, object], config=ConfigDict(hide_input_in_errors=True))
def _text_as_stored_on_span(span: Span, key: str, text: str) -> str:
from opentelemetry.sdk.trace import ReadableSpan
stored: Final = span.attributes.get(key) if isinstance(span, ReadableSpan) and span.attributes else None
return stored if isinstance(stored, str) else text
# Cap on credential-scoped providers held at once; each one owns an exporter thread.
_MAX_DYNAMIC_TRACER_PROVIDERS: Final = 256
@ -922,7 +941,11 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
value=value,
)
service_logging_span.set_status(Status(StatusCode.ERROR))
service_logging_span.set_status(
Status(
StatusCode.ERROR, _text_as_stored_on_span(service_logging_span, "error", error) if error else None
)
)
service_logging_span.end(end_time=_end_time_ns)
async def async_post_call_failure_hook(
@ -937,8 +960,6 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
parent_otel_span: Final = user_api_key_dict.parent_otel_span
if parent_otel_span is not None:
parent_otel_span.set_status(Status(StatusCode.ERROR))
# Stamp team attributes onto the SERVER (root) span too, so the
# trace root is team-filterable on the failure path like the
# child exception span below.
@ -960,13 +981,14 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
original_exception=original_exception,
traceback_str=traceback_str,
)
self._record_exception_on_span(
server_error_message: Final = self._record_exception_on_span(
span=parent_otel_span,
kwargs={
"exception": original_exception,
"standard_logging_object": {"error_information": error_information},
},
)
parent_otel_span.set_status(Status(StatusCode.ERROR, server_error_message))
# _record_exception_on_span only stamps when error_code is set;
# bare TypeError etc. has none, and the span is about to be ended.
@ -995,7 +1017,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
team_id=user_api_key_dict.team_id,
team_alias=user_api_key_dict.team_alias,
)
exception_logging_span.set_status(Status(StatusCode.ERROR))
exception_logging_span.set_status(Status(StatusCode.ERROR, server_error_message))
exception_logging_span.end(end_time=self._to_ns(datetime.now()))
# Emit guardrail spans for any guardrail invocations that
@ -2199,11 +2221,11 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
if self._gen_ai_semconv_latest_experimental:
span_kwargs["kind"] = self.span_kind.CLIENT
span = otel_tracer.start_span(**span_kwargs)
span.set_status(Status(StatusCode.ERROR))
self.set_attributes(span, kwargs, response_obj)
# Record exception information using OTEL standard method
self._record_exception_on_span(span=span, kwargs=kwargs)
span_error_message: Final = self._record_exception_on_span(span=span, kwargs=kwargs)
span.set_status(Status(StatusCode.ERROR, span_error_message))
span.end(end_time=self._to_ns(end_time))
else:
@ -2212,9 +2234,9 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
# Only set attributes if the span is still recording (not closed)
# Note: parent_otel_span is guaranteed to be not None here
if parent_otel_span.is_recording():
parent_otel_span.set_status(Status(StatusCode.ERROR))
self.set_attributes(parent_otel_span, kwargs, response_obj)
self._record_exception_on_span(span=parent_otel_span, kwargs=kwargs)
parent_error_message: Final = self._record_exception_on_span(span=parent_otel_span, kwargs=kwargs)
parent_otel_span.set_status(Status(StatusCode.ERROR, parent_error_message))
# Create span for guardrail information — ensure proper parenting (Issue #5)
guardrail_ctx: Final = self._resolve_guardrail_context(
@ -2232,13 +2254,15 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
):
parent_otel_span.end(end_time=self._to_ns(end_time))
def _record_exception_on_span(self, span: Span, kwargs: dict):
def _record_exception_on_span(self, span: Span, kwargs: _FailureKwargsView) -> str | None:
"""
Record exception information on the span using OTEL standard methods.
This extracts error information from StandardLoggingPayload and:
1. Uses span.record_exception() for the actual exception object (OTEL standard)
2. Sets structured error attributes from StandardLoggingPayloadErrorInformation
Returns the error message it recorded, if any, for the span status description.
"""
try:
from litellm.integrations._types.open_inference import (
@ -2253,10 +2277,10 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
span.record_exception(exception)
# Get StandardLoggingPayload for structured error information
standard_logging_payload: Final[StandardLoggingPayload | None] = kwargs.get("standard_logging_object")
standard_logging_payload: Final = kwargs.get("standard_logging_object")
if standard_logging_payload is None:
return
return None
# Extract error_information from StandardLoggingPayload
error_information: Final = standard_logging_payload.get("error_information")
@ -2270,59 +2294,67 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
key=ErrorAttributes.ERROR_MESSAGE,
value=error_str,
)
return
return _text_as_stored_on_span(span, ErrorAttributes.ERROR_MESSAGE, error_str) if error_str else None
# Set structured error attributes from StandardLoggingPayloadErrorInformation
if error_information.get("error_code"):
error_code: Final = error_information.get("error_code")
if error_code:
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_CODE,
value=error_information["error_code"],
value=error_code,
)
# Also expose under the OTel-standard name as an int
# (error_code is a str, may be non-numeric).
_error_code_val: Final = error_information["error_code"]
if _error_code_val is not None:
try:
self.safe_set_attribute(
span=span,
key=HTTP_RESPONSE_STATUS_CODE_ATTRIBUTE,
value=int(_error_code_val),
)
except (ValueError, TypeError):
pass
try:
self.safe_set_attribute(
span=span,
key=HTTP_RESPONSE_STATUS_CODE_ATTRIBUTE,
value=int(error_code),
)
except (ValueError, TypeError):
pass
if error_information.get("error_class"):
error_class: Final = error_information.get("error_class")
if error_class:
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_TYPE,
value=error_information["error_class"],
value=error_class,
)
if error_information.get("error_message"):
error_message: Final = error_information.get("error_message")
if error_message:
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_MESSAGE,
value=error_information["error_message"],
value=error_message,
)
if error_information.get("llm_provider"):
llm_provider: Final = error_information.get("llm_provider")
if llm_provider:
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_LLM_PROVIDER,
value=error_information["llm_provider"],
value=llm_provider,
)
if error_information.get("traceback"):
traceback_text: Final = error_information.get("traceback")
if traceback_text:
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_STACK_TRACE,
value=error_information["traceback"],
value=traceback_text,
)
except Exception as e:
verbose_logger.exception("OpenTelemetry: Error recording exception on span: %s", str(e))
return None
else:
if error_message:
return _text_as_stored_on_span(span, ErrorAttributes.ERROR_MESSAGE, error_message)
return error_class or None
def set_tools_attributes(self, span: Span, tools):
import json
@ -3518,8 +3550,6 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
key="exception",
value=str(_exception),
)
management_endpoint_span.set_status(Status(StatusCode.ERROR))
management_endpoint_span.end(end_time=_end_time_ns)
# The management wrapper has no other hook that closes the SERVER span.
from litellm.litellm_core_utils.litellm_logging import (
@ -3529,14 +3559,16 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
error_information: Final = StandardLoggingPayloadSetup.get_error_information(
original_exception=_exception,
)
parent_otel_span.set_status(Status(StatusCode.ERROR))
self._record_exception_on_span(
server_error_message: Final = self._record_exception_on_span(
span=parent_otel_span,
kwargs={
"exception": _exception,
"standard_logging_object": {"error_information": error_information},
},
)
management_endpoint_span.set_status(Status(StatusCode.ERROR, server_error_message))
management_endpoint_span.end(end_time=_end_time_ns)
parent_otel_span.set_status(Status(StatusCode.ERROR, server_error_message))
parent_otel_span.end(end_time=_end_time_ns)
def create_litellm_proxy_request_started_span(
@ -3596,19 +3628,20 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
span: Span | None,
exception: Exception | None,
status_code: int,
) -> None:
) -> str | None:
"""Stamp structured ``error.*`` attributes on the SERVER span from the
exception returned to the client, with ``error.code`` pinned to the real
response status. Idempotent (overwrites); emits no exception event."""
response status. Idempotent (overwrites); emits no exception event.
Returns the error text for the span status description."""
if span is None or exception is None:
return
return None
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
)
error_information: Final = StandardLoggingPayloadSetup.get_error_information(original_exception=exception)
error_information["error_code"] = str(status_code)
self._record_exception_on_span(
return self._record_exception_on_span(
span=span,
kwargs={"standard_logging_object": {"error_information": error_information}},
)

2
litellm/integrations/otel/README.md
View file

@ -91,7 +91,7 @@ way (`pod_lock`, `budget_reset`, `spend_queue`, `health_check`, `scheduler_queue
rather than a bare `redis.set`; `tests/unit/test_internal_context.py` scans every
module under `litellm/` and `enterprise/` that calls a shared cache or declares a
read or write on the request batch (`reserve_redis_batch_reads`,
`declare_batch_get`, `batch.mget`, `batch.set`, `batch.script`) and fails when
`batch.mget`, `batch.set`, `batch.script`) and fails when
one has no declared family, with the process-local `InMemoryCache` callers listed
as the only exemptions. A batch op carries the family that was active when it was
declared, so the routing prefetch armed before deployment selection

1
litellm/integrations/otel/logger.py
View file

@ -594,6 +594,7 @@ class OpenTelemetryV2(CustomLogger):
payload,
capture_content=self._capture_span_content(),
time_to_first_chunk_seconds=call.time_to_first_chunk_seconds,
completion_start_seconds=call.completion_start_seconds,
request_route=request_root_http_route(),
request_purpose=call.purpose,
trace=call.trace,

7
litellm/integrations/otel/mappers/langfuse.py
View file

@ -12,8 +12,10 @@ the JSON-serialized payloads. ``trace_attributes`` maps the caller's trace contr
import json
from collections.abc import Callable
from datetime import datetime, timezone
from typing import Final
from litellm.integrations.langfuse.langfuse_otel_attributes import serialize_langfuse_datetime
from litellm.integrations.otel.mappers.base import AttributeMap, AttrValue, SpanData
from litellm.integrations.otel.mappers.utils import (
collect,
@ -45,6 +47,11 @@ class LangfuseMapper:
"langfuse.observation.id": lambda d: d.identity.call_id or None,
"langfuse.trace.metadata.team_id": lambda d: d.identity.team_id or None,
"langfuse.trace.metadata.team_alias": lambda d: d.identity.team_alias or None,
"langfuse.observation.completion_start_time": lambda d: (
serialize_langfuse_datetime(datetime.fromtimestamp(d.completion_start_seconds, tz=timezone.utc))
if d.completion_start_seconds is not None
else None
),
}
# Sub-tables folded into their respective JSON blobs.

2
litellm/integrations/otel/model/metadata.py
View file

@ -258,6 +258,7 @@ class LLMCallEvent:
# needs to be reasonable for a span that never gets closed (a leak).
provisional_span_name: str
time_to_first_chunk_seconds: float | None
completion_start_seconds: float | None
trace: TraceControls
session_id: str | None
@ -280,6 +281,7 @@ class LLMCallEvent:
cache_hit=bool(payload.get("cache_hit")) if payload else False, # pyright: ignore[reportUnknownMemberType] # TypedDict .get on a str key is partially unknown
provisional_span_name=f"{operation.value} {model}".strip(),
time_to_first_chunk_seconds=time_to_first_chunk_seconds(kwargs),
completion_start_seconds=_epoch_seconds(kwargs.get("completion_start_time")),
trace=trace,
session_id=caller_session_id(kwargs, trace),
)

6
litellm/integrations/otel/model/payloads.py
View file

@ -429,6 +429,7 @@ class LLMCallSpanData:
choices_out: tuple[Mapping[str, object], ...] = ()
system_fingerprint: str | None = None
time_to_first_chunk_seconds: float | None = None
completion_start_seconds: float | None = None
# The requested output modality, set only on the routes that pin one (image
# generation, speech, transcription, OCR), and the litellm route itself, which
# keeps routes the convention folds into one operation distinguishable.
@ -450,6 +451,7 @@ class LLMCallSpanData:
payload: StandardLoggingPayload,
capture_content: bool = False,
time_to_first_chunk_seconds: float | None = None,
completion_start_seconds: float | None = None,
request_route: str | None = None,
request_purpose: str | None = None,
trace: TraceControls | None = None,
@ -481,6 +483,7 @@ class LLMCallSpanData:
embedding_output: Final = (
EmbeddingOutput.from_response(response) if operation is GenAIOperation.EMBEDDINGS else None
)
cost_lookup_failed: Final = payload.get("response_cost_failure_debug_info") is not None and not cache_hit
return cls(
operation=operation,
provider=resolve_provider(as_str(payload.get("custom_llm_provider"))),
@ -491,7 +494,7 @@ class LLMCallSpanData:
usage=LLMUsage.from_standard_logging_payload(payload),
finish_reasons=finish_reasons,
error=_parse_error(payload),
response_cost=as_float(payload.get("response_cost")),
response_cost=None if cost_lookup_failed else as_float(payload.get("response_cost")),
cost=LLMCost.from_breakdown(cast("Mapping[str, object] | None", payload.get("cost_breakdown"))),
server=None if cache_hit else ServerInfo.from_api_base(context.api_base),
identity=context.identity,
@ -502,6 +505,7 @@ class LLMCallSpanData:
choices_out=choices_out if capture_content else (),
system_fingerprint=as_str(response.get("system_fingerprint")),
time_to_first_chunk_seconds=time_to_first_chunk_seconds,
completion_start_seconds=completion_start_seconds,
output_type=resolve_output_type(call_type),
call_type=call_type or None,
request_route=request_route or context.identity.request_route,

1
litellm/integrations/otel/model/spans.py
View file

@ -214,6 +214,7 @@ _SERVICE_VERB_BY_CALL_TYPE: Final[dict[str, str]] = {
"increment_cache": "incr",
"async_increment": "incr",
"async_increment_pipeline": "incr",
"async_increment_if_present": "incr",
"delete_cache": "delete",
"async_delete_cache": "delete",
"async_rpush": "rpush",

64
litellm/integrations/prometheus.py
View file

@ -47,9 +47,11 @@ from litellm.litellm_core_utils.core_helpers import (
get_litellm_metadata_from_kwargs,
get_metadata_variable_name_from_kwargs,
)
from litellm.litellm_core_utils.get_llm_provider_logic import is_registered_custom_provider
from litellm.litellm_core_utils.service_tier_utils import (
get_service_tier_from_standard_logging_payload,
)
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
from litellm.models.end_user import LiteLLM_EndUserTable
from litellm.models.team import LiteLLM_TeamTable
from litellm.models.user import LiteLLM_UserTable
@ -60,6 +62,7 @@ from litellm.repositories.organization_repository import OrganizationRepository
from litellm.repositories.table_repositories import EndUserRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.router_utils.router_callbacks.track_deployment_metrics import find_deployment_metadata
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import (
@ -255,11 +258,32 @@ class _LabeledMetric:
_MetricLike: TypeAlias = "NoOpMetric | _LabeledMetric | MetricWrapperBase"
_SeriesLimitT: Final = TypeVar("_SeriesLimitT", int, float)
_REQUEST_METADATA_ADAPTER: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(Mapping[str, object])
_POSITIVE_SERIES_CAP: Final[TypeAdapter[int]] = TypeAdapter(Annotated[int, Field(gt=0)])
_POSITIVE_SERIES_TTL: Final[TypeAdapter[float]] = TypeAdapter(Annotated[float, Field(gt=0)])
_SERIES_CLEANUP_INTERVAL: Final[TypeAdapter[float]] = TypeAdapter(Annotated[float, Field(ge=0)])
_DEFAULT_SERIES_CLEANUP_INTERVAL_SECONDS: Final = 60.0
_ADDITIONAL_HEADERS: Final[TypeAdapter[Mapping[object, object]]] = TypeAdapter(Mapping[object, object])
_LOGGED_FIELDS: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(Mapping[str, object])
_NO_LOGGED_FIELDS: Final[Mapping[str, object]] = MappingProxyType({})
def _logged_fields(value: object) -> Mapping[str, object]:
return _LOGGED_FIELDS.validate_python(value) if isinstance(value, Mapping) else _NO_LOGGED_FIELDS
def _is_llm_provider(name: str) -> bool:
routable: Final = (
name in litellm.provider_list or JSONProviderRegistry.exists(name) or is_registered_custom_provider(name)
)
return routable and name not in litellm._known_custom_logger_compatible_callbacks # pyright: ignore[reportPrivateUsage] # the router treats these prefixes as prompt-management deployments
def _deployment_model_provider(deployment: object) -> str:
if not isinstance(deployment, str):
return ""
prefix, separator, _ = deployment.partition("/")
return prefix if separator and _is_llm_provider(prefix) else ""
def _number_or_none(value: object, limit: TypeAdapter[_SeriesLimitT]) -> _SeriesLimitT | None:
@ -1725,12 +1749,25 @@ class PrometheusLogger(CustomLogger):
# increment litellm_proxy_total_requests_metric for all successful requests
# (both streaming and non-streaming) in this single location to prevent
# double-counting that occurs when async_post_call_success_hook also increments
request_metadata: Final = {
**_REQUEST_METADATA_ADAPTER.validate_python(get_litellm_metadata_from_kwargs(kwargs)),
**_REQUEST_METADATA_ADAPTER.validate_python(_metadata),
}
original_model_group: Final = request_metadata.get("original_model_group")
request_enum_values: Final = replace(
enum_values,
requested_model=(
original_model_group
if isinstance(original_model_group, str) and original_model_group
else standard_logging_payload["model_group"]
),
)
PrometheusLogger._inc_labeled_counter(
self,
self.litellm_proxy_total_requests_metric,
"litellm_proxy_total_requests_metric",
enum_values,
label_context=label_context,
request_enum_values,
label_context=PrometheusLabelFactoryContext(request_enum_values),
)
def _increment_token_metrics(
@ -2846,6 +2883,25 @@ class PrometheusLogger(CustomLogger):
)
return None
@staticmethod
def _extract_deployment_api_provider(request_kwargs: Mapping[str, object]) -> str:
"""
Provider of the deployment a failed call was routed to: the resolved
``custom_llm_provider`` on ``litellm_params``, then the one recorded on
``standard_logging_object``, then the provider prefix of the deployment
model the router wrote into the request metadata (``azure/my-gpt`` ->
``azure``) when that prefix is an LLM provider the router accepts.
Returns an empty string otherwise.
"""
litellm_params: Final = _logged_fields(request_kwargs.get("litellm_params"))
provider: Final = litellm_params.get("custom_llm_provider") or _logged_fields(
request_kwargs.get("standard_logging_object")
).get("custom_llm_provider")
if isinstance(provider, str) and provider:
return provider
deployment_metadata: Final = find_deployment_metadata(litellm_params)
return _deployment_model_provider(deployment_metadata.get("deployment") if deployment_metadata else None)
@staticmethod
def _extract_api_provider_from_exception(exception: Exception) -> str | None:
if not isinstance(exception, litellm.exceptions.RateLimitError):
@ -3124,7 +3180,9 @@ class PrometheusLogger(CustomLogger):
label_litellm_model_name = litellm_model_name
label_model_id = model_id
label_api_base = api_base
label_api_provider = llm_provider
label_api_provider = self._extract_deployment_api_provider(
_LOGGED_FIELDS.validate_python(request_kwargs)
)
label_requested_model = model_group or litellm_model_name
label_model_group = model_group
else:

10
litellm/integrations/websearch_interception/transformation.py
View file

@ -5,7 +5,7 @@ Transforms between Anthropic/OpenAI tool_use format and LiteLLM search format.
"""
import json
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from typing import Final
from pydantic import TypeAdapter
@ -539,7 +539,8 @@ class WebSearchTransformation:
result: SearchResponse from litellm.asearch()
Returns:
Formatted text with Title, URL, Snippet for each result
Formatted text with Title, URL, Snippet for each result, or the response
without its provider_metadata (the caller's billing envelope) when there are no results
"""
# Convert SearchResponse to string
if hasattr(result, "results") and result.results:
@ -548,6 +549,9 @@ class WebSearchTransformation:
[f"Title: {r.title}\nURL: {r.url}\nSnippet: {r.snippet}" for r in result.results]
)
else:
search_result_text = str(result)
fields: Final[Mapping[str, object]] = dict(result)
search_result_text = " ".join(
f"{key}={value!r}" for key, value in fields.items() if key != "provider_metadata"
)
return search_result_text

11
litellm/interactions/streaming_iterator.py
View file

@ -11,6 +11,7 @@ from datetime import datetime
from typing import Any, Final
import httpx
from pydantic import ConfigDict, TypeAdapter
from litellm._logging import verbose_logger
from litellm.constants import STREAM_SSE_DONE_STRING
@ -25,6 +26,10 @@ from litellm.types.interactions import (
)
from litellm.utils import CustomStreamWrapper
_INTERACTIONS_STREAMING_CHUNK_ADAPTER: Final = TypeAdapter(
object, config=ConfigDict(strict=True, hide_input_in_errors=True)
)
class BaseInteractionsAPIStreamingIterator:
"""
@ -48,7 +53,7 @@ class BaseInteractionsAPIStreamingIterator:
self.finished = False
self.interactions_api_config = interactions_api_config
self.completed_response: InteractionsAPIStreamingResponse | None = None
self.start_time = datetime.now()
self.start_time = logging_obj.start_time
# set request kwargs
self.litellm_metadata = litellm_metadata
@ -83,10 +88,12 @@ class BaseInteractionsAPIStreamingIterator:
try:
# Parse the JSON chunk
parsed_chunk: Final = json.loads(stripped_chunk)
parsed_chunk: Final = _INTERACTIONS_STREAMING_CHUNK_ADAPTER.validate_python(json.loads(stripped_chunk))
# Format as InteractionsAPIStreamingResponse
if isinstance(parsed_chunk, dict):
if self.logging_obj.completion_start_time is None:
self.logging_obj.update_completion_start_time(completion_start_time=datetime.now())
streaming_response: Final = self.interactions_api_config.transform_streaming_response(
model=self.model,
parsed_chunk=parsed_chunk,

23
litellm/litellm_core_utils/cloud_storage_security.py
View file

@ -1,6 +1,7 @@
import posixpath
import re
from collections.abc import Mapping, Sequence
from itertools import accumulate, repeat
from types import MappingProxyType
from typing import Any, Final, cast
from urllib.parse import quote, unquote
@ -20,15 +21,33 @@ MANAGED_CLOUD_STORAGE_SCHEMES: Final = ("s3://", "gs://")
_MAPPING_PROXY_TYPE: Final[type] = type(MappingProxyType({}))
MAX_FILE_ID_DECODE_PASSES: Final = 8
def _unquote_once(current: str, _: str) -> str:
return unquote(current)
def _decodings(value: str) -> tuple[str, ...]:
return tuple(accumulate(repeat(value, MAX_FILE_ID_DECODE_PASSES + 2), _unquote_once))
def is_managed_cloud_storage_uri(file_id: str) -> bool:
"""
True if file_id is a raw cloud-storage object URI (e.g. ``s3://bucket/key``).
True if file_id is a raw cloud-storage object URI (e.g. ``s3://bucket/key``),
up to ``MAX_FILE_ID_DECODE_PASSES`` layers of percent-encoding deep, or an id
encoded deeper than that, which no client produces and which is treated as raw
rather than decoded any further.
These are internal provider artifacts. On the multi-tenant proxy they must be
retrieved through their managed unified file id so owner/team access is enforced;
a raw URI supplied by a caller bypasses that check.
"""
return isinstance(file_id, str) and file_id.startswith(MANAGED_CLOUD_STORAGE_SCHEMES)
if not isinstance(file_id, str):
return False
decodings: Final = _decodings(file_id)
settled: Final = decodings[-1] == decodings[-2]
return decodings[-1].startswith(MANAGED_CLOUD_STORAGE_SCHEMES) or not settled
_SAFE_OBJECT_COMPONENT_PATTERN: Final = re.compile(r"[^A-Za-z0-9._-]+")

33
litellm/litellm_core_utils/core_helpers.py
View file

@ -10,6 +10,7 @@ from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, cast, runtime_c
import httpx
from pydantic import TypeAdapter, ValidationError
import litellm
from litellm._logging import verbose_logger
from litellm.constants import KUBERNETES_POD_ROUTING_KEY
from litellm.litellm_core_utils.llm_response_utils.get_headers import get_provider_request_id
@ -93,6 +94,10 @@ def normalize_drop_params(value: object) -> bool | None:
return None
def drops_unsupported_params(drop_params: object) -> bool:
return litellm.drop_params is True or normalize_drop_params(drop_params) is True
def drop_params_flag(value: object, source: str, logger: logging.Logger) -> bool:
normalized: Final = normalize_drop_params(value)
if normalized is None and value is not None:
@ -515,7 +520,7 @@ def reconstruct_model_name(
# Helper functions used for OTEL logging
def get_parent_otel_span_from_kwargs(
kwargs: dict[str, object] | None = None,
kwargs: Mapping[str, object] | None = None,
) -> Span | None:
try:
if kwargs is None:
@ -607,10 +612,30 @@ def preserve_upstream_non_openai_attributes(
Preserve non-OpenAI attributes from the original chunk.
"""
# Access model_fields on the class, not the instance, to avoid Pydantic 2.11+ deprecation warnings
expected_keys: Final = set(type(model_response).model_fields.keys()).union({"usage"})
for key, value in original_chunk.model_dump().items():
if key not in expected_keys:
expected_keys: Final = frozenset(type(model_response).model_fields) | frozenset({"usage"})
candidate_keys: Final = tuple(
key
for key in (*type(original_chunk).model_fields, *(original_chunk.__pydantic_extra__ or {}))
if key not in expected_keys
)
if not candidate_keys:
return
candidate_values: Final[tuple[tuple[str, object], ...]] = tuple(
(
key,
cast( # cast-ok: Pydantic extras are scalar-checked before assignment
object,
getattr(original_chunk, key),
),
)
for key in candidate_keys
)
if all(value is None or isinstance(value, (str, int, float, bool)) for _, value in candidate_values):
for key, value in candidate_values:
setattr(model_response, key, value)
return
for key, value in original_chunk.model_dump(include=frozenset(candidate_keys)).items():
setattr(model_response, key, value)
def safe_deep_copy(data):

27
litellm/litellm_core_utils/credential_accessor.py
View file

@ -1,9 +1,15 @@
"""Utils for accessing credentials."""
from collections.abc import Mapping
from typing import Final
import litellm
from litellm.types.utils import CredentialItem
from litellm.types.router import holds_api_key_secret_reference
from litellm.types.utils import (
API_KEY_SECRET_REFERENCE_REJECTION,
CredentialItem,
credential_api_key_secret_reference,
)
class CredentialAccessor:
@ -21,7 +27,7 @@ class CredentialAccessor:
"""Safe accessor for credentials."""
credential: Final = CredentialAccessor.find_credential(credential_name)
return {} if credential is None else credential.credential_values.copy()
return {} if credential is None else dict(credential_api_key_secret_reference(credential.credential_values))
@staticmethod
def upsert_credentials(credentials: list[CredentialItem]):
@ -38,3 +44,20 @@ class CredentialAccessor:
break
else:
litellm.credential_list.append(credential)
def sends_api_key_secret_reference(params: Mapping[str, object]) -> bool:
named: Final = params.get("litellm_credential_name")
credential_values: Final = CredentialAccessor.get_credential_values(named) if isinstance(named, str) else {}
return holds_api_key_secret_reference({**credential_values, **params})
def reject_api_key_secret_reference(params: Mapping[str, object]) -> None:
if holds_api_key_secret_reference(params):
raise ValueError(API_KEY_SECRET_REFERENCE_REJECTION)
named: Final = params.get("litellm_credential_name")
if isinstance(named, str) and holds_api_key_secret_reference(CredentialAccessor.get_credential_values(named)):
raise ValueError(
f"Rejected Request: litellm_credential_name={named!r} names a credential holding an oidc/ api_key "
"secret reference, which only a deployment a proxy admin configures can attach."
)

1
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -27,6 +27,7 @@ AWS_CREDENTIAL_KWARGS_KEYS: Final = frozenset(
"aws_external_id",
"aws_session_tags",
"aws_bedrock_runtime_endpoint",
"aws_bedrock_control_plane_endpoint",
"aws_bedrock_project_id",
}
)

7
litellm/litellm_core_utils/get_model_cost_map.py
View file

@ -558,11 +558,14 @@ def get_model_cost_map_loaded_at() -> "datetime | None":
class AliasedCostEntry(dict[str, object]):
__slots__ = ("names",)
__slots__ = ("names", "registered_fields")
def __init__(self, entry: Mapping[str, object], names: tuple[str, ...]) -> None:
def __init__(
self, entry: Mapping[str, object], names: tuple[str, ...], registered_fields: frozenset[str] = frozenset()
) -> None:
super().__init__(entry)
self.names: tuple[str, ...] = names
self.registered_fields: frozenset[str] = registered_fields
def _alias_groups(alias_owners: Mapping[str, str]) -> Mapping[str, tuple[str, ...]]:

20
litellm/litellm_core_utils/hidden_params.py
View file

@ -1,6 +1,8 @@
from collections.abc import Iterator, MutableMapping
from collections.abc import Iterator, Mapping, MutableMapping
from typing import Final, Protocol, cast
from pydantic import BaseModel
from litellm.types.llms.base import HiddenParams
_HIDDEN_PARAMS_ATTR: Final = "_hidden_params"
@ -47,8 +49,22 @@ class HiddenParamsModelView(MutableMapping[str, object]):
return frozenset(self._hidden_params.model_fields_set) | frozenset(self._hidden_params.model_extra or {})
def stored_attribute(obj: object, name: str) -> object:
if not isinstance(obj, BaseModel):
return getattr(obj, name, None)
values: Final[Mapping[str, object]] = vars(obj)
if name in values:
return values[name]
extras: Final[Mapping[str, object]] = obj.__pydantic_extra__ or {}
return extras.get(name)
def _get_hidden_params_storage(obj: object) -> object | None:
return obj.get(_HIDDEN_PARAMS_ATTR) if isinstance(obj, dict) else getattr(obj, _HIDDEN_PARAMS_ATTR, None)
if isinstance(obj, dict):
return obj.get(_HIDDEN_PARAMS_ATTR)
if isinstance(obj, BaseModel) and _HIDDEN_PARAMS_ATTR in type(obj).__private_attributes__:
return getattr(obj, _HIDDEN_PARAMS_ATTR, None)
return stored_attribute(obj, _HIDDEN_PARAMS_ATTR)
def get_hidden_params_storage(obj: object) -> dict[str, object] | HiddenParams | None:

112
litellm/litellm_core_utils/litellm_logging.py
View file

@ -44,6 +44,7 @@ from litellm.constants import (
EMPTY_MAPPING,
INTERNAL_CALLBACK_NAMES,
KUBERNETES_POD_ROUTING_KEY,
LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL,
PROVIDER_REQUEST_ID_HEADERS,
REDACTED_BY_LITELLM,
)
@ -116,7 +117,8 @@ from litellm.litellm_core_utils.ptu_pricing import is_spilled_over_ptu_request
from litellm.litellm_core_utils.redact_messages import (
redact_message_input_output_from_custom_logger,
redact_message_input_output_from_logging,
redact_streaming_responses_for_custom_logger,
redacted_model_call_details_for_custom_logger,
redacted_response_obj_for_custom_logger,
should_redact_message_logging,
)
from litellm.litellm_core_utils.served_output_texts import (
@ -336,6 +338,7 @@ def _get_provider_request_id(original_exception: Exception) -> str | None:
# Cache custom pricing keys as frozenset for O(1) lookups instead of looping through 49 keys
_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
_METADATA_MAPPING: Final = TypeAdapter(Mapping[str, object])
_MODEL_INFO_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = _CUSTOM_PRICING_KEYS | DEPLOYMENT_SCOPED_PRICING_FIELDS
_UNSERIALIZABLE_METADATA_KEYS: Final[frozenset[str]] = frozenset(
("user_api_key_auth", "user_api_key_budget_reservation")
@ -508,7 +511,7 @@ def _published_pricing(deployment_model: str | None) -> ModelInfo | None:
return None
def _resolve_vertex_location_for_cost(
def resolve_vertex_location_for_cost(
custom_llm_provider: str | None,
litellm_params: Mapping[str, object] | None,
optional_params: Mapping[str, object] | None,
@ -994,6 +997,21 @@ class Logging(LiteLLMLoggingBaseClass):
return model_id
return None
def get_router_pricing_provider(self) -> str | None:
"""The deployment's ``model_info.pricing_provider``, the provider its cost-map lookups run under."""
litellm_params: Final = cast( # cast-ok: callback metadata is caller-provided
Mapping[str, object] | None, getattr(self, "litellm_params", None)
)
if not litellm_params:
return None
candidates: Final = (
_METADATA_MAPPING.validate_python(
_METADATA_MAPPING.validate_python(litellm_params.get(key) or {}).get("model_info") or {}
).get("pricing_provider")
for key in ("litellm_metadata", "metadata")
)
return next((provider for provider in candidates if isinstance(provider, str) and provider), None)
def _routed_deployment_model(self) -> str | None:
litellm_params: Final = self._model_call_details_view.get("litellm_params")
metadata: Final = (
@ -1531,15 +1549,20 @@ class Logging(LiteLLMLoggingBaseClass):
"LiteLLM.LoggingError: [Non-Blocking] Exception occurred while logging %s", e
)
self.model_call_details["api_call_start_time"] = datetime.datetime.now()
# Set-once first provider-handoff instant. api_call_start_time
# is overwritten on every retry, so it can't measure one-time
# preprocessing; pinning the first attempt excludes retry loops
# + backoff. Logging object only — must NOT go into
# litellm_params["metadata"] (caller request metadata, typed
# Dict[str, str], echoed downstream; a datetime breaks it).
if self.model_call_details.get("first_api_call_start_time") is None:
self.model_call_details["first_api_call_start_time"] = self.model_call_details["api_call_start_time"]
# A proxy-gate failure (auth, rate limit, client disconnect) fires pre_call for its callbacks
# without any provider handoff, so it must not stamp the handoff instants.
if LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL not in self.model_call_details:
self.model_call_details["api_call_start_time"] = datetime.datetime.now()
# Set-once first provider-handoff instant. api_call_start_time
# is overwritten on every retry, so it can't measure one-time
# preprocessing; pinning the first attempt excludes retry loops
# + backoff. Logging object only — must NOT go into
# litellm_params["metadata"] (caller request metadata, typed
# Dict[str, str], echoed downstream; a datetime breaks it).
if self.model_call_details.get("first_api_call_start_time") is None:
self.model_call_details["first_api_call_start_time"] = self.model_call_details[
"api_call_start_time"
]
# Input Integration Logging -> If you want to log the fact that an attempt to call the model was made
callbacks: Final = litellm.input_callback + (self.dynamic_input_callbacks or [])
for callback in callbacks:
@ -1933,6 +1956,8 @@ class Logging(LiteLLMLoggingBaseClass):
used for consistent cost calculation across response headers + logging integrations.
"""
self.model_call_details.pop("response_cost_failure_debug_information", None)
if cache_hit is None:
cache_hit = self.model_call_details.get("cache_hit", False)
@ -1982,7 +2007,7 @@ class Logging(LiteLLMLoggingBaseClass):
router_model_id = self.get_router_model_id()
## RESPONSE COST ##
custom_pricing: Final = self._custom_pricing_for(priced_result)
custom_pricing: Final = self.custom_pricing_for(priced_result)
prompt: Final = self._prompt_for_cost_calculation()
@ -2012,13 +2037,14 @@ class Logging(LiteLLMLoggingBaseClass):
"prompt": prompt,
"standard_built_in_tools_params": self.standard_built_in_tools_params,
"router_model_id": router_model_id,
"pricing_provider": self.get_router_pricing_provider(),
"litellm_logging_obj": self,
"data_residency": (
self.litellm_params.get("data_residency")
if hasattr(self, "litellm_params") and self.litellm_params
else None
),
"vertex_location": _resolve_vertex_location_for_cost(
"vertex_location": resolve_vertex_location_for_cost(
custom_llm_provider=cost_custom_llm_provider,
litellm_params=(self.litellm_params if hasattr(self, "litellm_params") else None),
optional_params=self.optional_params,
@ -2093,15 +2119,17 @@ class Logging(LiteLLMLoggingBaseClass):
previous_cost_breakdown: Final = self.cost_breakdown
previous_billed_token_rates: Final = self.billed_token_rates
debug_key: Final = "response_cost_failure_debug_information"
had_failure_debug_info: Final = debug_key in self.model_call_details
previous_failure_debug_info: Final = self.model_call_details.get(debug_key)
try:
cost: Final = self.response_cost_calculator(result=result, cache_hit=False, ignore_stamped_cost=True)
return UncachedPrice(cost=cost, cost_breakdown=self.cost_breakdown)
finally:
self.cost_breakdown = previous_cost_breakdown
self.billed_token_rates = previous_billed_token_rates
if not had_failure_debug_info:
if previous_failure_debug_info is None:
self.model_call_details.pop(debug_key, None)
else:
self.model_call_details[debug_key] = previous_failure_debug_info
def _record_zero_cost_diagnostic(
self,
@ -2154,8 +2182,8 @@ class Logging(LiteLLMLoggingBaseClass):
pricing: Final = pricing_entry_for_cost_calc(
model=model,
completion_response=result,
custom_llm_provider=custom_llm_provider,
custom_pricing=self._custom_pricing_for(result),
custom_llm_provider=self.get_router_pricing_provider() or custom_llm_provider,
custom_pricing=self.custom_pricing_for(result),
base_model=get_base_model_from_metadata(model_call_details=self.model_call_details),
router_model_id=router_model_id or self.get_router_model_id(),
region_name=_resolve_mantle_region_for_cost(
@ -2180,7 +2208,7 @@ class Logging(LiteLLMLoggingBaseClass):
usage=usage,
)
def _custom_pricing_for(self, result: object) -> bool:
def custom_pricing_for(self, result: object) -> bool:
litellm_params: Final = getattr(self, "litellm_params", None)
result_hidden_params: Final = getattr(result, HIDDEN_PARAMS_ATTR, None) or MappingProxyType({})
additional_headers: Final = (
@ -3479,8 +3507,10 @@ class Logging(LiteLLMLoggingBaseClass):
): # custom logger class
if self.stream and complete_streaming_response is None:
callback.log_stream_event(
kwargs=call_details,
response_obj=result,
kwargs=redacted_model_call_details_for_custom_logger(
model_call_details=call_details, custom_logger=callback
),
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
@ -3490,8 +3520,13 @@ class Logging(LiteLLMLoggingBaseClass):
result = call_details_view["complete_response"]
callback.log_success_event(
kwargs=callback.redact_standard_logging_payload_from_model_call_details(call_details),
response_obj=result,
kwargs=redacted_model_call_details_for_custom_logger(
model_call_details=callback.redact_standard_logging_payload_from_model_call_details(
call_details
),
custom_logger=callback,
),
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
@ -3605,6 +3640,14 @@ class Logging(LiteLLMLoggingBaseClass):
model_name=self.get_deployment_model_for_cost(),
litellm_params=self.litellm_params,
model_info=self.get_router_deployment_model_info(),
vertex_location=resolve_vertex_location_for_cost(
custom_llm_provider=self.custom_llm_provider,
litellm_params=_LITELLM_PARAMS.validate_python(
self.litellm_params # pyright: ignore[reportUnknownMemberType] # Logging.litellm_params is untyped
),
optional_params=None,
model=self.get_deployment_model_for_cost() or self.model,
),
)
set_hidden_param(result, "response_cost", batch_result.cost)
@ -3826,7 +3869,7 @@ class Logging(LiteLLMLoggingBaseClass):
model_call_details = callback.redact_standard_logging_payload_from_model_call_details(
model_call_details=model_call_details
)
model_call_details = redact_streaming_responses_for_custom_logger(
model_call_details = redacted_model_call_details_for_custom_logger(
model_call_details=model_call_details, custom_logger=callback
)
##################################
@ -3837,7 +3880,7 @@ class Logging(LiteLLMLoggingBaseClass):
kwargs=model_call_details,
response_obj=model_call_details["async_complete_streaming_response"]
if self.stream
else result,
else redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
guardrail_snapshot=self.shadow_eval_request_snapshot,
@ -3853,14 +3896,14 @@ class Logging(LiteLLMLoggingBaseClass):
else:
await callback.async_log_stream_event( # [TODO]: move this to being an async log stream event function
kwargs=model_call_details,
response_obj=result,
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
else:
await callback.async_log_success_event(
kwargs=model_call_details,
response_obj=result,
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
@ -6471,6 +6514,7 @@ class StandardLoggingPayloadSetup:
custom_llm_provider: str | None,
init_response_obj: object,
api_base: str | None = None,
router_model_id: str | None = None,
) -> StandardLoggingModelInformation:
model_cost_name: Final = select_model_name_for_cost_calc(
model=base_model if custom_pricing else None,
@ -6478,12 +6522,23 @@ class StandardLoggingPayloadSetup:
base_model=base_model,
custom_pricing=custom_pricing,
)
pricing_model_name: Final = (
select_model_name_for_cost_calc(
model=base_model if custom_pricing else None,
completion_response=init_response_obj,
base_model=base_model,
custom_pricing=custom_pricing,
router_model_id=router_model_id,
)
if router_model_id is not None
else model_cost_name
)
if model_cost_name is None:
model_cost_information = StandardLoggingModelInformation(model_map_key="", model_map_value=None)
else:
try:
_model_cost_information: Final = litellm.get_model_info(
model=model_cost_name,
model=pricing_model_name or model_cost_name,
custom_llm_provider=custom_llm_provider,
api_base=api_base,
)
@ -7171,7 +7226,7 @@ def get_standard_logging_object_payload(
# Fall back to the actual deployment model set by the router in metadata.
if base_model is None:
base_model = metadata.get("deployment")
custom_pricing: Final = use_custom_pricing_for_model(litellm_params=litellm_params)
custom_pricing: Final = logging_obj.custom_pricing_for(init_response_obj)
raw_response_cost: Final = kwargs.get("response_cost")
llm_response_cost: Final[float] = raw_response_cost or 0.0
guardrail_cost: Final = guardrail_information_cost(metadata.get("standard_logging_guardrail_information"))
@ -7193,6 +7248,7 @@ def get_standard_logging_object_payload(
custom_llm_provider=kwargs.get("custom_llm_provider"),
init_response_obj=init_response_obj,
api_base=litellm_params.get("api_base"),
router_model_id=logging_obj.get_router_model_id(),
)
error_information, error_str = StandardLoggingPayloadSetup.get_error_information_for_logging_payload(

184
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -285,7 +285,7 @@ def _parse_token_threshold(threshold: str) -> float:
def _parse_above_token_threshold(key: str) -> float:
return _parse_token_threshold(key.split("_above_")[1].split("_tokens")[0])
return _parse_token_threshold(key.rsplit("_above_", 1)[1].split("_tokens")[0])
def _prompt_exceeds_threshold(prompt_tokens: int, threshold: float, inclusive: bool) -> bool:
@ -356,7 +356,7 @@ def get_batch_cost_rates(model_info: ModelInfo, usage: Usage, custom_llm_provide
)
def _select_priced_tier(model_info: ModelInfo, usage: Usage) -> dict | None:
def _select_priced_tier(model_info: ModelInfo, usage: Usage) -> dict[str, object] | None:
tiered_pricing: Final = model_info.get("tiered_pricing")
if not isinstance(tiered_pricing, list) or not tiered_pricing:
return None
@ -545,6 +545,71 @@ class TokenRates:
return self.output_rate if self.reasoning_rate is None else self.reasoning_rate
def resolve_explicit_cache_read_cost_rate(
model_info: ModelInfo,
usage: Usage,
current_time: datetime | None,
service_tier: str | None = None,
) -> float | None:
cache_read_cost_key: Final = "explicit_cache_read_input_token_cost"
service_cache_read_cost_key: Final = _get_service_tier_cost_key(cache_read_cost_key, service_tier)
if (
model_info.get(service_cache_read_cost_key) is None
and model_info.get(cache_read_cost_key) is None
and model_info.get("tiered_pricing") is None
and model_info.get("off_peak_pricing") is None
):
return None
prompt_tokens_details: Final = usage.prompt_tokens_details
cache_type: Final = (
getattr(prompt_tokens_details, "cache_type", None) if prompt_tokens_details is not None else None
)
if parse_prompt_tokens_details(usage)["cache_hit_tokens"] <= 0 or cache_type != "ephemeral":
return None
tier: Final = _select_priced_tier(model_info=model_info, usage=usage)
off_peak: Final = _open_off_peak_block(model_info, current_time)
model_cache_read_cost_rate: Final = _get_cost_per_unit(
model_info,
service_cache_read_cost_key,
default_value=None,
)
tier_has_explicit_rate: Final = tier is not None and cache_read_cost_key in tier
off_peak_has_explicit_rate: Final = (
off_peak is not None and _parse_off_peak_rate(off_peak.get(cache_read_cost_key)) is not None
)
if model_cache_read_cost_rate is None and not tier_has_explicit_rate and not off_peak_has_explicit_rate:
return None
cache_read_cost_rate: Final = (
tier_rate(tier, cache_read_cost_key)
if tier is not None and tier_has_explicit_rate
else tier_rate(tier, "cache_read_input_token_cost", "input_cost_per_token")
if tier is not None
else model_cache_read_cost_rate
if model_cache_read_cost_rate is not None
else _get_cost_per_unit(
model_info,
_get_service_tier_cost_key("cache_read_input_token_cost", service_tier),
default_value=_get_cost_per_unit(model_info, "input_cost_per_token") or 0.0,
)
or 0.0
)
if off_peak is None:
return cache_read_cost_rate
mode_specific_off_peak_rate: Final = _parse_off_peak_rate(off_peak.get(cache_read_cost_key))
generic_off_peak_rate: Final = _parse_off_peak_rate(off_peak.get("cache_read_input_token_cost"))
return (
mode_specific_off_peak_rate
if mode_specific_off_peak_rate is not None
else generic_off_peak_rate
if generic_off_peak_rate is not None
else cache_read_cost_rate
)
def _parse_off_peak_rate(value: object) -> float | None:
if isinstance(value, bool):
return None
@ -683,10 +748,12 @@ def _get_token_base_cost(
# Optimization: collect threshold keys first to avoid sorting all model_info keys.
# Standard thresholds and thresholds suffixed for this request's service tier both count.
tier_key_suffix: Final = get_service_tier_cost_key("", service_tier)
threshold_prefixes: Final = tuple(f"{prefix}_above_" for prefix in _BATCH_RATE_PREFIXES)
threshold_keys: Final = [
k
for k in model_info
if k.startswith("input_cost_per_token_above_")
if k.startswith(threshold_prefixes)
and "_tokens" in k
and (
not k.endswith(_NON_STANDARD_THRESHOLD_SUFFIXES) or (tier_key_suffix != "" and k.endswith(tier_key_suffix))
)
@ -694,12 +761,12 @@ def _get_token_base_cost(
# Only sort the threshold keys (typically 1-2 keys instead of 66+)
threshold: float | None = None
for key in sorted(threshold_keys, key=_parse_above_token_threshold, reverse=True):
for key in sorted(threshold_keys, key=_parse_above_token_threshold):
value = model_info.get(key)
if value is not None:
try:
# Handle both formats: _above_128k_tokens and _above_128_tokens
threshold_str = key.split("_above_")[1].split("_tokens")[0]
threshold_str = key.rsplit("_above_", 1)[1].split("_tokens")[0]
threshold = _parse_above_token_threshold(key)
if _prompt_exceeds_threshold(usage.prompt_tokens, threshold, threshold_is_inclusive):
# Prefer a service_tier-specific above-threshold key when available,
@ -712,7 +779,7 @@ def _get_token_base_cost(
service_tier,
)
if service_tier
else key
else f"input_cost_per_token_above_{threshold_str}_tokens"
)
prompt_base_cost = cast(
float,
@ -768,8 +835,6 @@ def _get_token_base_cost(
)
cache_read_cost = get_cost_per_unit(model_info, cache_read_tiered_key, cache_read_cost)
break
except (IndexError, ValueError):
continue
except Exception:
@ -1088,6 +1153,25 @@ def parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsResu
)
def _modality_token_cost(
model_info: ModelInfo,
modality_cost_key: str,
tokens: int,
prompt_base_cost: float,
service_tier: str | None,
) -> float:
"""
Audio, image and video prompt tokens bill at their modality rate when the model has one.
Without one they are ordinary prompt tokens, so they bill at `prompt_base_cost`, which
`_get_token_base_cost` has already resolved for the service tier and long-context tier.
"""
tiered_key: Final = get_service_tier_cost_key(modality_cost_key, service_tier)
cost_key: Final = tiered_key if model_info.get(tiered_key) is not None else modality_cost_key
if model_info.get(cost_key) is None:
return float(tokens) * prompt_base_cost
return calculate_cost_component(model_info, cost_key, tokens)
def _calculate_input_cost(
prompt_tokens_details: PromptTokensDetailsResult,
model_info: ModelInfo,
@ -1099,8 +1183,14 @@ def _calculate_input_cost(
) -> float:
"""
Calculates the input cost for a given model, prompt tokens, and completion tokens.
A model priced per character (`input_cost_per_character`) bills the reported character count
in place of its text tokens, so the same prompt is never charged under both units.
"""
prompt_cost = float(prompt_tokens_details["text_tokens"]) * prompt_base_cost
bills_characters: Final = (
prompt_tokens_details["character_count"] > 0 and model_info.get("input_cost_per_character") is not None
)
prompt_cost = 0.0 if bills_characters else float(prompt_tokens_details["text_tokens"]) * prompt_base_cost
### CACHE READ COST - Now uses tiered pricing
cache_hit_audio_tokens: Final = prompt_tokens_details["cache_hit_audio_tokens"]
@ -1118,28 +1208,37 @@ def _calculate_input_cost(
if prompt_tokens_details["audio_tokens"] and not (
prompt_tokens_details["audio_length_seconds"] and model_info.get("input_cost_per_audio_per_second") is not None
):
audio_cost_key: Final = get_service_tier_cost_key("input_cost_per_audio_token", service_tier)
prompt_cost += calculate_cost_component(model_info, audio_cost_key, prompt_tokens_details["audio_tokens"])
prompt_cost += _modality_token_cost(
model_info,
"input_cost_per_audio_token",
prompt_tokens_details["audio_tokens"],
prompt_base_cost,
service_tier,
)
### IMAGE TOKEN COST
if prompt_tokens_details["image_tokens"] and not (
prompt_tokens_details["image_count"] and model_info.get("input_cost_per_image") is not None
):
# For image token costs:
# First check if input_cost_per_image_token is available. If not, default to generic input_cost_per_token.
image_token_cost_key = "input_cost_per_image_token"
if model_info.get(image_token_cost_key) is None:
image_token_cost_key = "input_cost_per_token"
prompt_cost += calculate_cost_component(model_info, image_token_cost_key, prompt_tokens_details["image_tokens"])
prompt_cost += _modality_token_cost(
model_info,
"input_cost_per_image_token",
prompt_tokens_details["image_tokens"],
prompt_base_cost,
service_tier,
)
### VIDEO TOKEN COST
if prompt_tokens_details["video_tokens"] and not (
prompt_tokens_details["video_length_seconds"] and model_info.get("input_cost_per_video_per_second") is not None
):
video_token_cost_key = "input_cost_per_video_token"
if model_info.get(video_token_cost_key) is None:
video_token_cost_key = "input_cost_per_token"
prompt_cost += calculate_cost_component(model_info, video_token_cost_key, prompt_tokens_details["video_tokens"])
prompt_cost += _modality_token_cost(
model_info,
"input_cost_per_video_token",
prompt_tokens_details["video_tokens"],
prompt_base_cost,
service_tier,
)
### CACHE WRITING COST - Now uses tiered pricing
if (
@ -1424,12 +1523,19 @@ def generic_cost_per_token(
current_time=billing_time,
threshold_is_inclusive=_uses_inclusive_token_thresholds(custom_llm_provider),
)
resolved_cache_read_cost: Final = resolve_explicit_cache_read_cost_rate(
model_info=resolved_model_info,
usage=usage,
current_time=billing_time,
service_tier=service_tier,
)
effective_cache_read_cost: Final = cache_read_cost if resolved_cache_read_cost is None else resolved_cache_read_cost
prompt_cost = _calculate_input_cost(
prompt_tokens_details=prompt_tokens_details,
model_info=resolved_model_info,
prompt_base_cost=prompt_base_cost,
cache_read_cost=cache_read_cost,
cache_read_cost=effective_cache_read_cost,
cache_creation_cost=cache_creation_cost,
cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
service_tier=service_tier,
@ -1624,6 +1730,16 @@ def _cost_map_billed_rates(
current_time=billing_time,
threshold_is_inclusive=_uses_inclusive_token_thresholds(custom_llm_provider),
)
provider_cache_read_cost_rate: Final = resolve_explicit_cache_read_cost_rate(
model_info=model_info,
usage=usage,
current_time=billing_time,
service_tier=service_tier,
)
effective_cache_read_cost_rate: Final = (
cache_read_cost_rate if provider_cache_read_cost_rate is None else provider_cache_read_cost_rate
)
reasoning_rate: Final = _resolve_billed_reasoning_rate(
model_info=model_info,
usage=usage,
@ -1645,9 +1761,9 @@ def _cost_map_billed_rates(
return BilledTokenRates(
input_cost_per_token=prompt_base_cost,
output_cost_per_token=completion_base_cost,
cache_read_input_token_cost=cache_read_cost_rate,
cache_read_input_token_cost=effective_cache_read_cost_rate,
cache_read_input_audio_token_cost=(
audio_cache_read_rate if audio_cache_read_rate is not None else cache_read_cost_rate
audio_cache_read_rate if audio_cache_read_rate is not None else effective_cache_read_cost_rate
),
cache_creation_input_token_cost=cache_creation_cost_rate,
cache_creation_input_token_cost_above_1hr=cache_creation_cost_above_1hr_rate,
@ -1664,20 +1780,23 @@ def get_billed_token_rates(
vertex_location: str | None = None,
current_time: datetime | None = None,
custom_cost_per_token: CostPerToken | None = None,
model_info: ModelInfo | None = None,
) -> BilledTokenRates | None:
"""Rates the cost calculator bills ``usage`` at, resolved exactly as the totals and the token-type
breakdown resolve them. None when the model's pricing cannot be resolved."""
if custom_cost_per_token is not None:
return _custom_pricing_rates(custom_cost_per_token)
try:
model_info: Final = apply_provider_cache_read_default(
get_model_info(model=model, custom_llm_provider=custom_llm_provider),
resolved_model_info: Final = apply_provider_cache_read_default(
model_info
if model_info is not None
else get_model_info(model=model, custom_llm_provider=custom_llm_provider),
custom_llm_provider,
)
except Exception: # noqa: BLE001 # get_model_info raises a bare Exception for an unmapped model: no rates
return None
return _cost_map_billed_rates(
model_info=model_info,
model_info=resolved_model_info,
usage=usage,
custom_llm_provider=custom_llm_provider,
service_tier=service_tier,
@ -1696,6 +1815,7 @@ def get_token_type_cost_breakdown(
vertex_location: str | None = None,
current_time: datetime | None = None,
custom_cost_per_token: CostPerToken | None = None,
model_info: ModelInfo | None = None,
) -> TokenTypeCostBreakdown:
"""
Provider-agnostic cost of reasoning and cache tokens, derived from the usage
@ -1720,6 +1840,7 @@ def get_token_type_cost_breakdown(
vertex_location=vertex_location,
current_time=current_time,
custom_cost_per_token=custom_cost_per_token,
model_info=model_info,
)
if rates is None:
return TokenTypeCostBreakdown(0.0, 0.0, 0.0)
@ -1772,6 +1893,13 @@ def calculate_prompt_caching_savings(
current_time=billed_at,
threshold_is_inclusive=_uses_inclusive_token_thresholds(custom_llm_provider),
)
provider_cache_read_cost: Final = resolve_explicit_cache_read_cost_rate(
model_info=model_info,
usage=usage,
current_time=billed_at,
service_tier=service_tier,
)
effective_cache_read_cost: Final = cache_read_cost if provider_cache_read_cost is None else provider_cache_read_cost
write_rate: Final = cache_creation_cost or prompt_base_cost
write_rate_1h: Final = cache_creation_cost_above_1hr or write_rate
prompt_tokens_details: Final = parse_prompt_tokens_details(usage)
@ -1786,7 +1914,7 @@ def calculate_prompt_caching_savings(
if details is not None
else None
)
read_discount: Final = cache_read_tokens * max(prompt_base_cost - cache_read_cost, 0.0)
read_discount: Final = cache_read_tokens * max(prompt_base_cost - effective_cache_read_cost, 0.0)
write_premium: Final = calculate_cache_writing_cost(
cache_creation_tokens=cache_creation_tokens,
cache_creation_token_details=cache_creation_details,

4
litellm/litellm_core_utils/prompt_templates/chat_template_tools.py
View file

@ -24,7 +24,9 @@ def _qwen3_5_tool_schemas(functions: Sequence[Mapping[str, object]]) -> str:
_TOOL_SECTIONS: Final = (
(
re.compile(r"deepseek-v4(?![.\d]|p\d)", re.IGNORECASE),
re.compile(
r"deepseek-v4(?![.\d]|p\d)|^(?:deepseek/)?deepseek-(?:chat|reasoner|flash)(?::free)?$", re.IGNORECASE
),
ChatTemplateToolSection(222, 0, _deepseek_v4_tool_schemas),
),
(re.compile(r"qwen3[._]?5(?!\d)", re.IGNORECASE), ChatTemplateToolSection(196, 5, _qwen3_5_tool_schemas)),

16
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -624,7 +624,7 @@ def update_messages_with_model_file_ids(
def update_responses_input_with_model_file_ids(
input: object,
model_id: str | None = None,
model_file_id_mapping: dict[str, dict[str, str]] | None = None,
model_file_id_mapping: Mapping[str, Mapping[str, str]] | None = None,
) -> object:
"""
Updates responses API input with provider-specific file IDs.
@ -665,7 +665,7 @@ def update_responses_input_with_model_file_ids(
for content_item in content:
if isinstance(content_item, dict) and content_item.get("type") == "input_file":
file_id = content_item.get("file_id")
if file_id:
if isinstance(file_id, str) and file_id:
provider_file_id = file_id # Default to original
# Check if we have a mapping for this file ID
@ -761,7 +761,7 @@ def _decode_vector_store_ids_in_tools(
def update_responses_tools_with_model_file_ids(
tools: list[dict[str, object]] | None,
model_id: str | None = None,
model_file_id_mapping: dict[str, dict[str, str]] | None = None,
model_file_id_mapping: Mapping[str, Mapping[str, str]] | None = None,
) -> list[dict[str, object]] | None:
"""
Updates responses API tools with provider-specific file IDs.
@ -2325,6 +2325,16 @@ def is_unsignable_thinking_block(block: object) -> bool:
return not (isinstance(signature, str) and len(signature) > 0)
def has_blank_redacted_data(block: object) -> bool:
match block:
case {"type": "redacted_thinking", "data": str() as data}:
return not data.strip()
case {"type": "redacted_thinking"}:
return True
case _:
return False
def native_redacted_thinking_data(data: str) -> str:
"""Claude's own `data` string for a redacted_thinking block.

272
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -6,10 +6,10 @@ import json
import mimetypes
import re
import xml.etree.ElementTree as ET
from collections.abc import Container, Iterable, Iterator, Mapping, Sequence
from collections.abc import Callable, Container, Iterable, Iterator, Mapping, Sequence
from enum import Enum
from types import MappingProxyType
from typing import Annotated, Any, Final, TypeAlias, TypedDict, cast, overload
from typing import Annotated, Any, Final, TypeAlias, TypedDict, TypeVar, cast, overload
from jinja2.sandbox import ImmutableSandboxedEnvironment
from pydantic import BaseModel, BeforeValidator, ConfigDict, JsonValue, TypeAdapter, ValidationError, model_validator
@ -61,6 +61,7 @@ from litellm.types.utils import GenericImageParsingChunk
from .common_utils import (
converse_redacted_content,
convert_content_list_to_str,
has_blank_redacted_data,
infer_content_type_from_url_and_content,
is_encrypted_reasoning_block,
is_non_content_values_set,
@ -1669,7 +1670,7 @@ def _replace_invalid_tool_use_id_chars(tool_use_id: str, invalid_chars: re.Patte
return invalid_chars.sub("_", tool_use_id) or _TOOL_USE_ID_FALLBACK
def _sanitize_anthropic_tool_use_id(tool_use_id: str) -> str:
def sanitize_anthropic_tool_use_id(tool_use_id: str) -> str:
"""Anthropic requires tool_use_id to match ^[a-zA-Z0-9_-]+$."""
return _replace_invalid_tool_use_id_chars(tool_use_id, _ANTHROPIC_TOOL_USE_ID_INVALID_CHARS)
@ -1815,7 +1816,7 @@ def convert_to_anthropic_tool_result(
tool_message: Final[ChatCompletionToolMessage] = message
tool_call_id: str = tool_message["tool_call_id"]
# Sanitize tool_use_id to match Anthropic's pattern requirement: ^[a-zA-Z0-9_-]+$
sanitized_tool_use_id = _sanitize_anthropic_tool_use_id(tool_call_id)
sanitized_tool_use_id = sanitize_anthropic_tool_use_id(tool_call_id)
# We can't determine from openai message format whether it's a successful or
# error call result so default to the successful result template
@ -1829,7 +1830,7 @@ def convert_to_anthropic_tool_result(
function_message: Final[ChatCompletionFunctionMessage] = message
tool_call_id = function_message.get("tool_call_id") or str(uuid.uuid4())
# Sanitize tool_use_id to match Anthropic's pattern requirement: ^[a-zA-Z0-9_-]+$
sanitized_tool_use_id = _sanitize_anthropic_tool_use_id(tool_call_id)
sanitized_tool_use_id = sanitize_anthropic_tool_use_id(tool_call_id)
anthropic_tool_result = AnthropicMessagesToolResultParam(
type="tool_result",
tool_use_id=sanitized_tool_use_id,
@ -2037,7 +2038,7 @@ def convert_to_anthropic_tool_invoke(
elif is_anthropic_tool_search_use(tool_id, tool_name) and tool_id not in answered_tool_ids:
continue
else:
sanitized_tool_id = _sanitize_anthropic_tool_use_id(tool_id)
sanitized_tool_id = sanitize_anthropic_tool_use_id(tool_id)
_anthropic_tool_use_param = AnthropicMessagesToolUseParam(
type="tool_use",
id=sanitized_tool_id,
@ -2550,22 +2551,125 @@ def _pair_tool_calls_with_results(messages: Sequence[AllMessageValues]) -> tuple
def _replayable_thinking_blocks(
thinking_blocks: list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock],
*,
drop_params: bool = False,
) -> list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock]:
keys_to_drop: Final = _UNSUPPORTED_THINKING_KEYS if drop_params else _STREAM_ONLY_THINKING_KEYS
return [
_without_stream_index(_with_native_redacted_data(block))
_without_keys(_with_native_redacted_data(block), keys_to_drop)
for block in thinking_blocks
if not is_unsignable_thinking_block(block)
if not (is_unsignable_thinking_block(block) or (drop_params and has_blank_redacted_data(block)))
]
def _without_stream_index(
def _as_thinking_block(item: object) -> ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock | None:
match item:
case {"type": "thinking" | "redacted_thinking"}:
return cast( # cast-ok: the mapping pattern matched a thinking block's type tag
ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, item
)
case _:
return None
def _thinking_identity(item: object) -> tuple[str | None, ...] | None:
match item:
case {"type": "redacted_thinking", "data": str() as data}:
return ("redacted_thinking", native_redacted_thinking_data(data))
case {"type": "thinking", "signature": str() as signature} if signature:
return ("thinking", signature)
case {"type": str() as kind} if kind in ("thinking", "redacted_thinking"):
fields: Final = cast(Mapping[str, object], item) # cast-ok: the mapping pattern matched
return _unsigned_thinking_identity(kind, fields.get("thinking"), fields.get("signature"))
case _:
return None
def _unsigned_thinking_identity(kind: str, thinking: object, signature: object) -> tuple[str | None, ...] | None:
match thinking, signature:
case (str() | None as text, str() | None as signed):
return (kind, text, signed)
case _:
return None
_Replayed: Final = TypeVar("_Replayed")
def _with_missing_thinking_blocks(
content: Iterable[_Replayed], thinking_blocks: Sequence[_Replayed]
) -> tuple[_Replayed, ...]:
content_items: Final = tuple(content)
block_identities: Final = tuple(map(_thinking_identity, thinking_blocks))
content_positions: Final[Mapping[tuple[object, ...] | None, int]] = MappingProxyType(
{
identity: index
for index, identity in reversed(tuple(enumerate(map(_thinking_identity, content_items))))
if identity is not None
}
)
anchors: Final = tuple(index for index, identity in enumerate(block_identities) if identity in content_positions)
if not anchors:
return (*thinking_blocks, *content_items)
missing_before: Final = MappingProxyType(
{
content_positions[block_identities[end]]: thinking_blocks[start + 1 : end]
for start, end in zip((-1, *anchors), anchors)
}
)
missing_after: Final = MappingProxyType(
{content_positions[block_identities[anchors[-1]]]: thinking_blocks[anchors[-1] + 1 :]}
)
return tuple(_interleave(content_items, missing_before, missing_after))
def _without_repeated_thinking(
items: Sequence[_Replayed], identity: Callable[[_Replayed], tuple[object, ...] | None]
) -> tuple[_Replayed, ...]:
identities: Final = tuple(map(identity, items))
first_positions: Final[Mapping[tuple[object, ...], int]] = MappingProxyType(
{key: index for index, key in reversed(tuple(enumerate(identities))) if key is not None}
)
return tuple(
item for index, (item, key) in enumerate(zip(items, identities)) if key is None or first_positions[key] == index
)
def _interleave(
content: Sequence[_Replayed],
before: Mapping[int, Sequence[_Replayed]],
after: Mapping[int, Sequence[_Replayed]],
) -> Iterator[_Replayed]:
for index, item in enumerate(content):
yield from before.get(index, ())
yield item
yield from after.get(index, ())
def _thinking_in_replay_order(
content: str | Iterable[object] | None, thinking_blocks: Sequence[object] | None
) -> tuple[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, ...]:
content_items: Final = content if isinstance(content, list) else ()
return tuple(
block
for item in _with_missing_thinking_blocks(content_items, thinking_blocks or ())
if (block := _as_thinking_block(item)) is not None
)
_STREAM_ONLY_THINKING_KEYS: Final = frozenset({"index"})
_UNSUPPORTED_THINKING_KEYS: Final = frozenset({"index", "cache_control"})
def _without_keys(
block: ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock,
keys: frozenset[str],
) -> ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock:
if "index" not in block:
if keys.isdisjoint(block):
return block
return cast( # cast-ok: dropping one key keeps the block's TypedDict shape
return cast( # cast-ok: dropping keys keeps the block's TypedDict shape
ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock,
{key: value for key, value in block.items() if key != "index"},
{key: value for key, value in block.items() if key not in keys},
)
@ -2629,6 +2733,7 @@ def anthropic_messages_pt(
messages: list[AllMessageValues],
model: str,
llm_provider: str,
drop_params: bool = False,
) -> _AnthropicMessageList:
"""
format messages for anthropic
@ -2816,7 +2921,9 @@ def anthropic_messages_pt(
_raw_thinking_blocks = assistant_content_block.get("thinking_blocks", None)
thinking_blocks = (
_replayable_thinking_blocks(_raw_thinking_blocks) if _raw_thinking_blocks is not None else None
_replayable_thinking_blocks(_raw_thinking_blocks, drop_params=drop_params)
if _raw_thinking_blocks is not None
else None
)
# Check if tool_calls contain server tool calls (web search, etc.)
@ -2951,42 +3058,28 @@ def anthropic_messages_pt(
# SEQUENTIAL MODE: No server tool calls, or no thinking blocks,
# or content is a list. Use the original sequential approach.
# When content is a list, check if it already contains thinking
# blocks inline. If so, skip prepending thinking_blocks to avoid
# duplication and preserve the original interleaved order.
# Fixes the gap where list-content messages bypass INTERLEAVED
# MODE and still get thinking blocks prepended out of order.
_content_is_list = "content" in assistant_content_block and isinstance(
assistant_content_block["content"], list
)
_content_list = assistant_content_block.get("content") if _content_is_list else None
_list_has_thinking = False
if _content_is_list and _content_list is not None:
for _item in _content_list:
if isinstance(_item, dict) and _item.get("type") in (
"thinking",
"redacted_thinking",
):
_list_has_thinking = True
break
if (
thinking_blocks is not None and not _list_has_thinking
thinking_blocks is not None and not _content_is_list
): # IMPORTANT: ADD THIS FIRST, ELSE ANTHROPIC WILL RAISE AN ERROR
assistant_content.extend(thinking_blocks)
if _content_is_list and _content_list is not None:
for m in _content_list:
for m in _with_missing_thinking_blocks(_content_list, thinking_blocks or ()):
if not isinstance(m, dict):
continue
# handle thinking blocks
text_block = cast(str, m.get("text", ""))
if m.get("type", "") == "thinking" and not is_unsignable_thinking_block(m):
anthropic_message: ChatCompletionThinkingBlock | AnthropicMessagesTextParam = cast(
ChatCompletionThinkingBlock, m
if (thinking_block := _as_thinking_block(m)) is not None:
assistant_content.extend(
_replayable_thinking_blocks([thinking_block], drop_params=drop_params)
)
assistant_content.append(anthropic_message)
continue
text_block = cast(str, m.get("text", ""))
# handle text
elif (
if (
m.get("type", "") == "text" and len(text_block) > 0
): # don't pass empty text blocks. anthropic api raises errors.
anthropic_message = AnthropicMessagesTextParam(type="text", text=text_block)
@ -3058,7 +3151,12 @@ def anthropic_messages_pt(
msg_i += 1
if assistant_content:
new_messages.append({"role": "assistant", "content": assistant_content})
new_messages.append(
{
"role": "assistant",
"content": list(_without_repeated_thinking(assistant_content, _thinking_identity)),
}
)
## MID-CONVERSATION SYSTEM MESSAGES (placement is the caller's job) ##
while msg_i < len(messages) and messages[msg_i]["role"] == "system":
@ -4190,6 +4288,16 @@ def _convert_to_bedrock_tool_call_result(
return content_block
def _reasoning_identity(block: BedrockContentBlock) -> tuple[object, ...] | None:
match block:
case {"reasoningContent": {"redactedContent": redacted}}:
return ("redacted_thinking", redacted)
case {"reasoningContent": {"reasoningText": {"signature": str() as signature}}} if signature:
return ("thinking", signature)
case _:
return None
def _deduplicate_bedrock_content_blocks(
blocks: list[BedrockContentBlock],
block_key: str,
@ -4673,6 +4781,7 @@ class BedrockConverseMessagesProcessor:
assistant_continue_message: str | ChatCompletionAssistantMessage | None = None,
deployment_model: str | None = None,
base_model: str | None = None,
drop_params: bool = False,
) -> list[BedrockMessageBlock]:
from litellm.llms.bedrock.common_utils import bedrock_converse_accepts_redacted_thinking
@ -4826,33 +4935,21 @@ class BedrockConverseMessagesProcessor:
)
_assistant_content = assistant_message_block.get("content", None)
thinking_blocks = assistant_message_block.get("thinking_blocks")
if thinking_blocks is not None:
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks, to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
_thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
)
if _assistant_content is not None and isinstance(_assistant_content, list):
assistants_parts: list[BedrockContentBlock] = []
for element in _assistant_content:
if isinstance(element, dict):
if element["type"] == "thinking" or element["type"] == "redacted_thinking":
thinking_block = BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks=[element], to_claude=to_claude
)
assistants_parts = (
BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=thinking_block,
assistant_parts=assistants_parts,
)
)
elif element["type"] == "text":
if element["type"] == "text":
# Skip completely empty strings to avoid blank content blocks
if element.get("text", "").strip():
assistants_part = BedrockContentBlock(text=element["text"])
@ -4871,7 +4968,9 @@ class BedrockConverseMessagesProcessor:
model=model,
deployment_model=deployment_model,
)
if _cache_point_block is not None:
if _cache_point_block is not None and not (
drop_params and _as_thinking_block(element) is not None
):
assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
elif _assistant_content is not None and isinstance(_assistant_content, str):
@ -4899,7 +4998,11 @@ class BedrockConverseMessagesProcessor:
msg_i += 1
assistant_content = _deduplicate_bedrock_content_blocks(assistant_content, "toolUse")
assistant_content = list(
_without_repeated_thinking(
_deduplicate_bedrock_content_blocks(assistant_content, "toolUse"), _reasoning_identity
)
)
assistant_content = _sort_bedrock_assistant_content_blocks(assistant_content)
if assistant_content:
@ -5083,6 +5186,7 @@ def _bedrock_converse_messages_pt(
assistant_continue_message: str | ChatCompletionAssistantMessage | None = None,
deployment_model: str | None = None,
base_model: str | None = None,
drop_params: bool = False,
) -> list[BedrockMessageBlock]:
"""
Converts given messages from OpenAI format to Bedrock format
@ -5244,35 +5348,21 @@ def _bedrock_converse_messages_pt(
)
_assistant_content = assistant_message_block.get("content", None)
thinking_blocks = assistant_message_block.get("thinking_blocks")
if thinking_blocks is not None:
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks, to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
_thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
)
if _assistant_content is not None and isinstance(_assistant_content, list):
assistants_parts: list[BedrockContentBlock] = []
for element in _assistant_content:
if isinstance(element, dict):
if element["type"] == "thinking" or element["type"] == "redacted_thinking":
thinking_block = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks=[element], to_claude=to_claude
)
)
assistants_parts = (
BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=thinking_block,
assistant_parts=assistants_parts,
)
)
elif element["type"] == "text":
if element["type"] == "text":
# AWS Bedrock doesn't allow empty or whitespace-only text content
# Skip completely empty strings to avoid blank content blocks
if element.get("text", "").strip():
@ -5292,7 +5382,9 @@ def _bedrock_converse_messages_pt(
model=model,
deployment_model=deployment_model,
)
if _cache_point_block is not None:
if _cache_point_block is not None and not (
drop_params and _as_thinking_block(element) is not None
):
assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
elif _assistant_content is not None and isinstance(_assistant_content, str):
@ -5313,7 +5405,11 @@ def _bedrock_converse_messages_pt(
msg_i += 1
assistant_content = _deduplicate_bedrock_content_blocks(assistant_content, "toolUse")
assistant_content = list(
_without_repeated_thinking(
_deduplicate_bedrock_content_blocks(assistant_content, "toolUse"), _reasoning_identity
)
)
assistant_content = _sort_bedrock_assistant_content_blocks(assistant_content)
if assistant_content:
@ -5900,21 +5996,21 @@ def _optional_tool_name(value: object) -> str | None:
def _split_tool_call_ids(calls: Sequence[tuple[str | None, int]]) -> tuple[tuple[str | None, ...], ...]:
taken: Final = frozenset(_sanitize_anthropic_tool_use_id(call_id) for call_id, _ in calls if call_id)
taken: Final = frozenset(sanitize_anthropic_tool_use_id(call_id) for call_id, _ in calls if call_id)
def fresh(call_id: str) -> Iterator[str]:
return filter(
lambda candidate: _sanitize_anthropic_tool_use_id(candidate) not in taken,
lambda candidate: sanitize_anthropic_tool_use_id(candidate) not in taken,
(f"{call_id}__concat_{n}" for n in itertools.count(1)),
)
suffixes: Final = MappingProxyType(
{_sanitize_anthropic_tool_use_id(call_id): fresh(call_id) for call_id, count in calls if call_id and count > 1}
{sanitize_anthropic_tool_use_id(call_id): fresh(call_id) for call_id, count in calls if call_id and count > 1}
)
return tuple(
(
call_id,
*(next(suffixes[_sanitize_anthropic_tool_use_id(call_id)]) for _ in range(count - 1)),
*(next(suffixes[sanitize_anthropic_tool_use_id(call_id)]) for _ in range(count - 1)),
)
if call_id
else (None,) * count

19
litellm/litellm_core_utils/prompt_templates/mid_conversation_system.py
View file

@ -49,7 +49,7 @@ from litellm.types.llms.openai import (
ChatCompletionUserMessage,
)
from .common_utils import is_unsignable_thinking_block
from .common_utils import has_blank_redacted_data, is_unsignable_thinking_block
CONVERTED_SYSTEM_NOTE: Final = (
"Operator note (not from the user): the following was originally a mid-conversation system-role reminder."
@ -267,27 +267,22 @@ def _block_index_by_message(blocks: Sequence[tuple[bool, tuple[int, ...]]]) -> M
def _thinking_block_renders(block: object) -> bool:
"""A thinking block the converter keeps: one Anthropic can verify, so never bridged encrypted reasoning."""
return message_field(block, "type") in _THINKING_BLOCK_TYPES and not is_unsignable_thinking_block(block)
return message_field(block, "type") in _THINKING_BLOCK_TYPES and not (
is_unsignable_thinking_block(block) or has_blank_redacted_data(block)
)
def _assistant_part_renders(part: object) -> bool:
"""A text part always renders: the converter pads empty text with a placeholder."""
part_type: Final = message_field(part, "type")
if part_type == "thinking":
if part_type in _THINKING_BLOCK_TYPES:
return _thinking_block_renders(part)
return part_type in _RENDERED_ASSISTANT_PART_TYPES or (
isinstance(part_type, str) and part_type.endswith("_tool_result")
)
def _separate_thinking_blocks_render(message: object, parts: Sequence[object]) -> bool:
"""``thinking_blocks`` reach the wire only when no inline thinking part claims the slot.
The converter skips the separate blocks as soon as the content list carries a
``thinking`` or ``redacted_thinking`` part, whether or not that part itself renders.
"""
if any(message_field(part, "type") in _THINKING_BLOCK_TYPES for part in parts):
return False
def _separate_thinking_blocks_render(message: object) -> bool:
return any(_thinking_block_renders(block) for block in parts_of(message_field(message, "thinking_blocks")))
@ -404,7 +399,7 @@ def _assistant_renders(message: object, answered: frozenset[str]) -> bool:
parts: Final = parts_of(content)
return (
any(_assistant_part_renders(part) for part in parts)
or _separate_thinking_blocks_render(message, parts)
or _separate_thinking_blocks_render(message)
or _tool_calls_render(message, answered)
or bool(message_field(message, "function_call"))
or bool(message_field(message_field(message, "provider_specific_fields"), "compaction_blocks"))

28
litellm/litellm_core_utils/reasoning_effort_utils.py
View file

@ -1,16 +1,19 @@
from typing import Literal
from typing import Final, Literal
import litellm
from litellm.constants import (
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
)
from litellm.exceptions import UnsupportedParamsError
from litellm.litellm_core_utils.core_helpers import as_str_mapping
OpenAIStyleReasoningEffort = Literal["minimal", "low", "medium", "high"]
def reasoning_effort_from_thinking_budget(
budget_tokens: int,
budget_tokens: float,
) -> OpenAIStyleReasoningEffort:
"""Bucket an Anthropic ``thinking.budget_tokens`` into an OpenAI-style
``reasoning_effort`` using the shared ``DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET``
@ -24,3 +27,24 @@ def reasoning_effort_from_thinking_budget(
if budget_tokens >= DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET:
return "low"
return "minimal"
def reasoning_effort_from_thinking(thinking: object, *, drop_params: bool) -> OpenAIStyleReasoningEffort | None:
fields: Final = as_str_mapping(thinking)
if fields is None or fields.get("type") != "enabled":
return None
budget: Final = fields.get("budget_tokens", 0)
if isinstance(budget, (int, float)) and not isinstance(budget, bool):
return reasoning_effort_from_thinking_budget(budget)
if drop_params or litellm.drop_params is True:
return None
raise non_numeric_thinking_budget_error(budget)
def non_numeric_thinking_budget_error(budget: object) -> UnsupportedParamsError:
return UnsupportedParamsError(
message=(
f"thinking.budget_tokens must be a number, got {type(budget).__name__}. To drop it, set "
"`litellm.drop_params=True`, or `drop_params: true` under `litellm_settings` on the proxy"
)
)

230
litellm/litellm_core_utils/redact_messages.py
View file

@ -10,10 +10,12 @@
import asyncio
import copy
import inspect
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from dataclasses import replace
from typing import TYPE_CHECKING, Any, Final
from pydantic import TypeAdapter
import litellm
from litellm.constants import REDACTED_BY_LITELLM
from litellm.integrations.custom_logger import CustomLogger
@ -40,36 +42,79 @@ else:
LiteLLMLoggingObject = Any
def _opted_out_of_message_logging(custom_logger: CustomLogger) -> bool:
return hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True
def redact_message_input_output_from_custom_logger(
litellm_logging_obj: LiteLLMLoggingObject, result, custom_logger: CustomLogger
):
if hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True:
) -> object:
if _opted_out_of_message_logging(custom_logger):
return perform_redaction(litellm_logging_obj.model_call_details, result, redact_streaming_responses=False)
return result
def redact_streaming_responses_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
def redacted_model_call_details_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
"""
Returns a copy of model_call_details whose streaming response entries are redacted deepcopies
when the custom logger has opted out of message logging. The shared model_call_details is left
untouched so other callbacks still receive the unredacted response.
Returns a copy of model_call_details with the request messages, instructions and streaming responses
redacted when the custom logger has opted out of message logging. The shared model_call_details is left
untouched so other callbacks and the spend log still receive the unredacted values.
"""
if not (hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True):
if not _opted_out_of_message_logging(custom_logger):
return model_call_details
redacted_entries: Final = {
streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
if model_call_details.get(streaming_key) is not None
return {
**{key: value for key, value in model_call_details.items() if key != SERVED_OUTPUT_TEXTS_KEY},
**_redacted_entries_for_opted_out_logger(model_call_details),
}
if not redacted_entries:
return model_call_details
return {**model_call_details, **redacted_entries}
def _redacted_streaming_response_copy(streaming_response):
redacted_response: Final = copy.deepcopy(streaming_response)
_redact_streaming_response(redacted_response)
return redacted_response
def _redacted_entries_for_opted_out_logger(model_call_details: Mapping[str, object]) -> Mapping[str, object]:
optional_params: Final = _as_mapping(model_call_details.get("optional_params"))
litellm_params: Final = _as_mapping(model_call_details.get("litellm_params"))
standard_logging_object: Final = _as_mapping(model_call_details.get("standard_logging_object"))
return {
"messages": [{"role": "user", "content": REDACTED_BY_LITELLM}],
"prompt": "",
"input": "",
**({"instructions": REDACTED_BY_LITELLM} if model_call_details.get("instructions") is not None else {}),
**({"optional_params": _with_redacted_instructions(optional_params)} if optional_params is not None else {}),
**({"litellm_params": redacted_litellm_params(litellm_params)} if litellm_params is not None else {}),
**(
{"standard_logging_object": _redact_standard_logging_object(standard_logging_object)}
if standard_logging_object is not None
else {}
),
**{
streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
for streaming_key in (
"complete_streaming_response",
"async_complete_streaming_response",
"complete_response",
)
if model_call_details.get(streaming_key) is not None
},
}
def redacted_response_obj_for_custom_logger(response_obj: object, custom_logger: CustomLogger) -> object:
if not _opted_out_of_message_logging(custom_logger) or not _is_redactable_response(response_obj):
return response_obj
return perform_redaction({}, response_obj)
def _is_redactable_response(response_obj: object) -> bool:
return isinstance(
response_obj, (litellm.ModelResponse, litellm.ResponsesAPIResponse, litellm.EmbeddingResponse)
) or (isinstance(response_obj, dict) and ("choices" in response_obj or "output" in response_obj))
def _redacted_streaming_response_copy(streaming_response: object) -> object:
redacted_response: Final = (
streaming_response
if isinstance(streaming_response, litellm.ResponsesAPIResponse)
else copy.deepcopy(streaming_response)
)
return _redact_streaming_response(redacted_response)
def _redact_streaming_response(streaming_response):
@ -77,10 +122,13 @@ def _redact_streaming_response(streaming_response):
for choice in streaming_response.choices:
_redact_choice_content(choice)
redact_vertex_ai_metadata_from_logged_object(streaming_response)
elif isinstance(streaming_response, litellm.ResponsesAPIResponse):
return perform_redaction({}, streaming_response)
elif hasattr(streaming_response, "output"):
_redact_responses_api_output(streaming_response.output)
if hasattr(streaming_response, "reasoning") and streaming_response.reasoning is not None:
streaming_response.reasoning = None
return streaming_response
def _redact_tool_calls(tool_calls) -> None:
@ -177,31 +225,75 @@ def _redact_responses_api_output(output_items):
output_item.input = REDACTED_BY_LITELLM
def _redact_responses_api_output_dict(output_items, redacted_str: str):
"""Helper to redact ResponsesAPIResponse output items in dict form."""
for output_item in output_items:
if not isinstance(output_item, dict):
continue
_RESPONSE_ITEMS: Final = TypeAdapter(tuple[object, ...])
if output_item.get("text") is not None:
output_item["text"] = redacted_str
if isinstance(output_item.get("content"), list):
for content_item in output_item["content"]:
if isinstance(content_item, dict) and content_item.get("text") is not None:
content_item["text"] = redacted_str
if isinstance(content_item, dict) and content_item.get("refusal") is not None:
content_item["refusal"] = redacted_str
def _redacted_responses_api_output_part(part: object, fields: tuple[str, ...], redacted_str: str) -> object:
if (part_mapping := _as_mapping(part)) is None:
return part
return {key: redacted_str if key in fields and value is not None else value for key, value in part_mapping.items()}
if output_item.get("type") == "reasoning" and isinstance(output_item.get("summary"), list):
for summary_item in output_item["summary"]:
if isinstance(summary_item, dict) and summary_item.get("text") is not None:
summary_item["text"] = redacted_str
if output_item.get("type") == "function_call" and "arguments" in output_item:
output_item["arguments"] = redacted_str
if output_item.get("type") == "custom_tool_call" and "input" in output_item:
output_item["input"] = redacted_str
def _redacted_responses_api_output_item(item: object, redacted_str: str) -> object:
if (item_mapping := _as_mapping(item)) is None:
return item
content: Final = item_mapping.get("content")
summary: Final = item_mapping.get("summary")
return {
**item_mapping,
**({"text": redacted_str} if item_mapping.get("text") is not None else {}),
**(
{
"content": [
_redacted_responses_api_output_part(part, ("text", "refusal"), redacted_str)
for part in _RESPONSE_ITEMS.validate_python(content)
]
}
if isinstance(content, list)
else {}
),
**(
{
"summary": [
_redacted_responses_api_output_part(part, ("text",), redacted_str)
for part in _RESPONSE_ITEMS.validate_python(summary)
]
}
if item_mapping.get("type") == "reasoning" and isinstance(summary, list)
else {}
),
**(
{"arguments": redacted_str}
if item_mapping.get("type") == "function_call" and "arguments" in item_mapping
else {}
),
**(
{"input": redacted_str}
if item_mapping.get("type") == "custom_tool_call" and "input" in item_mapping
else {}
),
}
def _redact_responses_api_output_dict(output_items: Sequence[object], redacted_str: str) -> Sequence[object]:
return [_redacted_responses_api_output_item(item, redacted_str) for item in output_items]
def _with_redacted_instructions(params: Mapping[str, object]) -> Mapping[str, object]:
return params if params.get("instructions") is None else {**params, "instructions": REDACTED_BY_LITELLM}
def _redacted_responses_api_response(
response: Mapping[str, object], *, redact_instructions: bool = True
) -> Mapping[str, object]:
output: Final = response.get("output")
redacted: Final = {
**response,
"output": _redact_responses_api_output_dict(_RESPONSE_ITEMS.validate_python(output), REDACTED_BY_LITELLM)
if isinstance(output, list)
else output,
}
return _with_redacted_instructions(redacted) if redact_instructions else redacted
def redacted_standard_logging_payload(payload: Mapping[str, object]) -> Mapping[str, object]:
@ -215,20 +307,25 @@ def redacted_standard_logging_payload(payload: Mapping[str, object]) -> Mapping[
return _redact_standard_logging_object(payload)
def _redact_standard_logging_object(payload: Mapping[str, object]) -> dict[str, object]:
def _redact_standard_logging_object(
payload: Mapping[str, object], *, redact_instructions: bool = True
) -> dict[str, object]:
standard_logging_object: Final = copy.deepcopy(without_classifier_audit(payload))
redacted_str: Final = REDACTED_BY_LITELLM
if standard_logging_object.get("messages") is not None:
standard_logging_object["messages"] = [{"role": "user", "content": redacted_str}]
model_parameters: Final = _as_mapping(standard_logging_object.get("model_parameters"))
if model_parameters is not None and redact_instructions:
standard_logging_object["model_parameters"] = _with_redacted_instructions(model_parameters)
response: Final = standard_logging_object.get("response")
if response is not None:
if isinstance(response, dict) and "output" in response:
# ResponsesAPIResponse format - redact content in output items
if isinstance(response.get("output"), list):
_redact_responses_api_output_dict(response["output"], redacted_str)
redact_vertex_ai_metadata_from_logged_object(response)
standard_logging_object["response"] = _redacted_responses_api_response(
response, redact_instructions=redact_instructions
)
redact_vertex_ai_metadata_from_logged_object(standard_logging_object["response"])
elif isinstance(response, dict) and "choices" in response:
# ModelResponse dict format - redact content in choices
if isinstance(response.get("choices"), list):
@ -320,11 +417,11 @@ def _redact_hidden_provider_specific_fields(params: Mapping[str, object]) -> Non
def redacted_litellm_params(params: Mapping[str, object], *, redact_request_body: bool = True) -> dict[str, object]:
"""
Copy of litellm_params without classifier audit fields or baseline route request parameters.
redact_request_body also redacts the nested proxy_server_request body.
redact_request_body also redacts the request instructions and the nested proxy_server_request body.
"""
request: Final = _as_mapping(params.get("proxy_server_request"))
return {
**params,
**(_with_redacted_instructions(params) if redact_request_body else params),
**{
key: _redacted_baseline_metadata(value)
for key, value in params.items()
@ -344,8 +441,9 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
redact_streaming_responses=False skips the in-place redaction of the shared streaming
response entries; per-callback redaction hands each opted-out callback its own redacted
copy via redact_streaming_responses_for_custom_logger instead. It also leaves the shared
proxy request body and hidden provider fields as they are, since the spend log reads them.
copy via redacted_model_call_details_for_custom_logger instead. It also leaves the shared
request instructions, proxy request body and hidden provider fields as they are, since the
spend log and the other callbacks read them.
"""
# Redact model_call_details
params: Final = _as_mapping(model_call_details.get("litellm_params"))
@ -358,16 +456,31 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
model_call_details["messages"] = [{"role": "user", "content": REDACTED_BY_LITELLM}]
model_call_details["prompt"] = ""
model_call_details["input"] = ""
optional_params: Final = _as_mapping(model_call_details.get("optional_params"))
if redact_streaming_responses:
model_call_details.update(
{
**({"instructions": REDACTED_BY_LITELLM} if model_call_details.get("instructions") is not None else {}),
**({"optional_params": _with_redacted_instructions(optional_params)} if optional_params else {}),
}
)
model_call_details.pop(SERVED_OUTPUT_TEXTS_KEY, None)
standard_logging_object: Final = model_call_details.get("standard_logging_object")
if isinstance(standard_logging_object, Mapping):
model_call_details["standard_logging_object"] = _redact_standard_logging_object(standard_logging_object)
model_call_details["standard_logging_object"] = _redact_standard_logging_object(
standard_logging_object, redact_instructions=redact_streaming_responses
)
redact_vertex_ai_metadata_from_litellm_params(model_call_details)
# Redact streaming response
if redact_streaming_responses and model_call_details.get("stream", False) is True:
for _streaming_key in ("complete_streaming_response", "async_complete_streaming_response"):
_redact_streaming_response(model_call_details.get(_streaming_key))
model_call_details.update(
{
streaming_key: _redact_streaming_response(model_call_details[streaming_key])
for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
if model_call_details.get(streaming_key) is not None
}
)
# Redact result
if result is not None:
@ -381,10 +494,7 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
# For async objects, return a simple redacted response without deepcopy
return {"text": REDACTED_BY_LITELLM}
if not (
isinstance(result, (litellm.ModelResponse, litellm.ResponsesAPIResponse, litellm.EmbeddingResponse))
or (isinstance(result, dict) and ("choices" in result or "output" in result))
):
if not _is_redactable_response(result):
return {"text": REDACTED_BY_LITELLM}
_result: Final = copy.deepcopy(result)
@ -399,14 +509,18 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
_redact_model_response_dict_choices(_result["choices"], REDACTED_BY_LITELLM)
redact_vertex_ai_metadata_from_logged_object(_result)
elif isinstance(_result, dict) and "output" in _result:
if isinstance(_result.get("output"), list):
_redact_responses_api_output_dict(_result["output"], REDACTED_BY_LITELLM)
return _redacted_responses_api_response(_result, redact_instructions=redact_streaming_responses)
elif isinstance(_result, litellm.ResponsesAPIResponse):
if hasattr(_result, "output"):
_redact_responses_api_output(_result.output)
# Redact reasoning field in ResponsesAPIResponse
if hasattr(_result, "reasoning") and _result.reasoning is not None:
_result.reasoning = None
return _result.model_copy(
update={"instructions": REDACTED_BY_LITELLM}
if redact_streaming_responses and _result.instructions is not None
else {}
)
elif isinstance(_result, litellm.EmbeddingResponse):
if hasattr(_result, "data") and _result.data is not None:
_result.data = []

23
litellm/litellm_core_utils/request_timeout_resolver.py
View file

@ -12,10 +12,15 @@ tell "user asked for this" from "nobody set it". This resolver answers that:
from __future__ import annotations
from typing import Final
from collections.abc import Mapping
from typing import Final, Literal, Protocol
from pydantic import TypeAdapter
from litellm.constants import DEFAULT_REQUEST_TIMEOUT_SECONDS
_TIMEOUTS_BY_SERVICE_TIER: Final[TypeAdapter[Mapping[str, float] | None]] = TypeAdapter(Mapping[str, float] | None)
def get_configured_request_timeout() -> float | None:
"""Return the explicitly-configured ``litellm.request_timeout``, else ``None``."""
@ -27,3 +32,19 @@ def get_configured_request_timeout() -> float | None:
if timeout != float(DEFAULT_REQUEST_TIMEOUT_SECONDS):
return timeout
return None
class _DeploymentParams(Protocol):
def get(self, key: str, /) -> object: ...
def get_service_tier_timeout(
service_tier: object,
litellm_params: _DeploymentParams,
key: Literal["timeout_by_service_tier", "stream_timeout_by_service_tier"],
) -> float | None:
"""Return the deployment's ``key`` timeout for the request's ``service_tier``, else ``None``."""
if not isinstance(service_tier, str):
return None
timeouts: Final = _TIMEOUTS_BY_SERVICE_TIER.validate_python(litellm_params.get(key))
return None if timeouts is None else timeouts.get(service_tier)

183
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -1,13 +1,13 @@
import base64
import builtins
import time
from collections.abc import Callable, Iterator, Mapping, Sequence
from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass, replace
from itertools import chain, groupby
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, Optional, TypeAlias, TypedDict, Union, cast
from typing import TYPE_CHECKING, Final, Optional, Protocol, TypeAlias, TypedDict, TypeVar, Union, cast
from pydantic import BaseModel
from pydantic import BaseModel, TypeAdapter, ValidationError
from typing_extensions import ReadOnly, Required
import litellm
@ -17,6 +17,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
get_content_from_model_response, # pyright: ignore[reportUnknownVariableType] # response param accepts an untyped dict
)
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAssistantContentValue,
ChatCompletionAudioDelta,
)
@ -36,12 +37,16 @@ from litellm.types.utils import (
Function,
FunctionCall,
LlmProviders,
Logprobs,
Message,
ModelResponse,
ModelResponseStream,
PromptTokensDetailsWrapper,
ServerToolUse,
StreamingChoices,
TextChoices,
TextCompletionResponse,
UnreportedUsage,
Usage,
)
from litellm.utils import print_verbose, token_counter
@ -857,7 +862,9 @@ class ChunkProcessor:
return reasoning_tokens
@staticmethod
def _extract_usage_chunk(chunk: "_UsageBearingChunk | ModelResponse | ModelResponseStream") -> Usage | None:
def extract_usage_chunk(
chunk: "_UsageBearingChunk | ModelResponse | ModelResponseStream | TextCompletionResponse",
) -> Usage | None:
usage_chunk: Usage | CompletionUsage | None = None
if hasattr(chunk, "usage") and chunk.usage is not None:
usage_chunk = chunk.usage
@ -870,6 +877,8 @@ class ChunkProcessor:
if isinstance(usage_chunk, dict):
return Usage(**usage_chunk)
if isinstance(usage_chunk, UnreportedUsage):
return None
if usage_chunk is None or isinstance(usage_chunk, Usage):
return usage_chunk
return Usage(**usage_chunk.model_dump())
@ -916,7 +925,7 @@ class ChunkProcessor:
cost: float | None = None
for chunk in chunks:
usage_chunk = self._extract_usage_chunk(chunk)
usage_chunk = self.extract_usage_chunk(chunk)
if usage_chunk is not None:
usage_chunk_dict = self._usage_chunk_calculation_helper(usage_chunk)
@ -1013,7 +1022,7 @@ class ChunkProcessor:
values: Final = [
value
for chunk in chunks
if (usage_chunk := self._extract_usage_chunk(chunk)) is not None
if (usage_chunk := self.extract_usage_chunk(chunk)) is not None
and isinstance(value := getattr(usage_chunk, field, None), str)
]
return values[-1] if values else None
@ -1086,7 +1095,7 @@ class ChunkProcessor:
chunks: Sequence["_UsageBearingChunk | ModelResponse"],
model: str,
completion_output: str,
messages: Sequence | None = None,
messages: Sequence[object] | None = None,
reasoning_tokens: int | None = None,
count_prompt_tokens: Callable[[], int] | None = None,
) -> Usage:
@ -1199,7 +1208,7 @@ class ChunkProcessor:
return returned_usage
def concatenate_base64_list(base64_strings: list[str]) -> str:
def concatenate_base64_list(base64_strings: Sequence[str]) -> str:
"""
Concatenates a list of base64-encoded strings.
@ -1216,35 +1225,115 @@ def concatenate_base64_list(base64_strings: list[str]) -> str:
return base64.b64encode(combined_bytes).decode("utf-8")
def stream_chunk_builder_text_completion(chunks: list, messages: Sequence | None = None) -> TextCompletionResponse:
id: Final = chunks[0]["id"]
object: Final = chunks[0]["object"]
created: Final = chunks[0]["created"]
model: Final = chunks[0]["model"]
system_fingerprint: Final = chunks[0].get("system_fingerprint", None)
finish_reason: Final = chunks[-1]["choices"][0]["finish_reason"]
logprobs: Final = chunks[-1]["choices"][0]["logprobs"]
_CHOICE_DICTS: Final = TypeAdapter(list[dict[str, object]])
content_list: Final = []
for chunk in chunks:
choices = chunk["choices"]
for choice in choices:
if choice is not None and hasattr(choice, "text") and choice.get("text") is not None:
_choice = choice.get("text")
content_list.append(_choice)
# Combine the "content" strings into a single string || combine the 'function' strings into a single string
combined_content: Final = "".join(content_list)
def _text_completion_chunk_from_mapping(chunk: Mapping[str, object]) -> TextCompletionResponse:
try:
prompt_tokens = token_counter(model=model, messages=messages)
except Exception: # don't allow this failing to block a complete streaming response from being returned
print_verbose("token_counter failed, assuming prompt tokens is 0")
prompt_tokens = 0
completion_tokens: Final = token_counter(
model=model,
text=combined_content,
count_response_tokens=True, # count_response_tokens is a Flag to tell token counter this is a response, No need to add extra tokens we do for input messages
return TextCompletionResponse.model_validate(chunk)
except ValidationError:
choices: Final = _CHOICE_DICTS.validate_python(chunk.get("choices", []))
return TextCompletionResponse.model_validate(
{**chunk, "choices": [{**choice, "logprobs": None} for choice in choices]}
)
def _as_text_completion_chunk(
chunk: TextCompletionResponse | ModelResponseStream | Mapping[str, object],
) -> TextCompletionResponse:
if isinstance(chunk, TextCompletionResponse):
return chunk
if isinstance(chunk, ModelResponseStream):
return TextCompletionResponse(
id=chunk.id,
created=chunk.created,
model=chunk.model,
choices=[TextChoices(index=choice.index, finish_reason=choice.finish_reason) for choice in chunk.choices],
usage=ChunkProcessor.extract_usage_chunk(chunk),
)
return _text_completion_chunk_from_mapping(chunk)
_LogprobsValue: Final = TypeVar("_LogprobsValue")
def _joined_or_none(fields: Iterable[Sequence[_LogprobsValue] | None]) -> list[_LogprobsValue] | None:
sent: Final = tuple(field for field in fields if field is not None)
return list(chain.from_iterable(sent)) if sent else None
class _CarriesLogprobs(Protocol):
@property
def logprobs(self) -> object: ...
def _sent_logprobs(choice: _CarriesLogprobs) -> object:
return choice.logprobs
def _merged_logprobs(choices: Sequence[TextChoices]) -> object:
sent: Final = tuple(logprobs for logprobs in map(_sent_logprobs, choices) if logprobs is not None)
parts: Final = tuple(logprobs for logprobs in sent if isinstance(logprobs, Logprobs))
if len(parts) < len(sent):
return sent[-1]
if not parts:
return None
return Logprobs(
text_offset=_joined_or_none(part.text_offset for part in parts),
token_logprobs=_joined_or_none(part.token_logprobs for part in parts),
tokens=_joined_or_none(part.tokens for part in parts),
top_logprobs=_joined_or_none(part.top_logprobs for part in parts),
)
def _text_of(choice: TextChoices) -> str:
return choice.text if isinstance(choice.text, str) else ""
def _index_of(choice: TextChoices) -> int:
try:
return int(choice.index or 0)
except (TypeError, ValueError, OverflowError):
return 0
def _rebuilt_text_choice(index: int, choices: Sequence[TextChoices]) -> TextChoices:
return TextChoices(
text="".join(map(_text_of, choices)),
index=index,
logprobs=_merged_logprobs(choices),
finish_reason=next((choice.finish_reason for choice in reversed(choices) if choice.finish_reason), None),
)
def _reported_usage(chunk: TextCompletionResponse) -> Usage | None:
usage: Final = ChunkProcessor.extract_usage_chunk(chunk)
return usage if usage is not None and (usage.prompt_tokens or usage.completion_tokens) else None
def stream_chunk_builder_text_completion(
chunks: Sequence[TextCompletionResponse | ModelResponseStream | Mapping[str, object]],
messages: Sequence[AllMessageValues | Message] | None = None,
) -> TextCompletionResponse:
text_chunks: Final = tuple(_as_text_completion_chunk(chunk) for chunk in chunks)
id: Final = text_chunks[0].id
object: Final = text_chunks[0].object
created: Final = text_chunks[0].created
model: Final = text_chunks[0].model
system_fingerprint: Final = text_chunks[0].get("system_fingerprint", None)
all_choices: Final = tuple(chain.from_iterable(chunk.choices for chunk in text_chunks))
choices: Final = [
_rebuilt_text_choice(index, tuple(group))
for index, group in groupby(sorted(all_choices, key=_index_of), key=_index_of)
] or [_rebuilt_text_choice(0, ())]
combined_content: Final = "".join(map(_text_of, choices))
reported_usages: Final = tuple(reported for reported in map(_reported_usage, text_chunks) if reported is not None)
usage: Final = ChunkProcessor(list(text_chunks), messages).calculate_usage(
chunks=tuple(_UsageBearingChunk(usage=reported) for reported in reported_usages),
model=model or "",
completion_output=combined_content,
messages=messages,
)
response: Final = {
@ -1253,19 +1342,8 @@ def stream_chunk_builder_text_completion(chunks: list, messages: Sequence | None
"created": created,
"model": model,
"system_fingerprint": system_fingerprint,
"choices": [
{
"text": combined_content,
"index": 0,
"logprobs": logprobs,
"finish_reason": finish_reason,
}
],
"usage": {
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": prompt_tokens + completion_tokens,
},
"choices": choices,
"usage": usage,
}
return TextCompletionResponse(**response)
@ -1416,9 +1494,14 @@ def stream_chunk_builder(
return None
## Route to the text completion logic
first_chunk_with_choices: Final = next((c for c in chunks if c.get("choices")), None)
if first_chunk_with_choices is not None and isinstance(
first_chunk_with_choices["choices"][0], litellm.utils.TextChoices
): # route to the text completion logic
if first_chunk_with_choices is not None and (
isinstance(first_chunk_with_choices["choices"][0], TextChoices)
or (
isinstance(first_chunk_with_choices["choices"][0], Mapping)
and "text" in first_chunk_with_choices["choices"][0]
and "delta" not in first_chunk_with_choices["choices"][0]
)
):
return stream_chunk_builder_text_completion(chunks=chunks, messages=messages)
model: Final = chunks[0]["model"]

306
litellm/litellm_core_utils/streaming_handler.py
View file

@ -15,6 +15,8 @@ from typing import Any, Final, NoReturn, Protocol, TypeVar, cast
import anyio
import httpx
from openai.types.chat import ChatCompletionChunk
from openai.types.chat.chat_completion_chunk import Choice as ChunkChoice
from openai.types.chat.chat_completion_chunk import ChoiceDelta
from pydantic import BaseModel, ValidationError
from typing_extensions import NotRequired, TypedDict
@ -22,7 +24,12 @@ import litellm
from litellm import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.hidden_params import HIDDEN_PARAMS_ATTR, get_hidden_params
from litellm.litellm_core_utils.hidden_params import (
HIDDEN_PARAMS_ATTR,
get_hidden_params,
set_hidden_param,
stored_attribute,
)
from litellm.litellm_core_utils.model_response_utils import (
is_model_response_stream_empty,
)
@ -41,6 +48,8 @@ from litellm.types.utils import (
ModelResponseStream,
PromptTokensDetailsWrapper,
StreamingChoices,
TextCompletionResponse,
UnreportedUsage,
Usage,
)
from litellm.types.utils import GenericStreamingChunk as GChunk
@ -60,6 +69,7 @@ FUNCTION_CALL_ATTRIBUTE: Final = "function_call"
_SYNC_ITER_EXHAUSTED: Final = object()
_GCHUNK_FIELDS: Final[frozenset] = frozenset(GChunk.__annotations__)
_CompleteStreamingResponse = ModelResponse | TextCompletionResponse | ModelResponseStream | None
_USAGE_COST_HEADER_PROVIDERS: Final[frozenset[str]] = frozenset({LlmProviders.OPENROUTER.value})
@ -88,6 +98,30 @@ def _apply_provider_fields(response: ModelResponseStream, fields: Mapping[str, o
setattr(response, key, value)
def _prepend_upstream_choice_extras(
choice: StreamingChoices, upstream_choice: ChunkChoice, extra_keys: frozenset[str]
) -> None:
if not extra_keys:
return
upstream_extras: Final[Mapping[str, object]] = upstream_choice.model_dump(include=set(extra_keys))
choice.__pydantic_extra__ = {**upstream_extras, **(choice.__pydantic_extra__ or {})} # rebind-ok: reuse choice
choice.__pydantic_fields_set__.update(upstream_extras)
def _holds_only_scalars(model: BaseModel) -> bool:
fields: Final[Iterator[tuple[str, object]]] = iter(model)
return all(value is None or isinstance(value, (str, int, float, bool)) for _, value in fields)
def _delta_from_upstream(upstream_delta: BaseModel | None) -> Delta | None:
if upstream_delta is None:
return Delta()
try:
return Delta(**dict(iter(upstream_delta)))
except Exception:
return None
def _is_transport_timeout(error: BaseException) -> bool:
return isinstance(error, httpx.TimeoutException) or (
isinstance(error, Timeout) and isinstance(error.__cause__, httpx.TimeoutException)
@ -252,6 +286,7 @@ class CustomStreamWrapper:
self.completion_stream = completion_stream
self.sent_first_chunk = False
self.sent_last_chunk = False
self._openai_handler_receives_chat_chunks = False
self._failure_logging: asyncio.Task[None] | None = None
self._closed = False
self._stream_created_time: float = time.time()
@ -308,7 +343,8 @@ class CustomStreamWrapper:
self.sent_stream_usage = False
self.send_stream_usage = True if self.check_send_stream_usage(self.stream_options) else False
self.tool_call = False
self.chunks: list = [] # keep track of the returned chunks - used for calculating the input/output tokens for stream options
# keep track of the returned chunks - used for calculating the input/output tokens for stream options
self.chunks: list[ModelResponseStream] = []
self._repeated_messages_count = 1
self.is_function_call = self.check_is_function_call(logging_obj=logging_obj)
self.created: int | None = None
@ -517,7 +553,7 @@ class CustomStreamWrapper:
last_content: Final = self.chunks[-1].choices[0].delta.content
if (
last_content is None or not isinstance(last_content, str) or len(last_content) <= 2
not isinstance(last_content, str) or len(last_content) <= 2
): # ignore empty content - https://github.com/BerriAI/litellm/issues/5158#issuecomment-2287156946
self._repeated_messages_count = 1
return
@ -789,7 +825,10 @@ class CustomStreamWrapper:
raise e
def model_response_creator(
self, chunk: dict | None = None, hidden_params: Mapping[str, object] | None = None
self,
chunk: dict | None = None,
hidden_params: Mapping[str, object] | None = None,
delta: Delta | None = None,
) -> ModelResponseStream:
_model: Final = self._cached_model_name
_logging_obj_llm_provider: Final = self._cached_logging_llm_provider
@ -802,7 +841,11 @@ class CustomStreamWrapper:
if chunk:
args.update({k: v for k, v in chunk.items() if k != "stream"})
model_response: Final = ModelResponseStream(**args)
model_response: Final = (
ModelResponseStream(**args)
if delta is None
else ModelResponseStream(**args, choices=[StreamingChoices(delta=delta)])
)
if self.response_id is not None:
model_response.id = self.response_id
elif model_response.id:
@ -943,6 +986,39 @@ class CustomStreamWrapper:
model_response.choices[0].delta = Delta(**_initial_delta)
return model_response
def _prebuilt_plain_delta(self, chunk: object) -> Delta | None:
if (
not self._openai_handler_receives_chat_chunks
or not isinstance(chunk, ChatCompletionChunk)
or not chunk.choices
):
return None
upstream_delta: Final = cast( # cast-ok: [LIT006] runtime delta may be absent
ChoiceDelta | None,
chunk.choices[0].delta,
)
if (
upstream_delta is None
or upstream_delta.function_call is not None
or upstream_delta.tool_calls is not None
or self._has_any_special_delta_attributes(upstream_delta)
):
return None
return _delta_from_upstream(upstream_delta)
def _set_role_on_fresh_delta(self, model_response: ModelResponseStream) -> None:
delta: Final = model_response.choices[0].delta
if self.sent_first_chunk is False:
delta.role = "assistant"
self.sent_first_chunk = True
return
if not _holds_only_scalars(delta):
self.strip_role_from_delta(model_response)
return
delta.role = None
if "provider_specific_fields" not in delta.model_fields_set:
delta.provider_specific_fields = None
def _has_special_delta_content(self, model_response: ModelResponseStream) -> bool:
"""
Check if the delta contains special content types (tool_calls, function_call, audio, or image).
@ -975,11 +1051,11 @@ class CustomStreamWrapper:
"""
return self.strip_role_from_delta(model_response)
def _has_special_delta_attribute(self, delta, attribute_name: str) -> bool:
def _has_special_delta_attribute(self, delta: object, attribute_name: str) -> bool:
"""
Check if delta has a specific attribute and it's not None.
"""
return delta is not None and getattr(delta, attribute_name, None) is not None
return delta is not None and stored_attribute(delta, attribute_name) is not None
def _copy_delta_attribute(self, source_delta, target_delta, attribute_name: str) -> None:
"""
@ -987,7 +1063,7 @@ class CustomStreamWrapper:
"""
setattr(target_delta, attribute_name, getattr(source_delta, attribute_name))
def _has_any_special_delta_attributes(self, delta) -> bool:
def _has_any_special_delta_attributes(self, delta: object) -> bool:
"""
Check if delta has any special attributes (audio, image).
"""
@ -1011,6 +1087,7 @@ class CustomStreamWrapper:
completion_obj: dict[str, Any],
model_response: ModelResponseStream,
response_obj: dict[str, Any],
plain_delta_created: bool = False,
):
from litellm.litellm_core_utils.core_helpers import (
preserve_upstream_non_openai_attributes,
@ -1030,32 +1107,66 @@ class CustomStreamWrapper:
original_chunk: Final = response_obj.get("original_chunk", None)
if original_chunk:
if len(original_chunk.choices) > 0:
choices: Final = []
for choice in original_chunk.choices:
try:
if isinstance(choice, BaseModel):
choice_json = choice.model_dump()
choice_json.pop(
"finish_reason", None
) # for mistral etc. which return a value in their last chunk (not-openai compatible).
choices.append(StreamingChoices(**choice_json))
except Exception:
choices.append(StreamingChoices())
setattr(model_response, "choices", choices)
original_chunk_for_reuse: Final[object] = cast( # cast-ok: runtime type-check before access
object,
original_chunk,
)
upstream_choice: Final = (
original_chunk_for_reuse.choices[0]
if plain_delta_created
and isinstance(original_chunk_for_reuse, ChatCompletionChunk)
and len(original_chunk_for_reuse.choices) == 1
else None
)
upstream_delta: Final = cast( # cast-ok: [LIT006] runtime delta may be absent
ChoiceDelta | None,
getattr(upstream_choice, "delta", None),
)
upstream_choice_extra_keys: Final = frozenset(
(upstream_choice.__pydantic_extra__ or {}) if upstream_choice is not None else ()
)
can_reuse_choice: Final = (
upstream_choice is not None
and upstream_delta is not None
and "enhancements" not in upstream_choice_extra_keys
and upstream_choice.logprobs is None
and _holds_only_scalars(upstream_delta)
)
if can_reuse_choice and upstream_choice is not None:
model_response.choices[0].finish_reason = None # rebind-ok: keep reused choice
model_response.choices[0].index = upstream_choice.index # rebind-ok: keep reused choice
_prepend_upstream_choice_extras(
model_response.choices[0],
upstream_choice,
upstream_choice_extra_keys - {"message"},
)
else:
choices: Final = []
for choice in original_chunk.choices:
try:
if isinstance(choice, BaseModel):
choice_json = choice.model_dump()
choice_json.pop(
"finish_reason", None
) # for mistral etc. which return a value in their last chunk (not-openai compatible).
choices.append(StreamingChoices(**choice_json))
except Exception:
choices.append(StreamingChoices())
setattr(model_response, "choices", choices)
else:
return
model_response.system_fingerprint = original_chunk.system_fingerprint
setattr(
model_response,
"citations",
getattr(original_chunk, "citations", None),
stored_attribute(original_chunk, "citations"),
)
preserve_upstream_non_openai_attributes(
model_response=model_response,
original_chunk=original_chunk,
)
model_response = self.strip_role_from_delta(model_response)
self._set_role_on_fresh_delta(model_response)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
"model_response.choices[0].delta: %s",
@ -1404,6 +1515,8 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
else: # openai / azure chat model
if isinstance(chunk, ChatCompletionChunk):
self._openai_handler_receives_chat_chunks = True
if self.custom_llm_provider in [
LlmProviders.AZURE.value,
LlmProviders.AZURE_AI.value,
@ -1468,8 +1581,11 @@ class CustomStreamWrapper:
provider_response_model: Final = _provider_response_model(chunk)
if provider_response_model is not None:
self._provider_response_model = provider_response_model
prebuilt_delta: Final = self._prebuilt_plain_delta(chunk)
plain_delta_created: bool = prebuilt_delta is not None # rebind-ok: the first chunk builds its delta below
model_response = self.model_response_creator(
hidden_params=_provider_hidden_params(chunk, self._provider_response_model)
hidden_params=_provider_hidden_params(chunk, self._provider_response_model),
delta=prebuilt_delta,
)
response_obj: dict[str, Any] = {}
try:
@ -1493,7 +1609,7 @@ class CustomStreamWrapper:
# enter this branch when no content has been passed in response
if hasattr(original_chunk, "id"):
model_response = self.set_model_id(original_chunk.id, model_response)
if hasattr(original_chunk, "provider_specific_fields"):
if stored_attribute(original_chunk, "provider_specific_fields") is not None:
model_response = self.copy_model_response_level_provider_specific_fields(
original_chunk, model_response
)
@ -1545,16 +1661,10 @@ class CustomStreamWrapper:
model_response.choices[0].delta = Delta()
elif self._has_any_special_delta_attributes(delta):
self._handle_special_delta_attributes(delta, model_response)
else:
try:
delta = (
dict()
if original_chunk.choices[0].delta is None
else dict(original_chunk.choices[0].delta)
)
model_response.choices[0].delta = Delta(**delta)
except Exception:
model_response.choices[0].delta = Delta()
elif prebuilt_delta is None:
plain_delta: Final = _delta_from_upstream(delta)
model_response.choices[0].delta = plain_delta if plain_delta is not None else Delta()
plain_delta_created = plain_delta is not None # rebind-ok: the first chunk builds its delta
else:
if self.send_stream_usage is True:
model_response.choices = []
@ -1581,6 +1691,7 @@ class CustomStreamWrapper:
completion_obj=completion_obj,
model_response=model_response,
response_obj=response_obj,
plain_delta_created=plain_delta_created,
)
return result
@ -1825,6 +1936,51 @@ class CustomStreamWrapper:
response.hidden_params["additional_headers"] = {}
response.hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] = _cost
def _builder_failure_response(self, error: Exception) -> ModelResponseStream | None:
verbose_logger.warning(
"stream_chunk_builder raised at end-of-stream (%s); logging best-effort usage from chunks.",
str(error),
)
try:
return self.model_response_creator(chunk={"usage": calculate_total_usage(chunks=self.chunks)})
except Exception:
return None
def _build_complete_streaming_response(self) -> _CompleteStreamingResponse:
try:
return litellm.stream_chunk_builder(
chunks=self.chunks,
messages=self.messages,
logging_obj=self.logging_obj,
count_prompt_tokens=self.count_prompt_tokens,
)
except Exception as e:
return self._builder_failure_response(e)
async def _async_build_complete_streaming_response(self) -> _CompleteStreamingResponse:
try:
return await asyncify(litellm.stream_chunk_builder)(
chunks=self.chunks,
messages=self.messages,
logging_obj=self.logging_obj,
count_prompt_tokens=self.count_prompt_tokens,
)
except Exception as e:
return self._builder_failure_response(e)
def _set_final_chunk_usage(self, chunk: ModelResponseStream) -> None:
reported_usage: Final = _reported_total_usage(chunks=self.chunks)
set_hidden_param(
chunk,
"usage",
UnreportedUsage(prompt_tokens=0, completion_tokens=0, total_tokens=0)
if reported_usage is None
else reported_usage,
)
self._last_returned_hidden_params = cast( # cast-ok: preserve dynamic mapping behavior
dict[str, object], getattr(chunk, HIDDEN_PARAMS_ATTR)
)
def __next__(self) -> "ModelResponseStream":
if self._closed:
raise StopIteration
@ -1905,12 +2061,7 @@ class CustomStreamWrapper:
continue
# add usage as hidden param
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
response_hidden_params_for_final_chunk = cast( # cast-ok: preserve dynamic mapping behavior
dict[str, object], getattr(response, HIDDEN_PARAMS_ATTR)
)
response_hidden_params_for_final_chunk["usage"] = usage
self._last_returned_hidden_params = response_hidden_params_for_final_chunk
self._set_final_chunk_usage(response)
# Add MCP metadata to final chunk if present
response = self._add_mcp_metadata_to_final_chunk(response)
# RETURN RESULT
@ -1918,30 +2069,7 @@ class CustomStreamWrapper:
except StopIteration:
if self.sent_last_chunk is True:
try:
complete_streaming_response = litellm.stream_chunk_builder(
chunks=self.chunks,
messages=self.messages,
logging_obj=self.logging_obj,
count_prompt_tokens=self.count_prompt_tokens,
)
except Exception as e:
# stream_chunk_builder can re-raise (as APIError) on large agentic
# streams. The raise originates inside this except-StopIteration block,
# so the sibling `except Exception` below does not catch it; it would
# escape __next__ and drop the request from SpendLogs. Recover
# best-effort usage from the raw chunks so cost is still tracked
verbose_logger.warning(
"stream_chunk_builder raised at end-of-stream (%s); logging best-effort usage from chunks.",
str(e),
)
try:
complete_streaming_response = self.model_response_creator(
chunk={"usage": calculate_total_usage(chunks=self.chunks)}
)
except Exception:
complete_streaming_response = None
complete_streaming_response: Final = self._build_complete_streaming_response()
response = self.model_response_creator()
if complete_streaming_response is not None:
self._propagate_usage_cost_to_hidden_params(complete_streaming_response, self.custom_llm_provider)
@ -2005,9 +2133,8 @@ class CustomStreamWrapper:
# last content chunk (stripped there), but never add the synthetic "stop" used when it sent none.
if self.received_finish_reason is not None or self.intermittent_finish_reason is not None:
self.chunks.append(processed_chunk)
if self.stream_options is None: # add usage as hidden param
usage = calculate_total_usage(chunks=self.chunks)
processed_chunk.hidden_params["usage"] = usage
if self.stream_options is None:
self._set_final_chunk_usage(processed_chunk)
## LOGGING
executor.submit(
self.run_success_logging_and_cache_storage,
@ -2122,12 +2249,7 @@ class CustomStreamWrapper:
# add usage as hidden param
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
processed_chunk_hidden_params = cast( # cast-ok: preserve dynamic mapping behavior
dict[str, object], getattr(processed_chunk, HIDDEN_PARAMS_ATTR)
)
processed_chunk_hidden_params["usage"] = usage
self._last_returned_hidden_params = processed_chunk_hidden_params
self._set_final_chunk_usage(processed_chunk)
# Call post-call streaming deployment hook for final chunk
if self.sent_last_chunk is True:
@ -2183,29 +2305,7 @@ class CustomStreamWrapper:
async def _finalize_completed_stream(self, cache_hit: bool) -> "ModelResponseStream":
if self.sent_last_chunk is True:
# log the final chunk with accurate streaming values
try:
complete_streaming_response = await asyncify(litellm.stream_chunk_builder)(
chunks=self.chunks,
messages=self.messages,
logging_obj=self.logging_obj,
count_prompt_tokens=self.count_prompt_tokens,
)
except Exception as e:
# see sync __next__: a raise from stream_chunk_builder inside this
# except handler escapes __anext__ and drops the request from SpendLogs.
# Recover best-effort usage from the raw chunks so cost is still tracked
verbose_logger.warning(
"stream_chunk_builder raised at end-of-stream (%s); logging best-effort usage from chunks.",
str(e),
)
try:
complete_streaming_response = self.model_response_creator(
chunk={"usage": calculate_total_usage(chunks=self.chunks)}
)
except Exception:
complete_streaming_response = None
complete_streaming_response: Final = await self._async_build_complete_streaming_response()
response: Final = self.model_response_creator()
if complete_streaming_response is not None:
self._propagate_usage_cost_to_hidden_params(complete_streaming_response, self.custom_llm_provider)
@ -2282,11 +2382,7 @@ class CustomStreamWrapper:
if self.received_finish_reason is not None or self.intermittent_finish_reason is not None:
self.chunks.append(processed_chunk)
if self.stream_options is None:
usage: Final = calculate_total_usage(chunks=self.chunks)
processed_chunk_hidden_params: Final = cast( # cast-ok: preserve dynamic mapping behavior
dict[str, object], getattr(processed_chunk, HIDDEN_PARAMS_ATTR)
)
processed_chunk_hidden_params["usage"] = usage
self._set_final_chunk_usage(processed_chunk)
# see sync __next__'s sibling branch: deliberately do NOT restore
# here - this chunk is still this call's own data, and restoring
# before returning it would corrupt the caller's own log
@ -2557,7 +2653,13 @@ def _coerce_token_details(
return details_type(**(raw if isinstance(raw, dict) else raw.model_dump()))
def calculate_total_usage(chunks: list[ModelResponse]) -> Usage:
def _reported_total_usage(chunks: Sequence[ModelResponse | ModelResponseStream]) -> Usage | None:
if not any("usage" in chunk and chunk["usage"] is not None for chunk in chunks):
return None
return calculate_total_usage(chunks=chunks)
def calculate_total_usage(chunks: Sequence[ModelResponse | ModelResponseStream]) -> Usage:
"""Assume most recent usage chunk has total usage uptil then."""
from litellm.litellm_core_utils.streaming_chunk_builder_utils import (
attach_cache_creation_token_details,

1
litellm/litellm_core_utils/tokenizers/deepseek_tokenizer.json Normal file
View file

File diff suppressed because one or more lines are too long

21
litellm/litellm_core_utils/tokenizers/deepseek_tokenizer_LICENSE Normal file
View file

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2023 DeepSeek
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

18
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -30,6 +30,10 @@ from litellm.llms.anthropic.pass_through.adapters.transformation import (
LiteLLMAnthropicMessagesAdapter,
is_provider_native_tool_dict,
)
from litellm.llms.anthropic.pass_through.convertible_blocks import (
keep_non_string_tool_result_texts,
scannable_tool_result_request,
)
from litellm.llms.anthropic.pass_through.messages.utils import filtered_safeguard_results, redacted_stop_details
from litellm.llms.anthropic.pass_through.stream_assembly import (
build_complete_streaming_response,
@ -493,8 +497,8 @@ class AnthropicMessagesHandler(BaseTranslation):
``get_streaming_string_so_far``): raw SSE ``bytes`` -- which may bundle
several events separated by a blank line -- and an already-parsed event
``dict``."""
if isinstance(item, dict):
return (_as_str_mapping(item),)
if (event := as_str_mapping(item)) is not None:
return (event,)
if not isinstance(item, (bytes, bytearray)):
return ()
return tuple(
@ -510,7 +514,7 @@ class AnthropicMessagesHandler(BaseTranslation):
chat_completion_compatible_request,
_tool_name_mapping,
) = LiteLLMAnthropicMessagesAdapter().translate_anthropic_to_openai(
anthropic_message_request=cast(AnthropicMessagesRequest, data.copy()),
anthropic_message_request=cast(AnthropicMessagesRequest, scannable_tool_result_request(data)),
preserve_midturn_system=True,
)
return chat_completion_compatible_request
@ -832,7 +836,7 @@ class AnthropicMessagesHandler(BaseTranslation):
@staticmethod
def _write_back_structured_messages(
data: dict, # mutable-ok: API message payload
data: dict[str, object], # mutable-ok: API message payload
structured_messages: list, # mutable-ok: API message payload
hoisted_system_message: object = None,
preserve_system_messages: bool = False,
@ -899,7 +903,7 @@ class AnthropicMessagesHandler(BaseTranslation):
for block in content:
if isinstance(block, dict) and block.get("type") == "thinking":
block.pop("cache_control", None)
data["messages"] = converted
data["messages"] = keep_non_string_tool_result_texts(converted, data.get("messages"))
@staticmethod
def _extract_midturn_system_text(
@ -1601,8 +1605,8 @@ class AnthropicMessagesHandler(BaseTranslation):
@staticmethod
def _rewrite_buffered_item(item: object, rewrite_event: _SSEEventRewriter) -> object:
if isinstance(item, dict):
return _rewritten_event(_as_str_mapping(item), rewrite_event)
if (event := as_str_mapping(item)) is not None:
return _rewritten_event(event, rewrite_event)
if isinstance(item, (bytes, bytearray)):
return AnthropicMessagesHandler._rewrite_sse_events(bytes(item), rewrite_event)
return item

34
litellm/llms/anthropic/chat/handler.py
View file

@ -28,6 +28,7 @@ from litellm.llms.base_llm.base_utils import (
ResponseFormatSchema,
names_a_response_format_property,
partial_response_format_answer,
response_format_answer_json,
response_format_envelope_contents,
split_response_format_answer,
)
@ -104,17 +105,17 @@ def _key_after_a_string_verdict(string_value_start: str, schema: ResponseFormatS
next_key: Final = _STRING_VALUE_THEN_KEY.match(string_value_start)
if next_key is None:
return "undecided"
return "pass_through" if next_key.group(1) in schema.property_names else "hold"
return "pass_through" if next_key.group(1) in schema.declared_names else "hold"
def _is_a_label(text: str, schema: ResponseFormatSchema, tool_name: str) -> bool:
return text in schema.property_names or text == tool_name
return text in schema.declared_names or text == tool_name
def _label_verdict(label: str, next_key: str, schema: ResponseFormatSchema, tool_name: str) -> _EnvelopeVerdict:
if _is_a_label(label, schema, tool_name):
return "hold"
return "pass_through" if next_key in schema.property_names else "undecided"
return "pass_through" if next_key in schema.declared_names else "undecided"
def _leading_labels_verdict(string_value_start: str, schema: ResponseFormatSchema, tool_name: str) -> _EnvelopeVerdict:
@ -159,9 +160,9 @@ def _prefix_verdict(buffered_json: str, schema: ResponseFormatSchema, tool_name:
return "hold"
if first_value is None and _NAME_VALUE_PREFIX.match(buffered_json):
return "undecided"
if first_key.group(1) == "values" and "values" not in schema.property_names:
if first_key.group(1) == "values" and "values" not in schema.declared_names:
return "hold"
if first_key.group(1) in schema.property_names or not schema.property_names:
if first_key.group(1) in schema.declared_names or not schema.declared_names:
return "pass_through"
return _first_value_verdict(buffered_json[first_key.end() :], schema, tool_name)
@ -172,13 +173,13 @@ def _held_answer(held: str, schema: ResponseFormatSchema) -> str:
except (json.JSONDecodeError, RecursionError):
return held
contents: Final = response_format_envelope_contents(payload, schema, RESPONSE_FORMAT_TOOL_NAME)
return held if contents is None else json.dumps(contents)
return held if contents is None else response_format_answer_json(contents)
def _split_answer_text(blocks: tuple[str, ...], schema: ResponseFormatSchema) -> str:
answer: Final = split_response_format_answer(blocks, schema)
if answer is not None:
return json.dumps(answer)
return response_format_answer_json(answer)
return "".join(_held_answer(block, schema) for block in blocks)
@ -1283,7 +1284,7 @@ class ModelResponseIterator:
return None
self._streamed_answer_keys = streamed | frozenset(fields)
separator: Final = ", " if streamed else ""
return f"{separator}{json.dumps(fields)[1:-1]}" if fields else ""
return f"{separator}{response_format_answer_json(fields)[1:-1]}" if fields else ""
def _flush_response_format_buffer(self) -> str:
schema: Final = self.response_format_schema
@ -1314,6 +1315,17 @@ class ModelResponseIterator:
return ""
return _split_answer_text(blocks, self.response_format_schema)
def cut_stream_chunk(self) -> ModelResponseStream | None:
unsent: Final = self._flush_response_format_buffer() if self._response_format_buffer.getvalue() else ""
text: Final = unsent + self._release_answer_close() + self._drain_split_answer_blocks()
if not text:
return None
return ModelResponseStream(
choices=[StreamingChoices(index=0, delta=Delta(content=text))],
id=self.response_id,
model=self.served_model,
)
def _handle_message_delta(
self, chunk: dict
) -> tuple[str, Usage | None, dict[str, Any] | None, Mapping[str, object]]:
@ -1428,6 +1440,9 @@ class ModelResponseIterator:
result = self._handle_accumulated_json_chunk(data_str="", is_final=True)
if result is not None:
return result
cut_stream_chunk = self.cut_stream_chunk()
if cut_stream_chunk is not None:
return cut_stream_chunk
raise StopIteration
except ValueError as e:
raise RuntimeError(f"Error receiving chunk from stream: {e}")
@ -1474,6 +1489,9 @@ class ModelResponseIterator:
result = self._handle_accumulated_json_chunk(data_str="", is_final=True)
if result is not None:
return result
cut_stream_chunk = self.cut_stream_chunk()
if cut_stream_chunk is not None:
return cut_stream_chunk
raise StopAsyncIteration
except ValueError as e:
raise RuntimeError(f"Error receiving chunk from stream: {e}")

35
litellm/llms/anthropic/chat/transformation.py
View file

@ -23,7 +23,7 @@ from litellm.constants import (
DEFAULT_REASONING_EFFORT_XHIGH_THINKING_BUDGET,
RESPONSE_FORMAT_TOOL_NAME,
)
from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params, map_finish_reason
from litellm.litellm_core_utils.prompt_templates.common_utils import (
DEFS_MAX_INLINED_BYTES,
is_object_schema,
@ -50,6 +50,7 @@ from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import
from litellm.llms.base_llm.base_utils import (
ResponseFormatSchema,
as_json_object,
response_format_answer_json,
response_format_schema,
split_response_format_answer,
type_to_response_format_param,
@ -125,6 +126,7 @@ from ..common_utils import (
refusal_provider_specific_fields,
requires_native_compaction_beta,
strip_advisor_blocks_from_messages,
tools_marked_for_a_tool_config_point,
)
if TYPE_CHECKING:
@ -162,6 +164,7 @@ _ANTHROPIC_TOOL_NAME_MAX_LEN: Final = 128
_DEFER_LOADING_ADAPTER: Final = TypeAdapter(bool)
_TOOL_FIELDS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_TOOLS_ADAPTER: Final = TypeAdapter(list[object])
_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_NATIVE_OUTPUT_STRING_FORMATS: Final = frozenset(
("date-time", "time", "date", "duration", "email", "hostname", "uri", "ipv4", "ipv6", "uuid")
)
@ -2264,6 +2267,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
anthropic_messages_pt,
)
drop_params: Final = litellm.drop_params is True or litellm_params.get("drop_params") is True
if "tools" not in optional_params and messages is not None and has_tool_call_blocks(messages):
optional_params["tools"], _ = self._map_tools(add_dummy_tool(custom_llm_provider="anthropic"))
@ -2290,7 +2295,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
AnthropicConfig._maybe_drop_speed_param(
model=model,
optional_params=optional_params,
drop_params=litellm.drop_params or litellm_params.get("drop_params") is True,
drop_params=drop_params,
custom_llm_provider=self.custom_llm_provider,
)
@ -2348,6 +2353,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
model=model,
messages=list(conversation),
llm_provider=self._resolved_provider,
drop_params=drop_params,
)
except Exception as e:
raise AnthropicError(
@ -2408,6 +2414,19 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
# Remove internal LiteLLM parameters that should not be sent to Anthropic API
optional_params.pop("is_vertex_request", None)
optional_params.pop("client_metadata", None)
request_params: Final = cast( # cast-ok: optional_params is the untyped dict of the shared transform signature
dict[str, object], optional_params
)
marked_tools: Final = tools_marked_for_a_tool_config_point(
request_params.get("tools"), request_params.pop("cache_control_injection_points", None)
)
if marked_tools is not None:
from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheControlHook
request_litellm_params: Final = cast( # cast-ok: litellm_params is the untyped dict of the shared signature
Mapping[str, object], litellm_params
)
AnthropicCacheControlHook.record_gateway_injection(request_litellm_params, 1)
# ``top_k`` is a provider-specific kwarg that bypasses
# ``map_openai_params``; gate it here, the single boundary shared by
@ -2422,11 +2441,19 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
drop_params=litellm_params.get("drop_params") is True,
output_key="top_k",
)
output_format: Final = request_params.get("output_format")
if (
output_format is not None
and not isinstance(output_format, Mapping)
and drops_unsupported_params(_LITELLM_PARAMS_ADAPTER.validate_python(litellm_params).get("drop_params"))
):
request_params.pop("output_format")
data: Final = {
"model": model,
"messages": anthropic_messages,
**optional_params,
**({} if marked_tools is None else {"tools": list(marked_tools)}),
}
self._apply_output_config(data=data, model=model, optional_params=optional_params)
@ -3151,7 +3178,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
)
if split_answer is not None:
return litellm.Message(content=json.dumps(split_answer))
return litellm.Message(content=response_format_answer_json(split_answer))
try:
payload: Final = _DECODED_JSON.validate_python(json.loads(json_mode_content_str))
except (json.JSONDecodeError, RecursionError):
@ -3159,7 +3186,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
return litellm.Message(content=json_mode_content_str)
if response_format_schema is not None:
return litellm.Message(
content=json.dumps(
content=response_format_answer_json(
unwrap_response_format_envelope(payload, response_format_schema, RESPONSE_FORMAT_TOOL_NAME)
)
)

138
litellm/llms/anthropic/common_utils.py
View file

@ -34,6 +34,7 @@ from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import
message_field,
parts_of,
)
from litellm.litellm_core_utils.reasoning_effort_utils import non_numeric_thinking_budget_error
from litellm.llms.anthropic.wif import (
aget_anthropic_wif_token,
anthropic_base_without_chat_suffix,
@ -50,6 +51,7 @@ from litellm.types.llms.anthropic import (
ANTHROPIC_OAUTH_TOKEN_PREFIX,
ANTHROPIC_SERVER_TOOL_USE_ID_PREFIX,
ANTHROPIC_THINKING_DISPLAY_UPDATES_BETA_HEADER,
ANTHROPIC_TOOL_SEARCH_TOOL_TYPES,
AllAnthropicToolsValues,
AnthropicMcpServerTool,
AnthropicMessagesToolChoice,
@ -111,7 +113,7 @@ _DATED_RELEASE_SUFFIX_RE: Final = re.compile(r"-\d{8}$")
_DOTTED_VERSION_RE: Final = re.compile(r"(\d)\.(\d)")
_CLAUDE_CODE_BILLING_HEADER_PREFIX: Final = "x-anthropic-billing-header:"
OBJECT_MAPPING_ADAPTER: Final = TypeAdapter(dict[object, object])
_CLAUDE_CODE_OBJECT_LIST_ADAPTER: Final = TypeAdapter(list[object])
OBJECT_LIST_ADAPTER: Final = TypeAdapter(list[object])
_DECODED_JSON: Final = TypeAdapter(object)
@ -160,6 +162,40 @@ def supports_anthropic_cache_control(model: str, custom_llm_provider: str | None
)
def _tool_config_point_control(points: object) -> Mapping[object, object] | None:
point_fields: Final = tuple(validated_object_mapping(point) for point in validated_object_list(points) or ())
tool_config_point: Final = next(
(fields for fields in point_fields if fields is not None and fields.get("location") == "tool_config"), None
)
if tool_config_point is None:
return None
control: Final = tool_config_point.get("control")
return {"type": "ephemeral"} if control is None else validated_object_mapping(control)
def tools_marked_for_a_tool_config_point(tools: object, points: object) -> tuple[object, ...] | None:
"""Put the ``tool_config`` point's breakpoint on the last tool Anthropic caches.
Anthropic caches tools as one prefix ending at the marked tool, so the last one carries
the mark, skipping tool-search tools whose transform drops cache markers. A client that
already marked a tool keeps its own breakpoint. Returns ``None`` when nothing is marked.
"""
control: Final = _tool_config_point_control(points)
tool_list: Final = validated_object_list(tools) or ()
tool_fields: Final = tuple(validated_object_mapping(tool) for tool in tool_list)
if control is None or any(fields is not None and "cache_control" in fields for fields in tool_fields):
return None
cacheable: Final = tuple(
(position, fields)
for position, fields in enumerate(tool_fields)
if fields is not None and fields.get("type") not in ANTHROPIC_TOOL_SEARCH_TOOL_TYPES
)
if not cacheable:
return None
position, fields = cacheable[-1]
return (*tool_list[:position], {**fields, "cache_control": control}, *tool_list[position + 1 :])
def is_claude_code_user_agent(user_agent: str) -> bool:
"""Claude Code sends its API calls through the Anthropic SDK as `claude-cli/<version>` and its own
fetches, such as gateway model discovery, as `claude-code/<version>`"""
@ -173,9 +209,9 @@ def validated_object_mapping(value: object) -> dict[object, object] | None:
return None
def _validated_claude_code_list(value: object) -> list[object] | None:
def validated_object_list(value: object) -> Sequence[object] | None:
try:
return _CLAUDE_CODE_OBJECT_LIST_ADAPTER.validate_python(value)
return OBJECT_LIST_ADAPTER.validate_python(value)
except ValidationError:
return None
@ -202,7 +238,7 @@ def _claude_code_billing_fields(text: str) -> tuple[tuple[str, str], ...] | None
def _claude_code_billing_texts(system: object) -> tuple[str, ...] | None:
if isinstance(system, str):
return (system,)
blocks: Final = _validated_claude_code_list(system)
blocks: Final = validated_object_list(system)
if blocks is None:
return None
block_mappings: Final = tuple(validated_object_mapping(block) for block in blocks)
@ -462,6 +498,27 @@ def _fetch_anthropic_model_ids(
raise Exception(f"Anthropic /v1/models did not terminate within {_MODEL_LIST_PAGE_CAP} pages.")
def _numeric_budget_tokens(budget_tokens: object) -> int | None:
if isinstance(budget_tokens, bool):
return None
if isinstance(budget_tokens, (int, float)):
return int(budget_tokens)
if not isinstance(budget_tokens, str):
return None
try:
return int(budget_tokens)
except ValueError:
return None
def _legacy_budget_tokens(budget_tokens: object, *, drop_params: bool) -> int | None:
if (numeric := _numeric_budget_tokens(budget_tokens)) is not None:
return numeric
if drop_params or litellm.drop_params is True:
return None
raise non_numeric_thinking_budget_error(budget_tokens)
class AnthropicModelInfo(BaseLLMModelInfo):
_workload_identity_eligible: ClassVar[bool] = True
@ -655,7 +712,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
optional_params: dict,
model: str,
param: str,
value: Any,
value: object,
drop_params: bool,
output_key: str,
) -> None:
@ -942,19 +999,19 @@ class AnthropicModelInfo(BaseLLMModelInfo):
if thinking is None or thinking.get("type") != "enabled":
return
budget_tokens: Final = thinking.get("budget_tokens") or 0
effort: Final = AnthropicModelInfo._legacy_budget_to_effort(
model=model,
budget_tokens=int(budget_tokens) if isinstance(budget_tokens, (int, float, str)) else 0,
custom_llm_provider=custom_llm_provider,
)
existing_output_config: Final = optional_params.get("output_config")
budget_tokens: Final = _legacy_budget_tokens(thinking.get("budget_tokens", 0), drop_params=drop_params)
display: Final = thinking.get("display")
supported_display: Final = {"display": display} if display in ("summarized", "omitted") else {}
carried_keys: Final = carried_thinking_keys(
thinking, rebuilt_keys=frozenset(("type", "budget_tokens", "display")), drop_params=drop_params
)
optional_params["thinking"] = {**carried_keys, "type": "adaptive", **supported_display}
if budget_tokens is None:
return
effort: Final = AnthropicModelInfo._legacy_budget_to_effort(
model=model, budget_tokens=budget_tokens, custom_llm_provider=custom_llm_provider
)
existing_output_config: Final = optional_params.get("output_config")
optional_params["output_config"] = {
"effort": effort,
**(existing_output_config if isinstance(existing_output_config, dict) else MappingProxyType({})),
@ -1731,6 +1788,21 @@ def strip_thinking_blocks_from_anthropic_messages_request_dict(
data.pop("thinking", None)
_DROP_ANTHROPIC_MESSAGE: Final = object()
def _strip_empty_content_blocks_from_message(message: object) -> object:
if not isinstance(message, dict) or not isinstance(message.get("content"), list):
return message
content: Final = message["content"]
filtered: Final = [
block for block in content if not _is_empty_text_block(block) and not is_empty_unsigned_thinking_block(block)
]
if len(filtered) == len(content):
return message
return {**message, "content": filtered} if filtered else _DROP_ANTHROPIC_MESSAGE
def strip_empty_content_blocks_from_anthropic_messages(
messages: Sequence[object],
) -> list[Any]:
@ -1765,18 +1837,11 @@ def strip_empty_content_blocks_from_anthropic_messages(
The caller's list and its content blocks are never mutated; modified
messages are returned as shallow copies with a fresh content list.
"""
out: Final[list[Any]] = []
for m in messages:
if not isinstance(m, dict) or not isinstance(m.get("content"), list):
out.append(m)
continue
content = m["content"]
filtered = [b for b in content if not _is_empty_text_block(b) and not is_empty_unsigned_thinking_block(b)]
if len(filtered) == len(content):
out.append(m)
elif filtered:
out.append({**m, "content": filtered})
return out
return [
stripped_message
for message in messages
if (stripped_message := _strip_empty_content_blocks_from_message(message)) is not _DROP_ANTHROPIC_MESSAGE
]
def _is_empty_text_block(block: object) -> bool:
@ -1864,18 +1929,15 @@ def sanitize_tool_use_ids_in_anthropic_messages(messages: Sequence[object]) -> l
and ``:`` — valid on the upstream provider but rejected by Anthropic when
the session is switched to a native Anthropic deployment.
"""
out: Final[list[Any]] = []
for m in messages:
if not isinstance(m, dict) or not isinstance(m.get("content"), list):
out.append(m)
continue
content = m["content"]
new_content = [_sanitize_tool_use_id_content_block(b) for b in content]
if new_content == content:
out.append(m)
else:
out.append({**m, "content": new_content})
return out
return [_sanitize_tool_use_ids_in_message(message) for message in messages]
def _sanitize_tool_use_ids_in_message(message: object) -> object:
if not isinstance(message, dict) or not isinstance(message.get("content"), list):
return message
content: Final = message["content"]
new_content: Final = [_sanitize_tool_use_id_content_block(block) for block in content]
return message if new_content == content else {**message, "content": new_content}
class _ReplayedSearchQuery(LiteLLMBaseModel):
@ -1963,8 +2025,8 @@ def tool_call_is_rebuilt_as_server_tool_use(
return (
find_anthropic_server_tool_result(
tool_call_id,
_validated_claude_code_list(fields.get("web_search_results")),
_validated_claude_code_list(fields.get("tool_results")),
validated_object_list(fields.get("web_search_results")),
validated_object_list(fields.get("tool_results")),
)
is not None
)

34
litellm/llms/anthropic/count_tokens/token_counter.py
View file

@ -2,6 +2,8 @@
Anthropic Token Counter implementation using the CountTokens API.
"""
from collections.abc import Mapping
from types import MappingProxyType
from typing import Any, Final
from pydantic import JsonValue
@ -10,6 +12,11 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import AuthenticationError
from litellm.llms.anthropic.count_tokens.handler import AnthropicCountTokensHandler
from litellm.llms.anthropic.count_tokens.transformation import (
AnthropicCountFields,
ChatCountTranslationError,
anthropic_count_fields,
)
from litellm.llms.base_llm.base_utils import BaseTokenCounter
from litellm.types.utils import CountContextManagement, LlmProviders, TokenCountMessagesFormat, TokenCountResponse
@ -38,6 +45,8 @@ class AnthropicTokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "anthropic",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
count_options: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
"""
Count tokens using Anthropic's CountTokens API.
@ -57,6 +66,30 @@ class AnthropicTokenCounter(BaseTokenCounter):
if not messages:
return None
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
match translated := await anthropic_count_fields(
model_to_use, messages, system, tools, messages_format, AnthropicConfig(), chat_params
):
case ChatCountTranslationError():
return translated.response(request_model, model_to_use, "anthropic_api")
case AnthropicCountFields():
return await self.count_tokens(
model_to_use=model_to_use,
messages=[dict(message) for message in translated.messages],
contents=contents,
deployment=deployment,
request_model=request_model,
tools=[dict(tool) for tool in translated.tools] if translated.tools is not None else None,
system=translated.system,
messages_format="anthropic",
output_format=output_format,
context_management=context_management,
count_options=translated.options,
)
case None:
pass
deployment = deployment or {}
litellm_params: Final = deployment.get("litellm_params", {})
api_base: Final = litellm_params.get("api_base")
@ -89,6 +122,7 @@ class AnthropicTokenCounter(BaseTokenCounter):
api_base=api_base,
tools=tools,
system=system,
optional_params=count_options,
)
if result is not None:

225
litellm/llms/anthropic/count_tokens/transformation.py
View file

@ -4,25 +4,37 @@ Anthropic CountTokens API transformation logic.
This module handles the transformation of requests to Anthropic's CountTokens API format.
"""
import copy
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from itertools import chain
from types import MappingProxyType
from typing import Final
from typing import (
Final,
Protocol,
TypeAlias,
cast, # noqa: TID251 # caller chat messages go to chat inference translation as-is
)
from pydantic import JsonValue, TypeAdapter
from pydantic import ConfigDict, JsonValue, TypeAdapter, with_config
from typing_extensions import NotRequired, ReadOnly, TypedDict
from litellm.constants import ANTHROPIC_TOKEN_COUNTING_BETA_VERSION
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import (
anthropic_system_blocks,
split_leading_system_run,
)
from litellm.llms.anthropic.common_utils import merge_anthropic_beta_headers
from litellm.llms.anthropic.wif import resolve_anthropic_base
from litellm.types.llms.openai import ChatCompletionImageObject
from litellm.types.llms.openai import AllMessageValues, ChatCompletionImageObject, ValidUserMessageContentTypes
from litellm.types.utils import TokenCountMessagesFormat, TokenCountResponse
_COUNT_REQUEST: Final = TypeAdapter(dict[str, JsonValue])
_SYSTEM_BLOCKS: Final = TypeAdapter(list[JsonValue])
_IMAGE_BLOCK: Final = TypeAdapter(ChatCompletionImageObject)
COUNT_TOKEN_OPTION_NAMES: Final = ("thinking", "tool_choice", "output_config")
_PLAIN_MESSAGE_KEYS: Final = frozenset({"role", "content"})
def _count_image(block: JsonValue) -> JsonValue:
@ -74,6 +86,213 @@ def _lift_leading_system(
return conversation, lifted or system
@dataclass(frozen=True, slots=True)
class AnthropicCountFields:
messages: tuple[Mapping[str, JsonValue], ...]
system: JsonValue
tools: tuple[Mapping[str, JsonValue], ...] | None
options: Mapping[str, JsonValue]
@dataclass(frozen=True, slots=True)
class ChatCountTranslationError:
message: str
def response(self, request_model: str, model_used: str, tokenizer_type: str) -> TokenCountResponse:
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_used,
tokenizer_type=tokenizer_type,
error=True,
error_message=self.message,
status_code=400,
rejected_before_provider_call=True,
)
@with_config(ConfigDict(extra="ignore"))
class _AnthropicChatRequest(TypedDict):
messages: ReadOnly[tuple[Mapping[str, JsonValue], ...]]
system: NotRequired[ReadOnly[JsonValue]]
tools: NotRequired[ReadOnly[tuple[Mapping[str, JsonValue], ...]]]
thinking: NotRequired[ReadOnly[JsonValue]]
tool_choice: NotRequired[ReadOnly[JsonValue]]
output_config: NotRequired[ReadOnly[JsonValue]]
output_format: NotRequired[ReadOnly[JsonValue]]
_ANTHROPIC_CHAT_REQUEST: Final = TypeAdapter(_AnthropicChatRequest)
_ChatParams: TypeAlias = dict[str, object]
_ChatMessages: TypeAlias = list[AllMessageValues]
class _ChatRequestTranslator(Protocol):
def map_openai_params(
self, non_default_params: _ChatParams, optional_params: _ChatParams, model: str, drop_params: bool
) -> _ChatParams: ...
def transform_request(
self,
model: str,
messages: _ChatMessages,
optional_params: _ChatParams,
litellm_params: _ChatParams,
headers: _ChatParams,
) -> _ChatParams: ...
def _needs_chat_translation(message: Mapping[str, object]) -> bool:
return message.get("role") == "tool" or not _PLAIN_MESSAGE_KEYS.issuperset(message.keys())
def _is_chat_tool(tool: Mapping[str, object]) -> bool:
from litellm.litellm_core_utils.prompt_templates.factory import is_flat_responses_tool
return is_flat_responses_tool(tool) or (tool.get("type") == "function" and "function" in tool)
def _has_non_chat_user_block(message: Mapping[str, object]) -> bool:
content: Final = message.get("content")
if message.get("role") != "user" or not isinstance(content, list):
return False
blocks: Final = _SYSTEM_BLOCKS.validate_python(content)
return any(isinstance(block, dict) and block.get("type") not in ValidUserMessageContentTypes for block in blocks)
def _gets_placeholder_name(tool: Mapping[str, object]) -> bool:
from litellm.llms.anthropic.pass_through.adapters.transformation import is_provider_native_tool_dict
name: Final = tool.get("name")
return not (_is_chat_tool(tool) or is_provider_native_tool_dict(tool) or (isinstance(name, str) and name.strip()))
def _untranslatable_input(
messages: Sequence[Mapping[str, object]], tools: Sequence[Mapping[str, object]]
) -> ChatCountTranslationError | None:
invalid_index: Final = next(
(index for index, message in enumerate(messages) if _has_non_chat_user_block(message)), None
)
if invalid_index is not None:
return ChatCountTranslationError(
message=f"Invalid user message at index {invalid_index}. "
"Please ensure all user messages are valid OpenAI chat completion messages."
)
unnamed_tool: Final = next(filter(_gets_placeholder_name, tools), None)
if unnamed_tool is not None:
return ChatCountTranslationError(
message=f"Invalid request for token counting: tool type {unnamed_tool.get('type')!r} "
"has no Chat Completions equivalent"
)
return None
def _system_messages(system: object) -> tuple[Mapping[str, object], ...]:
message: Final[Mapping[str, object]] = {"role": "system", "content": system}
return (message,) if isinstance(system, (str, list)) and system else ()
def _count_options(request: _AnthropicChatRequest) -> Mapping[str, JsonValue]:
"""Structured output reaches chat inference as the beta ``output_format``, which count_tokens
refuses; its GA ``output_config.format`` form counts the same schema the same way."""
output_config: Final = request.get("output_config")
output_format: Final = request.get("output_format")
counted_config: Final = {
**(output_config if isinstance(output_config, dict) else {}),
**({} if output_format is None else {"format": output_format}),
}
return MappingProxyType(
{
key: value
for key, value in (
("thinking", request.get("thinking")),
("tool_choice", request.get("tool_choice")),
("output_config", counted_config or None),
)
if value is not None
}
)
def _translated_chat_request(
model: str,
messages: Sequence[Mapping[str, object]],
system: object,
tools: Sequence[Mapping[str, object]],
chat_params: Mapping[str, object],
translator: _ChatRequestTranslator,
) -> _ChatParams:
from litellm.litellm_core_utils.prompt_templates.factory import chat_completion_tools_for
chat_tools: Final = [dict(chat_tool) for chat_tool in chain.from_iterable(map(chat_completion_tools_for, tools))]
openai_params: Final = {name: value for name, value in chat_params.items() if name != "drop_params"}
mapped_params: Final = {**openai_params, **({"tools": chat_tools} if chat_tools else {})}
chat_messages: Final = cast( # cast-ok: chat-shaped request messages, read the way chat inference reads them
_ChatMessages, copy.deepcopy([*_system_messages(system), *messages])
)
return translator.transform_request(
model=model,
messages=chat_messages,
optional_params=translator.map_openai_params(
mapped_params, {}, model, drop_params=chat_params.get("drop_params") is True
)
if mapped_params
else {},
litellm_params={},
headers={},
)
def anthropic_fields_for_chat_tool_use(
model: str,
messages: Sequence[Mapping[str, object]],
system: object,
tools: Sequence[Mapping[str, object]] | None,
translator: _ChatRequestTranslator,
chat_params: Mapping[str, object] = MappingProxyType({}),
) -> AnthropicCountFields | ChatCountTranslationError | None:
counted_tools: Final = tools or ()
if (
not chat_params
and not any(map(_is_chat_tool, counted_tools))
and not any(map(_needs_chat_translation, messages))
):
return None
untranslatable: Final = _untranslatable_input(messages, counted_tools)
if untranslatable is not None:
return untranslatable
from litellm.exceptions import BadRequestError
from litellm.llms.anthropic.common_utils import AnthropicError
try:
request: Final = _translated_chat_request(model, messages, system, counted_tools, chat_params, translator)
except (AnthropicError, BadRequestError) as e:
return ChatCountTranslationError(message=str(e.message))
except (ValueError, KeyError) as e:
return ChatCountTranslationError(message=f"Invalid request for token counting: {e}")
translated: Final = _ANTHROPIC_CHAT_REQUEST.validate_python(request)
return AnthropicCountFields(
messages=translated["messages"],
system=translated.get("system"),
tools=translated.get("tools"),
options=_count_options(translated),
)
async def anthropic_count_fields(
model: str,
messages: Sequence[Mapping[str, object]],
system: object,
tools: Sequence[Mapping[str, object]] | None,
messages_format: TokenCountMessagesFormat,
translator: _ChatRequestTranslator,
chat_params: Mapping[str, object] = MappingProxyType({}),
) -> AnthropicCountFields | ChatCountTranslationError | None:
if messages_format != "openai":
return None
return await asyncify(anthropic_fields_for_chat_tool_use)(model, messages, system, tools, translator, chat_params)
class AnthropicCountTokensConfig:
"""
Configuration and transformation logic for Anthropic CountTokens API.

35
litellm/llms/anthropic/pass_through/adapters/handler.py
View file

@ -13,6 +13,7 @@ from typing_extensions import TypedDict
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.asyncify import run_async_function
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObject
from litellm.llms.anthropic.pass_through.adapters.transformation import (
AnthropicAdapter,
@ -22,6 +23,7 @@ from litellm.llms.anthropic.pass_through.context_management import (
PolyfillResult,
apply_context_management,
)
from litellm.llms.anthropic.pass_through.convertible_blocks import reject_malformed_tool_result_text
from litellm.llms.anthropic.pass_through.safeguards import (
SafeguardsEvaluator,
build_safeguards_evaluator,
@ -329,8 +331,10 @@ async def _run_polyfill_if_enabled(
does not disable a valid spec because context_management is a supported
param). Raises ``AnthropicContextManagementError`` so the /v1/messages
endpoint can emit an Anthropic-format 400, unless ``drop_invalid_edits``
(``drop_params``) drops the invalid spec. All other exceptions are
best-effort swallowed (matches v0 behavior).
(``drop_params``) drops what is invalid: the malformed fields of a
``clear_tool_uses`` edit alone, or the whole spec for an invalid ``compact``
trigger. All other
exceptions are best-effort swallowed (matches v0 behavior).
"""
if not context_management_spec:
return None
@ -351,6 +355,7 @@ async def _run_polyfill_if_enabled(
sub_call_logging_kwargs=sub_call_logging_kwargs,
parent_trace_id=parent_trace_id,
count_only=count_only,
drop_invalid_edits=drop_invalid_edits,
)
except AnthropicContextManagementError:
if drop_invalid_edits:
@ -548,16 +553,25 @@ class LiteLLMMessagesToCompletionTransformationHandler:
model: Final = completion_kwargs.get("model", "")
custom_llm_provider: Final = completion_kwargs.get("custom_llm_provider")
deployment_model_info: Final = completion_kwargs.get("model_info")
if isinstance(reasoning_effort, str):
normalized = normalize_reasoning_effort_value(
reasoning_effort, model=model, custom_llm_provider=custom_llm_provider
reasoning_effort,
model=model,
custom_llm_provider=custom_llm_provider,
deployment_model_info=deployment_model_info,
)
if normalized != reasoning_effort:
completion_kwargs["reasoning_effort"] = normalized
elif isinstance(reasoning_effort, dict) and "effort" in reasoning_effort:
effort: Final = reasoning_effort["effort"]
normalized = normalize_reasoning_effort_value(effort, model=model, custom_llm_provider=custom_llm_provider)
normalized = normalize_reasoning_effort_value(
effort,
model=model,
custom_llm_provider=custom_llm_provider,
deployment_model_info=deployment_model_info,
)
if normalized != effort:
completion_kwargs["reasoning_effort"] = {
**reasoning_effort,
@ -638,6 +652,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
) = ANTHROPIC_ADAPTER.translate_completion_input_params_with_tool_mapping(
request_data,
custom_llm_provider=custom_llm_provider if isinstance(custom_llm_provider, str) else None,
drop_params=drops_unsupported_params(extra_kwargs.get("drop_params")),
)
if openai_request is None:
@ -724,13 +739,12 @@ class LiteLLMMessagesToCompletionTransformationHandler:
"""Handle non-Anthropic models asynchronously using the adapter"""
context_management: Final = kwargs.pop("context_management", None)
additional_drop_params: Final[list[str] | None] = kwargs.get("additional_drop_params", None)
drop_invalid_edits: Final = litellm.drop_params is True or (
"drop_params" in kwargs and kwargs["drop_params"] is True
)
litellm_router: Final[Router | None] = _router_or_proxy_fallback(kwargs.pop("litellm_router", None))
proxy_litellm_metadata, user_api_key_auth = _extract_proxy_litellm_metadata(kwargs)
request_kwargs: Final = _REQUEST_KWARGS.validate_python(kwargs)
reject_malformed_tool_result_text(messages, model, request_kwargs)
drop_invalid_edits: Final = drops_unsupported_params(request_kwargs.get("drop_params"))
sub_call_logging_kwargs: Final = proxy_sub_call_logging_kwargs(request_kwargs)
sub_call_trace_id: Final = proxy_sub_call_trace_id(request_kwargs)
@ -853,6 +867,9 @@ class LiteLLMMessagesToCompletionTransformationHandler:
**kwargs,
)
sync_request_kwargs: Final = _REQUEST_KWARGS.validate_python(kwargs)
reject_malformed_tool_result_text(messages, model, sync_request_kwargs)
# Run the context_management polyfill on the sync path too so that
# ``litellm.messages.create()`` callers don't silently lose edits like
# ``clear_tool_uses_20250919``. The dispatcher is async (so the
@ -860,9 +877,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
# bridge to it via ``run_async_function``.
context_management: Final = kwargs.pop("context_management", None)
additional_drop_params: Final[list[str] | None] = kwargs.get("additional_drop_params", None)
drop_invalid_edits: Final = litellm.drop_params is True or (
"drop_params" in kwargs and kwargs["drop_params"] is True
)
drop_invalid_edits: Final = drops_unsupported_params(sync_request_kwargs.get("drop_params"))
# Deliberately do NOT auto-attach the proxy ``llm_router`` here:
# ``run_async_function`` spawns a new event loop in a worker thread
# to bridge to the async dispatcher, but the proxy router's httpx

26
litellm/llms/anthropic/pass_through/adapters/transformation.py
View file

@ -8,7 +8,9 @@ from typing import TYPE_CHECKING, Any, Final, Literal, TypeAlias, TypeVar, cast
from pydantic import JsonValue, TypeAdapter
import litellm
from litellm.litellm_core_utils.core_helpers import normalize_drop_params
from litellm.litellm_core_utils.hidden_params import get_hidden_params
from litellm.llms.anthropic.pass_through.convertible_blocks import tool_result_text_part
from litellm.llms.anthropic.pass_through.utils import (
is_reasoning_auto_summary_enabled,
json_copy_or_original,
@ -129,7 +131,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
THOUGHT_SIGNATURE_SEPARATOR,
)
from litellm.litellm_core_utils.reasoning_effort_utils import (
reasoning_effort_from_thinking_budget,
reasoning_effort_from_thinking,
)
from litellm.llms.anthropic.common_utils import (
eager_input_streaming_flag,
@ -233,7 +235,7 @@ class AnthropicAdapter:
def __init__(self) -> None:
pass
def translate_completion_input_params(self, kwargs) -> ChatCompletionRequest | None:
def translate_completion_input_params(self, kwargs: Mapping[str, object]) -> ChatCompletionRequest | None:
"""
Translate Anthropic request params to OpenAI format.
@ -243,11 +245,14 @@ class AnthropicAdapter:
Note: Use translate_completion_input_params_with_tool_mapping() if you need
the tool name mapping for restoring original names in responses.
"""
result, _ = self.translate_completion_input_params_with_tool_mapping(kwargs)
result, _ = self.translate_completion_input_params_with_tool_mapping(
kwargs,
drop_params=litellm.drop_params is True or normalize_drop_params(kwargs.get("drop_params")) is True,
)
return result
def translate_completion_input_params_with_tool_mapping(
self, kwargs, *, custom_llm_provider: str | None = None
self, kwargs, *, custom_llm_provider: str | None = None, drop_params: bool = True
) -> tuple[ChatCompletionRequest | None, dict[str, str]]:
"""
Translate Anthropic request params to OpenAI format, returning tool name mapping.
@ -281,6 +286,7 @@ class AnthropicAdapter:
) = LiteLLMAnthropicMessagesAdapter().translate_anthropic_to_openai(
anthropic_message_request=request_body,
custom_llm_provider=custom_llm_provider,
drop_params=drop_params,
)
return translated_body, tool_name_mapping
@ -647,6 +653,8 @@ class LiteLLMAnthropicMessagesAdapter:
@staticmethod
def translate_anthropic_thinking_to_reasoning_effort(
thinking: AnthropicThinkingParam,
*,
drop_params: bool = True,
) -> str | None:
"""
Translate Anthropic's thinking parameter to OpenAI's reasoning_effort.
@ -665,7 +673,7 @@ class LiteLLMAnthropicMessagesAdapter:
if thinking_type == "disabled":
return "none"
elif thinking_type == "enabled":
return reasoning_effort_from_thinking_budget(thinking.get("budget_tokens", 0))
return reasoning_effort_from_thinking(thinking, drop_params=drop_params)
elif thinking_type == "adaptive":
# Adaptive thinking: effort is controlled by output_config.effort,
# not budget_tokens. Return a default; caller should override with
@ -1112,6 +1120,7 @@ class LiteLLMAnthropicMessagesAdapter:
new_kwargs: ChatCompletionRequest,
*,
custom_llm_provider: str | None = None,
drop_params: bool = True,
) -> None:
"""Translate Anthropic thinking to either thinking or reasoning_effort.
@ -1169,7 +1178,7 @@ class LiteLLMAnthropicMessagesAdapter:
return
reasoning_effort: Final = declared_effort or self.translate_anthropic_thinking_to_reasoning_effort(
cast(AnthropicThinkingParam, thinking)
cast(AnthropicThinkingParam, thinking), drop_params=drop_params
)
if not reasoning_effort:
return
@ -1227,6 +1236,7 @@ class LiteLLMAnthropicMessagesAdapter:
*,
custom_llm_provider: str | None = None,
preserve_midturn_system: bool = False,
drop_params: bool = True,
) -> tuple[ChatCompletionRequest, dict[str, str]]:
"""
This is used by the beta Anthropic Adapter, for translating anthropic `/v1/messages` requests to the openai format.
@ -1279,6 +1289,7 @@ class LiteLLMAnthropicMessagesAdapter:
anthropic_message_request=anthropic_message_request,
new_kwargs=new_kwargs,
custom_llm_provider=custom_llm_provider,
drop_params=drop_params,
)
## CONVERT STOP_SEQUENCES
self._translate_stop_sequences_to_openai(
@ -1334,7 +1345,8 @@ class LiteLLMAnthropicMessagesAdapter:
block: Final = cast(Mapping[str, object], item) # cast-ok: untrusted client payload
match block.get("type"):
case "text":
return ChatCompletionTextObject(type="text", text=str(block.get("text") or ""))
text: Final = tool_result_text_part(block)
return None if text is None else ChatCompletionTextObject(type="text", text=text)
case "image" | "document":
return self._tool_result_image_part(block.get("source"))
case "tool_reference":

25
litellm/llms/anthropic/pass_through/context_management/dispatcher.py
View file

@ -9,7 +9,12 @@ from litellm.litellm_core_utils.asyncify import asyncify
from litellm.types.llms.anthropic import AppliedEdit
from .constants import CLEAR_TOOL_USES_EDIT_TYPE, COMPACT_EDIT_TYPE
from .editors import apply_clear_tool_uses_20250919, apply_compact_20260112
from .editors import (
CheckedEdit,
apply_clear_tool_uses_20250919,
apply_compact_20260112,
check_clear_tool_uses_spec,
)
from .result import PolyfillResult
if TYPE_CHECKING:
@ -73,6 +78,12 @@ def _wrap_editor_return(
)
def _checked_edit(index: int, edit_spec: dict[str, object]) -> CheckedEdit:
if edit_spec.get("type") != CLEAR_TOOL_USES_EDIT_TYPE:
return CheckedEdit(edit=edit_spec, error=None)
return check_clear_tool_uses_spec(edit_spec, index)
async def apply_context_management(
*,
model: str,
@ -86,9 +97,14 @@ async def apply_context_management(
sub_call_logging_kwargs: "ProxySubCallLoggingKwargs | None" = None,
parent_trace_id: str | None = None,
count_only: bool = False,
drop_invalid_edits: bool = False,
) -> PolyfillResult:
"""Run edits in order; return a single ``PolyfillResult``.
Every ``clear_tool_uses_20250919`` edit is validated before any editor runs, so a malformed one raises its 400
before a ``compact_20260112`` summary call is paid for, or runs without its malformed fields under
``drop_invalid_edits``.
The dispatcher is async so async editors (``compact_20260112``) can
``await`` the configured summarization model. Sync editors run in a
worker thread so their token counts stay off the event loop;
@ -97,6 +113,11 @@ async def apply_context_management(
edits: Final = normalize_spec(context_management_spec)
if not edits:
return PolyfillResult(messages=messages, system=system, applied_edits=[])
checked: Final = tuple(_checked_edit(index, edit_spec) for index, edit_spec in enumerate(edits))
first_error: Final = next((check.error for check in checked if check.error is not None), None)
if first_error is not None and not drop_invalid_edits:
raise first_error
well_formed_edits: Final = tuple(check.edit for check in checked)
current_messages = messages
current_system = system
@ -104,7 +125,7 @@ async def apply_context_management(
aggregated_compaction_block = None
aggregated_iterations_usage = None
for edit_spec in edits:
for edit_spec in well_formed_edits:
edit_type = edit_spec.get("type")
editor = _EDITOR_REGISTRY.get(edit_type) if isinstance(edit_type, str) else None
if editor is None:

4
litellm/llms/anthropic/pass_through/context_management/editors/__init__.py
View file

@ -1,4 +1,4 @@
from .clear_tool_uses import apply_clear_tool_uses_20250919
from .clear_tool_uses import CheckedEdit, apply_clear_tool_uses_20250919, check_clear_tool_uses_spec
from .compact import apply_compact_20260112
__all__ = ["apply_clear_tool_uses_20250919", "apply_compact_20260112"]
__all__ = ["CheckedEdit", "apply_clear_tool_uses_20250919", "apply_compact_20260112", "check_clear_tool_uses_spec"]

287
litellm/llms/anthropic/pass_through/context_management/editors/clear_tool_uses.py
View file

@ -1,7 +1,23 @@
"""``clear_tool_uses_20250919`` polyfill (v0: ``trigger`` and ``keep`` only)."""
"""``clear_tool_uses_20250919`` polyfill, matching the Anthropic API's selection.
from typing import Any, Final, cast
``keep`` counts clearable tool uses only. Uses of ``exclude_tools`` are never cleared and never count toward a
``tool_uses`` trigger, and neither are uses whose result is 1 to 46 bytes of text, which the API leaves out of its
count, unless ``clear_tool_inputs`` covers their tool: then their input can still be cleared, and the API counts and
clears them like any other. Clearing goes oldest first in batches that free at least ``clear_at_least`` tokens,
counting the result and any input it clears, and leaves an incomplete last batch, so the cleared prefix only grows
between requests and each batch costs one prompt-cache miss. ``check_clear_tool_uses_spec`` answers a malformed field
the way the API's 400 does, so the dispatcher can reject the edit, or drop only its malformed fields, before any editor
runs.
"""
from collections.abc import Mapping, Sequence, Set
from dataclasses import dataclass
from functools import reduce
from itertools import chain
from types import MappingProxyType
from typing import Annotated, Any, Final, Literal, TypeVar, cast
from pydantic import BaseModel, ConfigDict, Field, StrictInt, TypeAdapter, ValidationError
from typing_extensions import ReadOnly, TypedDict
import litellm
@ -13,8 +29,39 @@ from ..constants import (
DEFAULT_INPUT_TOKENS_TRIGGER,
DEFAULT_KEEP_TOOL_USES,
)
from ..errors import AnthropicContextManagementError
from ..placeholders import build_cleared_tool_result_content
T = TypeVar("T")
# api.anthropic.com, measured 2026-10-19: text results of 1 to 46 bytes are never cleared or counted toward keep
_TINY_RESULT_BYTES: Final = 46
ToolNames = Literal["all"] | frozenset[str]
class _ClearAtLeast(BaseModel):
model_config = ConfigDict(frozen=True)
type: Literal["input_tokens"]
value: Annotated[StrictInt, Field(ge=0)]
class _ClearToolUsesFields(BaseModel):
model_config = ConfigDict(frozen=True)
clear_at_least: _ClearAtLeast | None = None
exclude_tools: tuple[str, ...] | None = None
clear_tool_inputs: bool | tuple[str, ...] | None = None
_BLOCKS: Final = TypeAdapter(tuple[Mapping[str, object], ...])
# api.anthropic.com, measured 2026-10-19: its wording for the two shapes pydantic words differently
_API_MESSAGES: Final = MappingProxyType(
{
"tuple_type": "Input should be a valid array",
"model_type": "Input does not match the expected shape.",
}
)
class ClearToolUsesEditSpec(TypedDict, total=False):
"""The ``clear_tool_uses_20250919`` entry of a ``context_management`` spec."""
@ -27,36 +74,125 @@ class ClearToolUsesEditSpec(TypedDict, total=False):
clear_tool_inputs: ReadOnly[object]
def _count_tool_uses(messages: list[dict[str, object]]) -> int:
"""Return the number of tool_use content blocks across all messages.
Only counts blocks with a string ``id`` to stay consistent with
:func:`_collect_tool_use_ids_in_order`, which is the source of truth for
which blocks are clearable.
"""
count = 0
for msg in messages:
content = msg.get("content")
if isinstance(content, list):
for block in content:
if isinstance(block, dict) and block.get("type") == "tool_use":
if isinstance(block.get("id"), str):
count += 1
return count
def _parse(adapter: TypeAdapter[T], value: object) -> T | None:
try:
return adapter.validate_python(value)
except ValidationError:
return None
def _collect_tool_use_ids_in_order(messages: list[dict[str, object]]) -> list[str]:
"""Return tool_use ids in the chronological order they appear in messages."""
ids: Final[list[str]] = []
for msg in messages:
content = msg.get("content")
if isinstance(content, list):
for block in content:
if isinstance(block, dict) and block.get("type") == "tool_use":
block_id = block.get("id")
if isinstance(block_id, str):
ids.append(block_id)
return ids
@dataclass(frozen=True, slots=True)
class CheckedEdit:
edit: dict[str, object]
error: AnthropicContextManagementError | None
def check_clear_tool_uses_spec(edit_spec: dict[str, object], index: int) -> CheckedEdit:
"""``edits[index]`` without its malformed fields, with the API's 400 for the first one, or None when there is none."""
try:
_ClearToolUsesFields.model_validate(edit_spec)
except ValidationError as error:
details: Final = error.errors(include_url=False)
malformed: Final = frozenset(detail["loc"][0] for detail in details)
first: Final = details[0]
location: Final = ".".join(map(str, ("edits", index, CLEAR_TOOL_USES_EDIT_TYPE, *first["loc"])))
return CheckedEdit(
edit={key: value for key, value in edit_spec.items() if key not in malformed},
error=AnthropicContextManagementError(
status_code=400,
message=f"context_management.{location}: {_API_MESSAGES.get(first['type'], first['msg'])}",
),
)
return CheckedEdit(edit=edit_spec, error=None)
def _input_tools(clear_tool_inputs: bool | tuple[str, ...] | None) -> ToolNames:
return "all" if clear_tool_inputs is True else frozenset(clear_tool_inputs or ())
def _all_blocks(messages: Sequence[Mapping[str, object]]) -> tuple[Mapping[str, object], ...]:
return tuple(chain.from_iterable(_parse(_BLOCKS, message.get("content")) or () for message in messages))
def _tool_use_ref(block: Mapping[str, object]) -> tuple[str, str | None] | None:
block_id: Final = block.get("id")
name: Final = block.get("name")
if block.get("type") != "tool_use" or not isinstance(block_id, str):
return None
return block_id, name if isinstance(name, str) else None
def _tool_uses_in_order(blocks: Sequence[Mapping[str, object]]) -> tuple[tuple[str, str | None], ...]:
refs: Final = (_tool_use_ref(block) for block in blocks)
return tuple(ref for ref in refs if ref is not None)
def _block_id(block: Mapping[str, object], block_type: str, id_key: str) -> str | None:
block_id: Final = block.get(id_key)
return block_id if block.get("type") == block_type and isinstance(block_id, str) else None
def _blocks_by_id(
blocks: Sequence[Mapping[str, object]], block_type: str, id_key: str
) -> Mapping[str, Mapping[str, object]]:
pairs: Final = ((_block_id(block, block_type, id_key), block) for block in blocks)
return {block_id: block for block_id, block in pairs if block_id is not None}
def _utf8_size(text: str) -> int:
return len(text.encode("utf-8", "surrogatepass"))
def _text_bytes(content: object) -> int | None:
"""Return the UTF-8 size of a text-only result, or None when it holds anything but text."""
if isinstance(content, str):
return _utf8_size(content)
parts: Final = _parse(_BLOCKS, content) or ()
texts: Final = tuple(part.get("text") for part in parts)
if any(part.get("type") != "text" or not isinstance(text, str) for part, text in zip(parts, texts)):
return None
return sum(_utf8_size(text) for text in texts if isinstance(text, str))
def _is_tiny(content: object) -> bool:
size: Final = _text_bytes(content)
return size is not None and 0 < size <= _TINY_RESULT_BYTES
def _tiny_result_ids(results: Mapping[str, Mapping[str, object]]) -> frozenset[str]:
"""Return ids whose result is 1 to 46 bytes of text; the API neither counts nor clears those uses."""
return frozenset(tool_use_id for tool_use_id, block in results.items() if _is_tiny(block.get("content")))
def _clears_input(name: str | None, input_tools: ToolNames) -> bool:
return input_tools == "all" or (name is not None and name in input_tools)
def _cleared_result(block: Mapping[str, object]) -> dict[str, object]:
return {**block, "content": build_cleared_tool_result_content(block.get("content"))}
def _freed_tokens(model: str, result: Mapping[str, object], use: Mapping[str, object], clears_input: bool) -> int:
"""Tokens that clearing this use frees: its result, and its input when ``clear_tool_inputs`` covers it."""
before: Final = [result, use] if clears_input else [result]
after: Final = [_cleared_result(result), {**use, "input": {}}] if clears_input else [_cleared_result(result)]
counts: Final = tuple(
litellm.token_counter(model=model, messages=[{"role": "user", "content": blocks}]) for blocks in (before, after)
)
return max(counts[0] - counts[1], 0)
def _select_in_batches(candidates: Sequence[str], tokens: Mapping[str, int], clear_at_least: int) -> tuple[str, ...]:
"""Oldest first, close a batch once it holds ``clear_at_least`` tokens; drop the incomplete last batch."""
def step(state: tuple[int, int], item: tuple[int, str]) -> tuple[int, int]:
batch_end, batch_tokens = state
index, tool_use_id = item
total: Final = batch_tokens + tokens[tool_use_id]
return (index + 1, 0) if total >= clear_at_least else (batch_end, total)
batch_end, _ = reduce(step, enumerate(candidates), (0, 0))
return tuple(candidates[:batch_end])
def _trigger_met(
@ -64,6 +200,7 @@ def _trigger_met(
model: str,
messages: list[dict[str, object]],
tools: list[dict[str, object]] | None,
counted_tool_uses: int,
) -> tuple[bool, int | None]:
"""Return (trigger_met, input_tokens if counted for reuse)."""
trigger_type: Final = trigger.get("type", "input_tokens")
@ -72,7 +209,7 @@ def _trigger_met(
if trigger_type == "tool_uses":
if not isinstance(threshold, int):
return False, None
return _count_tool_uses(messages) > threshold, None
return counted_tool_uses > threshold, None
if not isinstance(threshold, int):
threshold = DEFAULT_INPUT_TOKENS_TRIGGER
@ -132,11 +269,7 @@ def _clear_tool_results(
and block.get("type") == "tool_result"
and block.get("tool_use_id") in ids_to_clear
):
new_block = {
**block,
"content": build_cleared_tool_result_content(block.get("content")),
}
new_blocks.append(new_block)
new_blocks.append(_cleared_result(block))
mutated = True
cleared += 1
else:
@ -150,6 +283,23 @@ def _clear_tool_results(
return new_messages, cleared
def _without_input(block: Mapping[str, object], ids_to_clear: Set[str], tool_names: ToolNames) -> Mapping[str, object]:
match _tool_use_ref(block):
case (block_id, name) if block_id in ids_to_clear and _clears_input(name, tool_names):
return {**block, "input": dict[str, object]()}
case _:
return block
def _message_without_inputs(
message: dict[str, object], ids_to_clear: Set[str], tool_names: ToolNames
) -> dict[str, object]: # mutable-ok: the editor contract hands messages back as list[dict]
blocks: Final = _parse(_BLOCKS, message.get("content"))
if blocks is None:
return message
return {**message, "content": [_without_input(block, ids_to_clear, tool_names) for block in blocks]}
def apply_clear_tool_uses_20250919(
*,
model: str,
@ -158,15 +308,11 @@ def apply_clear_tool_uses_20250919(
system: str | list[dict[str, object]] | None,
edit_spec: ClearToolUsesEditSpec,
) -> tuple[list[dict[str, object]], AppliedEdit | None]:
"""Apply clear_tool_uses; return (messages, AppliedEdit or None)."""
ignored_knobs = [knob for knob in ("clear_at_least", "exclude_tools", "clear_tool_inputs") if knob in edit_spec]
for ignored_knob in ignored_knobs:
verbose_logger.warning(
"context_management polyfill: ignoring '%s' on %s "
"(supported only on Anthropic-family forwarding path in v0)",
ignored_knob,
CLEAR_TOOL_USES_EDIT_TYPE,
)
"""Apply clear_tool_uses to an edit ``check_clear_tool_uses_spec`` returned; return (messages, AppliedEdit or None)."""
fields: Final = _ClearToolUsesFields.model_validate(edit_spec)
clear_at_least: Final = 0 if fields.clear_at_least is None else fields.clear_at_least.value
exclude_tools: Final = frozenset(fields.exclude_tools or ())
input_tools: Final = _input_tools(fields.clear_tool_inputs)
trigger: Final[dict[str, object]] = edit_spec.get("trigger") or {
"type": "input_tokens",
@ -177,37 +323,56 @@ def apply_clear_tool_uses_20250919(
"value": DEFAULT_KEEP_TOOL_USES,
}
met, tokens_before = _trigger_met(trigger, model, messages, tools)
blocks: Final = _all_blocks(messages)
results: Final = _blocks_by_id(blocks, "tool_result", "tool_use_id")
uses: Final = _blocks_by_id(blocks, "tool_use", "id")
tiny: Final = _tiny_result_ids(results)
clearable: Final = tuple(
(tool_use_id, name)
for tool_use_id, name in _tool_uses_in_order(blocks)
if name not in exclude_tools and (tool_use_id not in tiny or _clears_input(name, input_tools))
)
met, counted_tokens = _trigger_met(trigger, model, messages, tools, len(clearable))
if not met:
return messages, None
keep_count: Final = _resolve_keep_count(keep)
tool_use_ids: Final = _collect_tool_use_ids_in_order(messages)
if len(tool_use_ids) <= keep_count:
if len(clearable) <= keep_count:
return messages, None
ids_to_clear: Final = set(tool_use_ids[: len(tool_use_ids) - keep_count])
# Never clear the latest completed tool_result (reply context).
last_completed_id: Final = _last_completed_tool_use_id(messages)
if last_completed_id is not None:
ids_to_clear.discard(last_completed_id)
candidates: Final = tuple(
(tool_use_id, name)
for tool_use_id, name in clearable[: len(clearable) - keep_count]
if tool_use_id in results and tool_use_id != last_completed_id
)
candidate_ids: Final = tuple(tool_use_id for tool_use_id, _ in candidates)
tokens: Final = {
tool_use_id: _freed_tokens(model, results[tool_use_id], uses[tool_use_id], _clears_input(name, input_tools))
for tool_use_id, name in (candidates if clear_at_least > 0 else ())
}
ids_to_clear: Final = frozenset(
_select_in_batches(candidate_ids, tokens, clear_at_least) if clear_at_least > 0 else candidate_ids
)
edited, cleared_count = _clear_tool_results(messages, ids_to_clear)
edited, cleared_count = _clear_tool_results(messages, set(ids_to_clear))
verbose_logger.debug("context_management polyfill: edited: %s", edited)
if cleared_count == 0:
return messages, None
final_messages: Final = (
[_message_without_inputs(message, ids_to_clear, input_tools) for message in edited] if input_tools else edited
)
if tokens_before is None:
tokens_before = litellm.token_counter(model=model, messages=messages, tools=cast(Any, tools))
tokens_after: Final = litellm.token_counter(model=model, messages=edited, tools=cast(Any, tools))
tokens_before: Final = (
counted_tokens
if counted_tokens is not None
else litellm.token_counter(model=model, messages=messages, tools=cast(Any, tools))
)
tokens_after: Final = litellm.token_counter(model=model, messages=final_messages, tools=cast(Any, tools))
cleared_input_tokens: Final = max(tokens_before - tokens_after, 0)
applied: Final[AppliedEdit] = {
return final_messages, {
"type": CLEAR_TOOL_USES_EDIT_TYPE,
"cleared_tool_uses": cleared_count,
"cleared_input_tokens": cleared_input_tokens,
}
if ignored_knobs:
applied["warnings"] = [f"{knob}_ignored" for knob in ignored_knobs]
return edited, applied

255
litellm/llms/anthropic/pass_through/convertible_blocks.py
View file

@ -1,11 +1,18 @@
from collections.abc import Iterator, Mapping, Sequence
from itertools import chain
import json
from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from functools import partial
from itertools import chain, groupby
from operator import itemgetter
from types import MappingProxyType
from typing import Final
from typing import Final, TypeAlias
from pydantic import JsonValue, TypeAdapter, ValidationError
from typing_extensions import assert_never
import litellm
from litellm.litellm_core_utils.core_helpers import as_str_mapping, drops_unsupported_params
from litellm.litellm_core_utils.prompt_templates.factory import sanitize_anthropic_tool_use_id
from litellm.llms.anthropic.pass_through.messages.utils import normalize_native_messages
from litellm.types.utils import TokenCountMessagesFormat
@ -104,3 +111,245 @@ def unconvertible_blocks_error(
)
case _:
assert_never(messages_format)
_OBJECTS: Final = TypeAdapter(tuple[object, ...])
def _list_items(value: object) -> tuple[object, ...]:
return _OBJECTS.validate_python(value) if isinstance(value, list) else ()
@dataclass(frozen=True, slots=True)
class _KeptPart:
block: object
scanned_text: str
_KeptParts: TypeAlias = Mapping[int, _KeptPart]
_NO_KEPT_PARTS: Final[_KeptParts] = MappingProxyType({})
def _scannable_text(text: object) -> str:
return json.dumps(text, default=str, ensure_ascii=False)
def _non_str_text(block: object) -> _KeptPart | None:
match as_str_mapping(block):
case {"type": "text", "text": str()}:
return None
case {"type": "text", "text": text}:
return _KeptPart(block=block, scanned_text=_scannable_text(text))
case _:
return None
def tool_result_text_part(block: object) -> str | None:
match block:
case {"type": "text", "text": str(text)}:
return text
case {"type": "text", "text": object()}:
return None
case {"type": "text"}:
return ""
case _:
return None
def tool_result_text_parts(tool_result: Mapping[str, object]) -> tuple[str, ...]:
parts: Final = map(tool_result_text_part, _list_items(tool_result.get("content")))
return tuple(part for part in parts if part is not None)
def _user_content_blocks(message: object) -> tuple[object, ...]:
match as_str_mapping(message):
case {"role": "user", "content": content}:
return _list_items(content)
case _:
return ()
def _with_user_blocks(message: object, transform: Callable[[object], object]) -> object:
match as_str_mapping(message):
case {"role": "user", "content": list()} as user_message:
return {**user_message, "content": [transform(block) for block in _user_content_blocks(user_message)]}
case _:
return message
def _non_string_text_errors(label: str, content: object) -> Iterator[str]:
return (
f"{label} content.{index}.text: Input should be a valid string"
for index, text_block in enumerate(_list_items(content))
if _non_str_text(text_block) is not None
)
def _tool_result_text_errors(block: object) -> Iterator[str]:
match block:
case {"type": "tool_result", "tool_use_id": object() as tool_use_id, "content": object() as content}:
return _non_string_text_errors(f"tool_result {tool_use_id}", content)
case {"type": "tool_result", "content": object() as content}:
return _non_string_text_errors("tool_result", content)
case _:
return iter(())
def malformed_tool_result_text_error(messages: Iterable[object]) -> str | None:
blocks: Final = chain.from_iterable(map(_user_content_blocks, messages))
return next(chain.from_iterable(map(_tool_result_text_errors, blocks)), None)
def malformed_tool_result_count_error(
messages: Iterable[Mapping[str, object]] | None,
messages_format: TokenCountMessagesFormat,
deployment: Mapping[str, object] | None,
) -> str | None:
if messages_format != "anthropic" or deployment_drops_params(deployment):
return None
return malformed_tool_result_text_error(messages or ())
def _scannable_part(part: object) -> object:
match as_str_mapping(part):
case {"type": "text", "text": str()}:
return part
case {"type": "text", "text": text} as text_block:
return {**text_block, "text": _scannable_text(text)}
case _:
return part
def _scannable_block(block: object) -> object:
match as_str_mapping(block):
case {"type": "tool_result", "content": list()} as tool_result:
return {
**tool_result,
"content": [_scannable_part(part) for part in _list_items(tool_result.get("content"))],
}
case _:
return block
def _scannable_messages(messages: object) -> object:
items: Final = _list_items(messages)
return [_with_user_blocks(message, _scannable_block) for message in items] if items else messages
def scannable_tool_result_request(request: object) -> object:
match as_str_mapping(request):
case {"messages": messages} as request_fields:
return {**request_fields, "messages": _scannable_messages(messages)}
case None:
return request
case request_fields:
return {**request_fields}
def _non_string_text_parts(content: object) -> _KeptParts:
return MappingProxyType(
{index: part for index, block in enumerate(_list_items(content)) if (part := _non_str_text(block)) is not None}
)
def _tool_result_non_string_texts(block: object) -> tuple[tuple[str, _KeptParts], ...]:
match as_str_mapping(block):
case {"type": "tool_result", "tool_use_id": str(tool_use_id)} as tool_result:
return ((sanitize_anthropic_tool_use_id(tool_use_id), _non_string_text_parts(tool_result.get("content"))),)
case _:
return ()
def _content_blocks(content: object) -> tuple[object, ...]:
match content:
case str() if content:
return ({"type": "text", "text": content},)
case _:
return _list_items(content)
def _restored_part(block: object, kept: _KeptPart | None) -> object:
return kept.block if kept is not None and tool_result_text_part(block) == kept.scanned_text else block
def _with_kept_parts(block: object, tool_result: Mapping[str, object], parts: _KeptParts) -> object:
if not parts:
return block
blocks: Final = _content_blocks(tool_result.get("content"))
return {**tool_result, "content": [_restored_part(part, parts.get(index)) for index, part in enumerate(blocks)]}
def _restored_block(block: object, kept: Mapping[str, Iterator[_KeptParts]]) -> object:
match as_str_mapping(block):
case {"type": "tool_result", "tool_use_id": str(tool_use_id)} as tool_result if (
sanitize_anthropic_tool_use_id(tool_use_id) in kept
):
return _with_kept_parts(
block, tool_result, next(kept[sanitize_anthropic_tool_use_id(tool_use_id)], _NO_KEPT_PARTS)
)
case _:
return block
def keep_non_string_tool_result_texts(rewritten: Sequence[object], original: object) -> Sequence[object]:
blocks: Final = chain.from_iterable(map(_user_content_blocks, _list_items(original)))
by_id: Final = tuple(sorted(chain.from_iterable(map(_tool_result_non_string_texts, blocks)), key=itemgetter(0)))
if not any(parts for _, parts in by_id):
return rewritten
kept_in_order: Final = MappingProxyType(
{
tool_use_id: iter(tuple(map(itemgetter(1), group)))
for tool_use_id, group in groupby(by_id, key=itemgetter(0))
}
)
return [_with_user_blocks(message, partial(_restored_block, kept=kept_in_order)) for message in rewritten]
def _without_non_string_text(block: object) -> object:
match as_str_mapping(block):
case {"type": "tool_result", "content": list()} as tool_result:
parts: Final = _list_items(tool_result.get("content"))
return {**tool_result, "content": [part for part in parts if _non_str_text(part) is None]}
case _:
return block
def _without_non_string_texts(
message: dict[str, object], # mutable-ok: AnthropicMessagesRequest.messages is typed list[dict]
) -> dict[str, object]: # mutable-ok: AnthropicMessagesRequest.messages is typed list[dict]
match message:
case {"role": "user", "content": list()}:
return {**message, "content": [_without_non_string_text(block) for block in _user_content_blocks(message)]}
case _:
return message
def drop_non_string_tool_result_texts(
messages: Sequence[dict[str, object]], # mutable-ok: AnthropicMessagesRequest.messages is typed list[dict]
drop_params: object,
) -> Sequence[dict[str, object]]: # mutable-ok: AnthropicMessagesRequest.messages is typed list[dict]
if not drops_unsupported_params(drop_params) or malformed_tool_result_text_error(messages) is None:
return messages
return [_without_non_string_texts(message) for message in messages]
def deployment_drops_params(deployment: Mapping[str, object] | None) -> bool:
match deployment:
case {"litellm_params": {"drop_params": object() as drop_params}}:
return drops_unsupported_params(drop_params)
case _:
return drops_unsupported_params(None)
def reject_malformed_tool_result_text(
messages: Iterable[Mapping[str, object]], model: str, forwarded_kwargs: Mapping[str, object]
) -> None:
if drops_unsupported_params(forwarded_kwargs.get("drop_params")):
return
error: Final = malformed_tool_result_text_error(messages)
if error is None:
return
provider: Final = forwarded_kwargs.get("custom_llm_provider")
raise litellm.BadRequestError(
message=error, model=model, llm_provider=provider if isinstance(provider, str) else ""
)

124
litellm/llms/anthropic/pass_through/messages/handler.py
View file

@ -15,10 +15,13 @@ from pydantic import TypeAdapter
import litellm
from litellm.anthropic_beta_headers_manager import aload_beta_headers_config
from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheControlHook
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.exception_mapping_utils import exception_type
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.anthropic.common_utils import (
strip_provider_specific_fields_from_anthropic_messages,
tools_marked_for_a_tool_config_point,
)
from litellm.llms.base_llm.anthropic_messages.transformation import (
BaseAnthropicMessagesConfig,
@ -26,6 +29,15 @@ from litellm.llms.base_llm.anthropic_messages.transformation import (
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.responses.mcp.tool_precheck import may_name_litellm_mcp_gateway
from litellm.router_utils.reasoning_effort_capability import (
RequestedReasoningEffort,
deployment_reasoning_effort_levels,
keeps_effort,
model_group_from_metadata,
resolve_deployment_reasoning_effort,
with_adaptive_thinking_as_effort,
)
from litellm.types.llms.anthropic import AllAnthropicToolsValues
from litellm.types.llms.anthropic_messages.anthropic_request import AnthropicMetadata
from litellm.types.llms.anthropic_messages.anthropic_response import (
@ -144,6 +156,48 @@ def _deployment_passes_through_anthropic_messages(model_info: object) -> bool:
return isinstance(supported_endpoints, (list, tuple)) and "/v1/messages" in supported_endpoints
def _with_output_config(params: Mapping[str, object], output_config: RequestedReasoningEffort) -> Mapping[str, object]:
rest: Final = {key: value for key, value in params.items() if key != "output_config"}
return rest if output_config is None else {**rest, "output_config": output_config}
def _with_deployment_output_effort(
params: Mapping[str, object],
*,
model: str,
custom_llm_provider: str,
kwargs: Mapping[str, object],
drop_params: bool,
) -> Mapping[str, object]:
deployment_model_info: Final = kwargs.get("model_info")
def resolve(requested: RequestedReasoningEffort) -> RequestedReasoningEffort:
return resolve_deployment_reasoning_effort(
requested,
model=model,
custom_llm_provider=custom_llm_provider,
deployment_model_info=deployment_model_info,
model_group=model_group_from_metadata(kwargs.get("litellm_metadata"), kwargs.get("metadata")),
drop_params=drop_params,
).value
def translate(source: Mapping[str, object]) -> Mapping[str, object]:
return with_adaptive_thinking_as_effort(
source, model=model, custom_llm_provider=custom_llm_provider, drop_params=drop_params
)
requested_output: Final = as_str_mapping(params.get("output_config"))
caller_output: Final = None if requested_output is None else resolve(requested_output)
caller_params: Final = params if requested_output is None else _with_output_config(params, caller_output)
if deployment_reasoning_effort_levels(deployment_model_info) is None:
return caller_params
if requested_output is not None and keeps_effort(requested_output, caller_output):
return _with_output_config(translate(params), caller_output)
translated: Final = translate(caller_params)
budget_output: Final = as_str_mapping(translated.get("output_config"))
return translated if budget_output is None else _with_output_config(translated, resolve(budget_output))
def _deployment_supports_cache_control_ttl(model_info: object) -> bool:
return isinstance(model_info, dict) and model_info.get("cache_control_ttl") is True
@ -601,38 +655,34 @@ def anthropic_messages_handler(
# downstream path (native passthrough and both bridges) gets real tools rather than a
# reference the provider cannot resolve. Popped from kwargs so it never reaches the provider.
skip_mcp_handler: Final = kwargs.pop("_skip_mcp_handler", False)
if not skip_mcp_handler and tools:
if not skip_mcp_handler and tools and may_name_litellm_mcp_gateway(tools): # pyright: ignore[reportUnknownArgumentType] # untyped tools list
from litellm.llms.anthropic.pass_through.messages.mcp_handler import (
anthropic_messages_with_mcp,
)
from litellm.responses.mcp.litellm_proxy_mcp_handler import (
LiteLLM_Proxy_MCP_Handler,
)
if LiteLLM_Proxy_MCP_Handler.should_use_litellm_mcp_gateway(tools=tools):
return anthropic_messages_with_mcp(
max_tokens=max_tokens,
messages=messages,
model=model,
metadata=metadata,
stop_sequences=stop_sequences,
stream=stream,
system=cast( # cast-ok: anthropic accepts list system blocks; handler signature is narrower
"str | None", system
),
temperature=temperature,
thinking=thinking,
tool_choice=tool_choice,
tools=tools,
top_k=top_k,
top_p=top_p,
container=container,
api_key=api_key,
api_base=api_base,
client=client,
custom_llm_provider=custom_llm_provider,
**kwargs,
)
return anthropic_messages_with_mcp(
max_tokens=max_tokens,
messages=messages,
model=model,
metadata=metadata,
stop_sequences=stop_sequences,
stream=stream,
system=cast( # cast-ok: anthropic accepts list system blocks; handler signature is narrower
"str | None", system
),
temperature=temperature,
thinking=thinking,
tool_choice=tool_choice,
tools=tools,
top_k=top_k,
top_p=top_p,
container=container,
api_key=api_key,
api_base=api_base,
client=client,
custom_llm_provider=custom_llm_provider,
**kwargs,
)
anthropic_messages_provider_config: Final = _native_messages_config(
model, custom_llm_provider, kwargs.get("model_info"), api_base
@ -701,7 +751,13 @@ def anthropic_messages_handler(
f"custom_llm_provider is required for Anthropic messages, passed in model={model}, custom_llm_provider={custom_llm_provider}"
)
local_vars.update(kwargs)
request_kwargs: Final = cast(dict[str, object], kwargs) # cast-ok: **kwargs of the handler is untyped
marked_tools: Final = tools_marked_for_a_tool_config_point(
local_vars.get("tools"), request_kwargs.pop("cache_control_injection_points", None)
)
if marked_tools is not None:
AnthropicCacheControlHook.record_gateway_injection(request_kwargs, 1)
local_vars.update({**request_kwargs, **({"tools": list(marked_tools)} if marked_tools is not None else {})})
anthropic_messages_optional_request_params: Final = (
AnthropicMessagesRequestUtils.get_requested_anthropic_messages_optional_param(
params=local_vars,
@ -727,7 +783,15 @@ def anthropic_messages_handler(
),
),
anthropic_messages_provider_config=anthropic_messages_provider_config,
anthropic_messages_optional_request_params=dict(anthropic_messages_optional_request_params),
anthropic_messages_optional_request_params=dict(
_with_deployment_output_effort(
anthropic_messages_optional_request_params,
model=model,
custom_llm_provider=custom_llm_provider,
kwargs=cast(Mapping[str, object], kwargs), # cast-ok: router kwargs arrive untyped
drop_params=litellm.drop_params is True or litellm_params.get("drop_params") is True,
)
),
_is_async=is_async,
client=client,
custom_llm_provider=custom_llm_provider,

3
litellm/llms/anthropic/pass_through/messages/interceptors/advisor.py
View file

@ -27,6 +27,7 @@ from litellm.types.llms.anthropic import ANTHROPIC_ADVISOR_TOOL_TYPE
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
)
from litellm.types.utils import API_KEY_SECRET_REFERENCE_REJECTION, OIDC_SECRET_REFERENCE_PREFIX
if TYPE_CHECKING:
from litellm.router import Router
@ -226,6 +227,8 @@ def _resolve_advisor_credentials(advisor_tool: dict) -> tuple[str | None, str |
return None, None
api_key: Final[str | None] = advisor_tool.get("api_key")
api_base: Final[str | None] = advisor_tool.get("api_base")
if isinstance(api_key, str) and api_key.startswith(OIDC_SECRET_REFERENCE_PREFIX):
raise litellm.BadRequestError(message=API_KEY_SECRET_REFERENCE_REJECTION, model=None, llm_provider=None)
if api_base is None:
return api_key, None
if not api_key:

3
litellm/llms/anthropic/pass_through/messages/response_cache.py
View file

@ -1,5 +1,6 @@
import re
from collections.abc import AsyncIterator, Mapping, Sequence
from datetime import datetime
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, cast
@ -153,6 +154,8 @@ class CachedAnthropicMessagesStreamIterator(BaseAnthropicMessagesStreamingIterat
await self._handle_streaming_logging(chunks)
raise StopAsyncIteration
chunk: Final = self.chunks[self.current_index]
if self.completion_start_time is None:
self.completion_start_time = datetime.now()
self.current_index += 1
return chunk

2
litellm/llms/anthropic/pass_through/messages/streaming_iterator.py
View file

@ -506,7 +506,7 @@ class BaseAnthropicMessagesStreamingIterator:
):
self.litellm_logging_obj = litellm_logging_obj
self.request_body = request_body
self.start_time = datetime.now()
self.start_time = litellm_logging_obj.start_time
self.completion_start_time: datetime | None = None
async def _handle_streaming_logging(self, collected_chunks: list[bytes], *, stream_teardown: bool = False):

14
litellm/llms/anthropic/pass_through/messages/transformation.py
View file

@ -7,7 +7,9 @@ import litellm
from litellm._logging import verbose_logger
from litellm.anthropic_beta_headers_manager import get_unsupported_headers, is_beta_header_supported_by_bundled_config
from litellm.exceptions import AuthenticationError
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.anthropic.pass_through.convertible_blocks import drop_non_string_tool_result_texts
from litellm.llms.anthropic.pass_through.safeguards import proxy_serves_safeguards
from litellm.llms.base_llm.anthropic_messages.transformation import (
BaseAnthropicMessagesConfig,
@ -626,6 +628,14 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
):
anthropic_messages_optional_request_params.pop("safeguards") # pyright: ignore[reportUnknownMemberType] # the request params arrive as a bare dict
output_format: Final[object] = anthropic_messages_optional_request_params.get("output_format") # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType] # the request params arrive as a bare dict
if (
output_format is not None
and not isinstance(output_format, Mapping)
and drops_unsupported_params(litellm_params.drop_params)
):
anthropic_messages_optional_request_params.pop("output_format") # pyright: ignore[reportUnknownMemberType] # the request params arrive as a bare dict
self._translate_reasoning_effort_to_anthropic(
model=model,
optional_params=anthropic_messages_optional_request_params,
@ -692,7 +702,9 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
messages = strip_advisor_blocks_from_messages(messages)
anthropic_messages_request: Final[AnthropicMessagesRequest] = AnthropicMessagesRequest(
messages=strip_encrypted_reasoning_blocks_from_anthropic_messages(messages),
messages=strip_encrypted_reasoning_blocks_from_anthropic_messages(
drop_non_string_tool_result_texts(messages, getattr(litellm_params, "drop_params", None))
),
max_tokens=max_tokens,
model=model,
**anthropic_messages_optional_request_params,

8
litellm/llms/anthropic/pass_through/responses_adapters/handler.py
View file

@ -14,7 +14,7 @@ from litellm.integrations.anthropic_cache_control_hook import (
AnthropicCacheControlHook,
supports_openai_prompt_cache_breakpoint,
)
from litellm.litellm_core_utils.core_helpers import normalize_drop_params
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
from litellm.litellm_core_utils.prompt_templates.common_utils import cache_control_as_prompt_cache_breakpoint
from litellm.responses.utils import auto_reasoning_summary
from litellm.types.integrations.custom_logger import CONVERTED_STREAM_KEYS
@ -32,6 +32,7 @@ from litellm.types.llms.openai import ResponsesAPIResponse
from litellm.utils import ProviderConfigManager
from ..adapters.handler import request_safeguards_evaluator
from ..convertible_blocks import reject_malformed_tool_result_text
from ..safeguards import with_safeguard_results
from ..utils import litellm_logging_obj_from_kwargs, local_model_name
from .streaming_iterator import AnthropicResponsesStreamWrapper
@ -96,7 +97,7 @@ def _prompt_cache_block_rewrite(model: str, forwarded_kwargs: Mapping[str, objec
return cache_control_as_prompt_cache_breakpoint
if _model_takes_prompt_cache_breakpoint(model, forwarded_kwargs):
return None
if normalize_drop_params(forwarded_kwargs.get("drop_params")) is True or litellm.drop_params is True:
if drops_unsupported_params(forwarded_kwargs.get("drop_params")):
return _without_prompt_cache_breakpoint
return None
@ -142,6 +143,7 @@ def _build_responses_kwargs(
Build the kwargs dict to pass directly to litellm.responses() / litellm.aresponses().
"""
forwarded_kwargs: Final = _forwarded_kwargs(extra_kwargs)
reject_malformed_tool_result_text(messages, model, forwarded_kwargs)
block_rewrite: Final = _prompt_cache_block_rewrite(model, forwarded_kwargs)
request_messages: Final[AnthropicRequestMessages] = (
[
@ -190,6 +192,7 @@ def _build_responses_kwargs(
model, forwarded_kwargs.get("custom_llm_provider")
),
auto_summary_level=auto_reasoning_summary(model, forwarded_kwargs.get("custom_llm_provider")),
drop_params=drops_unsupported_params(forwarded_kwargs.get("drop_params")),
)
# Normalize reasoning effort based on model capabilities
@ -207,6 +210,7 @@ def _build_responses_kwargs(
effort,
model=model,
custom_llm_provider=provider_hint if isinstance(provider_hint, str) else None,
deployment_model_info=forwarded_kwargs.get("model_info"),
)
if normalized != effort:
responses_kwargs["reasoning"] = {**reasoning, "effort": normalized}

42
litellm/llms/anthropic/pass_through/responses_adapters/streaming_iterator.py
View file

@ -2,7 +2,7 @@
## Translates OpenAI call to Anthropic `/v1/messages` format
import asyncio
from collections import deque
from collections.abc import AsyncIterator, Iterator, Mapping
from collections.abc import AsyncGenerator, Iterator, Mapping
from typing import TYPE_CHECKING, Any, Final
from pydantic import ConfigDict, field_validator
@ -21,7 +21,10 @@ from litellm.llms.anthropic.pass_through.messages.utils import (
)
from litellm.llms.anthropic.pass_through.safeguards import StreamedSafeguardResults
from litellm.llms.anthropic.pass_through.utils import anthropic_sse_frame
from litellm.responses.streaming_iterator import stream_error_status_and_message
from litellm.responses.streaming_iterator import (
partial_responses_incomplete_event_on_disconnect,
stream_error_status_and_message,
)
from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicUsage
from litellm.types.llms.base import LiteLLMBaseModel
@ -33,6 +36,7 @@ from .transformation import (
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObject
from litellm.llms.anthropic.pass_through.safeguards import SafeguardsEvaluator
from litellm.types.llms.openai import ResponseIncompleteEvent
class _UpstreamFailure(LiteLLMBaseModel):
@ -488,8 +492,38 @@ class AnthropicResponsesStreamWrapper:
raise StopAsyncIteration
async def async_anthropic_sse_wrapper(self) -> AsyncIterator[bytes]:
"""Yield SSE-encoded bytes for each Anthropic event chunk."""
def async_anthropic_sse_wrapper(self) -> "AnthropicResponsesSSEByteStream":
"""SSE-encoded bytes for each Anthropic event chunk."""
return AnthropicResponsesSSEByteStream(self)
async def _sse_frames(self) -> AsyncGenerator[bytes, None]:
safeguard_results: Final = StreamedSafeguardResults(self.safeguards_evaluator)
async for chunk in self:
yield anthropic_sse_frame(await safeguard_results.observe(chunk))
def partial_response_on_disconnect(self) -> "ResponseIncompleteEvent | None":
return partial_responses_incomplete_event_on_disconnect(self.responses_stream)
class AnthropicResponsesSSEByteStream:
"""
The byte stream the proxy relays for a default-route /v1/messages request.
Keeps the Responses API iterator reachable so a client disconnect can bill
the partial output the way native /v1/responses streams do.
"""
def __init__(self, wrapper: AnthropicResponsesStreamWrapper) -> None:
self._wrapper: Final = wrapper
self._frames: Final = wrapper._sse_frames() # pyright: ignore[reportPrivateUsage] # the wrapper's own byte stream
def __aiter__(self) -> "AnthropicResponsesSSEByteStream":
return self
async def __anext__(self) -> bytes:
return await self._frames.__anext__()
async def aclose(self) -> None:
await self._frames.aclose()
def partial_response_on_disconnect(self) -> "ResponseIncompleteEvent | None":
return self._wrapper.partial_response_on_disconnect()

68
litellm/llms/anthropic/pass_through/responses_adapters/transformation.py
View file

@ -18,8 +18,9 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
with_prompt_cache_breakpoint,
)
from litellm.litellm_core_utils.reasoning_effort_utils import (
reasoning_effort_from_thinking_budget,
reasoning_effort_from_thinking,
)
from litellm.llms.anthropic.pass_through.convertible_blocks import tool_result_text_parts
from litellm.llms.anthropic.pass_through.messages.utils import (
refusal_stop_details,
responses_output_refusal_text,
@ -371,10 +372,9 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
elif btype == "tool_result":
tool_use_id = block.get("tool_use_id", "")
inner = block.get("content")
tool_result_block = cast(Mapping[str, object], block) # cast-ok: TypedDict upcast
tool_breakpoint, covered_image_parts, later_image_parts = (
self._tool_result_images_by_breakpoint(
cast(Mapping[str, object], block) # cast-ok: TypedDict upcast
)
self._tool_result_images_by_breakpoint(tool_result_block)
)
document_candidates = (
tuple(
@ -391,10 +391,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
elif isinstance(inner, str):
output_text = inner
elif isinstance(inner, list):
parts = [
c.get("text", "") for c in inner if isinstance(c, dict) and c.get("type") == "text"
]
output_text = "\n".join(parts)
output_text = "\n".join(tool_result_text_parts(tool_result_block))
if covered_image_parts or later_image_parts:
output_text = (
f"{output_text}\n{TOOL_RESULT_IMAGE_PLACEHOLDER}"
@ -467,33 +464,32 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
return input_items
@staticmethod
def _translate_tool_to_responses_api(tool_dict: Mapping[str, object]) -> dict[str, object]:
tool_type: Final = tool_dict.get("type", "")
tool_name: Final = tool_dict.get("name", "")
# web_search tool
if (isinstance(tool_type, str) and tool_type.startswith("web_search")) or tool_name == "web_search":
return {"type": "web_search_preview"}
# Responses turns strict mode on when `strict` is omitted, silently rewriting
# `required` to every property. Anthropic tools are non-strict unless asked.
func_tool: Final[dict[str, object]] = {
"type": "function",
"name": tool_name,
"strict": bool(tool_dict.get("strict")),
}
if "description" in tool_dict:
func_tool["description"] = tool_dict["description"]
if "input_schema" in tool_dict:
func_tool["parameters"] = json_copy_or_original(tool_dict["input_schema"])
return func_tool
def translate_tools_to_responses_api(
self,
tools: list[AllAnthropicToolsValues],
) -> list[dict[str, object]]:
"""Convert Anthropic tool definitions to Responses API function tools."""
result: Final[list[dict[str, object]]] = []
for tool in tools:
tool_dict = cast(dict[str, Any], tool)
tool_type = tool_dict.get("type", "")
tool_name = tool_dict.get("name", "")
# web_search tool
if (isinstance(tool_type, str) and tool_type.startswith("web_search")) or tool_name == "web_search":
result.append({"type": "web_search_preview"})
continue
# Responses turns strict mode on when `strict` is omitted, silently rewriting
# `required` to every property. Anthropic tools are non-strict unless asked.
func_tool: dict[str, object] = {
"type": "function",
"name": tool_name,
"strict": bool(tool_dict.get("strict")),
}
if "description" in tool_dict:
func_tool["description"] = tool_dict["description"]
if "input_schema" in tool_dict:
func_tool["parameters"] = json_copy_or_original(tool_dict["input_schema"])
result.append(func_tool)
return result
return [self._translate_tool_to_responses_api(tool) for tool in tools]
@staticmethod
def translate_tool_choice_to_responses_api(
@ -545,6 +541,8 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
thinking: dict[str, object],
output_config: dict[str, object] | None = None,
auto_summary_level: str = "detailed",
*,
drop_params: bool = True,
) -> dict[str, object] | None:
"""
Convert Anthropic thinking param to Responses API reasoning param.
@ -565,9 +563,10 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
if isinstance(output_config, dict) and output_config.get("effort"):
effort = output_config["effort"]
elif thinking_type == "enabled":
raw_budget: Final = thinking.get("budget_tokens", 0)
budget_tokens: Final = int(raw_budget) if isinstance(raw_budget, (int, float)) else 0
effort = reasoning_effort_from_thinking_budget(budget_tokens)
budget_effort: Final = reasoning_effort_from_thinking(thinking, drop_params=drop_params)
if budget_effort is None:
return None
effort = budget_effort
else:
return None
@ -593,6 +592,8 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
anthropic_request: AnthropicMessagesRequest,
include_encrypted_reasoning: bool = True,
auto_summary_level: str = "detailed",
*,
drop_params: bool = True,
) -> dict[str, Any]:
"""
Translate a full Anthropic /v1/messages request dict to
@ -674,6 +675,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
thinking,
output_config=cast(dict[str, Any] | None, output_config),
auto_summary_level=auto_summary_level,
drop_params=drop_params,
)
if reasoning:
responses_kwargs["reasoning"] = reasoning

12
litellm/llms/anthropic/pass_through/utils.py
View file

@ -193,9 +193,13 @@ def normalize_reasoning_effort_value(
effort: str,
model: str,
custom_llm_provider: str | None = None,
deployment_model_info: object = None,
) -> str:
"""Lower a tier the deployment does not accept to the nearest one it does, leaving others alone.
A deployment whose own reasoning_effort_levels name a strength tier keeps the tier as sent, since
completion() and responses() round it onto those levels.
The accepted set is resolved by the same owner that answers ``/model_group/info``, so a level
the proxy advertises is a level this path forwards.
@ -210,7 +214,13 @@ def normalize_reasoning_effort_value(
if chain is None:
return effort
from litellm.router_utils.reasoning_effort_capability import resolve_supported_reasoning_efforts
from litellm.router_utils.reasoning_effort_capability import (
deployment_rounds_reasoning_effort,
resolve_supported_reasoning_efforts,
)
if deployment_rounds_reasoning_effort(deployment_model_info):
return effort
from litellm.utils import get_model_info
try:

4
litellm/llms/azure/chat/gpt_5_transformation.py
View file

@ -1,5 +1,6 @@
"""Support for Azure OpenAI gpt-5 model family."""
from collections.abc import Collection
from typing import ClassVar, Final
import litellm
@ -86,6 +87,8 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
model: str,
drop_params: bool,
api_version: str = "",
*,
declared_reasoning_efforts: Collection[str] = (),
) -> dict:
return OpenAIGPT5Config.map_openai_params(
self,
@ -93,6 +96,7 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
optional_params=optional_params,
model=model,
drop_params=drop_params,
declared_reasoning_efforts=declared_reasoning_efforts,
)
def transform_request(

3
litellm/llms/azure_ai/anthropic/count_tokens/handler.py
View file

@ -4,6 +4,7 @@ Azure AI Anthropic CountTokens API handler.
Uses httpx for HTTP requests with Azure authentication.
"""
from collections.abc import Mapping
from typing import Final
import httpx
@ -36,6 +37,7 @@ class AzureAIAnthropicCountTokensHandler(AzureAIAnthropicCountTokensConfig):
timeout: float | httpx.Timeout | None = None,
tools: list[dict[str, JsonValue]] | None = None,
system: object = None,
optional_params: Mapping[str, JsonValue] | None = None,
) -> dict[str, object]:
"""
Handle a CountTokens request using httpx with Azure authentication.
@ -66,6 +68,7 @@ class AzureAIAnthropicCountTokensHandler(AzureAIAnthropicCountTokensConfig):
messages=messages,
tools=tools,
system=system,
optional_params=optional_params,
)
verbose_logger.debug("Transformed request: %s", request_body)

34
litellm/llms/azure_ai/anthropic/count_tokens/token_counter.py
View file

@ -3,12 +3,19 @@ Azure AI Anthropic Token Counter implementation using the CountTokens API.
"""
import os
from collections.abc import Mapping
from types import MappingProxyType
from typing import Any, Final
from pydantic import JsonValue
import litellm
from litellm._logging import verbose_logger
from litellm.llms.anthropic.count_tokens.transformation import (
AnthropicCountFields,
ChatCountTranslationError,
anthropic_count_fields,
)
from litellm.llms.azure_ai.anthropic.count_tokens.handler import (
AzureAIAnthropicCountTokensHandler,
)
@ -55,6 +62,8 @@ class AzureAIAnthropicTokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "anthropic",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
count_options: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
"""
Count tokens using Azure AI Anthropic's CountTokens API.
@ -74,6 +83,30 @@ class AzureAIAnthropicTokenCounter(BaseTokenCounter):
if not messages:
return None
from litellm.llms.azure_ai.anthropic.transformation import AzureAnthropicConfig
match translated := await anthropic_count_fields(
model_to_use, messages, system, tools, messages_format, AzureAnthropicConfig(), chat_params
):
case ChatCountTranslationError():
return translated.response(request_model, model_to_use, "azure_ai_anthropic_api")
case AnthropicCountFields():
return await self.count_tokens(
model_to_use=model_to_use,
messages=[dict(message) for message in translated.messages],
contents=contents,
deployment=deployment,
request_model=request_model,
tools=[dict(tool) for tool in translated.tools] if translated.tools is not None else None,
system=translated.system,
messages_format="anthropic",
output_format=output_format,
context_management=context_management,
count_options=translated.options,
)
case None:
pass
deployment = deployment or {}
litellm_params: Final = deployment.get("litellm_params", {})
@ -101,6 +134,7 @@ class AzureAIAnthropicTokenCounter(BaseTokenCounter):
litellm_params=litellm_params,
tools=tools,
system=system,
optional_params=count_options,
)
if result is not None:

5
litellm/llms/azure_ai/chat/transformation.py
View file

@ -1,7 +1,7 @@
import copy
import enum
import re
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Collection, Iterator
from typing import TYPE_CHECKING, Final, cast
import httpx
@ -104,6 +104,8 @@ class AzureAIStudioConfig(OpenAIConfig):
optional_params: dict[str, object], # mutable-ok: OpenAIConfig.map_openai_params signature
model: str,
drop_params: bool,
*,
declared_reasoning_efforts: Collection[str] = (),
) -> dict[str, object]: # mutable-ok: OpenAIConfig.map_openai_params signature
if not azureAIGPT5Config.is_model_gpt_5_model(model):
return super().map_openai_params(
@ -117,6 +119,7 @@ class AzureAIStudioConfig(OpenAIConfig):
optional_params=optional_params,
model=model,
drop_params=drop_params,
declared_reasoning_efforts=declared_reasoning_efforts,
)
def _supports_stop_reason(self, model: str) -> bool:

52
litellm/llms/base_llm/base_utils.py
View file

@ -5,9 +5,11 @@ Utility functions for base LLM classes.
import copy
import itertools
import json
import re
from abc import ABC, abstractmethod
from collections.abc import Iterator, Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import Any, Final
from openai.lib import _parsing, _pydantic # pyright: ignore[reportPrivateUsage] # SDK parser internals
@ -39,6 +41,7 @@ class BaseTokenCounter(ABC):
messages_format: TokenCountMessagesFormat = "openai",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
pass
@ -172,7 +175,7 @@ _convert_tool_response_to_message = convert_tool_response_to_message
@dataclass(frozen=True, slots=True)
class ResponseFormatSchema:
property_names: frozenset[str]
declared_names: frozenset[str]
required_names: frozenset[str] = frozenset()
allows_undeclared_keys: bool = False
@ -216,9 +219,10 @@ def response_format_schema(value: object) -> ResponseFormatSchema | None:
properties: Final = as_json_object(document.get("properties", {}))
if properties is None:
return None
required_names: Final = _names(document.get("required", ()))
return ResponseFormatSchema(
property_names=frozenset(properties),
required_names=_names(document.get("required", ())),
declared_names=frozenset(properties) | required_names,
required_names=required_names,
allows_undeclared_keys=_allows_undeclared_keys(document),
)
@ -244,8 +248,9 @@ def _json_object_or_encoded_object(value: JsonValue) -> Mapping[str, JsonValue]
def _label_keys(payload_object: Mapping[str, JsonValue], schema: ResponseFormatSchema) -> frozenset[str]:
if not schema.required_names:
return frozenset()
declared: Final = schema.property_names | schema.required_names
return frozenset(key for key, value in payload_object.items() if isinstance(value, str) and key not in declared)
return frozenset(
key for key, value in payload_object.items() if isinstance(value, str) and key not in schema.declared_names
)
def _envelope_layer(
@ -256,7 +261,7 @@ def _envelope_layer(
)
label_keys: Final = name_keys or _label_keys(payload_object, schema)
wrapper_keys: Final = tuple(key for key in payload_object if key not in label_keys)
if len(wrapper_keys) != 1 or wrapper_keys[0] in schema.property_names:
if len(wrapper_keys) != 1 or wrapper_keys[0] in schema.declared_names:
return None
wrapped: Final = payload_object[wrapper_keys[0]]
inner: Final = _json_object_or_encoded_object(wrapped) if name_keys else as_json_object(wrapped)
@ -275,15 +280,16 @@ def _envelope_layers(
def _schema_object_inside_envelope(
payload_object: Mapping[str, JsonValue], schema: ResponseFormatSchema, tool_name: str
) -> Mapping[str, JsonValue] | None:
layers: Final = _envelope_layers(payload_object, schema, tool_name)
candidates: Final = layers if schema.property_names else itertools.islice(layers, 1)
return next((layer.inner for layer in candidates if _holds_the_schema_object(layer, schema)), None)
walk: Final = _envelope_layers(payload_object, schema, tool_name)
layers: Final = tuple(walk if schema.declared_names else itertools.islice(walk, 1))
schema_object: Final = next((layer.inner for layer in layers if _shares_a_schema_key(layer.inner, schema)), None)
if schema_object is not None:
return schema_object
return next((layer.inner for layer in layers if layer.names_the_tool), None)
def _holds_the_schema_object(layer: _EnvelopeLayer, schema: ResponseFormatSchema) -> bool:
if layer.names_the_tool:
return True
return not schema.accepts_a_lone_undeclared_key and not schema.property_names.isdisjoint(layer.inner)
def _shares_a_schema_key(candidate: Mapping[str, JsonValue], schema: ResponseFormatSchema) -> bool:
return not schema.accepts_a_lone_undeclared_key and not schema.declared_names.isdisjoint(candidate)
def response_format_envelope_contents(payload: object, schema: ResponseFormatSchema, tool_name: str) -> object | None:
@ -293,7 +299,15 @@ def response_format_envelope_contents(payload: object, schema: ResponseFormatSch
schema_object: Final = _schema_object_inside_envelope(payload_object, schema, tool_name)
if schema_object is not None:
return schema_object
return None if "values" in schema.property_names else payload_object.get("values")
return None if "values" in schema.declared_names else payload_object.get("values")
_LONE_SURROGATE: Final = re.compile("[\ud800-\udfff]")
def response_format_answer_json(value: object) -> str:
text: Final = json.dumps(value, ensure_ascii=False)
return json.dumps(value) if _LONE_SURROGATE.search(text) else text
def unwrap_response_format_envelope(payload: object, schema: ResponseFormatSchema, tool_name: str) -> object:
@ -305,7 +319,7 @@ def _field_names(
blocks: tuple[Mapping[str, JsonValue], ...], key: str, schema: ResponseFormatSchema
) -> tuple[str, ...] | None:
values: Final = (block[key] for block in blocks)
names: Final = tuple(value for value in values if isinstance(value, str) and value in schema.property_names)
names: Final = tuple(value for value in values if isinstance(value, str) and value in schema.declared_names)
return names if len(frozenset(names)) == len(blocks) else None
@ -324,7 +338,7 @@ def _split_answer_fields(
blocks: tuple[Mapping[str, JsonValue], ...], schema: ResponseFormatSchema
) -> Mapping[str, JsonValue] | None:
keys: Final = frozenset(blocks[0])
if not keys.isdisjoint(schema.property_names) or any(frozenset(block) != keys for block in blocks):
if not keys.isdisjoint(schema.declared_names) or any(frozenset(block) != keys for block in blocks):
return None
named: Final = tuple((key, names) for key in keys if (names := _field_names(blocks, key, schema)) is not None)
if len(named) != 1:
@ -340,7 +354,7 @@ def _merged_partial_answers(
blocks: tuple[Mapping[str, JsonValue], ...], schema: ResponseFormatSchema
) -> Mapping[str, JsonValue] | None:
keys: Final = frozenset(itertools.chain.from_iterable(blocks))
if not keys <= schema.property_names or len(keys) != sum(len(block) for block in blocks):
if not keys <= schema.declared_names or len(keys) != sum(len(block) for block in blocks):
return None
return dict(itertools.chain.from_iterable(block.items() for block in blocks))
@ -358,14 +372,14 @@ def split_response_format_answer(
def partial_response_format_answer(encoded_block: str, schema: ResponseFormatSchema) -> Mapping[str, JsonValue] | None:
block: Final = _json_object_or_encoded_object(encoded_block)
return block if block is not None and frozenset(block) <= schema.property_names else None
return block if block is not None and frozenset(block) <= schema.declared_names else None
def names_a_response_format_property(encoded_block: str, schema: ResponseFormatSchema) -> bool:
block: Final = _json_object_or_encoded_object(encoded_block)
if block is None:
return False
return any(isinstance(value, str) and value in schema.property_names for value in block.values())
return any(isinstance(value, str) and value in schema.declared_names for value in block.values())
def _dict_to_response_format_helper(response_format: dict, ref_template: str | None = None) -> dict:

Some files were not shown because too many files have changed in this diff Show more