scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-03 07:23:33 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 166

Merge remote-tracking branch 'origin/main' into litellm_fix_responses_stream_failure_billing

Browse source 
# Conflicts:
#	litellm/llms/anthropic/pass_through/responses_adapters/streaming_iterator.py
#	litellm/responses/streaming_iterator.py
#	tests/unit/responses/test_responses_streaming_iterator.py
This commit is contained in:
mateo-berri 2026-10-20 14:30:57 -07:00
commit 48bde72002
964 changed files with 28523 additions and 10011 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
.github/PULL_REQUEST_TEMPLATE/general.md vendored
View file

@ -4,6 +4,13 @@
Drop every section you have nothing to put in, heading included: a bare "## Relevant issues" or
"## Affected release" with nothing under it must not appear in the final description -->
<!-- Two or three plain sentences a teammate can read in 10 seconds, written the way you'd explain it in Slack
Say what the PR does for users and why, then anything that keeps it working over time (a sync job, a config someone owns)
No headings, bullets, tables, or code names here. Everything else goes inside the Details block below -->
<details>
<summary>Details</summary>
## TLDR
<!-- Fill in the bullets below and keep each one short and concrete: one line per bullet, roughly 10 words max
@ -164,3 +171,5 @@ Example checklists:
## Final Attestation
- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
</details>

41
cookbook/litellm_proxy_server/batch_api/bedrock/README.md Normal file
View file

@ -0,0 +1,41 @@
# Bedrock batch endpoints
Set `aws_bedrock_control_plane_endpoint` on a deployment when batch create, retrieve, and cancel requests must use a custom Bedrock control-plane endpoint. The proxy signs these requests for the configured URL. Without this setting, batch requests use the standard regional Bedrock endpoint
```yaml
model_list:
- model_name: bedrock-batch
litellm_params:
model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
aws_region_name: us-east-1
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
aws_batch_role_arn: os.environ/AWS_BATCH_ROLE_ARN
s3_output_bucket_name: os.environ/AWS_S3_OUTPUT_BUCKET_NAME
aws_bedrock_control_plane_endpoint: os.environ/BEDROCK_CONTROL_PLANE_ENDPOINT
```
Set `BEDROCK_CONTROL_PLANE_ENDPOINT` to your Bedrock control-plane VPC endpoint or compatible gateway base URL, without `/model-invocation-job`. The proxy adds the operation path. The endpoint must be reachable from the proxy, and the configured AWS region must match the endpoint's signing region
`api_base` and `aws_bedrock_runtime_endpoint` configure runtime inference separately. They do not override the batch control-plane endpoint. `s3_endpoint_url` configures S3 file access separately
The control-plane setting belongs in deployment configuration. Sending `aws_bedrock_control_plane_endpoint` in an API request body is rejected. There is no automatic `AWS_BEDROCK_CONTROL_PLANE_ENDPOINT` environment fallback, so reference the desired environment variable explicitly as shown above
Start the proxy with this configuration and set `LITELLM_API_BASE` to its URL. Set `BEDROCK_BATCH_INPUT_URI` to a valid input JSONL file in S3 that the batch service role can read
```bash
curl -sS "$LITELLM_API_BASE/v1/batches" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"bedrock-batch\",\"input_file_id\":\"$BEDROCK_BATCH_INPUT_URI\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\"}"
```
Use the returned `id` as `BATCH_ID` to retrieve or cancel the same job through the deployment's configured endpoint
```bash
curl -sS "$LITELLM_API_BASE/v1/batches/$BATCH_ID?model=bedrock-batch" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
curl -sS -X POST "$LITELLM_API_BASE/v1/batches/$BATCH_ID/cancel?model=bedrock-batch" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
```

164
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -4,6 +4,7 @@ Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if t
import asyncio
import json
import re
from collections.abc import Mapping, Sequence
from dataclasses import replace as dataclasses_replace
from datetime import datetime, timedelta, timezone
@ -39,7 +40,12 @@ if TYPE_CHECKING:
from litellm.types.utils import LiteLLMBatch
def _error_names_resource(error: Exception, resource_id: str) -> bool:
return resource_id in re.findall(r"""[^\s"'`]+""", str(error))
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
RECENT_BATCH_POLL_WINDOW: Final = timedelta(hours=48)
PROVIDER_TERMINAL_BATCH_STATUSES: Final[Tuple[str, ...]] = (
"completed",
@ -109,6 +115,11 @@ class CheckBatchCost:
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
self.batch_processed_support_confirmed: bool = False
self._poll_cursor: tuple[datetime, str] | None = None
self._poll_before: datetime = datetime.now(timezone.utc)
self._arrival_cursor: tuple[datetime, str] | None = None
self._arrival_before: datetime = self._poll_before
self._poll_arrivals: bool = False
self._line_item_stores: Final[set[asyncio.Task[None]]] = set()
self._line_item_store_turn: Final = asyncio.Lock()
self._poll_idle: Final = asyncio.Event()
@ -161,29 +172,9 @@ class CheckBatchCost:
f"(older than {MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days) as stale_expired"
)
if not self._has_batch_processed_column:
return
# A row already in a terminal status is never rewritten by the sweep above, so
# without this it keeps a poll-page slot forever and starves newer batches.
retired: Final = await _managed_object_table(self.prisma_client).update_many(
where={
"file_purpose": "batch",
"batch_processed": False,
"status": {"in": ["complete", "completed"]},
"created_at": {"lt": cutoff},
},
data={"batch_processed": True},
)
if retired > 0:
verbose_proxy_logger.warning(
f"CheckBatchCost: gave up on {retired} completed managed objects older than "
f"{MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days that were never costed"
)
async def _fallback_find_jobs(self) -> "Sequence[ManagedObjectRow]":
"""Query batch jobs without the batch_processed filter (for older schemas)."""
return await _managed_object_table(self.prisma_client).find_many(
return await self._find_jobs(
where={
"file_purpose": "batch",
"status": {
@ -197,10 +188,59 @@ class CheckBatchCost:
]
},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
async def _find_jobs(self, where: Mapping[str, object]) -> Sequence[ManagedObjectRow]:
arrivals: Final = self._poll_arrivals
page: Final = await self._find_job_page(where, arrivals=arrivals)
self._poll_arrivals = not arrivals if page else arrivals
if page:
return page
return await self._find_job_page(where, arrivals=not arrivals)
async def _find_job_page(self, where: Mapping[str, object], *, arrivals: bool) -> Sequence[ManagedObjectRow]:
cursor: Final = self._arrival_cursor if arrivals else self._poll_cursor
if cursor is None:
if arrivals:
self._arrival_before = datetime.now(timezone.utc)
else:
self._poll_before = datetime.now(timezone.utc) - RECENT_BATCH_POLL_WINDOW
before: Final = self._arrival_before if arrivals else self._poll_before
page: Final = await _managed_object_table(self.prisma_client).find_many(
where={
**where,
"created_at": {
"lte": before,
**({"gt": before - RECENT_BATCH_POLL_WINDOW} if arrivals else {}),
},
**(
{
"OR": [
{"created_at": {"gt": cursor[0]}},
{"created_at": cursor[0], "id": {"gt": cursor[1]}},
]
}
if cursor is not None
else {}
),
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order=[{"created_at": "asc"}, {"id": "asc"}],
)
next_cursor: Final = (page[-1].created_at, page[-1].id) if len(page) == MAX_OBJECTS_PER_POLL_CYCLE else None
if arrivals:
self._arrival_cursor = next_cursor
else:
self._poll_cursor = next_cursor
return page
def _unprocessed_snapshot(self, job: ManagedObjectRow) -> Mapping[str, object]:
return {
"id": job.id,
"updated_at": job.updated_at,
**({"batch_processed": False} if self._has_batch_processed_column else {}),
}
async def _retire_job(self, job: "ManagedObjectRow", reason: str) -> None:
"""
Take a row that can never be costed out of the poll page. Leaving it selectable
@ -208,24 +248,19 @@ class CheckBatchCost:
once enough such rows accumulate no newer batch is ever reached. Older schemas
without batch_processed can only be excluded through the status filter.
"""
data: Final = (
{"batch_processed": True}
if self._has_batch_processed_column
else {"status": "stale_expired"}
)
data: Final = {"batch_processed": True} if self._has_batch_processed_column else {"status": "stale_expired"}
try:
await _managed_object_table(self.prisma_client).update(
where={"id": job.id},
retired: Final = await _managed_object_table(self.prisma_client).update_many(
where=self._unprocessed_snapshot(job),
data=data,
)
except Exception as db_err:
verbose_proxy_logger.error(
f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}"
)
verbose_proxy_logger.error(f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}")
return
if retired == 0:
return
verbose_proxy_logger.warning(
f"CheckBatchCost: job {job.id} can never be costed ({reason}), "
"so it will no longer be polled"
f"CheckBatchCost: job {job.id} can never be costed ({reason}), so it will no longer be polled"
)
async def _claim_job_for_costing(self, job: "ManagedObjectRow", response: "LiteLLMBatch") -> bool:
@ -312,13 +347,13 @@ class CheckBatchCost:
from litellm.exceptions import NotFoundError
return isinstance(error, (NotFoundError, openai.NotFoundError)) and batch_id in str(error)
return isinstance(error, (NotFoundError, openai.NotFoundError)) and _error_names_resource(error, batch_id)
def _batch_deployment_exists(self, model_id: str) -> bool:
"""A 404 only proves the batch is gone when it came from the batch's own
deployment. Once that deployment leaves the router, default fallbacks can
silently send the retrieve to a provider that never saw the batch, so its
404 must not retire the row; the staleness sweep bounds it instead."""
404 must not retire a potentially recoverable row."""
return self.llm_router.get_deployment(model_id=model_id) is not None
@staticmethod
@ -333,7 +368,7 @@ class CheckBatchCost:
if not output_file_id:
return False
return isinstance(error, (NotFoundError, openai.NotFoundError)) and output_file_id in str(error)
return isinstance(error, (NotFoundError, openai.NotFoundError)) and _error_names_resource(error, output_file_id)
async def _finalize_unbilled_terminal_job(
self, job: "ManagedObjectRow", response: "LiteLLMBatch"
@ -355,18 +390,18 @@ class CheckBatchCost:
db_batch_object=job,
unified_batch_id=_is_base64_encoded_unified_file_id(job.unified_object_id),
)
update_data: Final[dict] = {
update_data: Final[dict[str, object]] = {
"status": response.status,
"file_object": response.model_dump_json(),
**({"batch_processed": True} if self._has_batch_processed_column else {}),
}
await _managed_object_table(self.prisma_client).update(
where={"id": job.id},
finalized: Final = await _managed_object_table(self.prisma_client).update_many(
where=self._unprocessed_snapshot(job),
data=update_data,
)
verbose_proxy_logger.info(
f"CheckBatchCost: marked job {job.id} as {response.status} in DB"
)
if finalized == 0:
return False
verbose_proxy_logger.info(f"CheckBatchCost: marked job {job.id} as {response.status} in DB")
return True
except Exception as db_err:
verbose_proxy_logger.error(
@ -817,20 +852,26 @@ class CheckBatchCost:
)
return None
try:
await logging_obj.async_success_handler(
result=response,
batch_cost=batch_result.cost,
batch_usage=batch_result.usage,
batch_models=batch_result.models,
batch_successful_requests=batch_result.successful_requests,
batch_failed_requests=batch_result.failed_requests,
batch_prompt_cost=batch_result.prompt_cost,
batch_completion_cost=batch_result.completion_cost,
)
except Exception:
await self._release_job_claim(job, original_status, original_file_object)
raise
from litellm.proxy.hooks.proxy_track_cost_callback import track_batch_accounting
with track_batch_accounting(logging_obj.litellm_call_id) as accounting:
try:
await logging_obj.async_success_handler(
result=response,
batch_cost=batch_result.cost,
batch_usage=batch_result.usage,
batch_models=batch_result.models,
batch_successful_requests=batch_result.successful_requests,
batch_failed_requests=batch_result.failed_requests,
batch_prompt_cost=batch_result.prompt_cost,
batch_completion_cost=batch_result.completion_cost,
)
finally:
if accounting.retryable_failure:
await self._release_job_claim(job, original_status, original_file_object)
if accounting.retryable_failure:
return None
# Record batch duration (completed_at - created_at)
if prom_logger and response.completed_at and response.created_at:
@ -1047,6 +1088,7 @@ class CheckBatchCost:
"""
try:
from litellm.integrations.prometheus import PrometheusLogger
prom_logger = PrometheusLogger.get_instance()
except Exception as e:
verbose_proxy_logger.error(f"CheckBatchCost: could not get Prometheus logger: {e}")
@ -1071,7 +1113,7 @@ class CheckBatchCost:
# endpoint may transition a batch to "complete" before
# CheckBatchCost runs. The batch_processed=False filter
# already prevents reprocessing finished batches.
jobs = await _managed_object_table(self.prisma_client).find_many(
jobs = await self._find_jobs(
where={
"file_purpose": "batch",
"batch_processed": False,
@ -1084,8 +1126,6 @@ class CheckBatchCost:
]
},
},
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
self.batch_processed_support_confirmed = True
except Exception as query_err:
@ -1152,7 +1192,7 @@ class CheckBatchCost:
continue
verbose_proxy_logger.error(
f"CheckBatchCost: failed to track cost for batch {batch_id} "
f"(job {job.id}); leaving it unprocessed so the next poll retries: {tracking_err}"
f"(job {job.id}): {tracking_err}"
)
self._record_error(prom_logger, "cost_tracking_error")
continue

26
enterprise/litellm_enterprise/proxy/common_utils/check_responses_cost.py
View file

@ -59,6 +59,19 @@ if TYPE_CHECKING:
from litellm.router import Router
_STORED_RESPONSE_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
class _RoutedModelInfo(BaseModel):
model_config = ConfigDict(frozen=True)
id: str | None = None
class _RoutedDeployment(BaseModel):
model_config = ConfigDict(frozen=True)
model_info: _RoutedModelInfo = _RoutedModelInfo()
_ROUTED_DEPLOYMENTS: Final = TypeAdapter(tuple[_RoutedDeployment, ...])
_RETRIEVE_RESPONSE_CALL_TYPE: Final = "aget_responses"
@ -314,10 +327,23 @@ class CheckResponsesCost:
INTERNAL_CALL_ORIGIN_METADATA_KEY: BACKGROUND_RESPONSE_COST_POLL_CALL_ORIGIN,
**target.model_metadata,
**(_deployment_metadata(deployment) if deployment is not None else {}),
**self._requested_model_group(target, job.team_id),
}
),
)
def _requested_model_group(self, target: _PollTarget, team_id: str | None) -> Mapping[str, object]:
requested: Final = target.model_metadata.get("model_group")
if target.deployment is None or not isinstance(requested, str):
return MappingProxyType({})
routed: Final = _ROUTED_DEPLOYMENTS.validate_python(
self.llm_router.get_model_list(model_name=requested, team_id=team_id) or ()
)
serving_id: Final = target.deployment.model_info.id
return MappingProxyType(
{"model_group": requested} if any(row.model_info.id == serving_id for row in routed) else {}
)
async def _bill_once(
self,
job: ManagedObjectRow,

62
enterprise/litellm_enterprise/proxy/common_utils/managed_object_attribution.py
View file

@ -9,6 +9,7 @@ from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime
from typing import TYPE_CHECKING, Final, Protocol, cast
from pydantic import TypeAdapter
@ -27,6 +28,12 @@ if TYPE_CHECKING:
class ManagedObjectRow(Protocol):
@property
def created_at(self) -> datetime: ...
@property
def updated_at(self) -> datetime: ...
@property
def id(self) -> str: ...
@ -107,8 +114,10 @@ class ManagedObjectCreatorAttribution:
user's alias, because a resolvable name is more useful on the spend row than a null.
user_api_key_org_id is resolved too: the spend update writer reads it off this metadata
to increment organization spend, so leaving it out drops the cost from org accounting.
A background response also carries its end user and the key, team, and user budgets, the
way the read that billed it before did, and never names a key-scoped creator as its user.
A background response also carries its end user, the creating key's project and its alias,
and the key, team, and user budgets and per-model budgets, the way the read that billed it
before did, and never names a key-scoped creator as its user. A key that sets no budget of
its own takes its budget tier's, the same rule auth applies.
"""
def __init__(self, prisma_client: PrismaClient, poller_name: str) -> None:
@ -126,9 +135,23 @@ class ManagedObjectCreatorAttribution:
"user_api_key_user_id": _creator_user_id(job.created_by),
**_shared_metadata(job, rows),
**({"user_api_key_end_user_id": end_user_id} if end_user_id else {}),
**_budget_fields(rows.key, "user_api_key_spend", "user_api_key_max_budget"),
**_budget_fields(rows.team, "user_api_key_team_spend", "user_api_key_team_max_budget"),
**_budget_fields(rows.user, "user_api_key_user_spend", "user_api_key_user_max_budget"),
"user_api_key_project_id": getattr(rows.key, "project_id", None),
"user_api_key_project_alias": getattr(
getattr(rows.key, "litellm_project_table", None), "project_alias", None
),
**_key_budget_fields(rows.key),
**_budget_fields(
rows.team,
"user_api_key_team_spend",
"user_api_key_team_max_budget",
"user_api_key_team_model_max_budget",
),
**_budget_fields(
rows.user,
"user_api_key_user_spend",
"user_api_key_user_max_budget",
"user_api_key_user_model_max_budget",
),
}
async def _rows(self, job: ManagedObjectRow, object_id: str) -> _CreatorRows:
@ -155,7 +178,9 @@ class ManagedObjectCreatorAttribution:
if not api_key or _is_cli_session_key(api_key, created_by):
return None
try:
return await _token_table(self._prisma_client).find_unique(where={"token": api_key})
return await _token_table(self._prisma_client).find_unique(
where={"token": api_key}, include={"litellm_budget_table": True, "litellm_project_table": True}
)
except Exception as e:
verbose_proxy_logger.error(f"{self._poller_name}: could not look up the creating key for {object_id}: {e}")
return None
@ -212,7 +237,28 @@ def _organization_id(row: object) -> str | None:
return cast("str | None", getattr(row, "organization_id", None)) if row is not None else None
def _budget_fields(row: object, spend_key: str, max_budget_key: str) -> Mapping[str, object]:
def _budget_fields(row: object, spend_key: str, max_budget_key: str, model_max_budget_key: str) -> Mapping[str, object]:
if row is None:
return {}
return {spend_key: getattr(row, "spend", None), max_budget_key: getattr(row, "max_budget", None)}
return {
spend_key: getattr(row, "spend", None),
max_budget_key: getattr(row, "max_budget", None),
model_max_budget_key: getattr(row, "model_max_budget", None),
}
def _key_budget_fields(key: object) -> Mapping[str, object]:
if key is None:
return {}
tier: Final = getattr(key, "litellm_budget_table", None)
return {
"user_api_key_spend": getattr(key, "spend", None),
"user_api_key_max_budget": _own_or_tier_budget(key, tier, "max_budget"),
"user_api_key_model_max_budget": _own_or_tier_budget(key, tier, "model_max_budget"),
}
def _own_or_tier_budget(key: object, tier: object, field: str) -> object:
own: Final = getattr(key, field, None)
tier_value: Final = getattr(tier, field, None)
return own if tier_value is None or own not in (None, {}) else tier_value

1
litellm/__init__.py
View file

@ -551,6 +551,7 @@ from litellm.litellm_core_utils.get_model_cost_map import get_model_cost_map, ma
model_cost = get_model_cost_map(url=model_cost_map_url)
cost_discount_config: Dict[str, float] = {} # Provider-specific cost discounts {"vertex_ai": 0.05} = 5% discount
apply_search_free_tier: bool = False
cost_margin_config: Dict[
str, Union[float, Dict[str, float]]
] = {} # Provider-specific or global cost margins. Examples:

2
litellm/a2a_protocol/litellm_completion_bridge/handler.py
View file

@ -80,7 +80,7 @@ class A2ACompletionBridgeHandler:
@staticmethod
def _build_completion_params(
params: dict[str, Any],
litellm_params: Mapping[str, Any],
litellm_params: Mapping[str, object],
api_base: str | None,
agent_extra_headers: Mapping[str, str] | None,
*,

4
litellm/a2a_protocol/main.py
View file

@ -602,9 +602,9 @@ def _build_streaming_logging_obj(
litellm_params: dict[str, object] | None,
metadata: dict[str, object] | None,
proxy_server_request: dict[str, object] | None,
start_time: datetime.datetime,
) -> Logging:
"""Build logging object for streaming A2A requests."""
start_time: Final = datetime.datetime.now()
model: Final = f"a2a_agent/{agent_name}"
logging_obj: Final = Logging(
@ -718,6 +718,7 @@ async def asend_message_streaming(
if request is None:
raise ValueError("request is required")
called_at: Final = datetime.datetime.now()
_raw_logging_obj: Final = kwargs.get("litellm_logging_obj")
logging_obj: Logging | None = _raw_logging_obj if isinstance(_raw_logging_obj, Logging) else None
@ -751,6 +752,7 @@ async def asend_message_streaming(
litellm_params=litellm_params,
metadata=metadata,
proxy_server_request=proxy_server_request,
start_time=called_at,
)
verbose_logger.info("A2A send_message_streaming request_id=%s, agent=%s", request.id, agent_name)

4
litellm/a2a_protocol/streaming_iterator.py
View file

@ -41,7 +41,7 @@ class A2AStreamingIterator:
self.request = request
self.logging_obj = logging_obj
self.agent_name = agent_name
self.start_time = datetime.now()
self.start_time = logging_obj.start_time
# Collect chunks for token counting
self.chunks: list[SendStreamingMessageResponse] = []
@ -54,6 +54,8 @@ class A2AStreamingIterator:
async def __anext__(self) -> "SendStreamingMessageResponse":
try:
chunk: Final = await self.stream.__anext__()
if self.logging_obj.completion_start_time is None:
self.logging_obj.update_completion_start_time(completion_start_time=datetime.now())
# Store chunk
self.chunks.append(chunk)

26
litellm/caching/dual_cache.py
View file

@ -59,16 +59,6 @@ class PendingBatchRead:
previous_access_times: dict[str, float | None]
@dataclass(frozen=True, slots=True)
class DeclaredBatchRead:
"""A ``async_batch_get_cache`` split in two: the memory half done, the Redis half declared on a ``RedisBatch``
so it rides that batch's next round trip, resolved later with ``async_resolve_batch_get``."""
keys: tuple[str, ...]
pending: PendingBatchRead
result: BatchResult[Mapping[str, object]] | None
def _log_deferred_increment_failure(future: asyncio.Future[float]) -> None:
if future.cancelled():
return
@ -382,22 +372,6 @@ class DualCache(BaseCache):
await self.in_memory_cache.async_set_cache(key, value, **self._backfill_kwargs(kwargs))
return merged
async def declare_batch_get(self, keys: Sequence[str], batch: RedisBatch) -> DeclaredBatchRead:
pending: Final = await self._prepare_batch_get(
list(keys),
local_only=False,
throttle_redis=False,
)
return DeclaredBatchRead(
keys=tuple(keys),
pending=pending,
result=batch.mget(pending.redis_keys) if pending.redis_keys else None,
)
async def async_resolve_batch_get(self, declared: DeclaredBatchRead) -> list[object | None]:
redis_result: Final = None if declared.result is None else await declared.result
return await self._apply_batch_get(declared.pending, redis_result)
async def async_batch_get_cache(
self,
keys: list,

47
litellm/caching/redis_cache.py
View file

@ -134,7 +134,12 @@ _INCREMENT_WITH_FLOOR_LUA: Final = (
"return count"
)
_INCREMENT_IF_PRESENT_LUA: Final = (
"if redis.call('EXISTS', KEYS[1]) == 0 then return false end return redis.call('INCRBYFLOAT', KEYS[1], ARGV[1])"
)
_LUA_COUNT: Final = TypeAdapter(int)
_LUA_OPTIONAL_TOTAL: Final[TypeAdapter[float | None]] = TypeAdapter(float | None)
_OPTIONAL_COUNTS: Final = TypeAdapter(tuple[int | None, ...])
@ -675,7 +680,7 @@ class RedisCache(BaseCache):
# redis namespaces
self.namespace = namespace
# for high traffic, we store the redis results in memory and then batch write to redis
self.redis_batch_writing_buffer: list = []
self.redis_batch_writing_buffer: list[tuple[str, object]] = []
if redis_flush_size is None:
self.redis_flush_size: int = 100
else:
@ -1568,6 +1573,46 @@ class RedisCache(BaseCache):
count: Final = await _redis_client.eval(_INCREMENT_WITH_FLOOR_LUA, 1, namespaced_key, value, ttl)
return _LUA_COUNT.validate_python(count)
@_redis_circuit_breaker_guard
async def async_increment_if_present(
self, key: str, value: float, parent_otel_span: Span | None = None
) -> float | None:
_redis_client: Final = self._async_commands()
namespaced_key: Final = self.check_and_fix_namespace(key=key)
start_time: Final = time.time()
try:
total: Final = _LUA_OPTIONAL_TOTAL.validate_python(
await _redis_client.eval(_INCREMENT_IF_PRESENT_LUA, 1, namespaced_key, value)
)
except Exception as e:
failed_at: Final = time.time()
asyncio.create_task(
self.service_logger_obj.async_service_failure_hook(
service=ServiceTypes.REDIS,
duration=failed_at - start_time,
error=e,
call_type="async_increment_if_present",
caller=_get_call_stack_info(),
start_time=start_time,
end_time=failed_at,
parent_otel_span=parent_otel_span,
)
)
raise
end_time: Final = time.time()
asyncio.create_task(
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=end_time - start_time,
call_type="async_increment_if_present",
caller=_get_call_stack_info(),
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
)
)
return total
async def flush_cache_buffer(self):
print_verbose(f"flushing to redis....reached size of buffer {len(self.redis_batch_writing_buffer)}")
await self.async_set_cache_pipeline(self.redis_batch_writing_buffer)

5
litellm/constants.py
View file

@ -1855,6 +1855,9 @@ SPEND_LOG_CLEANUP_BATCH_FAILURE_BACKOFF_SECONDS: Final = get_env_float(
SPEND_LOG_CLEANUP_RUN_BUDGET_SECONDS: Final = get_env_float("SPEND_LOG_CLEANUP_RUN_BUDGET_SECONDS", 300.0)
SPEND_LOG_CLEANUP_BATCH_TIMEOUT_SECONDS: Final = get_env_float("SPEND_LOG_CLEANUP_BATCH_TIMEOUT_SECONDS", 30.0)
SPEND_LOG_CLEANUP_REMAINING_COUNT_CAP: Final = get_env_int("SPEND_LOG_CLEANUP_REMAINING_COUNT_CAP", 100000)
SPEND_LOG_SIZE_CHECK_JOB_NAME: Final = "spend_log_size_check"
SPEND_LOG_SIZE_CHECK_INTERVAL: Final = os.getenv("SPEND_LOG_SIZE_CHECK_INTERVAL", "1d")
SPEND_LOG_SIZE_CLEANUP_RECHECK_BATCHES: Final = get_env_int("SPEND_LOG_SIZE_CLEANUP_RECHECK_BATCHES", 10)
SCHEDULED_JOB_SHUTDOWN_FINISH_TIMEOUT_SECONDS: Final = get_env_float(
"SCHEDULED_JOB_SHUTDOWN_FINISH_TIMEOUT_SECONDS", 5.0
)
@ -1976,6 +1979,7 @@ LITELLM_SETTINGS_SAFE_DB_OVERRIDES: Final = [
"public_model_groups",
"public_model_groups_links",
"cost_discount_config",
"apply_search_free_tier",
"cost_margin_config",
"block_requests_for_models_without_pricing",
"budget_exceeded_throttle_percentage",
@ -2316,6 +2320,7 @@ HARNESS_STDERR_TAIL_LINES: Final = 40
HARNESS_STREAM_READ_CHUNK_BYTES: Final = 64 * 1024
HARNESS_EVENT_QUEUE_MAX_SIZE: Final = 1024
HARNESS_PROCESS_KILL_GRACE_SECONDS: Final = 5.0
HARNESS_RESUME_COMMAND_TIMEOUT_SECONDS: Final = 30.0
HARNESS_SNAPSHOT_SKIP_DIRS: Final = frozenset(
{
".git",

26
litellm/cost_calculator.py
View file

@ -906,11 +906,15 @@ def _bedrock_pricing_info(
return model_info_with_pricing_overrides(profile, registration[1])
def _declares_rates(model_info: ModelInfo | None) -> bool:
entry: Final = _raw_cost_map_entry(model_info["key"]) if model_info is not None else None
def cost_map_key_prices_anything(key: str) -> bool:
entry: Final = _raw_cost_map_entry(key)
return entry is not None and _cost_map_entry_prices_anything(entry)
def _declares_rates(model_info: ModelInfo | None) -> bool:
return model_info is not None and cost_map_key_prices_anything(model_info["key"])
def _base_model_pricing_provider(base_model: str, custom_llm_provider: str | None) -> str | None:
"""The provider to price an explicit base_model under, when the deployment's own provider has
no rates for it (e.g. chatgpt/ subscription rows) but the base model's own row does"""
@ -1507,6 +1511,7 @@ def completion_cost(
litellm_model_name: str | None = None,
router_model_id: str | None = None,
litellm_logging_obj: LitellmLoggingObject | None = None,
pricing_provider: str | None = None,
### SERVICE TIER ###
service_tier: str | None = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
@ -1527,6 +1532,8 @@ def completion_cost(
"""
try:
call_type = infer_call_type(call_type, completion_response) or "completion"
if pricing_provider is not None:
custom_llm_provider = pricing_provider # rebind-ok: model_info.pricing_provider overrides the routing provider for cost-map lookups
if call_type == CallTypes.aresponses_websocket.value and isinstance(
completion_response, LiteLLMRealtimeStreamLoggingObject
@ -1552,6 +1559,7 @@ def completion_cost(
optional_params=optional_params,
custom_pricing=custom_pricing,
base_model=base_model,
pricing_provider=pricing_provider,
standard_built_in_tools_params=standard_built_in_tools_params,
litellm_model_name=litellm_model_name,
router_model_id=router_model_id,
@ -1642,10 +1650,10 @@ def completion_cost(
*(((model, request_pricing_provider),) if model is not None else ()),
)
potential_model_names: Final = (
raw_candidates if explicit_pricing else _priced_candidates_before_unpriced_rules(raw_candidates)
raw_candidates if custom_pricing is True else _priced_candidates_before_unpriced_rules(raw_candidates)
)
for idx, (model, pricing_provider) in enumerate(potential_model_names):
for idx, (model, candidate_pricing_provider) in enumerate(potential_model_names):
try:
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug("selected model name for cost calculation: %s", model)
@ -1712,7 +1720,9 @@ def completion_cost(
hidden_params = getattr(completion_response, "_hidden_params", None)
if hidden_params is not None:
custom_llm_provider = hidden_params.get("custom_llm_provider", custom_llm_provider or None)
custom_llm_provider = pricing_provider or hidden_params.get(
"custom_llm_provider", custom_llm_provider or None
)
region_name = hidden_params.get("region_name", region_name)
else:
@ -1998,7 +2008,7 @@ def completion_cost(
prompt_tokens=prompt_tokens or 0,
completion_tokens=completion_tokens or 0,
custom_llm_provider=_pricing_provider_for_cost_candidate(
candidate_provider=pricing_provider,
candidate_provider=candidate_pricing_provider,
request_provider=request_pricing_provider,
current_provider=custom_llm_provider,
),
@ -2102,7 +2112,7 @@ def completion_cost(
_token_type_breakdown = get_token_type_cost_breakdown(
model=model,
custom_llm_provider=_pricing_provider_for_cost_candidate(
candidate_provider=pricing_provider,
candidate_provider=candidate_pricing_provider,
request_provider=request_pricing_provider,
current_provider=custom_llm_provider,
),
@ -2225,6 +2235,7 @@ def response_cost_calculator(
litellm_model_name: str | None = None,
router_model_id: str | None = None,
litellm_logging_obj: LitellmLoggingObject | None = None,
pricing_provider: str | None = None,
### SERVICE TIER ###
service_tier: str | None = None, # for OpenAI service tier pricing
### DATA RESIDENCY ###
@ -2265,6 +2276,7 @@ def response_cost_calculator(
litellm_model_name=litellm_model_name,
router_model_id=router_model_id,
litellm_logging_obj=litellm_logging_obj,
pricing_provider=pricing_provider,
service_tier=service_tier,
data_residency=data_residency,
vertex_location=vertex_location,

43
litellm/decisions/call.py
View file

@ -11,6 +11,7 @@ from pydantic_core import to_jsonable_python
from typing_extensions import assert_never
import litellm
from litellm.cost_calculator import cost_map_key_prices_anything
from litellm.litellm_core_utils.core_helpers import RESPONSE_COST_HEADER, normalize_drop_params
from litellm.litellm_core_utils.get_provider_specific_headers import ProviderSpecificHeaderUtils
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -21,6 +22,7 @@ from litellm.litellm_core_utils.prompt_templates.image_handling import (
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.decisions.transformation import (
BaseDecisionsConfig,
DecisionsResponseValidationError,
image_media_type,
image_param,
input_images,
@ -427,7 +429,7 @@ def _jsonable_fallback(value: object) -> object:
def _logged_request_message(caller_request: object) -> ChatCompletionUserMessage:
caller_fields: Final = _LOGGED_FIELDS_ADAPTER.validate_python(
to_jsonable_python(caller_request, fallback=_jsonable_fallback)
to_jsonable_python(caller_request, fallback=_jsonable_fallback, bytes_mode="base64")
)
sent_fields: Final = json.dumps(
dict((name, value) for name, value in caller_fields.items() if value not in (None, [], {}))
@ -687,6 +689,13 @@ def _request_body(call: DecisionsCall) -> Mapping[str, object]:
return body
def _pricing_model(response: DecisionsIRResponse, call: DecisionsCall) -> str:
returned: Final = get_model_cost_key(f"{call.custom_llm_provider}/{response.model}")
if returned is not None and cost_map_key_prices_anything(returned):
return returned
return f"{call.custom_llm_provider}/{call.model}"
def _format_response(response: DecisionsIRResponse, call: DecisionsCall) -> DecisionsResponse | OpenAIDecisionResponse:
formatted: Final = _formatted_response(response, call)
provider_cost: Final = call.provider_config.provider_reported_cost(response)
@ -694,10 +703,7 @@ def _format_response(response: DecisionsIRResponse, call: DecisionsCall) -> Deci
{
"model": f"{call.custom_llm_provider}/{call.model}",
"custom_llm_provider": call.custom_llm_provider,
"provider_response_model": (
get_model_cost_key(f"{call.custom_llm_provider}/{response.model}")
or f"{call.custom_llm_provider}/{call.model}"
),
"provider_response_model": _pricing_model(response, call),
**({"additional_headers": {RESPONSE_COST_HEADER: provider_cost}} if provider_cost is not None else {}),
}
)
@ -727,18 +733,21 @@ def _formatted_response(
def _map_upstream_exception(error: Exception, call: DecisionsCall) -> Exception:
if isinstance(error, BaseLLMException) and error.status_code_is_synthesized:
provider_label: Final = f"{call.custom_llm_provider[0].upper()}{call.custom_llm_provider[1:]}Exception"
return litellm.APIConnectionError(
message=f"{provider_label} - {error.message}",
llm_provider=call.custom_llm_provider,
model=f"{call.custom_llm_provider}/{call.model}",
)
return litellm.exception_type(
model=f"{call.custom_llm_provider}/{call.model}",
custom_llm_provider=call.custom_llm_provider,
original_exception=error,
)
provider_label: Final = f"{call.custom_llm_provider[0].upper()}{call.custom_llm_provider[1:]}Exception"
model: Final = f"{call.custom_llm_provider}/{call.model}"
match error:
case DecisionsResponseValidationError():
return litellm.InternalServerError(
message=f"{provider_label} - {error.message}", llm_provider=call.custom_llm_provider, model=model
)
case BaseLLMException(status_code_is_synthesized=True):
return litellm.APIConnectionError(
message=f"{provider_label} - {error.message}", llm_provider=call.custom_llm_provider, model=model
)
case _:
return litellm.exception_type(
model=model, custom_llm_provider=call.custom_llm_provider, original_exception=error
)
async def asend(call: DecisionsCall) -> DecisionsResponse | OpenAIDecisionResponse:

4
litellm/google_genai/streaming_iterator.py
View file

@ -71,7 +71,7 @@ class BaseGoogleGenAIGenerateContentStreamingIterator:
):
self.litellm_logging_obj = litellm_logging_obj
self.request_body = request_body
self.start_time = datetime.now()
self.start_time = litellm_logging_obj.start_time
self.collected_chunks: list[bytes] = []
self.model = model
self.custom_llm_provider = custom_llm_provider
@ -192,6 +192,8 @@ class AsyncGoogleGenAIGenerateContentStreamingIterator(BaseGoogleGenAIGenerateCo
async def __anext__(self):
try:
chunk: Final = await _anext_google_genai_sse_chunk(self.stream_iterator)
if self.litellm_logging_obj.completion_start_time is None:
self.litellm_logging_obj.update_completion_start_time(completion_start_time=datetime.now())
self.collected_chunks.append(chunk)
return chunk
except StopAsyncIteration:

86
litellm/harness/handlers/cli_handler.py
View file

@ -12,17 +12,27 @@ from __future__ import annotations
import asyncio
import os
from collections import deque
from collections.abc import AsyncIterator, Sequence
from collections.abc import AsyncIterator, Mapping, Sequence
from dataclasses import dataclass
from typing import Final
from litellm._logging import verbose_logger
from litellm.constants import HARNESS_STDERR_TAIL_LINES, HARNESS_STREAM_READ_CHUNK_BYTES
from litellm.constants import (
HARNESS_RESUME_COMMAND_TIMEOUT_SECONDS,
HARNESS_STDERR_TAIL_LINES,
HARNESS_STREAM_READ_CHUNK_BYTES,
)
from litellm.harness.context import SessionContext
from litellm.harness.errors import HarnessInstallFailed, SandboxError
from litellm.harness.errors import HarnessError, HarnessInstallFailed, SandboxError
from litellm.harness.handlers.base import BaseHarnessHandler
from litellm.harness.sandbox.base import Process, Sandbox
from litellm.harness.sandbox.base import CompletedRun, Process, Sandbox
from litellm.harness.types import Event
from litellm.llms.base_llm.harness.transformation import BaseCLIHarnessConfig, HarnessSessionSetup
from litellm.llms.base_llm.harness.transformation import (
BaseCLIHarnessConfig,
HarnessSessionCopy,
HarnessSessionSetup,
HarnessTurnError,
)
from litellm.llms.base_llm.harness.utils import decode_json_line, read_skill_files
# Link <private_dir>/<dir> to a LiteLLM-owned cache dir so a later session can resume.
@ -81,6 +91,52 @@ async def persist_dir(sandbox: Sandbox, link_path: str, cache_subpath: str) -> N
)
@dataclass(frozen=True, slots=True)
class ResumeFailed:
reason: str
async def bounded_resume_run(
sandbox: Sandbox, cmd: Sequence[str], env: Mapping[str, str]
) -> CompletedRun | ResumeFailed:
try:
run: Final = await sandbox.run(cmd, env=env, timeout=HARNESS_RESUME_COMMAND_TIMEOUT_SECONDS)
except SandboxError as exc:
return ResumeFailed(str(exc))
return run if run.exit_code == 0 else ResumeFailed(run.stderr.strip() or f"exit code {run.exit_code}")
async def import_session_copy(sandbox: Sandbox, copy: HarnessSessionCopy, env: Mapping[str, str]) -> str | ResumeFailed:
try:
await sandbox.write(copy.path, copy.data)
except SandboxError as exc:
return ResumeFailed(str(exc))
imported: Final = await bounded_resume_run(sandbox, copy.import_command, env)
return imported if isinstance(imported, ResumeFailed) else copy.session_id
async def resumable_session_id(
config: BaseCLIHarnessConfig[object, object],
sandbox: Sandbox,
setup_env: Mapping[str, str],
private_dir: str,
native_session_id: str,
) -> str | ResumeFailed:
export_cmd: Final = config.transform_resume_export_command(native_session_id)
if export_cmd is None:
return native_session_id
exported: Final = await bounded_resume_run(sandbox, export_cmd, setup_env)
if isinstance(exported, ResumeFailed):
return exported
try:
copy: Final = config.transform_resume_copy(exported.stdout, private_dir, sandbox.workdir)
except HarnessError as exc:
return ResumeFailed(str(exc))
if copy is None:
return native_session_id
return await import_session_copy(sandbox, copy, setup_env)
async def copy_skills(sandbox: Sandbox, skills: Sequence[str], skills_root: str) -> None:
for skill in skills:
name = os.path.basename(os.path.realpath(os.fspath(skill)))
@ -89,13 +145,14 @@ async def copy_skills(sandbox: Sandbox, skills: Sequence[str], skills_root: str)
class CLIHarnessHandler(BaseHarnessHandler):
config: BaseCLIHarnessConfig
config: BaseCLIHarnessConfig[object, object]
def __init__(self, config: BaseCLIHarnessConfig) -> None:
super().__init__(config)
self._private_dir: str | None = None
self._setup: HarnessSessionSetup | None = None
self._native_id: str | None = None
self._pending_resume: str | None = None
self._proc: Process | None = None
async def start(self, ctx: SessionContext) -> None:
@ -119,6 +176,9 @@ class CLIHarnessHandler(BaseHarnessHandler):
async def turn(self, ctx: SessionContext, prompt: str) -> AsyncIterator[Event]:
if self._setup is None or self._private_dir is None:
raise RuntimeError("CLIHarnessHandler.turn() called before start()")
if self._pending_resume is not None:
self._native_id = await self._prepared_session_id(ctx, self._setup, self._private_dir, self._pending_resume)
self._pending_resume = None
request = self.config.transform_turn_request(ctx, self._setup, self._private_dir, prompt, self._native_id)
argv: Final = list(request.argv) # mutable-ok: Sandbox.exec takes list[str]
proc = await ctx.sandbox.exec(argv, env=request.env, cwd=request.cwd)
@ -158,4 +218,16 @@ class CLIHarnessHandler(BaseHarnessHandler):
return self._native_id
async def resume(self, ctx: SessionContext, native_session_id: str) -> None:
self._native_id = native_session_id
if native_session_id != self._native_id:
self._native_id = native_session_id
self._pending_resume = native_session_id
async def _prepared_session_id(
self, ctx: SessionContext, setup: HarnessSessionSetup, private_dir: str, saved_id: str
) -> str:
target: Final = await resumable_session_id(self.config, ctx.sandbox, setup.env, private_dir, saved_id)
match target:
case ResumeFailed(reason=reason):
raise HarnessTurnError(f"could not prepare session {saved_id} for resume: {reason}")
case session_id:
return session_id

2
litellm/harness/handlers/deepagents_handler.py
View file

@ -183,7 +183,7 @@ class DeepAgentsHandler(BaseHarnessHandler):
break
resume: dict[str, object] = {} # mutable-ok: Command(resume=) payload, filled per answered approval
for interrupt in state.interrupts:
decisions: list[dict[str, Any]] = [] # mutable-ok: HITL decisions collected across awaited approvals
decisions: list[dict[str, str]] = [] # mutable-ok: HITL decisions collected across awaited approvals
for request in approval_requests(getattr(interrupt, "value", None)):
approval = Approval(
tool=normalized_tool_name(str(request.get("name") or "")),

16
litellm/harness/sandbox/local.py
View file

@ -14,7 +14,7 @@ from typing import Final
from litellm.constants import HARNESS_PROCESS_KILL_GRACE_SECONDS
from litellm.harness.errors import SandboxError
from litellm.harness.sandbox.base import CompletedRun
from litellm.harness.sandbox.base import CompletedRun, Process
from litellm.harness.sandbox.snapshot import snapshot_local
_SECRET_PREFIXES: Final = (
@ -103,7 +103,7 @@ class SubprocessHandle:
await self._proc.wait()
async def _read_all(handle: SubprocessHandle) -> tuple[bytes, bytes, int]:
async def _read_all(handle: Process) -> tuple[bytes, bytes, int]:
if handle.stdin is not None:
handle.stdin.close()
stdout, stderr = await asyncio.gather(handle.stdout.read(), handle.stderr.read())
@ -111,13 +111,16 @@ async def _read_all(handle: SubprocessHandle) -> tuple[bytes, bytes, int]:
return stdout, stderr, exit_code
async def collect_output(handle: SubprocessHandle, cmd: Sequence[str], timeout: float | None) -> CompletedRun:
"""Close stdin, read stdout/stderr to EOF; kill and raise SandboxError on timeout."""
async def collect_output(handle: Process, cmd: Sequence[str], timeout: float | None) -> CompletedRun:
"""Close stdin, read stdout/stderr to EOF; kill on cancel, and kill and raise SandboxError on timeout."""
try:
stdout, stderr, code = await asyncio.wait_for(_read_all(handle), timeout)
except asyncio.TimeoutError:
await handle.kill()
raise SandboxError(f"command timed out after {timeout}s: {cmd[0]}")
except asyncio.CancelledError:
await handle.kill()
raise
return CompletedRun(
stdout=stdout.decode("utf-8", errors="replace"),
stderr=stderr.decode("utf-8", errors="replace"),
@ -180,14 +183,15 @@ class LocalSandbox:
self._check_open()
if not cmd:
raise SandboxError("exec() needs a non-empty command")
work_dir: Final = self._resolve_cwd(cwd)
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
cwd=self._resolve_cwd(cwd),
env=self.child_env(env),
cwd=work_dir,
env=self.child_env(MappingProxyType({**(env or {}), "PWD": work_dir})),
start_new_session=True,
)
except (FileNotFoundError, PermissionError) as exc:

21
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -342,6 +342,20 @@ EXTERNAL_BREAKPOINTS_STAMP: Final = "_litellm_external_breakpoints"
CUSTOM_LLM_PROVIDER_STAMP: Final = "_litellm_custom_llm_provider"
def _accepts_cache_control(part: object) -> bool:
match part:
case {"type": "thinking" | "redacted_thinking"}:
return False
case dict():
return True
case _:
return False
def _last_index_accepting_cache_control(parts: Sequence[object]) -> int | None:
return next((index for index in reversed(range(len(parts))) if _accepts_cache_control(parts[index])), None)
class AnthropicCacheControlHook(CustomPromptManagement):
@staticmethod
def _request_value(request_kwargs: object, key: str) -> object:
@ -740,10 +754,11 @@ class AnthropicCacheControlHook(CustomPromptManagement):
# 1. if string, insert cache control in the message
if isinstance(message_content, str):
message["cache_control"] = control
# 2. list of objects - only apply to last item per Anthropic spec
# 2. list of objects - only apply to the last item Claude accepts a breakpoint on
elif isinstance(message_content, list):
if len(message_content) > 0 and isinstance(message_content[-1], dict):
message_content[-1]["cache_control"] = control # pyright: ignore[reportGeneralTypeIssues] # loose runtime dict
markable_index: Final = _last_index_accepting_cache_control(message_content)
if markable_index is not None:
message_content[markable_index]["cache_control"] = control # pyright: ignore[reportGeneralTypeIssues] # loose runtime dict
return message
@staticmethod

13
litellm/integrations/arize/_utils.py
View file

@ -884,7 +884,7 @@ def _set_session_and_user_attrs(span: "Span", kwargs: dict, standard_logging_pay
safe_set_attribute(span, "litellm.key_alias", str(key_alias))
def _set_response_cost_attr(span: "Span", standard_logging_payload) -> None:
def _set_response_cost_attr(span: "Span", standard_logging_payload: Mapping[str, object] | None) -> None:
"""Emit cost attributes from the StandardLoggingPayload when present.
Uses the OpenInference `llm.cost.total` key so Arize / Phoenix can
@ -892,15 +892,22 @@ def _set_response_cost_attr(span: "Span", standard_logging_payload) -> None:
single total in `StandardLoggingPayload.response_cost`, so we cannot
split it into prompt/completion. We also keep the legacy
`llm.response.cost` key for back-compat with any consumer querying it.
When the price lookup failed, the call's total is unknown, so no cost is
emitted and the backend does not record the call as free. A guardrail fee
in `response_cost` stays on the guardrail span, as in OTel v2.
"""
if not isinstance(standard_logging_payload, dict):
return
cost: Final = standard_logging_payload.get("response_cost")
if cost is None:
cost_lookup_failed: Final = (
standard_logging_payload.get("response_cost_failure_debug_info") is not None
and standard_logging_payload.get("cache_hit") is not True
)
if not isinstance(cost, (int, float, str)) or cost_lookup_failed:
return
try:
cost_value: Final = float(cost)
except (TypeError, ValueError):
except ValueError:
return
safe_set_attribute(span, "llm.cost.total", cost_value)
safe_set_attribute(span, "llm.response.cost", cost_value)

24
litellm/integrations/datadog/datadog_llm_obs.py
View file

@ -12,7 +12,7 @@ import os
from collections.abc import Mapping, Sequence
from datetime import datetime
from types import MappingProxyType
from typing import Any, Final, Literal
from typing import Final, Literal
import httpx
from pydantic import ConfigDict, TypeAdapter, ValidationError
@ -54,6 +54,7 @@ from litellm.types.utils import (
PROMPT_CARRYING_GUARDRAIL_FIELDS,
PROMPT_QUOTING_ROUTING_DECISION_FIELDS,
CallTypes,
StandardLoggingMetadata,
StandardLoggingPayload,
StandardLoggingPayloadErrorInformation,
)
@ -93,8 +94,12 @@ _ROUTER_DIMENSIONS: Final[tuple[str, ...]] = ("router_tier", "router_cause", "ro
_COST_DIMENSIONS: Final[tuple[str, ...]] = ("team", "user", "key_alias", "model_group", *_ROUTER_DIMENSIONS)
def _metadata_of(standard_logging_payload: StandardLoggingPayload) -> Mapping[str, Any]:
metadata: Final = standard_logging_payload.get("metadata")
def _logging_metadata_of(standard_logging_payload: StandardLoggingPayload) -> StandardLoggingMetadata | None:
return standard_logging_payload.get("metadata")
def _metadata_of(standard_logging_payload: StandardLoggingPayload) -> Mapping[str, object]:
metadata: Final = _logging_metadata_of(standard_logging_payload)
return metadata or _EMPTY_MAPPING
@ -1057,14 +1062,23 @@ class DataDogLLMObsLogger(CustomBatchLogger):
# Get budget information from metadata
metadata: Final = _metadata_of(standard_logging_payload)
logging_metadata: Final = _logging_metadata_of(standard_logging_payload)
# API key max budget
user_api_key_max_budget: Final = metadata.get("user_api_key_max_budget")
user_api_key_max_budget: Final = (
logging_metadata["user_api_key_max_budget"]
if logging_metadata and "user_api_key_max_budget" in logging_metadata
else None
)
if user_api_key_max_budget is not None:
spend_metrics["user_api_key_max_budget"] = float(user_api_key_max_budget)
# API key spend
user_api_key_spend: Final = metadata.get("user_api_key_spend")
user_api_key_spend: Final = (
logging_metadata["user_api_key_spend"]
if logging_metadata and "user_api_key_spend" in logging_metadata
else None
)
if user_api_key_spend is not None:
try:
spend_metrics["user_api_key_spend"] = float(user_api_key_spend)

15
litellm/integrations/langfuse/langfuse_otel.py
View file

@ -1,7 +1,7 @@
import base64
import json
import os
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, Optional
@ -9,6 +9,7 @@ from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
from litellm.integrations.langfuse.langfuse_otel_attributes import (
LangfuseLLMObsOTELAttributes,
serialize_langfuse_datetime,
)
from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
@ -55,7 +56,7 @@ class LangfuseOtelLogger(OpenTelemetry):
LangfuseOtelLogger._set_langfuse_specific_attributes(span=span, kwargs=kwargs, response_obj=response_obj)
@staticmethod
def _extract_langfuse_metadata(kwargs: dict) -> dict:
def _extract_langfuse_metadata(kwargs: Mapping[str, Any]) -> dict:
"""
Extracts Langfuse metadata from the standard LiteLLM kwargs structure.
@ -221,7 +222,7 @@ class LangfuseOtelLogger(OpenTelemetry):
)
@staticmethod
def _set_langfuse_specific_attributes(span: Span, kwargs, response_obj):
def _set_langfuse_specific_attributes(span: Span, kwargs: Mapping[str, Any], response_obj):
"""
Sets Langfuse specific metadata attributes onto the OTEL span.
@ -255,6 +256,14 @@ class LangfuseOtelLogger(OpenTelemetry):
safe_dumps(messages),
)
completion_start_time: Final = kwargs.get("completion_start_time")
if isinstance(completion_start_time, datetime):
safe_set_attribute(
span,
LangfuseSpanAttributes.OBSERVATION_COMPLETION_START_TIME.value,
serialize_langfuse_datetime(completion_start_time),
)
LangfuseOtelLogger._set_observation_output(span=span, response_obj=response_obj)
@staticmethod

5
litellm/integrations/langfuse/langfuse_otel_attributes.py
View file

@ -6,6 +6,7 @@ Relevant Issue: https://github.com/BerriAI/litellm/issues/13764
import json
from collections.abc import Mapping
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final
from pydantic import BaseModel
@ -29,6 +30,10 @@ if TYPE_CHECKING:
from opentelemetry.trace import Span
def serialize_langfuse_datetime(value: datetime) -> str:
return json.dumps(value.astimezone().isoformat())
def get_output_content_by_type(
response_obj: None
| dict

3
litellm/integrations/langfuse/langfuse_sdk.py
View file

@ -39,6 +39,7 @@ from pydantic import ConfigDict
import litellm
from litellm._logging import verbose_logger
from litellm.integrations.langfuse.langfuse import PROMPT_CACHE_TTL_ENV, parse_langfuse_debug, whole_number
from litellm.integrations.langfuse.langfuse_otel_attributes import serialize_langfuse_datetime
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_httpx_client
from litellm.types.llms.base import LiteLLMBaseModel
@ -168,7 +169,7 @@ def _string_or_none(value: object) -> str | None:
def _serialize_datetime(value: object) -> str | None:
"""A datetime the way the SDK's ``EventSerializer`` sends one: a JSON string, naive values read as local time."""
if isinstance(value, datetime):
return safe_dumps(value.astimezone().isoformat())
return serialize_langfuse_datetime(value)
return _serialize(value)

2
litellm/integrations/otel/README.md
View file

@ -91,7 +91,7 @@ way (`pod_lock`, `budget_reset`, `spend_queue`, `health_check`, `scheduler_queue
rather than a bare `redis.set`; `tests/unit/test_internal_context.py` scans every
module under `litellm/` and `enterprise/` that calls a shared cache or declares a
read or write on the request batch (`reserve_redis_batch_reads`,
`declare_batch_get`, `batch.mget`, `batch.set`, `batch.script`) and fails when
`batch.mget`, `batch.set`, `batch.script`) and fails when
one has no declared family, with the process-local `InMemoryCache` callers listed
as the only exemptions. A batch op carries the family that was active when it was
declared, so the routing prefetch armed before deployment selection

1
litellm/integrations/otel/logger.py
View file

@ -594,6 +594,7 @@ class OpenTelemetryV2(CustomLogger):
payload,
capture_content=self._capture_span_content(),
time_to_first_chunk_seconds=call.time_to_first_chunk_seconds,
completion_start_seconds=call.completion_start_seconds,
request_route=request_root_http_route(),
request_purpose=call.purpose,
trace=call.trace,

7
litellm/integrations/otel/mappers/langfuse.py
View file

@ -12,8 +12,10 @@ the JSON-serialized payloads. ``trace_attributes`` maps the caller's trace contr
import json
from collections.abc import Callable
from datetime import datetime, timezone
from typing import Final
from litellm.integrations.langfuse.langfuse_otel_attributes import serialize_langfuse_datetime
from litellm.integrations.otel.mappers.base import AttributeMap, AttrValue, SpanData
from litellm.integrations.otel.mappers.utils import (
collect,
@ -45,6 +47,11 @@ class LangfuseMapper:
"langfuse.observation.id": lambda d: d.identity.call_id or None,
"langfuse.trace.metadata.team_id": lambda d: d.identity.team_id or None,
"langfuse.trace.metadata.team_alias": lambda d: d.identity.team_alias or None,
"langfuse.observation.completion_start_time": lambda d: (
serialize_langfuse_datetime(datetime.fromtimestamp(d.completion_start_seconds, tz=timezone.utc))
if d.completion_start_seconds is not None
else None
),
}
# Sub-tables folded into their respective JSON blobs.

2
litellm/integrations/otel/model/metadata.py
View file

@ -258,6 +258,7 @@ class LLMCallEvent:
# needs to be reasonable for a span that never gets closed (a leak).
provisional_span_name: str
time_to_first_chunk_seconds: float | None
completion_start_seconds: float | None
trace: TraceControls
session_id: str | None
@ -280,6 +281,7 @@ class LLMCallEvent:
cache_hit=bool(payload.get("cache_hit")) if payload else False, # pyright: ignore[reportUnknownMemberType] # TypedDict .get on a str key is partially unknown
provisional_span_name=f"{operation.value} {model}".strip(),
time_to_first_chunk_seconds=time_to_first_chunk_seconds(kwargs),
completion_start_seconds=_epoch_seconds(kwargs.get("completion_start_time")),
trace=trace,
session_id=caller_session_id(kwargs, trace),
)

6
litellm/integrations/otel/model/payloads.py
View file

@ -429,6 +429,7 @@ class LLMCallSpanData:
choices_out: tuple[Mapping[str, object], ...] = ()
system_fingerprint: str | None = None
time_to_first_chunk_seconds: float | None = None
completion_start_seconds: float | None = None
# The requested output modality, set only on the routes that pin one (image
# generation, speech, transcription, OCR), and the litellm route itself, which
# keeps routes the convention folds into one operation distinguishable.
@ -450,6 +451,7 @@ class LLMCallSpanData:
payload: StandardLoggingPayload,
capture_content: bool = False,
time_to_first_chunk_seconds: float | None = None,
completion_start_seconds: float | None = None,
request_route: str | None = None,
request_purpose: str | None = None,
trace: TraceControls | None = None,
@ -481,6 +483,7 @@ class LLMCallSpanData:
embedding_output: Final = (
EmbeddingOutput.from_response(response) if operation is GenAIOperation.EMBEDDINGS else None
)
cost_lookup_failed: Final = payload.get("response_cost_failure_debug_info") is not None and not cache_hit
return cls(
operation=operation,
provider=resolve_provider(as_str(payload.get("custom_llm_provider"))),
@ -491,7 +494,7 @@ class LLMCallSpanData:
usage=LLMUsage.from_standard_logging_payload(payload),
finish_reasons=finish_reasons,
error=_parse_error(payload),
response_cost=as_float(payload.get("response_cost")),
response_cost=None if cost_lookup_failed else as_float(payload.get("response_cost")),
cost=LLMCost.from_breakdown(cast("Mapping[str, object] | None", payload.get("cost_breakdown"))),
server=None if cache_hit else ServerInfo.from_api_base(context.api_base),
identity=context.identity,
@ -502,6 +505,7 @@ class LLMCallSpanData:
choices_out=choices_out if capture_content else (),
system_fingerprint=as_str(response.get("system_fingerprint")),
time_to_first_chunk_seconds=time_to_first_chunk_seconds,
completion_start_seconds=completion_start_seconds,
output_type=resolve_output_type(call_type),
call_type=call_type or None,
request_route=request_route or context.identity.request_route,

1
litellm/integrations/otel/model/spans.py
View file

@ -214,6 +214,7 @@ _SERVICE_VERB_BY_CALL_TYPE: Final[dict[str, str]] = {
"increment_cache": "incr",
"async_increment": "incr",
"async_increment_pipeline": "incr",
"async_increment_if_present": "incr",
"delete_cache": "delete",
"async_delete_cache": "delete",
"async_rpush": "rpush",

11
litellm/interactions/streaming_iterator.py
View file

@ -11,6 +11,7 @@ from datetime import datetime
from typing import Any, Final
import httpx
from pydantic import ConfigDict, TypeAdapter
from litellm._logging import verbose_logger
from litellm.constants import STREAM_SSE_DONE_STRING
@ -25,6 +26,10 @@ from litellm.types.interactions import (
)
from litellm.utils import CustomStreamWrapper
_INTERACTIONS_STREAMING_CHUNK_ADAPTER: Final = TypeAdapter(
object, config=ConfigDict(strict=True, hide_input_in_errors=True)
)
class BaseInteractionsAPIStreamingIterator:
"""
@ -48,7 +53,7 @@ class BaseInteractionsAPIStreamingIterator:
self.finished = False
self.interactions_api_config = interactions_api_config
self.completed_response: InteractionsAPIStreamingResponse | None = None
self.start_time = datetime.now()
self.start_time = logging_obj.start_time
# set request kwargs
self.litellm_metadata = litellm_metadata
@ -83,10 +88,12 @@ class BaseInteractionsAPIStreamingIterator:
try:
# Parse the JSON chunk
parsed_chunk: Final = json.loads(stripped_chunk)
parsed_chunk: Final = _INTERACTIONS_STREAMING_CHUNK_ADAPTER.validate_python(json.loads(stripped_chunk))
# Format as InteractionsAPIStreamingResponse
if isinstance(parsed_chunk, dict):
if self.logging_obj.completion_start_time is None:
self.logging_obj.update_completion_start_time(completion_start_time=datetime.now())
streaming_response: Final = self.interactions_api_config.transform_streaming_response(
model=self.model,
parsed_chunk=parsed_chunk,

23
litellm/litellm_core_utils/cloud_storage_security.py
View file

@ -1,6 +1,7 @@
import posixpath
import re
from collections.abc import Mapping, Sequence
from itertools import accumulate, repeat
from types import MappingProxyType
from typing import Any, Final, cast
from urllib.parse import quote, unquote
@ -20,15 +21,33 @@ MANAGED_CLOUD_STORAGE_SCHEMES: Final = ("s3://", "gs://")
_MAPPING_PROXY_TYPE: Final[type] = type(MappingProxyType({}))
MAX_FILE_ID_DECODE_PASSES: Final = 8
def _unquote_once(current: str, _: str) -> str:
return unquote(current)
def _decodings(value: str) -> tuple[str, ...]:
return tuple(accumulate(repeat(value, MAX_FILE_ID_DECODE_PASSES + 2), _unquote_once))
def is_managed_cloud_storage_uri(file_id: str) -> bool:
"""
True if file_id is a raw cloud-storage object URI (e.g. ``s3://bucket/key``).
True if file_id is a raw cloud-storage object URI (e.g. ``s3://bucket/key``),
up to ``MAX_FILE_ID_DECODE_PASSES`` layers of percent-encoding deep, or an id
encoded deeper than that, which no client produces and which is treated as raw
rather than decoded any further.
These are internal provider artifacts. On the multi-tenant proxy they must be
retrieved through their managed unified file id so owner/team access is enforced;
a raw URI supplied by a caller bypasses that check.
"""
return isinstance(file_id, str) and file_id.startswith(MANAGED_CLOUD_STORAGE_SCHEMES)
if not isinstance(file_id, str):
return False
decodings: Final = _decodings(file_id)
settled: Final = decodings[-1] == decodings[-2]
return decodings[-1].startswith(MANAGED_CLOUD_STORAGE_SCHEMES) or not settled
_SAFE_OBJECT_COMPONENT_PATTERN: Final = re.compile(r"[^A-Za-z0-9._-]+")

1
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -27,6 +27,7 @@ AWS_CREDENTIAL_KWARGS_KEYS: Final = frozenset(
"aws_external_id",
"aws_session_tags",
"aws_bedrock_runtime_endpoint",
"aws_bedrock_control_plane_endpoint",
"aws_bedrock_project_id",
}
)

61
litellm/litellm_core_utils/litellm_logging.py
View file

@ -117,7 +117,8 @@ from litellm.litellm_core_utils.ptu_pricing import is_spilled_over_ptu_request
from litellm.litellm_core_utils.redact_messages import (
redact_message_input_output_from_custom_logger,
redact_message_input_output_from_logging,
redact_streaming_responses_for_custom_logger,
redacted_model_call_details_for_custom_logger,
redacted_response_obj_for_custom_logger,
should_redact_message_logging,
)
from litellm.litellm_core_utils.served_output_texts import (
@ -337,6 +338,7 @@ def _get_provider_request_id(original_exception: Exception) -> str | None:
# Cache custom pricing keys as frozenset for O(1) lookups instead of looping through 49 keys
_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
_METADATA_MAPPING: Final = TypeAdapter(Mapping[str, object])
_MODEL_INFO_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = _CUSTOM_PRICING_KEYS | DEPLOYMENT_SCOPED_PRICING_FIELDS
_UNSERIALIZABLE_METADATA_KEYS: Final[frozenset[str]] = frozenset(
("user_api_key_auth", "user_api_key_budget_reservation")
@ -996,6 +998,21 @@ class Logging(LiteLLMLoggingBaseClass):
return model_id
return None
def get_router_pricing_provider(self) -> str | None:
"""The deployment's ``model_info.pricing_provider``, the provider its cost-map lookups run under."""
litellm_params: Final = cast( # cast-ok: callback metadata is caller-provided
Mapping[str, object] | None, getattr(self, "litellm_params", None)
)
if not litellm_params:
return None
candidates: Final = (
_METADATA_MAPPING.validate_python(
_METADATA_MAPPING.validate_python(litellm_params.get(key) or {}).get("model_info") or {}
).get("pricing_provider")
for key in ("litellm_metadata", "metadata")
)
return next((provider for provider in candidates if isinstance(provider, str) and provider), None)
def _routed_deployment_model(self) -> str | None:
litellm_params: Final = self._model_call_details_view.get("litellm_params")
metadata: Final = (
@ -1940,6 +1957,8 @@ class Logging(LiteLLMLoggingBaseClass):
used for consistent cost calculation across response headers + logging integrations.
"""
self.model_call_details.pop("response_cost_failure_debug_information", None)
if cache_hit is None:
cache_hit = self.model_call_details.get("cache_hit", False)
@ -1989,7 +2008,7 @@ class Logging(LiteLLMLoggingBaseClass):
router_model_id = self.get_router_model_id()
## RESPONSE COST ##
custom_pricing: Final = self._custom_pricing_for(priced_result)
custom_pricing: Final = self.custom_pricing_for(priced_result)
prompt: Final = self._prompt_for_cost_calculation()
@ -2019,6 +2038,7 @@ class Logging(LiteLLMLoggingBaseClass):
"prompt": prompt,
"standard_built_in_tools_params": self.standard_built_in_tools_params,
"router_model_id": router_model_id,
"pricing_provider": self.get_router_pricing_provider(),
"litellm_logging_obj": self,
"data_residency": (
self.litellm_params.get("data_residency")
@ -2100,15 +2120,17 @@ class Logging(LiteLLMLoggingBaseClass):
previous_cost_breakdown: Final = self.cost_breakdown
previous_billed_token_rates: Final = self.billed_token_rates
debug_key: Final = "response_cost_failure_debug_information"
had_failure_debug_info: Final = debug_key in self.model_call_details
previous_failure_debug_info: Final = self.model_call_details.get(debug_key)
try:
cost: Final = self.response_cost_calculator(result=result, cache_hit=False, ignore_stamped_cost=True)
return UncachedPrice(cost=cost, cost_breakdown=self.cost_breakdown)
finally:
self.cost_breakdown = previous_cost_breakdown
self.billed_token_rates = previous_billed_token_rates
if not had_failure_debug_info:
if previous_failure_debug_info is None:
self.model_call_details.pop(debug_key, None)
else:
self.model_call_details[debug_key] = previous_failure_debug_info
def _record_zero_cost_diagnostic(
self,
@ -2161,8 +2183,8 @@ class Logging(LiteLLMLoggingBaseClass):
pricing: Final = pricing_entry_for_cost_calc(
model=model,
completion_response=result,
custom_llm_provider=custom_llm_provider,
custom_pricing=self._custom_pricing_for(result),
custom_llm_provider=self.get_router_pricing_provider() or custom_llm_provider,
custom_pricing=self.custom_pricing_for(result),
base_model=get_base_model_from_metadata(model_call_details=self.model_call_details),
router_model_id=router_model_id or self.get_router_model_id(),
region_name=_resolve_mantle_region_for_cost(
@ -2187,7 +2209,7 @@ class Logging(LiteLLMLoggingBaseClass):
usage=usage,
)
def _custom_pricing_for(self, result: object) -> bool:
def custom_pricing_for(self, result: object) -> bool:
litellm_params: Final = getattr(self, "litellm_params", None)
result_hidden_params: Final = getattr(result, HIDDEN_PARAMS_ATTR, None) or MappingProxyType({})
additional_headers: Final = (
@ -3492,8 +3514,10 @@ class Logging(LiteLLMLoggingBaseClass):
): # custom logger class
if self.stream and complete_streaming_response is None:
callback.log_stream_event(
kwargs=call_details,
response_obj=result,
kwargs=redacted_model_call_details_for_custom_logger(
model_call_details=call_details, custom_logger=callback
),
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
@ -3503,8 +3527,13 @@ class Logging(LiteLLMLoggingBaseClass):
result = call_details_view["complete_response"]
callback.log_success_event(
kwargs=callback.redact_standard_logging_payload_from_model_call_details(call_details),
response_obj=result,
kwargs=redacted_model_call_details_for_custom_logger(
model_call_details=callback.redact_standard_logging_payload_from_model_call_details(
call_details
),
custom_logger=callback,
),
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
@ -3847,7 +3876,7 @@ class Logging(LiteLLMLoggingBaseClass):
model_call_details = callback.redact_standard_logging_payload_from_model_call_details(
model_call_details=model_call_details
)
model_call_details = redact_streaming_responses_for_custom_logger(
model_call_details = redacted_model_call_details_for_custom_logger(
model_call_details=model_call_details, custom_logger=callback
)
##################################
@ -3858,7 +3887,7 @@ class Logging(LiteLLMLoggingBaseClass):
kwargs=model_call_details,
response_obj=model_call_details["async_complete_streaming_response"]
if self.stream
else result,
else redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
guardrail_snapshot=self.shadow_eval_request_snapshot,
@ -3874,14 +3903,14 @@ class Logging(LiteLLMLoggingBaseClass):
else:
await callback.async_log_stream_event( # [TODO]: move this to being an async log stream event function
kwargs=model_call_details,
response_obj=result,
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
else:
await callback.async_log_success_event(
kwargs=model_call_details,
response_obj=result,
response_obj=redacted_response_obj_for_custom_logger(result, callback),
start_time=start_time,
end_time=end_time,
)
@ -7204,7 +7233,7 @@ def get_standard_logging_object_payload(
# Fall back to the actual deployment model set by the router in metadata.
if base_model is None:
base_model = metadata.get("deployment")
custom_pricing: Final = use_custom_pricing_for_model(litellm_params=litellm_params)
custom_pricing: Final = logging_obj.custom_pricing_for(init_response_obj)
raw_response_cost: Final = kwargs.get("response_cost")
llm_response_cost: Final[float] = raw_response_cost or 0.0
guardrail_cost: Final = guardrail_information_cost(metadata.get("standard_logging_guardrail_information"))

16
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -624,7 +624,7 @@ def update_messages_with_model_file_ids(
def update_responses_input_with_model_file_ids(
input: object,
model_id: str | None = None,
model_file_id_mapping: dict[str, dict[str, str]] | None = None,
model_file_id_mapping: Mapping[str, Mapping[str, str]] | None = None,
) -> object:
"""
Updates responses API input with provider-specific file IDs.
@ -665,7 +665,7 @@ def update_responses_input_with_model_file_ids(
for content_item in content:
if isinstance(content_item, dict) and content_item.get("type") == "input_file":
file_id = content_item.get("file_id")
if file_id:
if isinstance(file_id, str) and file_id:
provider_file_id = file_id # Default to original
# Check if we have a mapping for this file ID
@ -761,7 +761,7 @@ def _decode_vector_store_ids_in_tools(
def update_responses_tools_with_model_file_ids(
tools: list[dict[str, object]] | None,
model_id: str | None = None,
model_file_id_mapping: dict[str, dict[str, str]] | None = None,
model_file_id_mapping: Mapping[str, Mapping[str, str]] | None = None,
) -> list[dict[str, object]] | None:
"""
Updates responses API tools with provider-specific file IDs.
@ -2325,6 +2325,16 @@ def is_unsignable_thinking_block(block: object) -> bool:
return not (isinstance(signature, str) and len(signature) > 0)
def has_blank_redacted_data(block: object) -> bool:
match block:
case {"type": "redacted_thinking", "data": str() as data}:
return not data.strip()
case {"type": "redacted_thinking"}:
return True
case _:
return False
def native_redacted_thinking_data(data: str) -> str:
"""Claude's own `data` string for a redacted_thinking block.

256
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -6,10 +6,10 @@ import json
import mimetypes
import re
import xml.etree.ElementTree as ET
from collections.abc import Container, Iterable, Iterator, Mapping, Sequence
from collections.abc import Callable, Container, Iterable, Iterator, Mapping, Sequence
from enum import Enum
from types import MappingProxyType
from typing import Annotated, Any, Final, TypeAlias, TypedDict, cast, overload
from typing import Annotated, Any, Final, TypeAlias, TypedDict, TypeVar, cast, overload
from jinja2.sandbox import ImmutableSandboxedEnvironment
from pydantic import BaseModel, BeforeValidator, ConfigDict, JsonValue, TypeAdapter, ValidationError, model_validator
@ -61,6 +61,7 @@ from litellm.types.utils import GenericImageParsingChunk
from .common_utils import (
converse_redacted_content,
convert_content_list_to_str,
has_blank_redacted_data,
infer_content_type_from_url_and_content,
is_encrypted_reasoning_block,
is_non_content_values_set,
@ -2550,22 +2551,125 @@ def _pair_tool_calls_with_results(messages: Sequence[AllMessageValues]) -> tuple
def _replayable_thinking_blocks(
thinking_blocks: list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock],
*,
drop_params: bool = False,
) -> list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock]:
keys_to_drop: Final = _UNSUPPORTED_THINKING_KEYS if drop_params else _STREAM_ONLY_THINKING_KEYS
return [
_without_stream_index(_with_native_redacted_data(block))
_without_keys(_with_native_redacted_data(block), keys_to_drop)
for block in thinking_blocks
if not is_unsignable_thinking_block(block)
if not (is_unsignable_thinking_block(block) or (drop_params and has_blank_redacted_data(block)))
]
def _without_stream_index(
def _as_thinking_block(item: object) -> ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock | None:
match item:
case {"type": "thinking" | "redacted_thinking"}:
return cast( # cast-ok: the mapping pattern matched a thinking block's type tag
ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, item
)
case _:
return None
def _thinking_identity(item: object) -> tuple[str | None, ...] | None:
match item:
case {"type": "redacted_thinking", "data": str() as data}:
return ("redacted_thinking", native_redacted_thinking_data(data))
case {"type": "thinking", "signature": str() as signature} if signature:
return ("thinking", signature)
case {"type": str() as kind} if kind in ("thinking", "redacted_thinking"):
fields: Final = cast(Mapping[str, object], item) # cast-ok: the mapping pattern matched
return _unsigned_thinking_identity(kind, fields.get("thinking"), fields.get("signature"))
case _:
return None
def _unsigned_thinking_identity(kind: str, thinking: object, signature: object) -> tuple[str | None, ...] | None:
match thinking, signature:
case (str() | None as text, str() | None as signed):
return (kind, text, signed)
case _:
return None
_Replayed: Final = TypeVar("_Replayed")
def _with_missing_thinking_blocks(
content: Iterable[_Replayed], thinking_blocks: Sequence[_Replayed]
) -> tuple[_Replayed, ...]:
content_items: Final = tuple(content)
block_identities: Final = tuple(map(_thinking_identity, thinking_blocks))
content_positions: Final[Mapping[tuple[object, ...] | None, int]] = MappingProxyType(
{
identity: index
for index, identity in reversed(tuple(enumerate(map(_thinking_identity, content_items))))
if identity is not None
}
)
anchors: Final = tuple(index for index, identity in enumerate(block_identities) if identity in content_positions)
if not anchors:
return (*thinking_blocks, *content_items)
missing_before: Final = MappingProxyType(
{
content_positions[block_identities[end]]: thinking_blocks[start + 1 : end]
for start, end in zip((-1, *anchors), anchors)
}
)
missing_after: Final = MappingProxyType(
{content_positions[block_identities[anchors[-1]]]: thinking_blocks[anchors[-1] + 1 :]}
)
return tuple(_interleave(content_items, missing_before, missing_after))
def _without_repeated_thinking(
items: Sequence[_Replayed], identity: Callable[[_Replayed], tuple[object, ...] | None]
) -> tuple[_Replayed, ...]:
identities: Final = tuple(map(identity, items))
first_positions: Final[Mapping[tuple[object, ...], int]] = MappingProxyType(
{key: index for index, key in reversed(tuple(enumerate(identities))) if key is not None}
)
return tuple(
item for index, (item, key) in enumerate(zip(items, identities)) if key is None or first_positions[key] == index
)
def _interleave(
content: Sequence[_Replayed],
before: Mapping[int, Sequence[_Replayed]],
after: Mapping[int, Sequence[_Replayed]],
) -> Iterator[_Replayed]:
for index, item in enumerate(content):
yield from before.get(index, ())
yield item
yield from after.get(index, ())
def _thinking_in_replay_order(
content: str | Iterable[object] | None, thinking_blocks: Sequence[object] | None
) -> tuple[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock, ...]:
content_items: Final = content if isinstance(content, list) else ()
return tuple(
block
for item in _with_missing_thinking_blocks(content_items, thinking_blocks or ())
if (block := _as_thinking_block(item)) is not None
)
_STREAM_ONLY_THINKING_KEYS: Final = frozenset({"index"})
_UNSUPPORTED_THINKING_KEYS: Final = frozenset({"index", "cache_control"})
def _without_keys(
block: ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock,
keys: frozenset[str],
) -> ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock:
if "index" not in block:
if keys.isdisjoint(block):
return block
return cast( # cast-ok: dropping one key keeps the block's TypedDict shape
return cast( # cast-ok: dropping keys keeps the block's TypedDict shape
ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock,
{key: value for key, value in block.items() if key != "index"},
{key: value for key, value in block.items() if key not in keys},
)
@ -2629,6 +2733,7 @@ def anthropic_messages_pt(
messages: list[AllMessageValues],
model: str,
llm_provider: str,
drop_params: bool = False,
) -> _AnthropicMessageList:
"""
format messages for anthropic
@ -2816,7 +2921,9 @@ def anthropic_messages_pt(
_raw_thinking_blocks = assistant_content_block.get("thinking_blocks", None)
thinking_blocks = (
_replayable_thinking_blocks(_raw_thinking_blocks) if _raw_thinking_blocks is not None else None
_replayable_thinking_blocks(_raw_thinking_blocks, drop_params=drop_params)
if _raw_thinking_blocks is not None
else None
)
# Check if tool_calls contain server tool calls (web search, etc.)
@ -2951,42 +3058,28 @@ def anthropic_messages_pt(
# SEQUENTIAL MODE: No server tool calls, or no thinking blocks,
# or content is a list. Use the original sequential approach.
# When content is a list, check if it already contains thinking
# blocks inline. If so, skip prepending thinking_blocks to avoid
# duplication and preserve the original interleaved order.
# Fixes the gap where list-content messages bypass INTERLEAVED
# MODE and still get thinking blocks prepended out of order.
_content_is_list = "content" in assistant_content_block and isinstance(
assistant_content_block["content"], list
)
_content_list = assistant_content_block.get("content") if _content_is_list else None
_list_has_thinking = False
if _content_is_list and _content_list is not None:
for _item in _content_list:
if isinstance(_item, dict) and _item.get("type") in (
"thinking",
"redacted_thinking",
):
_list_has_thinking = True
break
if (
thinking_blocks is not None and not _list_has_thinking
thinking_blocks is not None and not _content_is_list
): # IMPORTANT: ADD THIS FIRST, ELSE ANTHROPIC WILL RAISE AN ERROR
assistant_content.extend(thinking_blocks)
if _content_is_list and _content_list is not None:
for m in _content_list:
for m in _with_missing_thinking_blocks(_content_list, thinking_blocks or ()):
if not isinstance(m, dict):
continue
# handle thinking blocks
text_block = cast(str, m.get("text", ""))
if m.get("type", "") == "thinking" and not is_unsignable_thinking_block(m):
anthropic_message: ChatCompletionThinkingBlock | AnthropicMessagesTextParam = cast(
ChatCompletionThinkingBlock, m
if (thinking_block := _as_thinking_block(m)) is not None:
assistant_content.extend(
_replayable_thinking_blocks([thinking_block], drop_params=drop_params)
)
assistant_content.append(anthropic_message)
continue
text_block = cast(str, m.get("text", ""))
# handle text
elif (
if (
m.get("type", "") == "text" and len(text_block) > 0
): # don't pass empty text blocks. anthropic api raises errors.
anthropic_message = AnthropicMessagesTextParam(type="text", text=text_block)
@ -3058,7 +3151,12 @@ def anthropic_messages_pt(
msg_i += 1
if assistant_content:
new_messages.append({"role": "assistant", "content": assistant_content})
new_messages.append(
{
"role": "assistant",
"content": list(_without_repeated_thinking(assistant_content, _thinking_identity)),
}
)
## MID-CONVERSATION SYSTEM MESSAGES (placement is the caller's job) ##
while msg_i < len(messages) and messages[msg_i]["role"] == "system":
@ -4190,6 +4288,16 @@ def _convert_to_bedrock_tool_call_result(
return content_block
def _reasoning_identity(block: BedrockContentBlock) -> tuple[object, ...] | None:
match block:
case {"reasoningContent": {"redactedContent": redacted}}:
return ("redacted_thinking", redacted)
case {"reasoningContent": {"reasoningText": {"signature": str() as signature}}} if signature:
return ("thinking", signature)
case _:
return None
def _deduplicate_bedrock_content_blocks(
blocks: list[BedrockContentBlock],
block_key: str,
@ -4673,6 +4781,7 @@ class BedrockConverseMessagesProcessor:
assistant_continue_message: str | ChatCompletionAssistantMessage | None = None,
deployment_model: str | None = None,
base_model: str | None = None,
drop_params: bool = False,
) -> list[BedrockMessageBlock]:
from litellm.llms.bedrock.common_utils import bedrock_converse_accepts_redacted_thinking
@ -4826,33 +4935,21 @@ class BedrockConverseMessagesProcessor:
)
_assistant_content = assistant_message_block.get("content", None)
thinking_blocks = assistant_message_block.get("thinking_blocks")
if thinking_blocks is not None:
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks, to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
_thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
)
if _assistant_content is not None and isinstance(_assistant_content, list):
assistants_parts: list[BedrockContentBlock] = []
for element in _assistant_content:
if isinstance(element, dict):
if element["type"] == "thinking" or element["type"] == "redacted_thinking":
thinking_block = BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks=[element], to_claude=to_claude
)
assistants_parts = (
BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=thinking_block,
assistant_parts=assistants_parts,
)
)
elif element["type"] == "text":
if element["type"] == "text":
# Skip completely empty strings to avoid blank content blocks
if element.get("text", "").strip():
assistants_part = BedrockContentBlock(text=element["text"])
@ -4871,7 +4968,9 @@ class BedrockConverseMessagesProcessor:
model=model,
deployment_model=deployment_model,
)
if _cache_point_block is not None:
if _cache_point_block is not None and not (
drop_params and _as_thinking_block(element) is not None
):
assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
elif _assistant_content is not None and isinstance(_assistant_content, str):
@ -4899,7 +4998,11 @@ class BedrockConverseMessagesProcessor:
msg_i += 1
assistant_content = _deduplicate_bedrock_content_blocks(assistant_content, "toolUse")
assistant_content = list(
_without_repeated_thinking(
_deduplicate_bedrock_content_blocks(assistant_content, "toolUse"), _reasoning_identity
)
)
assistant_content = _sort_bedrock_assistant_content_blocks(assistant_content)
if assistant_content:
@ -5083,6 +5186,7 @@ def _bedrock_converse_messages_pt(
assistant_continue_message: str | ChatCompletionAssistantMessage | None = None,
deployment_model: str | None = None,
base_model: str | None = None,
drop_params: bool = False,
) -> list[BedrockMessageBlock]:
"""
Converts given messages from OpenAI format to Bedrock format
@ -5244,35 +5348,21 @@ def _bedrock_converse_messages_pt(
)
_assistant_content = assistant_message_block.get("content", None)
thinking_blocks = assistant_message_block.get("thinking_blocks")
if thinking_blocks is not None:
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks, to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
converted_thinking_blocks = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
_thinking_in_replay_order(_assistant_content, thinking_blocks), to_claude=to_claude
)
)
assistant_content = BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=converted_thinking_blocks,
assistant_parts=assistant_content,
)
if _assistant_content is not None and isinstance(_assistant_content, list):
assistants_parts: list[BedrockContentBlock] = []
for element in _assistant_content:
if isinstance(element, dict):
if element["type"] == "thinking" or element["type"] == "redacted_thinking":
thinking_block = (
BedrockConverseMessagesProcessor.translate_thinking_blocks_to_reasoning_content_blocks(
thinking_blocks=[element], to_claude=to_claude
)
)
assistants_parts = (
BedrockConverseMessagesProcessor.add_thinking_blocks_to_assistant_content(
thinking_blocks=thinking_block,
assistant_parts=assistants_parts,
)
)
elif element["type"] == "text":
if element["type"] == "text":
# AWS Bedrock doesn't allow empty or whitespace-only text content
# Skip completely empty strings to avoid blank content blocks
if element.get("text", "").strip():
@ -5292,7 +5382,9 @@ def _bedrock_converse_messages_pt(
model=model,
deployment_model=deployment_model,
)
if _cache_point_block is not None:
if _cache_point_block is not None and not (
drop_params and _as_thinking_block(element) is not None
):
assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
elif _assistant_content is not None and isinstance(_assistant_content, str):
@ -5313,7 +5405,11 @@ def _bedrock_converse_messages_pt(
msg_i += 1
assistant_content = _deduplicate_bedrock_content_blocks(assistant_content, "toolUse")
assistant_content = list(
_without_repeated_thinking(
_deduplicate_bedrock_content_blocks(assistant_content, "toolUse"), _reasoning_identity
)
)
assistant_content = _sort_bedrock_assistant_content_blocks(assistant_content)
if assistant_content:

19
litellm/litellm_core_utils/prompt_templates/mid_conversation_system.py
View file

@ -49,7 +49,7 @@ from litellm.types.llms.openai import (
ChatCompletionUserMessage,
)
from .common_utils import is_unsignable_thinking_block
from .common_utils import has_blank_redacted_data, is_unsignable_thinking_block
CONVERTED_SYSTEM_NOTE: Final = (
"Operator note (not from the user): the following was originally a mid-conversation system-role reminder."
@ -267,27 +267,22 @@ def _block_index_by_message(blocks: Sequence[tuple[bool, tuple[int, ...]]]) -> M
def _thinking_block_renders(block: object) -> bool:
"""A thinking block the converter keeps: one Anthropic can verify, so never bridged encrypted reasoning."""
return message_field(block, "type") in _THINKING_BLOCK_TYPES and not is_unsignable_thinking_block(block)
return message_field(block, "type") in _THINKING_BLOCK_TYPES and not (
is_unsignable_thinking_block(block) or has_blank_redacted_data(block)
)
def _assistant_part_renders(part: object) -> bool:
"""A text part always renders: the converter pads empty text with a placeholder."""
part_type: Final = message_field(part, "type")
if part_type == "thinking":
if part_type in _THINKING_BLOCK_TYPES:
return _thinking_block_renders(part)
return part_type in _RENDERED_ASSISTANT_PART_TYPES or (
isinstance(part_type, str) and part_type.endswith("_tool_result")
)
def _separate_thinking_blocks_render(message: object, parts: Sequence[object]) -> bool:
"""``thinking_blocks`` reach the wire only when no inline thinking part claims the slot.
The converter skips the separate blocks as soon as the content list carries a
``thinking`` or ``redacted_thinking`` part, whether or not that part itself renders.
"""
if any(message_field(part, "type") in _THINKING_BLOCK_TYPES for part in parts):
return False
def _separate_thinking_blocks_render(message: object) -> bool:
return any(_thinking_block_renders(block) for block in parts_of(message_field(message, "thinking_blocks")))
@ -404,7 +399,7 @@ def _assistant_renders(message: object, answered: frozenset[str]) -> bool:
parts: Final = parts_of(content)
return (
any(_assistant_part_renders(part) for part in parts)
or _separate_thinking_blocks_render(message, parts)
or _separate_thinking_blocks_render(message)
or _tool_calls_render(message, answered)
or bool(message_field(message, "function_call"))
or bool(message_field(message_field(message, "provider_specific_fields"), "compaction_blocks"))

230
litellm/litellm_core_utils/redact_messages.py
View file

@ -10,10 +10,12 @@
import asyncio
import copy
import inspect
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from dataclasses import replace
from typing import TYPE_CHECKING, Any, Final
from pydantic import TypeAdapter
import litellm
from litellm.constants import REDACTED_BY_LITELLM
from litellm.integrations.custom_logger import CustomLogger
@ -40,36 +42,79 @@ else:
LiteLLMLoggingObject = Any
def _opted_out_of_message_logging(custom_logger: CustomLogger) -> bool:
return hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True
def redact_message_input_output_from_custom_logger(
litellm_logging_obj: LiteLLMLoggingObject, result, custom_logger: CustomLogger
):
if hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True:
) -> object:
if _opted_out_of_message_logging(custom_logger):
return perform_redaction(litellm_logging_obj.model_call_details, result, redact_streaming_responses=False)
return result
def redact_streaming_responses_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
def redacted_model_call_details_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
"""
Returns a copy of model_call_details whose streaming response entries are redacted deepcopies
when the custom logger has opted out of message logging. The shared model_call_details is left
untouched so other callbacks still receive the unredacted response.
Returns a copy of model_call_details with the request messages, instructions and streaming responses
redacted when the custom logger has opted out of message logging. The shared model_call_details is left
untouched so other callbacks and the spend log still receive the unredacted values.
"""
if not (hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True):
if not _opted_out_of_message_logging(custom_logger):
return model_call_details
redacted_entries: Final = {
streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
if model_call_details.get(streaming_key) is not None
return {
**{key: value for key, value in model_call_details.items() if key != SERVED_OUTPUT_TEXTS_KEY},
**_redacted_entries_for_opted_out_logger(model_call_details),
}
if not redacted_entries:
return model_call_details
return {**model_call_details, **redacted_entries}
def _redacted_streaming_response_copy(streaming_response):
redacted_response: Final = copy.deepcopy(streaming_response)
_redact_streaming_response(redacted_response)
return redacted_response
def _redacted_entries_for_opted_out_logger(model_call_details: Mapping[str, object]) -> Mapping[str, object]:
optional_params: Final = _as_mapping(model_call_details.get("optional_params"))
litellm_params: Final = _as_mapping(model_call_details.get("litellm_params"))
standard_logging_object: Final = _as_mapping(model_call_details.get("standard_logging_object"))
return {
"messages": [{"role": "user", "content": REDACTED_BY_LITELLM}],
"prompt": "",
"input": "",
**({"instructions": REDACTED_BY_LITELLM} if model_call_details.get("instructions") is not None else {}),
**({"optional_params": _with_redacted_instructions(optional_params)} if optional_params is not None else {}),
**({"litellm_params": redacted_litellm_params(litellm_params)} if litellm_params is not None else {}),
**(
{"standard_logging_object": _redact_standard_logging_object(standard_logging_object)}
if standard_logging_object is not None
else {}
),
**{
streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
for streaming_key in (
"complete_streaming_response",
"async_complete_streaming_response",
"complete_response",
)
if model_call_details.get(streaming_key) is not None
},
}
def redacted_response_obj_for_custom_logger(response_obj: object, custom_logger: CustomLogger) -> object:
if not _opted_out_of_message_logging(custom_logger) or not _is_redactable_response(response_obj):
return response_obj
return perform_redaction({}, response_obj)
def _is_redactable_response(response_obj: object) -> bool:
return isinstance(
response_obj, (litellm.ModelResponse, litellm.ResponsesAPIResponse, litellm.EmbeddingResponse)
) or (isinstance(response_obj, dict) and ("choices" in response_obj or "output" in response_obj))
def _redacted_streaming_response_copy(streaming_response: object) -> object:
redacted_response: Final = (
streaming_response
if isinstance(streaming_response, litellm.ResponsesAPIResponse)
else copy.deepcopy(streaming_response)
)
return _redact_streaming_response(redacted_response)
def _redact_streaming_response(streaming_response):
@ -77,10 +122,13 @@ def _redact_streaming_response(streaming_response):
for choice in streaming_response.choices:
_redact_choice_content(choice)
redact_vertex_ai_metadata_from_logged_object(streaming_response)
elif isinstance(streaming_response, litellm.ResponsesAPIResponse):
return perform_redaction({}, streaming_response)
elif hasattr(streaming_response, "output"):
_redact_responses_api_output(streaming_response.output)
if hasattr(streaming_response, "reasoning") and streaming_response.reasoning is not None:
streaming_response.reasoning = None
return streaming_response
def _redact_tool_calls(tool_calls) -> None:
@ -177,31 +225,75 @@ def _redact_responses_api_output(output_items):
output_item.input = REDACTED_BY_LITELLM
def _redact_responses_api_output_dict(output_items, redacted_str: str):
"""Helper to redact ResponsesAPIResponse output items in dict form."""
for output_item in output_items:
if not isinstance(output_item, dict):
continue
_RESPONSE_ITEMS: Final = TypeAdapter(tuple[object, ...])
if output_item.get("text") is not None:
output_item["text"] = redacted_str
if isinstance(output_item.get("content"), list):
for content_item in output_item["content"]:
if isinstance(content_item, dict) and content_item.get("text") is not None:
content_item["text"] = redacted_str
if isinstance(content_item, dict) and content_item.get("refusal") is not None:
content_item["refusal"] = redacted_str
def _redacted_responses_api_output_part(part: object, fields: tuple[str, ...], redacted_str: str) -> object:
if (part_mapping := _as_mapping(part)) is None:
return part
return {key: redacted_str if key in fields and value is not None else value for key, value in part_mapping.items()}
if output_item.get("type") == "reasoning" and isinstance(output_item.get("summary"), list):
for summary_item in output_item["summary"]:
if isinstance(summary_item, dict) and summary_item.get("text") is not None:
summary_item["text"] = redacted_str
if output_item.get("type") == "function_call" and "arguments" in output_item:
output_item["arguments"] = redacted_str
if output_item.get("type") == "custom_tool_call" and "input" in output_item:
output_item["input"] = redacted_str
def _redacted_responses_api_output_item(item: object, redacted_str: str) -> object:
if (item_mapping := _as_mapping(item)) is None:
return item
content: Final = item_mapping.get("content")
summary: Final = item_mapping.get("summary")
return {
**item_mapping,
**({"text": redacted_str} if item_mapping.get("text") is not None else {}),
**(
{
"content": [
_redacted_responses_api_output_part(part, ("text", "refusal"), redacted_str)
for part in _RESPONSE_ITEMS.validate_python(content)
]
}
if isinstance(content, list)
else {}
),
**(
{
"summary": [
_redacted_responses_api_output_part(part, ("text",), redacted_str)
for part in _RESPONSE_ITEMS.validate_python(summary)
]
}
if item_mapping.get("type") == "reasoning" and isinstance(summary, list)
else {}
),
**(
{"arguments": redacted_str}
if item_mapping.get("type") == "function_call" and "arguments" in item_mapping
else {}
),
**(
{"input": redacted_str}
if item_mapping.get("type") == "custom_tool_call" and "input" in item_mapping
else {}
),
}
def _redact_responses_api_output_dict(output_items: Sequence[object], redacted_str: str) -> Sequence[object]:
return [_redacted_responses_api_output_item(item, redacted_str) for item in output_items]
def _with_redacted_instructions(params: Mapping[str, object]) -> Mapping[str, object]:
return params if params.get("instructions") is None else {**params, "instructions": REDACTED_BY_LITELLM}
def _redacted_responses_api_response(
response: Mapping[str, object], *, redact_instructions: bool = True
) -> Mapping[str, object]:
output: Final = response.get("output")
redacted: Final = {
**response,
"output": _redact_responses_api_output_dict(_RESPONSE_ITEMS.validate_python(output), REDACTED_BY_LITELLM)
if isinstance(output, list)
else output,
}
return _with_redacted_instructions(redacted) if redact_instructions else redacted
def redacted_standard_logging_payload(payload: Mapping[str, object]) -> Mapping[str, object]:
@ -215,20 +307,25 @@ def redacted_standard_logging_payload(payload: Mapping[str, object]) -> Mapping[
return _redact_standard_logging_object(payload)
def _redact_standard_logging_object(payload: Mapping[str, object]) -> dict[str, object]:
def _redact_standard_logging_object(
payload: Mapping[str, object], *, redact_instructions: bool = True
) -> dict[str, object]:
standard_logging_object: Final = copy.deepcopy(without_classifier_audit(payload))
redacted_str: Final = REDACTED_BY_LITELLM
if standard_logging_object.get("messages") is not None:
standard_logging_object["messages"] = [{"role": "user", "content": redacted_str}]
model_parameters: Final = _as_mapping(standard_logging_object.get("model_parameters"))
if model_parameters is not None and redact_instructions:
standard_logging_object["model_parameters"] = _with_redacted_instructions(model_parameters)
response: Final = standard_logging_object.get("response")
if response is not None:
if isinstance(response, dict) and "output" in response:
# ResponsesAPIResponse format - redact content in output items
if isinstance(response.get("output"), list):
_redact_responses_api_output_dict(response["output"], redacted_str)
redact_vertex_ai_metadata_from_logged_object(response)
standard_logging_object["response"] = _redacted_responses_api_response(
response, redact_instructions=redact_instructions
)
redact_vertex_ai_metadata_from_logged_object(standard_logging_object["response"])
elif isinstance(response, dict) and "choices" in response:
# ModelResponse dict format - redact content in choices
if isinstance(response.get("choices"), list):
@ -320,11 +417,11 @@ def _redact_hidden_provider_specific_fields(params: Mapping[str, object]) -> Non
def redacted_litellm_params(params: Mapping[str, object], *, redact_request_body: bool = True) -> dict[str, object]:
"""
Copy of litellm_params without classifier audit fields or baseline route request parameters.
redact_request_body also redacts the nested proxy_server_request body.
redact_request_body also redacts the request instructions and the nested proxy_server_request body.
"""
request: Final = _as_mapping(params.get("proxy_server_request"))
return {
**params,
**(_with_redacted_instructions(params) if redact_request_body else params),
**{
key: _redacted_baseline_metadata(value)
for key, value in params.items()
@ -344,8 +441,9 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
redact_streaming_responses=False skips the in-place redaction of the shared streaming
response entries; per-callback redaction hands each opted-out callback its own redacted
copy via redact_streaming_responses_for_custom_logger instead. It also leaves the shared
proxy request body and hidden provider fields as they are, since the spend log reads them.
copy via redacted_model_call_details_for_custom_logger instead. It also leaves the shared
request instructions, proxy request body and hidden provider fields as they are, since the
spend log and the other callbacks read them.
"""
# Redact model_call_details
params: Final = _as_mapping(model_call_details.get("litellm_params"))
@ -358,16 +456,31 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
model_call_details["messages"] = [{"role": "user", "content": REDACTED_BY_LITELLM}]
model_call_details["prompt"] = ""
model_call_details["input"] = ""
optional_params: Final = _as_mapping(model_call_details.get("optional_params"))
if redact_streaming_responses:
model_call_details.update(
{
**({"instructions": REDACTED_BY_LITELLM} if model_call_details.get("instructions") is not None else {}),
**({"optional_params": _with_redacted_instructions(optional_params)} if optional_params else {}),
}
)
model_call_details.pop(SERVED_OUTPUT_TEXTS_KEY, None)
standard_logging_object: Final = model_call_details.get("standard_logging_object")
if isinstance(standard_logging_object, Mapping):
model_call_details["standard_logging_object"] = _redact_standard_logging_object(standard_logging_object)
model_call_details["standard_logging_object"] = _redact_standard_logging_object(
standard_logging_object, redact_instructions=redact_streaming_responses
)
redact_vertex_ai_metadata_from_litellm_params(model_call_details)
# Redact streaming response
if redact_streaming_responses and model_call_details.get("stream", False) is True:
for _streaming_key in ("complete_streaming_response", "async_complete_streaming_response"):
_redact_streaming_response(model_call_details.get(_streaming_key))
model_call_details.update(
{
streaming_key: _redact_streaming_response(model_call_details[streaming_key])
for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
if model_call_details.get(streaming_key) is not None
}
)
# Redact result
if result is not None:
@ -381,10 +494,7 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
# For async objects, return a simple redacted response without deepcopy
return {"text": REDACTED_BY_LITELLM}
if not (
isinstance(result, (litellm.ModelResponse, litellm.ResponsesAPIResponse, litellm.EmbeddingResponse))
or (isinstance(result, dict) and ("choices" in result or "output" in result))
):
if not _is_redactable_response(result):
return {"text": REDACTED_BY_LITELLM}
_result: Final = copy.deepcopy(result)
@ -399,14 +509,18 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
_redact_model_response_dict_choices(_result["choices"], REDACTED_BY_LITELLM)
redact_vertex_ai_metadata_from_logged_object(_result)
elif isinstance(_result, dict) and "output" in _result:
if isinstance(_result.get("output"), list):
_redact_responses_api_output_dict(_result["output"], REDACTED_BY_LITELLM)
return _redacted_responses_api_response(_result, redact_instructions=redact_streaming_responses)
elif isinstance(_result, litellm.ResponsesAPIResponse):
if hasattr(_result, "output"):
_redact_responses_api_output(_result.output)
# Redact reasoning field in ResponsesAPIResponse
if hasattr(_result, "reasoning") and _result.reasoning is not None:
_result.reasoning = None
return _result.model_copy(
update={"instructions": REDACTED_BY_LITELLM}
if redact_streaming_responses and _result.instructions is not None
else {}
)
elif isinstance(_result, litellm.EmbeddingResponse):
if hasattr(_result, "data") and _result.data is not None:
_result.data = []

72
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -13,7 +13,7 @@ Pattern Overview:
"""
import json
from collections.abc import Mapping, MutableSequence, Sequence
from collections.abc import Iterable, Mapping, MutableSequence, Sequence
from copy import deepcopy
from dataclasses import dataclass
from itertools import chain, repeat
@ -27,8 +27,10 @@ from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
from litellm.llms.anthropic.pass_through.adapters.transformation import (
OPENAI_MAX_TOOL_NAME_LENGTH,
LiteLLMAnthropicMessagesAdapter,
is_provider_native_tool_dict,
truncate_tool_name,
)
from litellm.llms.anthropic.pass_through.convertible_blocks import (
keep_non_string_tool_result_texts,
@ -202,6 +204,60 @@ def _is_client_tool_use(block: Mapping[str, object]) -> bool:
)
def _tool_use_name(block: object) -> str | None:
fields: Final = as_str_mapping(block)
if fields is None or fields.get("type") != "tool_use":
return None
return anthropic_tool_name(fields)
def _tool_use_names(message: object) -> tuple[str, ...]:
return tuple(name for name in map(_tool_use_name, list_field(message, "content")) if name is not None)
@dataclass(frozen=True, slots=True)
class _OriginalToolNames:
"""Caller tool names over the guardrail's limit, keyed by the shortened name the guardrail sees."""
declared: Mapping[str, str]
used: Mapping[str, str]
def _restorable_names(names: Iterable[str]) -> Mapping[str, str]:
listed: Final = tuple(names)
taken: Final = frozenset(listed)
shortened: Final = tuple(
(truncate_tool_name(name), name) for name in listed if len(name) > OPENAI_MAX_TOOL_NAME_LENGTH
)
return MappingProxyType({alias: name for alias, name in shortened if alias not in taken})
def _original_tool_names(request: Mapping[str, object]) -> _OriginalToolNames:
return _OriginalToolNames(
declared=_restorable_names(chain.from_iterable(map(anthropic_tool_names, list_field(request, "tools")))),
used=_restorable_names(chain.from_iterable(map(_tool_use_names, list_field(request, "messages")))),
)
def _restore_tool_use_names(messages: Sequence[object], original_names: Mapping[str, str]) -> None:
for message in messages:
for block in list_field(message, "content"):
if (name := _tool_use_name(block)) is not None and name in original_names and isinstance(block, dict):
block["name"] = original_names[name]
def _with_original_name(tool: ChatCompletionToolParam, original_names: Mapping[str, str]) -> ChatCompletionToolParam:
if "function" not in tool:
return tool
function: Final = tool["function"]
fields: Final = as_str_mapping(function)
name: Final = None if fields is None else fields.get("name")
original: Final = original_names.get(name) if isinstance(name, str) else None
if original is None:
return tool
return {**tool, "function": {**function, "name": original}}
def _write_back_system_block(system: object, block_idx: int, response: str) -> None:
if not isinstance(system, list):
return
@ -629,6 +685,9 @@ class AnthropicMessagesHandler(BaseTranslation):
model: Final = data.get("model")
if model:
inputs["model"] = model
original_tool_names: Final = _original_tool_names(
cast(Mapping[str, object], data) # cast-ok: Anthropic request body
)
guardrailed_inputs: Final = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=data,
@ -643,7 +702,9 @@ class AnthropicMessagesHandler(BaseTranslation):
anthropic_config: Final = AnthropicConfig()
anthropic_tools: Final[list[AllAnthropicToolsValues]] = []
for tool in guardrailed_tools:
converted_tool, mcp_server = anthropic_config.map_tool_helper(tool)
converted_tool, mcp_server = anthropic_config.map_tool_helper(
_with_original_name(tool, original_tool_names.declared)
)
if converted_tool is not None:
anthropic_tools.append(converted_tool)
# Note: MCP servers are handled separately in the main transformation
@ -676,6 +737,7 @@ class AnthropicMessagesHandler(BaseTranslation):
),
hoisted_system_message=hoisted_system_message,
preserve_system_messages=has_midturn_system_message,
original_tool_names=original_tool_names.used,
)
else:
if guardrailed_texts and len(guardrailed_texts) != len(scanned):
@ -840,6 +902,8 @@ class AnthropicMessagesHandler(BaseTranslation):
structured_messages: list, # mutable-ok: API message payload
hoisted_system_message: object = None,
preserve_system_messages: bool = False,
*,
original_tool_names: Mapping[str, str],
) -> None:
"""Write a guardrail's structured-message rewrite back without losing corrections."""
from litellm.litellm_core_utils.prompt_templates.factory import (
@ -903,6 +967,10 @@ class AnthropicMessagesHandler(BaseTranslation):
for block in content:
if isinstance(block, dict) and block.get("type") == "thinking":
block.pop("cache_control", None)
_restore_tool_use_names(
cast(Sequence[object], converted), # cast-ok: Anthropic message payload
original_tool_names,
)
data["messages"] = keep_non_string_tool_result_texts(converted, data.get("messages"))
@staticmethod

5
litellm/llms/anthropic/chat/transformation.py
View file

@ -2267,6 +2267,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
anthropic_messages_pt,
)
drop_params: Final = litellm.drop_params is True or litellm_params.get("drop_params") is True
if "tools" not in optional_params and messages is not None and has_tool_call_blocks(messages):
optional_params["tools"], _ = self._map_tools(add_dummy_tool(custom_llm_provider="anthropic"))
@ -2293,7 +2295,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
AnthropicConfig._maybe_drop_speed_param(
model=model,
optional_params=optional_params,
drop_params=litellm.drop_params or litellm_params.get("drop_params") is True,
drop_params=drop_params,
custom_llm_provider=self.custom_llm_provider,
)
@ -2351,6 +2353,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
model=model,
messages=list(conversation),
llm_provider=self._resolved_provider,
drop_params=drop_params,
)
except Exception as e:
raise AnthropicError(

55
litellm/llms/anthropic/common_utils.py
View file

@ -712,7 +712,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
optional_params: dict,
model: str,
param: str,
value: Any,
value: object,
drop_params: bool,
output_key: str,
) -> None:
@ -1788,6 +1788,21 @@ def strip_thinking_blocks_from_anthropic_messages_request_dict(
data.pop("thinking", None)
_DROP_ANTHROPIC_MESSAGE: Final = object()
def _strip_empty_content_blocks_from_message(message: object) -> object:
if not isinstance(message, dict) or not isinstance(message.get("content"), list):
return message
content: Final = message["content"]
filtered: Final = [
block for block in content if not _is_empty_text_block(block) and not is_empty_unsigned_thinking_block(block)
]
if len(filtered) == len(content):
return message
return {**message, "content": filtered} if filtered else _DROP_ANTHROPIC_MESSAGE
def strip_empty_content_blocks_from_anthropic_messages(
messages: Sequence[object],
) -> list[Any]:
@ -1822,18 +1837,11 @@ def strip_empty_content_blocks_from_anthropic_messages(
The caller's list and its content blocks are never mutated; modified
messages are returned as shallow copies with a fresh content list.
"""
out: Final[list[Any]] = []
for m in messages:
if not isinstance(m, dict) or not isinstance(m.get("content"), list):
out.append(m)
continue
content = m["content"]
filtered = [b for b in content if not _is_empty_text_block(b) and not is_empty_unsigned_thinking_block(b)]
if len(filtered) == len(content):
out.append(m)
elif filtered:
out.append({**m, "content": filtered})
return out
return [
stripped_message
for message in messages
if (stripped_message := _strip_empty_content_blocks_from_message(message)) is not _DROP_ANTHROPIC_MESSAGE
]
def _is_empty_text_block(block: object) -> bool:
@ -1921,18 +1929,15 @@ def sanitize_tool_use_ids_in_anthropic_messages(messages: Sequence[object]) -> l
and ``:`` — valid on the upstream provider but rejected by Anthropic when
the session is switched to a native Anthropic deployment.
"""
out: Final[list[Any]] = []
for m in messages:
if not isinstance(m, dict) or not isinstance(m.get("content"), list):
out.append(m)
continue
content = m["content"]
new_content = [_sanitize_tool_use_id_content_block(b) for b in content]
if new_content == content:
out.append(m)
else:
out.append({**m, "content": new_content})
return out
return [_sanitize_tool_use_ids_in_message(message) for message in messages]
def _sanitize_tool_use_ids_in_message(message: object) -> object:
if not isinstance(message, dict) or not isinstance(message.get("content"), list):
return message
content: Final = message["content"]
new_content: Final = [_sanitize_tool_use_id_content_block(block) for block in content]
return message if new_content == content else {**message, "content": new_content}
class _ReplayedSearchQuery(LiteLLMBaseModel):

34
litellm/llms/anthropic/count_tokens/token_counter.py
View file

@ -2,6 +2,8 @@
Anthropic Token Counter implementation using the CountTokens API.
"""
from collections.abc import Mapping
from types import MappingProxyType
from typing import Any, Final
from pydantic import JsonValue
@ -10,6 +12,11 @@ import litellm
from litellm._logging import verbose_logger
from litellm.exceptions import AuthenticationError
from litellm.llms.anthropic.count_tokens.handler import AnthropicCountTokensHandler
from litellm.llms.anthropic.count_tokens.transformation import (
AnthropicCountFields,
ChatCountTranslationError,
anthropic_count_fields,
)
from litellm.llms.base_llm.base_utils import BaseTokenCounter
from litellm.types.utils import CountContextManagement, LlmProviders, TokenCountMessagesFormat, TokenCountResponse
@ -38,6 +45,8 @@ class AnthropicTokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "anthropic",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
count_options: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
"""
Count tokens using Anthropic's CountTokens API.
@ -57,6 +66,30 @@ class AnthropicTokenCounter(BaseTokenCounter):
if not messages:
return None
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
match translated := await anthropic_count_fields(
model_to_use, messages, system, tools, messages_format, AnthropicConfig(), chat_params
):
case ChatCountTranslationError():
return translated.response(request_model, model_to_use, "anthropic_api")
case AnthropicCountFields():
return await self.count_tokens(
model_to_use=model_to_use,
messages=[dict(message) for message in translated.messages],
contents=contents,
deployment=deployment,
request_model=request_model,
tools=[dict(tool) for tool in translated.tools] if translated.tools is not None else None,
system=translated.system,
messages_format="anthropic",
output_format=output_format,
context_management=context_management,
count_options=translated.options,
)
case None:
pass
deployment = deployment or {}
litellm_params: Final = deployment.get("litellm_params", {})
api_base: Final = litellm_params.get("api_base")
@ -89,6 +122,7 @@ class AnthropicTokenCounter(BaseTokenCounter):
api_base=api_base,
tools=tools,
system=system,
optional_params=count_options,
)
if result is not None:

225
litellm/llms/anthropic/count_tokens/transformation.py
View file

@ -4,25 +4,37 @@ Anthropic CountTokens API transformation logic.
This module handles the transformation of requests to Anthropic's CountTokens API format.
"""
import copy
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from itertools import chain
from types import MappingProxyType
from typing import Final
from typing import (
Final,
Protocol,
TypeAlias,
cast, # noqa: TID251 # caller chat messages go to chat inference translation as-is
)
from pydantic import JsonValue, TypeAdapter
from pydantic import ConfigDict, JsonValue, TypeAdapter, with_config
from typing_extensions import NotRequired, ReadOnly, TypedDict
from litellm.constants import ANTHROPIC_TOKEN_COUNTING_BETA_VERSION
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.prompt_templates.mid_conversation_system import (
anthropic_system_blocks,
split_leading_system_run,
)
from litellm.llms.anthropic.common_utils import merge_anthropic_beta_headers
from litellm.llms.anthropic.wif import resolve_anthropic_base
from litellm.types.llms.openai import ChatCompletionImageObject
from litellm.types.llms.openai import AllMessageValues, ChatCompletionImageObject, ValidUserMessageContentTypes
from litellm.types.utils import TokenCountMessagesFormat, TokenCountResponse
_COUNT_REQUEST: Final = TypeAdapter(dict[str, JsonValue])
_SYSTEM_BLOCKS: Final = TypeAdapter(list[JsonValue])
_IMAGE_BLOCK: Final = TypeAdapter(ChatCompletionImageObject)
COUNT_TOKEN_OPTION_NAMES: Final = ("thinking", "tool_choice", "output_config")
_PLAIN_MESSAGE_KEYS: Final = frozenset({"role", "content"})
def _count_image(block: JsonValue) -> JsonValue:
@ -74,6 +86,213 @@ def _lift_leading_system(
return conversation, lifted or system
@dataclass(frozen=True, slots=True)
class AnthropicCountFields:
messages: tuple[Mapping[str, JsonValue], ...]
system: JsonValue
tools: tuple[Mapping[str, JsonValue], ...] | None
options: Mapping[str, JsonValue]
@dataclass(frozen=True, slots=True)
class ChatCountTranslationError:
message: str
def response(self, request_model: str, model_used: str, tokenizer_type: str) -> TokenCountResponse:
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_used,
tokenizer_type=tokenizer_type,
error=True,
error_message=self.message,
status_code=400,
rejected_before_provider_call=True,
)
@with_config(ConfigDict(extra="ignore"))
class _AnthropicChatRequest(TypedDict):
messages: ReadOnly[tuple[Mapping[str, JsonValue], ...]]
system: NotRequired[ReadOnly[JsonValue]]
tools: NotRequired[ReadOnly[tuple[Mapping[str, JsonValue], ...]]]
thinking: NotRequired[ReadOnly[JsonValue]]
tool_choice: NotRequired[ReadOnly[JsonValue]]
output_config: NotRequired[ReadOnly[JsonValue]]
output_format: NotRequired[ReadOnly[JsonValue]]
_ANTHROPIC_CHAT_REQUEST: Final = TypeAdapter(_AnthropicChatRequest)
_ChatParams: TypeAlias = dict[str, object]
_ChatMessages: TypeAlias = list[AllMessageValues]
class _ChatRequestTranslator(Protocol):
def map_openai_params(
self, non_default_params: _ChatParams, optional_params: _ChatParams, model: str, drop_params: bool
) -> _ChatParams: ...
def transform_request(
self,
model: str,
messages: _ChatMessages,
optional_params: _ChatParams,
litellm_params: _ChatParams,
headers: _ChatParams,
) -> _ChatParams: ...
def _needs_chat_translation(message: Mapping[str, object]) -> bool:
return message.get("role") == "tool" or not _PLAIN_MESSAGE_KEYS.issuperset(message.keys())
def _is_chat_tool(tool: Mapping[str, object]) -> bool:
from litellm.litellm_core_utils.prompt_templates.factory import is_flat_responses_tool
return is_flat_responses_tool(tool) or (tool.get("type") == "function" and "function" in tool)
def _has_non_chat_user_block(message: Mapping[str, object]) -> bool:
content: Final = message.get("content")
if message.get("role") != "user" or not isinstance(content, list):
return False
blocks: Final = _SYSTEM_BLOCKS.validate_python(content)
return any(isinstance(block, dict) and block.get("type") not in ValidUserMessageContentTypes for block in blocks)
def _gets_placeholder_name(tool: Mapping[str, object]) -> bool:
from litellm.llms.anthropic.pass_through.adapters.transformation import is_provider_native_tool_dict
name: Final = tool.get("name")
return not (_is_chat_tool(tool) or is_provider_native_tool_dict(tool) or (isinstance(name, str) and name.strip()))
def _untranslatable_input(
messages: Sequence[Mapping[str, object]], tools: Sequence[Mapping[str, object]]
) -> ChatCountTranslationError | None:
invalid_index: Final = next(
(index for index, message in enumerate(messages) if _has_non_chat_user_block(message)), None
)
if invalid_index is not None:
return ChatCountTranslationError(
message=f"Invalid user message at index {invalid_index}. "
"Please ensure all user messages are valid OpenAI chat completion messages."
)
unnamed_tool: Final = next(filter(_gets_placeholder_name, tools), None)
if unnamed_tool is not None:
return ChatCountTranslationError(
message=f"Invalid request for token counting: tool type {unnamed_tool.get('type')!r} "
"has no Chat Completions equivalent"
)
return None
def _system_messages(system: object) -> tuple[Mapping[str, object], ...]:
message: Final[Mapping[str, object]] = {"role": "system", "content": system}
return (message,) if isinstance(system, (str, list)) and system else ()
def _count_options(request: _AnthropicChatRequest) -> Mapping[str, JsonValue]:
"""Structured output reaches chat inference as the beta ``output_format``, which count_tokens
refuses; its GA ``output_config.format`` form counts the same schema the same way."""
output_config: Final = request.get("output_config")
output_format: Final = request.get("output_format")
counted_config: Final = {
**(output_config if isinstance(output_config, dict) else {}),
**({} if output_format is None else {"format": output_format}),
}
return MappingProxyType(
{
key: value
for key, value in (
("thinking", request.get("thinking")),
("tool_choice", request.get("tool_choice")),
("output_config", counted_config or None),
)
if value is not None
}
)
def _translated_chat_request(
model: str,
messages: Sequence[Mapping[str, object]],
system: object,
tools: Sequence[Mapping[str, object]],
chat_params: Mapping[str, object],
translator: _ChatRequestTranslator,
) -> _ChatParams:
from litellm.litellm_core_utils.prompt_templates.factory import chat_completion_tools_for
chat_tools: Final = [dict(chat_tool) for chat_tool in chain.from_iterable(map(chat_completion_tools_for, tools))]
openai_params: Final = {name: value for name, value in chat_params.items() if name != "drop_params"}
mapped_params: Final = {**openai_params, **({"tools": chat_tools} if chat_tools else {})}
chat_messages: Final = cast( # cast-ok: chat-shaped request messages, read the way chat inference reads them
_ChatMessages, copy.deepcopy([*_system_messages(system), *messages])
)
return translator.transform_request(
model=model,
messages=chat_messages,
optional_params=translator.map_openai_params(
mapped_params, {}, model, drop_params=chat_params.get("drop_params") is True
)
if mapped_params
else {},
litellm_params={},
headers={},
)
def anthropic_fields_for_chat_tool_use(
model: str,
messages: Sequence[Mapping[str, object]],
system: object,
tools: Sequence[Mapping[str, object]] | None,
translator: _ChatRequestTranslator,
chat_params: Mapping[str, object] = MappingProxyType({}),
) -> AnthropicCountFields | ChatCountTranslationError | None:
counted_tools: Final = tools or ()
if (
not chat_params
and not any(map(_is_chat_tool, counted_tools))
and not any(map(_needs_chat_translation, messages))
):
return None
untranslatable: Final = _untranslatable_input(messages, counted_tools)
if untranslatable is not None:
return untranslatable
from litellm.exceptions import BadRequestError
from litellm.llms.anthropic.common_utils import AnthropicError
try:
request: Final = _translated_chat_request(model, messages, system, counted_tools, chat_params, translator)
except (AnthropicError, BadRequestError) as e:
return ChatCountTranslationError(message=str(e.message))
except (ValueError, KeyError) as e:
return ChatCountTranslationError(message=f"Invalid request for token counting: {e}")
translated: Final = _ANTHROPIC_CHAT_REQUEST.validate_python(request)
return AnthropicCountFields(
messages=translated["messages"],
system=translated.get("system"),
tools=translated.get("tools"),
options=_count_options(translated),
)
async def anthropic_count_fields(
model: str,
messages: Sequence[Mapping[str, object]],
system: object,
tools: Sequence[Mapping[str, object]] | None,
messages_format: TokenCountMessagesFormat,
translator: _ChatRequestTranslator,
chat_params: Mapping[str, object] = MappingProxyType({}),
) -> AnthropicCountFields | ChatCountTranslationError | None:
if messages_format != "openai":
return None
return await asyncify(anthropic_fields_for_chat_tool_use)(model, messages, system, tools, translator, chat_params)
class AnthropicCountTokensConfig:
"""
Configuration and transformation logic for Anthropic CountTokens API.

3
litellm/llms/anthropic/pass_through/messages/response_cache.py
View file

@ -1,5 +1,6 @@
import re
from collections.abc import AsyncIterator, Mapping, Sequence
from datetime import datetime
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, cast
@ -153,6 +154,8 @@ class CachedAnthropicMessagesStreamIterator(BaseAnthropicMessagesStreamingIterat
await self._handle_streaming_logging(chunks)
raise StopAsyncIteration
chunk: Final = self.chunks[self.current_index]
if self.completion_start_time is None:
self.completion_start_time = datetime.now()
self.current_index += 1
return chunk

2
litellm/llms/anthropic/pass_through/messages/streaming_iterator.py
View file

@ -506,7 +506,7 @@ class BaseAnthropicMessagesStreamingIterator:
):
self.litellm_logging_obj = litellm_logging_obj
self.request_body = request_body
self.start_time = datetime.now()
self.start_time = litellm_logging_obj.start_time
self.completion_start_time: datetime | None = None
async def _handle_streaming_logging(self, collected_chunks: list[bytes], *, stream_teardown: bool = False):

43
litellm/llms/anthropic/pass_through/responses_adapters/streaming_iterator.py
View file

@ -2,7 +2,7 @@
## Translates OpenAI call to Anthropic `/v1/messages` format
import asyncio
from collections import deque
from collections.abc import AsyncIterator, Awaitable, Callable, Iterator, Mapping
from collections.abc import AsyncGenerator, Awaitable, Callable, Iterator, Mapping
from typing import TYPE_CHECKING, Any, Final
import anyio
@ -22,7 +22,11 @@ from litellm.llms.anthropic.pass_through.messages.utils import (
)
from litellm.llms.anthropic.pass_through.safeguards import StreamedSafeguardResults
from litellm.llms.anthropic.pass_through.utils import anthropic_sse_frame
from litellm.responses.streaming_iterator import map_stream_error_to_exception, stream_error_status_and_message
from litellm.responses.streaming_iterator import (
map_stream_error_to_exception,
partial_responses_incomplete_event_on_disconnect,
stream_error_status_and_message,
)
from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicUsage
from litellm.types.llms.base import LiteLLMBaseModel
@ -34,6 +38,7 @@ from .transformation import (
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObject
from litellm.llms.anthropic.pass_through.safeguards import SafeguardsEvaluator
from litellm.types.llms.openai import ResponseIncompleteEvent
class _UpstreamFailure(LiteLLMBaseModel):
@ -526,8 +531,11 @@ class AnthropicResponsesStreamWrapper:
raise StopAsyncIteration
async def async_anthropic_sse_wrapper(self) -> AsyncIterator[bytes]:
"""Yield SSE-encoded bytes for each Anthropic event chunk."""
def async_anthropic_sse_wrapper(self) -> "AnthropicResponsesSSEByteStream":
"""SSE-encoded bytes for each Anthropic event chunk."""
return AnthropicResponsesSSEByteStream(self)
async def _sse_frames(self) -> AsyncGenerator[bytes, None]:
safeguard_results: Final = StreamedSafeguardResults(self.safeguards_evaluator)
try:
async for chunk in self:
@ -537,3 +545,30 @@ class AnthropicResponsesStreamWrapper:
if not closing_superseded_attempt():
await self._report_unreported_failure()
def partial_response_on_disconnect(self) -> "ResponseIncompleteEvent | None":
return partial_responses_incomplete_event_on_disconnect(self.responses_stream)
class AnthropicResponsesSSEByteStream:
"""
The byte stream the proxy relays for a default-route /v1/messages request.
Keeps the Responses API iterator reachable so a client disconnect can bill
the partial output the way native /v1/responses streams do.
"""
def __init__(self, wrapper: AnthropicResponsesStreamWrapper) -> None:
self._wrapper: Final = wrapper
self._frames: Final = wrapper._sse_frames() # pyright: ignore[reportPrivateUsage] # the wrapper's own byte stream
def __aiter__(self) -> "AnthropicResponsesSSEByteStream":
return self
async def __anext__(self) -> bytes:
return await self._frames.__anext__()
async def aclose(self) -> None:
await self._frames.aclose()
def partial_response_on_disconnect(self) -> "ResponseIncompleteEvent | None":
return self._wrapper.partial_response_on_disconnect()

43
litellm/llms/anthropic/pass_through/responses_adapters/transformation.py
View file

@ -464,33 +464,32 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
return input_items
@staticmethod
def _translate_tool_to_responses_api(tool_dict: Mapping[str, object]) -> dict[str, object]:
tool_type: Final = tool_dict.get("type", "")
tool_name: Final = tool_dict.get("name", "")
# web_search tool
if (isinstance(tool_type, str) and tool_type.startswith("web_search")) or tool_name == "web_search":
return {"type": "web_search_preview"}
# Responses turns strict mode on when `strict` is omitted, silently rewriting
# `required` to every property. Anthropic tools are non-strict unless asked.
func_tool: Final[dict[str, object]] = {
"type": "function",
"name": tool_name,
"strict": bool(tool_dict.get("strict")),
}
if "description" in tool_dict:
func_tool["description"] = tool_dict["description"]
if "input_schema" in tool_dict:
func_tool["parameters"] = json_copy_or_original(tool_dict["input_schema"])
return func_tool
def translate_tools_to_responses_api(
self,
tools: list[AllAnthropicToolsValues],
) -> list[dict[str, object]]:
"""Convert Anthropic tool definitions to Responses API function tools."""
result: Final[list[dict[str, object]]] = []
for tool in tools:
tool_dict = cast(dict[str, Any], tool)
tool_type = tool_dict.get("type", "")
tool_name = tool_dict.get("name", "")
# web_search tool
if (isinstance(tool_type, str) and tool_type.startswith("web_search")) or tool_name == "web_search":
result.append({"type": "web_search_preview"})
continue
# Responses turns strict mode on when `strict` is omitted, silently rewriting
# `required` to every property. Anthropic tools are non-strict unless asked.
func_tool: dict[str, object] = {
"type": "function",
"name": tool_name,
"strict": bool(tool_dict.get("strict")),
}
if "description" in tool_dict:
func_tool["description"] = tool_dict["description"]
if "input_schema" in tool_dict:
func_tool["parameters"] = json_copy_or_original(tool_dict["input_schema"])
result.append(func_tool)
return result
return [self._translate_tool_to_responses_api(tool) for tool in tools]
@staticmethod
def translate_tool_choice_to_responses_api(

3
litellm/llms/azure_ai/anthropic/count_tokens/handler.py
View file

@ -4,6 +4,7 @@ Azure AI Anthropic CountTokens API handler.
Uses httpx for HTTP requests with Azure authentication.
"""
from collections.abc import Mapping
from typing import Final
import httpx
@ -36,6 +37,7 @@ class AzureAIAnthropicCountTokensHandler(AzureAIAnthropicCountTokensConfig):
timeout: float | httpx.Timeout | None = None,
tools: list[dict[str, JsonValue]] | None = None,
system: object = None,
optional_params: Mapping[str, JsonValue] | None = None,
) -> dict[str, object]:
"""
Handle a CountTokens request using httpx with Azure authentication.
@ -66,6 +68,7 @@ class AzureAIAnthropicCountTokensHandler(AzureAIAnthropicCountTokensConfig):
messages=messages,
tools=tools,
system=system,
optional_params=optional_params,
)
verbose_logger.debug("Transformed request: %s", request_body)

34
litellm/llms/azure_ai/anthropic/count_tokens/token_counter.py
View file

@ -3,12 +3,19 @@ Azure AI Anthropic Token Counter implementation using the CountTokens API.
"""
import os
from collections.abc import Mapping
from types import MappingProxyType
from typing import Any, Final
from pydantic import JsonValue
import litellm
from litellm._logging import verbose_logger
from litellm.llms.anthropic.count_tokens.transformation import (
AnthropicCountFields,
ChatCountTranslationError,
anthropic_count_fields,
)
from litellm.llms.azure_ai.anthropic.count_tokens.handler import (
AzureAIAnthropicCountTokensHandler,
)
@ -55,6 +62,8 @@ class AzureAIAnthropicTokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "anthropic",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
count_options: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
"""
Count tokens using Azure AI Anthropic's CountTokens API.
@ -74,6 +83,30 @@ class AzureAIAnthropicTokenCounter(BaseTokenCounter):
if not messages:
return None
from litellm.llms.azure_ai.anthropic.transformation import AzureAnthropicConfig
match translated := await anthropic_count_fields(
model_to_use, messages, system, tools, messages_format, AzureAnthropicConfig(), chat_params
):
case ChatCountTranslationError():
return translated.response(request_model, model_to_use, "azure_ai_anthropic_api")
case AnthropicCountFields():
return await self.count_tokens(
model_to_use=model_to_use,
messages=[dict(message) for message in translated.messages],
contents=contents,
deployment=deployment,
request_model=request_model,
tools=[dict(tool) for tool in translated.tools] if translated.tools is not None else None,
system=translated.system,
messages_format="anthropic",
output_format=output_format,
context_management=context_management,
count_options=translated.options,
)
case None:
pass
deployment = deployment or {}
litellm_params: Final = deployment.get("litellm_params", {})
@ -101,6 +134,7 @@ class AzureAIAnthropicTokenCounter(BaseTokenCounter):
litellm_params=litellm_params,
tools=tools,
system=system,
optional_params=count_options,
)
if result is not None:

2
litellm/llms/base_llm/base_utils.py
View file

@ -9,6 +9,7 @@ import re
from abc import ABC, abstractmethod
from collections.abc import Iterator, Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import Any, Final
from openai.lib import _parsing, _pydantic # pyright: ignore[reportPrivateUsage] # SDK parser internals
@ -40,6 +41,7 @@ class BaseTokenCounter(ABC):
messages_format: TokenCountMessagesFormat = "openai",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
pass

2
litellm/llms/base_llm/chat/transformation.py
View file

@ -428,7 +428,7 @@ class BaseConfig(ABC):
streaming_response: Union[Iterator[str], AsyncIterator[str], "ModelResponse"],
sync_stream: bool,
json_mode: bool | None = False,
) -> Any:
) -> object:
pass
def get_model_response_iterator_for_request(

43
litellm/llms/base_llm/decisions/transformation.py
View file

@ -8,6 +8,7 @@ from typing import Final
import httpx
from pydantic import TypeAdapter, ValidationError
from pydantic_core import ErrorDetails
from typing_extensions import assert_never
import litellm
@ -394,7 +395,9 @@ _MAPPING: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(Mapping[str, ob
_LABELS: Final[TypeAdapter[tuple[str, ...]]] = TypeAdapter(tuple[str, ...])
def _normalize_answer(value: object) -> Mapping[str, object]:
def _normalize_answer(value: object) -> object:
if not isinstance(value, Mapping):
return value
answer: Final = _MAPPING.validate_python(value)
legend: Final = answer.get("legend")
if answer.get("type") != "score" or not isinstance(legend, list):
@ -403,11 +406,36 @@ def _normalize_answer(value: object) -> Mapping[str, object]:
return {**answer, "legend": {str(index): label for index, label in enumerate(labels)}}
def parse_systemone_response(payload: object, request: DecisionsIRRequest) -> DecisionsIRResponse:
def _normalize_systemone_response(payload: object) -> object:
if not isinstance(payload, Mapping):
return payload
response: Final = _MAPPING.validate_python(payload)
answers: Final = _MAPPING.validate_python(response.get("answers"))
normalized: Final = {**response, "answers": {name: _normalize_answer(answer) for name, answer in answers.items()}}
return systemone_response_to_ir(_SYSTEMONE_RESPONSE_ADAPTER.validate_python(normalized), request)
answers: Final = response.get("answers")
if not isinstance(answers, Mapping):
return response
return {
**response,
"answers": {name: _normalize_answer(answer) for name, answer in _MAPPING.validate_python(answers).items()},
}
def _validation_detail(detail: ErrorDetails) -> str:
location: Final = ".".join(str(part) for part in detail["loc"])
return f"{location}: {detail['msg']}" if location else detail["msg"]
def _validation_summary(error: ValidationError) -> str:
return "; ".join(_validation_detail(detail) for detail in error.errors(include_url=False, include_input=False))
class DecisionsResponseValidationError(BaseLLMException):
pass
def parse_systemone_response(payload: object, request: DecisionsIRRequest) -> DecisionsIRResponse:
return systemone_response_to_ir(
_SYSTEMONE_RESPONSE_ADAPTER.validate_python(_normalize_systemone_response(payload)), request
)
def _systemone_answer(answer: DecisionsIRAnswer) -> DecisionAnswer | None:
@ -627,9 +655,10 @@ class BaseDecisionsConfig(ABC):
try:
return self.parse_response(payload, request)
except ValidationError as error:
raise BaseLLMException(
raise DecisionsResponseValidationError(
status_code=500,
message=f"Decisions provider '{custom_llm_provider}' returned an unexpected response: {error}",
message=f"Decisions provider '{custom_llm_provider}' returned an unexpected response: "
f"{error.title} {_validation_summary(error)}",
) from error
def get_error_class(

2
litellm/llms/base_llm/files/batch_records.py
View file

@ -18,6 +18,7 @@ def _responses_request_keys() -> frozenset[str]:
def responses_batch_body_to_chat_body(
openai_request_body: Mapping[str, object],
custom_llm_provider: str | None = None,
drop_params: bool | None = None,
) -> dict[str, object]: # mutable-ok: provider transforms take the bridged chat body as a plain dict
"""
Rewrite the body of an OpenAI `/v1/responses` batch record as a Chat Completions body.
@ -50,5 +51,6 @@ def responses_batch_body_to_chat_body(
input=chat_input,
responses_api_request=responses_request,
custom_llm_provider=custom_llm_provider,
drop_params=drop_params,
metadata=openai_request_body.get("metadata"),
)

2
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -185,7 +185,7 @@ class BaseTranslation(ABC):
request_data: dict | None = None,
stream_transform_sink: StreamTransformSink | None = None,
deliver_ended_stream_rewrites: bool = False,
) -> Any:
) -> object:
"""
Process output streaming response with guardrails.

22
litellm/llms/base_llm/harness/transformation.py
View file

@ -68,6 +68,16 @@ class HarnessTurnRequest:
cwd: str | None = None
@dataclass(frozen=True)
class HarnessSessionCopy:
"""A saved session under fresh ids: written to `path` in the sandbox, then loaded by `import_command`."""
session_id: str
path: str
data: bytes
import_command: Sequence[str]
@dataclass(frozen=True)
class HarnessTurnResponse:
"""What the runtime produced for one turn, after the process exited."""
@ -128,6 +138,18 @@ class BaseCLIHarnessConfig(BaseHarnessConfig[OptionsT], Generic[OptionsT, Stream
) -> HarnessTurnRequest:
"""argv / env / stdin for one turn. native_session_id is set after the first turn."""
def transform_resume_export_command(self, native_session_id: str) -> Sequence[str] | None:
"""Command that prints a saved session before resuming one this handler did not run.
None resumes the saved session as is. A non-zero exit or a timeout fails the turn, and the next turn
tries again.
"""
return None
def transform_resume_copy(self, exported: str, private_dir: str, workdir: str) -> HarnessSessionCopy | None:
"""The copy to import and continue, from the export command's stdout, or None to continue the original."""
return None
@abstractmethod
def create_stream_state(self) -> StreamStateT:
"""Fresh per-turn parser state."""

4
litellm/llms/base_llm/managed_resources/base_managed_resource.py
View file

@ -28,7 +28,7 @@ if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient as _PrismaClient
from litellm.router import Router as _Router
Span = _Span | Any
Span = _Span
InternalUsageCache = _InternalUsageCache
PrismaClient = _PrismaClient
Router = _Router
@ -550,7 +550,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
limit: int | None = None,
after: str | None = None,
additional_filters: Mapping[str, object] | None = None,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
List resources created by a user.

37
litellm/llms/bedrock/base_aws_llm.py
View file

@ -80,6 +80,12 @@ class _AssumeRoleParams(TypedDict):
Tags: ReadOnly[NotRequired[tuple[AwsSessionTag, ...]]]
class _StsClientKwargs(TypedDict):
verify: ReadOnly[bool | str]
endpoint_url: ReadOnly[NotRequired[str]]
region_name: ReadOnly[NotRequired[str]]
def _assume_role_params(
aws_role_name: str,
aws_session_name: str,
@ -275,7 +281,7 @@ class BaseAWSLLM(SignsRequestsWithAWS):
"aws_session_tags",
]
def _get_ssl_verify(self, ssl_verify: bool | str | None = None):
def _get_ssl_verify(self, ssl_verify: bool | str | None = None) -> bool | str:
"""
Get SSL verification setting for boto3 clients.
@ -875,15 +881,18 @@ class BaseAWSLLM(SignsRequestsWithAWS):
aws_sts_endpoint: str | None = None,
ssl_verify: bool | str | None = None,
aws_region_name: str | None = None,
) -> dict:
) -> _StsClientKwargs:
"""STS client kwargs with aligned endpoint_url and region_name (SigV4)."""
kwargs: Final[dict] = {"verify": self._get_ssl_verify(ssl_verify)}
if aws_sts_endpoint is not None:
kwargs["endpoint_url"] = aws_sts_endpoint
sts_region: Final = self._resolve_sts_region(aws_sts_endpoint, aws_region_name)
if sts_region is not None:
kwargs["region_name"] = sts_region
return kwargs
verify: Final = self._get_ssl_verify(ssl_verify)
match (aws_sts_endpoint, self._resolve_sts_region(aws_sts_endpoint, aws_region_name)):
case (None, None):
return _StsClientKwargs(verify=verify)
case (None, region):
return _StsClientKwargs(verify=verify, region_name=region)
case (endpoint, None):
return _StsClientKwargs(verify=verify, endpoint_url=endpoint)
case (endpoint, region):
return _StsClientKwargs(verify=verify, endpoint_url=endpoint, region_name=region)
def get_aws_region_name_for_non_llm_api_calls(
self,
@ -1813,13 +1822,13 @@ async def run_aws_signing(
)
async def sign_request_off_loop_if_aws(
async def run_off_loop_if_aws(
provider_config: object,
sign_request: Callable[_SignParams, _SignedRequest],
call: Callable[_SignParams, _SignedRequest],
/,
*args: _SignParams.args,
**kwargs: _SignParams.kwargs, # kwargs-ok: ParamSpec forwarding keeps the wrapped sign_request signature
**kwargs: _SignParams.kwargs, # kwargs-ok: ParamSpec forwarding keeps the wrapped call's signature
) -> _SignedRequest:
if isinstance(provider_config, SignsRequestsWithAWS):
return await run_aws_signing(sign_request, *args, **kwargs)
return sign_request(*args, **kwargs)
return await run_aws_signing(call, *args, **kwargs)
return call(*args, **kwargs)

26
litellm/llms/bedrock/batches/handler.py
View file

@ -5,7 +5,10 @@ from typing import TYPE_CHECKING, Final, Literal
from openai.types.batch import BatchRequestCounts
from openai.types.batch import Metadata as OpenAIBatchMetadata
from litellm.litellm_core_utils.aws_partition import get_aws_dns_suffix
from litellm.llms.bedrock.common_utils import (
bedrock_control_plane_base_url,
configured_bedrock_control_plane_endpoint,
)
from litellm.types.llms.bedrock import AwsAuthParams, AwsSessionTag
from litellm.types.utils import LiteLLMBatch
@ -156,10 +159,14 @@ class BedrockBatchesHandler:
aws_session_tags=aws_session_tags,
)
creds: Final = BedrockBatchesConfig().resolve_credentials(auth_params, region)
control_plane_endpoint: Final = configured_bedrock_control_plane_endpoint(
kwargs.get("aws_bedrock_control_plane_endpoint")
)
client: Final = boto3.client(
"bedrock",
region_name=region,
endpoint_url=control_plane_endpoint,
aws_access_key_id=creds.access_key,
aws_secret_access_key=creds.secret_key,
aws_session_token=creds.token,
@ -171,6 +178,7 @@ class BedrockBatchesHandler:
batch_id=batch_id,
aws_region_name=region,
logging_obj=logging_obj,
aws_bedrock_control_plane_endpoint=control_plane_endpoint,
**auth_params.model_dump(),
)
@ -270,7 +278,7 @@ class BedrockBatchesHandler:
batch_id: str,
aws_region_name: str | None = None,
logging_obj=None,
**kwargs,
**kwargs: object,
) -> "LiteLLMBatch":
"""
Handle ``GetModelInvocationJob`` status check for AWS Bedrock bulk batch
@ -294,7 +302,10 @@ class BedrockBatchesHandler:
``aws_session_token``, ``aws_profile_name``,
``aws_role_name``, ``aws_session_name``,
``aws_web_identity_token``, ``aws_sts_endpoint``,
``aws_external_id``, ``aws_session_tags``). Unknown keys are ignored.
``aws_external_id``, ``aws_session_tags``), plus
``aws_bedrock_control_plane_endpoint`` to send the call to a VPC
endpoint or gateway instead of the regional ``bedrock`` host.
Unknown keys are ignored.
Returns:
``LiteLLMBatch`` shaped like an OpenAI Batch resource.
@ -318,10 +329,14 @@ class BedrockBatchesHandler:
from litellm.llms.bedrock.batches.transformation import BedrockBatchesConfig
creds: Final = BedrockBatchesConfig().resolve_credentials(AwsAuthParams.model_validate(kwargs), region)
control_plane_endpoint: Final = configured_bedrock_control_plane_endpoint(
kwargs.get("aws_bedrock_control_plane_endpoint")
)
client: Final = boto3.client(
"bedrock",
region_name=region,
endpoint_url=control_plane_endpoint,
aws_access_key_id=creds.access_key,
aws_secret_access_key=creds.secret_key,
aws_session_token=creds.token,
@ -334,14 +349,13 @@ class BedrockBatchesHandler:
# `GetModelInvocationJob` accepts either form, but only the bare id
# produces a sensible-looking URL in logs.
url_path_id: Final = _extract_job_id_from_arn(batch_id) or batch_id
control_plane_base: Final = bedrock_control_plane_base_url(control_plane_endpoint, region)
logging_obj.pre_call(
input=batch_id,
api_key="",
additional_args={
"complete_input_dict": {"jobIdentifier": batch_id},
"api_base": (
f"https://bedrock.{region}.{get_aws_dns_suffix(region)}/model-invocation-job/{url_path_id}"
),
"api_base": f"{control_plane_base}/model-invocation-job/{url_path_id}",
},
)

24
litellm/llms/bedrock/batches/transformation.py
View file

@ -7,7 +7,7 @@ from typing import TYPE_CHECKING, Final, Literal, cast
from httpx import Headers, Response
from pydantic import TypeAdapter, ValidationError
from litellm.litellm_core_utils.aws_partition import get_aws_dns_suffix, is_bedrock_arn
from litellm.litellm_core_utils.aws_partition import is_bedrock_arn
from litellm.litellm_core_utils.cloud_storage_security import (
BEDROCK_MANAGED_S3_BATCH_PREFIX,
)
@ -32,6 +32,7 @@ from litellm.types.utils import LiteLLMBatch, LlmProviders, Usage
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import (
CommonBatchFilesUtils,
bedrock_control_plane_base_url,
merge_bedrock_aws_request_params,
resolve_s3_bucket_owner,
resolve_s3_encryption_key_id,
@ -42,7 +43,7 @@ if TYPE_CHECKING:
# Bedrock batch input files are uploaded as
# s3://bucket/litellm-bedrock-files-{model, ":" -> "-"}-{uuid4}.jsonl (see
# BedrockFilesTransformation._get_s3_object_name). A uuid4 is always 36 hex/dash
# _batch_s3_object_name in litellm/llms/bedrock/files/transformation.py). A uuid4 is always 36 hex/dash
# characters, so it can be stripped off the end unambiguously even though the
# model name itself may contain dashes.
_S3_BATCH_FILE_UUID_SUFFIX_PATTERN: Final = re.compile(
@ -177,11 +178,10 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
aws_region_name: Final = self._get_aws_region_name(request_params, model)
# Bedrock model invocation job endpoint
# Format: https://bedrock.{region}.{partition dns suffix}/model-invocation-job
bedrock_endpoint: Final = (
f"https://bedrock.{aws_region_name}.{get_aws_dns_suffix(aws_region_name)}/model-invocation-job"
control_plane_base: Final = bedrock_control_plane_base_url(
request_params.get("aws_bedrock_control_plane_endpoint"), aws_region_name
)
bedrock_endpoint: Final = f"{control_plane_base}/model-invocation-job"
return bedrock_endpoint
@ -279,9 +279,10 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
# Use common utility for AWS signing
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
aws_region_name: Final = self._get_aws_region_name(request_params, model)
endpoint_url: Final = (
f"https://bedrock.{aws_region_name}.{get_aws_dns_suffix(aws_region_name)}/model-invocation-job"
control_plane_base: Final = bedrock_control_plane_base_url(
request_params.get("aws_bedrock_control_plane_endpoint"), aws_region_name
)
endpoint_url: Final = f"{control_plane_base}/model-invocation-job"
signed_headers, signed_data = self.common_utils.sign_aws_request(
service_name="bedrock",
data=bedrock_request,
@ -431,12 +432,13 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
import urllib.parse as _ul
encoded_arn: Final = _ul.quote(batch_id, safe="")
endpoint_url: Final = (
f"https://bedrock.{region}.{get_aws_dns_suffix(region)}/model-invocation-job/{encoded_arn}"
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
control_plane_base: Final = bedrock_control_plane_base_url(
request_params.get("aws_bedrock_control_plane_endpoint"), region
)
endpoint_url: Final = f"{control_plane_base}/model-invocation-job/{encoded_arn}"
# Use common utility for AWS signing
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
signed_headers, _ = self.common_utils.sign_aws_request(
service_name="bedrock",
data={}, # GET request has no body

4
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -385,7 +385,7 @@ class AmazonConverseConfig(BaseConfig):
@staticmethod
def _requires_min_max_tokens(model: str) -> bool:
return re.search(r"openai\.gpt-\d|xai\.grok-", model) is not None
return re.search(r"openai\.gpt-\d|xai\.grok-|moonshotai\.kimi-k3", model) is not None
def _is_nova_2_model(self, model: str) -> bool:
"""
@ -2127,6 +2127,7 @@ class AmazonConverseConfig(BaseConfig):
llm_provider="bedrock_converse",
user_continue_message=litellm_params.pop("user_continue_message", None),
base_model=self._deployment_base_model(_DEPLOYMENT_PARAMS.validate_python(litellm_params)),
drop_params=litellm.drop_params is True or litellm_params.get("drop_params") is True,
),
capability_model,
)
@ -2207,6 +2208,7 @@ class AmazonConverseConfig(BaseConfig):
llm_provider="bedrock_converse",
user_continue_message=litellm_params.pop("user_continue_message", None),
base_model=self._deployment_base_model(_DEPLOYMENT_PARAMS.validate_python(litellm_params)),
drop_params=litellm.drop_params is True or litellm_params.get("drop_params") is True,
),
capability_model,
)

34
litellm/llms/bedrock/claude_platform/messages_transformation.py
View file

@ -1,10 +1,11 @@
from typing import Any, Final
from typing import Final
import litellm
from litellm.llms.anthropic.pass_through.messages.transformation import (
DEFAULT_ANTHROPIC_API_VERSION,
AnthropicMessagesConfig,
)
from litellm.llms.bedrock.base_aws_llm import run_aws_signing
from litellm.secret_managers.main import get_secret_str
from litellm.types.router import GenericLiteLLMParams
@ -22,14 +23,14 @@ class BedrockClaudePlatformMessagesConfig(BedrockClaudePlatformMixin, AnthropicM
def validate_anthropic_messages_environment(
self,
headers: dict,
headers: dict[str, str], # mutable-ok: overrides the base signature
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
messages: list[object], # mutable-ok: overrides the base signature
optional_params: dict[str, object], # mutable-ok: overrides the base signature
litellm_params: dict[str, object], # mutable-ok: overrides the base signature
api_key: str | None = None,
api_base: str | None = None,
) -> tuple[dict, str | None]:
) -> tuple[dict[str, str], str | None]: # mutable-ok: overrides the base signature
workspace_id: Final = self._get_workspace_id(optional_params, litellm_params)
if workspace_id is None:
raise litellm.AuthenticationError(
@ -66,6 +67,27 @@ class BedrockClaudePlatformMessagesConfig(BedrockClaudePlatformMixin, AnthropicM
api_base,
)
async def avalidate_anthropic_messages_environment(
self,
headers: dict[str, str], # mutable-ok: mirrors validate_anthropic_messages_environment
model: str,
messages: list[object], # mutable-ok: mirrors validate_anthropic_messages_environment
optional_params: dict[str, object], # mutable-ok: mirrors validate_anthropic_messages_environment
litellm_params: dict[str, object], # mutable-ok: mirrors validate_anthropic_messages_environment
api_key: str | None = None,
api_base: str | None = None,
) -> tuple[dict[str, str], str | None]: # mutable-ok: mirrors validate_anthropic_messages_environment
return await run_aws_signing(
self.validate_anthropic_messages_environment,
headers=headers,
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
api_key=api_key,
api_base=api_base,
)
def transform_anthropic_messages_request(
self,
model: str,

19
litellm/llms/bedrock/common_utils.py
View file

@ -138,6 +138,25 @@ def merge_bedrock_aws_request_params(
return request_params
def configured_bedrock_control_plane_endpoint(endpoint: object) -> str | None:
"""An ``aws_bedrock_control_plane_endpoint`` value without a trailing slash, or None when unset."""
if isinstance(endpoint, str) and endpoint.strip():
return endpoint.strip().rstrip("/")
return None
def bedrock_control_plane_base_url(endpoint: object, aws_region_name: str) -> str:
"""Base URL for Bedrock control-plane calls such as model invocation jobs.
A configured ``aws_bedrock_control_plane_endpoint`` (a VPC endpoint or a compatible gateway)
wins over the regional ``bedrock`` host. ``api_base`` is deliberately not used here, since on a
Bedrock deployment it points at the runtime host, which does not serve these APIs.
"""
return configured_bedrock_control_plane_endpoint(endpoint) or (
f"https://bedrock.{aws_region_name}.{get_aws_dns_suffix(aws_region_name)}"
)
def s3_static_key_pair(params: Mapping[str, object]) -> tuple[str, str] | None:
"""The s3_access_key_id / s3_secret_access_key pair when both are set, otherwise None."""
s3_access_key_id: Final = params.get("s3_access_key_id")

62
litellm/llms/bedrock/count_tokens/bedrock_token_counter.py
View file

@ -4,12 +4,15 @@ Bedrock Token Counter implementation using the CountTokens API.
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import Any, Final
from pydantic import JsonValue
from typing_extensions import assert_never
from litellm._logging import verbose_logger
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
from litellm.llms.anthropic.count_tokens.transformation import ChatCountTranslationError, anthropic_count_fields
from litellm.llms.base_llm.base_utils import BaseTokenCounter
from litellm.llms.bedrock.common_utils import BedrockError, get_bedrock_base_model
from litellm.llms.bedrock.count_tokens.handler import BedrockCountTokensHandler
@ -58,6 +61,17 @@ def _runtime_rejected_claude_model(outcome: CountTokensOutcome, resolved_model:
)
def _request_data(
model: str, messages: Sequence[Mapping[str, object]], tools: Sequence[Mapping[str, object]] | None, system: object
) -> Mapping[str, object]:
return {
"model": model,
"messages": [dict(message) for message in messages],
**({"tools": [dict(tool) for tool in tools]} if tools else {}),
**({"system": system} if system else {}),
}
class BedrockTokenCounter(BaseTokenCounter):
"""Counts on the configured Bedrock endpoint, with Mantle retry when Runtime rejects a Claude model"""
@ -81,13 +95,13 @@ class BedrockTokenCounter(BaseTokenCounter):
self,
handler: BedrockCountTokensHandler | BedrockMantleCountTokensHandler,
tokenizer_type: str,
request_data: dict[str, object],
request_data: Mapping[str, object],
litellm_params: dict[str, object],
resolved_model: str,
) -> CountTokensOutcome:
try:
result: Final = await handler.handle_count_tokens_request(
request_data=request_data,
request_data=dict(request_data),
litellm_params=litellm_params,
resolved_model=resolved_model,
client=self._client,
@ -115,30 +129,44 @@ class BedrockTokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "anthropic",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
if not messages:
return None
litellm_params: Final = (deployment or {}).get("litellm_params", {})
request_data: Final[dict[str, object]] = {
"model": model_to_use,
"messages": messages,
**({"tools": tools} if tools else {}),
**({"system": system} if system else {}),
}
use_mantle: Final = model_to_use.startswith("bedrock_mantle/")
resolved_model: Final = get_bedrock_base_model(model_to_use.removeprefix("bedrock_mantle/"))
counted_model: Final = model_to_use.removeprefix("bedrock_mantle/")
resolved_model: Final = get_bedrock_base_model(counted_model)
primary_tokenizer: Final = MANTLE_TOKENIZER_TYPE if use_mantle else RUNTIME_TOKENIZER_TYPE
translated: Final = (
await anthropic_count_fields(
counted_model, messages, system, tools, messages_format, AnthropicConfig(), chat_params
)
if is_mantle_claude_model(resolved_model)
else None
)
if isinstance(translated, ChatCountTranslationError):
return translated.response(request_model, model_to_use, primary_tokenizer)
litellm_params: Final = (deployment or {}).get("litellm_params", {})
runtime_request: Final = (
_request_data(model_to_use, messages, tools, system)
if translated is None
else _request_data(model_to_use, translated.messages, translated.tools, translated.system)
)
mantle_request: Final = runtime_request if translated is None else {**runtime_request, **translated.options}
primary: Final = await self._count_with(
self._mantle_handler if use_mantle else self._runtime_handler,
MANTLE_TOKENIZER_TYPE if use_mantle else RUNTIME_TOKENIZER_TYPE,
request_data,
litellm_params,
resolved_model,
primary: Final = (
await self._count_with(
self._mantle_handler, primary_tokenizer, mantle_request, litellm_params, resolved_model
)
if use_mantle
else await self._count_with(
self._runtime_handler, primary_tokenizer, runtime_request, litellm_params, resolved_model
)
)
outcome: Final = (
await self._count_with(
self._mantle_handler, MANTLE_TOKENIZER_TYPE, request_data, litellm_params, resolved_model
self._mantle_handler, MANTLE_TOKENIZER_TYPE, mantle_request, litellm_params, resolved_model
)
if not use_mantle and _runtime_rejected_claude_model(primary, resolved_model)
else primary

6
litellm/llms/bedrock/count_tokens/handler.py
View file

@ -48,7 +48,8 @@ class BedrockCountTokensHandler(BedrockCountTokensConfig):
verbose_logger.debug("Processing CountTokens request for resolved model: %s", resolved_model)
# Get AWS region using existing LiteLLM function
aws_region_name: Final = self.get_aws_region_name(
aws_region_name: Final = await run_aws_signing(
self.get_aws_region_name,
optional_params=litellm_params,
model=resolved_model,
model_id=None,
@ -64,7 +65,8 @@ class BedrockCountTokensHandler(BedrockCountTokensConfig):
# Get endpoint URL using simplified function
api_base: Final = litellm_params.get("api_base", None)
aws_bedrock_runtime_endpoint: Final = litellm_params.get("aws_bedrock_runtime_endpoint", None)
endpoint_url: Final = self.get_bedrock_count_tokens_endpoint(
endpoint_url: Final = await run_aws_signing(
self.get_bedrock_count_tokens_endpoint,
model=resolved_model,
aws_region_name=aws_region_name,
api_base=api_base,

13
litellm/llms/bedrock/count_tokens/mantle_handler.py
View file

@ -1,3 +1,4 @@
from collections.abc import Mapping
from typing import Final
import httpx
@ -6,7 +7,7 @@ from typing_extensions import NotRequired, ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
from litellm.llms.anthropic.count_tokens.transformation import AnthropicCountTokensConfig
from litellm.llms.anthropic.count_tokens.transformation import COUNT_TOKEN_OPTION_NAMES, AnthropicCountTokensConfig
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM, run_aws_signing
from litellm.llms.bedrock.common_utils import BedrockError, build_mantle_messages_url
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, get_async_httpx_client
@ -23,6 +24,7 @@ class MantleCountTokensRequest(TypedDict):
_COUNT_REQUEST: Final = TypeAdapter(MantleCountTokensRequest)
_COUNT_RESPONSE: Final = TypeAdapter(dict[str, JsonValue])
_COUNT_OPTIONS: Final[TypeAdapter[Mapping[str, JsonValue]]] = TypeAdapter(Mapping[str, JsonValue])
class BedrockMantleCountTokensHandler(BaseAWSLLM):
@ -52,15 +54,18 @@ class BedrockMantleCountTokensHandler(BaseAWSLLM):
) -> dict[str, JsonValue]:
try:
request: Final = _COUNT_REQUEST.validate_python(request_data)
aws_region_name: Final = self.get_aws_region_name(
optional_params=litellm_params, model=resolved_model, model_id=None
aws_region_name: Final = await run_aws_signing(
self.get_aws_region_name, optional_params=litellm_params, model=resolved_model, model_id=None
)
endpoint_url: Final = self.get_mantle_count_tokens_endpoint(aws_region_name)
endpoint_url: Final = await run_aws_signing(self.get_mantle_count_tokens_endpoint, aws_region_name)
body: Final = self._anthropic_config.transform_request_to_count_tokens(
model=resolved_model,
messages=request["messages"],
tools=request.get("tools"),
system=request.get("system"),
optional_params=_COUNT_OPTIONS.validate_python(
{name: value for name, value in request_data.items() if name in COUNT_TOKEN_OPTION_NAMES}
),
)
verbose_logger.debug("Making bedrock-mantle count_tokens request to: %s", endpoint_url)
api_key: Final = litellm_params.get("api_key")

9
litellm/llms/bedrock/count_tokens/transformation.py
View file

@ -52,12 +52,15 @@ def _sanitized_bedrock_tool_name(raw_name: JsonValue) -> str:
def _to_bedrock_tool_spec(tool: JsonValue) -> dict[str, JsonValue]:
fields: Final = _json_dict(tool)
name: Final = _sanitized_bedrock_tool_name(fields.get("name", ""))
chat_function: Final = _json_dict(fields.get("function"))
spec: Final = chat_function or fields
name: Final = _sanitized_bedrock_tool_name(spec.get("name", ""))
schema_field: Final = "parameters" if chat_function else "input_schema"
return {
"toolSpec": {
"name": name,
"description": fields.get("description") or name,
"inputSchema": {"json": fields.get("input_schema", {"type": "object", "properties": {}})},
"description": spec.get("description") or name,
"inputSchema": {"json": spec.get(schema_field, {"type": "object", "properties": {}})},
}
}

254
litellm/llms/bedrock/files/transformation.py
View file

@ -4,7 +4,7 @@ import os
import posixpath
import time
import xml.etree.ElementTree as ET
from collections.abc import Iterable, Mapping, MutableMapping, Sequence
from collections.abc import Iterable, Iterator, Mapping, MutableMapping, Sequence
from contextlib import suppress
from dataclasses import dataclass
from datetime import datetime
@ -23,6 +23,7 @@ from typing_extensions import ReadOnly
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.constants import BEDROCK_INVOKE_PROVIDERS_LITERAL
from litellm.exceptions import BadRequestError
from litellm.files.utils import FilesAPIUtils
from litellm.litellm_core_utils.aws_partition import get_aws_dns_suffix
from litellm.litellm_core_utils.cloud_storage_security import (
@ -324,7 +325,10 @@ _REJECTED_FILE_ID_REQUEST_URL: Final = "https://litellm.ai"
def _rejected_file_id(reason: ValueError) -> BedrockError:
message: Final = str(reason)
return _client_error(str(reason))
def _client_error(message: str) -> BedrockError:
return BedrockError(
status_code=400,
message=message,
@ -336,6 +340,92 @@ def _rejected_file_id(reason: ValueError) -> BedrockError:
)
@dataclass(frozen=True, slots=True)
class _BatchInputLine:
line_number: int
record: _OpenAIBatchRecord
_BATCH_RECORD_CONVERSION_ERRORS: Final = (AttributeError, LookupError, TypeError, ValueError, BadRequestError)
def _batch_line_error(line_number: int, problem: str) -> BedrockError:
return _client_error(f"Batch input file line {line_number} {problem}")
def _decode_batch_input(content: str | bytes) -> str:
if isinstance(content, str):
return content.removeprefix("\ufeff")
try:
return content.decode("utf-8-sig")
except UnicodeDecodeError as e:
raise _batch_line_error(content.count(b"\n", 0, e.start) + 1, f"is not valid UTF-8: {e.reason}") from e
def _load_batch_line(line_number: int, line: str) -> _OpenAIBatchRecord:
try:
return json.loads(line)
except json.JSONDecodeError as e:
raise _batch_line_error(line_number, f"is not valid JSON: {e.msg} at column {e.colno}") from e
except (ValueError, RecursionError) as e:
raise _batch_line_error(line_number, f"could not be parsed: {e}") from e
def _json_object(value: object) -> Mapping[str, object] | None:
return value if isinstance(value, dict) else None
def _batch_json_object(line_number: int, line: str) -> _OpenAIBatchRecord:
record: Final = _load_batch_line(line_number, line)
if _json_object(record) is None:
raise _batch_line_error(line_number, "is not a JSON object")
return record
def _batch_body_problem(record: Mapping[str, object]) -> str | None:
body: Final = _json_object(record.get("body"))
if body is None:
return "has a body that is missing or not a JSON object"
if "model" in body and not isinstance(body["model"], str):
return "has a body.model that is not a string"
return None
def _batch_input_line(line_number: int, line: str) -> _BatchInputLine:
record: Final = _batch_json_object(line_number, line)
problem: Final = _batch_body_problem(record)
if problem is not None:
raise _batch_line_error(line_number, problem)
return _BatchInputLine(line_number=line_number, record=record)
def _non_blank_lines(file_content: str) -> Iterator[tuple[int, str]]:
numbered_lines: Final = enumerate(file_content.splitlines(), start=1)
return ((line_number, line) for line_number, line in numbered_lines if line.strip())
def _parse_batch_jsonl(file_content: str) -> tuple[_BatchInputLine, ...]:
return tuple(_batch_input_line(line_number, line) for line_number, line in _non_blank_lines(file_content))
def _batch_file_model(file_content: str) -> str | None:
records: Final = tuple(
_batch_json_object(line_number, line) for line_number, line in _non_blank_lines(file_content)
)
if not records:
return None
body: Final = _json_object(records[0].get("body"))
model: Final = body.get("model") if body is not None else None
return model if isinstance(model, str) else ""
def _batch_s3_object_name(model: str) -> str:
safe_model: Final = sanitize_cloud_object_component(
model.removeprefix("bedrock/").replace(":", "-"), fallback="model"
)
return f"{BEDROCK_MANAGED_S3_BATCH_PREFIX}{safe_model}-{uuid.uuid4()}.jsonl"
def _resolve_managed_s3_object(file_id: str, litellm_params: Mapping[str, object]) -> tuple[str, str]:
configured_bucket_names: Final = get_configured_s3_bucket_names(litellm_params)
allow_legacy_cloud_file_ids: Final = should_allow_legacy_cloud_file_ids(litellm_params)
@ -545,29 +635,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
# File-like objects need to be read
content = file_content.read()
# Ensure content is string
if isinstance(content, bytes):
content = content.decode("utf-8")
return content
def _get_s3_object_name_from_batch_jsonl(
self,
openai_jsonl_content: Sequence[_OpenAIBatchRecord],
) -> str:
"""
Gets a unique S3 object name for the Bedrock batch processing job
named as: litellm-bedrock-files/{model}/{uuid}
"""
_model = openai_jsonl_content[0].get("body", {}).get("model", "")
# Remove bedrock/ prefix if present
_model = _model.removeprefix("bedrock/")
safe_model: Final = sanitize_cloud_object_component(_model.replace(":", "-"), fallback="model")
object_name: Final = f"{BEDROCK_MANAGED_S3_BATCH_PREFIX}{safe_model}-{uuid.uuid4()}.jsonl"
return object_name
return _decode_batch_input(content)
def get_object_name(self, extracted_file_data: ExtractedFileData, purpose: str) -> str:
"""
@ -581,11 +649,9 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
if purpose == "batch":
## 1. If jsonl, check if there's a model name
file_content: Final = self._get_content_from_openai_file(extracted_file_data_content)
# Split into lines and parse each line as JSON
openai_jsonl_content: Final = [json.loads(line) for line in file_content.splitlines() if line.strip()]
if len(openai_jsonl_content) > 0:
return self._get_s3_object_name_from_batch_jsonl(openai_jsonl_content)
batch_model: Final = _batch_file_model(file_content)
if batch_model is not None:
return _batch_s3_object_name(batch_model)
## 2. If not jsonl, store under a server-generated managed object name
filename: Final = extracted_file_data.get("filename")
@ -1080,42 +1146,60 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
}
}
"""
return [
self._transform_openai_batch_record(idx, openai_record, target_model)
for idx, openai_record in enumerate(openai_jsonl_content)
]
def transform_batch_input_lines(
self, batch_input_lines: Sequence[_BatchInputLine], target_model: str = ""
) -> tuple[_BedrockBatchRecord, ...]:
return tuple(
self._transform_batch_input_line(idx, batch_input_line, target_model)
for idx, batch_input_line in enumerate(batch_input_lines)
)
def _transform_batch_input_line(
self, idx: int, batch_input_line: _BatchInputLine, target_model: str
) -> _BedrockBatchRecord:
try:
return self._transform_openai_batch_record(idx, batch_input_line.record, target_model)
except _BATCH_RECORD_CONVERSION_ERRORS as e:
raise _batch_line_error(
batch_input_line.line_number, f"could not be converted to a Bedrock batch record: {e}"
) from e
def _transform_openai_batch_record(
self, idx: int, openai_record: _OpenAIBatchRecord, target_model: str
) -> _BedrockBatchRecord:
import litellm
bedrock_jsonl_content: Final = []
for idx, _openai_jsonl_content in enumerate(openai_jsonl_content):
# Extract the request body from OpenAI format
openai_body = _openai_jsonl_content.get("body", {})
record_model = openai_body.get("model", "")
resolved_model = litellm.model_alias_map.get(record_model, record_model)
model_for_transform, provider = self._resolve_batch_record_model_and_provider(
record_model=resolved_model, target_model=target_model
openai_body: Final = openai_record.get("body", {})
record_model: Final = openai_body.get("model", "")
resolved_model: Final = litellm.model_alias_map.get(record_model, record_model)
model_for_transform, provider = self._resolve_batch_record_model_and_provider(
record_model=resolved_model, target_model=target_model
)
# Route to the embedding transformer when the OpenAI batch line
# targets /v1/embeddings; every other endpoint shape is normalized
# to chat completions first. We branch here (rather than inside
# `_map_openai_to_bedrock_params`) so the chat helper keeps its
# narrow contract and the embedding helper can evolve independently.
record_kind: Final = self._classify_batch_record(openai_record)
model_input: Final = (
self._map_openai_embedding_to_bedrock_params(openai_request_body=openai_body, model=model_for_transform)
if record_kind is BedrockBatchRecordKind.EMBEDDING
else self._map_openai_to_bedrock_params(
openai_request_body=self._transform_batch_body_to_chat_body(openai_body, record_kind),
model=model_for_transform,
provider=provider,
)
# Route to the embedding transformer when the OpenAI batch line
# targets /v1/embeddings; every other endpoint shape is normalized
# to chat completions first. We branch here (rather than inside
# `_map_openai_to_bedrock_params`) so the chat helper keeps its
# narrow contract and the embedding helper can evolve independently.
record_kind = self._classify_batch_record(_openai_jsonl_content)
if record_kind is BedrockBatchRecordKind.EMBEDDING:
model_input = self._map_openai_embedding_to_bedrock_params(
openai_request_body=openai_body, model=model_for_transform
)
else:
model_input = self._map_openai_to_bedrock_params(
openai_request_body=self._transform_batch_body_to_chat_body(openai_body, record_kind),
model=model_for_transform,
provider=provider,
)
# Create Bedrock batch record
record_id = _openai_jsonl_content.get("custom_id", f"CALL{str(idx).zfill(7)}")
bedrock_record = {"recordId": record_id, "modelInput": model_input}
bedrock_jsonl_content.append(bedrock_record)
return bedrock_jsonl_content
)
return _BedrockBatchRecord(
recordId=openai_record.get("custom_id", f"CALL{str(idx).zfill(7)}"),
modelInput=model_input,
)
def transform_openai_jsonl_content_to_bedrock_jsonl_content(
self,
@ -1151,11 +1235,11 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
):
## Transform JSONL content to Bedrock format
original_file_content: Final = self._get_content_from_openai_file(extracted_file_data_content)
openai_jsonl_content = [json.loads(line) for line in original_file_content.splitlines() if line.strip()]
batch_input_lines: Final = _parse_batch_jsonl(original_file_content)
litellm_params_model: Final = litellm_params.get("model")
target_model: Final = model or (litellm_params_model if isinstance(litellm_params_model, str) else "")
bedrock_jsonl_content = self._transform_openai_jsonl_content_to_bedrock_jsonl_content(
openai_jsonl_content, target_model=target_model
bedrock_jsonl_content: Final = self.transform_batch_input_lines(
batch_input_lines, target_model=target_model
)
file_content = "\n".join(json.dumps(item) for item in bedrock_jsonl_content)
elif isinstance(extracted_file_data_content, bytes):
@ -1662,37 +1746,11 @@ class BedrockJsonlFilesTransformation:
# Read the content of the file
file_content: Final = self._get_content_from_openai_file(openai_file_content)
# Split into lines and parse each line as JSON
openai_jsonl_content: Final = [json.loads(line) for line in file_content.splitlines() if line.strip()]
bedrock_jsonl_content = self._transform_openai_jsonl_content_to_bedrock_jsonl_content(openai_jsonl_content)
batch_input_lines: Final = _parse_batch_jsonl(file_content)
bedrock_jsonl_content: Final = BedrockFilesConfig().transform_batch_input_lines(batch_input_lines)
bedrock_jsonl_string: Final = "\n".join(json.dumps(item) for item in bedrock_jsonl_content)
object_name: Final = self._get_s3_object_name(openai_jsonl_content=openai_jsonl_content)
return bedrock_jsonl_string, object_name
def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
self, openai_jsonl_content: Sequence[_OpenAIBatchRecord]
):
"""
Delegate to the main BedrockFilesConfig transformation method
"""
config: Final = BedrockFilesConfig()
return config.transform_openai_jsonl_content_to_bedrock_jsonl_content(openai_jsonl_content)
def _get_s3_object_name(
self,
openai_jsonl_content: Sequence[_OpenAIBatchRecord],
) -> str:
"""
Gets a unique S3 object name for the Bedrock batch processing job
named as: litellm-bedrock-files-{model}-{uuid}
"""
_model = openai_jsonl_content[0].get("body", {}).get("model", "")
# Remove bedrock/ prefix if present
_model = _model.removeprefix("bedrock/")
safe_model: Final = sanitize_cloud_object_component(_model.replace(":", "-"), fallback="model")
object_name: Final = f"{BEDROCK_MANAGED_S3_BATCH_PREFIX}{safe_model}-{uuid.uuid4()}.jsonl"
return object_name
first_model: Final = batch_input_lines[0].record.get("body", {}).get("model", "") if batch_input_lines else ""
return bedrock_jsonl_string, _batch_s3_object_name(first_model)
def _get_content_from_openai_file(self, openai_file_content: FileTypes) -> str:
"""
@ -1725,11 +1783,7 @@ class BedrockJsonlFilesTransformation:
# File-like objects need to be read
content = file_content.read()
# Ensure content is string
if isinstance(content, bytes):
content = content.decode("utf-8")
return content
return _decode_batch_input(content)
def transform_s3_bucket_response_to_openai_file_object(
self, create_file_data: CreateFileRequest, s3_upload_response: _S3UploadResponse

5
litellm/llms/bedrock/realtime/handler.py
View file

@ -337,10 +337,7 @@ class BedrockRealtime(BaseAWSLLM):
# Get AWS region
if aws_region_name is None:
optional_params: Final = {
"aws_region_name": aws_region_name,
}
aws_region_name = self._get_aws_region_name(optional_params, model)
aws_region_name = await run_aws_signing(self.get_aws_region_name, {}, model)
# Get endpoint URL
if api_base is not None:

2
litellm/llms/chatgpt/chat/transformation.py
View file

@ -72,7 +72,7 @@ class ChatGPTConfig(OpenAIConfig):
default_headers: Final = get_chatgpt_default_headers(api_key or "", account_id, session_id)
return {**default_headers, **validated_headers}
def post_stream_processing(self, stream: Any) -> Any:
def post_stream_processing(self, stream: Any) -> ChatGPTToolCallNormalizer:
return ChatGPTToolCallNormalizer(stream)
def map_openai_params(

74
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -104,7 +104,7 @@ from litellm.llms.base_llm.vector_store_files.transformation import (
BaseVectorStoreFilesConfig,
)
from litellm.llms.base_llm.videos.transformation import BaseVideoConfig
from litellm.llms.bedrock.base_aws_llm import SignsRequestsWithAWS, run_aws_signing, sign_request_off_loop_if_aws
from litellm.llms.bedrock.base_aws_llm import SignsRequestsWithAWS, run_aws_signing, run_off_loop_if_aws
from litellm.llms.custom_httpx.container_handler import raise_for_error_status
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
@ -2085,14 +2085,18 @@ class BaseLLMHTTPHandler:
raise ValueError(f"No provider config found for provider: {custom_llm_provider}")
# Validate environment and get headers
headers = provider_config.validate_environment(
headers = await run_off_loop_if_aws(
provider_config,
provider_config.validate_environment,
api_key=api_key,
api_base=api_base,
headers=headers or {},
)
# Transform the request first
data: Final = provider_config.transform_search_request(
data: Final = await run_off_loop_if_aws(
provider_config,
provider_config.transform_search_request,
query=query,
optional_params=optional_params,
api_key=api_key,
@ -2108,7 +2112,7 @@ class BaseLLMHTTPHandler:
api_key=api_key,
)
signed_headers, signed_json_body = await sign_request_off_loop_if_aws(
signed_headers, signed_json_body = await run_off_loop_if_aws(
provider_config,
provider_config.sign_request,
headers=headers,
@ -2236,7 +2240,7 @@ class BaseLLMHTTPHandler:
)
provider_config.transform_anthropic_messages_request_on_http_error(e=e, request_data=request_body)
recovered_errors = (*recovered_errors, e)
headers, signed_json_body = await sign_request_off_loop_if_aws(
headers, signed_json_body = await run_off_loop_if_aws(
provider_config,
provider_config.sign_request,
headers=headers,
@ -2405,7 +2409,9 @@ class BaseLLMHTTPHandler:
logging_obj.model_call_details.update(request_body)
# Make the request
request_url: Final = anthropic_messages_provider_config.get_complete_url(
request_url: Final = await run_off_loop_if_aws(
anthropic_messages_provider_config,
anthropic_messages_provider_config.get_complete_url,
api_base=api_base,
api_key=api_key,
model=model,
@ -2416,7 +2422,7 @@ class BaseLLMHTTPHandler:
stream=stream,
)
headers, signed_json_body = await sign_request_off_loop_if_aws(
headers, signed_json_body = await run_off_loop_if_aws(
anthropic_messages_provider_config,
anthropic_messages_provider_config.sign_request,
headers=headers,
@ -2967,7 +2973,9 @@ class BaseLLMHTTPHandler:
stream = response_api_optional_request_params.get("stream", False)
caller_requested_stream: Final = bool(stream or (extra_body or {}).get("stream"))
api_base: Final = responses_api_provider_config.get_complete_url(
api_base: Final = await run_off_loop_if_aws(
responses_api_provider_config,
responses_api_provider_config.get_complete_url,
api_base=litellm_params.api_base,
litellm_params=dict(litellm_params),
)
@ -3009,7 +3017,7 @@ class BaseLLMHTTPHandler:
fake_stream=fake_stream,
)
headers, signed_body = await sign_request_off_loop_if_aws(
headers, signed_body = await run_off_loop_if_aws(
responses_api_provider_config,
responses_api_provider_config.sign_request,
headers=validated_headers,
@ -3856,7 +3864,9 @@ class BaseLLMHTTPHandler:
litellm_params=litellm_params,
api_key=api_key,
)
complete_api_base: Final = provider_config.get_complete_file_url(
complete_api_base: Final = await run_off_loop_if_aws(
provider_config,
provider_config.get_complete_file_url,
api_base=api_base,
api_key=api_key,
model="",
@ -3867,7 +3877,9 @@ class BaseLLMHTTPHandler:
if not complete_api_base:
raise ValueError("api_base is required for create_file")
return await self.async_create_file(
transformed_request=provider_config.transform_create_file_request(
transformed_request=await run_off_loop_if_aws(
provider_config,
provider_config.transform_create_file_request,
model="",
create_file_data=create_file_data,
litellm_params=litellm_params,
@ -4334,7 +4346,9 @@ class BaseLLMHTTPHandler:
litellm_params=litellm_params,
api_key=api_key,
)
complete_api_base: Final = provider_config.get_complete_batch_url(
complete_api_base: Final = await run_off_loop_if_aws(
provider_config,
provider_config.get_complete_batch_url,
api_base=api_base,
api_key=api_key,
model=model,
@ -4345,7 +4359,9 @@ class BaseLLMHTTPHandler:
if not complete_api_base:
raise ValueError("api_base is required for create_batch")
return await self.async_create_batch(
transformed_request=provider_config.transform_create_batch_request(
transformed_request=await run_off_loop_if_aws(
provider_config,
provider_config.transform_create_batch_request,
model=model,
create_batch_data=create_batch_data,
litellm_params=litellm_params,
@ -4818,7 +4834,9 @@ class BaseLLMHTTPHandler:
**(extra_headers or {}),
}
api_base: Final = responses_api_provider_config.get_complete_url(
api_base: Final = await run_off_loop_if_aws(
responses_api_provider_config,
responses_api_provider_config.get_complete_url,
api_base=litellm_params.api_base,
litellm_params=dict(litellm_params),
)
@ -4836,7 +4854,7 @@ class BaseLLMHTTPHandler:
)
data = BaseResponsesAPIConfig.normalize_responses_api_request_dict(data)
headers, signed_body = await sign_request_off_loop_if_aws(
headers, signed_body = await run_off_loop_if_aws(
responses_api_provider_config,
responses_api_provider_config.sign_request,
headers=validated_headers,
@ -4962,7 +4980,9 @@ class BaseLLMHTTPHandler:
async_httpx_client = client
# Get URL and params from provider config
url, params = provider_config.transform_retrieve_file_request(
url, params = await run_off_loop_if_aws(
provider_config,
provider_config.transform_retrieve_file_request,
file_id=file_id,
optional_params={},
litellm_params=litellm_params,
@ -5089,7 +5109,9 @@ class BaseLLMHTTPHandler:
async_httpx_client = client
# Get URL and params from provider config
url, params = provider_config.transform_delete_file_request(
url, params = await run_off_loop_if_aws(
provider_config,
provider_config.transform_delete_file_request,
file_id=file_id,
optional_params={},
litellm_params=litellm_params,
@ -5214,7 +5236,9 @@ class BaseLLMHTTPHandler:
async_httpx_client = client
# Get URL and params from provider config
url, params = provider_config.transform_list_files_request(
url, params = await run_off_loop_if_aws(
provider_config,
provider_config.transform_list_files_request,
purpose=purpose,
optional_params={},
litellm_params=litellm_params,
@ -5298,7 +5322,9 @@ class BaseLLMHTTPHandler:
)
yield page_files[: MAX_FILE_LIST_LIMIT - listed_count]
listed_count += len(page_files)
next_request = self._next_listing_request(latest_page, provider_config, litellm_params, listed_count)
next_request = await run_off_loop_if_aws(
provider_config, self._next_listing_request, latest_page, provider_config, litellm_params, listed_count
)
if next_request is None:
return
url, params = next_request
@ -5524,7 +5550,9 @@ class BaseLLMHTTPHandler:
headers: dict,
logging_obj: LiteLLMLoggingObj,
) -> "_PreparedFileContentRequest":
url, params = provider_config.transform_file_content_request(
url, params = await run_off_loop_if_aws(
provider_config,
provider_config.transform_file_content_request,
file_content_request=file_content_request,
optional_params={},
litellm_params=litellm_params,
@ -10332,7 +10360,9 @@ class BaseLLMHTTPHandler:
if extra_headers:
headers.update(extra_headers)
api_base: Final = vector_store_provider_config.get_complete_url(
api_base: Final = await run_off_loop_if_aws(
vector_store_provider_config,
vector_store_provider_config.get_complete_url,
api_base=litellm_params.api_base,
litellm_params=dict(litellm_params),
)
@ -10366,7 +10396,7 @@ class BaseLLMHTTPHandler:
)
all_optional_params: Final[dict[str, object]] = dict(litellm_params)
all_optional_params.update(vector_store_search_optional_params or {})
headers, signed_json_body = await sign_request_off_loop_if_aws(
headers, signed_json_body = await run_off_loop_if_aws(
vector_store_provider_config,
vector_store_provider_config.sign_request,
headers=headers,

4
litellm/llms/gemini/common_utils.py
View file

@ -3,6 +3,7 @@ import datetime
import json
import math
from collections.abc import Mapping, Sequence
from types import MappingProxyType
from typing import Any, Final
import httpx
@ -498,6 +499,7 @@ class GoogleAIStudioTokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "openai",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
import copy
@ -522,7 +524,7 @@ class GoogleAIStudioTokenCounter(BaseTokenCounter):
)
if isinstance(request_view, VertexGeminiCountContextError):
return count_context_error_response(request_view, request_model, model_to_use)
count_context: Final = vertex_gemini_count_context(system=request_view.system, tools=tools)
count_context: Final = vertex_gemini_count_context(system=request_view.system, tools=tools, model=model_to_use)
if isinstance(count_context, VertexGeminiCountContextError):
return count_context_error_response(count_context, request_model, model_to_use)
blocks_error: Final = unconvertible_messages_error(

10
litellm/llms/gemini/google_genai/transformation.py
View file

@ -15,8 +15,9 @@ from litellm.llms.base_llm.google_genai.transformation import (
from litellm.llms.vertex_ai.common_utils import (
build_vertex_schema,
supports_response_json_schema,
with_integer_enums_that_fit,
)
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexLLM
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig, VertexLLM
from litellm.types.router import GenericLiteLLMParams
if TYPE_CHECKING:
@ -327,8 +328,11 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
else:
if json_schema_key is not None:
generate_content_config_dict.pop(json_schema_key)
generate_content_config_dict[schema_key] = build_vertex_schema(
parameters=deepcopy(value), add_property_ordering=True
generate_content_config_dict[schema_key] = with_integer_enums_that_fit(
lambda send_integer_enums: build_vertex_schema(
parameters=deepcopy(value), add_property_ordering=True, send_integer_enums=send_integer_enums
),
VertexGeminiConfig.is_gemini_3_or_newer(model),
)
def transform_generate_content_request(

16
litellm/llms/openai/responses/count_tokens/handler.py
View file

@ -34,7 +34,7 @@ class OpenAICountTokensHandler(OpenAICountTokensConfig):
tools: Sequence[Mapping[str, object]] | None = None,
instructions: str | None = None,
text: Mapping[str, object] | None = None,
) -> dict[str, object]:
) -> Mapping[str, object]:
"""
Handle a token counting request to OpenAI's Responses API.
@ -56,7 +56,19 @@ class OpenAICountTokensHandler(OpenAICountTokensConfig):
instructions=instructions,
text=text,
)
except ValueError as e:
verbose_logger.error("Error in CountTokens handler: %s", e)
raise OpenAIError(status_code=500, message=f"CountTokens processing error: {e}")
return await self.post_count_request(request_body, api_key, api_base, timeout)
async def post_count_request(
self,
request_body: Mapping[str, object],
api_key: str,
api_base: str | None = None,
timeout: float | httpx.Timeout | None = None,
) -> Mapping[str, object]:
try:
endpoint_url: Final = self.get_openai_count_tokens_endpoint(api_base)
verbose_logger.debug("Making request to: %s", endpoint_url)
@ -70,7 +82,7 @@ class OpenAICountTokensHandler(OpenAICountTokensConfig):
response: Final = await async_client.post(
endpoint_url,
headers=headers,
json=request_body,
json=dict(request_body),
timeout=request_timeout,
)

163
litellm/llms/openai/responses/count_tokens/token_counter.py
View file

@ -2,7 +2,9 @@
OpenAI Token Counter implementation using the Responses API /input_tokens endpoint.
"""
from collections.abc import Mapping, Sequence
from collections.abc import Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import Any, Final
import openai
@ -11,6 +13,8 @@ from typing_extensions import assert_never
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.core_helpers import as_str_mapping
from litellm.litellm_core_utils.tool_fields import DropPolicy
from litellm.llms.anthropic.pass_through.convertible_blocks import deployment_drops_params, unconvertible_blocks_error
from litellm.llms.anthropic.pass_through.responses_adapters.transformation import (
LiteLLMAnthropicToResponsesAPIAdapter,
@ -23,6 +27,7 @@ from litellm.llms.openai.responses.count_tokens.handler import (
from litellm.llms.openai.responses.count_tokens.transformation import (
OpenAICountTokensConfig,
ResponsesCountRequestParts,
native_count_request,
)
from litellm.llms.openai.workload_identity import (
OpenAIWorkloadIdentityConfig,
@ -50,6 +55,46 @@ def _workload_identity_config(
return e
@dataclass(frozen=True, slots=True)
class _CountCredential:
api_base: str | None
credential: str | OpenAIWorkloadIdentityConfig
def _count_credential(deployment: Mapping[str, object] | None, model_label: str) -> _CountCredential | OpenAIError:
litellm_params: Final = as_str_mapping(None if deployment is None else deployment.get("litellm_params")) or {}
api_base: Final = litellm_params.get("api_base")
api_key: Final = litellm_params.get("api_key")
credentials: Final = get_openai_credentials(
api_base=api_base if isinstance(api_base, str) else None,
api_key=api_key if isinstance(api_key, str) else None,
)
credential: Final = openai_static_api_key(credentials.api_key) or _workload_identity_config(litellm_params)
if isinstance(credential, OpenAIError):
return credential
if credential is None:
return OpenAIError(
status_code=401,
message=(
f"No OpenAI API key is set for model {model_label}, so tokens can't be counted. Set api_key or "
"workload identity on the deployment, or OPENAI_API_KEY in the proxy environment."
),
)
return _CountCredential(api_base=credentials.api_base, credential=credential)
def _count_error(model_to_use: str, request_model: str, message: str, status_code: int) -> TokenCountResponse:
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_to_use,
tokenizer_type="openai_api",
error=True,
error_message=message,
status_code=status_code,
)
class OpenAITokenCounter(BaseTokenCounter):
"""Token counter implementation for OpenAI provider using the Responses API."""
@ -102,6 +147,7 @@ class OpenAITokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "openai",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
"""
Count tokens using OpenAI's Responses API /input_tokens endpoint.
@ -129,41 +175,9 @@ class OpenAITokenCounter(BaseTokenCounter):
rejected_before_provider_call=True,
)
deployment = deployment or {}
litellm_params: Final = deployment.get("litellm_params", {})
credentials: Final = get_openai_credentials(
api_base=litellm_params.get("api_base"), api_key=litellm_params.get("api_key")
)
credential: Final = openai_static_api_key(credentials.api_key) or _workload_identity_config(litellm_params)
if isinstance(credential, OpenAIError):
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_to_use,
tokenizer_type="openai_api",
error=True,
error_message=credential.message,
status_code=credential.status_code,
)
if credential is None:
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_to_use,
tokenizer_type="openai_api",
error=True,
error_message=(
f"No OpenAI API key is set for model {request_model or model_to_use}, so tokens can't be "
"counted. Set api_key or workload identity on the deployment, or OPENAI_API_KEY in the proxy "
"environment."
),
status_code=401,
)
api_base: Final = credentials.api_base
resolved: Final = _count_credential(deployment, request_model or model_to_use)
if isinstance(resolved, OpenAIError):
return _count_error(model_to_use, request_model, resolved.message, resolved.status_code)
input_items, instructions, counted_tools, text = self._responses_request_parts(
model_to_use=model_to_use,
@ -178,13 +192,9 @@ class OpenAITokenCounter(BaseTokenCounter):
if not input_items:
return None
try:
api_key: Final = (
credential
if isinstance(credential, str)
else await asyncify(get_workload_identity_bearer_token)(credential)
)
result: Final = await openai_count_tokens_handler.handle_count_tokens_request(
return await self._counted(
resolved,
lambda api_key, api_base: openai_count_tokens_handler.handle_count_tokens_request(
model=model_to_use,
input=input_items,
api_key=api_key,
@ -192,7 +202,44 @@ class OpenAITokenCounter(BaseTokenCounter):
tools=counted_tools,
instructions=instructions,
text=text,
),
model_to_use,
request_model,
)
async def count_responses_request(
self,
model_to_use: str,
request: Mapping[str, object],
policy: DropPolicy,
deployment: Mapping[str, object] | None,
request_model: str,
) -> TokenCountResponse | None:
resolved: Final = _count_credential(deployment, request_model or model_to_use)
if isinstance(resolved, OpenAIError):
return _count_error(model_to_use, request_model, resolved.message, resolved.status_code)
body: Final = native_count_request(request, model_to_use, policy)
return await self._counted(
resolved,
lambda api_key, api_base: openai_count_tokens_handler.post_count_request(body, api_key, api_base),
model_to_use,
request_model,
)
@staticmethod
async def _counted(
resolved: _CountCredential,
send: Callable[[str, str | None], Awaitable[Mapping[str, object]]],
model_to_use: str,
request_model: str,
) -> TokenCountResponse | None:
try:
api_key: Final = (
resolved.credential
if isinstance(resolved.credential, str)
else await asyncify(get_workload_identity_bearer_token)(resolved.credential)
)
result: Final = await send(api_key, resolved.api_base)
if result is not None:
return TokenCountResponse.model_validate(
@ -206,36 +253,12 @@ class OpenAITokenCounter(BaseTokenCounter):
)
except OpenAIError as e:
verbose_logger.warning("OpenAI CountTokens API error: status=%s, message=%s", e.status_code, e.message)
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_to_use,
tokenizer_type="openai_api",
error=True,
error_message=e.message,
status_code=e.status_code,
)
return _count_error(model_to_use, request_model, e.message, e.status_code)
except openai.AuthenticationError as e:
verbose_logger.warning("OpenAI workload identity token exchange failed: %s", e.message)
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_to_use,
tokenizer_type="openai_api",
error=True,
error_message=e.message,
status_code=401,
)
return _count_error(model_to_use, request_model, e.message, 401)
except Exception as e:
verbose_logger.warning("Error calling OpenAI CountTokens API: %s", e)
return TokenCountResponse(
total_tokens=0,
request_model=request_model,
model_used=model_to_use,
tokenizer_type="openai_api",
error=True,
error_message=str(e),
status_code=500,
)
return _count_error(model_to_use, request_model, str(e), 500)
return None

36
litellm/llms/openai/responses/count_tokens/transformation.py
View file

@ -4,12 +4,15 @@ OpenAI Responses API token counting transformation logic.
This module handles the transformation of requests to OpenAI's /v1/responses/input_tokens endpoint.
"""
import copy
from collections.abc import Mapping, Sequence
from typing import Any, Final, Literal, TypeAlias
from openai.types.responses.input_token_count_params import InputTokenCountParams
from pydantic import JsonValue, TypeAdapter, ValidationError
from typing_extensions import ReadOnly, TypedDict
from litellm.litellm_core_utils.tool_fields import DropPolicy
from litellm.llms.anthropic.pass_through.convertible_blocks import malformed_tool_result_text_error
from litellm.llms.anthropic.pass_through.messages.utils import normalize_native_messages
from litellm.llms.anthropic.pass_through.responses_adapters.transformation import (
@ -17,6 +20,8 @@ from litellm.llms.anthropic.pass_through.responses_adapters.transformation impor
)
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.types.llms.anthropic import AnthropicMessagesRequest, AnthropicOutputSchema
from litellm.types.llms.openai import ResponsesAPIRequestParams
from litellm.types.utils import all_litellm_params
class ResponsesInputTextPart(TypedDict):
@ -340,3 +345,34 @@ class OpenAICountTokensConfig:
input_items: object, instructions: object, tools: object, text: object = None
) -> ResponsesCountRequestParts:
return _RESPONSES_COUNT_REQUEST_PARTS.validate_python((input_items, instructions, tools, text))
_COUNT_FIELDS_THE_SDK_TYPE_OMITS: Final = frozenset({"prompt"})
_COUNT_REQUEST_FIELDS: Final = (
InputTokenCountParams.__required_keys__ | InputTokenCountParams.__optional_keys__ | _COUNT_FIELDS_THE_SDK_TYPE_OMITS
)
_INFERENCE_REQUEST_FIELDS: Final = (
ResponsesAPIRequestParams.__required_keys__ | ResponsesAPIRequestParams.__optional_keys__
)
_LITELLM_REQUEST_FIELDS: Final = frozenset((*all_litellm_params, "drop_params", "additional_drop_params"))
def _forwards(field: str, policy: DropPolicy) -> bool:
if field in _LITELLM_REQUEST_FIELDS or field in policy.additional_drop_params:
return False
return field in _COUNT_REQUEST_FIELDS or (not policy.drop_params and field in _INFERENCE_REQUEST_FIELDS)
def _restored(field: str, value: object) -> object:
match field, value:
case "input", _:
return ResponsesAPIRequestUtils.restore_encrypted_content_item_ids_in_input(copy.deepcopy(value))
case "previous_response_id", str():
return ResponsesAPIRequestUtils.decode_previous_response_id_to_original_previous_response_id(value)
case _:
return value
def native_count_request(request: Mapping[str, object], model: str, policy: DropPolicy) -> Mapping[str, object]:
forwarded: Final = {field: _restored(field, value) for field, value in request.items() if _forwards(field, policy)}
return {**forwarded, "model": model}

115
litellm/llms/opencode/harness/transformation.py
View file

@ -8,17 +8,31 @@ repo's own opencode.json cannot redirect model calls. The token is never in argv
the config references it with `{file:<private_dir>/token}`. XDG dirs point at a persisted
LiteLLM-owned root so the user's opencode config and auth are never read, and the session
DB outlives a session for resume. Verified against opencode 1.14.41.
opencode files each session under the folder it ran in and only streams a resumed session's
events from that folder, so resuming one filed elsewhere (by an older LiteLLM that left PWD
unset, or from another sandbox) hangs. The first turn after a resume runs `opencode export` in
the sandbox, and when the session is filed elsewhere it imports a copy under fresh ids, filed
under the sandbox folder, and continues the copy. Import overwrites the folder of a session id it
already has, so reusing the ids would break the original wherever it was saved. Before 1.15.6,
import files a session under the folder named in its JSON, so the copy names the sandbox folder.
A failed or timed-out export or import fails that turn instead of hanging, and the next turn
tries again. Verified against opencode 1.14.41, 1.15.5, 1.15.6 and 1.19.3.
"""
from __future__ import annotations
import itertools
import json
from collections.abc import Mapping, Sequence
import secrets
import string
from collections.abc import Iterator, Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final
from pydantic import BaseModel, ConfigDict, JsonValue, TypeAdapter, ValidationError
from litellm.harness.errors import CapabilityUnsupported, HarnessError, OptionsMismatch
from litellm.harness.options import OpenCodeOptions
from litellm.harness.types import (
@ -33,6 +47,7 @@ from litellm.harness.types import (
)
from litellm.llms.base_llm.harness.transformation import (
BaseCLIHarnessConfig,
HarnessSessionCopy,
HarnessSessionSetup,
HarnessTurnError,
HarnessTurnRequest,
@ -57,6 +72,10 @@ OPENCODE_PROVIDER_NPM: Final = "@ai-sdk/openai-compatible"
# A fixed title skips opencode's extra title-generation model call on the first turn.
OPENCODE_SESSION_TITLE: Final = "litellm-harness"
TOKEN_FILENAME: Final = "token"
RESUME_COPY_FILENAME: Final = "resume-session.json"
ID_ORDERED_PREFIX_LENGTH: Final = 16
ID_RANDOM_LENGTH: Final = 14
ID_RANDOM_ALPHABET: Final = string.ascii_letters + string.digits
INSTRUCTIONS_FILENAME: Final = "instructions.md"
XDG_DIRNAME: Final = "xdg"
XDG_SUBDIRS: Final = ("config", "data", "state", "cache")
@ -162,6 +181,94 @@ class OpenCodeStreamState:
step_texts: Sequence[str] = ()
class ExportedPart(BaseModel):
model_config = ConfigDict(frozen=True)
id: str
class ExportedMessageInfo(BaseModel):
model_config = ConfigDict(frozen=True)
id: str
class ExportedMessage(BaseModel):
model_config = ConfigDict(frozen=True)
info: ExportedMessageInfo
parts: tuple[ExportedPart, ...]
class ExportedSessionInfo(BaseModel):
model_config = ConfigDict(frozen=True)
id: str
directory: str
class ExportedSession(BaseModel):
"""The ids and folder of an `opencode export`; every other field is copied as is."""
model_config = ConfigDict(frozen=True)
info: ExportedSessionInfo
messages: tuple[ExportedMessage, ...]
def ids(self) -> Iterator[str]:
yield self.info.id
for message in self.messages:
yield message.info.id
yield from (part.id for part in message.parts)
EXPORT_JSON: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
def fresh_id(old: str) -> str:
"""Keeps the type prefix and time-ordered part of an opencode id, so the copy sorts like the original."""
suffix: Final = "".join(secrets.choice(ID_RANDOM_ALPHABET) for _ in range(ID_RANDOM_LENGTH))
return f"{old[:ID_ORDERED_PREFIX_LENGTH]}{suffix}"
def with_id_fields_replaced(value: JsonValue, fresh: Mapping[str, str], key: str = "") -> JsonValue:
match value:
case str() if key == "id" or key.endswith("ID"):
return fresh.get(value, value)
case list():
return [with_id_fields_replaced(item, fresh, key) for item in value]
case dict():
return {name: with_id_fields_replaced(item, fresh, name) for name, item in value.items()}
case _:
return value
def filed_under(exported: JsonValue, workdir: str) -> JsonValue:
match exported:
case {"info": dict() as info}:
return {**exported, "info": {**info, "directory": workdir}}
case _:
return exported
def session_copy(exported: str, path: str, workdir: str) -> HarnessSessionCopy | None:
"""A copy of an exported session filed under workdir, or None when the session is already filed there."""
try:
raw: Final = EXPORT_JSON.validate_json(exported)
session: Final = ExportedSession.model_validate(raw)
except ValidationError as exc:
raise HarnessError(f"opencode export did not print a session: {exc.errors()[0]['msg']}") from exc
if session.info.directory == workdir:
return None
fresh: Final = MappingProxyType({old: fresh_id(old) for old in session.ids()})
return HarnessSessionCopy(
session_id=fresh[session.info.id],
path=path,
data=json.dumps(filed_under(with_id_fields_replaced(raw, fresh), workdir)).encode("utf-8"),
import_command=(OPENCODE_BINARY, "--pure", "import", path),
)
def _as_dict(value: object) -> Mapping[str, Any]:
return value if isinstance(value, dict) else MappingProxyType({})
@ -362,6 +469,12 @@ class OpenCodeHarnessConfig(BaseCLIHarnessConfig):
# The prompt goes on stdin; opencode appends non-TTY stdin to the message.
return HarnessTurnRequest(argv=argv, env=setup.env, stdin=turn_prompt(ctx, prompt), cwd=ctx.sandbox.workdir)
def transform_resume_export_command(self, native_session_id: str) -> Sequence[str]:
return (OPENCODE_BINARY, "--pure", "export", native_session_id)
def transform_resume_copy(self, exported: str, private_dir: str, workdir: str) -> HarnessSessionCopy | None:
return session_copy(exported, f"{private_dir}/{RESUME_COPY_FILENAME}", workdir)
def create_stream_state(self) -> OpenCodeStreamState:
return OpenCodeStreamState()

195
litellm/llms/vertex_ai/common_utils.py
View file

@ -1,11 +1,12 @@
import re
from collections.abc import Mapping
from collections.abc import Callable, Iterable, Mapping, Sequence
from copy import deepcopy
from datetime import datetime, timezone
from enum import Enum
from functools import lru_cache
from itertools import accumulate, chain, repeat, takewhile
from types import MappingProxyType
from typing import Any, Final, Literal, cast, get_type_hints
from typing import Any, Final, Literal, TypeVar, cast, get_type_hints
import httpx
from pydantic import JsonValue, TypeAdapter, ValidationError
@ -666,12 +667,89 @@ def _fix_enum_empty_strings(schema, depth=0):
_fix_enum_empty_strings(items, depth=depth + 1)
def _fix_enum_types(schema, depth=0):
"""Remove `enum` fields when the schema type is not string.
GEMINI_MAX_ENUM_CHARACTERS: Final = 2000
_GEMINI_EXACT_INT_MAX: Final = 2**53 - 1
Gemini / Vertex APIs only allow enums for string-typed fields. When an enum
is present on a non-string typed property (or when `anyOf` types do not
include a string type), remove the enum to avoid provider validation errors.
def _gemini_integer_enum_value(value: object) -> str | None:
match value:
case bool():
return None
case int() if -_GEMINI_EXACT_INT_MAX <= value <= _GEMINI_EXACT_INT_MAX:
return str(int(value))
case str() if (
re.fullmatch(r"-?[0-9]{1,16}", value) and -_GEMINI_EXACT_INT_MAX <= int(value) <= _GEMINI_EXACT_INT_MAX
):
return str(int(value))
case _:
return None
def _gemini_integer_enum(values: Sequence[object]) -> tuple[str, ...] | None:
if not values:
return None
converted: Final = tuple(text for value in values if (text := _gemini_integer_enum_value(value)) is not None)
return converted if len(converted) == len(values) else None
def _json_children(node: JsonValue) -> Iterable[JsonValue]:
match node:
case dict():
return node.values()
case list():
return node
case _:
return ()
def _next_json_level(level: tuple[JsonValue, ...], _: None) -> tuple[JsonValue, ...]:
return tuple(chain.from_iterable(_json_children(node) for node in level))
def _enum_size(node: JsonValue) -> tuple[bool, int] | None:
match node:
case {"enum": list() as values}:
return (str(node.get("type")).lower() == "integer", sum(len(str(value)) for value in values))
case _:
return None
def _integer_enums_fit(schema: JsonValue) -> bool:
levels: Final = takewhile(bool, accumulate(repeat(None), _next_json_level, initial=(schema,)))
enum_sizes: Final = tuple(size for node in chain.from_iterable(levels) if (size := _enum_size(node)) is not None)
return (
not any(is_integer for is_integer, _ in enum_sizes)
or sum(characters for _, characters in enum_sizes) <= GEMINI_MAX_ENUM_CHARACTERS
)
_GeminiRequestPart: Final = TypeVar("_GeminiRequestPart")
def with_integer_enums_that_fit(
build: Callable[[bool], _GeminiRequestPart], send_integer_enums: bool
) -> _GeminiRequestPart:
if not send_integer_enums:
return build(False)
with_integer_enums: Final = build(True)
if _integer_enums_fit(
cast(JsonValue, with_integer_enums) # cast-ok: built tools and schemas are JSON request data
):
return with_integer_enums
verbose_logger.debug(
"Gemini integer enums dropped: the request's enum values exceed %s characters", GEMINI_MAX_ENUM_CHARACTERS
)
return build(False)
def _fix_enum_types(schema, depth=0, send_integer_enums: bool = False):
"""Make `enum` fields acceptable to Gemini / Vertex.
Gemini / Vertex APIs only accept string enum values. With `send_integer_enums`,
an integer enum of values a double holds exactly is sent as type INTEGER with the
values as strings; the model still returns an integer.
On any other non-string typed property (or when `anyOf` types do not include a
string type), the enum is removed to avoid provider validation errors.
"""
if depth > DEFAULT_MAX_RECURSE_DEPTH:
raise ValueError(f"Max depth of {DEFAULT_MAX_RECURSE_DEPTH} exceeded while processing schema.")
@ -696,26 +774,37 @@ def _fix_enum_types(schema, depth=0):
break
if not keep_enum:
schema.pop("enum", None)
is_integer_type: Final = isinstance(schema_type, str) and schema_type.lower() == "integer"
integer_enum: Final = (
_gemini_integer_enum(cast(Sequence[object], schema["enum"])) # cast-ok: an untyped JSON list
if send_integer_enums and is_integer_type
else None
)
if integer_enum is not None:
schema["enum"] = list(integer_enum) # rebind-ok: this function edits the schema in place by design
else:
schema.pop("enum", None)
# Recurse into nested structures
properties: Final = schema.get("properties", None)
if properties is not None:
for _, value in properties.items():
_fix_enum_types(value, depth=depth + 1)
_fix_enum_types(value, depth=depth + 1, send_integer_enums=send_integer_enums)
items: Final = schema.get("items", None)
if items is not None:
_fix_enum_types(items, depth=depth + 1)
_fix_enum_types(items, depth=depth + 1, send_integer_enums=send_integer_enums)
anyof = schema.get("anyOf", None)
if anyof is not None and isinstance(anyof, list):
for item in anyof:
if isinstance(item, dict):
_fix_enum_types(item, depth=depth + 1)
_fix_enum_types(item, depth=depth + 1, send_integer_enums=send_integer_enums)
def build_vertex_schema(parameters: dict, add_property_ordering: bool = False):
def build_vertex_schema(
parameters: dict, add_property_ordering: bool = False, send_integer_enums: bool = False
) -> dict[str, object]:
"""
This is a modified version of https://github.com/google-gemini/generative-ai-python/blob/8f77cc6ac99937cd3a81299ecf79608b91b06bbb/google/generativeai/types/content_types.py#L419
@ -725,6 +814,7 @@ def build_vertex_schema(parameters: dict, add_property_ordering: bool = False):
parameters: dict - the json schema to build from
add_property_ordering: bool - whether to add propertyOrdering to the schema. This is only applicable to schemas for structured outputs. See
set_schema_property_ordering for more details.
send_integer_enums: bool - whether to send integer enums with string values instead of removing them
Returns:
parameters: dict - the input parameters, modified in place
"""
@ -750,7 +840,7 @@ def build_vertex_schema(parameters: dict, add_property_ordering: bool = False):
_fix_enum_empty_strings(parameters)
# Remove enums for non-string typed fields (Gemini requires enum only on strings)
_fix_enum_types(parameters)
_fix_enum_types(parameters, send_integer_enums=send_integer_enums)
# Handle empty items objects
process_items(parameters)
@ -1068,6 +1158,17 @@ def _convert_schema_types(schema, depth=0):
if field in schema:
item_schema[field] = deepcopy(schema[field])
any_of.append(item_schema)
elif t == "integer" and "enum" in schema and isinstance(schema["enum"], list):
any_of.append(
{
"type": t,
"enum": [
value
for value in cast(list[object], schema["enum"]) # cast-ok: an untyped JSON list
if value is not None
],
}
)
else:
# For primitive types, only include the type
any_of.append({"type": t})
@ -1263,6 +1364,52 @@ class VertexAITokenCounter(BaseTokenCounter):
return custom_llm_provider == LlmProviders.VERTEX_AI.value
async def _count_translated_claude_request(
self,
model_to_use: str,
messages: Sequence[Mapping[str, object]] | None,
deployment: Mapping[str, object],
request_model: str,
tools: Sequence[Mapping[str, object]] | None,
system: object | None,
messages_format: TokenCountMessagesFormat,
output_format: JsonValue,
context_management: CountContextManagement | None,
chat_params: Mapping[str, JsonValue],
) -> TokenCountResponse | None:
from litellm.llms.anthropic.count_tokens.transformation import (
AnthropicCountFields,
ChatCountTranslationError,
anthropic_count_fields,
)
from litellm.llms.vertex_ai.vertex_ai_partner_models.count_tokens.handler import count_tokens_publisher
if count_tokens_publisher(model_to_use) != "anthropic":
return None
from litellm.llms.vertex_ai.vertex_ai_partner_models.anthropic.transformation import VertexAIAnthropicConfig
match translated := await anthropic_count_fields(
model_to_use, messages or [], system, tools, messages_format, VertexAIAnthropicConfig(), chat_params
):
case ChatCountTranslationError():
return translated.response(request_model, model_to_use, "vertex_ai_partner_models")
case AnthropicCountFields():
return await self.count_tokens(
model_to_use=model_to_use,
messages=[dict(message) for message in translated.messages],
contents=None,
deployment=dict(deployment),
request_model=request_model,
tools=[dict(tool) for tool in translated.tools] if translated.tools is not None else None,
system=translated.system,
messages_format="anthropic",
output_format=output_format,
context_management=context_management,
count_options=translated.options,
)
case None:
return None
async def count_tokens(
self,
model_to_use: str,
@ -1275,6 +1422,8 @@ class VertexAITokenCounter(BaseTokenCounter):
messages_format: TokenCountMessagesFormat = "openai",
output_format: JsonValue = None,
context_management: CountContextManagement | None = None,
chat_params: Mapping[str, JsonValue] = MappingProxyType({}),
count_options: Mapping[str, JsonValue] = MappingProxyType({}),
) -> TokenCountResponse | None:
import copy
@ -1313,6 +1462,21 @@ class VertexAITokenCounter(BaseTokenCounter):
# Check if this is a partner model (Claude, Mistral, etc.)
if VertexAIPartnerModels.is_vertex_partner_model(model_to_use):
translated_count: Final = await self._count_translated_claude_request(
model_to_use=model_to_use,
messages=messages,
deployment=deployment,
request_model=request_model,
tools=tools,
system=system,
messages_format=messages_format,
output_format=output_format,
context_management=context_management,
chat_params=chat_params,
)
if translated_count is not None:
return translated_count
# Use partner models token counter
partner_models_handler: Final = VertexAIPartnerModels()
@ -1343,6 +1507,7 @@ class VertexAITokenCounter(BaseTokenCounter):
vertex_credentials=vertex_credentials,
system=system,
tools=tools,
count_options=count_options,
)
except (PartnerVertexAIError, httpx.HTTPStatusError) as e:
status_code: Final = e.response.status_code
@ -1390,7 +1555,9 @@ class VertexAITokenCounter(BaseTokenCounter):
)
if isinstance(request_view, VertexGeminiCountContextError):
return count_context_error_response(request_view, request_model, model_to_use)
count_context: Final = vertex_gemini_count_context(system=request_view.system, tools=tools)
count_context: Final = vertex_gemini_count_context(
system=request_view.system, tools=tools, model=model_to_use
)
if isinstance(count_context, VertexGeminiCountContextError):
return count_context_error_response(count_context, request_model, model_to_use)
blocks_error: Final = unconvertible_messages_error(

156
litellm/llms/vertex_ai/context_caching/known_cached_contents.py Normal file
View file

@ -0,0 +1,156 @@
import asyncio
import contextlib
import re
import threading
import time
import weakref
from collections.abc import Awaitable, Callable, Mapping
from datetime import datetime, timezone
from typing import Final, Literal
KNOWN_CACHE_EXPIRY_MARGIN_SECONDS: Final = 10.0
MAX_KNOWN_CACHED_CONTENTS: Final = 10_000
_CACHE_REJECTED_STATUS_CODES: Final = frozenset({400, 403, 404})
_NAMES_CACHED_CONTENT: Final = re.compile(r"cached?\s*content", re.IGNORECASE)
_FORBIDS_FIELDS_BESIDE_A_CACHE: Final = re.compile(r"system[_\s]instruction", re.IGNORECASE)
_RFC3339_FRACTION: Final = re.compile(r"\.(\d{1,6})\d*")
def names_rejected_cache(status_code: int, message: str) -> bool:
return (
status_code in _CACHE_REJECTED_STATUS_CODES
and _NAMES_CACHED_CONTENT.search(message) is not None
and _FORBIDS_FIELDS_BESIDE_A_CACHE.search(message) is None
)
def parse_expire_time(expire_time: object) -> float | None:
"""Parse a Vertex RFC 3339 ``expireTime`` (nanosecond precision, ``Z`` suffix) to epoch seconds."""
if not isinstance(expire_time, str) or not expire_time:
return None
normalized: Final = _RFC3339_FRACTION.sub(lambda m: "." + m.group(1), expire_time.replace("Z", "+00:00"), count=1)
try:
parsed: Final = datetime.fromisoformat(normalized)
except ValueError:
return None
if parsed.tzinfo is None:
return parsed.replace(tzinfo=timezone.utc).timestamp()
return parsed.timestamp()
class KnownCachedContents:
"""Process-local record of the cachedContents this process created or found.
Vertex meters every cachedContents list page against a per-minute, per-location quota,
so listing on every cache-marked request exhausts it under load. A cache is remembered
until ``KNOWN_CACHE_EXPIRY_MARGIN_SECONDS`` before the ``expireTime`` Google reported,
and after that requests list again, as they did before the record existed.
"""
def __init__(self, clock: Callable[[], float] = time.time) -> None:
self._clock: Final = clock
self._entries: dict[tuple[tuple[str, str], str], tuple[str, float]] = {} # mutable-ok: guarded by _lock
self._lock: Final = threading.Lock()
self._inflight: weakref.WeakKeyDictionary[ # mutable-ok: private, guarded by _lock
asyncio.AbstractEventLoop, dict[tuple[tuple[str, str], str], asyncio.Future[str | None]]
] = weakref.WeakKeyDictionary()
@staticmethod
def scope(
base_url: str,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
headers: Mapping[str, str],
vertex_project: str | None,
vertex_location: str | None,
) -> tuple[str, str]:
"""Where a cache name is valid: a Vertex project and location, or the project a Gemini key belongs to.
A custom ``api_base`` can replace the whole Vertex path, so the project and location are kept beside the URL.
The Gemini key is read from the headers actually sent, so an ``extra_headers`` override scopes the
cache to the caller's project. It is kept as-is in this in-memory scope and never logged or persisted.
"""
if custom_llm_provider != "gemini":
return (base_url, f"{vertex_project}/{vertex_location}")
sent_keys: Final = [value for name, value in headers.items() if name.lower() == "x-goog-api-key"]
return (base_url, sent_keys[-1] if sent_keys else "")
def has_not_expired(self, expire_time: object) -> bool:
expires_at: Final = parse_expire_time(expire_time)
return expires_at is None or expires_at > self._clock()
def get(self, scope: tuple[str, str], cache_key: str) -> str | None:
with self._lock:
entry: Final = self._entries.get((scope, cache_key))
if entry is None:
return None
name, reuse_until = entry
if reuse_until > self._clock():
return name
del self._entries[(scope, cache_key)]
return None
def remember(self, scope: tuple[str, str], cache_key: str, name: str | None, expire_time: object) -> None:
expires_at: Final = parse_expire_time(expire_time)
if not name or expires_at is None:
return
reuse_until: Final = expires_at - KNOWN_CACHE_EXPIRY_MARGIN_SECONDS
now: Final = self._clock()
if reuse_until <= now:
return
with self._lock:
if len(self._entries) >= MAX_KNOWN_CACHED_CONTENTS:
self._entries = {k: v for k, v in self._entries.items() if v[1] > now}
while len(self._entries) >= MAX_KNOWN_CACHED_CONTENTS:
del self._entries[next(iter(self._entries))]
self._entries[(scope, cache_key)] = (name, reuse_until)
def forget_rejected(self, cached_content: str | None, status_code: int) -> None:
if cached_content is None or status_code not in _CACHE_REJECTED_STATUS_CODES:
return
with self._lock:
self._entries = {key: entry for key, entry in self._entries.items() if entry[0] != cached_content}
async def single_flight(
self,
scope: tuple[str, str],
cache_key: str,
resolve: Callable[[], Awaitable[str | None]],
wait_seconds: float | None,
) -> str | None:
"""Run ``resolve`` once per (scope, cache_key) at a time within this event loop.
Concurrent requests for the same prefix wait for the first one's lookup-or-create
instead of each listing (and creating a duplicate cache). A waiter resolves on its
own when that first attempt fails or has not finished within ``wait_seconds``, and
never cancels it.
"""
loop: Final = asyncio.get_running_loop()
own: Final[asyncio.Future[str | None]] = loop.create_future()
with self._lock:
inflight: Final = self._inflight.setdefault(loop, {})
leader: Final = inflight.setdefault((scope, cache_key), own)
if leader is not own:
with contextlib.suppress(asyncio.TimeoutError):
if name := await asyncio.wait_for(asyncio.shield(leader), wait_seconds):
return name
return await resolve()
try:
resolved: Final = await resolve()
except BaseException:
own.set_result(None)
raise
else:
own.set_result(resolved)
return resolved
finally:
with self._lock:
inflight.pop((scope, cache_key), None)
def clear(self) -> None:
with self._lock:
self._entries.clear()
self._inflight.clear()
known_cached_contents: Final = KnownCachedContents()

74
litellm/llms/vertex_ai/context_caching/rejected_cache_retry.py Normal file
View file

@ -0,0 +1,74 @@
from collections.abc import Awaitable, Callable
from copy import deepcopy
from functools import partial
from typing import Final, TypeVar
import httpx
from litellm.litellm_core_utils.core_helpers import independent_snapshot
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.llms.openai import AllMessageValues
from litellm.types.llms.vertex_ai import RequestBody
from litellm.utils import is_cached_message
from .known_cached_contents import names_rejected_cache
Built: Final = TypeVar("Built")
Sent: Final = TypeVar("Sent")
def cache_rebuild(
build: Callable[..., Built], messages: list[AllMessageValues], optional_params: dict
) -> Callable[[], Built] | None:
"""Copies the inputs before the first transform, which pops keys from ``optional_params`` and ``messages``."""
if not any(map(is_cached_message, messages)):
return None
return partial(build, messages=deepcopy(messages), optional_params=independent_snapshot(optional_params))
def _status_and_message(error: httpx.HTTPStatusError | BaseLLMException) -> tuple[int, str]:
match error:
case httpx.HTTPStatusError():
return error.response.status_code, error.response.text
case BaseLLMException():
return error.status_code, error.message
def _cache_rejected(body: RequestBody, error: httpx.HTTPStatusError | BaseLLMException) -> bool:
return body.get("cachedContent") is not None and names_rejected_cache(*_status_and_message(error))
async def post_with_fresh_cache_async(
body: RequestBody,
rebuild: Callable[[], Awaitable[RequestBody]] | None,
post: Callable[[RequestBody], Awaitable[Sent]],
log_request: Callable[[RequestBody], None],
) -> tuple[RequestBody, Sent]:
try:
return body, await post(body)
except (httpx.HTTPStatusError, BaseLLMException) as error:
if rebuild is None or not _cache_rejected(body, error):
raise
fresh_body: Final = await rebuild()
log_request(fresh_body)
if fresh_body.get("cachedContent") == body.get("cachedContent"):
raise
return fresh_body, await post(fresh_body)
def post_with_fresh_cache_sync(
body: RequestBody,
rebuild: Callable[[], RequestBody] | None,
post: Callable[[RequestBody], Sent],
log_request: Callable[[RequestBody], None],
) -> tuple[RequestBody, Sent]:
try:
return body, post(body)
except (httpx.HTTPStatusError, BaseLLMException) as error:
if rebuild is None or not _cache_rejected(body, error):
raise
fresh_body: Final = rebuild()
log_request(fresh_body)
if fresh_body.get("cachedContent") == body.get("cachedContent"):
raise
return fresh_body, post(fresh_body)

179
litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
View file

@ -20,6 +20,7 @@ from litellm.utils import is_prompt_caching_valid_prompt
from ..common_utils import VertexAIError, get_vertex_base_url
from ..vertex_llm_base import VertexBase
from .known_cached_contents import KnownCachedContents, known_cached_contents
from .transformation import (
cached_messages_end_on_supported_turn,
separate_cached_messages,
@ -35,6 +36,7 @@ MAX_PAGINATION_PAGES: Final = 100 # Reasonable upper bound for pagination
class _CachedContentItem(TypedDict):
displayName: ReadOnly[NotRequired[object]]
name: ReadOnly[NotRequired[str | None]]
expireTime: ReadOnly[NotRequired[str | None]]
@with_config(ConfigDict(extra="allow", strict=True, hide_input_in_errors=True))
@ -51,6 +53,7 @@ class _CachedContentUsageMetadata(TypedDict):
@with_config(ConfigDict(extra="allow", strict=True, hide_input_in_errors=True))
class _CreatedCachedContent(TypedDict):
name: ReadOnly[NotRequired[str | None]]
expireTime: ReadOnly[NotRequired[str | None]]
usageMetadata: ReadOnly[NotRequired[_CachedContentUsageMetadata | None]]
@ -67,6 +70,14 @@ def _record_cache_creation_tokens(logging_obj: Logging, created: _CreatedCachedC
logging_obj.vertex_ai_cache_creation_input_tokens = max(tokens, 0) # rebind-ok: set provider count
def _wait_seconds(timeout: float | httpx.Timeout | None) -> float | None:
match timeout:
case httpx.Timeout():
return timeout.read
case _:
return timeout
class ContextCachingEndpoints(VertexBase):
"""
Covers context caching endpoints for Vertex AI + Google AI Studio
@ -74,8 +85,9 @@ class ContextCachingEndpoints(VertexBase):
v0: covers Google AI Studio
"""
def __init__(self) -> None:
def __init__(self, known_caches: KnownCachedContents = known_cached_contents) -> None:
super().__init__()
self._known_caches: Final = known_caches
def _get_token_and_url_context_caching(
self,
@ -129,7 +141,7 @@ class ContextCachingEndpoints(VertexBase):
self,
cache_key: str,
client: HTTPHandler,
headers: dict,
headers: dict[str, str], # mutable-ok: passed through to the HTTP client, which takes a dict
api_key: str,
api_base: str | None,
logging_obj: Logging,
@ -159,6 +171,12 @@ class ContextCachingEndpoints(VertexBase):
vertex_auth_header=vertex_auth_header,
model=model,
)
scope: Final = KnownCachedContents.scope(
base_url, custom_llm_provider, headers, vertex_project=vertex_project, vertex_location=vertex_location
)
known_name: Final = self._known_caches.get(scope, cache_key)
if known_name is not None:
return known_name
page_token: str | None = None
@ -201,8 +219,14 @@ class ContextCachingEndpoints(VertexBase):
# Check current page for matching cache_key
for cached_item in raw_response["cachedContents"]:
display_name = cached_item.get("displayName")
if display_name is not None and display_name == cache_key:
return cached_item.get("name")
if display_name is None or display_name != cache_key:
continue
expire_time = cached_item.get("expireTime")
if not self._known_caches.has_not_expired(expire_time):
continue
name = cached_item.get("name")
self._known_caches.remember(scope, cache_key, name, expire_time)
return name
# Check if there are more pages
page_token = raw_response.get("nextPageToken")
@ -216,7 +240,7 @@ class ContextCachingEndpoints(VertexBase):
self,
cache_key: str,
client: AsyncHTTPHandler,
headers: dict,
headers: dict[str, str], # mutable-ok: passed through to the HTTP client, which takes a dict
api_key: str,
api_base: str | None,
logging_obj: Logging,
@ -246,6 +270,12 @@ class ContextCachingEndpoints(VertexBase):
vertex_auth_header=vertex_auth_header,
model=model,
)
scope: Final = KnownCachedContents.scope(
base_url, custom_llm_provider, headers, vertex_project=vertex_project, vertex_location=vertex_location
)
known_name: Final = self._known_caches.get(scope, cache_key)
if known_name is not None:
return known_name
page_token: str | None = None
@ -288,8 +318,14 @@ class ContextCachingEndpoints(VertexBase):
# Check current page for matching cache_key
for cached_item in raw_response["cachedContents"]:
display_name = cached_item.get("displayName")
if display_name is not None and display_name == cache_key:
return cached_item.get("name")
if display_name is None or display_name != cache_key:
continue
expire_time = cached_item.get("expireTime")
if not self._known_caches.has_not_expired(expire_time):
continue
name = cached_item.get("name")
self._known_caches.remember(scope, cache_key, name, expire_time)
return name
# Check if there are more pages
page_token = raw_response.get("nextPageToken")
@ -451,6 +487,14 @@ class ContextCachingEndpoints(VertexBase):
raw_response_cached: Final = _CreatedCachedContentRoot.model_validate(response.json()).root
_record_cache_creation_tokens(logging_obj, raw_response_cached)
self._known_caches.remember(
KnownCachedContents.scope(
url, custom_llm_provider, headers, vertex_project=vertex_project, vertex_location=vertex_location
),
generated_cache_key,
raw_response_cached.get("name"),
raw_response_cached.get("expireTime"),
)
return (
non_cached_messages,
optional_params,
@ -548,68 +592,77 @@ class ContextCachingEndpoints(VertexBase):
generated_cache_key: Final = local_cache_obj.get_cache_key(
messages=cached_messages, tools=tools, tool_choice=tool_choice, model=model
)
google_cache_name: Final = await self.async_check_cache(
cache_key=generated_cache_key,
client=client,
headers=headers,
api_key=api_key,
api_base=api_base,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header,
model=model,
scope: Final = KnownCachedContents.scope(
url, custom_llm_provider, headers, vertex_project=vertex_project, vertex_location=vertex_location
)
if google_cache_name:
return non_cached_messages, optional_params, google_cache_name
## TRANSFORM REQUEST
cached_content_request_body: Final = transform_openai_messages_to_gemini_context_caching(
model=model,
messages=cached_messages,
cache_key=generated_cache_key,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
)
cached_content_request_body["tools"] = tools
if tool_choice is not None:
cached_content_request_body["toolConfig"] = tool_choice
## LOGGING
logging_obj.pre_call(
input=messages,
api_key="",
additional_args={
"complete_input_dict": cached_content_request_body,
"api_base": url,
"headers": headers,
},
)
try:
response: Final = await client.post(
url=url,
async def lookup_or_create() -> str | None:
google_cache_name: Final = await self.async_check_cache(
cache_key=generated_cache_key,
client=client,
headers=headers,
json=cached_content_request_body,
api_key=api_key,
api_base=api_base,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=vertex_auth_header,
model=model,
)
response.raise_for_status()
except httpx.HTTPStatusError as err:
error_code: Final = err.response.status_code
raise VertexAIError(status_code=error_code, message=err.response.text)
except httpx.TimeoutException:
raise VertexAIError(status_code=408, message="Timeout error occurred.")
raw_response_cached: Final = _CreatedCachedContentRoot.model_validate(response.json()).root
_record_cache_creation_tokens(logging_obj, raw_response_cached)
return (
non_cached_messages,
optional_params,
raw_response_cached.get("name"),
if google_cache_name:
return google_cache_name
## TRANSFORM REQUEST
cached_content_request_body: Final = transform_openai_messages_to_gemini_context_caching(
model=model,
messages=cached_messages,
cache_key=generated_cache_key,
custom_llm_provider=custom_llm_provider,
vertex_project=vertex_project,
vertex_location=vertex_location,
)
cached_content_request_body["tools"] = tools
if tool_choice is not None:
cached_content_request_body["toolConfig"] = tool_choice
## LOGGING
logging_obj.pre_call(
input=messages,
api_key="",
additional_args={
"complete_input_dict": cached_content_request_body,
"api_base": url,
"headers": headers,
},
)
try:
response: Final = await client.post(
url=url,
headers=headers,
json=cached_content_request_body,
)
response.raise_for_status()
except httpx.HTTPStatusError as err:
error_code: Final = err.response.status_code
raise VertexAIError(status_code=error_code, message=err.response.text)
except httpx.TimeoutException:
raise VertexAIError(status_code=408, message="Timeout error occurred.")
raw_response_cached: Final = _CreatedCachedContentRoot.model_validate(response.json()).root
_record_cache_creation_tokens(logging_obj, raw_response_cached)
self._known_caches.remember(
scope, generated_cache_key, raw_response_cached.get("name"), raw_response_cached.get("expireTime")
)
return raw_response_cached.get("name")
cache_name: Final = await self._known_caches.single_flight(
scope, generated_cache_key, lookup_or_create, _wait_seconds(timeout)
)
return non_cached_messages, optional_params, cache_name
def get_cache(self):
pass

16
litellm/llms/vertex_ai/count_tokens/context.py
View file

@ -78,6 +78,7 @@ def _is_function_tool(tool: Mapping[str, JsonValue]) -> bool:
def _map_function_tools(
tools: object | None,
model: str,
) -> tuple[Mapping[str, object], ...] | None | VertexGeminiCountContextError:
try:
tool_values: Final = _TOOLS.validate_python(tools or ())
@ -89,15 +90,21 @@ def _map_function_tools(
if not all(_has_object_schema(tool) for tool in function_tools):
return _INVALID_TOOLS
from litellm.llms.vertex_ai.common_utils import with_integer_enums_that_fit
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
chat_tools: Final = tuple(
chat_tool for tool in function_tools for chat_tool in chat_completion_tools_for(tool)
) # comprehension-ok: flattens each tool's translation
config: Final = VertexGeminiConfig()
try:
mapped_tools: Final = VertexGeminiConfig().map_function(
value=[dict(tool) for tool in chat_tools],
optional_params={},
mapped_tools: Final = with_integer_enums_that_fit(
lambda send_integer_enums: config.map_function(
value=[dict(tool) for tool in chat_tools],
optional_params={},
send_integer_enums=send_integer_enums,
),
VertexGeminiConfig.is_gemini_3_or_newer(model),
)
except (AttributeError, TypeError, ValueError):
return _INVALID_TOOLS
@ -110,11 +117,12 @@ def _map_function_tools(
def vertex_gemini_count_context(
system: object | None,
tools: object | None,
model: str,
) -> VertexGeminiCountContext | VertexGeminiCountContextError:
system_parts: Final = _system_parts(system)
if isinstance(system_parts, VertexGeminiCountContextError):
return system_parts
function_tools: Final = _map_function_tools(tools)
function_tools: Final = _map_function_tools(tools, model)
if isinstance(function_tools, VertexGeminiCountContextError):
return function_tools
return VertexGeminiCountContext(

1
litellm/llms/vertex_ai/files/transformation.py
View file

@ -1118,6 +1118,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
non_default_params=openai_request_body,
optional_params={},
drop_params=False,
allow_integer_enums=False,
)
return vertex_params

297
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -95,6 +95,13 @@ from ..common_utils import (
build_json_schema,
build_vertex_schema,
supports_response_json_schema,
with_integer_enums_that_fit,
)
from ..context_caching.known_cached_contents import known_cached_contents
from ..context_caching.rejected_cache_retry import (
cache_rebuild,
post_with_fresh_cache_async,
post_with_fresh_cache_sync,
)
from ..vertex_llm_base import VertexBase
from .grounding_requests import calculate_grounding_requests
@ -686,13 +693,14 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
return googleSearch, googleSearchRetrieval, enterpriseWebSearch, urlContext
def _map_function(self, value: list[dict], optional_params: dict) -> list[Tools]:
def _map_function(self, value: list[dict], optional_params: dict, send_integer_enums: bool = False) -> list[Tools]:
"""
Map OpenAI-style tools/functions to Vertex AI format.
Args:
value: List of tool definitions
optional_params: Request-scoped parameters to store retrieval config
send_integer_enums: Whether to send integer enums with string values
Returns:
List of mapped tools in Vertex AI format
@ -705,18 +713,18 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
googleSearch: dict | None = None
googleSearchRetrieval: dict | None = None
enterpriseWebSearch: dict | None = None
urlContext: dict | None = None
urlContext: dict[str, object] | None = None
code_execution: dict | None = None
googleMaps: dict | None = None
google_maps_retrieval_config: dict | None = None
computerUse: dict | None = None
# remove 'additionalProperties' from tools
schema_value: Final[JsonValue] = cast(JsonValue, value) # cast-ok: tool schemas come from JSON request data
schema_value: Final[JsonValue] = deepcopy(cast(JsonValue, value)) # cast-ok: tools are JSON request data
remove_additional_properties(schema_value)
# remove 'strict' from tools
remove_strict_from_schema(schema_value)
for tool in value:
for tool in cast(list[dict], schema_value): # cast-ok: a deep copy keeps the caller's tool shape
openai_function_object: ChatCompletionToolParamFunctionChunk | None = None
if "function" in tool: # tools list
_openai_function_object = ChatCompletionToolParamFunctionChunk(**tool["function"])
@ -726,7 +734,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
and _openai_function_object["parameters"] is not None
and isinstance(_openai_function_object["parameters"], dict)
): # OPENAI accepts JSON Schema, Google accepts OpenAPI schema.
_openai_function_object["parameters"] = build_vertex_schema(_openai_function_object["parameters"])
_openai_function_object["parameters"] = build_vertex_schema(
_openai_function_object["parameters"], send_integer_enums=send_integer_enums
)
openai_function_object = _openai_function_object
@ -869,21 +879,28 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
self,
value: list[dict[str, object]], # mutable-ok: mirrors override contract
optional_params: dict[str, object], # mutable-ok: mirrors override contract
send_integer_enums: bool = False,
) -> list[Tools]: # mutable-ok: mirrors override contract
return self._map_function(value, optional_params)
return self._map_function(value, optional_params, send_integer_enums)
def _map_response_schema(self, value: dict) -> dict:
def _map_response_schema(self, value: dict, send_integer_enums: bool = False) -> Mapping[str, object]:
old_schema = deepcopy(value)
if isinstance(old_schema, list):
for item in old_schema:
if isinstance(item, dict):
item = build_vertex_schema(parameters=item, add_property_ordering=True)
item = build_vertex_schema(
parameters=item, add_property_ordering=True, send_integer_enums=send_integer_enums
)
elif isinstance(old_schema, dict):
old_schema = build_vertex_schema(parameters=old_schema, add_property_ordering=True)
old_schema = build_vertex_schema(
parameters=old_schema, add_property_ordering=True, send_integer_enums=send_integer_enums
)
return old_schema
def apply_response_schema_transformation(self, value: dict, optional_params: dict, model: str):
def apply_response_schema_transformation(
self, value: dict, optional_params: dict, model: str, allow_integer_enums: bool = True
):
new_value = deepcopy(value)
# remove 'strict' from json schema (not supported by Gemini)
new_value = remove_strict_from_schema(new_value)
@ -925,7 +942,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
# - OpenAPI-style format (uppercase types)
# - No additionalProperties support
# - Requires propertyOrdering
optional_params["response_schema"] = self._map_response_schema(value=schema)
optional_params["response_schema"] = with_integer_enums_that_fit(
partial(self._map_response_schema, schema),
allow_integer_enums and self._is_gemini_3_or_newer(model),
)
@property
def accepts_reasoning_summary_in_reasoning_effort(self) -> bool:
@ -1243,6 +1263,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
optional_params: dict,
model: str,
drop_params: bool,
allow_integer_enums: bool = True,
) -> dict:
self._apply_include_server_side_tool_invocations(non_default_params, optional_params)
gemini_sampling_params_warned: bool = False
@ -1293,7 +1314,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
elif param == "max_tokens" or param == "max_completion_tokens":
optional_params["max_output_tokens"] = value
elif param == "response_format" and isinstance(value, dict):
self.apply_response_schema_transformation(value=value, optional_params=optional_params, model=model)
self.apply_response_schema_transformation(
value=value, optional_params=optional_params, model=model, allow_integer_enums=allow_integer_enums
)
elif param == "frequency_penalty":
if self._supports_penalty_parameters(model):
optional_params["frequency_penalty"] = value
@ -1306,7 +1329,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
optional_params["logprobs"] = value
elif (param == "tools" or param == "functions") and isinstance(value, list) and value:
# Pass optional_params so _map_function can add toolConfig if needed
mapped_tools = self._map_function(value=value, optional_params=optional_params)
mapped_tools = with_integer_enums_that_fit(
partial(self._map_function, value, optional_params),
allow_integer_enums and self._is_gemini_3_or_newer(model),
)
optional_params = self._add_tools_to_optional_params(optional_params, mapped_tools)
elif param == "tool_choice" and (isinstance(value, str) or isinstance(value, dict)):
_tool_choice_value = self.map_tool_choice_values(
@ -2786,7 +2812,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
litellm_params: dict,
api_key: str | dict | None = None,
api_base: str | None = None,
) -> dict:
) -> dict[str, str]:
default_headers: Final = {
"Content-Type": "application/json",
}
@ -2800,6 +2826,20 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
return default_headers
def _log_generate_request(
logging_obj: LoggingClass,
messages: Sequence[AllMessageValues],
api_base: str,
headers: Mapping[str, str],
body: RequestBody,
) -> None:
logging_obj.pre_call(
input=messages,
api_key="",
additional_args={"complete_input_dict": body, "api_base": api_base, "headers": headers},
)
async def make_call(
client: AsyncHTTPHandler | None, # module-level client
gemini_client: AsyncHTTPHandler | None, # if passed by user
@ -2832,6 +2872,7 @@ async def make_call(
response.raise_for_status()
except httpx.HTTPStatusError as e:
exception_string: Final = str(await e.response.aread())
known_cached_contents.forget_rejected(request_body.get("cachedContent"), e.response.status_code)
raise VertexAIError(
status_code=e.response.status_code,
message=VertexGeminiConfig().translate_exception_str(exception_string),
@ -2845,14 +2886,18 @@ async def make_call(
)
async def open_hop(body: RequestBody) -> httpx.Response:
return await client.post(
url_for(api_base, body),
headers=dict(await hop_headers()),
data=json.dumps(body),
stream=True,
logging_obj=logging_obj,
timeout=timeout,
)
try:
return await client.post(
url_for(api_base, body),
headers=dict(await hop_headers()),
data=json.dumps(body),
stream=True,
logging_obj=logging_obj,
timeout=timeout,
)
except httpx.HTTPStatusError as e:
known_cached_contents.forget_rejected(body.get("cachedContent"), e.response.status_code)
raise
completion_stream: Final = ModelResponseIterator(
streaming_response=response.aiter_lines(),
@ -2891,14 +2936,18 @@ def make_sync_call(
if client is None:
client = HTTPHandler() # Create a new client if none provided
response: Final = client.post(
url_for(api_base, request_body),
headers=headers,
data=data,
stream=True,
logging_obj=logging_obj,
timeout=timeout,
)
try:
response: Final = client.post(
url_for(api_base, request_body),
headers=headers,
data=data,
stream=True,
logging_obj=logging_obj,
timeout=timeout,
)
except httpx.HTTPStatusError as e:
known_cached_contents.forget_rejected(request_body.get("cachedContent"), e.response.status_code)
raise
if response.status_code != 200 and response.status_code != 201:
raise VertexAIError(
@ -2908,14 +2957,18 @@ def make_sync_call(
)
def open_hop(body: RequestBody) -> httpx.Response:
return client.post(
url_for(api_base, body),
headers=dict(hop_headers()),
data=json.dumps(body),
stream=True,
logging_obj=logging_obj,
timeout=timeout,
)
try:
return client.post(
url_for(api_base, body),
headers=dict(hop_headers()),
data=json.dumps(body),
stream=True,
logging_obj=logging_obj,
timeout=timeout,
)
except httpx.HTTPStatusError as e:
known_cached_contents.forget_rejected(body.get("cachedContent"), e.response.status_code)
raise
completion_stream: Final = ModelResponseIterator(
streaming_response=response.iter_lines(),
@ -2983,14 +3036,14 @@ class VertexLLM(VertexBase):
custom_llm_provider: Literal[
"vertex_ai", "vertex_ai_beta", "gemini"
], # if it's vertex_ai or gemini (google ai studio)
messages: list,
messages: list[AllMessageValues],
model_response: ModelResponse,
print_verbose: Callable,
data: dict,
timeout: float | httpx.Timeout | None,
encoding: object,
logging_obj,
stream,
logging_obj: LoggingClass,
stream: bool | None,
optional_params: dict,
litellm_params: dict,
logger_fn=None,
@ -3033,6 +3086,17 @@ class VertexLLM(VertexBase):
litellm_params=litellm_params,
)
rebuild: Final = cache_rebuild(
partial(
async_transform_request_body,
**data,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=auth_header,
),
messages,
optional_params,
)
request_body: Final = await async_transform_request_body(
**data,
vertex_project=vertex_project,
@ -3041,39 +3105,37 @@ class VertexLLM(VertexBase):
)
## LOGGING
logging_obj.pre_call(
input=messages,
api_key="",
additional_args={
"complete_input_dict": request_body,
"api_base": api_base,
"headers": headers,
},
log_request: Final = partial(_log_generate_request, logging_obj, messages, api_base, headers)
log_request(request_body)
open_stream: Final = partial(
make_call,
gemini_client=(client if client is not None and isinstance(client, AsyncHTTPHandler) else None),
api_base=api_base,
headers=headers,
model=model,
messages=messages,
logging_obj=logging_obj,
hop_headers=partial(
self._fresh_headers_async,
headers,
extra_headers,
vertex_credentials,
vertex_project,
custom_llm_provider,
),
timeout=timeout,
)
request_body_str: Final = json.dumps(request_body)
async def open_with_fresh_cache(client: AsyncHTTPHandler | None) -> ModelResponseIterator:
async def post(body: RequestBody) -> ModelResponseIterator:
return await open_stream(client=client, data=json.dumps(body), request_body=body)
return (await post_with_fresh_cache_async(request_body, rebuild, post, log_request))[1]
streaming_response: Final = CustomStreamWrapper(
completion_stream=None,
make_call=partial(
make_call,
gemini_client=(client if client is not None and isinstance(client, AsyncHTTPHandler) else None),
api_base=api_base,
headers=headers,
data=request_body_str,
model=model,
messages=messages,
logging_obj=logging_obj,
request_body=request_body,
hop_headers=partial(
self._fresh_headers_async,
headers,
extra_headers,
vertex_credentials,
vertex_project,
custom_llm_provider,
),
timeout=timeout,
),
make_call=open_with_fresh_cache,
model=model,
custom_llm_provider="vertex_ai_beta",
logging_obj=logging_obj,
@ -3083,7 +3145,7 @@ class VertexLLM(VertexBase):
async def async_completion(
self,
model: str,
messages: list,
messages: list[AllMessageValues],
model_response: ModelResponse,
print_verbose: Callable,
data: dict,
@ -3093,7 +3155,7 @@ class VertexLLM(VertexBase):
timeout: float | httpx.Timeout | None,
encoding: object,
logging_obj: LoggingClass,
stream,
stream: bool | None,
optional_params: dict,
litellm_params: dict,
logger_fn=None,
@ -3136,6 +3198,17 @@ class VertexLLM(VertexBase):
litellm_params=litellm_params,
)
rebuild: Final = cache_rebuild(
partial(
async_transform_request_body,
**data,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=auth_header,
),
messages,
optional_params,
)
request_body: Final = await async_transform_request_body(
**data,
vertex_project=vertex_project,
@ -3151,15 +3224,8 @@ class VertexLLM(VertexBase):
else:
client = client
## LOGGING
logging_obj.pre_call(
input=messages,
api_key="",
additional_args={
"complete_input_dict": request_body,
"api_base": api_base,
"headers": headers,
},
)
log_request: Final = partial(_log_generate_request, logging_obj, messages, api_base, headers)
log_request(request_body)
fresh_headers: Final = partial(
self._fresh_headers_async, headers, extra_headers, vertex_credentials, vertex_project, custom_llm_provider
@ -3175,6 +3241,7 @@ class VertexLLM(VertexBase):
)
response.raise_for_status()
except httpx.HTTPStatusError as err:
known_cached_contents.forget_rejected(body.get("cachedContent"), err.response.status_code)
raise VertexAIError(
status_code=err.response.status_code,
message=err.response.text,
@ -3188,14 +3255,14 @@ class VertexLLM(VertexBase):
)
return response
first_response: Final = await post(request_body)
sent_body, first_response = await post_with_fresh_cache_async(request_body, rebuild, post, log_request)
return _transform_long_decode_hops(
model=model,
hops=await collect_hops_async(parse_body(first_response), request_body, post),
hops=await collect_hops_async(parse_body(first_response), sent_body, post),
first_response=first_response,
model_response=model_response,
logging_obj=logging_obj,
request_body=request_body,
request_body=sent_body,
messages=messages,
)
@ -3320,6 +3387,17 @@ class VertexLLM(VertexBase):
)
## TRANSFORMATION ##
rebuild: Final = cache_rebuild(
partial(
sync_transform_request_body,
**transform_request_params,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_auth_header=auth_header,
),
messages,
optional_params,
)
data: Final = sync_transform_request_body(
**transform_request_params,
vertex_project=vertex_project,
@ -3328,15 +3406,8 @@ class VertexLLM(VertexBase):
)
## LOGGING
logging_obj.pre_call(
input=messages,
api_key="",
additional_args={
"complete_input_dict": data,
"api_base": url,
"headers": headers,
},
)
log_request: Final = partial(_log_generate_request, logging_obj, messages, url, headers)
log_request(data)
fresh_headers: Final = partial(
self._fresh_headers, headers, extra_headers, vertex_credentials, vertex_project, custom_llm_provider
@ -3344,22 +3415,27 @@ class VertexLLM(VertexBase):
## SYNC STREAMING CALL ##
if stream is True:
request_data_str: Final = json.dumps(data)
open_stream: Final = partial(
make_sync_call,
gemini_client=(client if client is not None and isinstance(client, HTTPHandler) else None),
api_base=url,
model=model,
messages=messages,
logging_obj=logging_obj,
headers=headers,
hop_headers=fresh_headers,
timeout=timeout,
)
def open_with_fresh_cache(client: HTTPHandler | None) -> ModelResponseIterator:
def post(body: RequestBody) -> ModelResponseIterator:
return open_stream(client=client, data=json.dumps(body), request_body=body)
return post_with_fresh_cache_sync(data, rebuild, post, log_request)[1]
streaming_response: Final = CustomStreamWrapper(
completion_stream=None,
make_call=partial(
make_sync_call,
gemini_client=(client if client is not None and isinstance(client, HTTPHandler) else None),
api_base=url,
data=request_data_str,
model=model,
messages=messages,
logging_obj=logging_obj,
headers=headers,
request_body=data,
hop_headers=fresh_headers,
timeout=timeout,
),
make_call=open_with_fresh_cache,
model=model,
custom_llm_provider="vertex_ai_beta",
logging_obj=logging_obj,
@ -3385,6 +3461,7 @@ class VertexLLM(VertexBase):
)
response.raise_for_status()
except httpx.HTTPStatusError as err:
known_cached_contents.forget_rejected(body.get("cachedContent"), err.response.status_code)
raise VertexAIError(
status_code=err.response.status_code,
message=err.response.text,
@ -3398,14 +3475,14 @@ class VertexLLM(VertexBase):
)
return response
first_response: Final = post(data)
sent_body, first_response = post_with_fresh_cache_sync(data, rebuild, post, log_request)
return _transform_long_decode_hops(
model=model,
hops=collect_hops_sync(parse_body(first_response), data, post),
hops=collect_hops_sync(parse_body(first_response), sent_body, post),
first_response=first_response,
model_response=model_response,
logging_obj=logging_obj,
request_body=data,
request_body=sent_body,
messages=messages,
)

4
litellm/llms/vertex_ai/vertex_ai_partner_models/main.py
View file

@ -1,6 +1,6 @@
# What is this?
## API Handler for calling Vertex AI Partner Models
from collections.abc import Callable
from collections.abc import Callable, Mapping
from enum import Enum
from types import MappingProxyType
from typing import Final
@ -266,6 +266,7 @@ class VertexAIPartnerModels(VertexBase):
vertex_credentials=None,
system: object | None = None,
tools: list[dict[str, object]] | None = None,
count_options: Mapping[str, object] = MappingProxyType({}),
):
"""
Count tokens for Vertex AI partner models (Anthropic Claude, Mistral, etc.)
@ -302,6 +303,7 @@ class VertexAIPartnerModels(VertexBase):
**MappingProxyType(
{key: value for key, value in (("system", system), ("tools", tools)) if value is not None}
),
**count_options,
}
# Prepare litellm_params with credentials

14
litellm/main.py
View file

@ -400,7 +400,11 @@ class LiteLLM:
class Chat:
def __init__(self, params, router_obj: "Router | None"):
def __init__(
self,
params: dict[str, object], # mutable-ok: pops the acompletion flag in place
router_obj: "Router | None",
):
self.params = params
if self.params.get("acompletion", False) is True:
self.params.pop("acompletion")
@ -480,6 +484,7 @@ async def acompletion(
service_tier: str | None = None,
store: bool | None = None,
prompt_cache_key: str | None = None,
prompt_cache_retention: str | None = None,
# set api_base, api_version, api_key
base_url: str | None = None,
api_version: str | None = None,
@ -657,6 +662,7 @@ async def acompletion(
"service_tier": service_tier,
"store": store,
"prompt_cache_key": prompt_cache_key,
"prompt_cache_retention": prompt_cache_retention,
"extra_headers": extra_headers,
"acompletion": True, # assuming this is a required parameter
"thinking": thinking,
@ -5356,6 +5362,7 @@ def completion(
service_tier: str | None = None,
store: bool | None = None,
prompt_cache_key: str | None = None,
prompt_cache_retention: str | None = None,
# soon to be deprecated params by OpenAI
functions: list | None = None,
function_call: str | None = None,
@ -5478,6 +5485,7 @@ def completion(
service_tier=service_tier,
store=store,
prompt_cache_key=prompt_cache_key,
prompt_cache_retention=prompt_cache_retention,
base_url=base_url,
api_version=api_version,
api_key=api_key,
@ -5843,6 +5851,7 @@ def completion(
"service_tier": service_tier,
"store": store,
"prompt_cache_key": prompt_cache_key,
"prompt_cache_retention": prompt_cache_retention,
"allowed_openai_params": allowed_openai_params,
"base_model": base_model,
}
@ -9143,7 +9152,7 @@ async def ahealth_check(
####### HELPER FUNCTIONS ################
## Set verbose to true -> ```litellm.set_verbose = True```
def print_verbose(print_statement):
def print_verbose(print_statement: object):
try:
verbose_logger.debug(print_statement)
if litellm.set_verbose:
@ -9247,6 +9256,7 @@ async def acount_tokens(
request_model=model,
tools=tools,
system=system,
messages_format="openai",
)
if result is not None and not result.error:
return result

318
litellm/model_prices_and_context_window_backup.json
View file

@ -4916,6 +4916,26 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure/eu/gpt-4o": {
"deprecation_date": "2027-04-14",
"cache_creation_input_token_cost": 1.38e-06,
"cache_read_input_token_cost": 1.375e-06,
"input_cost_per_token": 2.75e-06,
"input_cost_per_token_batches": 1.375e-06,
"litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.1e-05,
"output_cost_per_token_batches": 5.5e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/eu/gpt-4o-2024-11-20": {
"deprecation_date": "2027-04-14",
"cache_creation_input_token_cost": 1.38e-06,
@ -4936,6 +4956,25 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure/eu/gpt-4o-mini": {
"cache_read_input_token_cost": 8.3e-08,
"deprecation_date": "2027-04-14",
"input_cost_per_token": 1.65e-07,
"input_cost_per_token_batches": 8.3e-08,
"litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 6.6e-07,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/eu/gpt-4o-mini-2024-07-18": {
"cache_read_input_token_cost": 8.3e-08,
"deprecation_date": "2027-04-14",
@ -5247,6 +5286,25 @@
"supports_vision": true,
"cache_read_input_token_cost_batches": 2.75e-09
},
"azure/eu/o1": {
"cache_read_input_token_cost": 8.25e-06,
"deprecation_date": "2026-11-19",
"input_cost_per_token": 1.65e-05,
"input_cost_per_token_batches": 8.25e-06,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 6.6e-05,
"output_cost_per_token_batches": 3.3e-05,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/eu/o1-2024-12-17": {
"cache_read_input_token_cost": 8.25e-06,
"deprecation_date": "2026-11-19",
@ -5283,6 +5341,24 @@
"supports_prompt_caching": true,
"supports_vision": false
},
"azure/eu/o3-mini": {
"cache_read_input_token_cost": 6.05e-07,
"deprecation_date": "2026-11-19",
"input_cost_per_token": 1.21e-06,
"input_cost_per_token_batches": 6.05e-07,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 4.84e-06,
"output_cost_per_token_batches": 2.42e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": false
},
"azure/eu/o3-mini-2025-01-31": {
"cache_read_input_token_cost": 6.05e-07,
"deprecation_date": "2026-11-19",
@ -11540,6 +11616,26 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/gpt-4o": {
"deprecation_date": "2027-04-14",
"cache_creation_input_token_cost": 1.38e-06,
"cache_read_input_token_cost": 1.375e-06,
"input_cost_per_token": 2.75e-06,
"input_cost_per_token_batches": 1.375e-06,
"litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 1.1e-05,
"output_cost_per_token_batches": 5.5e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/gpt-4o-2024-11-20": {
"deprecation_date": "2027-04-14",
"cache_creation_input_token_cost": 1.38e-06,
@ -11560,6 +11656,25 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/gpt-4o-mini": {
"cache_read_input_token_cost": 8.3e-08,
"deprecation_date": "2027-04-14",
"input_cost_per_token": 1.65e-07,
"input_cost_per_token_batches": 8.3e-08,
"litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_token": 6.6e-07,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/gpt-4o-mini-2024-07-18": {
"cache_read_input_token_cost": 8.3e-08,
"deprecation_date": "2027-04-14",
@ -11871,6 +11986,25 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/o1": {
"cache_read_input_token_cost": 8.25e-06,
"deprecation_date": "2026-11-19",
"input_cost_per_token": 1.65e-05,
"input_cost_per_token_batches": 8.25e-06,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 6.6e-05,
"output_cost_per_token_batches": 3.3e-05,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/o1-2024-12-17": {
"cache_read_input_token_cost": 8.25e-06,
"deprecation_date": "2026-11-19",
@ -11907,6 +12041,39 @@
"supports_prompt_caching": true,
"supports_vision": false
},
"azure/us/o3": {
"deprecation_date": "2026-11-19",
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 2.2e-06,
"input_cost_per_token_batches": 1.1e-06,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 8.8e-06,
"output_cost_per_token_batches": 4.4e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/batch",
"/v1/responses"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_parallel_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/o3-2025-04-16": {
"deprecation_date": "2026-11-19",
"cache_read_input_token_cost": 5.5e-07,
@ -11940,6 +12107,24 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/o3-mini": {
"cache_read_input_token_cost": 6.05e-07,
"deprecation_date": "2026-11-19",
"input_cost_per_token": 1.21e-06,
"input_cost_per_token_batches": 6.05e-07,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 4.84e-06,
"output_cost_per_token_batches": 2.42e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": false
},
"azure/us/o3-mini-2025-01-31": {
"cache_read_input_token_cost": 6.05e-07,
"deprecation_date": "2026-11-19",
@ -11958,6 +12143,27 @@
"supports_tool_choice": true,
"supports_vision": false
},
"azure/us/o4-mini": {
"cache_read_input_token_cost": 3.03e-07,
"deprecation_date": "2026-11-19",
"input_cost_per_token": 1.21e-06,
"input_cost_per_token_batches": 6.05e-07,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 4.84e-06,
"output_cost_per_token_batches": 2.42e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'",
"supports_function_calling": true,
"supports_parallel_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"azure/us/o4-mini-2025-04-16": {
"cache_read_input_token_cost": 3.03e-07,
"deprecation_date": "2026-11-19",
@ -28742,6 +28948,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 1500, "period": "day", "shared_group": "gemini-2-flash"},
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token_batches": 5e-07,
"input_cost_per_token_batches": 1.5e-07,
@ -28856,6 +29063,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query"
},
"gemini-3-pro-image-preview": {
@ -28908,6 +29116,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-image": {
@ -28958,6 +29167,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-image-preview": {
@ -29001,6 +29211,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-lite-image": {
@ -29096,6 +29307,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -29157,6 +29369,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_audio_token_batches": 2.5e-07
@ -29216,6 +29429,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -29302,6 +29516,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 1500, "period": "day", "shared_group": "gemini-2-flash"},
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token_batches": 5e-08,
"input_cost_per_token_batches": 5e-08,
@ -29356,6 +29571,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 1500, "period": "day", "shared_group": "gemini-2-flash"},
"google_maps_grounding_cost_per_query": 0.025,
"supports_image_size": false
},
@ -29402,6 +29618,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 1500, "period": "day", "shared_group": "gemini-2-flash"},
"google_maps_grounding_cost_per_query": 0.025,
"supports_image_size": false
},
@ -29589,6 +29806,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 10000, "period": "day", "shared_group": "gemini-2.5-pro"},
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_token_above_200k_tokens_priority": 4.5e-06,
"input_cost_per_token_batches": 6.25e-07,
@ -29626,6 +29844,7 @@
"output_cost_per_token_priority": 2.16e-05,
"prompt_cache_min_tokens": 4096.0,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -29706,6 +29925,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -29762,6 +29982,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query"
},
"vertex_ai/gemini-3-flash-preview": {
@ -29786,6 +30007,7 @@
"output_cost_per_token_flex": 1.5e-06,
"output_cost_per_token_priority": 5.4e-06,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -29868,6 +30090,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_batches": 7.5e-07,
@ -29932,6 +30155,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -29992,6 +30216,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -30018,6 +30243,7 @@
"prompt_cache_min_tokens": 4096.0,
"regional_endpoint_uplift_multiplier": 1.1,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -30163,6 +30389,7 @@
"output_cost_per_token_priority": 2.16e-05,
"prompt_cache_min_tokens": 4096.0,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -30249,6 +30476,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -30576,6 +30804,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"supports_reasoning": false
},
@ -30620,6 +30849,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3.1-flash-lite-preview": {
@ -30681,6 +30911,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_audio_token_batches": 2.5e-07
@ -30830,6 +31061,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 1500, "period": "day", "shared_group": "gemini-2-flash"},
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token_batches": 5e-07,
"input_cost_per_audio_token_priority": 1.8e-06,
@ -30942,6 +31174,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"supports_reasoning": true
},
@ -31006,6 +31239,7 @@
"output_cost_per_token_batches": 1.5e-06,
"rpm": 1000.0,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://ai.google.dev/gemini-api/docs/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -31156,6 +31390,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 1500, "period": "day", "shared_group": "gemini-2-flash"},
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token_batches": 1.5e-07,
"input_cost_per_audio_token_priority": 5.4e-07,
@ -31361,6 +31596,7 @@
"search_context_size_medium": 0.035,
"search_context_size_high": 0.035
},
"search_free_requests": {"count": 10000, "period": "day", "shared_group": "gemini-2.5-pro"},
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_token_batches": 6.25e-07,
"input_cost_per_token_flex": 6.25e-07,
@ -31458,6 +31694,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_audio_token_batches": 2.5e-07,
@ -31519,6 +31756,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -31575,6 +31813,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_audio_token_batches": 5e-07,
@ -31638,6 +31877,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_batches": 7.5e-07,
@ -31703,6 +31943,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -31764,6 +32005,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -31825,6 +32067,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -31920,6 +32163,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_flex": 1e-06,
@ -31983,6 +32227,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_flex": 1e-06,
@ -32011,6 +32256,7 @@
"output_cost_per_token_flex": 1.5e-06,
"output_cost_per_token_priority": 5.4e-06,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -32125,6 +32371,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_token_batches": 7.5e-07,
@ -32189,6 +32436,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -32249,6 +32497,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -32274,6 +32523,7 @@
"output_cost_per_token_priority": 6.75e-06,
"prompt_cache_min_tokens": 4096.0,
"search_context_cost_per_query": {"search_context_size_low":0.014,"search_context_size_medium":0.014,"search_context_size_high":0.014},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
@ -52243,6 +52493,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -52305,6 +52556,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014,
"input_cost_per_audio_token_batches": 2.5e-07
@ -52365,6 +52617,7 @@
"search_context_size_medium": 0.014,
"search_context_size_high": 0.014
},
"search_free_requests": {"count": 5000, "period": "month", "shared_group": "gemini-3"},
"web_search_billing_unit": "per_query",
"google_maps_grounding_cost_per_query": 0.014
},
@ -61148,6 +61401,41 @@
"supports_tool_choice": false,
"supports_vision": true
},
"volcengine/doubao-seed-2-1-pro-260915": {
"cache_read_input_token_cost": 1.725e-07,
"input_cost_per_token": 8.625e-07,
"litellm_provider": "volcengine",
"max_input_tokens": 1024000,
"max_output_tokens": 256000,
"max_tokens": 256000,
"mode": "chat",
"output_cost_per_token": 4.3125e-06,
"source": "https://www.volcengine.com/docs/82379/1544106",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": false,
"supports_vision": true
},
"volcengine/doubao-seed-2-1-lite-260915": {
"cache_read_input_audio_token_cost": 3.45e-07,
"cache_read_input_token_cost": 2.3e-08,
"input_cost_per_audio_token": 1.725e-06,
"input_cost_per_token": 1.15e-07,
"litellm_provider": "volcengine",
"max_input_tokens": 1024000,
"max_output_tokens": 256000,
"max_tokens": 256000,
"mode": "chat",
"output_cost_per_token": 3.88126e-07,
"source": "https://www.volcengine.com/docs/82379/1544106",
"supports_audio_input": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": false,
"supports_vision": true
},
"volcengine/doubao-seed-2-1-turbo-260628": {
"cache_read_input_token_cost": 8.625e-08,
"input_cost_per_token": 4.3125e-07,
@ -67004,6 +67292,7 @@
"fireworks_ai/accounts/fireworks/routers/glm-5p3-flash-us": {
"cache_read_input_token_cost": 4.5e-08,
"cache_read_input_token_cost_priority": 5.625e-08,
"deprecation_date": "2026-11-07",
"input_cost_per_token": 2.25e-07,
"input_cost_per_token_priority": 2.8125e-07,
"litellm_provider": "fireworks_ai",
@ -67038,6 +67327,7 @@
"fireworks_ai/glm-5p3-flash-us": {
"cache_read_input_token_cost": 4.5e-08,
"cache_read_input_token_cost_priority": 5.625e-08,
"deprecation_date": "2026-11-07",
"input_cost_per_token": 2.25e-07,
"input_cost_per_token_priority": 2.8125e-07,
"litellm_provider": "fireworks_ai",
@ -70141,6 +70431,20 @@
"output_cost_per_token_batches": 2.42e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
},
"azure/eu/o3": {
"deprecation_date": "2026-11-19",
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 2.2e-06,
"input_cost_per_token_batches": 1.1e-06,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 8.8e-06,
"output_cost_per_token_batches": 4.4e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
},
"azure/eu/o3-2025-04-16": {
"deprecation_date": "2026-11-19",
"cache_read_input_token_cost": 5.5e-07,
@ -70164,6 +70468,20 @@
"output_cost_per_token": 4.4e-05,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
},
"azure/eu/o4-mini": {
"deprecation_date": "2026-11-19",
"cache_read_input_token_cost": 3.03e-07,
"input_cost_per_token": 1.21e-06,
"input_cost_per_token_batches": 6.05e-07,
"litellm_provider": "azure",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 4.84e-06,
"output_cost_per_token_batches": 2.42e-06,
"source": "https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Foundry%20Models'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'"
},
"azure/eu/o4-mini-2025-04-16": {
"deprecation_date": "2026-11-19",
"cache_read_input_token_cost": 3.03e-07,

3
litellm/models/organization_membership.py
View file

@ -6,7 +6,6 @@ Canonical definition for ``litellm_organizationmembership``. Re-exported from
"""
from datetime import datetime
from typing import Any
from pydantic import ConfigDict, model_validator
@ -24,7 +23,7 @@ class LiteLLM_OrganizationMembershipTable(LiteLLMPydanticObjectBase):
budget_id: str | None = None
created_at: datetime
updated_at: datetime
user: Any | None = None
user: object | None = None
litellm_budget_table: LiteLLM_BudgetTable | None = None
user_email: str | None = None

2
litellm/proxy/_experimental/out/404.html
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/404/index.html
View file

File diff suppressed because one or more lines are too long

39
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.__PAGE__.txt
View file

File diff suppressed because one or more lines are too long

64
litellm/proxy/_experimental/out/__next._full.txt
View file

File diff suppressed because one or more lines are too long

6
litellm/proxy/_experimental/out/__next._tree.txt
View file

@ -1,4 +1,4 @@
:HL["/litellm-asset-prefix/_next/static/chunks/1kid9zr1--h6y.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/3146e697tym4_.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/0ckv5j_fni3xm.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/06yuul4yvgbd0.css","style"]
:HL["/litellm-asset-prefix/_next/static/media/83afe278b6a6bb3c-s.p.2bn3s6zvc0dyp.woff2","font",{"crossOrigin":"","type":"font/woff2"}]
0:{"tree":{"name":"","param":null,"prefetchHints":4176,"slots":{"children":{"name":"(dashboard)","param":null,"prefetchHints":4192,"slots":{"children":{"name":"__PAGE__","param":null,"prefetchHints":4256,"slots":null}}}}},"staleTime":300,"buildId":"kXnLzJ6ylsRPmgSkCkCKM"}
0:{"tree":{"name":"","param":null,"prefetchHints":4176,"slots":{"children":{"name":"(dashboard)","param":null,"prefetchHints":4192,"slots":{"children":{"name":"__PAGE__","param":null,"prefetchHints":4256,"slots":null}}}}},"staleTime":300,"buildId":"u_MWVLx_WA9i-AM2JdCsp"}

1
litellm/proxy/_experimental/out/_next/static/chunks/0-p31wd20yje5.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-w0rr3df0htc.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-xrvklzr286n.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/006y36jxl8z-u.js
View file

File diff suppressed because one or more lines are too long

23
litellm/proxy/_experimental/out/_next/static/chunks/01cjt3b43zsyy.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/01f03qwhd6l7d.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/01msi0ehnlc7g.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/02erlqrgtvdvz.js
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more