scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-06 08:16:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 79

Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_stream_usage_cost_default

Browse source
This commit is contained in:
mateo-berri 2026-09-01 13:03:18 -07:00
commit ac53ea0756
332 changed files with 16494 additions and 3561 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
.circleci/config.yml
View file

@ -1483,7 +1483,7 @@ jobs:
- run:
name: Run tests
command: |
uv run --no-sync python -m pytest -vv tests/local_testing/test_basic_python_version.py -k "not v2_resolver"
uv run --no-sync python -m pytest -vv tests/local_testing/test_basic_python_version.py -k "not legacy_resolver"
installing_litellm_on_python_3_13:
docker:
@ -1507,9 +1507,9 @@ jobs:
- run:
name: Run tests
command: |
uv run --no-sync python -m pytest -v tests/local_testing/test_basic_python_version.py -k "not v2_resolver"
uv run --no-sync python -m pytest -v tests/local_testing/test_basic_python_version.py -k "not legacy_resolver"
installing_litellm_on_python_v2_migration_resolver:
installing_litellm_on_python_legacy_migration_resolver:
docker:
- *python312_image
- image: cimg/postgres:16.0@sha256:b125148bc76e8e8eee5eb3ad6020a3a14110a14e8192f1c645128afebe2e2f84
@ -1536,10 +1536,10 @@ jobs:
url: tcp://localhost:5432
timeout: "60"
- run:
name: Run v2 migration resolver proxy smoke test
name: Run legacy migration resolver proxy smoke test
command: |
uv run --no-sync python -m pytest -vv \
tests/local_testing/test_basic_python_version.py::test_litellm_proxy_server_config_no_general_settings_v2_resolver
tests/local_testing/test_basic_python_version.py::test_litellm_proxy_server_config_no_general_settings_legacy_resolver
helm_chart_testing:
machine:
@ -2879,7 +2879,8 @@ jobs:
command: |
if grep -q "Error: P1001: Can't reach database server at" docker_output.log && \
(grep -q "Database setup failed after multiple retries" docker_output.log || \
grep -q "ERROR: Application startup failed. Exiting." docker_output.log); then
grep -q "ERROR: Application startup failed. Exiting." docker_output.log || \
grep -q "Database migration cannot proceed" docker_output.log); then
echo "Expected error found. Test passed."
else
echo "Expected error not found. Test failed."
@ -3011,7 +3012,7 @@ workflows:
filters: *main_branches
- installing_litellm_on_python_3_13:
filters: *main_branches
- installing_litellm_on_python_v2_migration_resolver:
- installing_litellm_on_python_legacy_migration_resolver:
filters: *main_branches
- helm_chart_testing:
requires:

2
CLAUDE.md
View file

@ -37,7 +37,7 @@ Same applies for filing bug reports and feature requests, with .github/ISSUE_TEM
If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank
Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`; the Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, just tell me which URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, what fields to fill out, etc. along with the other commands to run in an ordered list, and I'll do it myself and post the screenshots after you make the PR
Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload 2>&1 | tee litellm.log`; the Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, just tell me which URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, what fields to fill out, etc. along with the other commands to run in an ordered list, and I'll do it myself and post the screenshots after you make the PR
If you ever write any human-facing text (pull requests, issues, commit messages, discussion posts, github comments, release notes, docs, etc.), always follow these guidelines to sound less AI-y:
- don't use emojis

2
README.md
View file

@ -354,6 +354,8 @@ curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
| [Petals (`petals`)](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ | | | | | | | |
| [Pinstripes (`pinstripes`)](https://docs.litellm.ai/docs/providers/pinstripes) | ✅ | ✅ | ✅ | | | | | | | |
| [Predibase (`predibase`)](https://docs.litellm.ai/docs/providers/predibase) | ✅ | ✅ | ✅ | | | | | | | |
| [Qwen AI Platform (`qwen_ai_platform`)](https://docs.litellm.ai/docs/providers/qwencloud) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | ✅ |
| [QwenCloud (`qwencloud`)](https://docs.litellm.ai/docs/providers/qwencloud) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | ✅ |
| [Recraft (`recraft`)](https://docs.litellm.ai/docs/providers/recraft) | | | | | ✅ | | | | | |
| [Replicate (`replicate`)](https://docs.litellm.ai/docs/providers/replicate) | ✅ | ✅ | ✅ | | | | | | | |
| [Sagemaker Chat (`sagemaker_chat`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | | | | | | | |

22
basedpyright-code-budget.json
View file

@ -1,9 +1,9 @@
{
"reportAny": {
"limit": 16171
"limit": 14076
},
"reportArgumentType": {
"limit": 2224
"limit": 2216
},
"reportAssignmentType": {
"limit": 319
@ -24,7 +24,7 @@
"limit": 19
},
"reportExplicitAny": {
"limit": 5199
"limit": 4128
},
"reportFunctionMemberAccess": {
"limit": 7
@ -42,7 +42,7 @@
"limit": 12
},
"reportIndexIssue": {
"limit": 35
"limit": 25
},
"reportInvalidTypeForm": {
"limit": 34
@ -54,10 +54,10 @@
"limit": 0
},
"reportMissingParameterType": {
"limit": 5611
"limit": 5601
},
"reportMissingTypeArgument": {
"limit": 15348
"limit": 15306
},
"reportMissingTypeStubs": {
"limit": 40
@ -105,13 +105,13 @@
"limit": 109
},
"reportUnknownMemberType": {
"limit": 38465
"limit": 38350
},
"reportUnknownParameterType": {
"limit": 19663
"limit": 19626
},
"reportUnknownVariableType": {
"limit": 30064
"limit": 29890
},
"reportUnnecessaryCast": {
"limit": 111
@ -123,7 +123,7 @@
"limit": 5
},
"reportUnnecessaryIsInstance": {
"limit": 828
"limit": 826
},
"reportUntypedBaseClass": {
"limit": 0
@ -141,6 +141,6 @@
"limit": 543
},
"reportUnusedVariable": {
"limit": 139
"limit": 137
}
}

26
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -5,7 +5,7 @@ Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if t
from dataclasses import replace as dataclasses_replace
from datetime import datetime, timedelta, timezone
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Dict, Final, List, Literal, Optional, Tuple, cast
from typing import TYPE_CHECKING, Final, List, Literal, Optional, Tuple, cast
from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
@ -87,7 +87,7 @@ class CheckBatchCost:
return
self.batch_processed_support_confirmed = True
async def _get_user_info(self, batch_id: str, user_id: Optional[str]) -> Dict[str, Any]:
async def _get_user_info(self, batch_id: str, user_id: Optional[str]) -> dict[str, str | None]:
"""
Look up user email and key alias by user_id for enriching the S3 callback metadata.
Returns a dict with user_api_key_user_email and user_api_key_alias (both may be None).
@ -97,8 +97,10 @@ class CheckBatchCost:
if not user_id:
return {}
try:
user_row = await self.prisma_client.db.litellm_usertable.find_unique(
where={"user_id": user_id}
user_row: prisma_models.LiteLLM_UserTable | None = (
await self.prisma_client.db.litellm_usertable.find_unique(
where={"user_id": user_id}
)
)
if user_row is None:
return {}
@ -115,8 +117,10 @@ class CheckBatchCost:
if not api_key:
return None
try:
key_row = await self.prisma_client.db.litellm_verificationtoken.find_unique(
where={"token": api_key}
key_row: prisma_models.LiteLLM_VerificationToken | None = (
await self.prisma_client.db.litellm_verificationtoken.find_unique(
where={"token": api_key}
)
)
return getattr(key_row, "key_alias", None) if key_row is not None else None
except Exception as e:
@ -128,8 +132,10 @@ class CheckBatchCost:
if not team_id:
return None
try:
team_row = await self.prisma_client.db.litellm_teamtable.find_unique(
where={"team_id": team_id}
team_row: prisma_models.LiteLLM_TeamTable | None = (
await self.prisma_client.db.litellm_teamtable.find_unique(
where={"team_id": team_id}
)
)
return getattr(team_row, "team_alias", None) if team_row is not None else None
except Exception as e:
@ -138,7 +144,7 @@ class CheckBatchCost:
async def _build_creator_attribution_metadata(
self, job: "LiteLLM_ManagedObjectTable", batch_id: str
) -> Dict[str, Any]:
) -> dict[str, object]:
"""
Rebuild the spend-tracking metadata for the key, team, and tags that created the
batch so the batch-cost spend log is attributed the same way a non-batch request
@ -152,7 +158,7 @@ class CheckBatchCost:
team_id = getattr(job, "team_id", None)
request_tags = getattr(job, "request_tags", None)
metadata: Dict[str, Any] = {
metadata: dict[str, object] = {
"user_api_key_user_id": job.created_by,
"user_api_key": api_key,
"user_api_key_team_id": team_id,

23
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -182,6 +182,10 @@ class _ManagedObjectTableActions(Protocol):
async def update_many(self, where: Mapping[str, object], data: Mapping[str, object]) -> int: ...
class _SchedulerWithJobLookup(Protocol):
def get_job(self, job_id: str) -> object: ...
class _CursorPageArgs(TypedDict, total=False):
cursor: Mapping[str, str]
skip: int
@ -853,7 +857,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
file_ids.append(file_id)
return file_ids
def get_file_ids_from_responses_input(self, input: Union[str, List[Dict[str, Any]]]) -> List[str]:
def get_file_ids_from_responses_input(self, input: Union[str, List[Dict[str, object]]]) -> List[str]:
"""
Gets file ids from responses API input.
@ -878,7 +882,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
# Check for direct input_file type
if item.get("type") == "input_file":
file_id = item.get("file_id")
if file_id:
if isinstance(file_id, str) and file_id:
file_ids.append(file_id)
# Check for input_file in content array
@ -887,7 +891,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
for content_item in content:
if isinstance(content_item, dict) and content_item.get("type") == "input_file":
file_id = content_item.get("file_id")
if file_id:
if isinstance(file_id, str) and file_id:
file_ids.append(file_id)
return file_ids
@ -1227,7 +1231,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
# Handle both output_file_id and error_file_id
for file_attr in ["output_file_id", "error_file_id"]:
file_id_value = getattr(response, file_attr, None)
file_id_value: str | None = getattr(response, file_attr, None)
if file_id_value and model_id:
decoded_output_file_id = _is_base64_encoded_unified_file_id(file_id_value)
if decoded_output_file_id and "llm_output_file_id," in decoded_output_file_id:
@ -1496,7 +1500,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
import litellm.proxy.proxy_server as proxy_server_module
# Check if the scheduler has the batch cost checking job registered
scheduler = getattr(proxy_server_module, "scheduler", None)
scheduler: Final[_SchedulerWithJobLookup | None] = getattr(proxy_server_module, "scheduler", None)
if scheduler is None:
return False
@ -1542,7 +1546,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
)
MAX_MATCHES_TO_RETURN = 10
batches = await self.prisma_client.db.litellm_managedobjecttable.find_many(
batches = await _managed_object_table(self.prisma_client).find_many(
where={
"file_purpose": "batch",
"batch_processed": False,
@ -1552,11 +1556,14 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
order={"created_at": "desc"},
)
referencing_batches = []
referencing_batches: Final[list[dict[str, object]]] = []
for batch in batches:
try:
# Parse the batch file_object to check for file references
batch_data = json.loads(batch.file_object) if isinstance(batch.file_object, str) else batch.file_object
decoded_file_object = _decode_json_blob(batch.file_object)
batch_data: Mapping[str, object] = (
decoded_file_object if isinstance(decoded_file_object, Mapping) else {}
)
# Extract file IDs from batch
# Batches typically reference the unified file ID in input_file_id

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.62"
version = "0.1.63"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.62"
version = "0.1.63"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

4
helm/litellm/templates/backend/deployment.yaml
View file

@ -7,6 +7,10 @@ metadata:
{{- include "litellm.commonLabels" . | nindent 4 }}
app.kubernetes.io/component: backend
spec:
{{- with .Values.backend.strategy }}
strategy:
{{- toYaml . | nindent 4 }}
{{- end }}
selector:
matchLabels:
{{- include "litellm.backend.selectorLabels" . | nindent 6 }}

4
helm/litellm/templates/gateway/deployment.yaml
View file

@ -7,6 +7,10 @@ metadata:
{{- include "litellm.commonLabels" . | nindent 4 }}
app.kubernetes.io/component: gateway
spec:
{{- with .Values.gateway.strategy }}
strategy:
{{- toYaml . | nindent 4 }}
{{- end }}
selector:
matchLabels:
{{- include "litellm.gateway.selectorLabels" . | nindent 6 }}

12
helm/litellm/templates/migrations-job.yaml
View file

@ -7,6 +7,8 @@
#
# Running this pre-upgrade closes the window where new application pods would
# otherwise serve traffic against the previous release's unmigrated schema.
# Argo CD users can swap the Helm hook for a PreSync hook through
# `migrationJob.hooks`, which re-runs the Job on every sync.
apiVersion: batch/v1
kind: Job
metadata:
@ -14,10 +16,18 @@ metadata:
labels:
{{- include "litellm.commonLabels" . | nindent 4 }}
app.kubernetes.io/component: migrations
{{- if or .Values.migrationJob.hooks.helm.enabled .Values.migrationJob.hooks.argocd.enabled }}
annotations:
{{- if .Values.migrationJob.hooks.helm.enabled }}
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation
helm.sh/hook-weight: "0"
helm.sh/hook-weight: {{ .Values.migrationJob.hooks.helm.weight | default "0" | quote }}
{{- end }}
{{- if .Values.migrationJob.hooks.argocd.enabled }}
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
{{- end }}
{{- end }}
spec:
backoffLimit: {{ .Values.migrationJob.backoffLimit }}
ttlSecondsAfterFinished: {{ .Values.migrationJob.ttlSecondsAfterFinished }}

4
helm/litellm/templates/ui/deployment.yaml
View file

@ -7,6 +7,10 @@ metadata:
{{- include "litellm.commonLabels" . | nindent 4 }}
app.kubernetes.io/component: ui
spec:
{{- with .Values.ui.strategy }}
strategy:
{{- toYaml . | nindent 4 }}
{{- end }}
selector:
matchLabels:
{{- include "litellm.ui.selectorLabels" . | nindent 6 }}

63
helm/litellm/tests/migration_job_hooks_tests.yaml Normal file
View file

@ -0,0 +1,63 @@
suite: test migrations Job hook annotations
templates:
- migrations-job.yaml
values:
- ./values/required.yaml
tests:
- it: runs as a Helm pre-install / pre-upgrade hook by default
asserts:
- equal:
path: metadata.annotations["helm.sh/hook"]
value: pre-install,pre-upgrade
- equal:
path: metadata.annotations["helm.sh/hook-delete-policy"]
value: before-hook-creation
- equal:
path: metadata.annotations["helm.sh/hook-weight"]
value: "0"
- notExists:
path: metadata.annotations["argocd.argoproj.io/hook"]
- it: adds the Argo CD PreSync hook when asked
set:
migrationJob.hooks.argocd.enabled: true
asserts:
- equal:
path: metadata.annotations["argocd.argoproj.io/hook"]
value: PreSync
- equal:
path: metadata.annotations["argocd.argoproj.io/hook-delete-policy"]
value: BeforeHookCreation
- it: drops the Helm hook so Argo CD owns the Job
set:
migrationJob.hooks.argocd.enabled: true
migrationJob.hooks.helm.enabled: false
asserts:
- equal:
path: metadata.annotations["argocd.argoproj.io/hook"]
value: PreSync
- notExists:
path: metadata.annotations["helm.sh/hook"]
- notExists:
path: metadata.annotations["helm.sh/hook-delete-policy"]
- notExists:
path: metadata.annotations["helm.sh/hook-weight"]
- it: renders an ordinary Job when both hooks are disabled
set:
migrationJob.hooks.helm.enabled: false
asserts:
- notExists:
path: metadata.annotations
- equal:
path: kind
value: Job
- it: honours a custom Helm hook weight
set:
migrationJob.hooks.helm.weight: "-5"
asserts:
- equal:
path: metadata.annotations["helm.sh/hook-weight"]
value: "-5"

66
helm/litellm/tests/rollout_strategy_tests.yaml Normal file
View file

@ -0,0 +1,66 @@
suite: test rolling update strategy on the component deployments
templates:
- gateway/deployment.yaml
- gateway/configmap.yaml
- backend/deployment.yaml
- ui/deployment.yaml
values:
- ./values/required.yaml
tests:
- it: leaves the strategy to Kubernetes defaults when unset
asserts:
- notExists:
path: spec.strategy
- it: renders the configured strategy on each deployment
set:
gateway.strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
backend.strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: "25%"
maxSurge: 2
ui.strategy:
type: Recreate
asserts:
- equal:
path: spec.strategy
value:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template: gateway/deployment.yaml
- equal:
path: spec.strategy
value:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 2
template: backend/deployment.yaml
- equal:
path: spec.strategy
value:
type: Recreate
template: ui/deployment.yaml
- it: keeps a component on the cluster default when only another one sets a strategy
set:
gateway.strategy:
type: Recreate
asserts:
- equal:
path: spec.strategy.type
value: Recreate
template: gateway/deployment.yaml
- notExists:
path: spec.strategy
template: backend/deployment.yaml
- notExists:
path: spec.strategy
template: ui/deployment.yaml

29
helm/litellm/values.yaml
View file

@ -75,6 +75,22 @@ serviceAccounts:
# generate` — the migration engine doesn't need the generated client.
migrationJob:
enabled: true
# Which controller is responsible for running the Job.
#
# `helm.enabled` renders the Helm pre-install / pre-upgrade hook, so the Job
# runs whenever `helm upgrade` sees a change to apply. `argocd.enabled`
# renders an Argo CD PreSync hook instead, which runs the Job on every sync
# even when the rendered manifests are unchanged: the way to re-run
# migrations on demand from a GitOps pipeline. Turning the Helm hook off
# while the Argo CD hook is on leaves the Job out of Helm's own upgrade
# path, which is what Argo CD users want since Argo, not Helm, applies the
# manifests.
hooks:
helm:
enabled: true
weight: "0"
argocd:
enabled: false
backoffLimit: 4
ttlSecondsAfterFinished: 120
# Wall-clock budget for the whole Job, shared across every `backoffLimit`
@ -257,6 +273,15 @@ gateway:
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 10
# Rolling update tuning for the gateway Deployment. Empty by default, so
# Kubernetes applies its own RollingUpdate defaults (25% maxSurge /
# 25% maxUnavailable). Example, for a surge-only rollout behind a load
# balancer that must never lose capacity:
# type: RollingUpdate
# rollingUpdate:
# maxUnavailable: 0
# maxSurge: 1
strategy: {}
# Optional startupProbe. Empty by default, so existing installs are unchanged
# and liveness/readiness apply from container start. Set it to gate
# liveness/readiness until a slow cold start finishes — a high failureThreshold
@ -369,6 +394,8 @@ backend:
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 10
# Same shape as gateway.strategy.
strategy: {}
# Optional startupProbe; same shape as gateway.startupProbe. Empty by default.
startupProbe: {}
hpa:
@ -433,6 +460,8 @@ ui:
httpGet: { path: /, port: http }
initialDelaySeconds: 2
periodSeconds: 10
# Same shape as gateway.strategy.
strategy: {}
# Optional startupProbe; same shape as gateway.startupProbe. Empty by default.
startupProbe: {}
hpa:

132
litellm-proxy-extras/litellm_proxy_extras/utils.py
View file

@ -7,7 +7,8 @@ import subprocess
import tempfile
import time
from pathlib import Path
from typing import Optional
from types import MappingProxyType
from typing import Final, Optional
from litellm_proxy_extras._logging import logger
from litellm_proxy_extras.replica_identity import (
@ -50,6 +51,17 @@ _SPEND_LOGS_PK_CLAUSE_RE = re.compile(
re.IGNORECASE,
)
_PRISMA_ATTEMPTS: Final = 4
_TRANSIENT_PRISMA_FAILURES: Final = MappingProxyType(
{
"deadlock detected": "a deadlock on the migration advisory lock (a concurrent migrate deploy)",
"P1001": "an unreachable database server",
"P1002": "a database server that timed out",
}
)
PARTITIONED_SPEND_LOGS_PUSH_ERROR = (
"LiteLLM_SpendLogs is a partitioned table (see db_scripts/partition_spend_logs.sql), "
"so its primary key must include the partition key (\"startTime\"). `prisma db push` "
@ -274,6 +286,23 @@ class ProxyExtrasDBManager:
env=prisma_env,
)
@staticmethod
def _transient_prisma_failure(stderr: str) -> str | None:
"""Why a failed prisma command is worth retrying, or None.
v1 retried every failure, so it absorbed a database that was not up yet
or another instance holding the migration lock. v2 fails fast, which is
right for a broken migration and wrong for these.
"""
return next(
(
reason
for marker, reason in _TRANSIENT_PRISMA_FAILURES.items()
if marker in stderr
),
None,
)
@staticmethod
def _is_permission_error(error_message: str) -> bool:
"""
@ -655,7 +684,7 @@ class ProxyExtrasDBManager:
@staticmethod
def _setup_database_v2(use_migrate: bool) -> bool:
"""
v2 migration resolver (opt-in via --use_v2_migration_resolver).
v2 migration resolver (what the proxy CLI selects by default).
Runs `prisma migrate deploy` and handles standard recovery paths
(P3005 baseline, P3009/P3018 idempotent errors). Critically, it does
@ -676,20 +705,46 @@ class ProxyExtrasDBManager:
original_dir = os.getcwd()
os.chdir(migrations_dir)
try:
subprocess.run(
[_get_prisma_command(), "db", "push", "--accept-data-loss"],
timeout=prisma_command_timeout(),
check=True,
env=_get_prisma_env(),
for attempt in range(_PRISMA_ATTEMPTS):
try:
subprocess.run(
[_get_prisma_command(), "db", "push", "--accept-data-loss"],
timeout=prisma_command_timeout(),
check=True,
capture_output=True,
text=True,
env=_get_prisma_env(),
)
return True
except subprocess.TimeoutExpired:
logger.info(
"prisma db push attempt %s timed out, retrying",
attempt + 1,
)
time.sleep(random.randrange(5, 15))
except subprocess.CalledProcessError as e:
stderr = e.stderr or ""
transient = ProxyExtrasDBManager._transient_prisma_failure(
stderr
)
# Re-raise as RuntimeError so proxy_cli.py's
# `except RuntimeError` catches it and exits cleanly.
if transient is None or attempt == _PRISMA_ATTEMPTS - 1:
raise RuntimeError(
f"prisma db push failed.\n\nDetail: {e}"
f"\n\nPrisma error:\n{stderr}"
) from e
logger.info(
"prisma db push attempt %s failed on %s, retrying. "
"Prisma error:\n%s",
attempt + 1,
transient,
stderr,
)
time.sleep(random.randrange(5, 15))
raise RuntimeError(
f"prisma db push failed after {_PRISMA_ATTEMPTS} attempts."
)
return True
except (
subprocess.CalledProcessError,
subprocess.TimeoutExpired,
) as e:
# Re-raise as RuntimeError so proxy_cli.py's
# `except RuntimeError` catches it and exits cleanly.
raise RuntimeError(f"prisma db push failed.\n\nDetail: {e}") from e
finally:
os.chdir(original_dir)
@ -699,7 +754,7 @@ class ProxyExtrasDBManager:
original_dir = os.getcwd()
os.chdir(migrations_dir)
try:
for attempt in range(4):
for attempt in range(_PRISMA_ATTEMPTS):
try:
result = subprocess.run(
[_get_prisma_command(), "migrate", "deploy"],
@ -814,16 +869,36 @@ class ProxyExtrasDBManager:
f"Manual intervention required.\n\nPrisma error:\n{stderr}"
) from e
raise RuntimeError(
"Database migration failed and cannot be auto-recovered. "
f"Manual intervention required.\n\nPrisma error:\n{stderr}"
) from e
transient = ProxyExtrasDBManager._transient_prisma_failure(stderr)
if transient is None:
raise RuntimeError(
"Database migration failed and cannot be auto-recovered. "
f"Manual intervention required.\n\nPrisma error:\n{stderr}"
) from e
if attempt == _PRISMA_ATTEMPTS - 1:
raise RuntimeError(
f"Database migration failed after "
f"{_PRISMA_ATTEMPTS} attempts on {transient}. "
"Check database connectivity and load."
f"\n\nPrisma error:\n{stderr}"
) from e
logger.info(
"prisma migrate deploy attempt %s failed on %s, retrying. "
"Prisma error:\n%s",
attempt + 1,
transient,
stderr,
)
time.sleep(random.randrange(5, 15))
continue
raise RuntimeError(
"Database migration failed after 4 attempts (retry loop "
"exhausted by timeouts or repeated idempotent-recovery "
"continues). Check database connectivity, load, and "
"_prisma_migrations ledger state."
f"Database migration failed after {_PRISMA_ATTEMPTS} "
"attempts (retry loop exhausted by timeouts or repeated "
"idempotent-recovery continues). Check database connectivity, "
"load, and _prisma_migrations ledger state."
)
finally:
os.chdir(original_dir)
@ -871,10 +946,11 @@ class ProxyExtrasDBManager:
Args:
use_migrate: Whether to use prisma migrate instead of db push
use_v2_resolver: Opt into the v2 migration resolver (safer during
use_v2_resolver: Run the v2 migration resolver (safer during
rolling deploys; does not run the diff-and-force recovery
that causes schema thrashing). Defaults to False for
backwards compatibility.
that causes schema thrashing). Defaults to False here so
direct callers keep the old behavior; the proxy CLI passes
True, so the proxy's runtime default is v2.
Returns:
bool: True if setup was successful, False otherwise
@ -892,7 +968,7 @@ class ProxyExtrasDBManager:
@staticmethod
def _run_migrations(use_migrate: bool, use_v2_resolver: bool) -> bool:
if use_v2_resolver:
logger.info("Using v2 migration resolver (--use_v2_migration_resolver)")
logger.info("Using v2 migration resolver")
return ProxyExtrasDBManager._setup_database_v2(use_migrate=use_migrate)
schema_path = ProxyExtrasDBManager._get_prisma_dir() + "/schema.prisma"

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.91"
version = "0.4.92"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.91"
version = "0.4.92"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

0
litellm-proxy-extras/tests/__init__.py
View file

242
litellm-proxy-extras/tests/test_setup_database_fail_fast.py
View file

@ -1,242 +0,0 @@
"""Regression tests for ProxyExtrasDBManager v2 migration resolver.
The v2 resolver is opt-in via `--use_v2_migration_resolver` / the
`use_v2_resolver=True` kwarg. These tests exercise the v2 path; the v1
(default) behavior is unchanged from pre-fix.
"""
import subprocess
from unittest.mock import patch
import pytest
from litellm_proxy_extras.utils import (
ProxyExtrasDBManager,
_max_migration_timestamp,
_migration_timestamp,
)
def _fake_migrate_deploy_failure(returncode: int, stderr: str):
def _run(*args, **kwargs):
raise subprocess.CalledProcessError(
returncode=returncode,
cmd=args[0],
stderr=stderr,
output="",
)
return _run
def test_v2_p3018_permission_error_raises_runtime_error(monkeypatch, tmp_path):
"""v2: a permission failure during migrate deploy raises RuntimeError."""
monkeypatch.setenv("DATABASE_URL", "postgresql://u:p@localhost:9/x")
monkeypatch.setattr(
ProxyExtrasDBManager, "_warn_if_db_ahead_of_head", lambda _: None
)
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
stderr = (
"Error: P3018\nMigration name: 20250326162113_baseline\n"
"Database error code: 42501\npermission denied for schema public"
)
with patch("subprocess.run", side_effect=_fake_migrate_deploy_failure(1, stderr)):
with pytest.raises(RuntimeError, match="permission"):
ProxyExtrasDBManager.setup_database(use_migrate=True, use_v2_resolver=True)
def test_v2_non_idempotent_p3009_raises_runtime_error(monkeypatch, tmp_path):
"""v2: a non-idempotent migration failure raises (no silent recovery)."""
monkeypatch.setenv("DATABASE_URL", "postgresql://u:p@localhost:9/x")
monkeypatch.setattr(
ProxyExtrasDBManager, "_warn_if_db_ahead_of_head", lambda _: None
)
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
stderr = (
"Error: P3009\nMigration `20260101000000_genuinely_broken` failed\n"
'Reason: syntax error at or near "BRKN" LINE 42'
)
with patch("subprocess.run", side_effect=_fake_migrate_deploy_failure(1, stderr)):
with pytest.raises(RuntimeError, match="cannot be auto-recovered"):
ProxyExtrasDBManager.setup_database(use_migrate=True, use_v2_resolver=True)
def test_strip_prisma_query_params_removes_connection_limit():
"""DATABASE_URLs with Prisma-specific params should be parseable by psycopg."""
url = "postgresql://u:p@h:5432/db?connection_limit=100&pool_timeout=60&sslmode=require"
stripped = ProxyExtrasDBManager._strip_prisma_query_params(url)
assert "connection_limit" not in stripped
assert "pool_timeout" not in stripped
assert "sslmode=require" in stripped
def test_strip_prisma_query_params_passthrough_no_query():
"""URLs without query strings are returned unchanged."""
url = "postgresql://u:p@h:5432/db"
assert ProxyExtrasDBManager._strip_prisma_query_params(url) == url
def test_migration_timestamp_extracts_leading_digits():
assert _migration_timestamp("20260101000000_add_foo") == 20260101000000
assert _migration_timestamp("20250326162113_baseline") == 20250326162113
def test_migration_timestamp_returns_zero_on_malformed():
assert _migration_timestamp("0_init") == 0
assert _migration_timestamp("not_a_migration") == 0
def test_max_migration_timestamp():
names = {"20250326000000_a", "20260415000000_b", "20251115000000_c"}
assert _max_migration_timestamp(names) == 20260415000000
def test_max_migration_timestamp_empty_set():
assert _max_migration_timestamp(set()) == 0
def test_v1_default_still_calls_resolve_all_migrations(monkeypatch, tmp_path):
"""v1 (default) continues to call _resolve_all_migrations on the happy path.
This is the existing buggy behavior — we're not fixing it in v1, only
offering v2 as opt-in. This test pins the default so that a future
inadvertent default flip is caught.
"""
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
# Stub `prisma migrate deploy` to claim success with pending migrations
# applied, which is the code path that triggers the legacy post-migration
# sanity check (a call to _resolve_all_migrations).
class FakeResult:
stdout = "Applied migration.\n"
stderr = ""
def fake_run(cmd, *args, **kwargs):
return FakeResult()
resolve_called = {"n": 0}
def fake_resolve(*args, **kwargs):
resolve_called["n"] += 1
monkeypatch.setattr("subprocess.run", fake_run)
monkeypatch.setattr(ProxyExtrasDBManager, "_resolve_all_migrations", fake_resolve)
ok = ProxyExtrasDBManager.setup_database(use_migrate=True) # v2 flag NOT set
assert ok is True
assert resolve_called["n"] == 1, "v1 default should still invoke the legacy path"
def test_v2_db_push_wraps_subprocess_error_as_runtime_error(monkeypatch, tmp_path):
"""v2: a failing `prisma db push` must raise RuntimeError, not leak
CalledProcessError past proxy_cli.py's `except RuntimeError`."""
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
stderr = "db push error"
with patch("subprocess.run", side_effect=_fake_migrate_deploy_failure(1, stderr)):
with pytest.raises(RuntimeError, match="prisma db push failed"):
ProxyExtrasDBManager.setup_database(use_migrate=False, use_v2_resolver=True)
def test_v2_warn_ahead_of_head_swallows_db_errors(monkeypatch, tmp_path):
"""_warn_if_db_ahead_of_head must never raise — it's informational.
Non-connection DB errors (e.g. InsufficientPrivilege from a user
without SELECT on _prisma_migrations) must be caught, not propagated.
"""
import psycopg
monkeypatch.setenv("DATABASE_URL", "postgresql://u:p@localhost:9/x")
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
class _FakeConn:
def __enter__(self):
return self
def __exit__(self, *a):
return False
def execute(self, *a, **kw):
# Simulate an InsufficientPrivilege (subclass of DatabaseError).
raise psycopg.errors.InsufficientPrivilege("permission denied")
def _fake_connect(*a, **kw):
return _FakeConn()
monkeypatch.setattr("psycopg.connect", _fake_connect)
# Must not raise.
ProxyExtrasDBManager._warn_if_db_ahead_of_head(str(tmp_path))
def test_v2_resolve_specific_migration_failure_raises_runtime_error(
monkeypatch, tmp_path
):
"""If marking a migration as applied fails inside P3009 idempotent
recovery, the subprocess error must be re-raised as RuntimeError so
proxy_cli.py catches it cleanly (instead of leaking CalledProcessError)."""
monkeypatch.setattr(
ProxyExtrasDBManager, "_warn_if_db_ahead_of_head", lambda _: None
)
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
monkeypatch.setattr(
ProxyExtrasDBManager, "_roll_back_migration", lambda *a, **kw: None
)
# First call: migrate deploy -> P3009 idempotent error.
# Recovery path tries _resolve_specific_migration; that also raises.
def _failing_resolve(*a, **kw):
raise subprocess.CalledProcessError(
returncode=1,
cmd="prisma migrate resolve --applied",
stderr="resolve failed",
output="",
)
monkeypatch.setattr(
ProxyExtrasDBManager, "_resolve_specific_migration", _failing_resolve
)
stderr = (
"Error: P3009\nMigration `20260101000000_some_migration` failed\n"
"relation already exists"
)
with patch("subprocess.run", side_effect=_fake_migrate_deploy_failure(1, stderr)):
with pytest.raises(
RuntimeError, match="Failed to mark migration .* as applied"
):
ProxyExtrasDBManager.setup_database(use_migrate=True, use_v2_resolver=True)
def test_v2_does_not_call_resolve_all_migrations(monkeypatch, tmp_path):
"""v2 must never call _resolve_all_migrations — that's the bug it fixes."""
monkeypatch.setattr(
ProxyExtrasDBManager, "_warn_if_db_ahead_of_head", lambda _: None
)
monkeypatch.setattr(ProxyExtrasDBManager, "_get_prisma_dir", lambda: str(tmp_path))
(tmp_path / "schema.prisma").write_text("// stub")
class FakeResult:
stdout = "Applied migration.\n"
stderr = ""
monkeypatch.setattr("subprocess.run", lambda *a, **kw: FakeResult())
resolve_called = {"n": 0}
monkeypatch.setattr(
ProxyExtrasDBManager,
"_resolve_all_migrations",
lambda *a, **kw: resolve_called.__setitem__("n", resolve_called["n"] + 1),
)
ok = ProxyExtrasDBManager.setup_database(use_migrate=True, use_v2_resolver=True)
assert ok is True
assert resolve_called["n"] == 0, "v2 must not invoke the diff-and-force recovery"

28
litellm/__init__.py
View file

@ -659,6 +659,8 @@ aiml_models: Set = set()
deepgram_models: Set = set()
elevenlabs_models: Set = set()
dashscope_models: Set = set()
qwencloud_models: Set = set()
qwen_ai_platform_models: Set = set()
moonshot_models: Set = set()
publicai_models: Set = set()
darkbloom_models: Set = set()
@ -909,6 +911,10 @@ def _populate_provider_model_sets(model_cost_map: Dict) -> None:
heroku_models.add(key)
elif value.get("litellm_provider") == "dashscope":
dashscope_models.add(key)
elif value.get("litellm_provider") == "qwencloud":
qwencloud_models.add(key)
elif value.get("litellm_provider") == "qwen_ai_platform":
qwen_ai_platform_models.add(key)
elif value.get("litellm_provider") == "modelscope":
modelscope_models.add(key)
elif value.get("litellm_provider") == "moonshot":
@ -1072,6 +1078,8 @@ model_list = list(
| deepgram_models
| elevenlabs_models
| dashscope_models
| qwencloud_models
| qwen_ai_platform_models
| moonshot_models
| publicai_models
| darkbloom_models
@ -1178,6 +1186,8 @@ def _build_models_by_provider() -> dict:
"elevenlabs": elevenlabs_models,
"heroku": heroku_models,
"dashscope": dashscope_models,
"qwencloud": qwencloud_models,
"qwen_ai_platform": qwen_ai_platform_models,
"modelscope": modelscope_models,
"moonshot": moonshot_models,
"publicai": publicai_models,
@ -2014,6 +2024,24 @@ if TYPE_CHECKING:
from .llms.dashscope.rerank.transformation import (
DashScopeRerankConfig as DashScopeRerankConfig,
)
from .llms.dashscope.qwencloud import (
QwenCloudChatConfig as QwenCloudChatConfig,
)
from .llms.dashscope.qwencloud import (
QwenCloudEmbeddingConfig as QwenCloudEmbeddingConfig,
)
from .llms.dashscope.qwencloud import (
QwenCloudRerankConfig as QwenCloudRerankConfig,
)
from .llms.dashscope.qwen_ai_platform import (
QwenAIPlatformChatConfig as QwenAIPlatformChatConfig,
)
from .llms.dashscope.qwen_ai_platform import (
QwenAIPlatformEmbeddingConfig as QwenAIPlatformEmbeddingConfig,
)
from .llms.dashscope.qwen_ai_platform import (
QwenAIPlatformRerankConfig as QwenAIPlatformRerankConfig,
)
from .llms.modelscope.chat.transformation import (
ModelScopeChatConfig as ModelScopeChatConfig,
)

10
litellm/_lazy_imports_registry.py
View file

@ -310,6 +310,8 @@ LLM_CONFIG_NAMES: Final = (
"GigaChatConfig",
"GigaChatEmbeddingConfig",
"DashScopeChatConfig",
"QwenCloudChatConfig",
"QwenAIPlatformChatConfig",
"ModelScopeChatConfig",
"MoonshotChatConfig",
"DockerModelRunnerChatConfig",
@ -1172,6 +1174,14 @@ _LLM_CONFIGS_IMPORT_MAP: Final = {
".llms.dashscope.chat.transformation",
"DashScopeChatConfig",
),
"QwenCloudChatConfig": (
".llms.dashscope.qwencloud",
"QwenCloudChatConfig",
),
"QwenAIPlatformChatConfig": (
".llms.dashscope.qwen_ai_platform",
"QwenAIPlatformChatConfig",
),
"GDCGeminiConfig": (
".llms.gdc.chat.transformation",
"GDCGeminiConfig",

9
litellm/caching/caching.py
View file

@ -12,6 +12,7 @@ import hashlib
import json
import time
import traceback
from collections.abc import Mapping
from enum import Enum
from typing import Any, Final
@ -506,7 +507,7 @@ class Cache:
def _get_cache_logic(
self,
cached_result: Any | None,
cached_result: object | None,
max_age: float | None,
):
"""
@ -538,8 +539,8 @@ class Cache:
return cached_result
@staticmethod
def _get_safe_cache_lookup_kwargs(kwargs: dict[str, Any]) -> dict[str, Any]:
cache_lookup_kwargs: Final[dict[str, Any]] = {}
def _get_safe_cache_lookup_kwargs(kwargs: Mapping[str, object]) -> dict[str, object]:
cache_lookup_kwargs: Final[dict[str, object]] = {}
for prompt_kwarg in ("messages", "input"):
if prompt_kwarg in kwargs:
cache_lookup_kwargs[prompt_kwarg] = kwargs[prompt_kwarg]
@ -552,7 +553,7 @@ class Cache:
@staticmethod
def _update_metadata_from_cache_lookup_kwargs(
original_kwargs: dict[str, Any], cache_lookup_kwargs: dict[str, Any]
original_kwargs: Mapping[str, object], cache_lookup_kwargs: Mapping[str, object]
) -> None:
original_metadata: Final = original_kwargs.get("metadata")
cache_lookup_metadata: Final = cache_lookup_kwargs.get("metadata")

18
litellm/caching/qdrant_semantic_cache.py
View file

@ -12,7 +12,7 @@ import ast
import asyncio
import json
import os
from typing import TYPE_CHECKING, Any, Final, cast
from typing import TYPE_CHECKING, Any, Final, Protocol, cast
import litellm
from litellm._logging import print_verbose
@ -39,6 +39,12 @@ if TYPE_CHECKING:
from litellm.router import Router
class _QdrantCollectionDetailsResponse(Protocol):
"""The qdrant `/collections/{name}` response, whose body is kept as an opaque JSON object."""
def json(self) -> dict[str, object]: ...
class QdrantSemanticCache(BaseCache):
CACHE_KEY_FIELD_NAME = "litellm_cache_key"
embedding_max_input_tokens: int | None = None
@ -115,15 +121,15 @@ class QdrantSemanticCache(BaseCache):
raise ValueError(f"Error from qdrant checking if /collections exist {collection_exists.text}")
if collection_exists.json()["result"]["exists"]:
collection_details = self.sync_client.get(
collection_details: _QdrantCollectionDetailsResponse = self.sync_client.get(
url=f"{self.qdrant_api_base}/collections/{self.collection_name}",
headers=self.headers,
)
self.collection_info = collection_details.json()
self.collection_info: dict[str, object] = collection_details.json()
print_verbose(f"Collection already exists.\nCollection details:{self.collection_info}")
self._ensure_cache_key_payload_index()
else:
quantization_params: dict[str, Any]
quantization_params: dict[str, dict[str, object]]
if quantization_config is None or quantization_config == "binary":
quantization_params = {
"binary": {
@ -214,7 +220,7 @@ class QdrantSemanticCache(BaseCache):
resolve_embedding_max_input_tokens(self.embedding_max_input_tokens, self.embedding_model, router),
)
def _get_embedding(self, prompt: str, metadata: dict[str, Any] | None = None) -> EmbeddingResponse:
def _get_embedding(self, prompt: str, metadata: dict[str, object] | None = None) -> EmbeddingResponse:
"""Embed via the proxy Router when it serves the model, else direct."""
try:
from litellm.proxy.proxy_server import llm_model_list, llm_router
@ -241,7 +247,7 @@ class QdrantSemanticCache(BaseCache):
num_retries=0,
)
async def _get_async_embedding(self, prompt: str, metadata: dict[str, Any] | None = None) -> EmbeddingResponse:
async def _get_async_embedding(self, prompt: str, metadata: dict[str, object] | None = None) -> EmbeddingResponse:
try:
from litellm.proxy.proxy_server import llm_model_list, llm_router
except ImportError:

57
litellm/caching/redis_cache.py
View file

@ -18,7 +18,7 @@ import time
from collections.abc import Awaitable, Callable, Sequence
from contextvars import ContextVar
from datetime import timedelta
from typing import TYPE_CHECKING, Any, Final, TypeVar, cast
from typing import TYPE_CHECKING, Any, Final, Protocol, TypeVar, cast
import litellm
from litellm._logging import print_verbose, verbose_logger
@ -58,6 +58,26 @@ else:
Span = Any
class _AsyncRedisCommands(Protocol):
"""Async redis commands this cache issues.
redis-py's type stubs omit these methods on RedisCluster, so the union returned by
init_async_client() is untyped at every call site without this protocol.
"""
def ping(self) -> Awaitable[bool]: ...
def delete(self, *names: str) -> Awaitable[int]: ...
def ttl(self, name: str) -> Awaitable[int]: ...
def rpush(self, name: str, *values: str | bytes | float) -> Awaitable[int]: ...
def lpop(self, name: str, count: int | None = None) -> Awaitable[object]: ...
def pipeline(self, transaction: bool = True) -> "Pipeline[bytes]": ...
def _get_call_stack_info(num_frames: int = 2) -> str:
"""
Get the function names from the previous 1-2 functions in the call stack.
@ -429,6 +449,9 @@ class RedisCache(BaseCache):
self.redis_async_client = redis_async_client
return redis_async_client
def _async_commands(self) -> _AsyncRedisCommands:
return self.init_async_client()
def check_and_fix_namespace(self, key: str) -> str:
"""
Make sure each key starts with the given namespace
@ -1055,19 +1078,17 @@ class RedisCache(BaseCache):
await self.async_set_cache_pipeline(self.redis_batch_writing_buffer)
self.redis_batch_writing_buffer = []
def _get_cache_logic(self, cached_response: Any):
def _get_cache_logic(self, cached_response: bytes | str | None):
"""
Common 'get_cache_logic' across sync + async redis client implementations
"""
if cached_response is None:
return cached_response
# cached_response is in `b{} convert it to ModelResponse
cached_response = cached_response.decode("utf-8") # Convert bytes to string
return None
decoded: Final = cached_response.decode("utf-8") if isinstance(cached_response, bytes) else cached_response
try:
cached_response = json.loads(cached_response) # Convert string to dictionary
return json.loads(decoded)
except Exception:
cached_response = ast.literal_eval(cached_response)
return cached_response
return ast.literal_eval(decoded)
def get_cache(self, key, parent_otel_span: Span | None = None, **kwargs):
try:
@ -1314,8 +1335,7 @@ class RedisCache(BaseCache):
raise e
async def ping(self) -> bool:
# typed as Any, redis python lib has incomplete type stubs for RedisCluster and does not include `ping`
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
start_time: Final = time.time()
print_verbose("Pinging Async Redis Cache")
try:
@ -1349,8 +1369,7 @@ class RedisCache(BaseCache):
@_redis_circuit_breaker_guard
async def delete_cache_keys(self, keys):
# typed as Any, redis python lib has incomplete type stubs for RedisCluster and does not include `delete`
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
keys = [self.check_and_fix_namespace(key=key) for key in keys]
# keys is a list, unpack it so it gets passed as individual elements to delete
await _redis_client.delete(*keys)
@ -1415,8 +1434,7 @@ class RedisCache(BaseCache):
@_redis_circuit_breaker_guard
async def async_delete_cache(self, key: str):
# typed as Any, redis python lib has incomplete type stubs for RedisCluster and does not include `delete`
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
key = self.check_and_fix_namespace(key=key)
# keys is str
return await _redis_client.delete(key)
@ -1523,8 +1541,7 @@ class RedisCache(BaseCache):
Redis ref: https://redis.io/docs/latest/commands/ttl/
"""
try:
# typed as Any, redis python lib has incomplete type stubs for RedisCluster and does not include `ttl`
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
key = self.check_and_fix_namespace(key=key)
ttl: Final = await _redis_client.ttl(key)
if ttl <= -1: # -1 means the key does not exist, -2 key does not exist
@ -1554,7 +1571,7 @@ class RedisCache(BaseCache):
Returns:
int: The length of the list after the push operation
"""
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
key = self.check_and_fix_namespace(key=key)
start_time: Final = time.time()
try:
@ -1621,7 +1638,7 @@ class RedisCache(BaseCache):
if len(rpush_list) == 0:
return []
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
start_time: Final = time.time()
try:
@ -1678,7 +1695,7 @@ class RedisCache(BaseCache):
parent_otel_span: Span | None = None,
**kwargs,
) -> Any | list[Any]:
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
key = self.check_and_fix_namespace(key=key)
start_time: Final = time.time()
print_verbose(f"LPOP from Redis list: key: {key}, count: {count}")
@ -1810,7 +1827,7 @@ class RedisCache(BaseCache):
if len(lpop_list) == 0:
return []
_redis_client: Final[Any] = self.init_async_client()
_redis_client: Final = self._async_commands()
start_time: Final = time.time()
try:

14
litellm/completion_extras/litellm_responses_transformation/handler.py
View file

@ -45,14 +45,14 @@ class ResponsesToCompletionBridgeHandler:
return bool(stream)
@staticmethod
def _is_preformatted_cached_chat_stream(result: Any) -> bool:
def _is_preformatted_cached_chat_stream(result: object) -> bool:
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
return isinstance(result, CustomStreamWrapper) and result.custom_llm_provider == "cached_response"
@staticmethod
def _coerce_response_object(
response_obj: Any,
response_obj: object,
hidden_params: dict | None,
) -> "ResponsesAPIResponse":
if isinstance(response_obj, ResponsesAPIResponse):
@ -78,8 +78,8 @@ class ResponsesToCompletionBridgeHandler:
for _ in stream_iter:
pass
completed: Final = getattr(stream_iter, "completed_response", None)
response_obj: Final = getattr(completed, "response", None) if completed else None
completed: Final[object] = getattr(stream_iter, "completed_response", None)
response_obj: Final[object] = getattr(completed, "response", None) if completed else None
if response_obj is None:
raise ValueError("Stream ended without a completed response")
@ -93,8 +93,8 @@ class ResponsesToCompletionBridgeHandler:
async for _ in stream_iter:
pass
completed: Final = getattr(stream_iter, "completed_response", None)
response_obj: Final = getattr(completed, "response", None) if completed else None
completed: Final[object] = getattr(stream_iter, "completed_response", None)
response_obj: Final[object] = getattr(completed, "response", None) if completed else None
if response_obj is None:
raise ValueError("Stream ended without a completed response")
@ -157,7 +157,7 @@ class ResponsesToCompletionBridgeHandler:
def completion(
self, *args, **kwargs
) -> Union[
Coroutine[Any, Any, Union["ModelResponse", "CustomStreamWrapper"]],
Coroutine[None, None, Union["ModelResponse", "CustomStreamWrapper"]],
"ModelResponse",
"CustomStreamWrapper",
]:

7
litellm/completion_extras/litellm_responses_transformation/transformation.py
View file

@ -212,7 +212,8 @@ def _tool_call_dict_from_output_item(item: Mapping[str, Any], index: int) -> _Ch
LiteLLMCompletionResponsesConfig,
)
is_custom: Final = item.get("type") == "custom_tool_call"
item_type: Final[object] = item.get("type")
is_custom: Final = item_type == "custom_tool_call"
arguments: Final = (item.get("input") if is_custom else item.get("arguments")) or ""
name: Final = item.get("name") or ("custom_tool" if is_custom else "")
function_chunk: Final = ChatCompletionToolCallFunctionChunk(name=name, arguments=arguments)
@ -222,7 +223,7 @@ def _tool_call_dict_from_output_item(item: Mapping[str, Any], index: int) -> _Ch
function=function_chunk,
index=index,
)
raw_provider_fields: Final = item.get("provider_specific_fields")
raw_provider_fields: Final[object] = item.get("provider_specific_fields")
if isinstance(raw_provider_fields, dict):
provider_specific_fields = raw_provider_fields
elif raw_provider_fields and hasattr(raw_provider_fields, "__dict__"):
@ -507,7 +508,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def _merge_responses_api_request_into_request_data(
self,
request_data: dict[str, Any],
request_data: dict[str, object],
responses_api_request: "ResponsesAPIOptionalRequestParams",
instructions: str | None,
) -> None:

14
litellm/constants.py
View file

@ -630,6 +630,8 @@ LITELLM_CHAT_PROVIDERS: Final = [
"nscale",
"nebius",
"dashscope",
"qwencloud",
"qwen_ai_platform",
"modelscope",
"moonshot",
"publicai",
@ -799,6 +801,7 @@ openai_compatible_endpoints: Final[list] = [
"inference.api.nscale.com/v1",
"api.studio.nebius.ai/v1",
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
"https://dashscope.aliyuncs.com/compatible-mode/v1",
"https://api-inference.modelscope.cn/v1",
"https://api.moonshot.ai/v1",
"https://api.publicai.co/v1",
@ -872,6 +875,8 @@ openai_compatible_providers: Final[list] = [
"nscale",
"nebius",
"dashscope",
"qwencloud",
"qwen_ai_platform",
"modelscope",
"moonshot",
"v0",
@ -902,6 +907,8 @@ openai_text_completion_compatible_providers: Final[list] = [ # providers that s
"featherless_ai",
"nebius",
"dashscope",
"qwencloud",
"qwen_ai_platform",
"modelscope",
"moonshot",
"publicai",
@ -1109,7 +1116,7 @@ nebius_models: Final[set] = set(
]
)
dashscope_models: Final[set] = set(
dashscope_models: Final[frozenset] = frozenset(
[
"qwen-turbo",
"qwen-plus",
@ -1124,6 +1131,10 @@ dashscope_models: Final[set] = set(
]
)
qwencloud_models: Final[frozenset] = frozenset(dashscope_models)
qwen_ai_platform_models: Final[frozenset] = frozenset(dashscope_models)
nebius_embedding_models: Final[set] = set(
[
"BAAI/bge-en-icl",
@ -1240,6 +1251,7 @@ BEDROCK_CONVERSE_MODELS: Final = [
"openai.gpt-oss-120b-1:0",
"anthropic.claude-haiku-4-5-20251001-v1:0",
"anthropic.claude-sonnet-4-5-20250929-v1:0",
"anthropic.claude-fable-5-1",
"anthropic.claude-fable-5",
"anthropic.claude-sonnet-5",
"anthropic.claude-opus-5",

34
litellm/cost_calculator.py
View file

@ -641,12 +641,12 @@ def cost_per_token(
return xai_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "lemonade":
return lemonade_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "dashscope":
elif custom_llm_provider in ("dashscope", "qwencloud", "qwen_ai_platform"):
from litellm.llms.dashscope.cost_calculator import (
cost_per_token as dashscope_cost_per_token,
)
return dashscope_cost_per_token(model=model, usage=usage_block)
return dashscope_cost_per_token(model=model, usage=usage_block, custom_llm_provider=custom_llm_provider)
elif custom_llm_provider == "azure_ai":
return azure_ai_cost_per_token(
model=model,
@ -1910,12 +1910,15 @@ def ocr_cost(
if credits is not None and cost_per_credit is not None:
return cost_per_credit * credits, 0.0
ocr_cost_per_page: float | None = None
if model_info is not None:
ocr_cost_per_page = model_info.get("ocr_cost_per_page")
ocr_cost_per_page: Final = model_info.get("ocr_cost_per_page") if model_info is not None else None
annotation_cost_per_page: Final = model_info.get("annotation_cost_per_page") if model_info is not None else None
annotation_rate: Final = annotation_cost_per_page if annotation_cost_per_page is not None else ocr_cost_per_page
pages_processed: Final = response.usage_info.pages_processed
if pages_processed is None:
annotation_pages: Final = response.usage_info.pages_processed_annotation or 0
has_billable_annotation_pages: Final = annotation_rate is not None and annotation_pages > 0
if pages_processed is None and not has_billable_annotation_pages:
if cost_per_credit is not None or ocr_cost_per_page is None:
# Surface missing usage data instead of silently under-reporting
# cost. The previous behavior raised ValueError; we now return 0.0
@ -1931,7 +1934,7 @@ def ocr_cost(
return 0.0, 0.0
raise ValueError("OCR response pages_processed is None")
if ocr_cost_per_page is None:
if ocr_cost_per_page is None and not has_billable_annotation_pages:
# No per-page pricing configured. Either the model is on credit-based
# pricing (and credits weren't returned, so the credit branch above did
# not match) or the model has no OCR pricing entry at all. Surface a
@ -1947,8 +1950,9 @@ def ocr_cost(
)
return 0.0, 0.0
total_ocr_processing_cost: Final[float] = ocr_cost_per_page * pages_processed
return total_ocr_processing_cost, 0.0
ocr_pages_cost: Final = (ocr_cost_per_page or 0.0) * (pages_processed or 0)
annotation_pages_cost: Final = (annotation_rate or 0.0) * annotation_pages
return ocr_pages_cost + annotation_pages_cost, 0.0
def vector_store_search_cost(
@ -2268,6 +2272,10 @@ def batch_cost_calculator(
return total_prompt_cost, total_completion_cost
def _attribute_value(obj: object, name: str) -> object:
return getattr(obj, name)
def _summable_prompt_token_fields(prompt_tokens_details: BaseModel) -> list[str]:
field_names: Final = list(type(prompt_tokens_details).model_fields)
if getattr(prompt_tokens_details, "cache_write_tokens", None) is None:
@ -2293,7 +2301,7 @@ class BaseTokenUsageProcessor:
for usage in usage_objects:
# Handle direct attributes by checking what exists in the model
for attr in dir(usage):
if not attr.startswith("_") and not callable(getattr(usage, attr)):
if not attr.startswith("_") and not callable(_attribute_value(usage, attr)):
current_val = getattr(combined, attr, 0)
new_val = getattr(usage, attr, 0)
if (
@ -2313,7 +2321,7 @@ class BaseTokenUsageProcessor:
if (
hasattr(usage.prompt_tokens_details, attr)
and not attr.startswith("_")
and not callable(getattr(usage.prompt_tokens_details, attr))
and not callable(_attribute_value(usage.prompt_tokens_details, attr))
):
current_val = getattr(combined.prompt_tokens_details, attr, 0) or 0
new_val = getattr(usage.prompt_tokens_details, attr, 0) or 0
@ -2332,7 +2340,9 @@ class BaseTokenUsageProcessor:
# Check what keys exist in the model's completion_tokens_details
# Access model_fields on the class, not the instance, to avoid Pydantic 2.11+ deprecation warnings
for attr in type(usage.completion_tokens_details).model_fields:
if not attr.startswith("_") and not callable(getattr(usage.completion_tokens_details, attr)):
if not attr.startswith("_") and not callable(
_attribute_value(usage.completion_tokens_details, attr)
):
current_val = getattr(combined.completion_tokens_details, attr, 0) or 0
new_val = getattr(usage.completion_tokens_details, attr, 0) or 0
if isinstance(new_val, (int, float)):

2
litellm/google_genai/adapters/transformation.py
View file

@ -722,7 +722,7 @@ class GoogleGenAIAdapter:
)
for tool_call in tool_calls:
if not hasattr(tool_call, "function"):
if not hasattr(tool_call, "function") or isinstance(tool_call, ChatCompletionDeltaCustomToolCall):
continue
# 3. Use `index` as the primary key for accumulation

30
litellm/google_genai/main.py
View file

@ -52,10 +52,10 @@ class GenerateContentSetupResult(BaseModel):
model_config: ClassVar[ConfigDict] = ConfigDict(arbitrary_types_allowed=True)
model: str
request_body: dict[str, Any]
request_body: dict[str, object]
custom_llm_provider: str
generate_content_provider_config: BaseGoogleGenAIGenerateContentConfig | None
generate_content_config_dict: dict[str, Any]
generate_content_config_dict: dict[str, object]
native_request_fields: dict[str, object]
litellm_params: GenericLiteLLMParams
litellm_logging_obj: LiteLLMLoggingObj
@ -68,7 +68,7 @@ class GenerateContentHelper:
@staticmethod
def mock_generate_content_response(
mock_response: str = "This is a mock response from Google GenAI generate_content.",
) -> dict[str, Any]:
) -> dict[str, object]:
"""Mock response for generate_content for testing purposes"""
return {
"text": mock_response,
@ -239,9 +239,9 @@ async def agenerate_content(
tools: ToolConfigDict | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -307,9 +307,9 @@ def generate_content(
tools: ToolConfigDict | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -397,9 +397,9 @@ async def agenerate_content_stream(
tools: ToolConfigDict | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -492,9 +492,9 @@ def generate_content_stream(
tools: ToolConfigDict | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,

26
litellm/images/main.py
View file

@ -3,7 +3,7 @@ import contextvars
import importlib
from collections.abc import Coroutine
from functools import partial
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, cast, overload
from typing import TYPE_CHECKING, Final, Literal, Optional, cast, overload
if TYPE_CHECKING:
from litellm.images.utils import ImageEditRequestUtils
@ -151,7 +151,7 @@ def image_generation(
*,
aimg_generation: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, ImageResponse]:
) -> Coroutine[object, object, ImageResponse]:
...
@ -197,7 +197,7 @@ def image_generation(
api_version: str | None = None,
custom_llm_provider=None,
**kwargs,
) -> ImageResponse | Coroutine[Any, Any, ImageResponse]:
) -> ImageResponse | Coroutine[object, object, ImageResponse]:
"""
Maps the https://api.openai.com/v1/images/generations endpoint.
@ -386,6 +386,8 @@ def image_generation(
litellm.LlmProviders.VERTEX_AI,
litellm.LlmProviders.OPENROUTER,
litellm.LlmProviders.DASHSCOPE,
litellm.LlmProviders.QWENCLOUD,
litellm.LlmProviders.QWEN_AI_PLATFORM,
):
if image_generation_config is None:
raise ValueError(f"image generation config is not supported for {custom_llm_provider}")
@ -723,14 +725,14 @@ def image_edit(
user: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
**kwargs,
) -> ImageResponse | Coroutine[Any, Any, ImageResponse]:
) -> ImageResponse | Coroutine[object, object, ImageResponse]:
"""
Maps the image edit functionality, similar to OpenAI's images/edits endpoint.
"""
@ -769,7 +771,7 @@ def image_edit(
images: Final = image if isinstance(image, list) else ([image] if image is not None else [])
headers_from_kwargs: Final = kwargs.get("headers")
merged_extra_headers: Final[dict[str, Any]] = {}
merged_extra_headers: Final[dict[str, object]] = {}
if isinstance(headers_from_kwargs, dict):
merged_extra_headers.update(headers_from_kwargs)
if isinstance(extra_headers, dict):
@ -974,9 +976,9 @@ async def aimage_edit(
user: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -1044,7 +1046,7 @@ async def aimage_edit(
)
def __getattr__(name: str) -> Any:
def __getattr__(name: str) -> type["ImageEditRequestUtils"]:
"""Lazy import handler for images.main module"""
if name == "ImageEditRequestUtils":
# Lazy load ImageEditRequestUtils to avoid heavy import from images.utils at module load time

24
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -545,7 +545,6 @@ class SlackAlerting(CustomBatchLogger):
# Get the appropriate budget alert type handler
budget_alert_class: Final = get_budget_alert_type(type)
_id: Final = budget_alert_class.get_id(user_info)
user_info_json: Final = user_info.model_dump(exclude_none=True)
user_info_str: Final = self._get_user_info_str(user_info)
event_message = budget_alert_class.get_event_message()
@ -575,7 +574,22 @@ class SlackAlerting(CustomBatchLogger):
webhook_event = WebhookEvent(
event=event,
event_message=event_message,
**user_info_json,
spend=user_info.spend,
max_budget=user_info.max_budget,
soft_budget=user_info.soft_budget,
token=user_info.token,
customer_id=user_info.customer_id,
user_id=user_info.user_id,
team_id=user_info.team_id,
team_alias=user_info.team_alias,
organization_id=user_info.organization_id,
user_email=user_info.user_email,
key_alias=user_info.key_alias,
projected_exceeded_date=user_info.projected_exceeded_date,
projected_spend=user_info.projected_spend,
event_group=user_info.event_group,
alert_emails=user_info.alert_emails,
max_budget_alert_emails=user_info.max_budget_alert_emails,
)
await self.send_alert(
message=event_message + "\n\n" + user_info_str,
@ -657,7 +671,7 @@ class SlackAlerting(CustomBatchLogger):
"""
Create a standard message for a budget alert
"""
_all_fields_as_dict: Final = user_info.model_dump(exclude_none=True)
_all_fields_as_dict: Final[dict[str, object]] = user_info.model_dump(exclude_none=True)
_all_fields_as_dict.pop("token")
msg = ""
for k, v in _all_fields_as_dict.items():
@ -1006,7 +1020,7 @@ class SlackAlerting(CustomBatchLogger):
except Exception:
pass
async def model_added_alert(self, model_name: str, litellm_model_name: str, passed_model_info: Any):
async def model_added_alert(self, model_name: str, litellm_model_name: str, passed_model_info: object):
base_model_from_user: Final = getattr(passed_model_info, "base_model", None)
model_info = {}
base_model = ""
@ -1973,7 +1987,7 @@ Model Info:
try:
message = f"`{event_name}`\n"
key_event_dict: Final = key_event.model_dump()
key_event_dict: Final[dict[str, object]] = key_event.model_dump()
# Add Created by information first
message += "*Action Done by:*\n"

72
litellm/integrations/arize/arize_phoenix_prompt_manager.py
View file

@ -3,10 +3,12 @@ Arize Phoenix prompt manager that integrates with LiteLLM's prompt management sy
Fetches prompt versions from Arize Phoenix and provides workspace-based access control.
"""
from typing import Any, Final
from collections.abc import Mapping, Sequence
from typing import Any, Final, cast
from jinja2 import DictLoader, select_autoescape
from jinja2.sandbox import ImmutableSandboxedEnvironment
from typing_extensions import ReadOnly, TypedDict
from litellm.integrations.custom_prompt_management import CustomPromptManagement
from litellm.integrations.prompt_management_base import (
@ -20,6 +22,31 @@ from litellm.types.utils import StandardCallbackDynamicParams
from .arize_phoenix_client import ArizePhoenixClient
class ArizePhoenixContentPart(TypedDict, total=False):
type: ReadOnly[str]
text: ReadOnly[str]
class ArizePhoenixTemplateMessage(TypedDict, total=False):
role: ReadOnly[str]
content: ReadOnly[Sequence[ArizePhoenixContentPart]]
class ArizePhoenixTemplateBody(TypedDict, total=False):
messages: ReadOnly[Sequence[ArizePhoenixTemplateMessage]]
class ArizePhoenixPromptMetadata(TypedDict):
model_name: ReadOnly[str | None]
model_provider: ReadOnly[str | None]
description: ReadOnly[str]
template_type: ReadOnly[str | None]
template_format: ReadOnly[str]
invocation_parameters: ReadOnly[Mapping[str, Mapping[str, object]]]
temperature: ReadOnly[float | None]
max_tokens: ReadOnly[int | None]
class ArizePhoenixPromptTemplate:
"""
Represents a prompt template loaded from Arize Phoenix.
@ -28,10 +55,10 @@ class ArizePhoenixPromptTemplate:
def __init__(
self,
template_id: str,
messages: list[dict[str, Any]],
metadata: dict[str, Any],
messages: Sequence[ArizePhoenixTemplateMessage],
metadata: ArizePhoenixPromptMetadata,
model: str | None = None,
):
) -> None:
self.template_id = template_id
self.messages = messages
self.metadata = metadata
@ -43,7 +70,7 @@ class ArizePhoenixPromptTemplate:
self.description = metadata.get("description", "")
self.template_format = metadata.get("template_format", "MUSTACHE")
def __repr__(self):
def __repr__(self) -> str:
return f"ArizePhoenixPromptTemplate(id='{self.template_id}', model='{self.model}')"
@ -109,7 +136,7 @@ class ArizePhoenixTemplateManager:
def _parse_prompt_data(self, data: dict[str, Any], prompt_version_id: str) -> ArizePhoenixPromptTemplate:
"""Parse Arize Phoenix prompt data and extract messages and metadata."""
template_data: Final = data.get("template", {})
template_data: Final[ArizePhoenixTemplateBody] = data.get("template", {})
messages: Final = template_data.get("messages", [])
# Extract invocation parameters
@ -129,7 +156,7 @@ class ArizePhoenixTemplateManager:
break
# Build metadata dictionary
metadata: Final = {
metadata: Final[ArizePhoenixPromptMetadata] = {
"model_name": data.get("model_name"),
"model_provider": data.get("model_provider"),
"description": data.get("description", ""),
@ -146,7 +173,9 @@ class ArizePhoenixTemplateManager:
metadata=metadata,
)
def render_template(self, template_id: str, variables: dict[str, Any] | None = None) -> list[AllMessageValues]:
def render_template(
self, template_id: str, variables: Mapping[str, object] | None = None
) -> list[AllMessageValues]:
"""Render a template with the given variables and return formatted messages."""
if template_id not in self.prompts:
raise ValueError(f"Template '{template_id}' not found")
@ -174,7 +203,9 @@ class ArizePhoenixTemplateManager:
# Combine rendered content
final_content = " ".join(rendered_content_parts)
rendered_messages.append({"role": role, "content": final_content})
rendered_messages.append(
cast("AllMessageValues", {"role": role, "content": final_content}) # cast-ok: Phoenix roles are OpenAI
)
return rendered_messages
@ -243,8 +274,8 @@ class ArizePhoenixPromptManager(CustomPromptManagement):
def get_prompt_template(
self,
prompt_id: str,
prompt_variables: dict[str, Any] | None = None,
) -> tuple[list[AllMessageValues], dict[str, Any]]:
prompt_variables: Mapping[str, object] | None = None,
) -> tuple[list[AllMessageValues], dict[str, object]]:
"""
Get a prompt template and render it with variables.
@ -263,7 +294,7 @@ class ArizePhoenixPromptManager(CustomPromptManagement):
rendered_messages: Final = self.prompt_manager.render_template(prompt_id, prompt_variables or {})
# Extract metadata
metadata: Final = {
metadata: Final[dict[str, object]] = {
"model": template.model,
"temperature": template.temperature,
"max_tokens": template.max_tokens,
@ -271,7 +302,7 @@ class ArizePhoenixPromptManager(CustomPromptManagement):
# Add additional invocation parameters
invocation_params: Final = template.invocation_parameters
provider_params = {}
provider_params: Mapping[str, object] = {}
if "openai" in invocation_params:
provider_params = invocation_params["openai"]
@ -289,12 +320,12 @@ class ArizePhoenixPromptManager(CustomPromptManagement):
self,
user_id: str | None,
messages: list[AllMessageValues],
function_call: dict[str, Any] | str | None = None,
litellm_params: dict[str, Any] | None = None,
function_call: dict[str, object] | str | None = None,
litellm_params: dict[str, object] | None = None,
prompt_id: str | None = None,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: dict[str, object] | None = None,
**kwargs,
) -> tuple[list[AllMessageValues], dict[str, Any] | None]:
) -> tuple[list[AllMessageValues], dict[str, object] | None]:
"""
Pre-call hook that processes the prompt template before making the LLM call.
"""
@ -335,9 +366,9 @@ class ArizePhoenixPromptManager(CustomPromptManagement):
except Exception as e:
# Log error but don't fail the call
import litellm
from litellm._logging import verbose_proxy_logger
litellm._logging.verbose_proxy_logger.error("Error in Arize Phoenix prompt pre_call_hook: %s", e)
verbose_proxy_logger.error("Error in Arize Phoenix prompt pre_call_hook: %s", e)
return messages, litellm_params
def get_available_prompts(self) -> list[str]:
@ -393,7 +424,8 @@ class ArizePhoenixPromptManager(CustomPromptManagement):
rendered_messages, prompt_metadata = self.get_prompt_template(prompt_id, prompt_variables)
# Extract model from metadata (if specified)
template_model: Final = prompt_metadata.get("model")
raw_template_model: Final = prompt_metadata.get("model")
template_model: Final = raw_template_model if isinstance(raw_template_model, str) else None
# Extract optional parameters from metadata
optional_params: Final = {}

37
litellm/integrations/bitbucket/bitbucket_prompt_manager.py
View file

@ -3,6 +3,7 @@ BitBucket prompt manager that integrates with LiteLLM's prompt management system
Fetches .prompt files from BitBucket repositories and provides team-based access control.
"""
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final
from jinja2 import DictLoader, select_autoescape
@ -65,7 +66,7 @@ class BitBucketTemplateManager:
def __init__(
self,
bitbucket_config: dict[str, Any],
bitbucket_config: Mapping[str, object],
prompt_id: str | None = None,
):
self.bitbucket_config = bitbucket_config
@ -123,7 +124,7 @@ class BitBucketTemplateManager:
template_content = content
# Parse YAML frontmatter
metadata: dict[str, Any] = {}
metadata: dict[str, object] = {}
if frontmatter_str:
try:
import yaml
@ -141,9 +142,9 @@ class BitBucketTemplateManager:
metadata=metadata,
)
def _parse_yaml_basic(self, yaml_str: str) -> dict[str, Any]:
def _parse_yaml_basic(self, yaml_str: str) -> dict[str, object]:
"""Basic YAML parser for simple cases when PyYAML is not available."""
result: Final[dict[str, Any]] = {}
result: Final[dict[str, object]] = {}
for line in yaml_str.split("\n"):
line = line.strip()
if ":" in line and not line.startswith("#"):
@ -162,7 +163,7 @@ class BitBucketTemplateManager:
result[key] = value.strip("\"'")
return result
def render_template(self, template_id: str, variables: dict[str, Any] | None = None) -> str:
def render_template(self, template_id: str, variables: Mapping[str, object] | None = None) -> str:
"""Render a template with the given variables."""
if template_id not in self.prompts:
raise ValueError(f"Template '{template_id}' not found")
@ -209,7 +210,7 @@ class BitBucketPromptManager(CustomPromptManagement):
def __init__(
self,
bitbucket_config: dict[str, Any],
bitbucket_config: Mapping[str, object],
prompt_id: str | None = None,
):
self.bitbucket_config = bitbucket_config
@ -234,7 +235,7 @@ class BitBucketPromptManager(CustomPromptManagement):
def get_prompt_template(
self,
prompt_id: str,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
) -> tuple[str, dict[str, Any]]:
"""
Get a prompt template and render it with variables.
@ -267,12 +268,12 @@ class BitBucketPromptManager(CustomPromptManagement):
self,
user_id: str | None,
messages: list[AllMessageValues],
function_call: dict[str, Any] | str | None = None,
litellm_params: dict[str, Any] | None = None,
function_call: Mapping[str, object] | str | None = None,
litellm_params: dict[str, object] | None = None,
prompt_id: str | None = None,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
**kwargs,
) -> tuple[list[AllMessageValues], dict[str, Any] | None]:
) -> tuple[list[AllMessageValues], dict[str, object] | None]:
"""
Pre-call hook that processes the prompt template before making the LLM call.
"""
@ -316,9 +317,9 @@ class BitBucketPromptManager(CustomPromptManagement):
except Exception as e:
# Log error but don't fail the call
import litellm
from litellm._logging import verbose_proxy_logger
litellm._logging.verbose_proxy_logger.error("Error in BitBucket prompt pre_call_hook: %s", e)
verbose_proxy_logger.error("Error in BitBucket prompt pre_call_hook: %s", e)
return messages, litellm_params
def _parse_prompt_to_messages(self, prompt_content: str) -> list[AllMessageValues]:
@ -384,14 +385,14 @@ class BitBucketPromptManager(CustomPromptManagement):
def post_call_hook(
self,
user_id: str | None,
response: Any,
response: object,
input_messages: list[AllMessageValues],
function_call: dict[str, Any] | str | None = None,
litellm_params: dict[str, Any] | None = None,
function_call: Mapping[str, object] | str | None = None,
litellm_params: Mapping[str, object] | None = None,
prompt_id: str | None = None,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
**kwargs,
) -> Any:
) -> object:
"""
Post-call hook for any post-processing after the LLM call.
"""

27
litellm/integrations/cloudzero/transform.py
View file

@ -19,14 +19,29 @@
"""Transform LiteLLM data to CloudZero AnyCost CBF format."""
from datetime import datetime
from typing import Any, Final
from typing import Final, SupportsFloat, SupportsIndex, SupportsInt
import polars as pl
from typing_extensions import Buffer
from ...types.integrations.cloudzero import CBFRecord
from .cz_resource_names import CZEntityType, CZRNGenerator
def _as_int(value: object) -> int:
"""The integer form of a spend table cell, computed the way :func:`int` computes it."""
if isinstance(value, (str, Buffer, SupportsInt, SupportsIndex)):
return int(value)
raise TypeError(f"int() argument must be a string or a number, not {type(value).__name__!r}")
def _as_float(value: object) -> float:
"""The floating point form of a spend table cell, computed the way :func:`float` computes it."""
if isinstance(value, (str, Buffer, SupportsFloat, SupportsIndex)):
return float(value)
raise TypeError(f"float() argument must be a string or a number, not {type(value).__name__!r}")
class CBFTransformer:
"""Transform LiteLLM usage data to CloudZero Billing Format (CBF)."""
@ -82,15 +97,15 @@ class CBFTransformer:
return pl.DataFrame(cbf_data)
def _create_cbf_record(self, row: dict[str, Any]) -> CBFRecord:
def _create_cbf_record(self, row: dict[str, object]) -> CBFRecord:
"""Create a single CBF record from LiteLLM daily spend row."""
# Parse date (daily spend tables use date strings like '2025-04-19')
usage_date: Final = self._parse_date(row.get("date"))
# Calculate total tokens
prompt_tokens: Final = int(row.get("prompt_tokens", 0))
completion_tokens: Final = int(row.get("completion_tokens", 0))
prompt_tokens: Final = _as_int(row.get("prompt_tokens", 0))
completion_tokens: Final = _as_int(row.get("completion_tokens", 0))
total_tokens: Final = prompt_tokens + completion_tokens
# Create CloudZero Resource Name (CZRN) as resource_id
@ -154,7 +169,7 @@ class CBFTransformer:
"time/usage_start": (
usage_date.isoformat() if usage_date else None
), # Required: ISO-formatted UTC datetime
"cost/cost": float(row.get("spend", 0.0)), # Required: billed cost
"cost/cost": _as_float(row.get("spend", 0.0)), # Required: billed cost
"resource/id": resource_id, # CZRN (CloudZero Resource Name)
# Usage metrics for token consumption
"usage/amount": total_tokens, # Numeric value of tokens consumed
@ -187,7 +202,7 @@ class CBFTransformer:
return CBFRecord(cbf_record)
def _parse_date(self, date_str) -> datetime | None:
def _parse_date(self, date_str: object) -> datetime | None:
"""Parse date string from daily spend tables (e.g., '2025-04-19')."""
if date_str is None:
return None

13
litellm/integrations/custom_guardrail.py
View file

@ -2,6 +2,7 @@ import contextvars
import hashlib
import os
import secrets
from collections.abc import Mapping
from datetime import datetime
from typing import TYPE_CHECKING, Any, ClassVar, Final, Literal, Optional, get_args
@ -227,13 +228,13 @@ class CustomGuardrail(CustomLogger):
)
super().__init__(**kwargs)
def render_violation_message(self, default: str, context: dict[str, Any] | None = None) -> str:
def render_violation_message(self, default: str, context: Mapping[str, object] | None = None) -> str:
"""Return a custom violation message if template is configured."""
if not self.violation_message_template:
return default
format_context: Final[dict[str, Any]] = {"default_message": default}
format_context: Final[dict[str, object]] = {"default_message": default}
if context:
format_context.update(context)
try:
@ -661,7 +662,7 @@ class CustomGuardrail(CustomLogger):
value: Final = self._get_admin_metadata(data).get("opted_out_global_guardrails")
return value if isinstance(value, list) else []
def _is_valid_response_type(self, result: Any) -> bool:
def _is_valid_response_type(self, result: object) -> bool:
"""
Check if result is a valid LLMResponseTypes instance.
@ -722,7 +723,7 @@ class CustomGuardrail(CustomLogger):
return None
return f"{_PRE_CALL_EXECUTED_TOKEN}:{name}"
def mark_pre_call_hook_ran(self, data: dict[str, Any]) -> None:
def mark_pre_call_hook_ran(self, data: dict[str, object]) -> None:
"""
Record that this guardrail's ``async_pre_call_hook`` already ran for this
request, so the deployment-level hook does not run it a second time.
@ -747,7 +748,7 @@ class CustomGuardrail(CustomLogger):
return
data["metadata"] = {PRE_CALL_EXECUTED_GUARDRAILS_KEY: [marker]}
def _pre_call_hook_already_ran(self, data: dict[str, Any]) -> bool:
def _pre_call_hook_already_ran(self, data: dict[str, object]) -> bool:
marker: Final = self._pre_call_marker()
if marker is None:
return False
@ -1170,7 +1171,7 @@ class CustomGuardrail(CustomLogger):
This gets logged on downsteam Langfuse, DataDog, etc.
"""
# Convert None to empty dict to satisfy type requirements
guardrail_response: dict[str, Any] | str = {} if response is None else response
guardrail_response: dict[str, object] | str = {} if response is None else response
# For apply_guardrail functions in custom_code_guardrail scenario,
# simplify the logged response to "allow", "deny", or "mask"

47
litellm/integrations/custom_logger.py
View file

@ -31,6 +31,9 @@ if TYPE_CHECKING:
from litellm.caching.caching import DualCache
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.anthropic_messages.transformation import (
BaseAnthropicMessagesConfig,
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.mcp import (
MCPPostCallResponseObject,
@ -39,7 +42,7 @@ if TYPE_CHECKING:
)
from litellm.types.router import PreRoutingHookResponse
Span = _Span | Any
Span = _Span
else:
Span = Any
LiteLLMLoggingObj = Any
@ -268,7 +271,9 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
) -> list[dict]:
return healthy_deployments
async def async_pre_call_deployment_hook(self, kwargs: dict[str, Any], call_type: CallTypes | None) -> dict | None:
async def async_pre_call_deployment_hook(
self, kwargs: dict[str, object], call_type: CallTypes | None
) -> dict | None:
"""
Allow modifying the request just before it's sent to the deployment.
@ -344,9 +349,9 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
async def async_post_call_streaming_deployment_hook(
self,
request_data: dict,
response_chunk: Any,
response_chunk: object,
call_type: CallTypes | None,
) -> Any | None:
) -> object | None:
"""
Allow modifying streaming chunks just before they're returned to the user.
@ -378,7 +383,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
"""
def translate_completion_output_params_streaming(
self, completion_stream: Any
self, completion_stream: object
) -> AdapterCompletionStreamWrapper | None:
"""
Translates the streaming chunk, from the OpenAI format to the custom format.
@ -418,9 +423,9 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
response: Any,
response: object,
request_headers: dict[str, str] | None = None,
litellm_call_info: dict[str, Any] | None = None,
litellm_call_info: dict[str, object] | None = None,
) -> dict[str, str] | None:
"""
Called after an LLM API call (success or failure) to allow injecting custom HTTP response headers.
@ -471,11 +476,11 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
) -> Any:
pass
async def async_logging_hook(self, kwargs: dict, result: Any, call_type: str) -> tuple[dict, Any]:
async def async_logging_hook(self, kwargs: dict, result: object, call_type: str) -> tuple[dict, object]:
"""For masking logged request/response. Return a modified version of the request/result."""
return kwargs, result
def logging_hook(self, kwargs: dict, result: Any, call_type: str) -> tuple[dict, Any]:
def logging_hook(self, kwargs: dict, result: object, call_type: str) -> tuple[dict, object]:
"""For masking logged request/response. Return a modified version of the request/result."""
return kwargs, result
@ -581,7 +586,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
async def async_should_run_agentic_loop(
self,
response: Any,
response: object,
model: str,
messages: list[dict],
tools: list[dict] | None,
@ -642,8 +647,8 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
tools: dict,
model: str,
messages: list[dict],
response: Any,
anthropic_messages_provider_config: Any,
response: object,
anthropic_messages_provider_config: "BaseAnthropicMessagesConfig | None",
anthropic_messages_optional_request_params: dict,
logging_obj: "LiteLLMLoggingObj",
stream: bool,
@ -711,8 +716,8 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
tools: dict,
model: str,
messages: list[dict],
response: Any,
anthropic_messages_provider_config: Any,
response: object,
anthropic_messages_provider_config: "BaseAnthropicMessagesConfig | None",
anthropic_messages_optional_request_params: dict,
logging_obj: "LiteLLMLoggingObj",
stream: bool,
@ -728,7 +733,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
async def async_post_agentic_loop_response_hook(
self,
response: Any,
response: object,
plan: AgenticLoopPlan,
kwargs: dict,
) -> Any:
@ -767,7 +772,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
async def async_should_run_chat_completion_agentic_loop(
self,
response: Any,
response: object,
model: str,
messages: list[dict],
tools: list[dict] | None,
@ -785,12 +790,12 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
tools: dict,
model: str,
messages: list[dict],
response: Any,
response: object,
optional_params: dict,
logging_obj: "LiteLLMLoggingObj",
stream: bool,
kwargs: dict,
) -> Any:
) -> object:
"""
Hook to execute chat completion agentic loop based on context from should_run hook.
"""
@ -800,7 +805,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
tools: dict,
model: str,
messages: list[dict],
response: Any,
response: object,
optional_params: dict,
logging_obj: "LiteLLMLoggingObj",
stream: bool,
@ -1056,7 +1061,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
def _redact_base64(
self,
value: Any,
value: object,
depth: int = 0,
max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER,
) -> object:
@ -1079,7 +1084,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
return value
def _should_keep_content(self, content: Any) -> bool:
def _should_keep_content(self, content: object) -> bool:
"""Return True if this content item should be retained."""
if not isinstance(content, dict):
return True

44
litellm/integrations/datadog/datadog.py
View file

@ -20,10 +20,11 @@ import time
import traceback
from collections.abc import Sequence
from datetime import datetime as datetimeObj
from typing import Any, Final
from typing import TYPE_CHECKING, Any, Final
import httpx
from httpx import Response
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
@ -62,6 +63,18 @@ from litellm.types.utils import StandardLoggingPayload
from ..additional_logging_utils import AdditionalLoggingUtils
if TYPE_CHECKING:
from fastapi import HTTPException
from litellm.proxy._types import UserAPIKeyAuth
class _DatadogLoggingKwargs(TypedDict, total=False):
"""The subset of logging ``kwargs`` that the Datadog payload builder reads."""
standard_logging_object: ReadOnly[StandardLoggingPayload | None]
# max number of logs DD API can accept
@ -87,6 +100,11 @@ def _resolve_dd_batch_size() -> int:
return max(1, min(value, DD_MAX_BATCH_SIZE))
def _span_attribute(span: object, name: str) -> object:
"""Read an optional attribute off whatever span object the active tracer hands back."""
return getattr(span, name, None)
class DataDogLogger(
CustomBatchLogger,
AdditionalLoggingUtils,
@ -271,9 +289,9 @@ class DataDogLogger(
self,
request_data: dict,
original_exception: Exception,
user_api_key_dict: Any,
user_api_key_dict: "UserAPIKeyAuth",
traceback_str: str | None = None,
) -> Any | None:
) -> "HTTPException | None":
"""
Log proxy-level failures (e.g. 401 auth, DB connection errors) to Datadog.
@ -297,7 +315,7 @@ class DataDogLogger(
status_code = int(_code)
# Use project-standard sanitized user context when running in proxy
user_context: dict[str, Any] = {}
user_context: dict[str, object] = {}
try:
from litellm.proxy.litellm_pre_call_utils import (
LiteLLMProxyRequestSetup,
@ -553,8 +571,8 @@ class DataDogLogger(
def create_datadog_logging_payload(
self,
kwargs: dict | Any,
response_obj: Any,
kwargs: _DatadogLoggingKwargs,
response_obj: object,
start_time: datetime.datetime,
end_time: datetime.datetime,
) -> DatadogPayload:
@ -562,8 +580,8 @@ class DataDogLogger(
Helper function to create a datadog payload for logging
Args:
kwargs (Union[dict, Any]): request kwargs
response_obj (Any): llm api response
kwargs: request kwargs, read for its standard logging object
response_obj: llm api response
start_time (datetime.datetime): start time of request
end_time (datetime.datetime): end time of request
@ -625,7 +643,7 @@ class DataDogLogger(
self,
payload: ServiceLoggerPayload,
error: str | None = "",
parent_otel_span: Any | None = None,
parent_otel_span: object = None,
start_time: datetimeObj | float | None = None,
end_time: float | datetimeObj | None = None,
event_metadata: dict | None = None,
@ -659,7 +677,7 @@ class DataDogLogger(
self,
payload: ServiceLoggerPayload,
error: str | None = "",
parent_otel_span: Any | None = None,
parent_otel_span: object = None,
start_time: datetimeObj | float | None = None,
end_time: float | datetimeObj | None = None,
event_metadata: dict | None = None,
@ -696,7 +714,7 @@ class DataDogLogger(
def _create_v0_logging_payload(
self,
kwargs: dict | Any,
kwargs: dict,
response_obj: Any,
start_time: datetime.datetime,
end_time: datetime.datetime,
@ -810,11 +828,11 @@ class DataDogLogger(
if current_span is None:
return None
trace_id: Final = getattr(current_span, "trace_id", None)
trace_id: Final = _span_attribute(current_span, "trace_id")
if trace_id is None:
return None
span_id: Final = getattr(current_span, "span_id", None)
span_id: Final = _span_attribute(current_span, "span_id")
trace_context: Final[dict[str, str]] = {"trace_id": str(trace_id)}
if span_id is not None:
trace_context["span_id"] = str(span_id)

19
litellm/integrations/datadog/datadog_llm_obs.py
View file

@ -9,6 +9,7 @@ API Reference: https://docs.datadoghq.com/llm_observability/setup/api/?tab=examp
import asyncio
import json
import os
from collections.abc import Mapping, Sequence
from datetime import datetime
from typing import Any, Final, Literal
@ -334,7 +335,7 @@ class DataDogLLMObsLogger(CustomBatchLogger):
def _get_response_messages(
self, standard_logging_payload: StandardLoggingPayload, call_type: str | None
) -> list[Any]:
) -> list[object]:
"""
Get the messages from the response object
@ -484,7 +485,7 @@ class DataDogLLMObsLogger(CustomBatchLogger):
# Default fallback for unknown or passthrough operations
return "llm"
def _ensure_string_content(self, messages: str | list[Any] | dict[Any, Any] | None) -> list[Any]:
def _ensure_string_content(self, messages: str | Sequence[object] | Mapping[object, object] | None) -> list[object]:
if messages is None:
return []
if isinstance(messages, str):
@ -495,11 +496,11 @@ class DataDogLLMObsLogger(CustomBatchLogger):
return [str(messages.get("content", ""))]
return []
def _get_dd_llm_obs_payload_metadata(self, standard_logging_payload: StandardLoggingPayload) -> dict[str, Any]:
def _get_dd_llm_obs_payload_metadata(self, standard_logging_payload: StandardLoggingPayload) -> dict[str, object]:
"""
Fields to track in DD LLM Observability metadata from litellm standard logging payload
"""
_metadata: Final[dict[str, Any]] = {
_metadata: Final[dict[str, object]] = {
"model_name": standard_logging_payload.get("model", "unknown"),
"model_provider": standard_logging_payload.get("custom_llm_provider", "unknown"),
"id": standard_logging_payload.get("id", "unknown"),
@ -647,7 +648,7 @@ class DataDogLLMObsLogger(CustomBatchLogger):
return spend_metrics
def _process_input_messages_preserving_tool_calls(self, messages: list[Any]) -> list[dict[str, Any]]:
def _process_input_messages_preserving_tool_calls(self, messages: Sequence[object]) -> list[dict[str, object]]:
"""
Process input messages while preserving tool_calls and tool message types.
@ -671,13 +672,13 @@ class DataDogLLMObsLogger(CustomBatchLogger):
return processed
@staticmethod
def _tool_calls_kv_pair(tool_calls: list[dict[str, Any]]) -> dict[str, Any]:
def _tool_calls_kv_pair(tool_calls: list[dict[str, Any]]) -> dict[str, object]:
"""
Extract tool call information into key-value pairs for Datadog metadata.
Similar to OpenTelemetry's implementation but adapted for Datadog's format.
"""
kv_pairs: Final[dict[str, Any]] = {}
kv_pairs: Final[dict[str, object]] = {}
for idx, tool_call in enumerate(tool_calls):
try:
# Extract tool call ID
@ -712,11 +713,11 @@ class DataDogLLMObsLogger(CustomBatchLogger):
return kv_pairs
def _extract_tool_call_metadata(self, standard_logging_payload: StandardLoggingPayload) -> dict[str, Any]:
def _extract_tool_call_metadata(self, standard_logging_payload: StandardLoggingPayload) -> dict[str, object]:
"""
Extract tool call information from both input messages and response for Datadog metadata.
"""
tool_call_metadata: Final[dict[str, Any]] = {}
tool_call_metadata: Final[dict[str, object]] = {}
try:
# Extract tool calls from input messages

25
litellm/integrations/dotprompt/prompt_manager.py
View file

@ -3,12 +3,21 @@ Based on Google's GenAI Kit dotprompt implementation: https://google.github.io/d
"""
import re
from collections.abc import Mapping
from pathlib import Path
from typing import Any, Final
import yaml
from jinja2 import DictLoader, select_autoescape
from jinja2.sandbox import ImmutableSandboxedEnvironment
from typing_extensions import NotRequired, ReadOnly, TypedDict
class _PromptFileJson(TypedDict):
"""JSON form of a .prompt file: rendered template text plus its frontmatter."""
content: ReadOnly[NotRequired[str]]
metadata: ReadOnly[NotRequired[dict[str, object]]]
def strip_version_suffix(prompt_id: str) -> str | None:
@ -167,7 +176,7 @@ class PromptManager:
template_id=prompt_id,
)
def _parse_frontmatter(self, content: str) -> tuple[dict[str, Any], str]:
def _parse_frontmatter(self, content: str) -> tuple[dict[str, object], str]:
"""Parse YAML frontmatter from prompt content."""
# Match YAML frontmatter between --- delimiters
frontmatter_pattern: Final = r"^---\s*\n(.*?)\n---\s*\n(.*)$"
@ -178,7 +187,7 @@ class PromptManager:
template_content = match.group(2)
try:
frontmatter = yaml.safe_load(frontmatter_yaml) or {}
frontmatter: dict[str, object] = yaml.safe_load(frontmatter_yaml) or {}
except yaml.YAMLError as e:
raise ValueError(f"Invalid YAML frontmatter: {e}")
else:
@ -191,7 +200,7 @@ class PromptManager:
def render(
self,
prompt_id: str,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
version: int | None = None,
) -> str:
"""
@ -231,7 +240,7 @@ class PromptManager:
except Exception as e:
raise ValueError(f"Error rendering template '{prompt_id}': {e}")
def _validate_input(self, variables: dict[str, Any], schema: dict[str, Any]) -> None:
def _validate_input(self, variables: Mapping[str, object], schema: Mapping[str, str]) -> None:
"""Basic validation of input variables against schema."""
for field_name, field_type in schema.items():
if field_name in variables:
@ -291,7 +300,7 @@ class PromptManager:
"""Get a list of all available prompt IDs."""
return list(self.prompts.keys())
def get_prompt_metadata(self, prompt_id: str) -> dict[str, Any] | None:
def get_prompt_metadata(self, prompt_id: str) -> dict[str, object] | None:
"""Get metadata for a specific prompt."""
template: Final = self.prompts.get(prompt_id)
return template.metadata if template else None
@ -302,12 +311,12 @@ class PromptManager:
if self.prompt_directory:
self._load_prompts()
def add_prompt(self, prompt_id: str, content: str, metadata: dict[str, Any] | None = None) -> None:
def add_prompt(self, prompt_id: str, content: str, metadata: dict[str, object] | None = None) -> None:
"""Add a prompt template programmatically."""
template: Final = PromptTemplate(content=content, metadata=metadata or {}, template_id=prompt_id)
self.prompts[prompt_id] = template
def prompt_file_to_json(self, file_path: str | Path) -> dict[str, Any]:
def prompt_file_to_json(self, file_path: str | Path) -> _PromptFileJson:
"""Convert a .prompt file to JSON format.
Args:
@ -324,7 +333,7 @@ class PromptManager:
return {"content": template_content.strip(), "metadata": frontmatter}
def json_to_prompt_file(self, prompt_data: dict[str, Any]) -> str:
def json_to_prompt_file(self, prompt_data: _PromptFileJson) -> str:
"""Convert JSON prompt data to .prompt file format.
Args:

35
litellm/integrations/galileo.py
View file

@ -6,10 +6,11 @@ import re
import uuid
from collections.abc import Mapping, Sequence
from datetime import datetime, timezone, tzinfo
from typing import Any, Final, TypedDict, cast
from typing import Any, Final, Protocol, cast
import httpx
from pydantic import BaseModel, Field
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
@ -35,6 +36,34 @@ GALILEO_CLOUD_API_BASE_URL: Final = "https://api.galileo.ai"
GALILEO_MAX_IN_MEMORY_RECORDS: Final = 1000
class _GalileoLoginBody(TypedDict):
"""Decoded body of the Galileo login response."""
access_token: ReadOnly[str]
class _GalileoLoginResponse(Protocol):
"""The login call's HTTP response, read for the access token it carries."""
def json(self) -> _GalileoLoginBody: ...
class _JsonResponse(Protocol):
"""An HTTP response read only for whatever JSON body it decodes to."""
def json(self) -> object: ...
def _login_access_token(response: _GalileoLoginResponse) -> str:
"""Read the bearer token out of a Galileo login response body."""
return response.json()["access_token"]
def _decoded_body(response: _JsonResponse) -> object:
"""Decode a response body without asserting anything about its shape."""
return response.json()
class GalileoStandardLoggingFields(TypedDict, total=False):
call_type: str
model: str
@ -156,7 +185,7 @@ class GalileoObserve(CustomLogger):
},
)
galileo_login_response.raise_for_status()
access_token: Final = galileo_login_response.json()["access_token"]
access_token: Final = _login_access_token(galileo_login_response)
self.headers = {
"accept": "application/json",
"Content-Type": "application/json",
@ -421,7 +450,7 @@ class GalileoObserve(CustomLogger):
try:
verbose_logger.debug(
"Galileo Logger HTTP error response json: %s",
response.json(),
_decoded_body(response),
)
except Exception:
pass

88
litellm/integrations/gitlab/gitlab_client.py
View file

@ -4,12 +4,80 @@ Now supports selecting a tag via `config["tag"]`; falls back to branch ("main").
"""
import base64
from typing import Any, Final
from collections.abc import Mapping, Sequence
from typing import Any, Final, Protocol, TypedDict
from urllib.parse import quote
from typing_extensions import ReadOnly
from litellm.llms.custom_httpx.http_handler import HTTPHandler
class GitLabFilePayload(TypedDict, total=False):
"""A repository-files API entry."""
content: ReadOnly[str]
encoding: ReadOnly[str]
class GitLabTreeEntry(TypedDict, total=False):
"""A repository-tree API entry."""
path: ReadOnly[str]
type: ReadOnly[str]
class GitLabBranch(TypedDict, total=False):
"""A repository-branches API entry."""
name: ReadOnly[str]
type: ReadOnly[str]
class GitLabFileMetadata(TypedDict):
"""The response headers a raw file request exposes as metadata."""
content_type: ReadOnly[str | None]
content_length: ReadOnly[str | None]
last_modified: ReadOnly[str | None]
class _FileJsonResponse(Protocol):
def json(self) -> GitLabFilePayload: ...
class _TreeJsonResponse(Protocol):
def json(self) -> Sequence[GitLabTreeEntry] | None: ...
class _ProjectJsonResponse(Protocol):
def json(self) -> Mapping[str, object]: ...
class _BranchesJsonResponse(Protocol):
def json(self) -> Sequence[GitLabBranch] | None: ...
def _file_payload(resp: _FileJsonResponse) -> GitLabFilePayload:
"""The JSON body of a repository-files response."""
return resp.json()
def _tree_entries(resp: _TreeJsonResponse) -> Sequence[GitLabTreeEntry]:
"""The entries of a repository-tree response."""
return resp.json() or []
def _project_info(resp: _ProjectJsonResponse) -> Mapping[str, object]:
"""The JSON body of a project response."""
return resp.json()
def _branch_entries(resp: _BranchesJsonResponse) -> Sequence[GitLabBranch] | None:
"""The JSON body of a repository-branches response."""
return resp.json()
class GitLabClient:
"""
Client for interacting with the GitLab API to fetch files.
@ -42,12 +110,12 @@ class GitLabClient:
self.project: str | int = project
self.access_token: str = str(access_token)
self.auth_method = config.get("auth_method", "token") # 'token' or 'oauth'
self.auth_method: str = config.get("auth_method", "token") # 'token' or 'oauth'
self.branch = config.get("branch", None)
if not self.branch:
self.branch = "main"
self.tag = config.get("tag")
self.base_url = config.get("base_url", "https://gitlab.com/api/v4")
self.base_url: str = config.get("base_url", "https://gitlab.com/api/v4")
if not all([self.project, self.access_token]):
raise ValueError("project and access_token are required")
@ -159,7 +227,7 @@ class GitLabClient:
if resp.status_code == 404:
return None
resp.raise_for_status()
data: Final = resp.json()
data: Final = _file_payload(resp)
content: Final = data.get("content")
encoding: Final = data.get("encoding", "")
if content and encoding == "base64":
@ -208,7 +276,7 @@ class GitLabClient:
return []
resp.raise_for_status()
data: Final = resp.json() or []
data: Final = _tree_entries(resp)
files: Final[list[str]] = []
for item in data:
if item.get("type") == "blob":
@ -229,13 +297,13 @@ class GitLabClient:
raise Exception("Authentication failed. Check your GitLab token and auth_method.")
raise Exception(f"Failed to list files in '{directory_path}': {e}")
def get_repository_info(self) -> dict[str, Any]:
def get_repository_info(self) -> Mapping[str, object]:
"""Get information about the project/repository."""
url: Final = f"{self.base_url}/projects/{self._project_enc}"
try:
resp: Final = self.http_handler.get(url, headers=self.headers)
resp.raise_for_status()
return resp.json()
return _project_info(resp)
except Exception as e:
raise Exception(f"Failed to get repository info: {e}")
@ -247,18 +315,18 @@ class GitLabClient:
except Exception:
return False
def get_branches(self) -> list[dict[str, Any]]:
def get_branches(self) -> list[GitLabBranch]:
"""Get list of branches in the repository."""
url: Final = f"{self.base_url}/projects/{self._project_enc}/repository/branches"
try:
resp: Final = self.http_handler.get(url, headers=self.headers)
resp.raise_for_status()
data: Final = resp.json()
data: Final = _branch_entries(resp)
return data if isinstance(data, list) else []
except Exception as e:
raise Exception(f"Failed to get branches: {e}")
def get_file_metadata(self, file_path: str, *, ref: str | None = None) -> dict[str, Any] | None:
def get_file_metadata(self, file_path: str, *, ref: str | None = None) -> GitLabFileMetadata | None:
"""
Get minimal metadata about a file via RAW endpoint headers at a given ref.

59
litellm/integrations/gitlab/gitlab_prompt_manager.py
View file

@ -2,10 +2,12 @@
GitLab prompt manager with configurable prompts folder.
"""
from typing import TYPE_CHECKING, Any, Final
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final, TypeVar
from jinja2 import DictLoader, select_autoescape
from jinja2.sandbox import ImmutableSandboxedEnvironment
from typing_extensions import ReadOnly, TypedDict
from litellm.integrations.custom_prompt_management import CustomPromptManagement
@ -24,6 +26,19 @@ from litellm.types.utils import StandardCallbackDynamicParams
GITLAB_PREFIX: Final = "gitlab::"
_ResponseT = TypeVar("_ResponseT")
class GitLabCachedPrompt(TypedDict):
id: ReadOnly[str]
path: ReadOnly[str]
content: ReadOnly[str]
metadata: ReadOnly[Mapping[str, object]]
model: ReadOnly[str | None]
temperature: ReadOnly[float | None]
max_tokens: ReadOnly[int | None]
optional_params: ReadOnly[Mapping[str, object]]
def encode_prompt_id(raw_id: str) -> str:
"""Convert GitLab path IDs like 'invoice/extract' → 'gitlab::invoice::extract'"""
@ -206,7 +221,7 @@ class GitLabTemplateManager:
result[key] = value.strip("\"'")
return result
def render_template(self, template_id: str, variables: dict[str, Any] | None = None) -> str:
def render_template(self, template_id: str, variables: Mapping[str, object] | None = None) -> str:
if template_id not in self.prompts:
raise ValueError(f"Template '{template_id}' not found")
template: Final = self.prompts[template_id]
@ -313,7 +328,7 @@ class GitLabPromptManager(CustomPromptManagement):
def get_prompt_template(
self,
prompt_id: str,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
*,
ref: str | None = None,
) -> tuple[str, dict[str, Any]]:
@ -338,13 +353,13 @@ class GitLabPromptManager(CustomPromptManagement):
self,
user_id: str | None,
messages: list[AllMessageValues],
function_call: dict[str, Any] | str | None = None,
litellm_params: dict[str, Any] | None = None,
function_call: Mapping[str, object] | str | None = None,
litellm_params: dict[str, object] | None = None,
prompt_id: str | None = None,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
prompt_version: str | None = None,
**kwargs,
) -> tuple[list[AllMessageValues], dict[str, Any] | None]:
) -> tuple[list[AllMessageValues], dict[str, object] | None]:
if not prompt_id:
return messages, litellm_params
try:
@ -377,9 +392,9 @@ class GitLabPromptManager(CustomPromptManagement):
return final_messages, litellm_params
except Exception as e:
import litellm
from litellm._logging import verbose_proxy_logger
litellm._logging.verbose_proxy_logger.error("Error in GitLab prompt pre_call_hook: %s", e)
verbose_proxy_logger.error("Error in GitLab prompt pre_call_hook: %s", e)
return messages, litellm_params
def _parse_prompt_to_messages(self, prompt_content: str) -> list[AllMessageValues]:
@ -435,14 +450,14 @@ class GitLabPromptManager(CustomPromptManagement):
def post_call_hook(
self,
user_id: str | None,
response: Any,
response: _ResponseT,
input_messages: list[AllMessageValues],
function_call: dict[str, Any] | str | None = None,
litellm_params: dict[str, Any] | None = None,
function_call: Mapping[str, object] | str | None = None,
litellm_params: Mapping[str, object] | None = None,
prompt_id: str | None = None,
prompt_variables: dict[str, Any] | None = None,
prompt_variables: Mapping[str, object] | None = None,
**kwargs,
) -> Any:
) -> _ResponseT:
return response
def get_available_prompts(self) -> list[str]:
@ -498,7 +513,7 @@ class GitLabPromptManager(CustomPromptManagement):
messages: Final = self._parse_prompt_to_messages(rendered_prompt)
template_model: Final = prompt_metadata.get("model")
optional_params: Final[dict[str, Any]] = {}
optional_params: Final[dict[str, object]] = {}
for param in [
"temperature",
"max_tokens",
@ -658,14 +673,14 @@ class GitLabPromptCache:
self.template_manager: GitLabTemplateManager = self.prompt_manager.prompt_manager
# In-memory stores
self._by_file: dict[str, dict[str, Any]] = {}
self._by_id: dict[str, dict[str, Any]] = {}
self._by_file: dict[str, GitLabCachedPrompt] = {}
self._by_id: dict[str, GitLabCachedPrompt] = {}
# -------------------------
# Public API
# -------------------------
def load_all(self, *, recursive: bool = True) -> dict[str, dict[str, Any]]:
def load_all(self, *, recursive: bool = True) -> dict[str, GitLabCachedPrompt]:
"""
Scan GitLab for all .prompt files under prompts_path, load and parse each,
and return the mapping of repo file path -> JSON-like dict.
@ -695,7 +710,7 @@ class GitLabPromptCache:
return self._by_id
def reload(self, *, recursive: bool = True) -> dict[str, dict[str, Any]]:
def reload(self, *, recursive: bool = True) -> dict[str, GitLabCachedPrompt]:
"""Clear the cache and re-load from GitLab."""
self._by_file.clear()
self._by_id.clear()
@ -709,11 +724,11 @@ class GitLabPromptCache:
"""Return the template IDs (relative to prompts_path, without extension) currently cached."""
return list(self._by_id.keys())
def get_by_file(self, file_path: str) -> dict[str, Any] | None:
def get_by_file(self, file_path: str) -> GitLabCachedPrompt | None:
"""Get a cached prompt JSON by repo file path."""
return self._by_file.get(file_path)
def get_by_id(self, prompt_id: str) -> dict[str, Any] | None:
def get_by_id(self, prompt_id: str) -> GitLabCachedPrompt | None:
"""Get a cached prompt JSON by prompt ID (relative to prompts_path)."""
if prompt_id in self._by_id:
return self._by_id[prompt_id]
@ -728,7 +743,7 @@ class GitLabPromptCache:
# Internals
# -------------------------
def _template_to_json(self, prompt_id: str, tmpl: GitLabPromptTemplate) -> dict[str, Any]:
def _template_to_json(self, prompt_id: str, tmpl: GitLabPromptTemplate) -> GitLabCachedPrompt:
"""
Normalize a GitLabPromptTemplate into a JSON-like dict that is easy to serialize.
"""

39
litellm/integrations/langfuse/langfuse.py
View file

@ -89,7 +89,7 @@ def _extract_cache_read_input_tokens(usage_obj) -> int:
# Check prompt_tokens_details.cached_tokens (used by Gemini and other providers)
if hasattr(usage_obj, "prompt_tokens_details"):
prompt_tokens_details: Final = getattr(usage_obj, "prompt_tokens_details", None)
prompt_tokens_details: Final[object] = getattr(usage_obj, "prompt_tokens_details", None)
if prompt_tokens_details is not None and hasattr(prompt_tokens_details, "cached_tokens"):
cached_tokens: Final = getattr(prompt_tokens_details, "cached_tokens", None)
if cached_tokens is not None and isinstance(cached_tokens, (int, float)) and cached_tokens > 0:
@ -623,9 +623,16 @@ class LangFuseLogger:
)
# Apply custom masking function if provided
if masking_function is not None and callable(masking_function):
input = self._apply_masking_function(input, masking_function)
output = self._apply_masking_function(output, masking_function)
masked_input: Final[object] = (
self._apply_masking_function(input, masking_function)
if masking_function is not None and callable(masking_function)
else input
)
masked_output: Final[object] = (
self._apply_masking_function(output, masking_function)
if masking_function is not None and callable(masking_function)
else output
)
clean_metadata = redact_user_api_key_info(metadata=clean_metadata)
@ -651,15 +658,15 @@ class LangFuseLogger:
# Special keys that are found in the function arguments and not the metadata
if "input" in update_trace_keys:
trace_params["input"] = input if not mask_input else "redacted-by-litellm"
trace_params["input"] = masked_input if not mask_input else "redacted-by-litellm"
if "output" in update_trace_keys:
trace_params["output"] = output if not mask_output else "redacted-by-litellm"
trace_params["output"] = masked_output if not mask_output else "redacted-by-litellm"
else: # don't overwrite an existing trace
trace_params = {
"id": trace_id,
"name": trace_name,
"session_id": session_id,
"input": input if not mask_input else "redacted-by-litellm",
"input": masked_input if not mask_input else "redacted-by-litellm",
"version": clean_metadata.pop(
"trace_version", clean_metadata.get("version", None)
), # If provided just version, it will applied to the trace as well, if applied a trace version it will take precedence
@ -669,9 +676,9 @@ class LangFuseLogger:
trace_params[key.replace("trace_", "")] = clean_metadata.pop(key, None)
if level == "ERROR":
trace_params["status_message"] = output
trace_params["status_message"] = masked_output
else:
trace_params["output"] = output if not mask_output else "redacted-by-litellm"
trace_params["output"] = masked_output if not mask_output else "redacted-by-litellm"
if debug is True or (isinstance(debug, str) and debug.lower() == "true"):
debug_metadata: Final = {
@ -708,7 +715,7 @@ class LangFuseLogger:
("aws_region_name", aws_region_name, bool(aws_region_name)),
("cache_hit", kwargs.get("cache_hit") or False, self._supports_tags() and "cache_hit" in kwargs),
)
enrichments: Final[Mapping[str, Any]] = {
enrichments: Final[Mapping[str, object]] = {
key: value for key, value, include in candidate_enrichments if include
}
@ -802,8 +809,8 @@ class LangFuseLogger:
"end_time": end_time,
"model": model_name,
"model_parameters": optional_params,
"input": input if not mask_input else "redacted-by-litellm",
"output": output if not mask_output else "redacted-by-litellm",
"input": masked_input if not mask_input else "redacted-by-litellm",
"output": masked_output if not mask_output else "redacted-by-litellm",
"usage": usage,
"usage_details": usage_details,
"metadata": {
@ -825,8 +832,8 @@ class LangFuseLogger:
prompt_management_metadata=prompt_management_metadata,
langfuse_client=self.Langfuse,
)
if output is not None and isinstance(output, str) and level == "ERROR":
generation_params["status_message"] = output
if masked_output is not None and isinstance(masked_output, str) and level == "ERROR":
generation_params["status_message"] = masked_output
if self._supports_completion_start_time():
generation_params["completion_start_time"] = kwargs.get("completion_start_time", None)
@ -935,7 +942,7 @@ class LangFuseLogger:
return Version(self.langfuse_sdk_version) >= Version("2.7.3")
@staticmethod
def _apply_masking_function(data: Any, masking_function: Callable[[Any], Any]) -> Any:
def _apply_masking_function(data: object, masking_function: Callable[[object], object]) -> object:
"""
Apply a masking function to data, handling different data types.
@ -1049,7 +1056,7 @@ def _add_prompt_to_generation_params(
generation_params: dict,
clean_metadata: dict,
prompt_management_metadata: StandardLoggingPromptManagementMetadata | None,
langfuse_client: Any,
langfuse_client: object,
) -> dict:
from langfuse import Langfuse
from langfuse.model import (

27
litellm/integrations/opik/opik.py
View file

@ -4,9 +4,12 @@ Opik Logger that logs LLM events to an Opik server
import asyncio
import traceback
from collections.abc import Mapping
from datetime import datetime
from typing import Any, Final
from typing_extensions import ReadOnly, TypedDict, Unpack
from litellm._logging import verbose_logger
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.llms.custom_httpx.http_handler import (
@ -23,7 +26,7 @@ except Exception:
opik_client = None
def _should_skip_event(kwargs: dict[str, Any]) -> bool:
def _should_skip_event(kwargs: Mapping[str, object]) -> bool:
"""Check if event should be skipped due to missing standard_logging_object."""
if kwargs.get("standard_logging_object") is None:
verbose_logger.debug("OpikLogger skipping event; no standard_logging_object found")
@ -31,12 +34,24 @@ def _should_skip_event(kwargs: dict[str, Any]) -> bool:
return False
class _OpikLoggerKwargs(TypedDict, total=False):
"""Constructor options accepted by ``OpikLogger``."""
project_name: ReadOnly[str | None]
url: ReadOnly[str | None]
api_key: ReadOnly[str | None]
workspace: ReadOnly[str | None]
batch_size: ReadOnly[int | None]
flush_interval: ReadOnly[int | None]
max_queue_size: ReadOnly[int | None]
class OpikLogger(CustomBatchLogger):
"""
Opik Logger for logging events to an Opik Server
"""
def __init__(self, **kwargs: Any) -> None:
def __init__(self, **kwargs: Unpack[_OpikLoggerKwargs]) -> None:
self.async_httpx_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.LoggingCallback)
self.sync_httpx_client = _get_httpx_client()
@ -95,7 +110,7 @@ class OpikLogger(CustomBatchLogger):
async def async_log_success_event(
self,
kwargs: dict[str, Any],
kwargs: dict[str, object],
response_obj: Any,
start_time: datetime,
end_time: datetime,
@ -163,7 +178,7 @@ class OpikLogger(CustomBatchLogger):
except Exception as e:
verbose_logger.exception("OpikLogger failed to log success event - %s\n%s", e, traceback.format_exc())
def _sync_send(self, url: str, headers: dict[str, str], batch: dict[str, Any]) -> None:
def _sync_send(self, url: str, headers: dict[str, str], batch: dict[str, object]) -> None:
try:
response: Final = self.sync_httpx_client.post(
url=url,
@ -178,7 +193,7 @@ class OpikLogger(CustomBatchLogger):
def log_success_event(
self,
kwargs: dict[str, Any],
kwargs: dict[str, object],
response_obj: Any,
start_time: datetime,
end_time: datetime,
@ -247,7 +262,7 @@ class OpikLogger(CustomBatchLogger):
except Exception as e:
verbose_logger.exception("OpikLogger failed to log success event - %s\n%s", e, traceback.format_exc())
async def _submit_batch(self, url: str, headers: dict[str, str], batch: dict[str, Any]) -> None:
async def _submit_batch(self, url: str, headers: dict[str, str], batch: dict[str, object]) -> None:
try:
response: Final = await self.async_httpx_client.post(
url=url,

21
litellm/integrations/opik/opik_payload_builder/extractors.py
View file

@ -1,6 +1,7 @@
"""Data extraction functions for Opik payload building."""
import json
from collections.abc import Mapping
from typing import Any, Final
from litellm import _logging
@ -35,8 +36,8 @@ def normalize_provider_name(provider: str | None) -> str | None:
def extract_opik_metadata(
litellm_metadata: dict[str, Any],
standard_logging_metadata: dict[str, Any],
litellm_metadata: Mapping[str, Any],
standard_logging_metadata: Mapping[str, Any],
) -> dict[str, Any]:
"""
Merge Opik metadata from three sources in increasing priority order:
@ -97,7 +98,7 @@ def extract_span_identifiers(
def extract_tags(
opik_metadata: dict[str, Any],
opik_metadata: Mapping[str, Any],
custom_llm_provider: str | None,
) -> list[str]:
"""
@ -122,7 +123,7 @@ def apply_proxy_header_overrides(
project_name: str,
tags: list[str],
thread_id: str | None,
proxy_headers: dict[str, Any],
proxy_headers: Mapping[str, str],
) -> tuple[str, list[str], str | None]:
"""
Apply overrides from proxy request headers (opik_* prefix).
@ -148,7 +149,7 @@ def apply_proxy_header_overrides(
thread_id = value
elif param_key == "tags":
try:
parsed_tags = json.loads(value)
parsed_tags: object = json.loads(value)
if isinstance(parsed_tags, list):
tags.extend(parsed_tags)
except (json.JSONDecodeError, TypeError):
@ -158,11 +159,11 @@ def apply_proxy_header_overrides(
def extract_and_build_metadata(
opik_metadata: dict[str, Any],
standard_logging_metadata: dict[str, Any],
standard_logging_object: dict[str, Any],
litellm_kwargs: dict[str, Any],
) -> dict[str, Any]:
opik_metadata: Mapping[str, object],
standard_logging_metadata: Mapping[str, object],
standard_logging_object: Mapping[str, object],
litellm_kwargs: Mapping[str, object],
) -> dict[str, object]:
"""
Build the complete metadata dictionary from all available sources.

45
litellm/integrations/otel/plumbing/metrics.py
View file

@ -11,9 +11,10 @@ identical metrics. The attribute cardinality filter is reused from v1 by import
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime
from typing import Any, Final, TypeAlias
from typing import Any, Final, Literal, Protocol, TypeAlias
from opentelemetry.metrics import Histogram, Meter
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
@ -151,6 +152,29 @@ METRIC_ATTRIBUTE_CEILING: Final[frozenset[str]] = frozenset(
BOUNDED_HIDDEN_PARAM_KEYS: Final[tuple[str, ...]] = ("model_id",)
class _TokenUsage(TypedDict, total=False):
"""The token counts a response's ``usage`` carries, as the recorder reads them."""
prompt_tokens: ReadOnly[int]
completion_tokens: ReadOnly[int]
class _ResponseView(Protocol):
"""The one read the recorder makes on a litellm response object."""
def get(self, key: Literal["usage"], /) -> _TokenUsage | None: ...
class _MetricKwargs(TypedDict, total=False):
"""The logging kwargs the recorder reads directly."""
call_type: ReadOnly[str | None]
litellm_params: ReadOnly[Mapping[str, object] | None]
response_cost: ReadOnly[float | None]
completion_start_time: ReadOnly[datetime | float | str | None]
api_call_start_time: ReadOnly[datetime | float | str | None]
def resolve_error_type(kwargs: Mapping[str, Any]) -> str:
"""The ``error.type`` value for a failed request.
@ -192,8 +216,8 @@ class GenAIMetricRecorder:
def record(
self,
kwargs: Mapping[str, Any],
response_obj: Any,
kwargs: _MetricKwargs,
response_obj: _ResponseView | None,
start_time: datetime,
end_time: datetime,
) -> None:
@ -218,7 +242,7 @@ class GenAIMetricRecorder:
def record_failure(
self,
kwargs: Mapping[str, Any],
kwargs: _MetricKwargs,
start_time: datetime,
end_time: datetime,
) -> None:
@ -342,7 +366,7 @@ class GenAIMetricRecorder:
# Per-metric recording
# ------------------------------------------------------------------ #
def _record_token_usage(self, response_obj: Any, common_attrs: dict) -> None:
def _record_token_usage(self, response_obj: _ResponseView | None, common_attrs: dict) -> None:
if not response_obj:
return
usage: Final = response_obj.get("usage")
@ -353,7 +377,7 @@ class GenAIMetricRecorder:
self._metrics.token_usage.record(usage.get("prompt_tokens", 0), attributes=in_attrs)
self._metrics.token_usage.record(usage.get("completion_tokens", 0), attributes=out_attrs)
def _record_time_to_first_token(self, kwargs: Mapping[str, Any], common_attrs: dict) -> None:
def _record_time_to_first_token(self, kwargs: _MetricKwargs, common_attrs: dict) -> None:
time_to_first_chunk: Final = time_to_first_chunk_seconds(kwargs)
if time_to_first_chunk is None:
return
@ -361,15 +385,14 @@ class GenAIMetricRecorder:
def _record_time_per_output_token(
self,
kwargs: Mapping[str, Any],
response_obj: Any,
kwargs: _MetricKwargs,
response_obj: _ResponseView | None,
end_time: datetime,
duration_s: float,
common_attrs: dict,
) -> None:
completion_tokens = None
if response_obj and (usage := response_obj.get("usage")):
completion_tokens = usage.get("completion_tokens")
usage: Final = response_obj.get("usage") if response_obj else None
completion_tokens: Final = usage.get("completion_tokens") if usage else None
if completion_tokens is None or completion_tokens <= 0:
return

62
litellm/integrations/posthog.py
View file

@ -12,7 +12,10 @@ For batching specific details see CustomBatchLogger class
import asyncio
import atexit
import os
from typing import Any, Final
from collections.abc import Mapping, Sequence
from typing import Final
from typing_extensions import ReadOnly, TypedDict
from litellm._logging import verbose_logger
from litellm._uuid import uuid
@ -34,6 +37,21 @@ from litellm.types.integrations.posthog import (
from litellm.types.utils import StandardCallbackDynamicParams, StandardLoggingPayload
class PostHogBatchPayload(TypedDict):
api_key: ReadOnly[str]
batch: ReadOnly[Sequence[PostHogEventPayload]]
class PostHogLiteLLMParams(TypedDict, total=False):
metadata: ReadOnly[Mapping[str, object]]
class PostHogLogKwargs(TypedDict, total=False):
standard_logging_object: ReadOnly[StandardLoggingPayload]
standard_callback_dynamic_params: ReadOnly[StandardCallbackDynamicParams]
litellm_params: ReadOnly[PostHogLiteLLMParams]
class PostHogLogger(CustomBatchLogger):
def __init__(self, **kwargs):
"""
@ -137,7 +155,7 @@ class PostHogLogger(CustomBatchLogger):
if len(self.log_queue) >= self.batch_size:
await self.flush_queue()
def create_posthog_event_payload(self, kwargs: dict[str, Any]) -> PostHogEventPayload:
def create_posthog_event_payload(self, kwargs: PostHogLogKwargs) -> PostHogEventPayload:
"""
Helper function to create a PostHog event payload for logging
@ -171,11 +189,11 @@ class PostHogLogger(CustomBatchLogger):
def _create_posthog_properties(
self,
standard_logging_object: StandardLoggingPayload,
kwargs: dict[str, Any],
kwargs: PostHogLogKwargs,
event_name: str,
) -> dict[str, Any]:
) -> dict[str, object]:
"""Create PostHog properties following LLM Analytics spec"""
properties: Final = {}
properties: Final[dict[str, object]] = {}
# Core model information
properties["$ai_model"] = self._safe_get(standard_logging_object, "model", "")
@ -211,16 +229,19 @@ class PostHogLogger(CustomBatchLogger):
properties["$ai_error"] = error_str
# Add trace properties
self._add_trace_properties(properties, kwargs)
self._add_trace_properties(properties, standard_logging_object, kwargs)
# Add custom metadata fields
self._add_custom_metadata_properties(properties, kwargs)
return properties
def _add_trace_properties(self, properties: dict[str, Any], kwargs: dict[str, Any]):
standard_logging_object: Final = self._safe_get(kwargs, "standard_logging_object", {})
def _add_trace_properties(
self,
properties: dict[str, object],
standard_logging_object: StandardLoggingPayload,
kwargs: PostHogLogKwargs,
) -> None:
trace_id: Final = self._safe_get(standard_logging_object, "trace_id", self._safe_uuid())
properties["$ai_trace_id"] = trace_id
@ -232,7 +253,7 @@ class PostHogLogger(CustomBatchLogger):
if parent_id:
properties["$ai_parent_id"] = parent_id
def _add_custom_metadata_properties(self, properties: dict[str, Any], kwargs: dict[str, Any]):
def _add_custom_metadata_properties(self, properties: dict[str, object], kwargs: PostHogLogKwargs) -> None:
"""Add custom metadata fields to PostHog properties"""
metadata: Final = self._extract_metadata(kwargs)
if not isinstance(metadata, dict):
@ -277,7 +298,7 @@ class PostHogLogger(CustomBatchLogger):
if key not in litellm_internal_fields:
properties[key] = value
def _get_distinct_id(self, standard_logging_object: StandardLoggingPayload, kwargs: dict[str, Any]) -> str:
def _get_distinct_id(self, standard_logging_object: StandardLoggingPayload, kwargs: PostHogLogKwargs) -> str:
metadata: Final = self._extract_metadata(kwargs)
user_id: Final = self._safe_get(metadata, "user_id")
if user_id:
@ -291,7 +312,7 @@ class PostHogLogger(CustomBatchLogger):
return self._safe_uuid()
def _get_credentials_for_request(self, kwargs: dict[str, Any]) -> tuple[str | None, str | None]:
def _get_credentials_for_request(self, kwargs: PostHogLogKwargs) -> tuple[str | None, str | None]:
"""
Get PostHog credentials for this request.
@ -334,7 +355,7 @@ class PostHogLogger(CustomBatchLogger):
verbose_logger.debug("[POSTHOG MOCK] Mock mode enabled - API calls will be intercepted")
# Group events by credentials for batch sending
batches_by_credentials: Final[dict[tuple[str, str], list]] = {}
batches_by_credentials: Final[dict[tuple[str, str], list[PostHogEventPayload]]] = {}
for item in self.log_queue:
key = (item["api_key"], item["api_url"])
if key not in batches_by_credentials:
@ -380,18 +401,19 @@ class PostHogLogger(CustomBatchLogger):
verbose_logger.error("PostHog: Failed to initialize async components: %s", e)
raise
def _extract_metadata(self, kwargs: dict[str, Any]) -> dict[str, Any]:
litellm_params: Final = kwargs.get("litellm_params", {}) or {}
return litellm_params.get("metadata", {}) or {}
def _extract_metadata(self, kwargs: PostHogLogKwargs) -> Mapping[str, object]:
litellm_params: Final[PostHogLiteLLMParams] = kwargs.get("litellm_params", {}) or {}
metadata: Final[Mapping[str, object]] = litellm_params.get("metadata", {}) or {}
return metadata
def _safe_uuid(self) -> str:
return str(uuid.uuid4())
def _create_posthog_payload(self, events: list, api_key: str) -> dict[str, Any]:
def _create_posthog_payload(self, events: Sequence[PostHogEventPayload], api_key: str) -> PostHogBatchPayload:
return {"api_key": api_key, "batch": events}
def _safe_get(self, obj: Any, key: str, default: Any = None) -> Any:
if obj is None or not hasattr(obj, "get"):
def _safe_get(self, obj: Mapping[str, object] | None, key: str, default: object = None) -> object:
if not isinstance(obj, Mapping):
return default
return obj.get(key, default)
@ -412,7 +434,7 @@ class PostHogLogger(CustomBatchLogger):
try:
# Group events by credentials (same logic as async_send_batch)
batches_by_credentials: Final[dict[tuple[str, str], list]] = {}
batches_by_credentials: Final[dict[tuple[str, str], list[PostHogEventPayload]]] = {}
for item in self.log_queue:
key = (item["api_key"], item["api_url"])
if key not in batches_by_credentials:

6
litellm/integrations/vector_store_integrations/vector_store_pre_call_hook.py
View file

@ -13,7 +13,7 @@ from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.llms.openai import AllMessageValues, ChatCompletionUserMessage
from litellm.types.prompts.init_prompts import PromptSpec
from litellm.types.utils import StandardCallbackDynamicParams
from litellm.types.utils import CallTypes, StandardCallbackDynamicParams
from litellm.types.vector_stores import (
LiteLLM_ManagedVectorStore,
VectorStoreResultContent,
@ -226,7 +226,7 @@ class VectorStorePreCallHook(CustomLogger):
self,
request_data: dict,
response: Any,
call_type: Any | None,
call_type: CallTypes | None,
) -> Any | None:
"""
Add search results to the response after successful LLM call.
@ -283,7 +283,7 @@ class VectorStorePreCallHook(CustomLogger):
self,
request_data: dict,
response_chunk: Any,
call_type: Any | None,
call_type: CallTypes | None,
) -> Any | None:
"""
Add search results to the final streaming chunk.

54
litellm/integrations/websearch_interception/handler.py
View file

@ -416,25 +416,15 @@ class WebSearchInterceptionLogger(CustomLogger):
if not tools:
return None
is_responses_call: Final = call_type in (CallTypes.responses, CallTypes.aresponses)
has_websearch: Final = (
any(is_web_search_tool_responses(tool) for tool in tools)
if is_responses_call
else any(is_web_search_tool(tool) for tool in tools)
)
if call_type in (CallTypes.responses, CallTypes.aresponses):
return self._convert_responses_tools(kwargs=kwargs, tools=tools)
# Check if any tool is a web search tool (native or already LiteLLM standard)
has_websearch: Final = any(is_web_search_tool(t) for t in tools)
if not has_websearch:
return None
if self.search_tool_name:
try:
from litellm.proxy.proxy_server import llm_router
except ImportError:
llm_router = None
self._select_search_tool_from_router(llm_router=llm_router)
if is_responses_call:
return self._convert_responses_tools(kwargs=kwargs, tools=tools)
verbose_logger.debug("WebSearchInterception: Converting native web_search tools to LiteLLM standard")
# If the client sent an Anthropic-native web_search_* tool, mark the
@ -1641,36 +1631,34 @@ class WebSearchInterceptionLogger(CustomLogger):
return None
def _select_search_tool_from_router(self, llm_router: object) -> "_SearchToolConfig | None":
search_tools: Final = list(getattr(llm_router, "search_tools", []) or [])
if llm_router is None or not hasattr(llm_router, "search_tools"):
return None
search_tools: Final = tuple(getattr(llm_router, "search_tools", None) or ())
return self._select_search_tool_from_list(search_tools=search_tools, source="router")
def _select_search_tool_from_list(
self,
search_tools: list[_SearchToolConfig],
search_tools: Sequence[_SearchToolConfig],
source: str,
) -> "_SearchToolConfig | None":
if self.search_tool_name:
matching_tools = [tool for tool in search_tools if tool.get("search_tool_name") == self.search_tool_name]
if not matching_tools:
raise ValueError(f"Configured search tool '{self.search_tool_name}' was not found")
selected_tool: Final = matching_tools[0]
litellm_params: Final = selected_tool.get("litellm_params")
selected_search_provider: Final = (
litellm_params.get("search_provider") if isinstance(litellm_params, Mapping) else None
matching_tools: Final = tuple(
tool for tool in search_tools if tool.get("search_tool_name") == self.search_tool_name
)
if not isinstance(selected_search_provider, str) or not selected_search_provider.strip():
raise ValueError(
f"Configured search tool '{self.search_tool_name}' does not define a valid search provider"
if matching_tools:
search_provider = (matching_tools[0].get("litellm_params", {}) or {}).get("search_provider")
verbose_logger.debug(
"WebSearchInterception: Found search tool '%s' from %s with provider '%s'",
self.search_tool_name,
source,
search_provider,
)
return matching_tools[0]
verbose_logger.debug(
"WebSearchInterception: Found search tool '%s' from %s with provider '%s'",
"WebSearchInterception: Search tool '%s' not found in %s, falling back to first available or perplexity",
self.search_tool_name,
source,
selected_search_provider,
)
return selected_tool
if search_tools:
first_tool: Final = search_tools[0]

12
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -536,6 +536,14 @@ def get_llm_provider(
)
def _dashscope_family_chat_config(custom_llm_provider: str) -> "litellm.DashScopeChatConfig":
if custom_llm_provider == "qwencloud":
return litellm.QwenCloudChatConfig()
if custom_llm_provider == "qwen_ai_platform":
return litellm.QwenAIPlatformChatConfig()
return litellm.DashScopeChatConfig()
def _get_openai_compatible_provider_info(
model: str,
api_base: str | None,
@ -785,11 +793,11 @@ def _get_openai_compatible_provider_info(
api_base,
dynamic_api_key,
) = litellm.HerokuChatConfig()._get_openai_compatible_provider_info(api_base, api_key)
elif custom_llm_provider == "dashscope":
elif custom_llm_provider in ("dashscope", "qwencloud", "qwen_ai_platform"):
(
api_base,
dynamic_api_key,
) = litellm.DashScopeChatConfig()._get_openai_compatible_provider_info(api_base, api_key)
) = _dashscope_family_chat_config(custom_llm_provider)._get_openai_compatible_provider_info(api_base, api_key)
elif custom_llm_provider == "modelscope":
(
api_base,

44
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
View file

@ -3,7 +3,7 @@ Helper utilities for tracking the cost of built-in tools.
"""
from collections.abc import Mapping
from typing import Any, Final, Literal
from typing import Final, Literal
import litellm
from litellm.constants import OPENAI_FILE_SEARCH_COST_PER_1K_CALLS
@ -16,6 +16,7 @@ from litellm.types.llms.openai import (
WebSearchOptions,
)
from litellm.types.utils import (
ChatCompletionAnnotation,
Message,
ModelInfo,
ModelResponse,
@ -49,7 +50,7 @@ class StandardBuiltInToolCostTracking:
@staticmethod
def get_cost_for_built_in_tools(
model: str,
response_object: Any,
response_object: object,
usage: Usage | None = None,
custom_llm_provider: str | None = None,
standard_built_in_tools_params: StandardBuiltInToolsParams | None = None,
@ -201,8 +202,7 @@ class StandardBuiltInToolCostTracking:
model_info: Final = StandardBuiltInToolCostTracking._safe_get_model_info(
model=model, custom_llm_provider=custom_llm_provider
)
file_search_raw: Final[Any] = standard_built_in_tools_params.get("file_search", {})
file_search_usage: Final[FileSearchTool | None] = FileSearchTool(**file_search_raw) if file_search_raw else None
file_search_usage: Final[FileSearchTool | None] = standard_built_in_tools_params.get("file_search") or None
# Convert model_info to dict and extract usage parameters
model_info_dict: Final = dict(model_info) if model_info is not None else None
@ -245,7 +245,7 @@ class StandardBuiltInToolCostTracking:
@staticmethod
def _extract_file_search_params(
file_search_usage: Any,
file_search_usage: object,
) -> tuple[float | None, float | None]:
"""Extract and convert file search parameters safely."""
storage_gb = None
@ -335,7 +335,7 @@ class StandardBuiltInToolCostTracking:
@staticmethod
def _extract_token_counts(
computer_use_usage: Any,
computer_use_usage: object,
) -> tuple[int | None, int | None]:
"""Extract and convert token counts safely."""
input_tokens = None
@ -351,9 +351,9 @@ class StandardBuiltInToolCostTracking:
return input_tokens, output_tokens
@staticmethod
def _safe_convert_to_int(value: Any) -> int | None:
def _safe_convert_to_int(value: object) -> int | None:
"""Safely convert a value to int."""
if value is not None:
if isinstance(value, (int, float, str)):
try:
return int(value)
except (TypeError, ValueError):
@ -381,7 +381,7 @@ class StandardBuiltInToolCostTracking:
return usage.model_copy(update={"server_tool_use": server_tool_use})
@staticmethod
def response_object_includes_web_search_call(response_object: Any, usage: Usage | None = None) -> bool:
def response_object_includes_web_search_call(response_object: object, usage: Usage | None = None) -> bool:
"""
Check if the response object includes a web search call.
@ -446,7 +446,7 @@ class StandardBuiltInToolCostTracking:
@staticmethod
def response_object_includes_file_search_call(
response_object: Any,
response_object: object,
) -> bool:
"""
Check if the response object includes a file search call.
@ -477,11 +477,11 @@ class StandardBuiltInToolCostTracking:
message: Message | None = getattr(choice, "message", None)
if message is None:
continue
if annotations := getattr(message, "annotations", None):
if len(annotations) > 0:
for annotation in annotations:
if annotation.get("type", None) == annotation_type:
return True
annotations: list[ChatCompletionAnnotation] | None = getattr(message, "annotations", None)
if annotations:
for annotation in annotations:
if annotation.get("type", None) == annotation_type:
return True
return False
@staticmethod
@ -522,10 +522,8 @@ class StandardBuiltInToolCostTracking:
if model_info is None:
return 0.0
search_context_raw: Final[Any] = model_info.get("search_context_cost_per_query", {})
search_context_pricing: Final[SearchContextCostPerQuery] = (
SearchContextCostPerQuery(**search_context_raw) if search_context_raw else SearchContextCostPerQuery()
)
search_context_raw: Final = model_info.get("search_context_cost_per_query")
search_context_pricing: Final[SearchContextCostPerQuery] = search_context_raw or SearchContextCostPerQuery()
if web_search_options.get("search_context_size", None) == "low":
return search_context_pricing.get("search_context_size_low", 0.0)
elif web_search_options.get("search_context_size", None) == "medium":
@ -545,10 +543,8 @@ class StandardBuiltInToolCostTracking:
"""
if model_info is None:
return 0.0
search_context_raw: Final[Any] = model_info.get("search_context_cost_per_query", {}) or {}
search_context_pricing: Final[SearchContextCostPerQuery] = (
SearchContextCostPerQuery(**search_context_raw) if search_context_raw else SearchContextCostPerQuery()
)
search_context_raw: Final = model_info.get("search_context_cost_per_query")
search_context_pricing: Final[SearchContextCostPerQuery] = search_context_raw or SearchContextCostPerQuery()
return search_context_pricing.get("search_context_size_medium", 0.0)
@staticmethod
@ -714,7 +710,7 @@ class StandardBuiltInToolCostTracking:
response_object: ModelResponse,
) -> bool:
for _choice in response_object.choices:
message = getattr(_choice, "message", None)
message: Message | None = getattr(_choice, "message", None)
if (
message is not None
and hasattr(message, "annotations")

20
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -555,10 +555,10 @@ def update_messages_with_model_file_ids(
def update_responses_input_with_model_file_ids(
input: Any,
input: object,
model_id: str | None = None,
model_file_id_mapping: dict[str, dict[str, str]] | None = None,
) -> str | list[dict[str, Any]]:
) -> object:
"""
Updates responses API input with provider-specific file IDs.
File IDs are always inside the content array, not as direct input_file items.
@ -639,8 +639,8 @@ def update_responses_input_with_model_file_ids(
def _decode_vector_store_ids_in_tools(
tools: list[dict[str, Any]] | None,
) -> list[dict[str, Any]] | None:
tools: list[dict[str, object]] | None,
) -> list[dict[str, object]] | None:
"""
Decodes unified (LiteLLM-managed) vector_store_ids in file_search tools to
provider-native IDs. Non-unified IDs are passed through unchanged.
@ -692,10 +692,10 @@ def _decode_vector_store_ids_in_tools(
def update_responses_tools_with_model_file_ids(
tools: list[dict[str, Any]] | None,
tools: list[dict[str, object]] | None,
model_id: str | None = None,
model_file_id_mapping: dict[str, dict[str, str]] | None = None,
) -> list[dict[str, Any]] | None:
) -> list[dict[str, object]] | None:
"""
Updates responses API tools with provider-specific file IDs.
@ -888,7 +888,7 @@ def extract_file_data(file_data: FileTypes) -> ExtractedFileData:
# ---------------------------------------------------------------------------
def _estimate_json_bytes(obj: Any) -> int:
def _estimate_json_bytes(obj: object) -> int:
"""Estimate the JSON-serialised byte size of ``obj`` without materialising
JSON. Walks iteratively (no recursion stack risk).
@ -1979,7 +1979,7 @@ def drop_tool_reference_parts_from_tool_messages(
return [_drop_tool_reference_parts(message) for message in messages] # mutable-ok: pipelines mutate message lists
def _attempt_json_repair(s: str) -> Any | None:
def _attempt_json_repair(s: str) -> object | None:
"""
Attempt to repair truncated JSON produced by LLM tool calls.
@ -2095,7 +2095,7 @@ def parse_tool_call_arguments(
raise ValueError(error_message) from original_error
def split_concatenated_json_objects(raw: str) -> list[dict[str, Any]]:
def split_concatenated_json_objects(raw: str) -> list[dict[str, object]]:
"""
Split a string that contains one or more concatenated JSON objects into
a list of parsed dicts.
@ -2131,7 +2131,7 @@ def split_concatenated_json_objects(raw: str) -> list[dict[str, Any]]:
return []
decoder: Final = json.JSONDecoder()
results: Final[list[dict[str, Any]]] = []
results: Final[list[dict[str, object]]] = []
idx = 0
length: Final = len(raw)

52
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -1694,6 +1694,18 @@ def convert_function_to_anthropic_tool_invoke(
raise e
def _find_server_tool_result(
tool_id: str,
web_search_results: Sequence[object] | None,
tool_results: Sequence[object] | None,
) -> dict[str, object] | None:
candidates: Final = (*(web_search_results or ()), *(tool_results or ()))
return next(
(result for result in candidates if isinstance(result, dict) and result.get("tool_use_id") == tool_id),
None,
)
def convert_to_anthropic_tool_invoke(
tool_calls: list[ChatCompletionAssistantToolCall],
web_search_results: list[Any] | None = None,
@ -1758,32 +1770,22 @@ def convert_to_anthropic_tool_invoke(
context="Anthropic tool invoke",
)
# Check if this is a server-side tool (web_search, tool_search, etc.)
# Server tool IDs start with "srvtoolu_"
if tool_id.startswith("srvtoolu_"):
# Create server_tool_use block instead of tool_use
_anthropic_server_tool_use: dict[str, object] = {
"type": "server_tool_use",
"id": tool_id,
"name": tool_name,
"input": tool_input,
}
anthropic_tool_invoke.append(_anthropic_server_tool_use)
# Add corresponding tool result if available.
# Check both web_search_results (web_search_tool_result / web_fetch_tool_result)
# and tool_results (bash_code_execution_tool_result, etc.)
_all_tool_results: list[Any] = []
if web_search_results:
_all_tool_results.extend(web_search_results)
if tool_results:
_all_tool_results.extend(tool_results)
for result in _all_tool_results:
if result.get("tool_use_id") == tool_id:
anthropic_tool_invoke.append(result)
break
server_tool_result = (
_find_server_tool_result(tool_id, web_search_results, tool_results)
if tool_id.startswith("srvtoolu_")
else None
)
if server_tool_result is not None:
anthropic_tool_invoke.append(
{
"type": "server_tool_use",
"id": tool_id,
"name": tool_name,
"input": tool_input,
}
)
anthropic_tool_invoke.append(server_tool_result)
else:
# Regular tool_use
sanitized_tool_id = _sanitize_anthropic_tool_use_id(tool_id)
_anthropic_tool_use_param = AnthropicMessagesToolUseParam(
type="tool_use",

2
litellm/litellm_core_utils/realtime_streaming.py
View file

@ -1500,6 +1500,6 @@ class RealTimeStreaming:
pass
def client_sent_openai_beta_realtime_header(websocket: Any) -> bool:
def client_sent_openai_beta_realtime_header(websocket: _ScopedWebSocket) -> bool:
"""True when the client WebSocket includes ``OpenAI-Beta: realtime=v1``."""
return RealTimeStreaming._detect_beta_header(websocket)

14
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -75,6 +75,18 @@ class _ContentChunk(TypedDict):
choices: Sequence[_ContentChoice]
class _FunctionCallDelta(TypedDict):
function_call: ReadOnly[FunctionCall]
class _FunctionCallChoice(TypedDict):
delta: ReadOnly[_FunctionCallDelta]
class _FunctionCallChunk(TypedDict):
choices: ReadOnly[Sequence[_FunctionCallChoice]]
class _AudioDelta(TypedDict, total=False):
audio: ChatCompletionAudioDelta | None
@ -590,7 +602,7 @@ class ChunkProcessor:
return tool_calls_list
def get_combined_function_call_content(self, function_call_chunks: list[dict[str, Any]]) -> FunctionCall:
def get_combined_function_call_content(self, function_call_chunks: Sequence["_FunctionCallChunk"]) -> FunctionCall:
argument_list: Final = []
delta = function_call_chunks[0]["choices"][0]["delta"]
function_call = delta.get("function_call", "")

2
litellm/litellm_core_utils/streaming_handler.py
View file

@ -862,6 +862,8 @@ class CustomStreamWrapper:
model_response: Final = ModelResponseStream(**args)
if self.response_id is not None:
model_response.id = self.response_id
elif model_response.id:
self.response_id = model_response.id
if self.system_fingerprint is not None:
model_response.system_fingerprint = self.system_fingerprint

43
litellm/litellm_core_utils/token_counter.py
View file

@ -4,8 +4,9 @@ import base64
import io
import struct
from collections.abc import Callable, Iterable, Mapping, Sequence
from typing import Any, Final, Literal, cast
from typing import Final, Literal, cast
import httpx
import tiktoken
import litellm
@ -171,6 +172,10 @@ def calculate_tiles_needed(
return total_tiles
def _unpack_ints(fmt: str, buffer: bytes) -> tuple[int, ...]:
return struct.unpack(fmt, buffer)
def get_image_type(image_data: bytes) -> str | None:
"""take an image (really only the first ~100 bytes max are needed)
and return 'png' 'gif' 'jpeg' 'webp' 'heic' or None. method added to
@ -210,9 +215,9 @@ def get_image_dimensions(
if data.startswith(("http://", "https://")):
try:
client: Final = _get_httpx_client()
response: Final = safe_get(client, data)
response: Final[httpx.Response] = safe_get(client, data)
max_bytes: Final = int(MAX_IMAGE_URL_DOWNLOAD_SIZE_MB * 1024 * 1024)
content_length: Final = response.headers.get("Content-Length")
content_length: Final[str | None] = response.headers.get("Content-Length")
if content_length is not None and int(content_length) > max_bytes:
pass # skip download; img_data stays None
else:
@ -229,10 +234,10 @@ def get_image_dimensions(
img_type: Final = get_image_type(img_data)
if img_type == "png":
w, h = struct.unpack(">LL", img_data[16:24])
w, h = _unpack_ints(">LL", img_data[16:24])
return w, h
elif img_type == "gif":
w, h = struct.unpack("<HH", img_data[6:10])
w, h = _unpack_ints("<HH", img_data[6:10])
return w, h
elif img_type == "jpeg":
with io.BytesIO(img_data) as fhandle:
@ -245,25 +250,25 @@ def get_image_dimensions(
while ord(byte) == 0xFF:
byte = fhandle.read(1)
ftype = ord(byte)
size = struct.unpack(">H", fhandle.read(2))[0] - 2
size = _unpack_ints(">H", fhandle.read(2))[0] - 2
fhandle.seek(1, 1)
h, w = struct.unpack(">HH", fhandle.read(4))
h, w = _unpack_ints(">HH", fhandle.read(4))
return w, h
elif img_type == "webp":
# For WebP, the dimensions are stored at different offsets depending on the format
# Check for VP8X (extended format)
if img_data[12:16] == b"VP8X":
w = struct.unpack("<I", img_data[24:27] + b"\x00")[0] + 1
h = struct.unpack("<I", img_data[27:30] + b"\x00")[0] + 1
w = _unpack_ints("<I", img_data[24:27] + b"\x00")[0] + 1
h = _unpack_ints("<I", img_data[27:30] + b"\x00")[0] + 1
return w, h
# Check for VP8 (lossy format)
elif img_data[12:16] == b"VP8 ":
w = struct.unpack("<H", img_data[26:28])[0] & 0x3FFF
h = struct.unpack("<H", img_data[28:30])[0] & 0x3FFF
w = _unpack_ints("<H", img_data[26:28])[0] & 0x3FFF
h = _unpack_ints("<H", img_data[28:30])[0] & 0x3FFF
return w, h
# Check for VP8L (lossless format)
elif img_data[12:16] == b"VP8L":
bits: Final = struct.unpack("<I", img_data[21:25])[0]
bits: Final = _unpack_ints("<I", img_data[21:25])[0]
w = (bits & 0x3FFF) + 1
h = ((bits >> 14) & 0x3FFF) + 1
return w, h
@ -420,8 +425,8 @@ def token_counter(
def _count_function_call_tokens(
key: str,
value: Any,
message: Mapping[str, Any],
value: object,
message: Mapping[str, object],
count_function: TokenCounterFunction,
) -> int:
"""
@ -587,7 +592,7 @@ def _fix_model_name(model: str) -> str:
def _count_image_tokens(
image_url: Any,
image_url: object,
use_default_image_token_count: bool,
) -> int:
"""
@ -627,7 +632,7 @@ def _count_image_tokens(
raise ValueError(f"Invalid image_url type: {type(image_url).__name__}. Expected str or dict with 'url' field.")
def _validate_anthropic_content(content: Mapping[str, Any]) -> type:
def _validate_anthropic_content(content: Mapping[str, object]) -> type:
"""
Validate and determine which Anthropic TypedDict applies.
@ -642,7 +647,7 @@ def _validate_anthropic_content(content: Mapping[str, Any]) -> type:
"tool_result": AnthropicMessagesToolResultParam,
}
expected_cls: Final = mapping.get(content_type)
expected_cls: Final = mapping.get(content_type) if isinstance(content_type, str) else None
if expected_cls is None:
raise ValueError(f"Unknown Anthropic content type: '{content_type}'")
@ -714,7 +719,7 @@ def _count_file_tokens(
def _count_anthropic_content(
content: Mapping[str, Any],
content: Mapping[str, object],
count_function: TokenCounterFunction,
use_default_image_token_count: bool,
default_token_count: int | None,
@ -729,7 +734,7 @@ def _count_anthropic_content(
avoiding hardcoded field names.
"""
typeddict_cls: Final = _validate_anthropic_content(content)
type_hints: Final = getattr(typeddict_cls, "__annotations__", {})
type_hints: Final[Mapping[str, object]] = getattr(typeddict_cls, "__annotations__", {})
tokens = 0
# Fields to skip (metadata/identifiers that don't contribute to prompt tokens)

27
litellm/llms/a2a/chat/guardrail_translation/handler.py
View file

@ -11,8 +11,11 @@ A2A Protocol Format:
"""
import json
from collections.abc import Sequence
from typing import TYPE_CHECKING, Any, Final, Optional
from typing_extensions import ReadOnly, TypedDict
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.types.utils import GenericGuardrailAPIInputs
@ -23,6 +26,13 @@ if TYPE_CHECKING:
from litellm.proxy._types import UserAPIKeyAuth
class _A2ATextPart(TypedDict, total=False):
"""The subset of an A2A message part this handler reads text from."""
kind: ReadOnly[str]
text: ReadOnly[str]
class A2AGuardrailHandler(BaseTranslation):
"""
Handler for processing A2A Protocol messages with guardrails.
@ -41,7 +51,7 @@ class A2AGuardrailHandler(BaseTranslation):
data: dict,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
) -> Any:
) -> dict:
"""
Process A2A input messages by applying guardrails to text content.
@ -214,12 +224,12 @@ class A2AGuardrailHandler(BaseTranslation):
async def process_output_streaming_response(
self,
responses_so_far: list[Any],
responses_so_far: list[object],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
request_data: dict | None = None,
) -> list[Any]:
) -> list[object]:
"""
Process A2A streaming output by applying guardrails to accumulated text.
@ -305,11 +315,12 @@ class A2AGuardrailHandler(BaseTranslation):
def _parse_streaming_responses(
self,
responses_so_far: list[Any],
) -> tuple[list[dict[str, Any] | None], list[tuple[int, dict[str, Any]]]]:
responses_so_far: list[object],
) -> tuple[list[dict[str, object] | None], list[tuple[int, dict[str, object]]]]:
"""Parse JSON-RPC items, returning aligned parsed list and valid entries."""
parsed: Final[list[dict[str, Any] | None]] = [None] * len(responses_so_far)
parsed: Final[list[dict[str, object] | None]] = [None] * len(responses_so_far)
for i, item in enumerate(responses_so_far):
obj: dict[str, object]
if isinstance(item, dict):
obj = item
elif isinstance(item, str):
@ -326,7 +337,7 @@ class A2AGuardrailHandler(BaseTranslation):
def _collect_text_from_parsed_chunks(
self,
valid_parsed: list[tuple[int, dict[str, Any]]],
valid_parsed: list[tuple[int, dict[str, object]]],
) -> tuple[str, list[int]]:
"""Collect text from parsed chunks, returning combined text and indices."""
from litellm.llms.a2a.common_utils import extract_text_from_a2a_response
@ -411,7 +422,7 @@ class A2AGuardrailHandler(BaseTranslation):
def _extract_texts_from_parts(
self,
parts: list[dict[str, Any]],
parts: Sequence[_A2ATextPart],
path: tuple[str, ...],
texts_to_check: list[str],
task_mappings: list[tuple[tuple[str, ...], int]],

30
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -100,16 +100,6 @@ InputWriteBackTarget = (
)
class _SSEDelta(TypedDict, total=False):
type: ReadOnly[str]
text: ReadOnly[str]
stop_reason: ReadOnly[str | None]
class _SSEEventData(TypedDict, total=False):
delta: ReadOnly[_SSEDelta]
def _as_str_mapping(value: Mapping[str, object]) -> Mapping[str, object]:
return value
@ -157,6 +147,16 @@ class ExtractedInput:
EMPTY_EXTRACTED_INPUT: Final = ExtractedInput(scanned=(), images=())
class _AnthropicSSEDelta(TypedDict, total=False):
type: ReadOnly[str]
text: ReadOnly[str]
stop_reason: ReadOnly[str | None]
class _AnthropicSSEEvent(TypedDict, total=False):
delta: ReadOnly[_AnthropicSSEDelta]
class AnthropicMessagesHandler(BaseTranslation):
"""Process Anthropic messages with guardrails.
@ -1247,8 +1247,8 @@ class AnthropicMessagesHandler(BaseTranslation):
# Only process content_block_delta events
if event_type == "content_block_delta" and data_line:
try:
data: _SSEEventData = json.loads(data_line)
delta = data.get("delta", {})
data: _AnthropicSSEEvent = json.loads(data_line)
delta: _AnthropicSSEDelta = data.get("delta", {})
if delta.get("type") == "text_delta":
text += delta.get("text", "")
except json.JSONDecodeError:
@ -1310,9 +1310,9 @@ class AnthropicMessagesHandler(BaseTranslation):
# Check for message_delta event with stop_reason
if event_type == "message_delta" and data_line:
try:
data: _SSEEventData = json.loads(data_line)
delta = data.get("delta", {})
stop_reason = delta.get("stop_reason")
data: _AnthropicSSEEvent = json.loads(data_line)
delta: _AnthropicSSEDelta = data.get("delta", {})
stop_reason: str | None = delta.get("stop_reason")
if stop_reason is not None:
return True
except json.JSONDecodeError:

34
litellm/llms/anthropic/chat/handler.py
View file

@ -66,6 +66,10 @@ if TYPE_CHECKING:
from litellm.llms.base_llm.chat.transformation import BaseConfig
def _loads_stream_chunk(payload: str) -> dict[str, object]:
return json.loads(payload)
async def make_call(
client: AsyncHTTPHandler | None,
api_base: str,
@ -78,7 +82,7 @@ async def make_call(
json_mode: bool,
speed: str | None = None,
tool_name_reverse_map: dict[str, str] | None = None,
) -> tuple[Any, httpx.Headers]:
) -> tuple["ModelResponseIterator", httpx.Headers]:
if client is None:
client = litellm.module_level_aclient
@ -93,7 +97,7 @@ async def make_call(
)
except httpx.HTTPStatusError as e:
error_headers = getattr(e, "headers", None)
error_response: Final = getattr(e, "response", None)
error_response: Final[object] = getattr(e, "response", None)
if error_headers is None and error_response:
error_headers = getattr(error_response, "headers", None)
raise AnthropicError(
@ -138,7 +142,7 @@ def make_sync_call(
json_mode: bool,
speed: str | None = None,
tool_name_reverse_map: dict[str, str] | None = None,
) -> tuple[Any, httpx.Headers]:
) -> tuple["ModelResponseIterator", httpx.Headers]:
if client is None:
client = litellm.module_level_client # re-use a module level client
@ -153,7 +157,7 @@ def make_sync_call(
)
except httpx.HTTPStatusError as e:
error_headers = getattr(e, "headers", None)
error_response: Final = getattr(e, "response", None)
error_response: Final[object] = getattr(e, "response", None)
if error_headers is None and error_response:
error_headers = getattr(error_response, "headers", None)
raise AnthropicError(
@ -292,7 +296,7 @@ class AnthropicChatCompletion(BaseLLM):
status_code: Final = getattr(e, "status_code", 500)
error_headers = getattr(e, "headers", None)
error_text = getattr(e, "text", str(e))
error_response: Final = getattr(e, "response", None)
error_response: Final[object] = getattr(e, "response", None)
if error_headers is None and error_response:
error_headers = getattr(error_response, "headers", None)
if error_response and hasattr(error_response, "text"):
@ -593,7 +597,7 @@ class AnthropicChatCompletion(BaseLLM):
status_code: Final = getattr(e, "status_code", 500)
error_headers = getattr(e, "headers", None)
error_text = getattr(e, "text", str(e))
error_response: Final = getattr(e, "response", None)
error_response: Final[object] = getattr(e, "response", None)
if error_headers is None and error_response:
error_headers = getattr(error_response, "headers", None)
if error_response and hasattr(error_response, "text"):
@ -664,10 +668,10 @@ class ModelResponseIterator:
# Accumulate web_search_tool_result blocks for multi-turn reconstruction
# See: https://github.com/BerriAI/litellm/issues/17737
self.web_search_results: list[dict[str, Any]] = []
self.web_search_results: list[dict[str, object]] = []
# Accumulate compaction blocks for multi-turn reconstruction
self.compaction_blocks: list[dict[str, Any]] = []
self.compaction_blocks: list[dict[str, object]] = []
# Accumulate streamed thinking text so final usage can split reasoning
# tokens from regular output tokens.
@ -727,7 +731,7 @@ class ModelResponseIterator:
str,
ChatCompletionToolCallChunk | None,
list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock],
dict[str, Any],
dict[str, object],
str | None,
]:
"""
@ -735,7 +739,7 @@ class ModelResponseIterator:
"""
text = ""
tool_use: ChatCompletionToolCallChunk | None = None
provider_specific_fields: Final = {}
provider_specific_fields: Final[dict[str, object]] = {}
reasoning_content: str | None = None
content_block: Final = ContentBlockDelta(**chunk)
thinking_blocks: list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock] = []
@ -809,8 +813,8 @@ class ModelResponseIterator:
def _handle_redacted_thinking_content(
self,
content_block_start: ContentBlockStart,
provider_specific_fields: dict[str, Any],
) -> tuple[list[ChatCompletionRedactedThinkingBlock], dict[str, Any]]:
provider_specific_fields: dict[str, object],
) -> tuple[list[ChatCompletionRedactedThinkingBlock], dict[str, object]]:
"""
Handle the redacted thinking content
"""
@ -878,7 +882,7 @@ class ModelResponseIterator:
tool_use: ChatCompletionToolCallChunk | None = None
finish_reason = ""
usage: Usage | None = None
provider_specific_fields: dict[str, Any] = {}
provider_specific_fields: dict[str, object] = {}
reasoning_content: str | None = None
thinking_blocks: list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock] | None = None
@ -1212,7 +1216,7 @@ class ModelResponseIterator:
# Try to parse as valid JSON first
try:
data_json: Final = json.loads(data_str)
data_json: Final = _loads_stream_chunk(data_str)
return self.chunk_parser(chunk=data_json)
except json.JSONDecodeError:
# Switch to accumulation mode and start accumulating
@ -1330,7 +1334,7 @@ class ModelResponseIterator:
str_line = str_line[index:]
if str_line.startswith("data:"):
data_json: Final = json.loads(str_line[5:])
data_json: Final = _loads_stream_chunk(str_line[5:])
return self.chunk_parser(chunk=data_json)
else:
return ModelResponseStream(id=self.response_id)

79
litellm/llms/anthropic/chat/transformation.py
View file

@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Any, Final, NoReturn, cast
import httpx
from pydantic import ValidationError
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm.constants import (
@ -125,7 +126,25 @@ else:
_ANTHROPIC_TOOL_NAME_INVALID_CHARS: Final = re.compile(r"[^a-zA-Z0-9_-]")
_ANTHROPIC_TOOL_NAME_MAX_LEN: Final = 128
_ENUM_TYPE_CHECKS: Final[Mapping[str, Callable[[Any], bool]]] = MappingProxyType(
class _AnthropicUsageIteration(TypedDict, total=False):
"""One entry of the ``usage.iterations`` array on an Anthropic response."""
input_tokens: ReadOnly[int | None]
output_tokens: ReadOnly[int | None]
cache_creation_input_tokens: ReadOnly[int | None]
cache_read_input_tokens: ReadOnly[int | None]
class _AnthropicToolResultBlock(TypedDict, total=False):
"""A ``*_tool_result`` content block on an Anthropic response."""
type: ReadOnly[str]
tool_use_id: ReadOnly[str]
content: ReadOnly[object]
_ENUM_TYPE_CHECKS: Final[Mapping[str, Callable[[object], bool]]] = MappingProxyType(
{
"null": lambda v: v is None,
"boolean": lambda v: isinstance(v, bool),
@ -440,7 +459,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
optional_params.pop("speed", None)
@staticmethod
def _raise_invalid_reasoning_effort(model: str, value: Any, llm_provider: str) -> NoReturn:
def _raise_invalid_reasoning_effort(model: str, value: object, llm_provider: str) -> NoReturn:
"""Raise a ``BadRequestError`` for an unrecognised ``reasoning_effort``.
Args:
@ -1466,7 +1485,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
if _tool_choice is not None:
optional_params["tool_choice"] = _tool_choice
optional_params["tool_choice"] = AnthropicConfig._apply_forced_tool_choice(
model=model, tool_choice=_tool_choice, drop_params=drop_params
)
elif param == "stream" and value is True:
optional_params["stream"] = value
elif param == "stop" and (isinstance(value, str) or isinstance(value, list)):
@ -1992,19 +2013,35 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
return data
def _apply_output_config(self, data: dict, model: str, optional_params: dict) -> None:
"""Validate and apply output_config to the request data."""
"""Validate and apply output_config to the request data.
The ``drop_params`` gate here is an effort gate: ``format`` is a
structured-output field, not an effort field, so it survives the drop
and is vetted where it is consumed (the map's
``supports_native_structured_output`` flag on emission paths).
"""
if "output_config" not in optional_params:
return
output_config: Final = optional_params.get("output_config")
if not output_config or not isinstance(output_config, dict):
return
if litellm.drop_params is True and not self._model_supports_effort_param(model, self._resolved_provider):
if (
litellm.drop_params is True
and any(key != "format" for key in output_config)
and not self._model_supports_effort_param(model, self._resolved_provider)
):
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
model,
)
optional_params.pop("output_config", None)
data.pop("output_config", None)
preserved_format: Final = output_config.get("format")
if preserved_format is None:
optional_params.pop("output_config", None)
data.pop("output_config", None)
return
format_only: Final = {"format": preserved_format} # mutable-ok: json body
optional_params["output_config"] = format_only # rebind-ok: out-param store
data["output_config"] = format_only # rebind-ok: out-param store
return
effort: Final = output_config.get("effort")
valid_efforts: Final = ["high", "medium", "low", "xhigh", "max"]
@ -2059,22 +2096,22 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
self, completion_response: dict
) -> tuple[
str,
list[Any] | None,
list[object] | None,
list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock] | None,
str | None,
list[ChatCompletionToolCallChunk],
list[Any] | None,
list[Any] | None,
list[Any] | None,
list[object] | None,
list[_AnthropicToolResultBlock] | None,
list[object] | None,
]:
text_content = ""
citations: list[Any] | None = None
citations: list[object] | None = None
thinking_blocks: list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock] | None = None
reasoning_content: str | None = None
tool_calls: Final[list[ChatCompletionToolCallChunk]] = []
web_search_results: list[Any] | None = None
tool_results: list[Any] | None = None
compaction_blocks: list[Any] | None = None
web_search_results: list[object] | None = None
tool_results: list[_AnthropicToolResultBlock] | None = None
compaction_blocks: list[object] | None = None
for idx, content in enumerate(completion_response["content"]):
if content["type"] == "text":
text_content += content["text"]
@ -2284,7 +2321,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
raw_speed: Final = _usage.get("speed")
resolved_speed: Final = raw_speed if isinstance(raw_speed, str) else speed
iterations: Final[list[Any] | None] = _usage.get("iterations")
iterations: Final[Sequence[_AnthropicUsageIteration] | None] = _usage.get("iterations")
if iterations:
prompt_tokens = sum(it.get("input_tokens", 0) or 0 for it in iterations)
completion_tokens = sum(it.get("output_tokens", 0) or 0 for it in iterations)
@ -2377,7 +2414,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
def _build_code_interpreter_results(
self,
tool_results: list[Any],
tool_results: Sequence[_AnthropicToolResultBlock],
code_by_id: dict[str, str],
container_id: str | None,
) -> list[OutputCodeInterpreterCall]:
@ -2403,11 +2440,11 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
def _build_provider_specific_fields(
self,
completion_response: dict,
citations: list[Any] | None,
citations: Sequence[object] | None,
thinking_blocks: list[ChatCompletionThinkingBlock | ChatCompletionRedactedThinkingBlock] | None,
web_search_results: list[Any] | None,
tool_results: list[Any] | None,
compaction_blocks: list[Any] | None,
web_search_results: Sequence[object] | None,
tool_results: Sequence[_AnthropicToolResultBlock] | None,
compaction_blocks: Sequence[object] | None,
tool_calls: list[ChatCompletionToolCallChunk],
) -> dict[str, Any]:
provider_specific_fields: Final[dict[str, Any]] = {

52
litellm/llms/anthropic/common_utils.py
View file

@ -28,10 +28,15 @@ from litellm.types.llms.anthropic import (
ANTHROPIC_OAUTH_TOKEN_PREFIX,
AllAnthropicToolsValues,
AnthropicMcpServerTool,
AnthropicMessagesToolChoice,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.proxy.model_listing import ModelInfoResponse
DROP_FORCED_TOOL_CHOICE_WARNING: Final = (
"Downgrading forced tool_choice to 'auto' for model=%s (drop_params=True): this model rejects tool_choice type "
"'any'/'tool' with a 400 because thinking is always on and a forced call would skip it."
)
DROP_DISABLED_THINKING_WARNING: Final = (
"Dropping `thinking={'type': 'disabled'}` for model=%s: thinking is always on for this model and cannot be "
"disabled (the alternative is a provider 400). The model will still think adaptively, its response can contain "
@ -320,6 +325,41 @@ class AnthropicModelInfo(BaseLLMModelInfo):
status_code=400,
)
@staticmethod
def forced_tool_use_downgraded(model: str, drop_params: bool) -> bool:
"""True when the model map flags the model with
``supports_forced_tool_use: false`` (Fable 5.1 / Mythos 5.1 400 on
``any``/``tool``) and ``drop_params`` asks for the ``auto`` downgrade;
raises a clean client-side 400 for such models without ``drop_params``."""
if AnthropicModelInfo._get_model_capability(model, "supports_forced_tool_use") is not False:
return False
if not (litellm.drop_params or drop_params):
raise litellm.utils.UnsupportedParamsError(
message=(
f"{model} does not support forced tool use (tool_choice='required' or a named tool). "
"Use tool_choice='auto' and tell the model in the prompt when to call the tool, or set "
"`litellm.drop_params = True` to downgrade to 'auto' automatically."
),
status_code=400,
)
litellm.verbose_logger.warning(DROP_FORCED_TOOL_CHOICE_WARNING, model)
return True
@staticmethod
def _apply_forced_tool_choice(
model: str,
tool_choice: AnthropicMessagesToolChoice,
drop_params: bool,
) -> AnthropicMessagesToolChoice:
if tool_choice["type"] not in ("any", "tool"):
return tool_choice
if not AnthropicModelInfo.forced_tool_use_downgraded(model, drop_params):
return tool_choice
disable_parallel: Final = tool_choice.get("disable_parallel_tool_use")
if disable_parallel is None:
return AnthropicMessagesToolChoice(type="auto")
return AnthropicMessagesToolChoice(type="auto", disable_parallel_tool_use=disable_parallel)
@staticmethod
def _strip_version_suffix(model: str) -> str:
at: Final = model.rfind("@")
@ -865,13 +905,9 @@ class AnthropicModelInfo(BaseLLMModelInfo):
f"Failed to fetch models from Anthropic. Status code: {response.status_code}, Response: {response.text}"
)
models: Final = response.json()["data"]
models: Final[Sequence[Mapping[str, str]]] = response.json()["data"]
litellm_model_names: Final = []
for model in models:
stripped_model_name = model["id"]
litellm_model_name = "anthropic/" + stripped_model_name
litellm_model_names.append(litellm_model_name)
litellm_model_names: Final = ["anthropic/" + model["id"] for model in models]
return litellm_model_names
def get_token_counter(self) -> BaseTokenCounter | None:
@ -1077,7 +1113,7 @@ def strip_empty_content_blocks_from_anthropic_messages(
return out
def _is_empty_text_block(block: Any) -> bool:
def _is_empty_text_block(block: object) -> bool:
if not isinstance(block, dict) or block.get("type") != "text":
return False
text: Final = block.get("text")
@ -1131,7 +1167,7 @@ def normalize_anthropic_tool_use_id(raw_id: str) -> str:
return sanitized or "tool_use_id"
def _sanitize_tool_use_id_content_block(block: Any) -> Any:
def _sanitize_tool_use_id_content_block(block: object) -> object:
if not isinstance(block, dict):
return block
block_type: Final = block.get("type")

64
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -18,6 +18,24 @@ TOOL_NAME_PREFIX_LENGTH: Final = OPENAI_MAX_TOOL_NAME_LENGTH - TOOL_NAME_HASH_LE
PROVIDERS_PROXYING_AN_UNKNOWN_BACKEND: Final = frozenset({"litellm_proxy"})
def _optional_attr(source: object, name: str) -> object:
return getattr(source, name, None)
def _as_string_mapping(value: object) -> Mapping[str, object] | None:
if isinstance(value, Mapping):
return value
return None
def _thought_signature(provider_specific_fields: object) -> str | None:
fields: Final = _as_string_mapping(provider_specific_fields)
if fields is None:
return None
signature: Final = fields.get("thought_signature")
return signature if isinstance(signature, str) else None
_ANTHROPIC_TOOL_SCHEMA_KEYS: Final = frozenset(
{"name", "type", "input_schema", "description", "cache_control", "strict"}
)
@ -56,7 +74,7 @@ def truncate_tool_name(name: str) -> str:
def create_tool_name_mapping(
tools: list[dict[str, Any]],
tools: Sequence[Mapping[str, object]],
) -> dict[str, str]:
"""
Create a mapping of truncated tool names to original names.
@ -70,6 +88,8 @@ def create_tool_name_mapping(
mapping: Final[dict[str, str]] = {}
for tool in tools:
original_name = tool.get("name", "")
if not isinstance(original_name, str):
continue
truncated_name = truncate_tool_name(original_name)
if truncated_name != original_name:
mapping[truncated_name] = original_name
@ -286,44 +306,44 @@ class LiteLLMAnthropicMessagesAdapter:
### FOR [BETA] `/v1/messages` endpoint support
def _extract_signature_from_tool_call(self, tool_call: Any) -> str | None:
def _extract_signature_from_tool_call(self, tool_call: object) -> str | None:
"""
Extract signature from a tool call's provider_specific_fields.
Only checks provider_specific_fields, not thinking blocks.
"""
signature = None
fields: Final = _optional_attr(tool_call, "provider_specific_fields")
if fields:
return _thought_signature(fields)
if hasattr(tool_call, "provider_specific_fields") and tool_call.provider_specific_fields:
if "thought_signature" in tool_call.provider_specific_fields:
signature = tool_call.provider_specific_fields["thought_signature"]
elif hasattr(tool_call.function, "provider_specific_fields") and tool_call.function.provider_specific_fields:
if "thought_signature" in tool_call.function.provider_specific_fields:
signature = tool_call.function.provider_specific_fields["thought_signature"]
function_fields: Final = _optional_attr(_optional_attr(tool_call, "function"), "provider_specific_fields")
if function_fields:
return _thought_signature(function_fields)
return signature
return None
def _extract_signature_from_tool_use_content(self, content: dict[str, Any]) -> str | None:
def _extract_signature_from_tool_use_content(self, content: Mapping[str, object]) -> str | None:
"""
Extract signature from a tool_use content block's provider_specific_fields.
"""
provider_specific_fields: Final = content.get("provider_specific_fields", {})
provider_specific_fields: Final = _as_string_mapping(content.get("provider_specific_fields", {}))
if provider_specific_fields:
return provider_specific_fields.get("signature")
signature: Final = provider_specific_fields.get("signature")
return signature if isinstance(signature, str) else None
return None
def _add_cache_control_if_applicable(
self,
source: Any,
target: Any,
source: object,
target: object,
model: str | None,
) -> None:
"""
Extract cache_control from source and add to target if it should be preserved.
This method accepts Any type to support both regular dicts and TypedDict objects.
TypedDict objects (like ChatCompletionTextObject, ChatCompletionImageObject, etc.)
are dicts at runtime but have specific types at type-check time. Using Any allows
this method to work with both while maintaining runtime correctness.
This method accepts an unconstrained type to support both regular dicts and
TypedDict objects. TypedDict objects (like ChatCompletionTextObject,
ChatCompletionImageObject, etc.) are dicts at runtime but have specific types at
type-check time, so the widest parameter type works with both.
Args:
source: Dict or TypedDict containing potential cache_control field
@ -751,7 +771,7 @@ class LiteLLMAnthropicMessagesAdapter:
return new_tools, tool_name_mapping
def translate_anthropic_output_format_to_openai(self, output_format: Any) -> dict[str, object] | None:
def translate_anthropic_output_format_to_openai(self, output_format: object) -> dict[str, object] | None:
"""
Translate Anthropic's output_format to OpenAI's response_format.
@ -1366,7 +1386,7 @@ class LiteLLMAnthropicMessagesAdapter:
@classmethod
def _first_positive_prompt_tokens_detail_value(cls, usage: Usage, field_names: tuple[str, ...]) -> int:
prompt_tokens_details: Final = getattr(usage, "prompt_tokens_details", None)
prompt_tokens_details: Final = _optional_attr(usage, "prompt_tokens_details")
if prompt_tokens_details is None:
return 0
@ -1374,7 +1394,7 @@ class LiteLLMAnthropicMessagesAdapter:
if isinstance(prompt_tokens_details, dict):
value = cls._positive_int(prompt_tokens_details.get(field_name))
else:
value = cls._positive_int(getattr(prompt_tokens_details, field_name, None))
value = cls._positive_int(_optional_attr(prompt_tokens_details, field_name))
if value > 0:
return value
return 0

27
litellm/llms/anthropic/experimental_pass_through/context_management/editors/compact.py
View file

@ -14,7 +14,7 @@ Mirrors Anthropic's native ``compact_20260112`` for non-Anthropic providers:
import re
from collections.abc import Awaitable, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Protocol, TypeVar, Union, cast
from typing import TYPE_CHECKING, Final, Literal, Optional, Protocol, TypeVar, Union, cast
from typing_extensions import NotRequired, ReadOnly, TypedDict, Unpack
@ -232,7 +232,7 @@ async def _check_summary_model_access(
key_models: Final = list(getattr(user_api_key_auth, "models", None) or [])
team_id: Final[str | None] = getattr(user_api_key_auth, "team_id", None)
team_model_aliases: Final = getattr(user_api_key_auth, "team_model_aliases", None)
team_model_aliases: Final[dict[str, str] | None] = getattr(user_api_key_auth, "team_model_aliases", None)
team_models: Final = list(getattr(user_api_key_auth, "team_models", None) or [])
user_id: Final[str | None] = getattr(user_api_key_auth, "user_id", None)
project_id: Final[str | None] = getattr(user_api_key_auth, "project_id", None)
@ -443,7 +443,9 @@ async def _check_summary_model_budget(
)
return False
end_user_model_max_budget: Final = getattr(user_api_key_auth, "end_user_model_max_budget", None)
end_user_model_max_budget: Final[dict[str, object] | None] = getattr(
user_api_key_auth, "end_user_model_max_budget", None
)
end_user_id: Final[str | None] = getattr(user_api_key_auth, "end_user_id", None)
if isinstance(end_user_model_max_budget, dict) and end_user_model_max_budget and end_user_id is not None:
try:
@ -854,8 +856,8 @@ def _extract_summary_text(raw: str | None) -> str | None:
def _system_to_openai_message(
system: str | list[dict[str, Any]] | None,
) -> Mapping[str, object] | None:
system: str | list[dict[str, object]] | None,
) -> dict[str, object] | None:
"""Translate Anthropic-shaped ``system`` to an OpenAI system message.
Accepts a bare string or a list of Anthropic content blocks; returns
@ -866,10 +868,10 @@ def _system_to_openai_message(
if isinstance(system, str):
return {"role": "system", "content": system} if system else None
if isinstance(system, list):
parts: Final[tuple[str, ...]] = tuple(
parts: Final[list[object]] = [
block.get("text", "") for block in system if isinstance(block, dict) and block.get("type") == "text"
)
joined: Final = "\n\n".join(part for part in parts if part)
]
joined: Final = "\n\n".join(part for part in parts if isinstance(part, str) and part)
return {"role": "system", "content": joined} if joined else None
return None
@ -951,7 +953,7 @@ async def _call_summary_model(
summary_model: str,
summary_messages: Sequence[Mapping[str, object]],
metadata: Mapping[str, object],
llm_router: object,
llm_router: Optional["Router"],
allowed_model_region: str | None = None,
max_tokens: int = COMPACT_SUMMARY_MAX_TOKENS,
) -> Union["ModelResponse", "CustomStreamWrapper"]:
@ -1036,10 +1038,9 @@ def _extract_usage(response: object) -> tuple[int, int]:
usage: Final[object] = getattr(response, "usage", None)
if usage is None:
return 0, 0
return (
int(getattr(usage, "prompt_tokens", 0) or 0),
int(getattr(usage, "completion_tokens", 0) or 0),
)
prompt_tokens: Final[int | None] = getattr(usage, "prompt_tokens", 0)
completion_tokens: Final[int | None] = getattr(usage, "completion_tokens", 0)
return int(prompt_tokens or 0), int(completion_tokens or 0)
def apply_client_compaction_block_history(

42
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py
View file

@ -179,14 +179,14 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
)
@staticmethod
def _assistant_block_group_key(indexed_block: tuple[int, Mapping[str, Any]]) -> str:
def _assistant_block_group_key(indexed_block: tuple[int, Mapping[str, object]]) -> str:
"""Group a run of consecutive thinking blocks together; keep every other block alone."""
index, block = indexed_block
return "thinking" if block.get("type") == "thinking" else f"block:{index}"
@classmethod
def _assistant_group_to_input_item(
cls, group: tuple[Mapping[str, Any], ...]
cls, group: tuple[Mapping[str, object], ...]
) -> dict[str, Any] | None: # mutable-ok: API message payload
first: Final = group[0]
btype: Final = first.get("type")
@ -206,7 +206,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
def translate_messages_to_responses_input(
self,
messages: list[AllAnthropicPassThroughMessageValues],
) -> list[dict[str, Any]]:
) -> list[dict[str, object]]:
"""
Convert Anthropic messages list to Responses API `input` items.
@ -220,7 +220,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
assistant thinking -> reasoning
assistant tool_use -> function_call
"""
input_items: Final[list[dict[str, Any]]] = []
input_items: Final[list[dict[str, object]]] = []
for m in messages:
if m["role"] == "system":
@ -248,7 +248,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
}
)
elif isinstance(content, list):
user_parts: list[dict[str, Any]] = []
user_parts: list[Mapping[str, object]] = []
tool_image_parts: list[dict[str, Any]] = [] # mutable-ok: json content parts
for block in content:
if not isinstance(block, dict):
@ -379,9 +379,9 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
def translate_tools_to_responses_api(
self,
tools: list[AllAnthropicToolsValues],
) -> list[dict[str, Any]]:
) -> list[dict[str, object]]:
"""Convert Anthropic tool definitions to Responses API function tools."""
result: Final[list[dict[str, Any]]] = []
result: Final[list[dict[str, object]]] = []
for tool in tools:
tool_dict = cast(dict[str, Any], tool)
tool_type = tool_dict.get("type", "")
@ -392,7 +392,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
continue
# Responses turns strict mode on when `strict` is omitted, silently rewriting
# `required` to every property. Anthropic tools are non-strict unless asked.
func_tool: dict[str, Any] = {
func_tool: dict[str, object] = {
"type": "function",
"name": tool_name,
"strict": bool(tool_dict.get("strict")),
@ -407,7 +407,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
@staticmethod
def translate_tool_choice_to_responses_api(
tool_choice: AnthropicMessagesToolChoice,
) -> str | dict[str, Any]:
) -> str | dict[str, object]:
"""Convert Anthropic tool_choice to Responses API tool_choice."""
tc_type: Final = tool_choice.get("type")
if tc_type == "any":
@ -420,8 +420,8 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
@staticmethod
def translate_context_management_to_responses_api(
context_management: dict[str, Any],
) -> list[dict[str, Any]] | None:
context_management: dict[str, object],
) -> list[dict[str, object]] | None:
"""
Convert Anthropic context_management dict to OpenAI Responses API array format.
@ -435,13 +435,13 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
if not isinstance(edits, list):
return None
result: Final[list[dict[str, Any]]] = []
result: Final[list[dict[str, object]]] = []
for edit in edits:
if not isinstance(edit, dict):
continue
edit_type = edit.get("type", "")
if edit_type == "compact_20260112":
entry: dict[str, Any] = {"type": "compaction"}
entry: dict[str, object] = {"type": "compaction"}
trigger = edit.get("trigger")
if isinstance(trigger, dict) and trigger.get("value") is not None:
entry["compact_threshold"] = int(trigger["value"])
@ -451,9 +451,9 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
@staticmethod
def translate_thinking_to_reasoning(
thinking: dict[str, Any],
output_config: dict[str, Any] | None = None,
) -> dict[str, Any] | None:
thinking: dict[str, object],
output_config: dict[str, object] | None = None,
) -> dict[str, object] | None:
"""
Convert Anthropic thinking param to Responses API reasoning param.
@ -473,12 +473,14 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
if isinstance(output_config, dict) and output_config.get("effort"):
effort = output_config["effort"]
elif thinking_type == "enabled":
effort = reasoning_effort_from_thinking_budget(thinking.get("budget_tokens", 0))
raw_budget: Final = thinking.get("budget_tokens", 0)
budget_tokens: Final = int(raw_budget) if isinstance(raw_budget, (int, float)) else 0
effort = reasoning_effort_from_thinking_budget(budget_tokens)
else:
return None
auto_summary: Final = is_reasoning_auto_summary_enabled()
result: Final[dict[str, Any]] = {"effort": effort}
result: Final[dict[str, object]] = {"effort": effort}
summary: Final = thinking.get("summary")
if summary:
result["summary"] = summary
@ -570,7 +572,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
# output_format / output_config.format -> text format
# output_format: {"type": "json_schema", "schema": {...}}
# output_config: {"format": {"type": "json_schema", "schema": {...}}}
output_format: Any = anthropic_request.get("output_format")
output_format: object = anthropic_request.get("output_format")
output_config = anthropic_request.get("output_config")
if not isinstance(output_format, dict) and isinstance(output_config, dict):
output_format = output_config.get("format")
@ -620,7 +622,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
ResponseReasoningItem,
)
content: Final[list[dict[str, Any]]] = []
content: Final[list[dict[str, object]]] = []
stop_reason: AnthropicFinishReason = "end_turn"
for item in response.output:

4
litellm/llms/anthropic/files/handler.py
View file

@ -2,7 +2,7 @@ import asyncio
import json
import time
from collections.abc import Coroutine
from typing import Any, Final
from typing import Final
import httpx
@ -116,7 +116,7 @@ class AnthropicFilesHandler:
api_key: str | None = None,
timeout: float | httpx.Timeout = 600.0,
max_retries: int | None = None,
) -> HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent]:
) -> HttpxBinaryResponseContent | Coroutine[object, object, HttpxBinaryResponseContent]:
"""
Retrieve file content from Anthropic.

16
litellm/llms/azure/azure.py
View file

@ -2,7 +2,7 @@ import asyncio
import json
import time
from collections.abc import Callable, Coroutine
from typing import Any, Final
from typing import Final
import httpx
from openai import (
@ -374,7 +374,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
except Exception as e:
status_code: Final = getattr(e, "status_code", 500)
error_headers = getattr(e, "headers", None)
error_response: Final = getattr(e, "response", None)
error_response: Final[object] = getattr(e, "response", None)
error_body: Final = getattr(e, "body", None)
if error_headers is None and error_response:
error_headers = getattr(error_response, "headers", None)
@ -392,7 +392,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
model: str,
api_base: str,
data: dict,
timeout: Any,
timeout: float | httpx.Timeout,
dynamic_params: bool,
model_response: ModelResponse,
logging_obj: LiteLLMLoggingObj,
@ -502,7 +502,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
dynamic_params: bool,
data: dict[str, object],
model: str,
timeout: Any,
timeout: float | httpx.Timeout,
max_retries: int,
azure_ad_token: str | None = None,
azure_ad_token_provider: Callable | None = None,
@ -578,7 +578,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
dynamic_params: bool,
data: dict,
model: str,
timeout: Any,
timeout: float | httpx.Timeout,
max_retries: int,
azure_ad_token: str | None = None,
azure_ad_token_provider: Callable | None = None,
@ -634,7 +634,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
except Exception as e:
status_code: Final = getattr(e, "status_code", 500)
error_headers = getattr(e, "headers", None)
error_response: Final = getattr(e, "response", None)
error_response: Final[object] = getattr(e, "response", None)
message: Final = getattr(e, "message", str(e))
error_body: Final = getattr(e, "body", None)
if error_headers is None and error_response:
@ -754,7 +754,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
aembedding=None,
headers: dict | None = None,
litellm_params: dict | None = None,
) -> EmbeddingResponse | Coroutine[Any, Any, EmbeddingResponse]:
) -> EmbeddingResponse | Coroutine[object, object, EmbeddingResponse]:
if headers:
optional_params["extra_headers"] = headers
if self._client_session is None:
@ -1268,7 +1268,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
headers["Authorization"] = f"Bearer {azure_ad_token}"
# init AzureOpenAI Client
azure_client_params: Final[dict[str, Any]] = self.initialize_azure_sdk_client(
azure_client_params: Final[dict[str, object]] = self.initialize_azure_sdk_client(
litellm_params=litellm_params or {},
api_key=api_key,
model_name=model or "",

20
litellm/llms/azure_ai/agents/handler.py
View file

@ -51,15 +51,13 @@ else:
AsyncHTTPHandler = Any
class _AzureRawAnnotation(TypedDict, total=False):
type: ReadOnly[str]
class _AzureRawAnnotation(ChatCompletionAnnotation, total=False):
text: ReadOnly[str]
start_index: ReadOnly[int]
end_index: ReadOnly[int]
url_citation: ReadOnly[ChatCompletionAnnotationURLCitation]
_TransformedAnnotation: TypeAlias = ChatCompletionAnnotation | _AzureRawAnnotation
_TransformedAnnotation: TypeAlias = ChatCompletionAnnotation
class _AzureText(TypedDict, total=False):
@ -223,18 +221,11 @@ class AzureAIAgentsHandler:
"""Build the ModelResponse from agent output."""
from litellm.types.utils import Choices, Message, Usage
message_kwargs: Final[dict[str, Any]] = {
"content": content,
"role": "assistant",
}
if annotations:
message_kwargs["annotations"] = annotations
model_response.choices = [
Choices(
finish_reason="stop",
index=0,
message=Message(**message_kwargs),
message=Message(content=content, role="assistant", annotations=annotations or None),
)
]
model_response.model = model
@ -655,9 +646,6 @@ class AzureAIAgentsHandler:
if data_str == "[DONE]":
# Send final chunk with finish_reason
final_delta_kwargs: dict[str, Any] = {"content": None}
if collected_annotations:
final_delta_kwargs["annotations"] = collected_annotations
final_chunk = ModelResponseStream(
id=response_id,
created=created,
@ -667,7 +655,7 @@ class AzureAIAgentsHandler:
StreamingChoices(
finish_reason="stop",
index=0,
delta=Delta(**final_delta_kwargs),
delta=Delta(content=None, annotations=collected_annotations or None),
)
],
)

59
litellm/llms/base_llm/managed_resources/base_managed_resource.py
View file

@ -5,7 +5,8 @@
import base64
import json
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, Final, Generic, TypeVar, cast
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final, Generic, Protocol, TypeVar, cast, runtime_checkable
from litellm import verbose_logger
from litellm.llms.base_llm.managed_resources.isolation import (
@ -38,6 +39,30 @@ else:
ResourceObjectType = TypeVar("ResourceObjectType")
@runtime_checkable
class _HasIdentifier(Protocol):
id: str
class _ManagedResourceRecord(Protocol[ResourceObjectType]):
unified_resource_id: str
resource_object: ResourceObjectType
def model_dump(self) -> dict[str, object]: ...
class _ManagedResourceTable(Protocol[ResourceObjectType]):
async def create(self, *, data: Mapping[str, object]) -> object: ...
async def find_first(self, *, where: Mapping[str, object]) -> _ManagedResourceRecord[ResourceObjectType] | None: ...
async def find_many(
self, *, where: Mapping[str, object], take: int, order: Mapping[str, str]
) -> list[_ManagedResourceRecord[ResourceObjectType]]: ...
async def delete(self, *, where: Mapping[str, object]) -> object: ...
class BaseManagedResource(ABC, Generic[ResourceObjectType]):
"""
Base class for managing resources with target_model_names support.
@ -64,6 +89,9 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
self.internal_usage_cache = internal_usage_cache
self.prisma_client = prisma_client
def _resource_table(self) -> _ManagedResourceTable[ResourceObjectType]:
return getattr(self.prisma_client.db, self.table_name)
# ============================================================================
# ABSTRACT METHODS
# ============================================================================
@ -137,7 +165,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
litellm_parent_otel_span: Span | None,
model_mappings: dict[str, str],
user_api_key_dict: UserAPIKeyAuth,
additional_db_fields: dict[str, Any] | None = None,
additional_db_fields: Mapping[str, object] | None = None,
) -> None:
"""
Store unified resource ID with model mappings in cache and database.
@ -153,7 +181,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
verbose_logger.info("Storing LiteLLM Managed %s with id=%s in cache", self.resource_type, unified_resource_id)
# Prepare cache data
cache_data: Final = {
cache_data: Final[dict[str, object]] = {
"unified_resource_id": unified_resource_id,
"resource_object": resource_object,
"model_mappings": model_mappings,
@ -176,7 +204,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
)
# Prepare database data
db_data: Final = {
db_data: Final[dict[str, object]] = {
"unified_resource_id": unified_resource_id,
"model_mappings": json.dumps(model_mappings),
"flat_model_resource_ids": list(model_mappings.values()),
@ -205,7 +233,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
db_data.update(additional_db_fields)
# Store in database
table: Final = getattr(self.prisma_client.db, self.table_name)
table: Final = self._resource_table()
result: Final = await table.create(data=db_data)
verbose_logger.debug(
@ -240,7 +268,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
return result
# Check database
table: Final = getattr(self.prisma_client.db, self.table_name)
table: Final = self._resource_table()
db_object: Final = await table.find_first(where={"unified_resource_id": unified_resource_id})
if db_object:
@ -264,7 +292,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
The deleted resource object or None if not found
"""
# Get old value from database
table: Final = getattr(self.prisma_client.db, self.table_name)
table: Final = self._resource_table()
initial_value: Final = await table.find_first(where={"unified_resource_id": unified_resource_id})
if initial_value is None:
@ -515,7 +543,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
user_api_key_dict: UserAPIKeyAuth,
limit: int | None = None,
after: str | None = None,
additional_filters: dict[str, Any] | None = None,
additional_filters: Mapping[str, object] | None = None,
) -> dict[str, Any]:
"""
List resources created by a user.
@ -533,7 +561,7 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
if owner_filter is None:
return build_list_page([])
where_clause: Final[dict[str, Any]] = {**owner_filter}
where_clause: Final[dict[str, object]] = {**owner_filter}
if after:
where_clause["id"] = {"gt": after}
@ -544,14 +572,14 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
# Fetch resources
fetch_limit: Final = limit or 20
table: Final = getattr(self.prisma_client.db, self.table_name)
table: Final = self._resource_table()
resources: Final = await table.find_many(
where=where_clause,
take=fetch_limit,
order={"created_at": "desc"},
)
resource_objects: Final[list[Any]] = []
resource_objects: Final[list[object]] = []
for resource in resources:
try:
# Stop once we have enough
@ -559,12 +587,13 @@ class BaseManagedResource(ABC, Generic[ResourceObjectType]):
break
# Parse resource object
resource_data = resource.resource_object
if isinstance(resource_data, str):
resource_data = json.loads(resource_data)
stored_resource = resource.resource_object
resource_data: object = (
json.loads(stored_resource) if isinstance(stored_resource, str) else stored_resource
)
# Set unified ID
if hasattr(resource_data, "id"):
if isinstance(resource_data, _HasIdentifier):
resource_data.id = resource.unified_resource_id
elif isinstance(resource_data, dict):
resource_data["id"] = resource.unified_resource_id

1
litellm/llms/base_llm/ocr/transformation.py
View file

@ -75,6 +75,7 @@ class OCRUsageInfo(LiteLLMPydanticObjectBase):
"""Usage information from OCR response."""
pages_processed: int | None = None
pages_processed_annotation: int | None = None
credits: float | None = None
doc_size_bytes: int | None = None

10
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -588,6 +588,10 @@ class AmazonConverseConfig(BaseConfig):
supported_params.append("context_management")
return supported_params
@staticmethod
def _auto_tool_choice() -> ToolChoiceValuesBlock:
return ToolChoiceValuesBlock(auto={})
def map_tool_choice_values(
self, model: str, tool_choice: str | dict, drop_params: bool
) -> ToolChoiceValuesBlock | None:
@ -600,10 +604,14 @@ class AmazonConverseConfig(BaseConfig):
status_code=400,
)
elif tool_choice == "required":
if AnthropicModelInfo.forced_tool_use_downgraded(model, drop_params):
return self._auto_tool_choice()
return ToolChoiceValuesBlock(any={})
elif tool_choice == "auto":
return ToolChoiceValuesBlock(auto={})
return self._auto_tool_choice()
elif isinstance(tool_choice, dict):
if AnthropicModelInfo.forced_tool_use_downgraded(model, drop_params):
return self._auto_tool_choice()
# only supported for anthropic + mistral models - https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ToolChoice.html
specific_tool: Final = SpecificToolChoiceBlock(
name=make_valid_bedrock_tool_name(tool_choice.get("function", {}).get("name", ""))

44
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -3,7 +3,6 @@ from typing import TYPE_CHECKING, Any, Final
import httpx
from litellm.anthropic_beta_headers_manager import filter_and_transform_beta_headers
from litellm.litellm_core_utils.litellm_logging import verbose_logger
from litellm.litellm_core_utils.prompt_templates.factory import (
convert_to_anthropic_image_obj,
)
@ -16,17 +15,16 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import (
convert_bedrock_invoke_output_format_to_inline_schema,
apply_bedrock_invoke_structured_output,
get_anthropic_beta_from_headers,
normalize_bedrock_opus_output_config_effort,
normalize_custom_field_on_tools,
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
strip_unsupported_bedrock_invoke_output_config_keys,
)
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
from litellm.utils import _supports_factory
if TYPE_CHECKING:
import tiktoken
@ -212,36 +210,14 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
anthropic_request.pop("model", None)
anthropic_request.pop("stream", None)
anthropic_request.pop("stream_chunk_size", None)
output_format: Final = anthropic_request.pop("output_format", None)
output_config_format: Final = pop_bedrock_invoke_output_config_format(anthropic_request)
if output_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_format,
request_body=anthropic_request,
)
elif output_config_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_config_format,
request_body=anthropic_request,
)
if not (
_supports_factory(
model=model,
custom_llm_provider="bedrock",
key="supports_output_config",
)
or AnthropicConfig._model_supports_effort_param(model, "bedrock")
):
if anthropic_request.pop("output_config", None) is not None:
verbose_logger.warning(
"Bedrock Invoke: stripping unsupported `output_config` for "
"model=%s — neither `supports_output_config` nor any "
"`supports_*_reasoning_effort` flag is set in "
"model_prices_and_context_window.json. Add the capability "
"flag to the model JSON entry if this model accepts "
"`output_config`.",
model,
)
apply_bedrock_invoke_structured_output(
model=model,
request_body=anthropic_request,
)
strip_unsupported_bedrock_invoke_output_config_keys(
model=model,
request_body=anthropic_request,
)
if "anthropic_version" not in anthropic_request:
anthropic_request["anthropic_version"] = self.anthropic_version

90
litellm/llms/bedrock/common_utils.py
View file

@ -177,6 +177,95 @@ def convert_bedrock_invoke_output_format_to_inline_schema(
request_body["messages"] = new_messages
def _bedrock_model_supports(model: str, key: str) -> bool:
from litellm.utils import _supports_factory
return _supports_factory(model=model, custom_llm_provider="bedrock", key=key)
def apply_bedrock_invoke_structured_output(
model: str,
request_body: dict[str, object], # mutable-ok: edited in place like siblings
) -> None:
"""
Route Anthropic structured-output params to what the Bedrock model supports.
Consumes the legacy top-level ``output_format`` and the newer
``output_config.format``, keeping the pre-existing precedence of the legacy
field when a request carries both. Models flagged
``supports_native_structured_output`` in the model map get the schema
forwarded as ``output_config.format``, which Bedrock relays to the model for
enforced structured output. For every other model the schema is inlined into
the last user message as best-effort text, with a warning because nothing
enforces it.
"""
legacy_output_format: Final = request_body.pop("output_format", None)
output_config_format: Final = pop_bedrock_invoke_output_config_format(request_body)
schema_format: Final = legacy_output_format if isinstance(legacy_output_format, dict) else output_config_format
if schema_format is None:
return
if _bedrock_model_supports(model, "supports_native_structured_output"):
existing_output_config: Final = request_body.get("output_config")
if isinstance(existing_output_config, dict):
existing_output_config["format"] = schema_format
else:
request_body["output_config"] = {"format": schema_format} # rebind-ok: out-param # mutable-ok: json
return
verbose_logger.warning(
"Bedrock Invoke: model=%s does not advertise `supports_native_structured_output` "
"in model_prices_and_context_window.json, so the JSON schema was inlined into "
"the last user message and is NOT enforced by the model.",
model,
)
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=schema_format,
request_body=request_body,
)
def strip_unsupported_bedrock_invoke_output_config_keys(
model: str,
request_body: dict[str, object], # mutable-ok: edited in place like siblings
) -> None:
"""
Drop ``output_config`` keys the Bedrock model does not accept.
``format`` survives unconditionally: it is only attached for models whose map
entry advertises ``supports_native_structured_output``. Effort-bearing keys
survive only when the map flags ``supports_output_config`` or a
``supports_*_reasoning_effort`` tier; otherwise they are dropped with a
warning so Bedrock does not reject the request.
"""
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
output_config: Final = request_body.get("output_config")
if not isinstance(output_config, dict):
return
if all(key == "format" for key in output_config):
return
if _bedrock_model_supports(model, "supports_output_config") or AnthropicConfig._model_supports_effort_param(
model, "bedrock"
):
return
verbose_logger.warning(
"Bedrock Invoke: stripping unsupported `output_config` keys for "
"model=%s: neither `supports_output_config` nor any "
"`supports_*_reasoning_effort` flag is set in "
"model_prices_and_context_window.json. Add the capability "
"flag to the model JSON entry if this model accepts "
"`output_config`.",
model,
)
preserved_format: Final = output_config.get("format")
if preserved_format is None:
request_body.pop("output_config", None)
else:
request_body["output_config"] = {"format": preserved_format} # rebind-ok: out-param # mutable-ok: json
def normalize_custom_field_on_tools(request_body: dict) -> None:
"""
Drop the ``custom`` field from each tool, first hoisting a boolean
@ -1487,6 +1576,7 @@ class CommonBatchFilesUtils:
aws_role_name=optional_params.get("aws_role_name"),
aws_web_identity_token=optional_params.get("aws_web_identity_token"),
aws_sts_endpoint=optional_params.get("aws_sts_endpoint"),
aws_external_id=optional_params.get("aws_external_id"),
)
# Prepare the request data

1
litellm/llms/bedrock/files/handler.py
View file

@ -113,6 +113,7 @@ class BedrockFilesHandler(BaseAWSLLM):
aws_role_name=optional_params.get("aws_role_name"),
aws_web_identity_token=optional_params.get("aws_web_identity_token"),
aws_sts_endpoint=optional_params.get("aws_sts_endpoint"),
aws_external_id=optional_params.get("aws_external_id"),
)
# Create S3 client

3
litellm/llms/bedrock/files/transformation.py
View file

@ -146,6 +146,7 @@ class _BedrockS3RequestParams(BaseModel):
aws_role_name: str | None = None
aws_web_identity_token: str | None = None
aws_sts_endpoint: str | None = None
aws_external_id: str | None = None
s3_region_name: str | None = None
s3_endpoint_url: str | None = None
@ -1029,6 +1030,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
aws_role_name=optional_params.get("aws_role_name"),
aws_web_identity_token=optional_params.get("aws_web_identity_token"),
aws_sts_endpoint=optional_params.get("aws_sts_endpoint"),
aws_external_id=optional_params.get("aws_external_id"),
)
# Calculate SHA256 hash of the content (REQUIRED for S3)
@ -1290,6 +1292,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
aws_role_name=request_params.aws_role_name,
aws_web_identity_token=request_params.aws_web_identity_token,
aws_sts_endpoint=request_params.aws_sts_endpoint,
aws_external_id=request_params.aws_external_id,
)
empty_body_hash: Final = hashlib.sha256(b"").hexdigest()

60
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -29,14 +29,14 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
AmazonInvokeConfig,
)
from litellm.llms.bedrock.common_utils import (
convert_bedrock_invoke_output_format_to_inline_schema,
apply_bedrock_invoke_structured_output,
ensure_bedrock_anthropic_messages_tool_names,
get_anthropic_beta_from_headers,
is_claude_4_5_on_bedrock,
normalize_bedrock_opus_output_config_effort,
normalize_custom_field_on_tools,
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
strip_unsupported_bedrock_invoke_output_config_keys,
)
from litellm.llms.bedrock.request_metadata import (
bedrock_request_metadata_headers,
@ -51,7 +51,6 @@ from litellm.types.llms.openai import AllMessageValues
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import GenericStreamingChunk, ModelResponseStream
from litellm.types.utils import GenericStreamingChunk as GChunk
from litellm.utils import _supports_factory
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -708,52 +707,25 @@ class AmazonAnthropicClaudeMessagesConfig(
# 4. Remove `ttl` field from cache_control in messages (Bedrock doesn't support it for older models)
self._remove_ttl_from_cache_control(anthropic_messages_request=anthropic_messages_request, model=model)
# 5. Convert structured-output params to inline schema.
# Bedrock Invoke doesn't support top-level `output_format`; its
# accepted `output_config` subset is also narrower than Anthropic's, so
# consume the newer `output_config.format` shape here instead of
# forwarding it as an unknown nested key.
# 5. Route structured-output params (`output_format` /
# `output_config.format`) to native enforcement or the inline-schema
# fallback, then strip `output_config` keys the model does not accept.
# Ref: https://github.com/BerriAI/litellm/issues/22797
existing_output_config: Final = anthropic_messages_request.get("output_config")
if isinstance(existing_output_config, dict):
anthropic_messages_request["output_config"] = dict(existing_output_config)
output_format: Final = anthropic_messages_request.pop("output_format", None)
output_config_format: Final = pop_bedrock_invoke_output_config_format(anthropic_messages_request)
if output_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_format,
request_body=anthropic_messages_request,
)
elif output_config_format:
convert_bedrock_invoke_output_format_to_inline_schema(
output_format=output_config_format,
request_body=anthropic_messages_request,
)
apply_bedrock_invoke_structured_output(
model=model,
request_body=anthropic_messages_request,
)
normalize_bedrock_opus_output_config_effort(
model=model,
output_config=anthropic_messages_request.get("output_config"),
)
# 5a. Bedrock Invoke supports output_config (effort) for Claude 4.6+ models,
# but older models do not — strip it to avoid request rejection.
# Ref: https://github.com/BerriAI/litellm/issues/22797
if not (
_supports_factory(
model=model,
custom_llm_provider="bedrock",
key="supports_output_config",
)
or AnthropicConfig._model_supports_effort_param(model, "bedrock")
):
if anthropic_messages_request.pop("output_config", None) is not None:
verbose_logger.warning(
"Bedrock Invoke: stripping unsupported `output_config` for "
"model=%s — neither `supports_output_config` nor any "
"`supports_*_reasoning_effort` flag is set in "
"model_prices_and_context_window.json. Add the capability "
"flag to the model JSON entry if this model accepts "
"`output_config`.",
model,
)
strip_unsupported_bedrock_invoke_output_config_keys(
model=model,
request_body=anthropic_messages_request,
)
# 5b. Hoist `custom.defer_loading` then drop `custom` (Bedrock doesn't support it)
# Ref: https://github.com/BerriAI/litellm/issues/22847
@ -774,9 +746,11 @@ class AmazonAnthropicClaudeMessagesConfig(
if filtered_betas:
anthropic_messages_request["anthropic_beta"] = filtered_betas
remaining_output_config: Final = anthropic_messages_request.get("output_config")
if (
litellm.drop_params is True
and "output_config" in anthropic_messages_request
and isinstance(remaining_output_config, dict)
and any(key != "format" for key in remaining_output_config)
and not AnthropicConfig._model_supports_effort_param(model, "bedrock")
):
verbose_logger.warning(

6
litellm/llms/bedrock/realtime/transformation.py
View file

@ -7,7 +7,7 @@ Transforms between OpenAI Realtime API format and Bedrock Nova Sonic format.
import base64
import json
import uuid as uuid_lib
from typing import Any, Final, cast
from typing import Final, cast
from pydantic import BaseModel
@ -633,7 +633,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
try:
json_message: Final = json.loads(message)
json_message: Final[dict[str, object]] = json.loads(message)
except json.JSONDecodeError:
verbose_logger.warning("Invalid JSON message: %s", message[:200])
return []
@ -1182,7 +1182,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
# Create a function call arguments done event
# This is a custom event format that matches what clients expect
function_call_event: Final[dict[str, Any]] = {
function_call_event: Final[dict[str, object]] = {
"type": "response.function_call_arguments.done",
"event_id": f"event_{uuid.uuid4()}",
"response_id": current_response_id,

56
litellm/llms/black_forest_labs/image_edit/handler.py
View file

@ -8,9 +8,11 @@ then we poll until the result is ready.
import asyncio
import time
from typing import Any, Final
from collections.abc import Coroutine, Mapping
from typing import Final, Protocol
import httpx
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
@ -33,6 +35,42 @@ from ..common_utils import (
from .transformation import BlackForestLabsImageEditConfig
class _BFLSubmitBody(TypedDict, total=False):
"""Decoded body of the BFL submit response, which hands back a polling URL."""
errors: ReadOnly[object]
polling_url: ReadOnly[str]
class _BFLPollBody(TypedDict, total=False):
"""Decoded body of a BFL polling response."""
status: ReadOnly[str]
class _BFLSubmitResponse(Protocol):
"""The submit call's HTTP response, read for its status, body text and decoded body."""
@property
def status_code(self) -> int: ...
@property
def text(self) -> str: ...
def json(self) -> _BFLSubmitBody: ...
class _BFLPollResponse(Protocol):
"""A polling call's HTTP response, read only for the task status it carries."""
def json(self) -> _BFLPollBody: ...
def _poll_status(response: _BFLPollResponse) -> str | None:
"""Read the task status out of a BFL polling response body."""
return response.json().get("status")
class BlackForestLabsImageEdit:
"""
Black Forest Labs Image Edit handler.
@ -53,10 +91,10 @@ class BlackForestLabsImageEdit:
litellm_params: GenericLiteLLMParams | dict,
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout | None,
extra_headers: dict[str, Any] | None = None,
extra_headers: Mapping[str, object] | None = None,
client: HTTPHandler | AsyncHTTPHandler | None = None,
aimage_edit: bool = False,
) -> ImageResponse | Any:
) -> ImageResponse | Coroutine[object, object, ImageResponse]:
"""
Main entry point for image edit requests.
@ -185,7 +223,7 @@ class BlackForestLabsImageEdit:
litellm_params: GenericLiteLLMParams | dict,
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout | None,
extra_headers: dict[str, Any] | None = None,
extra_headers: Mapping[str, object] | None = None,
client: AsyncHTTPHandler | None = None,
) -> ImageResponse:
"""
@ -281,7 +319,7 @@ class BlackForestLabsImageEdit:
def _poll_for_result_sync(
self,
initial_response: httpx.Response,
initial_response: _BFLSubmitResponse,
headers: dict,
sync_client: HTTPHandler,
max_wait: float = DEFAULT_MAX_POLLING_TIME,
@ -356,8 +394,7 @@ class BlackForestLabsImageEdit:
message=f"Polling failed: {response.text}",
)
data = response.json()
status = data.get("status")
status = _poll_status(response)
verbose_logger.debug("BFL poll status: %s", status)
@ -383,7 +420,7 @@ class BlackForestLabsImageEdit:
async def _poll_for_result_async(
self,
initial_response: httpx.Response,
initial_response: _BFLSubmitResponse,
headers: dict,
async_client: AsyncHTTPHandler,
max_wait: float = DEFAULT_MAX_POLLING_TIME,
@ -447,8 +484,7 @@ class BlackForestLabsImageEdit:
message=f"Polling failed: {response.text}",
)
data = response.json()
status = data.get("status")
status = _poll_status(response)
verbose_logger.debug("BFL poll status: %s", status)

35
litellm/llms/black_forest_labs/image_generation/handler.py
View file

@ -8,9 +8,11 @@ then we poll until the result is ready.
import asyncio
import time
from typing import Any, Final
from collections.abc import Coroutine, Mapping
from typing import Final, Protocol, TypedDict
import httpx
from typing_extensions import ReadOnly
import litellm
from litellm._logging import verbose_logger
@ -33,6 +35,23 @@ from ..common_utils import (
from .transformation import BlackForestLabsImageGenerationConfig
class _BFLTaskPayload(TypedDict, total=False):
"""The body BFL returns for a submitted or polled generation task."""
errors: ReadOnly[object]
polling_url: ReadOnly[str]
status: ReadOnly[str]
class _TaskJsonResponse(Protocol):
def json(self) -> _BFLTaskPayload: ...
def _task_payload(response: _TaskJsonResponse) -> _BFLTaskPayload:
"""The JSON body of a BFL task submission or poll response."""
return response.json()
class BlackForestLabsImageGeneration:
"""
Black Forest Labs Image Generation handler.
@ -53,10 +72,10 @@ class BlackForestLabsImageGeneration:
litellm_params: GenericLiteLLMParams | dict,
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout | None,
extra_headers: dict[str, Any] | None = None,
extra_headers: Mapping[str, str] | None = None,
client: HTTPHandler | AsyncHTTPHandler | None = None,
aimg_generation: bool = False,
) -> ImageResponse | Any:
) -> ImageResponse | Coroutine[object, object, ImageResponse]:
"""
Main entry point for image generation requests.
@ -187,7 +206,7 @@ class BlackForestLabsImageGeneration:
litellm_params: GenericLiteLLMParams | dict,
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout | None,
extra_headers: dict[str, Any] | None = None,
extra_headers: Mapping[str, str] | None = None,
client: AsyncHTTPHandler | None = None,
) -> ImageResponse:
"""
@ -305,7 +324,7 @@ class BlackForestLabsImageGeneration:
# Parse initial response to get polling URL
try:
response_data: Final = initial_response.json()
response_data: Final = _task_payload(initial_response)
except Exception as e:
raise BlackForestLabsError(
status_code=initial_response.status_code,
@ -350,7 +369,7 @@ class BlackForestLabsImageGeneration:
message=f"Polling failed: {response.text}",
)
data = response.json()
data = _task_payload(response)
status = data.get("status")
verbose_logger.debug("BFL poll status: %s", status)
@ -396,7 +415,7 @@ class BlackForestLabsImageGeneration:
# Parse initial response to get polling URL
try:
response_data: Final = initial_response.json()
response_data: Final = _task_payload(initial_response)
except Exception as e:
raise BlackForestLabsError(
status_code=initial_response.status_code,
@ -441,7 +460,7 @@ class BlackForestLabsImageGeneration:
message=f"Polling failed: {response.text}",
)
data = response.json()
data = _task_payload(response)
status = data.get("status")
verbose_logger.debug("BFL poll status: %s", status)

52
litellm/llms/codestral/completion/handler.py
View file

@ -4,9 +4,10 @@
import json
from collections.abc import Callable
from functools import partial
from typing import Final
from typing import Final, Protocol
import httpx
from typing_extensions import NotRequired, ReadOnly, TypedDict
import litellm
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
@ -23,6 +24,53 @@ from litellm.types.utils import TextChoices
from litellm.utils import CustomStreamWrapper, TextCompletionResponse
class _CodestralChoiceMessage(TypedDict):
"""`choices[].message` of a Codestral FIM completion."""
role: ReadOnly[NotRequired[str]]
content: ReadOnly[NotRequired[str | None]]
class _CodestralChoice(TypedDict):
"""One entry of `choices` in a Codestral FIM completion."""
index: ReadOnly[int]
message: ReadOnly[NotRequired[_CodestralChoiceMessage]]
finish_reason: ReadOnly[NotRequired[str | None]]
logprobs: ReadOnly[NotRequired[dict[str, object] | None]]
class _CodestralUsage(TypedDict):
"""Token accounting returned alongside a Codestral FIM completion."""
prompt_tokens: ReadOnly[NotRequired[int]]
completion_tokens: ReadOnly[NotRequired[int]]
total_tokens: ReadOnly[NotRequired[int]]
class _CodestralCompletionResponse(TypedDict):
"""Body returned by the Codestral `/v1/fim/completions` endpoint."""
id: ReadOnly[NotRequired[str]]
created: ReadOnly[NotRequired[int]]
model: ReadOnly[NotRequired[str]]
object: ReadOnly[NotRequired[str]]
usage: ReadOnly[NotRequired[_CodestralUsage]]
choices: ReadOnly[NotRequired[list[_CodestralChoice]]]
class _CodestralHTTPResponse(Protocol):
"""The Codestral completion response as this handler reads it."""
@property
def status_code(self) -> int: ...
@property
def text(self) -> str: ...
def json(self) -> _CodestralCompletionResponse: ...
class TextCompletionCodestralError(Exception):
def __init__(
self,
@ -115,7 +163,7 @@ class CodestralTextCompletion:
def process_text_completion_response(
self,
model: str,
response: httpx.Response,
response: _CodestralHTTPResponse,
model_response: TextCompletionResponse,
stream: bool,
logging_obj: LiteLLMLogging,

14
litellm/llms/dashscope/chat/transformation.py
View file

@ -54,6 +54,9 @@ class DashScopeChatConfig(OpenAIGPTConfig):
dynamic_api_key: Final = api_key or get_secret_str("DASHSCOPE_API_KEY")
return api_base, dynamic_api_key
def _resolve_chat_api_base(self, api_base: str | None) -> str:
return api_base or "https://dashscope.aliyuncs.com/compatible-mode/v1"
def get_complete_url(
self,
api_base: str | None,
@ -66,10 +69,7 @@ class DashScopeChatConfig(OpenAIGPTConfig):
"""
If api_base is not provided, use the default DashScope /chat/completions endpoint.
"""
if not api_base:
api_base = "https://dashscope.aliyuncs.com/compatible-mode/v1"
if not api_base.endswith("/chat/completions"):
api_base = f"{api_base}/chat/completions"
return api_base
resolved_api_base: Final = self._resolve_chat_api_base(api_base)
if resolved_api_base.endswith("/chat/completions"):
return resolved_api_base
return f"{resolved_api_base}/chat/completions"

80
litellm/llms/dashscope/common_utils.py
View file

@ -2,9 +2,89 @@
Common utilities for the DashScope LLM provider.
"""
from typing import TYPE_CHECKING
import httpx
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.secret_managers.main import get_secret_str
if TYPE_CHECKING:
from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig
from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
def get_dashscope_family_embedding_config(custom_llm_provider: str) -> "BaseEmbeddingConfig":
if custom_llm_provider == "qwencloud":
from litellm.llms.dashscope.qwencloud import QwenCloudEmbeddingConfig
return QwenCloudEmbeddingConfig()
if custom_llm_provider == "qwen_ai_platform":
from litellm.llms.dashscope.qwen_ai_platform import (
QwenAIPlatformEmbeddingConfig,
)
return QwenAIPlatformEmbeddingConfig()
from litellm.llms.dashscope.embed.transformation import DashScopeEmbeddingConfig
return DashScopeEmbeddingConfig()
def get_dashscope_family_rerank_config(custom_llm_provider: str) -> "BaseRerankConfig":
if custom_llm_provider == "qwencloud":
from litellm.llms.dashscope.qwencloud import QwenCloudRerankConfig
return QwenCloudRerankConfig()
if custom_llm_provider == "qwen_ai_platform":
from litellm.llms.dashscope.qwen_ai_platform import QwenAIPlatformRerankConfig
return QwenAIPlatformRerankConfig()
from litellm.llms.dashscope.rerank.transformation import DashScopeRerankConfig
return DashScopeRerankConfig()
def get_dashscope_family_image_generation_config(
custom_llm_provider: str,
) -> "BaseImageGenerationConfig":
if custom_llm_provider == "qwencloud":
from litellm.llms.dashscope.qwencloud import QwenCloudImageGenerationConfig
return QwenCloudImageGenerationConfig()
if custom_llm_provider == "qwen_ai_platform":
from litellm.llms.dashscope.qwen_ai_platform import (
QwenAIPlatformImageGenerationConfig,
)
return QwenAIPlatformImageGenerationConfig()
from litellm.llms.dashscope.image_generation.transformation import (
DashScopeImageGenerationConfig,
)
return DashScopeImageGenerationConfig()
def resolve_dashscope_family_api_key(custom_llm_provider: str, api_key: str | None) -> str | None:
if custom_llm_provider == "dashscope":
return api_key or get_secret_str("DASHSCOPE_API_KEY")
return api_key or get_secret_str(f"{custom_llm_provider.upper()}_API_KEY") or get_secret_str("DASHSCOPE_API_KEY")
def missing_dashscope_family_key_message(custom_llm_provider: str) -> str:
if custom_llm_provider == "qwencloud":
return (
"Missing API key for QwenCloud. Set QWENCLOUD_API_KEY or "
"DASHSCOPE_API_KEY environment variable or pass api_key parameter."
)
if custom_llm_provider == "qwen_ai_platform":
return (
"Missing API key for Qwen AI Platform. Set QWEN_AI_PLATFORM_API_KEY or "
"DASHSCOPE_API_KEY environment variable or pass api_key parameter."
)
return "Missing API key for DashScope. Set DASHSCOPE_API_KEY environment variable or pass api_key parameter."
class DashScopeError(BaseLLMException):

5
litellm/llms/dashscope/cost_calculator.py
View file

@ -110,7 +110,7 @@ def _calculate_completion_cost(
return (breakdown.completion_tokens * output_cost) + (breakdown.reasoning_tokens * reasoning_cost)
def cost_per_token(model: str, usage: Usage) -> tuple[float, float]:
def cost_per_token(model: str, usage: Usage, custom_llm_provider: str = "dashscope") -> tuple[float, float]:
"""
Calculate cost per token for Dashscope models.
@ -119,11 +119,12 @@ def cost_per_token(model: str, usage: Usage) -> tuple[float, float]:
Args:
model: Model name without provider prefix
usage: LiteLLM Usage block
custom_llm_provider: The provider id the request resolved to; dashscope or one of its brand aliases
Returns:
Tuple[float, float] - (prompt_cost_in_usd, completion_cost_in_usd)
"""
model_info: Final = get_model_info(model=model, custom_llm_provider="dashscope")
model_info: Final = get_model_info(model=model, custom_llm_provider=custom_llm_provider)
breakdown: Final = _extract_token_breakdown(usage)
raw_tiers: Final = model_info.get("tiered_pricing")
tiered_pricing: Final = raw_tiers if isinstance(raw_tiers, list) else None

26
litellm/llms/dashscope/embed/transformation.py
View file

@ -62,6 +62,17 @@ class DashScopeEmbeddingConfig(BaseEmbeddingConfig):
# for drop_params=False before this method is called.
return optional_params
def _resolve_api_key(self, api_key: str | None) -> str:
resolved_api_key: Final = api_key if api_key is not None else get_secret_str("DASHSCOPE_API_KEY")
if resolved_api_key is None:
raise ValueError(
"DashScope API key is required. Set 'DASHSCOPE_API_KEY' env var or pass api_key explicitly."
)
return resolved_api_key
def _resolve_embedding_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("DASHSCOPE_API_BASE") or DEFAULT_API_BASE
def validate_environment(
self,
headers: dict,
@ -72,17 +83,11 @@ class DashScopeEmbeddingConfig(BaseEmbeddingConfig):
api_key: str | None = None,
api_base: str | None = None,
) -> dict:
if api_key is None:
api_key = get_secret_str("DASHSCOPE_API_KEY")
if api_key is None:
raise ValueError(
"DashScope API key is required. Set 'DASHSCOPE_API_KEY' env var or pass api_key explicitly."
)
default_headers: Final = {
return {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}",
"Authorization": f"Bearer {self._resolve_api_key(api_key)}",
**headers,
}
return {**default_headers, **headers}
def get_complete_url(
self,
@ -93,8 +98,7 @@ class DashScopeEmbeddingConfig(BaseEmbeddingConfig):
litellm_params: dict,
stream: bool | None = None,
) -> str:
base = api_base or get_secret_str("DASHSCOPE_API_BASE") or DEFAULT_API_BASE
base = base.rstrip("/")
base: Final = self._resolve_embedding_api_base(api_base).rstrip("/")
if base.endswith("/embeddings"):
return base
return f"{base}/embeddings"

16
litellm/llms/dashscope/image_generation/transformation.py
View file

@ -91,6 +91,15 @@ class DashScopeImageGenerationConfig(BaseImageGenerationConfig):
mapped[k] = v
return mapped
def _resolve_api_key(self, api_key: str | None) -> str:
resolved_api_key: Final = api_key or get_secret_str("DASHSCOPE_API_KEY")
if not resolved_api_key:
raise ValueError("DASHSCOPE_API_KEY is not set")
return resolved_api_key
def _resolve_image_api_base(self, image_api_base: str | None) -> str:
return image_api_base or get_secret_str("DASHSCOPE_API_BASE_IMAGE") or DEFAULT_API_BASE
def get_complete_url(
self,
api_base: str | None,
@ -103,7 +112,7 @@ class DashScopeImageGenerationConfig(BaseImageGenerationConfig):
image_api_base: Final = (
api_base if api_base and not api_base.rstrip("/").endswith(CHAT_COMPATIBLE_MODE_PATH) else None
)
return image_api_base or get_secret_str("DASHSCOPE_API_BASE_IMAGE") or DEFAULT_API_BASE
return self._resolve_image_api_base(image_api_base)
def validate_environment(
self,
@ -115,10 +124,7 @@ class DashScopeImageGenerationConfig(BaseImageGenerationConfig):
api_key: str | None = None,
api_base: str | None = None,
) -> dict:
final_api_key: Final = api_key or get_secret_str("DASHSCOPE_API_KEY")
if not final_api_key:
raise ValueError("DASHSCOPE_API_KEY is not set")
headers["Authorization"] = f"Bearer {final_api_key}"
headers["Authorization"] = f"Bearer {self._resolve_api_key(api_key)}"
headers["Content-Type"] = "application/json"
return headers

62
litellm/llms/dashscope/qwen_ai_platform.py Normal file
View file

@ -0,0 +1,62 @@
from typing import Final
from litellm.secret_managers.main import get_secret_str
from .chat.transformation import DashScopeChatConfig
from .embed.transformation import DashScopeEmbeddingConfig
from .image_generation.transformation import DashScopeImageGenerationConfig
from .rerank.transformation import DashScopeRerankConfig
QWEN_AI_PLATFORM_API_BASE: Final = "https://dashscope.aliyuncs.com/compatible-mode/v1"
QWEN_AI_PLATFORM_RERANK_API_BASE: Final = "https://dashscope.aliyuncs.com/compatible-api/v1/reranks"
QWEN_AI_PLATFORM_IMAGE_API_BASE: Final = (
"https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation"
)
def _resolve_qwen_ai_platform_api_key(api_key: str | None) -> str | None:
return api_key or get_secret_str("QWEN_AI_PLATFORM_API_KEY") or get_secret_str("DASHSCOPE_API_KEY")
def _require_qwen_ai_platform_api_key(api_key: str | None) -> str:
resolved: Final = _resolve_qwen_ai_platform_api_key(api_key)
if resolved is None:
raise ValueError(
"Qwen AI Platform API key is required. Set 'QWEN_AI_PLATFORM_API_KEY' or 'DASHSCOPE_API_KEY' env var "
"or pass api_key explicitly."
)
return resolved
class QwenAIPlatformChatConfig(DashScopeChatConfig):
def _get_openai_compatible_provider_info(
self, api_base: str | None, api_key: str | None
) -> tuple[str | None, str | None]:
return self._resolve_chat_api_base(api_base), _resolve_qwen_ai_platform_api_key(api_key)
def _resolve_chat_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("QWEN_AI_PLATFORM_API_BASE") or QWEN_AI_PLATFORM_API_BASE
class QwenAIPlatformEmbeddingConfig(DashScopeEmbeddingConfig):
def _resolve_api_key(self, api_key: str | None) -> str:
return _require_qwen_ai_platform_api_key(api_key)
def _resolve_embedding_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("QWEN_AI_PLATFORM_API_BASE") or QWEN_AI_PLATFORM_API_BASE
class QwenAIPlatformRerankConfig(DashScopeRerankConfig):
def _resolve_api_key(self, api_key: str | None) -> str:
return _require_qwen_ai_platform_api_key(api_key)
def _resolve_rerank_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("QWEN_AI_PLATFORM_API_BASE_RERANK") or QWEN_AI_PLATFORM_RERANK_API_BASE
class QwenAIPlatformImageGenerationConfig(DashScopeImageGenerationConfig):
def _resolve_api_key(self, api_key: str | None) -> str:
return _require_qwen_ai_platform_api_key(api_key)
def _resolve_image_api_base(self, image_api_base: str | None) -> str:
return image_api_base or get_secret_str("QWEN_AI_PLATFORM_API_BASE_IMAGE") or QWEN_AI_PLATFORM_IMAGE_API_BASE

62
litellm/llms/dashscope/qwencloud.py Normal file
View file

@ -0,0 +1,62 @@
from typing import Final
from litellm.secret_managers.main import get_secret_str
from .chat.transformation import DashScopeChatConfig
from .embed.transformation import DashScopeEmbeddingConfig
from .image_generation.transformation import DashScopeImageGenerationConfig
from .rerank.transformation import DashScopeRerankConfig
QWENCLOUD_API_BASE: Final = "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
QWENCLOUD_RERANK_API_BASE: Final = "https://dashscope-intl.aliyuncs.com/compatible-api/v1/reranks"
QWENCLOUD_IMAGE_API_BASE: Final = (
"https://dashscope-intl.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation"
)
def _resolve_qwencloud_api_key(api_key: str | None) -> str | None:
return api_key or get_secret_str("QWENCLOUD_API_KEY") or get_secret_str("DASHSCOPE_API_KEY")
def _require_qwencloud_api_key(api_key: str | None) -> str:
resolved: Final = _resolve_qwencloud_api_key(api_key)
if resolved is None:
raise ValueError(
"QwenCloud API key is required. Set 'QWENCLOUD_API_KEY' or 'DASHSCOPE_API_KEY' env var "
"or pass api_key explicitly."
)
return resolved
class QwenCloudChatConfig(DashScopeChatConfig):
def _get_openai_compatible_provider_info(
self, api_base: str | None, api_key: str | None
) -> tuple[str | None, str | None]:
return self._resolve_chat_api_base(api_base), _resolve_qwencloud_api_key(api_key)
def _resolve_chat_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("QWENCLOUD_API_BASE") or QWENCLOUD_API_BASE
class QwenCloudEmbeddingConfig(DashScopeEmbeddingConfig):
def _resolve_api_key(self, api_key: str | None) -> str:
return _require_qwencloud_api_key(api_key)
def _resolve_embedding_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("QWENCLOUD_API_BASE") or QWENCLOUD_API_BASE
class QwenCloudRerankConfig(DashScopeRerankConfig):
def _resolve_api_key(self, api_key: str | None) -> str:
return _require_qwencloud_api_key(api_key)
def _resolve_rerank_api_base(self, api_base: str | None) -> str:
return api_base or get_secret_str("QWENCLOUD_API_BASE_RERANK") or QWENCLOUD_RERANK_API_BASE
class QwenCloudImageGenerationConfig(DashScopeImageGenerationConfig):
def _resolve_api_key(self, api_key: str | None) -> str:
return _require_qwencloud_api_key(api_key)
def _resolve_image_api_base(self, image_api_base: str | None) -> str:
return image_api_base or get_secret_str("QWENCLOUD_API_BASE_IMAGE") or QWENCLOUD_IMAGE_API_BASE

36
litellm/llms/dashscope/rerank/transformation.py
View file

@ -58,19 +58,30 @@ class DashScopeRerankConfig(BaseRerankConfig):
def __init__(self) -> None:
pass
def _resolve_api_key(self, api_key: str | None) -> str:
resolved_api_key: Final = api_key if api_key is not None else get_secret_str("DASHSCOPE_API_KEY")
if resolved_api_key is None:
raise ValueError(
"DashScope API key is required. Set 'DASHSCOPE_API_KEY' env var or pass api_key explicitly."
)
return resolved_api_key
def _resolve_rerank_api_base(self, api_base: str | None) -> str:
if api_base is not None:
return api_base
return get_secret_str("DASHSCOPE_API_BASE_RERANK") or DEFAULT_RERANK_URL
def get_complete_url(
self,
api_base: str | None,
model: str,
optional_params: dict | None = None,
) -> str:
if api_base is None:
api_base = get_secret_str("DASHSCOPE_API_BASE_RERANK") or DEFAULT_RERANK_URL
resolved_api_base: Final = self._resolve_rerank_api_base(api_base)
if resolved_api_base == DEFAULT_RERANK_URL:
return resolved_api_base
if api_base == DEFAULT_RERANK_URL:
return DEFAULT_RERANK_URL
cleaned: Final = api_base.rstrip("/")
cleaned: Final = resolved_api_base.rstrip("/")
if cleaned.endswith("/reranks") or cleaned.endswith("/rerank"):
return cleaned
@ -88,19 +99,12 @@ class DashScopeRerankConfig(BaseRerankConfig):
optional_params: dict | None = None,
litellm_params: Mapping[str, object] | None = None,
) -> dict:
if api_key is None:
api_key = get_secret_str("DASHSCOPE_API_KEY")
if api_key is None:
raise ValueError(
"DashScope API key is required. Set 'DASHSCOPE_API_KEY' env var or pass api_key explicitly."
)
default_headers: Final = {
"Authorization": f"Bearer {api_key}",
return {
"Authorization": f"Bearer {self._resolve_api_key(api_key)}",
"accept": "application/json",
"content-type": "application/json",
**headers,
}
return {**default_headers, **headers}
def get_supported_cohere_rerank_params(self, model: str) -> list:
return ["query", "documents", "top_n", "return_documents"]

39
litellm/llms/deepinfra/rerank/transformation.py
View file

@ -2,10 +2,11 @@
Translate between Cohere's `/rerank` format and Deepinfra's `/rerank` format.
"""
from collections.abc import Mapping
from typing import Any, Final
from collections.abc import Mapping, Sequence
from typing import Final, Protocol
import httpx
from typing_extensions import ReadOnly, TypedDict
from litellm._uuid import uuid
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
@ -24,6 +25,36 @@ from litellm.types.rerank import (
)
class _DeepinfraInferenceStatus(TypedDict, total=False):
"""The ``inference_status`` block of a DeepInfra rerank response."""
status: ReadOnly[str]
runtime_ms: ReadOnly[float]
cost: ReadOnly[float]
tokens_generated: ReadOnly[int]
tokens_input: ReadOnly[int]
class _DeepinfraRerankResponse(TypedDict, total=False):
"""Body of a DeepInfra ``/rerank`` response."""
scores: ReadOnly[Sequence[float]]
input_tokens: ReadOnly[int]
request_id: ReadOnly[str | None]
inference_status: ReadOnly[_DeepinfraInferenceStatus]
class _DeepinfraRerankResponseSource(Protocol):
"""The DeepInfra ``/rerank`` HTTP response, read for the body it decodes to."""
def json(self) -> _DeepinfraRerankResponse: ...
def _deepinfra_rerank_body(response: _DeepinfraRerankResponseSource) -> _DeepinfraRerankResponse:
"""Decode the body of a DeepInfra ``/rerank`` response."""
return response.json()
class DeepinfraRerankConfig(BaseRerankConfig):
"""
Deepinfra Rerank - Follows the same Spec as Cohere Rerank
@ -95,7 +126,7 @@ class DeepinfraRerankConfig(BaseRerankConfig):
model: str,
drop_params: bool,
query: str,
documents: list[str | dict[str, Any]],
documents: list[str | dict[str, object]],
custom_llm_provider: str | None = None,
top_n: int | None = None,
rank_fields: list[str] | None = None,
@ -150,7 +181,7 @@ class DeepinfraRerankConfig(BaseRerankConfig):
litellm_params: dict = {},
) -> RerankResponse:
try:
response_json: Final = raw_response.json()
response_json: Final = _deepinfra_rerank_body(raw_response)
logging_obj.post_call(original_response=raw_response.text)
# Extract the scores from the response

94
litellm/llms/gemini/common_utils.py
View file

@ -2,7 +2,7 @@ import base64
import datetime
import json
import math
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from typing import Any, Final
import httpx
@ -128,24 +128,35 @@ def is_gemini_image_model(model: str) -> bool:
return "gemini" in base_model
def _parse_image_config_string(raw_image_config: str, model: str) -> object:
try:
return json.loads(raw_image_config)
except json.JSONDecodeError as exc:
raise litellm.UnsupportedParamsError(
model=model,
message="`imageConfig` must be valid JSON when provided as a string.",
) from exc
def map_openai_image_params_to_gemini(
params: dict[str, Any],
params: Mapping[str, object],
model: str,
supported_params: Sequence[str],
optional_params: dict[str, Any] | None = None,
optional_params: Mapping[str, object] | None = None,
parse_image_config_string: bool = False,
) -> dict[str, Any]:
optional_params = optional_params or {}
) -> dict[str, object]:
already_mapped: Final[Mapping[str, object]] = optional_params or {}
filtered_params: Final = {key: value for key, value in params.items() if key in supported_params}
mapped_params: Final[dict[str, Any]] = {}
mapped_params: Final[dict[str, object]] = {}
if "n" in filtered_params and "n" not in optional_params:
if "n" in filtered_params and "n" not in already_mapped:
mapped_params["sampleCount"] = filtered_params["n"]
if "size" in filtered_params and "size" not in optional_params:
size_param: Final = filtered_params.get("size")
if isinstance(size_param, str) and "size" not in already_mapped:
image_config: Final = map_openai_size_to_gemini_image_config(
filtered_params["size"],
size_param,
model,
)
if image_config is not None:
@ -156,33 +167,30 @@ def map_openai_image_params_to_gemini(
if "imageSize" in image_config:
mapped_params["imageSize"] = image_config["imageSize"]
image_config_param = filtered_params.get("imageConfig")
if isinstance(image_config_param, str) and parse_image_config_string:
try:
image_config_param = json.loads(image_config_param)
except json.JSONDecodeError as exc:
raise litellm.UnsupportedParamsError(
model=model,
message="`imageConfig` must be valid JSON when provided as a string.",
) from exc
raw_image_config: Final = filtered_params.get("imageConfig")
image_config_param: Final[object] = (
_parse_image_config_string(raw_image_config, model)
if isinstance(raw_image_config, str) and parse_image_config_string
else raw_image_config
)
if isinstance(image_config_param, dict):
mapped_params["imageConfig"] = image_config_param
for key, value in filtered_params.items():
if key not in ("n", "size", "imageConfig", "tools", "web_search_options") and key not in optional_params:
if key not in ("n", "size", "imageConfig", "tools", "web_search_options") and key not in already_mapped:
mapped_params[key] = value
return mapped_params
def _dedupe_gemini_search_tools(tools: list[dict[str, Any]]) -> list[dict[str, Any]]:
def _dedupe_gemini_search_tools(tools: list[dict[str, object]]) -> list[dict[str, object]]:
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
search_tool_keys: Final = VertexGeminiConfig._search_tool_keys()
seen_search_keys: Final[set[str]] = set()
deduped_tools: Final[list[dict[str, Any]]] = []
deduped_tools: Final[list[dict[str, object]]] = []
for tool in tools:
if not isinstance(tool, dict):
@ -203,7 +211,7 @@ def _dedupe_gemini_search_tools(tools: list[dict[str, Any]]) -> list[dict[str, A
return deduped_tools
def _has_gemini_search_tool(tools: list[Any]) -> bool:
def _has_gemini_search_tool(tools: list[object]) -> bool:
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
@ -213,9 +221,9 @@ def _has_gemini_search_tool(tools: list[Any]) -> bool:
def map_gemini_image_tools_params(
non_default_params: dict[str, Any],
mapped_params: dict[str, Any],
) -> dict[str, Any]:
non_default_params: Mapping[str, object],
mapped_params: Mapping[str, object],
) -> dict[str, object]:
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
@ -239,21 +247,24 @@ def map_gemini_image_tools_params(
gemini_config._drop_search_tools_mixed_with_functions(result)
if isinstance(result.get("tools"), list):
result["tools"] = _dedupe_gemini_search_tools(result["tools"])
resolved_tools: Final = result.get("tools")
if isinstance(resolved_tools, list):
result["tools"] = _dedupe_gemini_search_tools(resolved_tools)
return result
def get_gemini_image_web_search_requests(
response_data: dict[str, Any],
response_data: Mapping[str, object],
) -> int | None:
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
grounding_metadata: Final[list[dict[str, Any]]] = []
for candidate in response_data.get("candidates", []):
raw_candidates: Final = response_data.get("candidates")
candidates: Final[list[object]] = raw_candidates if isinstance(raw_candidates, list) else []
grounding_metadata: Final[list[dict[str, object]]] = []
for candidate in candidates:
if not isinstance(candidate, dict):
continue
candidate_grounding = candidate.get("groundingMetadata")
@ -267,13 +278,14 @@ def get_gemini_image_web_search_requests(
def get_gemini_image_generation_config(
model: str,
optional_params: dict[str, Any],
) -> dict[str, Any]:
generation_config: Final[dict[str, Any]] = {"response_modalities": ["IMAGE", "TEXT"]}
optional_params: Mapping[str, object],
) -> dict[str, object]:
generation_config: Final[dict[str, object]] = {"response_modalities": ["IMAGE", "TEXT"]}
image_config: Final[dict[str, Any]] = {}
if isinstance(optional_params.get("imageConfig"), dict):
image_config.update(optional_params["imageConfig"])
raw_image_config: Final = optional_params.get("imageConfig")
image_config: Final[dict[str, object]] = {}
if isinstance(raw_image_config, dict):
image_config.update(raw_image_config)
if not supports_gemini_image_size(model):
image_config.pop("imageSize", None)
@ -398,7 +410,7 @@ class GeminiModelInfo(BaseLLMModelInfo):
f"Failed to fetch models from Gemini. Status code: {response.status_code}, Response: {response.json()}"
)
models: Final = response.json()["models"]
models: Final[list[dict[str, str]]] = response.json()["models"]
litellm_model_names: Final = self.process_model_name(models)
return litellm_model_names
@ -473,12 +485,12 @@ class GoogleAIStudioTokenCounter(BaseTokenCounter):
async def count_tokens(
self,
model_to_use: str,
messages: list[dict[str, Any]] | None,
contents: list[dict[str, Any]] | None,
messages: list[dict[str, object]] | None,
contents: list[dict[str, object]] | None,
deployment: dict[str, Any] | None = None,
request_model: str = "",
tools: list[dict[str, Any]] | None = None,
system: Any | None = None,
tools: list[dict[str, object]] | None = None,
system: object | None = None,
) -> TokenCountResponse | None:
import copy

38
litellm/llms/gemini/files/transformation.py
View file

@ -5,11 +5,13 @@ For vertex ai, check out the vertex_ai/files/handler.py file.
"""
import time
from typing import Any, Final, Literal
from collections.abc import Mapping
from typing import Final, Literal, TypedDict
from urllib.parse import urlparse
import httpx
from openai.types.file_deleted import FileDeleted
from typing_extensions import ReadOnly, Required
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.prompt_templates.common_utils import extract_file_data
@ -18,7 +20,6 @@ from litellm.llms.base_llm.files.transformation import (
BaseFilesConfig,
LiteLLMLoggingObj,
)
from litellm.types.llms.gemini import GeminiCreateFilesResponseObject
from litellm.types.llms.openai import (
AllMessageValues,
CreateFileRequest,
@ -31,6 +32,25 @@ from litellm.types.utils import LlmProviders
from ..common_utils import GeminiModelInfo
class _GeminiFileMetadata(TypedDict, total=False):
name: ReadOnly[str]
uri: ReadOnly[Required[str]]
displayName: ReadOnly[Required[str]]
mimeType: ReadOnly[str]
sizeBytes: ReadOnly[Required[str]]
createTime: ReadOnly[Required[str]]
updateTime: ReadOnly[str]
expirationTime: ReadOnly[str]
sha256Hash: ReadOnly[str]
state: ReadOnly[str]
source: ReadOnly[str]
error: ReadOnly[Mapping[str, object]]
class _GeminiCreateFileResponse(TypedDict):
file: ReadOnly[_GeminiFileMetadata]
class GoogleAIStudioFilesHandler(GeminiModelInfo, BaseFilesConfig):
def __init__(self):
pass
@ -41,14 +61,14 @@ class GoogleAIStudioFilesHandler(GeminiModelInfo, BaseFilesConfig):
def validate_environment(
self,
headers: dict[Any, Any],
headers: dict[str, str],
model: str,
messages: list[AllMessageValues],
optional_params: dict[Any, Any],
litellm_params: dict[Any, Any],
optional_params: dict[str, object],
litellm_params: dict[str, object],
api_key: str | None = None,
api_base: str | None = None,
) -> dict[Any, Any]:
) -> dict[str, str]:
"""
Validate environment and add Gemini API key to headers.
Google AI Studio uses x-goog-api-key header for authentication.
@ -164,9 +184,9 @@ class GoogleAIStudioFilesHandler(GeminiModelInfo, BaseFilesConfig):
Transform Gemini's file upload response into OpenAI-style FileObject
"""
try:
response_json: Final = raw_response.json()
response_json: Final[_GeminiCreateFileResponse] = raw_response.json()
response_object: Final = GeminiCreateFilesResponseObject(**response_json.get("file", {}))
response_object: Final = response_json["file"]
# Extract file information from Gemini response
@ -262,7 +282,7 @@ class GoogleAIStudioFilesHandler(GeminiModelInfo, BaseFilesConfig):
"""
try:
verbose_logger.debug("Retrieve file response: %s", raw_response.text)
response_json: Final = raw_response.json()
response_json: Final[_GeminiFileMetadata] = raw_response.json()
verbose_logger.debug("Response JSON: %s", response_json)
# Map Gemini state to OpenAI status
gemini_state: Final = response_json.get("state", "STATE_UNSPECIFIED")

62
litellm/llms/gemini/interactions/transformation.py
View file

@ -12,9 +12,10 @@ Schema versioning:
litellm.use_legacy_interactions_schema = True. Remove flag after June 8, 2026.
"""
from typing import TYPE_CHECKING, Any, Final
from typing import TYPE_CHECKING, Any, Final, Protocol, TypeAlias
import httpx
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
@ -41,6 +42,53 @@ else:
LiteLLMLoggingObj = Any
_JsonObject: TypeAlias = dict[str, object]
class _InteractionPayload(TypedDict, total=False):
"""JSON body of an Interactions API interaction, keyed as ``InteractionsAPIResponse`` fields."""
id: ReadOnly[str | None]
object: ReadOnly[str | None]
model: ReadOnly[str | None]
agent: ReadOnly[str | None]
status: ReadOnly[str | None]
created: ReadOnly[str | None]
updated: ReadOnly[str | None]
outputs: ReadOnly[list[_JsonObject] | None]
steps: ReadOnly[list[_JsonObject] | None]
usage: ReadOnly[_JsonObject | None]
class _CancelPayload(TypedDict, total=False):
"""JSON body of an Interactions API cancel response."""
id: ReadOnly[str | None]
status: ReadOnly[str | None]
class _InteractionPayloadSource(Protocol):
"""An Interactions API HTTP response, read for the interaction body it decodes to."""
def json(self) -> _InteractionPayload: ...
class _CancelPayloadSource(Protocol):
"""An Interactions API cancel HTTP response, read for the body it decodes to."""
def json(self) -> _CancelPayload: ...
def _interaction_body(response: _InteractionPayloadSource) -> _InteractionPayload:
"""Decode the body of an Interactions API interaction response."""
return response.json()
def _cancel_body(response: _CancelPayloadSource) -> _CancelPayload:
"""Decode the body of an Interactions API cancel response."""
return response.json()
class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
"""
Configuration for Google AI Studio Interactions API.
@ -143,7 +191,7 @@ class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
"""
use_legacy: Final[bool] = litellm.use_legacy_interactions_schema
request_body: Final[dict[str, Any]] = {}
request_body: Final[dict[str, object]] = {}
# Model or Agent (one required)
if model:
@ -189,7 +237,7 @@ class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
and (not isinstance(response_format, dict) or "mime_type" not in response_format)
):
# Wrap the legacy schema into the new polymorphic format.
new_rf: Final[dict[str, Any]] = {
new_rf: Final[dict[str, object]] = {
"type": "text",
"mime_type": response_mime_type,
}
@ -215,7 +263,7 @@ class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
if image_config is not None:
# Move image_config to response_format with type=image.
image_rf: Final[dict[str, Any]] = {"type": "image", **image_config}
image_rf: Final[_JsonObject] = {"type": "image", **image_config}
existing_rf: Final = request_body.get("response_format")
if existing_rf is None:
request_body["response_format"] = image_rf
@ -239,7 +287,7 @@ class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
original_response=raw_response.text,
additional_args={"complete_input_dict": {}},
)
raw_json: Final = raw_response.json()
raw_json: Final = _interaction_body(raw_response)
except Exception:
raise GeminiError(
message=raw_response.text,
@ -290,7 +338,7 @@ class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
logging_obj: LiteLLMLoggingObj,
) -> InteractionsAPIResponse:
try:
raw_json: Final = raw_response.json()
raw_json: Final = _interaction_body(raw_response)
except Exception:
raise GeminiError(
message=raw_response.text,
@ -355,7 +403,7 @@ class GoogleAIStudioInteractionsConfig(BaseInteractionsAPIConfig):
logging_obj: LiteLLMLoggingObj,
) -> CancelInteractionResult:
try:
raw_json: Final = raw_response.json()
raw_json: Final = _cancel_body(raw_response)
except Exception:
raise GeminiError(
message=raw_response.text,

61
litellm/llms/gemini/realtime/transformation.py
View file

@ -7,6 +7,8 @@ from collections import OrderedDict
from collections.abc import Mapping, Sequence
from typing import Any, Final, cast
from typing_extensions import ReadOnly, Required, TypedDict
import litellm
from litellm import verbose_logger
from litellm._uuid import uuid
@ -96,6 +98,23 @@ def _gemini_live_speech_config(voice: object) -> Mapping[str, object] | None:
return VertexGeminiConfig()._map_audio_params({"voice": voice})
class _GeminiLiveSetupEnvelope(TypedDict, total=False):
setup: ReadOnly[BidiGenerateContentSetup]
class _OpenAIRealtimeClientEvent(TypedDict, total=False):
type: ReadOnly[str]
audio: ReadOnly[Required[str]]
session: ReadOnly[dict[str, object]]
item: ReadOnly[dict[str, object]]
def _parse_setup(session_configuration_request: str) -> BidiGenerateContentSetup:
envelope: Final[_GeminiLiveSetupEnvelope] = json.loads(session_configuration_request)
empty_setup: Final[BidiGenerateContentSetup] = {}
return envelope.get("setup", empty_setup)
# Google bills Live transcription at an estimated 25 audio tokens/sec of input and
# 175 text tokens/min of output (ai.google.dev/gemini-api/docs/pricing).
GEMINI_LIVE_TRANSCRIBE_AUDIO_TOKENS_PER_SECOND: Final = 25
@ -130,7 +149,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
return True
@staticmethod
def _usage_detail_alias(details: Any, defaults: dict[str, int]) -> dict[str, Any]:
def _usage_detail_alias(details: Mapping[str, int | None] | None, defaults: dict[str, int]) -> dict[str, int]:
if not isinstance(details, dict):
return dict(defaults)
return {
@ -139,7 +158,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
}
@staticmethod
def _add_pipecat_usage_detail_aliases(usage_dict: dict[str, Any]) -> dict[str, Any]:
def _add_pipecat_usage_detail_aliases(usage_dict: dict[str, Any]) -> dict[str, object]:
usage_dict.setdefault(
"input_token_details",
GeminiRealtimeConfig._usage_detail_alias(
@ -222,8 +241,10 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
if not session_configuration_request:
return False
try:
setup: Final = json.loads(session_configuration_request).get("setup", {})
automatic_detection: Final = setup.get("realtimeInputConfig", {}).get("automaticActivityDetection", {})
setup: Final = _parse_setup(session_configuration_request)
automatic_detection: Final[object] = setup.get("realtimeInputConfig", {}).get(
"automaticActivityDetection", {}
)
return isinstance(automatic_detection, dict) and automatic_detection.get("disabled") is True
except (json.JSONDecodeError, TypeError, AttributeError):
return False
@ -406,7 +427,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
return "TEXT" if GeminiRealtimeConfig._is_text_only_live_model(model) else "AUDIO"
@staticmethod
def _coerce_response_modalities(model: str, modalities: Sequence[Any]) -> tuple[str, ...]:
def _coerce_response_modalities(model: str, modalities: Sequence[object]) -> tuple[str, ...]:
"""Swap responseModalities a Live model cannot produce: TEXT to AUDIO for
audio-only models, AUDIO to TEXT for text-only ones (e.g. transcribe-live)."""
normalized: Final = tuple(
@ -431,7 +452,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
def _handle_session_update(
self,
json_message: dict,
json_message: _OpenAIRealtimeClientEvent,
model: str,
session_configuration_request: str | None,
) -> list[str]:
@ -445,7 +466,8 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
with a 1007, tearing the session down). To carry tools/instructions, send
them on the first session.update before any conversation content.
"""
session_payload = json_message.get("session") or {}
empty_session: Final[dict[str, object]] = {}
session_payload = json_message.get("session") or empty_session
# Normalize GA-remapped fields (``output_modalities``,
# nested ``audio.input.transcription``,
# ``audio.input.turn_detection``) back to their flat beta keys so
@ -486,14 +508,15 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
verbose_logger.debug("Gemini Realtime: Ignoring session.update (setup already sent)")
return []
def _handle_conversation_item(self, json_message: dict) -> list[str]:
def _handle_conversation_item(self, json_message: _OpenAIRealtimeClientEvent) -> list[str]:
"""
Handle conversation.item.create for user text or function call output.
Converts OpenAI format to Gemini's clientContent (for user text) or
toolResponse (for function outputs).
"""
item: Final = json_message.get("item", {})
empty_item: Final[dict[str, object]] = {}
item: Final = json_message.get("item", empty_item)
item_type: Final = item.get("type")
if item_type == "function_call_output":
@ -524,7 +547,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
call_id,
)
function_response: Final[dict[str, Any]] = {"response": output_dict}
function_response: Final[dict[str, object]] = {"response": output_dict}
if self._include_function_response_id() and call_id:
function_response["id"] = call_id
if function_name:
@ -559,7 +582,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
) -> list[str]:
realtime_input_dict: BidiGenerateContentRealtimeInput = {}
try:
json_message: Final = json.loads(message)
json_message: Final[_OpenAIRealtimeClientEvent] = json.loads(message)
except json.JSONDecodeError:
if isinstance(message, bytes):
message_str = message.decode("utf-8", errors="replace")
@ -610,9 +633,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
session_configuration_request: str | None = None,
) -> OpenAIRealtimeStreamSessionEvents:
if session_configuration_request:
session_configuration_request_dict: BidiGenerateContentSetup = json.loads(
session_configuration_request
).get("setup", {})
session_configuration_request_dict: BidiGenerateContentSetup = _parse_setup(session_configuration_request)
else:
session_configuration_request_dict = {}
@ -663,7 +684,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
session_configuration_request_dict: BidiGenerateContentSetup = {}
if session_configuration_request is not None:
try:
session_configuration_request_dict = json.loads(session_configuration_request).get("setup", {})
session_configuration_request_dict = _parse_setup(session_configuration_request)
except json.JSONDecodeError:
session_configuration_request_dict = {}
generation_config: Final = session_configuration_request_dict.get("generationConfig", {})
@ -931,9 +952,9 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
return events
@staticmethod
def get_nested_value(obj: dict, path: str) -> Any:
def get_nested_value(obj: dict, path: str) -> object | None:
keys: Final = path.split(".")
current = obj
current: object = obj
for key in keys:
if isinstance(current, dict) and key in current:
current = current[key]
@ -1011,9 +1032,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
current_response_id = f"resp_{uuid.uuid4()}"
if session_configuration_request:
session_configuration_request_dict: BidiGenerateContentSetup = json.loads(
session_configuration_request
).get("setup", {})
session_configuration_request_dict: BidiGenerateContentSetup = _parse_setup(session_configuration_request)
else:
session_configuration_request_dict = {}
@ -1337,7 +1356,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
session_setup: BidiGenerateContentSetup = {}
if session_configuration_request is not None:
try:
session_setup = json.loads(session_configuration_request).get("setup", {})
session_setup = _parse_setup(session_configuration_request)
except (json.JSONDecodeError, TypeError):
session_setup = {}
tool_call_generation_config = session_setup.get("generationConfig", {}) or {}

36
litellm/llms/gemini/videos/transformation.py
View file

@ -1,4 +1,5 @@
import base64
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final
import httpx
@ -54,8 +55,13 @@ def _convert_image_to_gemini_format(image_file) -> dict[str, str]:
return {"bytesBase64Encoded": base64_encoded, "mimeType": mime_type}
def _json_payload(raw_response: httpx.Response) -> object:
"""Read an HTTP response body as an opaque JSON payload."""
return raw_response.json()
def _usage_video_resolution_from_parameters(
parameters: dict[str, Any],
parameters: Mapping[str, object],
) -> str | None:
"""Normalize Veo ``parameters.resolution`` for usage and cost tracking."""
res: Final = parameters.get("resolution")
@ -97,7 +103,7 @@ class GeminiVideoConfig(BaseVideoConfig):
video_create_optional_params: VideoCreateOptionalRequestParams,
model: str,
drop_params: bool,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
Map OpenAI-style parameters to Veo format.
@ -111,7 +117,7 @@ class GeminiVideoConfig(BaseVideoConfig):
All other params are passed through as-is to support Gemini-specific parameters.
"""
mapped_params: Final[dict[str, Any]] = {}
mapped_params: Final[dict[str, object]] = {}
# Get supported OpenAI params (exclude "model" and "prompt" which are handled separately)
supported_openai_params: Final = self.get_supported_openai_params(model)
@ -312,11 +318,11 @@ class GeminiVideoConfig(BaseVideoConfig):
- status: "processing"
- usage: includes duration_seconds and optional video_resolution for cost calculation
"""
response_data: Final = raw_response.json()
response_data: Final = _json_payload(raw_response)
# Parse response using Pydantic model for type safety
try:
operation_response: Final = GeminiLongRunningOperationResponse(**response_data)
operation_response: Final = GeminiLongRunningOperationResponse.model_validate(response_data)
except Exception as e:
raise ValueError(f"Failed to parse operation response: {e}")
@ -336,7 +342,7 @@ class GeminiVideoConfig(BaseVideoConfig):
model=model,
)
usage_data: Final[dict[str, Any]] = {}
usage_data: Final[dict[str, float | str]] = {}
if request_data:
parameters: Final = request_data.get("parameters", {})
duration: Final = parameters.get("durationSeconds") or DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
@ -367,7 +373,7 @@ class GeminiVideoConfig(BaseVideoConfig):
"""
operation_name: Final = extract_original_video_id(video_id)
url: Final = f"{api_base.rstrip('/')}/v1beta/{operation_name}"
params: Final[dict[str, Any]] = {}
params: Final[dict[str, object]] = {}
return url, params
@ -403,9 +409,9 @@ class GeminiVideoConfig(BaseVideoConfig):
}
}
"""
response_data: Final = raw_response.json()
response_data: Final = _json_payload(raw_response)
# Parse response using Pydantic model for type safety
operation_response: Final = GeminiLongRunningOperationResponse(**response_data)
operation_response: Final = GeminiLongRunningOperationResponse.model_validate(response_data)
operation_name: Final = operation_response.name
is_done: Final = operation_response.done
@ -443,9 +449,9 @@ class GeminiVideoConfig(BaseVideoConfig):
client: Final = litellm.module_level_client
status_response: Final = client.get(url=status_url, headers=headers)
status_response.raise_for_status()
response_data: Final = status_response.json()
response_data: Final = _json_payload(status_response)
operation_response: Final = GeminiLongRunningOperationResponse(**response_data)
operation_response: Final = GeminiLongRunningOperationResponse.model_validate(response_data)
if not operation_response.done:
raise ValueError(
@ -458,7 +464,7 @@ class GeminiVideoConfig(BaseVideoConfig):
generated_samples: Final = operation_response.response.generateVideoResponse.generatedSamples
download_url: Final = generated_samples[0].video.uri
params: Final[dict[str, Any]] = {}
params: Final[dict[str, object]] = {}
return download_url, params
@ -480,7 +486,7 @@ class GeminiVideoConfig(BaseVideoConfig):
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: dict[str, Any] | None = None,
extra_body: Mapping[str, object] | None = None,
) -> tuple[str, dict]:
"""
Video remix is not supported by Veo API.
@ -506,7 +512,7 @@ class GeminiVideoConfig(BaseVideoConfig):
after: str | None = None,
limit: int | None = None,
order: str | None = None,
extra_query: dict[str, Any] | None = None,
extra_query: Mapping[str, object] | None = None,
) -> tuple[str, dict]:
"""
Video list is not supported by Veo API.
@ -547,7 +553,7 @@ class GeminiVideoConfig(BaseVideoConfig):
"""Video delete is not supported."""
raise NotImplementedError("Video delete is not supported by Google Veo.")
def transform_video_create_character_request(self, name, video, api_base, litellm_params, headers):
def transform_video_create_character_request(self, name, video: object, api_base, litellm_params, headers):
raise NotImplementedError("video create character is not supported for Gemini")
def transform_video_create_character_response(self, raw_response, logging_obj):

7
litellm/llms/hosted_vllm/embedding/README.md
View file

@ -4,13 +4,12 @@ VLLM is a superset of OpenAI's `embedding` endpoint.
## `encoding_format`
For OpenAI-compatible embedding calls (including `openai/...` with a custom `api_base` pointing at vLLM), LiteLLM resolves `encoding_format` when it is not set on the request:
For OpenAI-compatible embedding calls (including `openai/...` with a custom `api_base` pointing at vLLM), LiteLLM resolves `encoding_format` when it is not set on the request. `hosted_vllm/...` models use a separate handler that never adds the field on its own, so this resolution applies to the `openai/...`-style routes only:
1. Explicit value on the embedding call (`encoding_format=...`).
2. Model config (`litellm_params.encoding_format` on the proxy `model_list` entry).
3. Environment variable `LITELLM_DEFAULT_EMBEDDING_ENCODING_FORMAT` (e.g. in `.env` or container env).
4. Default **`float`**.
That avoids forwarding `encoding_format=None` to the provider/SDK where some servers behave poorly.
If none of those is set, or the winning value is the literal string `none`, the field is omitted from the upstream request entirely (LiteLLM also bypasses the OpenAI SDK's own base64 default), so OpenAI-compatible servers that reject `encoding_format` keep working.
To pass provider-specific parameters, see [provider-specific params](https://docs.litellm.ai/docs/completion/provider_specific_params).
To pass provider-specific parameters, see [provider-specific params](https://docs.litellm.ai/docs/completion/provider_specific_params).

22
litellm/llms/huggingface/embedding/transformation.py
View file

@ -1,8 +1,9 @@
import json
import os
import time
from collections.abc import Sequence
from copy import deepcopy
from typing import TYPE_CHECKING, Any, Final
from typing import TYPE_CHECKING, Any, Final, Protocol
import httpx
@ -24,6 +25,8 @@ from litellm.utils import token_counter
from ..common_utils import HuggingFaceError, hf_task_list, hf_tasks, output_parser
if TYPE_CHECKING:
import tiktoken
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
LoggingClass = LiteLLMLoggingObj
@ -31,6 +34,12 @@ else:
LoggingClass = Any
class _TokenEncoding(Protocol):
"""Tokenizer handle the caller passes in; only `encode` is used, to count completion tokens."""
def encode(self, text: str, /) -> Sequence[object]: ...
tgi_models_cache = None
conv_models_cache = None
@ -369,7 +378,7 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
model_response: ModelResponse,
task: hf_tasks | None,
optional_params: dict,
encoding: Any,
encoding: "_TokenEncoding | None",
messages: list[AllMessageValues],
model: str,
):
@ -439,9 +448,10 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
if output_text is not None and len(output_text) > 0:
completion_tokens = 0
try:
completion_tokens = len(
encoding.encode(model_response["choices"][0]["message"].get("content", ""))
) ##[TODO] use the llama2 tokenizer here
if encoding is not None:
completion_tokens = len(
encoding.encode(model_response["choices"][0]["message"].get("content", ""))
) ##[TODO] use the llama2 tokenizer here
except Exception:
# this should remain non blocking we should not block a response returning if calculating usage fails
pass
@ -469,7 +479,7 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: "tiktoken.Encoding | None",
api_key: str | None = None,
json_mode: bool | None = None,
) -> ModelResponse:

140
litellm/llms/litellm_proxy/skills/code_execution.py
View file

@ -13,55 +13,109 @@ Generated files are returned directly in the response - no separate storage need
import base64
import json
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from enum import Enum
from typing import Any, Final, Protocol
from typing import Any, Final, Protocol, TypedDict
from typing_extensions import NotRequired, ReadOnly, TypedDict
from typing_extensions import ReadOnly
from litellm._logging import verbose_logger
class _ToolCallFunction(Protocol):
"""Function payload of an assistant tool call."""
name: str | None
arguments: str
class _ToolParameterSchema(TypedDict, total=False):
type: ReadOnly[str]
description: ReadOnly[str]
class _ToolCall(Protocol):
"""Tool call requested by the assistant on a chat completion choice."""
id: str
function: _ToolCallFunction
class _ToolArgumentSchema(TypedDict, total=False):
type: ReadOnly[str]
properties: ReadOnly[Mapping[str, _ToolParameterSchema]]
required: ReadOnly[Sequence[str]]
class _AssistantMessage(Protocol):
"""Assistant message carried by a chat completion choice."""
content: str | None
tool_calls: Sequence[_ToolCall] | None
class _OpenAIToolFunction(TypedDict, total=False):
name: ReadOnly[str]
description: ReadOnly[str]
parameters: ReadOnly[_ToolArgumentSchema]
class _CompletionChoice(Protocol):
"""Single choice of a chat completion response."""
finish_reason: str
message: _AssistantMessage
class _OpenAIToolSpec(TypedDict, total=False):
type: ReadOnly[str]
function: ReadOnly[_OpenAIToolFunction]
class _SandboxFile(TypedDict):
"""File generated inside the sandbox during a code execution run."""
class _AnthropicToolSpec(TypedDict, total=False):
name: ReadOnly[str]
description: ReadOnly[str]
input_schema: ReadOnly[_ToolArgumentSchema]
class _CodeExecutionArguments(TypedDict, total=False):
code: ReadOnly[str]
class _GeneratedFile(TypedDict, total=False):
name: ReadOnly[str]
mime_type: ReadOnly[str]
content_base64: ReadOnly[str]
size: ReadOnly[int]
class _SandboxGeneratedFile(TypedDict):
name: ReadOnly[str]
mime_type: ReadOnly[str]
content_base64: ReadOnly[str]
class _CodeExecutionArguments(TypedDict):
"""Arguments the model passes to the `litellm_code_execution` tool."""
class _SandboxExecutionResult(TypedDict):
success: ReadOnly[bool]
output: ReadOnly[str]
error: ReadOnly[str]
files: ReadOnly[Sequence[_SandboxGeneratedFile]]
code: NotRequired[ReadOnly[str]]
class _ExecutionResult(TypedDict, total=False):
iteration: ReadOnly[int]
success: ReadOnly[bool]
output: ReadOnly[str]
error: ReadOnly[str]
files: ReadOnly[Sequence[str]]
class _ToolCallFunction(Protocol):
name: str
arguments: str
class _ToolCall(Protocol):
id: str
function: _ToolCallFunction
class _AssistantMessage(Protocol):
content: str | None
tool_calls: Sequence[_ToolCall] | None
class _ResponseChoice(Protocol):
message: _AssistantMessage
finish_reason: str | None
class _CompletionResponse(Protocol):
choices: Sequence[_ResponseChoice]
class _CodeExecutionOutcome(TypedDict, total=False):
response: ReadOnly[_CompletionResponse | None]
files: ReadOnly[Sequence[_GeneratedFile]]
execution_results: ReadOnly[Sequence[_ExecutionResult]]
messages: ReadOnly[Sequence[dict[str, object]]]
max_iterations_reached: ReadOnly[bool]
def _parse_code_execution_arguments(serialized_arguments: str) -> _CodeExecutionArguments:
return json.loads(serialized_arguments)
class LiteLLMInternalTools(str, Enum):
@ -75,7 +129,7 @@ class LiteLLMInternalTools(str, Enum):
CODE_EXECUTION = "litellm_code_execution"
def get_litellm_code_execution_tool() -> dict[str, object]:
def get_litellm_code_execution_tool() -> _OpenAIToolSpec:
"""
Returns the litellm_code_execution tool definition in OpenAI format.
@ -96,7 +150,7 @@ def get_litellm_code_execution_tool() -> dict[str, object]:
}
def get_litellm_code_execution_tool_anthropic() -> dict[str, object]:
def get_litellm_code_execution_tool_anthropic() -> _AnthropicToolSpec:
"""
Returns the litellm_code_execution tool definition in Anthropic/messages API format.
@ -143,12 +197,12 @@ class CodeExecutionHandler:
async def execute_with_code_execution(
self,
model: str,
messages: list[dict],
tools: list[dict],
messages: list[dict[str, object]],
tools: list[_OpenAIToolSpec],
skill_files: dict[str, bytes],
skill_id: str | None = None,
**kwargs,
) -> dict[str, object]:
) -> _CodeExecutionOutcome:
"""
Execute an LLM call with automatic code execution handling.
@ -179,8 +233,8 @@ class CodeExecutionHandler:
)
current_messages: Final = list(messages)
generated_files: Final[list[dict[str, object]]] = [] # Files returned directly
execution_results: Final[list[dict[str, object]]] = []
generated_files: Final[list[_GeneratedFile]] = [] # Files returned directly
execution_results: Final[list[_ExecutionResult]] = []
executor: Final = SkillsSandboxExecutor(timeout=self.sandbox_timeout)
response: Any = None # Initialize to avoid possibly unbound error
@ -196,9 +250,9 @@ class CodeExecutionHandler:
**kwargs,
)
choice: _CompletionChoice = response.choices[0]
choice: _ResponseChoice = response.choices[0]
assistant_message = choice.message
stop_reason: str = choice.finish_reason
stop_reason = choice.finish_reason
# Build assistant message for conversation history
assistant_msg_dict: dict[str, object] = {
@ -236,19 +290,19 @@ class CodeExecutionHandler:
if tool_name == LiteLLMInternalTools.CODE_EXECUTION.value:
# Execute code in sandbox
try:
args: _CodeExecutionArguments = json.loads(tool_call.function.arguments)
code: str = args.get("code", "")
args = _parse_code_execution_arguments(tool_call.function.arguments)
code = args.get("code", "")
verbose_logger.debug("CodeExecutionHandler: Executing code (%s chars)", len(code))
exec_result = executor.execute(
exec_result: _SandboxExecutionResult = executor.execute(
code=code,
skill_files=skill_files,
)
verbose_logger.debug("CodeExecutionHandler: Execution result: %s", exec_result)
sandbox_files: Sequence[_SandboxFile] = exec_result["files"]
sandbox_files: Sequence[_SandboxGeneratedFile] = exec_result["files"]
execution_results.append(
{
@ -326,7 +380,7 @@ class CodeExecutionHandler:
}
def has_code_execution_tool(tools: list[dict] | None) -> bool:
def has_code_execution_tool(tools: list[_OpenAIToolSpec] | None) -> bool:
"""Check if litellm_code_execution tool is in the tools list."""
if not tools:
return False
@ -337,7 +391,7 @@ def has_code_execution_tool(tools: list[dict] | None) -> bool:
return False
def add_code_execution_tool(tools: list[dict] | None) -> list[dict]:
def add_code_execution_tool(tools: list[_OpenAIToolSpec] | None) -> list[_OpenAIToolSpec]:
"""Add litellm_code_execution tool if not already present."""
tools = tools or []
if not has_code_execution_tool(tools):

30
litellm/llms/nvidia_riva/audio_transcription/audio_utils.py
View file

@ -16,7 +16,7 @@ import io
import os
import tempfile
from dataclasses import dataclass
from typing import Any, Final, cast
from typing import Final, Protocol, cast
from litellm.llms.nvidia_riva.audio_transcription.transformation import (
RIVA_TARGET_NUM_CHANNELS,
@ -24,10 +24,30 @@ from litellm.llms.nvidia_riva.audio_transcription.transformation import (
)
from litellm.llms.nvidia_riva.common_utils import NvidiaRivaException
# Keep this as Any: the module intentionally avoids importing numpy at module
# import time (optional dependency), and project-wide mypy config evaluates this
# file in contexts where conditional type aliases can degrade to "FloatArray?".
FloatArray = Any
class FloatArray(Protocol):
"""Structural view of the ``numpy.ndarray`` surface this module relies on."""
@property
def ndim(self) -> int: ...
@property
def shape(self) -> tuple[int, ...]: ...
@property
def size(self) -> int: ...
def mean(self, axis: int) -> "FloatArray": ...
def ravel(self) -> "FloatArray": ...
def astype(self, dtype: object) -> "FloatArray": ...
def tobytes(self) -> bytes: ...
def __getitem__(self, key: object) -> "FloatArray": ...
def __mul__(self, other: float) -> "FloatArray": ...
_INSTALL_HINT = "Install Riva STT extras to enable automatic audio resampling: `pip install 'litellm[stt-nvidia-riva]'`"

41
litellm/llms/oci/common_utils.py
View file

@ -5,10 +5,11 @@ import os
import re
from dataclasses import dataclass
from email.utils import formatdate
from typing import Any, Final, Protocol
from typing import Final, Protocol
from urllib.parse import urlparse
import httpx
from pydantic import JsonValue
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@ -64,7 +65,7 @@ class OCISignerProtocol(Protocol):
See: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
"""
def do_request_sign(self, request: Any, *, enforce_content_headers: bool = False) -> None:
def do_request_sign(self, request: "OCIRequestWrapper", *, enforce_content_headers: bool = False) -> None:
pass
@ -113,7 +114,7 @@ def build_signature_string(method: str, path: str, headers: dict, signed_headers
return "\n".join(lines)
def load_private_key_from_str(key_str: str) -> Any:
def load_private_key_from_str(key_str: str) -> "rsa.RSAPrivateKey":
_require_cryptography()
key: Final = serialization.load_pem_private_key(
key_str.encode("utf-8"),
@ -124,7 +125,7 @@ def load_private_key_from_str(key_str: str) -> Any:
return key
def load_private_key_from_file(file_path: str) -> Any:
def load_private_key_from_file(file_path: str) -> "rsa.RSAPrivateKey":
"""Loads a private key from a file path."""
try:
with open(file_path, "r", encoding="utf-8") as f:
@ -421,16 +422,17 @@ OCI_JSON_TO_PYTHON_TYPES: Final[dict[str, str]] = {
}
def resolve_oci_schema_refs(schema: dict[str, Any]) -> dict[str, Any]:
def resolve_oci_schema_refs(schema: JsonValue) -> JsonValue:
"""Inline all ``$ref``/``$defs`` references — OCI does not support JSON Schema ``$ref``."""
defs: Final = schema.get("$defs", {})
resolving_stack: Final[set] = set()
raw_defs: Final = schema.get("$defs") if isinstance(schema, dict) else None
defs: Final[dict[str, JsonValue]] = raw_defs if isinstance(raw_defs, dict) else {}
resolving_stack: Final[set[str]] = set()
def _resolve(obj: Any) -> Any:
def _resolve(obj: JsonValue) -> JsonValue:
if isinstance(obj, dict):
if "$ref" in obj:
ref: Final = obj["$ref"]
if ref.startswith("#/$defs/"):
ref: Final = obj.get("$ref")
if ref is not None:
if isinstance(ref, str) and ref.startswith("#/$defs/"):
key: Final = ref.split("/")[-1]
if key in resolving_stack:
return {"type": "object"} # break cycles
@ -451,7 +453,7 @@ def resolve_oci_schema_refs(schema: dict[str, Any]) -> dict[str, Any]:
return resolved
def resolve_oci_schema_anyof(obj: Any) -> Any:
def resolve_oci_schema_anyof(obj: JsonValue) -> JsonValue:
"""Resolve Pydantic v2 ``Optional[T]`` → ``anyOf`` patterns.
Pydantic v2 emits ``{"anyOf": [{"type": "T"}, {"type": "null"}]}`` for
@ -459,10 +461,13 @@ def resolve_oci_schema_anyof(obj: Any) -> Any:
first non-null branch and merge top-level metadata into it.
"""
if isinstance(obj, dict):
if "anyOf" in obj and "type" not in obj:
non_null: Final = [t for t in obj["anyOf"] if not (isinstance(t, dict) and t.get("type") == "null")]
raw_any_of: Final = obj.get("anyOf")
if raw_any_of is not None and "type" not in obj:
branches: Final = raw_any_of if isinstance(raw_any_of, list) else []
non_null: Final = [t for t in branches if not (isinstance(t, dict) and t.get("type") == "null")]
if non_null:
resolved: Final = {**obj, **non_null[0]}
first: Final = non_null[0]
resolved: Final[dict[str, JsonValue]] = {**obj, **first} if isinstance(first, dict) else {**obj}
resolved.pop("anyOf", None)
return resolve_oci_schema_anyof(resolved)
return {k: resolve_oci_schema_anyof(v) for k, v in obj.items()}
@ -471,7 +476,7 @@ def resolve_oci_schema_anyof(obj: Any) -> Any:
return obj
def sanitize_oci_schema(schema: Any) -> Any:
def sanitize_oci_schema(schema: JsonValue) -> JsonValue:
"""Recursively remove OCI-incompatible fields from a JSON schema.
Strips ``title`` keys, removes ``None``-valued ``default`` entries,
@ -483,7 +488,7 @@ def sanitize_oci_schema(schema: Any) -> Any:
if not isinstance(schema, dict):
return schema
sanitized: Final[dict[str, Any]] = {}
sanitized: Final[dict[str, JsonValue]] = {}
for key, value in schema.items():
if key == "title":
continue
@ -513,7 +518,7 @@ def sanitize_oci_schema(schema: Any) -> Any:
return sanitized
def enrich_cohere_param_description(description: str, param_schema: dict[str, Any]) -> str:
def enrich_cohere_param_description(description: str, param_schema: dict[str, JsonValue]) -> str:
"""Embed schema constraints into a Cohere parameter description.
``CohereParameterDefinition`` only has ``type``, ``description``, and

40
litellm/llms/openai/chat/gpt_transformation.py
View file

@ -170,16 +170,20 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
if model != "gpt-3.5-turbo-16k" and model != "gpt-4": # gpt-4 does not support 'response_format'
model_specific_params.append("response_format")
# Normalize model name for responses API (e.g., "responses/gpt-4.1" -> "gpt-4.1")
model_for_check: Final = model.split("responses/", 1)[1] if "responses/" in model else model
if (
model_for_check in litellm.open_ai_chat_completion_models
) or model_for_check in litellm.open_ai_text_completion_models:
if OpenAIGPTConfig.is_openai_catalog_model(model):
model_specific_params.append(
"user"
) # user is not a param supported by all openai-compatible endpoints - e.g. azure ai
return base_params + model_specific_params
@staticmethod
def is_openai_catalog_model(model: str) -> bool:
model_for_check: Final = model.split("responses/", 1)[1] if "responses/" in model else model
return (
model_for_check in litellm.open_ai_chat_completion_models
or model_for_check in litellm.open_ai_text_completion_models
)
def _map_openai_params(
self,
non_default_params: dict,
@ -321,7 +325,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
@overload
def _transform_messages(
self, messages: list[AllMessageValues], model: str, is_async: Literal[True]
) -> Coroutine[Any, Any, list[AllMessageValues]]:
) -> Coroutine[object, object, list[AllMessageValues]]:
...
@overload
@ -337,7 +341,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
def _transform_messages(
self, messages: list[AllMessageValues], model: str, is_async: bool = False
) -> list[AllMessageValues] | Coroutine[Any, Any, list[AllMessageValues]]:
) -> list[AllMessageValues] | Coroutine[object, object, list[AllMessageValues]]:
"""OpenAI no longer supports image_url as a string, so we need to convert it to a dict"""
stripped_messages: Final = drop_tool_reference_parts_from_tool_messages(messages)
hoisted_messages: Final = hoist_images_from_tool_messages(stripped_messages)
@ -439,6 +443,8 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
optional_params["tools"] = tools
optional_params.pop("max_retries", None)
if not optional_params.get("tools") and not optional_params.get("functions"):
optional_params.pop("tool_choice", None)
return {
"model": model,
@ -469,6 +475,8 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
if tools is not None and len(tools) > 0:
optional_params["tools"] = tools
if self.__class__._is_base_class:
if not optional_params.get("tools") and not optional_params.get("functions"):
optional_params.pop("tool_choice", None)
return {
"model": model,
"messages": transformed_messages,
@ -493,8 +501,12 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
return None
tool_call_names: Final = get_tool_call_names(optional_params.get("tools", []))
try:
json_content: Final = json.loads(content)
if json_content.get("type") == "function" and json_content.get("name") in tool_call_names:
json_content: Final[object] = json.loads(content)
if (
isinstance(json_content, dict)
and json_content.get("type") == "function"
and json_content.get("name") in tool_call_names
):
return ChatCompletionMessageToolCall(
function=Function(
name=json_content.get("name"),
@ -618,7 +630,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
## RESPONSE OBJECT
try:
completion_response: Final = raw_response.json()
completion_response: Final[dict[str, object]] = raw_response.json()
except Exception as e:
response_headers: Final = getattr(raw_response, "headers", None)
raise OpenAIError(
@ -755,6 +767,14 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
)
class OpenAIUnknownModelConfig(OpenAIGPTConfig):
"""A model the openai provider does not recognize is typically a LiteLLM proxy alias, so
forward reasoning_effort and let the server decide whether it is supported."""
def get_supported_openai_params(self, model: str) -> list: # mutable-ok: inherited contract
return super().get_supported_openai_params(model) + ["reasoning_effort"] # mutable-ok: inherited contract
class OpenAIChatCompletionStreamingHandler(BaseModelResponseIterator):
def _map_reasoning_to_reasoning_content(self, choices: list) -> list:
"""

Some files were not shown because too many files have changed in this diff Show more