scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-06 07:48:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 112

Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_team_daily_activity_aggregated

Browse source
This commit is contained in:
ryan-crabbe-berri 2026-08-11 13:53:44 -07:00
commit f09766e2d7
115 changed files with 5844 additions and 1973 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
.github/ISSUE_TEMPLATE/feature_request.yml vendored
View file

@ -24,10 +24,53 @@ body:
validations:
required: true
- type: textarea
id: motivation
id: user-flow
attributes:
label: Motivation, pitch
description: Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., "I'm working on X and would like Y to be possible". If this is related to another GitHub issue, please link here too.
label: User Flow
description: |
Two ordered lists, "Before this feature (today)" and "After this feature (ideal user flow)", walking the same end user through the same task, written strictly from that user's seat. Every rule below applies.
- Describe the real application and the routes its users actually hit, not a generic scenario. Link any related GitHub issue or provider API docs
- Lead each list with one plain sentence saying where the flow dead-ends today and what it would let them do instead, then number the steps
- Every step is something the user does or observes: the HTTP method and full URL they hit, what they sent, and what visibly came back (status code, error text, the shape of an ID). UI steps name the page URL and what is on screen
- No LiteLLM internals: never name functions, files, DB tables, config classes, hooks, callbacks, or code paths. Ask for the behavior you need, not the implementation you imagine
- Keep the two lists step-for-step identical until they diverge, so the missing capability is obvious
- "Before this feature" is also where you show the workaround you're living with, which is what tells us how badly this is needed
placeholder: |
Before this feature (today): a developer batching nightly summaries has no way to mark those calls as low priority, so they compete with live traffic for the same rate limit
1. They send POST https://litellm-domain/v1/chat/completions for 500 documents in a loop
2. Around document 120 they start getting 429s naming the rpm limit, and their user-facing chat app starts getting them too
3. Their workaround is a hand-rolled sleep between calls, which stretches the batch to 3 hours and still collides at peak
After this feature (ideal user flow): the same batch runs as background work that yields to live traffic
1. The developer sends the same POST with "service_tier": "flex"
2. Batch calls queue behind interactive ones instead of 429ing, and the response comes back with the tier it was served at
3. The live chat app keeps returning 200s throughout the batch
4. https://litellm-domain/ui/?page=logs shows the batch requests tagged with that tier
validations:
required: true
- type: textarea
id: how-far-you-got
attributes:
label: How far you got
description: |
Run as many steps of the "After this feature (ideal user flow)" list as you can against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), then paste the commands (e.g., curl) and their full output, ending at the step that dead-ends. Every rule below applies.
- Say plainly what stopped you there, in user terms: the option you passed came back ignored, the response 400'd naming an unsupported field, there is no button on the page for it. This is what proves the feature is genuinely missing rather than undocumented
- No mocks. Where the flow involves a provider call, hit the real provider API, even if it costs real $. `pytest` commands are not enough
- Include the config.yaml (or SDK setup) and env vars the proxy ran with, plus the version or commit you were on. Keep the real values for env vars that aren't sensitive, and redact only the secrets: never paste a real API key, virtual key, database URL, or other credential, here or anywhere else in the issue
- If the provider already supports this, link their API docs and paste a direct call to them succeeding, so we can see the shape LiteLLM should be sending
- For UI asks: include screenshots of the page you got stuck on and its URL. Scrub keys and tokens out of screenshots too (for example, the virtual key is briefly shown in the panel right after you create a virtual key)
placeholder: |
Config / setup the proxy ran with:
Version or commit:
Commands and their full output, up to the step that dead-ends:
What stopped me there:
validations:
required: true
- type: dropdown

25
.github/scripts/triage_with_llm.py vendored
View file

@ -597,6 +597,13 @@ def build_issue_prompt(*, title: str, body: str) -> str:
that it does not today).
- Motivation / use case with a concrete example (config, API call,
UI flow, or scenario showing what's blocked today).
- END-TO-END EVIDENCE OF THE DEAD-END (set
`has_dead_end_evidence=true` only when this is present): a video,
a screenshot, or the exact command(s) actually run paired with
their real output, showing the point where the flow stops today.
Mocked or stubbed dependencies do NOT count, and an unfilled
template scaffold (bare headings, empty numbered lists) counts as
absent.
For an issue that is neither a bug report nor a feature request (a
question, support request, or discussion), PASS as long as it has a
@ -610,6 +617,7 @@ def build_issue_prompt(*, title: str, body: str) -> str:
"has_repro": boolean,
"has_expected_vs_actual": boolean,
"has_motivation_example": boolean,
"has_dead_end_evidence": boolean,
"missing": ["plain-english strings naming what is missing"],
"explanation": "1-2 sentence reasoning for the team to skim"
}}
@ -707,6 +715,10 @@ _ISSUE_BUG_LABELS: tuple[tuple[str, str], ...] = (
)
_ISSUE_FEATURE_LABELS: tuple[tuple[str, str], ...] = (
("has_motivation_example", "Motivation and concrete example"),
(
"has_dead_end_evidence",
"End-to-end evidence of the dead-end (video, screenshot, or command + real output)",
),
)
@ -838,8 +850,11 @@ def format_issue_close_comment(verdict: dict) -> str:
"video, a screenshot, or the exact commands you ran with their real output / "
"traceback) plus expected vs. actual behavior. Written steps with no run output, "
"video, or screenshot don't count, and mocked or stubbed runs don't count.\n"
" - For **feature requests**: a concrete description of what should change, plus a "
"use case and example (config / API call / UI flow).\n"
" - For **feature requests**: a concrete description of what should change, a "
"use case and example (config / API call / UI flow), plus end-to-end evidence of "
"the dead-end (a video, a screenshot, or the exact commands you ran with their "
"real output showing where the flow stops today). Mocked or stubbed runs don't "
"count.\n"
"2. Comment `@agent-shin reconsider`. I'll re-run triage and reopen the issue if it "
"now meets the bar. (GitHub doesn't let external authors reopen an issue a maintainer "
"or bot closed, so the comment-based reconsider is the reliable path.)\n"
@ -945,8 +960,10 @@ def format_grace_warning_issue_comment(verdict: dict) -> str:
"screenshot, or the exact commands you ran with their real output / traceback) plus "
"expected vs. actual behavior. Written steps with no run output don't count, and "
"mocked or stubbed runs don't count.\n"
"- For **feature requests**: a concrete description of what should change, plus a use "
"case and example (config / API call / UI flow).\n"
"- For **feature requests**: a concrete description of what should change, a use "
"case and example (config / API call / UI flow), plus end-to-end evidence of the "
"dead-end (a video, a screenshot, or the exact commands you ran with their real "
"output showing where the flow stops today). Mocked or stubbed runs don't count.\n"
"\n"
"**If the issue does get auto-closed in 2 hours**, comment `@agent-shin reconsider` "
"and I'll re-evaluate. If it now meets the bar, I'll reopen the issue.\n"

11
.github/workflows/test-linting.yml vendored
View file

@ -43,9 +43,10 @@ jobs:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
MERGE_BASE=$(gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
MERGE_BASE=$(retry gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
test -n "$MERGE_BASE"
git fetch --no-tags --depth=1 origin "$MERGE_BASE"
retry git fetch --no-tags --depth=1 origin "$MERGE_BASE"
echo "GATE_BASE_SHA=$MERGE_BASE" >> "$GITHUB_ENV"
- name: Set up Python
@ -161,7 +162,8 @@ jobs:
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
git fetch --no-tags --depth=1 origin "$BASE_SHA"
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
retry git fetch --no-tags --depth=1 origin "$BASE_SHA"
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@ -205,7 +207,8 @@ jobs:
GITGUARDIAN_API_KEY: ${{ secrets.GITGUARDIAN_API_KEY }}
run: |
if [ -n "$GITGUARDIAN_API_KEY" ]; then
git fetch --no-tags --unshallow origin
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
retry git fetch --no-tags --unshallow origin
uv tool run --from 'ggshield==1.48.0' ggshield secret scan repo .
else
echo "GITGUARDIAN_API_KEY not set, skipping ggshield scan"

2
CLAUDE.md
View file

@ -31,7 +31,7 @@ When creating PRs, don't set base to `main`. `litellm_internal_staging` is the d
When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule
Same applies for filing bug reports and .github/ISSUE_TEMPLATE/bug_report.yml
Same applies for filing bug reports and feature requests, with .github/ISSUE_TEMPLATE/bug_report.yml and .github/ISSUE_TEMPLATE/feature_request.yml, respectively
If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank

34
basedpyright-code-budget.json
View file

@ -1,15 +1,15 @@
{
"reportAny": {
"limit": 26391
"limit": 23919
},
"reportArgumentType": {
"limit": 2614
"limit": 2580
},
"reportAssignmentType": {
"limit": 327
"limit": 323
},
"reportAttributeAccessIssue": {
"limit": 514
"limit": 488
},
"reportCallIssue": {
"limit": 114
@ -18,13 +18,13 @@
"limit": 40
},
"reportDeprecated": {
"limit": 215
"limit": 213
},
"reportDuplicateImport": {
"limit": 19
},
"reportExplicitAny": {
"limit": 8319
"limit": 7573
},
"reportFunctionMemberAccess": {
"limit": 7
@ -54,10 +54,10 @@
"limit": 0
},
"reportMissingParameterType": {
"limit": 5825
"limit": 5719
},
"reportMissingTypeArgument": {
"limit": 15695
"limit": 15657
},
"reportMissingTypeStubs": {
"limit": 40
@ -72,7 +72,7 @@
"limit": 0
},
"reportOptionalMemberAccess": {
"limit": 1077
"limit": 1069
},
"reportOptionalOperand": {
"limit": 0
@ -99,37 +99,37 @@
"limit": 0
},
"reportUnknownArgumentType": {
"limit": 44996
"limit": 44832
},
"reportUnknownLambdaType": {
"limit": 113
},
"reportUnknownMemberType": {
"limit": 39643
"limit": 39269
},
"reportUnknownParameterType": {
"limit": 20132
"limit": 19988
},
"reportUnknownVariableType": {
"limit": 31153
"limit": 30923
},
"reportUnnecessaryCast": {
"limit": 118
},
"reportUnnecessaryComparison": {
"limit": 701
"limit": 699
},
"reportUnnecessaryContains": {
"limit": 5
},
"reportUnnecessaryIsInstance": {
"limit": 857
"limit": 853
},
"reportUntypedBaseClass": {
"limit": 0
},
"reportUntypedFunctionDecorator": {
"limit": 33
"limit": 27
},
"reportUnusedClass": {
"limit": 23
@ -138,7 +138,7 @@
"limit": 139
},
"reportUnusedImport": {
"limit": 555
"limit": 545
},
"reportUnusedVariable": {
"limit": 146

5
litellm-proxy-extras/litellm_proxy_extras/migrations/20260810000000_add_verificationtoken_settings_updated_at/migration.sql Normal file
View file

@ -0,0 +1,5 @@
-- AlterTable
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "settings_updated_at" TIMESTAMP(3);
-- AlterTable
ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN IF NOT EXISTS "settings_updated_at" TIMESTAMP(3);

2
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -452,6 +452,7 @@ model LiteLLM_VerificationToken {
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")
updated_by String?
settings_updated_at DateTime? @map("settings_updated_at")
last_active DateTime? // When this key was last used
rotation_count Int? @default(0) // Number of times key has been rotated
auto_rotate Boolean? @default(false) // Whether this key should be auto-rotated
@ -548,6 +549,7 @@ model LiteLLM_DeletedVerificationToken {
created_by String? // Original creator
updated_at DateTime? // Last update timestamp before deletion
updated_by String? // Last user who updated before deletion
settings_updated_at DateTime? // Last configuration change before deletion
last_active DateTime? // When this key was last used before deletion
rotation_count Int? @default(0)
auto_rotate Boolean? @default(false)

28
litellm/completion_extras/litellm_responses_transformation/handler.py
View file

@ -22,6 +22,7 @@ class ResponsesToCompletionBridgeHandlerInputKwargs(TypedDict):
model_response: "ModelResponse"
logging_obj: "LiteLLMLoggingObj"
custom_llm_provider: str
encoding: object
class ResponsesToCompletionBridgeHandler:
@ -102,35 +103,37 @@ class ResponsesToCompletionBridgeHandler:
from litellm import LiteLLMLoggingObj
from litellm.types.utils import ModelResponse
model: Final = kwargs.get("model")
typed_kwargs: Final[dict[str, object]] = kwargs
model: Final = typed_kwargs.get("model")
if model is None or not isinstance(model, str):
raise ValueError("model is required")
custom_llm_provider: Final = kwargs.get("custom_llm_provider")
custom_llm_provider: Final = typed_kwargs.get("custom_llm_provider")
if custom_llm_provider is None or not isinstance(custom_llm_provider, str):
raise ValueError("custom_llm_provider is required")
messages: Final = kwargs.get("messages")
messages: Final = typed_kwargs.get("messages")
if messages is None or not isinstance(messages, list):
raise ValueError("messages is required")
optional_params: Final = kwargs.get("optional_params")
optional_params: Final = typed_kwargs.get("optional_params")
if optional_params is None or not isinstance(optional_params, dict):
raise ValueError("optional_params is required")
litellm_params: Final = kwargs.get("litellm_params")
litellm_params: Final = typed_kwargs.get("litellm_params")
if litellm_params is None or not isinstance(litellm_params, dict):
raise ValueError("litellm_params is required")
headers: Final = kwargs.get("headers")
headers: Final = typed_kwargs.get("headers")
if headers is None or not isinstance(headers, dict):
raise ValueError("headers is required")
model_response: Final = kwargs.get("model_response")
model_response: Final = typed_kwargs.get("model_response")
if model_response is None or not isinstance(model_response, ModelResponse):
raise ValueError("model_response is required")
logging_obj: Final = kwargs.get("logging_obj")
logging_obj: Final = typed_kwargs.get("logging_obj")
if logging_obj is None or not isinstance(logging_obj, LiteLLMLoggingObj):
raise ValueError("logging_obj is required")
@ -143,6 +146,7 @@ class ResponsesToCompletionBridgeHandler:
model_response=model_response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
encoding=typed_kwargs.get("encoding"),
)
def completion(
@ -205,7 +209,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)
@ -230,7 +234,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)
@ -303,7 +307,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)
@ -328,7 +332,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)

101
litellm/completion_extras/litellm_responses_transformation/transformation.py
View file

@ -4,8 +4,8 @@ Handler for transforming /chat/completions api requests to litellm.responses req
import json
import os
from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Mapping
from typing import TYPE_CHECKING, Any, Final, Literal, Union, cast
from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, TypedDict, Union, cast
from openai.types.responses.custom_tool_param import CustomToolParam
from openai.types.responses.response_input_param import (
@ -45,6 +45,9 @@ from litellm.types.utils import GenericStreamingChunk, ModelResponseStream
if TYPE_CHECKING:
from openai.types.responses import ResponseInputImageParam
from openai.types.responses.response_text_config_param import (
ResponseTextConfigParam as ResponseText,
)
from pydantic import BaseModel
from litellm import LiteLLMLoggingObj, ModelResponse
@ -57,6 +60,19 @@ if TYPE_CHECKING:
ChatCompletionThinkingBlock,
OpenAIMessageContentListBlock,
)
from litellm.types.utils import Choices
class _ReasoningSummaryText(TypedDict):
type: str
text: str
class _BuiltReasoningItem(TypedDict):
type: Literal["reasoning"]
id: str
encrypted_content: str | None
summary: Sequence[_ReasoningSummaryText]
def _get_reasoning_items(
@ -72,13 +88,13 @@ def _get_reasoning_items(
def _build_reasoning_item(
item_id: str,
encrypted_content: str | None,
summary_raw: Any,
) -> dict[str, Any]:
summary_raw: Iterable[object] | None,
) -> _BuiltReasoningItem:
"""Build a ChatCompletionReasoningItem-shaped dict from raw response data.
Handles both pydantic objects (attribute access) and plain dicts.
"""
summary: Final[list[dict[str, Any]]] = []
summary: Final[list[_ReasoningSummaryText]] = []
for s in summary_raw or []:
if isinstance(s, dict):
summary.append({"type": s.get("type", "summary_text"), "text": s.get("text", "")})
@ -98,7 +114,7 @@ def _build_reasoning_item(
class _ChatToolCallDict(ChatCompletionToolCallChunk, total=False):
provider_specific_fields: Mapping[str, Any]
provider_specific_fields: Mapping[str, object]
def _tool_call_dict_from_output_item(item: Mapping[str, Any], index: int) -> _ChatToolCallDict:
@ -142,10 +158,10 @@ def _flat_responses_tool_choice(choice_type: str, name: str) -> ToolChoiceFuncti
def _reasoning_item_to_response_input(
r_item: ChatCompletionReasoningItem | dict[str, Any],
) -> dict[str, Any]:
r_item: ChatCompletionReasoningItem,
) -> dict[str, object]:
"""Convert a stored ChatCompletionReasoningItem back to a Responses API input item."""
r_input: Final[dict[str, Any]] = {
r_input: Final[dict[str, object]] = {
"type": "reasoning",
"id": r_item.get("id") or f"rs_{id(r_item)}",
# summary is always required by the Responses API, even when empty
@ -181,7 +197,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return _flat_responses_tool_choice(choice_type, nested_name)
return tool_choice
def _handle_raw_dict_response_item(self, item: dict[str, Any], index: int) -> tuple[Any | None, int]:
def _handle_raw_dict_response_item(self, item: dict[str, Any], index: int) -> tuple["Choices | None", int]:
"""
Handle raw dict response items from Responses API (e.g., GPT-5 Codex format).
@ -228,8 +244,8 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def convert_chat_completion_messages_to_responses_api(
self, messages: list["AllMessageValues"]
) -> tuple[list[Any], str | None]:
input_items: Final[list[Any]] = []
) -> tuple[list[object], str | None]:
input_items: Final[list[object]] = []
instructions: str | None = None
custom_tool_call_ids: Final = frozenset(
tool_call["id"]
@ -270,7 +286,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
# Convert tool message to function call output format
# The Responses API expects 'output' to be a list with input_text/input_image types
# Using list format for consistency across text and multimodal content
tool_output: list[dict[str, Any]]
tool_output: list[dict[str, object]]
if content is None:
tool_output = []
elif isinstance(content, str):
@ -308,7 +324,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
function = tool_call.get("function")
custom = tool_call.get("custom")
if function:
input_tool_call: dict[str, Any] = {
input_tool_call: dict[str, object] = {
"type": "function_call",
"call_id": tool_call["id"],
}
@ -376,15 +392,15 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
elif key == "web_search_options":
self._add_web_search_tool(responses_api_request, value)
def _build_sanitized_litellm_params(self, litellm_params: dict) -> dict[str, Any]:
def _build_sanitized_litellm_params(self, litellm_params: dict) -> dict[str, object]:
"""Build sanitized litellm_params with merged metadata."""
responses_optional_param_keys: Final = set(ResponsesAPIOptionalRequestParams.__annotations__.keys())
sanitized: Final[dict[str, Any]] = {
sanitized: Final[dict[str, object]] = {
key: value for key, value in litellm_params.items() if key not in responses_optional_param_keys
}
legacy_metadata: Final = litellm_params.get("metadata")
existing_litellm_metadata: Final = litellm_params.get("litellm_metadata")
merged_litellm_metadata: Final[dict[str, Any]] = {}
merged_litellm_metadata: Final[dict[str, object]] = {}
if isinstance(legacy_metadata, dict):
merged_litellm_metadata.update(legacy_metadata)
if isinstance(existing_litellm_metadata, dict):
@ -424,7 +440,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
litellm_params: dict,
headers: dict,
litellm_logging_obj: "LiteLLMLoggingObj",
client: Any | None = None,
client: object | None = None,
) -> dict:
(
input_items,
@ -498,9 +514,9 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
@staticmethod
def _convert_response_output_to_choices(
output_items: list[Any],
handle_raw_dict_callback: Callable | None = None,
) -> list[Any]:
output_items: Sequence[object],
handle_raw_dict_callback: Callable[..., tuple["Choices | None", int]] | None = None,
) -> list["Choices"]:
"""
Convert Responses API output items to chat completion choices.
@ -529,11 +545,11 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
choices: Final[list[Choices]] = []
index = 0
reasoning_content: str | None = None
pending_reasoning_item: dict[str, Any] | None = None
pending_reasoning_item: _BuiltReasoningItem | None = None
# Collect all tool calls to put them in a single choice
# (Chat Completions API expects all tool calls in one message)
accumulated_tool_calls: Final[list[dict[str, Any]]] = []
accumulated_tool_calls: Final[list[Mapping[str, object]]] = []
tool_call_index = 0
for item in output_items:
@ -640,7 +656,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return choices
@classmethod
def _extract_output_from_completed_event(cls, parsed_chunk: dict[str, Any]) -> list[dict[str, Any]] | None:
def _extract_output_from_completed_event(cls, parsed_chunk: Mapping[str, object]) -> list[dict[str, object]] | None:
response_payload: Final = parsed_chunk.get("response")
if not isinstance(response_payload, dict):
return None
@ -650,12 +666,12 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return cast(list[dict[str, Any]], response_output)
@classmethod
def _recover_output_items_from_raw_sse(cls, raw_sse: str | None) -> list[dict[str, Any]]:
def _recover_output_items_from_raw_sse(cls, raw_sse: str | None) -> list[dict[str, object]]:
if not raw_sse or not isinstance(raw_sse, str):
return []
recovered_output_items: Final[dict[int, dict[str, Any]]] = {}
recovered_text_only_items: Final[dict[int, dict[str, Any]]] = {}
recovered_output_items: Final[dict[int, dict[str, object]]] = {}
recovered_text_only_items: Final[dict[int, dict[str, object]]] = {}
for chunk in raw_sse.splitlines():
parsed_chunk = parse_sse_json_chunk(chunk)
@ -690,7 +706,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
# but text-only items at indices without a matching OUTPUT_ITEM_DONE
# must still be preserved (e.g. multi-output responses where some
# indices only emitted OUTPUT_TEXT_DONE).
merged_items: Final[dict[int, dict[str, Any]]] = {**recovered_text_only_items}
merged_items: Final[dict[int, dict[str, object]]] = {**recovered_text_only_items}
merged_items.update(recovered_output_items)
if merged_items:
@ -699,7 +715,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return []
@classmethod
def _recover_output_items_from_logging(cls, logging_obj: "LiteLLMLoggingObj") -> list[dict[str, Any]]:
def _recover_output_items_from_logging(cls, logging_obj: "LiteLLMLoggingObj") -> list[dict[str, object]]:
model_call_details: Final = getattr(logging_obj, "model_call_details", {}) or {}
original_response: Final = model_call_details.get("original_response")
return cls._recover_output_items_from_raw_sse(original_response)
@ -714,7 +730,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
messages: list["AllMessageValues"],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> "ModelResponse":
@ -788,7 +804,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
) -> BaseModelResponseIterator:
return OpenAiResponsesToChatCompletionStreamIterator(streaming_response, sync_stream, json_mode)
def _convert_content_str_to_input_text(self, content: str, role: str) -> dict[str, Any]:
def _convert_content_str_to_input_text(self, content: str, role: str) -> dict[str, object]:
if role == "user" or role == "system" or role == "tool":
return {"type": "input_text", "text": content}
else:
@ -825,13 +841,13 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def _convert_content_to_responses_format(
self,
content: str
| list[Any]
| list[object]
| Iterable[
Union["OpenAIMessageContentListBlock", "ChatCompletionThinkingBlock", "ChatCompletionRedactedThinkingBlock"]
]
| None,
role: str,
) -> list[dict[str, Any]]:
) -> list[dict[str, object]]:
"""Convert chat completion content to responses API format"""
from litellm.types.llms.openai import ChatCompletionImageObject
@ -973,7 +989,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return optional_params
def _map_reasoning_effort(self, reasoning_effort: str | dict[str, Any]) -> Reasoning | None:
def _map_reasoning_effort(self, reasoning_effort: str | Reasoning) -> Reasoning | None:
# If dict is passed, convert it directly to Reasoning object
if isinstance(reasoning_effort, dict):
return Reasoning(**reasoning_effort)
@ -1006,7 +1022,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def _add_web_search_tool(
self,
responses_api_request: ResponsesAPIOptionalRequestParams,
web_search_options: Any,
web_search_options: object,
) -> None:
"""
Add web search tool to responses API request.
@ -1024,14 +1040,14 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
tools = []
responses_api_request["tools"] = tools
web_search_tool: Final[dict[str, Any]] = {"type": "web_search"}
web_search_tool: Final[dict[str, object]] = {"type": "web_search"}
if isinstance(web_search_options, dict):
web_search_tool.update(web_search_options)
# Cast to Any to match the expected union type for tools list items
tools.append(cast(Any, web_search_tool))
def _transform_response_format_to_text_format(self, response_format: dict[str, Any] | Any) -> dict[str, Any] | None:
def _transform_response_format_to_text_format(self, response_format: object) -> "ResponseText | None":
"""
Transform Chat Completion response_format parameter to Responses API text.format parameter.
@ -1130,7 +1146,12 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
def __init__(self, streaming_response, sync_stream: bool, json_mode: bool | None = False):
def __init__(
self,
streaming_response: Union[Iterator[str], AsyncIterator[str], "ModelResponse", "BaseModel"],
sync_stream: bool,
json_mode: bool | None = False,
):
super().__init__(streaming_response, sync_stream, json_mode)
self._chat_completion_id: str | None = None
self._tool_call_index_map: dict[int, int] = {} # mutable-ok: per-stream accumulator state
@ -1387,7 +1408,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
finish_reason: Final = "tool_calls" if has_function_calls else "stop"
# Extract reasoning items with encrypted_content for round-tripping
completed_reasoning_items: list[dict[str, Any]] | None = None
completed_reasoning_items: list[_BuiltReasoningItem] | None = None
for item in output_items:
if not isinstance(item, dict) or item.get("type") != "reasoning":
continue
@ -1439,7 +1460,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
]
)
def chunk_parser(self, chunk: dict) -> "ModelResponseStream":
def chunk_parser(self, chunk: dict[str, object]) -> "ModelResponseStream":
"""
Parse a Responses API streaming chunk and convert to OpenAI format.

4
litellm/constants.py
View file

@ -1478,6 +1478,10 @@ CLOUDZERO_MAX_FETCHED_DATA_RECORDS: Final = int(os.getenv("CLOUDZERO_MAX_FETCHED
SPEND_LOG_CLEANUP_JOB_NAME: Final = "spend_log_cleanup"
KEY_ROTATION_JOB_NAME: Final = "litellm_key_rotation_job"
EXPIRED_UI_SESSION_KEY_CLEANUP_JOB_NAME: Final = "litellm_expired_ui_session_key_cleanup_job"
WEEKLY_SPEND_REPORT_JOB_ID: Final = "weekly_spend_report_job"
MONTHLY_SPEND_REPORT_JOB_ID: Final = "monthly_spend_report_job"
PROMETHEUS_FALLBACK_STATS_JOB_ID: Final = "prometheus_fallback_stats_job"
SLACK_DAILY_REPORT_LOCK_ID: Final = "slack_daily_report"
SPEND_LOG_RUN_LOOPS: Final = int(os.getenv("SPEND_LOG_RUN_LOOPS", 500))
SPEND_LOG_CLEANUP_BATCH_SIZE: Final = int(os.getenv("SPEND_LOG_CLEANUP_BATCH_SIZE", 1000))
SPEND_LOG_CLEANUP_MAX_CONSECUTIVE_BATCH_FAILURES = int(os.getenv("SPEND_LOG_CLEANUP_MAX_CONSECUTIVE_BATCH_FAILURES", 3))

7
litellm/images/main.py
View file

@ -315,7 +315,12 @@ def image_generation(
or get_secret_str("AZURE_API_KEY")
)
azure_ad_token: Final = optional_params.pop("azure_ad_token", None) or get_secret_str("AZURE_AD_TOKEN")
azure_ad_token_param: Final = optional_params.pop("azure_ad_token", None)
azure_ad_token: Final = (
azure_ad_token_param
if isinstance(azure_ad_token_param, str) and azure_ad_token_param
else get_secret_str("AZURE_AD_TOKEN")
)
# Create azure_ad_token_provider from tenant_id, client_id, client_secret if not already provided
if azure_ad_token_provider is None:

27
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -17,7 +17,7 @@ import litellm.litellm_core_utils.litellm_logging
import litellm.types
from litellm._logging import verbose_logger, verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.constants import HOURS_IN_A_DAY
from litellm.constants import HOURS_IN_A_DAY, SLACK_DAILY_REPORT_LOCK_ID
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.integrations.SlackAlerting.budget_alert_types import get_budget_alert_type
from litellm.integrations.SlackAlerting.hanging_request_check import (
@ -51,6 +51,7 @@ from .batching_handler import send_to_webhook, squash_payloads
from .utils import process_slack_alerting_variables
if TYPE_CHECKING:
from litellm.proxy.db.db_transaction_queue.pod_lock_manager import PodLockManager
from litellm.router import Router as _Router
Router = _Router
@ -1576,7 +1577,11 @@ Model Info:
except Exception:
pass
async def _run_scheduler_helper(self, llm_router) -> bool:
async def _run_scheduler_helper(
self,
llm_router,
pod_lock_manager: "PodLockManager | None" = None,
) -> bool:
"""
Returns:
- True -> report sent
@ -1601,6 +1606,16 @@ Model Info:
interval_seconds: Final = self.alerting_args.daily_report_frequency
if current_time - report_sent >= interval_seconds:
if (
pod_lock_manager is not None
and (
await pod_lock_manager.acquire_lock(
cronjob_id=SLACK_DAILY_REPORT_LOCK_ID, ttl=interval_seconds, allow_reentrant=False
)
)
is False
):
return False
# Sneak in the reporting logic here
await self.send_daily_reports(router=llm_router)
# Also, don't forget to update the report_sent time after sending the report!
@ -1612,7 +1627,11 @@ Model Info:
return report_sent_bool
async def _run_scheduled_daily_report(self, llm_router: Any | None = None):
async def _run_scheduled_daily_report(
self,
llm_router: Any | None = None,
pod_lock_manager: "PodLockManager | None" = None,
):
"""
If 'daily_reports' enabled
@ -1625,7 +1644,7 @@ Model Info:
if "daily_reports" in self.alert_types:
while True:
await self._run_scheduler_helper(llm_router=llm_router)
await self._run_scheduler_helper(llm_router=llm_router, pod_lock_manager=pod_lock_manager)
interval = random.randint(
self.alerting_args.report_check_interval - 3,
self.alerting_args.report_check_interval + 3,

30
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -382,19 +382,23 @@ class AnthropicCacheControlHook(CustomPromptManagement):
model: str,
custom_llm_provider: str | None,
tools: list | None = None,
enable_prompt_caching: bool | None = None,
) -> list[CacheControlInjectionPoint]:
"""Default breakpoints when ``litellm.enable_anthropic_prompt_caching`` is on.
Caches the system prompt and the trailing turn, so the stable prefix
(system + tools + history) is reused while the breakpoint advances with
the conversation. Returns [] (stand down) when the flag is off, the
provider does not consume cache_control breakpoints (only anthropic /
bedrock do), the model lacks prompt-caching support, or the request
already carries client-supplied cache_control.
``enable_prompt_caching`` is the per-request override (stamped from key
metadata by the proxy); True turns auto-injection on for this request
even when the global flag is off. Caches the system prompt and the
trailing turn, so the stable prefix (system + tools + history) is
reused while the breakpoint advances with the conversation. Returns []
(stand down) when neither flag is on, the provider does not consume
cache_control breakpoints (only anthropic / bedrock do), the model
lacks prompt-caching support, or the request already carries
client-supplied cache_control.
"""
import litellm
if litellm.enable_anthropic_prompt_caching is not True:
if litellm.enable_anthropic_prompt_caching is not True and enable_prompt_caching is not True:
return []
provider = custom_llm_provider
@ -433,6 +437,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
model: str,
custom_llm_provider: str | None,
tools: list | None = None,
enable_prompt_caching: bool | None = None,
) -> None:
"""For /chat/completions: resolve the injection points the request should carry.
@ -458,6 +463,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
model=model,
custom_llm_provider=custom_llm_provider,
tools=tools,
enable_prompt_caching=enable_prompt_caching,
)
if points:
non_default_params["cache_control_injection_points"] = points
@ -478,12 +484,17 @@ class AnthropicCacheControlHook(CustomPromptManagement):
judgment happens once per request; points a prior pass wrote back
carry the judged stamp and are never re-judged (see
``_should_stand_down``). When none are configured but
``litellm.enable_anthropic_prompt_caching`` is on, synthesize default
breakpoints for the native /v1/messages path. Pops the key from kwargs;
``litellm.enable_anthropic_prompt_caching`` or the per-request
``enable_prompt_caching`` kwarg (stamped from key metadata) is on,
synthesize default breakpoints for the native /v1/messages path. Pops
both keys from kwargs;
if remaining (non-message) points exist they are written back so
downstream transforms can handle them.
"""
typed_messages = cast(list[AllMessageValues], messages) # cast-ok: Anthropic-shaped dicts from v1/messages
enable_prompt_caching: Final = cast( # cast-ok: kwargs is untyped; key stamped as bool by the proxy
bool | None, kwargs.pop("enable_prompt_caching", None)
)
configured: Final = cast( # cast-ok: kwargs is untyped; this key only holds the documented injection-point list
list[CacheControlInjectionPoint] | None, kwargs.pop("cache_control_injection_points", None)
)
@ -497,6 +508,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
tools=tools,
model=model,
custom_llm_provider=custom_llm_provider,
enable_prompt_caching=enable_prompt_caching,
)
if not injection_points:
return messages, system

3
litellm/integrations/braintrust_mock_client.py
View file

@ -10,6 +10,7 @@ Usage:
import os
import time
from collections.abc import AsyncIterable, Iterable
from typing import Final
from urllib.parse import urlparse
@ -84,7 +85,7 @@ def _mock_http_handler_post(
timeout=None,
stream=False,
files=None,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
logging_obj=None,
):
"""Monkey-patched HTTPHandler.post that intercepts Braintrust calls with endpoint-specific responses."""

3
litellm/integrations/gcs_bucket/gcs_bucket_mock_client.py
View file

@ -9,6 +9,7 @@ Usage:
"""
import asyncio
from collections.abc import AsyncIterable, Iterable
from typing import Final
from litellm._logging import verbose_logger
@ -113,7 +114,7 @@ async def _mock_async_handler_delete(
headers=None,
timeout=None,
stream=False,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
):
"""Monkey-patched AsyncHTTPHandler.delete that intercepts GCS calls."""
# Only mock GCS API calls

19
litellm/integrations/generic_api/generic_api_callback.py
View file

@ -11,7 +11,7 @@ import json
import os
import re
import traceback
from typing import Any, Final, Literal
from typing import Final, Literal
import httpx
@ -158,7 +158,7 @@ class GenericAPILogger(CustomBatchLogger):
"endpoint not set for GenericAPILogger, GENERIC_LOGGER_ENDPOINT not found in environment variables"
)
self.headers: dict = self._get_headers(headers)
self.headers: dict[str, str] = self._get_headers(headers)
self.endpoint: str = endpoint
self.event_types: list[API_EVENT_TYPES] | None = event_types
self.callback_name: str | None = callback_name
@ -248,18 +248,15 @@ class GenericAPILogger(CustomBatchLogger):
await asyncio.sleep(delay)
async def _post_with_retries(self, data: str) -> httpx.Response:
post_kwargs: Final[dict[str, Any]] = {
"url": self.endpoint,
"headers": self.headers,
"data": data,
}
if self.timeout is not None:
post_kwargs["timeout"] = self.timeout
total_attempts: Final = self.max_retries + 1
for attempt in range(total_attempts):
try:
return await self.async_httpx_client.post(**post_kwargs)
return await self.async_httpx_client.post(
url=self.endpoint,
headers=self.headers,
data=data,
timeout=self.timeout,
)
except Exception as e:
is_last_attempt = attempt == self.max_retries
should_retry = self._should_retry_exception(e)

5
litellm/integrations/mock_client_factory.py
View file

@ -8,6 +8,7 @@ making actual network calls.
import asyncio
import json
from collections.abc import AsyncIterable, Iterable
from dataclasses import dataclass
from datetime import timedelta
from typing import Final, cast
@ -140,7 +141,7 @@ def create_mock_client_factory(config: MockClientConfig):
stream=False,
logging_obj=None,
files=None,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
):
"""Monkey-patched AsyncHTTPHandler.post that intercepts API calls."""
if isinstance(url, str) and _is_mock_url(url):
@ -193,7 +194,7 @@ def create_mock_client_factory(config: MockClientConfig):
timeout=None,
stream=False,
files=None,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
logging_obj=None,
):
"""Monkey-patched HTTPHandler.post that intercepts API calls."""

64
litellm/integrations/opentelemetry.py
View file

@ -1,7 +1,8 @@
import os
from collections.abc import Mapping
from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, cast
from typing import TYPE_CHECKING, Any, Final, TypedDict, cast
import litellm
from litellm._logging import verbose_logger
@ -37,9 +38,11 @@ from litellm.types.utils import (
# OpenTelemetry imports moved to individual functions to avoid import errors when not installed
if TYPE_CHECKING:
from opentelemetry.sdk.trace import TracerProvider as _SDKTracerProvider
from opentelemetry.sdk.trace.export import SpanExporter as _SpanExporter
from opentelemetry.trace import Context as _Context
from opentelemetry.trace import Span as _Span
from opentelemetry.trace import SpanKind as _SpanKind
from opentelemetry.trace import Tracer as _Tracer
from litellm.proxy._types import (
@ -61,6 +64,25 @@ else:
ManagementEndpointLoggingPayload = Any
Context = Any
class _StartSpanRequiredKwargs(TypedDict):
name: str
start_time: int
context: "Context | None"
class _StartSpanKwargs(_StartSpanRequiredKwargs, total=False):
kind: "_SpanKind"
class _UsageCompletionTokensView(TypedDict, total=False):
completion_tokens: int
class _ResponseWithUsageView(TypedDict, total=False):
usage: "_UsageCompletionTokensView | None"
LITELLM_TRACER_NAME: Final = os.getenv("OTEL_TRACER_NAME", "litellm")
LITELLM_METER_NAME: Final = os.getenv("LITELLM_METER_NAME", "litellm")
LITELLM_LOGGER_NAME: Final = os.getenv("LITELLM_LOGGER_NAME", "litellm")
@ -297,9 +319,9 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
config: OpenTelemetryConfig | None = None,
callback_name: str | None = None,
# injection points for testing
tracer_provider: Any | None = None,
logger_provider: Any | None = None,
meter_provider: Any | None = None,
tracer_provider: object | None = None,
logger_provider: object | None = None,
meter_provider: object | None = None,
**kwargs,
):
team_metadata_keys_override: Final = kwargs.pop("baggage_team_metadata_keys", None)
@ -325,7 +347,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
self.OTEL_EXPORTER = self.config.exporter
self.OTEL_ENDPOINT = self.config.endpoint
self.OTEL_HEADERS = self.config.headers
self._tracer_provider_cache: dict[str, Any] = {}
self._tracer_provider_cache: dict[str, _SDKTracerProvider] = {}
self._init_tracing(tracer_provider)
_debug_otel: Final = str(os.getenv("DEBUG_OTEL", "False")).lower()
@ -870,7 +892,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
def _emit_guardrail_spans_from_request_data(
self,
request_data: dict,
parent_span: Any | None,
parent_span: "Span | None",
) -> None:
"""Emit ``guardrail`` spans from the request's proxy-internal metadata bucket
(``standard_logging_guardrail_information``).
@ -896,7 +918,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
# kwargs["litellm_params"]["metadata"]["_otel_internal"]. Pass the
# SAME metadata dict the proxy populated so _handle_failure and
# this hook see the same dedupe markers.
kwargs: Final[dict[str, Any]] = {
kwargs: Final[dict[str, object]] = {
"litellm_params": {"metadata": metadata},
"standard_logging_object": {
"guardrail_information": guardrail_information,
@ -1257,13 +1279,13 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
response_obj,
start_time,
end_time,
context,
context: "Context | None",
):
from opentelemetry.trace import Status, StatusCode
otel_tracer: Final[Tracer] = self.get_tracer_to_use_for_request(kwargs)
span_kwargs: Final[dict[str, Any]] = {
span_kwargs: Final[_StartSpanKwargs] = {
"name": self._get_span_name(kwargs),
"start_time": self._to_ns(start_time),
"context": context,
@ -1454,7 +1476,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
) = _resolve_metric_attribute_filter(attributes)
self._metric_attr_filter_resolved = True
def _filter_metric_attributes(self, attrs: dict[str, Any]) -> dict[str, Any]:
def _filter_metric_attributes(self, attrs: dict[str, str]) -> dict[str, str]:
if not self._metric_attr_filter_resolved:
self._ensure_metric_attribute_filter()
if self._metric_attr_include is not None:
@ -1559,7 +1581,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
def _record_time_per_output_token_metric(
self,
kwargs: dict,
response_obj: Any | None,
response_obj: "_ResponseWithUsageView | None",
end_time: datetime,
duration_s: float,
common_attrs: dict,
@ -1775,10 +1797,10 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
@staticmethod
def _resolve_guardrail_context(
span: Any | None,
parent_span: Any | None,
fallback_ctx: Any | None,
) -> Any | None:
span: "Span | None",
parent_span: "Span | None",
fallback_ctx: "Context | None",
) -> "Context | None":
"""
Return a valid OTEL context for guardrail child spans so they are
never orphaned (Issue #5). Priority:
@ -1945,7 +1967,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
if should_create_primary_span:
# Span 1: Request sent to litellm SDK
otel_tracer: Final[Tracer] = self.get_tracer_to_use_for_request(kwargs)
span_kwargs: Final[dict[str, Any]] = {
span_kwargs: Final[_StartSpanKwargs] = {
"name": self._get_span_name(kwargs),
"start_time": self._to_ns(start_time),
"context": _parent_context,
@ -2131,10 +2153,10 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
@staticmethod
def _tool_calls_kv_pair(
tool_calls: list[ChatCompletionMessageToolCall],
) -> dict[str, Any]:
) -> dict[str, object]:
from litellm.proxy._types import SpanAttributes
kv_pairs: Final[dict[str, Any]] = {}
kv_pairs: Final[dict[str, object]] = {}
for idx, tool_call in enumerate(tool_calls):
_function = tool_call.get("function")
if not _function:
@ -2691,8 +2713,8 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
import json
try:
_raw_response = json.loads(_raw_response)
for param, val in _raw_response.items():
_parsed: Final[Mapping[str, object]] = json.loads(_raw_response)
for param, val in _parsed.items():
self.safe_set_attribute(
span=span,
key=f"llm.{custom_llm_provider}.{param}",
@ -2722,7 +2744,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
return int(dt * 1e9)
return int(dt.timestamp() * 1e9)
def _get_span_name(self, kwargs):
def _get_span_name(self, kwargs) -> str:
litellm_params: Final = kwargs.get("litellm_params", {})
metadata: Final = litellm_params.get("metadata") or {}
generation_name: Final = metadata.get("generation_name")

13
litellm/integrations/websearch_interception/handler.py
View file

@ -85,6 +85,11 @@ class _WebSearchSettingsView(TypedDict):
websearch_interception_params: WebSearchInterceptionConfig
class _SearchToolConfig(TypedDict, total=False):
search_tool_name: str
litellm_params: Mapping[str, object] | None
class WebSearchInterceptionLogger(CustomLogger):
"""
CustomLogger that intercepts WebSearch tool calls for models that don't
@ -1487,7 +1492,7 @@ class WebSearchInterceptionLogger(CustomLogger):
return None
def _select_search_tool_from_router(self, llm_router: object) -> dict[str, Any] | None:
def _select_search_tool_from_router(self, llm_router: object) -> "_SearchToolConfig | None":
if llm_router is None or not hasattr(llm_router, "search_tools"):
return None
search_tools: Final = list(getattr(llm_router, "search_tools") or [])
@ -1495,9 +1500,9 @@ class WebSearchInterceptionLogger(CustomLogger):
def _select_search_tool_from_list(
self,
search_tools: list[dict[str, Any]],
search_tools: list[_SearchToolConfig],
source: str,
) -> dict[str, Any] | None:
) -> "_SearchToolConfig | None":
if self.search_tool_name:
matching_tools = [tool for tool in search_tools if tool.get("search_tool_name") == self.search_tool_name]
if matching_tools:
@ -1692,7 +1697,7 @@ class WebSearchInterceptionLogger(CustomLogger):
@staticmethod
def initialize_from_proxy_config(
litellm_settings: dict[str, Any],
litellm_settings: Mapping[str, WebSearchInterceptionConfig],
callback_specific_params: Mapping[str, object],
) -> "WebSearchInterceptionLogger":
"""

57
litellm/litellm_core_utils/litellm_logging.py
View file

@ -10,7 +10,7 @@ import subprocess
import sys
import time
import traceback
from collections.abc import Callable, Mapping
from collections.abc import Callable, Mapping, Sequence
from datetime import datetime as dt_object
from functools import lru_cache
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast
@ -176,6 +176,9 @@ from .initialize_dynamic_callback_params import (
from .specialty_caches.dynamic_logging_cache import DynamicLoggingCache
if TYPE_CHECKING:
from mcp.types import EmbeddedResource, ImageContent, TextContent
from litellm.integrations.otel.logger import OpenTelemetryV2
from litellm.llms.base_llm.passthrough.transformation import BasePassthroughConfig
try:
from litellm_enterprise.enterprise_callbacks.callback_controls import (
@ -211,14 +214,30 @@ except Exception as e:
PagerDutyAlerting = CustomLogger
EnterpriseCallbackControls = None
EnterpriseStandardLoggingPayloadSetupVAR = None
_in_memory_loggers: Final[list[Any]] = []
if TYPE_CHECKING:
from litellm.integrations.generic_api.generic_api_callback import (
GenericAPILogger as _GenericAPILoggerCls,
)
_STANDARD_LOGGING_METADATA_KEYS: Final[frozenset] = frozenset(StandardLoggingMetadata.__annotations__.keys())
_GENERIC_API_LOGGER_CLS: Final = _GenericAPILoggerCls
_RESEND_EMAIL_LOGGER_FACTORY: Final = CustomLogger
_SENDGRID_EMAIL_LOGGER_FACTORY: Final = CustomLogger
_SMTP_EMAIL_LOGGER_FACTORY: Final = CustomLogger
_PAGERDUTY_ALERTING_FACTORY: Final = CustomLogger
else:
_GENERIC_API_LOGGER_CLS: Final = GenericAPILogger
_RESEND_EMAIL_LOGGER_FACTORY: Final = ResendEmailLogger
_SENDGRID_EMAIL_LOGGER_FACTORY: Final = SendGridEmailLogger
_SMTP_EMAIL_LOGGER_FACTORY: Final = SMTPEmailLogger
_PAGERDUTY_ALERTING_FACTORY: Final = PagerDutyAlerting
_in_memory_loggers: Final[list[CustomLogger]] = []
_STANDARD_LOGGING_METADATA_KEYS: Final[frozenset[str]] = frozenset(StandardLoggingMetadata.__annotations__.keys())
### GLOBAL VARIABLES ###
# Cache custom pricing keys as frozenset for O(1) lookups instead of looping through 49 keys
_CUSTOM_PRICING_KEYS: Final[frozenset] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
sentry_sdk_instance = None
capture_exception = None
@ -1285,7 +1304,9 @@ class Logging(LiteLLMLoggingBaseClass):
verbose_logger.exception("LiteLLM.LoggingError: [Non-Blocking] Exception occurred while logging %s", e)
return response_obj
def _parse_post_mcp_call_hook_response(self, response: MCPPostCallResponseObject | None) -> Any:
def _parse_post_mcp_call_hook_response(
self, response: MCPPostCallResponseObject | None
) -> "Sequence[TextContent | ImageContent | EmbeddedResource] | None":
"""
Parse the response from the post_mcp_tool_call_hook
@ -1729,7 +1750,7 @@ class Logging(LiteLLMLoggingBaseClass):
self.completion_start_time = completion_start_time
self.model_call_details["completion_start_time"] = self.completion_start_time
def normalize_logging_result(self, result: Any) -> Any:
def normalize_logging_result(self, result: Any) -> object:
"""
Some endpoints return a different type of result than what is expected by the logging system.
This function is used to normalize the result to the expected type.
@ -1765,7 +1786,7 @@ class Logging(LiteLLMLoggingBaseClass):
)
return logging_result
def _merge_hidden_params_from_response_into_metadata(self, logging_result: Any) -> None:
def _merge_hidden_params_from_response_into_metadata(self, logging_result: object) -> None:
"""
Copy response._hidden_params into litellm_params.metadata['hidden_params'].
@ -1826,7 +1847,9 @@ class Logging(LiteLLMLoggingBaseClass):
if (standard_logging_payload := self.model_call_details.get("standard_logging_object")) is not None:
emit_standard_logging_payload(standard_logging_payload)
def _build_standard_logging_payload(self, init_response_obj: Any, start_time: Any, end_time: Any) -> Any:
def _build_standard_logging_payload(
self, init_response_obj: object, start_time: Any, end_time: Any
) -> StandardLoggingPayload | None:
"""Build StandardLoggingPayload and accumulate its construction time."""
_start: Final = time.time()
payload: Final = get_standard_logging_object_payload(
@ -1947,7 +1970,7 @@ class Logging(LiteLLMLoggingBaseClass):
def _is_recognized_call_type_for_logging(
self,
logging_result: Any,
logging_result: object,
):
"""
Returns True if the call type is recognized for logging (eg. ModelResponse, ModelResponseStream, etc.)
@ -4216,7 +4239,7 @@ def _init_custom_logger_compatible_class(
for callback in _in_memory_loggers:
if isinstance(callback, PagerDutyAlerting):
return callback
pagerduty_logger: Final = PagerDutyAlerting(**custom_logger_init_args)
pagerduty_logger: Final = _PAGERDUTY_ALERTING_FACTORY(**custom_logger_init_args)
_in_memory_loggers.append(pagerduty_logger)
return pagerduty_logger
elif logging_integration == "anthropic_cache_control_hook":
@ -4246,7 +4269,7 @@ def _init_custom_logger_compatible_class(
return _gcs_pubsub_logger
elif logging_integration == "generic_api":
for callback in _in_memory_loggers:
if isinstance(callback, GenericAPILogger):
if isinstance(callback, _GENERIC_API_LOGGER_CLS):
return callback
generic_api_logger: Final = GenericAPILogger()
_in_memory_loggers.append(generic_api_logger)
@ -4255,21 +4278,21 @@ def _init_custom_logger_compatible_class(
for callback in _in_memory_loggers:
if isinstance(callback, ResendEmailLogger):
return callback
resend_email_logger: Final = ResendEmailLogger()
resend_email_logger: Final = _RESEND_EMAIL_LOGGER_FACTORY()
_in_memory_loggers.append(resend_email_logger)
return resend_email_logger
elif logging_integration == "sendgrid_email":
for callback in _in_memory_loggers:
if isinstance(callback, SendGridEmailLogger):
return callback
sendgrid_email_logger: Final = SendGridEmailLogger()
sendgrid_email_logger: Final = _SENDGRID_EMAIL_LOGGER_FACTORY()
_in_memory_loggers.append(sendgrid_email_logger)
return sendgrid_email_logger
elif logging_integration == "smtp_email":
for callback in _in_memory_loggers:
if isinstance(callback, SMTPEmailLogger):
return callback
smtp_email_logger: Final = SMTPEmailLogger()
smtp_email_logger: Final = _SMTP_EMAIL_LOGGER_FACTORY()
_in_memory_loggers.append(smtp_email_logger)
return smtp_email_logger
elif logging_integration == "humanloop":
@ -4336,7 +4359,7 @@ def _init_custom_logger_compatible_class(
return None
def _maybe_construct_otel_v2(callback_name: str, _in_memory_loggers: list) -> Any | None:
def _maybe_construct_otel_v2(callback_name: str, _in_memory_loggers: list[CustomLogger]) -> "OpenTelemetryV2 | None":
"""If ``LITELLM_OTEL_V2`` is on, build (or reuse) a single ``OpenTelemetryV2``
instance configured via the preset for ``callback_name``.
@ -4367,7 +4390,7 @@ def _maybe_construct_otel_v2(callback_name: str, _in_memory_loggers: list) -> An
return v2_logger
def _maybe_auto_initialize_arize_phoenix(_in_memory_loggers: list) -> None:
def _maybe_auto_initialize_arize_phoenix(_in_memory_loggers: list[CustomLogger]) -> None:
"""
Auto-initialize ArizePhoenixLogger when Phoenix env vars are detected.
@ -4594,7 +4617,7 @@ def get_custom_logger_compatible_class(
return callback
elif logging_integration == "generic_api":
for callback in _in_memory_loggers:
if isinstance(callback, GenericAPILogger):
if isinstance(callback, _GENERIC_API_LOGGER_CLS):
return callback
elif logging_integration == "resend_email":
for callback in _in_memory_loggers:

84
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -549,7 +549,7 @@ def _fetch_and_extract_template(
return chat_template, bos_token, eos_token
async def ahf_chat_template(model: str, messages: list, chat_template: Any | None = None):
async def ahf_chat_template(model: str, messages: list, chat_template: str | None = None):
"""HuggingFace chat template (async version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_aget_chat_template_file,
@ -576,7 +576,7 @@ async def ahf_chat_template(model: str, messages: list, chat_template: Any | Non
)
def hf_chat_template(model: str, messages: list, chat_template: Any | None = None):
def hf_chat_template(model: str, messages: list, chat_template: str | None = None):
"""HuggingFace chat template (sync version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_get_chat_template_file,
@ -1130,7 +1130,7 @@ def convert_to_azure_openai_messages(
def infer_protocol_value(
value: Any,
value: object,
) -> Literal[
"string_value",
"number_value",
@ -1702,7 +1702,9 @@ def convert_function_to_anthropic_tool_invoke(
_name: Final = get_attribute_or_key(function_call, "name") or ""
_arguments: Final = get_attribute_or_key(function_call, "arguments")
tool_input = parse_tool_call_arguments(_arguments, tool_name=_name, context="Anthropic function to tool invoke")
tool_input: Final = parse_tool_call_arguments(
_arguments, tool_name=_name, context="Anthropic function to tool invoke"
)
anthropic_tool_invoke: Final = [
AnthropicMessagesToolUseParam(
@ -1764,7 +1766,7 @@ def convert_to_anthropic_tool_invoke(
Fixes: https://github.com/BerriAI/litellm/issues/17737
"""
anthropic_tool_invoke: Final[list[AnthropicMessagesToolUseParam | dict[str, Any]]] = []
anthropic_tool_invoke: Final[list[AnthropicMessagesToolUseParam | dict[str, object]]] = []
for tool in tool_calls:
if not get_attribute_or_key(tool, "type") == "function":
@ -1785,7 +1787,7 @@ def convert_to_anthropic_tool_invoke(
# Server tool IDs start with "srvtoolu_"
if tool_id.startswith("srvtoolu_"):
# Create server_tool_use block instead of tool_use
_anthropic_server_tool_use: dict[str, Any] = {
_anthropic_server_tool_use: dict[str, object] = {
"type": "server_tool_use",
"id": tool_id,
"name": tool_name,
@ -2177,7 +2179,7 @@ def _is_orphaned_tool_result(
return False
def _declared_tool_call_ids(message: Mapping[str, Any]) -> frozenset[str]:
def _declared_tool_call_ids(message: Mapping[str, object]) -> frozenset[str]:
tool_calls: Final = message.get("tool_calls")
if not isinstance(tool_calls, list):
return frozenset()
@ -2186,7 +2188,7 @@ def _declared_tool_call_ids(message: Mapping[str, Any]) -> frozenset[str]:
)
def group_tool_exchanges(messages: Sequence[Mapping[str, Any]]) -> tuple[tuple[int, ...], ...]:
def group_tool_exchanges(messages: Sequence[Mapping[str, object]]) -> tuple[tuple[int, ...], ...]:
"""Group message indices into tool exchanges: an assistant row that made
tool calls, together with the tool rows answering the ids it declared.
@ -2204,7 +2206,7 @@ def group_tool_exchanges(messages: Sequence[Mapping[str, Any]]) -> tuple[tuple[i
return tuple(_iter_tool_exchange_groups(messages))
def _iter_tool_exchange_groups(messages: Sequence[Mapping[str, Any]]) -> Iterator[tuple[int, ...]]:
def _iter_tool_exchange_groups(messages: Sequence[Mapping[str, object]]) -> Iterator[tuple[int, ...]]:
index = 0
while index < len(messages):
declared = _declared_tool_call_ids(messages[index])
@ -2409,7 +2411,7 @@ def anthropic_messages_pt(
# Convert ChatCompletionImageUrlObject to dict if needed
image_url_value = m["image_url"]
if isinstance(image_url_value, str):
image_url_input: str | dict[str, Any] = image_url_value
image_url_input: str | dict[str, object] = image_url_value
else:
# ChatCompletionImageUrlObject or dict case - convert to dict
image_url_input = {
@ -3179,7 +3181,7 @@ def _load_image_from_url(image_url):
try:
# Send a GET request to the image URL
client: Final = HTTPHandler(concurrent_limit=1)
response: Final = safe_get(client, image_url)
response: Final[httpx.Response] = safe_get(client, image_url)
response.raise_for_status() # Raise an exception for HTTP errors
# Check the response's content type to ensure it is an image
@ -3382,7 +3384,7 @@ class BedrockImageProcessor:
@staticmethod
def _post_call_image_processing(response: httpx.Response, image_url: str = "") -> tuple[str, str]:
# Check the response's content type to ensure it is an image
content_type = response.headers.get("content-type")
content_type: str | None = response.headers.get("content-type")
# Use helper function to infer content type with fallback logic
content_type = infer_content_type_from_url_and_content(
@ -3406,7 +3408,7 @@ class BedrockImageProcessor:
params={"concurrent_limit": 1},
)
# Send a GET request to the image URL
response: Final = await async_safe_get(client, image_url)
response: Final[httpx.Response] = await async_safe_get(client, image_url)
response.raise_for_status() # Raise an exception for HTTP errors
return BedrockImageProcessor._post_call_image_processing(response, image_url)
@ -3419,7 +3421,7 @@ class BedrockImageProcessor:
try:
client: Final = HTTPHandler(concurrent_limit=1)
# Send a GET request to the image URL
response: Final = safe_get(client, image_url)
response: Final[httpx.Response] = safe_get(client, image_url)
response.raise_for_status() # Raise an exception for HTTP errors
return BedrockImageProcessor._post_call_image_processing(response, image_url)
@ -3967,6 +3969,36 @@ def _rename_duplicate_bedrock_document_names(
return contents
BEDROCK_DOCUMENT_PLACEHOLDER_TEXT: Final = "."
def _with_text_when_document_only(message: BedrockMessageBlock) -> BedrockMessageBlock:
blocks: Final = message["content"]
needs_text: Final = (
message["role"] == "user"
and any("document" in block for block in blocks)
and all("text" not in block for block in blocks)
)
if not needs_text:
return message
placeholder: Final = BedrockContentBlock(text=BEDROCK_DOCUMENT_PLACEHOLDER_TEXT)
cut: Final = len(blocks) - 1 if "cachePoint" in blocks[-1] else len(blocks)
return BedrockMessageBlock(role="user", content=[*blocks[:cut], placeholder, *blocks[cut:]])
def _ensure_document_messages_have_text(
contents: list[BedrockMessageBlock],
) -> list[BedrockMessageBlock]:
"""
Bedrock Converse rejects any user message that carries a document block
without a sibling text block ("A text block must be included when using
documents"), e.g. Claude Code sends the PDF as a document-only user turn.
Inject a placeholder text block, kept ahead of a trailing cachePoint so
the caller's cache boundary stays the final block.
"""
return [_with_text_when_document_only(message) for message in contents]
def _sort_bedrock_assistant_content_blocks(
blocks: list[BedrockContentBlock],
) -> list[BedrockContentBlock]:
@ -4535,7 +4567,7 @@ class BedrockConverseMessagesProcessor:
llm_provider=llm_provider,
)
return _rename_duplicate_bedrock_document_names(contents)
return _ensure_document_messages_have_text(_rename_duplicate_bedrock_document_names(contents))
@staticmethod
def translate_thinking_blocks_to_reasoning_content_blocks(
@ -4911,7 +4943,7 @@ def _bedrock_converse_messages_pt(
llm_provider=llm_provider,
)
return _rename_duplicate_bedrock_document_names(contents)
return _ensure_document_messages_have_text(_rename_duplicate_bedrock_document_names(contents))
def make_valid_bedrock_tool_name(input_tool_name: str) -> str:
@ -5328,10 +5360,10 @@ def get_attribute_or_key(tool_or_function, attribute, default=None):
class NormalizedToolCall(TypedDict):
id: str | None
name: str | None
arguments: dict[str, Any]
arguments: dict[str, object]
def _parse_tool_call_arguments(raw: Any, tool_name: str | None, context: str) -> dict[str, Any]:
def _parse_tool_call_arguments(raw: Any, tool_name: str | None, context: str) -> dict[str, object]:
# Anthropic's tool_use blocks already carry a parsed dict in "input";
# chat completions and the Responses API carry a JSON string that may be
# truncated by the model, so route those through the repair-aware parser.
@ -5352,12 +5384,12 @@ def _parse_tool_call_arguments(raw: Any, tool_name: str | None, context: str) ->
def _tool_calls_from_chat_completion_response(
response: Any, include_all_choices: bool = False
response: object, include_all_choices: bool = False
) -> list[NormalizedToolCall]:
choices: Final = get_attribute_or_key(response, "choices", None)
if not (isinstance(choices, list) and choices):
return []
tool_calls: Final[list[Any]] = []
tool_calls: Final[list[object]] = []
for choice in choices if include_all_choices else choices[:1]:
message = get_attribute_or_key(choice, "message", None)
choice_tool_calls = get_attribute_or_key(message, "tool_calls", None) if message else None
@ -5383,7 +5415,7 @@ def _tool_calls_from_chat_completion_response(
return result
def _tool_calls_from_responses_api_response(response: Any) -> list[NormalizedToolCall]:
def _tool_calls_from_responses_api_response(response: object) -> list[NormalizedToolCall]:
output: Final = get_attribute_or_key(response, "output", None)
if not isinstance(output, list):
return []
@ -5406,7 +5438,7 @@ def _tool_calls_from_responses_api_response(response: Any) -> list[NormalizedToo
return result
def _tool_calls_from_anthropic_messages_response(response: Any) -> list[NormalizedToolCall]:
def _tool_calls_from_anthropic_messages_response(response: object) -> list[NormalizedToolCall]:
content: Final = get_attribute_or_key(response, "content", None)
if not isinstance(content, list):
return []
@ -5425,7 +5457,7 @@ def _tool_calls_from_anthropic_messages_response(response: Any) -> list[Normaliz
return result
def get_tool_calls_from_response(response: Any, include_all_choices: bool = False) -> list[NormalizedToolCall]:
def get_tool_calls_from_response(response: object, include_all_choices: bool = False) -> list[NormalizedToolCall]:
"""
Extract tool/function calls from a response object into a normalized
``{"id", "name", "arguments"}`` shape, regardless of which API surface
@ -5456,7 +5488,7 @@ def get_tool_calls_from_response(response: Any, include_all_choices: bool = Fals
return []
def has_tool_with_name(tools: Any, tool_name: str) -> bool:
def has_tool_with_name(tools: object, tool_name: str) -> bool:
"""
Check whether a tools list (as sent to an LLM) includes a tool with the
given name, regardless of shape: OpenAI-style function tools
@ -5482,9 +5514,9 @@ def has_tool_with_name(tools: Any, tool_name: str) -> bool:
def resolve_structured_messages(
messages: list[dict[str, Any]] | None,
messages: list[dict[str, object]] | None,
request_kwargs: dict[str, Any],
) -> list[dict[str, Any]] | None:
) -> list[dict[str, object]] | None:
"""
Normalize a request's messages to OpenAI-spec chat-completions shape,
regardless of which API surface produced them (chat completions,

10
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -145,7 +145,7 @@ class ChunkProcessor:
if first_hidden_params.get("created_at"):
def _created_at(chunk: Any) -> int | float:
def _created_at(chunk: object) -> int | float:
if isinstance(chunk, dict):
params = chunk.get("_hidden_params", {})
else:
@ -158,7 +158,7 @@ class ChunkProcessor:
return chunks
def update_model_response_with_hidden_params(
self, model_response: ModelResponse, chunk: dict[str, Any] | None = None
self, model_response: ModelResponse, chunk: Mapping[str, dict[str, object]] | None = None
) -> ModelResponse:
if chunk is None:
return model_response
@ -176,7 +176,7 @@ class ChunkProcessor:
if not chunks:
return
model: Final = getattr(response, "model", None)
model: Final[str | None] = getattr(response, "model", None)
if not model:
return
@ -214,7 +214,7 @@ class ChunkProcessor:
)
@staticmethod
def _get_chunk_id(chunks: list[dict[str, Any]]) -> str:
def _get_chunk_id(chunks: Sequence[Mapping[str, str]]) -> str:
"""
Chunks:
[{"id": ""}, {"id": "1"}, {"id": "1"}]
@ -225,7 +225,7 @@ class ChunkProcessor:
return ""
@staticmethod
def _get_model_from_chunks(chunks: list[dict[str, Any]], first_chunk_model: str) -> str:
def _get_model_from_chunks(chunks: Sequence[Mapping[str, str]], first_chunk_model: str) -> str:
"""
Get the actual model from chunks, preferring a model that differs from the first chunk.

100
litellm/litellm_core_utils/streaming_handler.py
View file

@ -6,13 +6,14 @@ import logging
import threading
import time
import traceback
from collections.abc import AsyncIterator, Callable, Iterator
from collections.abc import AsyncIterator, Callable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from typing import Any, Final, NoReturn, TypeVar, cast
from typing import Any, Final, NoReturn, Protocol, TypeVar, cast
import anyio
import httpx
from pydantic import BaseModel
from typing_extensions import NotRequired, TypedDict
import litellm
from litellm import verbose_logger
@ -54,7 +55,7 @@ _SYNC_ITER_EXHAUSTED: Final = object()
_GCHUNK_FIELDS: Final[frozenset] = frozenset(GChunk.__annotations__)
def _next_sync_or_exhausted(it: Any) -> Any:
def _next_sync_or_exhausted(it: Any) -> object:
"""
Call next(it) from a thread and return _SYNC_ITER_EXHAUSTED on StopIteration.
@ -68,7 +69,7 @@ def _next_sync_or_exhausted(it: Any) -> Any:
return _SYNC_ITER_EXHAUSTED
def is_async_iterable(obj: Any) -> bool:
def is_async_iterable(obj: object) -> bool:
"""
Check if an object is an async iterable (can be used with 'async for').
@ -81,7 +82,7 @@ def is_async_iterable(obj: Any) -> bool:
return isinstance(obj, collections.abc.AsyncIterable)
def print_verbose(print_statement):
def print_verbose(print_statement: object):
try:
if litellm.set_verbose:
print(print_statement) # noqa: T201
@ -96,18 +97,70 @@ class _ProviderChunkParsed:
@dataclass(frozen=True, slots=True)
class _ProviderChunkEarlyReturn:
value: Any
value: "ModelResponseStream | None"
_ProviderChunkResult = _ProviderChunkParsed | _ProviderChunkEarlyReturn
class _PredibaseStreamData(TypedDict):
token: NotRequired[Mapping[str, str]]
details: Mapping[str, str]
generated_text: str | None
error: str | None
class _Ai21StreamData(TypedDict):
completions: Sequence[Mapping[str, Mapping[str, str]]]
class _MaritalkStreamData(TypedDict):
answer: str
class _NlpCloudStreamData(TypedDict):
generated_text: str
class _AlephAlphaStreamData(TypedDict):
completions: Sequence[Mapping[str, str]]
class _AzureStreamChoice(TypedDict):
delta: Mapping[str, str] | None
finish_reason: str | None
class _AzureStreamData(TypedDict):
choices: Sequence[_AzureStreamChoice]
class _BasetenModelOutput(TypedDict):
data: NotRequired[Sequence[str]]
class _BasetenStreamData(TypedDict):
token: NotRequired[Mapping[str, str]]
model_output: NotRequired["_BasetenModelOutput | str"]
completion: NotRequired[object]
class _DeltaDumpDict(TypedDict):
role: NotRequired[str | None]
tool_calls: NotRequired[Sequence[Mapping[str, object]]]
class _TextCompletionChoiceLike(Protocol):
text: str
finish_reason: str | None
class CustomStreamWrapper:
def __init__(
self,
completion_stream,
model,
logging_obj: Any,
logging_obj: LiteLLMLoggingObject,
custom_llm_provider: str | None = None,
stream_options=None,
make_call: Callable | None = None,
@ -186,7 +239,7 @@ class CustomStreamWrapper:
# Snapshot assumes self._hidden_params is populated from litellm_params
# at init and never mutated during the stream. If that ever changes,
# this cache must be removed.
self._base_hidden_params: dict[str, Any] = {
self._base_hidden_params: dict[str, object] = {
**self._hidden_params,
"response_cost": None,
}
@ -416,7 +469,7 @@ class CustomStreamWrapper:
finish_reason = ""
print_verbose(f"chunk: {chunk}")
if chunk.startswith("data:"):
data_json: Final = json.loads(chunk[5:])
data_json: Final[_PredibaseStreamData] = json.loads(chunk[5:])
print_verbose(f"data json: {data_json}")
if "token" in data_json and "text" in data_json["token"]:
text = data_json["token"]["text"]
@ -446,7 +499,7 @@ class CustomStreamWrapper:
def handle_ai21_chunk(self, chunk): # fake streaming
chunk = chunk.decode("utf-8")
data_json: Final = json.loads(chunk)
data_json: Final[_Ai21StreamData] = json.loads(chunk)
try:
text: Final = data_json["completions"][0]["data"]["text"]
is_finished: Final = True
@ -461,7 +514,7 @@ class CustomStreamWrapper:
def handle_maritalk_chunk(self, chunk): # fake streaming
chunk = chunk.decode("utf-8")
data_json: Final = json.loads(chunk)
data_json: Final[_MaritalkStreamData] = json.loads(chunk)
try:
text: Final = data_json["answer"]
is_finished: Final = True
@ -482,7 +535,7 @@ class CustomStreamWrapper:
if self.model and "dolphin" in self.model:
chunk = self.process_chunk(chunk=chunk)
else:
data_json: Final = json.loads(chunk)
data_json: Final[_NlpCloudStreamData] = json.loads(chunk)
chunk = data_json["generated_text"]
text = chunk
if "[DONE]" in text:
@ -499,7 +552,7 @@ class CustomStreamWrapper:
def handle_aleph_alpha_chunk(self, chunk):
chunk = chunk.decode("utf-8")
data_json: Final = json.loads(chunk)
data_json: Final[_AlephAlphaStreamData] = json.loads(chunk)
try:
text: Final = data_json["completions"][0]["completion"]
is_finished: Final = True
@ -527,7 +580,7 @@ class CustomStreamWrapper:
"finish_reason": finish_reason,
}
elif chunk.startswith("data:"):
data_json: Final = json.loads(chunk[5:]) # chunk.startswith("data:"):
data_json: Final[_AzureStreamData] = json.loads(chunk[5:]) # chunk.startswith("data:"):
try:
if len(data_json["choices"]) > 0:
delta: Final = data_json["choices"][0]["delta"]
@ -616,7 +669,7 @@ class CustomStreamWrapper:
text = ""
is_finished = False
finish_reason = None
choices: Final = getattr(chunk, "choices", [])
choices: Final[Sequence[_TextCompletionChoiceLike]] = getattr(chunk, "choices", [])
if len(choices) > 0:
text = choices[0].text
if choices[0].finish_reason is not None:
@ -637,7 +690,7 @@ class CustomStreamWrapper:
is_finished = False
finish_reason = None
usage = None
choices: Final = getattr(chunk, "choices", [])
choices: Final[Sequence[_TextCompletionChoiceLike]] = getattr(chunk, "choices", [])
if len(choices) > 0:
text = choices[0].text
if choices[0].finish_reason is not None:
@ -654,12 +707,12 @@ class CustomStreamWrapper:
except Exception as e:
raise e
def handle_baseten_chunk(self, chunk):
def handle_baseten_chunk(self, chunk) -> str:
try:
chunk = chunk.decode("utf-8")
if len(chunk) > 0:
if chunk.startswith("data:"):
data_json = json.loads(chunk[5:])
data_json: _BasetenStreamData = json.loads(chunk[5:])
if "token" in data_json and "text" in data_json["token"]:
return data_json["token"]["text"]
else:
@ -1325,13 +1378,14 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
if "usage" in response_obj is not None:
_codestral_usage: Final[Usage] = response_obj["usage"]
setattr(
model_response,
"usage",
litellm.Usage(
prompt_tokens=response_obj["usage"].prompt_tokens,
completion_tokens=response_obj["usage"].completion_tokens,
total_tokens=response_obj["usage"].total_tokens,
prompt_tokens=_codestral_usage.prompt_tokens,
completion_tokens=_codestral_usage.completion_tokens,
total_tokens=_codestral_usage.total_tokens,
),
)
elif self.custom_llm_provider == "azure_text":
@ -1474,7 +1528,7 @@ class CustomStreamWrapper:
is None
):
t.function.arguments = ""
_json_delta: Final = delta.model_dump()
_json_delta: Final[_DeltaDumpDict] = delta.model_dump()
if "role" not in _json_delta or _json_delta["role"] is None:
_json_delta["role"] = "assistant" # mistral's api returns role as None
if "tool_calls" in _json_delta and isinstance(_json_delta["tool_calls"], list):
@ -1744,7 +1798,7 @@ class CustomStreamWrapper:
usage.cost, copy it into _hidden_params so litellm's cost
calculator uses it instead of a token-based estimate.
"""
_usage: Final = getattr(response, "usage", None)
_usage: Final[Usage | None] = getattr(response, "usage", None)
if _usage is not None and hasattr(_usage, "cost") and _usage.cost is not None:
if "additional_headers" not in response._hidden_params:
response._hidden_params["additional_headers"] = {}

5
litellm/llms/anthropic/chat/handler.py
View file

@ -58,6 +58,7 @@ from ..common_utils import AnthropicError, process_anthropic_headers
from .transformation import ANTHROPIC_TOOL_NAME_REVERSE_MAP_KEY, AnthropicConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.llms.base_llm.chat.transformation import BaseConfig
@ -206,7 +207,7 @@ class AnthropicChatCompletion(BaseLLM):
client: AsyncHTTPHandler | None,
encoding,
api_key,
logging_obj,
logging_obj: "LiteLLMLoggingObj",
stream,
_is_function_call,
data: dict,
@ -324,7 +325,7 @@ class AnthropicChatCompletion(BaseLLM):
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: "LiteLLMLoggingObj",
optional_params: dict,
timeout: float | httpx.Timeout,
litellm_params: dict,

11
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -1004,6 +1004,17 @@ class LiteLLMAnthropicMessagesAdapter:
model: Final = new_kwargs.get("model", "")
if self.is_anthropic_claude_model(model) or self.is_bedrock_arn_model(model):
new_kwargs["thinking"] = thinking
# Adaptive thinking without its effort tier makes Bedrock Converse
# return zero reasoning blocks, so forward output_config (minus
# `format`, already translated to response_format) for Bedrock
# targets only: other bridged providers reject the raw param, and
# get_llm_provider strips the `bedrock/` prefix before this runs.
if model.startswith(("bedrock/", "converse/", "invoke/")) or self.is_bedrock_arn_model(model):
claude_output_config: Final = anthropic_message_request.get("output_config")
if isinstance(claude_output_config, dict):
effort_config: Final = {k: v for k, v in claude_output_config.items() if k != "format"}
if effort_config:
new_kwargs["output_config"] = effort_config # rebind-ok: out-param store like thinking above
return
reasoning_effort = self.translate_anthropic_thinking_to_reasoning_effort(cast(dict[str, Any], thinking))

6
litellm/llms/azure/azure.py
View file

@ -228,7 +228,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
litellm_params=litellm_params,
)
data = {"model": None, "messages": messages, **optional_params}
data: dict[str, object] = {"model": None, "messages": messages, **optional_params}
elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=litellm_params.get("base_model") or model):
data = litellm.AzureOpenAIGPT5Config().transform_request(
model=model,
@ -482,12 +482,12 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
def streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_base: str,
api_key: str | None,
api_version: str,
dynamic_params: bool,
data: dict,
data: dict[str, object],
model: str,
timeout: Any,
max_retries: int,

5
litellm/llms/azure/chat/o_series_handler.py
View file

@ -5,10 +5,11 @@ Written separately to handle faking streaming for o1 and o3 models.
"""
from collections.abc import Callable
from typing import TYPE_CHECKING, Any, Optional
from typing import TYPE_CHECKING, Optional
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.utils import ModelResponse
from ...openai.openai import OpenAIChatCompletion
@ -25,7 +26,7 @@ class AzureOpenAIO1ChatCompletion(BaseAzureLLM, OpenAIChatCompletion):
timeout: float | httpx.Timeout,
optional_params: dict,
litellm_params: dict,
logging_obj: Any,
logging_obj: LiteLLMLoggingObj,
model: str | None = None,
messages: list | None = None,
print_verbose: Callable | None = None,

9
litellm/llms/azure/completion/handler.py
View file

@ -3,6 +3,7 @@ from typing import Any, Final
from openai import AsyncAzureOpenAI, AzureOpenAI
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.prompt_templates.factory import prompt_factory
from litellm.utils import CustomStreamWrapper, ModelResponse, TextCompletionResponse
@ -39,9 +40,9 @@ class AzureTextCompletion(BaseAzureLLM):
azure_ad_token_provider: Callable | None,
print_verbose: Callable,
timeout,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params,
litellm_params,
litellm_params: dict[str, object],
logger_fn,
acompletion: bool = False,
headers: dict | None = None,
@ -246,7 +247,7 @@ class AzureTextCompletion(BaseAzureLLM):
def streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_base: str,
api_key: str | None,
api_version: str,
@ -299,7 +300,7 @@ class AzureTextCompletion(BaseAzureLLM):
async def async_streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_base: str,
api_key: str | None,
api_version: str,

3
litellm/llms/azure_ai/anthropic/handler.py
View file

@ -9,6 +9,7 @@ from typing import Final
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.anthropic.chat.handler import AnthropicChatCompletion
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
@ -40,7 +41,7 @@ class AzureAnthropicChatCompletion(AnthropicChatCompletion):
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
timeout: float | httpx.Timeout,
litellm_params: dict,

2
litellm/llms/azure_ai/chat/transformation.py
View file

@ -248,7 +248,7 @@ class AzureAIStudioConfig(OpenAIConfig):
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=encoding,
encoding=encoding if encoding is not None else None,
api_key=api_key,
json_mode=json_mode,
)

2
litellm/llms/bedrock/chat/converse_handler.py
View file

@ -89,7 +89,7 @@ class BedrockConverseLLM(BaseAWSLLM):
model_response: ModelResponse,
timeout: float | httpx.Timeout | None,
encoding,
logging_obj,
logging_obj: LiteLLMLoggingObject,
stream,
optional_params: dict,
litellm_params: dict,

14
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -80,6 +80,7 @@ from ..common_utils import (
bedrock_converse_supports_parallel_tool_use_config,
get_anthropic_beta_from_headers,
get_bedrock_tool_name,
is_bedrock_application_inference_profile_arn,
is_claude_4_5_on_bedrock,
normalize_bedrock_opus_output_config_effort,
)
@ -514,6 +515,7 @@ class AmazonConverseConfig(BaseConfig):
supported_params.append("tool_choice")
supported_params.append("thinking")
supported_params.append("reasoning_effort")
supported_params.append("output_config")
# For nova imported models, also add web_search_options
if "nova" in model.lower():
supported_params.append("web_search_options")
@ -564,6 +566,7 @@ class AmazonConverseConfig(BaseConfig):
):
supported_params.append("thinking")
supported_params.append("reasoning_effort")
supported_params.append("output_config")
if base_model.startswith("anthropic"):
supported_params.append("context_management")
@ -919,6 +922,10 @@ class AmazonConverseConfig(BaseConfig):
self._handle_reasoning_effort_parameter(
model=model, reasoning_effort=value, optional_params=optional_params
)
elif param == "output_config" and isinstance(value, dict):
mapped_output_config = dict(value)
normalize_bedrock_opus_output_config_effort(model=model, output_config=mapped_output_config)
optional_params["output_config"] = mapped_output_config # rebind-ok: out-param store like siblings
elif param == "context_management" and isinstance(value, (dict, list)):
self._map_context_management_param(value, optional_params)
if param == "requestMetadata":
@ -1312,7 +1319,12 @@ class AmazonConverseConfig(BaseConfig):
additional_request_params = filter_exceptions_from_params(additional_request_params)
if anthropic_output_config is not None and isinstance(anthropic_output_config, dict):
if base_model.startswith("anthropic"):
# Application inference profile ARNs hide the underlying model, so the
# effort ceiling and capability gates below cannot run; forward
# verbatim (like ``thinking``) and let Bedrock enforce.
if is_bedrock_application_inference_profile_arn(model):
additional_request_params["output_config"] = anthropic_output_config
elif base_model.startswith("anthropic"):
if litellm.drop_params is True and not AnthropicConfig._model_supports_effort_param(model, "bedrock"):
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,

33
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -372,8 +372,9 @@ class AmazonAnthropicClaudeMessagesConfig(
"""
Check if the model supports tool search on Bedrock.
On Amazon Bedrock, server-side tool search is supported on Claude Opus 4.5
and Claude Sonnet 4.5 with the tool-search-tool-2025-10-19 beta header.
The model map's ``supports_tool_search`` flag is authoritative when
``model`` resolves to an entry that sets it; the name patterns below
cover ids the map cannot resolve (ARNs, unlisted regional variants).
Ref: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool
@ -383,9 +384,12 @@ class AmazonAnthropicClaudeMessagesConfig(
Returns:
True if the model supports tool search on Bedrock
"""
catalog: Final = AnthropicModelInfo._get_provider_resolved_capability(model, "supports_tool_search", "bedrock")
if catalog is not None:
return catalog
model_lower: Final = model.lower()
# Supported models for tool search on Bedrock
supported_patterns: Final = [
# Opus 4.5
"opus-4.5",
@ -407,10 +411,16 @@ class AmazonAnthropicClaudeMessagesConfig(
"sonnet_4.6",
"sonnet-4-6",
"sonnet_4_6",
# NOTE: Opus 4.7 on Bedrock does not support server-side tool search
# as of launch (2026-04-16). Bedrock rejects the tool type with:
# "tool type 'tool_search_tool_..._20251119' is not supported for this model".
# Re-add the opus-4.7 patterns here once AWS announces support.
# Opus 4.7
"opus-4.7",
"opus_4.7",
"opus-4-7",
"opus_4_7",
# Haiku 4.5
"haiku-4.5",
"haiku_4.5",
"haiku-4-5",
"haiku_4_5",
]
return any(pattern in model_lower for pattern in supported_patterns)
@ -426,11 +436,10 @@ class AmazonAnthropicClaudeMessagesConfig(
"""
Adjust tool search beta header for Bedrock.
Bedrock requires a different beta header for tool search on Opus 4 models
when tool search is used without programmatic tool calling or input examples.
Note: On Amazon Bedrock, server-side tool search is only supported on Claude Opus 4
with the `tool-search-tool-2025-10-19` beta header.
Bedrock requires a different beta header for tool search than the
Anthropic API when tool search is used without programmatic tool
calling or input examples: `tool-search-tool-2025-10-19`, and only on
the models listed in `_supports_tool_search_on_bedrock`.
Ref: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool

4
litellm/llms/codestral/completion/handler.py
View file

@ -195,7 +195,7 @@ class CodestralTextCompletion:
print_verbose: Callable,
encoding,
api_key: str,
logging_obj,
logging_obj: LiteLLMLogging,
optional_params: dict,
timeout: float | httpx.Timeout,
acompletion=None,
@ -383,7 +383,7 @@ class CodestralTextCompletion:
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLogging,
data: dict,
timeout: float | httpx.Timeout,
optional_params=None,

2
litellm/llms/custom_httpx/aiohttp_handler.py
View file

@ -221,7 +221,7 @@ class BaseLLMAIOHTTPHandler:
timeout=timeout,
stream=stream,
files=files,
content=content,
content=content if content is not None else None,
params=params,
)
except httpx.HTTPStatusError as e:

61
litellm/llms/custom_httpx/http_handler.py
View file

@ -7,9 +7,9 @@ import ssl
import sys
import threading
import time
from collections.abc import Callable, Mapping
from collections.abc import AsyncIterable, Callable, Iterable, Mapping
from http.cookiejar import CookieJar, DefaultCookiePolicy
from typing import TYPE_CHECKING, Any, Final, Optional
from typing import TYPE_CHECKING, Any, Final, Optional, TypeAlias, TypedDict
import certifi
import httpx
@ -62,8 +62,23 @@ except Exception:
# https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.TCPConnector
_AIOHTTP_SUPPORTS_SOCKET_FACTORY: Final = "socket_factory" in inspect.signature(TCPConnector.__init__).parameters
_AddrInfo: TypeAlias = tuple[int | socket.AddressFamily, int | socket.SocketKind, int, str, tuple[object, ...]]
def _build_aiohttp_keepalive_socket_factory() -> Callable[[tuple[Any, ...]], socket.socket] | None:
_RequestContent: TypeAlias = str | bytes | Iterable[bytes] | AsyncIterable[bytes]
class _TCPConnectorKwargs(TypedDict, total=False):
local_addr: tuple[str, int] | None
ssl: "ssl.SSLContext | bool"
keepalive_timeout: float
ttl_dns_cache: int
enable_cleanup_closed: bool
limit: int
limit_per_host: int
socket_factory: Callable[[_AddrInfo], socket.socket]
def _build_aiohttp_keepalive_socket_factory() -> Callable[[_AddrInfo], socket.socket] | None:
"""
Build a socket_factory that enables SO_KEEPALIVE on aiohttp TCP sockets.
@ -78,7 +93,7 @@ def _build_aiohttp_keepalive_socket_factory() -> Callable[[tuple[Any, ...]], soc
if not AIOHTTP_SO_KEEPALIVE or not _AIOHTTP_SUPPORTS_SOCKET_FACTORY:
return None
def factory(addr_info: tuple[Any, ...]) -> socket.socket:
def factory(addr_info: _AddrInfo) -> socket.socket:
family, type_, proto = addr_info[0], addr_info[1], addr_info[2]
sock: Final = socket.socket(family=family, type=type_, proto=proto)
sock.setblocking(False)
@ -163,8 +178,8 @@ _STREAMING_ERROR_BODY_READ_EXECUTOR: Final = concurrent.futures.ThreadPoolExecut
def _prepare_request_data_and_content(
data: dict | str | bytes | None = None,
content: Any = None,
) -> tuple[dict | Mapping | None, Any]:
content: _RequestContent | None = None,
) -> tuple[dict | Mapping | None, _RequestContent | None]:
"""
Helper function to route data/content parameters correctly for httpx requests
@ -528,7 +543,7 @@ class AsyncHTTPHandler:
def __init__(
self,
timeout: float | httpx.Timeout | None = None,
event_hooks: Mapping[str, list[Callable[..., Any]]] | None = None,
event_hooks: Mapping[str, list[Callable[..., object]]] | None = None,
concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
client_alias: str | None = None, # name for client in logs
ssl_verify: VerifyTypes | None = None,
@ -566,7 +581,7 @@ class AsyncHTTPHandler:
def create_client(
self,
timeout: float | httpx.Timeout | None,
event_hooks: Mapping[str, list[Callable[..., Any]]] | None,
event_hooks: Mapping[str, list[Callable[..., object]]] | None,
ssl_verify: VerifyTypes | None = None,
shared_session: Optional["ClientSession"] = None,
) -> httpx.AsyncClient:
@ -648,7 +663,7 @@ class AsyncHTTPHandler:
stream: bool = False,
logging_obj: LiteLLMLoggingObject | None = None,
files: RequestFiles | None = None,
content: Any = None,
content: _RequestContent | None = None,
):
start_time: Final = time.time()
try:
@ -691,7 +706,7 @@ class AsyncHTTPHandler:
end_time: Final = time.time()
time_delta: Final = round(end_time - start_time, 3)
headers = {}
error_response: Final = getattr(e, "response", None)
error_response: Final[httpx.Response | None] = getattr(e, "response", None)
if error_response is not None:
for key, value in error_response.headers.items():
headers[f"response_headers-{key}"] = value
@ -716,7 +731,7 @@ class AsyncHTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
if timeout is None:
@ -755,7 +770,7 @@ class AsyncHTTPHandler:
await new_client.aclose()
except httpx.TimeoutException as e:
headers = {}
error_response: Final = getattr(e, "response", None)
error_response: Final[httpx.Response | None] = getattr(e, "response", None)
if error_response is not None:
for key, value in error_response.headers.items():
headers[f"response_headers-{key}"] = value
@ -780,7 +795,7 @@ class AsyncHTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
if timeout is None:
@ -819,7 +834,7 @@ class AsyncHTTPHandler:
await new_client.aclose()
except httpx.TimeoutException as e:
headers = {}
error_response: Final = getattr(e, "response", None)
error_response: Final[httpx.Response | None] = getattr(e, "response", None)
if error_response is not None:
for key, value in error_response.headers.items():
headers[f"response_headers-{key}"] = value
@ -844,7 +859,7 @@ class AsyncHTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
if timeout is None:
@ -895,7 +910,7 @@ class AsyncHTTPHandler:
params: dict | None = None,
headers: dict | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
"""
Making POST request for a single connection client.
@ -993,7 +1008,7 @@ class AsyncHTTPHandler:
def _get_ssl_connector_kwargs(
ssl_verify: bool | None = None,
ssl_context: ssl.SSLContext | None = None,
) -> dict[str, Any]:
) -> _TCPConnectorKwargs:
"""
Helper method to get SSL connector initialization arguments for aiohttp TCPConnector.
@ -1004,7 +1019,7 @@ class AsyncHTTPHandler:
Returns:
Dict with appropriate SSL configuration for TCPConnector
"""
connector_kwargs: Final[dict[str, Any]] = {
connector_kwargs: Final[_TCPConnectorKwargs] = {
"local_addr": ("0.0.0.0", 0) if litellm.force_ipv4 else None,
}
@ -1054,7 +1069,7 @@ class AsyncHTTPHandler:
verbose_logger.debug("Creating AiohttpTransport...")
transport_connector_kwargs: Final = {
transport_connector_kwargs: Final[_TCPConnectorKwargs] = {
"keepalive_timeout": AIOHTTP_KEEPALIVE_TIMEOUT,
"ttl_dns_cache": AIOHTTP_TTL_DNS_CACHE,
**connector_kwargs,
@ -1212,7 +1227,7 @@ class HTTPHandler:
stream: bool = False,
timeout: float | httpx.Timeout | None = None,
files: dict | RequestFiles | None = None,
content: Any = None,
content: _RequestContent | None = None,
logging_obj: LiteLLMLoggingObject | None = None,
):
try:
@ -1265,7 +1280,7 @@ class HTTPHandler:
headers: dict | None = None,
stream: bool = False,
timeout: float | httpx.Timeout | None = None,
content: Any = None,
content: _RequestContent | None = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
@ -1315,7 +1330,7 @@ class HTTPHandler:
headers: dict | None = None,
stream: bool = False,
timeout: float | httpx.Timeout | None = None,
content: Any = None,
content: _RequestContent | None = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
@ -1364,7 +1379,7 @@ class HTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)

64
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -5,7 +5,8 @@ import ssl
from collections.abc import AsyncIterator, Coroutine, Iterator, Mapping
from contextlib import asynccontextmanager
from functools import lru_cache
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypeVar, Union, cast, get_type_hints
from types import ModuleType
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypedDict, TypeVar, Union, cast, get_type_hints
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
import httpx
@ -148,6 +149,7 @@ from .http_handler import get_shared_realtime_ssl_context
if TYPE_CHECKING:
from aiohttp import ClientSession
from websockets.asyncio.client import ClientConnection
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -176,6 +178,19 @@ else:
_ResponseT = TypeVar("_ResponseT")
class _DeleteRequestKwargs(TypedDict, total=False):
url: str
headers: dict[str, str]
timeout: float | httpx.Timeout | None
json: dict[str, object]
class _MediaUploadKwargs(TypedDict, total=False):
headers: dict[str, str]
content: Iterator[bytes] | AsyncIterator[bytes]
timeout: float | httpx.Timeout
def _google_genai_streaming_hidden_params(
*,
api_base: str,
@ -1413,7 +1428,7 @@ class BaseLLMHTTPHandler:
headers: dict[str, object] | None,
provider_config: BaseOCRConfig,
litellm_params: dict,
) -> tuple[dict[str, Any], str, dict[str, Any], None]:
) -> tuple[dict[str, object], str, dict[str, object], None]:
"""
Shared logic for preparing OCR requests.
Returns: (headers, complete_url, data, files)
@ -1479,7 +1494,7 @@ class BaseLLMHTTPHandler:
headers: dict[str, object] | None,
provider_config: BaseOCRConfig,
litellm_params: dict,
) -> tuple[dict[str, Any], str, dict[str, Any], None]:
) -> tuple[dict[str, object], str, dict[str, object], None]:
"""
Async version of _prepare_ocr_request for providers that need async transforms.
Returns: (headers, complete_url, data, files)
@ -2361,14 +2376,14 @@ class BaseLLMHTTPHandler:
model: str,
input: str | ResponseInputParam,
custom_llm_provider: str,
response_api_optional_request_params: dict[str, Any],
response_api_optional_request_params: dict[str, object],
litellm_params: GenericLiteLLMParams,
logging_obj: LiteLLMLoggingObj,
) -> tuple[
str,
str | ResponseInputParam,
str,
dict[str, Any],
dict[str, object],
GenericLiteLLMParams,
]:
if not _has_pre_call_deployment_hook(logging_obj):
@ -2894,7 +2909,7 @@ class BaseLLMHTTPHandler:
},
)
delete_kwargs: Final[dict[str, Any]] = {
delete_kwargs: Final[_DeleteRequestKwargs] = {
"url": url,
"headers": headers,
"timeout": timeout,
@ -2984,7 +2999,7 @@ class BaseLLMHTTPHandler:
},
)
delete_kwargs: Final[dict[str, Any]] = {
delete_kwargs: Final[_DeleteRequestKwargs] = {
"url": url,
"headers": headers,
"timeout": timeout,
@ -3725,7 +3740,7 @@ class BaseLLMHTTPHandler:
timeout: float | httpx.Timeout | None,
) -> httpx.Response:
headers: Final = {**base_headers, "Content-Type": content_type}
kwargs: Final[dict[str, Any]] = {
kwargs: Final[_MediaUploadKwargs] = {
"headers": headers,
"content": self._iter_in_blocks(body_stream.iter_bytes(), self._MEDIA_UPLOAD_BLOCK_SIZE),
}
@ -3762,7 +3777,7 @@ class BaseLLMHTTPHandler:
break
yield cast(bytes, block)
kwargs: Final[dict[str, Any]] = {"headers": headers, "content": _abody()}
kwargs: Final[_MediaUploadKwargs] = {"headers": headers, "content": _abody()}
if timeout is not None:
kwargs["timeout"] = timeout
resp: Final = await client.client.post(url, **kwargs)
@ -5242,7 +5257,7 @@ class BaseLLMHTTPHandler:
def _wrap_responses_response_as_fake_stream(
self,
result: Any,
result: ResponsesAPIResponse,
model: str,
responses_api_provider_config: BaseResponsesAPIConfig,
logging_obj: "LiteLLMLoggingObj",
@ -5365,7 +5380,7 @@ class BaseLLMHTTPHandler:
async def _call_agentic_completion_hooks(
self,
response: Any,
response: object,
model: str,
messages: list[dict],
anthropic_messages_provider_config: "BaseAnthropicMessagesConfig",
@ -5536,7 +5551,7 @@ class BaseLLMHTTPHandler:
async def _call_agentic_chat_completion_hooks(
self,
response: Any,
response: ModelResponse,
model: str,
messages: list[dict],
optional_params: dict,
@ -5760,14 +5775,14 @@ class BaseLLMHTTPHandler:
@staticmethod
async def _open_realtime_backend_ws(
websockets_module: Any,
websockets_module: ModuleType,
url: str,
headers: dict,
ssl_context: Any,
ssl_context: bool | str | ssl.SSLContext,
*,
open_timeout: float = 8.0,
max_attempts: int = 3,
) -> Any:
) -> "ClientConnection":
"""Open the backend realtime websocket, retrying a hung open handshake.
The upstream Live handshake (e.g. Gemini Live) intermittently hangs on
@ -5826,7 +5841,6 @@ class BaseLLMHTTPHandler:
query_params: RealtimeQueryParams | None = None,
):
import websockets
from websockets.asyncio.client import ClientConnection
url: Final = provider_config.get_complete_url(api_base, model, api_key)
headers = provider_config.validate_environment(
@ -5844,12 +5858,12 @@ class BaseLLMHTTPHandler:
ssl_context.verify_mode = ssl.CERT_NONE
backend_ws: Final = await self._open_realtime_backend_ws(websockets, url, headers, ssl_context)
async with backend_ws:
_request_data: Final[dict[str, Any]] = {}
_request_data: Final[dict[str, object]] = {}
if litellm_metadata:
_request_data["litellm_metadata"] = litellm_metadata
realtime_streaming: Final = RealTimeStreaming(
websocket,
cast(ClientConnection, backend_ws),
backend_ws,
logging_obj,
provider_config,
model,
@ -6008,7 +6022,7 @@ class BaseLLMHTTPHandler:
)
else:
url = provider_config.get_complete_url(api_base=api_base, model=model or "", api_version=api_version)
headers: dict[str, Any] = provider_config.validate_environment(
headers: dict[str, object] = provider_config.validate_environment(
headers={}, model=model or "", api_key=api_key
)
else:
@ -6079,7 +6093,7 @@ class BaseLLMHTTPHandler:
if provider_config is not None:
url = provider_config.get_realtime_calls_url(api_base=api_base, model=model or "", api_version=api_version)
headers: dict[str, Any] = provider_config.get_realtime_calls_headers(ephemeral_key=openai_ephemeral_key)
headers: dict[str, object] = provider_config.get_realtime_calls_headers(ephemeral_key=openai_ephemeral_key)
else:
url = f"{api_base.rstrip('/')}/v1/realtime/calls"
headers = {
@ -6247,7 +6261,7 @@ class BaseLLMHTTPHandler:
yield backend
async with _backend_connection() as backend_ws:
_request_data: Final[dict[str, Any]] = {}
_request_data: Final[dict[str, object]] = {}
if litellm_metadata:
_request_data["litellm_metadata"] = litellm_metadata
@ -9444,7 +9458,7 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
extra_body=extra_body,
)
all_optional_params: Final[dict[str, Any]] = dict(litellm_params)
all_optional_params: Final[dict[str, object]] = dict(litellm_params)
all_optional_params.update(vector_store_search_optional_params or {})
headers, signed_json_body = vector_store_provider_config.sign_request(
headers=headers,
@ -9540,7 +9554,7 @@ class BaseLLMHTTPHandler:
extra_body=extra_body,
)
all_optional_params: Final[dict[str, Any]] = dict(litellm_params)
all_optional_params: Final[dict[str, object]] = dict(litellm_params)
all_optional_params.update(vector_store_search_optional_params or {})
headers, signed_json_body = vector_store_provider_config.sign_request(
@ -9860,7 +9874,7 @@ class BaseLLMHTTPHandler:
url: Final = api_base
params: Final[dict[str, Any]] = {}
params: Final[dict[str, object]] = {}
if after is not None:
params["after"] = after
if before is not None:
@ -9938,7 +9952,7 @@ class BaseLLMHTTPHandler:
url: Final = api_base
params: Final[dict[str, Any]] = {}
params: Final[dict[str, object]] = {}
if after is not None:
params["after"] = after
if before is not None:

2
litellm/llms/github_copilot/chat/transformation.py
View file

@ -277,7 +277,7 @@ class GithubCopilotConfig(OpenAIConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> "ModelResponse":

23
litellm/llms/oci/chat/transformation.py
View file

@ -10,7 +10,7 @@ implement the LiteLLM BaseConfig interface. Heavy-lifting lives in:
"""
import json
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Callable, Iterator
from typing import TYPE_CHECKING, Any, Final
import httpx
@ -713,8 +713,25 @@ class OCIChatConfig(BaseConfig):
class OCIStreamWrapper(CustomStreamWrapper):
"""Custom stream wrapper that dispatches OCI SSE chunks to the correct handler."""
def __init__(self, **kwargs: Any):
super().__init__(**kwargs)
def __init__(
self,
completion_stream: object,
model: str,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: str | None = None,
stream_options: object = None,
make_call: Callable[..., object] | None = None,
_response_headers: dict[str, object] | None = None,
) -> None:
super().__init__(
completion_stream=completion_stream,
model=model,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
stream_options=stream_options,
make_call=make_call,
_response_headers=_response_headers,
)
# Tracks whether any prior Cohere chunk in this stream has emitted
# tool calls. The Cohere handler uses this to decide whether the
# terminal consolidation chunk's tool calls are duplicates (suppress)

4
litellm/llms/openai/completion/handler.py
View file

@ -217,7 +217,7 @@ class OpenAITextCompletion(BaseLLM):
def streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_key: str,
data: dict,
headers: dict,
@ -274,7 +274,7 @@ class OpenAITextCompletion(BaseLLM):
async def async_streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_key: str,
data: dict,
headers: dict,

72
litellm/llms/openai/openai.py
View file

@ -1,6 +1,6 @@
import time
import types
from collections.abc import AsyncIterator, Callable, Coroutine, Iterable, Iterator
from collections.abc import AsyncIterator, Callable, Coroutine, Iterable, Iterator, Mapping
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, cast
from urllib.parse import urlparse
@ -61,16 +61,17 @@ class MistralEmbeddingConfig:
def __init__(
self,
) -> None:
locals_: Final = locals().copy()
locals_: Final[Mapping[str, object]] = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@classmethod
def get_config(cls):
config_attrs: Final[Mapping[str, object]] = cls.__dict__
return {
k: v
for k, v in cls.__dict__.items()
for k, v in config_attrs.items()
if not k.startswith("__")
and not isinstance(
v,
@ -153,7 +154,7 @@ class OpenAIConfig(BaseConfig):
top_p: int | None = None,
response_format: dict | None = None,
) -> None:
locals_: Final = locals().copy()
locals_: Final[Mapping[str, object]] = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@ -261,7 +262,7 @@ class OpenAIConfig(BaseConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> ModelResponse:
@ -299,7 +300,7 @@ class OpenAIConfig(BaseConfig):
streaming_response: Iterator[str] | AsyncIterator[str] | ModelResponse,
sync_stream: bool,
json_mode: bool | None = False,
) -> Any:
) -> "OpenAIChatCompletionResponseIterator":
return OpenAIChatCompletionResponseIterator(
streaming_response=streaming_response,
sync_stream=sync_stream,
@ -478,14 +479,14 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
async def _call_agentic_completion_hooks_openai(
self,
response: Any,
response: object,
model: str,
messages: list[dict],
optional_params: dict,
logging_obj: LiteLLMLoggingObj,
stream: bool,
litellm_params: dict,
) -> Any | None:
) -> object | None:
"""
Call agentic completion hooks for all custom loggers (OpenAI Chat Completions API).
@ -536,7 +537,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
kwargs_with_provider["custom_llm_provider"] = custom_llm_provider
# For OpenAI Chat Completions, use the chat completion agentic loop method
agentic_response = await callback.async_run_chat_completion_agentic_loop(
agentic_response: object = await callback.async_run_chat_completion_agentic_loop(
tools=tool_calls,
model=model,
messages=messages,
@ -580,7 +581,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
timeout: float | httpx.Timeout,
optional_params: dict,
litellm_params: dict,
logging_obj: Any,
logging_obj: LiteLLMLoggingObj,
model: str | None = None,
messages: list | None = None,
print_verbose: Callable | None = None,
@ -1590,7 +1591,7 @@ class OpenAIFilesAPI(BaseLLM):
client: OpenAI | AsyncOpenAI | None = None,
_is_async: bool = False,
) -> OpenAI | AsyncOpenAI | None:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
openai_client: OpenAI | AsyncOpenAI | None = None
if client is None:
data: Final = {}
@ -1628,7 +1629,7 @@ class OpenAIFilesAPI(BaseLLM):
max_retries: int | None,
organization: str | None,
client: OpenAI | AsyncOpenAI | None = None,
) -> OpenAIFileObject | Coroutine[Any, Any, OpenAIFileObject]:
) -> OpenAIFileObject | Coroutine[None, None, OpenAIFileObject]:
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
@ -1670,7 +1671,7 @@ class OpenAIFilesAPI(BaseLLM):
max_retries: int | None,
organization: str | None,
client: OpenAI | AsyncOpenAI | None = None,
) -> HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent]:
) -> HttpxBinaryResponseContent | Coroutine[None, None, HttpxBinaryResponseContent]:
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
@ -1948,7 +1949,7 @@ class OpenAIBatchesAPI(BaseLLM):
client: OpenAI | AsyncOpenAI | None = None,
_is_async: bool = False,
) -> OpenAI | AsyncOpenAI | None:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
openai_client: OpenAI | AsyncOpenAI | None = None
if client is None:
data: Final = {}
@ -1986,7 +1987,7 @@ class OpenAIBatchesAPI(BaseLLM):
max_retries: int | None,
organization: str | None,
client: OpenAI | AsyncOpenAI | None = None,
) -> LiteLLMBatch | Coroutine[Any, Any, LiteLLMBatch]:
) -> LiteLLMBatch | Coroutine[None, None, LiteLLMBatch]:
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
@ -2160,7 +2161,7 @@ class OpenAIAssistantsAPI(BaseLLM):
organization: str | None,
client: OpenAI | None = None,
) -> OpenAI:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
if client is None:
data: Final = {}
for k, v in received_args.items():
@ -2185,7 +2186,7 @@ class OpenAIAssistantsAPI(BaseLLM):
organization: str | None,
client: AsyncOpenAI | None = None,
) -> AsyncOpenAI:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
if client is None:
data: Final = {}
for k, v in received_args.items():
@ -2848,7 +2849,7 @@ class OpenAIAssistantsAPI(BaseLLM):
assistant_id: str,
additional_instructions: str | None,
instructions: str | None,
metadata: dict | None,
metadata: dict[str, str] | None,
model: str | None,
stream: bool | None,
tools: Iterable[AssistantToolParam] | None,
@ -2912,23 +2913,32 @@ class OpenAIAssistantsAPI(BaseLLM):
assistant_id: str,
additional_instructions: str | None,
instructions: str | None,
metadata: dict | None,
metadata: dict[str, str] | None,
model: str | None,
tools: Iterable[AssistantToolParam] | None,
event_handler: AssistantEventHandler | None,
) -> AssistantStreamManager[AssistantEventHandler]:
data: Final[dict[str, Any]] = {
"thread_id": thread_id,
"assistant_id": assistant_id,
"additional_instructions": additional_instructions,
"instructions": instructions,
"metadata": metadata,
"model": model,
"tools": tools,
}
runs_stream: Final = client.beta.threads.runs.stream
if event_handler is not None:
data["event_handler"] = event_handler
return client.beta.threads.runs.stream(**data)
return runs_stream(
thread_id=thread_id,
assistant_id=assistant_id,
additional_instructions=additional_instructions,
instructions=instructions,
metadata=metadata,
model=model,
tools=tools,
event_handler=event_handler,
)
return runs_stream(
thread_id=thread_id,
assistant_id=assistant_id,
additional_instructions=additional_instructions,
instructions=instructions,
metadata=metadata,
model=model,
tools=tools,
)
# fmt: off
@ -2984,7 +2994,7 @@ class OpenAIAssistantsAPI(BaseLLM):
assistant_id: str,
additional_instructions: str | None,
instructions: str | None,
metadata: dict | None,
metadata: dict[str, str] | None,
model: str | None,
stream: bool | None,
tools: Iterable[AssistantToolParam] | None,

5
litellm/llms/openai_like/chat/handler.py
View file

@ -12,6 +12,7 @@ import httpx
import litellm
from litellm import LlmProviders
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.bedrock.chat.invoke_handler import MockResponseIterator
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.databricks.streaming_utils import ModelResponseIterator
@ -112,7 +113,7 @@ class OpenAILikeChatHandler(OpenAILikeBase):
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLoggingObj,
stream,
data: dict,
optional_params=None,
@ -214,7 +215,7 @@ class OpenAILikeChatHandler(OpenAILikeBase):
print_verbose: Callable,
encoding,
api_key: str | None,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
acompletion=None,
litellm_params: dict = {},

5
litellm/llms/predibase/chat/handler.py
View file

@ -9,6 +9,7 @@ from typing import Final
import httpx
import litellm
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
get_async_httpx_client,
@ -59,7 +60,7 @@ class PredibaseChatCompletion:
print_verbose: Callable,
encoding,
api_key: str,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
litellm_params: dict,
tenant_id: str,
@ -250,7 +251,7 @@ class PredibaseChatCompletion:
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLoggingObj,
data: dict,
timeout: float | httpx.Timeout,
optional_params=None,

5
litellm/llms/replicate/chat/handler.py
View file

@ -6,6 +6,7 @@ from typing import Final
import litellm
from litellm.constants import REPLICATE_POLLING_DELAY_SECONDS
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,
@ -128,7 +129,7 @@ def completion(
print_verbose: Callable,
optional_params: dict,
litellm_params: dict,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_key,
encoding,
custom_prompt_dict={},
@ -246,7 +247,7 @@ async def async_completion(
input_data,
api_key,
api_base,
logging_obj,
logging_obj: LiteLLMLoggingObj,
print_verbose,
headers: dict,
) -> ModelResponse | CustomStreamWrapper:

8
litellm/llms/sagemaker/completion/handler.py
View file

@ -8,6 +8,7 @@ import httpx
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.custom_httpx.http_handler import (
_get_httpx_client,
@ -138,7 +139,7 @@ class SagemakerLLM(BaseAWSLLM):
model_response: ModelResponse,
print_verbose: Callable,
encoding,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
litellm_params: dict,
timeout: float | httpx.Timeout | None = None,
@ -431,17 +432,18 @@ class SagemakerLLM(BaseAWSLLM):
if not prepared_request.body:
raise ValueError("Prepared request body is empty")
stream_logging_obj: Final[LiteLLMLoggingObj] = logging_obj
completion_stream: Final = await self.make_async_call(
api_base=prepared_request.url,
headers=prepared_request.headers,
data=cast(str, prepared_request.body),
logging_obj=logging_obj,
logging_obj=stream_logging_obj,
)
streaming_response: Final = CustomStreamWrapper(
completion_stream=completion_stream,
model=model,
custom_llm_provider="sagemaker",
logging_obj=logging_obj,
logging_obj=stream_logging_obj,
)
# LOGGING

41
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -3,7 +3,7 @@
## Initial implementation - covers gemini + image gen calls
import json
import time
from collections.abc import Callable, Mapping
from collections.abc import Callable, Mapping, Sequence
from copy import deepcopy
from functools import partial
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast
@ -208,7 +208,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
presence_penalty: float | None = None,
seed: int | None = None,
) -> None:
locals_: Final = locals().copy()
locals_: Final[Mapping[str, object]] = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@ -1427,7 +1427,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
@staticmethod
def _extract_server_side_tool_invocations(
parts: list[HttpxPartType],
) -> list[dict[str, Any]] | None:
) -> list[dict[str, object]] | None:
"""Extract server-side tool invocations (toolCall/toolResponse) from parts.
These are returned by Gemini when context circulation is enabled
@ -1438,15 +1438,15 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
Returns:
List of server-side invocation dicts if any found, None otherwise.
"""
invocations: Final[list[dict[str, Any]]] = []
invocations: Final[list[dict[str, object]]] = []
# Index toolCalls by id so we can pair them with responses
tool_calls_by_id: Final[dict[str, dict[str, Any]]] = {}
tool_responses_by_id: Final[dict[str, dict[str, Any]]] = {}
tool_calls_by_id: Final[dict[str, dict[str, object]]] = {}
tool_responses_by_id: Final[dict[str, dict[str, object]]] = {}
for part in parts:
if "toolCall" in part:
tc = part["toolCall"]
entry: dict[str, Any] = {
entry: dict[str, object] = {
"tool_type": tc.get("toolType"),
"id": tc.get("id"),
"args": tc.get("args"),
@ -1753,7 +1753,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
response_tokens_details: CompletionTokensDetailsWrapper | None = None
usage_metadata: Final = completion_response["usageMetadata"]
def _get_token_count(detail: Mapping[str, Any]) -> int:
def _get_token_count(detail: Mapping[str, object]) -> int:
raw_token_count: Final = detail.get("tokenCount", detail.get("token_count", 0))
return raw_token_count if isinstance(raw_token_count, int) else 0
@ -2068,7 +2068,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
)
@staticmethod
def _get_stream_chunk_attr(chunk: Any, field_name: str) -> Any:
def _get_stream_chunk_attr(chunk: object, field_name: str) -> object:
if isinstance(chunk, dict):
value = chunk.get(field_name)
if value is not None:
@ -2110,10 +2110,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
def apply_assembled_streaming_response_metadata(
self,
response: ModelResponse,
chunks: list[Any],
chunks: list[object],
) -> None:
for field_name in VERTEX_AI_PROVIDER_METADATA_FIELDS:
merged: list[Any] = []
merged: list[object] = []
for chunk in chunks:
value = VertexGeminiConfig._get_stream_chunk_attr(chunk, field_name)
if not value:
@ -2214,8 +2214,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
functions: ChatCompletionToolCallFunctionChunk | None = None
thinking_blocks: list[ChatCompletionThinkingBlock] | None = None
reasoning_content: str | None = None
thought_signatures: Any | None = None
server_side_tool_invocations: list[dict[str, Any]] | None = None
thought_signatures: Sequence[str] | None = None
server_side_tool_invocations: list[dict[str, object]] | None = None
for idx, candidate in enumerate(_candidates):
if "content" not in candidate:
@ -2370,7 +2370,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> ModelResponse:
@ -2486,7 +2486,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
## ADD SERVICE TIER ##
if getattr(raw_response, "headers", None):
if service_tier := raw_response.headers.get("x-gemini-service-tier"):
service_tier: Final[str | None] = raw_response.headers.get("x-gemini-service-tier")
if service_tier:
if service_tier.lower() == "standard":
setattr(model_response, "service_tier", "default")
else:
@ -2660,7 +2661,7 @@ class VertexLLM(VertexBase):
print_verbose: Callable,
data: dict,
timeout: float | httpx.Timeout | None,
encoding,
encoding: object,
logging_obj,
stream,
optional_params: dict,
@ -2756,7 +2757,7 @@ class VertexLLM(VertexBase):
"vertex_ai", "vertex_ai_beta", "gemini"
], # if it's vertex_ai or gemini (google ai studio)
timeout: float | httpx.Timeout | None,
encoding,
encoding: object,
logging_obj,
stream,
optional_params: dict,
@ -2873,7 +2874,7 @@ class VertexLLM(VertexBase):
custom_llm_provider: Literal[
"vertex_ai", "vertex_ai_beta", "gemini"
], # if it's vertex_ai or gemini (google ai studio)
encoding,
encoding: object,
logging_obj,
optional_params: dict,
acompletion: bool,
@ -3122,7 +3123,7 @@ class ModelResponseIterator:
def _apply_stream_candidates(
self,
_candidates: list[Candidates],
model_response: Any,
model_response: "ModelResponseStream",
) -> tuple[list[dict], list[dict], list[dict], list[dict]]:
(
grounding_metadata,
@ -3200,7 +3201,7 @@ class ModelResponseIterator:
def _apply_stream_usage_metadata(
self,
processed_chunk: Any,
processed_chunk: GenerateContentResponseBody,
model_response: Any,
grounding_metadata: list[dict],
) -> Usage | None:

2
litellm/llms/vertex_ai/vertex_ai_non_gemini.py
View file

@ -28,7 +28,7 @@ class TextStreamer:
Fake streaming iterator for Vertex AI Model Garden calls
"""
def __init__(self, text):
def __init__(self, text: str):
self.text = text.split() # let's assume words as a streaming unit
self.index = 0

273
litellm/main.py
View file

@ -19,12 +19,12 @@ import random
import sys
import time
import traceback
from collections.abc import AsyncIterator, Coroutine, Iterable, Mapping
from collections.abc import AsyncIterator, Coroutine, Iterable, Mapping, Sequence
from concurrent import futures
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from copy import deepcopy
from functools import partial
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast, get_args
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Protocol, Union, cast, get_args
from litellm._logging import _redact_string
from litellm._uuid import uuid
@ -504,6 +504,7 @@ async def acompletion(
model=model,
custom_llm_provider=cast(str | None, custom_llm_provider), # cast-ok: read from untyped kwargs
tools=tools,
enable_prompt_caching=cast(bool | None, kwargs.get("enable_prompt_caching")), # cast-ok: untyped kwargs
)
if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (
@ -596,7 +597,7 @@ async def acompletion(
_, custom_llm_provider, _, _ = get_llm_provider(
model=model,
custom_llm_provider=custom_llm_provider,
api_base=completion_kwargs.get("base_url", None),
api_base=base_url,
)
fallbacks = fallbacks or litellm.model_fallbacks
@ -633,10 +634,10 @@ async def acompletion(
init_response: Final = await loop.run_in_executor(None, func_with_context)
if isinstance(init_response, dict) or isinstance(init_response, ModelResponse): ## CACHING SCENARIO
if isinstance(init_response, dict):
response = ModelResponse(**init_response)
response = _model_response_from_cached_dict(init_response)
response = init_response
elif asyncio.iscoroutine(init_response):
response = await init_response
response = await _resolve_dispatched_chat_response(init_response)
else:
response = init_response
@ -698,6 +699,20 @@ async def acompletion(
)
async def _resolve_dispatched_chat_response(
pending: Coroutine[object, object, ModelResponse | CustomStreamWrapper],
) -> ModelResponse | CustomStreamWrapper:
return await pending
def _model_response_from_cached_dict(cached_response_dict: Mapping[str, object]) -> ModelResponse:
return ModelResponse(**cached_response_dict)
def _transcription_response_from_cached_dict(cached_response_dict: Mapping[str, object]) -> TranscriptionResponse:
return TranscriptionResponse(**cached_response_dict)
async def _async_streaming(response, model, custom_llm_provider, args):
try:
print_verbose(f"received response in _async_streaming: {response}")
@ -983,12 +998,12 @@ def responses_api_bridge_check(
model: str,
custom_llm_provider: str,
web_search_options: OpenAIWebSearchOptions | None = None,
tools: list[Any] | None = None,
reasoning_effort: Any | None = None,
reasoning_summary: Any | None = None,
tools: Sequence[Mapping[str, object]] | None = None,
reasoning_effort: str | Mapping[str, object] | None = None,
reasoning_summary: object | None = None,
api_base: str | None = None,
) -> tuple[dict, str]:
model_info: dict[str, Any] = {}
model_info: dict[str, object] = {}
# Global flag: route ALL OpenAI chat completions through Responses API.
# Returns early with minimal model_info; callers only inspect the "mode" key.
@ -1110,6 +1125,22 @@ def _drop_input_examples_from_tools(
return cleaned_tools
class _ProxyAuthHeadersProvider(Protocol):
def get_auth_headers(self) -> Mapping[str, str]: ...
def _proxy_auth_headers(proxy_auth: _ProxyAuthHeadersProvider) -> Mapping[str, str]:
return proxy_auth.get_auth_headers()
def _provider_config_items(config: Mapping[str, object]) -> Iterable[tuple[str, object]]:
return config.items()
def _locals_snapshot(values: Mapping[str, object]) -> Mapping[str, object]:
return values
def _build_custom_pricing_entry(
custom_llm_provider: str,
kwargs: dict,
@ -1185,13 +1216,31 @@ def _register_custom_pricing_for_request(
)
def _dispatch_metadata(ctx: _CompletionDispatchContext) -> Mapping[str, object] | None:
return ctx.metadata
def _dispatch_client_http(ctx: _CompletionDispatchContext) -> HTTPHandler | AsyncHTTPHandler | None:
return ctx.client
def _dispatch_client_azure(
ctx: _CompletionDispatchContext,
) -> openai.AzureOpenAI | openai.AsyncAzureOpenAI | HTTPHandler | AsyncHTTPHandler | None:
return ctx.client
def _dispatch_client_openai(ctx: _CompletionDispatchContext) -> openai.OpenAI | openai.AsyncOpenAI | None:
return ctx.client
def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
_azure_detection_model: Final = ctx._azure_detection_model
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
api_version = ctx.api_version
client: Final = ctx.client
client: Final = _dispatch_client_azure(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
@ -1232,7 +1281,8 @@ def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
"AZURE_AD_TOKEN"
)
azure_ad_token_provider: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider_value: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider: Final = azure_ad_token_provider_value if callable(azure_ad_token_provider_value) else None
headers = headers or litellm.headers
@ -1244,7 +1294,7 @@ def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
if litellm.AzureOpenAIO1Config().is_o_series_model(model=_azure_detection_model):
## LOAD CONFIG - if set
config = litellm.AzureOpenAIO1Config.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > azure_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1273,7 +1323,7 @@ def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
else:
## LOAD CONFIG - if set
config = litellm.AzureOpenAIConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > azure_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1323,7 +1373,7 @@ def _complete_azure_text(ctx: _CompletionDispatchContext) -> _CompletionDispatch
api_base = ctx.api_base
api_key = ctx.api_key
api_version = ctx.api_version
client: Final = ctx.client
client: Final = _dispatch_client_azure(ctx)
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1358,7 +1408,8 @@ def _complete_azure_text(ctx: _CompletionDispatchContext) -> _CompletionDispatch
"AZURE_AD_TOKEN"
)
azure_ad_token_provider: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider_value: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider: Final = azure_ad_token_provider_value if callable(azure_ad_token_provider_value) else None
headers = headers or litellm.headers
@ -1367,7 +1418,7 @@ def _complete_azure_text(ctx: _CompletionDispatchContext) -> _CompletionDispatch
## LOAD CONFIG - if set
config: Final = litellm.AzureOpenAIConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > azure_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1415,7 +1466,7 @@ def _complete_deepseek(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1466,7 +1517,7 @@ def _complete_azure_ai(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
@ -1622,7 +1673,7 @@ def _complete_text_completion_openai(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_openai(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1654,7 +1705,7 @@ def _complete_text_completion_openai(
## LOAD CONFIG - if set
config: Final = litellm.OpenAITextCompletionConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_text_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1704,7 +1755,7 @@ def _complete_fireworks_ai(
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1755,7 +1806,7 @@ def _complete_heroku(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1805,7 +1856,7 @@ def _complete_ragflow(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1855,7 +1906,7 @@ def _complete_xai(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1906,7 +1957,7 @@ def _complete_groq(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1938,7 +1989,7 @@ def _complete_groq(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult
## LOAD CONFIG - if set
config: Final = litellm.GroqChatConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1970,7 +2021,7 @@ def _complete_bedrock_mantle(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1987,7 +2038,7 @@ def _complete_bedrock_mantle(
api_key = api_key or litellm.api_key or get_secret("BEDROCK_MANTLE_API_KEY")
headers = headers or litellm.headers
config: Final = litellm.BedrockMantleChatConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if k not in optional_params:
optional_params[k] = v
return base_llm_http_handler.completion(
@ -2014,7 +2065,7 @@ def _complete_a2a(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2077,7 +2128,7 @@ def _complete_gigachat(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2139,7 +2190,7 @@ def _complete_sap(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2155,7 +2206,7 @@ def _complete_sap(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
headers = headers or litellm.headers
## LOAD CONFIG - if set
config: Final = litellm.GenAIHubOrchestrationConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
@ -2187,7 +2238,7 @@ def _complete_aiohttp_openai(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
@ -2242,7 +2293,7 @@ def _complete_cometapi(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2291,7 +2342,7 @@ def _complete_minimax(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2337,7 +2388,7 @@ def _complete_hosted_vllm(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2383,7 +2434,7 @@ def _complete_custom_openai(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
custom_prompt_dict: Final = ctx.custom_prompt_dict
extra_headers = ctx.extra_headers
@ -2392,7 +2443,7 @@ def _complete_custom_openai(
logger_fn: Final = ctx.logger_fn
logging: Final = ctx.logging
messages: Final = ctx.messages
metadata: Final = ctx.metadata
metadata: Final = _dispatch_metadata(ctx)
model: Final = ctx.model
model_response: Final = ctx.model_response
optional_params: Final = ctx.optional_params
@ -2445,7 +2496,7 @@ def _complete_custom_openai(
## LOAD CONFIG - if set
config: Final = litellm.OpenAIConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
@ -2522,7 +2573,7 @@ def _complete_mistral(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2673,7 +2724,7 @@ def _complete_anthropic(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
custom_prompt_dict = ctx.custom_prompt_dict
headers: Final = ctx.headers
@ -2972,7 +3023,7 @@ def _complete_huggingface(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3015,7 +3066,7 @@ def _complete_oci(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3050,7 +3101,7 @@ def _complete_compactifai(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3126,7 +3177,7 @@ def _complete_databricks(ctx: _CompletionDispatchContext) -> _CompletionDispatch
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -3198,7 +3249,7 @@ def _complete_datarobot(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3235,7 +3286,7 @@ def _complete_openrouter(ctx: _CompletionDispatchContext) -> _CompletionDispatch
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -3273,7 +3324,7 @@ def _complete_openrouter(ctx: _CompletionDispatchContext) -> _CompletionDispatch
## Load Config
config: Final = litellm.OpenrouterConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if k == "extra_body":
# we use openai 'extra_body' to pass openrouter specific params - transforms, route, models
if "extra_body" in optional_params:
@ -3314,7 +3365,7 @@ def _complete_vercel_ai_gateway(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -3351,7 +3402,7 @@ def _complete_vercel_ai_gateway(
## Load Config
config: Final = litellm.VercelAIGatewayConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if k == "extra_body":
# we use openai 'extra_body' to pass vercel specific params - providerOptions
if "extra_body" in optional_params:
@ -3392,7 +3443,7 @@ def _complete_vertex_ai_beta(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3457,7 +3508,7 @@ def _complete_vertex_ai_beta(
def _complete_vertex_ai(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
custom_prompt_dict: Final = ctx.custom_prompt_dict
headers: Final = ctx.headers
@ -3754,7 +3805,7 @@ def _complete_text_completion_inception(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_openai(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logger_fn: Final = ctx.logger_fn
@ -3818,7 +3869,7 @@ def _complete_sagemaker_chat(
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3881,7 +3932,7 @@ def _complete_bedrock(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_prompt_dict = ctx.custom_prompt_dict
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4005,7 +4056,7 @@ def _complete_watsonx(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_prompt_dict: Final = ctx.custom_prompt_dict
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4044,7 +4095,7 @@ def _complete_watsonx_text(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -4156,7 +4207,7 @@ def _complete_ollama(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -4196,7 +4247,7 @@ def _complete_ollama_chat(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -4311,7 +4362,7 @@ def _complete_cloudflare(ctx: _CompletionDispatchContext) -> _CompletionDispatch
def _complete_petals(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
api_base = ctx.api_base
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
litellm_params: Final = ctx.litellm_params
logger_fn: Final = ctx.logger_fn
logging: Final = ctx.logging
@ -4353,7 +4404,7 @@ def _complete_snowflake(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client = ctx.client
client = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4441,7 +4492,7 @@ def _complete_gdc(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4480,7 +4531,7 @@ def _complete_bytez(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4520,7 +4571,7 @@ def _complete_lemonade(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4560,7 +4611,7 @@ def _complete_ovhcloud(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4603,6 +4654,10 @@ def _complete_ovhcloud(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
return response
def _custom_api_first_output(resp: httpx.Response | None) -> str:
return resp.json()["data"][0]["output"][0]
def _complete_custom(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
api_base: Final = ctx.api_base
headers: Final = ctx.headers
@ -4651,7 +4706,6 @@ def _complete_custom(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
**kwargs.get("extra_body", {}),
},
)
response_json: Final = resp.json()
"""
assume all responses from custom api_bases of this format:
{
@ -4665,7 +4719,7 @@ def _complete_custom(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
]
}
"""
string_response: Final = response_json["data"][0]["output"][0]
string_response: Final = _custom_api_first_output(resp)
## RESPONSE OBJECT
model_response.choices[0].message.content = string_response
model_response.created = int(time.time())
@ -4740,7 +4794,7 @@ def _complete_langgraph(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4789,7 +4843,7 @@ def _complete_langflow(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4947,7 +5001,7 @@ def completion(
thinking = validate_and_fix_thinking_param(thinking=thinking)
######### unpacking kwargs #####################
args: Final = locals()
args: Final = _locals_snapshot(locals())
# Set by the responses->completion fallback so completion() does not bridge
# back to the Responses API: that round-trip mutually recurses forever for a
@ -5038,7 +5092,7 @@ def completion(
# Inject proxy auth headers if configured
if litellm.proxy_auth is not None:
try:
proxy_headers: Final = litellm.proxy_auth.get_auth_headers()
proxy_headers: Final = _proxy_auth_headers(litellm.proxy_auth)
headers.update(proxy_headers)
except Exception as e:
verbose_logger.warning("Failed to get proxy auth headers: %s", e)
@ -5091,7 +5145,7 @@ def completion(
)
######## end of unpacking kwargs ###########
non_default_params: Final = get_non_default_completion_params(kwargs=kwargs)
litellm_params = {} # used to prevent unbound var errors
litellm_params: dict[str, object] = {} # used to prevent unbound var errors
## PROMPT MANAGEMENT HOOKS ##
from litellm.integrations.anthropic_cache_control_hook import (
@ -5105,6 +5159,7 @@ def completion(
model=model,
custom_llm_provider=cast(str | None, kwargs.get("custom_llm_provider")), # cast-ok: untyped kwargs
tools=tools,
enable_prompt_caching=cast(bool | None, kwargs.get("enable_prompt_caching")), # cast-ok: untyped kwargs
)
if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (
@ -5913,7 +5968,7 @@ def embedding(
*,
aembedding: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, EmbeddingResponse]:
) -> Coroutine[object, object, EmbeddingResponse]:
...
@ -5964,7 +6019,7 @@ def embedding(
litellm_call_id=None,
logger_fn=None,
**kwargs,
) -> EmbeddingResponse | Coroutine[Any, Any, EmbeddingResponse]:
) -> EmbeddingResponse | Coroutine[object, object, EmbeddingResponse]:
"""
Embedding function that calls an API to generate embeddings for the given input.
@ -6007,7 +6062,7 @@ def embedding(
# Inject proxy auth headers if configured
if litellm.proxy_auth is not None:
try:
proxy_headers: Final = litellm.proxy_auth.get_auth_headers()
proxy_headers: Final = _proxy_auth_headers(litellm.proxy_auth)
headers.update(proxy_headers)
except Exception as e:
verbose_logger.warning("Failed to get proxy auth headers: %s", e)
@ -6084,7 +6139,7 @@ def embedding(
if mock_response is not None:
return mock_embedding(model=model, mock_response=mock_response)
try:
response: EmbeddingResponse | Coroutine[Any, Any, EmbeddingResponse] | None = None
response: EmbeddingResponse | Coroutine[object, object, EmbeddingResponse] | None = None
if azure is True or custom_llm_provider == "azure":
# azure configs
@ -6387,7 +6442,7 @@ def embedding(
response = huggingface_embed.embedding(
model=model,
input=input,
encoding=_get_encoding(),
encoding=sys.modules[__name__].encoding,
api_key=api_key,
api_base=api_base,
logging_obj=logging,
@ -6990,6 +7045,20 @@ def embedding(
###### Text Completion ################
async def _resolve_dispatched_text_completion_response(
pending: Coroutine[
object,
object,
TextCompletionResponse | ModelResponse | CustomStreamWrapper | TextCompletionStreamWrapper,
],
) -> TextCompletionResponse | ModelResponse | CustomStreamWrapper | TextCompletionStreamWrapper:
return await pending
async def _resolve_pending_chat_response(pending: Coroutine[object, object, ModelResponse]) -> ModelResponse:
return await pending
@client
async def atext_completion(*args, **kwargs) -> TextCompletionResponse | TextCompletionStreamWrapper:
"""
@ -7015,7 +7084,7 @@ async def atext_completion(*args, **kwargs) -> TextCompletionResponse | TextComp
else:
response = init_response
elif asyncio.iscoroutine(init_response):
response = await init_response
response = await _resolve_dispatched_text_completion_response(init_response)
else:
response = init_response
@ -7040,7 +7109,7 @@ async def atext_completion(*args, **kwargs) -> TextCompletionResponse | TextComp
if isinstance(response, TextCompletionResponse):
return response
elif asyncio.iscoroutine(response):
response = await response
response = await _resolve_pending_chat_response(response)
text_completion_response = TextCompletionResponse()
text_completion_response = litellm.utils.LiteLLMResponseObjectHandler.convert_chat_to_text_completion(
@ -7330,11 +7399,11 @@ async def aadapter_completion(*, adapter_id: str, **kwargs) -> BaseModel | Adapt
async def aadapter_generate_content(
**kwargs,
) -> dict[str, Any] | AsyncIterator[bytes]:
) -> dict[str, object] | AsyncIterator[bytes]:
from litellm.google_genai.adapters.handler import GenerateContentToCompletionHandler
coro: Final = cast(
Coroutine[Any, Any, dict[str, Any] | AsyncIterator[bytes]],
Coroutine[object, object, dict[str, object] | AsyncIterator[bytes]],
GenerateContentToCompletionHandler.generate_content_handler(**kwargs, _is_async=True),
)
return await coro
@ -7486,7 +7555,7 @@ async def atranscription(*args, **kwargs) -> TranscriptionResponse:
# Await normally
init_response: Final = await loop.run_in_executor(None, func_with_context)
if isinstance(init_response, dict):
response = TranscriptionResponse(**init_response)
response = _transcription_response_from_cached_dict(init_response)
elif isinstance(init_response, TranscriptionResponse): ## CACHING SCENARIO
response = init_response
elif asyncio.iscoroutine(init_response):
@ -7541,7 +7610,7 @@ def transcription(
max_retries: int | None = None,
custom_llm_provider=None,
**kwargs,
) -> TranscriptionResponse | Coroutine[Any, Any, TranscriptionResponse]:
) -> TranscriptionResponse | Coroutine[object, object, TranscriptionResponse]:
"""
Calls openai + azure whisper endpoints.
@ -7608,7 +7677,7 @@ def transcription(
custom_llm_provider=custom_llm_provider,
)
response: TranscriptionResponse | Coroutine[Any, Any, TranscriptionResponse] | None = None
response: TranscriptionResponse | Coroutine[object, object, TranscriptionResponse] | None = None
provider_config: Final = ProviderConfigManager.get_provider_audio_transcription_config(
model=model,
@ -7842,7 +7911,7 @@ def speech(
custom_llm_provider: str | None = None,
aspeech: bool | None = None,
**kwargs,
) -> HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent]:
) -> HttpxBinaryResponseContent | Coroutine[object, object, HttpxBinaryResponseContent]:
user: Final = kwargs.get("user", None)
litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
proxy_server_request: Final = kwargs.get("proxy_server_request", None)
@ -7901,7 +7970,7 @@ def speech(
},
custom_llm_provider=custom_llm_provider,
)
response: HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent] | None = None
response: HttpxBinaryResponseContent | Coroutine[object, object, HttpxBinaryResponseContent] | None = None
if custom_llm_provider == "openai" or custom_llm_provider in litellm.openai_compatible_providers:
if voice is None or not (isinstance(voice, str)):
raise litellm.BadRequestError(
@ -8663,7 +8732,7 @@ def stream_chunk_builder(
]
if len(provider_specific_chunks) > 0:
combined_provider_fields: Final[dict[str, Any]] = {}
combined_provider_fields: Final[dict[str, object]] = {}
for chunk in provider_specific_chunks:
fields = chunk["choices"][0]["delta"]["provider_specific_fields"]
if isinstance(fields, dict):
@ -8728,7 +8797,7 @@ def stream_chunk_builder(
async def acount_tokens(
model: str,
messages: list[dict[str, Any]] | None = None,
messages: list[dict[str, object]] | None = None,
tools: list[dict[str, Any]] | None = None,
system: str | None = None,
api_key: str | None = None,
@ -8774,7 +8843,7 @@ async def acount_tokens(
api_base = dynamic_api_base
# Build deployment dict for the token counter
deployment: Final[dict[str, Any]] = {
deployment: Final[dict[str, object]] = {
"litellm_params": {
"model": model,
"api_key": api_key,
@ -8825,29 +8894,37 @@ async def acount_tokens(
# Cache for encoding to avoid repeated __getattr__ calls
_encoding_cache: Any | None = None
_encoding_cache: tiktoken.Encoding | None = None
def _get_encoding():
def _load_module_encoding() -> tiktoken.Encoding:
import sys
return sys.modules[__name__].encoding
def _get_encoding() -> tiktoken.Encoding:
"""Get encoding, loading it lazily if needed."""
global _encoding_cache
if _encoding_cache is None:
import sys
# Access via module to trigger __getattr__ if not cached
_encoding_cache = sys.modules[__name__].encoding
_encoding_cache = _load_module_encoding()
return _encoding_cache
def __getattr__(name: str) -> Any:
def _load_default_encoding() -> tiktoken.Encoding:
from litellm._lazy_imports import _get_default_encoding
return _get_default_encoding()
def __getattr__(name: str) -> tiktoken.Encoding:
"""Lazy import handler for main module"""
if name == "encoding":
# Use _get_default_encoding which properly sets TIKTOKEN_CACHE_DIR
# before loading tiktoken, ensuring the local cache is used
# instead of downloading from the internet
from litellm._lazy_imports import _get_default_encoding
_encoding: Final = _get_default_encoding()
_encoding: Final = _load_default_encoding()
# Cache it in the module's __dict__ for subsequent accesses
import sys

43
litellm/model_prices_and_context_window_backup.json
View file

@ -730,6 +730,7 @@
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -755,6 +756,7 @@
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -973,6 +975,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -1005,6 +1008,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1038,6 +1042,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1071,6 +1076,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1104,6 +1110,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1137,6 +1144,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1171,6 +1179,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1222,6 +1231,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1258,6 +1268,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1294,6 +1305,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1330,6 +1342,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1946,6 +1959,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -2203,6 +2217,7 @@
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2235,6 +2250,7 @@
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2267,6 +2283,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2299,6 +2316,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2331,6 +2349,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2363,6 +2382,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2430,6 +2450,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.2e-05,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2686,6 +2707,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2775,6 +2797,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -10826,6 +10849,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -10850,6 +10874,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -11005,6 +11030,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -11029,6 +11055,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -11811,6 +11838,7 @@
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -15998,6 +16026,7 @@
"input_cost_per_token": 1.1e-06,
"deprecation_date": "2026-10-15",
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -16213,6 +16242,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -22031,6 +22061,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.2e-05,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -22091,6 +22122,7 @@
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -26534,6 +26566,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -26563,6 +26596,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35259,6 +35293,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35417,6 +35452,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35451,6 +35487,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.44e-05,
"cache_read_input_token_cost_above_200k_tokens": 7.2e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35475,6 +35512,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35525,6 +35563,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35556,6 +35595,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35586,6 +35626,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -45975,6 +46016,7 @@
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 1.2e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -46000,6 +46042,7 @@
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 1.2e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,

1
litellm/models/verification_token.py
View file

@ -49,6 +49,7 @@ class LiteLLM_VerificationToken(LiteLLMPydanticObjectBase):
created_by: str | None = None
updated_at: datetime | None = None
updated_by: str | None = None
settings_updated_at: datetime | None = None
last_active: datetime | None = None
object_permission_id: str | None = None
object_permission: LiteLLM_ObjectPermissionTable | None = None

3
litellm/proxy/_experimental/mcp_server/oauth2_token_cache.py
View file

@ -145,9 +145,8 @@ class MCPOAuth2TokenCache(InMemoryCache):
server.server_id,
)
post_kwargs: Final = {"data": data, **({"headers": token_request.headers} if token_request.headers else {})}
try:
response: Final = await client.post(server.token_url, **post_kwargs)
response: Final = await client.post(server.token_url, data=data, headers=token_request.headers or None)
response.raise_for_status()
except httpx.HTTPStatusError as exc:
raise ValueError(

2
litellm/proxy/_types.py
View file

@ -1113,6 +1113,7 @@ class KeyRequestBase(GenerateRequestBase):
budget_id: str | None = None
tags: list[str] | None = None
disable_global_guardrails: bool | None = None
enable_prompt_caching: bool | None = None
throttle_on_budget_exceeded: bool | None = None
enforced_params: list[str] | None = None
allowed_routes: list | None = []
@ -4129,6 +4130,7 @@ LiteLLM_ManagementEndpoint_MetadataFields: Final = [
"enforced_batch_output_expires_after",
"enforced_file_expires_after",
"throttle_on_budget_exceeded",
"enable_prompt_caching",
]
LiteLLM_ManagementEndpoint_MetadataFields_Premium: Final = [

234
litellm/proxy/auth/auth_checks.py
View file

@ -13,8 +13,8 @@ import asyncio
import math
import re
import time
from collections.abc import Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, cast
from collections.abc import Iterator, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Protocol, cast
from fastapi import HTTPException, Request, status
from pydantic import BaseModel
@ -87,6 +87,7 @@ from litellm.proxy.utils import PrismaClient, ProxyLogging, log_db_metrics
from litellm.repositories.budget_repository import BudgetRepository
from litellm.repositories.object_permission_repository import ObjectPermissionRepository
from litellm.repositories.organization_repository import OrganizationRepository
from litellm.repositories.prisma_protocols import RowT_co
from litellm.repositories.project_repository import ProjectRepository
from litellm.repositories.table_repositories import (
AccessGroupRepository,
@ -110,11 +111,144 @@ from .auth_utils import get_model_from_request, get_request_route_template
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
Span = _Span | Any
Span = _Span
else:
Span = Any
class _PrismaDictableRow(Protocol):
def dict(self) -> Mapping[str, object]: ...
class _PrismaJWTKeyMappingRow(Protocol):
token: str
class _PrismaModelDumpRow(Protocol):
def model_dump(self) -> Mapping[str, object]: ...
class _PrismaTeamRow(Protocol):
def dict(self) -> Mapping[str, object]: ...
def model_dump(self) -> Mapping[str, object]: ...
class _PrismaVectorStoreRow(Protocol):
def dict(self) -> Mapping[str, object]: ...
def model_dump(self) -> Mapping[str, object]: ...
def __iter__(self) -> Iterator[tuple[str, object]]: ...
class _PrismaUserRow(Protocol):
user_id: str
organization_memberships: Sequence[LiteLLM_OrganizationMembershipTable | None] | None
def __iter__(self) -> Iterator[tuple[str, object]]: ...
class _PrismaAuthTable(Protocol[RowT_co]):
async def find_unique(
self, *, where: Mapping[str, object], include: Mapping[str, object] | None = None
) -> RowT_co | None: ...
async def find_first(
self, *, where: Mapping[str, object], include: Mapping[str, object] | None = None
) -> RowT_co | None: ...
async def find_many(
self,
*,
where: Mapping[str, object],
include: Mapping[str, object] | None = None,
take: int | None = None,
) -> Sequence[RowT_co]: ...
async def update(self, *, where: Mapping[str, object], data: Mapping[str, object]) -> RowT_co | None: ...
async def create(self, *, data: Mapping[str, object], include: Mapping[str, object] | None = None) -> RowT_co: ...
class _PrismaTableHolder(Protocol[RowT_co]):
@property
def table(self) -> _PrismaAuthTable[RowT_co]: ...
def _dictable_table(repo: _PrismaTableHolder[_PrismaDictableRow]) -> _PrismaAuthTable[_PrismaDictableRow]:
return repo.table
def _jwt_key_mapping_table(
repo: _PrismaTableHolder[_PrismaJWTKeyMappingRow],
) -> _PrismaAuthTable[_PrismaJWTKeyMappingRow]:
return repo.table
def _model_dump_table(repo: _PrismaTableHolder[_PrismaModelDumpRow]) -> _PrismaAuthTable[_PrismaModelDumpRow]:
return repo.table
def _team_table(repo: _PrismaTableHolder[_PrismaTeamRow]) -> _PrismaAuthTable[_PrismaTeamRow]:
return repo.table
def _vector_store_table(repo: _PrismaTableHolder[_PrismaVectorStoreRow]) -> _PrismaAuthTable[_PrismaVectorStoreRow]:
return repo.table
def _user_table(repo: _PrismaTableHolder[_PrismaUserRow]) -> _PrismaAuthTable[_PrismaUserRow]:
return repo.table
def _object_permission_table(
repo: _PrismaTableHolder[LiteLLM_ObjectPermissionTable],
) -> _PrismaAuthTable[LiteLLM_ObjectPermissionTable]:
return repo.table
class _PrismaTagRow(Protocol):
tag_name: str
def dict(self) -> Mapping[str, object]: ...
def _tag_table(repo: _PrismaTableHolder[_PrismaTagRow]) -> _PrismaAuthTable[_PrismaTagRow]:
return repo.table
class _RawCacheRead(Protocol):
async def async_get_cache(self, *, key: str) -> object: ...
def _raw_cache(cache: _RawCacheRead) -> _RawCacheRead:
return cache
class _BudgetCacheRead(Protocol):
async def async_get_cache(self, *, key: str) -> "LiteLLM_BudgetTable | Mapping[str, object] | None": ...
def _budget_cache(cache: _BudgetCacheRead) -> _BudgetCacheRead:
return cache
def _typed_request_body(request_body: dict) -> Mapping[str, object]:
return request_body
class _JsonLoadsObj(Protocol):
def __call__(self, data: str) -> object: ...
def _typed_json_loads(fn: _JsonLoadsObj) -> _JsonLoadsObj:
return fn
_safe_json_loads_obj: Final = _typed_json_loads(safe_json_loads)
last_db_access_time: Final = LimitedSizeOrderedDict(max_size=100)
db_cache_expiry: Final = DEFAULT_IN_MEMORY_TTL # refresh every 5s
@ -384,7 +518,7 @@ _GUARDRAIL_MODIFICATION_KEYS: Final[tuple] = (
)
def _guardrail_modification_check(request_body: dict, team_object: LiteLLM_TeamTable | None) -> None:
def _guardrail_modification_check(request_body: Mapping[str, object], team_object: LiteLLM_TeamTable | None) -> None:
"""
Reject user-supplied metadata flags that would modify guardrail behavior
unless the team has explicit permission. Checked keys include the plural
@ -399,7 +533,7 @@ def _guardrail_modification_check(request_body: dict, team_object: LiteLLM_TeamT
"""
from litellm.proxy.guardrails.guardrail_helpers import can_modify_guardrails
def _coerce_to_dict(container: Any) -> dict | None:
def _coerce_to_dict(container: object) -> dict | None:
"""Accept dict or JSON-string (from multipart/form-data or extra_body).
Without this, an attacker can smuggle guardrail keys past the check by
@ -411,11 +545,11 @@ def _guardrail_modification_check(request_body: dict, team_object: LiteLLM_TeamT
if isinstance(container, dict):
return container
if isinstance(container, str):
parsed: Final = safe_json_loads(container)
parsed: Final = _safe_json_loads_obj(container)
return parsed if isinstance(parsed, dict) else None
return None
def _user_requested_modification(container: Any) -> bool:
def _user_requested_modification(container: object) -> bool:
coerced: Final = _coerce_to_dict(container)
if coerced is None:
return False
@ -731,7 +865,7 @@ async def common_checks(
_enforce_user_param_check(general_settings, request, request_body, route)
_global_proxy_budget_check(global_proxy_spend, skip_all_budget_checks, route)
_guardrail_modification_check(request_body, team_object)
_guardrail_modification_check(_typed_request_body(request_body), team_object)
# 10 [OPTIONAL] Organization RBAC checks
organization_role_based_access_check(user_object=user_object, route=route, request_body=request_body)
@ -955,7 +1089,7 @@ async def get_default_end_user_budget(
# Fetch from database
try:
budget_record: Final = await BudgetRepository(prisma_client).table.find_unique(
budget_record: Final = await _dictable_table(BudgetRepository(prisma_client)).find_unique(
where={"budget_id": litellm.max_end_user_budget_id}
)
@ -1007,14 +1141,16 @@ async def get_team_member_default_budget(
cache_key: Final = f"team_member_default_budget:{budget_id}"
cached_budget: Final = await user_api_key_cache.async_get_cache(key=cache_key)
cached_budget: Final = await _budget_cache(user_api_key_cache).async_get_cache(key=cache_key)
if isinstance(cached_budget, LiteLLM_BudgetTable):
return cached_budget
if isinstance(cached_budget, dict):
return LiteLLM_BudgetTable.model_validate(cached_budget)
try:
budget_record: Final = await BudgetRepository(prisma_client).table.find_unique(where={"budget_id": budget_id})
budget_record: Final = await _dictable_table(BudgetRepository(prisma_client)).find_unique(
where={"budget_id": budget_id}
)
if budget_record is None:
verbose_proxy_logger.warning("Team-default member budget not found in database: %s", budget_id)
@ -1171,7 +1307,7 @@ async def get_end_user_object(
# Fetch from database
try:
response: Final = await EndUserRepository(prisma_client).table.find_unique(
response: Final = await _dictable_table(EndUserRepository(prisma_client)).find_unique(
where={"user_id": end_user_id},
include={"litellm_budget_table": True, "object_permission": True},
)
@ -1243,7 +1379,7 @@ async def resolve_and_validate_end_user_id(
return raw_end_user_id
cache_key: Final = f"end_user_validation:{raw_end_user_id}"
cached: Final = await user_api_key_cache.async_get_cache(key=cache_key)
cached: Final = await _raw_cache(user_api_key_cache).async_get_cache(key=cache_key)
if cached == "valid":
return raw_end_user_id
if cached == "invalid":
@ -1345,8 +1481,8 @@ async def get_tag_objects_batch(
if not tag_names:
return {}
tag_objects: Final = {}
uncached_tags: Final = []
tag_objects: Final = dict[str, LiteLLM_TagTable]()
uncached_tags: Final = list[str]()
# Try to get all tags from cache first
for tag_name in tag_names:
@ -1363,7 +1499,7 @@ async def get_tag_objects_batch(
# Batch fetch uncached tags from DB in one query
if uncached_tags:
try:
db_tags: Final = await TagRepository(prisma_client).table.find_many(
db_tags: Final = await _tag_table(TagRepository(prisma_client)).find_many(
where={"tag_name": {"in": uncached_tags}},
include={"litellm_budget_table": True},
)
@ -1457,7 +1593,7 @@ async def get_team_membership(
# else, check db
try:
response: Final = await TeamMembershipRepository(prisma_client).table.find_unique(
response: Final = await _dictable_table(TeamMembershipRepository(prisma_client)).find_unique(
where={"user_id_team_id": {"user_id": user_id, "team_id": team_id}},
include={"litellm_budget_table": True},
)
@ -1524,7 +1660,7 @@ def _should_check_db(key: str, last_db_access_time: LimitedSizeOrderedDict, db_c
return False
def _update_last_db_access_time(key: str, value: Any | None, last_db_access_time: LimitedSizeOrderedDict):
def _update_last_db_access_time(key: str, value: object | None, last_db_access_time: LimitedSizeOrderedDict):
last_db_access_time[key] = (value, time.time())
@ -1545,7 +1681,7 @@ def _get_role_based_permissions(
for role_based_permission in role_based_permissions:
if role_based_permission.role == rbac_role:
return getattr(role_based_permission, key)
return role_based_permission.models if key == "models" else role_based_permission.routes
return None
@ -1586,7 +1722,7 @@ async def _get_fuzzy_user_object(
prisma_client: PrismaClient,
sso_user_id: str | None = None,
user_email: str | None = None,
) -> LiteLLM_UserTable | None:
) -> "_PrismaUserRow | None":
"""
Checks if sso user is in db.
@ -1600,7 +1736,7 @@ async def _get_fuzzy_user_object(
response = None
if sso_user_id is not None:
response = await UserRepository(prisma_client).table.find_unique(
response = await _user_table(UserRepository(prisma_client)).find_unique(
where={"sso_user_id": sso_user_id},
include={"organization_memberships": True},
)
@ -1608,14 +1744,14 @@ async def _get_fuzzy_user_object(
if response is None and user_email is not None:
# Use case-insensitive query to handle emails with different casing
# This matches the pattern used in _check_duplicate_user_email
response = await UserRepository(prisma_client).table.find_first(
response = await _user_table(UserRepository(prisma_client)).find_first(
where={"user_email": {"equals": user_email, "mode": "insensitive"}},
include={"organization_memberships": True},
)
if response is not None and sso_user_id is not None: # update sso_user_id
asyncio.create_task( # background task to update user with sso id
UserRepository(prisma_client).table.update(
_user_table(UserRepository(prisma_client)).update(
where={"user_id": response.user_id},
data={"sso_user_id": sso_user_id},
)
@ -1698,7 +1834,7 @@ async def get_user_object(
)
if should_check_db:
response = await UserRepository(prisma_client).table.find_unique(
response = await _user_table(UserRepository(prisma_client)).find_unique(
where={"user_id": user_id}, include={"organization_memberships": True}
)
@ -1736,7 +1872,7 @@ async def get_user_object(
budget_duration=new_user_params["budget_duration"]
)
response = await UserRepository(prisma_client).table.create(
response = await _user_table(UserRepository(prisma_client)).create(
data=new_user_params,
include={"organization_memberships": True},
)
@ -1802,7 +1938,7 @@ async def get_user_object(
async def _cache_management_object(
key: str,
value: BaseModel | dict[str, Any],
value: BaseModel | Mapping[str, object],
user_api_key_cache: UserApiKeyCache,
proxy_logging_obj: ProxyLogging | None,
*,
@ -1916,8 +2052,10 @@ async def _delete_cache_key_object(
@log_db_metrics
async def _get_team_db_check(team_id: str, prisma_client: PrismaClient, team_id_upsert: bool | None = None):
response = await TeamRepository(prisma_client).table.find_unique(where={"team_id": team_id})
async def _get_team_db_check(
team_id: str, prisma_client: PrismaClient, team_id_upsert: bool | None = None
) -> "_PrismaTeamRow | None":
response = await _team_table(TeamRepository(prisma_client)).find_unique(where={"team_id": team_id})
if response is None and team_id_upsert:
from litellm.proxy.management_endpoints.team_endpoints import new_team
@ -1936,8 +2074,8 @@ async def _get_team_db_check(team_id: str, prisma_client: PrismaClient, team_id_
return response
async def _get_team_object_from_db(team_id: str, prisma_client: PrismaClient):
return await TeamRepository(prisma_client).table.find_unique(where={"team_id": team_id})
async def _get_team_object_from_db(team_id: str, prisma_client: PrismaClient) -> "_PrismaTeamRow | None":
return await _team_table(TeamRepository(prisma_client)).find_unique(where={"team_id": team_id})
async def _get_team_object_from_user_api_key_cache(
@ -2148,7 +2286,7 @@ async def get_access_object(
# Not in cache - fetch from DB
try:
response: Final = await AccessGroupRepository(prisma_client).table.find_unique(
response: Final = await _dictable_table(AccessGroupRepository(prisma_client)).find_unique(
where={"access_group_id": access_group_id}
)
@ -2224,7 +2362,7 @@ async def get_team_object_by_alias(
# Query database by team_alias
try:
teams: Final = await TeamRepository(prisma_client).table.find_many(where={"team_alias": team_alias})
teams: Final = await _team_table(TeamRepository(prisma_client)).find_many(where={"team_alias": team_alias})
if not teams:
raise HTTPException(
@ -2329,7 +2467,9 @@ async def get_org_object_by_alias(
# Query database by organization_alias
try:
orgs = await OrganizationRepository(prisma_client).table.find_many(where={"organization_alias": org_alias})
orgs = await _model_dump_table(OrganizationRepository(prisma_client)).find_many(
where={"organization_alias": org_alias}
)
if not orgs:
raise HTTPException(
@ -2546,7 +2686,7 @@ async def get_jwt_key_mapping_object(
Returns the hashed token (str) if a matching active mapping is found, else None.
"""
mapping: Final = await JWTKeyMappingRepository(prisma_client).table.find_first(
mapping: Final = await _jwt_key_mapping_table(JWTKeyMappingRepository(prisma_client)).find_first(
where={
"jwt_claim_name": jwt_claim_name,
"jwt_claim_value": jwt_claim_value,
@ -2674,7 +2814,7 @@ async def get_object_permission(
# else, check db
try:
response: Final = await ObjectPermissionRepository(prisma_client).table.find_unique(
response: Final = await _dictable_table(ObjectPermissionRepository(prisma_client)).find_unique(
where={"object_permission_id": object_permission_id}
)
@ -2730,7 +2870,7 @@ async def get_managed_vector_store_rows_by_uuids(
if not cache_misses:
return result
rows: Final = await ManagedVectorStoresRepository(prisma_client).table.find_many(
rows: Final = await _vector_store_table(ManagedVectorStoresRepository(prisma_client)).find_many(
where={"vector_store_id": {"in": cache_misses}},
take=len(cache_misses),
)
@ -2804,11 +2944,11 @@ async def get_org_object(
return deserialized_org
# else, check db
try:
query_kwargs: Final[dict[str, Any]] = {"where": {"organization_id": org_id}}
query_kwargs: Final[dict[str, Mapping[str, object]]] = {"where": {"organization_id": org_id}}
if include_budget_table:
query_kwargs["include"] = {"litellm_budget_table": True}
response: Final = await OrganizationRepository(prisma_client).table.find_unique(**query_kwargs)
response: Final = await _model_dump_table(OrganizationRepository(prisma_client)).find_unique(**query_kwargs)
except Exception:
# An operational failure (DB down, timeout, cache fault) is NOT the same fact as a confirmed
# missing row, and relabelling it as "doesn't exist" made every caller unable to tell them
@ -3763,7 +3903,7 @@ async def _virtual_key_soft_budget_check(
)
def _parse_email_list(raw: Any) -> list[str]:
def _parse_email_list(raw: str | Sequence[object] | None) -> list[str]:
"""Parse emails from a list or comma-separated string."""
if isinstance(raw, list):
return [e.strip() for e in raw if isinstance(e, str) and e.strip()]
@ -3773,7 +3913,7 @@ def _parse_email_list(raw: Any) -> list[str]:
def _normalize_alert_emails(
cfg: dict[str, Any] | None,
cfg: Mapping[str, str | Sequence[object] | None] | None,
) -> dict[str, list[str]]:
"""Coerce user-supplied threshold→recipients mapping to Dict[str, List[str]].
@ -3786,8 +3926,8 @@ def _normalize_alert_emails(
def _merge_budget_alert_email_configs(
global_cfg: dict[str, Any] | None,
per_key_cfg: dict[str, Any] | None,
global_cfg: Mapping[str, str | Sequence[object] | None] | None,
per_key_cfg: Mapping[str, str | Sequence[object] | None] | None,
) -> dict[str, list[str]] | None:
"""
Per-threshold additive merge: each threshold's recipient list is the union
@ -4294,7 +4434,7 @@ async def get_project_object(
return deserialized_project
# Fetch from DB
project_row: Final = await ProjectRepository(prisma_client).table.find_unique(
project_row: Final = await _model_dump_table(ProjectRepository(prisma_client)).find_unique(
where={"project_id": project_id},
include={"litellm_budget_table": True},
)
@ -4621,7 +4761,9 @@ async def vector_store_access_check(
#########################################################
# Check if the key can access the vector store
if valid_token is not None and valid_token.object_permission_id is not None:
key_object_permission: Final = await ObjectPermissionRepository(prisma_client).table.find_unique(
key_object_permission: Final = await _object_permission_table(
ObjectPermissionRepository(prisma_client)
).find_unique(
where={"object_permission_id": valid_token.object_permission_id},
)
if key_object_permission is not None:
@ -4633,7 +4775,9 @@ async def vector_store_access_check(
# Check if the team can access the vector store
if team_object is not None and team_object.object_permission_id is not None:
team_object_permission: Final = await ObjectPermissionRepository(prisma_client).table.find_unique(
team_object_permission: Final = await _object_permission_table(
ObjectPermissionRepository(prisma_client)
).find_unique(
where={"object_permission_id": team_object.object_permission_id},
)
if team_object_permission is not None:

113
litellm/proxy/common_request_processing.py
View file

@ -8,7 +8,7 @@ from collections.abc import AsyncGenerator, Callable, Mapping
from datetime import datetime
from functools import lru_cache
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal
from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, overload
import anyio
import httpx
@ -312,8 +312,49 @@ def _stream_usage_tracking_updates(
}
def _getattr_object(value: object, name: str, default: object = None) -> object:
return getattr(value, name, default)
class _UpstreamHttpResponse(Protocol):
@property
def status_code(self) -> int: ...
@property
def headers(self) -> httpx.Headers: ...
async def aread(self) -> bytes: ...
def _as_upstream_response(response: _UpstreamHttpResponse) -> _UpstreamHttpResponse:
return response
class _ReadsHeaderValues(Protocol):
def get(self, key: str, default: str = "") -> str: ...
def _as_header_reader(headers: _ReadsHeaderValues) -> _ReadsHeaderValues:
return headers
class _DispatchesSuccessHandlers(Protocol):
async def dispatch_success_handlers(
self,
result: object = None,
start_time: object = None,
end_time: object = None,
cache_hit: object = None,
prefer_async_handlers: bool = False,
) -> None: ...
def _as_success_dispatcher(logging_obj: _DispatchesSuccessHandlers) -> _DispatchesSuccessHandlers:
return logging_obj
def _serialize_http_exception_detail(
detail: Any,
detail: object,
) -> tuple[str, dict | None]:
"""
Convert an HTTPException.detail value into (message, structured_fields)
@ -343,7 +384,7 @@ def _serialize_http_exception_detail(
return str(detail), None
def _collect_response_file_search_vector_store_ids(data: dict[str, Any]) -> set[str]:
def _collect_response_file_search_vector_store_ids(data: Mapping[str, object]) -> set[str]:
vector_store_ids: Final[set[str]] = set()
tools: Final = data.get("tools")
if not isinstance(tools, list):
@ -370,7 +411,7 @@ def _collect_response_file_search_vector_store_ids(data: dict[str, Any]) -> set[
async def _authorize_response_file_search_vector_stores(
data: dict[str, Any],
data: Mapping[str, object],
user_api_key_dict: UserAPIKeyAuth,
) -> None:
vector_store_ids: Final = _collect_response_file_search_vector_store_ids(data)
@ -701,7 +742,7 @@ async def create_response(
# Preserve status code from HTTPException (e.g., guardrail blocks)
error_status: Final = getattr(e, "status_code", status.HTTP_500_INTERNAL_SERVER_ERROR)
raw_detail: Final = getattr(e, "detail", "Error processing stream start")
raw_detail: Final = _getattr_object(e, "detail", "Error processing stream start")
message, structured_fields = _serialize_http_exception_detail(raw_detail)
existing_fields: Final = getattr(e, "provider_specific_fields", None) or {}
@ -712,7 +753,7 @@ async def create_response(
# Match ProxyException.to_dict() shape so streaming and non-streaming
# error frames are byte-identical.
error_obj: Final[dict[str, Any]] = {
error_obj: Final[dict[str, object]] = {
"message": message,
"type": getattr(e, "type", "None"),
"param": getattr(e, "param", "None"),
@ -778,7 +819,7 @@ def _is_azure_model_router_request(model: str) -> bool:
def _override_openai_response_model(
*,
response_obj: Any,
response_obj: object,
requested_model: str,
log_context: str,
return_raw_model_name: bool = False,
@ -973,7 +1014,7 @@ def _log_llm_api_exception(e: Exception) -> None:
async def _cancel_llm_call_on_client_disconnect(
request: Request,
llm_api_call: "asyncio.Future[Any]",
llm_api_call: "asyncio.Future[object]",
disconnect_event: asyncio.Event,
) -> None:
try:
@ -1024,7 +1065,7 @@ class ProxyBaseLLMRequestProcessing:
version: str | None = None,
model_region: str | None = None,
response_cost: float | str | None = None,
hidden_params: dict | None = None,
hidden_params: Mapping[str, object] | None = None,
fastest_response_batch_completion: bool | None = None,
request_data: dict | None = {},
timeout: float | httpx.Timeout | None = None,
@ -1116,7 +1157,7 @@ class ProxyBaseLLMRequestProcessing:
@staticmethod
async def build_litellm_proxy_success_headers_from_llm_response(
*,
response: Any,
response: object,
request_data: dict,
request: Request,
user_api_key_dict: UserAPIKeyAuth,
@ -1907,7 +1948,7 @@ class ProxyBaseLLMRequestProcessing:
_captured_user_api_key_dict: Final = user_api_key_dict
_captured_logging_obj: Final = logging_obj
async def _on_deferred_stream_complete(assembled_response, cache_hit):
async def _on_deferred_stream_complete(assembled_response: object, cache_hit: object) -> None:
await ProxyBaseLLMRequestProcessing._run_deferred_stream_guardrails(
captured_data=_captured_data,
captured_user_api_key_dict=_captured_user_api_key_dict,
@ -2158,7 +2199,7 @@ class ProxyBaseLLMRequestProcessing:
@staticmethod
async def _record_container_owners_from_responses_if_needed(
response: Any,
response: object,
user_api_key_dict: UserAPIKeyAuth,
) -> None:
"""Register code-interpreter containers so follow-up file APIs pass ownership checks."""
@ -2181,7 +2222,7 @@ class ProxyBaseLLMRequestProcessing:
)
@staticmethod
def _extract_completed_responses_response(stream_response: Any) -> Any:
def _extract_completed_responses_response(stream_response: object) -> object:
"""Pull the assembled ``ResponsesAPIResponse`` off a streaming iterator.
``ResponsesAPIStreamingIterator`` stores the terminal stream event
@ -2191,17 +2232,17 @@ class ProxyBaseLLMRequestProcessing:
``ResponsesAPIResponse`` directly. Handle both shapes so the
container-ownership recording path can walk ``.output`` either way.
"""
completed: Final = getattr(stream_response, "completed_response", None)
completed: Final = _getattr_object(stream_response, "completed_response")
if completed is None:
return None
response_obj: Final = getattr(completed, "response", None)
response_obj: Final = _getattr_object(completed, "response")
if response_obj is not None:
return response_obj
return completed
@staticmethod
async def _wrap_responses_stream_for_container_ownership(
original_stream_response: Any,
original_stream_response: object,
wrapped_generator: Any,
user_api_key_dict: UserAPIKeyAuth,
):
@ -2300,12 +2341,13 @@ class ProxyBaseLLMRequestProcessing:
if isinstance(result, Response):
return result
content: Final = await result.aread()
upstream: Final = _as_upstream_response(result)
content: Final = await upstream.aread()
return Response(
content=content,
status_code=result.status_code,
status_code=upstream.status_code,
headers=HttpPassThroughEndpointHelpers.get_response_headers(
headers=result.headers,
headers=upstream.headers,
custom_headers=dict(fastapi_response.headers),
),
)
@ -2436,9 +2478,10 @@ class ProxyBaseLLMRequestProcessing:
HttpPassThroughEndpointHelpers,
)
upstream: Final = _as_upstream_response(response)
try:
response_status: Final[int] = response.status_code
content_type: Final[str] = response.headers.get("content-type", "")
response_status: Final[int] = upstream.status_code
content_type: Final[str] = _as_header_reader(upstream.headers).get("content-type", "")
except AttributeError:
return None
@ -2452,20 +2495,20 @@ class ProxyBaseLLMRequestProcessing:
return None
response_headers: Final = HttpPassThroughEndpointHelpers.get_response_headers(
headers=response.headers,
headers=upstream.headers,
custom_headers=custom_headers,
)
callback_headers: Final = await proxy_logging_obj.post_call_response_headers_hook(
data=self.data,
user_api_key_dict=user_api_key_dict,
response=response,
response=upstream,
request_headers=request_headers,
)
if callback_headers:
response_headers.update(callback_headers)
if is_event_stream:
body_bytes = await response.aread()
body_bytes = await upstream.aread()
modified_bytes: Final = await self._handle_event_stream_allm_passthrough_route(
body_bytes=body_bytes,
proxy_logging_obj=proxy_logging_obj,
@ -2478,7 +2521,7 @@ class ProxyBaseLLMRequestProcessing:
headers=response_headers,
)
body_bytes = await response.aread()
body_bytes = await upstream.aread()
try:
parsed: Final = _json.loads(body_bytes)
except (_json.JSONDecodeError, UnicodeDecodeError):
@ -2567,9 +2610,9 @@ class ProxyBaseLLMRequestProcessing:
async def _run_deferred_stream_guardrails(
captured_data: dict,
captured_user_api_key_dict: "UserAPIKeyAuth",
captured_logging_obj: Any,
captured_logging_obj: LiteLLMLoggingObj,
assembled_response: Any,
cache_hit: Any,
cache_hit: object,
) -> None:
"""
Run non-streaming post-call guardrail hooks on an assembled streaming
@ -2647,7 +2690,7 @@ class ProxyBaseLLMRequestProcessing:
# _is_sync_litellm_request (which only recognizes a subset of
# async markers stored in litellm_params).
asyncio.create_task(
captured_logging_obj.dispatch_success_handlers(
_as_success_dispatcher(captured_logging_obj).dispatch_success_handlers(
_response,
cache_hit=cache_hit,
start_time=None,
@ -2718,7 +2761,7 @@ class ProxyBaseLLMRequestProcessing:
headers = getattr(e, "headers", None) or {}
if not headers:
# Try to get headers from e.response.headers (httpx.Response)
_response: Final = getattr(e, "response", None)
_response: Final = _getattr_object(e, "response")
if _response is not None:
_response_headers: Final = getattr(_response, "headers", None)
if _response_headers:
@ -2750,7 +2793,7 @@ class ProxyBaseLLMRequestProcessing:
raise e
if isinstance(e, HTTPException):
raw_detail: Final = getattr(e, "detail", str(e))
raw_detail: Final = _getattr_object(e, "detail", str(e))
message, structured_fields = _serialize_http_exception_detail(raw_detail)
existing_fields: Final = getattr(e, "provider_specific_fields", None) or {}
if structured_fields:
@ -3043,8 +3086,16 @@ class ProxyBaseLLMRequestProcessing:
request=request,
)
@overload
@staticmethod
def _process_chunk_with_cost_injection(chunk: Any, model_name: str) -> Any:
def _process_chunk_with_cost_injection(chunk: bytes, model_name: str) -> bytes: ...
@overload
@staticmethod
def _process_chunk_with_cost_injection(chunk: object, model_name: str) -> object: ...
@staticmethod
def _process_chunk_with_cost_injection(chunk: object, model_name: str) -> object:
"""
Process a streaming chunk and inject cost information if enabled.

21
litellm/proxy/db/db_transaction_queue/pod_lock_manager.py
View file

@ -43,6 +43,7 @@ end
self,
cronjob_id: str,
ttl: int | None = None,
allow_reentrant: bool = True,
) -> bool | None:
"""
Attempt to acquire the lock for a specific cron job using Redis.
@ -53,6 +54,10 @@ end
ttl: Optional custom TTL in seconds. Defaults to DEFAULT_CRON_JOB_LOCK_TTL_SECONDS.
Use a longer TTL for jobs that may take longer than the default 60s
(e.g. key rotation with many keys).
allow_reentrant: With the default True, a pod that already holds the lock
acquires it again (leader election semantics). Pass False when the live
lock marks work as already done for this window, so not even the holder
may redo it before the TTL expires.
"""
if self.redis_cache is None:
verbose_proxy_logger.debug("redis_cache is None, skipping acquire_lock")
@ -88,7 +93,7 @@ end
if current_value is not None:
if isinstance(current_value, bytes):
current_value = current_value.decode("utf-8")
if current_value == self.pod_id:
if current_value == self.pod_id and allow_reentrant:
verbose_proxy_logger.info(
"Pod %s already holds the Redis lock for cronjob_id=%s",
self.pod_id,
@ -96,14 +101,12 @@ end
)
self._emit_acquired_lock_event(cronjob_id, self.pod_id)
return True
else:
verbose_proxy_logger.info(
"Spend tracking - pod %s could not acquire lock for cronjob_id=%s, "
"held by pod %s. Spend updates in Redis will wait for the leader pod to commit.",
self.pod_id,
cronjob_id,
current_value,
)
verbose_proxy_logger.info(
"Pod %s could not acquire lock for cronjob_id=%s, held by pod %s.",
self.pod_id,
cronjob_id,
current_value,
)
return False
except Exception as e:
verbose_proxy_logger.error("Error acquiring Redis lock for %s: %s", cronjob_id, e)

4
litellm/proxy/db/spend_log_tool_index.py
View file

@ -35,7 +35,7 @@ class ToolUsageTransaction:
total_tokens: int
def response_tool_call_names(completion_response: Any) -> tuple[str, ...]:
def response_tool_call_names(completion_response: object) -> tuple[str, ...]:
"""Tool names invoked in a completion response, in call order, for any response
surface get_tool_calls_from_response understands (chat completions, Responses
API output items, Anthropic Messages tool_use blocks). Reads every choice of
@ -59,7 +59,7 @@ def build_tool_usage_transaction(
mcp_namespaced_tool_name: str | None,
spend: float,
total_tokens: int,
completion_response: Any,
completion_response: object,
realtime_tool_calls: Any = None,
) -> ToolUsageTransaction | None:
"""None when the request invoked no tools. Realtime sessions carry invoked

43
litellm/proxy/hooks/parallel_request_limiter.py
View file

@ -20,6 +20,7 @@ from litellm.proxy.auth.auth_utils import (
from litellm.proxy.auth.budget_throttle import throttled_limit
from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
from litellm.proxy.hooks.rate_limiter_utils import resolve_llm_provider_for_rate_limit
from litellm.types.utils import Usage
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@ -33,6 +34,13 @@ else:
InternalUsageCache = Any
def _response_total_tokens(response_obj: object) -> int:
if not isinstance(response_obj, (ModelResponse, EmbeddingResponse, TextCompletionResponse)):
return 0
response_usage: Final = getattr(response_obj, "usage", None)
return response_usage.total_tokens if isinstance(response_usage, Usage) else 0
class CacheObject(TypedDict):
current_global_requests: dict | None
request_count_api_key: dict | None
@ -480,7 +488,7 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
) # don't block execution for cache updates
)
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
async def async_log_success_event(self, kwargs, response_obj: object, start_time, end_time):
from litellm.proxy.common_utils.callback_utils import (
get_model_group_from_litellm_kwargs,
)
@ -529,21 +537,18 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
current_minute: Final = datetime.now().strftime("%M")
precise_minute: Final = f"{current_date}-{current_hour}-{current_minute}"
total_tokens = 0
if isinstance(response_obj, (ModelResponse, EmbeddingResponse, TextCompletionResponse)):
total_tokens = response_obj.usage.total_tokens
total_tokens: int = _response_total_tokens(response_obj)
# ------------
# Update usage - API Key
# ------------
values_to_update_in_cache: Final = []
values_to_update_in_cache: Final[list[tuple[str, object]]] = []
if user_api_key is not None:
request_count_api_key = f"{user_api_key}::{precise_minute}::request_count"
current = await self.internal_usage_cache.async_get_cache(
current: dict[str, int] = await self.internal_usage_cache.async_get_cache(
key=request_count_api_key,
litellm_parent_otel_span=litellm_parent_otel_span,
) or {
@ -606,13 +611,7 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
# Update usage - User
# ------------
if user_api_key_user_id is not None:
total_tokens = 0
if isinstance(
response_obj,
(ModelResponse, EmbeddingResponse, TextCompletionResponse),
):
total_tokens = response_obj.usage.total_tokens
total_tokens = _response_total_tokens(response_obj)
request_count_api_key = f"{user_api_key_user_id}::{precise_minute}::request_count"
@ -638,13 +637,7 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
# Update usage - Team
# ------------
if user_api_key_team_id is not None:
total_tokens = 0
if isinstance(
response_obj,
(ModelResponse, EmbeddingResponse, TextCompletionResponse),
):
total_tokens = response_obj.usage.total_tokens
total_tokens = _response_total_tokens(response_obj)
request_count_api_key = f"{user_api_key_team_id}::{precise_minute}::request_count"
@ -670,13 +663,7 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
# Update usage - End User
# ------------
if user_api_key_end_user_id is not None:
total_tokens = 0
if isinstance(
response_obj,
(ModelResponse, EmbeddingResponse, TextCompletionResponse),
):
total_tokens = response_obj.usage.total_tokens
total_tokens = _response_total_tokens(response_obj)
request_count_api_key = f"{user_api_key_end_user_id}::{precise_minute}::request_count"

9
litellm/proxy/image_endpoints/endpoints.py
View file

@ -1,8 +1,10 @@
import asyncio
import io
import traceback
from typing import Final
import orjson
from fastapi import APIRouter, Depends, File, HTTPException, Request, Response, status
from fastapi import APIRouter, Depends, File, HTTPException, Request, Response, UploadFile, status
from fastapi.responses import ORJSONResponse
import litellm
@ -18,11 +20,6 @@ from litellm.types.llms.openai import ChatCompletionUserMessage
router: Final = APIRouter()
import io
from typing import Final
from fastapi import UploadFile
async def uploadfile_to_bytesio(upload: UploadFile) -> io.BytesIO:
"""

53
litellm/proxy/litellm_pre_call_utils.py
View file

@ -201,6 +201,7 @@ _UNTRUSTED_ROOT_CONTROL_FIELDS: Final = (
"mock_tool_calls",
"disable_global_guardrails",
"disable_global_guardrail",
"enable_prompt_caching",
"opted_out_global_guardrails",
"applied_guardrails",
"applied_policies",
@ -1333,6 +1334,9 @@ class LiteLLMProxyRequestSetup:
if "disable_fallbacks" in key_metadata and isinstance(key_metadata["disable_fallbacks"], bool):
data["disable_fallbacks"] = key_metadata["disable_fallbacks"]
if isinstance(key_metadata.get("enable_prompt_caching"), bool):
data["enable_prompt_caching"] = key_metadata["enable_prompt_caching"] # rebind-ok: data is an out-param
## KEY-LEVEL METADATA
data = LiteLLMProxyRequestSetup.add_management_endpoint_metadata_to_request_metadata(
data=data,
@ -1862,6 +1866,24 @@ async def add_litellm_data_to_request(
tags_to_add=project_metadata["tags"],
)
# inherited_tags: every tag key/team/project policy contributed, read
# directly from those three sources rather than snapshotted off the shared
# "tags" list. A pre-auth pass (apply_client_tag_policy_pre_auth, run from
# user_api_key_auth for _tag_max_budget_check) may already have merged the
# caller's own header tags into that same list before this function ever
# runs, so a snapshot taken here -- at any point in this function -- would
# misattribute caller-supplied tags as policy-backed. tag_based_routing.py's
# allow_fail_open reads this (rather than subtracting caller_tags from the
# final merged set) so a caller can't strip an inherited "!"/"&"
# constraint's protection just by resubmitting its exact value alongside a
# conflicting one.
_key_tags: Final = (key_metadata or MappingProxyType({})).get("tags") or ()
_team_tags: Final = team_metadata.get("tags") or ()
_project_tags: Final = project_metadata.get("tags") or ()
data[_metadata_variable_name]["inherited_tags"] = tuple( # rebind-ok: matches this file's data[...] mutation idiom
dict.fromkeys((*_key_tags, *_team_tags, *_project_tags))
)
## TEAM-LEVEL METADATA
data = LiteLLMProxyRequestSetup.add_management_endpoint_metadata_to_request_metadata(
data=data,
@ -1958,15 +1980,28 @@ async def add_litellm_data_to_request(
tags_to_add=tags,
)
if _metadata_variable_name != "metadata":
_user_metadata = data.get("metadata")
if isinstance(_user_metadata, dict):
_user_tags: Final = _user_metadata.get("tags")
if isinstance(_user_tags, list) and _user_tags:
data[_metadata_variable_name]["tags"] = LiteLLMProxyRequestSetup._merge_tags(
request_tags=data[_metadata_variable_name].get("tags"),
tags_to_add=_user_tags,
)
_caller_body_metadata: Final = data.get("metadata") if _metadata_variable_name != "metadata" else None
_caller_body_tags: Final = (
_caller_body_metadata.get("tags")
if isinstance(_caller_body_metadata, dict) and isinstance(_caller_body_metadata.get("tags"), list)
else None
)
if _caller_body_tags:
data[_metadata_variable_name]["tags"] = LiteLLMProxyRequestSetup._merge_tags( # rebind-ok: matches file idiom
request_tags=data[_metadata_variable_name].get("tags"),
tags_to_add=_caller_body_tags,
)
# caller_tags: exactly what this request itself supplied (x-litellm-tags header,
# body "tags", or body "metadata.tags" on litellm_metadata routes), never
# anything from key/team/project metadata. Read directly from the header and
# body values here, the same way inherited_tags above is read directly from
# key/team/project metadata -- neither is derived by inspecting the shared
# "tags" list, which a pre-auth pass (apply_client_tag_policy_pre_auth) may
# have already merged caller header tags into before this function runs.
data[_metadata_variable_name]["caller_tags"] = tuple( # rebind-ok: matches file idiom
dict.fromkeys((*(tags or ()), *(_caller_body_tags or ())))
)
# Team Callbacks controls
callback_settings_obj: Final = _get_dynamic_logging_metadata(

27
litellm/proxy/management_endpoints/key_management_endpoints.py
View file

@ -88,6 +88,7 @@ from litellm.proxy.management_endpoints.common_utils import (
from litellm.proxy.management_endpoints.model_management_endpoints import (
_add_model_to_db,
)
from litellm.proxy.management_helpers.key_settings_audit import with_settings_updated_at
from litellm.proxy.management_helpers.object_permission_utils import (
_set_object_permission,
attach_object_permission_to_dict,
@ -189,6 +190,16 @@ class _PrismaTableActions(Protocol[_PrismaRowT]):
) -> _PrismaRowT | None: ...
class _UserRowLike(Protocol):
user_id: str | None
user_email: str | None
user_alias: str | None
def model_dump(self) -> Mapping[str, object]: ...
def dict(self) -> Mapping[str, object]: ...
class _TxTables(Protocol):
litellm_proxymodeltable: _PrismaTableActions[object]
@ -1592,6 +1603,7 @@ async def generate_key_fn(
- policies: Optional[List[str]] - List of policy names to apply to the key. Policies define guardrails, conditions, and inheritance rules.
- disable_global_guardrails: Optional[bool] - Whether to disable global guardrails for the key.
- throttle_on_budget_exceeded: Optional[bool] - When the key exceeds its max_budget, throttle its tpm/rpm to the global budget_exceeded_throttle_percentage instead of blocking the key entirely.
- enable_prompt_caching: Optional[bool] - Auto-inject prompt caching breakpoints (Anthropic cache_control markers) on requests made with this key. Anthropic and Bedrock Claude models only.
- permissions: Optional[dict] - key-specific permissions. Currently just used for turning off pii masking (if connected). Example - {"pii": false}
- model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.
- budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}.
@ -2692,6 +2704,7 @@ async def update_key_fn(
- policies: Optional[List[str]] - List of policy names to apply to the key. Policies define guardrails, conditions, and inheritance rules.
- disable_global_guardrails: Optional[bool] - Whether to disable global guardrails for the key.
- throttle_on_budget_exceeded: Optional[bool] - When the key exceeds its max_budget, throttle its tpm/rpm to the global budget_exceeded_throttle_percentage instead of blocking the key entirely.
- enable_prompt_caching: Optional[bool] - Auto-inject prompt caching breakpoints (Anthropic cache_control markers) on requests made with this key. Anthropic and Bedrock Claude models only.
- prompts: Optional[List[str]] - List of prompts that the key is allowed to use.
- blocked: Optional[bool] - Whether the key is blocked
- aliases: Optional[dict] - Model aliases for the key - [Docs](https://litellm.vercel.app/docs/proxy/virtual_keys#model-aliases)
@ -4222,7 +4235,7 @@ def _transform_verification_tokens_to_deleted_records(
record = deleted_record.model_dump()
# Map org_id to organization_id (model uses org_id, but schema expects organization_id)
org_id_value = record.pop("org_id", None)
org_id_value: object = record.pop("org_id", None)
if org_id_value is not None:
record["organization_id"] = org_id_value
@ -4691,9 +4704,9 @@ async def _execute_virtual_key_regeneration(
grace_period=data.grace_period if data else None,
)
updated_token: Final = await VerificationTokenRepository(prisma_client).table.update(
updated_token: Final[Mapping[str, object] | None] = await VerificationTokenRepository(prisma_client).table.update(
where={"token": hashed_api_key},
data=jsonified_update_data,
data=with_settings_updated_at(jsonified_update_data),
)
updated_token_dict: Final[dict[str, object]] = dict(updated_token) if updated_token is not None else {}
updated_token_dict["key"] = new_token
@ -5988,7 +6001,9 @@ async def _list_key_helper(
created_by_ids: Final = [key.created_by for key in keys if key.created_by]
all_ids: Final = list(set(user_ids + created_by_ids)) # Remove duplicates
if all_ids:
users: Final = await UserRepository(prisma_client).table.find_many(where={"user_id": {"in": all_ids}})
users: Final[Sequence[_UserRowLike]] = await UserRepository(prisma_client).table.find_many(
where={"user_id": {"in": all_ids}}
)
user_map = {user.user_id: user for user in users}
# Prepare response
@ -6203,7 +6218,7 @@ async def block_key(
record: Final = await _prisma_table(VerificationTokenRepository(prisma_client)).update(
where={"token": hashed_token},
data={"blocked": True},
data=with_settings_updated_at({"blocked": True}),
)
## UPDATE KEY CACHE - invalidate so next read re-fetches from DB
@ -6316,7 +6331,7 @@ async def unblock_key(
record: Final = await _prisma_table(VerificationTokenRepository(prisma_client)).update(
where={"token": hashed_token},
data={"blocked": False},
data=with_settings_updated_at({"blocked": False}),
)
## UPDATE KEY CACHE - invalidate so next read re-fetches from DB

14
litellm/proxy/management_helpers/key_settings_audit.py Normal file
View file

@ -0,0 +1,14 @@
"""Audit stamping for virtual key configuration changes."""
from collections.abc import Mapping
from datetime import datetime, timezone
def with_settings_updated_at(data: Mapping[str, object]) -> dict[str, object]:
"""Stamp a key update payload with the time its configuration changed.
``updated_at`` carries Prisma's ``@updatedAt`` and so is rewritten by every
spend flush, which makes it useless for auditing; ``settings_updated_at`` is
written only from key-management write paths.
"""
return {**data, "settings_updated_at": datetime.now(timezone.utc)}

2
litellm/proxy/pass_through_endpoints/llm_provider_handlers/gemini_passthrough_logging_handler.py
View file

@ -92,7 +92,7 @@ class GeminiPassthroughLoggingHandler:
litellm_params={},
api_key="",
request_data={},
encoding=litellm.encoding,
encoding=getattr(litellm, "encoding", None),
)
kwargs = GeminiPassthroughLoggingHandler._create_gemini_response_logging_payload_for_generate_content(
litellm_model_response=litellm_model_response,

2
litellm/proxy/pass_through_endpoints/llm_provider_handlers/openai_passthrough_logging_handler.py
View file

@ -327,7 +327,7 @@ class OpenAIPassthroughLoggingHandler(BasePassthroughLoggingHandler):
optional_params=request_body.get("optional_params", {}),
api_key="",
request_data=request_body,
encoding=litellm.encoding,
encoding=getattr(litellm, "encoding", None),
json_mode=request_body.get("response_format", {}).get("type") == "json_object",
litellm_params=existing_litellm_params,
)

2
litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
View file

@ -133,7 +133,7 @@ class VertexPassthroughLoggingHandler:
litellm_params={},
api_key="",
request_data={},
encoding=litellm.encoding,
encoding=getattr(litellm, "encoding", None),
)
kwargs = VertexPassthroughLoggingHandler._create_vertex_response_logging_payload_for_generate_content(
litellm_model_response=litellm_model_response,

96
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
View file

@ -5,10 +5,10 @@ import json
import posixpath
import traceback
from base64 import b64encode
from collections.abc import AsyncGenerator, Mapping
from collections.abc import AsyncGenerator, Callable, Mapping
from datetime import datetime
from itertools import groupby
from typing import Any, Final, cast
from typing import Any, Final, TypedDict, cast
from urllib.parse import urlencode, urlparse
import httpx
@ -92,7 +92,7 @@ router: Final = APIRouter()
pass_through_endpoint_logging: Final = PassThroughEndpointLogging()
# Global registry to track registered pass-through routes and prevent memory leaks
_registered_pass_through_routes: Final[dict[str, dict[str, str | bool | list[str] | dict[str, Any]]]] = {}
_registered_pass_through_routes: Final[dict[str, dict[str, str | bool | list[str] | Mapping[str, object]]]] = {}
def get_response_body(response: httpx.Response) -> dict | None:
@ -1128,15 +1128,22 @@ async def pass_through_request(
else:
# SigV4-signed callers (Bedrock) supply the exact pre-signed bytes;
# otherwise httpx encodes the parsed JSON dict as before.
body_kwargs: Final[dict[str, Any]] = (
{"content": state_raw_body} if state_raw_body is not None else {"json": _parsed_body}
)
req: Final = async_client.build_request(
request.method,
url,
params=requested_query_params,
headers=headers,
**body_kwargs,
req: Final = (
async_client.build_request(
request.method,
url,
params=requested_query_params,
headers=headers,
content=state_raw_body,
)
if state_raw_body is not None
else async_client.build_request(
request.method,
url,
params=requested_query_params,
headers=headers,
json=_parsed_body,
)
)
response = await async_client.send(req, stream=stream)
@ -1584,9 +1591,15 @@ def _update_metadata_with_tags_in_header(request: Request, metadata: dict) -> di
return metadata
class _PassThroughRequestEnvelope(TypedDict, total=False):
query_params: Mapping[str, object] | None
custom_body: Mapping[str, object] | None
stream: bool | None
async def _parse_request_data_by_content_type(
request: Request,
) -> tuple[Any | None, Any | None, Any | None, Any | None]:
) -> tuple[object, object, None, bool | None]:
"""
Parse request data based on content type.
@ -1605,7 +1618,7 @@ async def _parse_request_data_by_content_type(
if "application/json" in content_type:
# ✅ Handle JSON
try:
body = await request.json()
body: _PassThroughRequestEnvelope = await request.json()
query_params_data = body.get("query_params")
custom_body_data = body.get("custom_body")
stream = body.get("stream")
@ -1646,7 +1659,7 @@ async def _parse_request_data_by_content_type(
def create_pass_through_route(
endpoint,
target: str,
custom_headers: Mapping[str, Any] | None = None,
custom_headers: Mapping[str, object] | None = None,
_forward_headers: bool | None = False,
_merge_query_params: bool | None = False,
dependencies: list | None = None,
@ -1656,7 +1669,7 @@ def create_pass_through_route(
is_streaming_request: bool | None = False,
query_params: dict | None = None,
default_query_params: dict | None = None,
guardrails: dict[str, Any] | None = None,
guardrails: dict[str, object] | None = None,
config_file_path: str | None = None,
timeout: float | None = None,
):
@ -1887,7 +1900,7 @@ async def websocket_passthrough_request(
# Initialize tracking variables
start_time: Final = datetime.now()
websocket_messages: Final[list[dict[str, Any]]] = []
websocket_messages: Final[list[dict[str, object]]] = []
litellm_call_id: Final = str(uuid.uuid4())
verbose_proxy_logger.info("WebSocket passthrough (%s): Starting WebSocket connection to %s", endpoint, target)
@ -1980,7 +1993,7 @@ async def websocket_passthrough_request(
)
### CALL HOOKS ### - modify incoming data / reject request before calling the model
websocket_data: dict[str, Any] = {}
websocket_data: dict[str, object] = {}
websocket_data = await proxy_logging_obj.pre_call_hook(
user_api_key_dict=user_api_key_dict,
data=websocket_data,
@ -2009,8 +2022,8 @@ async def websocket_passthrough_request(
await upstream_ws.close()
break
text_data = message.get("text")
bytes_data = message.get("bytes")
text_data: str | None = message.get("text")
bytes_data: bytes | None = message.get("bytes")
if text_data is not None:
# Try to extract model from client setup message for Vertex AI Live
@ -2086,7 +2099,7 @@ async def websocket_passthrough_request(
# Ensure raw_response is bytes before decoding
if isinstance(raw_response, str):
raw_response = raw_response.encode("ascii")
setup_response: Final = json.loads(raw_response.decode("ascii"))
setup_response: Final[Mapping[str, object]] = json.loads(raw_response.decode("ascii"))
verbose_proxy_logger.debug("Setup response: %s", setup_response)
# Extract model and provider from setup response for Vertex AI Live
@ -2129,7 +2142,7 @@ async def websocket_passthrough_request(
await websocket.send_bytes(upstream_message)
# Parse and collect for cost tracking
try:
message_data = json.loads(upstream_message.decode())
message_data: dict[str, object] = json.loads(upstream_message.decode())
websocket_messages.append(message_data)
except (json.JSONDecodeError, UnicodeDecodeError):
pass
@ -2315,7 +2328,8 @@ def _should_buffer_passthrough_response(response: httpx.Response) -> bool:
"""
if response.status_code >= 400:
return True
media_type: Final = response.headers.get("content-type", "").split(";")[0].strip().lower()
content_type_header: Final[str] = response.headers.get("content-type", "")
media_type: Final = content_type_header.split(";")[0].strip().lower()
return media_type in ("", "application/json") or media_type.endswith("+json")
@ -2368,7 +2382,7 @@ async def _relay_passthrough_response_bytes(
)
def _extract_model_from_vertex_ai_setup(setup_response: dict) -> str | None:
def _extract_model_from_vertex_ai_setup(setup_response: Mapping[str, object]) -> str | None:
"""
Extract the model name from Vertex AI Live setup response.
@ -2434,7 +2448,7 @@ class SafeRouteAdder:
def add_api_route_if_not_exists(
app: FastAPI,
path: str,
endpoint: Any,
endpoint: Callable[..., object],
methods: list[str],
dependencies: list | None = None,
) -> bool:
@ -2767,7 +2781,7 @@ def _get_combined_pass_through_endpoints(
async def _register_pass_through_endpoint(
endpoint: dict[str, Any] | PassThroughGenericEndpoint,
endpoint: dict[str, object] | PassThroughGenericEndpoint,
app: FastAPI,
premium_user: bool,
visited_endpoints: set[str],
@ -2783,8 +2797,8 @@ async def _register_pass_through_endpoint(
endpoint_data["id"] = str(uuid.uuid4())
endpoint_id: Final = cast(str, endpoint_data["id"])
target: Final = endpoint_data.get("target")
path: Final = endpoint_data.get("path")
target: Final[str | None] = endpoint_data.get("target")
path: Final[str | None] = endpoint_data.get("path")
if path is None:
raise ValueError("Path is required for pass-through endpoint")
@ -2792,7 +2806,7 @@ async def _register_pass_through_endpoint(
forward_headers: Final = endpoint_data.get("forward_headers")
merge_query_params: Final = endpoint_data.get("merge_query_params")
default_query_params: Final = endpoint_data.get("default_query_params")
auth: Final = endpoint_data.get("auth")
auth: Final[bool | str | None] = endpoint_data.get("auth")
dependencies = None
auth_enforced: Final = auth is not None and str(auth).lower() == "true"
@ -2951,12 +2965,12 @@ def _get_pass_through_endpoints_from_config() -> list[PassThroughGenericEndpoint
if isinstance(endpoint, dict):
endpoint_dict = dict(endpoint)
endpoint_dict["is_from_config"] = True
returned_endpoints.append(PassThroughGenericEndpoint(**endpoint_dict))
returned_endpoints.append(PassThroughGenericEndpoint.model_validate(endpoint_dict))
elif isinstance(endpoint, PassThroughGenericEndpoint):
# Create a copy with is_from_config=True
endpoint_dict = endpoint.model_dump()
endpoint_dict["is_from_config"] = True
returned_endpoints.append(PassThroughGenericEndpoint(**endpoint_dict))
returned_endpoints.append(PassThroughGenericEndpoint.model_validate(endpoint_dict))
except ValidationError as e:
verbose_proxy_logger.warning(
"Skipping malformed pass-through endpoint from config: %s",
@ -2994,11 +3008,11 @@ async def _get_pass_through_endpoints_from_db(
if isinstance(endpoint, dict):
endpoint_dict = dict(endpoint)
endpoint_dict["is_from_config"] = False
returned_endpoints.append(PassThroughGenericEndpoint(**endpoint_dict))
returned_endpoints.append(PassThroughGenericEndpoint.model_validate(endpoint_dict))
elif isinstance(endpoint, PassThroughGenericEndpoint):
endpoint_dict = endpoint.model_dump()
endpoint_dict["is_from_config"] = False
returned_endpoints.append(PassThroughGenericEndpoint(**endpoint_dict))
returned_endpoints.append(PassThroughGenericEndpoint.model_validate(endpoint_dict))
else:
# Find specific endpoint by ID
found_endpoint: Final = _find_endpoint_by_id(pass_through_endpoint_data, endpoint_id)
@ -3009,7 +3023,7 @@ async def _get_pass_through_endpoints_from_db(
else dict(found_endpoint)
)
endpoint_dict["is_from_config"] = False
returned_endpoints.append(PassThroughGenericEndpoint(**endpoint_dict))
returned_endpoints.append(PassThroughGenericEndpoint.model_validate(endpoint_dict))
return returned_endpoints
@ -3191,7 +3205,7 @@ async def update_pass_through_endpoints(
endpoint_dict.pop("is_from_config", None)
# Create updated endpoint object
updated_endpoint: Final = PassThroughGenericEndpoint(**endpoint_dict)
updated_endpoint: Final = PassThroughGenericEndpoint.model_validate(endpoint_dict)
# Update the list
pass_through_endpoint_data[endpoint_index] = endpoint_dict
@ -3212,9 +3226,10 @@ async def update_pass_through_endpoints(
_custom_headers: dict | None = updated_endpoint.headers or {}
_custom_headers = await set_env_variables_in_header(custom_headers=_custom_headers)
route_app: Final[FastAPI] = request.app
if updated_endpoint.include_subpath:
InitPassThroughEndpointHelpers.add_subpath_route(
app=request.app,
app=route_app,
path=updated_endpoint.path,
target=updated_endpoint.target,
custom_headers=_custom_headers,
@ -3231,7 +3246,7 @@ async def update_pass_through_endpoints(
)
else:
InitPassThroughEndpointHelpers.add_exact_path_route(
app=request.app,
app=route_app,
path=updated_endpoint.path,
target=updated_endpoint.target,
custom_headers=_custom_headers,
@ -3297,15 +3312,16 @@ async def create_pass_through_endpoints(
await update_config_general_settings(data=updated_data, user_api_key_dict=user_api_key_dict)
# Return the created endpoint with the generated ID
created_endpoint: Final = PassThroughGenericEndpoint(**data_dict)
created_endpoint: Final = PassThroughGenericEndpoint.model_validate(data_dict)
# Register the new route
_custom_headers: dict | None = created_endpoint.headers or {}
_custom_headers = await set_env_variables_in_header(custom_headers=_custom_headers)
route_app: Final[FastAPI] = request.app
if created_endpoint.include_subpath:
InitPassThroughEndpointHelpers.add_subpath_route(
app=request.app,
app=route_app,
path=created_endpoint.path,
target=created_endpoint.target,
custom_headers=_custom_headers,
@ -3322,7 +3338,7 @@ async def create_pass_through_endpoints(
)
else:
InitPassThroughEndpointHelpers.add_exact_path_route(
app=request.app,
app=route_app,
path=created_endpoint.path,
target=created_endpoint.target,
custom_headers=_custom_headers,

16
litellm/proxy/pass_through_endpoints/success_handler.py
View file

@ -122,7 +122,7 @@ class PassThroughEndpointLogging:
def normalize_llm_passthrough_logging_payload(
self,
httpx_response: httpx.Response,
response_body: dict | None,
response_body: dict | list[dict[str, object]] | None,
request_body: dict,
logging_obj: LiteLLMLoggingObj,
url_route: str,
@ -142,7 +142,7 @@ class PassThroughEndpointLogging:
if self.is_gemini_route(url_route, custom_llm_provider):
gemini_passthrough_logging_handler_result = GeminiPassthroughLoggingHandler.gemini_passthrough_handler(
httpx_response=httpx_response,
response_body=response_body or {},
response_body=response_body if isinstance(response_body, dict) else {},
logging_obj=logging_obj,
url_route=url_route,
result=result,
@ -172,7 +172,7 @@ class PassThroughEndpointLogging:
anthropic_passthrough_logging_handler_result: Final = (
AnthropicPassthroughLoggingHandler.anthropic_passthrough_handler(
httpx_response=httpx_response,
response_body=response_body or {},
response_body=response_body if isinstance(response_body, dict) else {},
logging_obj=logging_obj,
url_route=url_route,
result=result,
@ -189,7 +189,7 @@ class PassThroughEndpointLogging:
elif self.is_cohere_route(url_route):
cohere_passthrough_logging_handler_result = cohere_passthrough_logging_handler.cohere_passthrough_handler(
httpx_response=httpx_response,
response_body=response_body or {},
response_body=response_body if isinstance(response_body, dict) else {},
logging_obj=logging_obj,
url_route=url_route,
result=result,
@ -208,7 +208,7 @@ class PassThroughEndpointLogging:
openai_passthrough_logging_handler_result = OpenAIPassthroughLoggingHandler.openai_passthrough_handler(
httpx_response=httpx_response,
response_body=response_body or {},
response_body=response_body if isinstance(response_body, dict) else {},
logging_obj=logging_obj,
url_route=url_route,
result=result,
@ -224,7 +224,7 @@ class PassThroughEndpointLogging:
elif self.is_cursor_route(url_route, custom_llm_provider):
cursor_passthrough_logging_handler_result = CursorPassthroughLoggingHandler.cursor_passthrough_handler(
httpx_response=httpx_response,
response_body=response_body or {},
response_body=response_body if isinstance(response_body, dict) else {},
logging_obj=logging_obj,
url_route=url_route,
result=result,
@ -266,7 +266,7 @@ class PassThroughEndpointLogging:
async def pass_through_async_success_handler(
self,
httpx_response: httpx.Response,
response_body: dict | None,
response_body: dict | list[dict[str, object]] | None,
logging_obj: LiteLLMLoggingObj,
url_route: str,
result: str,
@ -285,7 +285,7 @@ class PassThroughEndpointLogging:
return
self.assemblyai_passthrough_logging_handler.assemblyai_passthrough_logging_handler(
httpx_response=httpx_response,
response_body=response_body or {},
response_body=response_body if isinstance(response_body, dict) else {},
logging_obj=logging_obj,
url_route=url_route,
result=result,

272
litellm/proxy/proxy_server.py
View file

@ -9,13 +9,14 @@ import random
import re
import secrets
import shutil
import socket
import subprocess
import sys
import threading
import time
import traceback
import warnings
from collections.abc import AsyncGenerator, AsyncIterator, Callable, Mapping
from collections.abc import AsyncGenerator, AsyncIterator, Callable, Mapping, MutableMapping, Sequence
from datetime import datetime, timedelta, timezone
from types import MappingProxyType, UnionType
from typing import (
@ -25,6 +26,8 @@ from typing import (
Literal,
NamedTuple,
Optional,
Protocol,
TypeAlias,
TypedDict,
Union,
cast,
@ -128,6 +131,7 @@ from litellm.utils import (
if TYPE_CHECKING:
from aiohttp import ClientSession
from fastapi.routing import APIRoute
from opentelemetry.trace import Span as _Span
from litellm.integrations.opentelemetry import OpenTelemetry
@ -137,7 +141,7 @@ else:
Span = Any
OpenTelemetry = Any
REALTIME_REQUEST_SCOPE_TEMPLATE: Final[dict[str, Any]] = {
REALTIME_REQUEST_SCOPE_TEMPLATE: Final[dict[str, object]] = {
"type": "http",
"method": "POST",
"path": "/v1/realtime",
@ -233,6 +237,8 @@ from litellm.constants import (
GLOBAL_PROXY_SPEND_CACHE_KEY,
LITELLM_PROXY_ADMIN_NAME,
LITELLM_PROXY_BUDGET_NAME,
MONTHLY_SPEND_REPORT_JOB_ID,
PROMETHEUS_FALLBACK_STATS_JOB_ID,
PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS,
PROXY_BATCH_POLLING_ENABLED,
PROXY_BATCH_POLLING_INTERVAL,
@ -240,6 +246,7 @@ from litellm.constants import (
PROXY_BUDGET_RESCHEDULER_MAX_TIME,
PROXY_BUDGET_RESCHEDULER_MIN_TIME,
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS,
WEEKLY_SPEND_REPORT_JOB_ID,
)
from litellm.exceptions import RejectedRequestError
from litellm.integrations.custom_guardrail import ModifyResponseException
@ -596,6 +603,7 @@ from litellm.proxy.utils import (
update_spend,
)
from litellm.proxy.video_endpoints.endpoints import router as video_router
from litellm.repositories.base_repository import SupportsModelDump
from litellm.repositories.credentials_repository import CredentialsRepository
from litellm.router import (
AssistantsTypedDict,
@ -870,6 +878,18 @@ async def proxy_shutdown_event():
cleanup_router_config_variables()
_AiohttpAddrInfo: TypeAlias = tuple[int | socket.AddressFamily, int | socket.SocketKind, int, str, tuple[object, ...]]
class _AiohttpConnectorKwargs(TypedDict, total=False):
keepalive_timeout: float
ttl_dns_cache: int
enable_cleanup_closed: bool
limit: int
limit_per_host: int
socket_factory: Callable[[_AiohttpAddrInfo], socket.socket]
async def _initialize_shared_aiohttp_session():
"""Initialize shared aiohttp session for connection reuse with connection limits."""
try:
@ -879,7 +899,7 @@ async def _initialize_shared_aiohttp_session():
_build_aiohttp_keepalive_socket_factory,
)
connector_kwargs: Final[dict[str, Any]] = {
connector_kwargs: Final[_AiohttpConnectorKwargs] = {
"keepalive_timeout": AIOHTTP_KEEPALIVE_TIMEOUT,
"ttl_dns_cache": AIOHTTP_TTL_DNS_CACHE,
}
@ -1234,7 +1254,7 @@ async def proxy_startup_event(app: FastAPI):
await proxy_shutdown_event()
def _generate_stable_operation_id(route: Any) -> str:
def _generate_stable_operation_id(route: "APIRoute") -> str:
operation_id = re.sub(r"\W", "_", f"{route.name}{route.path_format}")
route_methods: Final = sorted(route.methods or [])
if len(route_methods) == 1:
@ -1493,7 +1513,7 @@ async def openai_exception_handler(request: Request, exc: ProxyException):
def _close_dangling_otel_server_span(request: Request, status_code: int, exc: Exception | None = None) -> None:
parent_otel_span: Final = getattr(request.state, "parent_otel_span", None)
parent_otel_span: Final[_Span | None] = getattr(request.state, "parent_otel_span", None)
if parent_otel_span is None:
return
if open_telemetry_logger is None:
@ -1536,17 +1556,80 @@ async def management_problem_exception_handler(request: Request, exc: Management
return problem_response(exc.problem)
class _ConfigParamRow(Protocol):
param_name: str
param_value: Mapping[str, JsonValue] | None
class _ConfigOverridesRow(Protocol):
config_value: Mapping[str, JsonValue] | None
class _SSOConfigRow(Protocol):
sso_settings: MutableMapping[str, object]
class _UISettingsRow(Protocol):
ui_settings: Mapping[str, object] | str | None
class _InvitationLinkRow(Protocol):
user_id: str
expires_at: datetime
is_accepted: bool
accepted_at: datetime | None
created_by: str
class _UserTableRow(Protocol):
user_id: str
user_email: str | None
user_role: str
class _ModelTableRow(Protocol):
model_id: str | None
created_by: str | None
class _TTFTRow(TypedDict):
api_base: str
model: str
time_to_first_token: float
request_id: str
day: str
class _LatencyRow(TypedDict):
api_base: str | None
model: str
day: str
avg_latency_per_token: float
class _ExceptionRow(TypedDict, total=False):
combined_model_api_base: str
total_exceptions: int
exception_counts: Mapping[str, int]
class _ValidationErrorDetail(TypedDict):
loc: tuple[int | str, ...]
msg: str
@app.exception_handler(RequestValidationError)
async def otel_request_validation_exception_handler(request: Request, exc: RequestValidationError):
if request.url.path.startswith(MANAGEMENT_V1_PREFIX):
_close_dangling_otel_server_span(request, 400, exc=exc)
validation_errors: Final[Sequence[_ValidationErrorDetail]] = exc.errors()
return problem_response(
ProblemDetail(
type=f"{PROBLEM_TYPE_BASE}invalid-query-parameter",
title="Invalid query parameter",
status=400,
detail="; ".join(
f"{'.'.join(str(part) for part in error['loc'][1:])}: {error['msg']}" for error in exc.errors()
f"{'.'.join(str(part) for part in error['loc'][1:])}: {error['msg']}" for error in validation_errors
)
or "The request query parameters are invalid.",
)
@ -2150,13 +2233,14 @@ db_writer_client: AsyncHTTPHandler | None = None
### logger ###
def _resolve_typed_dict_type(typ):
def _resolve_typed_dict_type(typ: object):
"""Resolve the actual TypedDict class from a potentially wrapped type."""
from typing_extensions import _TypedDictMeta
origin: Final = get_origin(typ)
origin: Final[object] = get_origin(typ)
if origin is Union or origin is UnionType: # Check if it's a Union (like Optional)
for arg in get_args(typ):
union_args: Final[tuple[object, ...]] = get_args(typ)
for arg in union_args:
if isinstance(arg, _TypedDictMeta):
return arg
elif isinstance(typ, type) and isinstance(typ, dict):
@ -2164,12 +2248,13 @@ def _resolve_typed_dict_type(typ):
return None
def _resolve_pydantic_type(typ) -> list:
def _resolve_pydantic_type(typ: object) -> list:
"""Resolve the actual TypedDict class from a potentially wrapped type."""
origin: Final = get_origin(typ)
origin: Final[object] = get_origin(typ)
typs: Final = []
if origin is Union or origin is UnionType: # Check if it's a Union (like Optional)
for arg in get_args(typ):
union_args: Final[tuple[object, ...]] = get_args(typ)
for arg in union_args:
if arg is not None and "NoneType" not in str(arg):
typs.append(arg)
elif isinstance(typ, type) and isinstance(typ, BaseModel):
@ -2499,7 +2584,7 @@ async def increment_spend_counters(
increment=cost,
)
key_obj: Final = await user_api_key_cache.async_get_cache(key=hashed_token)
key_obj: Final[object] = await user_api_key_cache.async_get_cache(key=hashed_token)
if key_obj is None:
return
key_budget_limits = getattr(key_obj, "budget_limits", None) or (
@ -2530,7 +2615,7 @@ async def increment_spend_counters(
increment=cost,
)
team_obj: Final = await user_api_key_cache.async_get_cache(key=f"team_id:{scope_team_id}")
team_obj: Final[object] = await user_api_key_cache.async_get_cache(key=f"team_id:{scope_team_id}")
if team_obj is None:
return
team_budget_limits = getattr(team_obj, "budget_limits", None) or (
@ -2824,7 +2909,7 @@ async def _ensure_window_spend_counter_initialized(
async def _is_spend_counter_cache_warm(counter_key: str) -> bool:
if spend_counter_cache.redis_cache is not None:
try:
current_value: Final = await spend_counter_cache.redis_cache.async_get_cache(
current_value: Final[object] = await spend_counter_cache.redis_cache.async_get_cache(
key=counter_key,
)
if current_value is None:
@ -2896,7 +2981,7 @@ async def update_cache(
Put any alerting logic in here.
"""
values_to_update_in_cache: Final[list[tuple[Any, Any]]] = []
values_to_update_in_cache: Final[list[tuple[str, object]]] = []
### UPDATE KEY SPEND ###
async def _update_key_cache(token: str, response_cost: float):
@ -4108,7 +4193,9 @@ class ProxyConfig:
if prisma_client is None or not (general_settings.get("store_model_in_db", False) is True or store_model_in_db):
return
row = await ConfigRepository(prisma_client).table.find_first(where={"param_name": "environment_variables"})
row: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "environment_variables"}
)
existing: Final[dict] = dict(row.param_value) if row is not None and row.param_value is not None else {}
to_set: Final = {k: v for k, v in updates.items() if v is not None}
@ -5906,7 +5993,7 @@ class ProxyConfig:
4. Update router settings
"""
if llm_router is not None and prisma_client is not None:
db_router_settings: Final = await ConfigRepository(prisma_client).table.find_first(
db_router_settings: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "router_settings"}
)
@ -6391,7 +6478,9 @@ class ProxyConfig:
new_models=new_models, proxy_logging_obj=proxy_logging_obj
)
db_general_settings: Final = await get_config_param(prisma_client, "general_settings")
db_general_settings: Final[_ConfigParamRow | None] = await get_config_param(
prisma_client, "general_settings"
)
# update general settings
if db_general_settings is not None:
@ -6587,7 +6676,7 @@ class ProxyConfig:
"""
try:
sso_settings: Final = await call_with_db_reconnect_retry(
sso_settings: Final[_SSOConfigRow | None] = await call_with_db_reconnect_retry(
prisma_client,
lambda: SSOConfigRepository(prisma_client).table.find_unique(where={"id": "sso_config"}),
reason="init_sso_settings_in_db_lookup_failure",
@ -6618,7 +6707,7 @@ class ProxyConfig:
)
try:
db_record: Final = await call_with_db_reconnect_retry(
db_record: Final[_ConfigOverridesRow | None] = await call_with_db_reconnect_retry(
prisma_client,
lambda: ConfigOverridesRepository(prisma_client).table.find_unique(
where={"config_type": "hashicorp_vault"}
@ -6836,7 +6925,7 @@ class ProxyConfig:
from litellm.types.prompts.init_prompts import PromptSpec
try:
prompts_in_db: Final = await PromptRepository(prisma_client).table.find_many()
prompts_in_db: Final[Sequence[object]] = await PromptRepository(prisma_client).table.find_many()
for prompt in prompts_in_db:
# Convert DB object to dict and create versioned prompt_id
prompt_spec = self._get_prompt_spec_for_db_prompt(db_prompt=prompt)
@ -8448,7 +8537,9 @@ class ProxyStartupEvent:
if prisma_client is None:
return
db_record: Final = await UISettingsRepository(prisma_client).table.find_unique(where={"id": "ui_settings"})
db_record: Final[_UISettingsRow | None] = await UISettingsRepository(prisma_client).table.find_unique(
where={"id": "ui_settings"}
)
if db_record and db_record.ui_settings:
raw: Final = db_record.ui_settings
ui_settings: Final = json.loads(raw) if isinstance(raw, str) else dict(raw)
@ -8597,7 +8688,7 @@ class ProxyStartupEvent:
# but YAML config has False.
if store_model_in_db is not True and prisma_client is not None:
try:
_db_gs_record: Final = await ConfigRepository(prisma_client).table.find_first(
_db_gs_record: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "general_settings"}
)
if _db_gs_record is not None and isinstance(_db_gs_record.param_value, dict):
@ -9043,41 +9134,76 @@ class ProxyStartupEvent:
spend_report_frequency: Final[str] = general_settings.get("spend_report_frequency", "7d") or "7d"
days: Final = int(spend_report_frequency[:-1])
if spend_report_frequency[-1].lower() != "d":
raise ValueError("spend_report_frequency must be specified in days, e.g., '1d', '7d'")
if spend_report_frequency[-1].lower() != "d" or days <= 0:
raise ValueError("spend_report_frequency must be a positive number of days, e.g., '1d', '7d'")
pod_lock_manager: Final = proxy_logging_obj.db_spend_update_writer.pod_lock_manager
weekly_lock_ttl: Final = duration_in_seconds(spend_report_frequency) - 3600
async def _scheduled_weekly_spend_report() -> None:
# TTL spans the whole reporting window: each pod's interval anchor is its own
# boot time + jitter, so a shorter lock would let a later pod re-send the report.
# Minus an hour so the next window's first firer finds a free key
if (
await pod_lock_manager.acquire_lock(
cronjob_id=WEEKLY_SPEND_REPORT_JOB_ID, ttl=weekly_lock_ttl, allow_reentrant=False
)
is False
):
return
await proxy_logging_obj.slack_alerting_instance.send_weekly_spend_report(spend_report_frequency)
async def _scheduled_monthly_spend_report() -> None:
if (
await pod_lock_manager.acquire_lock(
cronjob_id=MONTHLY_SPEND_REPORT_JOB_ID, ttl=3600, allow_reentrant=False
)
is False
):
return
await proxy_logging_obj.slack_alerting_instance.send_monthly_spend_report()
scheduler.add_job(
proxy_logging_obj.slack_alerting_instance.send_weekly_spend_report,
_scheduled_weekly_spend_report,
"interval",
days=days,
next_run_time=datetime.now() + timedelta(seconds=10 + random.randint(0, 300)),
args=[spend_report_frequency],
id="weekly_spend_report_job",
id=WEEKLY_SPEND_REPORT_JOB_ID,
replace_existing=True,
misfire_grace_time=APSCHEDULER_MISFIRE_GRACE_TIME,
)
scheduler.add_job(
proxy_logging_obj.slack_alerting_instance.send_monthly_spend_report,
_scheduled_monthly_spend_report,
"cron",
day=1,
id="monthly_spend_report_job",
id=MONTHLY_SPEND_REPORT_JOB_ID,
replace_existing=True,
)
if os.getenv("PROMETHEUS_URL"):
from zoneinfo import ZoneInfo
async def _scheduled_fallback_stats() -> None:
if (
await pod_lock_manager.acquire_lock(
cronjob_id=PROMETHEUS_FALLBACK_STATS_JOB_ID, ttl=3600, allow_reentrant=False
)
is False
):
return
await proxy_logging_obj.slack_alerting_instance.send_fallback_stats_from_prometheus()
scheduler.add_job(
proxy_logging_obj.slack_alerting_instance.send_fallback_stats_from_prometheus,
_scheduled_fallback_stats,
"cron",
hour=PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS,
minute=0,
timezone=ZoneInfo("America/Los_Angeles"),
id="prometheus_fallback_stats_job",
id=PROMETHEUS_FALLBACK_STATS_JOB_ID,
replace_existing=True,
)
await proxy_logging_obj.slack_alerting_instance.send_fallback_stats_from_prometheus()
await _scheduled_fallback_stats()
@classmethod
async def _setup_prisma_client(
@ -10441,7 +10567,7 @@ async def vertex_ai_live_passthrough_endpoint(
None,
description="Override the Vertex AI region (for example, 'us-central1').",
),
user_api_key_dict=Depends(user_api_key_auth_websocket),
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth_websocket),
):
"""
Vertex AI Live API WebSocket Pass-through Endpoint
@ -10489,7 +10615,7 @@ async def realtime_websocket_endpoint(
None,
description="Comma-separated list of guardrail names to apply to this request.",
),
user_api_key_dict=Depends(user_api_key_auth_websocket),
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth_websocket),
):
requested_protocols: Final = [
p.strip() for p in (websocket.headers.get("sec-websocket-protocol") or "").split(",") if p.strip()
@ -10521,7 +10647,7 @@ async def realtime_websocket_endpoint(
# Only use explicit parameters, not all query params
query_params: Final = cast(RealtimeQueryParams, dict(_realtime_query_params_template(model, intent)))
data: dict[str, Any] = {
data: dict[str, object] = {
"model": route_model,
"websocket": websocket,
"query_params": query_params, # Only explicit params
@ -11654,7 +11780,7 @@ async def _check_if_model_is_user_added(
id = model.get("model_info", {}).get("id", None)
if id is None:
continue
db_model = await ModelRepository(prisma_client).table.find_unique(where={"model_id": id})
db_model: _ModelTableRow | None = await ModelRepository(prisma_client).table.find_unique(where={"model_id": id})
if db_model is not None:
if db_model.created_by == user_api_key_dict.user_id:
filtered_models.append(model)
@ -11838,7 +11964,7 @@ async def get_all_team_models(
team_db_objects_typed: list[LiteLLM_TeamTable] = []
if user_teams == "*":
team_db_objects = await TeamRepository(prisma_client).table.find_many()
team_db_objects: Sequence[SupportsModelDump] = await TeamRepository(prisma_client).table.find_many()
team_db_objects_typed = [
LiteLLM_TeamTable.model_validate(team_db_object.model_dump()) for team_db_object in team_db_objects
]
@ -11917,7 +12043,7 @@ async def _populate_team_access_on_models(
user_teams = "*"
direct_access_models = llm_router.get_model_ids(exclude_team_models=True) # has access to all models
elif user_api_key_dict.user_id is not None:
user_db_object: Final = await UserRepository(prisma_client).table.find_unique(
user_db_object: Final[SupportsModelDump | None] = await UserRepository(prisma_client).table.find_unique(
where={"user_id": user_api_key_dict.user_id}
)
if user_db_object is not None:
@ -12473,7 +12599,9 @@ def _team_models_resolve_to_names(team_models: list[str], access_groups: dict[st
async def _load_team_object_for_model_filter(team_id: str, prisma_client: PrismaClient) -> LiteLLM_TeamTable | None:
"""Load team row from DB; returns None if missing or on error."""
try:
team_db_object: Final = await TeamRepository(prisma_client).table.find_unique(where={"team_id": team_id})
team_db_object: Final[SupportsModelDump | None] = await TeamRepository(prisma_client).table.find_unique(
where={"team_id": team_id}
)
if team_db_object is None:
verbose_proxy_logger.warning("Team %s not found in database", team_id)
return None
@ -12522,7 +12650,7 @@ async def _gather_team_accessible_model_ids(
try:
if team_object.models and SpecialModelNames.all_proxy_models.value not in team_object.models:
_resolved_names: Final = _team_models_resolve_to_names(team_object.models, access_groups)
db_models: Final = await ModelRepository(prisma_client).table.find_many(
db_models: Final[Sequence[_ModelTableRow]] = await ModelRepository(prisma_client).table.find_many(
where={"model_name": {"in": _resolved_names}}
)
for db_model in db_models:
@ -12984,7 +13112,9 @@ async def model_streaming_metrics(
"""
_all_api_bases: Final = set()
db_response: Final = await prisma_client.db.query_raw(sql_query, _selected_model_group, startTime, endTime)
db_response: Final[Sequence[_TTFTRow] | None] = await prisma_client.db.query_raw(
sql_query, _selected_model_group, startTime, endTime
)
_daily_entries: dict = {} # {"Jun 23": {"model1": 0.002, "model2": 0.003}}
if db_response is not None:
for model_data in db_response:
@ -13106,7 +13236,7 @@ async def model_metrics(
avg_latency_per_token DESC;
"""
_all_api_bases: Final = set()
db_response: Final = await prisma_client.db.query_raw(
db_response: Final[Sequence[_LatencyRow] | None] = await prisma_client.db.query_raw(
sql_query, _selected_model_group, startTime, endTime, api_key, customer
)
_daily_entries: dict = {} # {"Jun 23": {"model1": 0.002, "model2": 0.003}}
@ -13297,7 +13427,9 @@ async def model_metrics_exceptions(
ORDER BY total_exceptions DESC
LIMIT 200;
"""
db_response: Final = await prisma_client.db.query_raw(sql_query, startTime, endTime, _selected_model_group, api_key)
db_response: Final[Sequence[_ExceptionRow] | None] = await prisma_client.db.query_raw(
sql_query, startTime, endTime, _selected_model_group, api_key
)
response: Final[list[dict]] = []
exception_types: Final = set()
@ -14459,7 +14591,9 @@ async def onboarding(invite_link: str, request: Request):
detail={"error": CommonProxyErrors.db_not_connected_error.value},
)
invite_obj: Final = await InvitationLinkRepository(prisma_client).table.find_unique(where={"id": invite_link})
invite_obj: Final[_InvitationLinkRow | None] = await InvitationLinkRepository(prisma_client).table.find_unique(
where={"id": invite_link}
)
if invite_obj is None:
raise HTTPException(status_code=401, detail={"error": "Invitation link does not exist in db."})
#### CHECK IF EXPIRED
@ -14477,7 +14611,9 @@ async def onboarding(invite_link: str, request: Request):
)
### GET USER OBJECT ###
user_obj: Final = await UserRepository(prisma_client).table.find_unique(where={"user_id": invite_obj.user_id})
user_obj: Final[_UserTableRow | None] = await UserRepository(prisma_client).table.find_unique(
where={"user_id": invite_obj.user_id}
)
if user_obj is None:
raise HTTPException(status_code=401, detail={"error": "User does not exist in db."})
@ -14649,7 +14785,9 @@ async def claim_onboarding_link(data: InvitationClaim, request: Request):
detail={"error": CommonProxyErrors.db_not_connected_error.value},
)
invite_obj = await InvitationLinkRepository(prisma_client).table.find_unique(where={"id": data.invitation_link})
invite_obj: Final[_InvitationLinkRow | None] = await InvitationLinkRepository(prisma_client).table.find_unique(
where={"id": data.invitation_link}
)
if invite_obj is None:
raise HTTPException(status_code=401, detail={"error": "Invitation link does not exist in db."})
#### CHECK IF EXPIRED
@ -14704,7 +14842,7 @@ async def claim_onboarding_link(data: InvitationClaim, request: Request):
)
### UPDATE USER OBJECT ###
user_obj: Final = await tx.litellm_usertable.update(
user_obj: Final[_UserTableRow | None] = await tx.litellm_usertable.update(
where={"user_id": invite_obj.user_id}, data={"password": hashed_pw}
)
@ -14930,7 +15068,7 @@ async def new_invitation(data: InvitationNew, user_api_key_dict: UserAPIKeyAuth
detail={"error": "You can only create invitations for users in your organization or team."},
)
response: Final = await create_invitation_for_user(
response: Final[object] = await create_invitation_for_user(
data=data,
user_api_key_dict=user_api_key_dict,
)
@ -14972,7 +15110,9 @@ async def invitation_info(invitation_id: str, user_api_key_dict: UserAPIKeyAuth
detail={"error": f"{CommonProxyErrors.not_allowed_access.value}, your role={user_api_key_dict.user_role}"},
)
response: Final = await InvitationLinkRepository(prisma_client).table.find_unique(where={"id": invitation_id})
response: Final[object] = await InvitationLinkRepository(prisma_client).table.find_unique(
where={"id": invitation_id}
)
if response is None:
raise HTTPException(
@ -15020,7 +15160,7 @@ async def invitation_update(
)
current_time: Final = litellm.utils.get_utc_datetime()
response: Final = await InvitationLinkRepository(prisma_client).table.update(
response: Final[object] = await InvitationLinkRepository(prisma_client).table.update(
where={"id": data.invitation_id},
data={
"id": data.invitation_id,
@ -15086,7 +15226,9 @@ async def invitation_delete(
# Org admins can only delete invitations they created
if is_other_admin and not is_proxy_admin:
invitation = await InvitationLinkRepository(prisma_client).table.find_unique(where={"id": data.invitation_id})
invitation: Final[_InvitationLinkRow | None] = await InvitationLinkRepository(prisma_client).table.find_unique(
where={"id": data.invitation_id}
)
if invitation is None:
raise HTTPException(
status_code=400,
@ -15098,7 +15240,9 @@ async def invitation_delete(
detail={"error": "Organization admins can only delete invitations they created."},
)
response: Final = await InvitationLinkRepository(prisma_client).table.delete(where={"id": data.invitation_id})
response: Final[object] = await InvitationLinkRepository(prisma_client).table.delete(
where={"id": data.invitation_id}
)
if response is None:
raise HTTPException(
@ -15136,7 +15280,9 @@ async def update_config(
raise Exception("No DB Connected")
async def _read_section(param_name: str) -> dict:
row: Final = await ConfigRepository(prisma_client).table.find_first(where={"param_name": param_name})
row: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": param_name}
)
if row is None or row.param_value is None:
return {}
return dict(row.param_value)
@ -15159,7 +15305,7 @@ async def update_config(
if config_info.general_settings is not None:
existing = await _read_section("general_settings")
before_general_settings: Final = copy.deepcopy(existing)
updates = config_info.general_settings.dict(exclude_none=True)
updates: Mapping[str, JsonValue] = config_info.general_settings.dict(exclude_none=True)
for k, v in updates.items():
if k == "alert_to_webhook_url":
if "alerting" not in existing:
@ -15599,7 +15745,7 @@ async def get_config_general_settings(
)
## get general settings from db
db_general_settings: Final = await ConfigRepository(prisma_client).table.find_first(
db_general_settings: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "general_settings"}
)
### pop the value
@ -15788,12 +15934,12 @@ async def get_config_list(
is_full_admin: Final = user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN
## get general settings from db
db_general_settings: Final = await ConfigRepository(prisma_client).table.find_first(
db_general_settings: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "general_settings"}
)
if db_general_settings is not None and db_general_settings.param_value is not None:
db_general_settings_dict = dict(db_general_settings.param_value)
db_general_settings_dict: Mapping[str, JsonValue] = dict(db_general_settings.param_value)
else:
db_general_settings_dict = {}
@ -15884,7 +16030,7 @@ async def get_config_list(
)
return_val.append(_response_obj)
db_litellm_settings_row: Final = await ConfigRepository(prisma_client).table.find_first(
db_litellm_settings_row: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "litellm_settings"}
)
db_litellm_settings: Final[dict] = (
@ -15961,7 +16107,7 @@ async def delete_config_general_settings(
)
## get general settings from db
db_general_settings: Final = await ConfigRepository(prisma_client).table.find_first(
db_general_settings: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_first(
where={"param_name": "general_settings"}
)
### pop the value
@ -16528,7 +16674,7 @@ async def reload_anthropic_beta_headers(
last_anthropic_beta_headers_reload = current_time.isoformat()
# Set force reload flag in database for other pods, preserving existing interval_hours
existing_beta_config: Final = await ConfigRepository(prisma_client).table.find_unique(
existing_beta_config: Final[_ConfigParamRow | None] = await ConfigRepository(prisma_client).table.find_unique(
where={"param_name": "anthropic_beta_headers_reload_config"}
)
existing_beta_interval = None
@ -17116,7 +17262,7 @@ async def _is_mcp_access_group_cached(name: str) -> bool:
)
cache_key: Final = f"mcp_access_group_exists:{name}"
cached: Final = await user_api_key_cache.async_get_cache(key=cache_key)
cached: Final[object] = await user_api_key_cache.async_get_cache(key=cache_key)
if cached is not None:
return bool(cached)
result: Final = bool(await MCPRequestHandler._get_mcp_servers_from_access_groups([name]))

5
litellm/proxy/rerank_endpoints/endpoints.py
View file

@ -1,5 +1,8 @@
#### Rerank Endpoints #####
import asyncio
from typing import Final
import orjson
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
from fastapi.responses import ORJSONResponse
@ -10,8 +13,6 @@ from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
router: Final = APIRouter()
import asyncio
from typing import Final
@router.post(

2
litellm/proxy/response_api_endpoints/endpoints.py
View file

@ -95,7 +95,7 @@ def _normalize_tool_dialect(
def _is_chat_completions_body(data: Mapping[str, Any]) -> bool:
messages: Final = data.get("messages")
if isinstance(messages, list) and len(messages) > 0:
if isinstance(messages, list) and messages:
return True
return "messages" in data and "input" not in data

2
litellm/proxy/schema.prisma
View file

@ -452,6 +452,7 @@ model LiteLLM_VerificationToken {
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")
updated_by String?
settings_updated_at DateTime? @map("settings_updated_at")
last_active DateTime? // When this key was last used
rotation_count Int? @default(0) // Number of times key has been rotated
auto_rotate Boolean? @default(false) // Whether this key should be auto-rotated
@ -548,6 +549,7 @@ model LiteLLM_DeletedVerificationToken {
created_by String? // Original creator
updated_at DateTime? // Last update timestamp before deletion
updated_by String? // Last user who updated before deletion
settings_updated_at DateTime? // Last configuration change before deletion
last_active DateTime? // When this key was last used before deletion
rotation_count Int? @default(0)
auto_rotate Boolean? @default(false)

109
litellm/proxy/utils.py
View file

@ -15,7 +15,7 @@ from dataclasses import dataclass, field
from datetime import date, datetime, timedelta, timezone
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from typing import TYPE_CHECKING, Any, ClassVar, Final, Literal, Optional, Union, cast, overload
from typing import TYPE_CHECKING, Any, ClassVar, Final, Literal, Optional, TypeVar, Union, cast, overload
from litellm import _custom_logger_compatible_callbacks_literal
from litellm.constants import (
@ -135,6 +135,7 @@ from litellm.proxy.hooks.sensitive_data_routing import (
_PROXY_SensitiveDataRoutingHandler,
)
from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup
from litellm.proxy.management_helpers.key_settings_audit import with_settings_updated_at
from litellm.proxy.policy_engine.pipeline_executor import PipelineExecutor
from litellm.repositories.budget_repository import BudgetRepository
from litellm.repositories.config_repository import ConfigRepository
@ -163,23 +164,26 @@ if TYPE_CHECKING:
from mcp.types import CallToolResult
from opentelemetry.trace import Span as _Span
from prisma.client import TransactionManager
from prisma.types import HttpConfig
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.models.team import LiteLLM_TeamTableCachedObj
from litellm.proxy.db.autorouter_session_rollup import AutoRouterTurnTransaction
from litellm.proxy.db.spend_log_tool_index import ToolUsageTransaction
Span = _Span | Any
Span = _Span | object
else:
Span = Any
_T: Final = TypeVar("_T")
unified_guardrail: Final = UnifiedLLMGuardrails()
NON_OPENAI_STREAM_GUARDRAIL_TRANSLATION_CALL_TYPES: "frozenset[CallTypes]" = frozenset({CallTypes.anthropic_messages})
def print_verbose(print_statement):
def print_verbose(print_statement: object):
"""
Prints the given `print_statement` to the console if `litellm.set_verbose` is True.
Also logs the `print_statement` at the debug level using `verbose_proxy_logger`.
@ -227,10 +231,10 @@ class InternalUsageCache:
async def async_get_cache(
self,
key,
key: str,
litellm_parent_otel_span: Span | None,
local_only: bool = False,
**kwargs,
**kwargs: object,
) -> Any:
return await self.dual_cache.async_get_cache(
key=key,
@ -241,11 +245,11 @@ class InternalUsageCache:
async def async_set_cache(
self,
key,
value,
key: str,
value: object,
litellm_parent_otel_span: Span | None,
local_only: bool = False,
**kwargs,
**kwargs: object,
) -> None:
return await self.dual_cache.async_set_cache(
key=key,
@ -257,10 +261,10 @@ class InternalUsageCache:
async def async_batch_set_cache(
self,
cache_list: list,
cache_list: list[tuple[str, object]],
litellm_parent_otel_span: Span | None,
local_only: bool = False,
**kwargs,
**kwargs: object,
) -> None:
return await self.dual_cache.async_set_cache_pipeline(
cache_list=cache_list,
@ -271,19 +275,19 @@ class InternalUsageCache:
async def async_batch_get_cache(
self,
keys: list,
keys: Sequence[str | None],
parent_otel_span: Span | None = None,
local_only: bool = False,
):
return await self.dual_cache.async_batch_get_cache(
keys=keys,
keys=list(keys),
parent_otel_span=parent_otel_span,
local_only=local_only,
)
async def async_increment_cache(
self,
key,
key: str,
value: float,
litellm_parent_otel_span: Span | None,
local_only: bool = False,
@ -299,10 +303,10 @@ class InternalUsageCache:
def set_cache(
self,
key,
value,
key: str,
value: object,
local_only: bool = False,
**kwargs,
**kwargs: object,
) -> None:
return self.dual_cache.set_cache(
key=key,
@ -313,9 +317,9 @@ class InternalUsageCache:
def get_cache(
self,
key,
key: str,
local_only: bool = False,
**kwargs,
**kwargs: object,
) -> Any:
return self.dual_cache.get_cache(
key=key,
@ -338,7 +342,7 @@ def _accepts_litellm_call_info(cb: CustomLogger) -> bool:
return _CALLBACK_ACCEPTS_CALL_INFO[key]
def _enrich_http_exception_with_guardrail_context(exc: BaseException, callback: Any) -> None:
def _enrich_http_exception_with_guardrail_context(exc: BaseException, callback: object) -> None:
"""
If `exc` is an HTTPException with a dict `detail`, mutate it in place to
add `guardrail_name` and `guardrail_mode` taken from the callback instance.
@ -391,7 +395,7 @@ class _CallbackCapabilities:
# Resolved CustomLogger callbacks in original order. Pre-resolving once
# avoids the per-request ``get_custom_logger_compatible_class`` walk for
# every string entry in ``litellm.callbacks``.
resolved_callbacks: tuple[Any, ...] = field(default_factory=tuple)
resolved_callbacks: tuple[object, ...] = field(default_factory=tuple)
class ProxyLogging:
@ -467,7 +471,10 @@ class ProxyLogging:
and not self.daily_report_started
):
asyncio.create_task(
self.slack_alerting_instance._run_scheduled_daily_report(llm_router=llm_router)
self.slack_alerting_instance._run_scheduled_daily_report(
llm_router=llm_router,
pod_lock_manager=self.db_spend_update_writer.pod_lock_manager,
)
) # RUN DAILY REPORT (if scheduled)
self.daily_report_started = True
@ -674,7 +681,7 @@ class ProxyLogging:
return synthetic_data
def _convert_llm_result_to_mcp_response(self, llm_result, request_obj) -> Any | None:
def _convert_llm_result_to_mcp_response(self, llm_result, request_obj) -> MCPPreCallResponseObject | None:
"""
Convert LLM guardrail result back to MCP response format.
"""
@ -800,7 +807,7 @@ class ProxyLogging:
verbose_proxy_logger.error("Error in manual argument parsing: %s", e)
return None
def _convert_llm_result_to_mcp_during_response(self, llm_result, request_obj) -> Any | None:
def _convert_llm_result_to_mcp_during_response(self, llm_result, request_obj) -> MCPDuringCallResponseObject | None:
"""
Convert LLM guardrail result back to MCP during call response format.
"""
@ -846,7 +853,7 @@ class ProxyLogging:
self,
response: MCPPreCallResponseObject,
original_request: MCPPreCallRequestObject,
) -> dict[str, Any]:
) -> Mapping[str, object]:
"""
Parse the response from the pre_mcp_tool_call_hook
@ -949,8 +956,8 @@ class ProxyLogging:
data: dict,
user_api_key_dict: UserAPIKeyAuth | None,
call_type: CallTypesLiteral,
response: Any | None = None,
) -> Any:
response: LLMResponseTypes | None = None,
) -> object:
"""
Execute a single guardrail's hook.
@ -1004,8 +1011,8 @@ class ProxyLogging:
data: dict,
user_api_key_dict: UserAPIKeyAuth | None,
call_type: CallTypesLiteral,
response: Any | None = None,
) -> Any:
response: LLMResponseTypes | None = None,
) -> object:
"""
Execute a guardrail using the router's load balancing.
@ -1140,8 +1147,8 @@ class ProxyLogging:
self,
data: dict,
litellm_logging_obj: Any,
prompt_id: Any,
prompt_version: Any,
prompt_id: str,
prompt_version: int | None,
call_type: CallTypesLiteral,
) -> None:
"""Process prompt template if applicable."""
@ -1362,8 +1369,8 @@ class ProxyLogging:
return None
litellm_logging_obj: Final = cast(Optional["LiteLLMLoggingObj"], data.get("litellm_logging_obj", None))
prompt_id: Final = data.get("prompt_id", None)
prompt_version: Final = data.get("prompt_version", None)
prompt_id: Final[str | None] = data.get("prompt_id", None)
prompt_version: Final[int | None] = data.get("prompt_version", None)
## PROMPT TEMPLATE CHECK ##
@ -1444,7 +1451,7 @@ class ProxyLogging:
if call_type == "call_mcp_tool" and user_api_key_dict is None:
continue
response = await _callback.async_pre_call_hook(
response: Exception | str | Mapping[str, object] | None = await _callback.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=self.call_details["user_api_key_cache"],
data=data,
@ -1612,7 +1619,7 @@ class ProxyLogging:
break
@staticmethod
async def _run_guardrail_with_metrics(callback: Any, coro: Awaitable[Any], hook_type: str) -> Any:
async def _run_guardrail_with_metrics(callback: object, coro: Awaitable[_T], hook_type: str) -> _T:
"""
Await `coro`, recording its latency and status to the
`litellm_guardrail_latency_seconds` metric under `hook_type`, and
@ -1644,8 +1651,8 @@ class ProxyLogging:
@staticmethod
async def _wrap_streaming_iterator_with_enrichment(
callback: Any, gen: AsyncGenerator[Any, None]
) -> AsyncGenerator[Any, None]:
callback: object, gen: AsyncGenerator[_T, None]
) -> AsyncGenerator[_T, None]:
"""
Yield from `gen`; if iteration raises an HTTPException with dict detail,
enrich the detail with the originating callback's `guardrail_name` and
@ -1690,11 +1697,11 @@ class ProxyLogging:
has_guardrail = False
has_pre_call_override = False
iterator_overrides: Final[list[tuple[Any, str]]] = [] # (callback, kind)
resolved_callbacks: Final[list[Any]] = []
resolved_callbacks: Final[list[CustomLogger]] = []
for callback in callbacks:
if isinstance(callback, str):
resolved: Any = litellm.litellm_core_utils.litellm_logging.get_custom_logger_compatible_class(
resolved = litellm.litellm_core_utils.litellm_logging.get_custom_logger_compatible_class(
cast(_custom_logger_compatible_callbacks_literal, callback)
)
else:
@ -2539,7 +2546,7 @@ class ProxyLogging:
self,
data: dict,
user_api_key_dict: UserAPIKeyAuth,
response: Any,
response: object,
request_headers: dict[str, str] | None = None,
) -> dict[str, str]:
"""
@ -2595,7 +2602,7 @@ class ProxyLogging:
return merged_headers
@staticmethod
def _build_litellm_call_info(data: dict, response: Any) -> dict[str, Any]:
def _build_litellm_call_info(data: dict, response: object) -> dict[str, object]:
"""
Build a normalized dict of routing metadata from response._hidden_params
and data, abstracting away the metadata vs litellm_metadata split.
@ -2872,7 +2879,7 @@ _DEPRECATED_KEY_CACHE_TTL_SECONDS: Final = 60
async def _lookup_deprecated_key(
db: Any,
db: PrismaWrapper | RoutingPrismaWrapper,
hashed_token: str,
) -> str | None:
"""
@ -2940,7 +2947,7 @@ def _config_cache_key(param_name: str) -> str:
return f"litellm_config:param:{param_name}"
def _pack_config_row(row: Any) -> dict[str, Any]:
def _pack_config_row(row: Any) -> dict[str, object]:
return {"param_name": row.param_name, "param_value": row.param_value}
@ -2952,7 +2959,7 @@ def _unpack_config_row(cached: Any) -> _ConfigRow | None:
return None
async def get_config_param(prisma_client: Any, param_name: str) -> Any | None:
async def get_config_param(prisma_client: "PrismaClient", param_name: str) -> Any | None:
"""Cached read of a LiteLLM_Config row; returns row, _ConfigRow shim, or None."""
cache_key: Final = _config_cache_key(param_name)
cached: Final = await litellm_config_cache.async_get_cache(cache_key)
@ -2960,7 +2967,7 @@ async def get_config_param(prisma_client: Any, param_name: str) -> Any | None:
return _unpack_config_row(cached)
row: Final = await prisma_client.get_generic_data(key="param_name", value=param_name, table_name="config")
cache_value: Final[Any] = _pack_config_row(row) if row is not None else _CONFIG_CACHE_MISS
cache_value: Final[Mapping[str, object] | str] = _pack_config_row(row) if row is not None else _CONFIG_CACHE_MISS
await litellm_config_cache.async_set_cache(cache_key, cache_value, ttl=LITELLM_CONFIG_CACHE_TTL_SECONDS)
return row
@ -2975,7 +2982,7 @@ async def invalidate_config_param(param_name: str) -> None:
await publish_config_param_change(param_name)
async def prefetch_config_params(prisma_client: Any, param_names: list[str]) -> None:
async def prefetch_config_params(prisma_client: "PrismaClient | None", param_names: list[str]) -> None:
"""Batch-load LiteLLM_Config rows into the cache with one find_many."""
if not param_names:
return
@ -2990,7 +2997,7 @@ async def prefetch_config_params(prisma_client: Any, param_names: list[str]) ->
by_name: Final = {row.param_name: row for row in rows}
for name in param_names:
row = by_name.get(name)
cache_value: Any = _pack_config_row(row) if row is not None else _CONFIG_CACHE_MISS
cache_value: Mapping[str, object] | str = _pack_config_row(row) if row is not None else _CONFIG_CACHE_MISS
await litellm_config_cache.async_set_cache(
_config_cache_key(name), cache_value, ttl=LITELLM_CONFIG_CACHE_TTL_SECONDS
)
@ -3017,7 +3024,7 @@ class PrismaClient:
self,
database_url: str,
proxy_logging_obj: ProxyLogging,
http_client: Any | None = None,
http_client: "HttpConfig | None" = None,
):
## init logging object
self.proxy_logging_obj = proxy_logging_obj
@ -3309,7 +3316,7 @@ class PrismaClient:
async def get_generic_data(
self,
key: str,
value: Any,
value: object,
table_name: Literal["users", "keys", "config", "spend"],
):
"""
@ -3996,7 +4003,7 @@ class PrismaClient:
db_data["token"] = token
response: Final = await VerificationTokenRepository(self).table.update(
where={"token": token},
data={**db_data},
data=with_settings_updated_at(db_data),
)
verbose_proxy_logger.debug("\033[91m" + f"DB Token Table update succeeded {response}" + "\033[0m")
_data: dict = {}
@ -5494,7 +5501,7 @@ class ProxyUpdateSpend:
prisma_client: PrismaClient,
db_writer_client: AsyncHTTPHandler | None,
proxy_logging_obj: ProxyLogging,
logs_to_process: list[dict[str, Any]] | None = None,
logs_to_process: list[dict[str, object]] | None = None,
):
BATCH_SIZE: Final = 1000 # Preferred size of each batch to write to the database
MAX_LOGS_PER_INTERVAL: Final = 10000 # Maximum number of logs to flush in a single interval
@ -6725,7 +6732,7 @@ def model_dump_with_preserved_fields(
obj: Any,
preserve_fields: list[str] | None = None,
exclude_unset: bool = True,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
Serialize a Pydantic model to a dictionary while preserving specific fields
even if they are None.

121
litellm/responses/main.py
View file

@ -1,6 +1,6 @@
import asyncio
import contextvars
from collections.abc import Coroutine, Iterable
from collections.abc import Coroutine, Iterable, Mapping
from functools import partial
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, cast
@ -53,6 +53,7 @@ from litellm.utils import (
)
if TYPE_CHECKING:
from fastapi import WebSocket
from mcp.types import Tool as MCPTool
else:
MCPTool = Any
@ -66,7 +67,7 @@ litellm_completion_transformation_handler: Final = LiteLLMCompletionTransformati
#################################################
def _has_file_search_tool(tools: Any | None) -> bool:
def _has_file_search_tool(tools: Iterable[Mapping[str, object]] | None) -> bool:
"""Return True if any tool in the list has type 'file_search'."""
if not tools:
return False
@ -132,7 +133,7 @@ async def aresponses_api_with_mcp(
instructions: str | None = None,
max_output_tokens: int | None = None,
prompt: PromptObject | None = None,
metadata: dict[str, Any] | None = None,
metadata: dict[str, object] | None = None,
parallel_tool_calls: bool | None = None,
previous_response_id: str | None = None,
reasoning: Reasoning | None = None,
@ -148,9 +149,9 @@ async def aresponses_api_with_mcp(
user: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -397,7 +398,7 @@ async def aresponses(
instructions: str | None = None,
max_output_tokens: int | None = None,
prompt: PromptObject | None = None,
metadata: dict[str, Any] | None = None,
metadata: dict[str, object] | None = None,
parallel_tool_calls: bool | None = None,
previous_response_id: str | None = None,
reasoning: Reasoning | None = None,
@ -416,9 +417,9 @@ async def aresponses(
safety_identifier: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -564,9 +565,9 @@ def _apply_prompt_management_to_responses_call(
custom_llm_provider: str | None,
litellm_logging_obj: LiteLLMLoggingObj | None,
kwargs: dict[str, Any],
local_vars: dict[str, Any],
local_vars: dict[str, object],
) -> tuple[str | ResponseInputParam, str, str | None]:
async_merged: Final = kwargs.pop("_async_prompt_merged_params", None)
async_merged: Final[Mapping[str, object] | None] = kwargs.pop("_async_prompt_merged_params", None)
if async_merged is not None:
for key, value in async_merged.items():
local_vars[key] = value
@ -633,7 +634,7 @@ def _normalize_openai_chat_completions_responses_model(model: str) -> tuple[str,
return f"openai/{remainder}", True
def _pop_use_chat_completions_api_kw(kwargs: dict[str, Any]) -> bool:
def _pop_use_chat_completions_api_kw(kwargs: dict[str, object]) -> bool:
"""Pop use_chat_completions_api; True when the chat-completions bridge is requested."""
use_cc: Final = kwargs.pop("use_chat_completions_api", None)
return bool(use_cc)
@ -643,7 +644,7 @@ def _resolve_model_provider_for_responses(
model: str,
custom_llm_provider: str | None,
litellm_params: GenericLiteLLMParams,
local_vars: dict[str, Any],
local_vars: dict[str, object],
) -> tuple[str, str | None]:
if custom_llm_provider is not None and not litellm_params.custom_llm_provider:
litellm_params.custom_llm_provider = custom_llm_provider
@ -668,7 +669,7 @@ def _apply_managed_file_id_mapping(
input: str | ResponseInputParam,
tools: Iterable[ToolParam] | None,
kwargs: dict[str, Any],
local_vars: dict[str, Any],
local_vars: dict[str, object],
) -> tuple[str | ResponseInputParam, Iterable[ToolParam] | None]:
model_file_id_mapping: Final = kwargs.get("model_file_id_mapping")
model_info_id = kwargs.get("model_info", {}).get("id") if isinstance(kwargs.get("model_info"), dict) else None
@ -706,7 +707,7 @@ def _responses_try_dispatch_mcp_gateway(
instructions: str | None,
max_output_tokens: int | None,
prompt: PromptObject | None,
metadata: dict[str, Any] | None,
metadata: dict[str, object] | None,
parallel_tool_calls: bool | None,
previous_response_id: str | None,
reasoning: Reasoning | None,
@ -719,9 +720,9 @@ def _responses_try_dispatch_mcp_gateway(
top_p: float | None,
truncation: Literal["auto", "disabled"] | None,
user: str | None,
extra_headers: dict[str, Any] | None,
extra_query: dict[str, Any] | None,
extra_body: dict[str, Any] | None,
extra_headers: dict[str, object] | None,
extra_query: dict[str, object] | None,
extra_body: dict[str, object] | None,
timeout: float | httpx.Timeout | None,
custom_llm_provider: str | None,
kwargs: dict[str, Any],
@ -778,7 +779,7 @@ def _responses_try_dispatch_emulated_file_search(
instructions: str | None,
max_output_tokens: int | None,
prompt: PromptObject | None,
metadata: dict[str, Any] | None,
metadata: dict[str, object] | None,
parallel_tool_calls: bool | None,
previous_response_id: str | None,
reasoning: Reasoning | None,
@ -795,14 +796,14 @@ def _responses_try_dispatch_emulated_file_search(
safety_identifier: str | None,
text_format: type[BaseModel] | dict | None,
allowed_openai_params: list[str] | None,
extra_headers: dict[str, Any] | None,
extra_query: dict[str, Any] | None,
extra_body: dict[str, Any] | None,
extra_headers: dict[str, object] | None,
extra_query: dict[str, object] | None,
extra_body: dict[str, object] | None,
timeout: float | httpx.Timeout | None,
custom_llm_provider: str | None,
kwargs: dict[str, Any],
_is_async: bool,
) -> Any | None:
) -> ResponsesAPIResponse | Coroutine[object, object, ResponsesAPIResponse] | None:
"""Return a response when emulated file_search handles the call; otherwise None."""
if not _has_file_search_tool(tools) or not (
responses_api_provider_config is None
@ -864,7 +865,7 @@ def responses(
instructions: str | None = None,
max_output_tokens: int | None = None,
prompt: PromptObject | None = None,
metadata: dict[str, Any] | None = None,
metadata: dict[str, object] | None = None,
parallel_tool_calls: bool | None = None,
previous_response_id: str | None = None,
reasoning: Reasoning | None = None,
@ -883,9 +884,9 @@ def responses(
safety_identifier: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
allowed_openai_params: list[str] | None = None,
@ -1148,9 +1149,9 @@ async def adelete_responses(
response_id: str,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -1209,14 +1210,14 @@ def delete_responses(
response_id: str,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
**kwargs,
) -> DeleteResponseResult | Coroutine[Any, Any, DeleteResponseResult]:
) -> DeleteResponseResult | Coroutine[object, object, DeleteResponseResult]:
"""
Synchronous version of the DELETE Responses API
@ -1299,9 +1300,9 @@ async def aget_responses(
response_id: str,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -1374,14 +1375,14 @@ def get_responses(
response_id: str,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
**kwargs,
) -> ResponsesAPIResponse | Coroutine[Any, Any, ResponsesAPIResponse]:
) -> ResponsesAPIResponse | Coroutine[object, object, ResponsesAPIResponse]:
"""
Fetch a response by its ID.
@ -1481,7 +1482,7 @@ async def alist_input_items(
include: list[str] | None = None,
limit: int = 20,
order: Literal["asc", "desc"] = "desc",
extra_headers: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
custom_llm_provider: str | None = None,
**kwargs,
@ -1537,11 +1538,11 @@ def list_input_items(
include: list[str] | None = None,
limit: int = 20,
order: Literal["asc", "desc"] = "desc",
extra_headers: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
custom_llm_provider: str | None = None,
**kwargs,
) -> dict | Coroutine[Any, Any, dict]:
) -> dict | Coroutine[object, object, dict]:
"""List input items for a response"""
local_vars: Final = locals()
try:
@ -1612,9 +1613,9 @@ async def acancel_responses(
response_id: str,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -1673,14 +1674,14 @@ def cancel_responses(
response_id: str,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
**kwargs,
) -> ResponsesAPIResponse | Coroutine[Any, Any, ResponsesAPIResponse]:
) -> ResponsesAPIResponse | Coroutine[object, object, ResponsesAPIResponse]:
"""
Synchronous version of the POST Responses API
@ -1766,9 +1767,9 @@ async def acompact_responses(
previous_response_id: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
@ -1844,14 +1845,14 @@ def compact_responses(
previous_response_id: str | None = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
timeout: float | httpx.Timeout | None = None,
# LiteLLM specific params,
custom_llm_provider: str | None = None,
**kwargs,
) -> ResponsesAPIResponse | Coroutine[Any, Any, ResponsesAPIResponse]:
) -> ResponsesAPIResponse | Coroutine[object, object, ResponsesAPIResponse]:
"""
Synchronous version of the POST Compact Responses API
@ -1975,7 +1976,7 @@ def _build_litellm_metadata_for_ws(kwargs: dict) -> dict:
@client
async def _aresponses_websocket(
model: str,
websocket: Any,
websocket: "WebSocket",
api_base: str | None = None,
api_key: str | None = None,
timeout: float | None = None,

6
litellm/responses/mcp/chat_completions_handler.py
View file

@ -233,7 +233,7 @@ async def acompletion_with_mcp(
self.follow_up_iterator = None
self.follow_up_exhausted = False
async def __aiter__(self):
def __aiter__(self):
return self
def _add_mcp_list_tools_to_chunk(self, chunk: ModelResponseStream) -> ModelResponseStream:
@ -497,12 +497,12 @@ async def acompletion_with_mcp(
# Create a wrapper class that delegates to our custom iterator
# We'll use a simple approach: just replace the __aiter__ method
class MCPStreamWrapper(CustomStreamWrapper):
def __init__(self, original_wrapper, custom_iterator):
def __init__(self, original_wrapper: CustomStreamWrapper, custom_iterator: MCPStreamingIterator):
# Initialize with the same parameters as original wrapper
super().__init__(
completion_stream=None,
model=getattr(original_wrapper, "model", "unknown"),
logging_obj=getattr(original_wrapper, "logging_obj", None),
logging_obj=original_wrapper.logging_obj,
custom_llm_provider=getattr(original_wrapper, "custom_llm_provider", None),
stream_options=getattr(original_wrapper, "stream_options", None),
make_call=getattr(original_wrapper, "make_call", None),

32
litellm/responses/streaming_iterator.py
View file

@ -9,7 +9,7 @@ from collections.abc import Awaitable, Callable, Mapping
from datetime import datetime
from functools import lru_cache
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal
from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, runtime_checkable
import httpx
from openai._streaming import SSEDecoder
@ -313,8 +313,10 @@ class BaseResponsesAPIStreamingIterator:
if encrypted_content and isinstance(encrypted_content, str):
model_id: Final = _model_id_from_metadata(self.litellm_metadata)
if model_id:
wrapped_content = ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
encrypted_content, model_id
wrapped_content: Final = (
ResponsesAPIRequestUtils._wrap_encrypted_content_with_model_id(
encrypted_content, model_id
)
)
setattr(item, "encrypted_content", wrapped_content)
@ -336,7 +338,9 @@ class BaseResponsesAPIStreamingIterator:
usage_obj: Final[ResponseAPIUsage | None] = getattr(response_obj, "usage", None)
if usage_obj is not None:
try:
cost: float | None = self.logging_obj._response_cost_calculator(result=response_obj)
cost: Final[float | None] = self.logging_obj._response_cost_calculator(
result=response_obj
)
if cost is not None:
setattr(usage_obj, "cost", cost)
except Exception:
@ -1029,6 +1033,16 @@ class CachedResponsesAPIStreamingIterator(BaseResponsesAPIStreamingIterator):
return evt
@runtime_checkable
class _HasModelDump(Protocol):
def model_dump(self, *, exclude_none: bool = ...) -> Mapping[str, object]: ...
@runtime_checkable
class _HasModelDumpJson(Protocol):
def model_dump_json(self, *, exclude_none: bool = ...) -> str: ...
def _dump_response_object(obj: Any) -> dict[str, Any]:
if hasattr(obj, "model_dump"):
return obj.model_dump()
@ -1358,7 +1372,7 @@ class ResponsesWebSocketStreaming:
# response.create frame to prevent deployment-substitution attacks.
self.authorized_model: str | None = authorized_model
def _should_store_event(self, event_obj: dict[str, object]) -> bool:
def _should_store_event(self, event_obj: Mapping[str, object]) -> bool:
return event_obj.get("type") in RESPONSES_WS_LOGGED_EVENT_TYPES
def _store_event(self, event: str | bytes | dict[str, object]) -> None:
@ -1636,7 +1650,7 @@ class ResponsesWebSocketStreaming:
metadata: Final = self.request_data.get("metadata")
raw_pii_tokens: Final = metadata.get("pii_tokens") if _is_json_object(metadata) else None
pii_tokens: Final[dict[str, str]] = raw_pii_tokens if _is_str_mapping(raw_pii_tokens) else {}
pii_tokens: Final[Mapping[str, str]] = raw_pii_tokens if _is_str_mapping(raw_pii_tokens) else {}
if not pii_tokens:
return response_str
@ -1883,11 +1897,11 @@ class ManagedResponsesWebSocketHandler:
def _serialize_chunk(chunk: Any) -> str | None:
"""Serialize a streaming chunk to a JSON string for WebSocket transmission."""
try:
if hasattr(chunk, "model_dump_json"):
if isinstance(chunk, _HasModelDumpJson):
return chunk.model_dump_json(exclude_none=True)
if hasattr(chunk, "model_dump"):
if isinstance(chunk, _HasModelDump):
return json.dumps(chunk.model_dump(exclude_none=True), default=str)
if isinstance(chunk, dict):
if _is_json_object(chunk):
return json.dumps(chunk, default=str)
return json.dumps(str(chunk))
except Exception as exc:

75
litellm/router.py
View file

@ -23,7 +23,7 @@ from collections import defaultdict
from collections.abc import AsyncGenerator, Callable, Generator, Mapping, Sequence
from functools import lru_cache
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypeVar, Union, cast
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypeAlias, TypeVar, Union, cast
import anyio
import httpx
@ -258,6 +258,14 @@ else:
QualityRouter = Any
PreRoutingHookResponse = Any
RouterStrategySelector: TypeAlias = (
LeastBusyLoggingHandler
| LowestCostLoggingHandler
| LowestLatencyLoggingHandler
| LowestTPMLoggingHandler
| LowestTPMLoggingHandler_v2
)
def _cost_value_as_float(value: str | float | None) -> float | None:
if value is None:
@ -402,6 +410,7 @@ class Router:
enable_pre_call_checks: bool = False,
enable_tag_filtering: bool = False,
tag_filtering_match_any: bool = True,
tag_routing_prefix: str = "",
plugins: list[RoutingPlugin] | None = None,
retry_after: int = 0, # min time to wait before retrying a failed request
retry_policy: RetryPolicy | dict | None = None, # set custom retries for different exceptions
@ -511,6 +520,7 @@ class Router:
self.enable_pre_call_checks = enable_pre_call_checks
self.enable_tag_filtering = enable_tag_filtering
self.tag_filtering_match_any = tag_filtering_match_any
self.tag_routing_prefix = tag_routing_prefix
from litellm._service_logger import ServiceLogging
self.service_logger_obj: ServiceLogging = ServiceLogging()
@ -728,7 +738,7 @@ class Router:
routing_strategy_args=routing_strategy_args,
)
self._init_routing_groups(self._routing_groups_input)
self._override_selectors: dict[str, Any] = {}
self._override_selectors: dict[str, RouterStrategySelector | None] = {}
self._override_selectors_lock = threading.Lock()
self.access_groups = None
## USAGE TRACKING ##
@ -921,13 +931,13 @@ class Router:
strategy: RoutingStrategy | str,
routing_strategy_args: dict,
register_callbacks: bool = True,
) -> Any | None:
) -> RouterStrategySelector | None:
"""
Constructs a strategy selector for a given strategy.
Returns None for `simple-shuffle` (no selector needed) and unknown
strategies.
"""
selector: Any | None = None
selector: RouterStrategySelector | None = None
match self._normalize_strategy(strategy):
case RoutingStrategy.LEAST_BUSY.value:
selector = LeastBusyLoggingHandler(router_cache=self.cache)
@ -964,7 +974,7 @@ class Router:
return selector
def _unregister_router_selectors(self, selectors: list[Any]) -> None:
def _unregister_router_selectors(self, selectors: Sequence[object]) -> None:
"""
Drop router-owned strategy selectors from litellm's global callback
lists by identity. Used before re-init (`routing_strategy_init` /
@ -1021,13 +1031,14 @@ class Router:
`"default"` group, whose selectors are the `self.<strategy>_logger`
attributes set up in `routing_strategy_init`.
"""
self._unregister_router_selectors(
[sel for selectors in getattr(self, "_group_selectors", {}).values() for sel in selectors.values()]
group_selectors: Final[Mapping[str, Mapping[str, RouterStrategySelector]]] = getattr(
self, "_group_selectors", {}
)
self._unregister_router_selectors([sel for selectors in group_selectors.values() for sel in selectors.values()])
self._routing_groups: dict[str, RoutingGroup] = {}
self._model_to_group: dict[str, str] = {}
self._group_selectors: dict[str, dict[str, Any]] = {}
self._group_selectors: dict[str, dict[str, RouterStrategySelector]] = {}
if not groups_input:
return
@ -1105,7 +1116,7 @@ class Router:
return None
return strategy
def _get_override_strategy_selector(self, strategy: str) -> Any | None:
def _get_override_strategy_selector(self, strategy: str) -> RouterStrategySelector | None:
"""
Returns the selector for a per-request strategy override.
@ -1126,7 +1137,9 @@ class Router:
)
return self._override_selectors[strategy]
def _get_routing_context(self, model: str, request_kwargs: dict | None = None) -> tuple[str | None, Any | None]:
def _get_routing_context(
self, model: str, request_kwargs: dict | None = None
) -> tuple[str | None, RouterStrategySelector | None]:
"""
Resolves the routing strategy and selector to use for the given model.
@ -1949,7 +1962,7 @@ class Router:
return silent_kwargs
def _silent_experiment_completion(self, silent_model: str, messages: list[Any], **kwargs):
def _silent_experiment_completion(self, silent_model: str, messages: Sequence[Mapping[str, str]], **kwargs):
"""
Run a silent experiment in the background (thread).
"""
@ -2299,7 +2312,7 @@ class Router:
# in __init__ rather than declaring it as a class field, so
# static narrowing doesn't expose it. Mirror the sync path
# (_completion_streaming_iterator) and pull via getattr.
chat: Final = getattr(built, "usage", None) if built is not None else None
chat: Final[object | None] = getattr(built, "usage", None) if built is not None else None
if chat is not None:
# getattr-with-default because the test path may
# substitute a SimpleNamespace lacking some fields;
@ -2393,7 +2406,7 @@ class Router:
# ResponseOutputMessageParam, ...) — annotating as List[Dict[str, Any]]
# rejects the list() spread of input_val. We cast the combined list to
# ResponseInputParam at the return.
base: list[Any]
base: list[object]
if isinstance(input_val, str):
base = [
{
@ -2406,7 +2419,7 @@ class Router:
base = list(input_val)
else:
base = []
continuation: Final[list[Any]] = [
continuation: Final[list[object]] = [
{
"type": "message",
"role": "developer",
@ -2783,7 +2796,7 @@ class Router:
return SyncFallbackStreamWrapper(stream_with_fallbacks())
async def _silent_experiment_acompletion(self, silent_model: str, messages: list[Any], **kwargs):
async def _silent_experiment_acompletion(self, silent_model: str, messages: Sequence[Mapping[str, str]], **kwargs):
"""
Run a silent experiment in the background.
"""
@ -3039,7 +3052,7 @@ class Router:
pass
def _stamp_failed_deployment_id_with_effective_model_info(
self, exception: Exception, deployment: Mapping[str, Any], kwargs: Mapping[str, Any]
self, exception: Exception, deployment: Mapping[str, object], kwargs: Mapping[str, object]
) -> None:
# A client-side-credential call gets a dynamic deployment id generated inside
# _update_kwargs_with_deployment and stamped into kwargs["model_info"]; stamping
@ -3562,8 +3575,8 @@ class Router:
model: str,
priority: int,
original_function: Callable,
args: tuple[Any, ...],
kwargs: dict[str, Any],
args: tuple[object, ...],
kwargs: dict[str, object],
):
parent_otel_span: Final = _get_parent_otel_span_from_kwargs(kwargs)
### FLOW ITEM ###
@ -4651,7 +4664,7 @@ class Router:
# fallback to the original reference for any non-picklable value.
# The original_generic_function is preserved so the per-attempt
# helper knows which underlying API to call on fallback.
fallback_kwargs: Final[dict[str, Any]] = kwargs.copy()
fallback_kwargs: Final[dict[str, object]] = kwargs.copy()
if isinstance(fallback_kwargs.get("litellm_metadata"), dict):
fallback_kwargs["litellm_metadata"] = safe_deep_copy(fallback_kwargs["litellm_metadata"])
if isinstance(fallback_kwargs.get("metadata"), dict):
@ -5698,7 +5711,7 @@ class Router:
def sync_wrapper(
custom_llm_provider: str | None = None,
client: Any | None = None,
client: object | None = None,
**kwargs,
):
return self._generic_api_call_with_fallbacks(original_function=original_function, **kwargs)
@ -5714,7 +5727,7 @@ class Router:
def vector_store_sync_wrapper(
custom_llm_provider: str | None = None,
client: Any | None = None,
client: object | None = None,
**kwargs,
):
if custom_llm_provider and "custom_llm_provider" not in kwargs:
@ -5736,7 +5749,7 @@ class Router:
def vector_store_file_sync_wrapper(
custom_llm_provider: str | None = None,
client: Any | None = None,
client: object | None = None,
**kwargs,
):
return original_function(
@ -5757,7 +5770,7 @@ class Router:
def managed_agents_sync_wrapper(
custom_llm_provider: str | None = None,
client: Any | None = None,
client: object | None = None,
**kwargs,
):
if custom_llm_provider and "custom_llm_provider" not in kwargs:
@ -7142,7 +7155,9 @@ class Router:
except Exception as e:
raise e
async def async_deployment_callback_on_failure(self, kwargs, completion_response: Any | None, start_time, end_time):
async def async_deployment_callback_on_failure(
self, kwargs, completion_response: object | None, start_time, end_time
):
"""
Update RPM usage for a deployment
"""
@ -7843,7 +7858,7 @@ class Router:
continue
if self._has_registered_strategy(self.adaptive_routers, model_name, tagged.tags):
continue
adaptive_router = complexity_router._ensure_adaptive_router()
adaptive_router: AdaptiveRouter | None = complexity_router._ensure_adaptive_router()
if adaptive_router is not None:
self.adaptive_routers[model_name] = [
*self.adaptive_routers.get(model_name, []),
@ -8131,7 +8146,7 @@ class Router:
self.provider_default_deployment_ids.append(deployment.model_info.id)
_team_id: Final = deployment.model_info.get("team_id")
_team_public_model_name: Final = deployment.model_info.get("team_public_model_name")
_team_public_model_name: Final[str | None] = deployment.model_info.get("team_public_model_name")
if _team_id is not None and _team_public_model_name is not None and "*" in _team_public_model_name:
if _team_id not in self.team_pattern_routers:
self.team_pattern_routers[_team_id] = PatternMatchRouter()
@ -9485,7 +9500,7 @@ class Router:
async def set_response_headers(
self,
response: Any,
response: object,
model_group: str | None = None,
request_kwargs: dict | None = None,
) -> Any:
@ -10109,6 +10124,7 @@ class Router:
"model_group_alias",
"enable_weighted_failover",
"enable_tag_filtering",
"tag_routing_prefix",
]
for var in vars_to_include:
@ -10146,6 +10162,7 @@ class Router:
"model_group_alias",
"enable_weighted_failover",
"enable_tag_filtering",
"tag_routing_prefix",
]
_int_settings: Final = [
@ -11318,7 +11335,7 @@ class Router:
@staticmethod
def _redact_prompt_text_if_needed(
request_kwargs: Mapping[str, Any],
request_kwargs: Mapping[str, object],
routing_decision: StandardLoggingRoutingDecision,
) -> StandardLoggingRoutingDecision:
"""Drop verbatim prompt text from the record when message logging is redacted.
@ -11680,7 +11697,7 @@ class Router:
flag. Used by credential-lookup helpers so passthrough file / batch endpoints
cannot bypass the pause by resolving credentials directly.
"""
model_info: Final = getattr(deployment, "model_info", None)
model_info: Final[object | None] = getattr(deployment, "model_info", None)
if model_info is None:
return False
return getattr(model_info, "blocked", None) is True

376
litellm/router_strategy/tag_based_routing.py
View file

@ -4,9 +4,12 @@ Use this to route requests between Teams
- If tags in request is a subset of tags in deployment, return deployment
- if deployments are set with default tags, return all default deployment
- If no default_deployments are set, return all deployments
- A "!tag" excludes deployments carrying that tag; a "&tag" requires it
"""
import re
from collections.abc import Mapping, Sequence
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal
from litellm._logging import verbose_logger
@ -114,14 +117,52 @@ def _match_deployment(
return None
def _split_tags(tags: list[str]) -> tuple[list[str], list[str]]:
positive: Final = [t for t in tags if not t.startswith("!")]
excluded: Final = [tag[1:] for tag in tags if tag.startswith("!") and len(tag) > 1]
return positive, excluded
def _bare_tag_value(tag: str) -> str | None:
# Mirrors _split_tags' own stripping rule exactly, so a confirmed value
# compares equal to whatever required_set/excluded_set/positive_tags end up
# holding for the same tag: a "&"/"!" marker is stripped only when something
# follows it; a lone marker with nothing after it parses to nothing in any
# of the three sets, so it must not become a confirmed value either.
if tag.startswith(("&", "!")):
return tag[1:] if len(tag) > 1 else None
return tag
def _strip_routing_prefix(tags: Sequence[str], prefix: str) -> tuple[tuple[str, ...], frozenset[str]]:
# Strips the configured routing-prefix marker from any tag carrying it, used
# exactly as configured with no delimiter auto-appended, and separately
# tracks the post-strip, post-marker-strip values that arrived prefixed: tags
# whose routing intent the caller declared explicitly, exempt from the "maybe
# foreign to this group" heuristics in _unknown_required_tag_hides_an_answer
# and _tag_known_to_group below. Confirmed values are compared against
# required_set/excluded_set downstream, which are themselves already stripped
# of their "&"/"!" marker by _split_tags -- confirmed must match that same
# bare form, not the raw post-prefix-strip value that still carries the
# marker character. An empty prefix must return every tag unconfirmed, not
# run every tag through str.startswith(""), which is trivially True for
# every string and would mark everything confirmed.
if not prefix:
return tuple(tags), frozenset()
rewritten: Final = tuple(t.removeprefix(prefix) for t in tags)
confirmed: Final = frozenset(
bare
for bare in (_bare_tag_value(t.removeprefix(prefix)) for t in tags if t.startswith(prefix))
if bare is not None
)
return rewritten, confirmed
def _split_tags(tags: Sequence[str]) -> tuple[tuple[str, ...], list[str], tuple[str, ...]]:
required: Final = tuple(tag[1:] for tag in tags if tag.startswith("&") and len(tag) > 1)
positive: Final = [
t for t in tags if not t.startswith("!") and not t.startswith("&")
] # mutable-ok: feeds _match_deployment's existing list[str]-typed request_tags param
excluded: Final = tuple(tag[1:] for tag in tags if tag.startswith("!") and len(tag) > 1)
return required, positive, excluded
def _exclude_deployments(
deployments: list[Any] | dict[Any, Any],
deployments: Sequence[Any] | Mapping[Any, Any],
excluded_set: frozenset[str],
) -> list[Any]:
if not excluded_set:
@ -129,24 +170,223 @@ def _exclude_deployments(
return [d for d in deployments if not excluded_set.intersection(d.get("litellm_params", {}).get("tags") or [])]
def _require_candidates(
candidates: list[Any],
def _require_all_tags(
deployments: Sequence[Any] | Mapping[Any, Any],
required_set: frozenset[str],
) -> tuple[Any, ...]:
if not required_set:
return tuple(deployments)
return tuple(d for d in deployments if required_set.issubset(d.get("litellm_params", {}).get("tags") or []))
def _default_tagged_pool(
deployments: Sequence[Any] | Mapping[Any, Any],
) -> tuple[Any, ...]:
defaults: Final = tuple(d for d in deployments if "default" in (d.get("litellm_params", {}).get("tags") or []))
return defaults if defaults else tuple(deployments)
def _known_tag_values(deployments: Sequence[Any] | Mapping[Any, Any]) -> frozenset[str]:
return frozenset(
tag for d in deployments for tag in (d.get("litellm_params", MappingProxyType({})).get("tags") or ())
)
def _unknown_required_tag_hides_an_answer(
healthy_deployments: Sequence[Any] | Mapping[Any, Any],
excluded_set: frozenset[str],
required_set: frozenset[str],
routing_confirmed: frozenset[str],
) -> bool:
# A caller-invented "&" tag (one no deployment in this group has ever carried)
# guarantees an empty required-AND result on its own, regardless of whether the
# rest of the request's required tags were satisfiable. Dropping the unknown
# tags and recomputing: if that reveals a specific, non-empty answer, the invented
# tag was the actual cause of the exhaustion, and fail-open must not paper over
# it. If every required tag is known, or none are, there's nothing hidden to
# protect: either the caller made a real, honestly-unsatisfiable ask (fail-open
# proceeds normally), or the whole required set is unrecognized noise with no
# narrower answer to hide behind it. routing_confirmed (tag_routing_prefix)
# counts as known too: the caller explicitly declared it a routing directive,
# so it is never treated as invented noise regardless of deployment vocabulary.
known_required: Final = required_set & (_known_tag_values(healthy_deployments) | routing_confirmed)
if not known_required or known_required == required_set:
return False
allowed: Final = _exclude_deployments(healthy_deployments, excluded_set)
return bool(_require_all_tags(allowed, known_required))
def _chain_allows_fail_open(
healthy_deployments: Sequence[Any] | Mapping[Any, Any],
excluded_set: frozenset[str],
required_set: frozenset[str],
routing_confirmed: frozenset[str],
) -> bool:
if _unknown_required_tag_hides_an_answer(healthy_deployments, excluded_set, required_set, routing_confirmed):
return False
return any((d.get("model_info") or {}).get("allow_fail_open") is True for d in healthy_deployments)
def _trusted_only_pool(
healthy_deployments: Sequence[Any] | Mapping[Any, Any],
excluded_set: frozenset[str],
required_set: frozenset[str],
inherited_excluded_set: frozenset[str] | None,
inherited_required_set: frozenset[str] | None,
) -> tuple[Any, ...]:
# inherited_*_set is None only when this request carries no origin information
# at all (e.g. direct SDK Router usage, bypassing the proxy layer that
# populates metadata.inherited_tags) -- treat every constraint as
# caller-controlled in that case (protected == empty), reproducing this
# function's pre-provenance behavior exactly: an unconditional fall-open to the
# full default-tagged pool, constraints discarded entirely. Otherwise, a tag
# value is protected the moment it has ANY inherited backing, even when the
# caller also happens to submit the identical value themselves -- set
# membership can't distinguish "this value came from policy" from "this value
# coincidentally matches policy," so presence in the inherited set (not
# absence from a caller-supplied set) is what must gate discardability. This
# is deliberately intersection with inherited_*_set, not subtraction of a
# caller-supplied set: subtraction would let a caller strip an inherited
# requirement's protection just by resubmitting its exact value alongside a
# conflicting one (e.g. inherited "&region:eu" plus caller "&region:eu"
# and "!region:eu" would otherwise cancel the inherited requirement out).
trusted_excluded: Final = (
frozenset[str]() if inherited_excluded_set is None else inherited_excluded_set & excluded_set
)
trusted_required: Final = (
frozenset[str]() if inherited_required_set is None else inherited_required_set & required_set
)
return _require_all_tags(_exclude_deployments(healthy_deployments, trusted_excluded), trusted_required)
def _resolve_or_fail_open(
pool: Sequence[Any],
healthy_deployments: Sequence[Any] | Mapping[Any, Any],
excluded_set: frozenset[str],
required_set: frozenset[str],
inherited_excluded_set: frozenset[str] | None,
inherited_required_set: frozenset[str] | None,
routing_confirmed: frozenset[str],
model: str,
request_tags: Any,
) -> list[Any]:
if not candidates:
raise ValueError(
f"{RouterErrors.no_deployments_with_tag_routing.value}. Passed model={model} and tags={request_tags}"
request_tags: object,
) -> tuple[Any, ...]:
if pool:
return tuple(pool)
if _chain_allows_fail_open(healthy_deployments, excluded_set, required_set, routing_confirmed):
# Fall open only within whatever still satisfies whichever constraints
# trace back to key/team policy. A constraint with no inherited backing at
# all (or, when inherited_tags is unavailable, any constraint at all) can
# be discarded; one inherited from key/team policy cannot -- if that alone
# is unsatisfiable, raise instead of silently routing around it.
trusted_pool: Final = _trusted_only_pool(
healthy_deployments, excluded_set, required_set, inherited_excluded_set, inherited_required_set
)
return candidates
if trusted_pool:
return _default_tagged_pool(trusted_pool)
raise ValueError(
f"{RouterErrors.no_deployments_with_tag_routing.value}. Passed model={model} and tags={request_tags}"
)
def _ban_only_base_pool(
deployments: list[Any] | dict[Any, Any],
) -> list[Any]:
# Mirrors untagged-request semantics so callers can't use !tags to escape the default pool.
defaults: Final = [d for d in deployments if "default" in (d.get("litellm_params", {}).get("tags") or [])]
return defaults if defaults else list(deployments)
def _resolve_constraint_only_pool(
healthy_deployments: Sequence[Any] | Mapping[Any, Any],
excluded_set: frozenset[str],
required_set: frozenset[str],
inherited_excluded_set: frozenset[str] | None,
inherited_required_set: frozenset[str] | None,
routing_confirmed: frozenset[str],
model: str,
request_tags: object,
) -> tuple[Any, ...]:
pool: Final = (
_require_all_tags(_exclude_deployments(healthy_deployments, excluded_set), required_set)
if required_set
else _exclude_deployments(_default_tagged_pool(healthy_deployments), excluded_set)
)
return _resolve_or_fail_open(
pool,
healthy_deployments,
excluded_set,
required_set,
inherited_excluded_set,
inherited_required_set,
routing_confirmed,
model,
request_tags,
)
def _all_deployments_or_fallback(
llm_router_instance: LitellmRouter,
model: str,
fallback: Sequence[Any] | Mapping[Any, Any],
) -> Sequence[Any] | Mapping[Any, Any]:
try:
return llm_router_instance._get_all_deployments(model_name=model)
except Exception: # noqa: BLE001 # fail safe toward today's healthy-only behavior on lookup errors
return fallback
def _chain_tag_filtering_override(
llm_router_instance: LitellmRouter,
model: str,
healthy_deployments: Sequence[Any] | Mapping[Any, Any],
) -> bool | None:
# Resolved from every deployment configured for this model group, not just the
# ones that survived cooldown/health filtering (async_get_healthy_deployments
# filters cooldowns before calling get_deployments_for_tag) -- otherwise the
# sole deployment carrying this group's only explicit override loses its effect
# the moment it's transiently unhealthy, silently falling back to the
# router-wide default and letting an attacker disable a chain's tag policy by
# repeatedly failing that one deployment into cooldown. Falls back to
# healthy_deployments on a lookup error, preserving today's behavior rather
# than crashing the request.
all_deployments: Final = _all_deployments_or_fallback(llm_router_instance, model, healthy_deployments)
for d in all_deployments:
value = (d.get("model_info") or MappingProxyType({})).get("enable_tag_filtering")
if value is not None:
return value
return None
def _inherited_constraint_sets(
inherited_tags: object, routing_prefix: str
) -> tuple[frozenset[str] | None, frozenset[str] | None]:
# None means no origin information is available at all (e.g. this request
# bypassed the proxy layer that populates metadata.inherited_tags, as direct
# SDK Router usage does) -- callers of this must treat that as "nothing is
# protected," not "nothing is inherited," see _trusted_only_pool.
# metadata.inherited_tags is a snapshot of whatever key/team/project policy
# merged into "tags" *before* this request's own caller-supplied tags were
# merged in on top (see litellm_pre_call_utils.py), so a value present here is
# policy-backed regardless of whether the caller also happens to submit the
# identical value. inherited_tags is stripped through the same routing_prefix
# as the main request tags so a policy-inherited prefixed tag still matches
# correctly against the (already-stripped) required_set/excluded_set computed
# from request_tags.
if not isinstance(inherited_tags, (list, tuple)):
return None, None
rewritten_inherited_tags: Final = _strip_routing_prefix(inherited_tags, routing_prefix)[0]
inherited_required, _inherited_positive, inherited_excluded = _split_tags(rewritten_inherited_tags)
return frozenset(inherited_required), frozenset(inherited_excluded)
def _tag_known_to_group(
llm_router_instance: LitellmRouter,
model: str,
positive_tags: Sequence[str],
routing_confirmed: frozenset[str],
) -> bool:
tag_set: Final = frozenset(positive_tags)
if tag_set & routing_confirmed:
return True
try:
all_deployments: Final = llm_router_instance._get_all_deployments(model_name=model)
except Exception: # noqa: BLE001 # fail safe toward "unrecognized" so lookup errors preserve the existing silent-fallback behavior
return False
return any(
tag_set.intersection(d.get("litellm_params", MappingProxyType({})).get("tags") or ()) for d in all_deployments
)
async def get_deployments_for_tag(
@ -161,24 +401,29 @@ async def get_deployments_for_tag(
Executes tag based filtering based on the tags in request metadata and the tags on the deployments
Runs when the router-level `enable_tag_filtering` is True or the request carries
`enable_tag_filtering=True` (set from key/team router_settings by the proxy).
A request-level False never disables a router-level True, so per-request settings
cannot escape an operator's global tag-routing policy.
Runs when the effective enable_tag_filtering is True. Effective value: a
request-level enable_tag_filtering=True (set from key/team router_settings by
the proxy) always wins; otherwise model_info.enable_tag_filtering on this model
group, if set on any of its deployments, overrides the router-wide default.
A request-level False never disables either of those, so per-request settings
cannot escape an operator's or a chain owner's tag-routing policy.
"""
request_enable_tag_filtering: Final = request_kwargs.get("enable_tag_filtering") if request_kwargs else None
if request_enable_tag_filtering is not True and llm_router_instance.enable_tag_filtering is not True:
return healthy_deployments
if request_kwargs is None:
if request_kwargs is None or not healthy_deployments:
verbose_logger.debug(
"get_deployments_for_tag: request_kwargs is None returning healthy_deployments: %s",
"get_deployments_for_tag: skipping tag filter (request_kwargs=%s, healthy_deployments=%s)",
request_kwargs,
healthy_deployments,
)
return healthy_deployments
if not healthy_deployments:
verbose_logger.debug("get_deployments_for_tag: empty or None healthy_deployments; skipping tag filter")
request_enable_tag_filtering: Final = request_kwargs.get("enable_tag_filtering")
chain_enable_tag_filtering: Final = _chain_tag_filtering_override(llm_router_instance, model, healthy_deployments)
chain_default: Final = (
chain_enable_tag_filtering
if chain_enable_tag_filtering is not None
else llm_router_instance.enable_tag_filtering
)
if request_enable_tag_filtering is not True and chain_default is not True:
return healthy_deployments
verbose_logger.debug("request metadata: %s", request_kwargs.get(metadata_variable_name))
@ -186,29 +431,52 @@ async def get_deployments_for_tag(
metadata: Final = request_kwargs[metadata_variable_name]
request_tags: Final = metadata.get("tags")
match_any: Final = llm_router_instance.tag_filtering_match_any
routing_prefix: Final = llm_router_instance.tag_routing_prefix or ""
# Build header strings for regex matching from what the proxy already stores.
# Currently we match against User-Agent; format matches "^User-Agent: claude-code/..."
user_agent: Final = metadata.get("user_agent", "")
header_strings: Final[list[str]] = [f"User-Agent: {user_agent}"] if user_agent else []
positive_tags, excluded_patterns = _split_tags(request_tags or [])
# A tag_routing_prefix-marked tag is stripped before matching -- everything
# downstream (_split_tags, deployment matching) works off the unprefixed
# value, exactly as if the caller had sent it unprefixed -- and its
# post-strip value is remembered in routing_confirmed as an explicit,
# caller-declared routing directive, exempt from the "maybe foreign to this
# group" heuristics that unprefixed tags still go through unchanged below.
rewritten_tags, routing_confirmed = _strip_routing_prefix(request_tags or [], routing_prefix)
required_tags, positive_tags, excluded_patterns = _split_tags(rewritten_tags)
inherited_required_set, inherited_excluded_set = _inherited_constraint_sets(
metadata.get("inherited_tags"), routing_prefix
)
excluded_set: Final = frozenset(excluded_patterns)
candidates: Final = _exclude_deployments(healthy_deployments, excluded_set)
required_set: Final = frozenset(required_tags)
allowed_deployments: Final = _exclude_deployments(healthy_deployments, excluded_set)
candidates: Final = _require_all_tags(allowed_deployments, required_set)
has_regex_deployments: Final = any(d.get("litellm_params", {}).get("tag_regex") for d in candidates)
has_tag_filter: Final = bool(positive_tags) or (bool(header_strings) and has_regex_deployments)
ban_only: Final = bool(excluded_set) and not has_tag_filter
has_positive_filter: Final = bool(positive_tags) or (
bool(header_strings) and has_regex_deployments and not required_set
)
constraint_only: Final = (bool(excluded_set) or bool(required_set)) and not has_positive_filter
if ban_only:
pool: Final = _exclude_deployments(_ban_only_base_pool(healthy_deployments), excluded_set)
return _require_candidates(pool, model, request_tags)
if constraint_only:
return _resolve_constraint_only_pool(
healthy_deployments,
excluded_set,
required_set,
inherited_excluded_set,
inherited_required_set,
routing_confirmed,
model,
request_tags,
)
new_healthy_deployments: Final[list[Any]] = []
default_deployments: Final[list[Any]] = []
if has_tag_filter:
if has_positive_filter:
verbose_logger.debug(
"get_deployments_for_tag routing: request_tags=%s user_agent=%s",
request_tags,
@ -245,9 +513,33 @@ async def get_deployments_for_tag(
default_deployments.append(deployment)
if len(new_healthy_deployments) == 0 and len(default_deployments) == 0:
raise ValueError(
f"{RouterErrors.no_deployments_with_tag_routing.value}."
f" Passed model={model} and tags={request_tags}"
return _resolve_or_fail_open(
(),
healthy_deployments,
excluded_set,
required_set,
inherited_excluded_set,
inherited_required_set,
routing_confirmed,
model,
request_tags,
)
if (
len(new_healthy_deployments) == 0
and positive_tags
and _tag_known_to_group(llm_router_instance, model, positive_tags, routing_confirmed)
):
return _resolve_or_fail_open(
(),
healthy_deployments,
excluded_set,
required_set,
inherited_excluded_set,
inherited_required_set,
routing_confirmed,
model,
request_tags,
)
return new_healthy_deployments if len(new_healthy_deployments) > 0 else default_deployments

2
litellm/types/completion.py
View file

@ -217,7 +217,7 @@ class _CompletionDispatchContext:
headers: dict
hf_model_name: str | None
kwargs: dict
litellm_params: dict
litellm_params: dict[str, object]
logger_fn: Callable | None
logging: LiteLLMLoggingObj
max_retries: int | None

1
litellm/types/llms/openai.py
View file

@ -929,6 +929,7 @@ class ChatCompletionRequest(TypedDict, total=False):
user: str
metadata: dict # litellm specific param
reasoning_effort: str # OpenAI o1/o3 reasoning parameter
output_config: Mapping[str, object] # Anthropic adaptive-thinking effort, bridged for Bedrock Claude
class ChatCompletionDeltaChunk(TypedDict, total=False):

15
litellm/types/router.py
View file

@ -123,6 +123,7 @@ class UpdateRouterConfig(BaseModel):
context_window_fallbacks: list[dict] | None = None
model_group_alias: dict[str, str | dict] | None = {}
enable_tag_filtering: bool | None = None
tag_routing_prefix: str | None = None
model_config = ConfigDict(protected_namespaces=())
@ -170,6 +171,20 @@ class ModelInfo(MirroredPricingParams):
ptu_effective_from: datetime.datetime | None = None
ptu_effective_to: datetime.datetime | None = None
# when tag-based routing's "!" or "&" constraints eliminate every deployment
# in this model group, fall back to the default-tagged pool instead of
# raising no_deployments_with_tag_routing. Defaults to False (raise), so
# existing "!" negation behavior is unchanged unless explicitly opted in.
allow_fail_open: bool | None = None
# per-model-group override for router_settings.enable_tag_filtering; unset
# defers to the router-wide default. Checked against any deployment in the
# group, so set it consistently across every deployment sharing this
# model_name. A request-level enable_tag_filtering=True (from key/team
# settings) still wins over this, exactly as it already does over the
# router-wide default.
enable_tag_filtering: bool | None = None
def __init__(self, id: str | int | None = None, **params) -> None:
if id is None:
id = str(uuid.uuid4()) # Generate a UUID if id is None or not provided

2
litellm/types/utils.py
View file

@ -152,6 +152,7 @@ class ProviderSpecificModelInfo(TypedDict, total=False):
supports_web_search: bool | None
supports_reasoning: bool | None
supports_adaptive_thinking: bool | None
supports_tool_search: bool | None
supports_mid_conversation_system: bool | None
supports_url_context: bool | None
supports_none_reasoning_effort: bool | None
@ -3467,6 +3468,7 @@ all_litellm_params = (
"caching_groups",
"ttl",
"cache",
"enable_prompt_caching",
"no-log",
"base_model",
"stream_timeout",

124
litellm/utils.py
View file

@ -234,9 +234,11 @@ except (ImportError, AttributeError, TypeError):
# Convert to str (if necessary)
claude_json_str = json.dumps(json_data)
import importlib.metadata
from collections.abc import Callable, Iterable, Mapping
from collections.abc import Callable, Iterable, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast, get_args
from litellm import utils as litellm_utils
# These are lazy loaded via __getattr__
from litellm.llms.base_llm.base_utils import (
BaseLLMModelInfo,
@ -263,6 +265,7 @@ if TYPE_CHECKING:
map_finish_reason,
process_response_headers,
)
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
from litellm.litellm_core_utils.dot_notation_indexing import (
delete_nested_value,
is_nested_path,
@ -351,6 +354,24 @@ if TYPE_CHECKING:
)
from litellm.llms.base_llm.videos.transformation import BaseVideoConfig
from litellm.llms.bedrock.common_utils import BedrockModelInfo
from litellm.llms.bedrock.embed.amazon_nova_transformation import (
AmazonNovaEmbeddingConfig,
)
from litellm.llms.bedrock.embed.amazon_titan_g1_transformation import (
AmazonTitanG1Config,
)
from litellm.llms.bedrock.embed.amazon_titan_multimodal_transformation import (
AmazonTitanMultimodalEmbeddingG1Config,
)
from litellm.llms.bedrock.embed.amazon_titan_v2_transformation import (
AmazonTitanV2Config,
)
from litellm.llms.bedrock.embed.cohere_transformation import (
BedrockCohereEmbeddingConfig,
)
from litellm.llms.bedrock.embed.twelvelabs_marengo_transformation import (
TwelveLabsMarengoEmbeddingConfig,
)
from litellm.llms.cohere.common_utils import CohereModelInfo
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.mistral.ocr.transformation import MistralOCRConfig
@ -574,7 +595,7 @@ def get_request_guardrails(kwargs: dict[str, Any]) -> list[str]:
return applied_guardrails
def get_applied_guardrails(kwargs: dict[str, Any]) -> list[str]:
def get_applied_guardrails(kwargs: dict[str, object]) -> list[str]:
"""
- Add 'default_on' guardrails to the list
- Add request guardrails to the list
@ -601,7 +622,7 @@ def load_credentials_from_list(kwargs: dict):
credential_name: Final = kwargs.get("litellm_credential_name")
if credential_name and litellm.credential_list:
credential_accessor: Final = CredentialAccessor.get_credential_values(credential_name)
credential_accessor: Final[Mapping[str, object]] = CredentialAccessor.get_credential_values(credential_name)
for key, value in credential_accessor.items():
if key not in kwargs:
kwargs[key] = value
@ -789,7 +810,7 @@ def function_setup(
function_id: Final[str | None] = kwargs["id"] if "id" in kwargs else None
## LAZY LOAD COROUTINE CHECKER ##
get_coroutine_checker_fn: Final = getattr(sys.modules[__name__], "get_coroutine_checker")
get_coroutine_checker_fn: Final = litellm_utils.get_coroutine_checker
coroutine_checker: Final = get_coroutine_checker_fn()
## DYNAMIC CALLBACKS ##
@ -925,7 +946,7 @@ def function_setup(
elif kwargs.get("messages", None):
messages = kwargs["messages"]
### PRE-CALL RULES ###
Rules: Final = getattr(sys.modules[__name__], "Rules")
Rules: Final = litellm_utils.Rules
if (
Rules.has_pre_call_rules()
and isinstance(messages, list)
@ -1033,7 +1054,7 @@ def function_setup(
)
contents_param: Final = args[1] if len(args) > 1 else kwargs.get("contents")
model_param: Final = args[0] if len(args) > 0 else kwargs.get("model", "")
model_param: Final[str] = args[0] if len(args) > 0 else kwargs.get("model", "")
if contents_param:
adapter: Final = GoogleGenAIAdapter()
@ -1078,7 +1099,7 @@ def function_setup(
)
## check if metadata is passed in
litellm_params: Final[dict[str, Any]] = {"api_base": ""}
litellm_params: Final[dict[str, object]] = {"api_base": ""}
if "metadata" in kwargs:
litellm_params["metadata"] = kwargs["metadata"]
if "litellm_metadata" in kwargs and isinstance(kwargs["litellm_metadata"], dict):
@ -1154,9 +1175,11 @@ def _get_wrapper_num_retries(kwargs: dict[str, Any], exception: Exception) -> tu
if num_retries is None:
num_retries = litellm.num_retries
if kwargs.get("retry_policy", None):
get_num_retries_from_retry_policy: Final = getattr(sys.modules[__name__], "get_num_retries_from_retry_policy")
reset_retry_policy: Final = getattr(sys.modules[__name__], "reset_retry_policy")
retry_policy_num_retries: Final = get_num_retries_from_retry_policy(
get_num_retries_from_retry_policy: Final[Callable[..., int | None]] = getattr(
sys.modules[__name__], "get_num_retries_from_retry_policy"
)
reset_retry_policy: Final = litellm_utils.reset_retry_policy
retry_policy_num_retries: Final[int | None] = get_num_retries_from_retry_policy(
exception=exception,
retry_policy=kwargs.get("retry_policy"),
)
@ -1167,7 +1190,7 @@ def _get_wrapper_num_retries(kwargs: dict[str, Any], exception: Exception) -> tu
return num_retries, kwargs
def _get_wrapper_timeout(kwargs: dict[str, Any], exception: Exception) -> float | int | httpx.Timeout | None:
def _get_wrapper_timeout(kwargs: dict[str, object], exception: Exception) -> float | int | httpx.Timeout | None:
"""
Get the timeout from the kwargs
Used for the wrapper functions.
@ -1179,7 +1202,7 @@ def _get_wrapper_timeout(kwargs: dict[str, Any], exception: Exception) -> float
def check_coroutine(value) -> bool:
get_coroutine_checker: Final = getattr(sys.modules[__name__], "get_coroutine_checker")
get_coroutine_checker: Final = litellm_utils.get_coroutine_checker
return get_coroutine_checker().is_async_callable(value)
@ -1207,7 +1230,7 @@ async def async_pre_call_deployment_hook(kwargs: dict[str, Any], call_type: str)
async def async_post_call_success_deployment_hook(
request_data: dict, response: Any, call_type: CallTypes | None
request_data: dict, response: object, call_type: CallTypes | None
) -> Any | None:
"""
Allow modifying / reviewing the response just after it's received from the deployment.
@ -1317,7 +1340,7 @@ def post_call_processing(
def client(original_function):
Rules: Final = getattr(sys.modules[__name__], "Rules")
Rules: Final = litellm_utils.Rules
rules_obj: Final = Rules()
@wraps(original_function)
@ -1551,10 +1574,10 @@ def client(original_function):
if call_type == CallTypes.completion.value:
num_retries = kwargs.get("num_retries", None) or litellm.num_retries or None
if kwargs.get("retry_policy", None):
get_num_retries_from_retry_policy = getattr(
get_num_retries_from_retry_policy: Callable[..., int | None] = getattr(
sys.modules[__name__], "get_num_retries_from_retry_policy"
)
reset_retry_policy = getattr(sys.modules[__name__], "reset_retry_policy")
reset_retry_policy = litellm_utils.reset_retry_policy
num_retries = get_num_retries_from_retry_policy(
exception=e,
retry_policy=kwargs.get("retry_policy"),
@ -1593,7 +1616,7 @@ def client(original_function):
get_num_retries_from_retry_policy = getattr(
sys.modules[__name__], "get_num_retries_from_retry_policy"
)
reset_retry_policy = getattr(sys.modules[__name__], "reset_retry_policy")
reset_retry_policy = litellm_utils.reset_retry_policy
num_retries = get_num_retries_from_retry_policy(
exception=e,
retry_policy=kwargs.get("retry_policy"),
@ -1939,7 +1962,7 @@ def client(original_function):
if not _is_streaming_response_for_correlation(result):
_restore_correlation_context_if_supported(logging_obj)
get_coroutine_checker: Final = getattr(sys.modules[__name__], "get_coroutine_checker")
get_coroutine_checker: Final = litellm_utils.get_coroutine_checker
is_coroutine: Final = get_coroutine_checker().is_async_callable(original_function)
# Return the appropriate wrapper based on the original function type
@ -1992,7 +2015,7 @@ _STREAMING_CALL_TYPES: Final = frozenset(
def _is_streaming_request(
kwargs: dict[str, Any],
kwargs: dict[str, object],
call_type: CallTypes | str,
) -> bool:
"""
@ -2323,7 +2346,7 @@ def supports_response_schema(model: str, custom_llm_provider: str | None = None)
"""
## GET LLM PROVIDER ##
try:
get_llm_provider: Final = getattr(sys.modules[__name__], "get_llm_provider")
get_llm_provider: Final = litellm_utils.get_llm_provider
model, custom_llm_provider, _, _ = get_llm_provider(model=model, custom_llm_provider=custom_llm_provider)
except Exception as e:
verbose_logger.debug(
@ -2700,7 +2723,7 @@ _CACHE_PRICING_FIELDS: Final = (
)
def _resolve_builtin_model_cost_entry(key: str, provider: str) -> dict[str, Any] | None:
def _resolve_builtin_model_cost_entry(key: str, provider: str) -> dict[str, object] | None:
"""Best-effort lookup of a built-in ``model_cost`` entry for a custom key
whose shape ``get_model_info`` cannot resolve (repeated provider prefixes
like ``bedrock/bedrock/bedrock/us.anthropic.claude-sonnet-4-6`` or region
@ -2992,7 +3015,7 @@ def get_optional_params_transcription(
passed_params.pop("OPENAI_TRANSCRIPTION_PARAMS")
custom_llm_provider = passed_params.pop("custom_llm_provider")
drop_params = passed_params.pop("drop_params")
special_params: Final = passed_params.pop("kwargs")
special_params: Final[Mapping[str, object]] = passed_params.pop("kwargs")
for k, v in special_params.items():
passed_params[k] = v
@ -3101,7 +3124,7 @@ def get_optional_params_image_gen(
provider_config = passed_params.pop("provider_config", None)
drop_params = passed_params.pop("drop_params", None)
additional_drop_params = passed_params.pop("additional_drop_params", None)
special_params: Final = passed_params.pop("kwargs")
special_params: Final[Mapping[str, object]] = passed_params.pop("kwargs")
for k, v in special_params.items():
if (
k.startswith("aws_")
@ -3133,7 +3156,7 @@ def get_optional_params_image_gen(
default_params=default_params,
additional_drop_params=additional_drop_params,
)
optional_params: dict[str, Any] = {}
optional_params: dict[str, object] = {}
## raise exception if non-default value passed for non-openai/azure embedding calls
def _check_valid_arg(supported_params):
@ -3365,7 +3388,14 @@ def get_optional_params_embeddings(
elif custom_llm_provider == "bedrock":
# if dimensions is in non_default_params -> pass it for model=bedrock/amazon.titan-embed-text-v2
if "amazon.titan-embed-text-v1" in model:
object: Any = litellm.AmazonTitanG1Config()
object: (
AmazonTitanG1Config
| AmazonTitanMultimodalEmbeddingG1Config
| AmazonTitanV2Config
| BedrockCohereEmbeddingConfig
| TwelveLabsMarengoEmbeddingConfig
| AmazonNovaEmbeddingConfig
) = litellm.AmazonTitanG1Config()
elif "amazon.titan-embed-image-v1" in model:
object = litellm.AmazonTitanMultimodalEmbeddingG1Config()
elif "amazon.titan-embed-text-v2:0" in model:
@ -4949,7 +4979,7 @@ def get_max_tokens(model: str) -> int | None:
response.raise_for_status() # Raise an exception for bad responses (4xx or 5xx)
# Parse the JSON response
config_json: Final = response.json()
config_json: Final[Mapping[str, int]] = response.json()
# Extract and return the max_position_embeddings
max_position_embeddings: Final = config_json.get("max_position_embeddings")
if max_position_embeddings is not None:
@ -4965,7 +4995,7 @@ def get_max_tokens(model: str) -> int | None:
return litellm.model_cost[model]["max_output_tokens"]
elif "max_tokens" in litellm.model_cost[model]:
return litellm.model_cost[model]["max_tokens"]
get_llm_provider: Final = getattr(sys.modules[__name__], "get_llm_provider")
get_llm_provider: Final = litellm_utils.get_llm_provider
model, custom_llm_provider, _, _ = get_llm_provider(model=model)
if custom_llm_provider == "huggingface":
max_tokens: Final = _get_max_position_embeddings(model_name=model)
@ -5253,7 +5283,7 @@ def _get_potential_model_names(model: str, custom_llm_provider: str | None) -> P
if custom_llm_provider is None:
# Get custom_llm_provider
try:
get_llm_provider: Final = getattr(sys.modules[__name__], "get_llm_provider")
get_llm_provider: Final = litellm_utils.get_llm_provider
split_model, custom_llm_provider, _, _ = get_llm_provider(model=model)
except Exception:
split_model = model
@ -5297,7 +5327,7 @@ def _get_max_position_embeddings(model_name: str) -> int | None:
response.raise_for_status() # Raise an exception for bad responses (4xx or 5xx)
# Parse the JSON response
config_json: Final = response.json()
config_json: Final[Mapping[str, int]] = response.json()
# Extract and return the max_position_embeddings
max_position_embeddings: Final = config_json.get("max_position_embeddings")
@ -5679,6 +5709,7 @@ def _get_model_info_helper(
supports_url_context=_model_info.get("supports_url_context", None),
supports_reasoning=_model_info.get("supports_reasoning", None),
supports_adaptive_thinking=_model_info.get("supports_adaptive_thinking", None),
supports_tool_search=_model_info.get("supports_tool_search", None),
supports_mid_conversation_system=_model_info.get("supports_mid_conversation_system", None),
supports_none_reasoning_effort=_model_info.get("supports_none_reasoning_effort", None),
supports_minimal_reasoning_effort=_model_info.get("supports_minimal_reasoning_effort", None),
@ -6066,7 +6097,7 @@ def validate_environment(
}
## EXTRACT LLM PROVIDER - if model name provided
try:
get_llm_provider: Final = getattr(sys.modules[__name__], "get_llm_provider")
get_llm_provider: Final = litellm_utils.get_llm_provider
_, custom_llm_provider, _, _ = get_llm_provider(model=model)
except Exception:
custom_llm_provider = None
@ -6543,7 +6574,7 @@ def _get_retry_after_from_exception_header(
# <http-date>". See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#syntax for
# details.
if response_headers is not None:
retry_header: Final = response_headers.get("retry-after")
retry_header: Final[str] = response_headers.get("retry-after")
try:
retry_after = int(retry_header)
except Exception:
@ -6634,7 +6665,7 @@ def register_prompt_template(
complete_model: Final = model
potential_models: Final = [complete_model]
try:
get_llm_provider: Final = getattr(sys.modules[__name__], "get_llm_provider")
get_llm_provider: Final = litellm_utils.get_llm_provider
model = get_llm_provider(model=model)[0]
potential_models.append(model)
except Exception:
@ -7276,7 +7307,7 @@ def _get_base_model_from_metadata(model_call_details=None):
return _base_model
metadata: Final = litellm_params.get("metadata") or {}
_get_base_model_from_litellm_call_metadata = getattr(
_get_base_model_from_litellm_call_metadata: Callable[..., str | None] = getattr(
sys.modules[__name__], "_get_base_model_from_litellm_call_metadata"
)
base_model_from_metadata: Final = _get_base_model_from_litellm_call_metadata(metadata=metadata)
@ -7969,7 +8000,7 @@ class ProviderConfigManager:
@staticmethod
def _get_cohere_config(model: str) -> BaseConfig:
"""Get Cohere config based on route."""
CohereModelInfo: Final = getattr(sys.modules[__name__], "CohereModelInfo")
CohereModelInfo: Final = litellm_utils.CohereModelInfo
route: Final = CohereModelInfo.get_cohere_route(model)
if route == "v2":
return litellm.CohereV2ChatConfig()
@ -9006,7 +9037,7 @@ class ProviderConfigManager:
return ReductoParseLegacyConfig()
return None
MistralOCRConfig: Final = getattr(sys.modules[__name__], "MistralOCRConfig")
MistralOCRConfig: Final = litellm_utils.MistralOCRConfig
PROVIDER_TO_CONFIG_MAP: Final = {
litellm.LlmProviders.MISTRAL: MistralOCRConfig,
}
@ -9285,13 +9316,14 @@ def extract_duration_from_srt_or_vtt(srt_or_vtt_content: str) -> float | None:
# Regular expression to match timestamps in the format "hh:mm:ss,ms" or "hh:mm:ss.ms"
timestamp_pattern: Final = r"(\d{2}):(\d{2}):(\d{2})[.,](\d{3})"
timestamps: Final = re.findall(timestamp_pattern, srt_or_vtt_content)
timestamps: Final[Sequence[tuple[str, str, str, str]]] = re.findall(timestamp_pattern, srt_or_vtt_content)
if not timestamps:
return None
# Convert timestamps to seconds and find the max (end time)
durations: Final = []
match: tuple[str, str, str, str]
for match in timestamps:
hours, minutes, seconds, milliseconds = map(int, match)
total_seconds = hours * 3600 + minutes * 60 + seconds + milliseconds / 1000.0
@ -9338,11 +9370,11 @@ def _add_path_to_api_base(api_base: str, ending_path: str) -> str:
return str(modified_url.copy_with(params=original_url.params))
def get_standard_openai_params(params: dict) -> dict:
def get_standard_openai_params(params: Mapping[str, object]) -> dict:
return {k: v for k, v in params.items() if k in litellm.OPENAI_CHAT_COMPLETION_PARAMS and v is not None}
def get_non_default_completion_params(kwargs: dict) -> dict:
def get_non_default_completion_params(kwargs: Mapping[str, object]) -> dict:
openai_params: Final = litellm.OPENAI_CHAT_COMPLETION_PARAMS
default_params: Final = openai_params + all_litellm_params
non_default_params: Final = {
@ -9352,7 +9384,7 @@ def get_non_default_completion_params(kwargs: dict) -> dict:
return non_default_params
def peek_reasoning_summary_aliases(optional_params: dict) -> Any | None:
def peek_reasoning_summary_aliases(optional_params: dict) -> object | None:
"""Read AI-SDK-style reasoning summary from optional_params or nested extra_body.
Uses key membership (not ``or`` chains) so falsy values like ``""`` are not skipped.
@ -9372,7 +9404,7 @@ def peek_reasoning_summary_aliases(optional_params: dict) -> Any | None:
def strip_reasoning_summary_aliases_from_optional_params(
optional_params: dict,
) -> tuple[dict, Any | None]:
) -> tuple[dict, object | None]:
"""Copy optional_params; remove reasoningSummary aliases from top-level and extra_body."""
op: Final = dict(optional_params)
rs_val = op.pop("reasoningSummary", None)
@ -9404,7 +9436,7 @@ def get_non_default_transcription_params(kwargs: dict) -> dict:
def add_openai_metadata(
metadata: Mapping[str, Any] | None,
metadata: Mapping[str, object] | None,
) -> dict[str, str] | None:
"""
Add metadata to openai optional parameters, excluding hidden params.
@ -9438,7 +9470,7 @@ def add_openai_metadata(
return visible_metadata.copy()
def get_requester_metadata(metadata: dict):
def get_requester_metadata(metadata: Mapping[str, object]):
if not metadata:
return None
@ -9498,7 +9530,7 @@ def return_raw_request(endpoint: CallTypes, kwargs: dict) -> RawRequestTypedDict
)
def jsonify_tools(tools: list[Any]) -> list[dict]:
def jsonify_tools(tools: Sequence[object]) -> list[dict]:
"""
Fixes https://github.com/BerriAI/litellm/issues/9321
@ -9524,9 +9556,9 @@ def get_empty_usage() -> Usage:
def should_run_mock_completion(
mock_response: Any | None,
mock_tool_calls: Any | None,
mock_timeout: Any | None,
mock_response: object | None,
mock_tool_calls: object | None,
mock_timeout: object | None,
) -> bool:
if mock_response or mock_tool_calls or mock_timeout:
return True

43
model_prices_and_context_window.json
View file

@ -730,6 +730,7 @@
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -755,6 +756,7 @@
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -973,6 +975,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -1005,6 +1008,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1038,6 +1042,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1071,6 +1076,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1104,6 +1110,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1137,6 +1144,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1171,6 +1179,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1222,6 +1231,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1258,6 +1268,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1294,6 +1305,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1330,6 +1342,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -1946,6 +1959,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
@ -2203,6 +2217,7 @@
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2235,6 +2250,7 @@
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2267,6 +2283,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2299,6 +2316,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2331,6 +2349,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2363,6 +2382,7 @@
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2430,6 +2450,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.2e-05,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2686,6 +2707,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -2775,6 +2797,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -10826,6 +10849,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -10850,6 +10874,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -11005,6 +11030,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -11029,6 +11055,7 @@
"cache_read_input_token_cost": 3.6e-07,
"input_cost_per_token": 3.6e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 8192,
"max_tokens": 8192,
@ -11811,6 +11838,7 @@
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -15998,6 +16026,7 @@
"input_cost_per_token": 1.1e-06,
"deprecation_date": "2026-10-15",
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -16213,6 +16242,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -22031,6 +22061,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.2e-05,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -22091,6 +22122,7 @@
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -26534,6 +26566,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -26563,6 +26596,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35259,6 +35293,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35417,6 +35452,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.32e-05,
"cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35451,6 +35487,7 @@
"cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.44e-05,
"cache_read_input_token_cost_above_200k_tokens": 7.2e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35475,6 +35512,7 @@
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35525,6 +35563,7 @@
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35556,6 +35595,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -35586,6 +35626,7 @@
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -45975,6 +46016,7 @@
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 1.2e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
@ -46000,6 +46042,7 @@
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 1.2e-06,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,

3
model_prices_and_context_window.schema.json
View file

@ -671,6 +671,9 @@
"supports_tool_choice": {
"type": "boolean"
},
"supports_tool_search": {
"type": "boolean"
},
"supports_url_context": {
"type": "boolean"
},

20
ruff-strict-budget.json
View file

@ -1,21 +1,21 @@
{
"ANN001": {
"limit": 3106
"limit": 3058
},
"ANN002": {
"limit": 71
},
"ANN003": {
"limit": 832
"limit": 827
},
"ANN201": {
"limit": 2023
"limit": 2022
},
"ANN202": {
"limit": 860
"limit": 855
},
"ANN204": {
"limit": 713
"limit": 712
},
"ANN205": {
"limit": 114
@ -24,7 +24,7 @@
"limit": 133
},
"ANN401": {
"limit": 1495
"limit": 1384
},
"ASYNC230": {
"limit": 11
@ -39,7 +39,7 @@
"limit": 504
},
"B009": {
"limit": 79
"limit": 64
},
"B010": {
"limit": 190
@ -171,7 +171,7 @@
"limit": 3
},
"RET504": {
"limit": 177
"limit": 176
},
"RUF012": {
"limit": 241
@ -201,7 +201,7 @@
"limit": 58
},
"SIM102": {
"limit": 322
"limit": 321
},
"SIM103": {
"limit": 119
@ -234,7 +234,7 @@
"limit": 5
},
"TID251": {
"limit": 1226
"limit": 1224
},
"TRY002": {
"limit": 524

2
schema.prisma
View file

@ -452,6 +452,7 @@ model LiteLLM_VerificationToken {
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")
updated_by String?
settings_updated_at DateTime? @map("settings_updated_at")
last_active DateTime? // When this key was last used
rotation_count Int? @default(0) // Number of times key has been rotated
auto_rotate Boolean? @default(false) // Whether this key should be auto-rotated
@ -548,6 +549,7 @@ model LiteLLM_DeletedVerificationToken {
created_by String? // Original creator
updated_at DateTime? // Last update timestamp before deletion
updated_by String? // Last user who updated before deletion
settings_updated_at DateTime? // Last configuration change before deletion
last_active DateTime? // When this key was last used before deletion
rotation_count Int? @default(0)
auto_rotate Boolean? @default(false)

4
tests/e2e/claude_code/pdf_input/test_bedrock_converse.py
View file

@ -88,10 +88,6 @@ def _build_minimal_pdf(marker: str) -> bytes:
return bytes(out)
@pytest.mark.skip(
reason="product bug LIT-4523: Bedrock Converse requires a text block with document; "
"re-enable when document-only content is handled"
)
@pytest.mark.covers("llm.messages.bedrock_converse.pdf_input.nonstream.works")
def test_pdf_input_bedrock_converse(compat_result, tmp_path):
base_url, api_key = require_proxy(compat_result)

4
tests/e2e/claude_code/thinking/test_bedrock_converse.py
View file

@ -54,10 +54,6 @@ def _has_thinking_block(events: Sequence[Mapping[str, Any]]) -> bool:
return False
@pytest.mark.skip(
reason="product bug LIT-4524: Bedrock Converse streaming Content block is not a text block; "
"re-enable when empty/mismatched content_block_delta is fixed"
)
@pytest.mark.covers("llm.messages.bedrock_converse.thinking.nonstream.works")
def test_thinking_bedrock_converse(compat_result):
"""Drive the `claude` CLI against the LiteLLM proxy with thinking

4
tests/e2e/claude_code/tool_search/test_bedrock_invoke.py
View file

@ -59,10 +59,6 @@ BEDROCK_INVOKE_MODELS = [
]
@pytest.mark.skip(
reason="product bug LIT-4522: Bedrock Invoke /v1/messages does not normalize "
"tool_search_tool_regex_20251119; re-enable when messages path matches chat path"
)
@pytest.mark.covers("llm.messages.bedrock_invoke.tool_search.nonstream.works")
def test_tool_search_bedrock_invoke(compat_result):
"""Probe `/v1/messages` with a `tool_search_tool_regex_20251119`

170
tests/test_litellm/integrations/SlackAlerting/test_slack_alerting.py
View file

@ -1,17 +1,21 @@
import asyncio
import datetime
import json
import os
import sys
import time
import unittest
from typing import List, Optional, Tuple
from typing import Final, List, Optional, Tuple
from unittest.mock import ANY, AsyncMock, MagicMock, Mock, patch
sys.path.insert(
0, os.path.abspath("../../..")
) # Adds the parent directory to the system-path
import pytest
sys.path.insert(0, os.path.abspath("../../..")) # Adds the parent directory to the system-path
import litellm
from litellm.caching.caching import DualCache
from litellm.integrations.SlackAlerting.slack_alerting import SlackAlerting
from litellm.proxy._types import CallInfo, Litellm_EntityType
from litellm.types.integrations.slack_alerting import SlackAlertingCacheKeys
class TestSlackAlerting(unittest.TestCase):
@ -20,37 +24,27 @@ class TestSlackAlerting(unittest.TestCase):
def test_get_percent_of_max_budget_left(self):
# Test case 1: When max_budget is None
user_info = CallInfo(
max_budget=None, spend=50.0, event_group=Litellm_EntityType.KEY
)
user_info = CallInfo(max_budget=None, spend=50.0, event_group=Litellm_EntityType.KEY)
result = self.slack_alerting._get_percent_of_max_budget_left(user_info)
self.assertEqual(result, 0.0)
# Test case 2: When max_budget is 0
user_info = CallInfo(
max_budget=0.0, spend=50.0, event_group=Litellm_EntityType.KEY
)
user_info = CallInfo(max_budget=0.0, spend=50.0, event_group=Litellm_EntityType.KEY)
result = self.slack_alerting._get_percent_of_max_budget_left(user_info)
self.assertEqual(result, 0.0)
# Test case 3: When spend is less than max_budget
user_info = CallInfo(
max_budget=100.0, spend=75.0, event_group=Litellm_EntityType.KEY
)
user_info = CallInfo(max_budget=100.0, spend=75.0, event_group=Litellm_EntityType.KEY)
result = self.slack_alerting._get_percent_of_max_budget_left(user_info)
self.assertEqual(result, 0.25)
# Test case 4: When spend equals max_budget
user_info = CallInfo(
max_budget=100.0, spend=100.0, event_group=Litellm_EntityType.KEY
)
user_info = CallInfo(max_budget=100.0, spend=100.0, event_group=Litellm_EntityType.KEY)
result = self.slack_alerting._get_percent_of_max_budget_left(user_info)
self.assertEqual(result, 0.0)
# Test case 5: When spend exceeds max_budget
user_info = CallInfo(
max_budget=100.0, spend=120.0, event_group=Litellm_EntityType.KEY
)
user_info = CallInfo(max_budget=100.0, spend=120.0, event_group=Litellm_EntityType.KEY)
result = self.slack_alerting._get_percent_of_max_budget_left(user_info)
self.assertEqual(result, -0.2)
@ -189,7 +183,9 @@ class TestSlackAlerting(unittest.TestCase):
# Test the specific formatting logic we're interested in
alert_type_formatted = f"Alert type: `{alert_type.name}`\n"
formatted_message = f"{alert_type_formatted}\n Level: `{level}`\nTimestamp: `{current_time}`\n\nMessage: {message}"
formatted_message = (
f"{alert_type_formatted}\n Level: `{level}`\nTimestamp: `{current_time}`\n\nMessage: {message}"
)
# Verify alert_type is in the formatted message as expected
self.assertIn("Alert type: `llm_exceptions`", formatted_message)
@ -214,9 +210,7 @@ class TestSlackAlerting(unittest.TestCase):
json.dumps(outage_value)
# Verify the specific error message
self.assertIn(
"Object of type set is not JSON serializable", str(context.exception)
)
self.assertIn("Object of type set is not JSON serializable", str(context.exception))
def test_fixed_redis_serialization(self):
"""Test that our fix resolves the Redis serialization error."""
@ -245,3 +239,133 @@ class TestSlackAlerting(unittest.TestCase):
)
self.assertEqual(parsed_data["alerts"], [408])
self.assertEqual(parsed_data["provider_region_id"], "vertex_aius-east1")
_REPORT_SENT_KEY: Final = SlackAlertingCacheKeys.report_sent_key.value
_DAILY_REPORT_FREQUENCY: Final = 900
async def _slack_alerting_with_due_daily_report() -> SlackAlerting:
slack_alerting: Final = SlackAlerting(
internal_usage_cache=DualCache(),
alerting_args={"daily_report_frequency": _DAILY_REPORT_FREQUENCY},
)
await slack_alerting.internal_usage_cache.async_set_cache(
key=_REPORT_SENT_KEY,
value=time.time() - _DAILY_REPORT_FREQUENCY - 1,
)
slack_alerting.send_daily_reports = AsyncMock()
return slack_alerting
async def _read_report_sent(slack_alerting: SlackAlerting) -> float:
return await slack_alerting.internal_usage_cache.async_get_cache(
key=_REPORT_SENT_KEY,
parent_otel_span=None,
)
@pytest.mark.asyncio
async def test_daily_report_skipped_when_another_pod_holds_the_lock():
"""regression: issue #14809 - every pod sent its own copy of the daily report.
The losing pod must also leave report_sent untouched so the winner's window still counts.
"""
slack_alerting: Final = await _slack_alerting_with_due_daily_report()
report_sent_before: Final = await _read_report_sent(slack_alerting)
pod_lock_manager: Final = AsyncMock()
pod_lock_manager.acquire_lock.return_value = False
result: Final = await slack_alerting._run_scheduler_helper(
llm_router=MagicMock(),
pod_lock_manager=pod_lock_manager,
)
assert result is False
slack_alerting.send_daily_reports.assert_not_awaited()
assert await _read_report_sent(slack_alerting) == report_sent_before
pod_lock_manager.acquire_lock.assert_awaited_once_with(
cronjob_id="slack_daily_report",
ttl=_DAILY_REPORT_FREQUENCY,
allow_reentrant=False,
)
@pytest.mark.asyncio
async def test_daily_report_sent_by_the_pod_that_wins_the_lock():
slack_alerting: Final = await _slack_alerting_with_due_daily_report()
report_sent_before: Final = await _read_report_sent(slack_alerting)
llm_router: Final = MagicMock()
pod_lock_manager: Final = AsyncMock()
pod_lock_manager.acquire_lock.return_value = True
result: Final = await slack_alerting._run_scheduler_helper(
llm_router=llm_router,
pod_lock_manager=pod_lock_manager,
)
assert result is True
slack_alerting.send_daily_reports.assert_awaited_once_with(router=llm_router)
assert await _read_report_sent(slack_alerting) > report_sent_before
pod_lock_manager.acquire_lock.assert_awaited_once_with(
cronjob_id="slack_daily_report",
ttl=_DAILY_REPORT_FREQUENCY,
allow_reentrant=False,
)
@pytest.mark.parametrize("lock_state", ["no_pod_lock_manager", "no_redis_configured"])
@pytest.mark.asyncio
async def test_daily_report_still_sent_without_a_working_lock(lock_state: str):
"""Single-pod parity: a missing lock manager, or one whose acquire_lock returns None
because redis isn't configured, must not suppress the report."""
slack_alerting: Final = await _slack_alerting_with_due_daily_report()
report_sent_before: Final = await _read_report_sent(slack_alerting)
llm_router: Final = MagicMock()
pod_lock_manager: Final = (
None if lock_state == "no_pod_lock_manager" else AsyncMock(acquire_lock=AsyncMock(return_value=None))
)
result: Final = await slack_alerting._run_scheduler_helper(
llm_router=llm_router,
pod_lock_manager=pod_lock_manager,
)
assert result is True
slack_alerting.send_daily_reports.assert_awaited_once_with(router=llm_router)
assert await _read_report_sent(slack_alerting) > report_sent_before
@pytest.mark.asyncio
async def test_daily_report_lock_not_attempted_before_the_interval_elapses():
"""The lock is a per-window marker, so a pod must not burn it on a check that isn't due yet."""
slack_alerting: Final = await _slack_alerting_with_due_daily_report()
await slack_alerting.internal_usage_cache.async_set_cache(key=_REPORT_SENT_KEY, value=time.time())
pod_lock_manager: Final = AsyncMock()
pod_lock_manager.acquire_lock.return_value = True
result: Final = await slack_alerting._run_scheduler_helper(
llm_router=MagicMock(),
pod_lock_manager=pod_lock_manager,
)
assert result is False
pod_lock_manager.acquire_lock.assert_not_awaited()
slack_alerting.send_daily_reports.assert_not_awaited()
@pytest.mark.asyncio
async def test_scheduled_daily_report_threads_the_pod_lock_manager_through():
"""The loop in _run_scheduled_daily_report is where the lock manager reaches the gate."""
slack_alerting: Final = SlackAlerting(alert_types=["daily_reports"])
pod_lock_manager: Final = AsyncMock()
slack_alerting._run_scheduler_helper = AsyncMock(side_effect=asyncio.CancelledError)
with pytest.raises(asyncio.CancelledError):
await slack_alerting._run_scheduled_daily_report(
llm_router=MagicMock(),
pod_lock_manager=pod_lock_manager,
)
_, kwargs = slack_alerting._run_scheduler_helper.await_args
assert kwargs["pod_lock_manager"] is pod_lock_manager

81
tests/test_litellm/integrations/test_anthropic_cache_control_hook.py
View file

@ -1728,6 +1728,87 @@ class TestEnableAnthropicPromptCaching:
assert result_msgs[-1]["content"][-1]["cache_control"] == {"type": "ephemeral"}
assert "cache_control" not in result_msgs[0]["content"][-1]
class TestPerKeyEnablePromptCaching:
"""Per-request enable_prompt_caching override (stamped from key metadata) with the global flag off."""
MESSAGES: List[AllMessageValues] = [
{"role": "system", "content": "a long system prompt"},
{"role": "user", "content": "latest turn"},
]
def _points(self, enable_prompt_caching, model="claude-sonnet-4-5", provider="anthropic", messages=None):
return AnthropicCacheControlHook.get_default_injection_points(
messages=copy.deepcopy(self.MESSAGES) if messages is None else messages,
system=None,
model=model,
custom_llm_provider=provider,
enable_prompt_caching=enable_prompt_caching,
)
def test_true_injects_with_global_flag_off(self):
assert litellm.enable_anthropic_prompt_caching is False
assert self._points(True) == [
{"location": "message", "role": "system", "index": None, "control": {"type": "ephemeral"}},
{"location": "message", "role": None, "index": -1, "control": {"type": "ephemeral"}},
]
@pytest.mark.parametrize("enable_prompt_caching", [False, None])
def test_false_and_none_fall_back_to_global_flag(self, enable_prompt_caching):
assert self._points(enable_prompt_caching) == []
def test_false_does_not_suppress_global_flag(self, monkeypatch):
monkeypatch.setattr(litellm, "enable_anthropic_prompt_caching", True)
assert [p["index"] for p in self._points(False)] == [None, -1]
def test_provider_gate_still_applies(self):
assert self._points(True, model="gpt-4o", provider="openai") == []
def test_unsupported_model_gate_still_applies(self):
assert self._points(True, model="anthropic.claude-3-5-sonnet-20240620-v1:0", provider="bedrock") == []
def test_client_markers_still_win(self):
messages = [
{"role": "system", "content": [{"type": "text", "text": "s", "cache_control": {"type": "ephemeral"}}]},
{"role": "user", "content": "latest turn"},
]
assert self._points(True, messages=messages) == []
def test_seed_injects_with_global_flag_off(self):
params: dict = {}
AnthropicCacheControlHook.maybe_seed_default_injection_points(
non_default_params=params,
messages=copy.deepcopy(self.MESSAGES),
model="claude-sonnet-4-5",
custom_llm_provider="anthropic",
enable_prompt_caching=True,
)
assert [p["index"] for p in params["cache_control_injection_points"]] == [None, -1]
def test_v1_messages_injects_and_pops_flag_from_kwargs(self):
kwargs: dict = {"enable_prompt_caching": True}
result_msgs, result_sys = AnthropicCacheControlHook.maybe_inject_cache_control(
[{"role": "user", "content": [{"type": "text", "text": "latest"}]}],
"a system prompt",
kwargs,
model="claude-sonnet-4-5",
custom_llm_provider="anthropic",
)
assert result_sys == [{"type": "text", "text": "a system prompt", "cache_control": {"type": "ephemeral"}}]
assert result_msgs[-1]["content"][-1]["cache_control"] == {"type": "ephemeral"}
assert "enable_prompt_caching" not in kwargs
def test_v1_messages_pops_flag_even_when_noop(self):
kwargs: dict = {"enable_prompt_caching": True}
AnthropicCacheControlHook.maybe_inject_cache_control(
[{"role": "user", "content": [{"type": "text", "text": "hi"}]}],
None,
kwargs,
model="gpt-4o",
custom_llm_provider="openai",
)
assert "enable_prompt_caching" not in kwargs
def test_v1_messages_is_noop_when_disabled(self):
messages = [{"role": "user", "content": [{"type": "text", "text": "hi"}]}]
result_msgs, result_sys = AnthropicCacheControlHook.maybe_inject_cache_control(

138
tests/test_litellm/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.py
View file

@ -8,6 +8,7 @@ import pytest
import litellm
from litellm.litellm_core_utils.prompt_templates.factory import (
BAD_MESSAGE_ERROR_STR,
BEDROCK_DOCUMENT_PLACEHOLDER_TEXT,
BedrockConverseMessagesProcessor,
BedrockImageProcessor,
_bedrock_converse_messages_pt,
@ -3269,3 +3270,140 @@ def test_group_tool_exchanges_is_linear_in_message_count():
assert len(groups) == 100_000
assert elapsed < 3.0, f"grouping 100k messages took {elapsed:.2f}s; suspect superlinear accumulation"
_PDF_DATA_URI = "data:application/pdf;base64," + base64.b64encode(b"%PDF-1.4 regression fixture").decode()
_PNG_DATA_URI = (
"data:image/png;base64,"
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="
)
def _text_blocks(message):
return [block["text"] for block in message["content"] if "text" in block]
def test_bedrock_converse_pdf_only_user_message_gets_text_block():
"""
Regression for LIT-4523: Claude Code sends a PDF as a user turn whose only
content is the document (an image_url part with a pdf data URI after the
/v1/messages -> completion bridge). Bedrock Converse rejects any user
message carrying a document without a sibling text block, so the builder
must inject a placeholder text block.
"""
messages = [
{
"role": "user",
"content": [{"type": "image_url", "image_url": {"url": _PDF_DATA_URI}}],
}
]
result = _bedrock_converse_messages_pt(
messages, "anthropic.claude-haiku-4-5", "bedrock"
)
assert len(result) == 1
assert any("document" in block for block in result[0]["content"])
assert _text_blocks(result[0]) == [BEDROCK_DOCUMENT_PLACEHOLDER_TEXT]
def test_bedrock_converse_document_with_text_gets_no_extra_text_block():
messages = [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": _PDF_DATA_URI}},
{"type": "text", "text": "summarize this"},
],
}
]
result = _bedrock_converse_messages_pt(
messages, "anthropic.claude-haiku-4-5", "bedrock"
)
assert _text_blocks(result[0]) == ["summarize this"]
def test_bedrock_converse_image_only_user_message_gets_no_text_block():
messages = [
{
"role": "user",
"content": [{"type": "image_url", "image_url": {"url": _PNG_DATA_URI}}],
}
]
result = _bedrock_converse_messages_pt(
messages, "anthropic.claude-haiku-4-5", "bedrock"
)
assert any("image" in block for block in result[0]["content"])
assert _text_blocks(result[0]) == []
def test_bedrock_converse_tool_round_trip_document_injects_text_before_cache_point():
"""
Claude Code shape: after a Read tool round trip, the document-only user
turn (with cache_control) merges into the toolResult message. The injected
text block must land before the trailing cachePoint so the cache boundary
stays the final block, and earlier turns must stay untouched.
"""
messages = [
{"role": "user", "content": "read the pdf"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "tooluse_pdf1",
"type": "function",
"function": {"name": "Read", "arguments": "{}"},
}
],
},
{"role": "tool", "tool_call_id": "tooluse_pdf1", "content": "read ok"},
{
"role": "user",
"content": [
{
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": "dGVzdA==",
},
"cache_control": {"type": "ephemeral"},
}
],
},
]
result = _bedrock_converse_messages_pt(
messages, "anthropic.claude-haiku-4-5", "bedrock"
)
assert _text_blocks(result[0]) == ["read the pdf"]
document_message = result[-1]
block_keys = [next(iter(block)) for block in document_message["content"]]
assert block_keys == ["toolResult", "document", "text", "cachePoint"]
assert _text_blocks(document_message) == [BEDROCK_DOCUMENT_PLACEHOLDER_TEXT]
@pytest.mark.asyncio
async def test_bedrock_converse_pdf_only_user_message_gets_text_block_async():
messages = [
{
"role": "user",
"content": [{"type": "image_url", "image_url": {"url": _PDF_DATA_URI}}],
}
]
result = await BedrockConverseMessagesProcessor._bedrock_converse_messages_pt_async(
messages=messages,
model="anthropic.claude-haiku-4-5",
llm_provider="bedrock",
)
assert len(result) == 1
assert any("document" in block for block in result[0]["content"])
assert _text_blocks(result[0]) == [BEDROCK_DOCUMENT_PLACEHOLDER_TEXT]

82
tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
View file

@ -1828,6 +1828,88 @@ def test_thinking_disabled_stays_plain_string_when_auto_summary_enabled():
assert new_kwargs["reasoning_effort"] == "none"
@pytest.mark.parametrize(
"model",
[
# SDK-style model with the provider prefix intact
"bedrock/converse/us.anthropic.claude-opus-4-7",
# what the bridge actually sees in the proxy: get_llm_provider has
# already stripped the `bedrock/` prefix by the time it translates
"converse/us.anthropic.claude-opus-4-7",
],
)
def test_adaptive_thinking_output_config_effort_preserved_for_claude_model(model):
"""
Regression: Claude Code drives adaptive thinking as `thinking: {"type": "adaptive"}`
plus `output_config: {"effort": "max"}`. The Claude branch of the thinking translator
forwarded `thinking` verbatim but returned early without reading `output_config`, and
the handler strips the raw key from extra_kwargs, so the effort tier never reached the
backend. On Bedrock Converse, adaptive thinking without effort streams zero reasoning
blocks. The `format` subkey must still be excluded (it is translated to
`response_format` separately).
"""
from litellm.types.llms.anthropic import AnthropicMessagesRequest
anthropic_request = AnthropicMessagesRequest(
model=model,
max_tokens=1024,
messages=[{"role": "user", "content": "hi"}],
thinking={"type": "adaptive"},
output_config={
"effort": "max",
"format": {"type": "json_schema", "schema": {"type": "object", "properties": {}}},
},
)
adapter = LiteLLMAnthropicMessagesAdapter()
openai_request, _ = adapter.translate_anthropic_to_openai(anthropic_message_request=anthropic_request)
assert openai_request["thinking"] == {"type": "adaptive"}
assert openai_request["output_config"] == {"effort": "max"}
assert "response_format" in openai_request
def test_adaptive_thinking_format_only_output_config_not_forwarded_for_claude_model():
"""When `output_config` carries only `format`, nothing effort-bearing remains, so the
translator must not forward an empty `output_config` dict."""
from litellm.types.llms.anthropic import AnthropicMessagesRequest
anthropic_request = AnthropicMessagesRequest(
model="bedrock/converse/us.anthropic.claude-opus-4-7",
max_tokens=1024,
messages=[{"role": "user", "content": "hi"}],
thinking={"type": "adaptive"},
output_config={"format": {"type": "json_schema", "schema": {"type": "object", "properties": {}}}},
)
adapter = LiteLLMAnthropicMessagesAdapter()
openai_request, _ = adapter.translate_anthropic_to_openai(anthropic_message_request=anthropic_request)
assert openai_request["thinking"] == {"type": "adaptive"}
assert "output_config" not in openai_request
def test_adaptive_thinking_output_config_not_forwarded_for_non_bedrock_claude_model():
"""`output_config` is forwarded only for Bedrock-destined Claude models. Other
Claude-through-bridge providers (e.g. openrouter) accept `thinking` but reject a raw
`output_config` param with UnsupportedParamsError when drop_params is off."""
from litellm.types.llms.anthropic import AnthropicMessagesRequest
anthropic_request = AnthropicMessagesRequest(
model="openrouter/anthropic/claude-opus-4-7",
max_tokens=1024,
messages=[{"role": "user", "content": "hi"}],
thinking={"type": "adaptive"},
output_config={"effort": "max"},
)
adapter = LiteLLMAnthropicMessagesAdapter()
openai_request, _ = adapter.translate_anthropic_to_openai(anthropic_message_request=anthropic_request)
assert openai_request["thinking"] == {"type": "adaptive"}
assert "output_config" not in openai_request
def test_stop_sequences_translated_to_stop_for_non_claude_model():
from litellm.types.llms.anthropic import AnthropicMessagesRequest

67
tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py
View file

@ -370,6 +370,73 @@ def test_output_config_effort_forwarded_into_additional_request_fields(model):
assert additional.get("output_config") == {"effort": "high"}
@pytest.mark.parametrize(
"model,effort,expected_effort",
[
("bedrock/converse/us.anthropic.claude-opus-4-7", "max", "max"),
("bedrock/converse/us.anthropic.claude-opus-4-6-v1", "xhigh", "max"),
],
)
def test_explicit_output_config_effort_mapped_for_adaptive_thinking_converse(model, effort, expected_effort):
"""Regression: Claude Code drives adaptive thinking as ``thinking: {"type":
"adaptive"}`` plus ``output_config: {"effort": ...}``. ``output_config`` must
be a supported openai param and survive ``map_openai_params`` (clamped to the
model's Bedrock effort ceiling), otherwise the Converse request carries
adaptive thinking without an effort tier and Bedrock streams zero
``reasoningContent`` blocks."""
config = AmazonConverseConfig()
assert "output_config" in config.get_supported_openai_params(model)
optional_params = config.map_openai_params(
non_default_params={
"thinking": {"type": "adaptive"},
"output_config": {"effort": effort},
},
optional_params={},
model=model,
drop_params=False,
)
assert optional_params["thinking"] == {"type": "adaptive"}
assert optional_params["output_config"] == {"effort": expected_effort}
def test_output_config_supported_param_for_arn_models_converse():
"""ARN model ids hide the underlying Claude model, so ``output_config`` must
be in the blanket ARN supported-params list too."""
config = AmazonConverseConfig()
arn_model = "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abcdef123456"
assert "output_config" in config.get_supported_openai_params(arn_model)
def test_output_config_effort_forwarded_for_application_inference_profile_arn():
"""Regression: opaque application inference profile ARNs cannot resolve a
base model, so the anthropic-only serialization gate dropped ``output_config``
while still sending ``thinking``: adaptive thinking with no effort tier, and
Bedrock streams zero ``reasoningContent`` blocks. The effort must be forwarded
verbatim (ceilings and capability gates are unknowable behind the alias) for
Bedrock to enforce."""
config = AmazonConverseConfig()
arn_model = "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abcdef123456"
result = config._transform_request(
model=arn_model,
messages=[{"role": "user", "content": "hi"}],
optional_params={
"maxTokens": 256,
"thinking": {"type": "adaptive"},
"output_config": {"effort": "max"},
},
litellm_params={},
headers={},
)
additional = result.get("additionalModelRequestFields", {})
assert additional.get("thinking") == {"type": "adaptive"}
assert additional.get("output_config") == {"effort": "max"}
def test_output_config_format_translated_to_native_output_config_converse():
"""``output_config.format`` becomes Bedrock ``outputConfig`` and is not forwarded raw."""
config = AmazonConverseConfig()

85
tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py
View file

@ -2474,6 +2474,91 @@ def test_filter_and_transform_beta_headers_passes_context_management_for_bedrock
assert out_converse == []
@pytest.mark.parametrize(
"model",
[
"us.anthropic.claude-haiku-4-5-20251001-v1:0",
"us.anthropic.claude-sonnet-4-5-20250929-v1:0",
"us.anthropic.claude-opus-4-7",
],
)
def test_bedrock_messages_tool_search_adds_beta_header(local_beta_headers_config, model):
"""
LIT-4522: Bedrock InvokeModel only admits ``tool_search_tool_*`` tool types
when the request body carries the ``tool-search-tool-2025-10-19`` beta;
without it Bedrock 400s with "Input tag 'tool_search_tool_regex_20251119'
... does not match any of the expected tags". The allowlist in
``_supports_tool_search_on_bedrock`` previously omitted Haiku 4.5 and
Opus 4.7, so the beta was silently dropped for those models and every
tool-search request failed. Verified live 2026-08-11: Bedrock returns 200
with ``server_tool_use`` for all three models once the beta is sent.
"""
from litellm.types.router import GenericLiteLLMParams
cfg = AmazonAnthropicClaudeMessagesConfig()
messages = [{"role": "user", "content": [{"type": "text", "text": "Hi"}]}]
optional_params = {
"max_tokens": 64,
"tools": [
{"type": "tool_search_tool_regex_20251119", "name": "tool_search_tool_regex"},
{
"name": "add_numbers",
"description": "Add two integers",
"input_schema": {
"type": "object",
"properties": {"a": {"type": "integer"}, "b": {"type": "integer"}},
"required": ["a", "b"],
},
},
],
}
result = cfg.transform_anthropic_messages_request(
model=model,
messages=messages,
anthropic_messages_optional_request_params=optional_params,
litellm_params=GenericLiteLLMParams(),
headers={},
)
assert "tool-search-tool-2025-10-19" in (result.get("anthropic_beta") or [])
def test_bedrock_messages_tool_search_model_map_flag_is_authoritative(local_model_cost_map, monkeypatch):
"""``supports_tool_search`` lives in the model map; the name patterns in
``_supports_tool_search_on_bedrock`` are only a fallback for ids the map
cannot resolve. Flipping the mapped entry's flag to ``False`` must win even
though the model name still matches the ``haiku-4-5`` pattern."""
import litellm
from litellm.llms.anthropic.common_utils import AnthropicModelInfo
model = "us.anthropic.claude-haiku-4-5-20251001-v1:0"
cfg = AmazonAnthropicClaudeMessagesConfig()
assert AnthropicModelInfo._get_provider_resolved_capability(model, "supports_tool_search", "bedrock") is True
assert cfg._supports_tool_search_on_bedrock(model) is True
monkeypatch.setitem(litellm.model_cost[model], "supports_tool_search", False)
litellm.get_model_info.cache_clear()
assert cfg._supports_tool_search_on_bedrock(model) is False
@pytest.mark.parametrize(
"model, expected",
[
pytest.param("us.anthropic.claude-opus-4-6-v99:9", True, id="unmapped_id_falls_back_to_patterns"),
pytest.param("anthropic.claude-3-5-sonnet-20240620-v1:0", False, id="mapped_entry_without_flag_no_pattern"),
],
)
def test_bedrock_messages_tool_search_pattern_fallback(local_model_cost_map, model, expected):
"""Ids the model map cannot resolve (or resolves without a
``supports_tool_search`` opinion) fall through to the name patterns, so
ARNs and unlisted regional variants of supported families keep working."""
cfg = AmazonAnthropicClaudeMessagesConfig()
assert cfg._supports_tool_search_on_bedrock(model) is expected
def test_bedrock_messages_thinking_shape_follows_exact_bedrock_entry_flag(
local_model_cost_map, monkeypatch

35
tests/test_litellm/proxy/db/db_transaction_queue/test_pod_lock_manager.py
View file

@ -7,9 +7,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from fastapi.testclient import TestClient
sys.path.insert(
0, os.path.abspath("../../..")
) # Adds the parent directory to the system path
sys.path.insert(0, os.path.abspath("../../..")) # Adds the parent directory to the system path
from litellm.constants import DEFAULT_CRON_JOB_LOCK_TTL_SECONDS
from litellm.proxy.db.db_transaction_queue.pod_lock_manager import PodLockManager
@ -310,9 +308,7 @@ async def test_lock_takeover_race_condition(mock_redis):
@pytest.mark.asyncio
async def test_release_lock_uses_atomic_compare_delete_script_when_available(
pod_lock_manager, mock_redis
):
async def test_release_lock_uses_atomic_compare_delete_script_when_available(pod_lock_manager, mock_redis):
"""
Test that release_lock prefers atomic compare-and-delete Lua script when
redis cache exposes script registration.
@ -323,12 +319,8 @@ async def test_release_lock_uses_atomic_compare_delete_script_when_available(
await pod_lock_manager.release_lock(cronjob_id="test_job")
lock_key = pod_lock_manager.get_redis_lock_key(cronjob_id="test_job")
mock_redis.async_register_script.assert_called_once_with(
PodLockManager._COMPARE_AND_DELETE_LOCK_SCRIPT
)
script_callable.assert_called_once_with(
keys=[lock_key], args=[json.dumps(pod_lock_manager.pod_id)]
)
mock_redis.async_register_script.assert_called_once_with(PodLockManager._COMPARE_AND_DELETE_LOCK_SCRIPT)
script_callable.assert_called_once_with(keys=[lock_key], args=[json.dumps(pod_lock_manager.pod_id)])
mock_redis.async_get_cache.assert_not_called()
mock_redis.async_delete_cache.assert_not_called()
@ -359,9 +351,7 @@ async def test_release_lock_lua_path_emits_released_event(pod_lock_manager, mock
with patch.object(pod_lock_manager, "_emit_released_lock_event") as mock_emit:
await pod_lock_manager.release_lock(cronjob_id="test_job")
mock_emit.assert_called_once_with(
cronjob_id="test_job", pod_id=pod_lock_manager.pod_id
)
mock_emit.assert_called_once_with(cronjob_id="test_job", pod_id=pod_lock_manager.pod_id)
class FakeRedisLockStore:
@ -437,9 +427,7 @@ async def test_release_lock_preserves_lock_held_by_other_pod():
@pytest.mark.asyncio
async def test_release_lock_falls_back_to_get_del_when_lua_execution_fails(
pod_lock_manager, mock_redis
):
async def test_release_lock_falls_back_to_get_del_when_lua_execution_fails(pod_lock_manager, mock_redis):
"""
Test that release_lock falls back to GET+DEL when Lua script execution
raises (e.g. Redis restart cleared loaded scripts).
@ -457,3 +445,14 @@ async def test_release_lock_falls_back_to_get_del_when_lua_execution_fails(
mock_redis.async_delete_cache.assert_called_once_with(lock_key)
# Cached script handle should be reset so next call re-registers
assert pod_lock_manager._release_lock_script is None
@pytest.mark.asyncio
async def test_acquire_lock_own_lock_not_reentrant(pod_lock_manager, mock_redis):
"""With allow_reentrant=False a live lock means the window's work is done, so even
the holder gets False; the default stays reentrant for leader-election callers."""
mock_redis.async_set_cache.return_value = False
mock_redis.async_get_cache.return_value = pod_lock_manager.pod_id
assert await pod_lock_manager.acquire_lock(cronjob_id="test_job", allow_reentrant=False) is False
assert await pod_lock_manager.acquire_lock(cronjob_id="test_job") is True

64
tests/test_litellm/proxy/db/test_db_spend_update_writer.py
View file

@ -2157,3 +2157,67 @@ async def test_daily_transaction_compression_saved_tokens_zero_when_absent():
assert transaction["compression_saved_tokens"] == 0
assert transaction["compression_savings_spend"] == 0
assert transaction["prompt_caching_savings_spend"] == 0
@pytest.mark.asyncio
async def test_commit_spend_updates_to_db_does_not_stamp_key_settings_updated_at():
"""Spend flushes must leave settings_updated_at alone, or it decays into
another `updated_at` and stops being an audit signal."""
db_writer = DBSpendUpdateWriter()
mock_batcher = MagicMock()
mock_batcher.litellm_verificationtoken = MagicMock()
mock_batcher.litellm_verificationtoken.update_many = MagicMock()
mock_batcher.litellm_usertable = MagicMock()
mock_batcher.litellm_usertable.update_many = MagicMock()
mock_batcher.litellm_teamtable = MagicMock()
mock_batcher.litellm_teamtable.update_many = MagicMock()
mock_batcher.litellm_teammembership = MagicMock()
mock_batcher.litellm_teammembership.update_many = MagicMock()
mock_batcher.litellm_organizationtable = MagicMock()
mock_batcher.litellm_organizationtable.update_many = MagicMock()
mock_batcher.litellm_tagtable = MagicMock()
mock_batcher.litellm_tagtable.update_many = MagicMock()
mock_batcher.litellm_agentstable = MagicMock()
mock_batcher.litellm_agentstable.update_many = MagicMock()
mock_transaction = AsyncMock()
mock_transaction.__aenter__ = AsyncMock(return_value=mock_transaction)
mock_transaction.__aexit__ = AsyncMock(return_value=False)
mock_transaction.batch_ = MagicMock(
return_value=AsyncMock(
__aenter__=AsyncMock(return_value=mock_batcher),
__aexit__=AsyncMock(return_value=False),
)
)
mock_prisma_client = MagicMock()
mock_prisma_client.db = MagicMock()
mock_prisma_client.db.tx = MagicMock(return_value=mock_transaction)
token = "hashed-token-abc"
response_cost = 0.25
db_spend_update_transactions = {
"user_list_transactions": {},
"end_user_list_transactions": {},
"key_list_transactions": {token: response_cost},
"team_list_transactions": {},
"team_member_list_transactions": {},
"org_list_transactions": {},
"tag_list_transactions": {},
"agent_list_transactions": {},
}
with patch("litellm.proxy.utils._raise_failed_update_spend_exception"):
await db_writer._commit_spend_updates_to_db(
prisma_client=mock_prisma_client,
n_retry_times=0,
proxy_logging_obj=MagicMock(),
db_spend_update_transactions=db_spend_update_transactions,
)
mock_batcher.litellm_verificationtoken.update_many.assert_called_once()
call_kwargs = mock_batcher.litellm_verificationtoken.update_many.call_args[1]
assert call_kwargs["where"] == {"token": token}
assert set(call_kwargs["data"]) == {"spend", "last_active"}
assert call_kwargs["data"]["spend"] == {"increment": response_cost}

125
tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py
View file

@ -1733,6 +1733,21 @@ async def test_update_service_account_works_with_team_id():
await prepare_key_update_data(data=data, existing_key_row=existing_key)
@pytest.mark.asyncio
@pytest.mark.parametrize("flag_value", [True, False])
async def test_update_key_enable_prompt_caching_folds_into_metadata(flag_value):
"""Top-level enable_prompt_caching on /key/update lands in key metadata, including flipping back to False."""
data = UpdateKeyRequest(key="sk-1", enable_prompt_caching=flag_value)
existing_key = LiteLLM_VerificationToken(
token="hashed", metadata={"enable_prompt_caching": not flag_value}
)
updated = await prepare_key_update_data(data=data, existing_key_row=existing_key)
assert updated["metadata"]["enable_prompt_caching"] is flag_value
assert "enable_prompt_caching" not in {k for k in updated if k != "metadata"}
@pytest.mark.asyncio
async def test_update_preserves_service_account_id_when_metadata_replaced():
"""
@ -2293,9 +2308,9 @@ async def test_unblock_key_supports_both_sk_and_hashed_tokens(monkeypatch):
)
# Verify that the database update was called with hashed token
mock_prisma_client.db.litellm_verificationtoken.update.assert_called_with(
where={"token": test_hashed_token}, data={"blocked": False}
)
sk_token_call = mock_prisma_client.db.litellm_verificationtoken.update.call_args.kwargs
assert sk_token_call["where"] == {"token": test_hashed_token}
assert sk_token_call["data"]["blocked"] is False
assert result == mock_key_record
@ -2313,9 +2328,9 @@ async def test_unblock_key_supports_both_sk_and_hashed_tokens(monkeypatch):
)
# Verify that the database update was called with the same hashed token
mock_prisma_client.db.litellm_verificationtoken.update.assert_called_with(
where={"token": test_hashed_token}, data={"blocked": False}
)
hashed_token_call = mock_prisma_client.db.litellm_verificationtoken.update.call_args.kwargs
assert hashed_token_call["where"] == {"token": test_hashed_token}
assert hashed_token_call["data"]["blocked"] is False
assert result == mock_key_record
@ -2849,9 +2864,10 @@ async def test_block_key_existing_key_succeeds(monkeypatch):
mock_prisma_client.db.litellm_verificationtoken.find_unique.assert_called_once_with(
where={"token": test_hashed_token}
)
mock_prisma_client.db.litellm_verificationtoken.update.assert_called_once_with(
where={"token": test_hashed_token}, data={"blocked": True}
)
mock_prisma_client.db.litellm_verificationtoken.update.assert_called_once()
block_call = mock_prisma_client.db.litellm_verificationtoken.update.call_args.kwargs
assert block_call["where"] == {"token": test_hashed_token}
assert block_call["data"]["blocked"] is True
assert result == mock_updated_record
@ -4717,6 +4733,7 @@ def test_transform_verification_tokens_to_deleted_records():
user_role=LitellmUserRoles.PROXY_ADMIN.value,
)
config_stamp = datetime(2026, 8, 10, 12, 30, 45, tzinfo=timezone.utc)
key1 = LiteLLM_VerificationToken(
token="hashed-token-1",
user_id="user-123",
@ -4733,6 +4750,7 @@ def test_transform_verification_tokens_to_deleted_records():
model_spend={},
soft_budget_cooldown=False,
allowed_routes=[],
settings_updated_at=config_stamp,
)
key2 = LiteLLM_VerificationToken(
@ -4775,6 +4793,7 @@ def test_transform_verification_tokens_to_deleted_records():
assert record1["token"] == "hashed-token-1"
assert record1["user_id"] == "user-123"
assert record1["team_id"] == "team-456"
assert record1["settings_updated_at"] == config_stamp
assert isinstance(record1["aliases"], str)
assert isinstance(record1["config"], str)
assert isinstance(record1["permissions"], str)
@ -15817,3 +15836,91 @@ async def test_regenerate_key_output_token_estimate_lowered_rejected_for_non_adm
assert exc.value.status_code == 403
assert "Only proxy admins can set" in str(exc.value.detail)
@pytest.mark.asyncio
async def test_execute_virtual_key_regeneration_stamps_settings_updated_at():
"""Regenerate rewrites the key's config, so it must move settings_updated_at."""
from datetime import datetime, timezone
from litellm.proxy._types import RegenerateKeyRequest
from litellm.proxy.management_endpoints.key_management_endpoints import (
_execute_virtual_key_regeneration,
)
mock_prisma_client = _make_regenerate_mock_prisma()
with _patch_regenerate_side_effects():
before = datetime.now(timezone.utc)
await _execute_virtual_key_regeneration(
prisma_client=mock_prisma_client,
key_in_db=_make_regenerate_existing_key(),
hashed_api_key="abc123",
key="abc123",
data=RegenerateKeyRequest(max_budget=42.0),
user_api_key_dict=_make_regenerate_user_api_key_dict(),
litellm_changed_by=None,
user_api_key_cache=MagicMock(),
proxy_logging_obj=MagicMock(),
)
after = datetime.now(timezone.utc)
sent = mock_prisma_client.db.litellm_verificationtoken.update.call_args.kwargs["data"]
assert sent["max_budget"] == 42.0
assert before <= sent["settings_updated_at"] <= after
@pytest.mark.asyncio
async def test_block_key_stamps_settings_updated_at(monkeypatch):
"""Blocking a key is a config change, not spend activity."""
from datetime import datetime, timezone
from litellm.proxy._types import BlockKeyRequest
from litellm.proxy.management_endpoints.key_management_endpoints import block_key
mock_prisma_client, _ = _setup_block_unblock_mocks(monkeypatch)
before = datetime.now(timezone.utc)
await block_key(
data=BlockKeyRequest(key="sk-test123456789"),
http_request=MagicMock(),
user_api_key_dict=UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-admin",
user_id="admin_user",
),
litellm_changed_by=None,
)
after = datetime.now(timezone.utc)
sent = mock_prisma_client.db.litellm_verificationtoken.update.call_args.kwargs["data"]
assert sent["blocked"] is True
assert before <= sent["settings_updated_at"] <= after
@pytest.mark.asyncio
async def test_unblock_key_stamps_settings_updated_at(monkeypatch):
"""Unblocking a key is a config change, not spend activity."""
from datetime import datetime, timezone
from litellm.proxy._types import BlockKeyRequest
from litellm.proxy.management_endpoints.key_management_endpoints import unblock_key
mock_prisma_client, _ = _setup_block_unblock_mocks(monkeypatch)
before = datetime.now(timezone.utc)
await unblock_key(
data=BlockKeyRequest(key="sk-test123456789"),
http_request=MagicMock(),
user_api_key_dict=UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
api_key="sk-admin",
user_id="admin_user",
),
litellm_changed_by=None,
)
after = datetime.now(timezone.utc)
sent = mock_prisma_client.db.litellm_verificationtoken.update.call_args.kwargs["data"]
assert sent["blocked"] is False
assert before <= sent["settings_updated_at"] <= after

184
tests/test_litellm/proxy/proxy_server/test_lifecycle.py
View file

@ -22,6 +22,7 @@ import inspect
import json
import logging
import os
from collections.abc import Awaitable, Callable
from typing import List, Optional, Union
from unittest.mock import AsyncMock, MagicMock, patch
@ -397,9 +398,7 @@ def test__redact_worker_config_for_logging_masks_nested_secret_fields():
"database_url": nested_db_url,
"database_extra_connection_params": {"password": nested_extra_pw},
"alert_to_webhook_url": {"budget_alerts": nested_webhook},
"pass_through_endpoints": [
{"path": "/up", "headers": {"Authorization": nested_bearer}}
],
"pass_through_endpoints": [{"path": "/up", "headers": {"Authorization": nested_bearer}}],
}
}
}
@ -451,16 +450,13 @@ def test_load_from_azure_key_vault_disabled_no_side_effect(monkeypatch):
import litellm
sentinel_secret_mgr = object()
monkeypatch.setattr(
litellm, "secret_manager_client", sentinel_secret_mgr, raising=False
)
monkeypatch.setattr(litellm, "secret_manager_client", sentinel_secret_mgr, raising=False)
result = load_from_azure_key_vault(use_azure_key_vault=False)
observed = {
"return_value": result,
"secret_manager_unchanged": litellm.secret_manager_client
is sentinel_secret_mgr,
"secret_manager_unchanged": litellm.secret_manager_client is sentinel_secret_mgr,
"called_with": False,
}
assert normalize(observed) == {
@ -614,9 +610,7 @@ def test_get_litellm_model_info_uses_base_model_for_lookup(monkeypatch):
observed = {
"called_arg": (
fake_get.call_args.args[0]
if fake_get.call_args.args
else fake_get.call_args.kwargs.get("model")
fake_get.call_args.args[0] if fake_get.call_args.args else fake_get.call_args.kwargs.get("model")
),
"returned_max_tokens": result.get("max_tokens"),
"returned_cost": result.get("input_cost_per_token"),
@ -663,9 +657,7 @@ def test_run_ollama_serve_invokes_subprocess_popen(monkeypatch):
def test_run_ollama_serve_popen_failure_is_swallowed(monkeypatch):
"""Popen raising OSError must NOT propagate — function logs and returns."""
monkeypatch.setattr(
ps.subprocess, "Popen", MagicMock(side_effect=OSError("no ollama binary"))
)
monkeypatch.setattr(ps.subprocess, "Popen", MagicMock(side_effect=OSError("no ollama binary")))
result = run_ollama_serve()
assert result is None
@ -685,8 +677,7 @@ async def test_proxy_startup_event_is_async_context_manager_with_expected_signat
observed = {
"param_count": len(sig.parameters),
"has_app_param": "app" in sig.parameters,
"wrapped_is_async": inspect.iscoroutinefunction(wrapped)
or inspect.isasyncgenfunction(wrapped),
"wrapped_is_async": inspect.iscoroutinefunction(wrapped) or inspect.isasyncgenfunction(wrapped),
"has_asynccontextmanager_wrapper": wrapped is not None,
}
assert normalize(observed) == {
@ -777,3 +768,164 @@ def test_proxy_startup_event_warns_for_global_budget_without_database():
assert budget_check_pos < warn_pos < next_startup_section_pos, (
"DB-less budget warning must run after Prisma setup and the DB-backed budget block"
)
# ---------------------------------------------------------------------------
# _initialize_slack_alerting_jobs — spend-report pod locking (issue #14809)
# ---------------------------------------------------------------------------
SlackAlertingJobs = dict[str, Callable[[], Awaitable[None]]]
def _make_slack_alerting_proxy_logging(acquire_lock_result: bool | None) -> MagicMock:
proxy_logging_obj = MagicMock()
proxy_logging_obj.slack_alerting_instance.alerting = ["slack"]
proxy_logging_obj.slack_alerting_instance.send_weekly_spend_report = AsyncMock()
proxy_logging_obj.slack_alerting_instance.send_monthly_spend_report = AsyncMock()
proxy_logging_obj.slack_alerting_instance.send_fallback_stats_from_prometheus = AsyncMock()
pod_lock_manager = proxy_logging_obj.db_spend_update_writer.pod_lock_manager
pod_lock_manager.acquire_lock = AsyncMock(return_value=acquire_lock_result)
pod_lock_manager.release_lock = AsyncMock()
return proxy_logging_obj
async def _init_slack_alerting_jobs(
acquire_lock_result: bool | None,
spend_report_frequency: str = "7d",
) -> tuple[SlackAlertingJobs, MagicMock]:
scheduler = MagicMock()
proxy_logging_obj = _make_slack_alerting_proxy_logging(acquire_lock_result)
await ProxyStartupEvent._initialize_slack_alerting_jobs(
scheduler=scheduler,
general_settings={"spend_report_frequency": spend_report_frequency},
proxy_logging_obj=proxy_logging_obj,
prisma_client=MagicMock(),
)
jobs = {call.kwargs["id"]: call.args[0] for call in scheduler.add_job.call_args_list}
return jobs, proxy_logging_obj
@pytest.mark.parametrize("spend_report_frequency", ["0d", "-1d", "7h"])
@pytest.mark.asyncio
async def test_initialize_slack_alerting_jobs_invalid_frequency_raises(spend_report_frequency: str):
"""A non-positive window used to become an every-second APScheduler interval, and now also
computes a negative lock TTL that expires instantly and suppresses the report for good.
match= is load-bearing: drop the guard and "-1d" still raises, but from duration_in_seconds."""
with pytest.raises(ValueError, match="positive number of days"):
await _init_slack_alerting_jobs(
acquire_lock_result=True,
spend_report_frequency=spend_report_frequency,
)
@pytest.mark.asyncio
async def test_weekly_spend_report_skipped_when_another_pod_holds_the_lock():
"""regression: issue #14809 - every pod ran its own weekly spend report job."""
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=False)
await jobs["weekly_spend_report_job"]()
proxy_logging_obj.slack_alerting_instance.send_weekly_spend_report.assert_not_awaited()
proxy_logging_obj.db_spend_update_writer.pod_lock_manager.acquire_lock.assert_awaited_once_with(
cronjob_id="weekly_spend_report_job",
ttl=7 * 86400 - 3600,
allow_reentrant=False,
)
@pytest.mark.parametrize("acquire_lock_result", [True, None])
@pytest.mark.asyncio
async def test_weekly_spend_report_sent_when_the_lock_is_free_or_absent(acquire_lock_result):
"""None means redis isn't configured; a single-pod deploy must still report."""
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=acquire_lock_result)
await jobs["weekly_spend_report_job"]()
proxy_logging_obj.slack_alerting_instance.send_weekly_spend_report.assert_awaited_once_with("7d")
@pytest.mark.asyncio
async def test_weekly_spend_report_lock_ttl_tracks_the_configured_window():
"""TTL is the window less an hour: long enough that no second pod re-sends inside the
window, short enough that the lock is gone before the next one opens. A fixed TTL would
break one end or the other as soon as spend_report_frequency changes."""
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=True, spend_report_frequency="1d")
await jobs["weekly_spend_report_job"]()
proxy_logging_obj.db_spend_update_writer.pod_lock_manager.acquire_lock.assert_awaited_once_with(
cronjob_id="weekly_spend_report_job",
ttl=86400 - 3600,
allow_reentrant=False,
)
proxy_logging_obj.slack_alerting_instance.send_weekly_spend_report.assert_awaited_once_with("1d")
@pytest.mark.asyncio
async def test_monthly_spend_report_skipped_when_another_pod_holds_the_lock():
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=False)
await jobs["monthly_spend_report_job"]()
proxy_logging_obj.slack_alerting_instance.send_monthly_spend_report.assert_not_awaited()
proxy_logging_obj.db_spend_update_writer.pod_lock_manager.acquire_lock.assert_awaited_once_with(
cronjob_id="monthly_spend_report_job",
ttl=3600,
allow_reentrant=False,
)
@pytest.mark.parametrize("acquire_lock_result", [True, None])
@pytest.mark.asyncio
async def test_monthly_spend_report_sent_when_the_lock_is_free_or_absent(acquire_lock_result):
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=acquire_lock_result)
await jobs["monthly_spend_report_job"]()
proxy_logging_obj.slack_alerting_instance.send_monthly_spend_report.assert_awaited_once_with()
@pytest.mark.asyncio
async def test_spend_report_locks_are_never_released():
"""The lock is a per-window marker, not a mutex: releasing it lets the next pod re-send."""
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=True)
await jobs["weekly_spend_report_job"]()
await jobs["monthly_spend_report_job"]()
proxy_logging_obj.db_spend_update_writer.pod_lock_manager.release_lock.assert_not_awaited()
@pytest.mark.asyncio
async def test_prometheus_fallback_stats_job_skipped_when_another_pod_holds_the_lock(monkeypatch):
"""The boot-time send goes through the same gate, so a losing pod sends nothing at all:
startup and the scheduled job both stay at zero."""
monkeypatch.setenv("PROMETHEUS_URL", "http://prometheus.invalid")
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=False)
send_fallback_stats = proxy_logging_obj.slack_alerting_instance.send_fallback_stats_from_prometheus
assert send_fallback_stats.await_count == 0
await jobs["prometheus_fallback_stats_job"]()
assert send_fallback_stats.await_count == 0
proxy_logging_obj.db_spend_update_writer.pod_lock_manager.acquire_lock.assert_awaited_with(
cronjob_id="prometheus_fallback_stats_job",
ttl=3600,
allow_reentrant=False,
)
assert proxy_logging_obj.db_spend_update_writer.pod_lock_manager.acquire_lock.await_count == 2
@pytest.mark.parametrize("acquire_lock_result", [True, None])
@pytest.mark.asyncio
async def test_prometheus_fallback_stats_job_runs_when_the_lock_is_free_or_absent(monkeypatch, acquire_lock_result):
monkeypatch.setenv("PROMETHEUS_URL", "http://prometheus.invalid")
jobs, proxy_logging_obj = await _init_slack_alerting_jobs(acquire_lock_result=acquire_lock_result)
send_fallback_stats = proxy_logging_obj.slack_alerting_instance.send_fallback_stats_from_prometheus
assert send_fallback_stats.await_count == 1
await jobs["prometheus_fallback_stats_job"]()
assert send_fallback_stats.await_count == 2

274
tests/test_litellm/proxy/test_litellm_pre_call_utils.py
View file

@ -688,6 +688,7 @@ async def test_add_litellm_data_to_request_strips_user_control_fields():
"mock_response": "free response",
"mock_tool_calls": [{"id": "call_1"}],
"disable_global_guardrails": True,
"enable_prompt_caching": True,
"routing_decision": {"cause": "forged", "routed_model": "spoofed"},
"metadata": copy.deepcopy(malicious_metadata),
"litellm_metadata": copy.deepcopy(malicious_metadata),
@ -705,6 +706,7 @@ async def test_add_litellm_data_to_request_strips_user_control_fields():
assert "mock_response" not in updated
assert "mock_tool_calls" not in updated
assert "disable_global_guardrails" not in updated
assert "enable_prompt_caching" not in updated
assert "routing_decision" not in updated
stripped_keys = {
@ -741,6 +743,42 @@ async def test_add_litellm_data_to_request_strips_user_control_fields():
assert "pillar_response_headers" not in snapshot_body["metadata"]
@pytest.mark.asyncio
@pytest.mark.parametrize(
"key_value, expected",
[(True, True), (False, False), ("yes", None), (None, None)],
)
async def test_key_metadata_enable_prompt_caching_promoted_to_request_root(key_value, expected):
"""Key metadata enable_prompt_caching is stamped onto the request root (bools only), even when the client spoofs it."""
request_mock = MagicMock(spec=Request)
request_mock.url.path = "/v1/chat/completions"
request_mock.url = MagicMock()
request_mock.url.__str__.return_value = "http://localhost/v1/chat/completions"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
data = {
"model": "claude-sonnet-4-5",
"messages": [{"role": "user", "content": "hello"}],
"enable_prompt_caching": "spoofed-by-client",
}
key_metadata = {} if key_value is None else {"enable_prompt_caching": key_value}
updated = await add_litellm_data_to_request(
data=data,
request=request_mock,
user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key", metadata=key_metadata),
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert updated.get("enable_prompt_caching") == expected
@pytest.mark.asyncio
@pytest.mark.parametrize(
"control_field",
@ -6309,3 +6347,239 @@ class TestPromotedTraceControlFields:
assert "litellm_metadata" not in updated
assert updated["metadata"]["trace_id"] == "trace-1"
assert updated["metadata"]["session_id"] == "session-1"
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_inherited_tags_excludes_caller_tags():
"""inherited_tags must carry only what key/team/project policy contributed,
never anything the caller's own request (header/body) supplied, even when the
caller resubmits the identical value -- it's a snapshot taken before the
caller's own tags are merged in, not a set difference against caller_tags.
tag_based_routing.py's allow_fail_open relies on this so a caller can't strip
an inherited constraint's protection by resubmitting its exact value."""
request_mock = MagicMock(spec=Request)
request_mock.url.path = "/v1/chat/completions"
request_mock.url = MagicMock()
request_mock.url.__str__.return_value = "http://localhost/v1/chat/completions"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
data = {
"model": "gpt-3.5-turbo",
# Caller resubmits the exact value the key policy also contributes.
"tags": ["key-supplied"],
}
user_api_key_dict = UserAPIKeyAuth(
api_key="hashed-key",
user_id="real-user",
metadata={"tags": ["key-supplied"]},
team_metadata={"tags": ["team-supplied"]},
spend=0.0,
max_budget=100.0,
model_max_budget={},
team_spend=0.0,
team_max_budget=200.0,
)
updated = await add_litellm_data_to_request(
data=data,
request=request_mock,
user_api_key_dict=user_api_key_dict,
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert set(updated["metadata"]["tags"]) == {"key-supplied", "team-supplied"}
assert set(updated["metadata"]["inherited_tags"]) == {"key-supplied", "team-supplied"}
assert tuple(updated["metadata"]["caller_tags"]) == ("key-supplied",)
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_inherited_tags_survives_pre_auth_header_merge():
"""Regression: apply_client_tag_policy_pre_auth (run from user_api_key_auth,
for _tag_max_budget_check) merges the caller's x-litellm-tags header into the
same metadata.tags list this function later reads from -- before this
function ever runs. A snapshot-based inherited_tags would misattribute that
caller-controlled value as policy-backed; inherited_tags must instead be read
directly from key/team/project metadata, immune to whatever the pre-auth pass
already merged into "tags"."""
request_mock = MagicMock(spec=Request)
request_mock.url.path = "/v1/chat/completions"
request_mock.url = MagicMock()
request_mock.url.__str__.return_value = "http://localhost/v1/chat/completions"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json", "x-litellm-tags": "caller-invented-tag"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
data: dict = {"model": "gpt-3.5-turbo"}
user_api_key_dict = UserAPIKeyAuth(
api_key="hashed-key",
user_id="real-user",
metadata={"tags": ["key-supplied"]},
team_metadata={},
spend=0.0,
max_budget=100.0,
model_max_budget={},
team_spend=0.0,
team_max_budget=200.0,
)
# Simulate the real request pipeline: the pre-auth merge runs first, on the
# same data dict, before add_litellm_data_to_request is ever called.
LiteLLMProxyRequestSetup.apply_client_tag_policy_pre_auth(
request=request_mock,
request_data=data,
user_api_key_dict=user_api_key_dict,
)
assert data["metadata"]["tags"] == ["caller-invented-tag"]
updated = await add_litellm_data_to_request(
data=data,
request=request_mock,
user_api_key_dict=user_api_key_dict,
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert set(updated["metadata"]["tags"]) == {"caller-invented-tag", "key-supplied"}
assert updated["metadata"]["inherited_tags"] == ("key-supplied",)
assert updated["metadata"]["caller_tags"] == ("caller-invented-tag",)
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_caller_tags_excludes_key_and_team_tags():
"""caller_tags must carry only what the caller itself sent (header + body
tags), never anything merged in from key/team metadata, even though the
merged "tags" field (used for matching) legitimately contains all three."""
request_mock = MagicMock(spec=Request)
request_mock.url.path = "/v1/chat/completions"
request_mock.url = MagicMock()
request_mock.url.__str__.return_value = "http://localhost/v1/chat/completions"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
data = {
"model": "gpt-3.5-turbo",
"tags": ["caller-supplied"],
}
user_api_key_dict = UserAPIKeyAuth(
api_key="hashed-key",
user_id="real-user",
metadata={"tags": ["key-supplied"]},
team_metadata={"tags": ["team-supplied"]},
spend=0.0,
max_budget=100.0,
model_max_budget={},
team_spend=0.0,
team_max_budget=200.0,
)
updated = await add_litellm_data_to_request(
data=data,
request=request_mock,
user_api_key_dict=user_api_key_dict,
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert set(updated["metadata"]["tags"]) == {"caller-supplied", "key-supplied", "team-supplied"}
assert tuple(updated["metadata"]["caller_tags"]) == ("caller-supplied",)
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_caller_tags_includes_header_tags():
"""The x-litellm-tags header is as much a caller-controlled input as the
body's "tags" field; both must land in caller_tags."""
request_mock = MagicMock(spec=Request)
request_mock.url.path = "/v1/chat/completions"
request_mock.url = MagicMock()
request_mock.url.__str__.return_value = "http://localhost/v1/chat/completions"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json", "x-litellm-tags": "header-tag"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
data = {"model": "gpt-3.5-turbo"}
user_api_key_dict = UserAPIKeyAuth(
api_key="hashed-key",
user_id="real-user",
metadata={"tags": ["key-supplied"]},
team_metadata={},
spend=0.0,
max_budget=100.0,
model_max_budget={},
team_spend=0.0,
team_max_budget=200.0,
)
updated = await add_litellm_data_to_request(
data=data,
request=request_mock,
user_api_key_dict=user_api_key_dict,
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert set(updated["metadata"]["tags"]) == {"header-tag", "key-supplied"}
assert tuple(updated["metadata"]["caller_tags"]) == ("header-tag",)
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_caller_tags_empty_when_caller_sends_nothing():
"""caller_tags must be present (an empty tuple), not absent, when the caller
supplied no tags of their own -- an empty-but-present value tells
tag_based_routing.py's allow_fail_open that any required/excluded tag on the
request is entirely inherited, not that no origin information is available.
"""
request_mock = MagicMock(spec=Request)
request_mock.url.path = "/v1/chat/completions"
request_mock.url = MagicMock()
request_mock.url.__str__.return_value = "http://localhost/v1/chat/completions"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
data = {"model": "gpt-3.5-turbo"}
user_api_key_dict = UserAPIKeyAuth(
api_key="hashed-key",
user_id="real-user",
metadata={"tags": ["key-supplied"]},
team_metadata={},
spend=0.0,
max_budget=100.0,
model_max_budget={},
team_spend=0.0,
team_max_budget=200.0,
)
updated = await add_litellm_data_to_request(
data=data,
request=request_mock,
user_api_key_dict=user_api_key_dict,
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert updated["metadata"]["tags"] == ["key-supplied"]
assert updated["metadata"]["caller_tags"] == ()

1241
tests/test_litellm/proxy/test_proxy_server.py
View file

File diff suppressed because it is too large Load diff

Some files were not shown because too many files have changed in this diff Show more