scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 36

Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/ui-ux-patterns-audit-1d04bc

Browse source 
The auto-router tests moved onto accessible queries here while staging added a
"Lite" preset and a default-model pin, so the option-label expectations take
staging's list read through this branch's visibleOptions helper.

Staging's new pin tests reached for antd's internal classes, which the lint rule
this branch enables rejects. The edit-modal cases now read the rendered
selection through one selectedValueIn helper, and the clear-affordance click in
ComplexityRouterConfig keeps a reasoned suppression since antd marks that icon
aria-hidden.
This commit is contained in:
Yuneng Jiang 2026-08-17 14:14:08 -07:00
commit 75abe09f47
No known key found for this signature in database
1177 changed files with 22595 additions and 10375 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

36
.github/pull_request_template.md vendored
View file

@ -64,12 +64,36 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
## Screenshots / Proof of Fix
<!-- Include screenshots, screen recordings, or command (e.g., curl) + output demonstrating that your changes work as expected
The proof must be completely e2e with no mocks, using, for example, actual LLM calls costing real $. `pytest` commands are not enough
For bug fixes: show reproduction before the fix and passing behavior after
Include the commit hash each proof was captured at, for both the before and the after runs
If the change applies to all three LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), include proof for every single one of them, not just one
For new features: show the feature working end-to-end
For UI changes: include before/after screenshots -->
The proof must be completely e2e with no mocks, using actual LLM calls costing real $$$ if applicable. `pytest` commands are not enough
Show ONLY the latest run: capture Before at the merge base and After at the PR's current tip, and when new commits change behavior, replace this whole section with the fresh run instead of stacking it on top of older ones. The run must be up to date. As soon as a new commit is made and it makes this PR description's after sha stale (it's no longer tip of PR), you must re-run the QA
Structure the section exactly as below: Before and After one heading level below this section, each naming the commit hash it was captured at, one lower-level heading per case inside each, the same case names in the same order on both sides, and numbered steps (command, observed output) under every case, never loose prose; shared setup (config, payloads) goes above Before, and with a single case, drop the case headings and number the steps directly
### Before (<hash>)
#### <case 1>
1. ...
2. ...
#### <case 2>
1. ...
### After (<hash>)
#### <case 1>
1. ...
2. ...
#### <case 2>
1. ...
For bug fixes: Before shows the reproduction, After shows the same steps passing
For new features: Before shows the capability missing, After shows it working end-to-end
If the change applies to all three LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), make each endpoint its own case, not just one
For UI changes: before/after screenshots under the same headings -->
## Type

2
CLAUDE.md
View file

@ -53,6 +53,8 @@ When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-bud
`make check` (f.k.a. `make pre-commit`, which still works identically as an alias) saves its complete output to a log file in .git (overwriting previous logs) and prints that path as its first and last output lines. To inspect a run, read or grep that log instead of re-running the multi-minute checks just to see a different slice
`make check`, `make lint`, `scripts/pre_commit_lint.sh`, and the standalone budget gates (`scripts/ruff_strict_gate.py`, `scripts/type_discipline_gate.py`, `scripts/type_check_gate.py`) each hold one of 2 machine-wide slots, so when other sessions or worktrees on the same box are already running heavy work, yours prints "all N machine-wide slots are busy; queueing" and then stays quiet until a slot frees. Give the command a long timeout and let it wait rather than killing it, retrying it, or assuming it hung. Don't change the # of machine-wide slots or make it unlimited by setting `LITELLM_GATE_SLOTS=0`
If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing `reportAny` / `reportExplicitAny` closer to their basedpyright ceilings, just validate it in the caller with Pydantic (a model or `TypeAdapter` that returns the typed thing or raises will do) and then pass the now typed variable in
If you get an LIT001 or LIT002 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in `tuple()` / `MappingProxyType()` / `frozenset()` instead of seeding an empty `list`/`dict`/`set` and mutating it over time. Ideally, `# mutable-ok` is never used; reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason

23
Makefile
View file

@ -4,11 +4,11 @@
.PHONY: help test test-unit test-unit-llms test-unit-proxy-guardrails test-unit-proxy-core test-unit-proxy-misc \
test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
info lint lint-dev lint-checks format \
info lint lint-inner lint-dev lint-checks format \
lint-basedpyright lint-e2e-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
install-dev install-proxy-dev install-test-deps install-hooks \
install-helm-unittest check-circular-imports check-import-safety check pre-commit \
install-helm-unittest check-circular-imports check-import-safety check check-inner pre-commit \
lint-install lint-fetch-base bootstrap
# Default target
@ -52,10 +52,17 @@ help:
@echo " make test-proxy-unit-b - Run proxy_unit_tests (p-z, ~28 files)"
@echo " make test-integration - Run integration tests"
@echo " make test-unit-helm - Run helm unit tests"
@echo ""
@echo "Heavy targets (check, lint) queue for LITELLM_GATE_SLOTS machine-wide"
@echo "slots (default 2; 0 disables) so parallel sessions don't thrash one machine."
UV := uv
UV_RUN := $(UV) run --no-sync
# Machine-wide slot queue for the heavy targets below; python3 + stdlib only, so
# it runs before any venv exists. See scripts/gate_slot_lock.py.
GATE_SLOT_LOCK := python3 scripts/gate_slot_lock.py
LINT_DEP_INSTALL ?= install-dev
LINT_E2E_DEP_INSTALL ?= lint-install
LINT_DEP_BASE ?= lint-fetch-base
@ -73,6 +80,8 @@ info:
install-dev:
$(UV) sync --inexact --frozen
# Deliberately unqueued: provisioning is I/O bound, so it doesn't need one of the
# machine-wide slots the CPU-bound gates below share.
bootstrap:
$(UV) sync --inexact --frozen --extra proxy --group proxy-dev --group e2e-dev
$(UV_RUN) python scripts/prisma_generate_if_needed.py
@ -229,7 +238,10 @@ check-import-safety: $(LINT_DEP_INSTALL)
# does (merge-base with origin/litellm_internal_staging). Setup (env sync, Prisma client,
# base fetch) runs once up front; the checks themselves are independent, so a sub-make
# fans them out with -j and the fast ones finish under basedpyright's shadow.
lint: lint-install lint-fetch-base
lint:
@$(GATE_SLOT_LOCK) $(MAKE) lint-inner
lint-inner: lint-install lint-fetch-base
$(MAKE) -j $(LINT_JOBS) $(LINT_OUTPUT_SYNC) LINT_DEP_INSTALL= LINT_E2E_DEP_INSTALL= LINT_DEP_BASE= lint-checks
lint-checks: lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright lint-e2e-basedpyright check-circular-imports check-import-safety
@ -244,7 +256,10 @@ lint-dev: lint-format-changed check-circular-imports check-import-safety
# test-linting.yml (Python), test-litellm-ui-build.yml's frontend-lint (dashboard), and
# check-ui-api-types.yml (API-type drift), skipping any whose files aren't in scope.
# Not auto-installed as a git hook so it never slows an unrelated human commit.
check: bootstrap
check:
@$(GATE_SLOT_LOCK) $(MAKE) check-inner
check-inner: bootstrap
./scripts/pre_commit_lint.sh
pre-commit:

26
basedpyright-code-budget.json
View file

@ -1,9 +1,9 @@
{
"reportAny": {
"limit": 22947
"limit": 22343
},
"reportArgumentType": {
"limit": 2579
"limit": 2578
},
"reportAssignmentType": {
"limit": 323
@ -24,13 +24,13 @@
"limit": 19
},
"reportExplicitAny": {
"limit": 7312
"limit": 6991
},
"reportFunctionMemberAccess": {
"limit": 7
},
"reportGeneralTypeIssues": {
"limit": 157
"limit": 154
},
"reportIncompatibleMethodOverride": {
"limit": 56
@ -54,10 +54,10 @@
"limit": 0
},
"reportMissingParameterType": {
"limit": 5707
"limit": 5681
},
"reportMissingTypeArgument": {
"limit": 15642
"limit": 15609
},
"reportMissingTypeStubs": {
"limit": 40
@ -72,7 +72,7 @@
"limit": 0
},
"reportOptionalMemberAccess": {
"limit": 1069
"limit": 1061
},
"reportOptionalOperand": {
"limit": 0
@ -99,19 +99,19 @@
"limit": 0
},
"reportUnknownArgumentType": {
"limit": 44776
"limit": 44709
},
"reportUnknownLambdaType": {
"limit": 113
"limit": 112
},
"reportUnknownMemberType": {
"limit": 39237
"limit": 39154
},
"reportUnknownParameterType": {
"limit": 19969
"limit": 19947
},
"reportUnknownVariableType": {
"limit": 30881
"limit": 30772
},
"reportUnnecessaryCast": {
"limit": 117
@ -123,7 +123,7 @@
"limit": 5
},
"reportUnnecessaryIsInstance": {
"limit": 853
"limit": 851
},
"reportUntypedBaseClass": {
"limit": 0

32
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -3,6 +3,7 @@ Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if t
"""
from datetime import datetime, timedelta, timezone
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Dict, Final, List, Optional, Tuple
from litellm._logging import verbose_proxy_logger
@ -51,6 +52,33 @@ class CheckBatchCost:
# Cached after the first poll cycle. Once we know the column is absent we skip
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
self.batch_processed_support_confirmed: bool = False
@staticmethod
def _is_missing_batch_processed_column_error(err: Exception) -> bool:
message: Final = str(err).lower()
return "batch_processed" in message or "unknown column" in message or "does not exist" in message
async def confirm_batch_processed_support(self) -> None:
"""
Probe the batch_processed column before the proxy serves traffic, so the retrieve
path never sees an unconfirmed poller on a schema that has the column and accounts
inline for a batch the first poll cycle then accounts again.
"""
try:
await self.prisma_client.db.litellm_managedobjecttable.find_first(
where={"file_purpose": "batch", "batch_processed": False}
)
except Exception as probe_err:
if not self._is_missing_batch_processed_column_error(probe_err):
verbose_proxy_logger.debug(
f"CheckBatchCost: batch_processed probe failed, the poll cycle will confirm support: {probe_err}"
)
return
self._has_batch_processed_column = False
verbose_proxy_logger.warning("CheckBatchCost: batch_processed column not found, querying without it")
return
self.batch_processed_support_confirmed = True
async def _get_user_info(self, batch_id: str, user_id: Optional[str]) -> Dict[str, Any]:
"""
@ -537,6 +565,7 @@ class CheckBatchCost:
credentials = self.llm_router.get_deployment_credentials_with_provider(model_id) or {}
_file_content = await afile_content(
file_id=raw_output_file_id,
_litellm_internal_model_credentials=MappingProxyType(dict(credentials)),
**credentials,
)
@ -722,8 +751,9 @@ class CheckBatchCost:
take=MAX_OBJECTS_PER_POLL_CYCLE,
order={"created_at": "asc"},
)
self.batch_processed_support_confirmed = True
except Exception as query_err:
if "batch_processed" not in str(query_err).lower() and "unknown column" not in str(query_err).lower() and "does not exist" not in str(query_err).lower():
if not self._is_missing_batch_processed_column_error(query_err):
raise
# Permanent schema gap — cache the result so future cycles skip straight to fallback
self._has_batch_processed_column = False

7
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -54,6 +54,9 @@ from litellm.proxy.openai_files_endpoints.common_utils import (
normalize_mime_type_for_provider,
resolve_managed_output_file_model_name,
)
from litellm.proxy.pass_through_endpoints.llm_provider_handlers.batch_attribution import (
request_tags_from_metadata,
)
from litellm.types.llms.openai import ( # pyright: ignore[reportAttributeAccessIssue]
AllMessageValues,
AsyncCursorPage,
@ -1146,6 +1149,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
## Check if unified_file_id is in the response
unified_file_id = response._hidden_params.get("unified_file_id") # managed file id
unified_batch_id = response._hidden_params.get("unified_batch_id") # managed batch id
is_batch_create: Final = unified_file_id is not None
model_id = cast(Optional[str], response._hidden_params.get("model_id"))
model_name = cast(Optional[str], response._hidden_params.get("model_name"))
@ -1216,6 +1220,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
model_mappings={model_id: provider_file_id},
user_api_key_dict=user_api_key_dict,
)
request_metadata: Final = data.get("litellm_metadata")
await self.store_unified_object_id(
unified_object_id=response.id,
file_object=response,
@ -1223,6 +1228,8 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
model_object_id=original_response_id,
file_purpose="batch",
user_api_key_dict=user_api_key_dict,
request_tags=request_tags_from_metadata(request_metadata if isinstance(request_metadata, dict) else {}),
persist_attribution=is_batch_create,
)
# Only record batch creation metric on actual create (not retrieve/cancel).

84
enterprise/litellm_enterprise/proxy/management_endpoints/project_endpoints.py
View file

@ -11,7 +11,7 @@ Endpoints for /project operations
#### PROJECT MANAGEMENT ####
import json
from collections.abc import Mapping, Sequence
from collections.abc import Sequence
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends, HTTPException, Request
@ -29,7 +29,11 @@ from litellm.proxy.utils import PrismaClient, handle_exception_on_proxy
if TYPE_CHECKING:
from prisma import models as prisma_models
from prisma.actions import LiteLLM_TeamTableActions
from prisma.actions import (
LiteLLM_ProjectTableActions,
LiteLLM_TeamTableActions,
LiteLLM_VerificationTokenActions,
)
router = APIRouter()
@ -39,6 +43,27 @@ def _team_table(prisma_client: PrismaClient) -> "LiteLLM_TeamTableActions[prisma
return team_table
def _project_table(prisma_client: PrismaClient) -> "LiteLLM_ProjectTableActions[prisma_models.LiteLLM_ProjectTable]":
project_table: LiteLLM_ProjectTableActions[prisma_models.LiteLLM_ProjectTable] = (
prisma_client.db.litellm_projecttable
)
return project_table
def _verification_token_table(
prisma_client: PrismaClient,
) -> "LiteLLM_VerificationTokenActions[prisma_models.LiteLLM_VerificationToken]":
verification_token_table: LiteLLM_VerificationTokenActions[prisma_models.LiteLLM_VerificationToken] = (
prisma_client.db.litellm_verificationtoken
)
return verification_token_table
def _jsonified(prisma_client: PrismaClient, payload: dict[str, object]) -> dict[str, object]:
jsonified: dict[str, object] = prisma_client.jsonify_object(payload)
return jsonified
async def _check_user_permission_for_project(
user_api_key_dict: UserAPIKeyAuth,
team_id: str | None,
@ -137,7 +162,7 @@ def _check_team_project_limits(
# --- Validate project models are a subset of team models ---
project_models = data.models
team_models = team_object.models or []
team_models: list[str] = team_object.models or []
if project_models and len(team_models) > 0:
# If team has 'all-proxy-models', skip validation as it allows all models
if SpecialModelNames.all_proxy_models.value not in team_models:
@ -188,11 +213,11 @@ async def _create_budget_for_project(
) -> str:
"""Create a budget for the project and return budget_id."""
budget_params = LiteLLM_BudgetTable.model_fields.keys()
_json_data: Mapping[str, object] = data.json(exclude_none=True)
_json_data: dict[str, object] = data.model_dump(exclude_none=True)
_budget_data = {k: v for k, v in _json_data.items() if k in budget_params}
budget_row = LiteLLM_BudgetTable.model_validate(_budget_data)
new_budget = prisma_client.jsonify_object(budget_row.json(exclude_none=True))
new_budget = _jsonified(prisma_client, budget_row.model_dump(exclude_none=True))
_budget: prisma_models.LiteLLM_BudgetTable = await prisma_client.db.litellm_budgettable.create(
data={
@ -227,7 +252,7 @@ async def _set_project_object_permission(
return None
def _remove_budget_fields_from_project_data(project_data: dict) -> dict:
def _remove_budget_fields_from_project_data(project_data: dict[str, object]) -> dict[str, object]:
"""
Remove budget fields from project data.
Budget fields belong to LiteLLM_BudgetTable, not LiteLLM_ProjectTable.
@ -396,9 +421,7 @@ async def new_project(
data.project_id = str(uuid.uuid4())
else:
# Check if project_id already exists
existing_project = await prisma_client.db.litellm_projecttable.find_unique(
where={"project_id": data.project_id}
)
existing_project = await _project_table(prisma_client).find_unique(where={"project_id": data.project_id})
if existing_project is not None:
raise ProxyException(
message=f"Project id = {data.project_id} already exists. Please use a different project id.",
@ -423,11 +446,14 @@ async def new_project(
)
# Create project row (following organization_endpoints.py pattern)
project_row = LiteLLM_ProjectTable(
**data.json(exclude_none=True),
object_permission_id=object_permission_id,
created_by=user_api_key_dict.user_id or litellm_proxy_admin_name,
updated_by=user_api_key_dict.user_id or litellm_proxy_admin_name,
project_row_payload: dict[str, object] = data.model_dump(exclude_none=True)
project_row = LiteLLM_ProjectTable.model_validate(
{
**project_row_payload,
"object_permission_id": object_permission_id,
"created_by": user_api_key_dict.user_id or litellm_proxy_admin_name,
"updated_by": user_api_key_dict.user_id or litellm_proxy_admin_name,
}
)
for field in LiteLLM_ManagementEndpoint_MetadataFields:
@ -438,7 +464,7 @@ async def new_project(
value=getattr(data, field),
)
new_project_row = prisma_client.jsonify_object(project_row.json(exclude_none=True))
new_project_row = _jsonified(prisma_client, project_row.model_dump(exclude_none=True))
# Remove budget fields (following organization_endpoints.py pattern)
new_project_row = _remove_budget_fields_from_project_data(new_project_row)
@ -560,7 +586,7 @@ async def update_project(
# Fetch existing project
existing_project: (
prisma_models.LiteLLM_ProjectTable | None
) = await prisma_client.db.litellm_projecttable.find_unique(where={"project_id": data.project_id})
) = await _project_table(prisma_client).find_unique(where={"project_id": data.project_id})
if existing_project is None:
raise ProxyException(
@ -617,8 +643,7 @@ async def update_project(
)
# Prepare update data
update_data = data.json(exclude_none=True, exclude={"project_id"})
update_data = prisma_client.jsonify_object(update_data)
update_data = _jsonified(prisma_client, data.model_dump(exclude_none=True, exclude={"project_id"}))
update_data["updated_by"] = user_api_key_dict.user_id or litellm_proxy_admin_name
# Handle budget updates
@ -660,9 +685,10 @@ async def update_project(
# Handle metadata fields
for field in LiteLLM_ManagementEndpoint_MetadataFields:
if field in update_data:
if update_data.get("metadata") is None:
update_data["metadata"] = {}
update_data["metadata"][field] = update_data.pop(field)
existing_metadata = update_data.get("metadata")
metadata_dict: dict[str, object] = existing_metadata if isinstance(existing_metadata, dict) else {}
metadata_dict[field] = update_data.pop(field)
update_data["metadata"] = metadata_dict
# Remove budget fields (following organization_endpoints.py pattern)
update_data = _remove_budget_fields_from_project_data(update_data)
@ -748,11 +774,11 @@ async def delete_project(
detail={"error": "Only admins can delete projects"},
)
deleted_projects = []
deleted_projects: list[prisma_models.LiteLLM_ProjectTable | None] = []
for project_id in data.project_ids:
# Check if project exists
existing_project = await prisma_client.db.litellm_projecttable.find_unique(where={"project_id": project_id})
existing_project = await _project_table(prisma_client).find_unique(where={"project_id": project_id})
if existing_project is None:
raise ProxyException(
@ -765,7 +791,7 @@ async def delete_project(
# Check if there are any keys associated with this project
associated_keys: Sequence[
prisma_models.LiteLLM_VerificationToken
] = await prisma_client.db.litellm_verificationtoken.find_many(where={"project_id": project_id})
] = await _verification_token_table(prisma_client).find_many(where={"project_id": project_id})
if len(associated_keys) > 0:
raise ProxyException(
@ -778,7 +804,7 @@ async def delete_project(
# Delete the project
deleted_project: (
prisma_models.LiteLLM_ProjectTable | None
) = await prisma_client.db.litellm_projecttable.delete(where={"project_id": project_id})
) = await _project_table(prisma_client).delete(where={"project_id": project_id})
await delete_cached_project_object(
project_id=project_id,
@ -829,7 +855,7 @@ async def project_info(
)
# Fetch project
project: prisma_models.LiteLLM_ProjectTable | None = await prisma_client.db.litellm_projecttable.find_unique(
project: prisma_models.LiteLLM_ProjectTable | None = await _project_table(prisma_client).find_unique(
where={"project_id": project_id},
include={"litellm_budget_table": True, "object_permission": True},
)
@ -901,7 +927,7 @@ async def list_projects(
if user_api_key_has_admin_view(user_api_key_dict):
projects: Sequence[
prisma_models.LiteLLM_ProjectTable
] = await prisma_client.db.litellm_projecttable.find_many(
] = await _project_table(prisma_client).find_many(
include={"litellm_budget_table": True, "object_permission": True}
)
else:
@ -911,9 +937,9 @@ async def list_projects(
user_record: prisma_models.LiteLLM_UserTable | None = await prisma_client.db.litellm_usertable.find_unique(
where={"user_id": user_api_key_dict.user_id},
)
user_team_ids: Sequence[str] = user_record.teams if user_record is not None and user_record.teams else []
user_team_ids: list[str] = user_record.teams if user_record is not None and user_record.teams else []
projects = await prisma_client.db.litellm_projecttable.find_many(
projects = await _project_table(prisma_client).find_many(
where={"team_id": {"in": user_team_ids}},
include={"litellm_budget_table": True, "object_permission": True},
)

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.55"
version = "0.1.56"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.55"
version = "0.1.56"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.85"
version = "0.4.86"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.85"
version = "0.4.86"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

4
litellm/__init__.py
View file

@ -26,6 +26,7 @@ def _dev_env_hot_reload_enabled() -> bool:
if os.getenv("LITELLM_MODE", "DEV") == "DEV":
_dotenv.load_dotenv(override=_dev_env_hot_reload_enabled())
from collections.abc import Sequence
from typing import (
Any,
Callable,
@ -217,6 +218,9 @@ add_user_information_to_llm_headers: Optional[bool] = (
overwrite_user_with_key_hash: bool = (
False # force the outgoing `user` param to the hashed api key, so providers see a stable, tamper-proof id
)
bedrock_request_metadata_fields: Optional[Sequence[str]] = (
None # allow-list of `user_api_key_*` fields (+ `spend_logs_metadata`) sent as Bedrock `requestMetadata`
)
store_audit_logs = False # Enterprise feature, allow users to see audit logs
skip_system_message_in_guardrail: bool = False
skip_tool_message_in_guardrail: bool = False

10
litellm/_redis.py
View file

@ -67,12 +67,20 @@ def _init_arg_names(cls: type) -> frozenset[str]:
Keyword-only parameters are included, and the MRO is walked because redis-py splits a
connection's parameters between ``AbstractConnection`` and its concrete subclasses.
Each ``__init__`` is unwrapped before introspection: redis-py >= 7.4 decorates
``AbstractConnection.__init__`` with ``@deprecated_args``, whose wrapper is declared
``(self, *args, **kwargs)`` — introspecting the wrapper directly loses every real
parameter (``socket_timeout`` included), which silently emptied this allowlist and
dropped the socket timeouts from url-configured connections. ``inspect.unwrap``
follows the ``__wrapped__`` chain to the true signature and is a no-op on
undecorated ``__init__``s.
"""
return frozenset(
name
for klass in inspect.getmro(cls)
if klass is not object
for spec in (inspect.getfullargspec(klass.__init__),)
for spec in (inspect.getfullargspec(inspect.unwrap(klass.__init__)),)
for name in spec.args + spec.kwonlyargs
)

18
litellm/batches/batch_utils.py
View file

@ -5,6 +5,7 @@ from typing import Any, Final, Literal
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.get_litellm_params import AWS_CREDENTIAL_KWARGS_KEYS
from litellm.litellm_core_utils.llm_cost_calc.utils import parse_prompt_tokens_details
from litellm.types.llms.openai import Batch
from litellm.types.utils import CallTypes, ModelInfo, Usage
@ -58,6 +59,17 @@ async def _handle_completed_batch(
model_name: Optional model name
litellm_params: Optional litellm parameters containing credentials (api_key, api_base, etc.)
"""
# A completed batch whose request lines all failed has no output file - the
# results are written to a separate error_file_id and output_file_id is None.
# There is nothing to price or measure, so report an empty result set instead
# of calling _fetch_batch_output_file_content, which raises on a missing
# output file. Without this guard the logging worker crashes on every
# aretrieve_batch poll and the completed batch's zero-cost accounting is lost.
# The generic retrieval helper keeps raising for callers that explicitly ask
# for a missing output file.
if batch.output_file_id is None:
return 0.0, Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0), []
file_content = await _fetch_batch_output_file_content(batch, custom_llm_provider, litellm_params=litellm_params)
if (
@ -295,7 +307,7 @@ def _extract_file_access_credentials(litellm_params: dict | None) -> dict:
if litellm_params:
# List of credential keys that should be passed to file operations
credential_keys: Final = [
credential_keys: Final = (
"api_key",
"api_base",
"api_version",
@ -309,7 +321,9 @@ def _extract_file_access_credentials(litellm_params: dict | None) -> dict:
"bucket_name",
"timeout",
"max_retries",
]
"_litellm_internal_model_credentials",
*AWS_CREDENTIAL_KWARGS_KEYS,
)
for key in credential_keys:
if key in litellm_params:
credentials[key] = litellm_params[key]

13
litellm/batches/main.py
View file

@ -22,6 +22,7 @@ from openai.types.batch import BatchRequestCounts
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.get_litellm_params import add_trusted_model_credentials_to_litellm_params
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.anthropic.batches.handler import AnthropicBatchesHandler
from litellm.llms.azure.batches.handler import AzureBatchesAPI
@ -527,6 +528,7 @@ def retrieve_batch(
custom_llm_provider=custom_llm_provider,
**kwargs,
)
add_trusted_model_credentials_to_litellm_params(litellm_params, kwargs)
if litellm_logging_obj is not None:
litellm_logging_obj.update_from_kwargs(
kwargs=kwargs,
@ -824,7 +826,7 @@ def list_batches(
async def acancel_batch(
batch_id: str,
model: str | None = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
metadata: dict[str, str] | None = None,
extra_headers: dict[str, str] | None = None,
extra_body: dict[str, str] | None = None,
@ -870,7 +872,7 @@ async def acancel_batch(
def cancel_batch(
batch_id: str,
model: str | None = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] | str = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] | str = "openai",
metadata: dict[str, str] | None = None,
extra_headers: dict[str, str] | None = None,
extra_body: dict[str, str] | None = None,
@ -991,9 +993,14 @@ def cancel_batch(
timeout=timeout,
max_retries=optional_params.max_retries,
)
elif custom_llm_provider == "bedrock":
response = BedrockBatchesHandler.cancel_batch(
batch_id=batch_id,
**kwargs,
)
else:
raise litellm.exceptions.BadRequestError(
message=f"LiteLLM doesn't support {custom_llm_provider} for 'cancel_batch'. Only 'openai', 'azure', and 'vertex_ai' are supported.",
message=f"LiteLLM doesn't support {custom_llm_provider} for 'cancel_batch'. Only 'openai', 'azure', 'vertex_ai', and 'bedrock' are supported.",
model="n/a",
llm_provider=custom_llm_provider,
response=httpx.Response(

36
litellm/caching/redis_cache.py
View file

@ -49,7 +49,7 @@ if TYPE_CHECKING:
cluster_pipeline = ClusterPipeline
async_redis_client = Redis
async_redis_cluster_client = RedisCluster
Span = _Span | Any
Span = _Span
else:
pipeline = Any
cluster_pipeline = Any
@ -625,7 +625,11 @@ class RedisCache(BaseCache):
f"{self.namespace}-{hashlib.sha256(script.encode()).hexdigest()[:16]}"
)
async def run_script(keys: Sequence[str], args: Sequence[Any], client: Any = None) -> Any:
async def run_script(
keys: Sequence[str],
args: Sequence[str | bytes | int | float],
client: object = None,
) -> object:
async def execute() -> object:
executor: Callable[..., Awaitable[Any]] | None = litellm.in_memory_llm_clients_cache.get_cache(
key=script_cache_key
@ -650,7 +654,11 @@ class RedisCache(BaseCache):
if hasattr(_redis_client, "register_script"):
registered_script: Final = _redis_client.register_script(script)
async def standalone_executor(keys: Sequence[str], args: Sequence[Any], client: Any = None) -> Any:
async def standalone_executor(
keys: Sequence[str],
args: Sequence[str | bytes | int | float],
client: object = None,
) -> object:
namespaced_keys: Final = tuple(self.check_and_fix_namespace(key=key) for key in keys)
return await registered_script(keys=namespaced_keys, args=args, client=client)
@ -659,7 +667,11 @@ class RedisCache(BaseCache):
if hasattr(_redis_client, "script_load"):
script_sha: Final = _redis_client.script_load(script)
async def cluster_executor(keys: Sequence[str], args: Sequence[Any], client: Any = None) -> Any:
async def cluster_executor(
keys: Sequence[str],
args: Sequence[str | bytes | int | float],
client: object = None,
) -> object:
namespaced_keys: Final = tuple(self.check_and_fix_namespace(key=key) for key in keys)
return await _redis_client.evalsha(script_sha, len(namespaced_keys), *namespaced_keys, *args)
@ -757,7 +769,7 @@ class RedisCache(BaseCache):
async def _pipeline_helper(
self,
pipe: pipeline | cluster_pipeline,
cache_list: list[tuple[Any, Any]],
cache_list: Sequence[tuple[str, object]],
ttl: float | None,
) -> list:
"""
@ -783,7 +795,9 @@ class RedisCache(BaseCache):
return results
@_redis_circuit_breaker_guard
async def async_set_cache_pipeline(self, cache_list: list[tuple[Any, Any]], ttl: float | None = None, **kwargs):
async def async_set_cache_pipeline(
self, cache_list: Sequence[tuple[str, object]], ttl: float | None = None, **kwargs
):
"""
Use Redis Pipelines for bulk write operations
"""
@ -795,7 +809,7 @@ class RedisCache(BaseCache):
start_time: Final = time.time()
print_verbose(f"Set Async Redis Cache: key list: {cache_list}\nttl={ttl}, redis_version={self.redis_version}")
cache_value: Final[Any] = None
cache_value: Final = None
try:
async with _redis_client.pipeline(transaction=False) as pipe:
results: Final = await self._pipeline_helper(pipe, cache_list, ttl)
@ -1074,7 +1088,7 @@ class RedisCache(BaseCache):
# NON blocking - notify users Redis is throwing an exception
verbose_logger.error("litellm.caching.caching: get() - Got exception from REDIS: ", e)
def _run_redis_mget_operation(self, keys: list[str]) -> list[Any]:
def _run_redis_mget_operation(self, keys: list[str]) -> Sequence[bytes | str | None]:
"""
Wrapper to call `mget` on the redis client
@ -1082,7 +1096,7 @@ class RedisCache(BaseCache):
"""
return self.redis_client.mget(keys=keys)
async def _async_run_redis_mget_operation(self, keys: list[str]) -> list[Any]:
async def _async_run_redis_mget_operation(self, keys: list[str]) -> Sequence[bytes | str | None]:
"""
Wrapper to call `mget` on the redis client
@ -1115,7 +1129,7 @@ class RedisCache(BaseCache):
cache_key = self.check_and_fix_namespace(key=cache_key or "")
_keys.append(cache_key)
start_time: Final = time.time()
results: Final[list] = self._run_redis_mget_operation(keys=_keys)
results: Final = self._run_redis_mget_operation(keys=_keys)
end_time: Final = time.time()
_duration: Final = end_time - start_time
self.service_logger_obj.service_success_hook(
@ -1522,7 +1536,7 @@ class RedisCache(BaseCache):
async def async_rpush(
self,
key: str,
values: list[Any],
values: Sequence[str | bytes | int | float],
parent_otel_span: Span | None = None,
**kwargs,
) -> int:

9
litellm/constants.py
View file

@ -141,6 +141,8 @@ LITELLM_UI_ALLOW_HEADERS: Final = [
"x-litellm-semantic-filter",
"x-litellm-semantic-filter-tools",
"x-litellm-adaptive-router-model",
"x-litellm-applied-guardrails",
"x-litellm-guardrail-scan-id",
]
# Gemini model-specific minimal thinking budget constants
@ -1591,6 +1593,13 @@ DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL: Final = 10
# in a single ``/{name1,name2,...}/mcp`` URL. Bounds the per-request DB / cache
# fan-out an authenticated caller can trigger by stuffing the path with tokens.
DEFAULT_MCP_NAMESPACE_CSV_MAX_TOKENS: Final = 16
# Ceilings on the cached auth registries; larger tables fall back to per-row lookups
# instead of holding an unbounded id set in every worker.
TAG_REGISTRY_MAX_SIZE: Final = 5000
END_USER_RESTRICTED_REGISTRY_MAX_SIZE: Final = 5000
# How long a failed registry load is remembered as "unusable", so a degraded Postgres
# is not re-scanned on every request on top of the per-id lookups it falls back to.
REGISTRY_ERROR_NEGATIVE_CACHE_TTL: Final = 30
# Sentry Scrubbing Configuration
SENTRY_DENYLIST: Final = [

158
litellm/containers/main.py
View file

@ -1,9 +1,11 @@
import asyncio
import contextvars
import json
from collections.abc import Coroutine
from collections.abc import Coroutine, Mapping
from functools import partial
from typing import Any, Final, Literal, overload
from typing import Final, Literal, overload
import httpx
import litellm
from litellm.constants import request_timeout as DEFAULT_REQUEST_TIMEOUT
@ -48,16 +50,16 @@ __all__ = [
@client
async def acreate_container(
name: str,
expires_after: dict[str, Any] | None = None,
expires_after: Mapping[str, object] | None = None,
file_ids: list[str] | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
# LiteLLM specific params,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerObject:
"""Asynchronously calls the `create_container` function with the given arguments and keyword arguments.
@ -120,9 +122,9 @@ async def acreate_container(
@overload
def create_container(
name: str,
expires_after: dict[str, Any] | None = None,
expires_after: Mapping[str, object] | None = None,
file_ids: list[str] | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -130,16 +132,16 @@ def create_container(
*,
acreate_container: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, ContainerObject]:
) -> Coroutine[object, object, ContainerObject]:
...
@overload
def create_container(
name: str,
expires_after: dict[str, Any] | None = None,
expires_after: Mapping[str, object] | None = None,
file_ids: list[str] | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -156,20 +158,20 @@ def create_container(
@client
def create_container(
name: str,
expires_after: dict[str, Any] | None = None,
expires_after: Mapping[str, object] | None = None,
file_ids: list[str] | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerObject | Coroutine[Any, Any, ContainerObject]:
) -> ContainerObject | Coroutine[object, object, ContainerObject]:
"""Create a container using the OpenAI Container API.
Currently supports OpenAI
@ -281,13 +283,13 @@ async def alist_containers(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerListResponse:
"""Asynchronously list containers.
@ -351,7 +353,7 @@ def list_containers(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -359,7 +361,7 @@ def list_containers(
*,
alist_containers: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, ContainerListResponse]:
) -> Coroutine[object, object, ContainerListResponse]:
...
@ -368,7 +370,7 @@ def list_containers(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -387,18 +389,18 @@ def list_containers(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerListResponse | Coroutine[Any, Any, ContainerListResponse]:
) -> ContainerListResponse | Coroutine[object, object, ContainerListResponse]:
"""List containers using the OpenAI Container API.
Currently supports OpenAI
@ -481,13 +483,13 @@ def list_containers(
@client
async def aretrieve_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerObject:
"""Asynchronously retrieve a container.
@ -545,7 +547,7 @@ async def aretrieve_container(
@overload
def retrieve_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -553,14 +555,14 @@ def retrieve_container(
*,
aretrieve_container: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, ContainerObject]:
) -> Coroutine[object, object, ContainerObject]:
...
@overload
def retrieve_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -577,18 +579,18 @@ def retrieve_container(
@client
def retrieve_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerObject | Coroutine[Any, Any, ContainerObject]:
) -> ContainerObject | Coroutine[object, object, ContainerObject]:
"""Retrieve a container using the OpenAI Container API.
Currently supports OpenAI
@ -696,13 +698,13 @@ def retrieve_container(
@client
async def adelete_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> DeleteContainerResult:
"""Asynchronously delete a container.
@ -760,7 +762,7 @@ async def adelete_container(
@overload
def delete_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -768,14 +770,14 @@ def delete_container(
*,
adelete_container: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, DeleteContainerResult]:
) -> Coroutine[object, object, DeleteContainerResult]:
...
@overload
def delete_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -792,18 +794,18 @@ def delete_container(
@client
def delete_container(
container_id: str,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> DeleteContainerResult | Coroutine[Any, Any, DeleteContainerResult]:
) -> DeleteContainerResult | Coroutine[object, object, DeleteContainerResult]:
"""Delete a container using the OpenAI Container API.
Currently supports OpenAI
@ -914,11 +916,11 @@ async def alist_container_files(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerFileListResponse:
"""Asynchronously list files in a container.
@ -985,7 +987,7 @@ def list_container_files(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600,
timeout: float | httpx.Timeout = 600,
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -993,7 +995,7 @@ def list_container_files(
*,
alist_container_files: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, ContainerFileListResponse]:
) -> Coroutine[object, object, ContainerFileListResponse]:
...
@ -1003,7 +1005,7 @@ def list_container_files(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600,
timeout: float | httpx.Timeout = 600,
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -1023,16 +1025,16 @@ def list_container_files(
after: str | None = None,
limit: int | None = None,
order: str | None = None,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerFileListResponse | Coroutine[Any, Any, ContainerFileListResponse]:
) -> ContainerFileListResponse | Coroutine[object, object, ContainerFileListResponse]:
"""List files in a container using the OpenAI Container API.
Currently supports OpenAI
@ -1125,11 +1127,11 @@ def list_container_files(
async def aupload_container_file(
container_id: str,
file: FileTypes,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerFileObject:
"""Asynchronously upload a file to a container.
@ -1211,7 +1213,7 @@ async def aupload_container_file(
def upload_container_file(
container_id: str,
file: FileTypes,
timeout=600,
timeout: float | httpx.Timeout = 600,
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -1219,7 +1221,7 @@ def upload_container_file(
*,
aupload_container_file: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, ContainerFileObject]:
) -> Coroutine[object, object, ContainerFileObject]:
...
@ -1227,7 +1229,7 @@ def upload_container_file(
def upload_container_file(
container_id: str,
file: FileTypes,
timeout=600,
timeout: float | httpx.Timeout = 600,
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
@ -1245,16 +1247,16 @@ def upload_container_file(
def upload_container_file(
container_id: str,
file: FileTypes,
timeout=600, # default to 10 minutes
timeout: float | httpx.Timeout = 600, # default to 10 minutes
api_key: str | None = None,
api_base: str | None = None,
api_version: str | None = None,
custom_llm_provider: Literal["openai", "azure", "azure_text"] = "openai",
extra_headers: dict[str, Any] | None = None,
extra_query: dict[str, Any] | None = None,
extra_body: dict[str, Any] | None = None,
extra_headers: dict[str, object] | None = None,
extra_query: dict[str, object] | None = None,
extra_body: dict[str, object] | None = None,
**kwargs,
) -> ContainerFileObject | Coroutine[Any, Any, ContainerFileObject]:
) -> ContainerFileObject | Coroutine[object, object, ContainerFileObject]:
"""Upload a file to a container using the OpenAI Container API.
This endpoint allows uploading files directly to a container session,

16
litellm/files/main.py
View file

@ -11,7 +11,6 @@ import time
import uuid as uuid_module
from collections.abc import Coroutine
from functools import partial
from types import MappingProxyType
from typing import Any, Final, Literal, cast
import httpx
@ -34,6 +33,7 @@ import litellm
from litellm import get_secret_str
from litellm.files.streaming import FileContentStreamingResponse
from litellm.files.types import FileContentProvider, FileContentStreamingResult
from litellm.litellm_core_utils.get_litellm_params import add_trusted_model_credentials_to_litellm_params
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.common_utils import get_azure_credentials
@ -85,14 +85,6 @@ bedrock_files_instance: Final = BedrockFilesHandler()
#################################################
def _add_trusted_model_credentials_to_litellm_params(
litellm_params_dict: dict[str, Any], kwargs: dict[str, Any]
) -> None:
trusted_model_credentials: Final = kwargs.get("_litellm_internal_model_credentials")
if isinstance(trusted_model_credentials, type(MappingProxyType({}))):
litellm_params_dict["_litellm_internal_model_credentials"] = trusted_model_credentials
@client
async def acreate_file(
file: FileTypes,
@ -372,7 +364,7 @@ def file_retrieve(
)
if provider_config is not None:
litellm_params_dict: Final = get_litellm_params(**kwargs)
_add_trusted_model_credentials_to_litellm_params(
add_trusted_model_credentials_to_litellm_params(
litellm_params_dict=litellm_params_dict,
kwargs=kwargs,
)
@ -494,7 +486,7 @@ def file_delete(
pass
optional_params: Final = GenericLiteLLMParams(**kwargs)
litellm_params_dict: Final = get_litellm_params(**kwargs)
_add_trusted_model_credentials_to_litellm_params(
add_trusted_model_credentials_to_litellm_params(
litellm_params_dict=litellm_params_dict,
kwargs=kwargs,
)
@ -834,7 +826,7 @@ def file_content(
try:
optional_params: Final = GenericLiteLLMParams(**kwargs)
litellm_params_dict: Final = get_litellm_params(**kwargs)
_add_trusted_model_credentials_to_litellm_params(
add_trusted_model_credentials_to_litellm_params(
litellm_params_dict=litellm_params_dict,
kwargs=kwargs,
)

5
litellm/integrations/custom_guardrail.py
View file

@ -102,6 +102,9 @@ class CustomGuardrail(CustomLogger):
# If True, during_call runs async_moderation_hook instead of the unified apply_guardrail path.
use_native_during_call_hook: ClassVar[bool] = False
# If True, every proxy lifecycle event runs this guardrail's own hooks, not apply_guardrail.
use_native_lifecycle_hooks: ClassVar[bool] = False
records_own_guardrail_information: ClassVar[bool] = False
def __init__(
@ -632,7 +635,7 @@ class CustomGuardrail(CustomLogger):
return type(self).apply_guardrail is not CustomGuardrail.apply_guardrail
def _deployment_pre_call_target(self) -> "CustomLogger":
if not self.uses_apply_guardrail_interface():
if not self.uses_apply_guardrail_interface() or self.use_native_lifecycle_hooks:
return self
try:
from litellm.proxy.utils import unified_guardrail

324
litellm/integrations/shadow_eval_logger.py
View file

@ -1,5 +1,6 @@
"""Shadow Eval Logger: samples a shadowed key's successful chat requests, duplicates each
against the job's other arm in a detached task (the auto-router for a forward job, the
"""Shadow Eval Logger: samples a shadowed key's successful LLM requests (chat completions,
Anthropic Messages, and Responses API surfaces, each normalized to chat shape), duplicates
each against the job's other arm in a detached task (the auto-router for a forward job, the
fixed baseline model for a reverse one), blind-judges real vs shadow, and appends one
``LiteLLM_ShadowEvalAttempt`` row (verdict or error) as the feature's only hot-path write.
Counts, status, and spend derive from those rows at read time, so nothing can disagree
@ -16,7 +17,7 @@ from operator import itemgetter
from types import MappingProxyType
from typing import TYPE_CHECKING, Final
from pydantic import BaseModel, ConfigDict, ValidationError, field_validator, model_validator
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError, field_validator, model_validator
from litellm._logging import verbose_logger
from litellm.caching.in_memory_cache import InMemoryCache
@ -60,7 +61,240 @@ _MAX_ERROR_CHARS: Final = 500
_EMPTY_METADATA: Final[Mapping[str, object]] = MappingProxyType({})
_SAMPLED_CALL_TYPES: Final = frozenset({"completion", "acompletion"})
# Typed boundaries around the owner transformations, which declare untyped returns:
# a request or message that fails this lenient shape check is skipped, never sampled.
_CHAT_REQUEST_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_CHAT_MESSAGES_ADAPTER: Final = TypeAdapter(tuple[Mapping[str, object], ...])
_MESSAGE_ITEMS_ADAPTER: Final = TypeAdapter(tuple[object, ...])
def _chat_messages(kwargs: Mapping[str, object]) -> tuple[Mapping[str, object], ...]:
raw: Final = kwargs.get("messages")
return tuple(m for m in raw if isinstance(m, Mapping)) if isinstance(raw, Sequence) else ()
def _proxy_wire_body(kwargs: Mapping[str, object]) -> Mapping[str, object]:
litellm_params: Final = kwargs.get("litellm_params")
request: Final = litellm_params.get("proxy_server_request") if isinstance(litellm_params, Mapping) else None
body: Final = request.get("body") if isinstance(request, Mapping) else None
return body if isinstance(body, Mapping) else _EMPTY_METADATA
def _chat_request_from_chat(
kwargs: Mapping[str, object], model_parameters: Mapping[str, object]
) -> Mapping[str, object]:
"""Chat requests are already chat-shaped: the logged model_parameters forward as-is."""
return MappingProxyType({**model_parameters, "messages": _chat_messages(kwargs)})
# Anthropic params the adapter copies through untranslated; the translatable set comes
# from the adapter itself at call time.
_ANTHROPIC_SAMPLING_PARAM_KEYS: Final = frozenset(("max_tokens", "temperature", "top_p", "top_k", "reasoning_effort"))
def _chat_request_from_anthropic_messages(
kwargs: Mapping[str, object], _model_parameters: Mapping[str, object]
) -> Mapping[str, object]:
"""/v1/messages logs surface-native block messages with ``system`` top-level: the
native provider path carries it in kwargs, the openai-compatible bridge path only in
the proxy's snapshot of the client's wire body. Params come from the wire body alone,
because the logged optional_params switch dialect per provider path (the bridge's
inner completion rewrites them to chat shape mid-flight); the adapter translates
them alongside the messages, and sampling params copy through untranslated."""
from litellm.llms.anthropic.experimental_pass_through.adapters.transformation import (
LiteLLMAnthropicMessagesAdapter,
)
adapter: Final = LiteLLMAnthropicMessagesAdapter()
wire_body: Final = _proxy_wire_body(kwargs)
system: Final = kwargs.get("system") or wire_body.get("system")
param_keys: Final = (
frozenset(adapter.translatable_anthropic_params()) | _ANTHROPIC_SAMPLING_PARAM_KEYS
) - frozenset(("messages", "system"))
request: Final = MappingProxyType(
dict(
(
*((k, v) for k, v in wire_body.items() if k in param_keys),
("model", str(kwargs.get("model") or "")),
("messages", _CHAT_MESSAGES_ADAPTER.validate_python(kwargs.get("messages") or ())),
*((("system", system),) if system is not None else ()),
)
)
)
translated, _ = adapter.translate_anthropic_to_openai(request) # pyright: ignore[reportArgumentType] # wire-body mapping is the surface's native request shape; the adapter is duck-typed and read-only here
return translated
def _chat_request_from_responses(
kwargs: Mapping[str, object], _model_parameters: Mapping[str, object]
) -> Mapping[str, object]:
"""/v1/responses logs the raw ``input`` under ``kwargs["messages"]``, an alias
function_setup creates for responses call types: a bare string, chat-shaped dicts,
or item dicts; ``instructions`` is the system prompt. Params come from the wire body
for the same reason as the messages surface; the transformer translates them with
the input (max_output_tokens to max_tokens, Responses tools to chat tools, reasoning
to reasoning_effort) and never reads surface-only keys like previous_response_id."""
from litellm.responses.litellm_completion_transformation.transformation import (
LiteLLMCompletionResponsesConfig,
)
from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
wire_body: Final = _proxy_wire_body(kwargs)
instructions: Final = kwargs.get("instructions") or wire_body.get("instructions")
responses_request: Final = MappingProxyType(
dict(
(
*((k, v) for k, v in wire_body.items() if k in ResponsesAPIOptionalRequestParams.__annotations__),
*((("instructions", instructions),) if instructions is not None else ()),
)
)
)
return _CHAT_REQUEST_ADAPTER.validate_python(
LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request( # pyright: ignore[reportUnknownMemberType] # transformer declares a bare dict return
model=str(kwargs.get("model") or ""),
input=kwargs.get("messages"), # pyright: ignore[reportArgumentType] # untyped callback kwargs; transformer validates shapes
responses_api_request=responses_request, # pyright: ignore[reportArgumentType] # wire-body dict filtered to the surface's own request keys; the transformer is duck-typed
)
)
def _chat_final_text(response_obj: object) -> str:
"""The assistant's text, or empty when the turn carries tool calls: only text-final
turns produce a judgeable A/B comparison."""
try:
message: Final = (
response_obj["choices"][0]["message"]
if isinstance(response_obj, Mapping)
else response_obj.choices[0].message # pyright: ignore[reportAttributeAccessIssue] # duck-typed ModelResponse
)
except (AttributeError, KeyError, IndexError, TypeError):
return ""
read: Final = message.get if isinstance(message, Mapping) else lambda key: getattr(message, key, None)
if read("tool_calls") or read("function_call"):
return ""
return extract_text_from_content(read("content"))
def _responses_final_text(response_obj: object) -> str:
"""The turn's aggregated output text, or empty when the turn carries tool calls. A
dict-shaped payload is validated into the owner type first, because ``output_text``
is a derived property rather than a serialized field, so it never exists on a dict;
a dict the owner type rejects is unjudgeable and skipped."""
from litellm.types.llms.openai import ResponsesAPIResponse
try:
response: Final = (
ResponsesAPIResponse.model_validate(response_obj) if isinstance(response_obj, Mapping) else response_obj
)
except ValidationError:
return ""
output: Final = getattr(response, "output", None)
if not isinstance(output, Sequence):
return ""
items: Final = tuple(item.model_dump() if isinstance(item, BaseModel) else item for item in output)
if any(
not isinstance(item, Mapping) or item.get("type") in ("function_call", "custom_tool_call") for item in items
):
return ""
return str(getattr(response, "output_text", "") or "")
class _SurfaceOps:
"""One row per sampled call_type: how its logged request becomes a chat-shaped
request (messages plus translated generation params) and how its response yields
the judgeable final text. Membership in this table IS the sampling allowlist;
unknown call types fail closed. ``wire_params`` marks the surfaces whose params
come from the proxy's wire-body snapshot, which is taken before the guardrail
pre-call hook: those rows must not sample a request a pre-call guardrail rewrote,
or the shadow call would replay content (tools, unmasked entities) the guardrail
removed."""
__slots__ = ("chat_request", "final_text", "wire_params")
def __init__(
self,
chat_request: Callable[[Mapping[str, object], Mapping[str, object]], Mapping[str, object]],
final_text: Callable[[object], str],
wire_params: bool,
) -> None:
self.chat_request = chat_request
self.final_text = final_text
self.wire_params = wire_params
_CHAT_OPS: Final = _SurfaceOps(_chat_request_from_chat, _chat_final_text, wire_params=False)
_ANTHROPIC_OPS: Final = _SurfaceOps(_chat_request_from_anthropic_messages, _chat_final_text, wire_params=True)
_RESPONSES_OPS: Final = _SurfaceOps(_chat_request_from_responses, _responses_final_text, wire_params=True)
# Guardrail hooks that never rewrite the outbound request: they run in parallel with
# the call, on the response, or on logged copies. Anything else (pre_call, pre_mcp_call,
# a future mode) counts as request-mutating, failing closed.
_NON_MUTATING_GUARDRAIL_MODES: Final = frozenset(
("during_call", "post_call", "logging_only", "during_mcp_call", "post_mcp_call", "realtime_input_transcription")
)
def _request_mutating_guardrail_ran(request_metadata: Mapping[str, object]) -> bool:
"""Whether a guardrail that can rewrite the outbound request ran on this one, read
from the same guardrail-information entries spend logging uses. str-enum modes
compare equal to their plain-string values, and an entry whose mode is missing or
unrecognized counts as mutating."""
raw: Final = request_metadata.get("standard_logging_guardrail_information")
entries: Final = raw if isinstance(raw, Sequence) else ()
modes_per_entry: Final = tuple(entry.get("guardrail_mode") for entry in entries if isinstance(entry, Mapping))
return any(
not all(
mode in _NON_MUTATING_GUARDRAIL_MODES for mode in (modes if isinstance(modes, list | tuple) else (modes,))
)
for modes in modes_per_entry
)
# Translated-request keys that never forward to the shadow call: identity and transport,
# not generation. Empty-list values (e.g. tools) carry nothing and are dropped with them.
_UNFORWARDED_REQUEST_KEYS: Final = frozenset(("model", "messages", "stream", "stream_options", "metadata"))
def _forwards_nothing(value: object) -> bool:
return value is None or (isinstance(value, list) and len(value) == 0)
def _judgeable_sample(
ops: _SurfaceOps,
kwargs: Mapping[str, object],
model_parameters: Mapping[str, object],
response_obj: object,
) -> tuple[tuple[Mapping[str, object], ...], Mapping[str, object], str] | None:
"""The normalized chat conversation, the forwardable generation params, and the
judgeable final text; None when this request's shapes cannot be sampled (tool-final
turn, empty text, or a shape the owner transformations reject)."""
try:
request: Final = ops.chat_request(kwargs, model_parameters)
items: Final = _MESSAGE_ITEMS_ADAPTER.validate_python(request.get("messages"))
messages: Final = _CHAT_MESSAGES_ADAPTER.validate_python(
tuple(m.model_dump(exclude_none=True) if isinstance(m, BaseModel) else m for m in items)
)
except Exception as e: # noqa: BLE001 # a rejected shape is skipped, never sampled
verbose_logger.debug("shadow_eval: request normalization failed, skipping: %s", e)
return None
real_text: Final = ops.final_text(response_obj)
if not messages or not real_text:
return None
params: Final = MappingProxyType(
{k: v for k, v in request.items() if k not in _UNFORWARDED_REQUEST_KEYS and not _forwards_nothing(v)}
)
return messages, params, real_text
_SURFACE_OPS: Final[Mapping[str, _SurfaceOps]] = MappingProxyType(
{
"completion": _CHAT_OPS,
"acompletion": _CHAT_OPS,
"anthropic_messages": _ANTHROPIC_OPS,
"aresponses": _RESPONSES_OPS,
"responses": _RESPONSES_OPS,
}
)
PAIRWISE_JUDGE_SYSTEM_PROMPT: Final = """You are an impartial quality judge comparing two responses to the same conversation.
@ -361,25 +595,36 @@ class ShadowEvalLogger(CustomLogger):
request_id: Final = payload.get("id") or ""
if not request_id:
return
if payload.get("call_type") not in _SAMPLED_CALL_TYPES:
return # only known chat-shaped traffic is comparable; unknown or missing types fail closed
raw_messages: Final = kwargs.get("messages")
messages: Final = (
tuple(m for m in raw_messages if isinstance(m, Mapping)) if isinstance(raw_messages, Sequence) else ()
)
control_tier: Final = _routed_tier(request_metadata)
ops: Final = _SURFACE_OPS.get(str(payload.get("call_type") or ""))
if ops is None:
return # only surfaces this table can normalize are comparable; unknown types fail closed
if ops.wire_params and _request_mutating_guardrail_ran(request_metadata):
return # the wire-body snapshot predates the rewrite; replaying it would resurrect stripped content
# A key can hold one job per direction, and a request routed by one job's
# router while bypassing the other's qualifies for both. Each is separately
# budgeted, so both fire.
for job in (await self._active_jobs()).get(str(api_key_hash), ()):
if datetime.now(timezone.utc) >= job.ends_at:
continue
if job.attempts + self._job_starts.get(job.id, 0) >= job.max_turns:
continue
if not _sample_hits(request_id, job.id, job.shadow_percentage):
continue
if _request_was_routed_by(request_metadata, job.router_name) != (job.direction == "reverse"):
continue
# budgeted, so both fire; the request is normalized once, and only when at
# least one job sampled it.
eligible: Final = tuple(
job
for job in (await self._active_jobs()).get(str(api_key_hash), ())
if datetime.now(timezone.utc) < job.ends_at
and job.attempts + self._job_starts.get(job.id, 0) < job.max_turns
and _sample_hits(request_id, job.id, job.shadow_percentage)
and _request_was_routed_by(request_metadata, job.router_name) == (job.direction == "reverse")
)
if not eligible:
return
sample: Final = _judgeable_sample(
ops,
kwargs,
MappingProxyType(dict(payload.get("model_parameters") or {})), # mutable-ok: frozen snapshot
response_obj,
)
if sample is None:
return
messages, shadow_params, real_text = sample
control_tier: Final = _routed_tier(request_metadata)
for job in eligible:
if self._inflight_shadow_tasks >= _MAX_CONCURRENT_SHADOW_TASKS:
return
self._job_starts[job.id] = self._job_starts.get(job.id, 0) + 1
@ -389,12 +634,10 @@ class ShadowEvalLogger(CustomLogger):
job=job,
request_id=request_id,
messages=messages,
response_obj=response_obj,
real_text=real_text,
real_model=payload.get("model") or "",
control_tier=control_tier,
model_parameters=MappingProxyType(
dict(payload.get("model_parameters") or {}) # mutable-ok: frozen snapshot
),
shadow_params=shadow_params,
parent_metadata=MappingProxyType(dict(request_metadata)), # mutable-ok: frozen snapshot
)
).add_done_callback(self._release_shadow_slot)
@ -411,10 +654,10 @@ class ShadowEvalLogger(CustomLogger):
job: ActiveShadowEvalJob,
request_id: str,
messages: Sequence[Mapping[str, object]],
response_obj: object,
real_text: str,
real_model: str,
control_tier: str | None,
model_parameters: Mapping[str, object],
shadow_params: Mapping[str, object],
parent_metadata: Mapping[str, object],
) -> None:
"""Budget gate -> shadow call -> blind judge -> one attempt row. The prisma gate
@ -424,15 +667,10 @@ class ShadowEvalLogger(CustomLogger):
try:
if prisma is None:
return
real_text: Final = self._extract_response_text(response_obj)
if not real_text or not messages:
return
if await _key_or_team_is_over_budget(parent_metadata):
return
shadow: Final = await self._call_router_shadow(
job.shadow_target, messages, model_parameters, parent_metadata
)
shadow: Final = await self._call_router_shadow(job.shadow_target, messages, shadow_params, parent_metadata)
if isinstance(shadow, _CallFailure):
await self._record_attempt(prisma, job, request_id, control_tier, outcome="error", error=shadow.error)
return
@ -510,7 +748,7 @@ class ShadowEvalLogger(CustomLogger):
self,
target_model: str,
messages: Sequence[Mapping[str, object]],
model_parameters: Mapping[str, object],
shadow_params: Mapping[str, object],
parent_metadata: Mapping[str, object],
) -> "_ShadowResponse | _CallFailure":
"""Send the prompt through the arm nobody was served: the auto-router under
@ -523,9 +761,6 @@ class ShadowEvalLogger(CustomLogger):
shadow_metadata: Final[dict[str, object]] = ( # mutable-ok: router writes its routing decision back
sanitized_forwardable_call_metadata(parent_metadata, SHADOW_EVAL_ROUTER_CALL_ORIGIN)
)
shadow_params: Final = { # mutable-ok: splatted as kwargs
k: v for k, v in model_parameters.items() if k not in ("stream", "metadata")
}
try:
response: Final = await router.acompletion(
model=target_model,
@ -538,7 +773,7 @@ class ShadowEvalLogger(CustomLogger):
except Exception as e: # noqa: BLE001 # provider errors become error rows, not crashes
verbose_logger.debug("shadow_eval: router call failed: %s", e)
return _CallFailure(f"shadow router call failed: {e}")
text: Final = self._extract_response_text(response)
text: Final = _chat_final_text(response)
if not text:
return _CallFailure("shadow router returned an empty response")
return _ShadowResponse(
@ -597,19 +832,6 @@ class ShadowEvalLogger(CustomLogger):
cost=_judge_call_cost(response),
)
@staticmethod
def _extract_response_text(response_obj: object) -> str:
"""Extract the assistant's text from a ModelResponse-shaped object or dict."""
try:
content: Final = (
response_obj["choices"][0]["message"]["content"]
if isinstance(response_obj, Mapping)
else response_obj.choices[0].message.content # pyright: ignore[reportAttributeAccessIssue] # duck-typed ModelResponse
)
except (AttributeError, KeyError, IndexError, TypeError):
return ""
return extract_text_from_content(content)
_EMPTY_JOBS: Final[Mapping[str, tuple[ActiveShadowEvalJob, ...]]] = MappingProxyType({})

18
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -1,3 +1,5 @@
from collections.abc import Mapping, MutableMapping
from types import MappingProxyType
from typing import Final
from litellm.llms.openai.data_residency import infer_openai_data_residency
@ -184,3 +186,19 @@ def get_litellm_params(
litellm_params[key] = kwargs[key]
return litellm_params
def add_trusted_model_credentials_to_litellm_params(
litellm_params_dict: MutableMapping[str, object], kwargs: Mapping[str, object]
) -> None:
"""
Carry the immutable server-side credential snapshot into litellm_params.
get_litellm_params has a fixed signature, so callers that need the snapshot to
survive into the logging object and the downstream file read have to re-add it. Only
a MappingProxyType is accepted, since providers resolve trusted configuration such
as a Bedrock file bucket from it and must not read a request-supplied mapping.
"""
trusted_model_credentials: Final = kwargs.get("_litellm_internal_model_credentials")
if isinstance(trusted_model_credentials, MappingProxyType):
litellm_params_dict["_litellm_internal_model_credentials"] = trusted_model_credentials

42
litellm/litellm_core_utils/litellm_logging.py
View file

@ -13,6 +13,7 @@ import traceback
from collections.abc import Callable, Mapping, Sequence
from datetime import datetime as dt_object
from functools import lru_cache
from types import TracebackType
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast
from httpx import Response
@ -1189,6 +1190,7 @@ class Logging(LiteLLMLoggingBaseClass):
self.model_call_details["additional_args"] = additional_args
self.model_call_details["log_event_type"] = "post_api_call"
attr: Literal["warning", "debug"]
if self.litellm_request_debug:
attr = "warning"
else:
@ -1802,7 +1804,7 @@ class Logging(LiteLLMLoggingBaseClass):
if self.model_call_details.get("litellm_params") is None:
return
metadata_hidden_params: Final = hidden_params.copy()
response_cost: Final = self.model_call_details.get("response_cost")
response_cost: Final[object] = self.model_call_details.get("response_cost")
if metadata_hidden_params.get("response_cost") is None and response_cost is not None:
metadata_hidden_params["response_cost"] = response_cost
@ -1844,7 +1846,10 @@ class Logging(LiteLLMLoggingBaseClass):
logging_result, start_time, end_time
)
if (standard_logging_payload := self.model_call_details.get("standard_logging_object")) is not None:
standard_logging_payload: Final[StandardLoggingPayload | None] = self.model_call_details.get(
"standard_logging_object"
)
if standard_logging_payload is not None:
emit_standard_logging_payload(standard_logging_payload)
def _build_standard_logging_payload(
@ -2109,7 +2114,7 @@ class Logging(LiteLLMLoggingBaseClass):
def _success_handler_body(
self,
result: Any = None, # heterogeneous response object; varies by call type (ANN401 ignored, see ruff-strict.toml)
result: object = None,
start_time: datetime.datetime | None = None,
end_time: datetime.datetime | None = None,
cache_hit: bool | None = None,
@ -2150,7 +2155,10 @@ class Logging(LiteLLMLoggingBaseClass):
self.model_call_details["standard_logging_object"] = self._build_standard_logging_payload(
complete_streaming_response, start_time, end_time
)
if (standard_logging_payload := self.model_call_details.get("standard_logging_object")) is not None:
standard_logging_payload: Final[StandardLoggingPayload | None] = self.model_call_details.get(
"standard_logging_object"
)
if standard_logging_payload is not None:
# Only emit for sync requests (async_success_handler handles async)
if is_sync_request:
emit_standard_logging_payload(standard_logging_payload)
@ -2981,7 +2989,7 @@ class Logging(LiteLLMLoggingBaseClass):
global_callbacks=litellm.failure_callback,
)
result = None # result sent to all loggers, init this to None incase it's not created
result: object = None # result sent to all loggers, init this to None incase it's not created
result = redact_message_input_output_from_logging(
model_call_details=(self.model_call_details if hasattr(self, "model_call_details") else {}),
@ -3395,11 +3403,11 @@ class Logging(LiteLLMLoggingBaseClass):
def _get_assembled_streaming_response(
self,
result: ModelResponse | TextCompletionResponse | ModelResponseStream | ResponseCompletedEvent | Any,
result: ModelResponse | TextCompletionResponse | ModelResponseStream | ResponseCompletedEvent | object,
start_time: datetime.datetime,
end_time: datetime.datetime,
is_async: bool,
streaming_chunks: list[Any],
streaming_chunks: list[object],
) -> ModelResponse | TextCompletionResponse | ResponsesAPIResponse | None:
if self.stream is not True:
return None
@ -3677,9 +3685,7 @@ def set_callbacks(callback_list, function_id=None):
from sentry_sdk.scrubber import EventScrubber
sentry_sdk_instance = sentry_sdk
sentry_trace_rate = (
os.environ.get("SENTRY_API_TRACE_RATE") if "SENTRY_API_TRACE_RATE" in os.environ else "1.0"
)
sentry_trace_rate = os.environ.get("SENTRY_API_TRACE_RATE", "1.0")
sentry_sample_rate = (
os.environ.get("SENTRY_API_SAMPLE_RATE") if "SENTRY_API_SAMPLE_RATE" in os.environ else "1.0"
)
@ -5150,13 +5156,13 @@ class StandardLoggingPayloadSetup:
# ProxyException uses .code, LiteLLM exceptions use .status_code,
# httpx.HTTPStatusError exposes status only as .response.status_code.
# Stringified for Prisma JSON compatibility.
error_code_attr: Final = getattr(original_exception, "code", None)
error_code_attr: Final[object] = getattr(original_exception, "code", None)
if error_code_attr is not None and str(error_code_attr) not in ("", "None"):
error_status: str = str(error_code_attr)
else:
status_code_attr = getattr(original_exception, "status_code", None)
status_code_attr: object = getattr(original_exception, "status_code", None)
if status_code_attr is None:
response_attr: Final = getattr(original_exception, "response", None)
response_attr: Final[object] = getattr(original_exception, "response", None)
status_code_attr = getattr(response_attr, "status_code", None)
error_status = str(status_code_attr) if status_code_attr is not None else ""
error_class: Final[str] = str(original_exception.__class__.__name__) if original_exception else ""
@ -5165,7 +5171,7 @@ class StandardLoggingPayloadSetup:
# Get traceback information (first 100 lines)
traceback_info = traceback_str or ""
if original_exception:
tb: Final = getattr(original_exception, "__traceback__", None)
tb: Final[TracebackType | None] = getattr(original_exception, "__traceback__", None)
if tb:
tb_lines: Final = traceback.format_tb(tb)
traceback_info += "".join(tb_lines[:MAXIMUM_TRACEBACK_LINES_TO_LOG]) # Limit to first 100 lines
@ -5276,11 +5282,11 @@ class StandardLoggingPayloadSetup:
"""
dynamic_litellm_session_id: Final = litellm_params.get("litellm_session_id")
dynamic_litellm_trace_id: Final = litellm_params.get("litellm_trace_id")
metadata: Final = litellm_params.get("metadata")
metadata: Final[Mapping[str, object] | None] = litellm_params.get("metadata")
metadata_session_id: Final = metadata.get("session_id") if metadata else None
metadata_trace_id: Final = metadata.get("trace_id") if metadata else None
ordered_candidates: Final[tuple[Any, Any, Any, Any]] = (
ordered_candidates: Final[tuple[object, object, object, object]] = (
(dynamic_litellm_trace_id, dynamic_litellm_session_id, metadata_trace_id, metadata_session_id)
if litellm.request_correlation_in_logs
else (dynamic_litellm_session_id, dynamic_litellm_trace_id, metadata_session_id, metadata_trace_id)
@ -5305,10 +5311,10 @@ class StandardLoggingPayloadSetup:
"""
if not litellm.request_correlation_in_logs:
return ""
dynamic_litellm_session_id: Final = litellm_params.get("litellm_session_id")
dynamic_litellm_session_id: Final[object] = litellm_params.get("litellm_session_id")
if dynamic_litellm_session_id:
return str(dynamic_litellm_session_id)
metadata: Final = litellm_params.get("metadata")
metadata: Final[Mapping[str, object] | None] = litellm_params.get("metadata")
metadata_session_id: Final = metadata.get("session_id") if metadata else None
if metadata_session_id:
return str(metadata_session_id)

6
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -1004,7 +1004,7 @@ def _has_legacy_defs(schema: object) -> bool:
return "definitions" in schema or (isinstance(components, dict) and isinstance(components.get("schemas"), dict))
# Schema-bomb budget for ``unpack_legacy_defs``: cap the cumulative JSON-byte
# Schema-bomb budget for ``$ref`` inlining: cap the cumulative JSON-byte
# size of every inlined target. A byte cap is the universal measure of
# expansion -- it simultaneously bounds ref-count fan-out, node-count
# amplification, and scalar-byte amplification (large ``description`` /
@ -1012,14 +1012,14 @@ def _has_legacy_defs(schema: object) -> bool:
# inline well under 1MB; 10MB sits two orders of magnitude above that, well
# below memory-pressure territory, and rejects request-supplied bombs before
# the proxy materialises them.
_LEGACY_DEFS_MAX_INLINED_BYTES: Final = 10_000_000
DEFS_MAX_INLINED_BYTES: Final = 10_000_000
def unpack_legacy_defs(
schema: dict,
*,
copy: bool = False,
max_inlined_bytes: int = _LEGACY_DEFS_MAX_INLINED_BYTES,
max_inlined_bytes: int = DEFS_MAX_INLINED_BYTES,
) -> dict:
"""Inline ``$ref``s backed by draft-04 ``definitions`` / OpenAPI
``components.schemas``. ``$defs`` is left untouched.

2
litellm/litellm_core_utils/realtime_streaming.py
View file

@ -745,6 +745,8 @@ class RealTimeStreaming:
for callback in litellm.callbacks:
if not isinstance(callback, CustomGuardrail):
continue
if callback.use_native_lifecycle_hooks:
continue
if id(callback) in _already_run:
continue
if not any(callback.should_run_guardrail(data=_check_data, event_type=et) for et in _realtime_event_types):

172
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -3,7 +3,9 @@ import time
from collections.abc import Iterator, Mapping, Sequence
from itertools import groupby
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, TypedDict, Union, cast
from typing import TYPE_CHECKING, Any, Final, TypeAlias, TypedDict, Union, cast
from typing_extensions import ReadOnly, Required
from litellm._logging import verbose_logger
from litellm.types.llms.openai import (
@ -14,6 +16,9 @@ from litellm.types.utils import (
CacheCreationTokenDetails,
ChatCompletionAudioResponse,
ChatCompletionCustomToolCallPayload,
ChatCompletionDeltaCustomToolCall,
ChatCompletionDeltaCustomToolCallPayload,
ChatCompletionDeltaToolCall,
ChatCompletionMessageCustomToolCall,
ChatCompletionMessageToolCall,
Choices,
@ -25,6 +30,7 @@ from litellm.types.utils import (
ModelResponseStream,
PromptTokensDetailsWrapper,
ServerToolUse,
StreamingChoices,
Usage,
)
from litellm.utils import print_verbose, token_counter
@ -79,6 +85,51 @@ class _AudioChunk(TypedDict):
choices: Sequence[_AudioChoice]
_ChunkHiddenParams: TypeAlias = dict[str, object]
class _BaseChunk(TypedDict, total=False):
id: ReadOnly[str]
object: ReadOnly[str]
created: ReadOnly[int]
model: ReadOnly[str]
system_fingerprint: ReadOnly[str | None]
choices: ReadOnly[Required[Sequence[StreamingChoices]]]
_hidden_params: ReadOnly[_ChunkHiddenParams]
class _ToolCallFunctionFragment(TypedDict, total=False):
name: ReadOnly[str]
arguments: ReadOnly[str]
provider_specific_fields: ReadOnly[dict[str, object]]
class _ToolCallCustomFragment(TypedDict, total=False):
name: ReadOnly[str]
input: ReadOnly[str]
class _ToolCallFragment(TypedDict, total=False):
index: ReadOnly[int]
id: ReadOnly[str | None]
type: ReadOnly[str | None]
function: ReadOnly[_ToolCallFunctionFragment | Function | None]
custom: ReadOnly[_ToolCallCustomFragment | None]
provider_specific_fields: ReadOnly[dict[str, object] | None]
class _ToolCallDelta(TypedDict, total=False):
tool_calls: ReadOnly[Sequence[_ToolCallFragment | ChatCompletionDeltaToolCall | ChatCompletionDeltaCustomToolCall]]
class _ToolCallChoice(TypedDict, total=False):
delta: ReadOnly[_ToolCallDelta]
class _ToolCallChunk(TypedDict):
choices: ReadOnly[Sequence[_ToolCallChoice]]
class _UsageBearingChunk(TypedDict, total=False):
usage: Usage | None
_hidden_params: Mapping[str, str]
@ -158,7 +209,7 @@ class ChunkProcessor:
return chunks
def update_model_response_with_hidden_params(
self, model_response: ModelResponse, chunk: Mapping[str, dict[str, object]] | None = None
self, model_response: ModelResponse, chunk: "_BaseChunk | None" = None
) -> ModelResponse:
if chunk is None:
return model_response
@ -214,18 +265,18 @@ class ChunkProcessor:
)
@staticmethod
def _get_chunk_id(chunks: Sequence[Mapping[str, str]]) -> str:
def _get_chunk_id(chunks: Sequence["_BaseChunk"]) -> str:
"""
Chunks:
[{"id": ""}, {"id": "1"}, {"id": "1"}]
"""
for chunk in chunks:
if chunk.get("id"):
return chunk["id"]
if chunk_id := chunk.get("id"):
return chunk_id
return ""
@staticmethod
def _get_model_from_chunks(chunks: Sequence[Mapping[str, str]], first_chunk_model: str) -> str:
def _get_model_from_chunks(chunks: Sequence["_BaseChunk"], first_chunk_model: str) -> str:
"""
Get the actual model from chunks, preferring a model that differs from the first chunk.
@ -241,7 +292,7 @@ class ChunkProcessor:
# Fall back to first chunk's model if no different model found
return first_chunk_model
def build_base_response(self, chunks: list[dict[str, Any]]) -> ModelResponse:
def build_base_response(self, chunks: Sequence["_BaseChunk"]) -> ModelResponse:
chunk = self.first_chunk
id: Final = ChunkProcessor._get_chunk_id(chunks)
object: Final = chunk["object"]
@ -292,7 +343,7 @@ class ChunkProcessor:
@staticmethod
def _iter_tool_call_fragments(
tool_call_chunks: Sequence[Mapping[str, Any]],
tool_call_chunks: Sequence["_ToolCallChunk"],
) -> Iterator[tuple[int, str, str]]:
for chunk in tool_call_chunks:
for choice in chunk["choices"]:
@ -306,21 +357,21 @@ class ChunkProcessor:
index = tool_call.get("index", 0)
function = tool_call.get("function")
if isinstance(function, dict):
if function.get("arguments"):
yield index, "arguments", function["arguments"]
elif getattr(function, "arguments", None):
yield index, "arguments", function.arguments
if fragment_arguments := function.get("arguments"):
yield index, "arguments", fragment_arguments
elif function_arguments := getattr(function, "arguments", None):
yield index, "arguments", function_arguments
custom = tool_call.get("custom")
if isinstance(custom, dict) and custom.get("input"):
yield index, "custom_input", custom["input"]
if isinstance(custom, dict) and (custom_input := custom.get("input")):
yield index, "custom_input", custom_input
else:
index = getattr(tool_call, "index", 0)
function = getattr(tool_call, "function", None)
if getattr(function, "arguments", None):
yield index, "arguments", function.arguments
if object_arguments := getattr(function, "arguments", None):
yield index, "arguments", object_arguments
custom = getattr(tool_call, "custom", None)
if getattr(custom, "input", None):
yield index, "custom_input", custom.input
if object_custom_input := getattr(custom, "input", None):
yield index, "custom_input", object_custom_input
@staticmethod
def _join_fragments_by_index_and_field(
@ -337,7 +388,7 @@ class ChunkProcessor:
)
def get_combined_tool_content(
self, tool_call_chunks: Sequence[Mapping[str, Any]]
self, tool_call_chunks: Sequence["_ToolCallChunk"]
) -> list[
ChatCompletionMessageToolCall | ChatCompletionMessageCustomToolCall
]: # mutable-ok: assigned verbatim to Message.tool_calls, a list field
@ -364,7 +415,7 @@ class ChunkProcessor:
has_function = "function" in tool_call and tool_call["function"] is not None
has_custom = "custom" in tool_call and tool_call["custom"] is not None
else:
has_function = hasattr(tool_call, "function") and tool_call.function is not None
has_function = getattr(tool_call, "function", None) is not None
has_custom = getattr(tool_call, "custom", None) is not None
if not has_function and not has_custom:
@ -387,61 +438,67 @@ class ChunkProcessor:
# Extract id, type, and function data (handle both dict and object)
if isinstance(tool_call, dict):
if tool_call.get("id"):
tool_call_map[index]["id"] = tool_call["id"]
if tool_call.get("type"):
tool_call_map[index]["type"] = tool_call["type"]
if fragment_id := tool_call.get("id"):
tool_call_map[index]["id"] = fragment_id
if fragment_type := tool_call.get("type"):
tool_call_map[index]["type"] = fragment_type
function = tool_call.get("function", {})
if isinstance(function, dict):
if function.get("name"):
tool_call_map[index]["name"] = function["name"]
if fragment_name := function.get("name"):
tool_call_map[index]["name"] = fragment_name
else:
# function is an object
if hasattr(function, "name") and function.name:
tool_call_map[index]["name"] = function.name
if function_name := getattr(function, "name", None):
tool_call_map[index]["name"] = function_name
custom = tool_call.get("custom")
if isinstance(custom, dict):
if custom.get("name"):
tool_call_map[index]["custom_name"] = custom["name"]
if custom_name := custom.get("name"):
tool_call_map[index]["custom_name"] = custom_name
else:
# tool_call is an object
if hasattr(tool_call, "id") and tool_call.id:
tool_call_map[index]["id"] = tool_call.id
if hasattr(tool_call, "type") and tool_call.type:
tool_call_map[index]["type"] = tool_call.type
if hasattr(tool_call, "function"):
if hasattr(tool_call.function, "name") and tool_call.function.name:
tool_call_map[index]["name"] = tool_call.function.name
if object_function_name := getattr(getattr(tool_call, "function", None), "name", None):
tool_call_map[index]["name"] = object_function_name
custom = getattr(tool_call, "custom", None)
if custom is not None:
if getattr(custom, "name", None):
tool_call_map[index]["custom_name"] = custom.name
object_custom: ChatCompletionDeltaCustomToolCallPayload | None = getattr(
tool_call, "custom", None
)
if object_custom is not None:
if getattr(object_custom, "name", None):
tool_call_map[index]["custom_name"] = object_custom.name
# Preserve provider_specific_fields from streaming chunks
provider_fields = None
provider_fields: object = None
if isinstance(tool_call, dict):
provider_fields = tool_call.get("provider_specific_fields")
if not provider_fields and isinstance(tool_call.get("function"), dict):
provider_fields = tool_call["function"].get("provider_specific_fields")
if not provider_fields and isinstance(fragment_function := tool_call.get("function"), dict):
provider_fields = fragment_function.get("provider_specific_fields")
else:
if hasattr(tool_call, "provider_specific_fields") and tool_call.provider_specific_fields:
provider_fields = tool_call.provider_specific_fields
elif (
hasattr(tool_call, "function")
and hasattr(tool_call.function, "provider_specific_fields")
and tool_call.function.provider_specific_fields
):
provider_fields = tool_call.function.provider_specific_fields
object_provider_fields: object = getattr(tool_call, "provider_specific_fields", None)
if object_provider_fields:
provider_fields = object_provider_fields
else:
function_provider_fields: object = getattr(
getattr(tool_call, "function", None),
"provider_specific_fields",
None,
)
if function_provider_fields:
provider_fields = function_provider_fields
if provider_fields:
# Merge provider_specific_fields if multiple chunks have them
if tool_call_map[index]["provider_specific_fields"] is None:
tool_call_map[index]["provider_specific_fields"] = {}
merged_provider_fields = tool_call_map[index]["provider_specific_fields"]
if merged_provider_fields is None:
merged_provider_fields = {}
tool_call_map[index]["provider_specific_fields"] = merged_provider_fields
if isinstance(provider_fields, dict):
tool_call_map[index]["provider_specific_fields"].update(provider_fields)
merged_provider_fields.update(provider_fields)
joined_fragments: Final = self._join_fragments_by_index_and_field(
self._iter_tool_call_fragments(tool_call_chunks)
@ -762,19 +819,14 @@ class ChunkProcessor:
server_tool_use = usage_chunk.server_tool_use
else:
server_tool_use = ServerToolUse.model_validate(usage_chunk.server_tool_use)
if (
usage_chunk_dict["prompt_tokens_details"] is not None
and getattr(
if usage_chunk_dict["prompt_tokens_details"] is not None:
chunk_web_search_requests: int | None = getattr(
usage_chunk_dict["prompt_tokens_details"],
"web_search_requests",
None,
)
is not None
):
web_search_requests = getattr(
usage_chunk_dict["prompt_tokens_details"],
"web_search_requests",
)
if chunk_web_search_requests is not None:
web_search_requests = chunk_web_search_requests
prompt_tokens_details = usage_chunk_dict["prompt_tokens_details"] or prompt_tokens_details

88
litellm/litellm_core_utils/streaming_handler.py
View file

@ -6,7 +6,7 @@ import logging
import threading
import time
import traceback
from collections.abc import AsyncIterator, Callable, Iterator, Mapping, Sequence
from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from typing import Any, Final, NoReturn, Protocol, TypeVar, cast
@ -155,6 +155,33 @@ class _TextCompletionChoiceLike(Protocol):
finish_reason: str | None
class _VertexFunctionCallLike(Protocol):
name: str
args: Mapping[str, Iterable[object]]
class _VertexPartLike(Protocol):
function_call: _VertexFunctionCallLike
class _VertexContentLike(Protocol):
parts: Sequence[_VertexPartLike]
class _VertexFinishReasonLike(Protocol):
name: str
class _VertexCandidateLike(Protocol):
content: _VertexContentLike
finish_reason: _VertexFinishReasonLike
class _VertexChunkLike(Protocol):
text: str
candidates: Sequence[_VertexCandidateLike]
class CustomStreamWrapper:
def __init__(
self,
@ -291,13 +318,13 @@ class CustomStreamWrapper:
that has since taken over the same Task/thread's context.
"""
try:
logging_obj: Final = getattr(self, "logging_obj", None)
logging_obj: Final[object | None] = getattr(self, "logging_obj", None)
if logging_obj is None:
return
method_name: Final = (
"_restore_correlation_context_if_unclaimed" if guarded else "_restore_correlation_context"
)
restore: Final = getattr(logging_obj, method_name, None)
restore: Final[Callable[[], object] | None] = getattr(logging_obj, method_name, None)
if restore is not None:
restore()
except Exception as restore_error: # noqa: BLE001 # best-effort cleanup; must not raise into the caller
@ -1261,18 +1288,18 @@ class CustomStreamWrapper:
raise Exception("An unknown error occurred with the stream")
self.received_finish_reason = "stop"
elif self.custom_llm_provider == "vertex_ai" and not isinstance(chunk, ModelResponseStream):
chunk = cast(Any, chunk)
vertex_chunk: Final = cast(_VertexChunkLike, chunk)
import proto
if hasattr(chunk, "candidates") is True:
if hasattr(vertex_chunk, "candidates") is True:
try:
try:
completion_obj["content"] = chunk.text
completion_obj["content"] = vertex_chunk.text
except Exception as e:
original_exception: Final = e
if "Part has no text." in str(e):
## check for function calling
function_call: Final = chunk.candidates[0].content.parts[0].function_call
function_call: Final = vertex_chunk.candidates[0].content.parts[0].function_call
args_dict: Final = {}
@ -1311,15 +1338,15 @@ class CustomStreamWrapper:
else:
raise original_exception
if (
hasattr(chunk.candidates[0], "finish_reason")
and chunk.candidates[0].finish_reason.name != "FINISH_REASON_UNSPECIFIED"
hasattr(vertex_chunk.candidates[0], "finish_reason")
and vertex_chunk.candidates[0].finish_reason.name != "FINISH_REASON_UNSPECIFIED"
): # every non-final chunk in vertex ai has this
self.received_finish_reason = map_finish_reason(chunk.candidates[0].finish_reason.name)
self.received_finish_reason = map_finish_reason(vertex_chunk.candidates[0].finish_reason.name)
except Exception:
if chunk.candidates[0].finish_reason.name == "SAFETY":
raise Exception(f"The response was blocked by VertexAI. {chunk}")
if vertex_chunk.candidates[0].finish_reason.name == "SAFETY":
raise Exception(f"The response was blocked by VertexAI. {vertex_chunk}")
else:
completion_obj["content"] = str(chunk)
completion_obj["content"] = str(vertex_chunk)
elif self.custom_llm_provider == "petals":
if self.completion_stream is None or len(self.completion_stream) == 0:
if self.received_finish_reason is not None:
@ -1357,13 +1384,14 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
if response_obj["usage"] is not None:
_text_completion_usage: Final[Usage] = response_obj["usage"]
setattr(
model_response,
"usage",
litellm.Usage(
prompt_tokens=response_obj["usage"].prompt_tokens,
completion_tokens=response_obj["usage"].completion_tokens,
total_tokens=response_obj["usage"].total_tokens,
prompt_tokens=_text_completion_usage.prompt_tokens,
completion_tokens=_text_completion_usage.completion_tokens,
total_tokens=_text_completion_usage.total_tokens,
),
)
elif self.custom_llm_provider == "text-completion-codestral":
@ -1395,15 +1423,17 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
elif self.custom_llm_provider == "cached_response":
chunk = cast(ModelResponseStream, chunk)
chunk_finish_reason: Final = chunk.choices[0].finish_reason
cached_chunk: Final = cast(ModelResponseStream, chunk)
chunk_finish_reason: Final = cached_chunk.choices[0].finish_reason
response_obj = {
"text": chunk.choices[0].delta.content,
"text": cached_chunk.choices[0].delta.content,
"is_finished": chunk_finish_reason is not None,
"finish_reason": chunk_finish_reason,
"original_chunk": chunk,
"original_chunk": cached_chunk,
"tool_calls": (
chunk.choices[0].delta.tool_calls if hasattr(chunk.choices[0].delta, "tool_calls") else None
cached_chunk.choices[0].delta.tool_calls
if hasattr(cached_chunk.choices[0].delta, "tool_calls")
else None
),
}
@ -1411,11 +1441,11 @@ class CustomStreamWrapper:
if response_obj["tool_calls"] is not None:
completion_obj["tool_calls"] = response_obj["tool_calls"]
print_verbose(f"completion obj content: {completion_obj['content']}")
if hasattr(chunk, "id"):
model_response.id = chunk.id
self.response_id = chunk.id
if hasattr(chunk, "system_fingerprint"):
self.system_fingerprint = chunk.system_fingerprint
if hasattr(cached_chunk, "id"):
model_response.id = cached_chunk.id
self.response_id = cached_chunk.id
if hasattr(cached_chunk, "system_fingerprint"):
self.system_fingerprint = cached_chunk.system_fingerprint
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
else: # openai / azure chat model
@ -2310,16 +2340,16 @@ class CustomStreamWrapper:
def _normalize_status_code(exc: Exception) -> int | None:
"""Best-effort status_code extraction."""
try:
code: Final = getattr(exc, "status_code", None)
code: Final[int | str | None] = getattr(exc, "status_code", None)
if code is not None:
return int(code)
except Exception:
pass
response: Final = getattr(exc, "response", None)
response: Final[object | None] = getattr(exc, "response", None)
if response is not None:
try:
status_code: Final = getattr(response, "status_code", None)
status_code: Final[int | str | None] = getattr(response, "status_code", None)
if status_code is not None:
return int(status_code)
except Exception:

65
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -13,7 +13,7 @@ Pattern Overview:
"""
import json
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from copy import deepcopy
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Final, cast
@ -61,6 +61,7 @@ if TYPE_CHECKING:
ModifyResponseException,
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
)
@ -123,7 +124,7 @@ class AnthropicMessagesHandler(BaseTranslation):
@staticmethod
def _build_streaming_usage_response(
responses_so_far: list[Any],
responses_so_far: list[object],
request_data: dict | None,
) -> ModelResponse | None:
chunks: Final = tuple(response for response in responses_so_far if isinstance(response, (str, bytes)))
@ -141,7 +142,7 @@ class AnthropicMessagesHandler(BaseTranslation):
self,
exc: "ModifyResponseException",
stream_started: bool = False,
responses_so_far: list[Any] | None = None,
responses_so_far: list[object] | None = None,
) -> list[bytes]:
"""
Build an Anthropic SSE sequence delivering the guardrail block message
@ -184,7 +185,7 @@ class AnthropicMessagesHandler(BaseTranslation):
)
return list(FakeAnthropicMessagesStreamIterator(response=block_response))
def _block_continuation_chunks(self, exc: "ModifyResponseException", responses_so_far: list[Any]) -> list[bytes]:
def _block_continuation_chunks(self, exc: "ModifyResponseException", responses_so_far: list[object]) -> list[bytes]:
"""Continue an already-started message: close the open content block,
append the block message as a new text block, then end the message --
without a second message_start."""
@ -234,7 +235,7 @@ class AnthropicMessagesHandler(BaseTranslation):
@staticmethod
def _content_block_state(
responses_so_far: list[Any],
responses_so_far: list[object],
) -> tuple[int | None, int | None]:
"""From the SSE chunks already sent to the client, return (open
content-block index or None, highest content-block index seen or None).
@ -260,7 +261,7 @@ class AnthropicMessagesHandler(BaseTranslation):
return open_index, max_index
@staticmethod
def _iter_sse_events(item: Any) -> list[dict]:
def _iter_sse_events(item: object) -> list[dict[str, object]]:
"""Yield the event-data dicts in one stream chunk.
Handles both formats this stream can carry (see
@ -271,14 +272,16 @@ class AnthropicMessagesHandler(BaseTranslation):
return [item]
if not isinstance(item, (bytes, bytearray)):
return []
events: Final[list[dict]] = []
events: Final[list[dict[str, object]]] = []
for block in item.decode("utf-8", errors="replace").split("\n\n"):
for line in block.split("\n"):
line = line.strip()
if not line.startswith("data:"):
continue
try:
parsed = json.loads(line[len("data:") :].strip())
parsed: str | int | float | bool | None | Sequence[object] | Mapping[str, object] = json.loads(
line[len("data:") :].strip()
)
except json.JSONDecodeError:
continue
if isinstance(parsed, dict):
@ -315,7 +318,7 @@ class AnthropicMessagesHandler(BaseTranslation):
self,
data: dict,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None = None,
litellm_logging_obj: "LiteLLMLoggingObj | None" = None,
) -> Any:
"""
Process input messages by applying guardrails to text content.
@ -467,8 +470,8 @@ class AnthropicMessagesHandler(BaseTranslation):
@staticmethod
def _openai_system_message_to_anthropic(
message: dict[str, Any],
) -> dict[str, Any] | None: # mutable-ok: API message payload
message: dict[str, object],
) -> dict[str, object] | None: # mutable-ok: API message payload
"""Convert an OpenAI system message to the client's Anthropic-shaped entry."""
content: Final = message.get("content")
if isinstance(content, str):
@ -477,14 +480,14 @@ class AnthropicMessagesHandler(BaseTranslation):
) # mutable-ok: API message payload
if not isinstance(content, list):
return None
blocks: Final[list[dict[str, Any]]] = [] # mutable-ok: API message payload
blocks: Final[list[dict[str, object]]] = [] # mutable-ok: API message payload
for block in content:
if not isinstance(block, dict) or block.get("type") != "text":
continue
text = block.get("text")
if not isinstance(text, str) or not text:
continue
anthropic_block: dict[str, Any] = { # mutable-ok: API message payload
anthropic_block: dict[str, object] = { # mutable-ok: API message payload
"type": "text",
"text": text,
} # mutable-ok: API message payload
@ -602,7 +605,7 @@ class AnthropicMessagesHandler(BaseTranslation):
@staticmethod
def _extract_midturn_system_text(
message: dict[str, Any], # mutable-ok: API message payload
message: Mapping[str, object],
msg_idx: int,
) -> ExtractedInput:
"""Match the adapter's filtering so positional guardrail write-back stays aligned."""
@ -636,7 +639,7 @@ class AnthropicMessagesHandler(BaseTranslation):
@classmethod
def _extract_input_text_and_images(
cls,
message: dict[str, Any],
message: Mapping[str, object],
msg_idx: int,
skip_system_message: bool = False,
skip_tool_message: bool = False,
@ -707,7 +710,7 @@ class AnthropicMessagesHandler(BaseTranslation):
@classmethod
def _extract_tool_result(
cls,
content_item: Mapping[str, Any],
content_item: Mapping[str, object],
msg_idx: int,
content_idx: int,
) -> ExtractedInput:
@ -736,7 +739,7 @@ class AnthropicMessagesHandler(BaseTranslation):
)
@staticmethod
def _image_sources(block: Mapping[str, Any]) -> tuple[str, ...]:
def _image_sources(block: Mapping[str, object]) -> tuple[str, ...]:
source: Final = block.get("source")
if not isinstance(source, Mapping):
return ()
@ -746,7 +749,7 @@ class AnthropicMessagesHandler(BaseTranslation):
async def _apply_guardrail_responses_to_input(
self,
messages: list[dict[str, Any]],
messages: list[dict[str, object]],
responses: list[str],
scanned: tuple[ScannedText, ...],
) -> None:
@ -788,10 +791,10 @@ class AnthropicMessagesHandler(BaseTranslation):
self,
response: "AnthropicMessagesResponse",
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None = None,
user_api_key_dict: Any | None = None,
litellm_logging_obj: "LiteLLMLoggingObj | None" = None,
user_api_key_dict: "UserAPIKeyAuth | None" = None,
request_data: dict | None = None,
) -> Any:
) -> "AnthropicMessagesResponse":
"""
Process output response by applying guardrails to text content and tool calls.
@ -869,8 +872,8 @@ class AnthropicMessagesHandler(BaseTranslation):
self,
responses_so_far: list[Any],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None = None,
user_api_key_dict: Any | None = None,
litellm_logging_obj: "LiteLLMLoggingObj | None" = None,
user_api_key_dict: "UserAPIKeyAuth | None" = None,
request_data: dict | None = None,
) -> list[Any]:
"""
@ -950,8 +953,8 @@ class AnthropicMessagesHandler(BaseTranslation):
def _prepare_request_data(
self,
request_data: dict | None,
response: Any,
user_api_key_dict: Any | None,
response: object,
user_api_key_dict: "UserAPIKeyAuth | None",
key: str,
) -> dict:
"""Ensure request_data has the response/responses_so_far key and metadata."""
@ -968,7 +971,7 @@ class AnthropicMessagesHandler(BaseTranslation):
return request_data
@staticmethod
def _get_response_content(response: Any) -> list[Any]:
def _get_response_content(response: object) -> list[Any]:
"""Extract content list from a dict or object response."""
if isinstance(response, dict):
return response.get("content", []) or []
@ -986,10 +989,10 @@ class AnthropicMessagesHandler(BaseTranslation):
) -> None:
"""Extract text, images, and tool calls from content blocks."""
for content_idx, content_block in enumerate(response_content):
block_dict: dict[str, Any] = {}
block_dict: dict[str, object] = {}
if isinstance(content_block, dict):
block_type = content_block.get("type")
block_dict = cast(dict[str, Any], content_block)
block_dict = cast(dict[str, object], content_block)
elif hasattr(content_block, "type"):
block_type = getattr(content_block, "type", None)
if hasattr(content_block, "model_dump"):
@ -1017,7 +1020,7 @@ class AnthropicMessagesHandler(BaseTranslation):
texts_to_check: list[str],
images_to_check: list[str],
tool_calls_to_check: list["ChatCompletionToolCallChunk"],
response: Any,
response: object,
) -> "GenericGuardrailAPIInputs":
"""Build GenericGuardrailAPIInputs with optional images, tool calls, model."""
inputs: Final = GenericGuardrailAPIInputs(texts=texts_to_check)
@ -1212,7 +1215,7 @@ class AnthropicMessagesHandler(BaseTranslation):
def _extract_output_text_and_images(
self,
content_block: dict[str, Any],
content_block: dict[str, object],
content_idx: int,
texts_to_check: list[str],
images_to_check: list[str],
@ -1282,7 +1285,7 @@ class AnthropicMessagesHandler(BaseTranslation):
# Handle both dict and Pydantic object content blocks
if isinstance(content_block, dict):
if content_block.get("type") == "text":
cast(dict[str, Any], content_block)["text"] = guardrail_response
cast(dict[str, object], content_block)["text"] = guardrail_response
elif hasattr(content_block, "type") and getattr(content_block, "type", None) == "text":
# Update Pydantic object's text attribute
if hasattr(content_block, "text"):

3
litellm/llms/anthropic/chat/transformation.py
View file

@ -1267,13 +1267,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
import copy
from litellm.litellm_core_utils.prompt_templates.common_utils import (
DEFS_MAX_INLINED_BYTES,
unpack_defs,
)
json_schema = copy.deepcopy(json_schema)
defs: Final = json_schema.pop("$defs", json_schema.pop("definitions", {}))
if defs:
unpack_defs(json_schema, defs)
unpack_defs(json_schema, defs, max_inlined_bytes=DEFS_MAX_INLINED_BYTES)
# Filter out unsupported fields for Anthropic's output_format API
filtered_schema: Final = self.filter_anthropic_output_schema(json_schema)

40
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -84,6 +84,7 @@ from litellm.types.llms.anthropic import (
AnthropicResponseContentBlockText,
AnthropicResponseContentBlockThinking,
AnthropicResponseContentBlockToolUse,
AnthropicThinkingParam,
AppliedEdit,
ContentBlockDelta,
ContentJsonBlockDelta,
@ -305,9 +306,9 @@ class LiteLLMAnthropicMessagesAdapter:
target["cache_control"] = cache_control
else:
# Fallback for non-dict objects (shouldn't happen in practice)
cast(dict[str, Any], target)["cache_control"] = cache_control
cast(dict[str, object], target)["cache_control"] = cache_control
def translatable_anthropic_params(self) -> list:
def translatable_anthropic_params(self) -> list[str]:
"""
Which anthropic params, we need to translate to the openai format.
"""
@ -323,7 +324,7 @@ class LiteLLMAnthropicMessagesAdapter:
"stop_sequences",
]
def _is_web_search_tool(self, tool: dict[str, Any]) -> bool:
def _is_web_search_tool(self, tool: Mapping[str, object]) -> bool:
"""
Check if a tool is an Anthropic web search tool.
@ -498,7 +499,7 @@ class LiteLLMAnthropicMessagesAdapter:
assistant_message_str = str(content)
elif isinstance(content, dict):
if content.get("type") == "text":
text_block: dict[str, Any] = {
text_block: dict[str, object] = {
"type": "text",
"text": content.get("text", ""),
}
@ -513,10 +514,12 @@ class LiteLLMAnthropicMessagesAdapter:
"name": tool_name,
"arguments": json.dumps(content.get("input", {})),
}
signature = self._extract_signature_from_tool_use_content(cast(dict[str, Any], content))
signature = self._extract_signature_from_tool_use_content(
cast(dict[str, object], content)
)
if signature:
provider_specific_fields: dict[str, Any] = (
provider_specific_fields: dict[str, object] = (
function_chunk.get("provider_specific_fields") or {}
)
provider_specific_fields["thought_signature"] = signature
@ -575,7 +578,7 @@ class LiteLLMAnthropicMessagesAdapter:
@staticmethod
def translate_anthropic_thinking_to_reasoning_effort(
thinking: dict[str, Any],
thinking: AnthropicThinkingParam,
) -> str | None:
"""
Translate Anthropic's thinking parameter to OpenAI's reasoning_effort.
@ -632,9 +635,9 @@ class LiteLLMAnthropicMessagesAdapter:
@staticmethod
def translate_thinking_for_model(
thinking: dict[str, Any],
thinking: AnthropicThinkingParam,
model: str,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
Translate Anthropic thinking parameter based on the target model.
@ -670,7 +673,7 @@ class LiteLLMAnthropicMessagesAdapter:
@staticmethod
def _apply_reasoning_summary_wrapping(
reasoning_effort: str,
thinking: dict[str, Any],
thinking: Mapping[str, object],
) -> Any:
"""
Apply the reasoning_effort/summary wrapping rules shared by every
@ -731,6 +734,7 @@ class LiteLLMAnthropicMessagesAdapter:
"input_schema",
"description",
"cache_control",
"strict",
"type",
]
@ -760,6 +764,8 @@ class LiteLLMAnthropicMessagesAdapter:
function_chunk["parameters"] = tool["input_schema"]
if "description" in tool:
function_chunk["description"] = tool["description"]
if "strict" in tool:
function_chunk["strict"] = bool(tool["strict"])
for k, v in tool.items():
if k not in mapped_tool_params: # pass additional computer kwargs
@ -770,7 +776,7 @@ class LiteLLMAnthropicMessagesAdapter:
return new_tools, tool_name_mapping
def translate_anthropic_output_format_to_openai(self, output_format: Any) -> dict[str, Any] | None:
def translate_anthropic_output_format_to_openai(self, output_format: Any) -> dict[str, object] | None:
"""
Translate Anthropic's output_format to OpenAI's response_format.
@ -889,7 +895,7 @@ class LiteLLMAnthropicMessagesAdapter:
model_name: Final = anthropic_message_request.get("model", "")
for block in system_content:
if isinstance(block, dict) and block.get("type") == "text":
text_block: dict[str, Any] = {
text_block: dict[str, object] = {
"type": "text",
"text": block.get("text", ""),
}
@ -959,7 +965,7 @@ class LiteLLMAnthropicMessagesAdapter:
web_search_tools: Final[list[AllAnthropicToolsValues]] = []
regular_tools: Final[list[AllAnthropicToolsValues]] = []
for tool in tools:
cast_tool = cast(dict[str, Any], tool)
cast_tool = cast(dict[str, object], tool)
if self._is_web_search_tool(cast_tool):
web_search_tools.append(cast(AllAnthropicToolsValues, tool))
else:
@ -1007,7 +1013,7 @@ class LiteLLMAnthropicMessagesAdapter:
new_kwargs["output_config"] = effort_config # rebind-ok: out-param store like thinking above
return
reasoning_effort = self.translate_anthropic_thinking_to_reasoning_effort(cast(dict[str, Any], thinking))
reasoning_effort = self.translate_anthropic_thinking_to_reasoning_effort(cast(AnthropicThinkingParam, thinking))
if not reasoning_effort:
return
@ -1020,7 +1026,7 @@ class LiteLLMAnthropicMessagesAdapter:
reasoning_effort = output_config["effort"]
new_kwargs["reasoning_effort"] = self._apply_reasoning_summary_wrapping(
reasoning_effort, cast(dict[str, Any], thinking)
reasoning_effort, cast(dict[str, object], thinking)
)
def _translate_output_format_to_openai(
@ -1040,7 +1046,7 @@ class LiteLLMAnthropicMessagesAdapter:
``output_format`` takes precedence when both are provided.
"""
output_format: Any = anthropic_message_request.get("output_format")
output_format: object = anthropic_message_request.get("output_format")
if not output_format:
output_config: Final = anthropic_message_request.get("output_config")
if isinstance(output_config, dict):
@ -1407,7 +1413,7 @@ class LiteLLMAnthropicMessagesAdapter:
if THOUGHT_SIGNATURE_SEPARATOR in raw_id:
parts = raw_id.split(THOUGHT_SIGNATURE_SEPARATOR, 1)
thought_sig = parts[1] if len(parts) > 1 else None
tool_block: dict[str, Any] = {
tool_block: dict[str, object] = {
"type": "tool_use",
"id": normalize_anthropic_tool_use_id(raw_id),
"name": tool_name,

85
litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py
View file

@ -3,7 +3,7 @@
import json
import traceback
from collections import deque
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Mapping
from typing import Any, Final
from litellm import verbose_logger
@ -68,6 +68,19 @@ class AnthropicResponsesStreamWrapper:
self._current_block_index += 1
return self._current_block_index
def _open_block(self, item_id: str | None, content_block: Mapping[str, Any]) -> int:
block_idx = self._next_block_index()
if item_id:
self._item_id_to_block_index[item_id] = block_idx
self._chunk_queue.append(
{
"type": "content_block_start",
"index": block_idx,
"content_block": content_block,
}
)
return block_idx
def _process_event(self, event: Any) -> None:
"""Convert one Responses API event into zero or more Anthropic chunks queued for emission."""
event_type = getattr(event, "type", None)
@ -93,47 +106,22 @@ class AnthropicResponsesStreamWrapper:
item_id = getattr(item, "id", None) or (item.get("id") if isinstance(item, dict) else None)
if item_type == "message":
block_idx = self._next_block_index()
if item_id:
self._item_id_to_block_index[item_id] = block_idx
self._chunk_queue.append(
{
"type": "content_block_start",
"index": block_idx,
"content_block": {"type": "text", "text": ""},
}
)
self._open_block(item_id, {"type": "text", "text": ""})
elif item_type == "function_call":
call_id: Final = (
getattr(item, "call_id", None) or (item.get("call_id") if isinstance(item, dict) else None) or ""
)
name = getattr(item, "name", None) or (item.get("name") if isinstance(item, dict) else None) or ""
block_idx = self._next_block_index()
if item_id:
self._item_id_to_block_index[item_id] = block_idx
self._pending_tool_ids[item_id] = call_id
self._chunk_queue.append(
self._open_block(
item_id,
{
"type": "content_block_start",
"index": block_idx,
"content_block": {
"type": "tool_use",
"id": call_id,
"name": name,
"input": {},
},
}
)
elif item_type == "reasoning":
block_idx = self._next_block_index()
if item_id:
self._item_id_to_block_index[item_id] = block_idx
self._chunk_queue.append(
{
"type": "content_block_start",
"index": block_idx,
"content_block": {"type": "thinking", "thinking": ""},
}
"type": "tool_use",
"id": call_id,
"name": name,
"input": {},
},
)
return
@ -146,16 +134,7 @@ class AnthropicResponsesStreamWrapper:
# Some providers (e.g. LMStudio) skip response.output_item.added,
# so no text block is open yet; synthesize content_block_start
# instead of emitting a delta with index -1
block_idx = self._next_block_index()
if item_id:
self._item_id_to_block_index[item_id] = block_idx
self._chunk_queue.append(
{
"type": "content_block_start",
"index": block_idx,
"content_block": {"type": "text", "text": ""},
}
)
block_idx = self._open_block(item_id, {"type": "text", "text": ""})
self._chunk_queue.append(
{
"type": "content_block_delta",
@ -169,11 +148,11 @@ class AnthropicResponsesStreamWrapper:
if event_type == "response.reasoning_summary_text.delta":
item_id = getattr(event, "item_id", None) or (event.get("item_id") if isinstance(event, dict) else None)
delta = getattr(event, "delta", "") or (event.get("delta", "") if isinstance(event, dict) else "")
block_idx = (
self._item_id_to_block_index.get(item_id, self._current_block_index)
if item_id
else self._current_block_index
)
block_idx = self._item_id_to_block_index.get(item_id, -1) if item_id else self._current_block_index
if block_idx < 0:
if not delta:
return
block_idx = self._open_block(item_id, {"type": "thinking", "thinking": ""})
self._chunk_queue.append(
{
"type": "content_block_delta",
@ -207,11 +186,9 @@ class AnthropicResponsesStreamWrapper:
item_id = (
getattr(item, "id", None) or (item.get("id") if isinstance(item, dict) else None) if item else None
)
block_idx = (
self._item_id_to_block_index.get(item_id, self._current_block_index)
if item_id
else self._current_block_index
)
block_idx = self._item_id_to_block_index.get(item_id, -1) if item_id else self._current_block_index
if block_idx < 0:
return
self._chunk_queue.append(
{
"type": "content_block_stop",

8
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py
View file

@ -266,7 +266,13 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
if (isinstance(tool_type, str) and tool_type.startswith("web_search")) or tool_name == "web_search":
result.append({"type": "web_search_preview"})
continue
func_tool: dict[str, Any] = {"type": "function", "name": tool_name}
# Responses turns strict mode on when `strict` is omitted, silently rewriting
# `required` to every property. Anthropic tools are non-strict unless asked.
func_tool: dict[str, Any] = {
"type": "function",
"name": tool_name,
"strict": bool(tool_dict.get("strict")),
}
if "description" in tool_dict:
func_tool["description"] = tool_dict["description"]
if "input_schema" in tool_dict:

15
litellm/llms/azure/chat/gpt_transformation.py
View file

@ -112,6 +112,16 @@ class AzureOpenAIConfig(BaseConfig):
"store",
]
@classmethod
def requires_max_completion_tokens(cls, model: str) -> bool:
"""Whether Azure rejects the legacy ``max_tokens`` key for this deployment.
Deliberately wider than ``AzureOpenAIGPT5Config.is_model_gpt_5_model``: the whole gpt-5
name family needs the rename, including the ``gpt-5-chat*`` models that are excluded from
the reasoning path by https://github.com/BerriAI/litellm/issues/13781.
"""
return "gpt-5" in model or "gpt5_series" in model
def _is_response_format_supported_model(self, model: str) -> bool:
"""
Determines if the model supports response_format.
@ -160,6 +170,7 @@ class AzureOpenAIConfig(BaseConfig):
api_version: str = "",
) -> dict:
supported_openai_params: Final = self.get_supported_openai_params(model)
renames_max_tokens: Final = self.requires_max_completion_tokens(model)
api_version_times: Final = api_version.split("-")
if len(api_version_times) >= 3:
@ -172,7 +183,9 @@ class AzureOpenAIConfig(BaseConfig):
api_version_day = None
for param, value in non_default_params.items():
if param == "tool_choice":
if param == "max_tokens" and renames_max_tokens:
optional_params.setdefault("max_completion_tokens", value)
elif param == "tool_choice":
"""
This parameter requires API version 2023-12-01-preview or later

156
litellm/llms/azure_ai/agents/handler.py
View file

@ -22,10 +22,11 @@ import asyncio
import json
import time
import uuid
from collections.abc import AsyncIterator, Callable
from typing import TYPE_CHECKING, Any, Final
from collections.abc import AsyncIterator, Awaitable, Mapping
from typing import TYPE_CHECKING, Any, Final, Protocol, TypeAlias, TypedDict
import httpx
from typing_extensions import ReadOnly
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.url_utils import encode_url_path_segment
@ -33,7 +34,11 @@ from litellm.llms.azure_ai.agents.transformation import (
AzureAIAgentsConfig,
AzureAIAgentsError,
)
from litellm.types.utils import ModelResponse
from litellm.types.llms.openai import (
ChatCompletionAnnotation,
ChatCompletionAnnotationURLCitation,
)
from litellm.types.utils import ModelResponse, ModelResponseStream
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -46,6 +51,69 @@ else:
AsyncHTTPHandler = Any
class _AzureRawAnnotation(TypedDict, total=False):
type: ReadOnly[str]
text: ReadOnly[str]
start_index: ReadOnly[int]
end_index: ReadOnly[int]
url_citation: ReadOnly[ChatCompletionAnnotationURLCitation]
_TransformedAnnotation: TypeAlias = ChatCompletionAnnotation | _AzureRawAnnotation
class _AzureText(TypedDict, total=False):
value: ReadOnly[str]
annotations: ReadOnly[list[_AzureRawAnnotation]]
class _AzureContentItem(TypedDict, total=False):
type: ReadOnly[str]
text: ReadOnly[_AzureText]
class _AzureMessage(TypedDict, total=False):
role: ReadOnly[str]
content: ReadOnly[list[_AzureContentItem]]
class _AzureMessagesData(TypedDict, total=False):
data: ReadOnly[list[_AzureMessage]]
class _CreatedObject(TypedDict):
id: ReadOnly[str]
class _RunError(TypedDict, total=False):
message: ReadOnly[str]
class _RunStatus(TypedDict, total=False):
status: ReadOnly[str]
last_error: ReadOnly[_RunError]
class _SSEDelta(TypedDict, total=False):
content: ReadOnly[list[_AzureContentItem]]
class _SSEEventData(TypedDict, total=False):
id: ReadOnly[str]
content: ReadOnly[list[_AzureContentItem]]
delta: ReadOnly[_SSEDelta]
class _SyncAgentRequest(Protocol):
def __call__(self, method: str, url: str, json_data: Mapping[str, object] | None = None) -> httpx.Response: ...
class _AsyncAgentRequest(Protocol):
def __call__(
self, method: str, url: str, json_data: Mapping[str, object] | None = None
) -> Awaitable[httpx.Response]: ...
class AzureAIAgentsHandler:
"""
Handler for Azure AI Agent Service.
@ -89,7 +157,9 @@ class AzureAIAgentsHandler:
# -------------------------------------------------------------------------
# Response Helpers
# -------------------------------------------------------------------------
def _extract_content_from_messages(self, messages_data: dict) -> tuple[str, list[dict[str, Any]] | None]:
def _extract_content_from_messages(
self, messages_data: _AzureMessagesData
) -> tuple[str, list[_TransformedAnnotation] | None]:
"""Extract assistant content and annotations from the messages response.
Returns (content, annotations) where annotations is a list of
@ -108,8 +178,8 @@ class AzureAIAgentsHandler:
def _transform_annotations(
self,
raw_annotations: list[dict[str, Any]] | None,
) -> list[dict[str, Any]] | None:
raw_annotations: list[_AzureRawAnnotation] | None,
) -> list[_TransformedAnnotation] | None:
"""Transform Azure AI Foundry annotations to OpenAI-compatible format.
Azure AI returns annotations like:
@ -123,11 +193,11 @@ class AzureAIAgentsHandler:
if not raw_annotations:
return None
result: Final[list[dict[str, Any]]] = []
result: Final[list[_TransformedAnnotation]] = []
for ann in raw_annotations:
ann_type = ann.get("type")
if ann_type == "url_citation":
url_citation = dict(ann.get("url_citation", {}))
url_citation: ChatCompletionAnnotationURLCitation = {**ann.get("url_citation", {})}
# Azure puts start/end_index at annotation level; OpenAI
# expects them inside url_citation
if "start_index" in ann and "start_index" not in url_citation:
@ -147,8 +217,8 @@ class AzureAIAgentsHandler:
content: str,
model_response: ModelResponse,
thread_id: str,
messages: list[dict[str, Any]],
annotations: list[dict[str, Any]] | None = None,
messages: list[dict[str, object]],
annotations: list[_TransformedAnnotation] | None = None,
) -> ModelResponse:
"""Build the ModelResponse from agent output."""
from litellm.types.utils import Choices, Message, Usage
@ -201,7 +271,7 @@ class AzureAIAgentsHandler:
api_key: str,
optional_params: dict,
headers: dict | None,
) -> tuple:
) -> tuple[dict[str, str], str, str, str | None, str]:
"""Prepare common parameters for completion.
Azure Foundry Agents API uses Bearer token authentication:
@ -241,7 +311,7 @@ class AzureAIAgentsHandler:
def completion(
self,
model: str,
messages: list[dict[str, Any]],
messages: list[dict[str, object]],
api_base: str,
api_key: str,
model_response: ModelResponse,
@ -266,7 +336,7 @@ class AzureAIAgentsHandler:
api_base,
) = self._prepare_completion_params(model, api_base, api_key, optional_params, headers)
def make_request(method: str, url: str, json_data: dict | None = None) -> httpx.Response:
def make_request(method: str, url: str, json_data: Mapping[str, object] | None = None) -> httpx.Response:
if method == "GET":
return client.get(url=url, headers=headers)
return client.post(
@ -290,14 +360,14 @@ class AzureAIAgentsHandler:
def _execute_agent_flow_sync(
self,
make_request: Callable,
make_request: _SyncAgentRequest,
api_base: str,
api_version: str,
agent_id: str,
thread_id: str | None,
messages: list[dict[str, Any]],
messages: list[dict[str, object]],
optional_params: dict,
) -> tuple[str, str, list[dict[str, Any]] | None]:
) -> tuple[str, str, list[_TransformedAnnotation] | None]:
"""Execute the agent flow synchronously. Returns (thread_id, content, annotations)."""
# Step 1: Create thread if not provided
@ -305,7 +375,8 @@ class AzureAIAgentsHandler:
verbose_logger.debug("Creating thread at: %s", self._build_thread_url(api_base, api_version))
response = make_request("POST", self._build_thread_url(api_base, api_version), {})
self._check_response(response, [200, 201], "Failed to create thread")
thread_id = response.json()["id"]
thread_data: Final[_CreatedObject] = response.json()
thread_id = thread_data["id"]
verbose_logger.debug("Created thread: %s", thread_id)
# At this point thread_id is guaranteed to be a string
@ -325,7 +396,8 @@ class AzureAIAgentsHandler:
response = make_request("POST", self._build_runs_url(api_base, thread_id, api_version), run_payload)
self._check_response(response, [200, 201], "Failed to create run")
run_id: Final = response.json()["id"]
run_data: Final[_CreatedObject] = response.json()
run_id: Final = run_data["id"]
verbose_logger.debug("Created run: %s", run_id)
# Step 4: Poll for completion
@ -334,13 +406,15 @@ class AzureAIAgentsHandler:
response = make_request("GET", status_url)
self._check_response(response, [200], "Failed to get run status")
status = response.json().get("status")
status_data: _RunStatus = response.json()
status = status_data.get("status")
verbose_logger.debug("Run status: %s", status)
if status == "completed":
break
elif status in ["failed", "cancelled", "expired"]:
error_msg = response.json().get("last_error", {}).get("message", "Unknown error")
error_data: _RunStatus = response.json()
error_msg = error_data.get("last_error", {}).get("message", "Unknown error")
raise AzureAIAgentsError(status_code=500, message=f"Run {status}: {error_msg}")
time.sleep(self.config.POLL_INTERVAL_SECONDS)
@ -351,7 +425,8 @@ class AzureAIAgentsHandler:
response = make_request("GET", self._build_list_messages_url(api_base, thread_id, api_version))
self._check_response(response, [200], "Failed to get messages")
content, annotations = self._extract_content_from_messages(response.json())
messages_data: Final[_AzureMessagesData] = response.json()
content, annotations = self._extract_content_from_messages(messages_data)
return thread_id, content, annotations
# -------------------------------------------------------------------------
@ -360,7 +435,7 @@ class AzureAIAgentsHandler:
async def acompletion(
self,
model: str,
messages: list[dict[str, Any]],
messages: list[dict[str, object]],
api_base: str,
api_key: str,
model_response: ModelResponse,
@ -389,7 +464,7 @@ class AzureAIAgentsHandler:
api_base,
) = self._prepare_completion_params(model, api_base, api_key, optional_params, headers)
async def make_request(method: str, url: str, json_data: dict | None = None) -> httpx.Response:
async def make_request(method: str, url: str, json_data: Mapping[str, object] | None = None) -> httpx.Response:
if method == "GET":
return await client.get(url=url, headers=headers)
return await client.post(
@ -413,14 +488,14 @@ class AzureAIAgentsHandler:
async def _execute_agent_flow_async(
self,
make_request: Callable,
make_request: _AsyncAgentRequest,
api_base: str,
api_version: str,
agent_id: str,
thread_id: str | None,
messages: list[dict[str, Any]],
messages: list[dict[str, object]],
optional_params: dict,
) -> tuple[str, str, list[dict[str, Any]] | None]:
) -> tuple[str, str, list[_TransformedAnnotation] | None]:
"""Execute the agent flow asynchronously. Returns (thread_id, content, annotations)."""
# Step 1: Create thread if not provided
@ -428,7 +503,8 @@ class AzureAIAgentsHandler:
verbose_logger.debug("Creating thread at: %s", self._build_thread_url(api_base, api_version))
response = await make_request("POST", self._build_thread_url(api_base, api_version), {})
self._check_response(response, [200, 201], "Failed to create thread")
thread_id = response.json()["id"]
thread_data: Final[_CreatedObject] = response.json()
thread_id = thread_data["id"]
verbose_logger.debug("Created thread: %s", thread_id)
# At this point thread_id is guaranteed to be a string
@ -448,7 +524,8 @@ class AzureAIAgentsHandler:
response = await make_request("POST", self._build_runs_url(api_base, thread_id, api_version), run_payload)
self._check_response(response, [200, 201], "Failed to create run")
run_id: Final = response.json()["id"]
run_data: Final[_CreatedObject] = response.json()
run_id: Final = run_data["id"]
verbose_logger.debug("Created run: %s", run_id)
# Step 4: Poll for completion
@ -457,13 +534,15 @@ class AzureAIAgentsHandler:
response = await make_request("GET", status_url)
self._check_response(response, [200], "Failed to get run status")
status = response.json().get("status")
status_data: _RunStatus = response.json()
status = status_data.get("status")
verbose_logger.debug("Run status: %s", status)
if status == "completed":
break
elif status in ["failed", "cancelled", "expired"]:
error_msg = response.json().get("last_error", {}).get("message", "Unknown error")
error_data: _RunStatus = response.json()
error_msg = error_data.get("last_error", {}).get("message", "Unknown error")
raise AzureAIAgentsError(status_code=500, message=f"Run {status}: {error_msg}")
await asyncio.sleep(self.config.POLL_INTERVAL_SECONDS)
@ -474,7 +553,8 @@ class AzureAIAgentsHandler:
response = await make_request("GET", self._build_list_messages_url(api_base, thread_id, api_version))
self._check_response(response, [200], "Failed to get messages")
content, annotations = self._extract_content_from_messages(response.json())
messages_data: Final[_AzureMessagesData] = response.json()
content, annotations = self._extract_content_from_messages(messages_data)
return thread_id, content, annotations
# -------------------------------------------------------------------------
@ -483,7 +563,7 @@ class AzureAIAgentsHandler:
async def acompletion_stream(
self,
model: str,
messages: list[dict[str, Any]],
messages: list[dict[str, object]],
api_base: str,
api_key: str,
logging_obj: LiteLLMLoggingObj,
@ -491,7 +571,7 @@ class AzureAIAgentsHandler:
litellm_params: dict,
timeout: float,
headers: dict | None = None,
) -> AsyncIterator:
) -> AsyncIterator[ModelResponseStream]:
"""Execute async streaming completion using Azure Agent Service with native SSE."""
import litellm
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
@ -505,12 +585,12 @@ class AzureAIAgentsHandler:
) = self._prepare_completion_params(model, api_base, api_key, optional_params, headers)
# Build payload for create-thread-and-run with streaming
thread_messages: Final = []
thread_messages: Final[list[dict[str, object]]] = []
for msg in messages:
if msg.get("role") in ["user", "system"]:
thread_messages.append({"role": "user", "content": msg.get("content", "")})
payload: Final[dict[str, Any]] = {
payload: Final[dict[str, object]] = {
"assistant_id": agent_id,
"stream": True,
}
@ -552,14 +632,14 @@ class AzureAIAgentsHandler:
self,
response: httpx.Response,
model: str,
) -> AsyncIterator:
) -> AsyncIterator[ModelResponseStream]:
"""Process SSE stream and yield OpenAI-compatible streaming chunks."""
from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices
response_id: Final = f"chatcmpl-{uuid.uuid4().hex[:8]}"
created: Final = int(time.time())
thread_id = None
collected_annotations: list[dict[str, Any]] | None = None
collected_annotations: list[_TransformedAnnotation] | None = None
current_event = None
@ -597,7 +677,7 @@ class AzureAIAgentsHandler:
return
try:
data = json.loads(data_str)
data: _SSEEventData = json.loads(data_str)
except json.JSONDecodeError:
continue

82
litellm/llms/bedrock/batches/handler.py
View file

@ -1,11 +1,14 @@
from datetime import datetime
from typing import Any, Final, cast
from typing import TYPE_CHECKING, Any, Final, cast
from openai.types.batch import BatchRequestCounts
from openai.types.batch import Metadata as OpenAIBatchMetadata
from litellm.types.utils import LiteLLMBatch
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
# AWS Bedrock model-invocation-job statuses → OpenAI Batch statuses.
# Mirrors the mapping used by `BedrockBatchesConfig.transform_create_batch_response`
# so create / retrieve return consistent statuses.
@ -22,6 +25,8 @@ _BEDROCK_MIJ_STATUS_TO_OPENAI: Final = {
"Expired": "expired",
}
_CANCEL_IDEMPOTENT_STATUSES: Final = frozenset({"cancelling", "cancelled", "completed", "failed", "expired"})
def _extract_region_from_bedrock_arn(arn: str) -> str | None:
"""ARN shape: ``arn:aws:bedrock:<region>:<account>:<type>/<id>``"""
@ -82,6 +87,81 @@ class BedrockBatchesHandler:
E.g. Twelve Labs Embedding Async Invoke
"""
@staticmethod
def cancel_batch(
batch_id: str,
aws_region_name: str | None = None,
logging_obj: "LiteLLMLoggingObj | None" = None,
aws_access_key_id: str | None = None,
aws_secret_access_key: str | None = None,
aws_session_token: str | None = None,
aws_session_name: str | None = None,
aws_profile_name: str | None = None,
aws_role_name: str | None = None,
aws_web_identity_token: str | None = None,
aws_sts_endpoint: str | None = None,
aws_external_id: str | None = None,
**kwargs: object, # kwargs-ok: litellm.cancel_batch forwards arbitrary user kwargs verbatim
) -> "LiteLLMBatch":
try:
import boto3
from botocore.exceptions import ClientError
except ImportError as exc:
raise ImportError("Missing boto3/botocore to call bedrock. Run 'pip install boto3'.") from exc
region: Final = aws_region_name or _extract_region_from_bedrock_arn(batch_id) or "us-east-1"
from litellm.llms.bedrock.batches.transformation import BedrockBatchesConfig
creds: Final = BedrockBatchesConfig().get_credentials(
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
aws_region_name=region,
aws_session_name=aws_session_name,
aws_profile_name=aws_profile_name,
aws_role_name=aws_role_name,
aws_web_identity_token=aws_web_identity_token,
aws_sts_endpoint=aws_sts_endpoint,
aws_external_id=aws_external_id,
)
client: Final = boto3.client(
"bedrock",
region_name=region,
aws_access_key_id=creds.access_key,
aws_secret_access_key=creds.secret_key,
aws_session_token=creds.token,
)
def job_status() -> "LiteLLMBatch":
return BedrockBatchesHandler._handle_model_invocation_job_status(
batch_id=batch_id,
aws_region_name=region,
logging_obj=logging_obj,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
aws_session_name=aws_session_name,
aws_profile_name=aws_profile_name,
aws_role_name=aws_role_name,
aws_web_identity_token=aws_web_identity_token,
aws_sts_endpoint=aws_sts_endpoint,
aws_external_id=aws_external_id,
)
try:
client.stop_model_invocation_job(jobIdentifier=batch_id)
except ClientError as e:
if e.response.get("Error", {}).get("Code") not in ("ValidationException", "ConflictException"):
raise
current_batch: Final = job_status()
if current_batch.status not in _CANCEL_IDEMPOTENT_STATUSES:
raise
return current_batch
return job_status()
@staticmethod
def _handle_async_invoke_status(batch_id: str, aws_region_name: str, logging_obj=None, **kwargs) -> "LiteLLMBatch":
"""

23
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -39,6 +39,12 @@ from litellm.llms.anthropic.chat.transformation import (
AnthropicConfig,
)
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.llms.bedrock.request_metadata import (
bedrock_request_metadata_headers,
bedrock_request_metadata_is_owned,
merge_bedrock_invoke_headers,
resolve_bedrock_request_metadata,
)
from litellm.types.llms.bedrock import *
from litellm.types.llms.openai import (
AllMessageValues,
@ -1652,6 +1658,13 @@ class AmazonConverseConfig(BaseConfig):
user_continue_message=litellm_params.pop("user_continue_message", None),
)
request_metadata: Final = resolve_bedrock_request_metadata(
litellm_params=litellm_params, caller_metadata=_data.get("requestMetadata")
)
if bedrock_request_metadata_is_owned():
_data.pop("requestMetadata", None)
if request_metadata is not None:
_data["requestMetadata"] = request_metadata
data: Final[RequestObject] = {"messages": bedrock_messages, **_data}
return data
@ -1705,6 +1718,13 @@ class AmazonConverseConfig(BaseConfig):
user_continue_message=litellm_params.pop("user_continue_message", None),
)
request_metadata: Final = resolve_bedrock_request_metadata(
litellm_params=litellm_params, caller_metadata=_data.get("requestMetadata")
)
if bedrock_request_metadata_is_owned():
_data.pop("requestMetadata", None)
if request_metadata is not None:
_data["requestMetadata"] = request_metadata
data: Final[RequestObject] = {"messages": bedrock_messages, **_data}
return data
@ -2258,7 +2278,8 @@ class AmazonConverseConfig(BaseConfig):
) -> dict:
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
return headers
owned_names, metadata_headers = bedrock_request_metadata_headers(litellm_params)
return merge_bedrock_invoke_headers(headers, (), metadata_headers, owned_names)
def should_fake_stream(
self,

11
litellm/llms/bedrock/chat/invoke_transformations/amazon_openai_transformation.py
View file

@ -13,6 +13,10 @@ import httpx
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.bedrock.common_utils import BedrockError
from litellm.llms.bedrock.request_metadata import (
bedrock_request_metadata_headers,
merge_bedrock_invoke_headers,
)
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.passthrough.utils import CommonUtils
from litellm.types.llms.openai import AllMessageValues
@ -169,9 +173,12 @@ class AmazonBedrockOpenAIConfig(OpenAIGPTConfig, BaseAWSLLM):
"""
Validate the environment and return headers.
For Bedrock, we don't need Bearer token auth since we use AWS SigV4.
For Bedrock, we don't need Bearer token auth since we use AWS SigV4. This path signs the
same ``/model/{id}/invoke`` endpoint as ``AmazonInvokeConfig``, so it owns the request
metadata header on the same terms rather than letting a caller supply it.
"""
return headers
owned_names, metadata_headers = bedrock_request_metadata_headers(litellm_params)
return merge_bedrock_invoke_headers(headers, (), metadata_headers, owned_names)
def get_error_class(self, error_message: str, status_code: int, headers: dict | httpx.Headers) -> BedrockError:
"""Return the appropriate error class for Bedrock."""

20
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
View file

@ -20,6 +20,10 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.llms.bedrock.chat.invoke_handler import make_call, make_sync_call
from litellm.llms.bedrock.common_utils import BedrockError
from litellm.llms.bedrock.request_metadata import (
bedrock_request_metadata_headers,
merge_bedrock_invoke_headers,
)
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,
@ -417,15 +421,13 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
api_base: str | None = None,
) -> dict:
raw_guardrail_config: Final = optional_params.pop("guardrailConfig", None)
if raw_guardrail_config is None:
return headers
existing_header_names: Final = frozenset(name.lower() for name in headers)
guardrail_headers: Final = {
name: value
for name, value in _bedrock_invoke_guardrail_headers(raw_guardrail_config).items()
if name.lower() not in existing_header_names
}
return {**headers, **guardrail_headers}
guardrail_headers: Final = (
()
if raw_guardrail_config is None
else tuple(_bedrock_invoke_guardrail_headers(raw_guardrail_config).items())
)
owned_names, metadata_headers = bedrock_request_metadata_headers(litellm_params)
return merge_bedrock_invoke_headers(headers, guardrail_headers, metadata_headers, owned_names)
def get_error_class(self, error_message: str, status_code: int, headers: dict | httpx.Headers) -> BaseLLMException:
return BedrockError(status_code=status_code, message=error_message)

26
litellm/llms/bedrock/files/transformation.py
View file

@ -64,6 +64,10 @@ from ..common_utils import BedrockError, merge_bedrock_aws_request_params, resol
# Same pattern as the `upload_url` handoff in `transform_create_file_request`.
S3_SIGNED_GET_HEADERS_PARAM: Final = "_s3_signed_get_headers"
# litellm_params key carrying the size of the body uploaded to S3, handed from
# `transform_create_file_request` to `transform_create_file_response`.
UPLOAD_CONTENT_LENGTH_PARAM: Final = "_s3_upload_content_length"
def _frozen_mapping(items: Iterable[tuple[str, object]]) -> Mapping[str, object]:
return MappingProxyType(dict(items))
@ -197,6 +201,18 @@ def get_configured_s3_bucket_name(litellm_params: Mapping[str, object]) -> str:
return bucket_name
def _uploaded_object_size(litellm_params: Mapping[str, object], raw_response: Response) -> int:
"""
S3 answers PutObject with an empty body, so the stored object size comes from the
signed request recorded by `transform_create_file_request`, not the response headers.
"""
uploaded_size: Final = litellm_params.get(UPLOAD_CONTENT_LENGTH_PARAM)
if isinstance(uploaded_size, int):
return uploaded_size
response_content_length: Final = raw_response.headers.get("Content-Length", "0")
return int(response_content_length) if response_content_length.isdigit() else 0
class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
"""
Config for Bedrock Files - handles S3 uploads for Bedrock batch processing
@ -924,6 +940,8 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
)
litellm_params["upload_url"] = api_base
upload_content_length: Final = len(file_content.encode("utf-8"))
litellm_params[UPLOAD_CONTENT_LENGTH_PARAM] = upload_content_length # rebind-ok: same handoff as upload_url
# Return a dict that tells the HTTP handler exactly what to do
return {
@ -1081,12 +1099,6 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
"""
Transform S3 File upload response into OpenAI-style FileObject
"""
# For S3 uploads, we typically get an ETag and other metadata
response_headers: Final = raw_response.headers
# Extract S3 object information from the response
# S3 PUT object returns ETag and other metadata in headers
content_length: Final[str] = response_headers.get("Content-Length", "0")
# Use the actual upload URL that was used for the S3 upload
upload_url: Final = litellm_params.get("upload_url")
file_id: str = ""
@ -1101,7 +1113,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
filename=filename,
created_at=int(time.time()), # Current timestamp
status="uploaded",
bytes=int(content_length) if content_length.isdigit() else 0,
bytes=_uploaded_object_size(litellm_params=litellm_params, raw_response=raw_response),
object="file",
)

39
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -1,4 +1,5 @@
from collections.abc import AsyncIterator
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, cast
import httpx
@ -37,6 +38,10 @@ from litellm.llms.bedrock.common_utils import (
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
)
from litellm.llms.bedrock.request_metadata import (
bedrock_request_metadata_headers,
merge_bedrock_invoke_headers,
)
from litellm.types.llms.anthropic import (
ANTHROPIC_BETA_HEADER_VALUES,
ANTHROPIC_TOOL_SEARCH_BETA_HEADER,
@ -89,7 +94,8 @@ class AmazonAnthropicClaudeMessagesConfig(
api_key: str | None = None,
api_base: str | None = None,
) -> tuple[dict, str | None]:
return headers, api_base
owned_names, metadata_headers = bedrock_request_metadata_headers(litellm_params)
return merge_bedrock_invoke_headers(headers, (), metadata_headers, owned_names), api_base
def sign_request(
self,
@ -956,13 +962,32 @@ class AmazonAnthropicClaudeMessagesStreamDecoder(AWSEventStreamDecoder):
Bedrock returns usage metrics using camelCase keys. Convert these to
the Anthropic `/v1/messages` specification so callers receive a
consistent response shape when streaming.
Token counts already present in the chunk's own Anthropic usage block
win over the invocationMetrics-derived ones, and cache token fields
(``cache_read_input_tokens`` / ``cache_creation_input_tokens`` on
``message_stop.usage``, or ``cacheReadInputTokenCount`` /
``cacheWriteInputTokenCount`` inside the invocation metrics) are
preserved: ``invocationMetrics.inputTokenCount`` excludes cache reads
and writes, so replacing the whole usage block with input/output counts
alone drops the cache breakdown, ``_promote_message_stop_usage`` has
nothing left to promote, and cache tokens end up billed at $0.
"""
amazon_bedrock_invocation_metrics: Final = chunk_data.pop("amazon-bedrock-invocationMetrics", {})
if amazon_bedrock_invocation_metrics:
anthropic_usage: Final = {}
if "inputTokenCount" in amazon_bedrock_invocation_metrics:
anthropic_usage["input_tokens"] = amazon_bedrock_invocation_metrics["inputTokenCount"]
if "outputTokenCount" in amazon_bedrock_invocation_metrics:
anthropic_usage["output_tokens"] = amazon_bedrock_invocation_metrics["outputTokenCount"]
chunk_data["usage"] = anthropic_usage
existing_usage: Final = chunk_data.get("usage")
preserved_usage: Final = existing_usage if isinstance(existing_usage, dict) else MappingProxyType({})
metrics_usage: Final = MappingProxyType(
{
anthropic_key: amazon_bedrock_invocation_metrics[metrics_key]
for anthropic_key, metrics_key in (
("input_tokens", "inputTokenCount"),
("output_tokens", "outputTokenCount"),
("cache_read_input_tokens", "cacheReadInputTokenCount"),
("cache_creation_input_tokens", "cacheWriteInputTokenCount"),
)
if metrics_key in amazon_bedrock_invocation_metrics
}
)
chunk_data["usage"] = {**metrics_usage, **preserved_usage}
return chunk_data

199
litellm/llms/bedrock/request_metadata.py Normal file
View file

@ -0,0 +1,199 @@
"""
Resolve AWS Bedrock ``requestMetadata`` from LiteLLM proxy identity and caller metadata.
Bedrock attaches request metadata to CloudTrail records and to the dimension AWS Cost
Explorer groups on, so everything here is opt-in: nothing is forwarded unless the operator
sets ``litellm.bedrock_request_metadata_fields`` (``litellm_settings`` on the proxy).
Two properties are load-bearing for that billing record and are asserted by the tests:
proxy identity is resolved first so it can never be evicted by caller-supplied pairs, and the
whole ``user_api_key_`` prefix is reserved so a caller cannot write a proxy-authoritative
looking key. Values that break Bedrock's constraints are dropped rather than sanitised or
rejected, because an operator flipping this setting on must not turn a working request into a
400 and a silently rewritten attribution key is worse than an absent one.
"""
from __future__ import annotations
import json
import re
from collections.abc import Mapping
from typing import Final
import litellm
BEDROCK_REQUEST_METADATA_HEADER: Final = "X-Amzn-Bedrock-Request-Metadata"
BEDROCK_REQUEST_METADATA_MAX_PAIRS: Final = 16
BEDROCK_REQUEST_METADATA_IDENTITY_PREFIX: Final = "user_api_key_"
BEDROCK_REQUEST_METADATA_CLIENT_FIELD: Final = "spend_logs_metadata"
_METADATA_PARAM_NAMES: Final[tuple[str, ...]] = ("metadata", "litellm_metadata")
_KEY_PATTERN: Final = re.compile(r"^[a-zA-Z0-9\s:_@$#=/+,.-]{1,256}$")
_VALUE_PATTERN: Final = re.compile(r"^[a-zA-Z0-9\s:_@$#=/+,.-]{0,256}$")
_OWNED_HEADER_NAMES: Final[frozenset[str]] = frozenset((BEDROCK_REQUEST_METADATA_HEADER.lower(),))
def _is_forwardable(key: str, value: str) -> bool:
return _KEY_PATTERN.match(key) is not None and _VALUE_PATTERN.match(value) is not None
def _text_pairs(source: object) -> tuple[tuple[str, str], ...]:
if not isinstance(source, Mapping):
return ()
return tuple((key, value) for key, value in source.items() if isinstance(key, str) and isinstance(value, str))
def _allowed_fields() -> tuple[str, ...]:
"""
The operator allow-list, deduplicated so a field repeated in config cannot consume a second
reserved slot and shrink the client budget for nothing. First occurrence wins, which keeps
the operator's declared precedence intact.
"""
configured: Final[object] = litellm.bedrock_request_metadata_fields
if not isinstance(configured, (list, tuple)):
return ()
fields: Final = tuple(str(field) for field in configured)
return tuple(field for index, field in enumerate(fields) if field not in fields[:index])
def _metadata_sources(litellm_params: Mapping[str, object] | None) -> tuple[Mapping[str, object], ...]:
"""``metadata`` on /v1/chat/completions, ``litellm_metadata`` on the LITELLM_METADATA_ROUTES."""
if litellm_params is None:
return ()
return tuple(
source
for name in _METADATA_PARAM_NAMES
for source in (litellm_params.get(name),)
if isinstance(source, Mapping)
)
def _identity_pairs(
sources: tuple[Mapping[str, object], ...],
allowed_fields: tuple[str, ...],
) -> tuple[tuple[str, str], ...]:
return tuple(
(field, value)
for field in allowed_fields
if field.startswith(BEDROCK_REQUEST_METADATA_IDENTITY_PREFIX)
for value in (_first_text(sources, field),)
if value is not None and _is_forwardable(field, value)
)[:BEDROCK_REQUEST_METADATA_MAX_PAIRS]
def _first_text(sources: tuple[Mapping[str, object], ...], field: str) -> str | None:
return next((value for source in sources if isinstance(value := source.get(field), str)), None)
def _client_pairs(
sources: tuple[Mapping[str, object], ...],
allowed_fields: tuple[str, ...],
caller_metadata: object,
budget: int,
) -> tuple[tuple[str, str], ...]:
spend_logs_pairs: Final = (
tuple(pair for source in sources for pair in _text_pairs(source.get(BEDROCK_REQUEST_METADATA_CLIENT_FIELD)))
if BEDROCK_REQUEST_METADATA_CLIENT_FIELD in allowed_fields
else ()
)
candidates: Final = tuple(
(key, value)
for key, value in (*_text_pairs(caller_metadata), *spend_logs_pairs)
if not key.startswith(BEDROCK_REQUEST_METADATA_IDENTITY_PREFIX) and _is_forwardable(key, value)
)
return tuple(
pair
for index, pair in enumerate(candidates)
if pair[0] not in tuple(earlier for earlier, _ in candidates[:index])
)[:budget]
def resolve_bedrock_request_metadata(
litellm_params: Mapping[str, object] | None,
caller_metadata: object = None,
) -> dict[str, str] | None:
"""
Resolve the ``requestMetadata`` pairs to send to Bedrock, or ``None`` when the feature is
off or nothing survives Bedrock's constraints. The result is a plain dict because it is
written straight onto the Converse body, which Bedrock types as ``dict[str, str]``.
``caller_metadata`` is any ``requestMetadata`` the caller passed explicitly. It has already
been validated (and rejected with a 400) by the Converse transformation, so it is only
filtered here for the reserved identity prefix and the remaining slot budget.
"""
allowed_fields: Final = _allowed_fields()
if not allowed_fields:
return None
sources: Final = _metadata_sources(litellm_params)
identity: Final = _identity_pairs(sources, allowed_fields)
client: Final = _client_pairs(
sources=sources,
allowed_fields=allowed_fields,
caller_metadata=caller_metadata,
budget=BEDROCK_REQUEST_METADATA_MAX_PAIRS - len(identity),
)
resolved: Final = {key: value for key, value in (*identity, *client)}
return resolved or None
def bedrock_request_metadata_is_owned() -> bool:
"""
Whether the proxy OWNS the request-metadata field and header name for this request.
Ownership follows the operator's opt-in alone, never whether anything resolved, because a
caller can suppress the resolver by omitting the allow-listed fields or by sending values
that all fail Bedrock's rules. Owned-but-empty has to mean "absent on the wire" rather than
"fall back to whatever the caller supplied", or the reserved-prefix guarantee is bypassable
by anyone who can make the resolver produce nothing.
"""
return bool(_allowed_fields())
def bedrock_request_metadata_headers(
litellm_params: Mapping[str, object] | None,
) -> tuple[frozenset[str], tuple[tuple[str, str], ...]]:
"""
The signed ``X-Amzn-Bedrock-Request-Metadata`` header for the Invoke paths, which have no
body field for request metadata.
Returns the header names the proxy OWNS and, separately, the pairs to send. Ownership is
reported whenever forwarding is enabled, including when nothing resolves, because a caller
can suppress the resolver (omit the allow-listed fields, or send values that all fail
Bedrock's rules) and an owned-but-empty result must still evict the caller's header rather
than fall back to it.
"""
if not bedrock_request_metadata_is_owned():
return frozenset(), ()
resolved: Final = resolve_bedrock_request_metadata(litellm_params)
if resolved is None:
return _OWNED_HEADER_NAMES, ()
return _OWNED_HEADER_NAMES, ((BEDROCK_REQUEST_METADATA_HEADER, json.dumps(resolved, separators=(",", ":"))),)
def merge_bedrock_invoke_headers(
headers: dict[str, str],
caller_owned: tuple[tuple[str, str], ...],
proxy_owned: tuple[tuple[str, str], ...],
proxy_owned_names: frozenset[str],
) -> dict[str, str]:
"""
Merge the ``X-Amzn-*`` headers the Invoke paths derive from params.
``caller_owned`` (the guardrail headers) defers to a header the caller already set, which is
the long-standing behaviour for those. ``proxy_owned_names`` are dropped from the caller's
headers unconditionally and re-supplied only from ``proxy_owned``, because those names carry
proxy-authenticated identity into an AWS billing record that the caller must not be able to
write. Names are compared case-insensitively so a caller cannot leave a second spelling in
the dict and let the transport pick the winner.
"""
if not caller_owned and not proxy_owned and not proxy_owned_names:
return headers
existing_names: Final = frozenset(name.lower() for name in headers)
return {
name: value
for name, value in (
*((n, v) for n, v in headers.items() if n.lower() not in proxy_owned_names),
*((n, v) for n, v in caller_owned if n.lower() not in existing_names),
*proxy_owned,
)
}

15
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -49,6 +49,7 @@ from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from litellm.llms.base_llm.ocr.transformation import BaseOCRConfig, OCRResponse
from litellm.llms.base_llm.realtime.http_transformation import BaseRealtimeHTTPConfig
from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig
from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
from litellm.llms.base_llm.responses.transformation import BaseResponsesAPIConfig
@ -5930,10 +5931,10 @@ class BaseLLMHTTPHandler:
self,
api_base: str,
api_key: str,
request_data: dict[str, Any],
request_data: dict[str, object],
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout,
provider_config: Any | None = None,
provider_config: BaseRealtimeHTTPConfig | None = None,
model: str | None = None,
extra_headers: dict[str, object] | None = None,
client: HTTPHandler | AsyncHTTPHandler | None = None,
@ -5963,10 +5964,10 @@ class BaseLLMHTTPHandler:
self,
api_base: str,
api_key: str,
request_data: dict[str, Any],
request_data: dict[str, object],
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout,
provider_config: Any | None = None,
provider_config: BaseRealtimeHTTPConfig | None = None,
model: str | None = None,
extra_headers: dict[str, object] | None = None,
client: HTTPHandler | AsyncHTTPHandler | None = None,
@ -5992,7 +5993,7 @@ class BaseLLMHTTPHandler:
endpoint: Literal["client_secrets", "transcription_sessions"],
api_base: str,
api_key: str,
request_data: dict[str, Any],
request_data: dict[str, object],
logging_obj: LiteLLMLoggingObj,
timeout: float | httpx.Timeout,
provider_config: Any | None = None,
@ -11077,7 +11078,7 @@ class BaseLLMHTTPHandler:
client: HTTPHandler | AsyncHTTPHandler | None = None,
stream: bool = False,
litellm_metadata: dict[str, object] | None = None,
system_instruction: Any | None = None,
system_instruction: object | None = None,
) -> Any:
"""
Handles Google GenAI generate content requests.
@ -11208,7 +11209,7 @@ class BaseLLMHTTPHandler:
client: AsyncHTTPHandler | None = None,
stream: bool = False,
litellm_metadata: dict[str, object] | None = None,
system_instruction: Any | None = None,
system_instruction: object | None = None,
) -> Any:
"""
Async version of the generate content handler.

1
litellm/llms/databricks/chat/transformation.py
View file

@ -733,6 +733,7 @@ class DatabricksChatResponseIterator(BaseModelResponseIterator):
created=chunk["created"],
model=chunk["model"],
choices=translated_choices,
usage=chunk.get("usage"),
)
except KeyError as e:
raise DatabricksException(

18
litellm/llms/fireworks_ai/chat/transformation.py
View file

@ -39,7 +39,11 @@ from ...openai.chat.gpt_transformation import (
OpenAIChatCompletionStreamingHandler,
OpenAIGPTConfig,
)
from ..common_utils import FireworksAIException, FireworksAIMixin
from ..common_utils import (
FireworksAIException,
FireworksAIMixin,
resolve_fireworks_resource_name,
)
def _extract_fireworks_hidden_params(payload: dict) -> dict:
@ -627,12 +631,10 @@ class FireworksAIConfig(FireworksAIMixin, OpenAIGPTConfig):
litellm_params: dict,
headers: dict,
) -> dict:
if not model.startswith("accounts/") and "#" not in model:
if model.endswith("-fast"):
model = f"accounts/fireworks/routers/{model}"
else:
model = f"accounts/fireworks/models/{model}"
messages = self._transform_messages_helper(messages=messages, model=model, litellm_params=litellm_params)
resolved_model: Final = resolve_fireworks_resource_name(model)
messages = self._transform_messages_helper(
messages=messages, model=resolved_model, litellm_params=litellm_params
)
if "tools" in optional_params and optional_params["tools"] is not None:
tools: Final = self._transform_tools(tools=optional_params["tools"])
optional_params["tools"] = tools
@ -646,7 +648,7 @@ class FireworksAIConfig(FireworksAIMixin, OpenAIGPTConfig):
"include_usage": True,
}
return super().transform_request(
model=model,
model=resolved_model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,

11
litellm/llms/fireworks_ai/common_utils.py
View file

@ -29,6 +29,17 @@ def get_fireworks_session_id(litellm_params: dict) -> str | None:
return None
def resolve_fireworks_resource_name(model: str) -> str:
stripped: Final = model.removeprefix("fireworks_ai/")
if stripped.startswith("accounts/") or "#" in stripped:
return stripped
if stripped.startswith(("routers/", "models/")):
return f"accounts/fireworks/{stripped}"
if stripped.endswith("-fast"):
return f"accounts/fireworks/routers/{stripped}"
return f"accounts/fireworks/models/{stripped}"
class FireworksAIMixin:
"""
Common Base Config functions across Fireworks AI Endpoints

7
litellm/llms/fireworks_ai/completion/transformation.py
View file

@ -13,7 +13,7 @@ from ..chat.transformation import (
FireworksAIConfig,
effort_from_chat_template_kwargs,
)
from ..common_utils import FireworksAIMixin
from ..common_utils import FireworksAIMixin, resolve_fireworks_resource_name
_TEXT_COMPLETION_STRIP_PARAMS: Final = (
frozenset({"truncate_prompt_tokens", "prompt_truncate_len"}) | NIM_VLLM_STRIP_PARAMS
@ -167,11 +167,8 @@ class FireworksAITextCompletionConfig(FireworksAIMixin, BaseTextCompletionConfig
translated_params: Final = self.map_extra_body_params(optional_params=optional_params, model=model)
prompt: Final = _transform_prompt(messages=messages)
if not model.startswith("accounts/") and "#" not in model:
model = f"accounts/fireworks/models/{model}"
data: Final = {
"model": model,
"model": resolve_fireworks_resource_name(model),
"prompt": prompt,
**translated_params,
}

38
litellm/model_prices_and_context_window_backup.json
View file

@ -12011,6 +12011,7 @@
"output_cost_per_token": 5e-06,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_native_structured_output": true,
"supports_computer_use": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
@ -12033,6 +12034,7 @@
"output_cost_per_token": 5e-06,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_native_structured_output": true,
"supports_computer_use": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
@ -12254,6 +12256,7 @@
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_native_structured_output": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
@ -18036,6 +18039,7 @@
"supports_tool_choice": true
},
"ft:gpt-3.5-turbo-0613": {
"deprecation_date": "2026-10-23",
"input_cost_per_token": 3e-06,
"litellm_provider": "openai",
"max_input_tokens": 4096,
@ -18047,6 +18051,7 @@
"supports_tool_choice": true
},
"ft:gpt-3.5-turbo-1106": {
"deprecation_date": "2026-10-23",
"input_cost_per_token": 3e-06,
"litellm_provider": "openai",
"max_input_tokens": 16385,
@ -22986,6 +22991,7 @@
"supports_tool_choice": true
},
"gpt-3.5-turbo-16k": {
"deprecation_date": "2026-10-23",
"input_cost_per_token": 3e-06,
"litellm_provider": "openai",
"max_input_tokens": 16385,
@ -23120,6 +23126,7 @@
"supports_vision": true
},
"gpt-4-turbo-preview": {
"deprecation_date": "2026-03-26",
"input_cost_per_token": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 128000,
@ -46512,6 +46519,21 @@
"rpm": 10,
"gemini_audio_only_live": true
},
"gemini/gemini-3.1-flash-tts-preview": {
"input_cost_per_token": 1e-06,
"litellm_provider": "gemini",
"max_input_tokens": 8192,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "audio_speech",
"output_cost_per_token": 2e-05,
"source": "https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-tts-preview",
"supported_endpoints": [
"/v1/audio/speech"
],
"tpm": 4000000,
"rpm": 10
},
"gemini-2.5-flash-preview-tts": {
"input_cost_per_token": 3e-07,
"litellm_provider": "gemini",
@ -47717,8 +47739,8 @@
"input_cost_per_token_cache_hit": 2.8e-09,
"litellm_provider": "deepseek",
"max_input_tokens": 1000000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"max_output_tokens": 393216,
"max_tokens": 393216,
"mode": "chat",
"output_cost_per_token": 2.8e-07,
"source": "https://api-docs.deepseek.com/quick_start/pricing",
@ -47743,8 +47765,8 @@
"input_cost_per_token_cache_hit": 3.625e-09,
"litellm_provider": "deepseek",
"max_input_tokens": 1000000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"max_output_tokens": 393216,
"max_tokens": 393216,
"mode": "chat",
"output_cost_per_token": 8.7e-07,
"source": "https://api-docs.deepseek.com/quick_start/pricing",
@ -47769,8 +47791,8 @@
"input_cost_per_token_cache_hit": 2.8e-09,
"litellm_provider": "deepseek",
"max_input_tokens": 1000000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"max_output_tokens": 393216,
"max_tokens": 393216,
"mode": "chat",
"output_cost_per_token": 2.8e-07,
"source": "https://api-docs.deepseek.com/quick_start/pricing",
@ -47795,8 +47817,8 @@
"input_cost_per_token_cache_hit": 3.625e-09,
"litellm_provider": "deepseek",
"max_input_tokens": 1000000,
"max_output_tokens": 8192,
"max_tokens": 8192,
"max_output_tokens": 393216,
"max_tokens": 393216,
"mode": "chat",
"output_cost_per_token": 8.7e-07,
"source": "https://api-docs.deepseek.com/quick_start/pricing",

4
litellm/passthrough/utils.py
View file

@ -18,6 +18,7 @@ _PASS_THROUGH_PROTECTED_HEADERS: Final[frozenset] = frozenset(
"x-goog-api-key",
"host",
"content-length",
"accept-encoding",
}
)
@ -69,6 +70,9 @@ class BasePassthroughUtils:
# Header We Should NOT forward
request_headers.pop("content-length", None)
request_headers.pop("host", None)
# accept-encoding must stay client-negotiated: forwarding e.g. "br" when
# the brotli package is absent relays undecodable bytes to the caller
request_headers.pop("accept-encoding", None)
custom_header_names: Final = {header_name.lower() for header_name in headers}
for header_name in list(request_headers.keys()):

11
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -83,7 +83,7 @@ class UnloadableEntitlementError(Exception):
def _parse_mcp_server_names_from_path(path: str, mcp_servers_header: list[str] | None = None) -> list[str] | None:
"""Resolve the single MCP server name a cold-start passthrough bypass may
target. Delegates parsing to
:meth:`MCPRequestHandler._extract_target_server_names_from_path` so the
:meth:`MCPRequestHandler.extract_target_server_names_from_path` so the
names used here always match the names downstream routing uses; returns
``None`` whenever the bypass must not activate (aggregate ``/mcp``,
multi-server CSV paths, or any other unrecognized path).
@ -94,7 +94,7 @@ def _parse_mcp_server_names_from_path(path: str, mcp_servers_header: list[str] |
header/path mismatch here is a sign of a confused or hostile caller —
refuse the cold-start bypass rather than admit anonymously based on the
path while the header advertises a stricter, non-passthrough target."""
servers: Final = MCPRequestHandler._extract_target_server_names_from_path(path)
servers: Final = MCPRequestHandler.extract_target_server_names_from_path(path)
if len(servers) != 1:
verbose_logger.debug(
"MCP cold-start: path %r resolved to %r; passthrough 401 bypass "
@ -215,7 +215,7 @@ def _is_gateway_dcr_challenge_scope(
return False
if _has_client_supplied_mcp_auth(mcp_auth_header, mcp_server_auth_headers):
return False
if len(MCPRequestHandler._extract_target_server_names_from_path(route)) == 0:
if len(MCPRequestHandler.extract_target_server_names_from_path(route)) == 0:
return True
return _gateway_dcr_challenge_target(route, mcp_servers, client_ip) is not None
@ -579,7 +579,7 @@ class MCPRequestHandler:
return oauth2_headers, raw_headers, mcp_auth_header, mcp_server_auth_headers
@staticmethod
def _extract_target_server_names_from_path(path: str) -> list[str]:
def extract_target_server_names_from_path(path: str) -> list[str]:
"""
Extract the target MCP server name(s) from the standard MCP transport
URL patterns: ``/mcp/{server_name_or_csv}[/...]`` and
@ -836,6 +836,7 @@ class MCPRequestHandler:
case SessionBearerAdmitted():
try:
admitted: Final = await MCPRequestHandler._reload_admitted_user(result.principal.user_id)
admitted.mcp_session_resource_server_id = result.principal.resource_server_id
await MCPRequestHandler._enforce_admitted_live_policy(
admitted=admitted, request=request, route=route
)
@ -1168,7 +1169,7 @@ class MCPRequestHandler:
(header/path TOCTOU). For non-``/mcp/...`` paths (where the path
does not encode targets), fall back to the header.
"""
path_targets: Final = MCPRequestHandler._extract_target_server_names_from_path(path)
path_targets: Final = MCPRequestHandler.extract_target_server_names_from_path(path)
if path_targets:
return path_targets
# Path did not resolve to /mcp/... targets — trust the header

148
litellm/proxy/_experimental/mcp_server/db.py
View file

@ -4,7 +4,7 @@ import hashlib
import json
from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, Any, Final, TypedDict, cast
from typing import TYPE_CHECKING, Any, Final, Protocol, TypedDict, TypeVar, cast
from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
@ -45,10 +45,47 @@ from litellm.types.mcp import MCPCredentials
if TYPE_CHECKING:
from prisma import models as prisma_db_models
from prisma import types as prisma_db_types
from prisma.actions import LiteLLM_MCPUserCredentialsActions, LiteLLM_MCPUserEnvVarsActions
from litellm.types.mcp_server.mcp_server_manager import MCPServer
_RowT = TypeVar("_RowT")
class _TableActions(Protocol[_RowT]):
async def find_unique(
self, where: Mapping[str, object], include: Mapping[str, object] | None = None
) -> _RowT | None: ...
async def find_many(
self,
take: int | None = None,
where: Mapping[str, object] | None = None,
order: Mapping[str, object] | None = None,
) -> list[_RowT]: ...
async def create(self, data: Mapping[str, object]) -> _RowT: ...
async def upsert(self, where: Mapping[str, object], data: Mapping[str, object]) -> _RowT: ...
async def update(self, where: Mapping[str, object], data: Mapping[str, object]) -> _RowT | None: ...
async def delete(self, where: Mapping[str, object]) -> _RowT | None: ...
async def delete_many(self, where: Mapping[str, object] | None = None) -> int: ...
class _UserEnvVarsTransactionClient(Protocol):
litellm_mcpuserenvvars: "_TableActions[prisma_db_models.LiteLLM_MCPUserEnvVars]"
async def execute_raw(self, query: str, *args: object) -> int: ...
class _UserEnvVarsTransaction(Protocol):
async def __aenter__(self) -> _UserEnvVarsTransactionClient: ...
async def __aexit__(self, exc_type: object, exc_value: object, traceback: object) -> bool | None: ...
_AUTH_FLOW_SCOPED_FIELDS: Final["frozenset[str]"] = frozenset(
{
"issuer",
@ -434,23 +471,54 @@ def _credentials_blob_to_mutable_dict(blob: str | Mapping[str, object]) -> dict[
return parsed_blob
def _mcp_server_table_actions(
prisma_client: PrismaClient,
) -> "_TableActions[prisma_db_models.LiteLLM_MCPServerTable]":
table: Final[_TableActions[prisma_db_models.LiteLLM_MCPServerTable]] = MCPServerRepository(prisma_client).table
return table
def _verification_token_table_actions(
prisma_client: PrismaClient,
) -> "_TableActions[prisma_db_models.LiteLLM_VerificationToken]":
table: Final[_TableActions[prisma_db_models.LiteLLM_VerificationToken]] = VerificationTokenRepository(
prisma_client
).table
return table
def _team_table_actions(
prisma_client: PrismaClient,
) -> "_TableActions[prisma_db_models.LiteLLM_TeamTable]":
table: Final[_TableActions[prisma_db_models.LiteLLM_TeamTable]] = TeamRepository(prisma_client).table
return table
def _oauth_client_table_actions(
prisma_client: PrismaClient,
) -> "_TableActions[prisma_db_models.LiteLLM_MCPServerOAuthClient]":
table: Final[_TableActions[prisma_db_models.LiteLLM_MCPServerOAuthClient]] = MCPServerOAuthClientRepository(
prisma_client
).table
return table
def _db_transaction_manager(prisma_client: PrismaClient) -> _UserEnvVarsTransaction:
manager: Final[_UserEnvVarsTransaction] = prisma_client.db.tx()
return manager
async def _db_find_mcp_server_rows(
prisma_client: PrismaClient,
where: "prisma_db_types.LiteLLM_MCPServerTableWhereInput | None" = None,
) -> "list[prisma_db_models.LiteLLM_MCPServerTable]":
rows: list[prisma_db_models.LiteLLM_MCPServerTable] = await MCPServerRepository(prisma_client).table.find_many(
where=where
)
return rows
return await _mcp_server_table_actions(prisma_client).find_many(where=where)
async def _db_find_mcp_server_row(
prisma_client: PrismaClient, server_id: str
) -> "prisma_db_models.LiteLLM_MCPServerTable | None":
row: prisma_db_models.LiteLLM_MCPServerTable | None = await MCPServerRepository(prisma_client).table.find_unique(
where={"server_id": server_id}
)
return row
return await _mcp_server_table_actions(prisma_client).find_unique(where={"server_id": server_id})
async def _db_update_mcp_server_row(
@ -467,19 +535,17 @@ async def _db_update_mcp_server_row(
def _user_credential_actions(
prisma_client: PrismaClient,
) -> "LiteLLM_MCPUserCredentialsActions[prisma_db_models.LiteLLM_MCPUserCredentials]":
table: Final[LiteLLM_MCPUserCredentialsActions[prisma_db_models.LiteLLM_MCPUserCredentials]] = (
MCPUserCredentialsRepository(prisma_client).table
)
) -> "_TableActions[prisma_db_models.LiteLLM_MCPUserCredentials]":
table: Final[_TableActions[prisma_db_models.LiteLLM_MCPUserCredentials]] = MCPUserCredentialsRepository(
prisma_client
).table
return table
def _user_env_var_actions(
prisma_client: PrismaClient,
) -> "LiteLLM_MCPUserEnvVarsActions[prisma_db_models.LiteLLM_MCPUserEnvVars]":
table: Final[LiteLLM_MCPUserEnvVarsActions[prisma_db_models.LiteLLM_MCPUserEnvVars]] = (
prisma_client.db.litellm_mcpuserenvvars
)
) -> "_TableActions[prisma_db_models.LiteLLM_MCPUserEnvVars]":
table: Final[_TableActions[prisma_db_models.LiteLLM_MCPUserEnvVars]] = prisma_client.db.litellm_mcpuserenvvars
return table
@ -501,7 +567,7 @@ async def _db_find_user_credential_rows(
async def _db_upsert_user_credential_row(
prisma_client: PrismaClient, user_id: str, server_id: str, credential_b64: str
) -> None:
await MCPUserCredentialsRepository(prisma_client).table.upsert(
await _user_credential_actions(prisma_client).upsert(
where={"user_id_server_id": {"user_id": user_id, "server_id": server_id}},
data={
"create": {
@ -592,9 +658,9 @@ async def get_mcp_servers(prisma_client: PrismaClient, server_ids: Iterable[str]
"""
Returns the matching mcp servers from the db with the server_ids
"""
_mcp_servers: Final[list[prisma_db_models.LiteLLM_MCPServerTable]] = await MCPServerRepository(
_mcp_servers: Final[list[prisma_db_models.LiteLLM_MCPServerTable]] = await _mcp_server_table_actions(
prisma_client
).table.find_many(
).find_many(
where={
"server_id": {"in": server_ids},
}
@ -612,9 +678,9 @@ async def get_mcp_servers_by_verificationtoken(prisma_client: PrismaClient, toke
"""
Returns the mcp servers from the db for the verification token
"""
verification_token_record: prisma_db_models.LiteLLM_VerificationToken | None = await VerificationTokenRepository(
prisma_client
).table.find_unique(
verification_token_record: (
prisma_db_models.LiteLLM_VerificationToken | None
) = await _verification_token_table_actions(prisma_client).find_unique(
where={
"token": token,
},
@ -633,7 +699,7 @@ async def get_mcp_servers_by_team(prisma_client: PrismaClient, team_id: str) ->
"""
Returns the mcp servers from the db for the team id
"""
team_record: prisma_db_models.LiteLLM_TeamTable | None = await TeamRepository(prisma_client).table.find_unique(
team_record: prisma_db_models.LiteLLM_TeamTable | None = await _team_table_actions(prisma_client).find_unique(
where={
"team_id": team_id,
},
@ -760,9 +826,9 @@ async def delete_mcp_server(
if deleted_server is not None:
credential_user_ids: list[str] = []
try:
credential_rows: Sequence[
prisma_db_models.LiteLLM_MCPUserCredentials
] = await prisma_client.db.litellm_mcpusercredentials.find_many(where={"server_id": server_id})
credential_rows: Sequence[prisma_db_models.LiteLLM_MCPUserCredentials] = await _user_credential_actions(
prisma_client
).find_many(where={"server_id": server_id})
credential_user_ids = [row.user_id for row in credential_rows]
except Exception as e: # noqa: BLE001 - enumeration is best-effort; cached tokens expire by TTL
verbose_proxy_logger.warning(
@ -771,9 +837,9 @@ async def delete_mcp_server(
e,
)
for model, label in (
(prisma_client.db.litellm_mcpusercredentials, "credential"),
(prisma_client.db.litellm_mcpuserenvvars, "env var"),
(prisma_client.db.litellm_mcpserveroauthclient, "OAuth client"),
(_user_credential_actions(prisma_client), "credential"),
(_user_env_var_actions(prisma_client), "env var"),
(_oauth_client_table_actions(prisma_client), "OAuth client"),
):
try:
await model.delete_many(where={"server_id": server_id})
@ -1042,9 +1108,9 @@ async def get_mcp_server_oauth_client_credentials(prisma_client: PrismaClient, s
LiteLLM_MCPServerTable row, so their dynamically registered client lives here keyed
by server_id. The returned value is the raw credentials blob for
``_get_persisted_dcr_credentials`` to parse."""
row: Final[prisma_db_models.LiteLLM_MCPServerOAuthClient | None] = await MCPServerOAuthClientRepository(
row: Final[prisma_db_models.LiteLLM_MCPServerOAuthClient | None] = await _oauth_client_table_actions(
prisma_client
).table.find_unique(where={"server_id": server_id})
).find_unique(where={"server_id": server_id})
if row is None:
return None
return row.credentials
@ -1062,7 +1128,7 @@ async def upsert_mcp_server_oauth_client_credentials(
encrypted: Final = encrypt_credentials(credentials=MCPCredentials(**credentials), encryption_key=_get_salt_key())
blob: Final = safe_dumps(encrypted)
await MCPServerOAuthClientRepository(prisma_client).table.upsert(
await _oauth_client_table_actions(prisma_client).upsert(
where={"server_id": server_id},
data={
"create": {"server_id": server_id, "credentials": blob},
@ -1109,21 +1175,21 @@ async def rotate_mcp_server_credentials_master_key(prisma_client: PrismaClient,
continue
update_data["updated_by"] = touched_by
await MCPServerRepository(prisma_client).table.update(
await _mcp_server_table_actions(prisma_client).update(
where={"server_id": mcp_server.server_id},
data=update_data,
)
updated += 1
oauth_clients: Final[list[prisma_db_models.LiteLLM_MCPServerOAuthClient]] = await MCPServerOAuthClientRepository(
oauth_clients: Final[list[prisma_db_models.LiteLLM_MCPServerOAuthClient]] = await _oauth_client_table_actions(
prisma_client
).table.find_many()
).find_many()
oauth_updated = 0
for oauth_client in oauth_clients:
rotated_credentials = _reencrypt_mcp_credentials_blob(oauth_client.credentials, new_master_key)
if rotated_credentials is None:
continue
await MCPServerOAuthClientRepository(prisma_client).table.update(
await _oauth_client_table_actions(prisma_client).update(
where={"server_id": oauth_client.server_id},
data={"credentials": rotated_credentials},
)
@ -1813,7 +1879,9 @@ async def get_mcp_submissions(
along with a summary count breakdown by approval_status.
Mirrors get_guardrail_submissions() from guardrail_endpoints.py.
"""
rows: list[prisma_db_models.LiteLLM_MCPServerTable] = await MCPServerRepository(prisma_client).table.find_many(
rows: Final[list[prisma_db_models.LiteLLM_MCPServerTable]] = await _mcp_server_table_actions(
prisma_client
).find_many(
where={"submitted_at": {"not": None}},
order={"submitted_at": "desc"},
take=500, # safety cap; paginate if needed in a future iteration
@ -1915,7 +1983,7 @@ async def merge_user_env_vars(
"big",
signed=True,
)
async with prisma_client.db.tx() as tx:
async with _db_transaction_manager(prisma_client) as tx:
await tx.execute_raw("SELECT pg_advisory_xact_lock($1::bigint)", lock_key)
row: Final[prisma_db_models.LiteLLM_MCPUserEnvVars | None] = await tx.litellm_mcpuserenvvars.find_unique(
where={"user_id_server_id": {"user_id": user_id, "server_id": server_id}}

4
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -1655,6 +1655,7 @@ async def authorize(
code_challenge_method: str | None = None,
response_type: str | None = None,
scope: str | None = None,
resource: str | None = None,
):
# Redirect to real OAuth provider with PKCE support
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
@ -1671,6 +1672,7 @@ async def authorize(
code_challenge_method=code_challenge_method,
response_type=response_type,
session_user_id=_session_cookie_user_id(request),
resource=resource,
)
lookup_name: Final[str | None] = mcp_server_name or client_id
@ -1721,6 +1723,7 @@ async def token_endpoint(
code_verifier: str = Form(None),
refresh_token: str | None = Form(None),
scope: str | None = Form(None),
resource: str | None = Form(None),
mcp_server_name: str | None = None,
):
"""
@ -1753,6 +1756,7 @@ async def token_endpoint(
master_key=master_key,
reload_user=_reload_active_user_by_id,
cache=user_api_key_cache,
resource=resource,
)
lookup_name: Final = mcp_server_name or client_id

92
litellm/proxy/_experimental/mcp_server/gateway_dcr_flow.py
View file

@ -56,6 +56,8 @@ from litellm._logging import verbose_logger
from litellm.caching.caching import DualCache
from litellm.proxy._experimental.mcp_server.oauth_utils import (
TOKEN_NO_CACHE_HEADERS,
canonical_resource_uri,
canonicalize_url_identity,
get_request_base_url,
is_loopback_redirect_host,
validate_redirect_uri_shape,
@ -77,6 +79,7 @@ from litellm.proxy.common_utils.encrypt_decrypt_utils import (
decrypt_value_helper,
encrypt_value_helper,
)
from litellm.types.mcp_server.mcp_server_manager import MCPServer
GATEWAY_DCR_CLIENT_ID_PREFIX: Final = "llm_dcrc_"
"""Marker prefix on every gateway-issued DCR client_id so the root authorize/token
@ -169,6 +172,7 @@ class _ConnectFlow(BaseModel):
code_challenge: str = Field(min_length=1)
jti: str = Field(min_length=1)
exp: int
resource_server_id: str | None = None
class _GatewayAuthCode(BaseModel):
@ -185,6 +189,7 @@ class _GatewayAuthCode(BaseModel):
jti: str = Field(min_length=1)
iat: int
exp: int
resource_server_id: str | None = None
def is_gateway_dcr_client_id(client_id: str | None) -> bool:
@ -204,7 +209,13 @@ def _oauth_error(status_code: int, error: str, description: str) -> JSONResponse
def _seal(prefix: str, payload: BaseModel) -> str:
return prefix + encrypt_value_helper(payload.model_dump_json())
"""Serialized ``exclude_none`` for the same reason session JWTs are minted that way: an
optional claim that is unset never reaches the wire, so during a rolling deploy a blob
sealed by a new pod without the new claim set stays byte-compatible with predating pods
whose strict models forbid unknown keys. This holds for every sealed artifact and every
future optional claim by construction; it requires each optional field to default to
``None`` so reopening restores exactly what was sealed."""
return prefix + encrypt_value_helper(payload.model_dump_json(exclude_none=True))
_SealedModelT = TypeVar("_SealedModelT", bound=BaseModel)
@ -320,6 +331,44 @@ def relative_request_url(request: Request) -> str:
return f"{path}?{request.url.query}" if request.url.query else path
def resolve_scoped_resource_server(request: Request, resource: str | None) -> MCPServer | None:
"""Resolve an RFC 8707 ``resource`` value to the single gateway-managed oauth2 server it
names, or ``None`` for every other shape: absent, the aggregate resource, a foreign
host, an unparseable value, a multi-server path, an unknown name, or any server mode the
keyless gateway flow does not serve (whose protected-resource metadata never directs a
client here). ``None`` means the flow stays unscoped and byte-identical to today, so a
hostile or confused ``resource`` can never widen anything; a resolved server only ever
NARROWS the session via the sealed scope.
Resolution is an IDENTITY question, deliberately free of the per-IP visibility filter:
access is enforced where it belongs (grant intersection at admission, IP checks on the
MCP routes), while filtering here would mint an entitlement-wide UNSCOPED bearer exactly
when the caller asked to narrow, and would let authorize-time vs token-time IP drift
turn a matching redemption into a spurious ``invalid_target``."""
if resource is None:
return None
canonical: Final = canonical_resource_uri(resource)
if canonical is None:
return None
base: Final = canonicalize_url_identity(get_request_base_url(request))
if canonical == f"{base}/mcp" or not canonical.startswith(f"{base}/"):
return None
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import ( # noqa: PLC0415 # proxy import cycle
MCPRequestHandler,
)
from litellm.proxy._experimental.mcp_server.mcp_server_manager import ( # noqa: PLC0415 # proxy import cycle
global_mcp_server_manager,
)
names: Final = MCPRequestHandler.extract_target_server_names_from_path(canonical[len(base) :])
if len(names) != 1:
return None
server: Final = global_mcp_server_manager.get_mcp_server_by_name(names[0])
if server is None or not server.is_gateway_managed_oauth2:
return None
return server
def aggregate_authorize(
request: Request,
client_id: str,
@ -329,11 +378,16 @@ def aggregate_authorize(
code_challenge_method: str | None,
response_type: str | None,
session_user_id: str | None,
resource: str | None = None,
) -> Response:
"""The aggregate authorize verb: validate the client, require S256 PKCE, interpose
LiteLLM sign-in, and hand the browser to the connect page with the flow sealed into a
per-flow cookie.
A per-server RFC 8707 ``resource`` naming a gateway-managed oauth2 server scopes the
flow to that one server: the scope is sealed into the flow, carried into the code, and
bound into the session token, while the connect page interlude runs exactly as before.
Validation failures respond directly with 400 and never redirect: per RFC 6749
section 4.1.2.1 an unvalidated redirect URI must not receive an error redirect, and
once the client is at fault there is no trusted place to send the browser.
@ -358,6 +412,7 @@ def aggregate_authorize(
login_url: Final = f"{base_url}/sso/key/generate?{urlencode({'return_to': relative_request_url(request)})}"
return RedirectResponse(login_url, status_code=303)
now: Final = datetime.now(timezone.utc)
scoped_server: Final = resolve_scoped_resource_server(request, resource)
handle: Final = secrets.token_urlsafe(24)
flow: Final = _ConnectFlow(
user_id=session_user_id,
@ -367,6 +422,7 @@ def aggregate_authorize(
code_challenge=code_challenge,
jti=secrets.token_urlsafe(24),
exp=int(now.timestamp()) + CONNECT_FLOW_TTL_SECONDS,
resource_server_id=scoped_server.server_id if scoped_server is not None else None,
)
connect_url: Final = _append_query_params(
f"{base_url}/ui/connect",
@ -455,6 +511,7 @@ async def complete_connect_flow(
jti=secrets.token_urlsafe(24),
iat=int(now.timestamp()),
exp=int(now.timestamp()) + code_ttl,
resource_server_id=flow.resource_server_id,
),
)
params: Final = {"code": code, **({"state": flow.state} if flow.state else {})}
@ -587,6 +644,20 @@ def _reload_failure_response(failure: ReloadUserFailure) -> Response:
assert_never(failure)
def _resource_conflicts_with_scope(
request: Request, resource: str | None, sealed_resource_server_id: str | None
) -> bool:
"""True when a scoped grant is being redeemed for a DIFFERENT resource than the one
sealed into it (RFC 8707 section 2.2: reject with ``invalid_target``). An absent
``resource`` never conflicts (the sealed scope still binds the minted session), and an
unscoped grant ignores the parameter entirely, exactly as the endpoint always has, so
no pre-existing client breaks."""
if sealed_resource_server_id is None or resource is None:
return False
resolved: Final = resolve_scoped_resource_server(request, resource)
return resolved is None or resolved.server_id != sealed_resource_server_id
async def aggregate_token(
request: Request,
grant_type: str,
@ -598,6 +669,7 @@ async def aggregate_token(
master_key: str | None,
reload_user: ReloadUser,
cache: DualCache,
resource: str | None = None,
) -> Response:
"""The aggregate token verb: authorization_code and refresh_token grants for the
identity-only session pair. Every path re-validates the litellm user live before
@ -609,10 +681,12 @@ async def aggregate_token(
now: Final = datetime.now(timezone.utc)
if grant_type == "authorization_code":
return await _authorization_code_grant(
request=request,
code=code,
redirect_uri=redirect_uri,
client_id=client_id,
code_verifier=code_verifier,
resource=resource,
keys=keys,
now=now,
reload_user=reload_user,
@ -620,8 +694,10 @@ async def aggregate_token(
)
if grant_type == "refresh_token":
return await _refresh_token_grant(
request=request,
refresh_token=refresh_token,
client_id=client_id,
resource=resource,
keys=keys,
now=now,
reload_user=reload_user,
@ -631,10 +707,12 @@ async def aggregate_token(
async def _authorization_code_grant(
request: Request,
code: str | None,
redirect_uri: str | None,
client_id: str,
code_verifier: str | None,
resource: str | None,
keys: SessionKeys,
now: datetime,
reload_user: ReloadUser,
@ -651,6 +729,8 @@ async def _authorization_code_grant(
return _oauth_error(400, "invalid_grant", "the authorization code has expired")
if client_id != parsed.client_id or redirect_uri != parsed.redirect_uri:
return _oauth_error(400, "invalid_grant", "the authorization code was issued to a different client")
if _resource_conflicts_with_scope(request, resource, parsed.resource_server_id):
return _oauth_error(400, "invalid_target", "resource does not match the scope this code was issued for")
if not _pkce_verifier_matches(code_verifier, parsed.code_challenge):
return _oauth_error(400, "invalid_grant", "PKCE verification failed")
# Revalidate the user BEFORE claiming the code, so a transient DB outage (a retryable
@ -666,12 +746,18 @@ async def _authorization_code_grant(
parsed.exp - int(now.timestamp()) + _CLAIM_TTL_BUFFER_SECONDS,
):
return _oauth_error(400, "invalid_grant", "the authorization code was already used")
return _session_token_pair(SessionPrincipal(user_id=parsed.user_id, client_id=client_id), keys, now)
return _session_token_pair(
SessionPrincipal(user_id=parsed.user_id, client_id=client_id, resource_server_id=parsed.resource_server_id),
keys,
now,
)
async def _refresh_token_grant(
request: Request,
refresh_token: str | None,
client_id: str,
resource: str | None,
keys: SessionKeys,
now: datetime,
reload_user: ReloadUser,
@ -682,6 +768,8 @@ async def _refresh_token_grant(
opened: Final = open_session_refresh_bearer(refresh_token, keys, now, expected_client_id=client_id)
if not isinstance(opened, SessionRefreshOpened):
return _oauth_error(400, "invalid_grant", "the refresh token is invalid for this client")
if _resource_conflicts_with_scope(request, resource, opened.principal.resource_server_id):
return _oauth_error(400, "invalid_target", "resource does not match the scope this token was issued for")
failure: Final = await reload_user(opened.principal.user_id)
if failure is not None:
return _reload_failure_response(failure)

111
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -13,7 +13,7 @@ import json
import os
import re
import time
from collections.abc import AsyncIterator, Callable, Sequence
from collections.abc import AsyncIterator, Callable, Mapping, Sequence
from contextlib import asynccontextmanager
from typing import TYPE_CHECKING, Any, Final, Literal, TypeAlias, TypedDict, cast
from urllib.parse import ParseResult, urlparse
@ -46,6 +46,9 @@ from litellm.constants import (
)
from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.experimental_mcp_client.client import MCPClient, MCPSigV4Auth
from litellm.integrations.custom_guardrail import (
_sync_guardrail_info_to_logging_obj, # pyright: ignore[reportPrivateUsage] - the same bridge @log_guardrail_information uses; reimplementing it here would fork the metadata-key logic
)
from litellm.litellm_core_utils.url_utils import SSRFError, async_safe_get
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
@ -162,6 +165,7 @@ if TYPE_CHECKING:
from mcp.types import CreateMessageRequestParams
from litellm.caching.caching import InMemoryCache
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.mcp_server.mcp_toolset import MCPToolset
try:
@ -1233,6 +1237,35 @@ def _create_elicitation_callback():
return _elicitation_callback
def _record_mcp_guardrail_evaluations(
synthetic_llm_data: dict[str, Any], # mutable-ok: `_sync_guardrail_info_to_logging_obj` takes a concrete dict
litellm_logging_obj: "LiteLLMLoggingObj | None",
) -> None:
"""Bridge guardrail decision records off an MCP synthetic request onto the request's logger.
MCP guardrails run against a throwaway LLM-shaped dict from
``ProxyLogging._convert_mcp_to_llm_format``, so ``@log_guardrail_information``
files ``standard_logging_guardrail_information`` in that dict's metadata bucket,
which ``get_standard_logging_object_payload`` never reads. Native (non-unified)
guardrails receive no ``logging_obj`` kwarg, so the decorator cannot bridge on
their behalf; this calls the same helper it would have.
Only the decision records move. The synthetic request's messages and tool
arguments stay behind: they can carry end-user data, and the monitor needs none
of it.
"""
if litellm_logging_obj is None:
return
try:
_sync_guardrail_info_to_logging_obj(synthetic_llm_data, litellm_logging_obj)
except Exception as e: # noqa: BLE001 # callers run this from a `finally` on the block path
# The breadth is the point. Narrowing to the knowable AttributeError/TypeError
# would let an unexpected type escape that ``finally`` and replace the guardrail's
# block with a bookkeeping error.
verbose_logger.warning("Failed to record MCP guardrail evaluation for logging: %s", e)
class MCPServerManager:
_STDIO_ENV_TEMPLATE_PATTERN = re.compile(r"^\$\{(X-[^}]+)\}$")
@ -2491,6 +2524,18 @@ class MCPServerManager:
open_ids.update(submitted_server_ids)
return open_ids
@staticmethod
def _admitted_session_resource_scope(user_api_key_auth: UserAPIKeyAuth | None) -> str | None:
"""The single server an admitted session subject's bearer was scoped to at authorize
time (RFC 8707 resource), or None for every other principal shape and for unscoped
sessions. Read at every return path of :meth:`get_allowed_mcp_servers`, including
the exception fallback, and applied AFTER every union (grants, operator-open,
submitted) because the scope is a ceiling over the whole reachable set; a resolver
fault therefore never widens a scoped bearer to the allow-all set."""
if user_api_key_auth is None or not _is_mcp_admitted_user_subject(user_api_key_auth):
return None
return user_api_key_auth.mcp_session_resource_server_id
async def get_allowed_mcp_servers(self, user_api_key_auth: UserAPIKeyAuth | None = None) -> list[str]:
"""
Get the allowed MCP Servers for the user.
@ -2600,13 +2645,19 @@ class MCPServerManager:
if len(combined_servers) == 0:
verbose_logger.debug("No allowed MCP Servers found for user api key auth.")
return list(combined_servers)
scope = MCPServerManager._admitted_session_resource_scope(user_api_key_auth)
return [server_id for server_id in combined_servers if scope is None or server_id == scope]
except Exception: # noqa: BLE001
verbose_logger.exception(
"Failed to get allowed MCP servers; team-level object_permission "
"grants may be dropped. Falling back to global and submitted servers."
)
return list(dict.fromkeys(allow_all_server_ids + submitted_server_ids))
scope = MCPServerManager._admitted_session_resource_scope(user_api_key_auth)
return [
server_id
for server_id in dict.fromkeys(allow_all_server_ids + submitted_server_ids)
if scope is None or server_id == scope
]
async def resolve_toolset_tool_permissions(
self,
@ -4555,6 +4606,7 @@ class MCPServerManager:
proxy_logging_obj: ProxyLogging | None,
server: MCPServer,
raw_headers: dict[str, str] | None = None,
litellm_logging_obj: "LiteLLMLoggingObj | None" = None,
) -> dict[str, Any]:
"""
Run pre-call checks and guardrail hooks for an MCP tool call.
@ -4564,6 +4616,10 @@ class MCPServerManager:
present. An absent logger must never be able to turn an authorization
decision into a no-op.
``litellm_logging_obj`` is the request's logger, and it is what lands a
``pre_mcp_call`` evaluation (or a block) on the spend-log row the Guardrails
Monitor counts. It stays optional so callers that do no logging are unchanged.
Returns a dict that may contain:
- "arguments": hook-modified tool arguments (only if changed)
- "extra_headers": headers injected by pre_mcp_call guardrail hooks
@ -4622,8 +4678,13 @@ class MCPServerManager:
# Create MCP request object for processing
mcp_request_obj: Final = proxy_logging_obj._create_mcp_request_object_from_kwargs(pre_hook_kwargs)
# Convert to LLM format for existing guardrail compatibility
# Convert to LLM format for existing guardrail compatibility.
# Unified guardrails read the seeded logger off the request dict and pass it
# into ``apply_guardrail``, so ``@log_guardrail_information`` bridges their
# evaluations itself; the ``finally`` below covers native guardrails, which
# never receive it. Same seeding the pass-through routes do.
synthetic_llm_data: Final = proxy_logging_obj._convert_mcp_to_llm_format(mcp_request_obj, pre_hook_kwargs)
synthetic_llm_data["litellm_logging_obj"] = litellm_logging_obj
try:
# Use standard pre_call_hook
@ -4648,6 +4709,12 @@ class MCPServerManager:
# Re-raise guardrail exceptions to properly fail the MCP call
verbose_logger.error("Guardrail blocked MCP tool call pre call: %s", e)
raise e
finally:
# ``finally`` rather than after the ``try``: a block raises straight out of
# here, and the failure spend-log row that "Total Blocked" counts is built
# from this logger further up the stack, so the record has to be attached
# before the exception leaves this frame.
_record_mcp_guardrail_evaluations(synthetic_llm_data, litellm_logging_obj)
return hook_result
@ -4659,8 +4726,14 @@ class MCPServerManager:
user_api_key_auth: UserAPIKeyAuth | None,
proxy_logging_obj: ProxyLogging,
start_time: datetime.datetime,
litellm_logging_obj: "LiteLLMLoggingObj | None" = None,
):
"""Create and return a during hook task for MCP tool calls."""
"""Create and return a during hook task for MCP tool calls.
``litellm_logging_obj`` is the request's logger; see ``pre_call_tool_check``.
The task is awaited before the tool call's success logging runs, so a
``during_mcp_call`` evaluation recorded on it is serialized with that call.
"""
from litellm.types.llms.base import HiddenParams
from litellm.types.mcp import MCPDuringCallRequestObject
@ -4679,15 +4752,23 @@ class MCPServerManager:
"user_api_key_auth": user_api_key_auth,
}
# Seeded for the same reason as in ``pre_call_tool_check``.
synthetic_llm_data: Final = proxy_logging_obj._convert_mcp_to_llm_format(request_obj, during_hook_kwargs)
synthetic_llm_data["litellm_logging_obj"] = litellm_logging_obj
return asyncio.create_task(
proxy_logging_obj.during_call_hook(
user_api_key_dict=user_api_key_auth,
data=synthetic_llm_data,
call_type=CallTypes.call_mcp_tool.value,
)
)
# Wrapped so the bridge runs inside the task: the caller only holds the task and
# gathers it later, so there is no other point that still sees a block here.
async def _run_during_call_hook() -> Mapping[str, Any] | None:
try:
return await proxy_logging_obj.during_call_hook(
user_api_key_dict=user_api_key_auth,
data=synthetic_llm_data,
call_type=CallTypes.call_mcp_tool.value,
)
finally:
_record_mcp_guardrail_evaluations(synthetic_llm_data, litellm_logging_obj)
return asyncio.create_task(_run_during_call_hook())
def _get_call_semaphore(self, mcp_server: MCPServer) -> asyncio.Semaphore | None:
limit: Final = mcp_server.max_concurrent_requests
@ -5216,6 +5297,7 @@ class MCPServerManager:
oauth2_headers: dict[str, str] | None = None,
raw_headers: dict[str, str] | None = None,
host_progress_callback: Callable | None = None,
litellm_logging_obj: "LiteLLMLoggingObj | None" = None,
) -> CallToolResult:
"""
Call a tool with the given name and arguments
@ -5228,6 +5310,9 @@ class MCPServerManager:
mcp_auth_header: MCP auth header (deprecated)
mcp_server_auth_headers: Optional dict of server-specific auth headers {server_alias: auth_value}
proxy_logging_obj: Optional ProxyLogging object for hook integration
litellm_logging_obj: Optional request logger the guardrail hooks record
their evaluations onto, so MCP guardrail activity reaches the
Guardrails Monitor. See ``pre_call_tool_check``
Returns:
@ -5258,6 +5343,7 @@ class MCPServerManager:
proxy_logging_obj=proxy_logging_obj,
server=mcp_server,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
if "arguments" in hook_result:
arguments = hook_result["arguments"]
@ -5272,6 +5358,7 @@ class MCPServerManager:
user_api_key_auth=user_api_key_auth,
proxy_logging_obj=proxy_logging_obj,
start_time=start_time,
litellm_logging_obj=litellm_logging_obj,
)
tasks.append(during_hook_task)

4
litellm/proxy/_experimental/mcp_server/oauth_utils.py
View file

@ -633,7 +633,7 @@ def canonicalize_url_identity(url: str) -> str:
return urlunparse((scheme, netloc, parsed.path.rstrip("/"), "", "", ""))
def _canonical_resource_uri(url: str) -> str | None:
def canonical_resource_uri(url: str) -> str | None:
"""Canonicalize an upstream MCP server URL into an RFC 8707 resource identifier.
Keeps only the scheme, host, port and path, which is the shape the MCP authorization spec's
@ -693,7 +693,7 @@ def resolve_upstream_resource(mcp_server: "MCPServer") -> str | None:
mcp_server.server_id,
)
return None
canonical: Final = _canonical_resource_uri(mcp_server.url)
canonical: Final = canonical_resource_uri(mcp_server.url)
if canonical is None:
verbose_logger.warning(
"MCP server %s sets upstream_resource=auto but its url is not an absolute URI, so no "

16
litellm/proxy/_experimental/mcp_server/outbound_credentials/session_token.py
View file

@ -85,11 +85,18 @@ class SessionPrincipal(BaseModel):
enforced at use time rather than frozen at mint time. ``client_id`` is the (stateless,
gateway-sealed) DCR client identifier the token was issued to; the token endpoint
requires it to match on the refresh grant.
``resource_server_id`` is the single MCP server this session was authorized for when
the client requested a per-server RFC 8707 resource at authorize time, or ``None`` for
the aggregate scope. It is a RESTRICTION carried for admission to intersect against
the live grant resolution, never a grant by itself; the refresh grant re-mints from
this principal so the restriction survives rotation.
"""
model_config = ConfigDict(frozen=True)
user_id: str = Field(min_length=1)
client_id: str = Field(min_length=1)
resource_server_id: str | None = None
class SessionKeys(BaseModel):
@ -186,6 +193,7 @@ class _SessionClaims(BaseModel):
kind: SessionTokenKind
user_id: str = Field(min_length=1)
client_id: str = Field(min_length=1)
resource_server_id: str | None = None
def is_session_token(candidate: str) -> bool:
@ -286,9 +294,10 @@ def _mint(
kind=kind,
user_id=principal.user_id,
client_id=principal.client_id,
resource_server_id=principal.resource_server_id,
)
token: Final = prefix + jwt.encode(
claims.model_dump(), keys.signing_key.get_secret_value(), algorithm=_SESSION_JWT_ALGORITHM
claims.model_dump(exclude_none=True), keys.signing_key.get_secret_value(), algorithm=_SESSION_JWT_ALGORITHM
)
size_bytes: Final = len(token.encode("utf-8"))
if size_bytes > MAX_SESSION_TOKEN_BYTES:
@ -323,7 +332,10 @@ def _open(
if now.timestamp() >= claims.exp:
return SessionExpired()
return OpenedSessionToken(
principal=SessionPrincipal(user_id=claims.user_id, client_id=claims.client_id), jti=claims.jti
principal=SessionPrincipal(
user_id=claims.user_id, client_id=claims.client_id, resource_server_id=claims.resource_server_id
),
jti=claims.jti,
)

17
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -2824,6 +2824,7 @@ if MCP_AVAILABLE:
proxy_logging_obj=proxy_logging_obj,
server=mcp_server,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
# `pre_call_tool_check` may return guardrail-modified
# arguments; honor them on the local path too.
@ -2962,6 +2963,7 @@ if MCP_AVAILABLE:
proxy_logging_obj=proxy_logging_obj,
server=prefix_server,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)
if "arguments" in hook_result:
arguments = hook_result["arguments"] # pyright: ignore[reportAny] # hook returns untyped args
@ -3149,6 +3151,20 @@ if MCP_AVAILABLE:
traceback_str: Final = traceback.format_exc(limit=MAXIMUM_TRACEBACK_LINES_TO_LOG)
from litellm.proxy.proxy_server import proxy_logging_obj
# Ordering is load-bearing. ``_ProxyDBLogger.async_post_call_failure_hook``,
# reached below, writes the failure spend-log row from this logger's
# ``standard_logging_object``, which only exists once the failure handlers
# have run. Flush them first or the row lands with
# ``guardrail_information=None`` and a guardrail block is never counted.
#
# Not double-logged: both handlers gate on ``should_run_logging`` and then
# mark it, so the ``@client`` wrapper's own post-raise logging no-ops on this
# logger, same as ``_fire_mcp_tool_call_logging`` does for ``isError=True``.
if litellm_logging_obj is not None:
end_time: Final = datetime.now() # noqa: DTZ005 # naive to match `start_time`, which it is subtracted from
litellm_logging_obj.failure_handler(e, traceback_str, start_time, end_time)
await litellm_logging_obj.async_failure_handler(e, traceback_str, start_time, end_time)
if proxy_logging_obj and user_api_key_auth:
await proxy_logging_obj.post_call_failure_hook(
request_data=kwargs,
@ -3326,6 +3342,7 @@ if MCP_AVAILABLE:
raw_headers=raw_headers,
proxy_logging_obj=proxy_logging_obj,
host_progress_callback=host_progress_callback,
litellm_logging_obj=litellm_logging_obj,
)
verbose_logger.debug("CALL TOOL RESULT: %s", call_tool_result)
return call_tool_result

2
litellm/proxy/_experimental/out/404.html
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/404/index.html
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.__PAGE__.txt
View file

@ -1,9 +1,9 @@
1:"$Sreact.fragment"
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"ClientPageRoot"]
3:I[871135,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js","/litellm-asset-prefix/_next/static/chunks/0catil7su1yp5.js","/litellm-asset-prefix/_next/static/chunks/2cngn5bal3278.js","/litellm-asset-prefix/_next/static/chunks/34canl9e2fj8w.js","/litellm-asset-prefix/_next/static/chunks/0vvnul8uez9kj.js","/litellm-asset-prefix/_next/static/chunks/3vu26x-a1_slz.js","/litellm-asset-prefix/_next/static/chunks/1vquuz09jxl5_.js","/litellm-asset-prefix/_next/static/chunks/3_3dj4vdy-3xy.js","/litellm-asset-prefix/_next/static/chunks/0ex44ljfg9dp9.js","/litellm-asset-prefix/_next/static/chunks/0xcneptpo0s76.js","/litellm-asset-prefix/_next/static/chunks/08uoywqkfbbbt.js","/litellm-asset-prefix/_next/static/chunks/0g_w4tf2inv3i.js","/litellm-asset-prefix/_next/static/chunks/1iakmimqrlpn0.js","/litellm-asset-prefix/_next/static/chunks/17-6zku8f68gf.js","/litellm-asset-prefix/_next/static/chunks/0f5fel02jwglw.js","/litellm-asset-prefix/_next/static/chunks/0fk0i3e2aixp7.js","/litellm-asset-prefix/_next/static/chunks/3ytz29phknzsy.js","/litellm-asset-prefix/_next/static/chunks/0dbvgsc7ha049.js","/litellm-asset-prefix/_next/static/chunks/0aoel7yrv88fp.js","/litellm-asset-prefix/_next/static/chunks/2v54hze4wuham.js","/litellm-asset-prefix/_next/static/chunks/1axupaiywv5s2.js","/litellm-asset-prefix/_next/static/chunks/0zy8o1br4cxj_.js","/litellm-asset-prefix/_next/static/chunks/1uz3jt-tj9lkf.js","/litellm-asset-prefix/_next/static/chunks/2bwy4wke9jrlh.js","/litellm-asset-prefix/_next/static/chunks/2cz4e0-p1l3hf.js","/litellm-asset-prefix/_next/static/chunks/2tkpj7d49kuht.js","/litellm-asset-prefix/_next/static/chunks/323l6h8s7ahat.js","/litellm-asset-prefix/_next/static/chunks/1cea03gg5a_c7.js","/litellm-asset-prefix/_next/static/chunks/2hu1vyy-5pv13.js","/litellm-asset-prefix/_next/static/chunks/199uwr871eene.js","/litellm-asset-prefix/_next/static/chunks/105643dvf00hu.js","/litellm-asset-prefix/_next/static/chunks/29l3pao1xfkc3.js","/litellm-asset-prefix/_next/static/chunks/111jj26rg98nb.js","/litellm-asset-prefix/_next/static/chunks/12wsfsljxg4xv.js","/litellm-asset-prefix/_next/static/chunks/112n0hv3cc2rg.js","/litellm-asset-prefix/_next/static/chunks/31azy9hywrzm7.js","/litellm-asset-prefix/_next/static/chunks/3pu9plov1btip.js","/litellm-asset-prefix/_next/static/chunks/3mz07lvvrbciz.js","/litellm-asset-prefix/_next/static/chunks/1lndy6n7cwvrq.js","/litellm-asset-prefix/_next/static/chunks/09l_m9l1emin2.js","/litellm-asset-prefix/_next/static/chunks/2c90xukbd3il6.js","/litellm-asset-prefix/_next/static/chunks/0ikhgrs0xvkyu.js","/litellm-asset-prefix/_next/static/chunks/23-g73xaw3kap.js"],"default"]
6:I[897367,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"OutletBoundary"]
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"ClientPageRoot"]
3:I[871135,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js","/litellm-asset-prefix/_next/static/chunks/21b4hw_igldhz.js","/litellm-asset-prefix/_next/static/chunks/2_hxghav3pe9j.js","/litellm-asset-prefix/_next/static/chunks/0g_w4tf2inv3i.js","/litellm-asset-prefix/_next/static/chunks/2i0218zvrsasm.js","/litellm-asset-prefix/_next/static/chunks/1gwzs-8xkvx8f.js","/litellm-asset-prefix/_next/static/chunks/1oob52g5gib5j.js","/litellm-asset-prefix/_next/static/chunks/3ytz29phknzsy.js","/litellm-asset-prefix/_next/static/chunks/1yok3x3_3gr1p.js","/litellm-asset-prefix/_next/static/chunks/0aoel7yrv88fp.js","/litellm-asset-prefix/_next/static/chunks/2mxcro_n8i1ef.js","/litellm-asset-prefix/_next/static/chunks/3xciut9pzmr7-.js","/litellm-asset-prefix/_next/static/chunks/1jkcw8ug0uobj.js","/litellm-asset-prefix/_next/static/chunks/3k3r6waxmnsvu.js","/litellm-asset-prefix/_next/static/chunks/3l0glczkblv8_.js","/litellm-asset-prefix/_next/static/chunks/1cea03gg5a_c7.js","/litellm-asset-prefix/_next/static/chunks/2ca0bgyj3-r_j.js","/litellm-asset-prefix/_next/static/chunks/29nmr1sywlx25.js","/litellm-asset-prefix/_next/static/chunks/0gh1eppc9ekzh.js","/litellm-asset-prefix/_next/static/chunks/3hk5c4q5k-j7x.js","/litellm-asset-prefix/_next/static/chunks/2mhbxmykyh83f.js","/litellm-asset-prefix/_next/static/chunks/1xk5l9lxa0dv-.js","/litellm-asset-prefix/_next/static/chunks/26e7zpdybuhtq.js","/litellm-asset-prefix/_next/static/chunks/2c90xukbd3il6.js","/litellm-asset-prefix/_next/static/chunks/0m-cn894wctv5.js","/litellm-asset-prefix/_next/static/chunks/3cw_k7_vr9pcu.js","/litellm-asset-prefix/_next/static/chunks/12wsfsljxg4xv.js","/litellm-asset-prefix/_next/static/chunks/2udc_95331vyv.js","/litellm-asset-prefix/_next/static/chunks/3mkd81u36rwju.js","/litellm-asset-prefix/_next/static/chunks/0kh9ov64og3-k.js","/litellm-asset-prefix/_next/static/chunks/3580ki1m5g-sx.js","/litellm-asset-prefix/_next/static/chunks/2xuwoxcnxuv39.js","/litellm-asset-prefix/_next/static/chunks/3bwziv83xzehe.js"],"default"]
6:I[897367,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"OutletBoundary"]
7:"$Sreact.suspense"
0:{"rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/2v54hze4wuham.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/1axupaiywv5s2.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0zy8o1br4cxj_.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/1uz3jt-tj9lkf.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/2bwy4wke9jrlh.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/2cz4e0-p1l3hf.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/2tkpj7d49kuht.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/323l6h8s7ahat.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/1cea03gg5a_c7.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/2hu1vyy-5pv13.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/199uwr871eene.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/105643dvf00hu.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/29l3pao1xfkc3.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/111jj26rg98nb.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/12wsfsljxg4xv.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/112n0hv3cc2rg.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/31azy9hywrzm7.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/3pu9plov1btip.js","async":true}],["$","script","script-18",{"src":"/litellm-asset-prefix/_next/static/chunks/3mz07lvvrbciz.js","async":true}],["$","script","script-19",{"src":"/litellm-asset-prefix/_next/static/chunks/1lndy6n7cwvrq.js","async":true}],["$","script","script-20",{"src":"/litellm-asset-prefix/_next/static/chunks/09l_m9l1emin2.js","async":true}],["$","script","script-21",{"src":"/litellm-asset-prefix/_next/static/chunks/2c90xukbd3il6.js","async":true}],["$","script","script-22",{"src":"/litellm-asset-prefix/_next/static/chunks/0ikhgrs0xvkyu.js","async":true}],["$","script","script-23",{"src":"/litellm-asset-prefix/_next/static/chunks/23-g73xaw3kap.js","async":true}]],["$","$L6",null,{"children":["$","$7",null,{"name":"Next.MetadataOutlet","children":"$@8"}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"HynDchE8aLeEewsZVNDO8"}
0:{"rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/3l0glczkblv8_.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/1cea03gg5a_c7.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/2ca0bgyj3-r_j.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/29nmr1sywlx25.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0gh1eppc9ekzh.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/3hk5c4q5k-j7x.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/2mhbxmykyh83f.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/1xk5l9lxa0dv-.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/26e7zpdybuhtq.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/2c90xukbd3il6.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/0m-cn894wctv5.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/3cw_k7_vr9pcu.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/12wsfsljxg4xv.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/2udc_95331vyv.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/3mkd81u36rwju.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/0kh9ov64og3-k.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/3580ki1m5g-sx.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/2xuwoxcnxuv39.js","async":true}],["$","script","script-18",{"src":"/litellm-asset-prefix/_next/static/chunks/3bwziv83xzehe.js","async":true}]],["$","$L6",null,{"children":["$","$7",null,{"name":"Next.MetadataOutlet","children":"$@8"}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"TeJ852IBdcKgsOMzGKY73"}
4:{}
5:"$0:rsc:props:children:0:props:serverProvidedParams:params"
8:null

10
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.txt
View file

@ -1,7 +1,7 @@
1:"$Sreact.fragment"
2:I[92825,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"ClientSegmentRoot"]
3:I[216370,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js","/litellm-asset-prefix/_next/static/chunks/0catil7su1yp5.js","/litellm-asset-prefix/_next/static/chunks/2cngn5bal3278.js","/litellm-asset-prefix/_next/static/chunks/34canl9e2fj8w.js","/litellm-asset-prefix/_next/static/chunks/0vvnul8uez9kj.js","/litellm-asset-prefix/_next/static/chunks/3vu26x-a1_slz.js","/litellm-asset-prefix/_next/static/chunks/1vquuz09jxl5_.js","/litellm-asset-prefix/_next/static/chunks/3_3dj4vdy-3xy.js","/litellm-asset-prefix/_next/static/chunks/0ex44ljfg9dp9.js","/litellm-asset-prefix/_next/static/chunks/0xcneptpo0s76.js","/litellm-asset-prefix/_next/static/chunks/08uoywqkfbbbt.js","/litellm-asset-prefix/_next/static/chunks/0g_w4tf2inv3i.js","/litellm-asset-prefix/_next/static/chunks/1iakmimqrlpn0.js","/litellm-asset-prefix/_next/static/chunks/17-6zku8f68gf.js","/litellm-asset-prefix/_next/static/chunks/0f5fel02jwglw.js","/litellm-asset-prefix/_next/static/chunks/0fk0i3e2aixp7.js","/litellm-asset-prefix/_next/static/chunks/3ytz29phknzsy.js","/litellm-asset-prefix/_next/static/chunks/0dbvgsc7ha049.js","/litellm-asset-prefix/_next/static/chunks/0aoel7yrv88fp.js"],"default"]
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"default"]
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"default"]
0:{"rsc":["$","$1","c",{"children":[[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0catil7su1yp5.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/2cngn5bal3278.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/34canl9e2fj8w.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/0vvnul8uez9kj.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/3vu26x-a1_slz.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/1vquuz09jxl5_.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/3_3dj4vdy-3xy.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/0ex44ljfg9dp9.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/0xcneptpo0s76.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/08uoywqkfbbbt.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/0g_w4tf2inv3i.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/1iakmimqrlpn0.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/17-6zku8f68gf.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/0f5fel02jwglw.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/0fk0i3e2aixp7.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/3ytz29phknzsy.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/0dbvgsc7ha049.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/0aoel7yrv88fp.js","async":true}]],["$","$L2",null,{"Component":"$3","slots":{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]},"serverProvidedParams":{"params":{},"promises":["$@6"]}}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"HynDchE8aLeEewsZVNDO8"}
2:I[92825,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"ClientSegmentRoot"]
3:I[216370,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js","/litellm-asset-prefix/_next/static/chunks/21b4hw_igldhz.js","/litellm-asset-prefix/_next/static/chunks/2_hxghav3pe9j.js","/litellm-asset-prefix/_next/static/chunks/0g_w4tf2inv3i.js","/litellm-asset-prefix/_next/static/chunks/2i0218zvrsasm.js","/litellm-asset-prefix/_next/static/chunks/1gwzs-8xkvx8f.js","/litellm-asset-prefix/_next/static/chunks/1oob52g5gib5j.js","/litellm-asset-prefix/_next/static/chunks/3ytz29phknzsy.js","/litellm-asset-prefix/_next/static/chunks/1yok3x3_3gr1p.js","/litellm-asset-prefix/_next/static/chunks/0aoel7yrv88fp.js","/litellm-asset-prefix/_next/static/chunks/2mxcro_n8i1ef.js","/litellm-asset-prefix/_next/static/chunks/3xciut9pzmr7-.js","/litellm-asset-prefix/_next/static/chunks/1jkcw8ug0uobj.js","/litellm-asset-prefix/_next/static/chunks/3k3r6waxmnsvu.js"],"default"]
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"default"]
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"default"]
0:{"rsc":["$","$1","c",{"children":[[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/21b4hw_igldhz.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/2_hxghav3pe9j.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0g_w4tf2inv3i.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/2i0218zvrsasm.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/1gwzs-8xkvx8f.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/1oob52g5gib5j.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/3ytz29phknzsy.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/1yok3x3_3gr1p.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/0aoel7yrv88fp.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/2mxcro_n8i1ef.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/3xciut9pzmr7-.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/1jkcw8ug0uobj.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/3k3r6waxmnsvu.js","async":true}]],["$","$L2",null,{"Component":"$3","slots":{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]},"serverProvidedParams":{"params":{},"promises":["$@6"]}}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"TeJ852IBdcKgsOMzGKY73"}
6:"$0:rsc:props:children:1:props:serverProvidedParams:params"

61
litellm/proxy/_experimental/out/__next._full.txt
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/__next._head.txt
View file

@ -1,6 +1,6 @@
1:"$Sreact.fragment"
2:I[897367,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"ViewportBoundary"]
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"MetadataBoundary"]
2:I[897367,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"ViewportBoundary"]
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"MetadataBoundary"]
4:"$Sreact.suspense"
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"IconMark"]
0:{"rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.3arlap5n8tyzg.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"/get_favicon"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"HynDchE8aLeEewsZVNDO8"}
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"IconMark"]
0:{"rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.3arlap5n8tyzg.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"/get_favicon"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"TeJ852IBdcKgsOMzGKY73"}

16
litellm/proxy/_experimental/out/__next._index.txt
View file

@ -1,10 +1,10 @@
1:"$Sreact.fragment"
2:I[12985,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"NuqsAdapter"]
3:I[867271,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"default"]
4:I[71195,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"default"]
5:I[557951,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"AuthProvider"]
6:I[339756,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"default"]
7:I[837457,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js"],"default"]
2:I[12985,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"NuqsAdapter"]
3:I[867271,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"default"]
4:I[71195,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"default"]
5:I[557951,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"AuthProvider"]
6:I[339756,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"default"]
7:I[837457,["/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js"],"default"]
:HL["/litellm-asset-prefix/_next/static/chunks/1kid9zr1--h6y.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/1u9cxkx771jnb.css","style"]
0:{"rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/1kid9zr1--h6y.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/1u9cxkx771jnb.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/20mvgyvrlrdla.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/1jfookxfajkeo.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/3fzya67r1y1fl.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"children":["$","$L5",null,{"children":["$","$L6",null,{"parallelRouterKey":"children","template":["$","$L7",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"HynDchE8aLeEewsZVNDO8"}
:HL["/litellm-asset-prefix/_next/static/chunks/0cefehsj9nby1.css","style"]
0:{"rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/1kid9zr1--h6y.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/0cefehsj9nby1.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/19frz_r2jewoi.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/3_zdkdwptdu3w.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/1ntn7efqc-iiw.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/1dg0y22lcfxz2.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"children":["$","$L5",null,{"children":["$","$L6",null,{"parallelRouterKey":"children","template":["$","$L7",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"TeJ852IBdcKgsOMzGKY73"}

4
litellm/proxy/_experimental/out/__next._tree.txt
View file

@ -1,4 +1,4 @@
:HL["/litellm-asset-prefix/_next/static/chunks/1kid9zr1--h6y.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/1u9cxkx771jnb.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/0cefehsj9nby1.css","style"]
:HL["/litellm-asset-prefix/_next/static/media/83afe278b6a6bb3c-s.p.2bn3s6zvc0dyp.woff2","font",{"crossOrigin":"","type":"font/woff2"}]
0:{"tree":{"name":"","param":null,"prefetchHints":16,"slots":{"children":{"name":"(dashboard)","param":null,"prefetchHints":0,"slots":{"children":{"name":"__PAGE__","param":null,"prefetchHints":0,"slots":null}}}}},"staleTime":300,"buildId":"HynDchE8aLeEewsZVNDO8"}
0:{"tree":{"name":"","param":null,"prefetchHints":16,"slots":{"children":{"name":"(dashboard)","param":null,"prefetchHints":0,"slots":{"children":{"name":"__PAGE__","param":null,"prefetchHints":0,"slots":null}}}}},"staleTime":300,"buildId":"TeJ852IBdcKgsOMzGKY73"}

0
litellm/proxy/_experimental/out/_next/static/HynDchE8aLeEewsZVNDO8/_buildManifest.js → litellm/proxy/_experimental/out/_next/static/TeJ852IBdcKgsOMzGKY73/_buildManifest.js
View file

0
litellm/proxy/_experimental/out/_next/static/HynDchE8aLeEewsZVNDO8/_clientMiddlewareManifest.js → litellm/proxy/_experimental/out/_next/static/TeJ852IBdcKgsOMzGKY73/_clientMiddlewareManifest.js
View file

0
litellm/proxy/_experimental/out/_next/static/HynDchE8aLeEewsZVNDO8/_ssgManifest.js → litellm/proxy/_experimental/out/_next/static/TeJ852IBdcKgsOMzGKY73/_ssgManifest.js
View file

216
litellm/proxy/_experimental/out/_next/static/chunks/0013wgjn81q8k.js Normal file
View file

File diff suppressed because one or more lines are too long

10
litellm/proxy/_experimental/out/_next/static/chunks/012_6ra8fo7np.js Normal file
View file

File diff suppressed because one or more lines are too long

41
litellm/proxy/_experimental/out/_next/static/chunks/01q1b-t4kl710.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/01txrb6bft5s5.js Normal file
View file

File diff suppressed because one or more lines are too long

7
litellm/proxy/_experimental/out/_next/static/chunks/01vm1oq9am52a.js Normal file
View file

File diff suppressed because one or more lines are too long

10
litellm/proxy/_experimental/out/_next/static/chunks/02-alpsjfp5b7.js
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/_next/static/chunks/02a2ogfa2h8o3.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/02emq9hm7g5fm.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/03c3h-nx-fb3y.js
View file

File diff suppressed because one or more lines are too long

31
litellm/proxy/_experimental/out/_next/static/chunks/048wkdcpsnwne.js
View file

File diff suppressed because one or more lines are too long

13
litellm/proxy/_experimental/out/_next/static/chunks/05287rwl48hh2.js
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/17y1q5sh_9s-g.js → litellm/proxy/_experimental/out/_next/static/chunks/05qv3czmeg-cb.js
View file

File diff suppressed because one or more lines are too long

4
litellm/proxy/_experimental/out/_next/static/chunks/0769vspoaelaf.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/07sexl9lqn8w6.js
View file

File diff suppressed because one or more lines are too long

420
litellm/proxy/_experimental/out/_next/static/chunks/07unod8edrfqd.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/07wi_3yhi4wcx.js Normal file
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/_next/static/chunks/08ldu6o5kiz9c.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/08ps_exix4aud.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/08uoywqkfbbbt.js
View file

File diff suppressed because one or more lines are too long

35
litellm/proxy/_experimental/out/_next/static/chunks/09advjwzkn7qu.js Normal file
View file

@ -0,0 +1,35 @@
(globalThis.TURBOPACK||(globalThis.TURBOPACK=[])).push(["object"==typeof document?document.currentScript:void 0,515288,e=>{"use strict";var r=e.i(843476),t=e.i(271645),a=e.i(115504);let s=t.forwardRef(({className:e,size:t="default",...s},d)=>(0,r.jsx)("div",{ref:d,"data-slot":"card","data-size":t,className:(0,a.cn)("group/card flex flex-col gap-(--card-spacing) rounded-xl bg-card py-(--card-spacing) text-sm text-card-foreground shadow-xs ring-1 ring-foreground/10 [--card-spacing:--spacing(6)] has-[>img:first-child]:pt-0 data-[size=sm]:[--card-spacing:--spacing(4)] *:[img:first-child]:rounded-t-xl *:[img:last-child]:rounded-b-xl",e),...s}));s.displayName="Card";let d=t.forwardRef(({className:e,...t},s)=>(0,r.jsx)("div",{ref:s,"data-slot":"card-header",className:(0,a.cn)("group/card-header @container/card-header grid auto-rows-min items-start gap-1 rounded-t-xl px-(--card-spacing) has-data-[slot=card-action]:grid-cols-[1fr_auto] has-data-[slot=card-description]:grid-rows-[auto_auto] [.border-b]:pb-(--card-spacing)",e),...t}));d.displayName="CardHeader";let i=t.forwardRef(({className:e,...t},s)=>(0,r.jsx)("div",{ref:s,"data-slot":"card-title",className:(0,a.cn)("text-base leading-normal font-medium group-data-[size=sm]/card:text-sm",e),...t}));i.displayName="CardTitle";let o=t.forwardRef(({className:e,...t},s)=>(0,r.jsx)("div",{ref:s,"data-slot":"card-description",className:(0,a.cn)("text-sm text-muted-foreground",e),...t}));o.displayName="CardDescription";let n=t.forwardRef(({className:e,...t},s)=>(0,r.jsx)("div",{ref:s,"data-slot":"card-action",className:(0,a.cn)("col-start-2 row-span-2 row-start-1 self-start justify-self-end",e),...t}));n.displayName="CardAction";let l=t.forwardRef(({className:e,...t},s)=>(0,r.jsx)("div",{ref:s,"data-slot":"card-content",className:(0,a.cn)("px-(--card-spacing)",e),...t}));l.displayName="CardContent";let c=t.forwardRef(({className:e,...t},s)=>(0,r.jsx)("div",{ref:s,"data-slot":"card-footer",className:(0,a.cn)("flex items-center rounded-b-xl px-(--card-spacing) [.border-t]:pt-(--card-spacing)",e),...t}));c.displayName="CardFooter",e.s(["Card",0,s,"CardAction",0,n,"CardContent",0,l,"CardDescription",0,o,"CardFooter",0,c,"CardHeader",0,d,"CardTitle",0,i])},972520,e=>{"use strict";let r=(0,e.i(475254).default)("arrow-right",[["path",{d:"M5 12h14",key:"1ays0h"}],["path",{d:"m12 5 7 7-7 7",key:"xquz4c"}]]);e.s(["ArrowRight",0,r],972520)},411929,e=>{"use strict";var r=e.i(843476),t=e.i(271645),a=e.i(972520),s=e.i(174886),d=e.i(519455),i=e.i(515288),o=e.i(624687),n=e.i(571303),l=e.i(602869),c=e.i(727749);let u=({accessToken:e})=>{let[u,m]=(0,t.useState)(`{
"model": "openai/gpt-4o",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Explain quantum computing in simple terms"
}
],
"temperature": 0.7,
"max_tokens": 500,
"stream": true
}`),[p,f]=(0,t.useState)(""),[x,h]=(0,t.useState)(!1),g=async()=>{h(!0);try{let s;try{s=JSON.parse(u)}catch(e){c.default.fromBackend("Invalid JSON in request body"),h(!1);return}let d={call_type:"completion",request_body:s};if(!e){c.default.fromBackend("No access token found"),h(!1);return}let i=await (0,l.transformRequestCall)(e,d);if(i.raw_request_api_base&&i.raw_request_body){var r,t,a;let e,s,d=(r=i.raw_request_api_base,t=i.raw_request_body,a=i.raw_request_headers||{},e=JSON.stringify(t,null,2).split("\n").map(e=>` ${e}`).join("\n"),s=Object.entries(a).map(([e,r])=>`-H '${e}: ${r}'`).join(" \\\n "),`curl -X POST \\
${r} \\
${s?`${s} \\
`:""}-H 'Content-Type: application/json' \\
-d '{
${e}
}'`);f(d),c.default.success("Request transformed successfully")}else{let e="string"==typeof i?i:JSON.stringify(i);f(e),c.default.info("Transformed request received in unexpected format")}}catch(e){console.error("Error transforming request:",e),c.default.fromBackend("Failed to transform request")}finally{h(!1)}};return(0,r.jsxs)("div",{className:"p-2",children:[(0,r.jsx)("h1",{className:"text-lg font-medium text-foreground",children:"Playground"}),(0,r.jsx)("p",{className:"text-sm text-muted-foreground",children:"See how LiteLLM transforms your request for the specified provider."}),(0,r.jsxs)("div",{className:"mt-4 grid grid-cols-1 gap-4 lg:grid-cols-2",children:[(0,r.jsxs)(i.Card,{children:[(0,r.jsxs)(i.CardHeader,{children:[(0,r.jsx)(i.CardTitle,{className:"text-2xl font-bold",children:"Original Request"}),(0,r.jsx)(i.CardDescription,{children:"The request you would send to LiteLLM /chat/completions endpoint."})]}),(0,r.jsx)(i.CardContent,{children:(0,r.jsx)(o.Textarea,{className:"h-72 resize-none p-4 font-mono text-sm field-sizing-fixed",value:u,onChange:e=>m(e.target.value),onKeyDown:e=>{(e.metaKey||e.ctrlKey)&&"Enter"===e.key&&(e.preventDefault(),g())},placeholder:"Press Cmd/Ctrl + Enter to transform"})}),(0,r.jsx)(i.CardFooter,{className:"justify-end",children:(0,r.jsxs)(d.Button,{onClick:g,disabled:x,children:[(0,r.jsx)("span",{children:"Transform"}),x?(0,r.jsx)(n.UiLoadingSpinner,{className:"size-4"}):(0,r.jsx)(a.ArrowRight,{})]})})]}),(0,r.jsxs)(i.Card,{children:[(0,r.jsxs)(i.CardHeader,{children:[(0,r.jsx)(i.CardTitle,{className:"text-2xl font-bold",children:"Transformed Request"}),(0,r.jsx)(i.CardDescription,{children:"How LiteLLM transforms your request for the specified provider."}),(0,r.jsx)("p",{className:"mt-2 text-xs text-muted-foreground",children:"Note: Sensitive headers are not shown."})]}),(0,r.jsx)(i.CardContent,{children:(0,r.jsxs)("div",{className:"relative rounded-md bg-muted",children:[(0,r.jsx)("pre",{className:"h-72 overflow-auto p-4 font-mono text-sm",children:p||`curl -X POST \\
https://api.openai.com/v1/chat/completions \\
-H 'Authorization: Bearer sk-xxx' \\
-H 'Content-Type: application/json' \\
-d '{
"model": "gpt-4",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
}
],
"temperature": 0.7
}'`}),(0,r.jsx)(d.Button,{variant:"ghost",size:"icon-sm","aria-label":"Copy to clipboard",className:"absolute top-2 right-2",onClick:()=>{navigator.clipboard.writeText(p||""),c.default.success("Copied to clipboard")},children:(0,r.jsx)(s.Copy,{})})]})})]})]}),(0,r.jsx)("div",{className:"mt-4 text-right",children:(0,r.jsxs)("p",{className:"text-sm text-muted-foreground",children:["Found an error? File an issue"," ",(0,r.jsx)("a",{className:"underline underline-offset-4",href:"https://github.com/BerriAI/litellm/issues",target:"_blank",rel:"noopener noreferrer",children:"here"}),"."]})})]})};var m=e.i(135214);e.s(["default",0,function(){let{accessToken:e}=(0,m.default)();return(0,r.jsx)(u,{accessToken:e})}],411929)}]);

4
litellm/proxy/_experimental/out/_next/static/chunks/09l_m9l1emin2.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/09w33nm2cbgkq.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0_vlfpb8phl0v.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0an9ovyhmjka9.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0bc2jtre_083a.js Normal file
View file

File diff suppressed because one or more lines are too long

3
litellm/proxy/_experimental/out/_next/static/chunks/0catil7su1yp5.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0cb9ynx_16337.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0cefehsj9nby1.css Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0d0ty29xv4qhj.js
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/0d398pudg-p7u.js
View file

File diff suppressed because one or more lines are too long

55
litellm/proxy/_experimental/out/_next/static/chunks/0dbvgsc7ha049.js
View file

File diff suppressed because one or more lines are too long

56
litellm/proxy/_experimental/out/_next/static/chunks/0dcwq2i45vhog.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0dhxm4s1uxvr1.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0dvq9v45hkxnf.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0e5-pjnljk_pz.js
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more