scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-08 22:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 78

Merge branch 'litellm_internal_staging' into feature/improve-gigachat-provider

Browse source
This commit is contained in:
KnyazSh 2026-07-02 16:21:53 +00:00
commit 8bf1f21c38
963 changed files with 29331 additions and 2652 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.git-blame-ignore-revs
View file

@ -13,7 +13,7 @@
7edf3a9cb55548b143df1692f4ed7c4681d7fcf7
# style: reformat litellm/ with ruff format (#31317)
430b5b8f1b12dc261a49fda99ac5d1b22381a428
17bfd415aeb5a57fb646b5cc67da1c730aa7c50b
# style: unify ruff format width on 120 (#31518)
3dfbeabe626d203ac9de86024519d9a96c484ce4
48b5a5a0cc5a694a11219416ee0b6eb6e620e74e

4
.github/workflows/codspeed.yml vendored
View file

@ -21,7 +21,7 @@ concurrency:
jobs:
benchmarks:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@ -48,6 +48,8 @@ jobs:
uv run --frozen --no-default-groups
--with pytest==8.3.5
--with pytest-codspeed==4.3.0
--with "mcp>=1.26.0,<2.0"
--with "a2a-sdk>=1.1.0,<2.0"
pytest
-p pytest_codspeed.plugin
tests/benchmarks/

11
.gitignore vendored
View file

@ -50,6 +50,8 @@ litellm/proxy/tests/package-lock.json
ui/litellm-dashboard/.next
ui/litellm-dashboard/node_modules
ui/litellm-dashboard/next-env.d.ts
ui/litellm-dashboard/package.json
ui/litellm-dashboard/package-lock.json
deploy/charts/litellm/*.tgz
deploy/charts/litellm/charts/*
deploy/charts/*.tgz
@ -85,12 +87,17 @@ litellm/proxy/db/migrations/*
litellm/proxy/migrations/*config.yaml
litellm/proxy/migrations/*
litellm/proxy/to_delete_loadtest_work/*
config.yaml
tests/litellm/litellm_core_utils/llm_cost_calc/log.txt
tests/test_custom_dir/*
test.py
litellm_config.yaml
!.github/observatory/litellm_config.yaml
.cursor
litellm/proxy/to_delete_loadtest_work/*
update_model_cost_map.py
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py
scripts/test_vertex_ai_search.py
LAZY_LOADING_IMPROVEMENTS.md
STABILIZATION_TODO.md
@ -123,7 +130,3 @@ crash.*.log
# pytest coverage data
.coverage
# _experimental/out UI build output
# (both componentized and non-componentized build the UI on project release)
litellm/proxy/_experimental/out/

14
CLAUDE.md
View file

@ -1,8 +1,7 @@
Do not write comments unless they are absolutely necessary to explain some very complex business logic. Please clean up if there are comments that are not absolutely necessary. Do not remove comments that are unrelated to the addition of the code of this PR
Explanation: code comments are, in a way, a violation of DRY code. You must update logic in two locations to change the code and "hard to change" is literally the definition of tech debt. We should instead aim to write code that is intuitive to the reader, while being both easy to maintain and high performance
Do not write any comments (existing comments can stay) unless explicitly asked to in a user (not system) prompt
Don't assume that the existing code is correct or the right way of doing things / good coding patterns. In fact, there are a lot of bad coding practices, overly complex code, code smells, etc. If something doesn't look right, speak up. Feel free to break existing patterns or question weird existing code to make new code high quality, as in:
- correct
- secure
- performant
@ -34,15 +33,17 @@ If you ever make public-facing PR descriptions, comments, issues, commit message
Don't hesitate to use values in .env to get needed API keys and other secrets, as long as you never add them to conversation history, commit them, or include them in GitHub issues / PRs
Run tests, format your code, and lint your code before each commit
Python max line length is 120, not 88
When you fix violations gated by `ruff-strict-budget.json` or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered baselines so the ceilings ratchet down instead of leaving stale headroom
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed. It only runs when there are staged frontend and/or backend changes and calculates violations, generates types, etc. based on the worktree, so stage what you need or stash/delete unwanted files in litellm/ or ui/ (where backend and frontend lint run, respectively) before running it. If it fails because dashboard api types are stale, it already regenerated them for you. You just need to stage the schema.d.ts, re-run `make pre-commit` to confirm it passes, and commit
When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-budget.json`, or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered limits so the ceilings ratchet down instead of leaving stale headroom. It measures the working tree, so it must contain exactly the fixes you're committing
If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing `reportAny` / `reportExplicitAny` closer to their basedpyright ceilings, just validate it in the caller with Pydantic (a model or `TypeAdapter` that returns the typed thing or raises will do) and then pass the now typed variable in
If you get an LIT001 or LIT002 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in `tuple()` / `frozenset()` instead of seeding an empty `list`/`dict`/`set` and mutating it over time. Ideally `# mutable-ok` is never used; reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason
Ask to commit and push your work when you're done (or if you're confident that your code is good and works, just do it)
Commit and push your work when you're done without asking
When you must use real LLM models to, for example, write e2e tests, write a QA runbook, etc., make sure to use the latest models (doesn't have to be smartest, can also be a modern small, fast one. No strong preference for smart vs fast here, just use something modern) as of the year and month of the current date. Do a web search as necessary to figure that out
@ -70,6 +71,7 @@ Follow these coding conventions for new/updated code (a three-line fix in a lega
- No monster files or god objects
- No file sprawl: deliberate file and folder structure
- Standard over hand-rolled: use the official SDK or a library where one exists; where none does, follow industry standards instead of inventing local conventions
- API-fragmentation-aware: when logic must branch on which API surface produced or consumes data (e.g. chat completions vs Anthropic Messages vs Responses API shapes), proactively look for an existing shared helper (e.g. `litellm_core_utils/prompt_templates/factory.py`) before writing per-surface parsing in the new module; if none exists, add one there instead of duplicating the same format-detection logic in every new guardrail/integration
Follow conventional commits for commit names and PR titles

87
Makefile
View file

@ -5,10 +5,11 @@
test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
info lint lint-dev format \
lint-basedpyright lint-basedpyright-budget-update \
lint-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
install-dev install-proxy-dev install-test-deps install-hooks \
install-helm-unittest check-circular-imports check-import-safety
install-helm-unittest check-circular-imports check-import-safety pre-commit \
lint-install lint-fetch-base
# Default target
help:
@ -20,17 +21,18 @@ help:
@echo " make install-test-deps - Install the full local test environment"
@echo " make install-helm-unittest - Install helm unittest plugin"
@echo " make install-hooks - Install git hooks (Conventional Commits + Branches)"
@echo " make pre-commit - Run CI-equivalent lint on staged files (run before committing)"
@echo " make format - Apply ruff format code formatting"
@echo " make format-check - Check ruff format code formatting (matches CI)"
@echo " make lint - Run all linting (Ruff, basedpyright, format check, circular imports, import safety)"
@echo " make lint-ruff - Run Ruff linting only"
@echo " make lint-basedpyright - Run basedpyright strict, gated by per-rule error counts"
@echo " make lint-basedpyright-budget-update - Re-capture the basedpyright per-rule budget (ratchet)"
@echo " make lint-basedpyright-budget-update - Ratchet basedpyright limits down by what this branch fixed"
@echo " make lint-format - Check ruff format formatting (matches CI)"
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its ceiling"
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its limit"
@echo " make lint-gate - Strict ruff gate in CI-parity mode (fetches staging, simulates the merge)"
@echo " make lint-ruff-budget-update - Re-capture per-rule baselines in ruff-strict-budget.json (ratchet)"
@echo " make lint-budget-update - Re-capture all ratchet budgets (ruff + basedpyright)"
@echo " make lint-ruff-budget-update - Ratchet ruff-strict-budget.json limits down by what this branch fixed"
@echo " make lint-budget-update - Ratchet all budgets down (ruff + type-discipline + basedpyright)"
@echo " make check-circular-imports - Check for circular imports"
@echo " make check-import-safety - Check import safety"
@echo " make test - Run all tests"
@ -56,8 +58,11 @@ info:
@echo "UV: $(UV)"
# Installation targets
# --inexact: sync the locked deps without pruning anything already installed, so running
# a lint/format target doesn't tear the proxy extras (prisma, websockets, ...) out from
# under a dev's venv (CI installs its own env per job, so it is unaffected by this).
install-dev:
$(UV) sync --frozen
$(UV) sync --inexact --frozen
install-proxy-dev:
$(UV) sync --frozen --group proxy-dev --extra proxy
@ -83,13 +88,38 @@ install-hooks:
# Formatting
# Wrap width is ruff.toml's single source of truth (line-length = 120), shared by the
# formatter, E501, and the import sorter so there's no 88-vs-120 split to reconcile.
# formatter and the import sorter so there's no 88-vs-120 split to reconcile.
format: install-dev
cd litellm && $(UV_RUN) ruff format --exclude '/enterprise/' . && cd ..
format-check: install-dev
cd litellm && $(UV_RUN) ruff format --check --exclude '/enterprise/' . && cd ..
# Single fetch of the PR base so the delta-based gates below share one network round
# trip instead of each re-fetching when chained from `lint`.
lint-fetch-base:
git fetch origin litellm_internal_staging
# Mirror test-linting.yml's lint job environment: the proxy-dev group plus a generated
# Prisma client, so basedpyright resolves the same modules CI does (without the generated
# client the DB wrappers typed against it degrade to Unknown, drifting the budget from
# CI's). --inexact tops up the venv instead of pruning the proxy extras gen:api and the
# running proxy need.
lint-install:
$(UV) sync --inexact --frozen --group proxy-dev
$(UV_RUN) prisma generate --schema litellm/proxy/schema.prisma
# Diff-scoped format check, identical to test-linting.yml's "Check ruff format" step:
# only the litellm Python files changed vs the base are checked, so a pre-existing
# format issue elsewhere doesn't block an unrelated commit.
lint-format-check-changed: install-dev lint-fetch-base
@files=$$(git diff --name-only origin/litellm_internal_staging...HEAD -- 'litellm/**/*.py' | grep -v '^litellm/enterprise/' || true); \
if [ -z "$$files" ]; then \
echo "No changed litellm Python files to format-check."; \
else \
echo "$$files" | xargs $(UV_RUN) ruff format --check --exclude '/enterprise/'; \
fi
# Linting targets
lint-ruff: install-dev
cd litellm && $(UV_RUN) ruff check . && cd ..
@ -126,11 +156,17 @@ lint-ruff-FULL-dev: install-dev
if [ -n "$$files" ]; then echo "$$files" | xargs $(UV_RUN) ruff check; \
else echo "No changed .py files to check."; fi
lint-basedpyright: install-dev
git fetch origin litellm_internal_staging
lint-basedpyright: install-dev lint-fetch-base
($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --base origin/litellm_internal_staging
lint-basedpyright-budget-update: install-dev
# Type-discipline budget (mutable collections / casts / type guards / kwargs /
# unexplained suppressions), the test-linting.yml step `make lint` used to omit.
lint-type-discipline: install-dev lint-fetch-base
$(UV_RUN) python scripts/type_discipline_gate.py --base origin/litellm_internal_staging
# --update lowers each limit by what this branch fixed since its branch point, so
# it needs the base ref fetched to resolve the merge-base.
lint-basedpyright-budget-update: install-dev lint-fetch-base
($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --update
lint-format: format-check
@ -140,15 +176,17 @@ lint-ruff-budget: install-dev
# Strict gate, invoked the same way CI does in test-linting.yml so a local pass
# means the CI check will pass too.
lint-gate: install-dev
git fetch origin litellm_internal_staging
lint-gate: install-dev lint-fetch-base
$(UV_RUN) python scripts/ruff_strict_gate.py --base origin/litellm_internal_staging
lint-ruff-budget-update: install-dev
lint-ruff-budget-update: install-dev lint-fetch-base
$(UV_RUN) python scripts/ruff_strict_gate.py --update
# Ratchet all budgets in one shot (ruff strict + basedpyright)
lint-budget-update: lint-ruff-budget-update lint-basedpyright-budget-update
lint-type-discipline-budget-update: install-dev lint-fetch-base
$(UV_RUN) python scripts/type_discipline_gate.py --update
# Ratchet all budgets in one shot (ruff strict + type-discipline + basedpyright)
lint-budget-update: lint-ruff-budget-update lint-type-discipline-budget-update lint-basedpyright-budget-update
check-circular-imports: install-dev
cd litellm && $(UV_RUN) python ../tests/documentation_tests/test_circular_imports.py && cd ..
@ -156,12 +194,25 @@ check-circular-imports: install-dev
check-import-safety: install-dev
@$(UV_RUN) python -c "from litellm import *; print('[from litellm import *] OK! no issues!');" || (echo '🚨 import failed, this means you introduced unprotected imports! 🚨'; exit 1)
# Combined linting (matches test-linting.yml workflow)
lint: format-check lint-ruff lint-basedpyright check-circular-imports check-import-safety lint-ruff-budget
# Combined linting, isomorphic to test-linting.yml's lint job so a local pass means a
# green CI lint: it installs the same env (proxy-dev + generated Prisma client) and then
# runs the diff-scoped ruff format check, whole-tree ruff check, the strict-rule /
# type-discipline / basedpyright budgets as a delta vs the base, then the circular-import
# and import-safety checks. Steps that compare against the base resolve it the same way CI
# does (merge-base with origin/litellm_internal_staging). lint-install is first so the
# Prisma client exists before basedpyright runs.
lint: lint-install lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright check-circular-imports check-import-safety
# Faster linting for local development (only checks changed code)
lint-dev: lint-format-changed check-circular-imports check-import-safety
# Run the gating CI checks against your staged files right before committing. Mirrors
# test-linting.yml (Python), test-litellm-ui-build.yml's frontend-lint (dashboard), and
# check-ui-api-types.yml (API-type drift), skipping any whose files you didn't stage.
# Not auto-installed as a git hook so it never slows an unrelated human commit.
pre-commit:
./scripts/pre_commit_lint.sh
# Testing targets
test: install-test-deps
$(UV_RUN) pytest tests/

144
basedpyright-code-budget.json
View file

@ -1,194 +1,146 @@
{
"reportAny": {
"baseline": 24989,
"slack": 2500
"limit": 37484
},
"reportArgumentType": {
"baseline": 1814,
"slack": 180
"limit": 2721
},
"reportAssignmentType": {
"baseline": 220,
"slack": 22
"limit": 330
},
"reportAttributeAccessIssue": {
"baseline": 346,
"slack": 35
"limit": 519
},
"reportCallIssue": {
"baseline": 87,
"slack": 10
"limit": 131
},
"reportConstantRedefinition": {
"baseline": 39,
"slack": 4
"limit": 59
},
"reportDeprecated": {
"baseline": 217,
"slack": 22
"limit": 326
},
"reportDuplicateImport": {
"baseline": 28,
"slack": 3
"limit": 42
},
"reportExplicitAny": {
"baseline": 6931,
"slack": 700
"limit": 10397
},
"reportFunctionMemberAccess": {
"baseline": 7,
"slack": 3
"limit": 11
},
"reportGeneralTypeIssues": {
"baseline": 151,
"slack": 15
"limit": 227
},
"reportIncompatibleMethodOverride": {
"baseline": 52,
"slack": 5
"limit": 78
},
"reportIncompatibleVariableOverride": {
"baseline": 8,
"slack": 3
"limit": 12
},
"reportInconsistentOverload": {
"baseline": 12,
"slack": 3
"limit": 18
},
"reportIndexIssue": {
"baseline": 26,
"slack": 3
"limit": 39
},
"reportInvalidTypeForm": {
"baseline": 23,
"slack": 3
"limit": 35
},
"reportInvalidTypeVarUse": {
"baseline": 2,
"slack": 3
"limit": 5
},
"reportMatchNotExhaustive": {
"baseline": 1,
"slack": 0
"limit": 2
},
"reportMissingParameterType": {
"baseline": 3933,
"slack": 390
"limit": 5900
},
"reportMissingTypeArgument": {
"baseline": 10612,
"slack": 1000
"limit": 15918
},
"reportMissingTypeStubs": {
"baseline": 27,
"slack": 10
"limit": 41
},
"reportOperatorIssue": {
"baseline": 6,
"slack": 3
"limit": 9
},
"reportOptionalCall": {
"baseline": 4,
"slack": 3
"limit": 7
},
"reportOptionalIterable": {
"baseline": 3,
"slack": 3
"limit": 6
},
"reportOptionalMemberAccess": {
"baseline": 724,
"slack": 72
"limit": 1086
},
"reportOptionalOperand": {
"baseline": 3,
"slack": 3
"limit": 6
},
"reportOptionalSubscript": {
"baseline": 11,
"slack": 3
"limit": 17
},
"reportPossiblyUnboundVariable": {
"baseline": 52,
"slack": 10
"limit": 78
},
"reportPrivateUsage": {
"baseline": 1625,
"slack": 160
"limit": 2438
},
"reportRedeclaration": {
"baseline": 8,
"slack": 3
"limit": 12
},
"reportReturnType": {
"baseline": 126,
"slack": 100
"limit": 226
},
"reportTypedDictNotRequiredAccess": {
"baseline": 20,
"slack": 3
"limit": 30
},
"reportUndefinedVariable": {
"baseline": 2,
"slack": 3
"limit": 5
},
"reportUnknownArgumentType": {
"baseline": 30603,
"slack": 3000
"limit": 45905
},
"reportUnknownLambdaType": {
"baseline": 75,
"slack": 10
"limit": 113
},
"reportUnknownMemberType": {
"baseline": 27037,
"slack": 2500
"limit": 40556
},
"reportUnknownParameterType": {
"baseline": 13612,
"slack": 1000
"limit": 20418
},
"reportUnknownVariableType": {
"baseline": 21445,
"slack": 2000
"limit": 32168
},
"reportUnnecessaryCast": {
"baseline": 118,
"slack": 10
"limit": 177
},
"reportUnnecessaryComparison": {
"baseline": 683,
"slack": 100
"limit": 1025
},
"reportUnnecessaryContains": {
"baseline": 4,
"slack": 3
"limit": 7
},
"reportUnnecessaryIsInstance": {
"baseline": 808,
"slack": 80
"limit": 1212
},
"reportUntypedBaseClass": {
"baseline": 110,
"slack": 11
"limit": 165
},
"reportUntypedFunctionDecorator": {
"baseline": 22,
"slack": 3
"limit": 33
},
"reportUnusedClass": {
"baseline": 22,
"slack": 3
"limit": 33
},
"reportUnusedFunction": {
"baseline": 137,
"slack": 10
"limit": 206
},
"reportUnusedImport": {
"baseline": 670,
"slack": 50
"limit": 1005
},
"reportUnusedVariable": {
"baseline": 865,
"slack": 50
"limit": 1298
}
}

4
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py
View file

@ -239,6 +239,7 @@ class BaseEmailLogger(CustomLogger):
max_budget_info=max_budget_info,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
email_footer=email_params.signature,
)
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,
@ -311,6 +312,7 @@ class BaseEmailLogger(CustomLogger):
max_budget_info=max_budget_info,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
email_footer=email_params.signature,
)
# Send email to all recipients
@ -379,6 +381,7 @@ class BaseEmailLogger(CustomLogger):
alert_threshold=alert_threshold_str,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
email_footer=email_params.signature,
)
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,
@ -403,6 +406,7 @@ class BaseEmailLogger(CustomLogger):
alert_threshold=alert_threshold_str,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
email_footer=email_params.signature,
)
await self.send_email(
from_email=self.DEFAULT_LITELLM_EMAIL,

233
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -13,6 +13,8 @@ from litellm.constants import (
)
if TYPE_CHECKING:
from litellm.integrations.prometheus import PrometheusLogger
from litellm.proxy._types import LiteLLM_ManagedObjectTable
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
@ -26,6 +28,7 @@ class CheckBatchCost:
proxy_logging_obj: "ProxyLogging",
prisma_client: "PrismaClient",
llm_router: "Router",
track_unmanaged_vertex_batch_cost: bool = False,
):
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
@ -33,6 +36,7 @@ class CheckBatchCost:
self.proxy_logging_obj: ProxyLogging = proxy_logging_obj
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router
self._track_unmanaged_vertex_batch_cost = track_unmanaged_vertex_batch_cost
# Cached after the first poll cycle. Once we know the column is absent we skip
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
@ -97,6 +101,182 @@ class CheckBatchCost:
order={"created_at": "asc"},
)
@staticmethod
def _record_error(
prom_logger: Optional["PrometheusLogger"], error_type: str
) -> None:
if prom_logger is not None:
prom_logger.record_check_batch_cost_error(error_type)
def _resolve_job_routing(
self,
job: "LiteLLM_ManagedObjectTable",
prom_logger: Optional["PrometheusLogger"],
) -> Optional[Tuple[str, str]]:
"""
Resolve (model_id, batch_id) for a managed-object row, where model_id is a router
deployment id and batch_id is the raw provider batch id.
Managed batches encode both in a base64 unified id. Unmanaged Vertex batches, created with
a raw gs:// input_file_id, store the raw provider job id as unified_object_id; when
track_unmanaged_vertex_batch_cost is enabled the model is derived from the gs:// path and
mapped to a configured vertex_ai deployment. Returns None (recording a metric) when the row
can't be routed.
"""
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
get_batch_id_from_unified_batch_id,
get_model_id_from_unified_batch_id,
)
unified_object_id = job.unified_object_id
decoded = _is_base64_encoded_unified_file_id(unified_object_id)
if decoded:
model_id = get_model_id_from_unified_batch_id(decoded)
if model_id is None:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid model id"
)
self._record_error(prom_logger, "invalid_model_id")
return None
return model_id, get_batch_id_from_unified_batch_id(decoded)
if self._track_unmanaged_vertex_batch_cost:
return self._resolve_unmanaged_vertex_routing(job, prom_logger)
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid unified object id"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
def _resolve_unmanaged_vertex_routing(
self,
job: "LiteLLM_ManagedObjectTable",
prom_logger: Optional["PrometheusLogger"],
) -> Optional[Tuple[str, str]]:
from litellm.llms.vertex_ai.batches.transformation import (
VertexAIBatchTransformation,
)
input_file_id = self._get_input_file_id(job)
if not VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
input_file_id
):
verbose_proxy_logger.info(
f"Skipping job {job.unified_object_id}: not an unmanaged vertex batch "
"(no gs:// input_file_id with a publishers/ model path)"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
assert input_file_id is not None # narrowed by is_unmanaged_gcs_batch_input_file_id
bare_model_name = VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
input_file_id
)
deployment_id = self._get_vertex_ai_deployment_id_for_bare_model(
bare_model_name
)
if deployment_id is None:
verbose_proxy_logger.info(
f"Skipping unmanaged vertex batch {job.unified_object_id}: no vertex_ai "
f"deployment configured for model {bare_model_name}"
)
self._record_error(prom_logger, "unmanaged_no_matching_deployment")
return None
return deployment_id, job.unified_object_id
def _get_vertex_ai_deployment_id_for_bare_model(
self, bare_model_name: str
) -> Optional[str]:
model_group = self.llm_router.resolve_model_name_from_model_id(bare_model_name)
deployment_id = (
self._get_vertex_ai_deployment_id(model_group) if model_group else None
)
if deployment_id is not None:
return deployment_id
return self._get_vertex_ai_deployment_id_from_matching_deployments(
bare_model_name
)
def _get_vertex_ai_deployment_id_from_matching_deployments(
self, bare_model_name: str
) -> Optional[str]:
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
for deployment in self.llm_router.get_model_list(model_name=None) or []:
litellm_params = deployment.get("litellm_params") or {}
actual_model = litellm_params.get("model")
if not isinstance(actual_model, str):
continue
if not self._is_bare_model_match(actual_model, bare_model_name):
continue
try:
_, llm_provider, _, _ = get_llm_provider(
model=actual_model,
custom_llm_provider=litellm_params.get("custom_llm_provider"),
)
except Exception:
continue
if llm_provider != "vertex_ai":
continue
model_info = deployment.get("model_info") or {}
deployment_id = model_info.get("id")
if isinstance(deployment_id, str):
return deployment_id
return None
@staticmethod
def _is_bare_model_match(actual_model: str, bare_model_name: str) -> bool:
return (
actual_model == bare_model_name
or actual_model.endswith(f"/{bare_model_name}")
or actual_model.endswith(f":{bare_model_name}")
)
def _get_vertex_ai_deployment_id(self, model_group: str) -> Optional[str]:
"""
Returns the first deployment id for `model_group` whose provider is vertex_ai,
skipping deployments from other providers that happen to share the model group name.
"""
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
for deployment_id in self.llm_router.get_model_ids(model_name=model_group):
deployment_info = self.llm_router.get_deployment(model_id=deployment_id)
if deployment_info is None:
continue
try:
_, llm_provider, _, _ = get_llm_provider(
model=deployment_info.litellm_params.model,
custom_llm_provider=deployment_info.litellm_params.custom_llm_provider,
)
except Exception:
continue
if llm_provider == "vertex_ai":
return deployment_id
return None
@staticmethod
def _get_input_file_id(job: "LiteLLM_ManagedObjectTable") -> Optional[str]:
import json
from litellm.types.utils import LiteLLMBatch
file_object = job.file_object
if isinstance(file_object, str):
try:
file_object = json.loads(file_object)
except (json.JSONDecodeError, ValueError):
return None
if not isinstance(file_object, dict):
return None
try:
return LiteLLMBatch.model_validate(file_object).input_file_id
except Exception:
return None
async def check_batch_cost(self):
"""
Check if the batch JOB has been tracked.
@ -114,8 +294,6 @@ class CheckBatchCost:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
get_batch_id_from_unified_batch_id,
get_model_id_from_unified_batch_id,
)
try:
@ -172,31 +350,10 @@ class CheckBatchCost:
else:
jobs = await self._fallback_find_jobs()
for job in jobs:
# get the model from the job
unified_object_id = job.unified_object_id
decoded_unified_object_id = _is_base64_encoded_unified_file_id(
unified_object_id
)
if not decoded_unified_object_id:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid unified object id"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("invalid_unified_id")
continue
else:
unified_object_id = decoded_unified_object_id
model_id = get_model_id_from_unified_batch_id(unified_object_id)
batch_id = get_batch_id_from_unified_batch_id(unified_object_id)
if model_id is None:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid model id"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("invalid_model_id")
routing = self._resolve_job_routing(job, prom_logger)
if routing is None:
continue
model_id, batch_id = routing
verbose_proxy_logger.info(
f"Querying model ID: {model_id} for cost and usage of batch ID: {batch_id}"
@ -213,7 +370,7 @@ class CheckBatchCost:
)
except Exception as e:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because of error querying model ID: {model_id} for cost and usage of batch ID: {batch_id}: {e}"
f"Skipping job {job.unified_object_id} because of error querying model ID: {model_id} for cost and usage of batch ID: {batch_id}: {e}"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("provider_retrieval_error")
@ -287,7 +444,7 @@ class CheckBatchCost:
deployment_info = self.llm_router.get_deployment(model_id=model_id)
if deployment_info is None:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid deployment info"
f"Skipping job {job.unified_object_id} because it is not a valid deployment info"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("deployment_not_found")
@ -413,6 +570,26 @@ class CheckBatchCost:
f"CheckBatchCost: failed to mark job {job.id} complete in DB: {db_err}"
)
elif response.status in ("failed", "expired", "cancelled"):
try:
update_data = {
"status": response.status,
"file_object": response.model_dump_json(),
}
if self._has_batch_processed_column:
update_data["batch_processed"] = True
await self.prisma_client.db.litellm_managedobjecttable.update(
where={"id": job.id},
data=update_data,
)
verbose_proxy_logger.info(
f"CheckBatchCost: marked job {job.id} as {response.status} in DB"
)
except Exception as db_err:
verbose_proxy_logger.error(
f"CheckBatchCost: failed to mark job {job.id} as {response.status} in DB: {db_err}"
)
# Record polling run metrics (always, even if nothing was processed)
if prom_logger:
prom_logger.record_check_batch_cost_run(

16
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -125,23 +125,33 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"team_id": user_api_key_dict.team_id,
"updated_by": user_api_key_dict.user_id,
}
update_data = {
"model_mappings": json.dumps(model_mappings),
"flat_model_file_ids": list(model_mappings.values()),
"updated_by": user_api_key_dict.user_id,
}
if file_object is not None:
db_data["file_object"] = file_object.model_dump_json()
file_object_json = file_object.model_dump_json()
db_data["file_object"] = file_object_json
update_data["file_object"] = file_object_json
# Extract storage metadata from hidden params if present
hidden_params = getattr(file_object, "_hidden_params", {}) or {}
if "storage_backend" in hidden_params:
db_data["storage_backend"] = hidden_params["storage_backend"]
update_data["storage_backend"] = hidden_params["storage_backend"]
if "storage_url" in hidden_params:
db_data["storage_url"] = hidden_params["storage_url"]
update_data["storage_url"] = hidden_params["storage_url"]
verbose_logger.debug(
f"Storage metadata: storage_backend={db_data.get('storage_backend')}, "
f"storage_url={db_data.get('storage_url')}"
)
result = await self.prisma_client.db.litellm_managedfiletable.create(
data=db_data
result = await self.prisma_client.db.litellm_managedfiletable.upsert(
where={"unified_file_id": file_id},
data={"create": db_data, "update": update_data},
)
verbose_logger.debug(
f"LiteLLM Managed File object with id={file_id} stored in db: {result}"

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.44"
version = "0.1.45"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.44"
version = "0.1.45"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

2
litellm-proxy-extras/litellm_proxy_extras/migrations/20260626120000_add_mcp_tool_search_enabled/migration.sql Normal file
View file

@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "LiteLLM_ObjectPermissionTable" ADD COLUMN "mcp_tool_search_enabled" BOOLEAN;

1
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -279,6 +279,7 @@ model LiteLLM_ObjectPermissionTable {
blocked_tools String[] @default([]) // Tool names blocked for any key/team/user with this permission
mcp_toolsets String[] @default([]) // Toolset IDs granted to this key/team/user
search_tools String[] @default([]) // search_tool_name values this key/team/user may call
mcp_tool_search_enabled Boolean?
teams LiteLLM_TeamTable[]
projects LiteLLM_ProjectTable[]
verification_tokens LiteLLM_VerificationToken[]

3
litellm/__init__.py
View file

@ -263,6 +263,8 @@ azure_key: Optional[str] = None
anthropic_key: Optional[str] = None
replicate_key: Optional[str] = None
bytez_key: Optional[str] = None
gdc_key: Optional[str] = None
gdc_api_base: Optional[str] = None
cohere_key: Optional[str] = None
infinity_key: Optional[str] = None
clarifai_key: Optional[str] = None
@ -1787,6 +1789,7 @@ if TYPE_CHECKING:
from .llms.nvidia_nim.embed import (
NvidiaNimEmbeddingConfig as NvidiaNimEmbeddingConfig,
)
from .llms.gdc.chat.transformation import GDCGeminiConfig as GDCGeminiConfig
# Type stubs for lazy-loaded config instances
openaiOSeriesConfig: OpenAIOSeriesConfig

5
litellm/_lazy_imports_registry.py
View file

@ -323,6 +323,7 @@ LLM_CONFIG_NAMES = (
"SnowflakeEmbeddingConfig",
"AmazonNovaChatConfig",
"SonioxAudioTranscriptionConfig",
"GDCGeminiConfig",
)
# Types that support lazy loading via _lazy_import_types
@ -1157,6 +1158,10 @@ _LLM_CONFIGS_IMPORT_MAP = {
".llms.dashscope.chat.transformation",
"DashScopeChatConfig",
),
"GDCGeminiConfig": (
".llms.gdc.chat.transformation",
"GDCGeminiConfig",
),
"ModelScopeChatConfig": (
".llms.modelscope.chat.transformation",
"ModelScopeChatConfig",

15
litellm/_redis.py
View file

@ -23,7 +23,11 @@ from litellm._redis_credential_provider import (
GCPIAMCredentialProvider,
_generate_gcp_iam_access_token,
)
from litellm.constants import REDIS_CONNECTION_POOL_TIMEOUT, REDIS_SOCKET_TIMEOUT
from litellm.constants import (
REDIS_CLUSTER_HEALTH_CHECK_INTERVAL,
REDIS_CONNECTION_POOL_TIMEOUT,
REDIS_SOCKET_TIMEOUT,
)
from litellm.litellm_core_utils.sensitive_data_masker import SensitiveDataMasker
from ._logging import verbose_logger
@ -102,6 +106,8 @@ def _get_redis_cluster_kwargs(client=None):
"max_connections",
"socket_timeout",
"socket_connect_timeout",
"health_check_interval",
"socket_keepalive",
}
return available_args
@ -579,6 +585,13 @@ def get_redis_async_client(
new_startup_nodes.append(ClusterNode(**item))
cluster_kwargs.pop("startup_nodes", None)
# Default to a periodic health check + TCP keepalive so a connection silently dropped
# by a cluster restart (e.g. ElastiCache Serverless maintenance) is revalidated and
# reconnected before reuse instead of stalling in re-initialization; an explicit value
# from config still wins.
cluster_kwargs.setdefault("health_check_interval", REDIS_CLUSTER_HEALTH_CHECK_INTERVAL)
cluster_kwargs.setdefault("socket_keepalive", True)
# Create async RedisCluster with IAM token as password if available
cluster_client = async_redis.RedisCluster(
startup_nodes=new_startup_nodes,

6
litellm/constants.py
View file

@ -332,6 +332,10 @@ REDIS_CONNECTION_POOL_TIMEOUT = int(os.getenv("REDIS_CONNECTION_POOL_TIMEOUT", 5
REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD = int(os.getenv("REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD", 5))
REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT = int(os.getenv("REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT", 60))
REDIS_CIRCUIT_BREAKER_ENABLED = os.getenv("REDIS_CIRCUIT_BREAKER_ENABLED", "true").lower() == "true"
# Seconds of idle before a Redis cluster connection is validated with a PING and
# reconnected if dead, so a connection silently dropped by a cluster restart
# (e.g. ElastiCache Serverless maintenance) is not reused while broken
REDIS_CLUSTER_HEALTH_CHECK_INTERVAL = 25
# Default Redis major version to assume when version cannot be determined
# Using 7 as it's the modern version that supports LPOP with count parameter
DEFAULT_REDIS_MAJOR_VERSION = int(os.getenv("DEFAULT_REDIS_MAJOR_VERSION", 7))
@ -456,6 +460,7 @@ LITELLM_CHAT_PROVIDERS = [
"openai",
"openai_like",
"bytez",
"gdc",
"xai",
"custom_openai",
"text-completion-openai",
@ -1123,6 +1128,7 @@ BEDROCK_CONVERSE_MODELS = [
"anthropic.claude-haiku-4-5-20251001-v1:0",
"anthropic.claude-sonnet-4-5-20250929-v1:0",
"anthropic.claude-fable-5",
"anthropic.claude-sonnet-5",
"anthropic.claude-opus-4-8",
"anthropic.claude-opus-4-7",
"anthropic.claude-opus-4-6-v1:0",

97
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -1,9 +1,12 @@
"""
This hook is used to inject cache control directives into the messages of a chat completion.
This hook is used to inject cache control directives into messages.
Users can define
- `cache_control_injection_points` in the completion params and litellm will inject the cache control directives into the messages at the specified injection points.
Supported for both `v1/chat/completions` (via the prompt-management hook) and
`v1/messages` (via `apply_to_anthropic_messages_request`).
"""
import copy
@ -225,6 +228,98 @@ class AnthropicCacheControlHook(CustomPromptManagement):
message_content[-1]["cache_control"] = control # type: ignore
return message
@staticmethod
def apply_to_anthropic_messages_request(
messages: List[Dict],
system: str | list | None,
injection_points: List[CacheControlInjectionPoint],
) -> Tuple[List[Dict], str | list | None, List[CacheControlInjectionPoint]]:
"""Apply cache control injection for the Anthropic-native v1/messages endpoint.
Returns (messages, system, remaining_non_message_points).
"""
if not injection_points:
return messages, system, []
processed_messages: List[Dict] = copy.deepcopy(messages)
processed_system = copy.deepcopy(system) if system is not None else None
message_points: List[CacheControlMessageInjectionPoint] = []
system_points: List[CacheControlMessageInjectionPoint] = []
remaining_points: List[CacheControlInjectionPoint] = []
for point in injection_points:
if point.get("location") == "message":
msg_point = cast(CacheControlMessageInjectionPoint, point)
if msg_point.get("role") == "system":
system_points.append(msg_point)
else:
message_points.append(msg_point)
else:
remaining_points.append(point)
reserved_blocks = 1 if any(p.get("location") == "tool_config" for p in remaining_points) else 0
max_blocks = MAX_CACHE_CONTROL_BLOCKS - reserved_blocks
used_blocks = sum(
AnthropicCacheControlHook._count_cache_control_blocks(cast(AllMessageValues, msg))
for msg in processed_messages
)
if isinstance(processed_system, list):
used_blocks += sum(
1 for b in processed_system if isinstance(b, dict) and b.get("cache_control") is not None
)
if system_points and processed_system is not None and used_blocks < max_blocks:
system_already_has_cc = isinstance(processed_system, list) and any(
isinstance(b, dict) and b.get("cache_control") is not None for b in processed_system
)
if not system_already_has_cc:
control = system_points[0].get("control") or ChatCompletionCachedContent(type="ephemeral")
if isinstance(processed_system, str):
processed_system = [{"type": "text", "text": processed_system, "cache_control": control}]
used_blocks += 1
elif len(processed_system) > 0 and isinstance(processed_system[-1], dict):
processed_system[-1] = {**processed_system[-1], "cache_control": control}
used_blocks += 1
for i, msg in enumerate(processed_messages):
content = msg.get("content")
if isinstance(content, str):
processed_messages[i] = {**msg, "content": [{"type": "text", "text": content}]}
processed_messages = AnthropicCacheControlHook._apply_message_injections(
points=message_points,
messages=cast(List[AllMessageValues], processed_messages),
max_blocks=max_blocks - used_blocks,
)
return processed_messages, processed_system, remaining_points
@staticmethod
def maybe_inject_cache_control(
messages: List[Dict],
system: str | list | None,
kwargs: Dict[str, Any],
) -> Tuple[List[Dict], str | list | None]:
"""Extract cache_control_injection_points from kwargs and apply if present.
Pops the key from kwargs; if remaining (non-message) points exist they
are written back so downstream transforms can handle them.
"""
injection_points = kwargs.pop("cache_control_injection_points", None)
if not injection_points:
return messages, system
messages, system, remaining = AnthropicCacheControlHook.apply_to_anthropic_messages_request(
messages=messages,
system=system,
injection_points=injection_points,
)
if remaining:
kwargs["cache_control_injection_points"] = remaining
return messages, system
@property
def integration_name(self) -> str:
"""Return the integration name for this hook."""

71
litellm/integrations/code_interpreter_interception/handler.py
View file

@ -40,9 +40,11 @@ from litellm.types.utils import (
LITELLM_CODE_EXECUTION_TOOL_NAME = "litellm_code_execution"
_INTERCEPTION_ACTIVE_KEY = "_code_interpreter_interception_active"
_SANDBOX_KEY = "_code_interpreter_interception_sandbox_key"
_SESSION_SCOPED_KEY = "_code_interpreter_interception_session_scoped"
_CONVERTED_STREAM_KEY = "_code_interpreter_interception_converted_stream"
_LITELLM_METADATA_KEY = "litellm_metadata"
_CACHE_TTL_SECONDS = 15 * 60
_SESSION_SCOPED_PER_IDENTITY_CAP = 10
class CodeExecutionToolCall(TypedDict, total=False):
@ -107,6 +109,20 @@ class ChatCompletionFunctionToolChoice(TypedDict):
CodeExecutionFunctionToolChoice = ResponsesFunctionToolChoice | ChatCompletionFunctionToolChoice
def _extract_session_id(kwargs: dict[str, Any]) -> str | None:
for meta_key in ("metadata", "litellm_metadata"):
meta = kwargs.get(meta_key)
if isinstance(meta, dict):
sid = meta.get("session_id")
if sid and isinstance(sid, str):
return sid
return None
def _extract_identity(kwargs: dict[str, Any]) -> str:
return kwargs.get("user_api_key_hash") or ""
def _resolve_sandbox_tool(sandbox_tool_name: str | None) -> dict[str, Any] | None:
try:
from litellm.sandbox.sandbox_tools import resolve_sandbox_tool
@ -140,7 +156,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
self.enabled_providers = enabled_providers
self.sandbox_tool_name = sandbox_tool_name
self.sandbox_config = sandbox_config
self._container_cache: dict[str, tuple[Any, dict[str, Any] | None, float]] = {}
self._container_cache: dict[str, tuple[Any, dict[str, Any] | None, float, str | None]] = {}
@classmethod
def from_config_yaml(cls, config: CodeInterpreterInterceptionConfig) -> "CodeInterpreterInterceptionLogger":
@ -191,7 +207,13 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
return None
kwargs[_INTERCEPTION_ACTIVE_KEY] = True
kwargs[_SANDBOX_KEY] = uuid.uuid4().hex
session_id = _extract_session_id(kwargs)
if session_id:
identity = _extract_identity(kwargs)
kwargs[_SANDBOX_KEY] = f"{identity}:{session_id}" if identity else session_id
kwargs[_SESSION_SCOPED_KEY] = True
else:
kwargs[_SANDBOX_KEY] = uuid.uuid4().hex
if kwargs.get("stream"):
kwargs["stream"] = False
kwargs[_CONVERTED_STREAM_KEY] = True
@ -217,6 +239,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
if not is_interception_internal_key(key)
and not key.startswith("_agentic_loop")
and key != "max_agentic_loops"
and key != _SESSION_SCOPED_KEY
}
if filtered_metadata:
kwargs[_LITELLM_METADATA_KEY] = filtered_metadata
@ -227,7 +250,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
def _write_interception_metadata(kwargs: dict[str, Any]) -> None:
metadata = kwargs.get(_LITELLM_METADATA_KEY)
metadata = dict(metadata) if isinstance(metadata, dict) else {}
for key in (_INTERCEPTION_ACTIVE_KEY, _SANDBOX_KEY, _CONVERTED_STREAM_KEY):
for key in (_INTERCEPTION_ACTIVE_KEY, _SANDBOX_KEY, _SESSION_SCOPED_KEY, _CONVERTED_STREAM_KEY):
if key in kwargs:
metadata[key] = kwargs[key]
kwargs[_LITELLM_METADATA_KEY] = metadata
@ -347,7 +370,9 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
await self._prune_expired_cache()
tool_calls = cast(list[CodeExecutionToolCall], tools.get("tool_calls", []))
sandbox_key = kwargs.get(_SANDBOX_KEY)
container, params = await self._get_or_create_container(cache_key=sandbox_key)
is_session = bool(kwargs.get(_SESSION_SCOPED_KEY))
identity = _extract_identity(kwargs) if is_session else None
container, params = await self._get_or_create_container(cache_key=sandbox_key, identity=identity)
try:
container_id = cast(str | None, getattr(container, "id", None))
@ -404,6 +429,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
metadata={
"tool_type": "code_interpreter",
"sandbox_key": sandbox_key or "",
"is_session_scoped": bool(kwargs.get(_SESSION_SCOPED_KEY)),
"code_interpreter_calls": code_interpreter_calls,
},
)
@ -419,7 +445,9 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
await self._prune_expired_cache()
tool_calls = cast(list[CodeExecutionToolCall], tools.get("tool_calls", []))
sandbox_key = cast(str | None, kwargs.get(_SANDBOX_KEY))
container, params = await self._get_or_create_container(cache_key=sandbox_key)
is_session = bool(kwargs.get(_SESSION_SCOPED_KEY))
identity = _extract_identity(cast(dict[str, Any], kwargs)) if is_session else None
container, params = await self._get_or_create_container(cache_key=sandbox_key, identity=identity)
try:
container_id = cast(str | None, getattr(container, "id", None))
@ -455,6 +483,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
metadata={
"tool_type": "code_interpreter",
"sandbox_key": sandbox_key or "",
"is_session_scoped": bool(kwargs.get(_SESSION_SCOPED_KEY)),
"code_interpreter_calls": code_interpreter_calls,
"response_format": "openai",
},
@ -489,6 +518,8 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
async def async_agentic_loop_cleanup_hook(self, plan: AgenticLoopPlan, kwargs: dict) -> None:
metadata = plan.metadata or {} if plan else {}
if metadata.get("is_session_scoped"):
return
await self._delete_container_for_cache_key(metadata.get("sandbox_key"))
@staticmethod
@ -520,7 +551,8 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
async def async_post_agentic_loop_response_hook(self, response: Any, plan: AgenticLoopPlan, kwargs: dict) -> Any:
metadata = plan.metadata or {} if plan else {}
await self._delete_container_for_cache_key(metadata.get("sandbox_key"))
if not metadata.get("is_session_scoped"):
await self._delete_container_for_cache_key(metadata.get("sandbox_key"))
calls = metadata.get("code_interpreter_calls")
if not calls:
@ -565,17 +597,32 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
return f"[execution error] {message}"
return getattr(result, "stdout", "") or ""
async def _get_or_create_container(self, cache_key: str | None) -> tuple[Any, dict[str, Any] | None]:
async def _get_or_create_container(
self,
cache_key: str | None,
identity: str | None = None,
) -> tuple[Any, dict[str, Any] | None]:
if cache_key:
cached = self._container_cache.get(cache_key)
if cached is not None:
self._container_cache[cache_key] = (cached[0], cached[1], time.time(), cached[3])
return cached[0], cached[1]
container, params = await self._create_container()
if cache_key:
self._container_cache[cache_key] = (container, params, time.time())
if identity is not None:
await self._evict_lru_session_if_over_cap(identity)
self._container_cache[cache_key] = (container, params, time.time(), identity)
return container, params
async def _evict_lru_session_if_over_cap(self, identity: str) -> None:
identity_entries = [(k, v) for k, v in self._container_cache.items() if v[3] == identity]
if len(identity_entries) < _SESSION_SCOPED_PER_IDENTITY_CAP:
return
lru_key, lru_entry = min(identity_entries, key=lambda item: item[1][2])
self._container_cache.pop(lru_key, None)
await self._delete_container(container=lru_entry[0], params=lru_entry[1])
async def _create_container(self) -> tuple[Any, dict[str, Any] | None]:
if self.sandbox_config is not None:
return await self.sandbox_config.acreate_sandbox(), None
@ -739,12 +786,8 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
now = time.time()
expired = [
(cache_key, container, params)
for cache_key, (
container,
params,
created_at,
) in self._container_cache.items()
if now - created_at > _CACHE_TTL_SECONDS
for cache_key, (container, params, last_accessed, *_) in self._container_cache.items()
if now - last_accessed > _CACHE_TTL_SECONDS
]
for cache_key, container, params in expired:
self._container_cache.pop(cache_key, None)

9
litellm/integrations/email_templates/templates.py
View file

@ -81,8 +81,7 @@ SOFT_BUDGET_ALERT_EMAIL_TEMPLATE = """
If you have any questions, please send an email to {email_support_contact} <br /> <br />
Best, <br />
The LiteLLM team <br />
{email_footer}
"""
TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE = """
@ -105,8 +104,7 @@ TEAM_SOFT_BUDGET_ALERT_EMAIL_TEMPLATE = """
If you have any questions, please send an email to {email_support_contact} <br /> <br />
Best, <br />
The LiteLLM team <br />
{email_footer}
"""
MAX_BUDGET_ALERT_EMAIL_TEMPLATE = """
@ -129,6 +127,5 @@ MAX_BUDGET_ALERT_EMAIL_TEMPLATE = """
If you have any questions, please send an email to {email_support_contact} <br /> <br />
Best, <br />
The LiteLLM team <br />
{email_footer}
"""

4
litellm/integrations/otel/__init__.py
View file

@ -32,11 +32,13 @@ from litellm.integrations.otel.model.payloads import (
LLMCallSpanData,
LLMRequestParams,
LLMUsage,
MCPListToolsSpanData,
MCPToolCallSpanData,
ProxyRequestSpanData,
ServerInfo,
ServiceSpanData,
SpanError,
is_mcp_list_tools,
is_mcp_tool_call,
)
from litellm.integrations.otel.model.semconv import (
@ -106,6 +108,7 @@ __all__ = [
"LLMCallSpanData",
"LLMRequestParams",
"LLMUsage",
"MCPListToolsSpanData",
"MCPToolCallSpanData",
"ProxyRequestSpanData",
"RequestContext",
@ -113,6 +116,7 @@ __all__ = [
"ServerInfo",
"ServiceSpanData",
"SpanError",
"is_mcp_list_tools",
"is_mcp_tool_call",
"promoted_baggage",
]

21
litellm/integrations/otel/emitter.py
View file

@ -4,7 +4,7 @@ from collections import OrderedDict
from typing import Callable, Sequence
from opentelemetry.context import Context
from opentelemetry.trace import Span, Tracer
from opentelemetry.trace import Link, Span, Tracer
from opentelemetry.trace.status import Status, StatusCode
from litellm.integrations.otel.model.config import OpenTelemetryV2Config
@ -13,6 +13,7 @@ from litellm.integrations.otel.mappers.base import AttributeMapper, SpanData
from litellm.integrations.otel.model.payloads import (
GuardrailSpanData,
LLMCallSpanData,
MCPListToolsSpanData,
MCPToolCallSpanData,
ServiceSpanData,
)
@ -23,6 +24,7 @@ from litellm.integrations.otel.model.spans import (
SpanRole,
guardrail_span_name,
llm_call_span_name,
mcp_list_tools_span_name,
mcp_tool_call_span_name,
service_span_name,
)
@ -33,6 +35,7 @@ from litellm.integrations.otel.model.spans import (
_NAME_BUILDERS: dict[SpanRole, Callable[..., str]] = {
SpanRole.LLM_CALL: llm_call_span_name,
SpanRole.MCP_TOOL_CALL: mcp_tool_call_span_name,
SpanRole.MCP_LIST_TOOLS: mcp_list_tools_span_name,
SpanRole.GUARDRAIL: guardrail_span_name,
# DB_CALL and SERVICE are both built from ServiceSpanData; they differ only in
# span kind (CLIENT vs INTERNAL) and attribute vocabulary, not in naming.
@ -74,18 +77,21 @@ class SpanEmitter:
start_time_ns: int | None = None,
*,
tracer: Tracer | None = None,
links: Sequence[Link] | None = None,
) -> Span:
"""Start a span for ``role`` without dedup or attribute mapping.
For callers that own and manage their own span lifecycle. ``tracer``
overrides the bound tracer for this span only, used for per-request
multi-tenant credential routing.
multi-tenant credential routing. ``links`` records related-but-not-parent
spans (e.g. the transport span of an MCP message, per MCP semconv).
"""
return (tracer or self._tracer).start_span(
name,
context=parent_context,
kind=to_otel_span_kind(SPAN_REGISTRY[role].kind),
start_time=start_time_ns,
links=list(links) if links else None,
)
def _seen(self, dedup_key: str | None, role: SpanRole) -> bool:
@ -116,16 +122,23 @@ class SpanEmitter:
start_time_ns: int | None = None,
end_time_ns: int | None = None,
tracer: Tracer | None = None,
links: Sequence[Link] | None = None,
) -> Span | None:
"""Emit one complete span: dedup, start, map attributes, status, end.
Return the span, or ``None`` if it was deduplicated away. ``tracer``
overrides the bound tracer for this span, used for per-request routing.
``links`` records related-but-not-parent spans (the transport span of an
MCP message).
"""
# LLM-call and MCP tool-call spans carry a dedup key (their request's
# call id), so a sync+async double-firing coalesces. ``isinstance`` narrows
# the type for mypy and keeps the engine free of duck-typed attribute reads.
dedup_key = data.identity.call_id if isinstance(data, (LLMCallSpanData, MCPToolCallSpanData)) else None
dedup_key = (
data.identity.call_id
if isinstance(data, (LLMCallSpanData, MCPToolCallSpanData, MCPListToolsSpanData))
else None
)
if self._seen(dedup_key, role):
return None
span = self.start_span(
@ -134,6 +147,7 @@ class SpanEmitter:
parent_context=parent_context,
start_time_ns=start_time_ns,
tracer=tracer,
links=links,
)
self.finish_span(role, span, data, end_time_ns=end_time_ns)
return span
@ -166,6 +180,7 @@ class SpanEmitter:
(
LLMCallSpanData,
MCPToolCallSpanData,
MCPListToolsSpanData,
ServiceSpanData,
GuardrailSpanData,
),

85
litellm/integrations/otel/logger.py
View file

@ -5,7 +5,7 @@ from contextlib import contextmanager
from datetime import datetime
from typing import TYPE_CHECKING, Any, Callable, Iterator, Mapping, Sequence, cast
from opentelemetry.context import attach, get_current
from opentelemetry.context import Context, attach, get_current
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import Span, Tracer, get_current_span, use_span
@ -17,6 +17,7 @@ from litellm.integrations.otel.model.config import OpenTelemetryV2Config
from litellm.integrations.otel.plumbing.context import (
is_recordable_span,
request_root_span,
resolve_mcp_span_context,
resolve_parent_context,
resolve_request_span_context,
set_request_baggage,
@ -32,9 +33,11 @@ from litellm.integrations.otel.model.metadata import (
from litellm.integrations.otel.model.payloads import (
GuardrailSpanData,
LLMCallSpanData,
MCPListToolsSpanData,
MCPToolCallSpanData,
ServiceSpanData,
SpanError,
is_mcp_list_tools,
is_mcp_tool_call,
)
from litellm.integrations.otel.plumbing.metrics import (
@ -218,6 +221,8 @@ class OpenTelemetryV2(CustomLogger):
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
if self._emit_mcp_tool_call(kwargs, start_time, end_time):
return
if self._emit_mcp_list_tools(kwargs, start_time, end_time):
return
self._close_llm_call(kwargs, start_time, end_time)
self._record_metrics(kwargs, response_obj, start_time, end_time)
@ -242,8 +247,24 @@ class OpenTelemetryV2(CustomLogger):
async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
if self._emit_mcp_tool_call(kwargs, start_time, end_time):
return
if self._emit_mcp_list_tools(kwargs, start_time, end_time):
return
self._close_llm_call(kwargs, start_time, end_time)
def _seed_identity_baggage(self, identity: RequestIdentity, model: str | None, context: Context) -> Context:
"""Seed authenticated request-identity Baggage onto ``context`` so the Baggage
processor stamps team/key/metadata onto the span. Identity is read from the
parsed payload, never the client's ``params._meta`` carrier, so it can't be
spoofed."""
bag = promoted_baggage(
identity,
model,
promoted_keys=tuple(self.config.baggage_promoted_keys),
metadata_keys=tuple(self.config.baggage_metadata_keys),
team_metadata_keys=tuple(self.config.baggage_team_metadata_keys),
)
return set_request_baggage(bag, context=context) if bag else context
def _emit_mcp_tool_call(
self,
kwargs: Mapping[str, Any],
@ -254,10 +275,12 @@ class OpenTelemetryV2(CustomLogger):
MCP tool calls reach the success/failure callbacks like any other request
(with ``call_type`` ``call_mcp_tool``), but they are not LLM calls and have
no ``pre_call`` carrier — so they get their own CLIENT span here, parented
to the request's server span. Returns whether it handled the event, so the
caller skips the LLM-call path. The whole span is emitted at once (there is
no boundary to open it at), deduped on the call id by the emitter.
no ``pre_call`` carrier — so they get their own CLIENT span here. Per the MCP
semconv it parents to the trace context the client propagated in
``params._meta`` (or starts a new root) and links the transport span, rather
than nesting under the HTTP/session span. Returns whether it handled the
event, so the caller skips the LLM-call path. The whole span is emitted at
once (there is no boundary to open it at), deduped on the call id.
"""
raw_payload = kwargs.get("standard_logging_object")
if not raw_payload or not is_mcp_tool_call(cast(Mapping[str, object], raw_payload)):
@ -271,12 +294,51 @@ class OpenTelemetryV2(CustomLogger):
# as a phantom LLM span.
if data.identity.call_id:
self._open_llm_calls.pop(data.identity.call_id, None)
parent_context, links = resolve_mcp_span_context()
parent_context = self._seed_identity_baggage(data.identity, None, parent_context)
self._emitter.emit(
SpanRole.MCP_TOOL_CALL,
data,
parent_context=resolve_request_span_context(),
parent_context=parent_context,
start_time_ns=to_ns(start_time),
end_time_ns=to_ns(end_time),
links=links,
)
return True
def _emit_mcp_list_tools(
self,
kwargs: Mapping[str, object],
start_time: datetime | float | None,
end_time: datetime | float | None,
) -> bool:
"""Emit an MCP ``tools/list`` span when the closed request was a discovery call.
Like a tool call, listing reaches the success/failure callbacks (here with
``call_type`` ``list_mcp_tools``) with no ``pre_call`` carrier, so it gets its
own CLIENT span. Per the MCP semconv it parents to the ``params._meta`` trace
context (or starts a new root) and links the transport span, rather than
nesting under the HTTP/session span. Returns whether it handled the event so
the caller skips the LLM-call path.
"""
raw_payload = kwargs.get("standard_logging_object")
if not raw_payload or not is_mcp_list_tools(cast(Mapping[str, object], raw_payload)):
return False
payload = cast("StandardLoggingPayload", raw_payload)
data = MCPListToolsSpanData.from_standard_logging_payload(
payload, capture_content=self.config.capture_span_content
)
if data.identity.call_id:
self._open_llm_calls.pop(data.identity.call_id, None)
parent_context, links = resolve_mcp_span_context()
parent_context = self._seed_identity_baggage(data.identity, None, parent_context)
self._emitter.emit(
SpanRole.MCP_LIST_TOOLS,
data,
parent_context=parent_context,
start_time_ns=to_ns(start_time),
end_time_ns=to_ns(end_time),
links=links,
)
return True
@ -319,16 +381,7 @@ class OpenTelemetryV2(CustomLogger):
# root span — parent to it (ambient fallback on the SDK path). Seed identity
# Baggage so the span — and the SDK path, which has none — is labeled
# consistently.
parent_ctx = resolve_request_span_context()
bag = promoted_baggage(
data.identity,
data.request_model,
promoted_keys=tuple(self.config.baggage_promoted_keys),
metadata_keys=tuple(self.config.baggage_metadata_keys),
team_metadata_keys=tuple(self.config.baggage_team_metadata_keys),
)
if bag:
parent_ctx = set_request_baggage(bag, context=parent_ctx)
parent_ctx = self._seed_identity_baggage(data.identity, data.request_model, resolve_request_span_context())
return self._emitter.emit(
SpanRole.LLM_CALL,
data,

3
litellm/integrations/otel/mappers/base.py
View file

@ -7,6 +7,7 @@ from typing_extensions import Protocol, runtime_checkable
from litellm.integrations.otel.model.payloads import (
GuardrailSpanData,
LLMCallSpanData,
MCPListToolsSpanData,
MCPToolCallSpanData,
ServiceSpanData,
)
@ -20,7 +21,7 @@ AttributeMap = dict[str, AttrValue]
# The closed set of span-data types the engine routes through the mapper chain.
# Server spans (PROXY_REQUEST + management routes) belong to the mounted FastAPI
# instrumentor, not the mapper chain.
SpanData = LLMCallSpanData | MCPToolCallSpanData | GuardrailSpanData | ServiceSpanData
SpanData = LLMCallSpanData | MCPToolCallSpanData | MCPListToolsSpanData | GuardrailSpanData | ServiceSpanData
@runtime_checkable

12
litellm/integrations/otel/mappers/genai.py
View file

@ -19,6 +19,7 @@ from litellm.integrations.otel.mappers.utils import (
from litellm.integrations.otel.model.payloads import (
GuardrailSpanData,
LLMCallSpanData,
MCPListToolsSpanData,
MCPToolCallSpanData,
ServiceSpanData,
ToolDefinition,
@ -100,6 +101,15 @@ class GenAIMapper:
f"{LiteLLM.COST_PREFIX}total": lambda d: d.response_cost,
}
# A tools/list discovery span: the method and session only. Per semconv it must
# NOT carry gen_ai.operation.name (execute_tool) or gen_ai.tool.name — those are
# for tool calls, and listing executes no tool.
_MCP_LIST_ATTRS: dict[str, Callable[[MCPListToolsSpanData], AttrValue | None]] = {
MCP.METHOD_NAME: lambda d: d.method,
MCP.SESSION_ID: lambda d: d.session_id,
LiteLLM.CALL_ID: lambda d: d.identity.call_id or None,
}
_GUARDRAIL_ATTRS: dict[str, Callable[[GuardrailSpanData], AttrValue | None]] = {
LiteLLM.GUARDRAIL_NAME: lambda d: d.guardrail_name,
LiteLLM.GUARDRAIL_MODE: lambda d: d.mode,
@ -130,6 +140,8 @@ class GenAIMapper:
return self._llm_call(data)
case MCPToolCallSpanData():
return collect(self._MCP_ATTRS, data)
case MCPListToolsSpanData():
return collect(self._MCP_LIST_ATTRS, data)
case GuardrailSpanData():
return self._guardrail(data)
case ServiceSpanData():

38
litellm/integrations/otel/model/payloads.py
View file

@ -37,12 +37,14 @@ __all__ = [
"LLMCost",
"LLMRequestParams",
"LLMUsage",
"MCPListToolsSpanData",
"MCPToolCallSpanData",
"ProxyRequestSpanData",
"ServerInfo",
"ServiceSpanData",
"SpanError",
"ToolDefinition",
"is_mcp_list_tools",
"is_mcp_tool_call",
]
@ -415,6 +417,42 @@ def is_mcp_tool_call(payload: Mapping[str, object]) -> bool:
return bool(_mcp_tool_call_metadata(payload)) or (payload.get("call_type") == "call_mcp_tool")
@dataclass(frozen=True)
class MCPListToolsSpanData:
"""One MCP ``tools/list`` discovery call, parsed from a closed request's payload.
The proxy is an MCP *client* enumerating an upstream server's tools, so this is
a CLIENT span. It carries neither ``gen_ai.operation.name`` nor ``gen_ai.tool.name``:
the GenAI semconv sets ``execute_tool`` (and the tool name) only for tool *calls*,
and listing executes no tool.
"""
method: str
session_id: str | None
error: SpanError | None
identity: RequestIdentity
@classmethod
def from_standard_logging_payload(
cls, payload: StandardLoggingPayload, capture_content: bool = False
) -> MCPListToolsSpanData:
# The list-tools logging path does not thread an MCP session id into the
# payload (only the tool-call path stamps ``mcp_tool_call_metadata``), so
# there is none to read here; ``mcp.session.id`` is simply omitted.
return cls(
method=MCPMethod.TOOLS_LIST.value,
session_id=None,
error=_parse_error(payload),
identity=RequestContext.from_standard_logging_payload(payload).identity,
)
def is_mcp_list_tools(payload: Mapping[str, object]) -> bool:
"""Whether a closed request's payload is an MCP ``tools/list`` discovery call
rather than a tool call or an LLM call — true when the call type says so."""
return payload.get("call_type") == "list_mcp_tools"
# --- service event_metadata sanitization ------------------------------------ #
# Substrings (case-insensitive) of keys that must never reach a span: secrets,

36
litellm/integrations/otel/model/spans.py
View file

@ -18,6 +18,13 @@ before the LLM call even starts), so a guardrail is a sibling of the LLM call,
not a child of it. The emitter parents every span to the ambient OTel context
(the active server span), which matches this.
MCP spans (``MCP_TOOL_CALL``, ``MCP_LIST_TOOLS``) are intentionally NOT in this
tree. Per the OTel GenAI MCP semconv, MCP and the HTTP transport are independent
contexts, so an MCP span parents to the trace context the client propagated in
``params._meta`` (or starts its own root when none is propagated) and records the
``PROXY_REQUEST`` transport span as a span *link*, never a parent. The registry
encodes this as ``parent=None, links=PROXY_REQUEST``.
Not every service call becomes a span — :func:`span_role_for_service` decides:
- ``DB_CALL`` (CLIENT) — outbound datastores (redis, postgres,
@ -46,6 +53,7 @@ if TYPE_CHECKING:
from litellm.integrations.otel.model.payloads import (
GuardrailSpanData,
LLMCallSpanData,
MCPListToolsSpanData,
MCPToolCallSpanData,
ProxyRequestSpanData,
ServiceSpanData,
@ -56,6 +64,7 @@ class SpanRole(str, Enum):
PROXY_REQUEST = "proxy_request"
LLM_CALL = "llm_call"
MCP_TOOL_CALL = "mcp_tool_call"
MCP_LIST_TOOLS = "mcp_list_tools"
GUARDRAIL = "guardrail"
DB_CALL = "db_call"
SERVICE = "service"
@ -74,14 +83,24 @@ class SpanSpec:
role: SpanRole
kind: LiteLLMSpanKind
parent: SpanRole | None
links: SpanRole | None = None
SPAN_REGISTRY: dict[SpanRole, SpanSpec] = {
SpanRole.PROXY_REQUEST: SpanSpec(SpanRole.PROXY_REQUEST, LiteLLMSpanKind.SERVER, parent=None),
SpanRole.LLM_CALL: SpanSpec(SpanRole.LLM_CALL, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
# The proxy is an MCP client to the upstream server it dispatches the tool
# call to, so this is a CLIENT span, sibling of the LLM call under the request.
SpanRole.MCP_TOOL_CALL: SpanSpec(SpanRole.MCP_TOOL_CALL, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
# MCP and the HTTP transport are independent contexts (OTel GenAI MCP semconv),
# so an MCP span does not nest under the transport span. The proxy is an MCP
# client to the upstream server, so it's a CLIENT span; it parents to the trace
# context the client propagated in ``params._meta`` (or starts its own root when
# none is propagated) and records the PROXY_REQUEST transport span as a span
# *link*, never a parent — hence ``parent=None, links=PROXY_REQUEST``.
SpanRole.MCP_TOOL_CALL: SpanSpec(
SpanRole.MCP_TOOL_CALL, LiteLLMSpanKind.CLIENT, parent=None, links=SpanRole.PROXY_REQUEST
),
SpanRole.MCP_LIST_TOOLS: SpanSpec(
SpanRole.MCP_LIST_TOOLS, LiteLLMSpanKind.CLIENT, parent=None, links=SpanRole.PROXY_REQUEST
),
SpanRole.GUARDRAIL: SpanSpec(SpanRole.GUARDRAIL, LiteLLMSpanKind.INTERNAL, parent=SpanRole.PROXY_REQUEST),
SpanRole.DB_CALL: SpanSpec(SpanRole.DB_CALL, LiteLLMSpanKind.CLIENT, parent=SpanRole.PROXY_REQUEST),
SpanRole.SERVICE: SpanSpec(SpanRole.SERVICE, LiteLLMSpanKind.INTERNAL, parent=SpanRole.PROXY_REQUEST),
@ -163,6 +182,12 @@ def mcp_tool_call_span_name(data: "MCPToolCallSpanData") -> str:
return f"{data.method} {data.tool_name}".strip()
def mcp_list_tools_span_name(data: "MCPListToolsSpanData") -> str:
"""``"{mcp.method.name}"`` i.e. ``"tools/list"`` — no low-cardinality target, so
the method name alone names the span (MCP semconv)."""
return data.method
def proxy_request_span_name(data: "ProxyRequestSpanData") -> str:
"""``"{method} {route}"`` (HTTP semconv)."""
return f"{data.http_method} {data.route}".strip()
@ -179,7 +204,8 @@ def service_span_name(data: "ServiceSpanData") -> str:
def root_roles() -> list[SpanRole]:
"""Roles that start a new trace (no in-process parent)."""
"""Roles with no in-process parent. They start a new trace unless they adopt a
remote parent (e.g. an MCP span joining the client's propagated context)."""
return [role for role, spec in SPAN_REGISTRY.items() if spec.parent is None]
@ -196,6 +222,8 @@ def validate_registry(
raise ValueError(f"SPAN_REGISTRY[{role}] has mismatched role {spec.role}")
if spec.parent is not None and spec.parent not in reg:
raise ValueError(f"span role {role} declares unknown parent {spec.parent}")
if spec.links is not None and spec.links not in reg:
raise ValueError(f"span role {role} declares unknown link target {spec.links}")
missing = [role for role in SpanRole if role not in reg]
if missing:
raise ValueError(f"SPAN_REGISTRY is missing roles: {missing}")

61
litellm/integrations/otel/plumbing/context.py
View file

@ -1,11 +1,11 @@
"""Trace-context + Baggage helpers."""
from contextvars import ContextVar
from contextvars import ContextVar, Token
from typing import Mapping
from opentelemetry import baggage
from opentelemetry.context import Context, get_current
from opentelemetry.trace import Span, get_current_span, set_span_in_context
from opentelemetry.trace import Link, Span, get_current_span, set_span_in_context
from opentelemetry.trace.propagation.tracecontext import (
TraceContextTextMapPropagator,
)
@ -47,6 +47,31 @@ def request_root_span() -> "Span | None":
return span if is_recordable_span(span) else None
# The W3C trace-context carrier (``traceparent``/``tracestate``/``baggage``) the
# MCP client propagated in the current request's ``params._meta``. The MCP gateway
# sets it per message so the MCP span can parent to the client's span rather than
# to the transport. A ``ContextVar`` because, like the root-span anchor, it must
# ride the request task and be readable by the inline success-logging callback.
_mcp_message_trace_carrier: "ContextVar[Mapping[str, str] | None]" = ContextVar(
"litellm_otel_mcp_message_trace_carrier", default=None
)
def set_mcp_message_trace_carrier(
carrier: "Mapping[str, str] | None",
) -> "Token[Mapping[str, str] | None]":
"""Stash the current MCP message's propagated trace-context carrier.
Returns the reset token; the caller must reset it once the message is handled
so the carrier never leaks to the next message on the same session task.
"""
return _mcp_message_trace_carrier.set(carrier)
def reset_mcp_message_trace_carrier(token: "Token[Mapping[str, str] | None]") -> None:
_mcp_message_trace_carrier.reset(token)
def set_request_baggage(values: Mapping[str, str], context: Context | None = None) -> Context:
"""Return a context with ``values`` written into Baggage."""
ctx = context
@ -104,6 +129,38 @@ def resolve_request_span_context() -> Context:
return get_current()
def resolve_mcp_span_context(
carrier: "Mapping[str, str] | None" = None,
) -> "tuple[Context, tuple[Link, ...]]":
"""Parent context + links for an MCP message span, per the OTel GenAI MCP semconv.
MCP and the underlying transport (HTTP) are independent lifecycles — one
streamable-HTTP session multiplexes many messages, so nesting the message span
under the HTTP/session span is wrong (it renders the message at the session's
start, skewed by however long the session has been open). Instead:
* parent to the trace context the client propagated in the request's
``params._meta`` (a *remote* parent), and
* record the transport/session span as a *link*, never the parent.
Only trace context (``traceparent``/``tracestate``) is extracted, never the
client's W3C Baggage: ``params._meta`` is caller-controlled, and the otel
baggage processor stamps allowlisted baggage keys (``litellm.team.id``,
``litellm.metadata.*``, ...) onto the span as attributes, so honoring remote
baggage would let a client spoof a span's identity attribution.
With no propagated context the returned context carries no span, so the span
starts its own root trace (still linked to the transport). The base context is
explicitly empty so an absent ``traceparent`` can never fall through to the
ambient (stale session) span.
"""
source = carrier if carrier is not None else _mcp_message_trace_carrier.get()
parent = _PROPAGATOR.extract(dict(source or {}), context=Context())
transport = request_root_span()
links = (Link(transport.get_span_context()),) if transport is not None else ()
return parent, links
def is_recordable_span(obj: object) -> bool:
"""True if ``obj`` is a live span with a valid context (safe to parent under)."""
if not isinstance(obj, Span):

34
litellm/integrations/otel/runtime.py
View file

@ -8,7 +8,23 @@ identity unconditionally.
"""
from contextlib import contextmanager
from typing import Any, Iterator
from functools import cache
from typing import Any, Callable, Iterator, Optional
@cache
def _otel_runtime() -> "Optional[tuple[Callable[[str], Any], Callable[..., None]]]":
"""Resolve the SDK-backed hooks once and cache the outcome, absence included.
CPython never caches a failed import, so without this memoization every call
site re-attempts the import on each request; when the OTel SDK is not installed
that re-scans ``sys.path`` and contends on the import lock on the hot path.
"""
try:
from litellm.integrations.otel import logger
except Exception:
return None
return (logger.phase_span, logger.seed_request_identity)
@contextmanager
@ -18,21 +34,17 @@ def phase_span(name: str) -> "Iterator[Any]":
Yields ``None`` (a plain no-op) when the OTel SDK is unavailable or V2 is not
the active logger.
"""
try:
from litellm.integrations.otel.logger import phase_span as _phase_span
except Exception:
runtime = _otel_runtime()
if runtime is None:
yield None
return
with _phase_span(name) as span:
with runtime[0](name) as span:
yield span
def seed_request_identity(user_api_key_dict: Any, model: Any = None) -> None:
"""Seed request-identity Baggage at the auth boundary (no-op without V2)."""
try:
from litellm.integrations.otel.logger import (
seed_request_identity as _seed_request_identity,
)
except Exception:
runtime = _otel_runtime()
if runtime is None:
return
_seed_request_identity(user_api_key_dict, model=model)
runtime[1](user_api_key_dict, model=model)

153
litellm/integrations/prometheus.py
View file

@ -49,12 +49,16 @@ from litellm.proxy._types import (
from litellm.repositories.organization_repository import OrganizationRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.integrations.prometheus import *
from litellm.types.integrations.prometheus import (
_sanitize_prometheus_label_name,
_sanitize_prometheus_label_value,
)
from litellm.types.utils import StandardLoggingPayload
from litellm.types.utils import (
StandardLoggingGuardrailInformation,
StandardLoggingPayload,
)
if TYPE_CHECKING:
from apscheduler.schedulers.asyncio import AsyncIOScheduler
@ -65,6 +69,8 @@ else:
class PrometheusLogger(CustomLogger):
# Class variables or attributes
_ADDITIVE_GUARDRAIL_MODES = frozenset((GuardrailEventHooks.pre_call.value, GuardrailEventHooks.post_call.value))
@staticmethod
def get_instance() -> Optional["PrometheusLogger"]:
"""Find the PrometheusLogger instance from litellm.callbacks, if registered."""
@ -343,6 +349,14 @@ class PrometheusLogger(CustomLogger):
buckets=self.latency_buckets,
)
self.litellm_overhead_with_guardrails_latency_metric = self._histogram_factory(
"litellm_overhead_with_guardrails_latency_metric",
"Total internal latency (seconds) added by LiteLLM, including "
"pre/post-call guardrails (excludes the LLM API call)",
labelnames=self.get_labels_for_metric("litellm_overhead_with_guardrails_latency_metric"),
buckets=self.latency_buckets,
)
# Request queue time metric
self.litellm_request_queue_time_metric = self._histogram_factory(
"litellm_request_queue_time_seconds",
@ -579,6 +593,21 @@ class PrometheusLogger(CustomLogger):
labelnames=[],
)
########################################
# MCP Tool Call Metrics
########################################
self.litellm_mcp_tool_calls_total = self._counter_factory(
name="litellm_mcp_tool_calls_total",
documentation="Total MCP tool calls, segmented by tool and server name",
labelnames=self.get_labels_for_metric("litellm_mcp_tool_calls_total"),
)
self.litellm_mcp_tool_call_spend_metric = self._counter_factory(
name="litellm_mcp_tool_call_spend_metric",
documentation="Total spend on MCP tool calls, segmented by tool and server name",
labelnames=self.get_labels_for_metric("litellm_mcp_tool_call_spend_metric"),
)
except Exception as e:
print_verbose(f"Got exception on init prometheus client {str(e)}")
raise e
@ -1001,6 +1030,67 @@ class PrometheusLogger(CustomLogger):
self._cached_metric_labels[metric_name] = filtered_labels
return filtered_labels
@staticmethod
def _guardrail_is_additive(info: StandardLoggingGuardrailInformation) -> bool:
mode = info.get("guardrail_mode")
modes = mode if isinstance(mode, list) else [mode]
mode_values = frozenset(
m.value if isinstance(m, GuardrailEventHooks) else m for m in modes if isinstance(m, str)
)
return bool(mode_values) and mode_values <= PrometheusLogger._ADDITIVE_GUARDRAIL_MODES
@staticmethod
def _get_guardrail_overhead_seconds(
standard_logging_payload: StandardLoggingPayload,
) -> float:
"""Seconds of additive guardrail time (pre/post-call only) on the payload.
during_call guardrails run concurrently with the LLM call, so their
wall-clock overlaps the provider call and is not additive overhead;
logging_only and MCP modes never block the user-facing response. A
guardrail counts only when every mode it carries is pre/post-call, so a
mixed list such as ["pre_call", "during_call"] is excluded.
guardrail_information is typed as a list, but some guardrails assign a
single dict directly, so normalize that shape to a one-item list.
"""
guardrail_information = standard_logging_payload.get("guardrail_information")
entries: list[StandardLoggingGuardrailInformation] = (
[cast("StandardLoggingGuardrailInformation", guardrail_information)]
if isinstance(guardrail_information, dict)
else guardrail_information or []
)
return sum(
(float(info.get("duration") or 0.0) for info in entries if PrometheusLogger._guardrail_is_additive(info)),
0.0,
)
def _set_overhead_with_guardrails_metric(
self,
standard_logging_payload: StandardLoggingPayload,
enum_values: UserAPIKeyLabelValues,
label_context: Optional[PrometheusLabelFactoryContext] = None,
) -> None:
"""Record litellm_overhead_with_guardrails_latency_metric (seconds): SDK overhead +
pre/post-call guardrail time. Recorded outside the SDK-overhead gate so
guardrail-only overhead is still captured when litellm_overhead_time_ms
is 0 or absent.
"""
litellm_overhead_time_ms = standard_logging_payload["hidden_params"].get("litellm_overhead_time_ms")
guardrail_overhead_seconds = self._get_guardrail_overhead_seconds(standard_logging_payload)
if litellm_overhead_time_ms is None and guardrail_overhead_seconds <= 0:
return
labels = prometheus_label_factory(
supported_enum_labels=self.get_labels_for_metric(
metric_name="litellm_overhead_with_guardrails_latency_metric"
),
enum_values=enum_values,
label_context=label_context,
)
self.litellm_overhead_with_guardrails_latency_metric.labels(**labels).observe(
((litellm_overhead_time_ms or 0.0) / 1000) + guardrail_overhead_seconds
)
def _track_end_user_metric_series(
self,
metric: Any,
@ -1225,6 +1315,13 @@ class PrometheusLogger(CustomLogger):
label_context=label_context,
)
# MCP tool call metrics
self._increment_mcp_tool_call_metrics(
standard_logging_payload=standard_logging_payload,
enum_values=enum_values,
response_cost=response_cost,
)
# increment litellm_proxy_total_requests_metric for all successful requests
# (both streaming and non-streaming) in this single location to prevent
# double-counting that occurs when async_post_call_success_hook also increments
@ -1446,6 +1543,49 @@ class PrometheusLogger(CustomLogger):
amount=float(provider_cache_creation_tokens),
)
def _increment_mcp_tool_call_metrics(
self,
standard_logging_payload: StandardLoggingPayload,
enum_values: UserAPIKeyLabelValues,
response_cost: float,
) -> None:
metadata = standard_logging_payload.get("metadata")
if not isinstance(metadata, dict):
return
mcp_meta = metadata.get("mcp_tool_call_metadata")
if not isinstance(mcp_meta, dict):
return
mcp_enum_values = UserAPIKeyLabelValues(
mcp_tool_name=mcp_meta.get("name"),
mcp_server_name=mcp_meta.get("mcp_server_name"),
hashed_api_key=enum_values.hashed_api_key,
api_key_alias=enum_values.api_key_alias,
team=enum_values.team,
team_alias=enum_values.team_alias,
user=enum_values.user,
end_user=enum_values.end_user,
)
mcp_label_context = PrometheusLabelFactoryContext(mcp_enum_values)
PrometheusLogger._inc_labeled_counter(
self,
self.litellm_mcp_tool_calls_total,
"litellm_mcp_tool_calls_total",
mcp_enum_values,
label_context=mcp_label_context,
)
if response_cost > 0:
PrometheusLogger._inc_labeled_counter(
self,
self.litellm_mcp_tool_call_spend_metric,
"litellm_mcp_tool_call_spend_metric",
mcp_enum_values,
label_context=mcp_label_context,
amount=response_cost,
)
async def _increment_remaining_budget_metrics(
self,
user_api_team: Optional[str],
@ -2346,6 +2486,12 @@ class PrometheusLogger(CustomLogger):
litellm_overhead_time_ms / 1000
) # set as seconds
self._set_overhead_with_guardrails_metric(
standard_logging_payload=standard_logging_payload,
enum_values=enum_values,
label_context=label_context,
)
if remaining_requests:
"""
"model_group",
@ -3723,6 +3869,10 @@ def _get_combined_custom_metadata_from_standard_logging_payload(
) -> Dict[str, Any]:
"""
Combine the metadata sources that can supply custom Prometheus labels.
Includes top-level scalar fields from the standard logging metadata (e.g.
user_api_key_project_alias, user_api_key_team_alias) so they are accessible
via custom_prometheus_metadata_labels configuration.
"""
if not isinstance(standard_logging_payload, dict):
return {}
@ -3736,6 +3886,7 @@ def _get_combined_custom_metadata_from_standard_logging_payload(
spend_logs_metadata = standard_logging_metadata.get("spend_logs_metadata")
return {
**{k: v for k, v in standard_logging_metadata.items() if not isinstance(v, dict)},
**(requester_metadata if isinstance(requester_metadata, dict) else {}),
**(user_api_key_auth_metadata if isinstance(user_api_key_auth_metadata, dict) else {}),
**(spend_logs_metadata if isinstance(spend_logs_metadata, dict) else {}),

52
litellm/integrations/websearch_interception/handler.py
View file

@ -31,6 +31,7 @@ from litellm.types.integrations.websearch_interception import (
WebSearchInterceptionConfig,
)
from litellm.types.integrations.custom_logger import (
CHAT_COMPLETION_AGENTIC_SURFACE,
AgenticLoopPlan,
AgenticLoopRequestPatch,
)
@ -119,21 +120,26 @@ class WebSearchInterceptionLogger(CustomLogger):
if self.enabled_providers is not None and provider_str not in self.enabled_providers:
return None
# Only short-circuit for providers without native Anthropic Messages
# support. Providers that have a BaseAnthropicMessagesConfig (bedrock,
# vertex_ai, azure_ai, anthropic) already use the agentic loop, which
# includes a follow-up LLM call to synthesize the answer from search
# results. Short-circuiting those would skip that synthesis step and
# return raw search text — a regression for existing users.
# Only short-circuit for providers whose Anthropic Messages agentic loop
# does not run web_search itself. Providers that have a
# BaseAnthropicMessagesConfig which handles web search natively (bedrock,
# vertex_ai, azure_ai, anthropic) already perform the search plus a
# follow-up LLM synthesis step; short-circuiting those would skip that
# synthesis and return raw search text — a regression for existing users.
#
# github_copilot has a BaseAnthropicMessagesConfig (added for thinking
# passthrough) but does not handle web_search natively, so its config
# returns handles_web_search_natively() == False and we still short-circuit
# web-search-only requests against it.
try:
provider_enum = LlmProviders(provider_str)
anthropic_config = ProviderConfigManager.get_provider_anthropic_messages_config(
model=model, provider=provider_enum
)
if anthropic_config is not None:
if anthropic_config is not None and anthropic_config.handles_web_search_natively():
verbose_logger.debug(
f"WebSearchInterception: Skipping short-circuit for {provider_str} "
"(provider has native Anthropic Messages support, using agentic loop)"
"(provider handles web search natively via the agentic loop)"
)
return None
except (ValueError, Exception):
@ -440,12 +446,16 @@ class WebSearchInterceptionLogger(CustomLogger):
custom_llm_provider: str,
kwargs: Dict,
) -> Tuple[bool, Dict]:
"""
Check if WebSearch tool interception is needed for Anthropic Messages API.
This is the legacy method for Anthropic-style responses.
For chat completions, use async_should_run_chat_completion_agentic_loop instead.
"""
if kwargs.get("_agentic_loop_api_surface") == CHAT_COMPLETION_AGENTIC_SURFACE:
return await self.async_should_run_chat_completion_agentic_loop(
response=response,
model=model,
messages=messages,
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
)
verbose_logger.debug(f"WebSearchInterception: Hook called! provider={custom_llm_provider}, stream={stream}")
verbose_logger.debug(f"WebSearchInterception: Response type: {type(response)}")
@ -629,6 +639,18 @@ class WebSearchInterceptionLogger(CustomLogger):
stream: bool,
kwargs: Dict,
) -> AgenticLoopPlan:
if kwargs.get("_agentic_loop_api_surface") == CHAT_COMPLETION_AGENTIC_SURFACE:
return await self.async_build_chat_completion_agentic_loop_plan(
tools=tools,
model=model,
messages=messages,
response=response,
optional_params=anthropic_messages_optional_request_params,
logging_obj=logging_obj,
stream=stream,
kwargs=kwargs,
)
tool_calls = tools["tool_calls"]
thinking_blocks = tools.get("thinking_blocks", [])
request_patch, structured_results = await self._build_anthropic_request_patch(
@ -1088,6 +1110,7 @@ class WebSearchInterceptionLogger(CustomLogger):
raise ValueError("WebSearchInterception: missing follow-up messages")
params = dict(optional_params)
params.update(request_patch.optional_params)
params.pop("tool_choice", None)
return await litellm.acompletion(
model=request_patch.model or model,
messages=request_patch.messages,
@ -1203,6 +1226,7 @@ class WebSearchInterceptionLogger(CustomLogger):
if k
not in {
"tools",
"tool_choice",
"extra_body",
"model_alias_map",
"stream_response",

18
litellm/litellm_core_utils/chat_completion_agentic_loop.py
View file

@ -137,8 +137,8 @@ async def _execute_chat_completion_agentic_plan(
optional_params_for_followup = {**optional_params, **patch.optional_params}
if patch.tools is not None:
optional_params_for_followup["tools"] = patch.tools
if "tool_choice" not in patch.optional_params:
optional_params_for_followup.pop("tool_choice", None)
if "tool_choice" not in patch.optional_params:
optional_params_for_followup.pop("tool_choice", None)
kwargs_for_followup = _filter_followup_kwargs(kwargs)
kwargs_for_followup.update(
@ -206,10 +206,11 @@ async def maybe_run_chat_completion_agentic_loop(
for callback in callbacks:
if not isinstance(callback, CustomLogger):
continue
if not _gate_overridden(callback):
continue
gate_kwargs = {
hook_kwargs = {
**kwargs,
"_agentic_loop_api_surface": CHAT_COMPLETION_AGENTIC_SURFACE,
"custom_llm_provider": custom_llm_provider,
@ -222,7 +223,7 @@ async def maybe_run_chat_completion_agentic_loop(
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
kwargs=gate_kwargs,
kwargs=hook_kwargs,
)
except Exception as e:
verbose_logger.exception(
@ -243,11 +244,6 @@ async def maybe_run_chat_completion_agentic_loop(
)
try:
plan_kwargs = {
**kwargs,
"_agentic_loop_api_surface": CHAT_COMPLETION_AGENTIC_SURFACE,
"custom_llm_provider": custom_llm_provider,
}
if not _build_plan_overridden(callback):
return await callback.async_run_agentic_loop(
tools=tool_calls,
@ -258,7 +254,7 @@ async def maybe_run_chat_completion_agentic_loop(
anthropic_messages_optional_request_params=optional_params,
logging_obj=logging_obj,
stream=stream,
kwargs=plan_kwargs,
kwargs=hook_kwargs,
)
plan = await callback.async_build_agentic_loop_plan(
@ -270,7 +266,7 @@ async def maybe_run_chat_completion_agentic_loop(
anthropic_messages_optional_request_params=optional_params,
logging_obj=logging_obj,
stream=stream,
kwargs=plan_kwargs,
kwargs=hook_kwargs,
)
if plan.response_override is not None:

2
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -449,6 +449,8 @@ def get_llm_provider(
# bytez models
elif model.startswith("bytez/"):
custom_llm_provider = "bytez"
elif model.startswith("gdc/"):
custom_llm_provider = "gdc"
elif model.startswith("lemonade/"):
custom_llm_provider = "lemonade"
elif model.startswith("heroku/"):

9
litellm/litellm_core_utils/litellm_logging.py
View file

@ -4725,7 +4725,7 @@ class StandardLoggingPayloadSetup:
api_base: Optional[str] = None,
) -> StandardLoggingModelInformation:
model_cost_name = _select_model_name_for_cost_calc(
model=None,
model=base_model if custom_pricing else None,
completion_response=init_response_obj, # type: ignore
base_model=base_model,
custom_pricing=custom_pricing,
@ -5271,6 +5271,11 @@ def get_standard_logging_object_payload(
## Get model cost information ##
base_model = _get_base_model_from_metadata(model_call_details=kwargs)
# The router overrides completion_response.model to the model-group alias before
# this payload is built, so cost-map lookup via that alias always misses.
# Fall back to the actual deployment model set by the router in metadata.
if base_model is None:
base_model = metadata.get("deployment")
custom_pricing = use_custom_pricing_for_model(litellm_params=litellm_params)
raw_response_cost = kwargs.get("response_cost")
response_cost: float = raw_response_cost or 0.0
@ -5393,7 +5398,7 @@ def get_standard_logging_object_payload(
def emit_standard_logging_payload(payload: StandardLoggingPayload):
if os.getenv("LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD"):
try:
print(json.dumps(payload, indent=4, default=str)) # noqa: T201
print(json.dumps(payload, indent=4, default=str), flush=True) # noqa: T201
except Exception as e: # noqa: BLE001
verbose_logger.exception("Error serializing standard logging payload for debug output: {}".format(str(e)))

154
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -6,7 +6,7 @@ import mimetypes
import re
import xml.etree.ElementTree as ET
from enum import Enum
from typing import Any, Dict, List, Optional, Set, Tuple, Union, cast, overload
from typing import Any, Dict, List, Optional, Set, Tuple, TypedDict, Union, cast, overload
from jinja2.sandbox import ImmutableSandboxedEnvironment
@ -5035,15 +5035,18 @@ def _bedrock_tools_pt(tools: List, model: Optional[str] = None) -> List[BedrockT
]
"""
from litellm.llms.bedrock.common_utils import (
get_bedrock_base_model,
bedrock_converse_supports_strict_tools,
normalize_json_schema_custom_types_to_object,
)
from litellm.litellm_core_utils.prompt_templates.common_utils import unpack_defs
_valid_json_schema_root_types = frozenset(("array", "boolean", "integer", "null", "number", "object", "string"))
# Only Claude on Bedrock honours strict tool schemas; other families
# (Nova, Llama, GPT-OSS) reject the strict field outright.
supports_strict_tools = bool(model and get_bedrock_base_model(model).startswith("anthropic"))
# (Nova, Llama, GPT-OSS) reject the strict field outright. Opus 4.7/4.8
# also reject `strict` on Bedrock Converse (see #31582) — their validator
# maps toolSpec to the native Anthropic tool shape, which has no strict
# field, even though Anthropic's native API accepts it as a top-level key.
supports_strict_tools = bool(model and bedrock_converse_supports_strict_tools(model))
tool_block_list: List[BedrockToolBlock] = []
for tool_idx, tool in enumerate(tools):
# Check if tool is already a BedrockToolBlock (e.g., systemTool for Nova grounding)
@ -5322,3 +5325,146 @@ def get_attribute_or_key(tool_or_function, attribute, default=None):
if hasattr(tool_or_function, attribute):
return getattr(tool_or_function, attribute)
return tool_or_function.get(attribute, default)
class NormalizedToolCall(TypedDict):
id: Optional[str]
name: Optional[str]
arguments: dict[str, Any]
def _parse_tool_call_arguments(raw: Any, tool_name: Optional[str], context: str) -> dict[str, Any]:
# Anthropic's tool_use blocks already carry a parsed dict in "input";
# chat completions and the Responses API carry a JSON string that may be
# truncated by the model, so route those through the repair-aware parser.
if isinstance(raw, dict):
return raw
if not isinstance(raw, str):
return {}
from litellm.litellm_core_utils.prompt_templates.common_utils import (
parse_tool_call_arguments,
)
try:
parsed = parse_tool_call_arguments(raw, tool_name=tool_name, context=context)
except ValueError as e:
verbose_logger.warning("Failed to parse tool call arguments: %s", e)
return {}
return parsed if isinstance(parsed, dict) else {}
def _tool_calls_from_chat_completion_response(response: Any) -> list[NormalizedToolCall]:
choices = get_attribute_or_key(response, "choices", None)
if not (isinstance(choices, list) and choices):
return []
message = get_attribute_or_key(choices[0], "message", None)
tool_calls = get_attribute_or_key(message, "tool_calls", None) if message else None
if not isinstance(tool_calls, list):
return []
result: list[NormalizedToolCall] = []
for tc in tool_calls:
fn = get_attribute_or_key(tc, "function", None)
if fn is None:
continue
name = get_attribute_or_key(fn, "name")
result.append(
NormalizedToolCall(
id=get_attribute_or_key(tc, "id"),
name=name,
arguments=_parse_tool_call_arguments(
get_attribute_or_key(fn, "arguments", "{}"),
tool_name=name,
context="chat completions",
),
)
)
return result
def _tool_calls_from_responses_api_response(response: Any) -> list[NormalizedToolCall]:
output = get_attribute_or_key(response, "output", None)
if not isinstance(output, list):
return []
result: list[NormalizedToolCall] = []
for item in output:
if get_attribute_or_key(item, "type") != "function_call":
continue
name = get_attribute_or_key(item, "name")
result.append(
NormalizedToolCall(
id=get_attribute_or_key(item, "call_id") or get_attribute_or_key(item, "id"),
name=name,
arguments=_parse_tool_call_arguments(
get_attribute_or_key(item, "arguments", "{}"),
tool_name=name,
context="responses API",
),
)
)
return result
def _tool_calls_from_anthropic_messages_response(response: Any) -> list[NormalizedToolCall]:
content = get_attribute_or_key(response, "content", None)
if not isinstance(content, list):
return []
result: list[NormalizedToolCall] = []
for block in content:
if get_attribute_or_key(block, "type") != "tool_use":
continue
raw_input = get_attribute_or_key(block, "input", {})
result.append(
NormalizedToolCall(
id=get_attribute_or_key(block, "id"),
name=get_attribute_or_key(block, "name"),
arguments=raw_input if isinstance(raw_input, dict) else {},
)
)
return result
def get_tool_calls_from_response(response: Any) -> list[NormalizedToolCall]:
"""
Extract tool/function calls from a response object into a normalized
``{"id", "name", "arguments"}`` shape, regardless of which API surface
produced it: chat completions (``choices[].message.tool_calls``),
the Responses API (``output`` items of type ``function_call``), or the
Anthropic Messages API (``content`` blocks of type ``tool_use``).
Callers that only care about a specific tool should filter the result by
``name`` themselves -- this returns every tool call found.
"""
for extractor in (
_tool_calls_from_chat_completion_response,
_tool_calls_from_responses_api_response,
_tool_calls_from_anthropic_messages_response,
):
tool_calls = extractor(response)
if tool_calls:
return tool_calls
return []
def has_tool_with_name(tools: Any, tool_name: str) -> bool:
"""
Check whether a tools list (as sent to an LLM) includes a tool with the
given name, regardless of shape: OpenAI-style function tools
(``{"type": "function", "function": {"name": ...}}``) or Anthropic's
native tool shape (a top-level ``"name"``, e.g.
``{"name": ..., "input_schema": ...}``). Anthropic's documented client
tool format doesn't require a ``"type"`` key at all -- ``"custom"`` is
only one of several possible values -- so any non-OpenAI-shaped tool is
matched on its top-level ``"name"``.
"""
if not isinstance(tools, list):
return False
for tool in tools:
if not isinstance(tool, dict):
continue
function = tool.get("function")
if tool.get("type") == "function" and isinstance(function, dict):
if function.get("name") == tool_name:
return True
elif tool.get("name") == tool_name:
return True
return False

6
litellm/litellm_core_utils/prompt_templates/image_handling.py
View file

@ -72,6 +72,9 @@ def _process_image_response(response: Response, url: str) -> str:
async def async_convert_url_to_base64(url: str) -> str:
if url.startswith("data:") and ";base64," in url:
return url
# If MAX_IMAGE_URL_DOWNLOAD_SIZE_MB is 0, block all image downloads
if MAX_IMAGE_URL_DOWNLOAD_SIZE_MB == 0:
raise litellm.ImageFetchError(
@ -95,6 +98,9 @@ async def async_convert_url_to_base64(url: str) -> str:
def convert_url_to_base64(url: str) -> str:
if url.startswith("data:") and ";base64," in url:
return url
# If MAX_IMAGE_URL_DOWNLOAD_SIZE_MB is 0, block all image downloads
if MAX_IMAGE_URL_DOWNLOAD_SIZE_MB == 0:
raise litellm.ImageFetchError(

7
litellm/litellm_core_utils/realtime_streaming.py
View file

@ -5,6 +5,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Protocol, Union, ca
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig
from litellm.types.llms.openai import (
OpenAIRealtimeEvents,
@ -315,8 +316,10 @@ class RealTimeStreaming:
self.logging_obj.model_call_details["realtime_tools"] = self.session_tools
self.logging_obj.model_call_details["realtime_tool_calls"] = self.tool_calls
## ASYNC LOGGING
# Create an event loop for the new thread
asyncio.create_task(self.logging_obj.async_success_handler(self.messages))
# Route through the bounded logging worker (per-coroutine timeout +
# concurrency cap) instead of a bare create_task, so a slow callback
# can't leave suspended tasks pinning each call's response in memory.
GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue(self.logging_obj.async_success_handler(self.messages))
## SYNC LOGGING
executor.submit(self.logging_obj.success_handler(self.messages))

43
litellm/litellm_core_utils/token_counter.py
View file

@ -407,6 +407,37 @@ def token_counter(
return num_tokens
def _count_function_call_tokens(
key: str,
value: Any,
message: Mapping[str, Any],
count_function: TokenCounterFunction,
) -> int:
"""
Count tokens contributed by an assistant message's tool/function call payload.
Handles both the modern `tool_calls` list and the legacy OpenAI
`function_call` dict. Only the `arguments` string is counted (matching the
existing tool_calls behavior); names are accounted for elsewhere via the
tool/function definitions and `tool_choice`.
"""
if key == "tool_calls":
if not isinstance(value, List):
raise ValueError(f"Unsupported type {type(value)} for key tool_calls in message {message}")
total = 0
for tool_call in value:
if "function" not in tool_call:
raise ValueError(f"Unsupported tool call {tool_call} must contain a function key")
function_arguments = tool_call["function"].get("arguments", "")
total += count_function(str(function_arguments))
return total
if key == "function_call":
if not isinstance(value, Mapping):
raise ValueError(f"Unsupported type {type(value)} for key function_call in message {message}")
return count_function(str(value.get("arguments", "")))
raise ValueError(f"Unexpected key {key!r}; expected 'tool_calls' or 'function_call'")
def _count_messages(
params: _MessageCountParams,
messages: List[AllMessageValues],
@ -430,16 +461,8 @@ def _count_messages(
for key, value in message.items():
if value is None:
pass
elif key == "tool_calls":
if isinstance(value, List):
for tool_call in value:
if "function" in tool_call:
function_arguments = tool_call["function"].get("arguments", [])
num_tokens += params.count_function(str(function_arguments))
else:
raise ValueError(f"Unsupported tool call {tool_call} must contain a function key")
else:
raise ValueError(f"Unsupported type {type(value)} for key tool_calls in message {message}")
elif key in ("tool_calls", "function_call"):
num_tokens += _count_function_call_tokens(key, value, message, params.count_function)
elif isinstance(value, str):
num_tokens += params.count_function(value)
if key == "name":

37
litellm/llms/anthropic/experimental_pass_through/messages/handler.py
View file

@ -61,6 +61,19 @@ def _should_route_to_responses_api(custom_llm_provider: Optional[str]) -> bool:
return custom_llm_provider in _RESPONSES_API_PROVIDERS
def _deployment_passes_through_anthropic_messages(model_info: object) -> bool:
"""Whether the deployment opted into forwarding /v1/messages untranslated.
The opt-in is ``model_info.supported_endpoints`` containing ``"/v1/messages"``,
declared per deployment in config.yaml and plumbed here as ``kwargs["model_info"]``
by the router.
"""
if not isinstance(model_info, dict):
return False
supported_endpoints = model_info.get("supported_endpoints")
return isinstance(supported_endpoints, (list, tuple)) and "/v1/messages" in supported_endpoints
####### ENVIRONMENT VARIABLES ###################
# Initialize any necessary instances or variables here
base_llm_http_handler = BaseLLMHTTPHandler()
@ -186,7 +199,7 @@ async def anthropic_messages(
metadata: Optional[Dict] = None,
stop_sequences: Optional[List[str]] = None,
stream: Optional[bool] = False,
system: Optional[str] = None,
system: Optional[Union[str, list]] = None,
temperature: Optional[float] = None,
thinking: Optional[Dict] = None,
tool_choice: Optional[Dict] = None,
@ -217,6 +230,12 @@ async def anthropic_messages(
# ids like ``functions.Bash:0`` that violate Anthropic's id pattern.
messages = sanitize_tool_use_ids_in_anthropic_messages(messages)
from litellm.integrations.anthropic_cache_control_hook import (
AnthropicCacheControlHook,
)
messages, system = AnthropicCacheControlHook.maybe_inject_cache_control(messages, system, kwargs)
original_stream = stream or kwargs.get("_websearch_interception_converted_stream", False)
# Execute pre-request hooks to allow CustomLoggers to modify request.
@ -362,7 +381,7 @@ def anthropic_messages_handler(
metadata: Optional[Dict] = None,
stop_sequences: Optional[List[str]] = None,
stream: Optional[bool] = False,
system: Optional[str] = None,
system: Optional[Union[str, list]] = None,
temperature: Optional[float] = None,
thinking: Optional[Dict] = None,
tool_choice: Optional[Dict] = None,
@ -399,6 +418,12 @@ def anthropic_messages_handler(
messages = strip_empty_text_blocks_from_anthropic_messages(messages)
messages = sanitize_tool_use_ids_in_anthropic_messages(messages)
from litellm.integrations.anthropic_cache_control_hook import (
AnthropicCacheControlHook,
)
messages, system = AnthropicCacheControlHook.maybe_inject_cache_control(messages, system, kwargs)
metadata = validate_anthropic_api_metadata(metadata)
local_vars = locals()
@ -456,6 +481,14 @@ def anthropic_messages_handler(
model=model,
provider=litellm.LlmProviders(custom_llm_provider),
)
if anthropic_messages_provider_config is None and _deployment_passes_through_anthropic_messages(
kwargs.get("model_info")
):
from litellm.llms.openai_like.messages.transformation import (
OpenAILikeAnthropicMessagesConfig,
)
anthropic_messages_provider_config = OpenAILikeAnthropicMessagesConfig()
if anthropic_messages_provider_config is None:
# Route to Responses API for OpenAI / Azure, chat/completions for everything else.
_shared_kwargs = dict(

24
litellm/llms/base_llm/anthropic_messages/transformation.py
View file

@ -103,6 +103,30 @@ class BaseAnthropicMessagesConfig(ABC):
"""
return headers, None
def should_filter_anthropic_beta_headers(self) -> bool:
"""
Whether ``anthropic-beta`` header values should be filtered down to the
ones the routed provider supports before the upstream request.
Cross-provider translation paths (bedrock, vertex_ai, ...) need this so
unsupported betas are dropped. Configs that forward natively to an
Anthropic-compatible endpoint return False to pass betas through verbatim.
"""
return True
def handles_web_search_natively(self) -> bool:
"""
Whether the upstream this config routes to executes ``web_search`` tools
itself as part of its Anthropic Messages agentic loop.
The web-search interception handler short-circuits web-search-only
requests (running the search itself and returning synthetic results) only
for providers that do NOT. Providers whose agentic loop already performs
the search plus a follow-up synthesis step (bedrock, vertex_ai, ...)
return True so those requests flow through untouched.
"""
return True
def get_async_streaming_response_iterator(
self,
model: str,

47
litellm/llms/bedrock/common_utils.py
View file

@ -4,9 +4,11 @@ from __future__ import annotations
Common utilities used across bedrock chat/embedding/image generation
"""
import contextlib
import functools
import json
import os
import re
from typing import (
TYPE_CHECKING,
Any,
@ -718,6 +720,51 @@ def is_claude_4_5_on_bedrock(model: str) -> bool:
return any(pattern in model_lower for pattern in claude_4_5_patterns)
_BEDROCK_MODEL_VERSION_SUFFIX_RE = re.compile(r"-v\d+(?::\d+)?$")
def bedrock_converse_supports_strict_tools(model: str) -> bool:
"""
Whether ``toolSpec.strict`` can be forwarded to Bedrock Converse for ``model``.
Non-Anthropic Bedrock families (Nova, Llama, GPT-OSS) reject the field
outright. Anthropic models forward it unless their entry in
``model_prices_and_context_window.json`` sets
``bedrock_converse_supports_strict_tools: false`` — Bedrock routes those
(Opus 4.7/4.8, see #31582) through a stricter validator that rejects the
``strict`` key on ``toolSpec`` even though Anthropic's native API accepts
it as a top-level tool field.
"""
base = get_bedrock_base_model(model)
if not base.startswith("anthropic"):
return False
flag = _get_bedrock_converse_strict_tools_flag(base)
return flag if flag is not None else True
def _get_bedrock_converse_strict_tools_flag(base_model: str) -> Optional[bool]:
candidates = dict.fromkeys((base_model, _BEDROCK_MODEL_VERSION_SUFFIX_RE.sub("", base_model)))
for candidate in candidates:
with contextlib.suppress(Exception):
model_info = get_cached_model_info()(
model=candidate,
custom_llm_provider="bedrock",
)
flag = model_info.get("bedrock_converse_supports_strict_tools")
if isinstance(flag, bool):
return flag
model_cost_key = model_info.get("key")
if isinstance(model_cost_key, str):
local_flag = (
_get_local_model_cost_map().get(model_cost_key, {}).get("bedrock_converse_supports_strict_tools")
)
if isinstance(local_flag, bool):
return local_flag
return None
def normalize_bedrock_opus_output_config_effort(model: str, output_config: Any) -> None:
"""
Normalize Anthropic ``output_config.effort`` values for Bedrock Opus ids.

37
litellm/llms/bedrock/realtime/handler.py
View file

@ -5,6 +5,7 @@ This uses aws_sdk_bedrock_runtime for bidirectional streaming with Nova Sonic.
"""
import asyncio
import contextlib
import json
from typing import Any, Optional
@ -156,12 +157,19 @@ class BedrockRealtime(BaseAWSLLM):
session_state: dict,
):
"""Forward messages from client WebSocket to Bedrock stream."""
try:
from aws_sdk_bedrock_runtime.models import (
BidirectionalInputPayloadPart,
InvokeModelWithBidirectionalStreamInputChunk,
)
from aws_sdk_bedrock_runtime.models import (
BidirectionalInputPayloadPart,
InvokeModelWithBidirectionalStreamInputChunk,
)
async def send_to_bedrock(bedrock_message: str) -> None:
event = InvokeModelWithBidirectionalStreamInputChunk(
value=BidirectionalInputPayloadPart(bytes_=bedrock_message.encode("utf-8"))
)
await bedrock_stream.input_stream.send(event)
verbose_proxy_logger.debug(f"Bedrock Realtime: Sent to Bedrock: {bedrock_message[:200]}")
try:
while True:
# Receive message from client
message = await client_ws.receive_text()
@ -176,19 +184,15 @@ class BedrockRealtime(BaseAWSLLM):
# Send transformed messages to Bedrock
for bedrock_message in transformed_messages:
event = InvokeModelWithBidirectionalStreamInputChunk(
value=BidirectionalInputPayloadPart(bytes_=bedrock_message.encode("utf-8"))
)
await bedrock_stream.input_stream.send(event)
verbose_proxy_logger.debug(f"Bedrock Realtime: Sent to Bedrock: {bedrock_message[:200]}")
await send_to_bedrock(bedrock_message)
except Exception as e:
verbose_proxy_logger.debug(f"Client to Bedrock forwarding ended: {e}", exc_info=True)
# Close the Bedrock stream input
try:
for close_message in transformation_config.session_close_messages():
with contextlib.suppress(Exception):
await send_to_bedrock(close_message)
with contextlib.suppress(Exception):
await bedrock_stream.input_stream.close()
except Exception:
pass
async def _forward_bedrock_to_client(
self,
@ -206,6 +210,10 @@ class BedrockRealtime(BaseAWSLLM):
output = await bedrock_stream.await_output()
result = await output[1].receive()
if result is None:
verbose_proxy_logger.debug("Bedrock Realtime: Bedrock stream ended")
break
if result.value and result.value.bytes_:
bedrock_response = result.value.bytes_.decode("utf-8")
verbose_proxy_logger.debug(f"Bedrock Realtime: Received from Bedrock: {bedrock_response[:200]}")
@ -252,6 +260,7 @@ class BedrockRealtime(BaseAWSLLM):
except Exception as e:
verbose_proxy_logger.debug(f"Bedrock to client forwarding ended: {e}", exc_info=True)
finally:
# Close the client WebSocket
try:
await client_ws.close()

146
litellm/llms/bedrock/realtime/transformation.py
View file

@ -4,14 +4,18 @@ This file contains the transformation logic for Bedrock Nova Sonic realtime API.
Transforms between OpenAI Realtime API format and Bedrock Nova Sonic format.
"""
import base64
import json
import uuid as uuid_lib
from typing import Any, List, Optional, Union
from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig
from litellm.llms.bedrock.realtime.trigger_audio import ready_trigger_pcm
from litellm.types.llms.openai import (
OpenAIRealtimeContentPartDone,
OpenAIRealtimeDoneEvent,
@ -35,6 +39,17 @@ from litellm.types.realtime import (
from litellm.utils import get_empty_usage
class BedrockContentEnd(BaseModel):
stopReason: Optional[str] = None
TRIGGER_AUDIO_SAMPLE_RATE_HERTZ = 16000
TRIGGER_AUDIO_BYTES_PER_SECOND = TRIGGER_AUDIO_SAMPLE_RATE_HERTZ * 2
TRIGGER_LEADING_SILENCE = bytes(TRIGGER_AUDIO_BYTES_PER_SECOND // 2)
TRIGGER_TRAILING_SILENCE = bytes(TRIGGER_AUDIO_BYTES_PER_SECOND * 3)
TRIGGER_AUDIO_CHUNK_SIZE = 1024
class BedrockRealtimeConfig(BaseRealtimeConfig):
"""Configuration for Bedrock Nova Sonic realtime transformations."""
@ -43,6 +58,8 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
self.prompt_name = str(uuid_lib.uuid4())
self.content_name = str(uuid_lib.uuid4())
self.audio_content_name = str(uuid_lib.uuid4())
self.prompt_started = False
self.client_audio_streamed = False
# Default configuration values
# Inference configuration
@ -247,6 +264,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
prompt_start = {"event": {"promptStart": prompt_start_config}}
messages.append(json.dumps(prompt_start))
self.prompt_started = True
# Send system prompt if provided
instructions = session_config.get("instructions")
@ -304,8 +322,22 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
verbose_logger.debug("Handling input_audio_buffer.append")
self.client_audio_streamed = True
messages: List[str] = []
if hasattr(self, "_audio_content_started") and self._audio_content_sample_rate != self.input_sample_rate_hertz:
mismatched_content_end = {
"event": {
"contentEnd": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
}
}
}
messages.append(json.dumps(mismatched_content_end))
delattr(self, "_audio_content_started")
self.audio_content_name = str(uuid_lib.uuid4())
# Check if we need to start audio content
if not hasattr(self, "_audio_content_started"):
audio_content_start = {
@ -329,6 +361,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
}
messages.append(json.dumps(audio_content_start))
self._audio_content_started = True
self._audio_content_sample_rate = self.input_sample_rate_hertz
# Send audio chunk
audio_data = json_message.get("audio", "")
@ -383,7 +416,6 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
verbose_logger.debug("Handling conversation.item.create")
messages: List[str] = []
item = json_message.get("item", {})
item_type = item.get("type")
@ -392,6 +424,8 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
if item_type == "function_call_output":
return self.transform_conversation_item_create_tool_result_event(json_message)
messages: list[str] = []
# Handle regular message
if item_type == "message":
content = item.get("content", [])
@ -443,6 +477,12 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
"""
Transform response.create event to Bedrock format.
Nova Sonic only starts generating after it detects user speech, so text-only
sessions never get a response on their own. Injecting a short spoken "ready"
utterance (followed by silence) makes the model respond to the pending
interactive text input. Sessions where the client streams its own audio rely
on Nova Sonic's built-in turn detection instead.
Args:
json_message: OpenAI response.create message
@ -450,8 +490,53 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
verbose_logger.debug("Handling response.create")
# Bedrock starts generating automatically, no explicit trigger needed
return []
if not self.prompt_started or self.client_audio_streamed:
return []
messages: list[str] = []
if not hasattr(self, "_audio_content_started"):
trigger_content_start = {
"event": {
"contentStart": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
"type": "AUDIO",
"interactive": True,
"role": "USER",
"audioInputConfiguration": {
"mediaType": self.input_media_type,
"sampleRateHertz": TRIGGER_AUDIO_SAMPLE_RATE_HERTZ,
"sampleSizeBits": self.input_sample_size_bits,
"channelCount": self.input_channel_count,
"audioType": self.input_audio_type,
"encoding": self.input_encoding,
},
}
}
}
messages.append(json.dumps(trigger_content_start))
self._audio_content_started = True
self._audio_content_sample_rate = TRIGGER_AUDIO_SAMPLE_RATE_HERTZ
messages.extend(self._response_trigger_audio_messages())
return messages
def _response_trigger_audio_messages(self) -> list[str]:
pcm = TRIGGER_LEADING_SILENCE + ready_trigger_pcm() + TRIGGER_TRAILING_SILENCE
return [
json.dumps(
{
"event": {
"audioInput": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
"content": base64.b64encode(pcm[offset : offset + TRIGGER_AUDIO_CHUNK_SIZE]).decode(),
}
}
}
)
for offset in range(0, len(pcm), TRIGGER_AUDIO_CHUNK_SIZE)
]
def transform_response_cancel_event(self, json_message: dict) -> List[str]:
"""
@ -467,6 +552,35 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
# Send interrupt signal if needed
return []
def session_close_messages(self) -> list[str]:
"""
Build the Bedrock events that gracefully close the session
(contentEnd for any open audio content, promptEnd, sessionEnd).
Returns:
List of Bedrock format messages (JSON strings)
"""
if not self.prompt_started:
return []
messages: list[str] = []
if hasattr(self, "_audio_content_started"):
audio_content_end = {
"event": {
"contentEnd": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
}
}
}
messages.append(json.dumps(audio_content_end))
delattr(self, "_audio_content_started")
messages.append(json.dumps({"event": {"promptEnd": {"promptName": self.prompt_name}}}))
messages.append(json.dumps({"event": {"sessionEnd": {}}}))
self.prompt_started = False
return messages
def transform_realtime_request(
self,
message: str,
@ -837,10 +951,11 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
Optional[ALL_DELTA_TYPES],
]:
"""
Transform Bedrock promptEnd event to OpenAI response.done.
Transform a Bedrock end-of-response event (promptEnd, completionEnd, or an
END_TURN contentEnd) to OpenAI response.done.
Args:
event: Bedrock promptEnd event
event: Bedrock event that ends the response
current_response_id: Current response ID
current_conversation_id: Current conversation ID
@ -848,7 +963,18 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
Tuple of (events, reset_output_item_id, reset_response_id, reset_delta_type)
"""
verbose_logger.debug("Handling promptEnd")
return self._response_done_events(current_response_id, current_conversation_id)
def _response_done_events(
self,
current_response_id: Optional[str],
current_conversation_id: Optional[str],
) -> tuple[
List[OpenAIRealtimeEvents],
Optional[str],
Optional[str],
Optional[ALL_DELTA_TYPES],
]:
if not current_response_id or not current_conversation_id:
return [], None, None, None
@ -1084,6 +1210,14 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
current_delta_chunks,
)
returned_messages.extend(events)
if BedrockContentEnd.model_validate(event["contentEnd"]).stopReason == "END_TURN":
(
done_events,
current_output_item_id,
current_response_id,
current_delta_type,
) = self._response_done_events(current_response_id, current_conversation_id)
returned_messages.extend(done_events)
elif "toolUse" in event:
events, tool_call_id, tool_name = self.transform_tool_use_event(
@ -1093,7 +1227,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
# Store tool call info for potential use
verbose_logger.debug(f"Tool use event: {tool_name} (ID: {tool_call_id})")
elif "promptEnd" in event:
elif "promptEnd" in event or "completionEnd" in event:
(
events,
current_output_item_id,

208
litellm/llms/bedrock/realtime/trigger_audio.py Normal file
View file

@ -0,0 +1,208 @@
"""
Pre-rendered spoken "ready" trigger audio (16kHz, 16-bit, mono PCM), generated with Amazon Polly.
Amazon Nova Sonic v1 only starts generating after it hears the user speak, so text-only realtime
sessions inject this short utterance to trigger a response (same approach as Pipecat's
AWSNovaSonicLLMService assistant-response trigger).
"""
import base64
import gzip
from functools import lru_cache
READY_TRIGGER_PCM_16KHZ_MONO_GZIP_B64 = (
"H4sIANGpRWoC/517dXQcR9Bnw+Duis3MzMwkc8xsxxTZMTMzM0PMmJhBjpmZmWKQZSaxFrQ80H0lJXf3vXf/nev17ExPQ3Xh"
"r2wPQv8/f/D/uMP/zxv8f64YkSyiWU1AIpCcRSqyICtQCApF4SgSRaHsKCfKjfKhgqgIKo5KobKoPKqEqqFaqC5qgBqjFqg1"
"ao86o+6oF+qDYtAgNBSNQKPQODQeTUZT0Uw0B81D89ECtAgtRcvRMrQCrQRahVajNWgtWo/+QBvQJrQRbQbagraibWg70Da4"
"2/rf7zbozxyxCcaug1krYZXFaCGai2ajGWgKmgi7jYFdh6ABqB/qjXqgLqgd8NUcNQUe66CaqDJwXQaVhDMUgNNkh7NZ4bSc"
"69zLXTyVJ/Kv/BN/z9/w5/w+0B1+g1/kZ/gpfoL/zQ/xA/wvvovv5Jv5Fmjr+Tq+lq/iS/kioPl8Fp+eRRP4KKBhfBDvw3vw"
"Lrw978jb8Fa8JW8B1JQ34PV5Q2iZ1wa8Ea/Da/Ja0Gpm9TTmzXgT3hzGt+a/wNz2vBPM78TbwtyW0NuIV+fVeCWgCrwyUHmg"
"yll9NWClOrwurNA0a42GMLYerFoN3pf7j8rwYkBFeWFehOcHKshz8dw8Ow/nUTyCh/IwaDYuc4mrPATIBs8WeJMNxmTnOWF0"
"Tp4DnsK4lYscc8405mNulsFc0FKYg9lZKjQX3KXDcxJck+E5haWxRPYd2k/2hX2G9gkonr1nX6H3C7RkGOmB1QjsGckLAK+1"
"QAqdeQwfD7Jdz3fzk/we/4f/5D4uoVyoGOizOdjbMLCw+WBBO1Esuo7uoZfoE/qJnCgARq7iUByGs+G8OGcWReAQbMUCDqBk"
"FI9eoGvoGFjVSrCdAWAr9VBhsHsHf8EP87m8F8gV88dsBxvFajKJvTS3mjFmOdNnXDMWG12NUoZPf6jv0afpnfRKejadaena"
"B+219kqL0xK0oBamV9E761P1I3qcLhhVjL7GEuOckWAUMHube02nWZ/NY7dYBFjJZR4JNnsRqbgr3o7f4WykNZlJDpEH5AfR"
"SRgtQMvTarQJbUZb0JZA7Wjr/1o7+gs8N6K1aRmal9qoSRLJC3KF7CHLyUTSl7QhtUkxkptkIyEklOQghUgF0oh0IINg/WVk"
"O9lLjsI+u8kfZC4ZQ/qQZqQ6jM9DchIbkQglfpyKP+Cb+DDejBfiEbgDrogx/gftQWNRbaTx23wBjwYbuMwmsfLsq7nZ7Ghi"
"84IxHs7r1A/rA/Wiery2UeukRWiPg4uD0UEeuBSYFWgQEAPP/dv9k/xd/K387fy9/DP9x/2p/mqB1QEjMCvIg/M1m75Tr2m8"
"NuaZ5dhrNoaHQnzIDpzUIzdJLXqS5hEmCVcFJpQXu4gjxelAk8XBYgexgiiLP4XLwmphkNBAyCkk0yt0Be1HK1JCX5JdZDSp"
"D2d7htfhLmAXX9FR0H5XVAPlQDp4/xN+BezsCHj6Xrge4cfA9y/wS/w6nPUxWN97/h2ihIObXISoWASiXwxEtDPIjRriTTiA"
"B5N3pD19RH8RbgilxWXiZ7GI1EUaLU2QxkojpcHSUGmGtEO6KX2UXkvbpYaSQ7wq/iXuEleKQ8XS4gdhsVBXEISjoNd3ZBzJ"
"S27hCbgMfgeRTUGHeXfuY6tZFfbQ7G+mGmOMJH2UTvRNWnZtfjA8eBjk+srfx+/x7fQ18/m8+739veW8yPvec8nzl2ejZzXQ"
"Ns9pz2dPiLeld4+3iO+pb6t/RWBr8IL2UY8wm7Pp/AC6huPIF/pOuC1ul4bKuZVDSnF1kfpUDbPUs/xq6WmpaZEtD9TJaqi6"
"QymlHJAj5KnSM7GYOF64SYvTtSSC7MB1cALE6WbIC/GyG8QaESJKAfDm2TyWx3EZInAryA9LQfZvwOor4c54PF4B8juGn+A0"
"HEWqgAf0JkPIeDKFTCOryA5yiaQSG42mI+hf9DvNLvQXrgg2sZ+4V/wilpH6SFulT1JReYJ8WU6QsynFlbxASPkiX5S3yNPk"
"0fJkebm8RO4ox0lNpbNia/Gj0BEsYxxYhZ1cJOdJE3IWM8h0vfgpFsVGmaeNaOOL3kK/ohXV2gfnBL74R/q/+rr57N6V3ibe"
"gGevp4fnm3uuu7T7bcaJjJkZTTMsGfddm11jXF1dfVyzXBdcpTOeZFx2f/HU8j33rww207Obz9l0VJqcoBXEQ1KyXEgtYgla"
"TlgL2ZbZ/rGl2GwheUKUkHhbN9t1a6S1iWWM2kepJxeTUoQLdD5pgWuiWnwWe282Ac3vNhZBNBllTDBWGDuNNwYzmpixZgTb"
"ysqBhJtDDOyL43E0nCw/HUPP0jTwlipCKaGQQIR4eotupcNoPcrJY3KP3CEmaUKnUZ22F24KMeItcbKUXf5DrqUsUxxKe3Wc"
"+lBNVQVLW8twy2kLs2S3NrY2tFqsryytLN/VieoLpaEyV/ZLG6VGUlDcKo4VO4uR4mOhs3CI/iQyyY3t/CmLNfcaF/TT2sXg"
"+0BUoLd/r4/4fvde80ieTu6/M1Jc0a5LzvzOBY4P9qH2bPaE9Pj0T+mivaZ9sv20XbfncNR3vHY8dHbPWOQp6v9VSzXX4wpi"
"HWWK5Zw1xtbd9t46yeq0PLHkt26y9rXlDukSEh1yz5ZgPWwppw6VN4qKsIL0wQPQJH6WtQH9Kmwge8em8ZFoNP6NtKDXAf5E"
"Co/AunLQzPiZi26gFYTzgiSGiYeEdHqKNMKneW0WahY2Ruo+7Yi2SzsBkT6P3lZfol/Tdb2+McLYblw2TNDEA/MhU5FANgnD"
"5BWWniEnw6ZHDIqcGZk7ck7EzfAe4ZXDK4WvCufhvSPORFyJGBnxPJyHpYW+DEm1mdYm1o2WoDpCjVOqKmvlEvJtaaqUXWom"
"NhLiSS5cmzczF+rZtBzBsYEE/xV/G/8nXz/feu87j+zp5l6TkTfjlquFq4DrunOUs6bzo2OsQ3Vst3ezlwdJNrBXg2uUXU8v"
"YRccp5293WX85fSd/C4tIt9UW1jnW7mljWWk2k5NV1Msw21KaL6wa2F/hrUPXWOba/lFmShlE2/SPeQNfoZsaDS/xtYyCRAS"
"Q71IN+qhbYRswl+0Ab1N/OQcfSa0l47Km5X3ynilh/xSjBIakivoL/6aDWcCm21+MmoYO/WC+m1tnrZUO6p91urq8/THeg8j"
"YHw28/Ih6DO+TW+KcfJE4KVw6O2wHeHFImIi9PDocCNsadiksPSwjLA3YTfCqoZ1Cz0fcsrmsi6ydrXOtn6z5rAxqwJetNBa"
"w/qPpYflndoRYtVpeYpUW/TRu2QktqKebJtxR7se8Phue85nNHGZjuoOh32MvYy9kP1D+iOgielqesv0kukF0wdCs6ZPT7+V"
"Xsf+1Z5gz+ko6Ryc8dHr19aiuaKpbrC1BaA02rpcfaNUV7tbGtkehFQJ9YSk2aKsK9SH8hWxibCFzEM9WG/Dqbm02UYCy0W+"
"CbOlmfJ4eZf0U7whrpAaKVUsH6wOW/mQt7bBtlfWn5azqqjYxao0G+pu1tK7a39px/U4Y7hZzaxvJGoDg7kCp/yJ/seBGE00"
"jplTeE9cmhYSp0mfpXjJK54RHgmzxDxyjCU8pFrYwvBLYUbIV2tXdYjslC5IyZJPPqUsVwOWYrZFIQmhxcECK4d+tcVa21kr"
"2mJDDobVi4iNiIooGUZCcljnqcWUSPmBGCGEkv68jtle7x787v/V53UnuD44CtuT0r+m77B3dMQ4tzvDnDmcC+3v01ukR6Tt"
"TU1NiUlpnKKkzk4bm97GUdn5wFHa2cQx3jnE0zJoQy7Rp26x9rP+Zsmh7lJKKHOVEeoPJUbpLn8SL4rVwffO0R0oYAwPCv6u"
"vqf+/noV/ogchlKFC7eFH+JB+YCabmliZdZctiW2DiH3Q1qFjg2Js2ZT44VpaKJRVOOBtYGMwG/BaUF74GwgIrg9uF7bodcF"
"9HZTb2M8Z+GkjlhfGaJ6lRVyZbEEnYC9fBZCdKT4WZYtsyDnzVRtymepqFQTOLog51Z3WIbZzoa8CG0YViwsLLRqyGXbUlux"
"0Blh9cIPRoRHvo14H/4+9KXtpkVTHssnpcLSafEvYSe1ksm8tnldEwN7vf3d91yHXYMztrtPex54FnoKu486v9u/p81Lq2Zf"
"51jhDHfddaxOb5w2K+Vz0vDk2sk1UtypQ1MrplZKr5ihBYeRnmopm2RrrtaX5otLxFNSYbWvdZ/NbZmjZJPzilNoE4xMR6CJ"
"d7a7jO+e9pYdJctETTwtxNB89K6QJEdaZ4WsCWsUPjl8emT5qKURR0K3qIwW5tX1ZsHHwdMQ15jWMVjMH+mb7X3oUwLvA9O1"
"3kYF3pRUFB/LLdVcahl5s5CPHAL0nIRvkXByGM1k7dkhXhDF4TkkkgSIRNcLprRV2WM5bb1pfWxZot5TvZarNlfIr6GvQ3rb"
"zlrXWivbqobst32wVrW+VAurMapb3Q8Svw6RppVYXahEz5JpaIy5WGvk/8tb1fsNUEwzb0fvBPdJl+o8YGfpetqxlE7JD5OV"
"ZHeyJd2bylJbptLEbgkzEub8nJrUPqUa2F2T9J0Zn/RE8aytctigkHVyfmrisUKI8kW9Z+mulpSGkXGsvDnUSAi6vH9nrHYd"
"zNjjL8C60fFSXmm+MFxoJc6QhlpWhA4Mt0SWiwyNWBs+MnS+db9yQtjO8xjN9ViwpSFGV+2Cv5D3c0ZOT/NAV8PkB/FG8oHO"
"EHaJQ+W2ynO5gtiQLIQKWoUKeTeqz18YX4I5AgW110YaTyd++kpoK42RXyjvLZutO61rbKlwHWfbZntqq2UbaTmtDJDPSivE"
"Z+Im+Z1yXm2reuWncmu5lrxIvqx8VzR5tXwEUO99uh3FsbV6seBNX3VvVY+asTVjbUaM+45nrPuOa7Kjpb1C6viU96lbU3+H"
"bLAqvaj9YdqQVG9S++SMpEpJ+ZOvJfZPupU8zLFBmyuWCskRut26Rub4LkvlbmGuckC9r1SUDpJu5qLACt8Zz09XlPOKc73n"
"czBonsPRtBO5C7V/FD0rb7A5w/dGXc/2W9THsDO2CEsZOU24Txriv9EP/oYdMO/q6wKPvfc8m7xlghfMmmSdWFdC0kBxn9hf"
"6iwtkBLFPmJNcZjgpnXJYtSLbdE2Bry+rv7DOkVf6BbpqhQQZ0tHpRRptdoSLOpL6PzQyyFNbYssJRVBuihECcWFC5Ku/KME"
"lFFKafmlRKSOUi55plLUkmLxWH9YTXWaPFdcCRmpBA81rgU/Brr48/sEr83TMyObe2LGtYyKGdQlgVfOs/9m/9W+1RnpLOTa"
"6yzuLOOw2KemD087n5or7XJqVNrxlI2OzcEmwjXLGltTWz7rXsmHR6D2Qg85XGmu9FduSHVJvLbYW9Q9OKOFu6N/oLGTj0WD"
"+RjeEkXjz2SfOF5Ntk0NC4scEtUj8n3IA6Wl+E4whfeCIrYQPuKnrKYR0P4MouDvwWjdyk+SseIieaKsgLzaiGWl3+UcgL/t"
"4kmxoxCPu3HNGKiN97cEVHAv8IFtIa3EpVJuqb+0Uv6hlLXeCu0R3i78aHjZ8KuhD6wB5Z3UX2wijhFVmaiXFEWeJM2VqskN"
"lGfqaUtXywBLJ8tjS0OrWx0sj4DYeAQ/Rk9YUb2Xv41HdI/wxPk3BqoHznkPuG46cju+2rO7Jrlfuld5Orh7uMY6UtNcKcVT"
"XqRMs/d2fAFLy5WamPwlMT4h2rmcpakTbLct9eQUWhg9QOWkebarIdstneUV4lK0UkvwXM8o5U7y7AycMP38lrkkuMbPgkvM"
"R/ik/MV6JvRC2OKw26HlQ2pZtsrvpMUKs3yw3Fbmih5eTTvvzevt6m+vjTFGQs1o5fXYXwyjFrgHnSp2lgeotS0n1HxKfekG"
"nYldPC8aiUrxDuYzbV1wj57MMvA3GifIYkF5gtJC/R3QbnnrGdvvIYNCRoSsty2zrJEjxDHCNGGcmENi4haxkVhQbCkmin+K"
"1aQSck8lQemozlOGSSWFFqQCNnkzNkDPHsjh++p54tnofe4p7NnnphkhzucO2RHpPORIcrRycecl++r0hmkd0i6nXUvLbk+2"
"b007n9wv0f+z18/eSdczTqCnwEV1eSjNiYuwYby1MMYSb8WW5/IF8Q46qKV6DrivePr79wSjjFxGpDbKN80bHqhp3ECquFZ5"
"py5XO6uJ6lJLH6jdHljKWT+ABMrL+ehD9lgrGiyiLTT2mR+Nq3p+QJUHg80NiU/A2+lyyCo7lA9ST7G4oJFtpAytJ8wX2pN1"
"bKNeWauqt2Nf8HlaR9SEUmJJKb/cTj6gRFjyWX22waHZQh/aBloqKWmARLLJY+Rlsk1OETeLzaTfpUZyqNxdKg4I5Kh0XEwX"
"2gr5hFPUiVvxFMMZXO2zeX96Xnp+95z0rPBU8sx1j3P5HGccVR0THCcc+ZzvHbvBzkTHqfS/0lhq7rRHqS9SHiUfT56WJCVO"
"TinnsbM4saFcU/ydKLgl7k5LyonKCbWepaKlitKcxurEl99d2X3Jt16/ZlbVW/o2ua+5a/h6Bp+bY8k46aO62/YjZKtttGW0"
"ck8pad0Y2jS8avhj2yC5Iz5ohhvT9C/aJL2u3kZfpGXTbgS1wOBAC20qy0PGCNehIh1J+9PG9D3dJBQUDNwY70EneF+eyErx"
"bmgk/5v7UVN8nRQUZovD5EbKW6WQpT6g7t8tFdWDUGVvV+up1dRR6gGlkdJPOSvHS3OkKOmq6Befi7PEOKjLiJhIV5BsuBeP"
"NQO6qB/S2gUf+Gv5Zd8nTzvPM3fpjDEZbd0lM+yueq5RjvaOk/Yj6eUcOR2b06um9Ujbl/otJZAyJmVHct/k8Ylm6iNfPL8m"
"vBaX0p1ordnabEZqWVjotfAjEQNDk6RkZg9sDEzQn7JCZCfpx5r4H7lyu2q7b/mi9EW8HDWB1gq9hWV0jlBL2WcLhPYM/xg+"
"OmyRdbtUmJbDyxGF+nMiyYcKGPmCBX2LQKtj/cP10eCPV9l9VpdNNZuzs+gQHSzekcbJ/SQH7Yr38VH8Mxol7FBmWpLU29IY"
"KuMfqBn9JtUCpDrBgi1vlbbyWOkPaay8Us1u7R1SPjTNWly9IX2Vsit31G7WrdZfLS3Ur1I5KVaYT0fSKjSO7MLvmUNvHGwW"
"OOO/4nvtb+Xv7D/rO+dxZSxxDnDUdPZ3+jOKeVhGWsZOxxzHLOcNRzVHFfu+tI72nvYSzvN2nLYh+U5i/bRX3oJsjbBOGMP3"
"aFv9/Q2H8FvInPDVYTVsWP1NmihZAOk9JUfJCuGm+EkYgmP1OP8dL/Jv1xawW7gj+Qc/RJfZIfYnukvbyolqYVunkCe2RdZK"
"lpxqbzla2gtoNsz6QE0QFdzD1PQocx27z/2skDlK9wQHBI8Ex2pD9InGZvMMX49aoVb8EPOa7dkPvpjWlG+rpS3V1WxyEbCT"
"dKmv+tLa1bbF2tkSq4yS8ggzSXVynJwRFkh15GxSSZqEuqC9uK2wTFovj5SvSjOlN2JRsZ+whBakaaQSbUkvkTr4BDtrtDRK"
"GgWMMMOr/2oc02toAwMj/O39NPA60Dp4NoACjfzcN8g/33/eX9Pfz7/LPzJQIdDN/8i71PvIq/na+6f6fb7FvrK+VpDB6wXL"
"6T2NXUayXkq36KqxyVzBf6ApuC9ug/fg2aQY7Uo70qm0mNBHHCqFyPelUCm/+EYYLo6Ucsqq3FzaJVYRK4klxThhidBYaCa0"
"FiYLc+HuJq1I15NhZBcJp9VobjqHfMF/4K34LN6HS+JDaACajY4CXv0FHectuMRdzMds/CNY4ho2iDVi9Vg1VpR9M/+B6h2z"
"7mw2W88WsXXsFrPwxfwHb4I+o214FqlOT9FywkqhipgkXpNmyW65PlS4awFreJXcaj91rjpWbawSNZtaDjDtVSW7ckPeJJ+Q"
"H8lz5ZbyFSlacon7xV/FIuJrYa1QWfhEe9PDgJhHYILvAzYcjt7zXHwy1MjXzbNmZ/O2Ud9I0LkeYvyp59evaCF6V/2HVl+7"
"FFS1odo+bZWWR3MGfcHcWh+oD19oH7Uw3dSi9G76Dv2+Lhn9jM9GTXOYudzMxY6z3jyBt0NnUGmcgJeTNyQDNL2KZhOmCjeF"
"/cJzIVVIFz5BZXUCetqAVOsKOj0K8hxKB9ESdCqpSsaTg2Qy0fGvuDy+jQlZgp+hTegaKoPD8E70htfgE3gldB81Rvl5GkM8"
"hl/ij/g4OFNP/op3Qh3Rc76fS+hPFIeeo1j0BMWji8BREfwPfoR34qW4J/6EB0HkOIcL4J7ER6uLTQERHRMKiUfF/NI06ZM0"
"UE6Rdfmn7JQny4XlgDRIVpQIpZscKp0SPdJguZvwinQVeymX5FhiomJ0txAmuPAEEi3cEeNJB3bfIKiqsFFYgzqbETw77sH7"
"m3HGbUZxU15AfxJYbj4jt9DcQE1Pu8AEQA+GVs/XxP3I89pX3dfWc8izxVfQ9ynjJkTWFs4Qd3bPNvfbjJ0ZroyJbuQu7C7p"
"669PRXdISTyC1TNv80pCbmWEJcaaCnk3TGksL5AXKw3UfOohiGOfxIOCHw9jHY1nxjeWD6Ww7sbrYOFAw0CJYIhe1jzDCnCF"
"tTE/mJv5e9yUWoQVgiFsFpk4TmwmCDQnrSnMEE+Jd6gPcLLf+M2MZTX4HLbTKA45aoB2XCuiT9ZT9KrGZOOm0d7MzZqxsUzh"
"VxChX+gGmkSO0q/SHMs72y3bQ8vfag+L23Y47GB4IIyF/hXSPWSibYxtrO2sLbstzXJWraO+VLyyJNcBlBgqD5BbSNuEUPqV"
"nKIvwVZa0f34CLrOh/BjLM50QcyYpY/TfIFFvl7e7p47npKemu5Ql+z43V4rfaq9rn2m470jxWHY+9gD6Rn2ac69rnoZ21yf"
"nE8dCY7Rjp7OEMc654yMyoFpbANgg799Yd5xwSn4jFLWFmodofwqrKbJQnO1ZsjQsCahf1vbK1ukW2J1MVY4KcwFO9kudMOD"
"zc6Qp8ELeAymGHGXPkILaJfN3XiesEDcKRSnPch0spdeEqNkopyXndJj0SL0IrPwc2yjOv1OP+EdbIveReun7zQT2F2WYTzT"
"OgVZ4FZwmE7NLhA5Nppe44KBWU/+FY0kPehPuoLaqIcspZWFdWKsfMyywXbEdkBdIbeVN1h+C8uIbBDVOqJ9SJp1sG1m2KjI"
"6KgGkRfCZodE2wxrrDVo/cP6wTrXGmrppNjhLKOEN0J7abTUWcT0JW/DnpqLWCLfxAexNvqXwFH/BH/dAA52CTz2j/HW8XQG"
"lLnD293bx1PD43V/gadmnuvuhxnRbu555Z3pu+BZ7lrveuh65on3fvTMz3DZ36UfcO7z3te7Glv09to37atxlX8lqXK0Jdq6"
"3XLOEms9YIsIuRA6I+x0+NDwAaHRNq5slffJR2Qmq1I+YQr6aAzQl+kB46d5wLip19RcgRrB6dpdoxl7xfbwKMRRbfwb/pPs"
"p5XFNGmjlCB2E24SCy1KI0FTC2lu3IntN44bb4x044QxXN+rXQxUDZQIJPsbBotqHfW7ekv9oD7COGpW4xfRX+QSXSh4BK9Q"
"XzwjSvI4Ja/6UPkgf5RqSM2lF1DlfJcaSYOkgdI0eZTUWxorFZEHKK3VDLWL5ara1nLU8tmaIyQyJNVW0NbaehckftzaDk7K"
"1dNKXXmrVFBKEqPEAO1FC5ICOMh/5cO4G07iN8eZd40YyIHVjdzGMWO1EW6YelP9qH5Hf6/XNsrqY7SZmk+rb9zTYoJ3/N99"
"Df2T/C8Dq7SPgak+X0bLjK/uEr6R/k3+Hd5ZGcmO2/aSzrYZZXzegKwtDxjeBb6+2p/sJCkF3nxTXCMa4i35rGVZSPbQ1qGX"
"bLr1hWWv2lTNrf6i3lZ+in+SQzzNzM5izR1mE7OF/jTo9yf6EwOdtK96DVMzC7IE8xY7juKJKnaUvksDZIu8Vm6jvFA2qyfU"
"HGpRpadUV4wWooU8wgTaj1RBSWyYed/IYVY0b5g5TMmw6o81ppc3j7E/+GP+FOJ1EFUnOWgd4bQ4X5okHYT4c0s8DDX1KDmn"
"PA+qyGNCUaGTUFvYQJNJKGSFEXgjqoVa8OmQOynqie6iUtiNp5Ph9BJUtyOl7vJ+ZaQ6Sa1guWIRrUnWN9bxttfW65aClli1"
"r+pRHsjT5ZJSM/EFrUQfkglkAN6B7vETPJ2/52/5CI54bV6GV0P90Xv0GvVG1RFFQV4V18VLcF4UZAvZUraJSXwcY6YXrOpc"
"cF+wmjZYuwvect230fvYG+W96W3stXuXeZO9g72jfdhnerODx2ieGH/LwIGA7o13L3ZJbp/ns39y8F6wTGCd76gvEDxi1kHh"
"uBtksHD8lZhCWfmJMkph8g95qDJFWaeUV+Yrk2SvtFE8S+/i+ugyyG8f97L3RiftRKBioFPwnvZOf6pP174G04O59QqmzNei"
"vWgMesGroI54EJ0LFv6LcJdmh5PXhXiiCDPpFvIJsmsRXodnwNkqsjNmS3Oi4dSHAzZYaNSGDH/Q/N2MMFeb81lfNA5bSDw+"
"iC/j4WQlHSLUESPF08JZ+gQQcZTAaFPhDBVpA9IJf4As3BNvwX6UyntwkS/lW/gD/pKf5PlQTkTQRJDtZfwDHyOlSVFSjvwk"
"u2gh+ZY63BqrtBRvCLklXW1giwm5Z30ia/QsLS19scyw7lW3i4/xKvQNmbSJ1EJsSbsjJ3sC2cxEGvbgDngef8Pqs3GsJMqJ"
"LbQVOYiK8aGsNHei4kSmS/BBHg8WWYZ15dPRH3yXWc7YqjXWFL2+cc7w6D+D4wO5/B/8CwNbg9eDawNvfMu8y71nfZ/8yYGQ"
"YJVAuH+dL+hL9r8OSNru4NZAG8Cgyb7K/iHglaavju+NZ61nmfeQ/0bgQ4AGogNnA5u0O0ZDbgHc4uQd0QXcX2gnXZRySylC"
"vBAQfpME+bX0u5hBF5I5+Djglt7oGy/MT5qTjIL6ccizTfUJxndjmPGrflOL0IdB/OsAqKY72sdX8UkgwdV0hDBAuEcdZB/J"
"DnKaKPygfahIOuI/0G20D61Cy3gO/sP8Zo40k81TZln203SZO808LIZl4624xsuiQ+gn6g54NgeZSl4Dzi1L29OF9A96gQ6m"
"PWkZWp6eIevJIjKQNCTpeAh+DzY1DC1Dh9F2NBhFoIN8B//ER6FkZMWv0Snwyc+oLt6N47EXNFyG5APMlRcyZRM8F88B9DUN"
"VUWf+Tn0lGymi2htfBAsoh3ZQI+LE0VVmItr4MWkgrhPOif+IswhGjbwG1oBcvAq6sFVcB2chseSAnQJ4fgFmouao79RNL6B"
"r+LO+BmvxPvBiS5Bj4pdfAnbY3KzIH/Bz4NdTDaHGSWMQ8ZaswfrYYYZT7VnWjc9ylhotDEW6wO1O5AZLmiadlcrpcnatuCc"
"YKvg8uCtYB1tp7Zfq6sVB0RcWOurubRhejP9kvY4GAjm1dZoKcHdwa+B9oE431jfLX/rYDDYJZgW6BS4F6ivXTR6A25dzXez"
"KrwsjhS+iYa0X7oIOL28RJQNUJ8fl03psLhfKCu8A83NJs/xO/SUt2A1TQm8DvKfqZjTjRCjqNHT/MEOg9ev5PNA3m9REO8E"
"NFBHWAzzFtDFQmnxunhPdAoXaCnamJ6morALPH0XyLcxmoI+ITvUAU4+lecDC1vMG6NnaD9YksZrQ3XQAktkA7lOupAK5DeC"
"6T+0uTBCoLBCY1qf7qC1AIlFCvkA2dwFNC2CHTzG1fFKpPNlwM02fpd7uJcfBPlrrBTE0rloCrlLCN2FV+MreAf9IVSVTHGU"
"aNI0ekEoJFWXR0itxLpCPL0D9b5T2C10oJOIieuQaWQumUU+YahV0W9oLWIoDb1EfVB+tIB35Im8NVqCBiEVjeBNAft/56Mh"
"lqSDN3xjn1kDfhsqkY38LcvDyrOS7G+WDFSUXTTzmQ3NoeYnM948aYabF4zZxjJjr+ExrhlrjMZGFcOlNzfyGtkhC/cGSRcy"
"DF3XX+g39WqGYUjmPWORscAoYqwyJhjbjQ+Aek9BpXkCaiMXRH9Zp8ZnvYeeDlXSDn2LKfEcqCiKBT4eo5NkidBa7CEmCYOE"
"CUJZMZdUVfoilhJLCoOpi1yCk4aSebgfeKmfLWBlWS3WkxVgDyDmTjefm9HMyabwhXw7j+V+QNeHcQhpCxIqC1X3JFIeasZH"
"oOupJCdxQUSeDh6ajNvh2agMYmDzSZDVXvO2/ALgjHdmPkZ5V96Mn4PqcxeL5PGQFRag0Wge1K+FoaJ9g29BDLgD9ZJK+pGl"
"pDv47Rt8AE/HM3B9XBG/RVsgOiXye4ATN4D91AWPSmSPGeEvWSrLz/vw8lCZKVCpreU/uZ1f5M/5bIjLR8FXn5G+OD9+hr/R"
"cDFB/CKMoflJFdpPWC01kz4Jg+hkUonE0F1CmnCbRpGf6ADEj1nATUWcG5Xgw1lr9hL2ymCbWV7GjarGF+Oc2Zc1YGvNw0Zl"
"o7MRajrNtuyC6TdqGaVAQwUg0iWZVcwWRub/kSpnjDBaGYn6bEBWrfWaemn9ofZKOwTe30JbrH3VtukrdFnvpaUHPwYfapHG"
"buO23l+bERS1QfpOI8FoYVzRNgUnB4fq1c3b5gTDDk8tgh20WP2AEWkc0ioHjwRqB84ECgdzabX0aVBHm8FhwU3BNvpHcw/4"
"wmIWYdYyv7OLWBDaCXPJSbCNENyZVhIThLx0HqrH/+RV8ESSg0xGvVl1UzZfQ4Um8pHsF7OYEdSnGtPMOnD2juwzVO5fzSfs"
"B98OmX0haohKodVoEG5F/oQa+ig2UDbcEGrwAmQcXgQV8Vq+HuJFgJ/mRfmvLNlMNXuy2+wHm8/c5iPTY66ASroxj+ClOGdN"
"wFIqo2gUwKWon06k+0k0vSdYAH80VprLJQBhF5WvKbvUi8ouaZyYU2wAttxYGiEuojvxLaizk7CJo/FRHsZGm6/MBawo38QO"
"md+N341NRjtTZArbbJrGTyPJKGHOMLeaG8w+ZoaxzmhvdDfyg34GgCY362+1G1qaVhF0lK7N1eppWCukjdUua7u07oCE/9Yu"
"aY+04lB7RulYb6iv1f36Ez1WZxDF8xtljUHGTqODEWF81csY0fB8yzhvtATNn9bP68eMp8ZRw69P1wfoY2BORV3TdmvttSda"
"DtD/St2mH9WG6E0MK1TFPVgFVpltZFd5PDqJd+D7+DTW8ABikmq0CN1EOpM+pAy5jffjXTgP3gY5qAh6D9k0yA6yADvPtrAB"
"rB3ryiaxdNYKUGVR3gAiZHPIyHlxBdwCb8Yy+YMECKI7yQVyk1Sko2lniml7cgnk2BM8OTeJxhPAj1uDd37OnIcW8t58HQ/y"
"pqgGikQPOEJ7kIBDwC/forKQM7dDbGgC8fsFROcA3g94oL+gizHSXvEBDRO4sEQ+qvRRo5RYQFfHxXrKR7WKmiAFhexCTaG/"
"9Ctg+TNCfTIMFyHT6Sc6nwZxKJoBEaQmPoXjkIx+5QIgiWdZ/z9tOrcD6nrKPjAXa80L8sesGgsCpkgzK7LZLJrNM+1GqpHd"
"3GdeNqeapc32Rro+3GgMz83NP4yuoPPFRnmwv1um1exnLDGY8c58b640J5oPjUuGag4yR5uFoAYZZa6A3wwjxGxh7jYlFsZe"
"mH+aE0wCWPU31oodBQ4UvoddAzk35GtAxicgwiWznFAfVASMUZUfYe1ZDXYKeJ3F3piNzApmN/O8mWJeMTuayGxjPgZLvccO"
"sBOsMh8Lc74AbmwNES832Uzy0xi6mt6nNkDN+4UzQpywWTgEGNovHBGWC4uEacIGYYXQWsgvvIEs5qTRQlXhDh0A6GkT/Uwj"
"hDDhHO1LR9A9lIKMswkfICcvAmT1lHrobcjZeagVcFs8OUQWkpLkOF6J/8Br8C84ATDUTPQraomaoCoomZ+BU+3kY/gQPpoP"
"4Pn5a+ZmYZBn4yD6XmYzWQfWh/0JVrcSbG4Q6wTeHcdk7mGxEOcNFs3nQBwewpvwipBHdJ4TneLt+Ax+E3Lpab6cl+aJIL/T"
"7B82mjnN2SCjbmZuwBxTjOZGTfCmTB/abQwxphnfjJVQHeaFODLf3GN2gsxaAmTdgSUCqlzDMmP/FPan2d8caNrYNjaDOSBa"
"5THXmB3YWeBvBLsPntWTPWCD4BTPYN5A0GIhfoyf4934PqaZ7VhuyC4ePoEfg3eH2G98EFqBHMD9L5BNjqJeeBXuiL+gE6g5"
"5KjF5CHIrT5g0t+JQAfR6bQLIJVcgFcH0aX0OX0GuptBp9FdoEFF8NFXUKXsAVzzmJo0lxAuIEDK72gcdVMXzSOUFpLpXnqc"
"vqcBKgspNJ4mgo7O0GP0Mj0H1610Dm1NI6lB/iEXyRrItDlJCkSFvoCCnOgaZNHeKDs6C3KtDNXObfYXG8PysX/MhWZb8zez"
"u1nHDBiPjT+NbXANMweDV3SGqjjaXG5WAinMYb8wysLheoF9Z59YP3bHRCChC2wR42Cru83rUEHbANEkmTpg+UKAbxLN/FDn"
"bGI3IDdeYW1YX9DgHL4L9D6DTQdrqMj/gHqrFlSqMvjqKf6RL+CxLDfkkO6sC8g0lV0xx5onzPqgncfA7XKzgfnSfAkYys2G"
"slxgSRfZYO7ml3l2/gkqqFjI6REoFOLcfrivgOYDTUTd0Ep0ErBlFTwI98R9cCwOJ+1ISzIOsA0nrcAb9tBVdBlIuq1wVYgR"
"GgrFhYrCDuG94BO2CNWFdFpDOCDsgTc24Ss9CXrjtKTwnY6lw0GjW0CTo0GfPWgL2px2h5o9B5UgTn4mxyBG9iZNSSo+D1G4"
"J7ZAvmuOCiGM4gCbNOGP2E6wycxqPYaVY5/MJ6YXrLQAE1kEK8JGwpt1bDf7BtF5KoxvxkuAbxREf6HJyAOy6QxW+BdKRfkQ"
"Bvm9ZHY+BCLgHrbRPGyeZD7+hhfgCeZ4sybrwXsgO7/BcrLGbDHvDRK5xW+x9Swc8Noy9JaH8f3sKqvAu/OZPIT3Y7VZL5bE"
"7oDH5mdW0Fpb8N+hrAlzmcUga5wDPxwPmf4j+MV4sKUVIPvi/A5vC7IuDRX/Rn4B4v8OyP3loHpdwHOh8+g71HgR6AO3gh4a"
"44E4HY0EnRTEx7AbqqKWuDTuhr/g/qQu8eG1gPFU0gFQZyNSk+g4P0S5O+Qc5LPGgPoXkPPQJpI2gP0WgH/NIc2BOoB/7YAZ"
"MaQaaUbGQN5aQlqRPKQErNiF1CGYnMDH8Vf8Cf+N22IRJ0EV2BgXww4UA/KrApzfBn7qgd084gSeqwO37yDuzIOYdgEs9A6c"
"sy54+C98OOS3pTyGV+HR0NOfb+Vn+QNAv1P5Zs4ArRZBtVBhVAxVg6ozFs6dAtcZaCj6A71AOXADQD/zwQZi0D9QzXaAKDoD"
"0FFtFI/64SlYwNXQd94eEdwf94J3ZaFyqYycaDzOh7cgBSE0AOJKdvwTdUZfuJN3hrwbD/VtEcTBBzqDhb8EVDoD6qVBaCfS"
"kIyj8DW0FWT+DZXANjivHR0DPrzoF1wV18YI5JAT+gfgybg3rgnv2wFPeyBLL4GePhDJNuFrED+24EOABg7B3TnQzVl4vxIw"
"xd+AfXeCL+2E380wFmo4sPIL+B6+iY/i61B7fcXpMP4ESP0baPcFfgn19k5A6/vwZbwY/4qbAw4cj3/HkwBDExyGa+HfQEIN"
"sQR6aYy742kQSfNhDjh7CnjuQOBPxSWhuh6AO8HJ7qPHyAZzauMfYGFX4akYrgZzn0DN74c5Q2F9N5xVwiNhz93AcyWwtNX4"
"FQ7CGWbiIVBP3McZ2AOn7I1H47v4LT6Jx+GyIIVecP4FuD2OAC1UwIPh7S+YAo/FYeQ6OPkUsNhieAR+CLXINuCtLm4C1xP4"
"COyVDeuoAIxsB/uXwCnoOHoEVpADZP8CUNAZdAc9QXHIA56bBHyGg9YzUC7cFFaoB3G6GlBFeOoBZy8Ke/qRgq1YQ270APR4"
"Al1Cr9EhqHJmgkedh7sD4Ge7AGUdQgkoACgsDr1D6VmaL4Jz47xYhhq1DkgtJ1h9KPSaIJGnMOYruoXuocuQxcehsWAzM9Bc"
"NAZNh+sk8M3xsGLm10br0Z/oAljwKRj7Cj2E2BMLfFyD5+3ob5h9Aew4HnoWoo5gZ3HAwQPwoxhA6KPA0r6g56g9VHkR8HQQ"
"bHUVaoUk8KwY4Pwi6oWyoVyoDRoIz0MAL3ohducCPFEc+fg/PJVbsr7LskFzAJ7rhPqiquAPAioP/tMcVYZxAYhxGbw8zHXy"
"q4BAbvJX/BvEtDi+B/LzTOi5Dv47iQ/iXSDa9uUT+TTeizeCyDkK3o4E/LUavPYoP86vwHUt5KVrsO9tvgXmDOBzwdO381kw"
"PhQwc33o6Qy45ht7zt6yCMApeXgG5P9tkJdUHgkY4x92GCiOBSGuMvYOMtMz5oPa2M88gPzesIcQo++xF5AHr8P1NvzugAz4"
"AOgji4fK8SWsYGcCz8utXIV6pwjE+/K8NuxUCCJsLdi9HUScXLBvDog+9Xh96E/KmpON5wSk9Ra4OQVR28841Lqn2XZATKks"
"FNZKhVG3YQcJcFQYd7LPEN3DYJ98gJZdgNMMVpiXhCedpQDHCE4j8gCMT8j6gkmHmvYDcPg16ykF8lECy/wXZD9gqcx/S9Zh"
"fjjwLMFqHDCdwA0WAs0LY3U4T6YMMr+I4jDHBOlk7v+NMajJ3bDmOzj9VxhpByl9YjdBMq9h/ZfsFrsE0jnFjgBCPMk2sFWA"
"HLextWw5mwd36yB7zQe0PIdNAyy/AbLRQshQw9kU6N/H/mDLoKoZAW0zzN4I2Ws0GwLzFrAlMHoYIJe+bCqMmQH9fWHWDMjA"
"2wCH7oCxu2DXJ+wV6O4+aOwVcPMJatFPwNFd6HsE7Snc3QVOrwCufQTvnwLPz2HOY/ae/QSJp8GJ3kCzg4R0kIcGJwsyK+gw"
"G2iVQZ+Y9YWZyhFPgLVTs74ZSwRJPIP93rMvgFlT4PoeUFYCzMWcgEw1kLKPabCeAjoUuBfeuOGdBTK2BTSX+RUagvUj4UmC"
"0UF4p4AeEcjfAfIPQnOBTtJhxwSgL6DTz7DHBzhB5lnuw1nuAd1kf4PMTwFie5aF2U6BPG7AGd8D3YOnk5D/n8Cch9B/HGqb"
"83DuNzD7HNh+LEjkJTxfh/7tgFxOgA7Psr2AGdYB0jkNY87CmL9g5EUY9Qxs9Ro7A5b5AlZ4AatfgBkXofch+ETmzifg/Y0s"
"zu5D73XofZxlH09h/AewoR8gtUzJpcPZvf/5WVrWd3SMUZCaAXWuASdnIBkdbA+BNEx4MqE/CM0DErHDDCfc+0G6Bsz8l4ws"
"+8xcReaZI3WwXpYlvUSQpi9rTS9I8SM0b9a85CzNpWRpiAK+/Q7c2WFU5peCOoxxQKOgEQY7uWCcEzTmyfKLn2BdicBFpgc4"
"4ZoEazmybMYH751gHQ6Yk8m9BnNSss6dBL0JWfvHAb2DnTNl8SVrpWSYw4CLf79htIK1RPIoiFyleTGoggtlURmIKqXA54tA"
"1V8BYkwT3hjiSQ2oResDzmzD2/JWvDn8doZ4GcMH8t+zrkP5CIiM0wHrTAGaDL9zAN8tgjYPouVivgqq240QRxfyJXC/EbDR"
"Zoihq/kGiMKxgJNO8YOAy49CXL4FNdRtoJtQDb/g7wD3v4Xfp0AvIYa/4E/+o8fQnkM9/5rH8w8Q239Cre+CPGEHvOOG2O/M"
"ujqA0uDXBe8/8s8w7itP4Mk8Ba6JWTM8PIn/gJYIfZlfun6AFd/Avp8z/44aKB5mpMCoVOj5B3Z9Du/igJeHwN8d4OE1PL3L"
"4vMb7O3nBmA9g5tQqQazyOAUWSFXyUgEnGaDKisfygF1iIwsKArlRjmhT0SZowXokSEneoBnDZ4yv0PWss4QhDU1HgB+0+HZ"
"CyfL/NtqB/xqWd/oemBfjWd+s4wgQ2a+C3IF8imGJy/wYoV9MneKhL0iUQHA4uWBiqKCgDCrZH2pXAdQaUVUKeu75frQakNG"
"rQitMqoAWLMkUGkYWzyLMr91LoTyojyQo3PBunlh5QhYNxKFoXDYQYLz/PvVNAb+vf/x6wPOU7Iknwb3buA+Fe7c8NYP5M3i"
"WoNz/vsHAW4lsIqUJbd/v8KWQH4yXKX/VpeyrlAI//elNs36Jf/d/89vuDMl+b+/7/73G+/MHTL34P9jv/97/ffd/3369/d/"
"AYxHlHJ2PgAA"
)
@lru_cache(maxsize=1)
def ready_trigger_pcm() -> bytes:
return gzip.decompress(base64.b64decode(READY_TRIGGER_PCM_16KHZ_MONO_GZIP_B64))

11
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -1986,7 +1986,8 @@ class BaseLLMHTTPHandler:
api_base=api_base,
)
headers = update_headers_with_filtered_beta(headers=headers, provider=custom_llm_provider)
if anthropic_messages_provider_config.should_filter_anthropic_beta_headers():
headers = update_headers_with_filtered_beta(headers=headers, provider=custom_llm_provider)
logging_obj.update_from_kwargs(
kwargs=kwargs,
@ -2111,7 +2112,7 @@ class BaseLLMHTTPHandler:
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs={**kwargs, "api_key": api_key} if api_key else kwargs,
)
return initial_response
else:
@ -2121,6 +2122,10 @@ class BaseLLMHTTPHandler:
logging_obj=logging_obj,
)
# Inject api_key into kwargs so follow-up calls in agentic hooks can
# authenticate. api_key is a named param here (not in kwargs), so
# _prepare_followup_kwargs would miss it otherwise.
kwargs_for_agentic = {**kwargs, "api_key": api_key} if api_key else kwargs
# Call agentic completion hooks (non-streaming path only)
final_response = await self._call_agentic_completion_hooks(
response=initial_response,
@ -2131,7 +2136,7 @@ class BaseLLMHTTPHandler:
logging_obj=logging_obj,
stream=False,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs=kwargs_for_agentic,
)
return self._maybe_wrap_in_fake_stream(

0
litellm/llms/gdc/__init__.py Normal file
View file

0
litellm/llms/gdc/chat/__init__.py Normal file
View file

285
litellm/llms/gdc/chat/transformation.py Normal file
View file

@ -0,0 +1,285 @@
"""
GDC Gemini chat completion transformation
"""
import json
import os
import re
import threading
from typing import Any, Final
from urllib.parse import urlsplit
import litellm
from litellm.llms.openai_like.chat.transformation import OpenAILikeChatConfig
class GDCGeminiConfig(OpenAILikeChatConfig):
supports_vertex_params: bool = True # Tell LiteLLM utilities not to strip vertex_ params
_GDCH_CREDENTIAL_TYPE: Final[str] = "gdch_service_account"
_PATH_ID_PATTERN: Final[re.Pattern[str]] = re.compile(r"^[a-zA-Z0-9_-]+$")
def __init__(self, **kwargs: Any) -> None:
super().__init__(**kwargs)
self._creds_lock = threading.Lock()
self._gdch_creds_cache: dict = {}
def get_supported_openai_params(self, model: str) -> list:
return [
"vertex_project",
"vertex_location",
] + super().get_supported_openai_params(model)
def _resolve_project(self, optional_params: dict, litellm_params: dict) -> str | None:
return (
litellm_params.get("vertex_project")
or litellm_params.get("vertex_ai_project")
or getattr(litellm, "vertex_project", None)
or optional_params.get("vertex_project")
or optional_params.get("vertex_ai_project")
)
def _resolve_location(self, optional_params: dict, litellm_params: dict) -> str | None:
return (
litellm_params.get("vertex_location")
or litellm_params.get("vertex_ai_location")
or getattr(litellm, "vertex_location", None)
or optional_params.get("vertex_location")
or optional_params.get("vertex_ai_location")
)
def _effective_project(self, api_base: str, optional_params: dict, litellm_params: dict) -> str | None:
match = re.search(r"/v1/projects/([^/]+)", api_base)
if match:
return match.group(1)
return self._resolve_project(optional_params, litellm_params)
def _validate_path_id(self, value: str, field: str, model: str) -> str:
if not self._PATH_ID_PATTERN.match(value):
raise litellm.utils.AuthenticationError(
message=f"{field} must be a plain identifier of letters, digits, hyphens or underscores.",
llm_provider="gdc",
model=model,
)
return value
def get_complete_url(
self,
api_base: str | None,
api_key: str | None,
model: str,
optional_params: dict,
litellm_params: dict,
stream: bool | None = None,
) -> str:
api_base = api_base or litellm.gdc_api_base or litellm.api_base
if not api_base:
raise litellm.utils.AuthenticationError(
message="api_base/host is required for GDC Gemini. Please set it or pass it.",
llm_provider="gdc",
model=model,
)
if not api_base.startswith("http"):
api_base = f"https://{api_base}"
api_base = api_base.rstrip("/")
if "/v1/projects/" in api_base:
return api_base
project = self._resolve_project(optional_params, litellm_params)
if not project:
raise litellm.utils.AuthenticationError(
message="project is required for GDC Gemini. Please pass vertex_project.",
llm_provider="gdc",
model=model,
)
location = self._resolve_location(optional_params, litellm_params)
if not location:
raise litellm.utils.AuthenticationError(
message="location is required for GDC Gemini. Please pass vertex_location.",
llm_provider="gdc",
model=model,
)
project = self._validate_path_id(project, "vertex_project", model)
location = self._validate_path_id(location, "vertex_location", model)
return f"{api_base}/v1/projects/{project}/locations/{location}/chat/completions"
def _read_env_bool(self, val: Any, env_var: str, default: bool = True) -> bool | str:
def _parse(s: str) -> bool | str:
cleaned = s.strip().lower()
if cleaned in ("false", "0", "no", "off"):
return False
if cleaned in ("true", "1", "yes", "on"):
return True
return s
if val is not None:
if isinstance(val, str):
return _parse(val)
return val
_env_val = os.getenv(env_var)
if _env_val is None:
return default
return _parse(_env_val)
def _fetch_auth(self, gdch_creds: Any, ssl_verify: bool | str) -> None:
import requests
from google.auth.transport import requests as auth_requests
auth_session = requests.Session()
auth_session.verify = ssl_verify
auth_request = auth_requests.Request(session=auth_session)
gdch_creds.refresh(auth_request)
def _cached_fetch_token(self, creds: Any, audience: str, ssl_verify: bool | str, api_key: str | None = None) -> str:
# Key cache by both audience and credential identity to prevent cross-caller contamination
cache_key = (audience.rstrip("/"), api_key or str(id(creds)))
with self._creds_lock:
if cache_key not in self._gdch_creds_cache:
self._gdch_creds_cache[cache_key] = creds.with_gdch_audience(audience.rstrip("/"))
gdch_creds = self._gdch_creds_cache[cache_key]
if not getattr(gdch_creds, "valid", False) or not getattr(gdch_creds, "token", None):
self._fetch_auth(gdch_creds, ssl_verify)
token = gdch_creds.token
return token
def _load_creds_from_key(self, api_key: str) -> tuple[Any, bool]:
import google.auth
try:
json_obj = json.loads(api_key)
except json.JSONDecodeError:
return None, False
if not isinstance(json_obj, dict) or json_obj.get("type") != self._GDCH_CREDENTIAL_TYPE:
raise ValueError(
"GDC only accepts a GDCH service account credential as a JSON api_key "
'(expected "type": "gdch_service_account"). Other Google credential types are '
"rejected so their token or external-account endpoints cannot drive server-side requests."
)
creds, _ = google.auth.load_credentials_from_dict(json_obj)
return creds, True
def validate_environment(
self,
headers: dict,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: str | None = None,
api_base: str | None = None,
) -> dict:
import google.auth.exceptions
api_base = api_base or litellm.gdc_api_base or litellm.api_base
if not api_base:
raise litellm.utils.AuthenticationError(
message="api_base/host is required for GDC Gemini. Please set it or pass it.",
llm_provider="gdc",
model=model,
)
if not api_key:
raise litellm.utils.AuthenticationError(
message="api_key is required for GDC Gemini. Please pass your service account string or token as the api_key.",
llm_provider="gdc",
model=model,
)
project = self._effective_project(api_base, optional_params, litellm_params)
if not project:
raise litellm.utils.AuthenticationError(
message="project is required for GDC Gemini. Please pass vertex_project.",
llm_provider="gdc",
model=model,
)
project = self._validate_path_id(project, "vertex_project", model)
_audience_parts = urlsplit(api_base if api_base.startswith("http") else f"https://{api_base}")
audience = f"{_audience_parts.scheme}://{_audience_parts.netloc}"
try:
creds, is_service_account = self._load_creds_from_key(api_key)
except (
google.auth.exceptions.GoogleAuthError,
ValueError,
TypeError,
KeyError,
AttributeError,
) as e:
raise litellm.utils.AuthenticationError(
message=f"Failed to load service account credentials from api_key: {str(e)}",
llm_provider="gdc",
model=model,
) from e
if creds is not None:
ssl_verify = self._read_env_bool(litellm_params.get("ssl_verify"), "SSL_VERIFY", default=True)
if self._read_env_bool(litellm_params.get("gdc_token_caching"), "GDC_TOKEN_CACHING", default=False):
token = self._cached_fetch_token(creds, audience, ssl_verify, api_key)
else:
gdch_creds = creds.with_gdch_audience(audience)
self._fetch_auth(gdch_creds, ssl_verify)
token = gdch_creds.token
headers["Authorization"] = f"Bearer {token}"
if "Authorization" not in headers and not is_service_account:
headers["Authorization"] = f"Bearer {api_key}"
# Standardize necessary metadata headers
if "content-type" not in headers and "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
stale_quota_headers = tuple(h for h in headers if h.lower() == "x-goog-user-project")
for stale in stale_quota_headers:
headers.pop(stale, None)
headers["x-goog-user-project"] = f"projects/{project}"
return headers
def transform_request(
self,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transforms the request to the GDC provider
"""
if model.startswith("gdc/"):
model = model.split("/", 1)[1]
data = super().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
# Remove extra params used for routing/auth
for param in [
"vertex_project",
"vertex_ai_project",
"vertex_location",
"vertex_ai_location",
"ssl_verify",
"gdc_token_caching",
]:
data.pop(param, None)
return data

0
litellm/llms/github_copilot/messages/__init__.py Normal file
View file

118
litellm/llms/github_copilot/messages/transformation.py Normal file
View file

@ -0,0 +1,118 @@
from typing import Any, Optional
from litellm.exceptions import AuthenticationError
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
from ..authenticator import Authenticator
from ..common_utils import (
DEFAULT_GITHUB_COPILOT_API_BASE,
GetAPIKeyError,
get_copilot_default_headers,
)
_MESSAGES_PROXY_API_VERSION = "2026-06-01"
class GithubCopilotAnthropicMessagesConfig(AnthropicMessagesConfig):
"""
GitHub Copilot implementation of Anthropic messages API.
Routes requests to Copilot's /v1/messages endpoint with appropriate authentication and headers.
"""
def __init__(self) -> None:
super().__init__()
self.authenticator = Authenticator()
def handles_web_search_natively(self) -> bool:
"""
Copilot's /v1/messages endpoint does not execute ``web_search`` tools, so
the interception handler must short-circuit web-search-only requests
instead of routing them here.
"""
return False
def should_filter_anthropic_beta_headers(self) -> bool:
"""
Copilot's /v1/messages is a native Anthropic Messages passthrough, so
``anthropic-beta`` values injected by ``_update_headers_with_anthropic_beta``
(context_management, structured outputs, ...) must reach the upstream
verbatim. The default provider-scoped filter would drop them because
github_copilot has no entry in ``anthropic_beta_headers_config.json``.
"""
return False
def validate_anthropic_messages_environment(
self,
headers: dict,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> tuple[dict, Optional[str]]:
"""
Validate environment for GitHub Copilot and add Copilot-specific headers.
The caller-supplied ``api_base`` is intentionally ignored. Routing this
request anywhere other than the authenticated Copilot endpoint would
leak the Copilot bearer token to a caller-controlled URL.
"""
# Always use the Copilot endpoint resolved from the authenticated
# session, never the caller-supplied api_base. rstrip so a
# tenant-specific base with a trailing slash does not yield a
# double-slash URL once "/v1/messages" is appended downstream.
dynamic_api_base = (self.authenticator.get_api_base() or DEFAULT_GITHUB_COPILOT_API_BASE).rstrip("/")
try:
dynamic_api_key = self.authenticator.get_api_key()
except GetAPIKeyError as e:
raise AuthenticationError(
model=model,
llm_provider="github_copilot",
message=str(e),
)
# Merge Copilot headers with provided headers
copilot_headers = get_copilot_default_headers(dynamic_api_key)
for key, value in copilot_headers.items():
if key not in headers:
headers[key] = value
headers["openai-intent"] = "messages-proxy"
headers["x-interaction-type"] = "messages-proxy"
headers["x-github-api-version"] = _MESSAGES_PROXY_API_VERSION
if "anthropic-version" not in headers:
headers["anthropic-version"] = "2023-06-01"
headers = self._update_headers_with_anthropic_beta(
headers, optional_params, custom_llm_provider="github_copilot"
)
return headers, dynamic_api_base
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
"""
Return the complete URL for GitHub Copilot /v1/messages endpoint.
``api_base`` here is the value already resolved by
``validate_anthropic_messages_environment`` (the authenticated Copilot
host), not the raw caller-supplied base — that one is discarded there to
avoid leaking the Copilot bearer token to a caller-controlled URL. We
reuse it to avoid a second authenticator read, falling back to a fresh
resolution only if it was not provided.
"""
resolved = (api_base or self.authenticator.get_api_base() or DEFAULT_GITHUB_COPILOT_API_BASE).rstrip("/")
if not resolved.endswith("/v1/messages"):
resolved = f"{resolved}/v1/messages"
return resolved

0
litellm/llms/openai_like/messages/__init__.py Normal file
View file

69
litellm/llms/openai_like/messages/transformation.py Normal file
View file

@ -0,0 +1,69 @@
from typing import Any, Optional
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
class OpenAILikeAnthropicMessagesConfig(AnthropicMessagesConfig):
"""
Forwards Anthropic /v1/messages requests to an OpenAI-compatible server that
also natively exposes the Anthropic Messages API, with no translation.
Opted into per deployment via ``model_info.supported_endpoints`` containing
``"/v1/messages"``. The inbound Anthropic payload (system, cache_control,
thinking, tools, ...) is forwarded essentially unchanged to
``{api_base}/v1/messages``, so Anthropic-only features that the
Anthropic->OpenAI translation would otherwise drop are preserved. Response
parsing and streaming are inherited from the native Anthropic config.
"""
def validate_anthropic_messages_environment(
self,
headers: dict[str, str],
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> tuple[dict[str, str], Optional[str]]:
present = {key.lower() for key in headers}
needs_auth = bool(api_key) and "authorization" not in present and "x-api-key" not in present
defaults: dict[str, str] = {
**({"authorization": f"Bearer {api_key}"} if needs_auth else {}),
**({"anthropic-version": DEFAULT_ANTHROPIC_API_VERSION} if "anthropic-version" not in present else {}),
**({"content-type": "application/json"} if "content-type" not in present else {}),
}
combined = {**headers, **defaults}
normalized = {
("anthropic-beta" if key.lower() == "anthropic-beta" else key): value for key, value in combined.items()
}
merged = self._update_headers_with_anthropic_beta(
headers=normalized,
optional_params=optional_params,
)
return merged, api_base
def should_filter_anthropic_beta_headers(self) -> bool:
return False
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
if not api_base:
raise ValueError("api_base is required to forward Anthropic /v1/messages to a native endpoint")
base = api_base.rstrip("/")
if base.endswith("/v1/messages"):
return base
if base.endswith("/v1"):
base = base[: -len("/v1")]
return f"{base}/v1/messages"

20
litellm/llms/vertex_ai/batches/transformation.py
View file

@ -1,4 +1,4 @@
from typing import Any, Dict
from typing import Any, Dict, Optional
from litellm._uuid import uuid
from litellm.llms.vertex_ai.common_utils import (
@ -47,7 +47,7 @@ class VertexAIBatchTransformation:
) -> LiteLLMBatch:
return LiteLLMBatch(
id=cls._get_batch_id_from_vertex_ai_batch_response(response),
completion_window="24hrs",
completion_window="24h",
created_at=_convert_vertex_datetime_to_openai_datetime(vertex_datetime=response.get("createTime", "")),
endpoint="",
input_file_id=cls._get_input_file_id_from_vertex_ai_batch_response(response),
@ -207,3 +207,19 @@ class VertexAIBatchTransformation:
parts = model_path.split("/")
model = f"publishers/{'/'.join(parts[:3])}"
return model
@classmethod
def is_unmanaged_gcs_batch_input_file_id(cls, input_file_id: Optional[str]) -> bool:
"""
Returns True if `input_file_id` is a raw gs:// Vertex batch input file (i.e. not a
LiteLLM-managed unified file id) with a `publishers/` model path that
`_get_model_from_gcs_file` can parse.
"""
return input_file_id is not None and input_file_id.startswith("gs://") and "publishers/" in input_file_id
@classmethod
def get_bare_model_name_from_gcs_file(cls, gcs_file_uri: str) -> str:
"""
Extracts the bare model name (e.g. "gemini-1.5-flash-001") from a gcs file uri.
"""
return cls._get_model_from_gcs_file(gcs_file_uri).rsplit("/", 1)[-1]

16
litellm/llms/vertex_ai/image_generation/vertex_gemini_transformation.py
View file

@ -1,6 +1,8 @@
import os
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from litellm._logging import verbose_logger
import httpx
import litellm
@ -52,6 +54,7 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
return [
"n",
"size",
"imageConfig",
"aspectRatio",
"aspect_ratio",
"imageSize",
@ -83,7 +86,12 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
mapped_params["aspectRatio"] = v
elif k in ("imageSize", "image_size"):
mapped_params["imageSize"] = v
elif k not in ("tools", "web_search_options"):
elif k == "imageConfig":
if isinstance(v, dict):
mapped_params["imageConfig"] = v
else:
verbose_logger.warning("imageConfig must be a dict, got %s — ignoring.", type(v).__name__)
elif k not in ("tools", "web_search_options", "imageConfig"):
mapped_params[k] = v
mapped_params = map_gemini_image_tools_params(non_default_params, mapped_params)
@ -211,16 +219,14 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
# Prepare generation config
generation_config: Dict[str, Any] = {"responseModalities": ["IMAGE"]}
# Handle image-specific config parameters
image_config: Dict[str, Any] = {}
# Seed from user-supplied imageConfig dict; flat params are overlaid for backward compat.
image_config: Dict[str, Any] = dict(optional_params.get("imageConfig") or {})
# Map aspectRatio
if "aspectRatio" in optional_params:
image_config["aspectRatio"] = optional_params["aspectRatio"]
elif "aspect_ratio" in optional_params:
image_config["aspectRatio"] = optional_params["aspect_ratio"]
# Map imageSize (for Gemini 3 Pro)
if "imageSize" in optional_params:
image_config["imageSize"] = optional_params["imageSize"]
elif "image_size" in optional_params:

43
litellm/main.py
View file

@ -210,6 +210,7 @@ from .llms.bedrock.embed.embedding import BedrockEmbedding
from .llms.bedrock.image_edit.handler import BedrockImageEdit
from .llms.bedrock.image_generation.image_handler import BedrockImageGeneration
from .llms.bytez.chat.transformation import BytezChatConfig
from .llms.gdc.chat.transformation import GDCGeminiConfig
from .llms.clarifai.chat.transformation import ClarifaiConfig
from .llms.codestral.completion.handler import CodestralTextCompletion
from .llms.cohere.embed import handler as cohere_embed
@ -318,6 +319,7 @@ google_batch_embeddings = GoogleBatchEmbeddings()
vertex_partner_models_chat_completion = VertexAIPartnerModels()
vertex_gemma_chat_completion = VertexAIGemmaModels()
vertex_model_garden_chat_completion = VertexAIModelGardenModels()
gdc_transformation = GDCGeminiConfig()
# vertex_text_to_speech is now replaced by VertexAITextToSpeechConfig
sagemaker_llm = SagemakerLLM()
watsonx_chat_completion = WatsonXChatHandler()
@ -4336,6 +4338,45 @@ def _complete_gradient_ai(ctx: _CompletionDispatchContext) -> _CompletionDispatc
)
def _complete_gdc(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client = ctx.client
custom_llm_provider = ctx.custom_llm_provider
headers = ctx.headers
litellm_params = ctx.litellm_params
logging = ctx.logging
messages = ctx.messages
model = ctx.model
model_response = ctx.model_response
optional_params = ctx.optional_params
stream = ctx.stream
timeout = ctx.timeout
api_key = api_key or litellm.gdc_key or get_secret_str("GDC_API_KEY") or litellm.api_key
api_base = api_base or litellm.gdc_api_base or get_secret_str("GDC_API_BASE") or litellm.api_base
return base_llm_http_handler.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
api_key=api_key,
api_base=api_base,
acompletion=acompletion,
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
timeout=timeout, # type: ignore
client=client,
custom_llm_provider=custom_llm_provider,
encoding=_get_encoding(),
stream=stream,
provider_config=gdc_transformation,
)
def _complete_bytez(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion = ctx.acompletion
api_base = ctx.api_base
@ -5536,6 +5577,8 @@ def completion( # type: ignore
elif custom_llm_provider == "gradient_ai":
response = _complete_gradient_ai(_dispatch_ctx)
elif custom_llm_provider == "gdc":
response = _complete_gdc(_dispatch_ctx)
elif custom_llm_provider == "bytez":
response = _complete_bytez(_dispatch_ctx)
elif custom_llm_provider == "lemonade":

377
litellm/model_prices_and_context_window_backup.json
View file

@ -1154,6 +1154,7 @@
"bedrock_output_config_effort_ceiling": "max"
},
"anthropic.claude-opus-4-7": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
@ -1203,6 +1204,7 @@
"supports_output_config": true
},
"global.anthropic.claude-opus-4-7": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
@ -1237,6 +1239,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"us.anthropic.claude-opus-4-7": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
@ -1271,6 +1274,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"eu.anthropic.claude-opus-4-7": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
@ -1305,6 +1309,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"au.anthropic.claude-opus-4-7": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
@ -1471,6 +1476,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"anthropic.claude-opus-4-8": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
@ -1505,6 +1511,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"global.anthropic.claude-opus-4-8": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
@ -1539,6 +1546,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"us.anthropic.claude-opus-4-8": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
@ -1573,6 +1581,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"eu.anthropic.claude-opus-4-8": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
@ -1607,6 +1616,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"au.anthropic.claude-opus-4-8": {
"bedrock_converse_supports_strict_tools": false,
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
@ -1641,6 +1651,7 @@
"bedrock_output_config_effort_ceiling": "xhigh"
},
"jp.anthropic.claude-opus-4-7": {
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 6.875e-06,
"cache_read_input_token_cost": 5.5e-07,
"input_cost_per_token": 5.5e-06,
@ -1671,6 +1682,204 @@
"supports_max_reasoning_effort": true,
"supports_minimal_reasoning_effort": true
},
"anthropic.claude-sonnet-5": {
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_1hr": 4e-06,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh"
},
"global.anthropic.claude-sonnet-5": {
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_1hr": 4e-06,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh"
},
"us.anthropic.claude-sonnet-5": {
"cache_creation_input_token_cost": 2.75e-06,
"cache_creation_input_token_cost_above_1hr": 4.4e-06,
"cache_read_input_token_cost": 2.2e-07,
"input_cost_per_token": 2.2e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh"
},
"eu.anthropic.claude-sonnet-5": {
"cache_creation_input_token_cost": 2.75e-06,
"cache_creation_input_token_cost_above_1hr": 4.4e-06,
"cache_read_input_token_cost": 2.2e-07,
"input_cost_per_token": 2.2e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh"
},
"au.anthropic.claude-sonnet-5": {
"cache_creation_input_token_cost": 2.75e-06,
"cache_creation_input_token_cost_above_1hr": 4.4e-06,
"cache_read_input_token_cost": 2.2e-07,
"input_cost_per_token": 2.2e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh"
},
"jp.anthropic.claude-sonnet-5": {
"cache_creation_input_token_cost": 2.75e-06,
"cache_creation_input_token_cost_above_1hr": 4.4e-06,
"cache_read_input_token_cost": 2.2e-07,
"input_cost_per_token": 2.2e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": true,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh"
},
"anthropic.claude-sonnet-4-6": {
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 3.75e-06,
@ -1884,7 +2093,8 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"bedrock_converse_supports_strict_tools": false
},
"anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
@ -2211,7 +2421,8 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"bedrock_converse_supports_strict_tools": false
},
"assemblyai/best": {
"input_cost_per_second": 3.333e-05,
@ -2511,6 +2722,36 @@
"supports_tool_choice": true,
"supports_vision": true
},
"azure_ai/claude-sonnet-5": {
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_1hr": 4e-06,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "azure_ai",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true
},
"azure_ai/claude-sonnet-4-6": {
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 3.75e-06,
@ -10245,6 +10486,40 @@
"supports_vision": true,
"supports_web_search": true
},
"claude-sonnet-5": {
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_1hr": 4e-06,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "anthropic",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true,
"provider_specific_entry": {
"us": 1.1
},
"supports_output_config": true
},
"claude-sonnet-4-6": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_1hr": 6e-06,
@ -14551,7 +14826,8 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"bedrock_converse_supports_strict_tools": false
},
"eu.anthropic.claude-sonnet-4-5-20250929-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
@ -18914,7 +19190,8 @@
"max_tokens": 16000,
"mode": "chat",
"supported_endpoints": [
"/v1/chat/completions"
"/v1/chat/completions",
"/v1/messages"
],
"supports_function_calling": true,
"supports_parallel_function_calling": true,
@ -18927,7 +19204,8 @@
"max_tokens": 16000,
"mode": "chat",
"supported_endpoints": [
"/v1/chat/completions"
"/v1/chat/completions",
"/v1/messages"
],
"supports_function_calling": true,
"supports_parallel_function_calling": true,
@ -18980,7 +19258,8 @@
"max_tokens": 16000,
"mode": "chat",
"supported_endpoints": [
"/v1/chat/completions"
"/v1/chat/completions",
"/v1/messages"
],
"supports_function_calling": true,
"supports_parallel_function_calling": true,
@ -19809,7 +20088,8 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"bedrock_converse_supports_strict_tools": false
},
"global.anthropic.claude-haiku-4-5-20251001-v1:0": {
"cache_creation_input_token_cost": 1.25e-06,
@ -32893,7 +33173,8 @@
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_vision": true,
"bedrock_converse_supports_strict_tools": false
},
"us.deepseek.r1-v1:0": {
"input_cost_per_token": 1.35e-06,
@ -34944,6 +35225,36 @@
"supports_tool_choice": true,
"supports_vision": true
},
"vertex_ai/claude-sonnet-5": {
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_1hr": 4e-06,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "vertex_ai-anthropic_models",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true
},
"vertex_ai/claude-sonnet-4-6": {
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 3.75e-06,
@ -42381,6 +42692,36 @@
"search_context_size_high": 0.035
}
},
"vertex_ai/claude-sonnet-5@default": {
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_1hr": 4e-06,
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "vertex_ai-anthropic_models",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true
},
"vertex_ai/claude-sonnet-4-6@default": {
"supports_adaptive_thinking": true,
"cache_creation_input_token_cost": 3.75e-06,
@ -42565,6 +42906,26 @@
"supports_tool_choice": true,
"supports_vision": true
},
"bedrock_mantle/xai.grok-4.3": {
"input_cost_per_token": 1.25e-06,
"output_cost_per_token": 2.5e-06,
"cache_read_input_token_cost": 2e-07,
"litellm_provider": "bedrock_mantle",
"max_input_tokens": 131072,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
],
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"source": "https://aws.amazon.com/bedrock/pricing/"
},
"volcengine/doubao-seed-2-0-pro-260215": {
"litellm_provider": "volcengine",
"max_input_tokens": 256000,

1
litellm/models/end_user.py
View file

@ -21,6 +21,7 @@ class LiteLLM_EndUserTable(LiteLLMPydanticObjectBase):
spend: float = 0.0
allowed_model_region: Optional[Literal["eu", "us"]] = None
default_model: Optional[str] = None
budget_id: Optional[str] = None
litellm_budget_table: Optional[LiteLLM_BudgetTable] = None
object_permission_id: Optional[str] = None
object_permission: Optional[LiteLLM_ObjectPermissionTable] = None

1
litellm/models/object_permission.py
View file

@ -24,3 +24,4 @@ class LiteLLM_ObjectPermissionTable(LiteLLMPydanticObjectBase):
mcp_toolsets: Optional[List[str]] = None
blocked_tools: Optional[List[str]] = []
search_tools: Optional[List[str]] = []
mcp_tool_search_enabled: Optional[bool] = None

6
litellm/proxy/_experimental/mcp_server/AGENTS.md
View file

@ -41,6 +41,7 @@ litellm/proxy/_experimental/mcp_server/
sampling_handler.py # MCP sampling to LiteLLM completion flow
elicitation_handler.py # MCP elicitation relay flow
semantic_tool_filter.py # semantic filtering of available MCP tools
tool_search.py # opt-in virtual tools (mcp_tool_search + mcp_tool_call) for large catalogs
guardrail_translation/
handler.py # MCP guardrail result translation
sse_transport.py # SSE transport implementation
@ -79,6 +80,11 @@ module materially harder to understand.
encryption need focused tests for both allowed and rejected paths.
- Avoid adding comments to new code unless they explain non-obvious security or
protocol behavior. Prefer clear names and small functions.
- The virtual tool path (`tool_search.py`, gated by `mcp_tool_search_enabled`)
must mirror the normal tool flow: IP filtering, server allowlist, per-key tool
permissions, no-accessible-server rejection, per-request auth headers, server
scope, error to `isError` conversion, and spend logging. Reuse `_list_mcp_tools`
and `execute_mcp_tool` rather than reimplementing any of these checks.
## Tests

17
litellm/proxy/_experimental/mcp_server/db.py
View file

@ -1228,6 +1228,23 @@ def _remaining_token_seconds(expires_at: str | None) -> int | None:
return remaining if remaining > 0 else None
async def get_active_submitted_mcp_server_ids_for_user(
prisma_client: PrismaClient,
user_id: str,
) -> list[str]:
"""Return active BYOM servers submitted by this user (creator visibility)."""
if not user_id:
return []
rows = await MCPServerRepository(prisma_client).table.find_many(
where={
"submitted_by": user_id,
"approval_status": MCPApprovalStatus.active,
},
)
return [row.server_id for row in rows]
async def approve_mcp_server(
prisma_client: PrismaClient,
server_id: str,

107
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -1246,6 +1246,67 @@ class MCPServerManager:
"""Return server IDs that bypass per-key restrictions."""
return [server.server_id for server in self.get_registry().values() if server.allow_all_keys is True]
@staticmethod
def get_byom_submitted_servers_cache_key(user_id: str) -> str:
return f"byom_submitted_servers:{user_id}"
async def invalidate_byom_submitted_servers_cache(self, user_id: str | None) -> None:
if not user_id:
return
try:
from litellm.proxy.proxy_server import user_api_key_cache
await user_api_key_cache.async_delete_cache(key=self.get_byom_submitted_servers_cache_key(user_id))
except Exception as e: # noqa: BLE001
verbose_logger.warning(f"Failed to invalidate BYOM submitted MCP server cache: {str(e)}")
async def _get_active_submitted_mcp_server_ids_for_user(
self, user_api_key_auth: UserAPIKeyAuth | None
) -> list[str]:
submitter_user_id = getattr(user_api_key_auth, "user_id", None) if user_api_key_auth else None
if not submitter_user_id:
return []
try:
from litellm.proxy._experimental.mcp_server.db import ( # noqa: PLC0415
get_active_submitted_mcp_server_ids_for_user,
)
from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
except Exception as e: # noqa: BLE001
verbose_logger.warning(f"Failed to load BYOM submitted MCP server cache dependencies: {str(e)}")
return []
byom_cache_key = self.get_byom_submitted_servers_cache_key(submitter_user_id)
submitted_server_ids: list[str] | None = None
try:
cached_submitted_server_ids = await user_api_key_cache.async_get_cache(key=byom_cache_key)
if cached_submitted_server_ids is not None:
submitted_server_ids = cast(list[str], cached_submitted_server_ids)
except Exception as e: # noqa: BLE001
verbose_logger.warning(f"Failed to read BYOM submitted MCP server cache: {str(e)}")
if submitted_server_ids is None:
if prisma_client is None:
submitted_server_ids = []
else:
try:
submitted_server_ids = await get_active_submitted_mcp_server_ids_for_user(
prisma_client, submitter_user_id
)
except Exception as e: # noqa: BLE001
verbose_logger.warning(f"Failed to read BYOM submitted MCP servers from database: {str(e)}")
submitted_server_ids = []
try:
await user_api_key_cache.async_set_cache(
key=byom_cache_key,
value=submitted_server_ids,
ttl=60,
)
except Exception as e: # noqa: BLE001
verbose_logger.warning(f"Failed to write BYOM submitted MCP server cache: {str(e)}")
return [server_id for server_id in submitted_server_ids if self.get_mcp_server_by_id(server_id) is not None]
async def get_allowed_mcp_servers(self, user_api_key_auth: Optional[UserAPIKeyAuth] = None) -> List[str]:
"""
Get the allowed MCP Servers for the user.
@ -1259,25 +1320,30 @@ class MCPServerManager:
allow_all_server_ids = self.get_allow_all_keys_server_ids()
# The key explicitly opted out of every MCP server. Return zero before
# layering on allow_all_keys or submitted servers so the opt-out is absolute.
key_object_permission = user_api_key_auth.object_permission if user_api_key_auth else None
if key_object_permission is not None and (
SpecialMCPServerNames.no_mcp_servers.value in (key_object_permission.mcp_servers or [])
):
return []
# Check if object_permission.mcp_servers is explicitly set (not None, empty list is valid)
has_explicit_object_permission = key_object_permission is not None and (
key_object_permission.mcp_servers is not None
)
if has_explicit_object_permission:
verbose_logger.debug(f"Object permission mcp_servers explicitly set: {key_object_permission.mcp_servers}")
# BYOM creator visibility never widens a key that was explicitly scoped:
# only keys without their own mcp_servers list get submitted servers unioned in.
submitted_server_ids = (
[]
if has_explicit_object_permission
else await self._get_active_submitted_mcp_server_ids_for_user(user_api_key_auth)
)
try:
# The key explicitly opted out of every MCP server. Return zero before
# layering on allow_all_keys servers so the opt-out is absolute.
key_object_permission = user_api_key_auth.object_permission if user_api_key_auth else None
if key_object_permission is not None and (
SpecialMCPServerNames.no_mcp_servers.value in (key_object_permission.mcp_servers or [])
):
return []
# Check if object_permission.mcp_servers is explicitly set
has_explicit_object_permission = False
if user_api_key_auth and user_api_key_auth.object_permission:
# Check if mcp_servers is explicitly set (not None, empty list is valid)
if user_api_key_auth.object_permission.mcp_servers is not None:
has_explicit_object_permission = True
verbose_logger.debug(
f"Object permission mcp_servers explicitly set: {user_api_key_auth.object_permission.mcp_servers}"
)
# If admin but NO explicit object permission, get all servers
if user_api_key_auth and _user_has_admin_view(user_api_key_auth) and not has_explicit_object_permission:
verbose_logger.debug("Admin user without explicit object_permission - returning all servers")
@ -1299,6 +1365,7 @@ class MCPServerManager:
in_toolset_scope = _mcp_active_toolset_id.get() is not None
if not in_toolset_scope:
combined_servers.update(allow_all_server_ids)
combined_servers.update(submitted_server_ids)
# For anonymous callers (no user_id, no role), also surface any
# servers the operator has opted into upstream-delegated auth.
@ -1331,9 +1398,9 @@ class MCPServerManager:
except Exception: # noqa: BLE001
verbose_logger.exception(
"Failed to get allowed MCP servers; team-level object_permission "
"grants may be dropped. Falling back to global servers only."
"grants may be dropped. Falling back to global and submitted servers."
)
return allow_all_server_ids
return list(dict.fromkeys(allow_all_server_ids + submitted_server_ids))
async def resolve_toolset_tool_permissions(
self,

112
litellm/proxy/_experimental/mcp_server/rest_endpoints.py
View file

@ -77,6 +77,7 @@ if MCP_AVAILABLE:
ListMCPToolsRestAPIResponseObject,
MCPInfo,
MCPServer,
_fire_mcp_success_logging,
_tool_name_matches,
execute_mcp_tool,
filter_tools_by_allowed_tools,
@ -84,6 +85,24 @@ if MCP_AVAILABLE:
########################################################
############ MCP Server REST API Routes #################
async def _safe_fire_mcp_success_logging(
logging_obj: Optional[Any],
result: Any,
start_time: datetime,
end_time: datetime,
) -> None:
if logging_obj is None:
return
logging_results = await asyncio.gather(
_fire_mcp_success_logging(logging_obj, result, start_time, end_time),
return_exceptions=True,
)
logging_error = logging_results[0]
if isinstance(logging_error, asyncio.CancelledError):
raise logging_error
if isinstance(logging_error, BaseException):
verbose_logger.warning("MCP tool success logging failed (continuing): %s", logging_error)
def _get_server_auth_header(
server,
mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
@ -569,6 +588,21 @@ if MCP_AVAILABLE:
include_disabled_tools and user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN
)
if apply_tool_filters and getattr(
getattr(user_api_key_dict, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
from litellm.proxy._experimental.mcp_server.tool_search import (
get_virtual_tool_definitions,
)
return {
"tools": get_virtual_tool_definitions(),
"error": None,
"message": "Successfully retrieved tools",
}
# Extract auth headers from request
headers = request.headers
raw_headers_from_request = dict(headers)
@ -727,6 +761,77 @@ if MCP_AVAILABLE:
try:
data = await request.json()
tool_name = data.get("name")
tool_arguments = data.get("arguments") or {}
from litellm.proxy._experimental.mcp_server.tool_search import (
MCP_TOOL_CALL_TOOL_NAME,
MCP_TOOL_SEARCH_TOOL_NAME,
coerce_top_k,
handle_mcp_tool_call,
handle_mcp_tool_search,
)
if tool_name in (MCP_TOOL_SEARCH_TOOL_NAME, MCP_TOOL_CALL_TOOL_NAME):
if not getattr(
getattr(user_api_key_dict, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
raise HTTPException(
status_code=403,
detail={
"error": "forbidden",
"message": f"{tool_name} requires mcp_tool_search_enabled on the key",
},
)
rest_client_ip = IPAddressUtils.get_mcp_client_ip(request)
(
virtual_mcp_auth_header,
virtual_mcp_server_auth_headers,
virtual_raw_headers,
) = _extract_mcp_headers_from_request(request, MCPRequestHandler)
virtual_oauth2_headers = MCPRequestHandler._get_oauth2_headers_from_headers(request.headers)
if tool_name == MCP_TOOL_SEARCH_TOOL_NAME:
return await handle_mcp_tool_search(
query=tool_arguments.get("query", ""),
top_k=coerce_top_k(tool_arguments.get("top_k", 5)),
user_api_key_dict=user_api_key_dict,
client_ip=rest_client_ip,
mcp_auth_header=virtual_mcp_auth_header,
mcp_server_auth_headers=virtual_mcp_server_auth_headers,
oauth2_headers=virtual_oauth2_headers,
raw_headers=virtual_raw_headers,
)
else: # MCP_TOOL_CALL_TOOL_NAME
# Run the same pre-call pipeline as the normal call path so the
# tool execution is spend-logged and guardrail-checked.
(
_,
virtual_logging_obj,
) = await ProxyBaseLLMRequestProcessing(data=data).common_processing_pre_call_logic(
request=request,
user_api_key_dict=user_api_key_dict,
proxy_config=proxy_config,
route_type=CallTypes.call_mcp_tool.value,
proxy_logging_obj=proxy_logging_obj,
general_settings=general_settings,
)
_tool_start_time = datetime.now()
result = await handle_mcp_tool_call(
tool_name=tool_arguments.get("tool_name", ""),
arguments=tool_arguments.get("arguments") or {},
user_api_key_dict=user_api_key_dict,
client_ip=rest_client_ip,
mcp_auth_header=virtual_mcp_auth_header,
mcp_server_auth_headers=virtual_mcp_server_auth_headers,
oauth2_headers=virtual_oauth2_headers,
raw_headers=virtual_raw_headers,
litellm_logging_obj=virtual_logging_obj,
)
await _safe_fire_mcp_success_logging(virtual_logging_obj, result, _tool_start_time, datetime.now())
return result
# Validate required parameters early
server_id = data.get("server_id")
if not server_id:
@ -738,7 +843,6 @@ if MCP_AVAILABLE:
},
)
tool_name = data.get("name")
if not tool_name:
raise HTTPException(
status_code=400,
@ -748,8 +852,6 @@ if MCP_AVAILABLE:
},
)
tool_arguments = data.get("arguments") or {}
proxy_base_llm_response_processor = ProxyBaseLLMRequestProcessing(data=data)
(
data,
@ -796,11 +898,12 @@ if MCP_AVAILABLE:
user_oauth_extra_headers = await _get_user_oauth_extra_headers(target_server, user_api_key_dict)
# Call execute_mcp_tool directly (permission checks already done)
_tool_start_time = datetime.now()
result = await execute_mcp_tool(
name=tool_name,
arguments=tool_arguments,
allowed_mcp_servers=allowed_mcp_servers,
start_time=datetime.now(),
start_time=_tool_start_time,
user_api_key_auth=data.get("user_api_key_auth"),
mcp_auth_header=data.get("mcp_auth_header"),
mcp_server_auth_headers=data.get("mcp_server_auth_headers"),
@ -809,6 +912,7 @@ if MCP_AVAILABLE:
litellm_logging_obj=data.get("litellm_logging_obj"),
requested_server_id=canonical_server_id,
)
await _safe_fire_mcp_success_logging(logging_obj, result, _tool_start_time, datetime.now())
return result
except MCPMissingUserEnvVarsError as e:
verbose_logger.info(

330
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -10,8 +10,8 @@ import contextvars
import hashlib
import json
import time
import types
import traceback
import types
import uuid
from datetime import datetime
from typing import (
@ -37,13 +37,17 @@ from starlette.types import Message, Receive, Scope, Send
from litellm._logging import verbose_logger
from litellm.constants import MAXIMUM_TRACEBACK_LINES_TO_LOG
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
MCPRequestHandler,
)
from litellm.proxy._experimental.mcp_server.exceptions import MCPUpstreamAuthError
from litellm.proxy._experimental.mcp_server.discoverable_endpoints import (
get_request_base_url,
)
from litellm.proxy._experimental.mcp_server.exceptions import MCPUpstreamAuthError
from litellm.proxy._experimental.mcp_server.mcp_context import (
_mcp_active_toolset_id,
_mcp_gateway_initialize_instructions,
@ -59,10 +63,6 @@ from litellm.proxy._experimental.mcp_server.utils import (
get_server_prefix,
iter_known_server_prefixes,
)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.proxy._types import (
ProxyException,
SpecialMCPServerNames,
@ -122,9 +122,12 @@ def _write_byok_cred_cache(user_id: str, server_id: str, credential: Optional[st
# TODO: Make this a util function for litellm client usage
MCP_AVAILABLE: bool = True
try:
import weakref
from mcp import ReadResourceResult, Resource
from mcp.server import Server
from mcp.server.lowlevel.helper_types import ReadResourceContents
from mcp.server.session import ServerSession as _McpServerSession
from mcp.types import (
BlobResourceContents,
GetPromptResult,
@ -132,8 +135,6 @@ try:
TextResourceContents,
Tool,
)
from mcp.server.session import ServerSession as _McpServerSession
import weakref
# Robust auth lookup keyed by session_object.
_session_obj_auth_storage: "weakref.WeakKeyDictionary[Any, MCPAuthenticatedUser]" = weakref.WeakKeyDictionary()
@ -229,6 +230,56 @@ def _jsonrpc_text_has_top_level_method(text: str) -> bool:
return False
def _mcp_meta_trace_carrier(req_ctx: object) -> Optional[dict[str, str]]:
"""The W3C trace context (``traceparent``/``tracestate``) the MCP client
propagated in the request's ``params._meta`` (SEP-414), or ``None``.
Per the OTel MCP semconv the MCP span parents to this propagated context rather
than to the HTTP/session transport (which is recorded as a link instead), so a
streamable-HTTP session that multiplexes many messages does not glue every
message under the session's first request. The client's W3C Baggage is
deliberately excluded: it is caller-controlled, and the otel baggage processor
stamps allowlisted baggage keys (``litellm.team.id``, ``litellm.metadata.*``,
...) onto the span, so honoring remote baggage would let a client spoof a
span's identity attribution.
"""
meta = getattr(req_ctx, "meta", None)
extra = getattr(meta, "model_extra", None)
if not isinstance(extra, dict):
return None
carrier = {key: extra[key] for key in ("traceparent", "tracestate") if isinstance(extra.get(key), str)}
return carrier or None
def _otel_set_mcp_trace_carrier(carrier: Optional[dict[str, str]]) -> object:
"""Stash ``carrier`` for the otel_v2 MCP span and return a reset token, or
``None`` when otel_v2 is unavailable. Lazily imported so opentelemetry stays an
optional dependency."""
try:
from litellm.integrations.otel.plumbing.context import (
set_mcp_message_trace_carrier,
)
return set_mcp_message_trace_carrier(carrier)
except ImportError:
return None
def _otel_reset_mcp_trace_carrier(token: object) -> None:
"""Clear the per-message trace carrier so it never leaks to the next message on
the same session task. Paired with ``_otel_set_mcp_trace_carrier``."""
if token is None:
return
try:
from litellm.integrations.otel.plumbing.context import (
reset_mcp_message_trace_carrier,
)
reset_mcp_message_trace_carrier(token)
except ImportError:
return
def _proxy_exception_to_http_exception(exc: ProxyException) -> HTTPException:
"""Map a ``ProxyException`` to an ``HTTPException`` that preserves its real
status code and headers.
@ -253,14 +304,14 @@ def _proxy_exception_to_http_exception(exc: ProxyException) -> HTTPException:
if MCP_AVAILABLE:
from mcp.server import Server
from mcp.server.lowlevel.server import NotificationOptions
from mcp.server.models import InitializationOptions
# Import auth context variables and middleware
from mcp.server.auth.middleware.auth_context import (
AuthContextMiddleware,
auth_context_var,
)
from mcp.server.lowlevel.server import NotificationOptions
from mcp.server.models import InitializationOptions
try:
from mcp.server.streamable_http_manager import StreamableHTTPSessionManager
@ -595,8 +646,10 @@ if MCP_AVAILABLE:
_session_reset_token = None
if req_ctx:
_session_reset_token = active_mcp_session_var.set(req_ctx.session)
_trace_token = None
try:
_trace_token = _otel_set_mcp_trace_carrier(_mcp_meta_trace_carrier(req_ctx))
# Get user authentication from context variable
(
user_api_key_auth,
@ -612,6 +665,19 @@ if MCP_AVAILABLE:
verbose_logger.debug(
f"MCP list_tools - MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}"
)
if getattr(
getattr(user_api_key_auth, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
from mcp.types import Tool
from litellm.proxy._experimental.mcp_server.tool_search import (
get_virtual_tool_definitions,
)
return [Tool(**d) for d in get_virtual_tool_definitions()]
# Get mcp_servers from context variable
verbose_logger.debug("MCP list_tools - Calling _list_mcp_tools")
tools = await _list_mcp_tools(
@ -632,9 +698,154 @@ if MCP_AVAILABLE:
# This prevents the HTTP stream from failing and allows the client to get a response
return []
finally:
_otel_reset_mcp_trace_carrier(_trace_token)
if _session_reset_token is not None:
active_mcp_session_var.reset(_session_reset_token)
def _capture_host_progress_callback(host_server) -> Optional[Callable]:
"""Return a progress-forwarding callback bound to the host MCP session.
Returns ``None`` when the host did not supply a progress token.
"""
try:
host_ctx = host_server.request_context
except Exception as e:
verbose_logger.warning(f"Could not capture host progress context: {e}")
return None
if not (host_ctx and hasattr(host_ctx, "meta") and host_ctx.meta):
return None
host_token = getattr(host_ctx.meta, "progressToken", None)
if not (host_token and hasattr(host_ctx, "session") and host_ctx.session):
return None
host_session = host_ctx.session
async def forward_progress(progress: float, total: Optional[float]):
"""Forward progress notifications from external MCP to Host"""
try:
await host_session.send_progress_notification(
progress_token=host_token,
progress=progress,
total=total,
)
verbose_logger.debug(f"Forwarded progress {progress}/{total} to Host")
except Exception as e:
verbose_logger.error(f"Failed to forward progress to Host: {e}")
verbose_logger.debug(f"Host progressToken captured: {host_token[:8]}...")
return forward_progress
async def _build_virtual_call_logging_obj(
name: str,
arguments: dict[str, Any],
user_api_key_auth: UserAPIKeyAuth,
) -> Optional[LiteLLMLoggingObj]:
"""Run the pre-call pipeline (guardrails + logging setup) for a virtual
mcp_tool_call so the SSE path spend-logs like the REST path."""
from fastapi import Request
from litellm.proxy.common_request_processing import (
ProxyBaseLLMRequestProcessing,
)
from litellm.proxy.proxy_server import (
general_settings,
proxy_config,
proxy_logging_obj,
)
request = Request(
scope={
"type": "http",
"method": "POST",
"path": "/mcp/tools/call",
"headers": [(b"content-type", b"application/json")],
}
)
_, virtual_logging_obj = await ProxyBaseLLMRequestProcessing(
data={"name": name, "arguments": arguments}
).common_processing_pre_call_logic(
request=request,
user_api_key_dict=user_api_key_auth,
proxy_config=proxy_config,
route_type=CallTypes.call_mcp_tool.value,
proxy_logging_obj=proxy_logging_obj,
general_settings=general_settings,
)
return virtual_logging_obj
async def _dispatch_virtual_mcp_tool(
name: str,
arguments: Optional[dict[str, Any]],
user_api_key_auth: Optional[UserAPIKeyAuth],
client_ip: Optional[str],
mcp_servers: Optional[list[str]] = None,
mcp_auth_header: Optional[str] = None,
mcp_server_auth_headers: Optional[dict[str, dict[str, str]]] = None,
oauth2_headers: Optional[dict[str, str]] = None,
raw_headers: Optional[dict[str, str]] = None,
) -> Optional[CallToolResult]:
"""Handle the mcp_tool_search / mcp_tool_call virtual tools.
Returns a CallToolResult when ``name`` is a virtual tool, else ``None`` so
the caller falls through to normal tool routing.
"""
from litellm.proxy._experimental.mcp_server.tool_search import (
MCP_TOOL_CALL_TOOL_NAME,
MCP_TOOL_SEARCH_TOOL_NAME,
coerce_top_k,
handle_mcp_tool_call,
handle_mcp_tool_search,
)
if name not in (MCP_TOOL_SEARCH_TOOL_NAME, MCP_TOOL_CALL_TOOL_NAME):
return None
if not getattr(
getattr(user_api_key_auth, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
return CallToolResult(
content=[
TextContent(
type="text",
text=f"Tool {name} requires mcp_tool_search_enabled on the key",
)
],
isError=True,
)
args = arguments or {}
if name == MCP_TOOL_SEARCH_TOOL_NAME:
return await handle_mcp_tool_search(
query=args.get("query", ""),
top_k=coerce_top_k(args.get("top_k", 5)),
user_api_key_dict=user_api_key_auth,
client_ip=client_ip,
mcp_servers=mcp_servers,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
)
assert user_api_key_auth is not None # guaranteed by the flag check above
virtual_logging_obj = await _build_virtual_call_logging_obj(
name=name, arguments=args, user_api_key_auth=user_api_key_auth
)
return await handle_mcp_tool_call(
tool_name=args.get("tool_name", ""),
arguments=args.get("arguments") or {},
user_api_key_dict=user_api_key_auth,
client_ip=client_ip,
mcp_servers=mcp_servers,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=virtual_logging_obj,
)
@server.call_tool()
async def mcp_server_tool_call(name: str, arguments: Dict[str, Any] | None) -> CallToolResult:
"""
@ -648,18 +859,21 @@ if MCP_AVAILABLE:
HTTPException: If tool not found or arguments missing
"""
from fastapi import Request
from mcp.server.lowlevel.server import request_ctx
from mcp.types import CallToolResult
from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.proxy.litellm_pre_call_utils import add_litellm_data_to_request
from litellm.proxy.proxy_server import proxy_config
from mcp.types import CallToolResult
from mcp.server.lowlevel.server import request_ctx
req_ctx = request_ctx.get(None)
_session_reset_token = None
if req_ctx:
_session_reset_token = active_mcp_session_var.set(req_ctx.session)
_trace_token = None
try:
_trace_token = _otel_set_mcp_trace_carrier(_mcp_meta_trace_carrier(req_ctx))
# Validate arguments
(
user_api_key_auth,
@ -675,31 +889,25 @@ if MCP_AVAILABLE:
)
verbose_logger.debug(f"MCP mcp_server_tool_call - User API Key Auth from context: {user_api_key_auth}")
host_progress_callback = None
try:
host_ctx = server.request_context
if host_ctx and hasattr(host_ctx, "meta") and host_ctx.meta:
host_token = getattr(host_ctx.meta, "progressToken", None)
if host_token and hasattr(host_ctx, "session") and host_ctx.session:
host_session = host_ctx.session
async def forward_progress(progress: float, total: Optional[float]):
"""Forward progress notifications from external MCP to Host"""
try:
await host_session.send_progress_notification(
progress_token=host_token,
progress=progress,
total=total,
)
verbose_logger.debug(f"Forwarded progress {progress}/{total} to Host")
except Exception as e:
verbose_logger.error(f"Failed to forward progress to Host: {e}")
host_progress_callback = forward_progress
verbose_logger.debug(f"Host progressToken captured: {host_token[:8]}...")
except Exception as e:
verbose_logger.warning(f"Could not capture host progress context: {e}")
try:
# Inside this try so virtual-tool errors convert to isError
# CallToolResult instead of raising out of the protocol handler.
virtual_tool_result = await _dispatch_virtual_mcp_tool(
name=name,
arguments=arguments,
user_api_key_auth=user_api_key_auth,
client_ip=_client_ip,
mcp_servers=mcp_servers,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
)
if virtual_tool_result is not None:
return virtual_tool_result
host_progress_callback = _capture_host_progress_callback(server)
# Create a body date for logging
body_data = {"name": name, "arguments": arguments}
# Set trace/session id from raw_headers so spend logs and logging_obj stay consistent (same as A2A)
@ -778,6 +986,7 @@ if MCP_AVAILABLE:
return response
finally:
_otel_reset_mcp_trace_carrier(_trace_token)
if _session_reset_token is not None:
active_mcp_session_var.reset(_session_reset_token)
@ -1472,6 +1681,8 @@ if MCP_AVAILABLE:
log_list_tools_to_spendlogs: bool = False,
list_tools_log_source: Optional[str] = None,
litellm_trace_id: Optional[str] = None,
request_tags: Optional[list[str]] = None,
client_ip: Optional[str] = None,
) -> List[MCPTool]:
"""
Helper method to fetch tools from MCP servers based on server filtering criteria.
@ -1514,6 +1725,7 @@ if MCP_AVAILABLE:
"litellm_trace_id": effective_litellm_trace_id,
"metadata": {
"spend_logs_metadata": spend_logs_metadata,
**({"tags": request_tags} if request_tags else {}),
},
# Provide a small input payload for standard logging
"input": [
@ -1559,6 +1771,7 @@ if MCP_AVAILABLE:
allowed_mcp_servers = await _get_allowed_mcp_servers(
user_api_key_auth=user_api_key_auth,
mcp_servers=mcp_servers,
client_ip=client_ip,
)
# Pre-fetch OAuth credentials only when at least one server uses OAuth2,
@ -1643,12 +1856,13 @@ if MCP_AVAILABLE:
)
return filtered_tools
except MCPUpstreamAuthError:
# Surface upstream 401/403 to the outer handler so the
# client receives a proper WWW-Authenticate challenge
# instead of a silently empty tool list. Without this
# re-raise the broad ``except Exception`` below would
# swallow the auth error.
raise
# Absorb so one unauthenticated server does not empty every other server's
# tools. Surfacing the upstream 401 to the client as a re-auth challenge is
# intentionally not done here: raising from this list handler cannot produce a
# 401 + WWW-Authenticate (the MCP session manager serializes it as a JSON-RPC
# error), so that belongs in a request-scope preemptive check, tracked separately.
verbose_logger.debug(f"MCP list_tools: omitting {server.name}; it needs upstream auth")
return []
except Exception as e:
verbose_logger.exception(f"Error getting tools from server {server.name}: {str(e)}")
return []
@ -1687,7 +1901,9 @@ if MCP_AVAILABLE:
end_time = datetime.now()
try:
await litellm_logging_obj.async_success_handler(
result=all_tools,
result=[
tool.model_dump(mode="json") if isinstance(tool, MCPTool) else tool for tool in all_tools
],
start_time=list_tools_start_time,
end_time=end_time,
)
@ -1967,6 +2183,7 @@ if MCP_AVAILABLE:
raw_headers: Optional[Dict[str, str]] = None,
log_list_tools_to_spendlogs: bool = False,
list_tools_log_source: Optional[str] = None,
client_ip: Optional[str] = None,
) -> List[MCPTool]:
"""
List all available MCP tools.
@ -1976,6 +2193,7 @@ if MCP_AVAILABLE:
mcp_auth_header: Optional auth header for MCP server (deprecated)
mcp_servers: Optional list of server names/aliases to filter by
mcp_server_auth_headers: Optional dict of server-specific auth headers {server_alias: auth_value}
client_ip: Client IP for IP-based server access control
Returns:
List[MCPTool]: Combined list of tools from all accessible servers
@ -1999,6 +2217,7 @@ if MCP_AVAILABLE:
raw_headers=raw_headers,
log_list_tools_to_spendlogs=log_list_tools_to_spendlogs,
list_tools_log_source=list_tools_log_source,
client_ip=client_ip,
)
verbose_logger.debug(f"Successfully fetched {len(managed_tools)} tools from managed MCP servers")
except Exception as e:
@ -2526,6 +2745,22 @@ if MCP_AVAILABLE:
return response
async def _fire_mcp_success_logging(
logging_obj: LiteLLMLoggingObj,
result: Any,
start_time: datetime,
end_time: datetime,
) -> None:
logging_obj.post_call(original_response=result)
await logging_obj.async_post_mcp_tool_call_hook(
kwargs=logging_obj.model_call_details,
response_obj=result,
start_time=start_time,
end_time=end_time,
)
logging_obj.call_type = CallTypes.call_mcp_tool.value
await logging_obj.async_success_handler(result=result, start_time=start_time, end_time=end_time)
@client
async def call_mcp_tool(
name: str,
@ -2597,16 +2832,7 @@ if MCP_AVAILABLE:
raise
if litellm_logging_obj:
litellm_logging_obj.post_call(original_response=response)
end_time = datetime.now()
await litellm_logging_obj.async_post_mcp_tool_call_hook(
kwargs=litellm_logging_obj.model_call_details,
response_obj=response,
start_time=start_time,
end_time=end_time,
)
litellm_logging_obj.call_type = CallTypes.call_mcp_tool.value
await litellm_logging_obj.async_success_handler(result=response, start_time=start_time, end_time=end_time)
await _fire_mcp_success_logging(litellm_logging_obj, response, start_time, datetime.now())
return response
async def mcp_get_prompt(

157
litellm/proxy/_experimental/mcp_server/tool_search.py Normal file
View file

@ -0,0 +1,157 @@
from __future__ import annotations
import json
from datetime import datetime
from typing import TYPE_CHECKING, Any, Optional
if TYPE_CHECKING:
from mcp.types import CallToolResult
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
MCP_TOOL_SEARCH_TOOL_NAME: str = "mcp_tool_search"
MCP_TOOL_CALL_TOOL_NAME: str = "mcp_tool_call"
def coerce_top_k(value: Any, default: int = 5) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def search_tools(query: str, tools: list[dict[str, Any]], top_k: int = 5) -> list[dict[str, Any]]:
if not query:
return []
tokens = query.lower().split()
def _score(tool: dict[str, Any]) -> int:
haystack = (tool.get("name", "") + " " + tool.get("description", "")).lower()
return sum(1 for t in tokens if t in haystack)
scored = ((s, tool) for tool in tools if (s := _score(tool)) > 0)
return [tool for _, tool in sorted(scored, key=lambda x: x[0], reverse=True)[:top_k]]
def get_virtual_tool_definitions() -> list[dict[str, Any]]:
return [
{
"name": MCP_TOOL_SEARCH_TOOL_NAME,
"description": "Search for MCP tools by keyword. Returns top matching tools with names, descriptions, and input schemas.",
"inputSchema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Keywords to search for in tool names and descriptions.",
},
"top_k": {
"type": "integer",
"description": "Maximum number of results to return.",
"default": 5,
},
},
"required": ["query"],
},
},
{
"name": MCP_TOOL_CALL_TOOL_NAME,
"description": "Call an MCP tool by name with the given arguments.",
"inputSchema": {
"type": "object",
"properties": {
"tool_name": {
"type": "string",
"description": "The exact name of the MCP tool to call.",
},
"arguments": {
"type": "object",
"description": "Arguments to pass to the tool.",
},
},
"required": ["tool_name"],
},
},
]
async def handle_mcp_tool_search(
query: str,
top_k: int,
user_api_key_dict: UserAPIKeyAuth,
client_ip: Optional[str] = None,
mcp_servers: Optional[list[str]] = None,
mcp_auth_header: Optional[str] = None,
mcp_server_auth_headers: Optional[dict[str, dict[str, str]]] = None,
oauth2_headers: Optional[dict[str, str]] = None,
raw_headers: Optional[dict[str, str]] = None,
) -> CallToolResult:
from mcp.types import CallToolResult, TextContent
from litellm.proxy._experimental.mcp_server.server import _list_mcp_tools
mcp_tools = await _list_mcp_tools(
user_api_key_auth=user_api_key_dict,
mcp_servers=mcp_servers,
client_ip=client_ip,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
)
tools = [
{
"name": t.name,
"description": t.description or "",
"inputSchema": t.inputSchema,
}
for t in mcp_tools
]
results = search_tools(query, tools, top_k)
return CallToolResult(content=[TextContent(type="text", text=json.dumps(results))], isError=False)
async def handle_mcp_tool_call(
tool_name: str,
arguments: dict[str, Any],
user_api_key_dict: UserAPIKeyAuth,
client_ip: Optional[str] = None,
mcp_servers: Optional[list[str]] = None,
mcp_auth_header: Optional[str] = None,
mcp_server_auth_headers: Optional[dict[str, dict[str, str]]] = None,
oauth2_headers: Optional[dict[str, str]] = None,
raw_headers: Optional[dict[str, str]] = None,
litellm_logging_obj: Optional[LiteLLMLoggingObj] = None,
) -> CallToolResult:
from litellm.proxy._experimental.mcp_server.server import (
_get_allowed_mcp_servers,
execute_mcp_tool,
)
allowed_mcp_servers = await _get_allowed_mcp_servers(
user_api_key_auth=user_api_key_dict,
mcp_servers=mcp_servers,
client_ip=client_ip,
)
# Reject before dispatch when the key has no accessible servers; otherwise an
# unprefixed local tool name would fall through to the local registry in
# execute_mcp_tool, which has no server permission check.
if not allowed_mcp_servers:
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="User not allowed to call this tool.")
return await execute_mcp_tool(
name=tool_name,
arguments=arguments,
allowed_mcp_servers=allowed_mcp_servers,
start_time=datetime.now(),
user_api_key_auth=user_api_key_dict,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)

1
litellm/proxy/_experimental/out/404.html Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/404/index.html Normal file
View file

File diff suppressed because one or more lines are too long

9
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.__PAGE__.txt Normal file
View file

@ -0,0 +1,9 @@
1:"$Sreact.fragment"
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ClientPageRoot"]
3:I[871135,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","/litellm-asset-prefix/_next/static/chunks/0whkizop7gd0~.js","/litellm-asset-prefix/_next/static/chunks/0-ih8xcz_89nt.js","/litellm-asset-prefix/_next/static/chunks/0pd5zl~lciww9.js","/litellm-asset-prefix/_next/static/chunks/02ihc5xweq16v.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/0mzw3maijoev6.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/04amwk-x_vjxu.js","/litellm-asset-prefix/_next/static/chunks/0-dhh1_d1.b1u.js","/litellm-asset-prefix/_next/static/chunks/0pwkd9r.mc_ee.js","/litellm-asset-prefix/_next/static/chunks/011mgw.-67gs_.js","/litellm-asset-prefix/_next/static/chunks/0~-ovi6c4wjt1.js","/litellm-asset-prefix/_next/static/chunks/0_y-b9_d9dsuv.js","/litellm-asset-prefix/_next/static/chunks/0c2apcdkbqq0o.js","/litellm-asset-prefix/_next/static/chunks/0zrbitbm~0koh.js","/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","/litellm-asset-prefix/_next/static/chunks/0ngre0.s4-ej6.js","/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","/litellm-asset-prefix/_next/static/chunks/05t1k89l9tc3s.js","/litellm-asset-prefix/_next/static/chunks/17n.qg70cy9.9.js","/litellm-asset-prefix/_next/static/chunks/00q4mtjboprhm.js","/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","/litellm-asset-prefix/_next/static/chunks/0-3i_.uof35pm.js","/litellm-asset-prefix/_next/static/chunks/14566-_ogh-19.js","/litellm-asset-prefix/_next/static/chunks/0w39dn9x3dp9g.js","/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","/litellm-asset-prefix/_next/static/chunks/0v1rxqc1hqmrl.js","/litellm-asset-prefix/_next/static/chunks/0c4pfjjue0uc-.js"],"default"]
6:I[897367,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"OutletBoundary"]
7:"$Sreact.suspense"
0:{"rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/011mgw.-67gs_.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0~-ovi6c4wjt1.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0_y-b9_d9dsuv.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/0c2apcdkbqq0o.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0zrbitbm~0koh.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/0ngre0.s4-ej6.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/05t1k89l9tc3s.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/17n.qg70cy9.9.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/00q4mtjboprhm.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/0-3i_.uof35pm.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/14566-_ogh-19.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/0w39dn9x3dp9g.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/0v1rxqc1hqmrl.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/0c4pfjjue0uc-.js","async":true}]],["$","$L6",null,{"children":["$","$7",null,{"name":"Next.MetadataOutlet","children":"$@8"}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}
4:{}
5:"$0:rsc:props:children:0:props:serverProvidedParams:params"
8:null

7
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.txt Normal file
View file

@ -0,0 +1,7 @@
1:"$Sreact.fragment"
2:I[92825,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ClientSegmentRoot"]
3:I[216370,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","/litellm-asset-prefix/_next/static/chunks/0whkizop7gd0~.js","/litellm-asset-prefix/_next/static/chunks/0-ih8xcz_89nt.js","/litellm-asset-prefix/_next/static/chunks/0pd5zl~lciww9.js","/litellm-asset-prefix/_next/static/chunks/02ihc5xweq16v.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/0mzw3maijoev6.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/04amwk-x_vjxu.js","/litellm-asset-prefix/_next/static/chunks/0-dhh1_d1.b1u.js","/litellm-asset-prefix/_next/static/chunks/0pwkd9r.mc_ee.js"],"default"]
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
0:{"rsc":["$","$1","c",{"children":[[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0whkizop7gd0~.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0-ih8xcz_89nt.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0pd5zl~lciww9.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/02ihc5xweq16v.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/0mzw3maijoev6.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/04amwk-x_vjxu.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/0-dhh1_d1.b1u.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/0pwkd9r.mc_ee.js","async":true}]],["$","$L2",null,{"Component":"$3","slots":{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]},"serverProvidedParams":{"params":{},"promises":["$@6"]}}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}
6:"$0:rsc:props:children:1:props:serverProvidedParams:params"

30
litellm/proxy/_experimental/out/__next._full.txt Normal file
View file

File diff suppressed because one or more lines are too long

6
litellm/proxy/_experimental/out/__next._head.txt Normal file
View file

@ -0,0 +1,6 @@
1:"$Sreact.fragment"
2:I[897367,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ViewportBoundary"]
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"MetadataBoundary"]
4:"$Sreact.suspense"
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"IconMark"]
0:{"rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.0~dgapwhi~75y.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"/get_favicon"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}

9
litellm/proxy/_experimental/out/__next._index.txt Normal file
View file

@ -0,0 +1,9 @@
1:"$Sreact.fragment"
2:I[867271,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
3:I[71195,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
4:I[557951,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"AuthProvider"]
5:I[339756,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
6:I[837457,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
:HL["/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/0i77.0u.82o9u.css","style"]
0:{"rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/0i77.0u.82o9u.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","template":["$","$L6",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}

4
litellm/proxy/_experimental/out/__next._tree.txt Normal file
View file

@ -0,0 +1,4 @@
:HL["/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/0i77.0u.82o9u.css","style"]
:HL["/litellm-asset-prefix/_next/static/media/83afe278b6a6bb3c-s.p.0q-301v4kxxnr.woff2","font",{"crossOrigin":"","type":"font/woff2"}]
0:{"tree":{"name":"","param":null,"prefetchHints":16,"slots":{"children":{"name":"(dashboard)","param":null,"prefetchHints":0,"slots":{"children":{"name":"__PAGE__","param":null,"prefetchHints":0,"slots":null}}}}},"staleTime":300,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}

16
litellm/proxy/_experimental/out/_next/static/5rDiFx0t_mOGYmV_8kSkw/_buildManifest.js Normal file
View file

@ -0,0 +1,16 @@
self.__BUILD_MANIFEST = {
"__rewrites": {
"afterFiles": [],
"beforeFiles": [
{
"source": "/litellm-asset-prefix/_next/:path+",
"destination": "/_next/:path+"
}
],
"fallback": []
},
"sortedPages": [
"/_app",
"/_error"
]
};self.__BUILD_MANIFEST_CB && self.__BUILD_MANIFEST_CB()

1
litellm/proxy/_experimental/out/_next/static/5rDiFx0t_mOGYmV_8kSkw/_clientMiddlewareManifest.js Normal file
View file

@ -0,0 +1 @@
self.__MIDDLEWARE_MATCHERS = [];self.__MIDDLEWARE_MATCHERS_CB && self.__MIDDLEWARE_MATCHERS_CB()

1
litellm/proxy/_experimental/out/_next/static/5rDiFx0t_mOGYmV_8kSkw/_ssgManifest.js Normal file
View file

@ -0,0 +1 @@
self.__SSG_MANIFEST=new Set([]);self.__SSG_MANIFEST_CB&&self.__SSG_MANIFEST_CB()

2
litellm/proxy/_experimental/out/_next/static/chunks/0-3i_.uof35pm.js Normal file
View file

File diff suppressed because one or more lines are too long

12
litellm/proxy/_experimental/out/_next/static/chunks/0-4tg9f~_a3b~.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-85n.4jrc2vv.js Normal file
View file

File diff suppressed because one or more lines are too long

3
litellm/proxy/_experimental/out/_next/static/chunks/0-dhh1_d1.b1u.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-f.2po-pctaa.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-ih8xcz_89nt.js Normal file
View file

File diff suppressed because one or more lines are too long

143
litellm/proxy/_experimental/out/_next/static/chunks/0.4.bbjx7y007.js Normal file
View file

File diff suppressed because one or more lines are too long

10
litellm/proxy/_experimental/out/_next/static/chunks/0.bx44y-6~tug.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0.yiw37jc_bvi.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/00cy3g~l27g1y.js Normal file
View file

File diff suppressed because one or more lines are too long

420
litellm/proxy/_experimental/out/_next/static/chunks/00jwo~_zp.35~.js Normal file
View file

File diff suppressed because one or more lines are too long

3
litellm/proxy/_experimental/out/_next/static/chunks/00p.gft-l.6p..js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/00pl5r0.xdcua.js Normal file
View file

File diff suppressed because one or more lines are too long

4
litellm/proxy/_experimental/out/_next/static/chunks/00q4mtjboprhm.js Normal file
View file

File diff suppressed because one or more lines are too long

10
litellm/proxy/_experimental/out/_next/static/chunks/011mgw.-67gs_.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/01_xjyxcb1uco.js Normal file
View file

File diff suppressed because one or more lines are too long

3
litellm/proxy/_experimental/out/_next/static/chunks/01xm1xt.gmrff.js Normal file
View file

File diff suppressed because one or more lines are too long

4
litellm/proxy/_experimental/out/_next/static/chunks/01y._o853f7le.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/01~uswbzv7_90.js Normal file
View file

File diff suppressed because one or more lines are too long

4
litellm/proxy/_experimental/out/_next/static/chunks/022.sz94ycw4x.js Normal file
View file

File diff suppressed because one or more lines are too long

8
litellm/proxy/_experimental/out/_next/static/chunks/02813b2b-kz98.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/02c1-r_khzb89.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/02ihc5xweq16v.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/02nrwvikmd-wf.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/02oicwo.~e~ak.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/036wlkuzplhfz.js Normal file
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more