scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 36

Merge pull request #32027 from BerriAI/litellm_internal_staging
Some checks failed
CodeQL / Analyze (actions) (push) Has been cancelled
Details
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
Details
CodeQL / Analyze (python) (push) Has been cancelled
Details
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Details
Helm unit test / unit-test (push) Has been cancelled
Details
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Details
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
Details

Browse source 
chore(ci): promote internal staging to main
This commit is contained in:
yuneng-jiang 2026-07-03 12:47:22 -07:00 • committed by GitHub
commit badc1414d3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1182 changed files with 43152 additions and 7038 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.git-blame-ignore-revs
View file

@ -13,7 +13,7 @@
7edf3a9cb55548b143df1692f4ed7c4681d7fcf7
# style: reformat litellm/ with ruff format (#31317)
430b5b8f1b12dc261a49fda99ac5d1b22381a428
17bfd415aeb5a57fb646b5cc67da1c730aa7c50b
# style: unify ruff format width on 120 (#31518)
3dfbeabe626d203ac9de86024519d9a96c484ce4
48b5a5a0cc5a694a11219416ee0b6eb6e620e74e

4
.github/workflows/codspeed.yml vendored
View file

@ -4,11 +4,9 @@ on:
push:
branches:
- main
- litellm_internal_staging
pull_request:
branches:
- main
- litellm_internal_staging
# Allow CodSpeed to trigger backtest performance analysis
# in order to generate initial data
workflow_dispatch:
@ -23,7 +21,7 @@ concurrency:
jobs:
benchmarks:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:

32
.github/workflows/create-release.yml vendored
View file

@ -122,10 +122,28 @@ jobs:
makeLatest = (!latestVersion || isAtLeast(newVersion, latestVersion)) ? "true" : "false";
}
try {
await github.rest.git.createRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: `refs/tags/${tag}`,
sha: commitHash,
});
} catch (error) {
if (error.status !== 422) throw error;
const existing = await github.rest.git.getRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: `tags/${tag}`,
});
if (existing.data.object.sha !== commitHash) {
throw new Error(`Tag ${tag} already exists at ${existing.data.object.sha}, expected ${commitHash}`);
}
}
const response = await github.rest.repos.createRelease({
draft: true,
generate_release_notes: true,
target_commitish: commitHash,
name: tag,
owner: context.repo.owner,
prerelease: isPrerelease,
@ -138,11 +156,21 @@ jobs:
owner: context.repo.owner,
repo: context.repo.repo,
release_id: response.data.id,
tag_name: tag,
body: updatedBody,
draft: false,
make_latest: makeLatest,
});
if (!isPrerelease) {
await github.rest.repos.updateRelease({
owner: context.repo.owner,
repo: context.repo.repo,
release_id: response.data.id,
tag_name: tag,
make_latest: makeLatest,
});
}
} catch (error) {
core.setFailed(error.message);
}

11
.gitignore vendored
View file

@ -50,6 +50,8 @@ litellm/proxy/tests/package-lock.json
ui/litellm-dashboard/.next
ui/litellm-dashboard/node_modules
ui/litellm-dashboard/next-env.d.ts
ui/litellm-dashboard/package.json
ui/litellm-dashboard/package-lock.json
deploy/charts/litellm/*.tgz
deploy/charts/litellm/charts/*
deploy/charts/*.tgz
@ -85,12 +87,17 @@ litellm/proxy/db/migrations/*
litellm/proxy/migrations/*config.yaml
litellm/proxy/migrations/*
litellm/proxy/to_delete_loadtest_work/*
config.yaml
tests/litellm/litellm_core_utils/llm_cost_calc/log.txt
tests/test_custom_dir/*
test.py
litellm_config.yaml
!.github/observatory/litellm_config.yaml
.cursor
litellm/proxy/to_delete_loadtest_work/*
update_model_cost_map.py
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py
scripts/test_vertex_ai_search.py
LAZY_LOADING_IMPROVEMENTS.md
STABILIZATION_TODO.md
@ -124,6 +131,4 @@ crash.*.log
# pytest coverage data
.coverage
# _experimental/out UI build output
# (both componentized and non-componentized build the UI on project release)
litellm/proxy/_experimental/out/
ui/litellm-dashboard/out/

9
CLAUDE.md
View file

@ -17,6 +17,8 @@ Same thing for bug fixes. The tests should make it so that this specific bug can
`tests/test_litellm/` mirrors `litellm/` in a parallel path (see `tests/test_litellm/readme.md`). Name tests `test_<filename>.py`, but always match the existing test file in the directory you touch — many provider dirs use longer descriptive names (e.g. `test_anthropic_chat_transformation.py`) to avoid ambiguity across sibling folders. For bug fixes, extend the existing mapped test file rather than creating a new one. Only create a new test file for a new feature (provider, endpoint, or transformation module) that has no mapped test yet, following that directory's naming convention (or `test_<filename>.py` if you're the first test there). One focused regression test beats many shallow ones
End-to-end tests belong in `tests/e2e/` and must follow the harness conventions documented in that directory's `CLAUDE.md`
When creating PRs, don't set base to `main`. `litellm_internal_staging` serves that purpose
Always use @.github/pull_request_template.md as a guide for your PR body
@ -33,9 +35,11 @@ If you ever make public-facing PR descriptions, comments, issues, commit message
Don't hesitate to use values in .env to get needed API keys and other secrets, as long as you never add them to conversation history, commit them, or include them in GitHub issues / PRs
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed
Python max line length is 120, not 88
When you fix violations gated by `ruff-strict-budget.json` or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered baselines so the ceilings ratchet down instead of leaving stale headroom
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed. It only runs when there are staged frontend and/or backend changes and calculates violations, generates types, etc. based on the worktree, so stage what you need or stash/delete unwanted files in litellm/ or ui/ (where backend and frontend lint run, respectively) before running it. If it fails because dashboard api types are stale, it already regenerated them for you. You just need to stage the schema.d.ts, re-run `make pre-commit` to confirm it passes, and commit
When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-budget.json`, or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered limits so the ceilings ratchet down instead of leaving stale headroom. It measures the working tree, so it must contain exactly the fixes you're committing
If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing `reportAny` / `reportExplicitAny` closer to their basedpyright ceilings, just validate it in the caller with Pydantic (a model or `TypeAdapter` that returns the typed thing or raises will do) and then pass the now typed variable in
@ -69,6 +73,7 @@ Follow these coding conventions for new/updated code (a three-line fix in a lega
- No monster files or god objects
- No file sprawl: deliberate file and folder structure
- Standard over hand-rolled: use the official SDK or a library where one exists; where none does, follow industry standards instead of inventing local conventions
- API-fragmentation-aware: when logic must branch on which API surface produced or consumes data (e.g. chat completions vs Anthropic Messages vs Responses API shapes), proactively look for an existing shared helper (e.g. `litellm_core_utils/prompt_templates/factory.py`) before writing per-surface parsing in the new module; if none exists, add one there instead of duplicating the same format-detection logic in every new guardrail/integration
Follow conventional commits for commit names and PR titles

58
Makefile
View file

@ -4,8 +4,8 @@
.PHONY: help test test-unit test-unit-llms test-unit-proxy-guardrails test-unit-proxy-core test-unit-proxy-misc \
test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
info lint lint-dev format \
lint-basedpyright lint-basedpyright-budget-update \
info lint lint-dev lint-checks format \
lint-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
install-dev install-proxy-dev install-test-deps install-hooks \
install-helm-unittest check-circular-imports check-import-safety pre-commit \
@ -27,12 +27,12 @@ help:
@echo " make lint - Run all linting (Ruff, basedpyright, format check, circular imports, import safety)"
@echo " make lint-ruff - Run Ruff linting only"
@echo " make lint-basedpyright - Run basedpyright strict, gated by per-rule error counts"
@echo " make lint-basedpyright-budget-update - Re-capture the basedpyright per-rule budget (ratchet)"
@echo " make lint-basedpyright-budget-update - Ratchet basedpyright limits down by what this branch fixed"
@echo " make lint-format - Check ruff format formatting (matches CI)"
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its ceiling"
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its limit"
@echo " make lint-gate - Strict ruff gate in CI-parity mode (fetches staging, simulates the merge)"
@echo " make lint-ruff-budget-update - Re-capture per-rule baselines in ruff-strict-budget.json (ratchet)"
@echo " make lint-budget-update - Re-capture all ratchet budgets (ruff + basedpyright)"
@echo " make lint-ruff-budget-update - Ratchet ruff-strict-budget.json limits down by what this branch fixed"
@echo " make lint-budget-update - Ratchet all budgets down (ruff + type-discipline + basedpyright)"
@echo " make check-circular-imports - Check for circular imports"
@echo " make check-import-safety - Check import safety"
@echo " make test - Run all tests"
@ -53,6 +53,11 @@ help:
UV := uv
UV_RUN := $(UV) run --no-sync
LINT_DEP_INSTALL ?= install-dev
LINT_DEP_BASE ?= lint-fetch-base
LINT_JOBS := $(shell sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4)
LINT_OUTPUT_SYNC := $(if $(filter output-sync,$(.FEATURES)),--output-sync=target,)
# Show info
info:
@echo "UV: $(UV)"
@ -88,7 +93,7 @@ install-hooks:
# Formatting
# Wrap width is ruff.toml's single source of truth (line-length = 120), shared by the
# formatter, E501, and the import sorter so there's no 88-vs-120 split to reconcile.
# formatter and the import sorter so there's no 88-vs-120 split to reconcile.
format: install-dev
cd litellm && $(UV_RUN) ruff format --exclude '/enterprise/' . && cd ..
@ -107,12 +112,12 @@ lint-fetch-base:
# running proxy need.
lint-install:
$(UV) sync --inexact --frozen --group proxy-dev
$(UV_RUN) prisma generate --schema litellm/proxy/schema.prisma
$(UV_RUN) python scripts/prisma_generate_if_needed.py
# Diff-scoped format check, identical to test-linting.yml's "Check ruff format" step:
# only the litellm Python files changed vs the base are checked, so a pre-existing
# format issue elsewhere doesn't block an unrelated commit.
lint-format-check-changed: install-dev lint-fetch-base
lint-format-check-changed: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
@files=$$(git diff --name-only origin/litellm_internal_staging...HEAD -- 'litellm/**/*.py' | grep -v '^litellm/enterprise/' || true); \
if [ -z "$$files" ]; then \
echo "No changed litellm Python files to format-check."; \
@ -121,7 +126,7 @@ lint-format-check-changed: install-dev lint-fetch-base
fi
# Linting targets
lint-ruff: install-dev
lint-ruff: $(LINT_DEP_INSTALL)
cd litellm && $(UV_RUN) ruff check . && cd ..
# faster linter for developing ...
@ -156,15 +161,17 @@ lint-ruff-FULL-dev: install-dev
if [ -n "$$files" ]; then echo "$$files" | xargs $(UV_RUN) ruff check; \
else echo "No changed .py files to check."; fi
lint-basedpyright: install-dev lint-fetch-base
lint-basedpyright: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --base origin/litellm_internal_staging
# Type-discipline budget (mutable collections / casts / type guards / kwargs /
# unexplained suppressions), the test-linting.yml step `make lint` used to omit.
lint-type-discipline: install-dev lint-fetch-base
lint-type-discipline: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
$(UV_RUN) python scripts/type_discipline_gate.py --base origin/litellm_internal_staging
lint-basedpyright-budget-update: install-dev
# --update lowers each limit by what this branch fixed since its branch point, so
# it needs the base ref fetched to resolve the merge-base.
lint-basedpyright-budget-update: install-dev lint-fetch-base
($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --update
lint-format: format-check
@ -174,19 +181,22 @@ lint-ruff-budget: install-dev
# Strict gate, invoked the same way CI does in test-linting.yml so a local pass
# means the CI check will pass too.
lint-gate: install-dev lint-fetch-base
lint-gate: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
$(UV_RUN) python scripts/ruff_strict_gate.py --base origin/litellm_internal_staging
lint-ruff-budget-update: install-dev
lint-ruff-budget-update: install-dev lint-fetch-base
$(UV_RUN) python scripts/ruff_strict_gate.py --update
# Ratchet all budgets in one shot (ruff strict + basedpyright)
lint-budget-update: lint-ruff-budget-update lint-basedpyright-budget-update
lint-type-discipline-budget-update: install-dev lint-fetch-base
$(UV_RUN) python scripts/type_discipline_gate.py --update
check-circular-imports: install-dev
# Ratchet all budgets in one shot (ruff strict + type-discipline + basedpyright)
lint-budget-update: lint-ruff-budget-update lint-type-discipline-budget-update lint-basedpyright-budget-update
check-circular-imports: $(LINT_DEP_INSTALL)
cd litellm && $(UV_RUN) python ../tests/documentation_tests/test_circular_imports.py && cd ..
check-import-safety: install-dev
check-import-safety: $(LINT_DEP_INSTALL)
@$(UV_RUN) python -c "from litellm import *; print('[from litellm import *] OK! no issues!');" || (echo '🚨 import failed, this means you introduced unprotected imports! 🚨'; exit 1)
# Combined linting, isomorphic to test-linting.yml's lint job so a local pass means a
@ -194,9 +204,13 @@ check-import-safety: install-dev
# runs the diff-scoped ruff format check, whole-tree ruff check, the strict-rule /
# type-discipline / basedpyright budgets as a delta vs the base, then the circular-import
# and import-safety checks. Steps that compare against the base resolve it the same way CI
# does (merge-base with origin/litellm_internal_staging). lint-install is first so the
# Prisma client exists before basedpyright runs.
lint: lint-install lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright check-circular-imports check-import-safety
# does (merge-base with origin/litellm_internal_staging). Setup (env sync, Prisma client,
# base fetch) runs once up front; the checks themselves are independent, so a sub-make
# fans them out with -j and the fast ones finish under basedpyright's shadow.
lint: lint-install lint-fetch-base
$(MAKE) -j $(LINT_JOBS) $(LINT_OUTPUT_SYNC) LINT_DEP_INSTALL= LINT_DEP_BASE= lint-checks
lint-checks: lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright check-circular-imports check-import-safety
# Faster linting for local development (only checks changed code)
lint-dev: lint-format-changed check-circular-imports check-import-safety

144
basedpyright-code-budget.json
View file

@ -1,194 +1,146 @@
{
"reportAny": {
"baseline": 24989,
"slack": 2500
"limit": 37484
},
"reportArgumentType": {
"baseline": 1814,
"slack": 180
"limit": 2721
},
"reportAssignmentType": {
"baseline": 220,
"slack": 22
"limit": 330
},
"reportAttributeAccessIssue": {
"baseline": 346,
"slack": 35
"limit": 519
},
"reportCallIssue": {
"baseline": 87,
"slack": 10
"limit": 131
},
"reportConstantRedefinition": {
"baseline": 39,
"slack": 4
"limit": 59
},
"reportDeprecated": {
"baseline": 217,
"slack": 22
"limit": 326
},
"reportDuplicateImport": {
"baseline": 28,
"slack": 3
"limit": 42
},
"reportExplicitAny": {
"baseline": 6931,
"slack": 700
"limit": 10397
},
"reportFunctionMemberAccess": {
"baseline": 7,
"slack": 3
"limit": 11
},
"reportGeneralTypeIssues": {
"baseline": 151,
"slack": 15
"limit": 227
},
"reportIncompatibleMethodOverride": {
"baseline": 52,
"slack": 5
"limit": 78
},
"reportIncompatibleVariableOverride": {
"baseline": 8,
"slack": 3
"limit": 12
},
"reportInconsistentOverload": {
"baseline": 12,
"slack": 3
"limit": 18
},
"reportIndexIssue": {
"baseline": 26,
"slack": 3
"limit": 39
},
"reportInvalidTypeForm": {
"baseline": 23,
"slack": 3
"limit": 35
},
"reportInvalidTypeVarUse": {
"baseline": 2,
"slack": 3
"limit": 5
},
"reportMatchNotExhaustive": {
"baseline": 1,
"slack": 0
"limit": 2
},
"reportMissingParameterType": {
"baseline": 3933,
"slack": 390
"limit": 5900
},
"reportMissingTypeArgument": {
"baseline": 10612,
"slack": 1000
"limit": 15918
},
"reportMissingTypeStubs": {
"baseline": 27,
"slack": 10
"limit": 41
},
"reportOperatorIssue": {
"baseline": 6,
"slack": 3
"limit": 9
},
"reportOptionalCall": {
"baseline": 4,
"slack": 3
"limit": 7
},
"reportOptionalIterable": {
"baseline": 3,
"slack": 3
"limit": 6
},
"reportOptionalMemberAccess": {
"baseline": 724,
"slack": 72
"limit": 1086
},
"reportOptionalOperand": {
"baseline": 3,
"slack": 3
"limit": 6
},
"reportOptionalSubscript": {
"baseline": 11,
"slack": 3
"limit": 17
},
"reportPossiblyUnboundVariable": {
"baseline": 52,
"slack": 10
"limit": 78
},
"reportPrivateUsage": {
"baseline": 1625,
"slack": 160
"limit": 2438
},
"reportRedeclaration": {
"baseline": 8,
"slack": 3
"limit": 12
},
"reportReturnType": {
"baseline": 126,
"slack": 100
"limit": 226
},
"reportTypedDictNotRequiredAccess": {
"baseline": 20,
"slack": 3
"limit": 30
},
"reportUndefinedVariable": {
"baseline": 2,
"slack": 3
"limit": 5
},
"reportUnknownArgumentType": {
"baseline": 30603,
"slack": 3000
"limit": 45905
},
"reportUnknownLambdaType": {
"baseline": 75,
"slack": 10
"limit": 113
},
"reportUnknownMemberType": {
"baseline": 27037,
"slack": 2500
"limit": 40556
},
"reportUnknownParameterType": {
"baseline": 13612,
"slack": 1000
"limit": 20418
},
"reportUnknownVariableType": {
"baseline": 21445,
"slack": 2000
"limit": 32168
},
"reportUnnecessaryCast": {
"baseline": 118,
"slack": 10
"limit": 177
},
"reportUnnecessaryComparison": {
"baseline": 683,
"slack": 100
"limit": 1025
},
"reportUnnecessaryContains": {
"baseline": 4,
"slack": 3
"limit": 7
},
"reportUnnecessaryIsInstance": {
"baseline": 808,
"slack": 80
"limit": 1212
},
"reportUntypedBaseClass": {
"baseline": 110,
"slack": 11
"limit": 165
},
"reportUntypedFunctionDecorator": {
"baseline": 22,
"slack": 3
"limit": 33
},
"reportUnusedClass": {
"baseline": 22,
"slack": 3
"limit": 33
},
"reportUnusedFunction": {
"baseline": 137,
"slack": 10
"limit": 206
},
"reportUnusedImport": {
"baseline": 670,
"slack": 50
"limit": 1005
},
"reportUnusedVariable": {
"baseline": 865,
"slack": 50
"limit": 1297
}
}

2
docker/build_admin_ui.sh
View file

@ -57,8 +57,6 @@ source ~/.nvm/nvm.sh
nvm install v18.17.0
nvm use v18.17.0
# copy _enterprise.json from this directory to /ui/litellm-dashboard, and rename it to ui_colors.json
cp enterprise/enterprise_ui/enterprise_colors.json ui/litellm-dashboard/ui_colors.json
# cd in to /ui/litellm-dashboard
cd ui/litellm-dashboard

233
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -13,6 +13,8 @@ from litellm.constants import (
)
if TYPE_CHECKING:
from litellm.integrations.prometheus import PrometheusLogger
from litellm.proxy._types import LiteLLM_ManagedObjectTable
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
@ -26,6 +28,7 @@ class CheckBatchCost:
proxy_logging_obj: "ProxyLogging",
prisma_client: "PrismaClient",
llm_router: "Router",
track_unmanaged_vertex_batch_cost: bool = False,
):
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
@ -33,6 +36,7 @@ class CheckBatchCost:
self.proxy_logging_obj: ProxyLogging = proxy_logging_obj
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router
self._track_unmanaged_vertex_batch_cost = track_unmanaged_vertex_batch_cost
# Cached after the first poll cycle. Once we know the column is absent we skip
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
@ -97,6 +101,182 @@ class CheckBatchCost:
order={"created_at": "asc"},
)
@staticmethod
def _record_error(
prom_logger: Optional["PrometheusLogger"], error_type: str
) -> None:
if prom_logger is not None:
prom_logger.record_check_batch_cost_error(error_type)
def _resolve_job_routing(
self,
job: "LiteLLM_ManagedObjectTable",
prom_logger: Optional["PrometheusLogger"],
) -> Optional[Tuple[str, str]]:
"""
Resolve (model_id, batch_id) for a managed-object row, where model_id is a router
deployment id and batch_id is the raw provider batch id.
Managed batches encode both in a base64 unified id. Unmanaged Vertex batches, created with
a raw gs:// input_file_id, store the raw provider job id as unified_object_id; when
track_unmanaged_vertex_batch_cost is enabled the model is derived from the gs:// path and
mapped to a configured vertex_ai deployment. Returns None (recording a metric) when the row
can't be routed.
"""
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
get_batch_id_from_unified_batch_id,
get_model_id_from_unified_batch_id,
)
unified_object_id = job.unified_object_id
decoded = _is_base64_encoded_unified_file_id(unified_object_id)
if decoded:
model_id = get_model_id_from_unified_batch_id(decoded)
if model_id is None:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid model id"
)
self._record_error(prom_logger, "invalid_model_id")
return None
return model_id, get_batch_id_from_unified_batch_id(decoded)
if self._track_unmanaged_vertex_batch_cost:
return self._resolve_unmanaged_vertex_routing(job, prom_logger)
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid unified object id"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
def _resolve_unmanaged_vertex_routing(
self,
job: "LiteLLM_ManagedObjectTable",
prom_logger: Optional["PrometheusLogger"],
) -> Optional[Tuple[str, str]]:
from litellm.llms.vertex_ai.batches.transformation import (
VertexAIBatchTransformation,
)
input_file_id = self._get_input_file_id(job)
if not VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
input_file_id
):
verbose_proxy_logger.info(
f"Skipping job {job.unified_object_id}: not an unmanaged vertex batch "
"(no gs:// input_file_id with a publishers/ model path)"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
assert input_file_id is not None # narrowed by is_unmanaged_gcs_batch_input_file_id
bare_model_name = VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
input_file_id
)
deployment_id = self._get_vertex_ai_deployment_id_for_bare_model(
bare_model_name
)
if deployment_id is None:
verbose_proxy_logger.info(
f"Skipping unmanaged vertex batch {job.unified_object_id}: no vertex_ai "
f"deployment configured for model {bare_model_name}"
)
self._record_error(prom_logger, "unmanaged_no_matching_deployment")
return None
return deployment_id, job.unified_object_id
def _get_vertex_ai_deployment_id_for_bare_model(
self, bare_model_name: str
) -> Optional[str]:
model_group = self.llm_router.resolve_model_name_from_model_id(bare_model_name)
deployment_id = (
self._get_vertex_ai_deployment_id(model_group) if model_group else None
)
if deployment_id is not None:
return deployment_id
return self._get_vertex_ai_deployment_id_from_matching_deployments(
bare_model_name
)
def _get_vertex_ai_deployment_id_from_matching_deployments(
self, bare_model_name: str
) -> Optional[str]:
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
for deployment in self.llm_router.get_model_list(model_name=None) or []:
litellm_params = deployment.get("litellm_params") or {}
actual_model = litellm_params.get("model")
if not isinstance(actual_model, str):
continue
if not self._is_bare_model_match(actual_model, bare_model_name):
continue
try:
_, llm_provider, _, _ = get_llm_provider(
model=actual_model,
custom_llm_provider=litellm_params.get("custom_llm_provider"),
)
except Exception:
continue
if llm_provider != "vertex_ai":
continue
model_info = deployment.get("model_info") or {}
deployment_id = model_info.get("id")
if isinstance(deployment_id, str):
return deployment_id
return None
@staticmethod
def _is_bare_model_match(actual_model: str, bare_model_name: str) -> bool:
return (
actual_model == bare_model_name
or actual_model.endswith(f"/{bare_model_name}")
or actual_model.endswith(f":{bare_model_name}")
)
def _get_vertex_ai_deployment_id(self, model_group: str) -> Optional[str]:
"""
Returns the first deployment id for `model_group` whose provider is vertex_ai,
skipping deployments from other providers that happen to share the model group name.
"""
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
for deployment_id in self.llm_router.get_model_ids(model_name=model_group):
deployment_info = self.llm_router.get_deployment(model_id=deployment_id)
if deployment_info is None:
continue
try:
_, llm_provider, _, _ = get_llm_provider(
model=deployment_info.litellm_params.model,
custom_llm_provider=deployment_info.litellm_params.custom_llm_provider,
)
except Exception:
continue
if llm_provider == "vertex_ai":
return deployment_id
return None
@staticmethod
def _get_input_file_id(job: "LiteLLM_ManagedObjectTable") -> Optional[str]:
import json
from litellm.types.utils import LiteLLMBatch
file_object = job.file_object
if isinstance(file_object, str):
try:
file_object = json.loads(file_object)
except (json.JSONDecodeError, ValueError):
return None
if not isinstance(file_object, dict):
return None
try:
return LiteLLMBatch.model_validate(file_object).input_file_id
except Exception:
return None
async def check_batch_cost(self):
"""
Check if the batch JOB has been tracked.
@ -114,8 +294,6 @@ class CheckBatchCost:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
get_batch_id_from_unified_batch_id,
get_model_id_from_unified_batch_id,
)
try:
@ -172,31 +350,10 @@ class CheckBatchCost:
else:
jobs = await self._fallback_find_jobs()
for job in jobs:
# get the model from the job
unified_object_id = job.unified_object_id
decoded_unified_object_id = _is_base64_encoded_unified_file_id(
unified_object_id
)
if not decoded_unified_object_id:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid unified object id"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("invalid_unified_id")
continue
else:
unified_object_id = decoded_unified_object_id
model_id = get_model_id_from_unified_batch_id(unified_object_id)
batch_id = get_batch_id_from_unified_batch_id(unified_object_id)
if model_id is None:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid model id"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("invalid_model_id")
routing = self._resolve_job_routing(job, prom_logger)
if routing is None:
continue
model_id, batch_id = routing
verbose_proxy_logger.info(
f"Querying model ID: {model_id} for cost and usage of batch ID: {batch_id}"
@ -213,7 +370,7 @@ class CheckBatchCost:
)
except Exception as e:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because of error querying model ID: {model_id} for cost and usage of batch ID: {batch_id}: {e}"
f"Skipping job {job.unified_object_id} because of error querying model ID: {model_id} for cost and usage of batch ID: {batch_id}: {e}"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("provider_retrieval_error")
@ -287,7 +444,7 @@ class CheckBatchCost:
deployment_info = self.llm_router.get_deployment(model_id=model_id)
if deployment_info is None:
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid deployment info"
f"Skipping job {job.unified_object_id} because it is not a valid deployment info"
)
if prom_logger:
prom_logger.record_check_batch_cost_error("deployment_not_found")
@ -413,6 +570,26 @@ class CheckBatchCost:
f"CheckBatchCost: failed to mark job {job.id} complete in DB: {db_err}"
)
elif response.status in ("failed", "expired", "cancelled"):
try:
update_data = {
"status": response.status,
"file_object": response.model_dump_json(),
}
if self._has_batch_processed_column:
update_data["batch_processed"] = True
await self.prisma_client.db.litellm_managedobjecttable.update(
where={"id": job.id},
data=update_data,
)
verbose_proxy_logger.info(
f"CheckBatchCost: marked job {job.id} as {response.status} in DB"
)
except Exception as db_err:
verbose_proxy_logger.error(
f"CheckBatchCost: failed to mark job {job.id} as {response.status} in DB: {db_err}"
)
# Record polling run metrics (always, even if nothing was processed)
if prom_logger:
prom_logger.record_check_batch_cost_run(

16
enterprise/litellm_enterprise/proxy/hooks/managed_files.py
View file

@ -125,23 +125,33 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"team_id": user_api_key_dict.team_id,
"updated_by": user_api_key_dict.user_id,
}
update_data = {
"model_mappings": json.dumps(model_mappings),
"flat_model_file_ids": list(model_mappings.values()),
"updated_by": user_api_key_dict.user_id,
}
if file_object is not None:
db_data["file_object"] = file_object.model_dump_json()
file_object_json = file_object.model_dump_json()
db_data["file_object"] = file_object_json
update_data["file_object"] = file_object_json
# Extract storage metadata from hidden params if present
hidden_params = getattr(file_object, "_hidden_params", {}) or {}
if "storage_backend" in hidden_params:
db_data["storage_backend"] = hidden_params["storage_backend"]
update_data["storage_backend"] = hidden_params["storage_backend"]
if "storage_url" in hidden_params:
db_data["storage_url"] = hidden_params["storage_url"]
update_data["storage_url"] = hidden_params["storage_url"]
verbose_logger.debug(
f"Storage metadata: storage_backend={db_data.get('storage_backend')}, "
f"storage_url={db_data.get('storage_url')}"
)
result = await self.prisma_client.db.litellm_managedfiletable.create(
data=db_data
result = await self.prisma_client.db.litellm_managedfiletable.upsert(
where={"unified_file_id": file_id},
data={"create": db_data, "update": update_data},
)
verbose_logger.debug(
f"LiteLLM Managed File object with id={file_id} stored in db: {result}"

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.45"
version = "0.1.46"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.45"
version = "0.1.46"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

2
litellm-proxy-extras/litellm_proxy_extras/migrations/20260626120000_add_mcp_tool_search_enabled/migration.sql Normal file
View file

@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "LiteLLM_ObjectPermissionTable" ADD COLUMN "mcp_tool_search_enabled" BOOLEAN;

2
litellm-proxy-extras/litellm_proxy_extras/migrations/20260629000000_add_max_concurrent_requests_to_mcp_server_table/migration.sql Normal file
View file

@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "max_concurrent_requests" INTEGER;

5
litellm-proxy-extras/litellm_proxy_extras/migrations/20260630190000_add_budget_fallbacks_to_litellm_verification_token/migration.sql Normal file
View file

@ -0,0 +1,5 @@
-- AlterTable
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "budget_fallbacks" JSONB NOT NULL DEFAULT '{}';
-- AlterTable
ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN IF NOT EXISTS "budget_fallbacks" JSONB NOT NULL DEFAULT '{}';

4
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -279,6 +279,7 @@ model LiteLLM_ObjectPermissionTable {
blocked_tools String[] @default([]) // Tool names blocked for any key/team/user with this permission
mcp_toolsets String[] @default([]) // Toolset IDs granted to this key/team/user
search_tools String[] @default([]) // search_tool_name values this key/team/user may call
mcp_tool_search_enabled Boolean?
teams LiteLLM_TeamTable[]
projects LiteLLM_ProjectTable[]
verification_tokens LiteLLM_VerificationToken[]
@ -337,6 +338,7 @@ model LiteLLM_MCPServerTable {
byok_api_key_help_url String?
source_url String?
timeout Float?
max_concurrent_requests Int?
// BYOM submission lifecycle
approval_status String? @default("active")
submitted_by String?
@ -417,6 +419,7 @@ model LiteLLM_VerificationToken {
access_group_ids String[] @default([])
model_spend Json @default("{}")
model_max_budget Json @default("{}")
budget_fallbacks Json @default("{}")
budget_id String?
organization_id String?
object_permission_id String?
@ -510,6 +513,7 @@ model LiteLLM_DeletedVerificationToken {
access_group_ids String[] @default([])
model_spend Json @default("{}")
model_max_budget Json @default("{}")
budget_fallbacks Json @default("{}")
router_settings Json? @default("{}")
budget_id String?
organization_id String?

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.74"
version = "0.4.75"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.74"
version = "0.4.75"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

11
litellm/__init__.py
View file

@ -263,6 +263,8 @@ azure_key: Optional[str] = None
anthropic_key: Optional[str] = None
replicate_key: Optional[str] = None
bytez_key: Optional[str] = None
gdc_key: Optional[str] = None
gdc_api_base: Optional[str] = None
cohere_key: Optional[str] = None
infinity_key: Optional[str] = None
clarifai_key: Optional[str] = None
@ -586,6 +588,7 @@ gemini_models: Set = set()
xai_models: Set = set()
zai_models: Set = set()
deepseek_models: Set = set()
tencent_models: Set = set()
runwayml_models: Set = set()
azure_ai_models: Set = set()
jina_ai_models: Set = set()
@ -799,6 +802,8 @@ def add_known_models(model_cost_map: Optional[Dict] = None):
fal_ai_models.add(key)
elif value.get("litellm_provider") == "deepseek":
deepseek_models.add(key)
elif value.get("litellm_provider") == "tencent":
tencent_models.add(key)
elif value.get("litellm_provider") == "runwayml":
runwayml_models.add(key)
elif value.get("litellm_provider") == "meta_llama":
@ -1091,6 +1096,7 @@ models_by_provider: dict = {
"zai": zai_models,
"fal_ai": fal_ai_models,
"deepseek": deepseek_models,
"tencent": tencent_models,
"runwayml": runwayml_models,
"mistral": mistral_chat_models,
"azure_ai": azure_ai_models,
@ -1787,6 +1793,7 @@ if TYPE_CHECKING:
from .llms.nvidia_nim.embed import (
NvidiaNimEmbeddingConfig as NvidiaNimEmbeddingConfig,
)
from .llms.gdc.chat.transformation import GDCGeminiConfig as GDCGeminiConfig
# Type stubs for lazy-loaded config instances
openaiOSeriesConfig: OpenAIOSeriesConfig
@ -1801,6 +1808,9 @@ if TYPE_CHECKING:
from .llms.deepseek.chat.transformation import (
DeepSeekChatConfig as _DeepSeekChatConfig,
)
from .llms.tencent.chat.transformation import (
TencentChatConfig as _TencentChatConfig,
)
from .llms.sap.chat.transformation import (
GenAIHubOrchestrationConfig as _GenAIHubOrchestrationConfig,
)
@ -1843,6 +1853,7 @@ if TYPE_CHECKING:
# Type stubs for lazy-loaded config classes (to help mypy understand types)
VLLMConfig: Type[_VLLMConfig]
DeepSeekChatConfig: Type[_DeepSeekChatConfig]
TencentChatConfig: Type[_TencentChatConfig]
GenAIHubOrchestrationConfig: Type[_GenAIHubOrchestrationConfig]
GenAIHubEmbeddingConfig: Type[_GenAIHubEmbeddingConfig]
AzureOpenAIO1Config: Type[_AzureOpenAIO1Config]

7
litellm/_lazy_imports_registry.py
View file

@ -284,6 +284,7 @@ LLM_CONFIG_NAMES = (
"LiteLLMProxyChatConfig",
"VLLMConfig",
"DeepSeekChatConfig",
"TencentChatConfig",
"LMStudioChatConfig",
"LmStudioEmbeddingConfig",
"NscaleConfig",
@ -323,6 +324,7 @@ LLM_CONFIG_NAMES = (
"SnowflakeEmbeddingConfig",
"AmazonNovaChatConfig",
"SonioxAudioTranscriptionConfig",
"GDCGeminiConfig",
)
# Types that support lazy loading via _lazy_import_types
@ -1095,6 +1097,7 @@ _LLM_CONFIGS_IMPORT_MAP = {
),
"VLLMConfig": (".llms.vllm.completion.transformation", "VLLMConfig"),
"DeepSeekChatConfig": (".llms.deepseek.chat.transformation", "DeepSeekChatConfig"),
"TencentChatConfig": (".llms.tencent.chat.transformation", "TencentChatConfig"),
"LMStudioChatConfig": (".llms.lm_studio.chat.transformation", "LMStudioChatConfig"),
"LmStudioEmbeddingConfig": (
".llms.lm_studio.embed.transformation",
@ -1157,6 +1160,10 @@ _LLM_CONFIGS_IMPORT_MAP = {
".llms.dashscope.chat.transformation",
"DashScopeChatConfig",
),
"GDCGeminiConfig": (
".llms.gdc.chat.transformation",
"GDCGeminiConfig",
),
"ModelScopeChatConfig": (
".llms.modelscope.chat.transformation",
"ModelScopeChatConfig",

30
litellm/a2a_protocol/main.py
View file

@ -129,6 +129,33 @@ def _set_agent_id_on_logging_obj(
litellm_logging_obj.model_call_details["agent_id"] = agent_id
_A2A_COST_PARAM_KEYS = ("cost_per_query", "input_cost_per_token", "output_cost_per_token")
def _set_litellm_params_on_logging_obj(
kwargs: dict[str, Any],
litellm_params: dict[str, Any],
) -> None:
"""
Merge the agent's pricing params into model_call_details["litellm_params"]
so A2ACostCalculator can read them.
The non-streaming path reuses the proxy-built logging object, whose
litellm_params already carries metadata / proxy_server_request / user-key
context, so merge the pricing keys in rather than replacing the dict.
"""
logging_obj = kwargs.get("litellm_logging_obj")
if logging_obj is None:
return
cost_params = {key: litellm_params[key] for key in _A2A_COST_PARAM_KEYS if litellm_params.get(key) is not None}
if not cost_params:
return
existing = logging_obj.model_call_details.get("litellm_params") or {}
logging_obj.model_call_details["litellm_params"] = {**existing, **cost_params}
def _get_a2a_model_info(a2a_client: Any, kwargs: Dict[str, Any]) -> str:
"""
Extract agent info and set model/custom_llm_provider for cost tracking.
@ -477,6 +504,9 @@ async def asend_message(
completion_tokens=completion_tokens,
)
# Merge agent pricing params into the logging obj so cost is calculated
_set_litellm_params_on_logging_obj(kwargs=kwargs, litellm_params=litellm_params)
# Set agent_id on logging obj for SpendLogs tracking
_set_agent_id_on_logging_obj(kwargs=kwargs, agent_id=agent_id)

7
litellm/a2a_protocol/utils.py
View file

@ -121,8 +121,13 @@ class A2ARequestUtils:
Returns:
Tuple of (prompt_tokens, completion_tokens, total_tokens)
"""
# Count input tokens
# Count input tokens. Dump the message to a dict first so extraction hits
# the dict branch — request-side parts are a2a-sdk Part RootModels whose
# kind/text live on part.root, which the object branch cannot read. This
# mirrors how the response side already works (it operates on model_dump).
input_message = A2ARequestUtils.get_input_message_from_request(request)
if input_message is not None and hasattr(input_message, "model_dump"):
input_message = input_message.model_dump(mode="json")
input_text = A2ARequestUtils.extract_text_from_message(input_message)
prompt_tokens = A2ARequestUtils.count_tokens(input_text)

3
litellm/constants.py
View file

@ -460,6 +460,7 @@ LITELLM_CHAT_PROVIDERS = [
"openai",
"openai_like",
"bytez",
"gdc",
"xai",
"custom_openai",
"text-completion-openai",
@ -507,6 +508,7 @@ LITELLM_CHAT_PROVIDERS = [
"text-completion-codestral",
"text-completion-inception",
"deepseek",
"tencent",
"sambanova",
"maritalk",
"cloudflare",
@ -728,6 +730,7 @@ openai_compatible_providers: List = [
"volcengine",
"codestral",
"deepseek",
"tencent",
"deepinfra",
"perplexity",
"xinference",

5
litellm/cost_calculator.py
View file

@ -52,6 +52,9 @@ from litellm.llms.databricks.cost_calculator import (
from litellm.llms.deepseek.cost_calculator import (
cost_per_token as deepseek_cost_per_token,
)
from litellm.llms.tencent.cost_calculator import (
cost_per_token as tencent_cost_per_token,
)
from litellm.llms.fireworks_ai.cost_calculator import (
cost_per_token as fireworks_ai_cost_per_token,
)
@ -625,6 +628,8 @@ def cost_per_token(
return gemini_cost_per_token(model=model, usage=usage_block, service_tier=service_tier)
elif custom_llm_provider == "deepseek":
return deepseek_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "tencent":
return tencent_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "perplexity":
return perplexity_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "xai":

6
litellm/exceptions.py
View file

@ -1165,12 +1165,18 @@ class ModifyResponseException(Exception):
request_data: Dict[str, Any],
guardrail_name: Optional[str] = None,
detection_info: Optional[Dict[str, Any]] = None,
original_response: Optional[Any] = None,
):
self.message = message
self.model = model
self.request_data = request_data
self.guardrail_name = guardrail_name
self.detection_info = detection_info or {}
# The LLM response that was blocked (post-call). Carries the real token
# usage the upstream call consumed, so the synthetic block response can
# report it instead of discarding it. None for pre-call blocks (the LLM
# was never invoked).
self.original_response = original_response
super().__init__(message)

97
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -1,9 +1,12 @@
"""
This hook is used to inject cache control directives into the messages of a chat completion.
This hook is used to inject cache control directives into messages.
Users can define
- `cache_control_injection_points` in the completion params and litellm will inject the cache control directives into the messages at the specified injection points.
Supported for both `v1/chat/completions` (via the prompt-management hook) and
`v1/messages` (via `apply_to_anthropic_messages_request`).
"""
import copy
@ -225,6 +228,98 @@ class AnthropicCacheControlHook(CustomPromptManagement):
message_content[-1]["cache_control"] = control # type: ignore
return message
@staticmethod
def apply_to_anthropic_messages_request(
messages: List[Dict],
system: str | list | None,
injection_points: List[CacheControlInjectionPoint],
) -> Tuple[List[Dict], str | list | None, List[CacheControlInjectionPoint]]:
"""Apply cache control injection for the Anthropic-native v1/messages endpoint.
Returns (messages, system, remaining_non_message_points).
"""
if not injection_points:
return messages, system, []
processed_messages: List[Dict] = copy.deepcopy(messages)
processed_system = copy.deepcopy(system) if system is not None else None
message_points: List[CacheControlMessageInjectionPoint] = []
system_points: List[CacheControlMessageInjectionPoint] = []
remaining_points: List[CacheControlInjectionPoint] = []
for point in injection_points:
if point.get("location") == "message":
msg_point = cast(CacheControlMessageInjectionPoint, point)
if msg_point.get("role") == "system":
system_points.append(msg_point)
else:
message_points.append(msg_point)
else:
remaining_points.append(point)
reserved_blocks = 1 if any(p.get("location") == "tool_config" for p in remaining_points) else 0
max_blocks = MAX_CACHE_CONTROL_BLOCKS - reserved_blocks
used_blocks = sum(
AnthropicCacheControlHook._count_cache_control_blocks(cast(AllMessageValues, msg))
for msg in processed_messages
)
if isinstance(processed_system, list):
used_blocks += sum(
1 for b in processed_system if isinstance(b, dict) and b.get("cache_control") is not None
)
if system_points and processed_system is not None and used_blocks < max_blocks:
system_already_has_cc = isinstance(processed_system, list) and any(
isinstance(b, dict) and b.get("cache_control") is not None for b in processed_system
)
if not system_already_has_cc:
control = system_points[0].get("control") or ChatCompletionCachedContent(type="ephemeral")
if isinstance(processed_system, str):
processed_system = [{"type": "text", "text": processed_system, "cache_control": control}]
used_blocks += 1
elif len(processed_system) > 0 and isinstance(processed_system[-1], dict):
processed_system[-1] = {**processed_system[-1], "cache_control": control}
used_blocks += 1
for i, msg in enumerate(processed_messages):
content = msg.get("content")
if isinstance(content, str):
processed_messages[i] = {**msg, "content": [{"type": "text", "text": content}]}
processed_messages = AnthropicCacheControlHook._apply_message_injections(
points=message_points,
messages=cast(List[AllMessageValues], processed_messages),
max_blocks=max_blocks - used_blocks,
)
return processed_messages, processed_system, remaining_points
@staticmethod
def maybe_inject_cache_control(
messages: List[Dict],
system: str | list | None,
kwargs: Dict[str, Any],
) -> Tuple[List[Dict], str | list | None]:
"""Extract cache_control_injection_points from kwargs and apply if present.
Pops the key from kwargs; if remaining (non-message) points exist they
are written back so downstream transforms can handle them.
"""
injection_points = kwargs.pop("cache_control_injection_points", None)
if not injection_points:
return messages, system
messages, system, remaining = AnthropicCacheControlHook.apply_to_anthropic_messages_request(
messages=messages,
system=system,
injection_points=injection_points,
)
if remaining:
kwargs["cache_control_injection_points"] = remaining
return messages, system
@property
def integration_name(self) -> str:
"""Return the integration name for this hook."""

6
litellm/integrations/azure_sentinel/azure_sentinel.py
View file

@ -61,13 +61,15 @@ class AzureSentinelLogger(CustomBatchLogger):
client_secret (str, optional): Azure Client Secret for OAuth2 authentication.
If not provided, will use AZURE_SENTINEL_CLIENT_SECRET or AZURE_CLIENT_SECRET env var.
audit_stream_name (str, optional): Stream name from DCR for audit logs.
If not provided, audit logs use the standard stream name.
If not provided, will use AZURE_SENTINEL_AUDIT_STREAM_NAME env var or the standard stream name.
"""
self.async_httpx_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.LoggingCallback)
resolved_dcr_immutable_id = dcr_immutable_id or os.getenv("AZURE_SENTINEL_DCR_IMMUTABLE_ID")
resolved_stream_name = stream_name or os.getenv("AZURE_SENTINEL_STREAM_NAME") or "Custom-LiteLLM"
resolved_audit_stream_name = audit_stream_name or resolved_stream_name
resolved_audit_stream_name = (
audit_stream_name or os.getenv("AZURE_SENTINEL_AUDIT_STREAM_NAME") or resolved_stream_name
)
resolved_endpoint = endpoint or os.getenv("AZURE_SENTINEL_ENDPOINT")
resolved_tenant_id = tenant_id or os.getenv("AZURE_SENTINEL_TENANT_ID") or os.getenv("AZURE_TENANT_ID")
resolved_client_id = client_id or os.getenv("AZURE_SENTINEL_CLIENT_ID") or os.getenv("AZURE_CLIENT_ID")

71
litellm/integrations/code_interpreter_interception/handler.py
View file

@ -40,9 +40,11 @@ from litellm.types.utils import (
LITELLM_CODE_EXECUTION_TOOL_NAME = "litellm_code_execution"
_INTERCEPTION_ACTIVE_KEY = "_code_interpreter_interception_active"
_SANDBOX_KEY = "_code_interpreter_interception_sandbox_key"
_SESSION_SCOPED_KEY = "_code_interpreter_interception_session_scoped"
_CONVERTED_STREAM_KEY = "_code_interpreter_interception_converted_stream"
_LITELLM_METADATA_KEY = "litellm_metadata"
_CACHE_TTL_SECONDS = 15 * 60
_SESSION_SCOPED_PER_IDENTITY_CAP = 10
class CodeExecutionToolCall(TypedDict, total=False):
@ -107,6 +109,20 @@ class ChatCompletionFunctionToolChoice(TypedDict):
CodeExecutionFunctionToolChoice = ResponsesFunctionToolChoice | ChatCompletionFunctionToolChoice
def _extract_session_id(kwargs: dict[str, Any]) -> str | None:
for meta_key in ("metadata", "litellm_metadata"):
meta = kwargs.get(meta_key)
if isinstance(meta, dict):
sid = meta.get("session_id")
if sid and isinstance(sid, str):
return sid
return None
def _extract_identity(kwargs: dict[str, Any]) -> str:
return kwargs.get("user_api_key_hash") or ""
def _resolve_sandbox_tool(sandbox_tool_name: str | None) -> dict[str, Any] | None:
try:
from litellm.sandbox.sandbox_tools import resolve_sandbox_tool
@ -140,7 +156,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
self.enabled_providers = enabled_providers
self.sandbox_tool_name = sandbox_tool_name
self.sandbox_config = sandbox_config
self._container_cache: dict[str, tuple[Any, dict[str, Any] | None, float]] = {}
self._container_cache: dict[str, tuple[Any, dict[str, Any] | None, float, str | None]] = {}
@classmethod
def from_config_yaml(cls, config: CodeInterpreterInterceptionConfig) -> "CodeInterpreterInterceptionLogger":
@ -191,7 +207,13 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
return None
kwargs[_INTERCEPTION_ACTIVE_KEY] = True
kwargs[_SANDBOX_KEY] = uuid.uuid4().hex
session_id = _extract_session_id(kwargs)
if session_id:
identity = _extract_identity(kwargs)
kwargs[_SANDBOX_KEY] = f"{identity}:{session_id}" if identity else session_id
kwargs[_SESSION_SCOPED_KEY] = True
else:
kwargs[_SANDBOX_KEY] = uuid.uuid4().hex
if kwargs.get("stream"):
kwargs["stream"] = False
kwargs[_CONVERTED_STREAM_KEY] = True
@ -217,6 +239,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
if not is_interception_internal_key(key)
and not key.startswith("_agentic_loop")
and key != "max_agentic_loops"
and key != _SESSION_SCOPED_KEY
}
if filtered_metadata:
kwargs[_LITELLM_METADATA_KEY] = filtered_metadata
@ -227,7 +250,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
def _write_interception_metadata(kwargs: dict[str, Any]) -> None:
metadata = kwargs.get(_LITELLM_METADATA_KEY)
metadata = dict(metadata) if isinstance(metadata, dict) else {}
for key in (_INTERCEPTION_ACTIVE_KEY, _SANDBOX_KEY, _CONVERTED_STREAM_KEY):
for key in (_INTERCEPTION_ACTIVE_KEY, _SANDBOX_KEY, _SESSION_SCOPED_KEY, _CONVERTED_STREAM_KEY):
if key in kwargs:
metadata[key] = kwargs[key]
kwargs[_LITELLM_METADATA_KEY] = metadata
@ -347,7 +370,9 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
await self._prune_expired_cache()
tool_calls = cast(list[CodeExecutionToolCall], tools.get("tool_calls", []))
sandbox_key = kwargs.get(_SANDBOX_KEY)
container, params = await self._get_or_create_container(cache_key=sandbox_key)
is_session = bool(kwargs.get(_SESSION_SCOPED_KEY))
identity = _extract_identity(kwargs) if is_session else None
container, params = await self._get_or_create_container(cache_key=sandbox_key, identity=identity)
try:
container_id = cast(str | None, getattr(container, "id", None))
@ -404,6 +429,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
metadata={
"tool_type": "code_interpreter",
"sandbox_key": sandbox_key or "",
"is_session_scoped": bool(kwargs.get(_SESSION_SCOPED_KEY)),
"code_interpreter_calls": code_interpreter_calls,
},
)
@ -419,7 +445,9 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
await self._prune_expired_cache()
tool_calls = cast(list[CodeExecutionToolCall], tools.get("tool_calls", []))
sandbox_key = cast(str | None, kwargs.get(_SANDBOX_KEY))
container, params = await self._get_or_create_container(cache_key=sandbox_key)
is_session = bool(kwargs.get(_SESSION_SCOPED_KEY))
identity = _extract_identity(cast(dict[str, Any], kwargs)) if is_session else None
container, params = await self._get_or_create_container(cache_key=sandbox_key, identity=identity)
try:
container_id = cast(str | None, getattr(container, "id", None))
@ -455,6 +483,7 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
metadata={
"tool_type": "code_interpreter",
"sandbox_key": sandbox_key or "",
"is_session_scoped": bool(kwargs.get(_SESSION_SCOPED_KEY)),
"code_interpreter_calls": code_interpreter_calls,
"response_format": "openai",
},
@ -489,6 +518,8 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
async def async_agentic_loop_cleanup_hook(self, plan: AgenticLoopPlan, kwargs: dict) -> None:
metadata = plan.metadata or {} if plan else {}
if metadata.get("is_session_scoped"):
return
await self._delete_container_for_cache_key(metadata.get("sandbox_key"))
@staticmethod
@ -520,7 +551,8 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
async def async_post_agentic_loop_response_hook(self, response: Any, plan: AgenticLoopPlan, kwargs: dict) -> Any:
metadata = plan.metadata or {} if plan else {}
await self._delete_container_for_cache_key(metadata.get("sandbox_key"))
if not metadata.get("is_session_scoped"):
await self._delete_container_for_cache_key(metadata.get("sandbox_key"))
calls = metadata.get("code_interpreter_calls")
if not calls:
@ -565,17 +597,32 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
return f"[execution error] {message}"
return getattr(result, "stdout", "") or ""
async def _get_or_create_container(self, cache_key: str | None) -> tuple[Any, dict[str, Any] | None]:
async def _get_or_create_container(
self,
cache_key: str | None,
identity: str | None = None,
) -> tuple[Any, dict[str, Any] | None]:
if cache_key:
cached = self._container_cache.get(cache_key)
if cached is not None:
self._container_cache[cache_key] = (cached[0], cached[1], time.time(), cached[3])
return cached[0], cached[1]
container, params = await self._create_container()
if cache_key:
self._container_cache[cache_key] = (container, params, time.time())
if identity is not None:
await self._evict_lru_session_if_over_cap(identity)
self._container_cache[cache_key] = (container, params, time.time(), identity)
return container, params
async def _evict_lru_session_if_over_cap(self, identity: str) -> None:
identity_entries = [(k, v) for k, v in self._container_cache.items() if v[3] == identity]
if len(identity_entries) < _SESSION_SCOPED_PER_IDENTITY_CAP:
return
lru_key, lru_entry = min(identity_entries, key=lambda item: item[1][2])
self._container_cache.pop(lru_key, None)
await self._delete_container(container=lru_entry[0], params=lru_entry[1])
async def _create_container(self) -> tuple[Any, dict[str, Any] | None]:
if self.sandbox_config is not None:
return await self.sandbox_config.acreate_sandbox(), None
@ -739,12 +786,8 @@ class CodeInterpreterInterceptionLogger(CustomLogger):
now = time.time()
expired = [
(cache_key, container, params)
for cache_key, (
container,
params,
created_at,
) in self._container_cache.items()
if now - created_at > _CACHE_TTL_SECONDS
for cache_key, (container, params, last_accessed, *_) in self._container_cache.items()
if now - last_accessed > _CACHE_TTL_SECONDS
]
for cache_key, container, params in expired:
self._container_cache.pop(cache_key, None)

111
litellm/integrations/prometheus.py
View file

@ -4,6 +4,7 @@
from __future__ import annotations
import asyncio
import math
import os
import sys
from datetime import datetime, timedelta
@ -65,6 +66,26 @@ if TYPE_CHECKING:
else:
AsyncIOScheduler = Any
_DEFAULT_BUDGET_METRICS_PER_REQUEST_TIMEOUT = 5.0
def _get_budget_metrics_per_request_timeout() -> float:
raw = os.getenv("PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT")
if raw is None:
return _DEFAULT_BUDGET_METRICS_PER_REQUEST_TIMEOUT
try:
parsed = float(raw)
except ValueError:
parsed = None
if parsed is None or not math.isfinite(parsed) or parsed <= 0:
verbose_logger.debug(
"[Non-Blocking] Prometheus: invalid PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT=%r; using default %ss.",
raw,
_DEFAULT_BUDGET_METRICS_PER_REQUEST_TIMEOUT,
)
return _DEFAULT_BUDGET_METRICS_PER_REQUEST_TIMEOUT
return parsed
class PrometheusLogger(CustomLogger):
# Class variables or attributes
@ -593,6 +614,21 @@ class PrometheusLogger(CustomLogger):
labelnames=[],
)
########################################
# MCP Tool Call Metrics
########################################
self.litellm_mcp_tool_calls_total = self._counter_factory(
name="litellm_mcp_tool_calls_total",
documentation="Total MCP tool calls, segmented by tool and server name",
labelnames=self.get_labels_for_metric("litellm_mcp_tool_calls_total"),
)
self.litellm_mcp_tool_call_spend_metric = self._counter_factory(
name="litellm_mcp_tool_call_spend_metric",
documentation="Total spend on MCP tool calls, segmented by tool and server name",
labelnames=self.get_labels_for_metric("litellm_mcp_tool_call_spend_metric"),
)
except Exception as e:
print_verbose(f"Got exception on init prometheus client {str(e)}")
raise e
@ -1300,6 +1336,13 @@ class PrometheusLogger(CustomLogger):
label_context=label_context,
)
# MCP tool call metrics
self._increment_mcp_tool_call_metrics(
standard_logging_payload=standard_logging_payload,
enum_values=enum_values,
response_cost=response_cost,
)
# increment litellm_proxy_total_requests_metric for all successful requests
# (both streaming and non-streaming) in this single location to prevent
# double-counting that occurs when async_post_call_success_hook also increments
@ -1521,6 +1564,49 @@ class PrometheusLogger(CustomLogger):
amount=float(provider_cache_creation_tokens),
)
def _increment_mcp_tool_call_metrics(
self,
standard_logging_payload: StandardLoggingPayload,
enum_values: UserAPIKeyLabelValues,
response_cost: float,
) -> None:
metadata = standard_logging_payload.get("metadata")
if not isinstance(metadata, dict):
return
mcp_meta = metadata.get("mcp_tool_call_metadata")
if not isinstance(mcp_meta, dict):
return
mcp_enum_values = UserAPIKeyLabelValues(
mcp_tool_name=mcp_meta.get("name"),
mcp_server_name=mcp_meta.get("mcp_server_name"),
hashed_api_key=enum_values.hashed_api_key,
api_key_alias=enum_values.api_key_alias,
team=enum_values.team,
team_alias=enum_values.team_alias,
user=enum_values.user,
end_user=enum_values.end_user,
)
mcp_label_context = PrometheusLabelFactoryContext(mcp_enum_values)
PrometheusLogger._inc_labeled_counter(
self,
self.litellm_mcp_tool_calls_total,
"litellm_mcp_tool_calls_total",
mcp_enum_values,
label_context=mcp_label_context,
)
if response_cost > 0:
PrometheusLogger._inc_labeled_counter(
self,
self.litellm_mcp_tool_call_spend_metric,
"litellm_mcp_tool_call_spend_metric",
mcp_enum_values,
label_context=mcp_label_context,
amount=response_cost,
)
async def _increment_remaining_budget_metrics(
self,
user_api_team: Optional[str],
@ -1542,7 +1628,15 @@ class PrometheusLogger(CustomLogger):
_user_spend = _metadata.get("user_api_key_user_spend", None)
_user_max_budget = _metadata.get("user_api_key_user_max_budget", None)
results = await asyncio.gather(
# Bound the per-request budget-metric emission so that slow Redis/DB
# lookups under load cannot consume the whole LoggingWorker watchdog
# (LOGGING_WORKER_MAX_TIME_PER_COROUTINE, default 20s) and get the entire
# success-logging event cancelled. Budget gauges are also refreshed by the
# periodic cron every PROMETHEUS_BUDGET_METRICS_REFRESH_INTERVAL_MINUTES,
# so dropping one slow per-request emission only loses sub-cron real-time
# detail, not correctness.
budget_metrics_timeout = _get_budget_metrics_per_request_timeout()
gather_coro = asyncio.gather(
self._set_api_key_budget_metrics_after_api_request(
user_api_key=user_api_key,
user_api_key_alias=user_api_key_alias,
@ -1569,6 +1663,16 @@ class PrometheusLogger(CustomLogger):
),
return_exceptions=True,
)
try:
results = await asyncio.wait_for(gather_coro, timeout=budget_metrics_timeout)
except asyncio.TimeoutError:
verbose_logger.debug(
"[Non-Blocking] Prometheus: per-request budget metric emission "
"exceeded %ss under load; skipping (values are refreshed by the "
"periodic budget-metrics cron job).",
budget_metrics_timeout,
)
return
for i, r in enumerate(results):
if isinstance(r, Exception):
verbose_logger.debug(
@ -3804,6 +3908,10 @@ def _get_combined_custom_metadata_from_standard_logging_payload(
) -> Dict[str, Any]:
"""
Combine the metadata sources that can supply custom Prometheus labels.
Includes top-level scalar fields from the standard logging metadata (e.g.
user_api_key_project_alias, user_api_key_team_alias) so they are accessible
via custom_prometheus_metadata_labels configuration.
"""
if not isinstance(standard_logging_payload, dict):
return {}
@ -3817,6 +3925,7 @@ def _get_combined_custom_metadata_from_standard_logging_payload(
spend_logs_metadata = standard_logging_metadata.get("spend_logs_metadata")
return {
**{k: v for k, v in standard_logging_metadata.items() if not isinstance(v, dict)},
**(requester_metadata if isinstance(requester_metadata, dict) else {}),
**(user_api_key_auth_metadata if isinstance(user_api_key_auth_metadata, dict) else {}),
**(spend_logs_metadata if isinstance(spend_logs_metadata, dict) else {}),

25
litellm/integrations/s3_v2.py
View file

@ -54,6 +54,7 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
s3_strip_base64_files: bool = False,
s3_use_key_prefix: bool = False,
s3_use_virtual_hosted_style: bool = False,
s3_server_side_encryption: Optional[str] = None,
s3_callback_params_override: Optional[dict] = None,
**kwargs,
):
@ -92,6 +93,7 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
s3_strip_base64_files=s3_strip_base64_files,
s3_use_key_prefix=s3_use_key_prefix,
s3_use_virtual_hosted_style=s3_use_virtual_hosted_style,
s3_server_side_encryption=s3_server_side_encryption,
)
verbose_logger.debug(f"s3 logger using endpoint url {s3_endpoint_url}")
@ -145,6 +147,7 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
s3_strip_base64_files: bool = False,
s3_use_key_prefix: bool = False,
s3_use_virtual_hosted_style: bool = False,
s3_server_side_encryption: Optional[str] = None,
params_source: Optional[dict] = None,
):
"""
@ -194,6 +197,8 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
bool(params.get("s3_use_virtual_hosted_style", False)) or s3_use_virtual_hosted_style
)
self.s3_server_side_encryption = params.get("s3_server_side_encryption") or s3_server_side_encryption
return
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
@ -273,6 +278,7 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
async def async_upload_data_to_s3(self, batch_logging_element: s3BatchLoggingElement):
try:
import base64
import hashlib
import requests
@ -317,14 +323,23 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
# Calculate SHA256 hash of the content
content_hash = hashlib.sha256(json_string.encode("utf-8")).hexdigest()
content_md5 = base64.b64encode(
hashlib.md5(json_string.encode("utf-8"), usedforsecurity=False).digest()
).decode()
# Prepare the request
headers = {
"Content-Type": "application/json",
"Content-MD5": content_md5,
"x-amz-content-sha256": content_hash,
"Content-Language": "en",
"Content-Disposition": f'inline; filename="{batch_logging_element.s3_object_download_filename}"',
"Cache-Control": "private, immutable, max-age=31536000, s-maxage=0",
**(
{"x-amz-server-side-encryption": self.s3_server_side_encryption}
if self.s3_server_side_encryption
else {}
),
}
req = requests.Request("PUT", url, data=json_string, headers=headers)
prepped = req.prepare()
@ -447,6 +462,7 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
def upload_data_to_s3(self, batch_logging_element: s3BatchLoggingElement):
try:
import base64
import hashlib
import requests
@ -482,14 +498,23 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
# Calculate SHA256 hash of the content
content_hash = hashlib.sha256(json_string.encode("utf-8")).hexdigest()
content_md5 = base64.b64encode(
hashlib.md5(json_string.encode("utf-8"), usedforsecurity=False).digest()
).decode()
# Prepare the request
headers = {
"Content-Type": "application/json",
"Content-MD5": content_md5,
"x-amz-content-sha256": content_hash,
"Content-Language": "en",
"Content-Disposition": f'inline; filename="{batch_logging_element.s3_object_download_filename}"',
"Cache-Control": "private, immutable, max-age=31536000, s-maxage=0",
**(
{"x-amz-server-side-encryption": self.s3_server_side_encryption}
if self.s3_server_side_encryption
else {}
),
}
req = requests.Request("PUT", url, data=json_string, headers=headers)
prepped = req.prepare()

52
litellm/integrations/websearch_interception/handler.py
View file

@ -31,6 +31,7 @@ from litellm.types.integrations.websearch_interception import (
WebSearchInterceptionConfig,
)
from litellm.types.integrations.custom_logger import (
CHAT_COMPLETION_AGENTIC_SURFACE,
AgenticLoopPlan,
AgenticLoopRequestPatch,
)
@ -119,21 +120,26 @@ class WebSearchInterceptionLogger(CustomLogger):
if self.enabled_providers is not None and provider_str not in self.enabled_providers:
return None
# Only short-circuit for providers without native Anthropic Messages
# support. Providers that have a BaseAnthropicMessagesConfig (bedrock,
# vertex_ai, azure_ai, anthropic) already use the agentic loop, which
# includes a follow-up LLM call to synthesize the answer from search
# results. Short-circuiting those would skip that synthesis step and
# return raw search text — a regression for existing users.
# Only short-circuit for providers whose Anthropic Messages agentic loop
# does not run web_search itself. Providers that have a
# BaseAnthropicMessagesConfig which handles web search natively (bedrock,
# vertex_ai, azure_ai, anthropic) already perform the search plus a
# follow-up LLM synthesis step; short-circuiting those would skip that
# synthesis and return raw search text — a regression for existing users.
#
# github_copilot has a BaseAnthropicMessagesConfig (added for thinking
# passthrough) but does not handle web_search natively, so its config
# returns handles_web_search_natively() == False and we still short-circuit
# web-search-only requests against it.
try:
provider_enum = LlmProviders(provider_str)
anthropic_config = ProviderConfigManager.get_provider_anthropic_messages_config(
model=model, provider=provider_enum
)
if anthropic_config is not None:
if anthropic_config is not None and anthropic_config.handles_web_search_natively():
verbose_logger.debug(
f"WebSearchInterception: Skipping short-circuit for {provider_str} "
"(provider has native Anthropic Messages support, using agentic loop)"
"(provider handles web search natively via the agentic loop)"
)
return None
except (ValueError, Exception):
@ -440,12 +446,16 @@ class WebSearchInterceptionLogger(CustomLogger):
custom_llm_provider: str,
kwargs: Dict,
) -> Tuple[bool, Dict]:
"""
Check if WebSearch tool interception is needed for Anthropic Messages API.
This is the legacy method for Anthropic-style responses.
For chat completions, use async_should_run_chat_completion_agentic_loop instead.
"""
if kwargs.get("_agentic_loop_api_surface") == CHAT_COMPLETION_AGENTIC_SURFACE:
return await self.async_should_run_chat_completion_agentic_loop(
response=response,
model=model,
messages=messages,
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
)
verbose_logger.debug(f"WebSearchInterception: Hook called! provider={custom_llm_provider}, stream={stream}")
verbose_logger.debug(f"WebSearchInterception: Response type: {type(response)}")
@ -629,6 +639,18 @@ class WebSearchInterceptionLogger(CustomLogger):
stream: bool,
kwargs: Dict,
) -> AgenticLoopPlan:
if kwargs.get("_agentic_loop_api_surface") == CHAT_COMPLETION_AGENTIC_SURFACE:
return await self.async_build_chat_completion_agentic_loop_plan(
tools=tools,
model=model,
messages=messages,
response=response,
optional_params=anthropic_messages_optional_request_params,
logging_obj=logging_obj,
stream=stream,
kwargs=kwargs,
)
tool_calls = tools["tool_calls"]
thinking_blocks = tools.get("thinking_blocks", [])
request_patch, structured_results = await self._build_anthropic_request_patch(
@ -1088,6 +1110,7 @@ class WebSearchInterceptionLogger(CustomLogger):
raise ValueError("WebSearchInterception: missing follow-up messages")
params = dict(optional_params)
params.update(request_patch.optional_params)
params.pop("tool_choice", None)
return await litellm.acompletion(
model=request_patch.model or model,
messages=request_patch.messages,
@ -1203,6 +1226,7 @@ class WebSearchInterceptionLogger(CustomLogger):
if k
not in {
"tools",
"tool_choice",
"extra_body",
"model_alias_map",
"stream_response",

18
litellm/litellm_core_utils/chat_completion_agentic_loop.py
View file

@ -137,8 +137,8 @@ async def _execute_chat_completion_agentic_plan(
optional_params_for_followup = {**optional_params, **patch.optional_params}
if patch.tools is not None:
optional_params_for_followup["tools"] = patch.tools
if "tool_choice" not in patch.optional_params:
optional_params_for_followup.pop("tool_choice", None)
if "tool_choice" not in patch.optional_params:
optional_params_for_followup.pop("tool_choice", None)
kwargs_for_followup = _filter_followup_kwargs(kwargs)
kwargs_for_followup.update(
@ -206,10 +206,11 @@ async def maybe_run_chat_completion_agentic_loop(
for callback in callbacks:
if not isinstance(callback, CustomLogger):
continue
if not _gate_overridden(callback):
continue
gate_kwargs = {
hook_kwargs = {
**kwargs,
"_agentic_loop_api_surface": CHAT_COMPLETION_AGENTIC_SURFACE,
"custom_llm_provider": custom_llm_provider,
@ -222,7 +223,7 @@ async def maybe_run_chat_completion_agentic_loop(
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
kwargs=gate_kwargs,
kwargs=hook_kwargs,
)
except Exception as e:
verbose_logger.exception(
@ -243,11 +244,6 @@ async def maybe_run_chat_completion_agentic_loop(
)
try:
plan_kwargs = {
**kwargs,
"_agentic_loop_api_surface": CHAT_COMPLETION_AGENTIC_SURFACE,
"custom_llm_provider": custom_llm_provider,
}
if not _build_plan_overridden(callback):
return await callback.async_run_agentic_loop(
tools=tool_calls,
@ -258,7 +254,7 @@ async def maybe_run_chat_completion_agentic_loop(
anthropic_messages_optional_request_params=optional_params,
logging_obj=logging_obj,
stream=stream,
kwargs=plan_kwargs,
kwargs=hook_kwargs,
)
plan = await callback.async_build_agentic_loop_plan(
@ -270,7 +266,7 @@ async def maybe_run_chat_completion_agentic_loop(
anthropic_messages_optional_request_params=optional_params,
logging_obj=logging_obj,
stream=stream,
kwargs=plan_kwargs,
kwargs=hook_kwargs,
)
if plan.response_override is not None:

6
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -446,6 +446,8 @@ def get_llm_provider(
# bytez models
elif model.startswith("bytez/"):
custom_llm_provider = "bytez"
elif model.startswith("gdc/"):
custom_llm_provider = "gdc"
elif model.startswith("lemonade/"):
custom_llm_provider = "lemonade"
elif model.startswith("heroku/"):
@ -650,6 +652,10 @@ def _get_openai_compatible_provider_info(
api_base = api_base or get_secret("DEEPSEEK_API_BASE") or "https://api.deepseek.com/beta" # type: ignore
dynamic_api_key = api_key or get_secret_str("DEEPSEEK_API_KEY")
elif custom_llm_provider == "tencent":
api_base = api_base or get_secret("TENCENT_API_BASE") or "https://tokenhub-intl.tencentcloudmaas.com/v1"
dynamic_api_key = api_key or get_secret_str("TENCENT_API_KEY")
elif custom_llm_provider == "fireworks_ai":
# fireworks is openai compatible, we just need to set this to custom_openai and have the api_base be https://api.fireworks.ai/inference/v1
(

2
litellm/litellm_core_utils/get_supported_openai_params.py
View file

@ -106,6 +106,8 @@ def get_supported_openai_params(
return litellm.VLLMConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "deepseek":
return litellm.DeepSeekChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "tencent":
return litellm.TencentChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "cohere_chat" or custom_llm_provider == "cohere":
return litellm.CohereChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "maritalk":

29
litellm/litellm_core_utils/initialize_dynamic_callback_params.py
View file

@ -1,7 +1,23 @@
from typing import Dict, Optional
from typing import Any, Dict, Iterator, Optional
from litellm.types.utils import StandardCallbackDynamicParams
_CLIENT_CALLBACK_METADATA_SLOTS: tuple[str, ...] = ("litellm_metadata", "metadata")
def iter_client_callback_metadata_dicts(
kwargs: dict[str, Any],
) -> Iterator[tuple[str, dict[str, Any]]]:
litellm_params = kwargs.get("litellm_params")
if isinstance(litellm_params, dict):
nested = litellm_params.get("metadata")
if isinstance(nested, dict):
yield "litellm_params.metadata", nested
for key in _CLIENT_CALLBACK_METADATA_SLOTS:
candidate = kwargs.get(key)
if isinstance(candidate, dict):
yield key, candidate
def _is_env_reference(value: object) -> bool:
return isinstance(value, str) and "os.environ/" in value
@ -55,6 +71,7 @@ _supported_callback_params = [
"dd_site",
"dd_agent_host",
"dd_agent_port",
"turn_off_message_logging",
]
_request_blocked_callback_params = {
@ -87,19 +104,13 @@ def initialize_standard_callback_dynamic_params(
validate_no_callback_env_reference(param, _param_value, source="request body")
standard_callback_dynamic_params[param] = _param_value # type: ignore
# 2. Fallback: check "metadata" or "litellm_params" -> "metadata"
metadata = (kwargs.get("metadata") or {}).copy()
litellm_params = kwargs.get("litellm_params") or {}
if isinstance(litellm_params, dict):
metadata.update(litellm_params.get("metadata") or {})
if isinstance(metadata, dict):
for slot_label, metadata in iter_client_callback_metadata_dicts(kwargs):
for param in _supported_callback_params:
if param in _request_blocked_callback_params:
continue
if param not in standard_callback_dynamic_params and param in metadata:
_param_value = metadata.get(param)
validate_no_callback_env_reference(param, _param_value, source="metadata")
validate_no_callback_env_reference(param, _param_value, source=slot_label)
standard_callback_dynamic_params[param] = _param_value # type: ignore
return standard_callback_dynamic_params

9
litellm/litellm_core_utils/litellm_logging.py
View file

@ -4722,7 +4722,7 @@ class StandardLoggingPayloadSetup:
api_base: Optional[str] = None,
) -> StandardLoggingModelInformation:
model_cost_name = _select_model_name_for_cost_calc(
model=None,
model=base_model if custom_pricing else None,
completion_response=init_response_obj, # type: ignore
base_model=base_model,
custom_pricing=custom_pricing,
@ -5268,6 +5268,11 @@ def get_standard_logging_object_payload(
## Get model cost information ##
base_model = _get_base_model_from_metadata(model_call_details=kwargs)
# The router overrides completion_response.model to the model-group alias before
# this payload is built, so cost-map lookup via that alias always misses.
# Fall back to the actual deployment model set by the router in metadata.
if base_model is None:
base_model = metadata.get("deployment")
custom_pricing = use_custom_pricing_for_model(litellm_params=litellm_params)
raw_response_cost = kwargs.get("response_cost")
response_cost: float = raw_response_cost or 0.0
@ -5389,7 +5394,7 @@ def get_standard_logging_object_payload(
def emit_standard_logging_payload(payload: StandardLoggingPayload):
if os.getenv("LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD"):
print(json.dumps(payload, indent=4)) # noqa: T201
print(json.dumps(payload, indent=4), flush=True) # noqa: T201
def get_standard_logging_metadata(

154
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -6,7 +6,7 @@ import mimetypes
import re
import xml.etree.ElementTree as ET
from enum import Enum
from typing import Any, Dict, List, Optional, Set, Tuple, Union, cast, overload
from typing import Any, Dict, List, Optional, Set, Tuple, TypedDict, Union, cast, overload
from jinja2.sandbox import ImmutableSandboxedEnvironment
@ -5035,15 +5035,18 @@ def _bedrock_tools_pt(tools: List, model: Optional[str] = None) -> List[BedrockT
]
"""
from litellm.llms.bedrock.common_utils import (
get_bedrock_base_model,
bedrock_converse_supports_strict_tools,
normalize_json_schema_custom_types_to_object,
)
from litellm.litellm_core_utils.prompt_templates.common_utils import unpack_defs
_valid_json_schema_root_types = frozenset(("array", "boolean", "integer", "null", "number", "object", "string"))
# Only Claude on Bedrock honours strict tool schemas; other families
# (Nova, Llama, GPT-OSS) reject the strict field outright.
supports_strict_tools = bool(model and get_bedrock_base_model(model).startswith("anthropic"))
# (Nova, Llama, GPT-OSS) reject the strict field outright. Opus 4.7/4.8
# also reject `strict` on Bedrock Converse (see #31582) — their validator
# maps toolSpec to the native Anthropic tool shape, which has no strict
# field, even though Anthropic's native API accepts it as a top-level key.
supports_strict_tools = bool(model and bedrock_converse_supports_strict_tools(model))
tool_block_list: List[BedrockToolBlock] = []
for tool_idx, tool in enumerate(tools):
# Check if tool is already a BedrockToolBlock (e.g., systemTool for Nova grounding)
@ -5322,3 +5325,146 @@ def get_attribute_or_key(tool_or_function, attribute, default=None):
if hasattr(tool_or_function, attribute):
return getattr(tool_or_function, attribute)
return tool_or_function.get(attribute, default)
class NormalizedToolCall(TypedDict):
id: Optional[str]
name: Optional[str]
arguments: dict[str, Any]
def _parse_tool_call_arguments(raw: Any, tool_name: Optional[str], context: str) -> dict[str, Any]:
# Anthropic's tool_use blocks already carry a parsed dict in "input";
# chat completions and the Responses API carry a JSON string that may be
# truncated by the model, so route those through the repair-aware parser.
if isinstance(raw, dict):
return raw
if not isinstance(raw, str):
return {}
from litellm.litellm_core_utils.prompt_templates.common_utils import (
parse_tool_call_arguments,
)
try:
parsed = parse_tool_call_arguments(raw, tool_name=tool_name, context=context)
except ValueError as e:
verbose_logger.warning("Failed to parse tool call arguments: %s", e)
return {}
return parsed if isinstance(parsed, dict) else {}
def _tool_calls_from_chat_completion_response(response: Any) -> list[NormalizedToolCall]:
choices = get_attribute_or_key(response, "choices", None)
if not (isinstance(choices, list) and choices):
return []
message = get_attribute_or_key(choices[0], "message", None)
tool_calls = get_attribute_or_key(message, "tool_calls", None) if message else None
if not isinstance(tool_calls, list):
return []
result: list[NormalizedToolCall] = []
for tc in tool_calls:
fn = get_attribute_or_key(tc, "function", None)
if fn is None:
continue
name = get_attribute_or_key(fn, "name")
result.append(
NormalizedToolCall(
id=get_attribute_or_key(tc, "id"),
name=name,
arguments=_parse_tool_call_arguments(
get_attribute_or_key(fn, "arguments", "{}"),
tool_name=name,
context="chat completions",
),
)
)
return result
def _tool_calls_from_responses_api_response(response: Any) -> list[NormalizedToolCall]:
output = get_attribute_or_key(response, "output", None)
if not isinstance(output, list):
return []
result: list[NormalizedToolCall] = []
for item in output:
if get_attribute_or_key(item, "type") != "function_call":
continue
name = get_attribute_or_key(item, "name")
result.append(
NormalizedToolCall(
id=get_attribute_or_key(item, "call_id") or get_attribute_or_key(item, "id"),
name=name,
arguments=_parse_tool_call_arguments(
get_attribute_or_key(item, "arguments", "{}"),
tool_name=name,
context="responses API",
),
)
)
return result
def _tool_calls_from_anthropic_messages_response(response: Any) -> list[NormalizedToolCall]:
content = get_attribute_or_key(response, "content", None)
if not isinstance(content, list):
return []
result: list[NormalizedToolCall] = []
for block in content:
if get_attribute_or_key(block, "type") != "tool_use":
continue
raw_input = get_attribute_or_key(block, "input", {})
result.append(
NormalizedToolCall(
id=get_attribute_or_key(block, "id"),
name=get_attribute_or_key(block, "name"),
arguments=raw_input if isinstance(raw_input, dict) else {},
)
)
return result
def get_tool_calls_from_response(response: Any) -> list[NormalizedToolCall]:
"""
Extract tool/function calls from a response object into a normalized
``{"id", "name", "arguments"}`` shape, regardless of which API surface
produced it: chat completions (``choices[].message.tool_calls``),
the Responses API (``output`` items of type ``function_call``), or the
Anthropic Messages API (``content`` blocks of type ``tool_use``).
Callers that only care about a specific tool should filter the result by
``name`` themselves -- this returns every tool call found.
"""
for extractor in (
_tool_calls_from_chat_completion_response,
_tool_calls_from_responses_api_response,
_tool_calls_from_anthropic_messages_response,
):
tool_calls = extractor(response)
if tool_calls:
return tool_calls
return []
def has_tool_with_name(tools: Any, tool_name: str) -> bool:
"""
Check whether a tools list (as sent to an LLM) includes a tool with the
given name, regardless of shape: OpenAI-style function tools
(``{"type": "function", "function": {"name": ...}}``) or Anthropic's
native tool shape (a top-level ``"name"``, e.g.
``{"name": ..., "input_schema": ...}``). Anthropic's documented client
tool format doesn't require a ``"type"`` key at all -- ``"custom"`` is
only one of several possible values -- so any non-OpenAI-shaped tool is
matched on its top-level ``"name"``.
"""
if not isinstance(tools, list):
return False
for tool in tools:
if not isinstance(tool, dict):
continue
function = tool.get("function")
if tool.get("type") == "function" and isinstance(function, dict):
if function.get("name") == tool_name:
return True
elif tool.get("name") == tool_name:
return True
return False

6
litellm/litellm_core_utils/prompt_templates/image_handling.py
View file

@ -72,6 +72,9 @@ def _process_image_response(response: Response, url: str) -> str:
async def async_convert_url_to_base64(url: str) -> str:
if url.startswith("data:") and ";base64," in url:
return url
# If MAX_IMAGE_URL_DOWNLOAD_SIZE_MB is 0, block all image downloads
if MAX_IMAGE_URL_DOWNLOAD_SIZE_MB == 0:
raise litellm.ImageFetchError(
@ -95,6 +98,9 @@ async def async_convert_url_to_base64(url: str) -> str:
def convert_url_to_base64(url: str) -> str:
if url.startswith("data:") and ";base64," in url:
return url
# If MAX_IMAGE_URL_DOWNLOAD_SIZE_MB is 0, block all image downloads
if MAX_IMAGE_URL_DOWNLOAD_SIZE_MB == 0:
raise litellm.ImageFetchError(

21
litellm/llms/a2a/chat/transformation.py
View file

@ -10,7 +10,7 @@ import httpx
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import Choices, Message, ModelResponse
from litellm.types.utils import Choices, Message, ModelResponse, Usage
from ..common_utils import (
A2AError,
@ -312,6 +312,25 @@ class A2AConfig(BaseConfig):
# Set ID from response
model_response.id = response_json.get("id", str(uuid.uuid4()))
# A2A agents don't return token usage; estimate it so per-token pricing
# produces real cost and callers don't receive usage of 0/0/0.
try:
from litellm.utils import token_counter
prompt_tokens = token_counter(model="gpt-3.5-turbo", messages=messages)
completion_tokens = token_counter(model="gpt-3.5-turbo", text=text, count_response_tokens=True)
setattr(
model_response,
"usage",
Usage(
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=prompt_tokens + completion_tokens,
),
)
except Exception: # noqa: BLE001 - best-effort estimate; a tokenizer hiccup must not break the response
pass
return model_response
def get_model_response_iterator(

199
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -48,7 +48,10 @@ from litellm.types.utils import (
)
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.custom_guardrail import (
CustomGuardrail,
ModifyResponseException,
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
@ -70,6 +73,170 @@ class AnthropicMessagesHandler(BaseTranslation):
super().__init__()
self.adapter = LiteLLMAnthropicMessagesAdapter()
@staticmethod
def _build_streaming_usage_response(
responses_so_far: list[Any],
request_data: Optional[dict],
) -> Optional[ModelResponse]:
chunks = tuple(response for response in responses_so_far if isinstance(response, (str, bytes)))
if not chunks:
return None
try:
return AnthropicPassthroughLoggingHandler._build_usage_only_response_from_chunks(
all_chunks=chunks,
model=str((request_data or {}).get("model") or ""),
)
except (AttributeError, TypeError, ValueError):
return None
def build_block_sse_chunks(
self,
exc: "ModifyResponseException",
stream_started: bool = False,
responses_so_far: Optional[list[Any]] = None,
) -> list[bytes]:
"""
Build an Anthropic SSE sequence delivering the guardrail block message
and terminating the stream cleanly.
- ``stream_started`` False (buffered / pre-stream): nothing has been
sent, so emit a complete standalone message (message_start ->
content_block_* -> message_delta -> message_stop) via
FakeAnthropicMessagesStreamIterator, the same converter the
/v1/messages pre-stream block handler uses.
- ``stream_started`` True (sampling / detect-only end-of-stream): real
chunks were already sent, so *continue* the in-progress message --
close the open content block, append the block message as a new text
block, then end the message. Emitting a second ``message_start`` here
would make Anthropic clients reject the stream.
"""
if stream_started:
return self._block_continuation_chunks(exc, responses_so_far or [])
return self._standalone_block_chunks(exc)
def _standalone_block_chunks(self, exc: "ModifyResponseException") -> list[bytes]:
import uuid
from litellm.llms.anthropic.experimental_pass_through.messages.fake_stream_iterator import (
FakeAnthropicMessagesStreamIterator,
)
from litellm.llms.base_llm.guardrail_translation.utils import (
blocked_response_usage,
)
from litellm.types.utils import AnthropicMessagesResponse
block_response = AnthropicMessagesResponse(
id=f"msg_{uuid.uuid4()}",
type="message",
role="assistant",
content=[{"type": "text", "text": exc.message}],
model=exc.model,
stop_reason="end_turn",
usage=blocked_response_usage(getattr(exc, "original_response", None)),
)
return list(FakeAnthropicMessagesStreamIterator(response=block_response))
def _block_continuation_chunks(self, exc: "ModifyResponseException", responses_so_far: list[Any]) -> list[bytes]:
"""Continue an already-started message: close the open content block,
append the block message as a new text block, then end the message --
without a second message_start."""
from litellm.llms.base_llm.guardrail_translation.utils import (
blocked_response_usage,
)
def _sse(event_type: str, payload: dict) -> bytes:
return f"event: {event_type}\ndata: {json.dumps(payload)}\n\n".encode()
output_tokens = blocked_response_usage(getattr(exc, "original_response", None))["output_tokens"]
open_index, max_index = self._content_block_state(responses_so_far)
new_index = (max_index + 1) if max_index is not None else 0
chunks: list[bytes] = []
if open_index is not None:
chunks.append(_sse("content_block_stop", {"type": "content_block_stop", "index": open_index}))
chunks += [
_sse(
"content_block_start",
{
"type": "content_block_start",
"index": new_index,
"content_block": {"type": "text", "text": ""},
},
),
_sse(
"content_block_delta",
{
"type": "content_block_delta",
"index": new_index,
"delta": {"type": "text_delta", "text": exc.message},
},
),
_sse("content_block_stop", {"type": "content_block_stop", "index": new_index}),
_sse(
"message_delta",
{
"type": "message_delta",
"delta": {"stop_reason": "end_turn", "stop_sequence": None},
"usage": {"output_tokens": output_tokens},
},
),
_sse("message_stop", {"type": "message_stop"}),
]
return chunks
@staticmethod
def _content_block_state(
responses_so_far: list[Any],
) -> tuple[Optional[int], Optional[int]]:
"""From the SSE chunks already sent to the client, return (open
content-block index or None, highest content-block index seen or None).
A single streamed item may bundle multiple SSE events (raw bytes) or be
an already-parsed event dict, so every event across every item is
considered -- matching how ``get_streaming_string_so_far`` reads the
same stream."""
open_indices: set[int] = set()
max_index: Optional[int] = None
for item in responses_so_far:
for data in AnthropicMessagesHandler._iter_sse_events(item):
event_type = data.get("type")
index = data.get("index")
if not isinstance(index, int):
continue
if event_type == "content_block_start":
open_indices.add(index)
max_index = index if max_index is None else max(max_index, index)
elif event_type == "content_block_stop":
open_indices.discard(index)
open_index = max(open_indices) if open_indices else None
return open_index, max_index
@staticmethod
def _iter_sse_events(item: Any) -> list[dict]:
"""Yield the event-data dicts in one stream chunk.
Handles both formats this stream can carry (see
``get_streaming_string_so_far``): raw SSE ``bytes`` -- which may bundle
several events separated by a blank line -- and an already-parsed event
``dict``."""
if isinstance(item, dict):
return [item]
if not isinstance(item, (bytes, bytearray)):
return []
events: list[dict] = []
for block in item.decode("utf-8", errors="replace").split("\n\n"):
for line in block.split("\n"):
line = line.strip()
if not line.startswith("data:"):
continue
try:
parsed = json.loads(line[len("data:") :].strip())
except json.JSONDecodeError:
continue
if isinstance(parsed, dict):
events.append(parsed)
return events
def _translate_to_openai(self, data: dict) -> ChatCompletionRequest:
"""Translate Anthropic request to OpenAI chat completion format."""
(
@ -406,6 +573,8 @@ class AnthropicMessagesHandler(BaseTranslation):
Get the string so far, check the apply guardrail to the string so far, and return the list of responses so far.
"""
from litellm.integrations.custom_guardrail import ModifyResponseException
has_ended = self._check_streaming_has_ended(responses_so_far)
if has_ended:
# build the model response from the responses_so_far
@ -430,25 +599,35 @@ class AnthropicMessagesHandler(BaseTranslation):
if tool_calls_list:
guardrail_inputs["tool_calls"] = tool_calls_list
_guardrailed_inputs = (
await guardrail_to_apply.apply_guardrail( # allow rejecting the response, if invalid
try:
_guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=guardrail_inputs,
request_data=request_data if request_data is not None else {},
input_type="response",
logging_obj=litellm_logging_obj,
)
)
except ModifyResponseException as e:
if e.original_response is None:
e.original_response = built_response or self._build_streaming_usage_response(
responses_so_far, request_data
)
raise
else:
verbose_proxy_logger.debug("Skipping output guardrail - model response has no choices")
return responses_so_far
string_so_far = self.get_streaming_string_so_far(responses_so_far)
_guardrailed_inputs = await guardrail_to_apply.apply_guardrail( # allow rejecting the response, if invalid
inputs={"texts": [string_so_far]},
request_data=request_data if request_data is not None else {},
input_type="response",
logging_obj=litellm_logging_obj,
)
try:
_guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs={"texts": [string_so_far]},
request_data=request_data if request_data is not None else {},
input_type="response",
logging_obj=litellm_logging_obj,
)
except ModifyResponseException as e:
if e.original_response is None:
e.original_response = self._build_streaming_usage_response(responses_so_far, request_data)
raise
return responses_so_far
def _prepare_request_data(

16
litellm/llms/anthropic/experimental_pass_through/messages/handler.py
View file

@ -199,7 +199,7 @@ async def anthropic_messages(
metadata: Optional[Dict] = None,
stop_sequences: Optional[List[str]] = None,
stream: Optional[bool] = False,
system: Optional[str] = None,
system: Optional[Union[str, list]] = None,
temperature: Optional[float] = None,
thinking: Optional[Dict] = None,
tool_choice: Optional[Dict] = None,
@ -230,6 +230,12 @@ async def anthropic_messages(
# ids like ``functions.Bash:0`` that violate Anthropic's id pattern.
messages = sanitize_tool_use_ids_in_anthropic_messages(messages)
from litellm.integrations.anthropic_cache_control_hook import (
AnthropicCacheControlHook,
)
messages, system = AnthropicCacheControlHook.maybe_inject_cache_control(messages, system, kwargs)
original_stream = stream or kwargs.get("_websearch_interception_converted_stream", False)
# Execute pre-request hooks to allow CustomLoggers to modify request.
@ -375,7 +381,7 @@ def anthropic_messages_handler(
metadata: Optional[Dict] = None,
stop_sequences: Optional[List[str]] = None,
stream: Optional[bool] = False,
system: Optional[str] = None,
system: Optional[Union[str, list]] = None,
temperature: Optional[float] = None,
thinking: Optional[Dict] = None,
tool_choice: Optional[Dict] = None,
@ -412,6 +418,12 @@ def anthropic_messages_handler(
messages = strip_empty_text_blocks_from_anthropic_messages(messages)
messages = sanitize_tool_use_ids_in_anthropic_messages(messages)
from litellm.integrations.anthropic_cache_control_hook import (
AnthropicCacheControlHook,
)
messages, system = AnthropicCacheControlHook.maybe_inject_cache_control(messages, system, kwargs)
metadata = validate_anthropic_api_metadata(metadata)
local_vars = locals()

356
litellm/llms/azure_ai/ocr/document_intelligence/transformation.py
View file

@ -15,6 +15,7 @@ from typing import Any, Dict
from urllib.parse import quote
import httpx
from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.url_utils import SSRFError, assert_same_origin
@ -38,6 +39,30 @@ from litellm.secret_managers.main import get_secret_str
AZURE_DOCUMENT_INTELLIGENCE_API_KEY_ENV_VAR = "AZURE_DOCUMENT_INTELLIGENCE_API_KEY"
class AzureDocumentIntelligenceLine(BaseModel):
content: str | None = None
class AzureDocumentIntelligencePage(BaseModel):
pageNumber: int | None = None
width: float | None = None
height: float | None = None
unit: str | None = None
lines: tuple[AzureDocumentIntelligenceLine, ...] = ()
class AzureDocumentIntelligenceAnalyzeResult(BaseModel):
content: str | None = None
pages: tuple[AzureDocumentIntelligencePage, ...] = ()
tables: list[dict[str, object]] | None = None
keyValuePairs: list[dict[str, object]] | None = None
class AzureDocumentIntelligenceOperation(BaseModel):
status: str | None = None
analyzeResult: AzureDocumentIntelligenceAnalyzeResult | None = None
class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
"""
Azure Document Intelligence OCR transformation configuration.
@ -67,11 +92,14 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
(1-based, e.g. "1-3,5,7-9"). To keep the public request shape
aligned with Mistral OCR, callers pass `pages` using Mistral
semantics — a list of 0-based integers — or a pre-formatted
Azure-style string. Other Mistral-specific params (e.g.
Azure-style string. Azure DI also exposes a `features` query
parameter enabling add-on capabilities (e.g. "keyValuePairs",
"languages"), passed as a list of feature names or a
comma-separated string. Other Mistral-specific params (e.g.
`include_image_base64`) are not supported by Azure DI and are
ignored during transformation.
"""
return ["pages"]
return ["pages", "features"]
def map_ocr_params(
self,
@ -85,16 +113,18 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
Translates Mistral-style `pages` (list[int], 0-based) into Azure's
`pages` query string (1-based, e.g. "1,2,3" or "1-3,5"). A raw
string that already matches Azure's format is passed through
unchanged.
unchanged. `features` (list[str] or comma-separated string) is
normalized into Azure's comma-joined `features` query string.
"""
pages = non_default_params.get("pages")
if pages is None:
return optional_params
normalized = self._normalize_pages_param(pages)
if normalized:
optional_params["pages"] = normalized
return optional_params
features = non_default_params.get("features")
normalized_pages = self._normalize_pages_param(pages) if pages is not None else ""
normalized_features = self._normalize_features_param(features) if features is not None else ""
return {
**optional_params,
**({"pages": normalized_pages} if normalized_pages else {}),
**({"features": normalized_features} if normalized_features else {}),
}
@staticmethod
def _normalize_pages_param(pages: Any) -> str:
@ -140,6 +170,39 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
raise ValueError("`pages` must be a list[int] (0-based, Mistral-style) or a string like '1-3,5,7-9'.")
@staticmethod
def _normalize_features_param(features: object) -> str:
"""
Convert a caller-provided `features` value to Azure DI's query-string
form (comma-joined feature names, e.g. "keyValuePairs,languages").
Accepted inputs:
- list[str]: feature names like ["keyValuePairs", "languages"].
- str: a single feature name or comma-separated names.
"""
invalid_features_error = ValueError(
f"Invalid `features` for Azure Document Intelligence: {features!r}. "
f"Expected a list of feature names or a comma-separated string like "
f"'keyValuePairs' or 'keyValuePairs,languages'."
)
if isinstance(features, str):
raw_tokens = features.split(",")
elif isinstance(features, list):
if len(features) == 0:
return ""
raw_tokens = [feature for feature in features if isinstance(feature, str)]
if len(raw_tokens) != len(features):
raise invalid_features_error
else:
raise invalid_features_error
tokens = tuple(token.strip() for token in raw_tokens)
feature_pattern = re.compile(r"^[A-Za-z][A-Za-z0-9]*$")
if not all(feature_pattern.match(token) for token in tokens):
raise invalid_features_error
return ",".join(tokens)
def validate_environment(
self,
headers: Dict,
@ -228,13 +291,15 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
f"?api-version={AZURE_DOCUMENT_INTELLIGENCE_API_VERSION}"
)
# Azure DI accepts `pages` as a query param (1-based, e.g. "1-3,5").
# Azure DI accepts `pages` (1-based, e.g. "1-3,5") and `features`
# (comma-joined names, e.g. "keyValuePairs") as query params.
# `optional_params` has already been normalized in `map_ocr_params`.
pages = optional_params.get("pages") if optional_params else None
if pages:
url += f"&pages={quote(str(pages), safe=',-')}"
features = optional_params.get("features") if optional_params else None
pages_query = f"&pages={quote(str(pages), safe=',-')}" if pages else ""
features_query = f"&features={quote(str(features), safe=',')}" if features else ""
return url
return f"{url}{pages_query}{features_query}"
def _extract_base64_from_data_uri(self, data_uri: str) -> str:
"""
@ -328,27 +393,15 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
return OCRRequestData(data=data, files=None)
def _extract_page_markdown(self, page_data: Dict[str, Any]) -> str:
"""
Extract text from Azure DI page and format as markdown.
Azure DI provides text in 'lines' array. We concatenate them with newlines.
Args:
page_data: Azure DI page object
Returns:
Markdown-formatted text
"""
lines = page_data.get("lines", [])
if not lines:
return ""
# Extract text content from each line
text_lines = [line.get("content", "") for line in lines]
# Join with newlines to preserve structure
return "\n".join(text_lines)
def _transform_azure_page(self, azure_page: AzureDocumentIntelligencePage) -> OCRPage:
page_number = azure_page.pageNumber if azure_page.pageNumber is not None else 1
markdown = "\n".join(line.content or "" for line in azure_page.lines)
dimensions = self._convert_dimensions(
width=azure_page.width if azure_page.width is not None else 8.5,
height=azure_page.height if azure_page.height is not None else 11,
unit=azure_page.unit if azure_page.unit is not None else "inch",
)
return OCRPage(index=page_number - 1, markdown=markdown, dimensions=dimensions)
def _convert_dimensions(self, width: float, height: float, unit: str) -> OCRPageDimensions:
"""
@ -526,6 +579,52 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
retry_after = self._get_retry_after(response=response)
await asyncio.sleep(retry_after)
def _get_polling_target(self, raw_response: httpx.Response) -> tuple[str, Dict[str, str]]:
operation_url = raw_response.headers.get("Operation-Location")
if not operation_url:
raise ValueError("Azure Document Intelligence returned 202 but no Operation-Location header found")
# Reject cross-origin polling URLs — the auth headers
# below would otherwise leak to whatever URL the upstream
# (or an attacker-controlled upstream) returns. VERIA-51.
try:
assert_same_origin(operation_url, str(raw_response.request.url))
except SSRFError as ssrf_err:
raise ValueError(f"Azure Document Intelligence: rejected polling URL ({ssrf_err})")
poll_headers = {"Ocp-Apim-Subscription-Key": raw_response.request.headers.get("Ocp-Apim-Subscription-Key", "")}
return operation_url, poll_headers
def _transform_completed_response(self, model: str, raw_response: httpx.Response) -> OCRResponse:
"""
Transform a completed Azure Document Intelligence analyze operation
into the Mistral OCR response shape, preserving Azure-native
`analyzeResult` fields (`content`, `tables`, `keyValuePairs`) as
top-level response fields.
"""
operation = AzureDocumentIntelligenceOperation.model_validate(raw_response.json())
verbose_logger.debug(f"Azure Document Intelligence response status: {operation.status}")
if operation.status != "succeeded":
raise ValueError(f"Azure Document Intelligence analysis failed with status: {operation.status}")
analyze_result = (
operation.analyzeResult if operation.analyzeResult is not None else AzureDocumentIntelligenceAnalyzeResult()
)
mistral_pages = [self._transform_azure_page(azure_page) for azure_page in analyze_result.pages]
usage_info = OCRUsageInfo(pages_processed=len(mistral_pages), doc_size_bytes=None)
return OCRResponse(
pages=mistral_pages,
model=model,
usage_info=usage_info,
object="ocr",
content=analyze_result.content,
tables=analyze_result.tables,
keyValuePairs=analyze_result.keyValuePairs,
)
def transform_ocr_response(
self,
model: str,
@ -552,11 +651,13 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
"unit": "inch",
"lines": [{"content": "text", "boundingBox": [...]}]
}
]
],
"tables": [...],
"keyValuePairs": [...]
}
}
Mistral OCR format:
Mistral OCR format (with Azure-native fields preserved):
{
"pages": [
{
@ -567,7 +668,10 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
],
"model": "azure_ai/doc-intelligence/prebuilt-layout",
"usage_info": {"pages_processed": 1},
"object": "ocr"
"object": "ocr",
"content": "Full document text...",
"tables": [...],
"keyValuePairs": [...]
}
Args:
@ -578,86 +682,17 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
Returns:
OCRResponse in Mistral format
"""
try:
# Check if we got 202 Accepted (async operation started)
if raw_response.status_code == 202:
verbose_logger.debug("Azure DI returned 202 Accepted, polling operation...")
if raw_response.status_code != 202:
return self._transform_completed_response(model=model, raw_response=raw_response)
# Get Operation-Location header
operation_url = raw_response.headers.get("Operation-Location")
if not operation_url:
raise ValueError("Azure Document Intelligence returned 202 but no Operation-Location header found")
# Reject cross-origin polling URLs — the auth headers
# below would otherwise leak to whatever URL the upstream
# (or an attacker-controlled upstream) returns. VERIA-51.
try:
assert_same_origin(operation_url, str(raw_response.request.url))
except SSRFError as ssrf_err:
raise ValueError(f"Azure Document Intelligence: rejected polling URL ({ssrf_err})")
# Get headers for polling (need auth)
poll_headers = {
"Ocp-Apim-Subscription-Key": raw_response.request.headers.get("Ocp-Apim-Subscription-Key", "")
}
# Get timeout from kwargs or use default
timeout_secs = AZURE_OPERATION_POLLING_TIMEOUT
# Poll until operation completes
raw_response = self._poll_operation_sync(
operation_url=operation_url,
headers=poll_headers,
timeout_secs=timeout_secs,
)
# Now parse the completed response
response_json = raw_response.json()
verbose_logger.debug(f"Azure Document Intelligence response status: {response_json.get('status')}")
# Check if request succeeded
status = response_json.get("status")
if status != "succeeded":
raise ValueError(f"Azure Document Intelligence analysis failed with status: {status}")
# Extract analyze result
analyze_result = response_json.get("analyzeResult", {})
azure_pages = analyze_result.get("pages", [])
# Transform pages to Mistral format
mistral_pages = []
for azure_page in azure_pages:
page_number = azure_page.get("pageNumber", 1)
index = page_number - 1 # Convert to 0-based index
# Extract markdown text
markdown = self._extract_page_markdown(azure_page)
# Convert dimensions
width = azure_page.get("width", 8.5)
height = azure_page.get("height", 11)
unit = azure_page.get("unit", "inch")
dimensions = self._convert_dimensions(width=width, height=height, unit=unit)
# Build OCR page
ocr_page = OCRPage(index=index, markdown=markdown, dimensions=dimensions)
mistral_pages.append(ocr_page)
# Build usage info
usage_info = OCRUsageInfo(pages_processed=len(mistral_pages), doc_size_bytes=None)
# Return Mistral OCR response
return OCRResponse(
pages=mistral_pages,
model=model,
usage_info=usage_info,
object="ocr",
)
except Exception as e:
verbose_logger.error(f"Error parsing Azure Document Intelligence response: {e}")
raise e
verbose_logger.debug("Azure DI returned 202 Accepted, polling operation...")
operation_url, poll_headers = self._get_polling_target(raw_response)
completed_response = self._poll_operation_sync(
operation_url=operation_url,
headers=poll_headers,
timeout_secs=AZURE_OPERATION_POLLING_TIMEOUT,
)
return self._transform_completed_response(model=model, raw_response=completed_response)
async def async_transform_ocr_response(
self,
@ -680,81 +715,14 @@ class AzureDocumentIntelligenceOCRConfig(BaseOCRConfig):
Returns:
OCRResponse in Mistral format
"""
try:
# Check if we got 202 Accepted (async operation started)
if raw_response.status_code == 202:
verbose_logger.debug("Azure DI returned 202 Accepted, polling operation (async)...")
if raw_response.status_code != 202:
return self._transform_completed_response(model=model, raw_response=raw_response)
# Get Operation-Location header
operation_url = raw_response.headers.get("Operation-Location")
if not operation_url:
raise ValueError("Azure Document Intelligence returned 202 but no Operation-Location header found")
# Reject cross-origin polling URLs (see sync path). VERIA-51.
try:
assert_same_origin(operation_url, str(raw_response.request.url))
except SSRFError as ssrf_err:
raise ValueError(f"Azure Document Intelligence: rejected polling URL ({ssrf_err})")
# Get headers for polling (need auth)
poll_headers = {
"Ocp-Apim-Subscription-Key": raw_response.request.headers.get("Ocp-Apim-Subscription-Key", "")
}
# Get timeout from kwargs or use default
timeout_secs = AZURE_OPERATION_POLLING_TIMEOUT
# Poll until operation completes (async)
raw_response = await self._poll_operation_async(
operation_url=operation_url,
headers=poll_headers,
timeout_secs=timeout_secs,
)
# Now parse the completed response
response_json = raw_response.json()
verbose_logger.debug(f"Azure Document Intelligence response status: {response_json.get('status')}")
# Check if request succeeded
status = response_json.get("status")
if status != "succeeded":
raise ValueError(f"Azure Document Intelligence analysis failed with status: {status}")
# Extract analyze result
analyze_result = response_json.get("analyzeResult", {})
azure_pages = analyze_result.get("pages", [])
# Transform pages to Mistral format
mistral_pages = []
for azure_page in azure_pages:
page_number = azure_page.get("pageNumber", 1)
index = page_number - 1 # Convert to 0-based index
# Extract markdown text
markdown = self._extract_page_markdown(azure_page)
# Convert dimensions
width = azure_page.get("width", 8.5)
height = azure_page.get("height", 11)
unit = azure_page.get("unit", "inch")
dimensions = self._convert_dimensions(width=width, height=height, unit=unit)
# Build OCR page
ocr_page = OCRPage(index=index, markdown=markdown, dimensions=dimensions)
mistral_pages.append(ocr_page)
# Build usage info
usage_info = OCRUsageInfo(pages_processed=len(mistral_pages), doc_size_bytes=None)
# Return Mistral OCR response
return OCRResponse(
pages=mistral_pages,
model=model,
usage_info=usage_info,
object="ocr",
)
except Exception as e:
verbose_logger.error(f"Error parsing Azure Document Intelligence response (async): {e}")
raise e
verbose_logger.debug("Azure DI returned 202 Accepted, polling operation (async)...")
operation_url, poll_headers = self._get_polling_target(raw_response)
completed_response = await self._poll_operation_async(
operation_url=operation_url,
headers=poll_headers,
timeout_secs=AZURE_OPERATION_POLLING_TIMEOUT,
)
return self._transform_completed_response(model=model, raw_response=completed_response)

13
litellm/llms/base_llm/anthropic_messages/transformation.py
View file

@ -114,6 +114,19 @@ class BaseAnthropicMessagesConfig(ABC):
"""
return True
def handles_web_search_natively(self) -> bool:
"""
Whether the upstream this config routes to executes ``web_search`` tools
itself as part of its Anthropic Messages agentic loop.
The web-search interception handler short-circuits web-search-only
requests (running the search itself and returning synthetic results) only
for providers that do NOT. Providers whose agentic loop already performs
the search plus a follow-up synthesis step (bedrock, vertex_ai, ...)
return True so those requests flow through untouched.
"""
return True
def get_async_streaming_response_iterator(
self,
model: str,

29
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -2,7 +2,10 @@ from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, Dict, List, Optional
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.custom_guardrail import (
CustomGuardrail,
ModifyResponseException,
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.llms.openai import AllMessageValues
@ -98,6 +101,30 @@ class BaseTranslation(ABC):
"""
return responses_so_far
def build_block_sse_chunks(
self,
exc: "ModifyResponseException",
stream_started: bool = False,
responses_so_far: Optional[list[Any]] = None,
) -> Optional[list[bytes]]:
"""
Build the streaming chunks that deliver a guardrail block message and
cleanly terminate the stream in this provider's wire format.
``stream_started`` is True when real chunks were already sent to the
client: the result must *continue* the in-progress message (e.g. close
the open content block and append the block message) rather than start
a new one, which clients reject. ``responses_so_far`` provides the prior
chunks needed to do so. When False, nothing has been sent and a
standalone block message is emitted.
Returns None when the format has no safe terminator; the caller then
re-raises ``exc`` so the proxy can surface a clean error instead.
Override in provider subclasses that support synthesizing a block
stream.
"""
return None
def get_structured_messages(self, data: dict) -> Optional[List["AllMessageValues"]]:
"""
Convert request data to OpenAI-spec structured messages.

92
litellm/llms/base_llm/guardrail_translation/utils.py
View file

@ -1,10 +1,100 @@
from __future__ import annotations
from typing import Any, List
import json
from typing import Any, List, Optional
from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicUsage
from litellm.types.llms.openai import AllMessageValues
def _anthropic_stream_chunk_events(item: Any) -> list[dict]:
if isinstance(item, dict):
return [item]
if isinstance(item, bytes):
chunk = item.decode("utf-8", errors="replace")
elif isinstance(item, str):
chunk = item
else:
return []
events: list[dict] = []
for block in chunk.split("\n\n"):
for line in block.splitlines():
stripped = line.strip()
if not stripped.startswith("data:"):
continue
payload = stripped[len("data:") :].strip()
if not payload or payload == "[DONE]":
continue
try:
parsed = json.loads(payload)
except json.JSONDecodeError:
continue
if isinstance(parsed, dict):
events.append(parsed)
return events
def _usage_from_anthropic_stream_chunks(original_response: list[Any]) -> Optional[AnthropicUsage]:
input_tokens = 0
output_tokens = 0
found_usage = False
for item in original_response:
for event in _anthropic_stream_chunk_events(item):
event_type = event.get("type")
if event_type == "message_start":
message = event.get("message") or {}
usage_obj = message.get("usage") or {}
elif event_type == "message_delta":
usage_obj = event.get("usage") or {}
else:
usage_obj = {}
if not isinstance(usage_obj, dict):
continue
if usage_obj.get("input_tokens") is not None:
input_tokens = int(usage_obj.get("input_tokens") or 0)
found_usage = True
if usage_obj.get("output_tokens") is not None:
output_tokens = int(usage_obj.get("output_tokens") or 0)
found_usage = True
if not found_usage:
return None
return AnthropicUsage(input_tokens=input_tokens, output_tokens=output_tokens)
def blocked_response_usage(original_response: Optional[Any]) -> AnthropicUsage:
"""
Token usage for a synthetic guardrail-blocked response.
A post-call block replaces the LLM's response with the violation message,
but the upstream call already consumed tokens -- report that real usage
(carried on ``ModifyResponseException.original_response``) rather than
discarding it. Pre-call blocks never invoked the LLM (no original_response),
so usage is zero.
"""
usage_obj: Any = None
if isinstance(original_response, list):
stream_usage = _usage_from_anthropic_stream_chunks(original_response)
if stream_usage is not None:
return stream_usage
elif isinstance(original_response, dict):
usage_obj = original_response.get("usage")
elif original_response is not None:
usage_obj = getattr(original_response, "usage", None)
def _tokens(key: str, fallback_key: str) -> int:
if isinstance(usage_obj, dict):
return int(usage_obj.get(key, usage_obj.get(fallback_key, 0)) or 0)
return int(getattr(usage_obj, key, getattr(usage_obj, fallback_key, 0)) or 0)
return AnthropicUsage(
input_tokens=_tokens("input_tokens", "prompt_tokens"),
output_tokens=_tokens("output_tokens", "completion_tokens"),
)
def effective_skip_system_message_for_guardrail(guardrail_to_apply: Any) -> bool:
per = getattr(guardrail_to_apply, "skip_system_message_in_guardrail", None)
if per is not None:

3
litellm/llms/base_llm/ocr/transformation.py
View file

@ -70,6 +70,9 @@ class OCRResponse(LiteLLMPydanticObjectBase):
model: str
document_annotation: Any | None = None
usage_info: OCRUsageInfo | None = None
content: str | None = None
tables: list[dict[str, object]] | None = None
keyValuePairs: list[dict[str, object]] | None = None
object: str = "ocr"
model_config = {"extra": "allow"}

28
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -76,6 +76,7 @@ from litellm.utils import (
from ..common_utils import (
BedrockError,
BedrockModelInfo,
bedrock_converse_supports_parallel_tool_use_config,
get_anthropic_beta_from_headers,
get_bedrock_tool_name,
is_claude_4_5_on_bedrock,
@ -1106,18 +1107,28 @@ class AmazonConverseConfig(BaseConfig):
if cache_control is None:
return None
cache_point = CachePointBlock(type="default")
if isinstance(cache_control, dict) and "ttl" in cache_control:
ttl = cache_control["ttl"]
if ttl in ["5m", "1h"] and model is not None:
if is_claude_4_5_on_bedrock(model):
cache_point["ttl"] = ttl
cache_point = self._build_cache_point_block(cache_control, model)
if block_type == "system":
return SystemContentBlock(cachePoint=cache_point)
else:
return ContentBlock(cachePoint=cache_point)
@staticmethod
def _build_cache_point_block(control: Optional[dict], model: Optional[str] = None) -> CachePointBlock:
"""Build a Bedrock ``cachePoint`` block from an OpenAI-style ``cache_control``/``control`` dict.
``type`` is always ``"default"`` (the only value Bedrock's Converse API
accepts). ``ttl`` is only honored for models that support extended TTL
caching (Claude 4.5 family on Bedrock).
"""
cache_point = CachePointBlock(type="default")
if isinstance(control, dict) and "ttl" in control:
ttl = control["ttl"]
if ttl in ["5m", "1h"] and model is not None and is_claude_4_5_on_bedrock(model):
cache_point["ttl"] = ttl
return cache_point
def _transform_system_message(
self, messages: List[AllMessageValues], model: Optional[str] = None
) -> Tuple[List[AllMessageValues], List[SystemContentBlock]]:
@ -1241,7 +1252,7 @@ class AmazonConverseConfig(BaseConfig):
# Handle parallel_tool_calls configuration
parallel_tool_use_config = additional_request_params.pop("_parallel_tool_use_config", None)
if parallel_tool_use_config is not None and is_claude_4_5_on_bedrock(model):
if parallel_tool_use_config is not None and bedrock_converse_supports_parallel_tool_use_config(model):
for key, value in parallel_tool_use_config.items():
if (
key in additional_request_params
@ -1526,7 +1537,8 @@ class AmazonConverseConfig(BaseConfig):
if cache_injection_points and len(bedrock_tools) > 0:
for point in cache_injection_points:
if point.get("location") == "tool_config":
bedrock_tools.append({"cachePoint": {"type": "default"}})
cache_point = self._build_cache_point_block(point.get("control"), model)
bedrock_tools.append(ToolBlock(cachePoint=cache_point))
break
bedrock_tool_config: Optional[ToolConfigBlock] = None

45
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
View file

@ -5,6 +5,7 @@ from functools import partial
from typing import TYPE_CHECKING, Any, List, Optional, Tuple, Union, cast, get_args
import httpx
from pydantic import TypeAdapter, ValidationError
import litellm
from litellm._logging import verbose_logger
@ -24,6 +25,7 @@ from litellm.llms.custom_httpx.http_handler import (
HTTPHandler,
_get_httpx_client,
)
from litellm.types.llms.bedrock import GuardrailConfigBlock
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse, Usage
from litellm.utils import CustomStreamWrapper
@ -37,6 +39,38 @@ else:
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
_GUARDRAIL_CONFIG_VALIDATOR: "TypeAdapter[GuardrailConfigBlock]" = TypeAdapter(GuardrailConfigBlock)
_GUARDRAIL_CONFIG_EXPECTED_FORMAT = (
"{'guardrailIdentifier': str, 'guardrailVersion': str, 'trace': 'enabled'|'disabled'|'enabled_full'}"
)
def _bedrock_invoke_guardrail_headers(raw_guardrail_config: object) -> "dict[str, str]":
try:
guardrail_config = _GUARDRAIL_CONFIG_VALIDATOR.validate_python(raw_guardrail_config)
except ValidationError as e:
raise BedrockError(
status_code=400,
message="Invalid guardrailConfig={}. Expected format: {}. Error: {}".format(
raw_guardrail_config, _GUARDRAIL_CONFIG_EXPECTED_FORMAT, e
),
)
if "guardrailIdentifier" not in guardrail_config:
raise BedrockError(
status_code=400,
message="guardrailConfig={} is missing 'guardrailIdentifier'. Expected format: {}".format(
raw_guardrail_config, _GUARDRAIL_CONFIG_EXPECTED_FORMAT
),
)
trace = guardrail_config.get("trace")
candidate_headers = {
"X-Amzn-Bedrock-GuardrailIdentifier": guardrail_config.get("guardrailIdentifier"),
"X-Amzn-Bedrock-GuardrailVersion": guardrail_config.get("guardrailVersion"),
"X-Amzn-Bedrock-Trace": trace.upper() if trace is not None else None,
}
return {name: value for name, value in candidate_headers.items() if value is not None}
class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
def __init__(self, **kwargs):
@ -390,7 +424,16 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
return headers
raw_guardrail_config = optional_params.pop("guardrailConfig", None)
if raw_guardrail_config is None:
return headers
existing_header_names = frozenset(name.lower() for name in headers)
guardrail_headers = {
name: value
for name, value in _bedrock_invoke_guardrail_headers(raw_guardrail_config).items()
if name.lower() not in existing_header_names
}
return {**headers, **guardrail_headers}
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]

95
litellm/llms/bedrock/common_utils.py
View file

@ -4,9 +4,11 @@ from __future__ import annotations
Common utilities used across bedrock chat/embedding/image generation
"""
import contextlib
import functools
import json
import os
import re
from typing import (
TYPE_CHECKING,
Any,
@ -683,39 +685,72 @@ def get_bedrock_base_model(model: str) -> str:
return model
def bedrock_converse_supports_parallel_tool_use_config(model: str) -> bool:
return any(
(litellm.model_cost.get(candidate) or {}).get("supports_parallel_tool_use_config") is True
for candidate in (model, get_bedrock_base_model(model))
)
def is_claude_4_5_on_bedrock(model: str) -> bool:
"""
Check if the model is a Claude 4.5 model on Bedrock.
Claude 4.5 models support prompt caching with '5m' and '1h' TTL on Bedrock.
Check if the model supports Bedrock prompt caching with an extended '1h' TTL
(in addition to the default 5m TTL).
Backed by the ``cache_creation_input_token_cost_above_1hr`` field in
``model_prices_and_context_window.json`` instead of a hardcoded list of
model-name patterns, so newly released models pick up support as soon as
their pricing entry ships, with no code change required here.
"""
model_lower = model.lower()
claude_4_5_patterns = [
"sonnet-4.5",
"sonnet_4.5",
"sonnet-4-5",
"sonnet_4_5",
"haiku-4.5",
"haiku_4.5",
"haiku-4-5",
"haiku_4_5",
"opus-4.5",
"opus_4.5",
"opus-4-5",
"opus_4_5",
"sonnet-4.6",
"sonnet_4.6",
"sonnet-4-6",
"sonnet_4_6",
"opus-4.6",
"opus_4.6",
"opus-4-6",
"opus_4_6",
"opus-4.7",
"opus_4.7",
"opus-4-7",
"opus_4_7",
]
return any(pattern in model_lower for pattern in claude_4_5_patterns)
return any(
(litellm.model_cost.get(candidate) or {}).get("cache_creation_input_token_cost_above_1hr") is not None
for candidate in (model, get_bedrock_base_model(model))
)
_BEDROCK_MODEL_VERSION_SUFFIX_RE = re.compile(r"-v\d+(?::\d+)?$")
def bedrock_converse_supports_strict_tools(model: str) -> bool:
"""
Whether ``toolSpec.strict`` can be forwarded to Bedrock Converse for ``model``.
Non-Anthropic Bedrock families (Nova, Llama, GPT-OSS) reject the field
outright. Anthropic models forward it unless their entry in
``model_prices_and_context_window.json`` sets
``bedrock_converse_supports_strict_tools: false`` — Bedrock routes those
(Opus 4.7/4.8, see #31582) through a stricter validator that rejects the
``strict`` key on ``toolSpec`` even though Anthropic's native API accepts
it as a top-level tool field.
"""
base = get_bedrock_base_model(model)
if not base.startswith("anthropic"):
return False
flag = _get_bedrock_converse_strict_tools_flag(base)
return flag if flag is not None else True
def _get_bedrock_converse_strict_tools_flag(base_model: str) -> Optional[bool]:
candidates = dict.fromkeys((base_model, _BEDROCK_MODEL_VERSION_SUFFIX_RE.sub("", base_model)))
for candidate in candidates:
with contextlib.suppress(Exception):
model_info = get_cached_model_info()(
model=candidate,
custom_llm_provider="bedrock",
)
flag = model_info.get("bedrock_converse_supports_strict_tools")
if isinstance(flag, bool):
return flag
model_cost_key = model_info.get("key")
if isinstance(model_cost_key, str):
local_flag = (
_get_local_model_cost_map().get(model_cost_key, {}).get("bedrock_converse_supports_strict_tools")
)
if isinstance(local_flag, bool):
return local_flag
return None
def normalize_bedrock_opus_output_config_effort(model: str, output_config: Any) -> None:

37
litellm/llms/bedrock/realtime/handler.py
View file

@ -5,6 +5,7 @@ This uses aws_sdk_bedrock_runtime for bidirectional streaming with Nova Sonic.
"""
import asyncio
import contextlib
import json
from typing import Any, Optional
@ -156,12 +157,19 @@ class BedrockRealtime(BaseAWSLLM):
session_state: dict,
):
"""Forward messages from client WebSocket to Bedrock stream."""
try:
from aws_sdk_bedrock_runtime.models import (
BidirectionalInputPayloadPart,
InvokeModelWithBidirectionalStreamInputChunk,
)
from aws_sdk_bedrock_runtime.models import (
BidirectionalInputPayloadPart,
InvokeModelWithBidirectionalStreamInputChunk,
)
async def send_to_bedrock(bedrock_message: str) -> None:
event = InvokeModelWithBidirectionalStreamInputChunk(
value=BidirectionalInputPayloadPart(bytes_=bedrock_message.encode("utf-8"))
)
await bedrock_stream.input_stream.send(event)
verbose_proxy_logger.debug(f"Bedrock Realtime: Sent to Bedrock: {bedrock_message[:200]}")
try:
while True:
# Receive message from client
message = await client_ws.receive_text()
@ -176,19 +184,15 @@ class BedrockRealtime(BaseAWSLLM):
# Send transformed messages to Bedrock
for bedrock_message in transformed_messages:
event = InvokeModelWithBidirectionalStreamInputChunk(
value=BidirectionalInputPayloadPart(bytes_=bedrock_message.encode("utf-8"))
)
await bedrock_stream.input_stream.send(event)
verbose_proxy_logger.debug(f"Bedrock Realtime: Sent to Bedrock: {bedrock_message[:200]}")
await send_to_bedrock(bedrock_message)
except Exception as e:
verbose_proxy_logger.debug(f"Client to Bedrock forwarding ended: {e}", exc_info=True)
# Close the Bedrock stream input
try:
for close_message in transformation_config.session_close_messages():
with contextlib.suppress(Exception):
await send_to_bedrock(close_message)
with contextlib.suppress(Exception):
await bedrock_stream.input_stream.close()
except Exception:
pass
async def _forward_bedrock_to_client(
self,
@ -206,6 +210,10 @@ class BedrockRealtime(BaseAWSLLM):
output = await bedrock_stream.await_output()
result = await output[1].receive()
if result is None:
verbose_proxy_logger.debug("Bedrock Realtime: Bedrock stream ended")
break
if result.value and result.value.bytes_:
bedrock_response = result.value.bytes_.decode("utf-8")
verbose_proxy_logger.debug(f"Bedrock Realtime: Received from Bedrock: {bedrock_response[:200]}")
@ -252,6 +260,7 @@ class BedrockRealtime(BaseAWSLLM):
except Exception as e:
verbose_proxy_logger.debug(f"Bedrock to client forwarding ended: {e}", exc_info=True)
finally:
# Close the client WebSocket
try:
await client_ws.close()

146
litellm/llms/bedrock/realtime/transformation.py
View file

@ -4,14 +4,18 @@ This file contains the transformation logic for Bedrock Nova Sonic realtime API.
Transforms between OpenAI Realtime API format and Bedrock Nova Sonic format.
"""
import base64
import json
import uuid as uuid_lib
from typing import Any, List, Optional, Union
from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig
from litellm.llms.bedrock.realtime.trigger_audio import ready_trigger_pcm
from litellm.types.llms.openai import (
OpenAIRealtimeContentPartDone,
OpenAIRealtimeDoneEvent,
@ -35,6 +39,17 @@ from litellm.types.realtime import (
from litellm.utils import get_empty_usage
class BedrockContentEnd(BaseModel):
stopReason: Optional[str] = None
TRIGGER_AUDIO_SAMPLE_RATE_HERTZ = 16000
TRIGGER_AUDIO_BYTES_PER_SECOND = TRIGGER_AUDIO_SAMPLE_RATE_HERTZ * 2
TRIGGER_LEADING_SILENCE = bytes(TRIGGER_AUDIO_BYTES_PER_SECOND // 2)
TRIGGER_TRAILING_SILENCE = bytes(TRIGGER_AUDIO_BYTES_PER_SECOND * 3)
TRIGGER_AUDIO_CHUNK_SIZE = 1024
class BedrockRealtimeConfig(BaseRealtimeConfig):
"""Configuration for Bedrock Nova Sonic realtime transformations."""
@ -43,6 +58,8 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
self.prompt_name = str(uuid_lib.uuid4())
self.content_name = str(uuid_lib.uuid4())
self.audio_content_name = str(uuid_lib.uuid4())
self.prompt_started = False
self.client_audio_streamed = False
# Default configuration values
# Inference configuration
@ -247,6 +264,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
prompt_start = {"event": {"promptStart": prompt_start_config}}
messages.append(json.dumps(prompt_start))
self.prompt_started = True
# Send system prompt if provided
instructions = session_config.get("instructions")
@ -304,8 +322,22 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
verbose_logger.debug("Handling input_audio_buffer.append")
self.client_audio_streamed = True
messages: List[str] = []
if hasattr(self, "_audio_content_started") and self._audio_content_sample_rate != self.input_sample_rate_hertz:
mismatched_content_end = {
"event": {
"contentEnd": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
}
}
}
messages.append(json.dumps(mismatched_content_end))
delattr(self, "_audio_content_started")
self.audio_content_name = str(uuid_lib.uuid4())
# Check if we need to start audio content
if not hasattr(self, "_audio_content_started"):
audio_content_start = {
@ -329,6 +361,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
}
messages.append(json.dumps(audio_content_start))
self._audio_content_started = True
self._audio_content_sample_rate = self.input_sample_rate_hertz
# Send audio chunk
audio_data = json_message.get("audio", "")
@ -383,7 +416,6 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
verbose_logger.debug("Handling conversation.item.create")
messages: List[str] = []
item = json_message.get("item", {})
item_type = item.get("type")
@ -392,6 +424,8 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
if item_type == "function_call_output":
return self.transform_conversation_item_create_tool_result_event(json_message)
messages: list[str] = []
# Handle regular message
if item_type == "message":
content = item.get("content", [])
@ -443,6 +477,12 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
"""
Transform response.create event to Bedrock format.
Nova Sonic only starts generating after it detects user speech, so text-only
sessions never get a response on their own. Injecting a short spoken "ready"
utterance (followed by silence) makes the model respond to the pending
interactive text input. Sessions where the client streams its own audio rely
on Nova Sonic's built-in turn detection instead.
Args:
json_message: OpenAI response.create message
@ -450,8 +490,53 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
List of Bedrock format messages (JSON strings)
"""
verbose_logger.debug("Handling response.create")
# Bedrock starts generating automatically, no explicit trigger needed
return []
if not self.prompt_started or self.client_audio_streamed:
return []
messages: list[str] = []
if not hasattr(self, "_audio_content_started"):
trigger_content_start = {
"event": {
"contentStart": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
"type": "AUDIO",
"interactive": True,
"role": "USER",
"audioInputConfiguration": {
"mediaType": self.input_media_type,
"sampleRateHertz": TRIGGER_AUDIO_SAMPLE_RATE_HERTZ,
"sampleSizeBits": self.input_sample_size_bits,
"channelCount": self.input_channel_count,
"audioType": self.input_audio_type,
"encoding": self.input_encoding,
},
}
}
}
messages.append(json.dumps(trigger_content_start))
self._audio_content_started = True
self._audio_content_sample_rate = TRIGGER_AUDIO_SAMPLE_RATE_HERTZ
messages.extend(self._response_trigger_audio_messages())
return messages
def _response_trigger_audio_messages(self) -> list[str]:
pcm = TRIGGER_LEADING_SILENCE + ready_trigger_pcm() + TRIGGER_TRAILING_SILENCE
return [
json.dumps(
{
"event": {
"audioInput": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
"content": base64.b64encode(pcm[offset : offset + TRIGGER_AUDIO_CHUNK_SIZE]).decode(),
}
}
}
)
for offset in range(0, len(pcm), TRIGGER_AUDIO_CHUNK_SIZE)
]
def transform_response_cancel_event(self, json_message: dict) -> List[str]:
"""
@ -467,6 +552,35 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
# Send interrupt signal if needed
return []
def session_close_messages(self) -> list[str]:
"""
Build the Bedrock events that gracefully close the session
(contentEnd for any open audio content, promptEnd, sessionEnd).
Returns:
List of Bedrock format messages (JSON strings)
"""
if not self.prompt_started:
return []
messages: list[str] = []
if hasattr(self, "_audio_content_started"):
audio_content_end = {
"event": {
"contentEnd": {
"promptName": self.prompt_name,
"contentName": self.audio_content_name,
}
}
}
messages.append(json.dumps(audio_content_end))
delattr(self, "_audio_content_started")
messages.append(json.dumps({"event": {"promptEnd": {"promptName": self.prompt_name}}}))
messages.append(json.dumps({"event": {"sessionEnd": {}}}))
self.prompt_started = False
return messages
def transform_realtime_request(
self,
message: str,
@ -837,10 +951,11 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
Optional[ALL_DELTA_TYPES],
]:
"""
Transform Bedrock promptEnd event to OpenAI response.done.
Transform a Bedrock end-of-response event (promptEnd, completionEnd, or an
END_TURN contentEnd) to OpenAI response.done.
Args:
event: Bedrock promptEnd event
event: Bedrock event that ends the response
current_response_id: Current response ID
current_conversation_id: Current conversation ID
@ -848,7 +963,18 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
Tuple of (events, reset_output_item_id, reset_response_id, reset_delta_type)
"""
verbose_logger.debug("Handling promptEnd")
return self._response_done_events(current_response_id, current_conversation_id)
def _response_done_events(
self,
current_response_id: Optional[str],
current_conversation_id: Optional[str],
) -> tuple[
List[OpenAIRealtimeEvents],
Optional[str],
Optional[str],
Optional[ALL_DELTA_TYPES],
]:
if not current_response_id or not current_conversation_id:
return [], None, None, None
@ -1084,6 +1210,14 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
current_delta_chunks,
)
returned_messages.extend(events)
if BedrockContentEnd.model_validate(event["contentEnd"]).stopReason == "END_TURN":
(
done_events,
current_output_item_id,
current_response_id,
current_delta_type,
) = self._response_done_events(current_response_id, current_conversation_id)
returned_messages.extend(done_events)
elif "toolUse" in event:
events, tool_call_id, tool_name = self.transform_tool_use_event(
@ -1093,7 +1227,7 @@ class BedrockRealtimeConfig(BaseRealtimeConfig):
# Store tool call info for potential use
verbose_logger.debug(f"Tool use event: {tool_name} (ID: {tool_call_id})")
elif "promptEnd" in event:
elif "promptEnd" in event or "completionEnd" in event:
(
events,
current_output_item_id,

208
litellm/llms/bedrock/realtime/trigger_audio.py Normal file
View file

@ -0,0 +1,208 @@
"""
Pre-rendered spoken "ready" trigger audio (16kHz, 16-bit, mono PCM), generated with Amazon Polly.
Amazon Nova Sonic v1 only starts generating after it hears the user speak, so text-only realtime
sessions inject this short utterance to trigger a response (same approach as Pipecat's
AWSNovaSonicLLMService assistant-response trigger).
"""
import base64
import gzip
from functools import lru_cache
READY_TRIGGER_PCM_16KHZ_MONO_GZIP_B64 = (
"H4sIANGpRWoC/517dXQcR9Bnw+Duis3MzMwkc8xsxxTZMTMzM0PMmJhBjpmZmWKQZSaxFrQ80H0lJXf3vXf/nev17ExPQ3Xh"
"r2wPQv8/f/D/uMP/zxv8f64YkSyiWU1AIpCcRSqyICtQCApF4SgSRaHsKCfKjfKhgqgIKo5KobKoPKqEqqFaqC5qgBqjFqg1"
"ao86o+6oF+qDYtAgNBSNQKPQODQeTUZT0Uw0B81D89ECtAgtRcvRMrQCrQRahVajNWgtWo/+QBvQJrQRbQbagraibWg70Da4"
"2/rf7zbozxyxCcaug1krYZXFaCGai2ajGWgKmgi7jYFdh6ABqB/qjXqgLqgd8NUcNQUe66CaqDJwXQaVhDMUgNNkh7NZ4bSc"
"69zLXTyVJ/Kv/BN/z9/w5/w+0B1+g1/kZ/gpfoL/zQ/xA/wvvovv5Jv5Fmjr+Tq+lq/iS/kioPl8Fp+eRRP4KKBhfBDvw3vw"
"Lrw978jb8Fa8JW8B1JQ34PV5Q2iZ1wa8Ea/Da/Ja0Gpm9TTmzXgT3hzGt+a/wNz2vBPM78TbwtyW0NuIV+fVeCWgCrwyUHmg"
"yll9NWClOrwurNA0a42GMLYerFoN3pf7j8rwYkBFeWFehOcHKshz8dw8Ow/nUTyCh/IwaDYuc4mrPATIBs8WeJMNxmTnOWF0"
"Tp4DnsK4lYscc8405mNulsFc0FKYg9lZKjQX3KXDcxJck+E5haWxRPYd2k/2hX2G9gkonr1nX6H3C7RkGOmB1QjsGckLAK+1"
"QAqdeQwfD7Jdz3fzk/we/4f/5D4uoVyoGOizOdjbMLCw+WBBO1Esuo7uoZfoE/qJnCgARq7iUByGs+G8OGcWReAQbMUCDqBk"
"FI9eoGvoGFjVSrCdAWAr9VBhsHsHf8EP87m8F8gV88dsBxvFajKJvTS3mjFmOdNnXDMWG12NUoZPf6jv0afpnfRKejadaena"
"B+219kqL0xK0oBamV9E761P1I3qcLhhVjL7GEuOckWAUMHube02nWZ/NY7dYBFjJZR4JNnsRqbgr3o7f4WykNZlJDpEH5AfR"
"SRgtQMvTarQJbUZb0JZA7Wjr/1o7+gs8N6K1aRmal9qoSRLJC3KF7CHLyUTSl7QhtUkxkptkIyEklOQghUgF0oh0IINg/WVk"
"O9lLjsI+u8kfZC4ZQ/qQZqQ6jM9DchIbkQglfpyKP+Cb+DDejBfiEbgDrogx/gftQWNRbaTx23wBjwYbuMwmsfLsq7nZ7Ghi"
"84IxHs7r1A/rA/Wiery2UeukRWiPg4uD0UEeuBSYFWgQEAPP/dv9k/xd/K387fy9/DP9x/2p/mqB1QEjMCvIg/M1m75Tr2m8"
"NuaZ5dhrNoaHQnzIDpzUIzdJLXqS5hEmCVcFJpQXu4gjxelAk8XBYgexgiiLP4XLwmphkNBAyCkk0yt0Be1HK1JCX5JdZDSp"
"D2d7htfhLmAXX9FR0H5XVAPlQDp4/xN+BezsCHj6Xrge4cfA9y/wS/w6nPUxWN97/h2ihIObXISoWASiXwxEtDPIjRriTTiA"
"B5N3pD19RH8RbgilxWXiZ7GI1EUaLU2QxkojpcHSUGmGtEO6KX2UXkvbpYaSQ7wq/iXuEleKQ8XS4gdhsVBXEISjoNd3ZBzJ"
"S27hCbgMfgeRTUGHeXfuY6tZFfbQ7G+mGmOMJH2UTvRNWnZtfjA8eBjk+srfx+/x7fQ18/m8+739veW8yPvec8nzl2ejZzXQ"
"Ns9pz2dPiLeld4+3iO+pb6t/RWBr8IL2UY8wm7Pp/AC6huPIF/pOuC1ul4bKuZVDSnF1kfpUDbPUs/xq6WmpaZEtD9TJaqi6"
"QymlHJAj5KnSM7GYOF64SYvTtSSC7MB1cALE6WbIC/GyG8QaESJKAfDm2TyWx3EZInAryA9LQfZvwOor4c54PF4B8juGn+A0"
"HEWqgAf0JkPIeDKFTCOryA5yiaQSG42mI+hf9DvNLvQXrgg2sZ+4V/wilpH6SFulT1JReYJ8WU6QsynFlbxASPkiX5S3yNPk"
"0fJkebm8RO4ox0lNpbNia/Gj0BEsYxxYhZ1cJOdJE3IWM8h0vfgpFsVGmaeNaOOL3kK/ohXV2gfnBL74R/q/+rr57N6V3ibe"
"gGevp4fnm3uuu7T7bcaJjJkZTTMsGfddm11jXF1dfVyzXBdcpTOeZFx2f/HU8j33rww207Obz9l0VJqcoBXEQ1KyXEgtYgla"
"TlgL2ZbZ/rGl2GwheUKUkHhbN9t1a6S1iWWM2kepJxeTUoQLdD5pgWuiWnwWe282Ac3vNhZBNBllTDBWGDuNNwYzmpixZgTb"
"ysqBhJtDDOyL43E0nCw/HUPP0jTwlipCKaGQQIR4eotupcNoPcrJY3KP3CEmaUKnUZ22F24KMeItcbKUXf5DrqUsUxxKe3Wc"
"+lBNVQVLW8twy2kLs2S3NrY2tFqsryytLN/VieoLpaEyV/ZLG6VGUlDcKo4VO4uR4mOhs3CI/iQyyY3t/CmLNfcaF/TT2sXg"
"+0BUoLd/r4/4fvde80ieTu6/M1Jc0a5LzvzOBY4P9qH2bPaE9Pj0T+mivaZ9sv20XbfncNR3vHY8dHbPWOQp6v9VSzXX4wpi"
"HWWK5Zw1xtbd9t46yeq0PLHkt26y9rXlDukSEh1yz5ZgPWwppw6VN4qKsIL0wQPQJH6WtQH9Kmwge8em8ZFoNP6NtKDXAf5E"
"Co/AunLQzPiZi26gFYTzgiSGiYeEdHqKNMKneW0WahY2Ruo+7Yi2SzsBkT6P3lZfol/Tdb2+McLYblw2TNDEA/MhU5FANgnD"
"5BWWniEnw6ZHDIqcGZk7ck7EzfAe4ZXDK4WvCufhvSPORFyJGBnxPJyHpYW+DEm1mdYm1o2WoDpCjVOqKmvlEvJtaaqUXWom"
"NhLiSS5cmzczF+rZtBzBsYEE/xV/G/8nXz/feu87j+zp5l6TkTfjlquFq4DrunOUs6bzo2OsQ3Vst3ezlwdJNrBXg2uUXU8v"
"YRccp5293WX85fSd/C4tIt9UW1jnW7mljWWk2k5NV1Msw21KaL6wa2F/hrUPXWOba/lFmShlE2/SPeQNfoZsaDS/xtYyCRAS"
"Q71IN+qhbYRswl+0Ab1N/OQcfSa0l47Km5X3ynilh/xSjBIakivoL/6aDWcCm21+MmoYO/WC+m1tnrZUO6p91urq8/THeg8j"
"YHw28/Ih6DO+TW+KcfJE4KVw6O2wHeHFImIi9PDocCNsadiksPSwjLA3YTfCqoZ1Cz0fcsrmsi6ydrXOtn6z5rAxqwJetNBa"
"w/qPpYflndoRYtVpeYpUW/TRu2QktqKebJtxR7se8Phue85nNHGZjuoOh32MvYy9kP1D+iOgielqesv0kukF0wdCs6ZPT7+V"
"Xsf+1Z5gz+ko6Ryc8dHr19aiuaKpbrC1BaA02rpcfaNUV7tbGtkehFQJ9YSk2aKsK9SH8hWxibCFzEM9WG/Dqbm02UYCy0W+"
"CbOlmfJ4eZf0U7whrpAaKVUsH6wOW/mQt7bBtlfWn5azqqjYxao0G+pu1tK7a39px/U4Y7hZzaxvJGoDg7kCp/yJ/seBGE00"
"jplTeE9cmhYSp0mfpXjJK54RHgmzxDxyjCU8pFrYwvBLYUbIV2tXdYjslC5IyZJPPqUsVwOWYrZFIQmhxcECK4d+tcVa21kr"
"2mJDDobVi4iNiIooGUZCcljnqcWUSPmBGCGEkv68jtle7x787v/V53UnuD44CtuT0r+m77B3dMQ4tzvDnDmcC+3v01ukR6Tt"
"TU1NiUlpnKKkzk4bm97GUdn5wFHa2cQx3jnE0zJoQy7Rp26x9rP+Zsmh7lJKKHOVEeoPJUbpLn8SL4rVwffO0R0oYAwPCv6u"
"vqf+/noV/ogchlKFC7eFH+JB+YCabmliZdZctiW2DiH3Q1qFjg2Js2ZT44VpaKJRVOOBtYGMwG/BaUF74GwgIrg9uF7bodcF"
"9HZTb2M8Z+GkjlhfGaJ6lRVyZbEEnYC9fBZCdKT4WZYtsyDnzVRtymepqFQTOLog51Z3WIbZzoa8CG0YViwsLLRqyGXbUlux"
"0Blh9cIPRoRHvo14H/4+9KXtpkVTHssnpcLSafEvYSe1ksm8tnldEwN7vf3d91yHXYMztrtPex54FnoKu486v9u/p81Lq2Zf"
"51jhDHfddaxOb5w2K+Vz0vDk2sk1UtypQ1MrplZKr5ihBYeRnmopm2RrrtaX5otLxFNSYbWvdZ/NbZmjZJPzilNoE4xMR6CJ"
"d7a7jO+e9pYdJctETTwtxNB89K6QJEdaZ4WsCWsUPjl8emT5qKURR0K3qIwW5tX1ZsHHwdMQ15jWMVjMH+mb7X3oUwLvA9O1"
"3kYF3pRUFB/LLdVcahl5s5CPHAL0nIRvkXByGM1k7dkhXhDF4TkkkgSIRNcLprRV2WM5bb1pfWxZot5TvZarNlfIr6GvQ3rb"
"zlrXWivbqobst32wVrW+VAurMapb3Q8Svw6RppVYXahEz5JpaIy5WGvk/8tb1fsNUEwzb0fvBPdJl+o8YGfpetqxlE7JD5OV"
"ZHeyJd2bylJbptLEbgkzEub8nJrUPqUa2F2T9J0Zn/RE8aytctigkHVyfmrisUKI8kW9Z+mulpSGkXGsvDnUSAi6vH9nrHYd"
"zNjjL8C60fFSXmm+MFxoJc6QhlpWhA4Mt0SWiwyNWBs+MnS+db9yQtjO8xjN9ViwpSFGV+2Cv5D3c0ZOT/NAV8PkB/FG8oHO"
"EHaJQ+W2ynO5gtiQLIQKWoUKeTeqz18YX4I5AgW110YaTyd++kpoK42RXyjvLZutO61rbKlwHWfbZntqq2UbaTmtDJDPSivE"
"Z+Im+Z1yXm2reuWncmu5lrxIvqx8VzR5tXwEUO99uh3FsbV6seBNX3VvVY+asTVjbUaM+45nrPuOa7Kjpb1C6viU96lbU3+H"
"bLAqvaj9YdqQVG9S++SMpEpJ+ZOvJfZPupU8zLFBmyuWCskRut26Rub4LkvlbmGuckC9r1SUDpJu5qLACt8Zz09XlPOKc73n"
"czBonsPRtBO5C7V/FD0rb7A5w/dGXc/2W9THsDO2CEsZOU24Txriv9EP/oYdMO/q6wKPvfc8m7xlghfMmmSdWFdC0kBxn9hf"
"6iwtkBLFPmJNcZjgpnXJYtSLbdE2Bry+rv7DOkVf6BbpqhQQZ0tHpRRptdoSLOpL6PzQyyFNbYssJRVBuihECcWFC5Ku/KME"
"lFFKafmlRKSOUi55plLUkmLxWH9YTXWaPFdcCRmpBA81rgU/Brr48/sEr83TMyObe2LGtYyKGdQlgVfOs/9m/9W+1RnpLOTa"
"6yzuLOOw2KemD087n5or7XJqVNrxlI2OzcEmwjXLGltTWz7rXsmHR6D2Qg85XGmu9FduSHVJvLbYW9Q9OKOFu6N/oLGTj0WD"
"+RjeEkXjz2SfOF5Ntk0NC4scEtUj8n3IA6Wl+E4whfeCIrYQPuKnrKYR0P4MouDvwWjdyk+SseIieaKsgLzaiGWl3+UcgL/t"
"4kmxoxCPu3HNGKiN97cEVHAv8IFtIa3EpVJuqb+0Uv6hlLXeCu0R3i78aHjZ8KuhD6wB5Z3UX2wijhFVmaiXFEWeJM2VqskN"
"lGfqaUtXywBLJ8tjS0OrWx0sj4DYeAQ/Rk9YUb2Xv41HdI/wxPk3BqoHznkPuG46cju+2rO7Jrlfuld5Orh7uMY6UtNcKcVT"
"XqRMs/d2fAFLy5WamPwlMT4h2rmcpakTbLct9eQUWhg9QOWkebarIdstneUV4lK0UkvwXM8o5U7y7AycMP38lrkkuMbPgkvM"
"R/ik/MV6JvRC2OKw26HlQ2pZtsrvpMUKs3yw3Fbmih5eTTvvzevt6m+vjTFGQs1o5fXYXwyjFrgHnSp2lgeotS0n1HxKfekG"
"nYldPC8aiUrxDuYzbV1wj57MMvA3GifIYkF5gtJC/R3QbnnrGdvvIYNCRoSsty2zrJEjxDHCNGGcmENi4haxkVhQbCkmin+K"
"1aQSck8lQemozlOGSSWFFqQCNnkzNkDPHsjh++p54tnofe4p7NnnphkhzucO2RHpPORIcrRycecl++r0hmkd0i6nXUvLbk+2"
"b007n9wv0f+z18/eSdczTqCnwEV1eSjNiYuwYby1MMYSb8WW5/IF8Q46qKV6DrivePr79wSjjFxGpDbKN80bHqhp3ECquFZ5"
"py5XO6uJ6lJLH6jdHljKWT+ABMrL+ehD9lgrGiyiLTT2mR+Nq3p+QJUHg80NiU/A2+lyyCo7lA9ST7G4oJFtpAytJ8wX2pN1"
"bKNeWauqt2Nf8HlaR9SEUmJJKb/cTj6gRFjyWX22waHZQh/aBloqKWmARLLJY+Rlsk1OETeLzaTfpUZyqNxdKg4I5Kh0XEwX"
"2gr5hFPUiVvxFMMZXO2zeX96Xnp+95z0rPBU8sx1j3P5HGccVR0THCcc+ZzvHbvBzkTHqfS/0lhq7rRHqS9SHiUfT56WJCVO"
"TinnsbM4saFcU/ydKLgl7k5LyonKCbWepaKlitKcxurEl99d2X3Jt16/ZlbVW/o2ua+5a/h6Bp+bY8k46aO62/YjZKtttGW0"
"ck8pad0Y2jS8avhj2yC5Iz5ohhvT9C/aJL2u3kZfpGXTbgS1wOBAC20qy0PGCNehIh1J+9PG9D3dJBQUDNwY70EneF+eyErx"
"bmgk/5v7UVN8nRQUZovD5EbKW6WQpT6g7t8tFdWDUGVvV+up1dRR6gGlkdJPOSvHS3OkKOmq6Befi7PEOKjLiJhIV5BsuBeP"
"NQO6qB/S2gUf+Gv5Zd8nTzvPM3fpjDEZbd0lM+yueq5RjvaOk/Yj6eUcOR2b06um9Ujbl/otJZAyJmVHct/k8Ylm6iNfPL8m"
"vBaX0p1ordnabEZqWVjotfAjEQNDk6RkZg9sDEzQn7JCZCfpx5r4H7lyu2q7b/mi9EW8HDWB1gq9hWV0jlBL2WcLhPYM/xg+"
"OmyRdbtUmJbDyxGF+nMiyYcKGPmCBX2LQKtj/cP10eCPV9l9VpdNNZuzs+gQHSzekcbJ/SQH7Yr38VH8Mxol7FBmWpLU29IY"
"KuMfqBn9JtUCpDrBgi1vlbbyWOkPaay8Us1u7R1SPjTNWly9IX2Vsit31G7WrdZfLS3Ur1I5KVaYT0fSKjSO7MLvmUNvHGwW"
"OOO/4nvtb+Xv7D/rO+dxZSxxDnDUdPZ3+jOKeVhGWsZOxxzHLOcNRzVHFfu+tI72nvYSzvN2nLYh+U5i/bRX3oJsjbBOGMP3"
"aFv9/Q2H8FvInPDVYTVsWP1NmihZAOk9JUfJCuGm+EkYgmP1OP8dL/Jv1xawW7gj+Qc/RJfZIfYnukvbyolqYVunkCe2RdZK"
"lpxqbzla2gtoNsz6QE0QFdzD1PQocx27z/2skDlK9wQHBI8Ex2pD9InGZvMMX49aoVb8EPOa7dkPvpjWlG+rpS3V1WxyEbCT"
"dKmv+tLa1bbF2tkSq4yS8ggzSXVynJwRFkh15GxSSZqEuqC9uK2wTFovj5SvSjOlN2JRsZ+whBakaaQSbUkvkTr4BDtrtDRK"
"GgWMMMOr/2oc02toAwMj/O39NPA60Dp4NoACjfzcN8g/33/eX9Pfz7/LPzJQIdDN/8i71PvIq/na+6f6fb7FvrK+VpDB6wXL"
"6T2NXUayXkq36KqxyVzBf6ApuC9ug/fg2aQY7Uo70qm0mNBHHCqFyPelUCm/+EYYLo6Ucsqq3FzaJVYRK4klxThhidBYaCa0"
"FiYLc+HuJq1I15NhZBcJp9VobjqHfMF/4K34LN6HS+JDaACajY4CXv0FHectuMRdzMds/CNY4ho2iDVi9Vg1VpR9M/+B6h2z"
"7mw2W88WsXXsFrPwxfwHb4I+o214FqlOT9FywkqhipgkXpNmyW65PlS4awFreJXcaj91rjpWbawSNZtaDjDtVSW7ckPeJJ+Q"
"H8lz5ZbyFSlacon7xV/FIuJrYa1QWfhEe9PDgJhHYILvAzYcjt7zXHwy1MjXzbNmZ/O2Ud9I0LkeYvyp59evaCF6V/2HVl+7"
"FFS1odo+bZWWR3MGfcHcWh+oD19oH7Uw3dSi9G76Dv2+Lhn9jM9GTXOYudzMxY6z3jyBt0NnUGmcgJeTNyQDNL2KZhOmCjeF"
"/cJzIVVIFz5BZXUCetqAVOsKOj0K8hxKB9ESdCqpSsaTg2Qy0fGvuDy+jQlZgp+hTegaKoPD8E70htfgE3gldB81Rvl5GkM8"
"hl/ij/g4OFNP/op3Qh3Rc76fS+hPFIeeo1j0BMWji8BREfwPfoR34qW4J/6EB0HkOIcL4J7ER6uLTQERHRMKiUfF/NI06ZM0"
"UE6Rdfmn7JQny4XlgDRIVpQIpZscKp0SPdJguZvwinQVeymX5FhiomJ0txAmuPAEEi3cEeNJB3bfIKiqsFFYgzqbETw77sH7"
"m3HGbUZxU15AfxJYbj4jt9DcQE1Pu8AEQA+GVs/XxP3I89pX3dfWc8izxVfQ9ynjJkTWFs4Qd3bPNvfbjJ0ZroyJbuQu7C7p"
"669PRXdISTyC1TNv80pCbmWEJcaaCnk3TGksL5AXKw3UfOohiGOfxIOCHw9jHY1nxjeWD6Ww7sbrYOFAw0CJYIhe1jzDCnCF"
"tTE/mJv5e9yUWoQVgiFsFpk4TmwmCDQnrSnMEE+Jd6gPcLLf+M2MZTX4HLbTKA45aoB2XCuiT9ZT9KrGZOOm0d7MzZqxsUzh"
"VxChX+gGmkSO0q/SHMs72y3bQ8vfag+L23Y47GB4IIyF/hXSPWSibYxtrO2sLbstzXJWraO+VLyyJNcBlBgqD5BbSNuEUPqV"
"nKIvwVZa0f34CLrOh/BjLM50QcyYpY/TfIFFvl7e7p47npKemu5Ql+z43V4rfaq9rn2m470jxWHY+9gD6Rn2ac69rnoZ21yf"
"nE8dCY7Rjp7OEMc654yMyoFpbANgg799Yd5xwSn4jFLWFmodofwqrKbJQnO1ZsjQsCahf1vbK1ukW2J1MVY4KcwFO9kudMOD"
"zc6Qp8ELeAymGHGXPkILaJfN3XiesEDcKRSnPch0spdeEqNkopyXndJj0SL0IrPwc2yjOv1OP+EdbIveReun7zQT2F2WYTzT"
"OgVZ4FZwmE7NLhA5Nppe44KBWU/+FY0kPehPuoLaqIcspZWFdWKsfMyywXbEdkBdIbeVN1h+C8uIbBDVOqJ9SJp1sG1m2KjI"
"6KgGkRfCZodE2wxrrDVo/cP6wTrXGmrppNjhLKOEN0J7abTUWcT0JW/DnpqLWCLfxAexNvqXwFH/BH/dAA52CTz2j/HW8XQG"
"lLnD293bx1PD43V/gadmnuvuhxnRbu555Z3pu+BZ7lrveuh65on3fvTMz3DZ36UfcO7z3te7Glv09to37atxlX8lqXK0Jdq6"
"3XLOEms9YIsIuRA6I+x0+NDwAaHRNq5slffJR2Qmq1I+YQr6aAzQl+kB46d5wLip19RcgRrB6dpdoxl7xfbwKMRRbfwb/pPs"
"p5XFNGmjlCB2E24SCy1KI0FTC2lu3IntN44bb4x044QxXN+rXQxUDZQIJPsbBotqHfW7ekv9oD7COGpW4xfRX+QSXSh4BK9Q"
"XzwjSvI4Ja/6UPkgf5RqSM2lF1DlfJcaSYOkgdI0eZTUWxorFZEHKK3VDLWL5ara1nLU8tmaIyQyJNVW0NbaehckftzaDk7K"
"1dNKXXmrVFBKEqPEAO1FC5ICOMh/5cO4G07iN8eZd40YyIHVjdzGMWO1EW6YelP9qH5Hf6/XNsrqY7SZmk+rb9zTYoJ3/N99"
"Df2T/C8Dq7SPgak+X0bLjK/uEr6R/k3+Hd5ZGcmO2/aSzrYZZXzegKwtDxjeBb6+2p/sJCkF3nxTXCMa4i35rGVZSPbQ1qGX"
"bLr1hWWv2lTNrf6i3lZ+in+SQzzNzM5izR1mE7OF/jTo9yf6EwOdtK96DVMzC7IE8xY7juKJKnaUvksDZIu8Vm6jvFA2qyfU"
"HGpRpadUV4wWooU8wgTaj1RBSWyYed/IYVY0b5g5TMmw6o81ppc3j7E/+GP+FOJ1EFUnOWgd4bQ4X5okHYT4c0s8DDX1KDmn"
"PA+qyGNCUaGTUFvYQJNJKGSFEXgjqoVa8OmQOynqie6iUtiNp5Ph9BJUtyOl7vJ+ZaQ6Sa1guWIRrUnWN9bxttfW65aClli1"
"r+pRHsjT5ZJSM/EFrUQfkglkAN6B7vETPJ2/52/5CI54bV6GV0P90Xv0GvVG1RFFQV4V18VLcF4UZAvZUraJSXwcY6YXrOpc"
"cF+wmjZYuwvect230fvYG+W96W3stXuXeZO9g72jfdhnerODx2ieGH/LwIGA7o13L3ZJbp/ns39y8F6wTGCd76gvEDxi1kHh"
"uBtksHD8lZhCWfmJMkph8g95qDJFWaeUV+Yrk2SvtFE8S+/i+ugyyG8f97L3RiftRKBioFPwnvZOf6pP174G04O59QqmzNei"
"vWgMesGroI54EJ0LFv6LcJdmh5PXhXiiCDPpFvIJsmsRXodnwNkqsjNmS3Oi4dSHAzZYaNSGDH/Q/N2MMFeb81lfNA5bSDw+"
"iC/j4WQlHSLUESPF08JZ+gQQcZTAaFPhDBVpA9IJf4As3BNvwX6UyntwkS/lW/gD/pKf5PlQTkTQRJDtZfwDHyOlSVFSjvwk"
"u2gh+ZY63BqrtBRvCLklXW1giwm5Z30ia/QsLS19scyw7lW3i4/xKvQNmbSJ1EJsSbsjJ3sC2cxEGvbgDngef8Pqs3GsJMqJ"
"LbQVOYiK8aGsNHei4kSmS/BBHg8WWYZ15dPRH3yXWc7YqjXWFL2+cc7w6D+D4wO5/B/8CwNbg9eDawNvfMu8y71nfZ/8yYGQ"
"YJVAuH+dL+hL9r8OSNru4NZAG8Cgyb7K/iHglaavju+NZ61nmfeQ/0bgQ4AGogNnA5u0O0ZDbgHc4uQd0QXcX2gnXZRySylC"
"vBAQfpME+bX0u5hBF5I5+Djglt7oGy/MT5qTjIL6ccizTfUJxndjmPGrflOL0IdB/OsAqKY72sdX8UkgwdV0hDBAuEcdZB/J"
"DnKaKPygfahIOuI/0G20D61Cy3gO/sP8Zo40k81TZln203SZO808LIZl4624xsuiQ+gn6g54NgeZSl4Dzi1L29OF9A96gQ6m"
"PWkZWp6eIevJIjKQNCTpeAh+DzY1DC1Dh9F2NBhFoIN8B//ER6FkZMWv0Snwyc+oLt6N47EXNFyG5APMlRcyZRM8F88B9DUN"
"VUWf+Tn0lGymi2htfBAsoh3ZQI+LE0VVmItr4MWkgrhPOif+IswhGjbwG1oBcvAq6sFVcB2chseSAnQJ4fgFmouao79RNL6B"
"r+LO+BmvxPvBiS5Bj4pdfAnbY3KzIH/Bz4NdTDaHGSWMQ8ZaswfrYYYZT7VnWjc9ylhotDEW6wO1O5AZLmiadlcrpcnatuCc"
"YKvg8uCtYB1tp7Zfq6sVB0RcWOurubRhejP9kvY4GAjm1dZoKcHdwa+B9oE431jfLX/rYDDYJZgW6BS4F6ivXTR6A25dzXez"
"KrwsjhS+iYa0X7oIOL28RJQNUJ8fl03psLhfKCu8A83NJs/xO/SUt2A1TQm8DvKfqZjTjRCjqNHT/MEOg9ev5PNA3m9REO8E"
"NFBHWAzzFtDFQmnxunhPdAoXaCnamJ6morALPH0XyLcxmoI+ITvUAU4+lecDC1vMG6NnaD9YksZrQ3XQAktkA7lOupAK5DeC"
"6T+0uTBCoLBCY1qf7qC1AIlFCvkA2dwFNC2CHTzG1fFKpPNlwM02fpd7uJcfBPlrrBTE0rloCrlLCN2FV+MreAf9IVSVTHGU"
"aNI0ekEoJFWXR0itxLpCPL0D9b5T2C10oJOIieuQaWQumUU+YahV0W9oLWIoDb1EfVB+tIB35Im8NVqCBiEVjeBNAft/56Mh"
"lqSDN3xjn1kDfhsqkY38LcvDyrOS7G+WDFSUXTTzmQ3NoeYnM948aYabF4zZxjJjr+ExrhlrjMZGFcOlNzfyGtkhC/cGSRcy"
"DF3XX+g39WqGYUjmPWORscAoYqwyJhjbjQ+Aek9BpXkCaiMXRH9Zp8ZnvYeeDlXSDn2LKfEcqCiKBT4eo5NkidBa7CEmCYOE"
"CUJZMZdUVfoilhJLCoOpi1yCk4aSebgfeKmfLWBlWS3WkxVgDyDmTjefm9HMyabwhXw7j+V+QNeHcQhpCxIqC1X3JFIeasZH"
"oOupJCdxQUSeDh6ajNvh2agMYmDzSZDVXvO2/ALgjHdmPkZ5V96Mn4PqcxeL5PGQFRag0Wge1K+FoaJ9g29BDLgD9ZJK+pGl"
"pDv47Rt8AE/HM3B9XBG/RVsgOiXye4ATN4D91AWPSmSPGeEvWSrLz/vw8lCZKVCpreU/uZ1f5M/5bIjLR8FXn5G+OD9+hr/R"
"cDFB/CKMoflJFdpPWC01kz4Jg+hkUonE0F1CmnCbRpGf6ADEj1nATUWcG5Xgw1lr9hL2ymCbWV7GjarGF+Oc2Zc1YGvNw0Zl"
"o7MRajrNtuyC6TdqGaVAQwUg0iWZVcwWRub/kSpnjDBaGYn6bEBWrfWaemn9ofZKOwTe30JbrH3VtukrdFnvpaUHPwYfapHG"
"buO23l+bERS1QfpOI8FoYVzRNgUnB4fq1c3b5gTDDk8tgh20WP2AEWkc0ioHjwRqB84ECgdzabX0aVBHm8FhwU3BNvpHcw/4"
"wmIWYdYyv7OLWBDaCXPJSbCNENyZVhIThLx0HqrH/+RV8ESSg0xGvVl1UzZfQ4Um8pHsF7OYEdSnGtPMOnD2juwzVO5fzSfs"
"B98OmX0haohKodVoEG5F/oQa+ig2UDbcEGrwAmQcXgQV8Vq+HuJFgJ/mRfmvLNlMNXuy2+wHm8/c5iPTY66ASroxj+ClOGdN"
"wFIqo2gUwKWon06k+0k0vSdYAH80VprLJQBhF5WvKbvUi8ouaZyYU2wAttxYGiEuojvxLaizk7CJo/FRHsZGm6/MBawo38QO"
"md+N341NRjtTZArbbJrGTyPJKGHOMLeaG8w+ZoaxzmhvdDfyg34GgCY362+1G1qaVhF0lK7N1eppWCukjdUua7u07oCE/9Yu"
"aY+04lB7RulYb6iv1f36Ez1WZxDF8xtljUHGTqODEWF81csY0fB8yzhvtATNn9bP68eMp8ZRw69P1wfoY2BORV3TdmvttSda"
"DtD/St2mH9WG6E0MK1TFPVgFVpltZFd5PDqJd+D7+DTW8ABikmq0CN1EOpM+pAy5jffjXTgP3gY5qAh6D9k0yA6yADvPtrAB"
"rB3ryiaxdNYKUGVR3gAiZHPIyHlxBdwCb8Yy+YMECKI7yQVyk1Sko2lniml7cgnk2BM8OTeJxhPAj1uDd37OnIcW8t58HQ/y"
"pqgGikQPOEJ7kIBDwC/forKQM7dDbGgC8fsFROcA3g94oL+gizHSXvEBDRO4sEQ+qvRRo5RYQFfHxXrKR7WKmiAFhexCTaG/"
"9Ctg+TNCfTIMFyHT6Sc6nwZxKJoBEaQmPoXjkIx+5QIgiWdZ/z9tOrcD6nrKPjAXa80L8sesGgsCpkgzK7LZLJrNM+1GqpHd"
"3GdeNqeapc32Rro+3GgMz83NP4yuoPPFRnmwv1um1exnLDGY8c58b640J5oPjUuGag4yR5uFoAYZZa6A3wwjxGxh7jYlFsZe"
"mH+aE0wCWPU31oodBQ4UvoddAzk35GtAxicgwiWznFAfVASMUZUfYe1ZDXYKeJ3F3piNzApmN/O8mWJeMTuayGxjPgZLvccO"
"sBOsMh8Lc74AbmwNES832Uzy0xi6mt6nNkDN+4UzQpywWTgEGNovHBGWC4uEacIGYYXQWsgvvIEs5qTRQlXhDh0A6GkT/Uwj"
"hDDhHO1LR9A9lIKMswkfICcvAmT1lHrobcjZeagVcFs8OUQWkpLkOF6J/8Br8C84ATDUTPQraomaoCoomZ+BU+3kY/gQPpoP"
"4Pn5a+ZmYZBn4yD6XmYzWQfWh/0JVrcSbG4Q6wTeHcdk7mGxEOcNFs3nQBwewpvwipBHdJ4TneLt+Ax+E3Lpab6cl+aJIL/T"
"7B82mjnN2SCjbmZuwBxTjOZGTfCmTB/abQwxphnfjJVQHeaFODLf3GN2gsxaAmTdgSUCqlzDMmP/FPan2d8caNrYNjaDOSBa"
"5THXmB3YWeBvBLsPntWTPWCD4BTPYN5A0GIhfoyf4934PqaZ7VhuyC4ePoEfg3eH2G98EFqBHMD9L5BNjqJeeBXuiL+gE6g5"
"5KjF5CHIrT5g0t+JQAfR6bQLIJVcgFcH0aX0OX0GuptBp9FdoEFF8NFXUKXsAVzzmJo0lxAuIEDK72gcdVMXzSOUFpLpXnqc"
"vqcBKgspNJ4mgo7O0GP0Mj0H1610Dm1NI6lB/iEXyRrItDlJCkSFvoCCnOgaZNHeKDs6C3KtDNXObfYXG8PysX/MhWZb8zez"
"u1nHDBiPjT+NbXANMweDV3SGqjjaXG5WAinMYb8wysLheoF9Z59YP3bHRCChC2wR42Cru83rUEHbANEkmTpg+UKAbxLN/FDn"
"bGI3IDdeYW1YX9DgHL4L9D6DTQdrqMj/gHqrFlSqMvjqKf6RL+CxLDfkkO6sC8g0lV0xx5onzPqgncfA7XKzgfnSfAkYys2G"
"slxgSRfZYO7ml3l2/gkqqFjI6REoFOLcfrivgOYDTUTd0Ep0ErBlFTwI98R9cCwOJ+1ISzIOsA0nrcAb9tBVdBlIuq1wVYgR"
"GgrFhYrCDuG94BO2CNWFdFpDOCDsgTc24Ss9CXrjtKTwnY6lw0GjW0CTo0GfPWgL2px2h5o9B5UgTn4mxyBG9iZNSSo+D1G4"
"J7ZAvmuOCiGM4gCbNOGP2E6wycxqPYaVY5/MJ6YXrLQAE1kEK8JGwpt1bDf7BtF5KoxvxkuAbxREf6HJyAOy6QxW+BdKRfkQ"
"Bvm9ZHY+BCLgHrbRPGyeZD7+hhfgCeZ4sybrwXsgO7/BcrLGbDHvDRK5xW+x9Swc8Noy9JaH8f3sKqvAu/OZPIT3Y7VZL5bE"
"7oDH5mdW0Fpb8N+hrAlzmcUga5wDPxwPmf4j+MV4sKUVIPvi/A5vC7IuDRX/Rn4B4v8OyP3loHpdwHOh8+g71HgR6AO3gh4a"
"44E4HY0EnRTEx7AbqqKWuDTuhr/g/qQu8eG1gPFU0gFQZyNSk+g4P0S5O+Qc5LPGgPoXkPPQJpI2gP0WgH/NIc2BOoB/7YAZ"
"MaQaaUbGQN5aQlqRPKQErNiF1CGYnMDH8Vf8Cf+N22IRJ0EV2BgXww4UA/KrApzfBn7qgd084gSeqwO37yDuzIOYdgEs9A6c"
"sy54+C98OOS3pTyGV+HR0NOfb+Vn+QNAv1P5Zs4ArRZBtVBhVAxVg6ozFs6dAtcZaCj6A71AOXADQD/zwQZi0D9QzXaAKDoD"
"0FFtFI/64SlYwNXQd94eEdwf94J3ZaFyqYycaDzOh7cgBSE0AOJKdvwTdUZfuJN3hrwbD/VtEcTBBzqDhb8EVDoD6qVBaCfS"
"kIyj8DW0FWT+DZXANjivHR0DPrzoF1wV18YI5JAT+gfgybg3rgnv2wFPeyBLL4GePhDJNuFrED+24EOABg7B3TnQzVl4vxIw"
"xd+AfXeCL+2E380wFmo4sPIL+B6+iY/i61B7fcXpMP4ESP0baPcFfgn19k5A6/vwZbwY/4qbAw4cj3/HkwBDExyGa+HfQEIN"
"sQR6aYy742kQSfNhDjh7CnjuQOBPxSWhuh6AO8HJ7qPHyAZzauMfYGFX4akYrgZzn0DN74c5Q2F9N5xVwiNhz93AcyWwtNX4"
"FQ7CGWbiIVBP3McZ2AOn7I1H47v4LT6Jx+GyIIVecP4FuD2OAC1UwIPh7S+YAo/FYeQ6OPkUsNhieAR+CLXINuCtLm4C1xP4"
"COyVDeuoAIxsB/uXwCnoOHoEVpADZP8CUNAZdAc9QXHIA56bBHyGg9YzUC7cFFaoB3G6GlBFeOoBZy8Ke/qRgq1YQ270APR4"
"Al1Cr9EhqHJmgkedh7sD4Ge7AGUdQgkoACgsDr1D6VmaL4Jz47xYhhq1DkgtJ1h9KPSaIJGnMOYruoXuocuQxcehsWAzM9Bc"
"NAZNh+sk8M3xsGLm10br0Z/oAljwKRj7Cj2E2BMLfFyD5+3ob5h9Aew4HnoWoo5gZ3HAwQPwoxhA6KPA0r6g56g9VHkR8HQQ"
"bHUVaoUk8KwY4Pwi6oWyoVyoDRoIz0MAL3ohducCPFEc+fg/PJVbsr7LskFzAJ7rhPqiquAPAioP/tMcVYZxAYhxGbw8zHXy"
"q4BAbvJX/BvEtDi+B/LzTOi5Dv47iQ/iXSDa9uUT+TTeizeCyDkK3o4E/LUavPYoP86vwHUt5KVrsO9tvgXmDOBzwdO381kw"
"PhQwc33o6Qy45ht7zt6yCMApeXgG5P9tkJdUHgkY4x92GCiOBSGuMvYOMtMz5oPa2M88gPzesIcQo++xF5AHr8P1NvzugAz4"
"AOgji4fK8SWsYGcCz8utXIV6pwjE+/K8NuxUCCJsLdi9HUScXLBvDog+9Xh96E/KmpON5wSk9Ra4OQVR28841Lqn2XZATKks"
"FNZKhVG3YQcJcFQYd7LPEN3DYJ98gJZdgNMMVpiXhCedpQDHCE4j8gCMT8j6gkmHmvYDcPg16ykF8lECy/wXZD9gqcx/S9Zh"
"fjjwLMFqHDCdwA0WAs0LY3U4T6YMMr+I4jDHBOlk7v+NMajJ3bDmOzj9VxhpByl9YjdBMq9h/ZfsFrsE0jnFjgBCPMk2sFWA"
"HLextWw5mwd36yB7zQe0PIdNAyy/AbLRQshQw9kU6N/H/mDLoKoZAW0zzN4I2Ws0GwLzFrAlMHoYIJe+bCqMmQH9fWHWDMjA"
"2wCH7oCxu2DXJ+wV6O4+aOwVcPMJatFPwNFd6HsE7Snc3QVOrwCufQTvnwLPz2HOY/ae/QSJp8GJ3kCzg4R0kIcGJwsyK+gw"
"G2iVQZ+Y9YWZyhFPgLVTs74ZSwRJPIP93rMvgFlT4PoeUFYCzMWcgEw1kLKPabCeAjoUuBfeuOGdBTK2BTSX+RUagvUj4UmC"
"0UF4p4AeEcjfAfIPQnOBTtJhxwSgL6DTz7DHBzhB5lnuw1nuAd1kf4PMTwFie5aF2U6BPG7AGd8D3YOnk5D/n8Cch9B/HGqb"
"83DuNzD7HNh+LEjkJTxfh/7tgFxOgA7Psr2AGdYB0jkNY87CmL9g5EUY9Qxs9Ro7A5b5AlZ4AatfgBkXofch+ETmzifg/Y0s"
"zu5D73XofZxlH09h/AewoR8gtUzJpcPZvf/5WVrWd3SMUZCaAXWuASdnIBkdbA+BNEx4MqE/CM0DErHDDCfc+0G6Bsz8l4ws"
"+8xcReaZI3WwXpYlvUSQpi9rTS9I8SM0b9a85CzNpWRpiAK+/Q7c2WFU5peCOoxxQKOgEQY7uWCcEzTmyfKLn2BdicBFpgc4"
"4ZoEazmybMYH751gHQ6Yk8m9BnNSss6dBL0JWfvHAb2DnTNl8SVrpWSYw4CLf79htIK1RPIoiFyleTGoggtlURmIKqXA54tA"
"1V8BYkwT3hjiSQ2oResDzmzD2/JWvDn8doZ4GcMH8t+zrkP5CIiM0wHrTAGaDL9zAN8tgjYPouVivgqq240QRxfyJXC/EbDR"
"Zoihq/kGiMKxgJNO8YOAy49CXL4FNdRtoJtQDb/g7wD3v4Xfp0AvIYa/4E/+o8fQnkM9/5rH8w8Q239Cre+CPGEHvOOG2O/M"
"ujqA0uDXBe8/8s8w7itP4Mk8Ba6JWTM8PIn/gJYIfZlfun6AFd/Avp8z/44aKB5mpMCoVOj5B3Z9Du/igJeHwN8d4OE1PL3L"
"4vMb7O3nBmA9g5tQqQazyOAUWSFXyUgEnGaDKisfygF1iIwsKArlRjmhT0SZowXokSEneoBnDZ4yv0PWss4QhDU1HgB+0+HZ"
"CyfL/NtqB/xqWd/oemBfjWd+s4wgQ2a+C3IF8imGJy/wYoV9MneKhL0iUQHA4uWBiqKCgDCrZH2pXAdQaUVUKeu75frQakNG"
"rQitMqoAWLMkUGkYWzyLMr91LoTyojyQo3PBunlh5QhYNxKFoXDYQYLz/PvVNAb+vf/x6wPOU7Iknwb3buA+Fe7c8NYP5M3i"
"WoNz/vsHAW4lsIqUJbd/v8KWQH4yXKX/VpeyrlAI//elNs36Jf/d/89vuDMl+b+/7/73G+/MHTL34P9jv/97/ffd/3369/d/"
"AYxHlHJ2PgAA"
)
@lru_cache(maxsize=1)
def ready_trigger_pcm() -> bytes:
return gzip.decompress(base64.b64decode(READY_TRIGGER_PCM_16KHZ_MONO_GZIP_B64))

8
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -2112,7 +2112,7 @@ class BaseLLMHTTPHandler:
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs={**kwargs, "api_key": api_key} if api_key else kwargs,
)
return initial_response
else:
@ -2122,6 +2122,10 @@ class BaseLLMHTTPHandler:
logging_obj=logging_obj,
)
# Inject api_key into kwargs so follow-up calls in agentic hooks can
# authenticate. api_key is a named param here (not in kwargs), so
# _prepare_followup_kwargs would miss it otherwise.
kwargs_for_agentic = {**kwargs, "api_key": api_key} if api_key else kwargs
# Call agentic completion hooks (non-streaming path only)
final_response = await self._call_agentic_completion_hooks(
response=initial_response,
@ -2132,7 +2136,7 @@ class BaseLLMHTTPHandler:
logging_obj=logging_obj,
stream=False,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs=kwargs_for_agentic,
)
return self._maybe_wrap_in_fake_stream(

0
litellm/llms/gdc/__init__.py Normal file
View file

0
litellm/llms/gdc/chat/__init__.py Normal file
View file

285
litellm/llms/gdc/chat/transformation.py Normal file
View file

@ -0,0 +1,285 @@
"""
GDC Gemini chat completion transformation
"""
import json
import os
import re
import threading
from typing import Any, Final
from urllib.parse import urlsplit
import litellm
from litellm.llms.openai_like.chat.transformation import OpenAILikeChatConfig
class GDCGeminiConfig(OpenAILikeChatConfig):
supports_vertex_params: bool = True # Tell LiteLLM utilities not to strip vertex_ params
_GDCH_CREDENTIAL_TYPE: Final[str] = "gdch_service_account"
_PATH_ID_PATTERN: Final[re.Pattern[str]] = re.compile(r"^[a-zA-Z0-9_-]+$")
def __init__(self, **kwargs: Any) -> None:
super().__init__(**kwargs)
self._creds_lock = threading.Lock()
self._gdch_creds_cache: dict = {}
def get_supported_openai_params(self, model: str) -> list:
return [
"vertex_project",
"vertex_location",
] + super().get_supported_openai_params(model)
def _resolve_project(self, optional_params: dict, litellm_params: dict) -> str | None:
return (
litellm_params.get("vertex_project")
or litellm_params.get("vertex_ai_project")
or getattr(litellm, "vertex_project", None)
or optional_params.get("vertex_project")
or optional_params.get("vertex_ai_project")
)
def _resolve_location(self, optional_params: dict, litellm_params: dict) -> str | None:
return (
litellm_params.get("vertex_location")
or litellm_params.get("vertex_ai_location")
or getattr(litellm, "vertex_location", None)
or optional_params.get("vertex_location")
or optional_params.get("vertex_ai_location")
)
def _effective_project(self, api_base: str, optional_params: dict, litellm_params: dict) -> str | None:
match = re.search(r"/v1/projects/([^/]+)", api_base)
if match:
return match.group(1)
return self._resolve_project(optional_params, litellm_params)
def _validate_path_id(self, value: str, field: str, model: str) -> str:
if not self._PATH_ID_PATTERN.match(value):
raise litellm.utils.AuthenticationError(
message=f"{field} must be a plain identifier of letters, digits, hyphens or underscores.",
llm_provider="gdc",
model=model,
)
return value
def get_complete_url(
self,
api_base: str | None,
api_key: str | None,
model: str,
optional_params: dict,
litellm_params: dict,
stream: bool | None = None,
) -> str:
api_base = api_base or litellm.gdc_api_base or litellm.api_base
if not api_base:
raise litellm.utils.AuthenticationError(
message="api_base/host is required for GDC Gemini. Please set it or pass it.",
llm_provider="gdc",
model=model,
)
if not api_base.startswith("http"):
api_base = f"https://{api_base}"
api_base = api_base.rstrip("/")
if "/v1/projects/" in api_base:
return api_base
project = self._resolve_project(optional_params, litellm_params)
if not project:
raise litellm.utils.AuthenticationError(
message="project is required for GDC Gemini. Please pass vertex_project.",
llm_provider="gdc",
model=model,
)
location = self._resolve_location(optional_params, litellm_params)
if not location:
raise litellm.utils.AuthenticationError(
message="location is required for GDC Gemini. Please pass vertex_location.",
llm_provider="gdc",
model=model,
)
project = self._validate_path_id(project, "vertex_project", model)
location = self._validate_path_id(location, "vertex_location", model)
return f"{api_base}/v1/projects/{project}/locations/{location}/chat/completions"
def _read_env_bool(self, val: Any, env_var: str, default: bool = True) -> bool | str:
def _parse(s: str) -> bool | str:
cleaned = s.strip().lower()
if cleaned in ("false", "0", "no", "off"):
return False
if cleaned in ("true", "1", "yes", "on"):
return True
return s
if val is not None:
if isinstance(val, str):
return _parse(val)
return val
_env_val = os.getenv(env_var)
if _env_val is None:
return default
return _parse(_env_val)
def _fetch_auth(self, gdch_creds: Any, ssl_verify: bool | str) -> None:
import requests
from google.auth.transport import requests as auth_requests
auth_session = requests.Session()
auth_session.verify = ssl_verify
auth_request = auth_requests.Request(session=auth_session)
gdch_creds.refresh(auth_request)
def _cached_fetch_token(self, creds: Any, audience: str, ssl_verify: bool | str, api_key: str | None = None) -> str:
# Key cache by both audience and credential identity to prevent cross-caller contamination
cache_key = (audience.rstrip("/"), api_key or str(id(creds)))
with self._creds_lock:
if cache_key not in self._gdch_creds_cache:
self._gdch_creds_cache[cache_key] = creds.with_gdch_audience(audience.rstrip("/"))
gdch_creds = self._gdch_creds_cache[cache_key]
if not getattr(gdch_creds, "valid", False) or not getattr(gdch_creds, "token", None):
self._fetch_auth(gdch_creds, ssl_verify)
token = gdch_creds.token
return token
def _load_creds_from_key(self, api_key: str) -> tuple[Any, bool]:
import google.auth
try:
json_obj = json.loads(api_key)
except json.JSONDecodeError:
return None, False
if not isinstance(json_obj, dict) or json_obj.get("type") != self._GDCH_CREDENTIAL_TYPE:
raise ValueError(
"GDC only accepts a GDCH service account credential as a JSON api_key "
'(expected "type": "gdch_service_account"). Other Google credential types are '
"rejected so their token or external-account endpoints cannot drive server-side requests."
)
creds, _ = google.auth.load_credentials_from_dict(json_obj)
return creds, True
def validate_environment(
self,
headers: dict,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: str | None = None,
api_base: str | None = None,
) -> dict:
import google.auth.exceptions
api_base = api_base or litellm.gdc_api_base or litellm.api_base
if not api_base:
raise litellm.utils.AuthenticationError(
message="api_base/host is required for GDC Gemini. Please set it or pass it.",
llm_provider="gdc",
model=model,
)
if not api_key:
raise litellm.utils.AuthenticationError(
message="api_key is required for GDC Gemini. Please pass your service account string or token as the api_key.",
llm_provider="gdc",
model=model,
)
project = self._effective_project(api_base, optional_params, litellm_params)
if not project:
raise litellm.utils.AuthenticationError(
message="project is required for GDC Gemini. Please pass vertex_project.",
llm_provider="gdc",
model=model,
)
project = self._validate_path_id(project, "vertex_project", model)
_audience_parts = urlsplit(api_base if api_base.startswith("http") else f"https://{api_base}")
audience = f"{_audience_parts.scheme}://{_audience_parts.netloc}"
try:
creds, is_service_account = self._load_creds_from_key(api_key)
except (
google.auth.exceptions.GoogleAuthError,
ValueError,
TypeError,
KeyError,
AttributeError,
) as e:
raise litellm.utils.AuthenticationError(
message=f"Failed to load service account credentials from api_key: {str(e)}",
llm_provider="gdc",
model=model,
) from e
if creds is not None:
ssl_verify = self._read_env_bool(litellm_params.get("ssl_verify"), "SSL_VERIFY", default=True)
if self._read_env_bool(litellm_params.get("gdc_token_caching"), "GDC_TOKEN_CACHING", default=False):
token = self._cached_fetch_token(creds, audience, ssl_verify, api_key)
else:
gdch_creds = creds.with_gdch_audience(audience)
self._fetch_auth(gdch_creds, ssl_verify)
token = gdch_creds.token
headers["Authorization"] = f"Bearer {token}"
if "Authorization" not in headers and not is_service_account:
headers["Authorization"] = f"Bearer {api_key}"
# Standardize necessary metadata headers
if "content-type" not in headers and "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
stale_quota_headers = tuple(h for h in headers if h.lower() == "x-goog-user-project")
for stale in stale_quota_headers:
headers.pop(stale, None)
headers["x-goog-user-project"] = f"projects/{project}"
return headers
def transform_request(
self,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transforms the request to the GDC provider
"""
if model.startswith("gdc/"):
model = model.split("/", 1)[1]
data = super().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
# Remove extra params used for routing/auth
for param in [
"vertex_project",
"vertex_ai_project",
"vertex_location",
"vertex_ai_location",
"ssl_verify",
"gdc_token_caching",
]:
data.pop(param, None)
return data

0
litellm/llms/github_copilot/messages/__init__.py Normal file
View file

118
litellm/llms/github_copilot/messages/transformation.py Normal file
View file

@ -0,0 +1,118 @@
from typing import Any, Optional
from litellm.exceptions import AuthenticationError
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
from ..authenticator import Authenticator
from ..common_utils import (
DEFAULT_GITHUB_COPILOT_API_BASE,
GetAPIKeyError,
get_copilot_default_headers,
)
_MESSAGES_PROXY_API_VERSION = "2026-06-01"
class GithubCopilotAnthropicMessagesConfig(AnthropicMessagesConfig):
"""
GitHub Copilot implementation of Anthropic messages API.
Routes requests to Copilot's /v1/messages endpoint with appropriate authentication and headers.
"""
def __init__(self) -> None:
super().__init__()
self.authenticator = Authenticator()
def handles_web_search_natively(self) -> bool:
"""
Copilot's /v1/messages endpoint does not execute ``web_search`` tools, so
the interception handler must short-circuit web-search-only requests
instead of routing them here.
"""
return False
def should_filter_anthropic_beta_headers(self) -> bool:
"""
Copilot's /v1/messages is a native Anthropic Messages passthrough, so
``anthropic-beta`` values injected by ``_update_headers_with_anthropic_beta``
(context_management, structured outputs, ...) must reach the upstream
verbatim. The default provider-scoped filter would drop them because
github_copilot has no entry in ``anthropic_beta_headers_config.json``.
"""
return False
def validate_anthropic_messages_environment(
self,
headers: dict,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> tuple[dict, Optional[str]]:
"""
Validate environment for GitHub Copilot and add Copilot-specific headers.
The caller-supplied ``api_base`` is intentionally ignored. Routing this
request anywhere other than the authenticated Copilot endpoint would
leak the Copilot bearer token to a caller-controlled URL.
"""
# Always use the Copilot endpoint resolved from the authenticated
# session, never the caller-supplied api_base. rstrip so a
# tenant-specific base with a trailing slash does not yield a
# double-slash URL once "/v1/messages" is appended downstream.
dynamic_api_base = (self.authenticator.get_api_base() or DEFAULT_GITHUB_COPILOT_API_BASE).rstrip("/")
try:
dynamic_api_key = self.authenticator.get_api_key()
except GetAPIKeyError as e:
raise AuthenticationError(
model=model,
llm_provider="github_copilot",
message=str(e),
)
# Merge Copilot headers with provided headers
copilot_headers = get_copilot_default_headers(dynamic_api_key)
for key, value in copilot_headers.items():
if key not in headers:
headers[key] = value
headers["openai-intent"] = "messages-proxy"
headers["x-interaction-type"] = "messages-proxy"
headers["x-github-api-version"] = _MESSAGES_PROXY_API_VERSION
if "anthropic-version" not in headers:
headers["anthropic-version"] = "2023-06-01"
headers = self._update_headers_with_anthropic_beta(
headers, optional_params, custom_llm_provider="github_copilot"
)
return headers, dynamic_api_base
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
"""
Return the complete URL for GitHub Copilot /v1/messages endpoint.
``api_base`` here is the value already resolved by
``validate_anthropic_messages_environment`` (the authenticated Copilot
host), not the raw caller-supplied base — that one is discarded there to
avoid leaking the Copilot bearer token to a caller-controlled URL. We
reuse it to avoid a second authenticator read, falling back to a fresh
resolution only if it was not provided.
"""
resolved = (api_base or self.authenticator.get_api_base() or DEFAULT_GITHUB_COPILOT_API_BASE).rstrip("/")
if not resolved.endswith("/v1/messages"):
resolved = f"{resolved}/v1/messages"
return resolved

10
litellm/llms/openai/responses/guardrail_translation/handler.py
View file

@ -45,6 +45,7 @@ from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionToolCallChunk,
ChatCompletionToolParam,
ResponsesAPIStreamEvents,
)
from litellm.types.responses.main import (
GenericResponseOutputItem,
@ -586,7 +587,14 @@ class OpenAIResponsesHandler(BaseTranslation):
"""
Check if the streaming has ended.
"""
return all(response.choices[0].finish_reason is not None for response in responses_so_far)
if not responses_so_far:
return False
terminal_types = {
ResponsesAPIStreamEvents.RESPONSE_COMPLETED.value,
ResponsesAPIStreamEvents.RESPONSE_FAILED.value,
ResponsesAPIStreamEvents.RESPONSE_INCOMPLETE.value,
}
return responses_so_far[-1].get("type") in terminal_types
def get_streaming_string_so_far(self, responses_so_far: List[Any]) -> str:
"""

0
litellm/llms/tencent/__init__.py Normal file
View file

0
litellm/llms/tencent/chat/__init__.py Normal file
View file

68
litellm/llms/tencent/chat/transformation.py Normal file
View file

@ -0,0 +1,68 @@
"""
Translates from OpenAI's `/v1/chat/completions` to Tencent TokenHub's
OpenAI-compatible endpoint.
"""
from typing import Optional
from litellm.secret_managers.main import get_secret_str
from litellm.utils import supports_reasoning
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
class TencentChatConfig(OpenAIGPTConfig):
def get_supported_openai_params(self, model: str) -> list:
params = super().get_supported_openai_params(model)
if supports_reasoning(model, custom_llm_provider="tencent"):
params.extend(["thinking", "reasoning_effort"])
return params
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
optional_params = super().map_openai_params(non_default_params, optional_params, model, drop_params)
thinking_value = optional_params.pop("thinking", None)
reasoning_effort = optional_params.pop("reasoning_effort", None)
if thinking_value is not None:
if isinstance(thinking_value, dict):
optional_params["thinking"] = thinking_value
elif reasoning_effort is not None and reasoning_effort != "none":
optional_params["thinking"] = {"type": "enabled"}
return optional_params
def _get_openai_compatible_provider_info(
self, api_base: Optional[str], api_key: Optional[str]
) -> tuple[Optional[str], Optional[str]]:
api_base = api_base or get_secret_str("TENCENT_API_BASE") or "https://tokenhub-intl.tencentcloudmaas.com/v1"
dynamic_api_key = api_key or get_secret_str("TENCENT_API_KEY")
return api_base, dynamic_api_key
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
if not api_base:
api_base = "https://tokenhub-intl.tencentcloudmaas.com/v1"
api_base = api_base.rstrip("/")
if api_base.endswith("/chat/completions"):
return api_base
if not api_base.endswith("/v1"):
api_base = f"{api_base}/v1"
return f"{api_base}/chat/completions"

6
litellm/llms/tencent/cost_calculator.py Normal file
View file

@ -0,0 +1,6 @@
from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
from litellm.types.utils import Usage
def cost_per_token(model: str, usage: Usage) -> tuple[float, float]:
return generic_cost_per_token(model=model, usage=usage, custom_llm_provider="tencent")

0
litellm/llms/tencent/messages/__init__.py Normal file
View file

85
litellm/llms/tencent/messages/transformation.py Normal file
View file

@ -0,0 +1,85 @@
"""
Tencent Anthropic-compatible messages transformation config.
Tencent TokenHub exposes an Anthropic-compatible Messages API endpoint
alongside its standard OpenAI-compatible chat completions endpoint.
"""
from typing import Any, Optional
import litellm
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
from litellm.secret_managers.main import get_secret_str
class TencentAnthropicMessagesConfig(AnthropicMessagesConfig):
"""
Tencent TokenHub exposes an Anthropic-compatible Messages API.
Unlike the chat completions endpoint (which uses /v1), the Anthropic
endpoint may use a different base URL. Configure via
TENCENT_ANTHROPIC_API_BASE or TENCENT_API_BASE.
"""
@property
def custom_llm_provider(self) -> Optional[str]:
return "tencent"
def should_strip_billing_metadata(self) -> bool:
return True
@staticmethod
def get_api_key(api_key: Optional[str] = None) -> Optional[str]:
return api_key or get_secret_str("TENCENT_API_KEY") or litellm.api_key
@staticmethod
def get_api_base(api_base: Optional[str] = None) -> str:
return (
api_base
or get_secret_str("TENCENT_ANTHROPIC_API_BASE")
or get_secret_str("TENCENT_API_BASE")
or "https://tokenhub-intl.tencentcloudmaas.com"
)
def validate_anthropic_messages_environment(
self,
headers: dict,
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> tuple[dict, Optional[str]]:
return super().validate_anthropic_messages_environment(
headers=headers,
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
api_key=self.get_api_key(api_key=api_key),
api_base=api_base,
)
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
base_url = self.get_api_base(api_base=api_base).rstrip("/")
if base_url.endswith("/v1/messages"):
return base_url
if base_url.endswith("/v1/chat/completions"):
base_url = base_url[: -len("/v1/chat/completions")]
elif base_url.endswith("/v1"):
base_url = base_url[: -len("/v1")]
return f"{base_url}/v1/messages"

270
litellm/llms/tinyfish/search/transformation.py
View file

@ -6,53 +6,42 @@ Docs: https://docs.tinyfish.ai/search-api
from __future__ import annotations
from typing import Literal, TypedDict
import json
from typing import Literal
from urllib.parse import urlencode
import httpx
from pydantic import BaseModel, TypeAdapter, ValidationError
from pydantic import TypeAdapter, ValidationError
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.search.transformation import (
BaseSearchConfig,
SearchResponse,
SearchResult,
)
from litellm.secret_managers.main import get_secret_str
class _TinyfishSearchRequestRequired(TypedDict):
query: str
class TinyfishSearchRequest(_TinyfishSearchRequestRequired, total=False):
location: str
language: str
page: int
include_thumbnail: bool
max_results: int
class _TinyfishResultItem(BaseModel, frozen=True):
title: str = ""
url: str = ""
snippet: str = ""
class _TinyfishApiResponse(BaseModel, frozen=True):
results: tuple[_TinyfishResultItem, ...] = ()
_UrlEncodableParams = TypeAdapter(dict[str, str | int | bool])
_StrList = TypeAdapter(list[str])
_StrFrozenSet = TypeAdapter(frozenset[str])
_TINYFISH_PARAMS_KEY = "_tinyfish_params"
_TINYFISH_DOCS_URL = "https://docs.tinyfish.ai/search-api"
_TINYFISH_RESULT_CAP = 10 # TinyFish's natural per-page SERP ceiling
class TinyfishSearchConfig(BaseSearchConfig):
TINYFISH_API_BASE = "https://api.search.tinyfish.ai"
def __init__(self) -> None:
super().__init__()
# Threaded from transform_search_request → transform_search_response so the
# response slice honors the caller's max_results without re-sending it on
# the wire (TinyFish doesn't honor it server-side). Safe because the
# config is instantiated per-call via ProviderConfigManager.
self._caller_max_results: int | None = None
@staticmethod
def ui_friendly_name() -> str:
return "TinyFish"
@ -97,36 +86,77 @@ class TinyfishSearchConfig(BaseSearchConfig):
optional_params: dict[str, object],
**kwargs: object,
) -> dict[str, object]:
"""
Transform a LiteLLM search request to TinyFish's querystring format.
Maps LiteLLM's unified-spec params (see
``BaseSearchConfig.get_supported_perplexity_optional_params``) to
TinyFish equivalents:
- ``query`` (str or list[str]) → ``query`` (list joined by spaces)
- ``country`` → ``location``
- ``search_domain_filter`` (list[str]) → folded into the query as
``(<query>) (site:a OR site:b ...)`` (TinyFish has no first-class
field today; see ML-2084 for the planned ``include_domains``)
- ``max_results`` → not sent on the wire; stashed on
``self._caller_max_results`` for client-side response truncation
(TinyFish doesn't honor it server-side)
- ``max_tokens_per_page`` → silently dropped (no TinyFish equivalent)
Any other ``optional_params`` keys are forwarded to TinyFish as-is.
dict/list values are JSON-encoded so they survive ``urlencode``.
Returns:
``{_TINYFISH_PARAMS_KEY: <dict of querystring entries>}``.
``get_complete_url`` reads this back to build the final URL.
"""
resolved_query = " ".join(query) if isinstance(query, list) else query
request_data: TinyfishSearchRequest = {"query": resolved_query}
country = optional_params.get("country")
if isinstance(country, str):
request_data["location"] = country
raw_max = optional_params.get("max_results")
if isinstance(raw_max, (int, float, str)):
request_data["max_results"] = max(1, min(int(raw_max), 20))
try:
domains = _StrList.validate_python(optional_params.get("search_domain_filter"))
except (ValidationError, TypeError):
domains = []
if domains:
request_data["query"] = _append_domain_filters(request_data["query"], domains)
resolved_query = _append_domain_filters(resolved_query, domains)
result_data: dict[str, object] = dict(request_data)
request_data: dict[str, object] = {"query": resolved_query}
country = optional_params.get("country")
if isinstance(country, str):
request_data["location"] = country
# max_results is enforced client-side on the response (TinyFish ignores
# the param and always returns ~10). Clamp to [1, 10] and stash on self
# so transform_search_response can slice without re-reading the URL.
raw_max = optional_params.get("max_results")
if isinstance(raw_max, (int, float, str)):
try:
self._caller_max_results = max(1, min(int(raw_max), _TINYFISH_RESULT_CAP))
except (ValueError, TypeError, OverflowError):
# OverflowError covers int(float('inf')) and similar non-finite floats.
verbose_logger.warning(
"TinyFish Search: max_results=%r is not a valid integer; ignoring.",
raw_max,
)
raw_supported: object = (
self.get_supported_perplexity_optional_params() # any-ok: base class returns bare set
)
supported_perplexity = _StrFrozenSet.validate_python(raw_supported)
for param, value in optional_params.items():
if param not in supported_perplexity and param not in result_data:
result_data[param] = value
if param not in supported_perplexity and param not in request_data:
# `fetch` expects a JSON-encoded object on the wire; accept the
# natural Python dict form and serialize here so callers don't
# have to pre-stringify.
if isinstance(value, dict):
value = json.dumps(value, separators=(",", ":"))
# `urlencode` would render Python bool as "True"/"False"
# (capitalized). ux-labs validators require lowercase
# "true"/"false" (e.g. `include_thumbnail`); normalize here.
elif isinstance(value, bool):
value = "true" if value else "false"
request_data[param] = value
return {_TINYFISH_PARAMS_KEY: result_data}
return {_TINYFISH_PARAMS_KEY: request_data}
def transform_search_response(
self,
@ -134,24 +164,158 @@ class TinyfishSearchConfig(BaseSearchConfig):
logging_obj: LiteLLMLoggingObj | None,
**kwargs: object,
) -> SearchResponse:
raw_json: object = raw_response.json() # any-ok: httpx Response.json() -> Any
parsed = _TinyfishApiResponse.model_validate(raw_json)
"""
Transform a TinyFish response to LiteLLM's unified ``SearchResponse``.
max_results_str: str = "20"
if raw_response.request:
raw_param: object = raw_response.request.url.params.get( # any-ok: httpx QueryParams.get() -> Any
"max_results", "20"
Mappings (per-result):
- ``title`` → ``SearchResult.title`` (defaults to ``""`` if missing/null)
- ``url`` → ``SearchResult.url`` (defaults to ``""``)
- ``snippet`` → ``SearchResult.snippet`` (defaults to ``""``)
- all other per-result fields (``position``, ``site_name``,
``thumbnail_url``, ``fetch``, ``fetch_error``, ...) ride through as
extras on ``SearchResult`` via its ``extra="allow"`` config.
Top-level ``parameter_warnings`` (see ML-2085) is read when present and
each entry is re-fired via ``verbose_logger.warning``. Absent or
malformed entries are silently skipped — never throws.
Error paths routed through ``self._wrap_error`` for uniform
``"TinyFish Search: <msg>. See <docs> for details."`` wrapping:
- non-2xx HTTP status (caught here because ``AsyncHTTPHandler.get``
does not call ``raise_for_status``)
- 200 with non-JSON body
- 200 with valid JSON whose shape doesn't satisfy ``SearchResponse``
Returns:
``SearchResponse`` truncated to ``self._caller_max_results`` (or
``_TINYFISH_RESULT_CAP`` when the caller didn't set ``max_results``).
"""
# AsyncHTTPHandler.get does not call raise_for_status, so non-2xx
# responses arrive here looking successful. Dispatch through
# get_error_class so callers see a uniform attributed error.
if not (200 <= raw_response.status_code < 300):
raise self._wrap_error(
error_message=raw_response.text,
status_code=raw_response.status_code,
headers=dict(raw_response.headers),
)
max_results_str = str(raw_param)
max_results: int = min(int(max_results_str), 20)
results = [
SearchResult(title=item.title, url=item.url, snippet=item.snippet) for item in parsed.results[:max_results]
]
try:
raw_json: object = raw_response.json() # any-ok: httpx Response.json() -> Any
except json.JSONDecodeError:
raise self._wrap_error(
error_message=f"Expected JSON response, got: {raw_response.text[:200]}",
status_code=raw_response.status_code,
headers=dict(raw_response.headers),
)
return SearchResponse(results=results, object="search")
_default_missing_result_fields(raw_json)
try:
parsed = SearchResponse.model_validate(raw_json)
except ValidationError as e:
raise self._wrap_error(
error_message=(f"Response shape does not match LiteLLM's SearchResponse schema: {e}"),
status_code=raw_response.status_code,
headers=dict(raw_response.headers),
)
_emit_parameter_warnings(parsed)
max_results = self._caller_max_results or _TINYFISH_RESULT_CAP
return SearchResponse(results=list(parsed.results[:max_results]))
def _wrap_error(
self,
error_message: str,
status_code: int,
headers: dict[str, str],
) -> Exception:
"""
Build an attributed ``BaseLLMException`` from a TinyFish error body.
Used only at the call sites we control inside
``transform_search_response`` (non-2xx, JSONDecodeError, ValidationError).
Not an override of ``BaseSearchConfig.get_error_class``: that path is
left to inherit from the base so it auto-picks-up any future LiteLLM
improvements. Trade-off: network failures (routed through LiteLLM
core's ``_handle_error`` → ``BaseSearchConfig.get_error_class``) won't
carry the ``TinyFish Search:`` prefix — the bare error already names
the host in the URL, so attribution is implicit there.
"""
# ux-labs frontend wraps every error body as {"error": {"code", "message", "details"?}}.
# Best-effort unwrap to surface the inner message; fall back to the raw body
# for non-ux-labs responses (CDN HTML pages, other JSON envelopes, plain text).
inner_message = error_message
try:
body: object = json.loads(error_message) # any-ok: json.loads -> Any
if isinstance(body, dict):
error_obj: object = body.get("error") # any-ok: untyped dict
if isinstance(error_obj, dict):
candidate: object = error_obj.get("message") # any-ok: untyped dict
if isinstance(candidate, str) and candidate:
inner_message = candidate
except (json.JSONDecodeError, TypeError):
pass
return BaseLLMException(
status_code=status_code,
message=f"TinyFish Search: {inner_message}. See {_TINYFISH_DOCS_URL} for details.",
headers=headers,
)
def _append_domain_filters(query: str, domains: list[str]) -> str:
domain_clauses = " OR ".join(f"site:{d}" for d in domains)
return f"({query}) ({domain_clauses})"
def _default_missing_result_fields(raw_json: object) -> None:
"""Default missing/null title/url/snippet to "" on each result item in place.
SearchResult requires these three fields; a degraded TinyFish result flows
through with empty strings instead of failing the whole call.
"""
if not isinstance(raw_json, dict):
return
results_in = raw_json.get("results")
if not isinstance(results_in, list):
return
for item in results_in:
if not isinstance(item, dict):
continue
for field in ("title", "url", "snippet"):
if not isinstance(item.get(field), str):
item[field] = ""
def _emit_parameter_warnings(parsed: SearchResponse) -> None:
"""Re-fire TinyFish-side ``parameter_warnings`` (see ML-2085) as warnings.
Defensive: skip silently on any shape we don't recognize so a malformed
entry (or an early/partial rollout of the field) never throws.
Schema per entry: ``{type, parameter, message, docs_url?}``.
"""
warnings_field: object = (
getattr(parsed, "parameter_warnings", None) # any-ok: extras=allow field
)
if not isinstance(warnings_field, list):
return
for entry in warnings_field:
if not isinstance(entry, dict):
continue
warning_type: object = entry.get("type") # any-ok: untyped dict
parameter: object = entry.get("parameter") # any-ok: untyped dict
message: object = entry.get("message") # any-ok: untyped dict
if not isinstance(warning_type, str) or not warning_type:
continue
if not isinstance(parameter, str) or not parameter:
continue
if not isinstance(message, str) or not message:
continue
verbose_logger.warning(
"TinyFish Search parameter_warning (%s) `%s`: %s",
warning_type,
parameter,
message,
)

20
litellm/llms/vertex_ai/batches/transformation.py
View file

@ -1,4 +1,4 @@
from typing import Any, Dict
from typing import Any, Dict, Optional
from litellm._uuid import uuid
from litellm.llms.vertex_ai.common_utils import (
@ -47,7 +47,7 @@ class VertexAIBatchTransformation:
) -> LiteLLMBatch:
return LiteLLMBatch(
id=cls._get_batch_id_from_vertex_ai_batch_response(response),
completion_window="24hrs",
completion_window="24h",
created_at=_convert_vertex_datetime_to_openai_datetime(vertex_datetime=response.get("createTime", "")),
endpoint="",
input_file_id=cls._get_input_file_id_from_vertex_ai_batch_response(response),
@ -207,3 +207,19 @@ class VertexAIBatchTransformation:
parts = model_path.split("/")
model = f"publishers/{'/'.join(parts[:3])}"
return model
@classmethod
def is_unmanaged_gcs_batch_input_file_id(cls, input_file_id: Optional[str]) -> bool:
"""
Returns True if `input_file_id` is a raw gs:// Vertex batch input file (i.e. not a
LiteLLM-managed unified file id) with a `publishers/` model path that
`_get_model_from_gcs_file` can parse.
"""
return input_file_id is not None and input_file_id.startswith("gs://") and "publishers/" in input_file_id
@classmethod
def get_bare_model_name_from_gcs_file(cls, gcs_file_uri: str) -> str:
"""
Extracts the bare model name (e.g. "gemini-1.5-flash-001") from a gcs file uri.
"""
return cls._get_model_from_gcs_file(gcs_file_uri).rsplit("/", 1)[-1]

24
litellm/llms/vertex_ai/image_generation/vertex_gemini_transformation.py
View file

@ -1,5 +1,7 @@
import os
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from typing import TYPE_CHECKING, Any, Optional
from litellm._logging import verbose_logger
import httpx
@ -52,6 +54,7 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
return [
"n",
"size",
"imageConfig",
"aspectRatio",
"aspect_ratio",
"imageSize",
@ -83,7 +86,12 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
mapped_params["aspectRatio"] = v
elif k in ("imageSize", "image_size"):
mapped_params["imageSize"] = v
elif k not in ("tools", "web_search_options"):
elif k == "imageConfig":
if isinstance(v, dict):
mapped_params["imageConfig"] = v
else:
verbose_logger.warning("imageConfig must be a dict, got %s — ignoring.", type(v).__name__)
elif k not in ("tools", "web_search_options", "imageConfig"):
mapped_params[k] = v
mapped_params = map_gemini_image_tools_params(non_default_params, mapped_params)
@ -167,7 +175,7 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
@ -209,18 +217,16 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
contents = [{"role": "user", "parts": [{"text": prompt}]}]
# Prepare generation config
generation_config: Dict[str, Any] = {"responseModalities": ["IMAGE"]}
generation_config: dict[str, Any] = {"responseModalities": ["IMAGE"]}
# Handle image-specific config parameters
image_config: Dict[str, Any] = {}
# Seed from user-supplied imageConfig dict; flat params are overlaid for backward compat.
image_config: dict[str, Any] = dict(optional_params.get("imageConfig") or {})
# Map aspectRatio
if "aspectRatio" in optional_params:
image_config["aspectRatio"] = optional_params["aspectRatio"]
elif "aspect_ratio" in optional_params:
image_config["aspectRatio"] = optional_params["aspect_ratio"]
# Map imageSize (for Gemini 3 Pro)
if "imageSize" in optional_params:
image_config["imageSize"] = optional_params["imageSize"]
elif "image_size" in optional_params:
@ -235,7 +241,7 @@ class VertexAIGeminiImageGenerationConfig(BaseImageGenerationConfig, VertexLLM):
elif "n" in optional_params:
generation_config["candidateCount"] = optional_params["n"]
request_body: Dict[str, Any] = {
request_body: dict[str, Any] = {
"contents": contents,
"generationConfig": generation_config,
}

43
litellm/main.py
View file

@ -210,6 +210,7 @@ from .llms.bedrock.embed.embedding import BedrockEmbedding
from .llms.bedrock.image_edit.handler import BedrockImageEdit
from .llms.bedrock.image_generation.image_handler import BedrockImageGeneration
from .llms.bytez.chat.transformation import BytezChatConfig
from .llms.gdc.chat.transformation import GDCGeminiConfig
from .llms.clarifai.chat.transformation import ClarifaiConfig
from .llms.codestral.completion.handler import CodestralTextCompletion
from .llms.cohere.embed import handler as cohere_embed
@ -318,6 +319,7 @@ google_batch_embeddings = GoogleBatchEmbeddings()
vertex_partner_models_chat_completion = VertexAIPartnerModels()
vertex_gemma_chat_completion = VertexAIGemmaModels()
vertex_model_garden_chat_completion = VertexAIModelGardenModels()
gdc_transformation = GDCGeminiConfig()
# vertex_text_to_speech is now replaced by VertexAITextToSpeechConfig
sagemaker_llm = SagemakerLLM()
watsonx_chat_completion = WatsonXChatHandler()
@ -4336,6 +4338,45 @@ def _complete_gradient_ai(ctx: _CompletionDispatchContext) -> _CompletionDispatc
)
def _complete_gdc(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client = ctx.client
custom_llm_provider = ctx.custom_llm_provider
headers = ctx.headers
litellm_params = ctx.litellm_params
logging = ctx.logging
messages = ctx.messages
model = ctx.model
model_response = ctx.model_response
optional_params = ctx.optional_params
stream = ctx.stream
timeout = ctx.timeout
api_key = api_key or litellm.gdc_key or get_secret_str("GDC_API_KEY") or litellm.api_key
api_base = api_base or litellm.gdc_api_base or get_secret_str("GDC_API_BASE") or litellm.api_base
return base_llm_http_handler.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
api_key=api_key,
api_base=api_base,
acompletion=acompletion,
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
timeout=timeout, # type: ignore
client=client,
custom_llm_provider=custom_llm_provider,
encoding=_get_encoding(),
stream=stream,
provider_config=gdc_transformation,
)
def _complete_bytez(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion = ctx.acompletion
api_base = ctx.api_base
@ -5533,6 +5574,8 @@ def completion( # type: ignore
elif custom_llm_provider == "gradient_ai":
response = _complete_gradient_ai(_dispatch_ctx)
elif custom_llm_provider == "gdc":
response = _complete_gdc(_dispatch_ctx)
elif custom_llm_provider == "bytez":
response = _complete_bytez(_dispatch_ctx)
elif custom_llm_provider == "lemonade":

1477
litellm/model_prices_and_context_window_backup.json
View file

File diff suppressed because it is too large Load diff

1
litellm/models/mcp_server.py
View file

@ -93,6 +93,7 @@ class LiteLLM_MCPServerTable(LiteLLMPydanticObjectBase):
has_user_credential: Optional[bool] = None
source_url: Optional[str] = None
timeout: Optional[float] = None
max_concurrent_requests: Optional[int] = None
approval_status: Optional[str] = Field(
default="active",
description="Approval status: 'pending_review', 'active', 'rejected'",

1
litellm/models/object_permission.py
View file

@ -24,3 +24,4 @@ class LiteLLM_ObjectPermissionTable(LiteLLMPydanticObjectBase):
mcp_toolsets: Optional[List[str]] = None
blocked_tools: Optional[List[str]] = []
search_tools: Optional[List[str]] = []
mcp_tool_search_enabled: Optional[bool] = None

1
litellm/models/verification_token.py
View file

@ -39,6 +39,7 @@ class LiteLLM_VerificationToken(LiteLLMPydanticObjectBase):
permissions: Dict = {}
model_spend: Dict = {}
model_max_budget: Dict = {}
budget_fallbacks: dict[str, list[str]] = {}
soft_budget_cooldown: bool = False
blocked: Optional[bool] = None
litellm_budget_table: Optional[dict] = None

6
litellm/proxy/_experimental/mcp_server/AGENTS.md
View file

@ -41,6 +41,7 @@ litellm/proxy/_experimental/mcp_server/
sampling_handler.py # MCP sampling to LiteLLM completion flow
elicitation_handler.py # MCP elicitation relay flow
semantic_tool_filter.py # semantic filtering of available MCP tools
tool_search.py # opt-in virtual tools (mcp_tool_search + mcp_tool_call) for large catalogs
guardrail_translation/
handler.py # MCP guardrail result translation
sse_transport.py # SSE transport implementation
@ -79,6 +80,11 @@ module materially harder to understand.
encryption need focused tests for both allowed and rejected paths.
- Avoid adding comments to new code unless they explain non-obvious security or
protocol behavior. Prefer clear names and small functions.
- The virtual tool path (`tool_search.py`, gated by `mcp_tool_search_enabled`)
must mirror the normal tool flow: IP filtering, server allowlist, per-key tool
permissions, no-accessible-server rejection, per-request auth headers, server
scope, error to `isError` conversion, and spend logging. Reuse `_list_mcp_tools`
and `execute_mcp_tool` rather than reimplementing any of these checks.
## Tests

17
litellm/proxy/_experimental/mcp_server/db.py
View file

@ -1228,6 +1228,23 @@ def _remaining_token_seconds(expires_at: str | None) -> int | None:
return remaining if remaining > 0 else None
async def get_active_submitted_mcp_server_ids_for_user(
prisma_client: PrismaClient,
user_id: str,
) -> list[str]:
"""Return active BYOM servers submitted by this user (creator visibility)."""
if not user_id:
return []
rows = await MCPServerRepository(prisma_client).table.find_many(
where={
"submitted_by": user_id,
"approval_status": MCPApprovalStatus.active,
},
)
return [row.server_id for row in rows]
async def approve_mcp_server(
prisma_client: PrismaClient,
server_id: str,

237
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -3,12 +3,13 @@ import html as _html
import json
import time
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Tuple
from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
import httpx
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse
from pydantic import BaseModel, ValidationError
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import (
@ -31,11 +32,11 @@ from litellm.proxy.common_utils.encrypt_decrypt_utils import (
)
from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
from litellm.proxy.utils import get_server_root_path
from litellm.types.mcp import MCPAuth
from litellm.types.mcp import MCPAuth, MCPCredentials
from litellm.types.mcp_server.mcp_server_manager import MCPServer
if TYPE_CHECKING:
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy._types import LiteLLM_MCPServerTable, UserAPIKeyAuth
# TTL cache for upstream OAuth metadata fetched from pass-through MCP servers.
# Keeps us from hammering the upstream IdP on each discovery request.
@ -390,6 +391,46 @@ async def _store_per_user_token_server_side(
)
def _raise_if_not_oauth2(mcp_server: MCPServer) -> None:
"""Reject a non-oauth2 server from the gateway's OAuth authorize/token/register flow."""
if mcp_server.auth_type == MCPAuth.oauth2:
return
raise HTTPException(
status_code=400,
detail={
"error": "server_not_oauth2",
"message": (
f"MCP server '{mcp_server.server_name or mcp_server.name}' does not use OAuth "
f"(auth_type={mcp_server.auth_type}). This server does not support the authorization-code "
"flow; it has no client_id, authorize, token, or registration endpoint. "
"Access is controlled by the server's configured auth_type and access groups"
),
},
)
def _raise_unless_oauth2_discovery_server(
mcp_server: Optional[MCPServer],
mcp_server_name: Optional[str],
description: str,
) -> None:
"""404 a NAMED discovery request unless it resolves to an oauth2 server.
A named server that is unknown (or hidden from the caller) and one that exists
but is non-oauth2 both return the same 404, so the well-known discovery paths
cannot be used to enumerate non-OAuth server names. Root discovery (no name) is
unaffected, and pass-through servers are resolved by the caller before this runs.
"""
if mcp_server_name is None:
return
if mcp_server is not None and mcp_server.auth_type == MCPAuth.oauth2:
return
raise HTTPException(
status_code=404,
detail=f"MCP server '{mcp_server_name}' is {description}",
)
async def authorize_with_server(
request: Request,
mcp_server: MCPServer,
@ -457,6 +498,7 @@ async def exchange_token_with_server(
refresh_token: Optional[str] = None,
scope: Optional[str] = None,
):
_raise_if_not_oauth2(mcp_server)
if grant_type not in ("authorization_code", "refresh_token"):
raise HTTPException(status_code=400, detail="Unsupported grant_type")
@ -573,6 +615,178 @@ async def exchange_token_with_server(
return JSONResponse(result, headers=TOKEN_NO_CACHE_HEADERS)
class _DcrClientRegistration(BaseModel):
"""RFC 7591 dynamic client registration response, narrowed to the fields the gateway
must persist to authenticate later token-endpoint calls. Extra members are ignored."""
client_id: str
client_secret: Optional[str] = None
token_endpoint_auth_method: Optional[str] = None
class _PersistedDcrCredentials(BaseModel):
client_id: Optional[str] = None
client_secret: Optional[str] = None
token_endpoint_auth_method: Optional[str] = None
def _get_persisted_dcr_credentials(credentials: object) -> Optional[_PersistedDcrCredentials]:
if not credentials:
return None
try:
return (
_PersistedDcrCredentials.model_validate_json(credentials)
if isinstance(credentials, str)
else _PersistedDcrCredentials.model_validate(credentials)
)
except ValidationError:
return None
def _decrypt_persisted_dcr_credential(value: Optional[str], key: str) -> Optional[str]:
if value is None:
return None
return decrypt_value_helper(
value=value,
key=key,
exception_type="debug",
return_original_value=True,
)
def _apply_persisted_dcr_credentials(mcp_server: MCPServer, credentials: _PersistedDcrCredentials) -> bool:
client_id = _decrypt_persisted_dcr_credential(credentials.client_id, "client_id")
if not client_id:
return False
mcp_server.client_id = client_id
mcp_server.client_secret = _decrypt_persisted_dcr_credential(credentials.client_secret, "client_secret")
mcp_server.token_endpoint_auth_method = credentials.token_endpoint_auth_method
return True
async def _get_persisted_mcp_server_with_dcr_client_id(
mcp_server: MCPServer,
) -> Optional[tuple["LiteLLM_MCPServerTable", _PersistedDcrCredentials]]:
from litellm.proxy._experimental.mcp_server.db import get_mcp_server # noqa: PLC0415
from litellm.proxy.utils import get_prisma_client_or_throw # noqa: PLC0415
try:
prisma_client = get_prisma_client_or_throw("Database not connected. Cannot read MCP OAuth client registration.")
persisted_mcp_server = await get_mcp_server(
prisma_client=prisma_client,
server_id=mcp_server.server_id,
)
except Exception as exc: # noqa: BLE001
verbose_logger.debug(
"register_client_with_server: failed to read persisted DCR client registration for server_id=%s: %s",
mcp_server.server_id,
exc,
)
return None
if persisted_mcp_server is None:
return None
credentials = _get_persisted_dcr_credentials(persisted_mcp_server.credentials)
if credentials is None or not credentials.client_id:
return None
return persisted_mcp_server, credentials
async def _reuse_persisted_dcr_client_if_available(mcp_server: MCPServer) -> bool:
persisted = await _get_persisted_mcp_server_with_dcr_client_id(mcp_server)
if persisted is None:
return False
persisted_mcp_server, credentials = persisted
if not _apply_persisted_dcr_credentials(mcp_server, credentials):
return False
from litellm.proxy._experimental.mcp_server.mcp_server_manager import ( # noqa: PLC0415
global_mcp_server_manager,
)
try:
await global_mcp_server_manager.update_server(persisted_mcp_server)
except Exception as exc: # noqa: BLE001
verbose_logger.warning(
"register_client_with_server: failed to refresh persisted DCR client registration for server_id=%s: %s",
mcp_server.server_id,
exc,
)
return bool(mcp_server.client_id)
DcrRegistrationPersistenceResult = Literal["persisted", "reused", "failed"]
async def _persist_dcr_client_registration(
mcp_server: MCPServer, registration_response: object
) -> DcrRegistrationPersistenceResult:
"""Persist the dynamically registered OAuth client (RFC 7591) onto the MCP server row.
The interactive authorization_code flow mints a ``client_id`` via Dynamic Client
Registration that discovery cannot re-derive; without persisting it the autonomous
``refresh_token`` grant has no client identity, so an expired access token forces a
full re-authorization instead of a silent refresh. Mirrors the ``encrypt_credentials``
write that ``client_credentials`` and token exchange already use. Failures are logged,
never raised: registration still returns to the caller even when persistence fails.
"""
try:
registration = _DcrClientRegistration.model_validate(registration_response)
except ValidationError as exc:
verbose_logger.warning(
"register_client_with_server: DCR response has no usable client_id for server_id=%s; "
"client registration not persisted (%s)",
mcp_server.server_id,
exc,
)
return "failed"
if await _reuse_persisted_dcr_client_if_available(mcp_server):
return "reused"
credentials: MCPCredentials = {
"client_id": registration.client_id,
**({"client_secret": registration.client_secret} if registration.client_secret is not None else {}),
**(
{"token_endpoint_auth_method": "client_secret_basic"}
if registration.token_endpoint_auth_method == "client_secret_basic"
else {}
),
}
from litellm.proxy._experimental.mcp_server.db import update_mcp_server # noqa: PLC0415
from litellm.proxy._experimental.mcp_server.mcp_server_manager import ( # noqa: PLC0415
global_mcp_server_manager,
)
from litellm.proxy._types import UpdateMCPServerRequest # noqa: PLC0415
from litellm.proxy.utils import get_prisma_client_or_throw # noqa: PLC0415
try:
prisma_client = get_prisma_client_or_throw(
"Database not connected. Cannot persist MCP OAuth client registration."
)
updated_row = await update_mcp_server(
prisma_client=prisma_client,
data=UpdateMCPServerRequest(
server_id=mcp_server.server_id,
credentials=credentials,
**({"token_url": mcp_server.token_url} if mcp_server.token_url else {}),
),
touched_by="mcp_oauth_dcr",
)
await global_mcp_server_manager.update_server(updated_row)
return "persisted"
except Exception as exc: # noqa: BLE001
verbose_logger.warning(
"register_client_with_server: failed to persist DCR client registration for server_id=%s: %s",
mcp_server.server_id,
exc,
)
return "failed"
async def register_client_with_server(
request: Request,
mcp_server: MCPServer,
@ -581,7 +795,9 @@ async def register_client_with_server(
response_types: Optional[list],
token_endpoint_auth_method: Optional[str],
fallback_client_id: Optional[str] = None,
persist_credentials: bool = False,
):
_raise_if_not_oauth2(mcp_server)
request_base_url = get_request_base_url(request)
dummy_return = {
"client_id": fallback_client_id or mcp_server.server_name,
@ -589,7 +805,10 @@ async def register_client_with_server(
"redirect_uris": [f"{request_base_url}/callback"],
}
if mcp_server.client_id and mcp_server.client_secret:
if mcp_server.client_id:
return dummy_return
if await _reuse_persisted_dcr_client_if_available(mcp_server):
return dummy_return
if mcp_server.authorization_url is None:
@ -625,6 +844,11 @@ async def register_client_with_server(
token_response = response.json()
if persist_credentials:
persistence_result = await _persist_dcr_client_registration(mcp_server, token_response)
if persistence_result == "reused":
return dummy_return
return JSONResponse(token_response)
@ -655,6 +879,7 @@ async def authorize(
mcp_server = _resolve_oauth2_server_for_root_endpoints(client_ip=client_ip)
if mcp_server is None:
raise HTTPException(status_code=404, detail="MCP server not found")
_raise_if_not_oauth2(mcp_server)
# Use server's stored client_id when caller doesn't supply one.
# Raise a clear error instead of passing an empty string — an empty
# client_id would silently produce a broken authorization URL.
@ -1063,6 +1288,8 @@ async def _build_oauth_protected_resource_response(
detail=(f"Upstream oauth-protected-resource metadata unavailable for MCP server {mcp_server.name!r}"),
)
_raise_unless_oauth2_discovery_server(mcp_server, mcp_server_name, "not an OAuth-protected resource")
return {
"authorization_servers": [
(f"{request_base_url}/{mcp_server_name}" if mcp_server_name else f"{request_base_url}")
@ -1149,6 +1376,8 @@ def _build_oauth_authorization_server_response(
if mcp_server_name:
mcp_server = global_mcp_server_manager.get_mcp_server_by_name(mcp_server_name, client_ip=client_ip)
_raise_unless_oauth2_discovery_server(mcp_server, mcp_server_name, "not an OAuth authorization server")
return {
"issuer": request_base_url, # point to your proxy
"authorization_endpoint": authorization_endpoint,

498
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

File diff suppressed because it is too large Load diff

112
litellm/proxy/_experimental/mcp_server/rest_endpoints.py
View file

@ -77,6 +77,7 @@ if MCP_AVAILABLE:
ListMCPToolsRestAPIResponseObject,
MCPInfo,
MCPServer,
_fire_mcp_success_logging,
_tool_name_matches,
execute_mcp_tool,
filter_tools_by_allowed_tools,
@ -84,6 +85,24 @@ if MCP_AVAILABLE:
########################################################
############ MCP Server REST API Routes #################
async def _safe_fire_mcp_success_logging(
logging_obj: Optional[Any],
result: Any,
start_time: datetime,
end_time: datetime,
) -> None:
if logging_obj is None:
return
logging_results = await asyncio.gather(
_fire_mcp_success_logging(logging_obj, result, start_time, end_time),
return_exceptions=True,
)
logging_error = logging_results[0]
if isinstance(logging_error, asyncio.CancelledError):
raise logging_error
if isinstance(logging_error, BaseException):
verbose_logger.warning("MCP tool success logging failed (continuing): %s", logging_error)
def _get_server_auth_header(
server,
mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
@ -569,6 +588,21 @@ if MCP_AVAILABLE:
include_disabled_tools and user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN
)
if apply_tool_filters and getattr(
getattr(user_api_key_dict, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
from litellm.proxy._experimental.mcp_server.tool_search import (
get_virtual_tool_definitions,
)
return {
"tools": get_virtual_tool_definitions(),
"error": None,
"message": "Successfully retrieved tools",
}
# Extract auth headers from request
headers = request.headers
raw_headers_from_request = dict(headers)
@ -727,6 +761,77 @@ if MCP_AVAILABLE:
try:
data = await request.json()
tool_name = data.get("name")
tool_arguments = data.get("arguments") or {}
from litellm.proxy._experimental.mcp_server.tool_search import (
MCP_TOOL_CALL_TOOL_NAME,
MCP_TOOL_SEARCH_TOOL_NAME,
coerce_top_k,
handle_mcp_tool_call,
handle_mcp_tool_search,
)
if tool_name in (MCP_TOOL_SEARCH_TOOL_NAME, MCP_TOOL_CALL_TOOL_NAME):
if not getattr(
getattr(user_api_key_dict, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
raise HTTPException(
status_code=403,
detail={
"error": "forbidden",
"message": f"{tool_name} requires mcp_tool_search_enabled on the key",
},
)
rest_client_ip = IPAddressUtils.get_mcp_client_ip(request)
(
virtual_mcp_auth_header,
virtual_mcp_server_auth_headers,
virtual_raw_headers,
) = _extract_mcp_headers_from_request(request, MCPRequestHandler)
virtual_oauth2_headers = MCPRequestHandler._get_oauth2_headers_from_headers(request.headers)
if tool_name == MCP_TOOL_SEARCH_TOOL_NAME:
return await handle_mcp_tool_search(
query=tool_arguments.get("query", ""),
top_k=coerce_top_k(tool_arguments.get("top_k", 5)),
user_api_key_dict=user_api_key_dict,
client_ip=rest_client_ip,
mcp_auth_header=virtual_mcp_auth_header,
mcp_server_auth_headers=virtual_mcp_server_auth_headers,
oauth2_headers=virtual_oauth2_headers,
raw_headers=virtual_raw_headers,
)
else: # MCP_TOOL_CALL_TOOL_NAME
# Run the same pre-call pipeline as the normal call path so the
# tool execution is spend-logged and guardrail-checked.
(
_,
virtual_logging_obj,
) = await ProxyBaseLLMRequestProcessing(data=data).common_processing_pre_call_logic(
request=request,
user_api_key_dict=user_api_key_dict,
proxy_config=proxy_config,
route_type=CallTypes.call_mcp_tool.value,
proxy_logging_obj=proxy_logging_obj,
general_settings=general_settings,
)
_tool_start_time = datetime.now()
result = await handle_mcp_tool_call(
tool_name=tool_arguments.get("tool_name", ""),
arguments=tool_arguments.get("arguments") or {},
user_api_key_dict=user_api_key_dict,
client_ip=rest_client_ip,
mcp_auth_header=virtual_mcp_auth_header,
mcp_server_auth_headers=virtual_mcp_server_auth_headers,
oauth2_headers=virtual_oauth2_headers,
raw_headers=virtual_raw_headers,
litellm_logging_obj=virtual_logging_obj,
)
await _safe_fire_mcp_success_logging(virtual_logging_obj, result, _tool_start_time, datetime.now())
return result
# Validate required parameters early
server_id = data.get("server_id")
if not server_id:
@ -738,7 +843,6 @@ if MCP_AVAILABLE:
},
)
tool_name = data.get("name")
if not tool_name:
raise HTTPException(
status_code=400,
@ -748,8 +852,6 @@ if MCP_AVAILABLE:
},
)
tool_arguments = data.get("arguments") or {}
proxy_base_llm_response_processor = ProxyBaseLLMRequestProcessing(data=data)
(
data,
@ -796,11 +898,12 @@ if MCP_AVAILABLE:
user_oauth_extra_headers = await _get_user_oauth_extra_headers(target_server, user_api_key_dict)
# Call execute_mcp_tool directly (permission checks already done)
_tool_start_time = datetime.now()
result = await execute_mcp_tool(
name=tool_name,
arguments=tool_arguments,
allowed_mcp_servers=allowed_mcp_servers,
start_time=datetime.now(),
start_time=_tool_start_time,
user_api_key_auth=data.get("user_api_key_auth"),
mcp_auth_header=data.get("mcp_auth_header"),
mcp_server_auth_headers=data.get("mcp_server_auth_headers"),
@ -809,6 +912,7 @@ if MCP_AVAILABLE:
litellm_logging_obj=data.get("litellm_logging_obj"),
requested_server_id=canonical_server_id,
)
await _safe_fire_mcp_success_logging(logging_obj, result, _tool_start_time, datetime.now())
return result
except MCPMissingUserEnvVarsError as e:
verbose_logger.info(

261
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -10,8 +10,8 @@ import contextvars
import hashlib
import json
import time
import types
import traceback
import types
import uuid
from datetime import datetime
from typing import (
@ -37,13 +37,17 @@ from starlette.types import Message, Receive, Scope, Send
from litellm._logging import verbose_logger
from litellm.constants import MAXIMUM_TRACEBACK_LINES_TO_LOG
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
MCPRequestHandler,
)
from litellm.proxy._experimental.mcp_server.exceptions import MCPUpstreamAuthError
from litellm.proxy._experimental.mcp_server.discoverable_endpoints import (
get_request_base_url,
)
from litellm.proxy._experimental.mcp_server.exceptions import MCPUpstreamAuthError
from litellm.proxy._experimental.mcp_server.mcp_context import (
_mcp_active_toolset_id,
_mcp_gateway_initialize_instructions,
@ -59,10 +63,6 @@ from litellm.proxy._experimental.mcp_server.utils import (
get_server_prefix,
iter_known_server_prefixes,
)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.proxy._types import (
ProxyException,
SpecialMCPServerNames,
@ -122,9 +122,12 @@ def _write_byok_cred_cache(user_id: str, server_id: str, credential: Optional[st
# TODO: Make this a util function for litellm client usage
MCP_AVAILABLE: bool = True
try:
import weakref
from mcp import ReadResourceResult, Resource
from mcp.server import Server
from mcp.server.lowlevel.helper_types import ReadResourceContents
from mcp.server.session import ServerSession as _McpServerSession
from mcp.types import (
BlobResourceContents,
GetPromptResult,
@ -132,8 +135,6 @@ try:
TextResourceContents,
Tool,
)
from mcp.server.session import ServerSession as _McpServerSession
import weakref
# Robust auth lookup keyed by session_object.
_session_obj_auth_storage: "weakref.WeakKeyDictionary[Any, MCPAuthenticatedUser]" = weakref.WeakKeyDictionary()
@ -303,14 +304,14 @@ def _proxy_exception_to_http_exception(exc: ProxyException) -> HTTPException:
if MCP_AVAILABLE:
from mcp.server import Server
from mcp.server.lowlevel.server import NotificationOptions
from mcp.server.models import InitializationOptions
# Import auth context variables and middleware
from mcp.server.auth.middleware.auth_context import (
AuthContextMiddleware,
auth_context_var,
)
from mcp.server.lowlevel.server import NotificationOptions
from mcp.server.models import InitializationOptions
try:
from mcp.server.streamable_http_manager import StreamableHTTPSessionManager
@ -664,6 +665,19 @@ if MCP_AVAILABLE:
verbose_logger.debug(
f"MCP list_tools - MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}"
)
if getattr(
getattr(user_api_key_auth, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
from mcp.types import Tool
from litellm.proxy._experimental.mcp_server.tool_search import (
get_virtual_tool_definitions,
)
return [Tool(**d) for d in get_virtual_tool_definitions()]
# Get mcp_servers from context variable
verbose_logger.debug("MCP list_tools - Calling _list_mcp_tools")
tools = await _list_mcp_tools(
@ -688,6 +702,150 @@ if MCP_AVAILABLE:
if _session_reset_token is not None:
active_mcp_session_var.reset(_session_reset_token)
def _capture_host_progress_callback(host_server) -> Optional[Callable]:
"""Return a progress-forwarding callback bound to the host MCP session.
Returns ``None`` when the host did not supply a progress token.
"""
try:
host_ctx = host_server.request_context
except Exception as e:
verbose_logger.warning(f"Could not capture host progress context: {e}")
return None
if not (host_ctx and hasattr(host_ctx, "meta") and host_ctx.meta):
return None
host_token = getattr(host_ctx.meta, "progressToken", None)
if not (host_token and hasattr(host_ctx, "session") and host_ctx.session):
return None
host_session = host_ctx.session
async def forward_progress(progress: float, total: Optional[float]):
"""Forward progress notifications from external MCP to Host"""
try:
await host_session.send_progress_notification(
progress_token=host_token,
progress=progress,
total=total,
)
verbose_logger.debug(f"Forwarded progress {progress}/{total} to Host")
except Exception as e:
verbose_logger.error(f"Failed to forward progress to Host: {e}")
verbose_logger.debug(f"Host progressToken captured: {host_token[:8]}...")
return forward_progress
async def _build_virtual_call_logging_obj(
name: str,
arguments: dict[str, Any],
user_api_key_auth: UserAPIKeyAuth,
) -> Optional[LiteLLMLoggingObj]:
"""Run the pre-call pipeline (guardrails + logging setup) for a virtual
mcp_tool_call so the SSE path spend-logs like the REST path."""
from fastapi import Request
from litellm.proxy.common_request_processing import (
ProxyBaseLLMRequestProcessing,
)
from litellm.proxy.proxy_server import (
general_settings,
proxy_config,
proxy_logging_obj,
)
request = Request(
scope={
"type": "http",
"method": "POST",
"path": "/mcp/tools/call",
"headers": [(b"content-type", b"application/json")],
}
)
_, virtual_logging_obj = await ProxyBaseLLMRequestProcessing(
data={"name": name, "arguments": arguments}
).common_processing_pre_call_logic(
request=request,
user_api_key_dict=user_api_key_auth,
proxy_config=proxy_config,
route_type=CallTypes.call_mcp_tool.value,
proxy_logging_obj=proxy_logging_obj,
general_settings=general_settings,
)
return virtual_logging_obj
async def _dispatch_virtual_mcp_tool(
name: str,
arguments: Optional[dict[str, Any]],
user_api_key_auth: Optional[UserAPIKeyAuth],
client_ip: Optional[str],
mcp_servers: Optional[list[str]] = None,
mcp_auth_header: Optional[str] = None,
mcp_server_auth_headers: Optional[dict[str, dict[str, str]]] = None,
oauth2_headers: Optional[dict[str, str]] = None,
raw_headers: Optional[dict[str, str]] = None,
) -> Optional[CallToolResult]:
"""Handle the mcp_tool_search / mcp_tool_call virtual tools.
Returns a CallToolResult when ``name`` is a virtual tool, else ``None`` so
the caller falls through to normal tool routing.
"""
from litellm.proxy._experimental.mcp_server.tool_search import (
MCP_TOOL_CALL_TOOL_NAME,
MCP_TOOL_SEARCH_TOOL_NAME,
coerce_top_k,
handle_mcp_tool_call,
handle_mcp_tool_search,
)
if name not in (MCP_TOOL_SEARCH_TOOL_NAME, MCP_TOOL_CALL_TOOL_NAME):
return None
if not getattr(
getattr(user_api_key_auth, "object_permission", None),
"mcp_tool_search_enabled",
False,
):
return CallToolResult(
content=[
TextContent(
type="text",
text=f"Tool {name} requires mcp_tool_search_enabled on the key",
)
],
isError=True,
)
args = arguments or {}
if name == MCP_TOOL_SEARCH_TOOL_NAME:
return await handle_mcp_tool_search(
query=args.get("query", ""),
top_k=coerce_top_k(args.get("top_k", 5)),
user_api_key_dict=user_api_key_auth,
client_ip=client_ip,
mcp_servers=mcp_servers,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
)
assert user_api_key_auth is not None # guaranteed by the flag check above
virtual_logging_obj = await _build_virtual_call_logging_obj(
name=name, arguments=args, user_api_key_auth=user_api_key_auth
)
return await handle_mcp_tool_call(
tool_name=args.get("tool_name", ""),
arguments=args.get("arguments") or {},
user_api_key_dict=user_api_key_auth,
client_ip=client_ip,
mcp_servers=mcp_servers,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=virtual_logging_obj,
)
@server.call_tool()
async def mcp_server_tool_call(name: str, arguments: Dict[str, Any] | None) -> CallToolResult:
"""
@ -701,11 +859,12 @@ if MCP_AVAILABLE:
HTTPException: If tool not found or arguments missing
"""
from fastapi import Request
from mcp.server.lowlevel.server import request_ctx
from mcp.types import CallToolResult
from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.proxy.litellm_pre_call_utils import add_litellm_data_to_request
from litellm.proxy.proxy_server import proxy_config
from mcp.types import CallToolResult
from mcp.server.lowlevel.server import request_ctx
req_ctx = request_ctx.get(None)
_session_reset_token = None
@ -730,31 +889,25 @@ if MCP_AVAILABLE:
)
verbose_logger.debug(f"MCP mcp_server_tool_call - User API Key Auth from context: {user_api_key_auth}")
host_progress_callback = None
try:
host_ctx = server.request_context
if host_ctx and hasattr(host_ctx, "meta") and host_ctx.meta:
host_token = getattr(host_ctx.meta, "progressToken", None)
if host_token and hasattr(host_ctx, "session") and host_ctx.session:
host_session = host_ctx.session
async def forward_progress(progress: float, total: Optional[float]):
"""Forward progress notifications from external MCP to Host"""
try:
await host_session.send_progress_notification(
progress_token=host_token,
progress=progress,
total=total,
)
verbose_logger.debug(f"Forwarded progress {progress}/{total} to Host")
except Exception as e:
verbose_logger.error(f"Failed to forward progress to Host: {e}")
host_progress_callback = forward_progress
verbose_logger.debug(f"Host progressToken captured: {host_token[:8]}...")
except Exception as e:
verbose_logger.warning(f"Could not capture host progress context: {e}")
try:
# Inside this try so virtual-tool errors convert to isError
# CallToolResult instead of raising out of the protocol handler.
virtual_tool_result = await _dispatch_virtual_mcp_tool(
name=name,
arguments=arguments,
user_api_key_auth=user_api_key_auth,
client_ip=_client_ip,
mcp_servers=mcp_servers,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
)
if virtual_tool_result is not None:
return virtual_tool_result
host_progress_callback = _capture_host_progress_callback(server)
# Create a body date for logging
body_data = {"name": name, "arguments": arguments}
# Set trace/session id from raw_headers so spend logs and logging_obj stay consistent (same as A2A)
@ -1528,6 +1681,8 @@ if MCP_AVAILABLE:
log_list_tools_to_spendlogs: bool = False,
list_tools_log_source: Optional[str] = None,
litellm_trace_id: Optional[str] = None,
request_tags: Optional[list[str]] = None,
client_ip: Optional[str] = None,
) -> List[MCPTool]:
"""
Helper method to fetch tools from MCP servers based on server filtering criteria.
@ -1570,6 +1725,7 @@ if MCP_AVAILABLE:
"litellm_trace_id": effective_litellm_trace_id,
"metadata": {
"spend_logs_metadata": spend_logs_metadata,
**({"tags": request_tags} if request_tags else {}),
},
# Provide a small input payload for standard logging
"input": [
@ -1615,6 +1771,7 @@ if MCP_AVAILABLE:
allowed_mcp_servers = await _get_allowed_mcp_servers(
user_api_key_auth=user_api_key_auth,
mcp_servers=mcp_servers,
client_ip=client_ip,
)
# Pre-fetch OAuth credentials only when at least one server uses OAuth2,
@ -1744,7 +1901,9 @@ if MCP_AVAILABLE:
end_time = datetime.now()
try:
await litellm_logging_obj.async_success_handler(
result=all_tools,
result=[
tool.model_dump(mode="json") if isinstance(tool, MCPTool) else tool for tool in all_tools
],
start_time=list_tools_start_time,
end_time=end_time,
)
@ -2024,6 +2183,7 @@ if MCP_AVAILABLE:
raw_headers: Optional[Dict[str, str]] = None,
log_list_tools_to_spendlogs: bool = False,
list_tools_log_source: Optional[str] = None,
client_ip: Optional[str] = None,
) -> List[MCPTool]:
"""
List all available MCP tools.
@ -2033,6 +2193,7 @@ if MCP_AVAILABLE:
mcp_auth_header: Optional auth header for MCP server (deprecated)
mcp_servers: Optional list of server names/aliases to filter by
mcp_server_auth_headers: Optional dict of server-specific auth headers {server_alias: auth_value}
client_ip: Client IP for IP-based server access control
Returns:
List[MCPTool]: Combined list of tools from all accessible servers
@ -2056,6 +2217,7 @@ if MCP_AVAILABLE:
raw_headers=raw_headers,
log_list_tools_to_spendlogs=log_list_tools_to_spendlogs,
list_tools_log_source=list_tools_log_source,
client_ip=client_ip,
)
verbose_logger.debug(f"Successfully fetched {len(managed_tools)} tools from managed MCP servers")
except Exception as e:
@ -2583,6 +2745,22 @@ if MCP_AVAILABLE:
return response
async def _fire_mcp_success_logging(
logging_obj: LiteLLMLoggingObj,
result: Any,
start_time: datetime,
end_time: datetime,
) -> None:
logging_obj.post_call(original_response=result)
await logging_obj.async_post_mcp_tool_call_hook(
kwargs=logging_obj.model_call_details,
response_obj=result,
start_time=start_time,
end_time=end_time,
)
logging_obj.call_type = CallTypes.call_mcp_tool.value
await logging_obj.async_success_handler(result=result, start_time=start_time, end_time=end_time)
@client
async def call_mcp_tool(
name: str,
@ -2654,16 +2832,7 @@ if MCP_AVAILABLE:
raise
if litellm_logging_obj:
litellm_logging_obj.post_call(original_response=response)
end_time = datetime.now()
await litellm_logging_obj.async_post_mcp_tool_call_hook(
kwargs=litellm_logging_obj.model_call_details,
response_obj=response,
start_time=start_time,
end_time=end_time,
)
litellm_logging_obj.call_type = CallTypes.call_mcp_tool.value
await litellm_logging_obj.async_success_handler(result=response, start_time=start_time, end_time=end_time)
await _fire_mcp_success_logging(litellm_logging_obj, response, start_time, datetime.now())
return response
async def mcp_get_prompt(

157
litellm/proxy/_experimental/mcp_server/tool_search.py Normal file
View file

@ -0,0 +1,157 @@
from __future__ import annotations
import json
from datetime import datetime
from typing import TYPE_CHECKING, Any, Optional
if TYPE_CHECKING:
from mcp.types import CallToolResult
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
MCP_TOOL_SEARCH_TOOL_NAME: str = "mcp_tool_search"
MCP_TOOL_CALL_TOOL_NAME: str = "mcp_tool_call"
def coerce_top_k(value: Any, default: int = 5) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def search_tools(query: str, tools: list[dict[str, Any]], top_k: int = 5) -> list[dict[str, Any]]:
if not query:
return []
tokens = query.lower().split()
def _score(tool: dict[str, Any]) -> int:
haystack = (tool.get("name", "") + " " + tool.get("description", "")).lower()
return sum(1 for t in tokens if t in haystack)
scored = ((s, tool) for tool in tools if (s := _score(tool)) > 0)
return [tool for _, tool in sorted(scored, key=lambda x: x[0], reverse=True)[:top_k]]
def get_virtual_tool_definitions() -> list[dict[str, Any]]:
return [
{
"name": MCP_TOOL_SEARCH_TOOL_NAME,
"description": "Search for MCP tools by keyword. Returns top matching tools with names, descriptions, and input schemas.",
"inputSchema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Keywords to search for in tool names and descriptions.",
},
"top_k": {
"type": "integer",
"description": "Maximum number of results to return.",
"default": 5,
},
},
"required": ["query"],
},
},
{
"name": MCP_TOOL_CALL_TOOL_NAME,
"description": "Call an MCP tool by name with the given arguments.",
"inputSchema": {
"type": "object",
"properties": {
"tool_name": {
"type": "string",
"description": "The exact name of the MCP tool to call.",
},
"arguments": {
"type": "object",
"description": "Arguments to pass to the tool.",
},
},
"required": ["tool_name"],
},
},
]
async def handle_mcp_tool_search(
query: str,
top_k: int,
user_api_key_dict: UserAPIKeyAuth,
client_ip: Optional[str] = None,
mcp_servers: Optional[list[str]] = None,
mcp_auth_header: Optional[str] = None,
mcp_server_auth_headers: Optional[dict[str, dict[str, str]]] = None,
oauth2_headers: Optional[dict[str, str]] = None,
raw_headers: Optional[dict[str, str]] = None,
) -> CallToolResult:
from mcp.types import CallToolResult, TextContent
from litellm.proxy._experimental.mcp_server.server import _list_mcp_tools
mcp_tools = await _list_mcp_tools(
user_api_key_auth=user_api_key_dict,
mcp_servers=mcp_servers,
client_ip=client_ip,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
)
tools = [
{
"name": t.name,
"description": t.description or "",
"inputSchema": t.inputSchema,
}
for t in mcp_tools
]
results = search_tools(query, tools, top_k)
return CallToolResult(content=[TextContent(type="text", text=json.dumps(results))], isError=False)
async def handle_mcp_tool_call(
tool_name: str,
arguments: dict[str, Any],
user_api_key_dict: UserAPIKeyAuth,
client_ip: Optional[str] = None,
mcp_servers: Optional[list[str]] = None,
mcp_auth_header: Optional[str] = None,
mcp_server_auth_headers: Optional[dict[str, dict[str, str]]] = None,
oauth2_headers: Optional[dict[str, str]] = None,
raw_headers: Optional[dict[str, str]] = None,
litellm_logging_obj: Optional[LiteLLMLoggingObj] = None,
) -> CallToolResult:
from litellm.proxy._experimental.mcp_server.server import (
_get_allowed_mcp_servers,
execute_mcp_tool,
)
allowed_mcp_servers = await _get_allowed_mcp_servers(
user_api_key_auth=user_api_key_dict,
mcp_servers=mcp_servers,
client_ip=client_ip,
)
# Reject before dispatch when the key has no accessible servers; otherwise an
# unprefixed local tool name would fall through to the local registry in
# execute_mcp_tool, which has no server permission check.
if not allowed_mcp_servers:
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="User not allowed to call this tool.")
return await execute_mcp_tool(
name=tool_name,
arguments=arguments,
allowed_mcp_servers=allowed_mcp_servers,
start_time=datetime.now(),
user_api_key_auth=user_api_key_dict,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
oauth2_headers=oauth2_headers,
raw_headers=raw_headers,
litellm_logging_obj=litellm_logging_obj,
)

1
litellm/proxy/_experimental/out/404.html Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/404/index.html Normal file
View file

File diff suppressed because one or more lines are too long

9
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.__PAGE__.txt Normal file
View file

@ -0,0 +1,9 @@
1:"$Sreact.fragment"
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ClientPageRoot"]
3:I[871135,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","/litellm-asset-prefix/_next/static/chunks/0whkizop7gd0~.js","/litellm-asset-prefix/_next/static/chunks/0-ih8xcz_89nt.js","/litellm-asset-prefix/_next/static/chunks/0pd5zl~lciww9.js","/litellm-asset-prefix/_next/static/chunks/02ihc5xweq16v.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/0mzw3maijoev6.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/04amwk-x_vjxu.js","/litellm-asset-prefix/_next/static/chunks/0-dhh1_d1.b1u.js","/litellm-asset-prefix/_next/static/chunks/0pwkd9r.mc_ee.js","/litellm-asset-prefix/_next/static/chunks/011mgw.-67gs_.js","/litellm-asset-prefix/_next/static/chunks/0~-ovi6c4wjt1.js","/litellm-asset-prefix/_next/static/chunks/0_y-b9_d9dsuv.js","/litellm-asset-prefix/_next/static/chunks/0c2apcdkbqq0o.js","/litellm-asset-prefix/_next/static/chunks/0zrbitbm~0koh.js","/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","/litellm-asset-prefix/_next/static/chunks/0ngre0.s4-ej6.js","/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","/litellm-asset-prefix/_next/static/chunks/05t1k89l9tc3s.js","/litellm-asset-prefix/_next/static/chunks/17n.qg70cy9.9.js","/litellm-asset-prefix/_next/static/chunks/00q4mtjboprhm.js","/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","/litellm-asset-prefix/_next/static/chunks/0-3i_.uof35pm.js","/litellm-asset-prefix/_next/static/chunks/14566-_ogh-19.js","/litellm-asset-prefix/_next/static/chunks/0w39dn9x3dp9g.js","/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","/litellm-asset-prefix/_next/static/chunks/0v1rxqc1hqmrl.js","/litellm-asset-prefix/_next/static/chunks/0c4pfjjue0uc-.js"],"default"]
6:I[897367,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"OutletBoundary"]
7:"$Sreact.suspense"
0:{"rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/011mgw.-67gs_.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0~-ovi6c4wjt1.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0_y-b9_d9dsuv.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/0c2apcdkbqq0o.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0zrbitbm~0koh.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/0sx3mu2_l9g_y.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/0ngre0.s4-ej6.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/0l7em-5kjv49e.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/05t1k89l9tc3s.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/17n.qg70cy9.9.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/00q4mtjboprhm.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/0el08tticy_20.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/0-3i_.uof35pm.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/14566-_ogh-19.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/0w39dn9x3dp9g.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/0q6~n4y84cejn.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/0v1rxqc1hqmrl.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/0c4pfjjue0uc-.js","async":true}]],["$","$L6",null,{"children":["$","$7",null,{"name":"Next.MetadataOutlet","children":"$@8"}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}
4:{}
5:"$0:rsc:props:children:0:props:serverProvidedParams:params"
8:null

7
litellm/proxy/_experimental/out/__next.!KGRhc2hib2FyZCk.txt Normal file
View file

@ -0,0 +1,7 @@
1:"$Sreact.fragment"
2:I[92825,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ClientSegmentRoot"]
3:I[216370,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","/litellm-asset-prefix/_next/static/chunks/0whkizop7gd0~.js","/litellm-asset-prefix/_next/static/chunks/0-ih8xcz_89nt.js","/litellm-asset-prefix/_next/static/chunks/0pd5zl~lciww9.js","/litellm-asset-prefix/_next/static/chunks/02ihc5xweq16v.js","/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","/litellm-asset-prefix/_next/static/chunks/0mzw3maijoev6.js","/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","/litellm-asset-prefix/_next/static/chunks/04amwk-x_vjxu.js","/litellm-asset-prefix/_next/static/chunks/0-dhh1_d1.b1u.js","/litellm-asset-prefix/_next/static/chunks/0pwkd9r.mc_ee.js"],"default"]
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
0:{"rsc":["$","$1","c",{"children":[[["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0whkizop7gd0~.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0-ih8xcz_89nt.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0pd5zl~lciww9.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/02ihc5xweq16v.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/0lg.6rbfsd-l9.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/0mzw3maijoev6.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/043q3g5-5-aju.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/04amwk-x_vjxu.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/0-dhh1_d1.b1u.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/0pwkd9r.mc_ee.js","async":true}]],["$","$L2",null,{"Component":"$3","slots":{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]},"serverProvidedParams":{"params":{},"promises":["$@6"]}}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}
6:"$0:rsc:props:children:1:props:serverProvidedParams:params"

30
litellm/proxy/_experimental/out/__next._full.txt Normal file
View file

File diff suppressed because one or more lines are too long

6
litellm/proxy/_experimental/out/__next._head.txt Normal file
View file

@ -0,0 +1,6 @@
1:"$Sreact.fragment"
2:I[897367,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"ViewportBoundary"]
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"MetadataBoundary"]
4:"$Sreact.suspense"
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"IconMark"]
0:{"rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.0~dgapwhi~75y.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"/get_favicon"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}

9
litellm/proxy/_experimental/out/__next._index.txt Normal file
View file

@ -0,0 +1,9 @@
1:"$Sreact.fragment"
2:I[867271,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
3:I[71195,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
4:I[557951,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"AuthProvider"]
5:I[339756,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
6:I[837457,["/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js"],"default"]
:HL["/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/0i77.0u.82o9u.css","style"]
0:{"rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/0i77.0u.82o9u.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/0n.a~e5dwfnkn.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/0.4.bbjx7y007.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/0pidya1qvuvx8.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","template":["$","$L6",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]}]]}],"isPartial":false,"staleTime":300,"varyParams":null,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}

4
litellm/proxy/_experimental/out/__next._tree.txt Normal file
View file

@ -0,0 +1,4 @@
:HL["/litellm-asset-prefix/_next/static/chunks/05qmwjqau64bz.css","style"]
:HL["/litellm-asset-prefix/_next/static/chunks/0i77.0u.82o9u.css","style"]
:HL["/litellm-asset-prefix/_next/static/media/83afe278b6a6bb3c-s.p.0q-301v4kxxnr.woff2","font",{"crossOrigin":"","type":"font/woff2"}]
0:{"tree":{"name":"","param":null,"prefetchHints":16,"slots":{"children":{"name":"(dashboard)","param":null,"prefetchHints":0,"slots":{"children":{"name":"__PAGE__","param":null,"prefetchHints":0,"slots":null}}}}},"staleTime":300,"buildId":"5rDiFx0t_mOGYmV_8kSkw"}

16
litellm/proxy/_experimental/out/_next/static/5rDiFx0t_mOGYmV_8kSkw/_buildManifest.js Normal file
View file

@ -0,0 +1,16 @@
self.__BUILD_MANIFEST = {
"__rewrites": {
"afterFiles": [],
"beforeFiles": [
{
"source": "/litellm-asset-prefix/_next/:path+",
"destination": "/_next/:path+"
}
],
"fallback": []
},
"sortedPages": [
"/_app",
"/_error"
]
};self.__BUILD_MANIFEST_CB && self.__BUILD_MANIFEST_CB()

1
litellm/proxy/_experimental/out/_next/static/5rDiFx0t_mOGYmV_8kSkw/_clientMiddlewareManifest.js Normal file
View file

@ -0,0 +1 @@
self.__MIDDLEWARE_MATCHERS = [];self.__MIDDLEWARE_MATCHERS_CB && self.__MIDDLEWARE_MATCHERS_CB()

1
litellm/proxy/_experimental/out/_next/static/5rDiFx0t_mOGYmV_8kSkw/_ssgManifest.js Normal file
View file

@ -0,0 +1 @@
self.__SSG_MANIFEST=new Set([]);self.__SSG_MANIFEST_CB&&self.__SSG_MANIFEST_CB()

2
litellm/proxy/_experimental/out/_next/static/chunks/0-3i_.uof35pm.js Normal file
View file

File diff suppressed because one or more lines are too long

12
litellm/proxy/_experimental/out/_next/static/chunks/0-4tg9f~_a3b~.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-85n.4jrc2vv.js Normal file
View file

File diff suppressed because one or more lines are too long

3
litellm/proxy/_experimental/out/_next/static/chunks/0-dhh1_d1.b1u.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-f.2po-pctaa.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0-ih8xcz_89nt.js Normal file
View file

File diff suppressed because one or more lines are too long

143
litellm/proxy/_experimental/out/_next/static/chunks/0.4.bbjx7y007.js Normal file
View file

File diff suppressed because one or more lines are too long

10
litellm/proxy/_experimental/out/_next/static/chunks/0.bx44y-6~tug.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/0.yiw37jc_bvi.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/00cy3g~l27g1y.js Normal file
View file

File diff suppressed because one or more lines are too long

420
litellm/proxy/_experimental/out/_next/static/chunks/00jwo~_zp.35~.js Normal file
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more