scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-10 22:41:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 83

Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339)
Some checks failed
OSS Daily Guardrails / Run OSS daily safe checks (push) Has been cancelled
Details

Browse source 
This reverts commit 90f495f8dc.
This commit is contained in:
yuneng-jiang 2026-07-14 19:32:25 -07:00 • committed by GitHub
parent 90f495f8dc
commit 6372ca32c1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1753 changed files with 15625 additions and 76021 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/CODEOWNERS vendored
View file

@ -1,2 +0,0 @@
/ui/ @yuneng-jiang @ryan-crabbe-berri
/litellm/proxy/_experimental/out/ @yuneng-jiang @ryan-crabbe-berri

48
.github/actions/detect-backend-changes/action.yml vendored
View file

@ -1,48 +0,0 @@
name: "Detect backend-relevant changes"
description: >-
Classify the pull request's changed files with .circleci/scripts/classify_changes.sh
and expose decision=run|skip. decision=skip means only ui/**, **.md or **.mdx files
changed, so callers can short-circuit expensive steps while the job still completes
successfully and satisfies its required status check. The decision defaults to run for
any non pull_request event or whenever the changed set cannot be resolved, so tests are
never skipped when the classification is uncertain.
outputs:
decision:
description: "run when backend-relevant files changed, otherwise skip"
value: ${{ steps.classify.outputs.decision }}
runs:
using: composite
steps:
- id: classify
shell: bash
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -uo pipefail
if [ -z "${BASE_SHA:-}" ]; then
echo "detect-backend-changes: not a pull_request event; running job"
echo "decision=run" >> "${GITHUB_OUTPUT}"
exit 0
fi
if ! git fetch --no-tags --depth=1 origin "${BASE_SHA}" >/dev/null 2>&1; then
echo "detect-backend-changes: could not fetch base ${BASE_SHA}; running job"
echo "decision=run" >> "${GITHUB_OUTPUT}"
exit 0
fi
changed="$(git diff --name-only "${BASE_SHA}" HEAD 2>/dev/null)" || {
echo "detect-backend-changes: git diff failed; running job"
echo "decision=run" >> "${GITHUB_OUTPUT}"
exit 0
}
if [ -z "${changed}" ]; then
echo "detect-backend-changes: no changed files vs ${BASE_SHA}; skipping job"
echo "decision=skip" >> "${GITHUB_OUTPUT}"
exit 0
fi
echo "detect-backend-changes: changed files vs ${BASE_SHA}:"
printf '%s\n' "${changed}" | sed 's/^/ /'
decision="$(printf '%s\n' "${changed}" | bash .circleci/scripts/classify_changes.sh backend)" || decision="run"
echo "detect-backend-changes: decision=${decision}"
echo "decision=${decision}" >> "${GITHUB_OUTPUT}"

47
.github/actions/setup-uv-with-retries/action.yml vendored
View file

@ -1,47 +0,0 @@
name: "Set up uv with retries"
description: >-
Install uv via astral-sh/setup-uv, retrying on transient failures. Even with
an exact pinned version, the action resolves the artifact URL by fetching
https://raw.githubusercontent.com/astral-sh/versions/main/v1/uv.ndjson in a
single request with no retry, timeout, or fallback, so one connection-level
network error ("fetch failed") fails the whole job before any test runs.
Retrying the full step covers the manifest fetch and the binary download.
inputs:
version:
description: "uv version to install"
required: true
runs:
using: composite
steps:
- name: Set up uv (attempt 1)
id: attempt-1
continue-on-error: true
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: ${{ inputs.version }}
- name: Wait before attempt 2
if: steps.attempt-1.outcome == 'failure'
shell: bash
run: sleep 15
- name: Set up uv (attempt 2)
id: attempt-2
if: steps.attempt-1.outcome == 'failure'
continue-on-error: true
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: ${{ inputs.version }}
- name: Wait before attempt 3
if: steps.attempt-2.outcome == 'failure'
shell: bash
run: sleep 30
- name: Set up uv (attempt 3)
if: steps.attempt-2.outcome == 'failure'
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: ${{ inputs.version }}

24
.github/pull_request_template.md vendored
View file

@ -41,27 +41,3 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
✅ Test
## Changes
## QA runbook
<!-- Only needed when your PR edits tests/e2e; delete this section otherwise
For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one top-level bullet per test giving its pytest node id followed by what it proves in plain words, then a nested "- [ ]" checklist where each item is a concrete action (route, request body, expected response) and the final item is the sanity-check step shown in the examples. Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
Example checklists:
- tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_rpm_limit_blocks_over_limit - a key allowed 2 requests a minute serves exactly 2 and refuses the 3rd
- [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
- [ ] Send three /v1/chat/completions requests with that key inside one minute
- [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
- [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
- tests/e2e/management/test_management_e2e.py::TestModelRoutes::test_model_create_appears_in_ui - a deployment created through the API shows up on the Admin UI models page
- [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
- [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
- [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
-->
### Final Attestation
- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

15
.github/workflows/_test-unit-base.yml vendored
View file

@ -45,25 +45,19 @@ jobs:
name: Run tests
runs-on: ubuntu-latest
timeout-minutes: ${{ inputs.timeout-minutes }}
outputs:
decision: ${{ steps.changes.outputs.decision }}
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
@ -78,19 +72,16 @@ jobs:
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
env:
PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
run: |
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
- name: Run tests
if: steps.changes.outputs.decision != 'skip'
env:
TEST_PATH: ${{ inputs.test-path }}
MAX_FAILURES: ${{ inputs.max-failures }}
@ -123,7 +114,7 @@ jobs:
fi
- name: Save coverage report
if: always() && steps.changes.outputs.decision != 'skip'
if: always()
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
name: coverage-${{ inputs.artifact-name }}-${{ github.run_id }}-${{ github.run_attempt }}
@ -133,7 +124,7 @@ jobs:
upload-coverage:
name: Upload coverage to Codecov
needs: run
if: always() && needs.run.outputs.decision != 'skip'
if: always()
runs-on: ubuntu-latest
permissions:
contents: read

2
.github/workflows/auto_update_price_and_context_window.yml vendored
View file

@ -18,7 +18,7 @@ jobs:
with:
persist-credentials: false
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
- name: Update JSON Data

2
.github/workflows/check-ui-api-types.yml vendored
View file

@ -31,7 +31,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

2
.github/workflows/codspeed.yml vendored
View file

@ -37,7 +37,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

4
.github/workflows/guard-main-branch.yml vendored
View file

@ -31,12 +31,12 @@ jobs:
echo "PR head repo: $HEAD_REPO"
echo "PR head branch: $HEAD_REF"
if [ "$HEAD_REPO" != "$BASE_REPO" ]; then
echo "::error::PRs to main must originate from the canonical repository ($BASE_REPO), not a fork ($HEAD_REPO). External contributors should open PRs against the current daily OSS branch (named litellm_oss_daily_YYYY_MM_DD; a fresh one is cut each weekday, so target the most recent) instead."
echo "::error::PRs to main must originate from the canonical repository ($BASE_REPO), not a fork ($HEAD_REPO). External contributors should open PRs against the 'litellm_oss_staging' branch instead."
exit 1
fi
if [ "$HEAD_REF" = "litellm_internal_staging" ] || [[ "$HEAD_REF" == litellm_hotfix_?* ]]; then
echo "Allowed source branch."
exit 0
fi
echo "::error::PRs to main must originate from 'litellm_internal_staging' or a 'litellm_hotfix_*' branch. Got: '$HEAD_REF'. If this is a contribution, retarget the PR against the current daily OSS branch (named litellm_oss_daily_YYYY_MM_DD; a fresh one is cut each weekday, so target the most recent) instead."
echo "::error::PRs to main must originate from 'litellm_internal_staging' or a 'litellm_hotfix_*' branch. Got: '$HEAD_REF'. If this is a contribution, retarget the PR against 'litellm_oss_staging' instead."
exit 1

2
.github/workflows/mutation-test.yml vendored
View file

@ -39,7 +39,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

2
.github/workflows/oss_daily_guardrails.yml vendored
View file

@ -35,7 +35,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

2
.github/workflows/test-code-quality.yml vendored
View file

@ -38,7 +38,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

16
.github/workflows/test-linting.yml vendored
View file

@ -33,7 +33,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
@ -48,7 +48,7 @@ jobs:
- name: Install dependencies
run: |
uv sync --frozen --group proxy-dev --group e2e-dev
uv sync --frozen --group proxy-dev
# basedpyright resolves Prisma's generated client (litellm/proxy/schema.prisma)
# only after `prisma generate` writes prisma/client.py et al. Without this the
@ -107,16 +107,6 @@ jobs:
run: |
(uv run --no-sync basedpyright --outputjson || true) | uv run --no-sync python scripts/type_check_gate.py --base "$BASE_SHA"
- name: Check tests/e2e basedpyright (zero errors)
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
if git diff --name-only --diff-filter=ACMRD "$BASE_SHA"...HEAD -- 'tests/e2e/**/*.py' | grep -q .; then
uv run --no-sync basedpyright tests/e2e
else
echo "No changed tests/e2e Python files; skipping."
fi
- name: Check for circular imports
run: |
cd litellm
@ -172,7 +162,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

76
.github/workflows/test-litellm-ui-build.yml vendored
View file

@ -36,3 +36,79 @@ jobs:
- name: Build
run: npm run build
frontend-lint:
runs-on: ubuntu-latest
timeout-minutes: 8
defaults:
run:
working-directory: ui/litellm-dashboard
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 0
persist-credentials: false
- name: Collect changed files
id: changed
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
: > "$RUNNER_TEMP/prettier_files.txt"
: > "$RUNNER_TEMP/eslint_files.txt"
while IFS= read -r f; do
[ -f "$f" ] || continue
case "$f" in
*.js | *.jsx | *.ts | *.tsx | *.mjs | *.cjs)
printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt"
printf '%s\n' "$f" >> "$RUNNER_TEMP/eslint_files.txt" ;;
*.json | *.css | *.scss | *.md | *.mdx | *.yml | *.yaml | *.html)
printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt" ;;
esac
done < <(git diff --name-only --diff-filter=ACMR --relative "$BASE_SHA"...HEAD -- .)
if [ -s "$RUNNER_TEMP/prettier_files.txt" ] || [ -s "$RUNNER_TEMP/eslint_files.txt" ]; then
echo "has_files=true" >> "$GITHUB_OUTPUT"
else
echo "has_files=false" >> "$GITHUB_OUTPUT"
echo "No lintable UI files changed in this PR; nothing to check."
fi
- name: Setup Node.js
if: steps.changed.outputs.has_files == 'true'
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
with:
node-version: "20"
cache: "npm"
cache-dependency-path: ui/litellm-dashboard/package-lock.json
- name: Install dependencies
if: steps.changed.outputs.has_files == 'true'
run: npm ci
- name: Lint changed files (prettier + eslint)
if: steps.changed.outputs.has_files == 'true'
run: |
prettier_files=()
eslint_files=()
while IFS= read -r f; do prettier_files+=("$f"); done < "$RUNNER_TEMP/prettier_files.txt"
while IFS= read -r f; do eslint_files+=("$f"); done < "$RUNNER_TEMP/eslint_files.txt"
status=0
if [ ${#prettier_files[@]} -gt 0 ]; then
echo "::group::Prettier (${#prettier_files[@]} files)"
npx prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
echo "::endgroup::"
fi
if [ ${#eslint_files[@]} -gt 0 ]; then
echo "::group::ESLint (${#eslint_files[@]} files)"
npx eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
echo "::endgroup::"
fi
exit $status
- name: Check lint budgets
if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
run: |
npx eslint . -f json -o "$RUNNER_TEMP/lint-report.json" || true
node scripts/check-lint-budgets.mjs "$RUNNER_TEMP/lint-report.json" eslint-budgets.json --check eslint-metrics.json

92
.github/workflows/test-litellm-ui-lint.yml vendored
View file

@ -1,92 +0,0 @@
name: UI Lint
permissions:
contents: read
on:
pull_request:
branches:
- main
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
jobs:
frontend-lint:
runs-on: ubuntu-latest
timeout-minutes: 8
defaults:
run:
working-directory: ui/litellm-dashboard
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 0
persist-credentials: false
- name: Collect changed files
id: changed
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
: > "$RUNNER_TEMP/prettier_files.txt"
: > "$RUNNER_TEMP/eslint_files.txt"
while IFS= read -r f; do
[ -f "$f" ] || continue
case "$f" in
*.js | *.jsx | *.ts | *.tsx | *.mjs | *.cjs)
printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt"
printf '%s\n' "$f" >> "$RUNNER_TEMP/eslint_files.txt" ;;
*.json | *.css | *.scss | *.md | *.mdx | *.yml | *.yaml | *.html)
printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt" ;;
esac
done < <(git diff --name-only --diff-filter=ACMR --relative "$BASE_SHA"...HEAD -- .)
if [ -s "$RUNNER_TEMP/prettier_files.txt" ] || [ -s "$RUNNER_TEMP/eslint_files.txt" ]; then
echo "has_files=true" >> "$GITHUB_OUTPUT"
else
echo "has_files=false" >> "$GITHUB_OUTPUT"
echo "No lintable UI files changed in this PR; nothing to check."
fi
- name: Setup Node.js
if: steps.changed.outputs.has_files == 'true'
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
with:
node-version: "20"
cache: "npm"
cache-dependency-path: ui/litellm-dashboard/package-lock.json
- name: Install dependencies
if: steps.changed.outputs.has_files == 'true'
run: npm ci
- name: Lint changed files (prettier + eslint)
if: steps.changed.outputs.has_files == 'true'
run: |
prettier_files=()
eslint_files=()
while IFS= read -r f; do prettier_files+=("$f"); done < "$RUNNER_TEMP/prettier_files.txt"
while IFS= read -r f; do eslint_files+=("$f"); done < "$RUNNER_TEMP/eslint_files.txt"
status=0
if [ ${#prettier_files[@]} -gt 0 ]; then
echo "::group::Prettier (${#prettier_files[@]} files)"
npx prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
echo "::endgroup::"
fi
if [ ${#eslint_files[@]} -gt 0 ]; then
echo "::group::ESLint (${#eslint_files[@]} files)"
npx eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
echo "::endgroup::"
fi
exit $status
- name: Check lint budgets
if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
run: |
npx eslint . -f json -o "$RUNNER_TEMP/lint-report.json" || true
node scripts/check-lint-budgets.mjs "$RUNNER_TEMP/lint-report.json" eslint-budgets.json
- name: Check for dead code (knip)
if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
run: npm run knip:ci

2
.github/workflows/test-mcp.yml vendored
View file

@ -32,7 +32,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

2
.github/workflows/test-semgrep.yml vendored
View file

@ -31,7 +31,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

2
.github/workflows/test-terraform-provider.yml vendored
View file

@ -74,7 +74,7 @@ jobs:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"

9
.github/workflows/test-unit-documentation.yml vendored
View file

@ -32,17 +32,13 @@ jobs:
path: docs/my-website
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
@ -57,12 +53,10 @@ jobs:
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
env:
PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
run: |
@ -70,7 +64,6 @@ jobs:
# Run the same documentation tests that CircleCI ran (as direct Python scripts)
- name: Run documentation validation tests
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync python ./tests/documentation_tests/test_env_keys.py
uv run --no-sync python ./tests/documentation_tests/test_router_settings.py

9
.github/workflows/test-unit-proxy-legacy.yml vendored
View file

@ -49,17 +49,13 @@ jobs:
with:
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
@ -74,19 +70,16 @@ jobs:
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
env:
PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
run: |
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
- name: Run tests - ${{ matrix.test-group.name }}
if: steps.changes.outputs.decision != 'skip'
env:
TEST_PATH: ${{ matrix.test-group.path }}
run: |

7
.gitignore vendored
View file

@ -106,13 +106,6 @@ STABILIZATION_TODO.md
**/coverage
test-config
# Claude Code compatibility-matrix pytest artifact (CI-only output).
compat-results.json
compat-results.json.shards/
compat-rate-limit-summary.json
# Matrix JSON produced by the daily-cron publisher (pushed to litellm-docs).
compatibility-matrix.json
# ---------- Terraform ----------
# Provider binaries + module cache — regenerated by `terraform init`.
**/.terraform/

4
CLAUDE.md
View file

@ -19,7 +19,7 @@ Same thing for bug fixes. The tests should make it so that this specific bug can
End-to-end tests belong in `tests/e2e/` and must follow the harness conventions documented in that directory's `CLAUDE.md`
When creating PRs, don't set base to `main`. `litellm_internal_staging` serves that purpose for internal contributors; external / OSS contributions target the current daily OSS branch instead, named `litellm_oss_daily_YYYY_MM_DD` (a fresh one is cut each weekday, so use the most recent)
When creating PRs, don't set base to `main`. `litellm_internal_staging` serves that purpose
When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule
@ -39,8 +39,6 @@ Don't hesitate to use values in .env to get needed API keys and other secrets, a
Python max line length is 120, not 88
On a fresh worktree or clone, run `make bootstrap` before anything else. It provisions everything tests, `make pre-commit`, and a local proxy need
Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed. It only runs when there are staged frontend and/or backend changes and calculates violations, generates types, etc. based on the worktree, so stage what you need or stash/delete unwanted files in litellm/ or ui/ (where backend and frontend lint run, respectively) before running it. If it fails because dashboard api types are stale, it already regenerated them for you. You just need to stage the schema.d.ts, re-run `make pre-commit` to confirm it passes, and commit
When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-budget.json`, or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered limits so the ceilings ratchet down instead of leaving stale headroom. It measures the working tree, so it must contain exactly the fixes you're committing

2
CONTRIBUTING.md
View file

@ -322,7 +322,7 @@ npm run build
## Submitting Your PR
1. **Push your branch**: `git push origin your-feature-branch`
2. **Create a PR**: Go to GitHub and open a pull request against the current daily OSS branch, named `litellm_oss_daily_YYYY_MM_DD`. A fresh one is cut each weekday, so pick the most recent from the [branch list](https://github.com/BerriAI/litellm/branches/all?query=litellm_oss_daily). Do not target `main`.
2. **Create a PR**: Go to GitHub and create a pull request
3. **Fill out the PR template**: Provide clear description of changes
4. **Wait for review**: Maintainers will review and provide feedback
5. **Address feedback**: Make requested changes and push updates

28
Makefile
View file

@ -5,16 +5,15 @@
test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
info lint lint-dev lint-checks format \
lint-basedpyright lint-e2e-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
lint-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
install-dev install-proxy-dev install-test-deps install-hooks \
install-helm-unittest check-circular-imports check-import-safety pre-commit \
lint-install lint-fetch-base bootstrap
lint-install lint-fetch-base
# Default target
help:
@echo "Available commands:"
@echo " make bootstrap - Provision a fresh clone/worktree"
@echo " make install-dev - Install development dependencies"
@echo " make install-proxy-dev - Install proxy development dependencies"
@echo " make install-dev-ci - Install dev dependencies (CI-compatible, pins OpenAI)"
@ -28,7 +27,6 @@ help:
@echo " make lint - Run all linting (Ruff, basedpyright, format check, circular imports, import safety)"
@echo " make lint-ruff - Run Ruff linting only"
@echo " make lint-basedpyright - Run basedpyright strict, gated by per-rule error counts"
@echo " make lint-e2e-basedpyright - Run basedpyright over tests/e2e (zero errors allowed)"
@echo " make lint-basedpyright-budget-update - Ratchet basedpyright limits down by what this branch fixed"
@echo " make lint-format - Check ruff format formatting (matches CI)"
@echo " make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its limit"
@ -56,7 +54,6 @@ UV := uv
UV_RUN := $(UV) run --no-sync
LINT_DEP_INSTALL ?= install-dev
LINT_E2E_DEP_INSTALL ?= lint-install
LINT_DEP_BASE ?= lint-fetch-base
LINT_JOBS := $(shell sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4)
LINT_OUTPUT_SYNC := $(if $(filter output-sync,$(.FEATURES)),--output-sync=target,)
@ -72,18 +69,6 @@ info:
install-dev:
$(UV) sync --inexact --frozen
bootstrap:
$(UV) sync --inexact --frozen --extra proxy --group proxy-dev --group e2e-dev
$(UV_RUN) python scripts/prisma_generate_if_needed.py
cd ui/litellm-dashboard && npm ci --no-audit --no-fund
@main_root=$$(git worktree list --porcelain | head -1 | sed 's/^worktree //'); \
if [ "$$main_root" != "$$(git rev-parse --show-toplevel)" ] && [ -f "$$main_root/.env" ] && [ ! -f .env ]; then \
cp "$$main_root/.env" .env && echo "bootstrap: copied .env from $$main_root"; \
else \
echo "bootstrap: .env left untouched"; \
fi
@echo "bootstrap: done"
install-proxy-dev:
$(UV) sync --frozen --group proxy-dev --extra proxy
@ -126,7 +111,7 @@ lint-fetch-base:
# CI's). --inexact tops up the venv instead of pruning the proxy extras gen:api and the
# running proxy need.
lint-install:
$(UV) sync --inexact --frozen --group proxy-dev --group e2e-dev
$(UV) sync --inexact --frozen --group proxy-dev
$(UV_RUN) python scripts/prisma_generate_if_needed.py
# Diff-scoped format check, identical to test-linting.yml's "Check ruff format" step:
@ -179,9 +164,6 @@ lint-ruff-FULL-dev: install-dev
lint-basedpyright: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --base origin/litellm_internal_staging
lint-e2e-basedpyright: $(LINT_E2E_DEP_INSTALL)
$(UV_RUN) basedpyright tests/e2e
# Type-discipline budget (mutable collections / casts / type guards / kwargs /
# unexplained suppressions), the test-linting.yml step `make lint` used to omit.
lint-type-discipline: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
@ -226,9 +208,9 @@ check-import-safety: $(LINT_DEP_INSTALL)
# base fetch) runs once up front; the checks themselves are independent, so a sub-make
# fans them out with -j and the fast ones finish under basedpyright's shadow.
lint: lint-install lint-fetch-base
$(MAKE) -j $(LINT_JOBS) $(LINT_OUTPUT_SYNC) LINT_DEP_INSTALL= LINT_E2E_DEP_INSTALL= LINT_DEP_BASE= lint-checks
$(MAKE) -j $(LINT_JOBS) $(LINT_OUTPUT_SYNC) LINT_DEP_INSTALL= LINT_DEP_BASE= lint-checks
lint-checks: lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright lint-e2e-basedpyright check-circular-imports check-import-safety
lint-checks: lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright check-circular-imports check-import-safety
# Faster linting for local development (only checks changed code)
lint-dev: lint-format-changed check-circular-imports check-import-safety

13
README.md
View file

@ -552,12 +552,17 @@ The Terraform modules live at [`terraform/litellm/aws/`](./terraform/litellm/aws
2. Run dependent services `docker-compose up db prometheus`
#### Backend
1. Run `make bootstrap`
2. Start proxy backend: `uv run python litellm/proxy/proxy_cli.py`
1. (In root) create virtual environment `python -m venv .venv`
2. Activate virtual environment `source .venv/bin/activate`
3. Install dependencies `uv sync --all-extras --group proxy-dev`
4. `uv run prisma generate`
5. `prisma generate`
6. Start proxy backend `python litellm/proxy/proxy_cli.py`
#### Frontend
1. Navigate to `ui/litellm-dashboard` (dependencies were already installed w/ `make bootstrap`)
2. Start dashboard: `npm run dev`
1. Navigate to `ui/litellm-dashboard`
2. Install dependencies `npm install`
3. Run `npm run dev` to start the dashboard
### Verify Docker Image Signatures

1
backend/routes/allowlist.py
View file

@ -46,7 +46,6 @@ BACKEND_PATH_PREFIXES: tuple[str, ...] = (
"/fallback",
"/fallbacks",
"/cache_settings",
"/coordination_redis/",
"/cost_tracking",
"/cost/",
"/credentials",

10
basedpyright-code-budget.json
View file

@ -57,7 +57,7 @@
"limit": 5900
},
"reportMissingTypeArgument": {
"limit": 15903
"limit": 15918
},
"reportMissingTypeStubs": {
"limit": 41
@ -105,13 +105,13 @@
"limit": 113
},
"reportUnknownMemberType": {
"limit": 40539
"limit": 40541
},
"reportUnknownParameterType": {
"limit": 20403
"limit": 20418
},
"reportUnknownVariableType": {
"limit": 32141
"limit": 32151
},
"reportUnnecessaryCast": {
"limit": 177
@ -123,7 +123,7 @@
"limit": 7
},
"reportUnnecessaryIsInstance": {
"limit": 1209
"limit": 1212
},
"reportUntypedBaseClass": {
"limit": 165

120
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -29,7 +29,7 @@ class CheckBatchCost:
proxy_logging_obj: "ProxyLogging",
prisma_client: "PrismaClient",
llm_router: "Router",
track_unmanaged_batch_cost: bool = False,
track_unmanaged_vertex_batch_cost: bool = False,
):
from litellm.proxy.utils import PrismaClient, ProxyLogging
from litellm.router import Router
@ -37,7 +37,7 @@ class CheckBatchCost:
self.proxy_logging_obj: ProxyLogging = proxy_logging_obj
self.prisma_client: PrismaClient = prisma_client
self.llm_router: Router = llm_router
self._track_unmanaged_batch_cost = track_unmanaged_batch_cost
self._track_unmanaged_vertex_batch_cost = track_unmanaged_vertex_batch_cost
# Cached after the first poll cycle. Once we know the column is absent we skip
# the guaranteed-failing primary query on every subsequent cycle.
self._has_batch_processed_column: bool = True
@ -118,11 +118,11 @@ class CheckBatchCost:
Resolve (model_id, batch_id) for a managed-object row, where model_id is a router
deployment id and batch_id is the raw provider batch id.
Managed batches encode both in a base64 unified id. Unmanaged batches (created outside
LiteLLM's own /v1/batches with a raw input_file_id) store the raw provider job id as
unified_object_id instead; when track_unmanaged_batch_cost is enabled the model is derived
from the provider-specific input_file_id layout (Vertex gs:// or Bedrock s3://) and mapped
to a matching deployment. Returns None (recording a metric) when the row can't be routed.
Managed batches encode both in a base64 unified id. Unmanaged Vertex batches, created with
a raw gs:// input_file_id, store the raw provider job id as unified_object_id; when
track_unmanaged_vertex_batch_cost is enabled the model is derived from the gs:// path and
mapped to a configured vertex_ai deployment. Returns None (recording a metric) when the row
can't be routed.
"""
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
@ -142,43 +142,8 @@ class CheckBatchCost:
return None
return model_id, get_batch_id_from_unified_batch_id(decoded)
if self._track_unmanaged_batch_cost:
from litellm.llms.bedrock.batches.transformation import (
BedrockBatchesConfig,
)
from litellm.llms.vertex_ai.batches.transformation import (
VertexAIBatchTransformation,
)
input_file_id = self._get_input_file_id(job)
if VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
input_file_id
):
assert input_file_id is not None # narrowed by is_unmanaged_gcs_batch_input_file_id
return self._resolve_unmanaged_provider_routing(
job=job,
prom_logger=prom_logger,
llm_provider="vertex_ai",
bare_model_name=VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
input_file_id
),
)
if BedrockBatchesConfig.is_unmanaged_s3_batch_input_file_id(input_file_id):
assert input_file_id is not None # narrowed by is_unmanaged_s3_batch_input_file_id
return self._resolve_unmanaged_provider_routing(
job=job,
prom_logger=prom_logger,
llm_provider="bedrock",
bare_model_name=BedrockBatchesConfig.get_bare_model_name_from_s3_file(
input_file_id
),
)
verbose_proxy_logger.info(
f"Skipping job {unified_object_id}: not a recognized unmanaged batch "
"(no gs:// or s3:// input_file_id with an embedded model)"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
if self._track_unmanaged_vertex_batch_cost:
return self._resolve_unmanaged_vertex_routing(job, prom_logger)
verbose_proxy_logger.info(
f"Skipping job {unified_object_id} because it is not a valid unified object id"
@ -186,17 +151,36 @@ class CheckBatchCost:
self._record_error(prom_logger, "invalid_unified_id")
return None
def _resolve_unmanaged_provider_routing(
def _resolve_unmanaged_vertex_routing(
self,
job: "LiteLLM_ManagedObjectTable",
prom_logger: Optional["PrometheusLogger"],
llm_provider: str,
bare_model_name: str,
) -> Optional[Tuple[str, str]]:
deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
from litellm.llms.vertex_ai.batches.transformation import (
VertexAIBatchTransformation,
)
input_file_id = self._get_input_file_id(job)
if not VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
input_file_id
):
verbose_proxy_logger.info(
f"Skipping job {job.unified_object_id}: not an unmanaged vertex batch "
"(no gs:// input_file_id with a publishers/ model path)"
)
self._record_error(prom_logger, "invalid_unified_id")
return None
assert input_file_id is not None # narrowed by is_unmanaged_gcs_batch_input_file_id
bare_model_name = VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
input_file_id
)
deployment_id = self._get_vertex_ai_deployment_id_for_bare_model(
bare_model_name
)
if deployment_id is None:
verbose_proxy_logger.info(
f"Skipping unmanaged {llm_provider} batch {job.unified_object_id}: no {llm_provider} "
f"Skipping unmanaged vertex batch {job.unified_object_id}: no vertex_ai "
f"deployment configured for model {bare_model_name}"
)
self._record_error(prom_logger, "unmanaged_no_matching_deployment")
@ -204,22 +188,22 @@ class CheckBatchCost:
return deployment_id, job.unified_object_id
def _get_deployment_id_for_bare_model(
self, bare_model_name: str, llm_provider: str
def _get_vertex_ai_deployment_id_for_bare_model(
self, bare_model_name: str
) -> Optional[str]:
model_group = self.llm_router.resolve_model_name_from_model_id(bare_model_name)
deployment_id = (
self._get_deployment_id_for_provider(model_group, llm_provider) if model_group else None
self._get_vertex_ai_deployment_id(model_group) if model_group else None
)
if deployment_id is not None:
return deployment_id
return self._get_deployment_id_from_matching_deployments(
bare_model_name, llm_provider
return self._get_vertex_ai_deployment_id_from_matching_deployments(
bare_model_name
)
def _get_deployment_id_from_matching_deployments(
self, bare_model_name: str, llm_provider: str
def _get_vertex_ai_deployment_id_from_matching_deployments(
self, bare_model_name: str
) -> Optional[str]:
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
@ -231,13 +215,13 @@ class CheckBatchCost:
if not self._is_bare_model_match(actual_model, bare_model_name):
continue
try:
_, deployment_llm_provider, _, _ = get_llm_provider(
_, llm_provider, _, _ = get_llm_provider(
model=actual_model,
custom_llm_provider=litellm_params.get("custom_llm_provider"),
)
except Exception:
continue
if deployment_llm_provider != llm_provider:
if llm_provider != "vertex_ai":
continue
model_info = deployment.get("model_info") or {}
deployment_id = model_info.get("id")
@ -247,21 +231,15 @@ class CheckBatchCost:
@staticmethod
def _is_bare_model_match(actual_model: str, bare_model_name: str) -> bool:
# Bedrock model ids may have ":" replaced with "-" in the S3 object key (see
# BedrockBatchesConfig.get_bare_model_name_from_s3_file), so normalize both sides;
# a no-op for providers like vertex_ai whose model ids never contain a colon.
normalized_actual = actual_model.replace(":", "-")
normalized_bare = bare_model_name.replace(":", "-")
return (
normalized_actual == normalized_bare
or normalized_actual.endswith(f"/{normalized_bare}")
actual_model == bare_model_name
or actual_model.endswith(f"/{bare_model_name}")
or actual_model.endswith(f":{bare_model_name}")
)
def _get_deployment_id_for_provider(
self, model_group: str, llm_provider: str
) -> Optional[str]:
def _get_vertex_ai_deployment_id(self, model_group: str) -> Optional[str]:
"""
Returns the first deployment id for `model_group` whose provider is `llm_provider`,
Returns the first deployment id for `model_group` whose provider is vertex_ai,
skipping deployments from other providers that happen to share the model group name.
"""
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
@ -271,13 +249,13 @@ class CheckBatchCost:
if deployment_info is None:
continue
try:
_, deployment_llm_provider, _, _ = get_llm_provider(
_, llm_provider, _, _ = get_llm_provider(
model=deployment_info.litellm_params.model,
custom_llm_provider=deployment_info.litellm_params.custom_llm_provider,
)
except Exception:
continue
if deployment_llm_provider == llm_provider:
if llm_provider == "vertex_ai":
return deployment_id
return None

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.50"
version = "0.1.49"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.50"
version = "0.1.49"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

7
helm/litellm-helm/templates/_helpers.tpl
View file

@ -76,13 +76,10 @@ so fall back to "default" (or an explicit override) to avoid a cyclic dependency
{{- end }}
{{/*
Get redis service name.
The bundled Redis subchart only serves sentinel in "replication" architecture
(it rejects standalone + sentinel outright), and in that mode the sentinel
Service is named "<release>-redis", not "<release>-redis-master".
Get redis service name
*/}}
{{- define "litellm.redis.serviceName" -}}
{{- if .Values.redis.sentinel.enabled -}}
{{- if and (eq .Values.redis.architecture "standalone") .Values.redis.sentinel.enabled -}}
{{- printf "%s-%s" .Release.Name (default "redis" .Values.redis.nameOverride | trunc 63 | trimSuffix "-") -}}
{{- else -}}
{{- printf "%s-%s-master" .Release.Name (default "redis" .Values.redis.nameOverride | trunc 63 | trimSuffix "-") -}}

15
helm/litellm-helm/templates/configmap-litellm.yaml
View file

@ -1,22 +1,9 @@
{{- if .Values.proxyConfigMap.create }}
{{- $config := deepCopy .Values.proxy_config }}
{{- if and .Values.redis.enabled (dig "coordination" "enabled" true .Values.redis) }}
{{- $generalSettings := (get $config "general_settings") | default dict }}
{{- if not (hasKey $generalSettings "coordination_redis") }}
{{- $coordinationRedis := dict "host" "os.environ/REDIS_HOST" "port" "os.environ/REDIS_PORT" "password" "os.environ/REDIS_PASSWORD" }}
{{- if .Values.redis.sentinel.enabled }}
{{- $sentinelNode := list (include "litellm.redis.serviceName" .) (include "litellm.redis.port" . | int) }}
{{- $coordinationRedis = dict "sentinel_nodes" (list $sentinelNode) "service_name" (default "mymaster" .Values.redis.sentinel.masterSet) "password" "os.environ/REDIS_PASSWORD" }}
{{- end }}
{{- $_ := set $generalSettings "coordination_redis" $coordinationRedis }}
{{- $_ := set $config "general_settings" $generalSettings }}
{{- end }}
{{- end }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "litellm.fullname" . }}-config
data:
config.yaml: |
{{ $config | toYaml | indent 6 }}
{{ .Values.proxy_config | toYaml | indent 6 }}
{{- end }}

143
helm/litellm-helm/tests/coordination_redis_tests.yaml
View file

@ -1,143 +0,0 @@
suite: test coordination redis
templates:
- configmap-litellm.yaml
- deployment.yaml
tests:
- it: should not render coordination_redis when redis is disabled
template: configmap-litellm.yaml
set:
redis.enabled: false
asserts:
- notMatchRegex:
path: data["config.yaml"]
pattern: coordination_redis
- it: should not emit redis env vars when redis is disabled
template: deployment.yaml
set:
redis.enabled: false
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: RELEASE-NAME-redis-master
any: true
- it: should render coordination_redis pointing at the bundled redis when enabled
template: configmap-litellm.yaml
set:
redis.enabled: true
asserts:
- matchRegex:
path: data["config.yaml"]
pattern: "coordination_redis:\n host: os.environ/REDIS_HOST\n password: os.environ/REDIS_PASSWORD\n port: os.environ/REDIS_PORT\n"
- matchRegex:
path: data["config.yaml"]
pattern: "master_key: os.environ/PROXY_MASTER_KEY"
- it: should emit redis env vars backing the coordination_redis os.environ refs
template: deployment.yaml
set:
redis.enabled: true
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: RELEASE-NAME-redis-master
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PORT
value: "6379"
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: RELEASE-NAME-redis
key: redis-password
- it: should not render coordination_redis when coordination is opted out
template: configmap-litellm.yaml
set:
redis.enabled: true
redis.coordination.enabled: false
asserts:
- notMatchRegex:
path: data["config.yaml"]
pattern: coordination_redis
- it: should keep emitting redis env vars when coordination is opted out
template: deployment.yaml
set:
redis.enabled: true
redis.coordination.enabled: false
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: RELEASE-NAME-redis-master
- it: should not clobber a user supplied coordination_redis block
template: configmap-litellm.yaml
set:
redis.enabled: true
proxy_config.general_settings.coordination_redis:
url: os.environ/COORDINATION_REDIS_URL
asserts:
- matchRegex:
path: data["config.yaml"]
pattern: "coordination_redis:\n url: os.environ/COORDINATION_REDIS_URL\n"
- notMatchRegex:
path: data["config.yaml"]
pattern: "host: os.environ/REDIS_HOST"
- it: should render sentinel_nodes and service_name in sentinel mode
template: configmap-litellm.yaml
set:
redis.enabled: true
redis.architecture: replication
redis.sentinel.enabled: true
asserts:
# The sentinel Service the redis subchart renders is "<release>-redis", and a
# plain client cannot speak the sentinel protocol, so host/port must not appear
- matchRegex:
path: data["config.yaml"]
pattern: "coordination_redis:\n password: os.environ/REDIS_PASSWORD\n sentinel_nodes:\n - - RELEASE-NAME-redis\n - 26379\n service_name: mymaster\n"
- notMatchRegex:
path: data["config.yaml"]
pattern: "host: os.environ/REDIS_HOST"
- it: should carry a custom sentinel masterSet into service_name
template: configmap-litellm.yaml
set:
redis.enabled: true
redis.architecture: replication
redis.sentinel.enabled: true
redis.sentinel.masterSet: litellm-master
asserts:
- matchRegex:
path: data["config.yaml"]
pattern: "service_name: litellm-master"
- it: should point REDIS_HOST at the sentinel service in sentinel mode
template: deployment.yaml
set:
redis.enabled: true
redis.architecture: replication
redis.sentinel.enabled: true
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: RELEASE-NAME-redis
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PORT
value: "26379"

22
helm/litellm-helm/values.yaml
View file

@ -331,28 +331,12 @@ postgresql:
# secretKeys:
# userPasswordKey: password
# Redis is the proxy's coordination store: cross-pod tpm/rpm rate limits, spend
# tracking, and the pod lock manager. Enabling this deploys the bundled Redis
# subchart, wires REDIS_HOST / REDIS_PORT / REDIS_PASSWORD into the proxy, and
# renders a `general_settings.coordination_redis` block into the proxy config.
#
# To point at an existing Redis instead, leave `enabled: false` and pass a
# secret for REDIS_HOST, REDIS_PORT, REDIS_PASSWORD or REDIS_URL; the proxy
# falls back to those env vars for coordination. Set `cache: true` in the proxy
# config only if you also want LLM response caching, which is independent of
# coordination
#
# When `redis.sentinel.enabled` is set, the coordination block is rendered with
# `sentinel_nodes` and `service_name` (from `redis.sentinel.masterSet`) instead
# of host/port, because a plain Redis client cannot talk to the sentinel port
# requires cache: true in config file
# either enable this or pass a secret for REDIS_HOST, REDIS_PORT, REDIS_PASSWORD or REDIS_URL
# with cache: true to use existing redis instance
redis:
enabled: false
architecture: standalone
coordination:
# Set to false to keep the bundled Redis for response caching only and leave
# `general_settings.coordination_redis` out of the rendered config. A
# `coordination_redis` block you define yourself in `proxy_config` always wins
enabled: true
# Prisma migration job settings
migrationJob:

13
helm/litellm/templates/_helpers.tpl
View file

@ -213,10 +213,6 @@ harmless no-op for the Job and authoritative for the app pods.
*/}}
- name: DISABLE_SCHEMA_UPDATE
value: "true"
{{/* These feed the proxy's coordination Redis (cross-pod rate limits, spend
tracking, pod lock manager) via its REDIS_* env fallback. An explicit
`general_settings.coordination_redis` block in proxy_config takes
precedence over anything emitted here. */}}
{{- if $root.Values.redis.host }}
- name: REDIS_HOST
value: {{ $root.Values.redis.host | quote }}
@ -230,11 +226,10 @@ harmless no-op for the Job and authoritative for the app pods.
key: {{ $root.Values.redis.passwordSecret.passwordKey | default "password" }}
{{- end }}
{{- if $root.Values.redis.cluster }}
{{/* The proxy falls back to REDIS_CLUSTER_NODES (JSON) to build a cluster-mode
coordination client when `general_settings.coordination_redis` is absent
and no plain-Redis response cache is configured. We seed with the single
configured endpoint; the cluster client discovers the remaining nodes from
CLUSTER SLOTS at startup. */}}
{{/* The proxy's Cache() reads REDIS_CLUSTER_NODES as JSON and constructs a
RedisClusterCache when it's set (litellm/caching/caching.py:169-192).
We seed with the single configured endpoint — the cluster client
discovers the remaining nodes from CLUSTER SLOTS at startup. */}}
- name: REDIS_CLUSTER_NODES
value: {{ printf "[{\"host\":%q,\"port\":%v}]" $root.Values.redis.host (int $root.Values.redis.port) | quote }}
{{- end }}

109
helm/litellm/tests/redis_env_tests.yaml
View file

@ -1,109 +0,0 @@
suite: test redis coordination env vars
templates:
- gateway/deployment.yaml
- gateway/configmap.yaml
- backend/deployment.yaml
values:
- ./values/required.yaml
tests:
- it: gateway omits redis env vars when no host is configured
template: gateway/deployment.yaml
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: redis.example.com
any: true
- notContains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_CLUSTER_NODES
any: true
- it: gateway emits host, port and password when redis is configured
template: gateway/deployment.yaml
set:
redis.host: redis.example.com
redis.port: 6380
redis.passwordSecret.name: redis-secret
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: redis.example.com
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PORT
value: "6380"
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-secret
key: password
- it: backend emits the same redis env vars so both pods coordinate on one redis
template: backend/deployment.yaml
set:
redis.host: redis.example.com
redis.passwordSecret.name: redis-secret
redis.passwordSecret.passwordKey: redis-password
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: redis.example.com
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-secret
key: redis-password
- it: gateway omits REDIS_PASSWORD for an auth-less redis
template: gateway/deployment.yaml
set:
redis.host: redis.example.com
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_PASSWORD
any: true
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_HOST
value: redis.example.com
- it: gateway seeds REDIS_CLUSTER_NODES from host and port in cluster mode
template: gateway/deployment.yaml
set:
redis.host: redis.example.com
redis.port: 6380
redis.cluster: true
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_CLUSTER_NODES
value: '[{"host":"redis.example.com","port":6380}]'
- it: gateway omits REDIS_CLUSTER_NODES when cluster mode is off
template: gateway/deployment.yaml
set:
redis.host: redis.example.com
asserts:
- notContains:
path: spec.template.spec.containers[0].env
content:
name: REDIS_CLUSTER_NODES
any: true

13
helm/litellm/values.yaml
View file

@ -100,18 +100,7 @@ database:
usernameKey: username
passwordKey: password
# Optional Redis. Leave host empty to disable.
#
# This is the proxy's coordination store: cross-pod tpm/rpm rate limits, spend
# tracking, and the pod lock manager. The chart emits REDIS_HOST / REDIS_PORT /
# REDIS_PASSWORD, which the proxy picks up through its coordination Redis env
# fallback. Response caching is separate and off unless you enable it in
# `proxy_config.litellm_settings.cache`.
#
# For full control, define `general_settings.coordination_redis` in
# `proxy_config` (host/port/password/username/url/ssl/startup_nodes/
# sentinel_nodes/sentinel_password/service_name, each accepting os.environ/VAR
# refs). An explicit block overrides these env vars.
# Optional Redis (caching, rate limiting). Leave host empty to disable.
#
# Set `cluster: true` for Redis Cluster mode (e.g. AWS ElastiCache Cluster,
# self-hosted Redis Cluster). The chart emits REDIS_CLUSTER_NODES from

2
litellm-proxy-extras/litellm_proxy_extras/migrations/20260710000000_add_dcr_bridge_to_mcp_server_table/migration.sql
View file

@ -1,2 +0,0 @@
-- AlterTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "dcr_bridge" BOOLEAN;

6
litellm-proxy-extras/litellm_proxy_extras/migrations/20260713230852_add_key_type_to_litellm_verification_token/migration.sql
View file

@ -1,6 +0,0 @@
-- AlterTable
ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN "key_type" TEXT;
-- AlterTable
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN "key_type" TEXT;

3
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -339,7 +339,6 @@ model LiteLLM_MCPServerTable {
available_on_public_internet Boolean @default(true)
delegate_auth_to_upstream Boolean @default(false)
oauth_passthrough Boolean @default(false)
dcr_bridge Boolean?
is_byok Boolean @default(false)
byok_description String[] @default([])
byok_api_key_help_url String?
@ -422,7 +421,6 @@ model LiteLLM_VerificationToken {
budget_reset_at DateTime?
allowed_cache_controls String[] @default([])
allowed_routes String[] @default([])
key_type String?
policies String[] @default([])
access_group_ids String[] @default([])
model_spend Json @default("{}")
@ -517,7 +515,6 @@ model LiteLLM_DeletedVerificationToken {
budget_reset_at DateTime?
allowed_cache_controls String[] @default([])
allowed_routes String[] @default([])
key_type String?
policies String[] @default([])
access_group_ids String[] @default([])
model_spend Json @default("{}")

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.77"
version = "0.4.75"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.77"
version = "0.4.75"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

16
litellm/_redis.py
View file

@ -325,19 +325,8 @@ def _get_redis_client_logic(**env_overrides):
value = get_secret(v) # type: ignore
env_overrides[k] = value
environment_kwargs = _redis_kwargs_from_environment()
# An explicitly configured connection target outranks REDIS_URL from the
# environment. Without this, the url branch below strips the caller's
# host/port/password and silently connects to whatever REDIS_URL names.
caller_named_a_target = any(
env_overrides.get(key) is not None for key in ("host", "startup_nodes", "sentinel_nodes")
)
if caller_named_a_target and env_overrides.get("url") is None:
environment_kwargs.pop("url", None)
redis_kwargs = {
**environment_kwargs,
**_redis_kwargs_from_environment(),
**env_overrides,
}
@ -689,8 +678,9 @@ def get_redis_connection_pool(
redis_kwargs["credential_provider"] = GCPIAMCredentialProvider(redis_connect_func._gcp_service_account)
connection_class = async_redis.Connection
if redis_kwargs.pop("ssl", False):
if "ssl" in redis_kwargs:
connection_class = async_redis.SSLConnection
redis_kwargs.pop("ssl", None)
redis_kwargs["connection_class"] = connection_class
return async_redis.BlockingConnectionPool(timeout=REDIS_CONNECTION_POOL_TIMEOUT, **redis_kwargs)

21
litellm/batches/batch_utils.py
View file

@ -118,7 +118,6 @@ def _batch_cost_calculator(
total_cost = _get_batch_job_cost_from_file_content(
file_content_dictionary=file_content_dictionary,
custom_llm_provider=custom_llm_provider,
model_name=model_name,
model_info=model_info,
)
verbose_logger.debug("total_cost=%s", total_cost)
@ -364,7 +363,6 @@ def _count_entry_tokens(
def _get_batch_job_cost_from_file_content(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic"] = "openai",
model_name: Optional[str] = None,
model_info: Optional[ModelInfo] = None,
) -> float:
"""
@ -379,15 +377,9 @@ def _get_batch_job_cost_from_file_content(
for _item in file_content_dictionary:
if _batch_response_was_successful(_item, custom_llm_provider):
_response_body = _get_response_from_batch_job_output_file(_item, custom_llm_provider)
if model_info is not None or custom_llm_provider in ("anthropic", "bedrock"):
if model_info is not None or custom_llm_provider == "anthropic":
usage = _get_batch_job_usage_from_response_body(_response_body, custom_llm_provider)
# Bedrock batch output lines report a short internal model id
# (e.g. "claude-sonnet-4-6") that is not in the cost map; use the
# deployment model name for pricing when available.
if custom_llm_provider == "bedrock" and model_name:
model = model_name
else:
model = _response_body.get("model") or model_name or ""
model = _response_body.get("model", "")
prompt_cost, completion_cost = batch_cost_calculator(
usage=usage,
model=model,
@ -493,7 +485,7 @@ def _get_batch_job_usage_from_response_body(response_body: dict, custom_llm_prov
"""
Get the tokens of a batch job from the response body
"""
if custom_llm_provider in ("anthropic", "bedrock"):
if custom_llm_provider == "anthropic":
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
return AnthropicConfig().calculate_usage(
@ -521,8 +513,6 @@ def _get_response_from_batch_job_output_file(batch_job_output_file: dict, custom
"""
if custom_llm_provider == "anthropic":
return _get_anthropic_result_from_batch_results_line(batch_job_output_file).get("message", None) or {}
if custom_llm_provider == "bedrock":
return batch_job_output_file.get("modelOutput", None) or {}
_response: dict = batch_job_output_file.get("response", None) or {}
_response_body = _response.get("body", None) or {}
return _response_body
@ -533,12 +523,9 @@ def _batch_response_was_successful(batch_job_output_file: dict, custom_llm_provi
Check if the batch job response was successful
OpenAI-shaped output rows report ``response.status_code == 200``; Anthropic
message batch results lines report ``result.type == "succeeded"``; Bedrock
batch output lines report ``modelOutput`` (and no ``error``).
message batch results lines report ``result.type == "succeeded"``.
"""
if custom_llm_provider == "anthropic":
return _get_anthropic_result_from_batch_results_line(batch_job_output_file).get("type") == "succeeded"
if custom_llm_provider == "bedrock":
return batch_job_output_file.get("modelOutput") is not None and batch_job_output_file.get("error") is None
_response: dict = batch_job_output_file.get("response", None) or {}
return _response.get("status_code", None) == 200

2
litellm/constants.py
View file

@ -715,7 +715,6 @@ openai_compatible_endpoints: List = [
"https://api.clarifai.com/v2/ext/openai/v1",
"https://api.libertai.io/v1",
"https://pinstripes.io/v1",
"https://api.meta.ai/v1",
]
@ -782,7 +781,6 @@ openai_compatible_providers: List = [
"ragflow",
"pinstripes", # Pinstripes - JSON-configured provider
"darkbloom",
"meta", # Meta Model API (Muse Spark) - JSON-configured provider
]
openai_text_completion_compatible_providers: List = [ # providers that support `/v1/completions`
"together_ai",

6
litellm/cost_calculator.py
View file

@ -760,11 +760,7 @@ def _select_model_name_for_cost_calc(
if custom_pricing is True:
if router_model_id is not None and router_model_id in litellm.model_cost:
entry = litellm.model_cost[router_model_id]
if (
entry.get("input_cost_per_token") is not None
or entry.get("input_cost_per_second") is not None
or entry.get("tiered_pricing") is not None
):
if entry.get("input_cost_per_token") is not None or entry.get("input_cost_per_second") is not None:
return_model = router_model_id
else:
return_model = model

41
litellm/experimental_mcp_client/client.py
View file

@ -382,25 +382,15 @@ class MCPClient:
if root_cause is not None and isinstance(in_flight_error, asyncio.CancelledError):
raise root_cause from in_flight_error
async def run_with_session(
self,
operation: Callable[[ClientSession], Awaitable[TSessionResult]],
*,
quiet_on_error: bool = False,
) -> TSessionResult:
"""Open a session, run the provided coroutine, and clean up.
quiet_on_error demotes the failure line to debug for callers that own the exception
(call_tool / list_tools under raise_on_error), so an expected pass-through re-auth does
not emit a warning per call; every other caller keeps the operator-visible warning."""
async def run_with_session(self, operation: Callable[[ClientSession], Awaitable[TSessionResult]]) -> TSessionResult:
"""Open a session, run the provided coroutine, and clean up."""
http_client: Optional[httpx.AsyncClient] = None
try:
self._last_initialize_instructions = None
transport_ctx, http_client = self._create_transport_context()
return await self._execute_session_operation(transport_ctx, operation)
except Exception:
_log = verbose_logger.debug if quiet_on_error else verbose_logger.warning
_log("MCP client run_with_session failed for %s", self.server_url or "stdio")
verbose_logger.warning("MCP client run_with_session failed for %s", self.server_url or "stdio")
raise
finally:
if http_client is not None:
@ -501,7 +491,7 @@ class MCPClient:
return await session.list_tools()
try:
result = await self.run_with_session(_list_tools_operation, quiet_on_error=raise_on_error)
result = await self.run_with_session(_list_tools_operation)
tool_count = len(result.tools)
tool_names = [tool.name for tool in result.tools]
verbose_logger.info(f"MCP client listed {tool_count} tools from {self.server_url or 'stdio'}: {tool_names}")
@ -511,13 +501,7 @@ class MCPClient:
raise
except Exception as e:
error_type = type(e).__name__
# Mirror call_tool: when the caller opted into raise_on_error it owns the exception and
# logs it at the fitting level (an expected pass-through re-auth 401 is info, not an
# error), so log at debug here to avoid an error-level line + traceback that would trip
# error-rate alerts on that expected signal. The swallow path still logs the full
# exception because nothing downstream will surface the failure.
_log = verbose_logger.debug if raise_on_error else verbose_logger.exception
_log(
verbose_logger.exception(
f"MCP client list_tools failed - "
f"Error Type: {error_type}, "
f"Error: {str(e)}, "
@ -526,8 +510,7 @@ class MCPClient:
)
# Check if it's a stream/connection error
if "BrokenResourceError" in error_type or "Broken" in error_type:
_log_broken = verbose_logger.debug if raise_on_error else verbose_logger.error
_log_broken(
verbose_logger.error(
"MCP client detected broken connection/stream during list_tools - "
"the MCP server may have crashed, disconnected, or timed out"
)
@ -584,7 +567,7 @@ class MCPClient:
)
try:
tool_result = await self.run_with_session(_call_tool_operation, quiet_on_error=raise_on_error)
tool_result = await self.run_with_session(_call_tool_operation)
verbose_logger.info(f"MCP client tool call '{call_tool_request_params.name}' completed successfully")
return tool_result
except asyncio.CancelledError:
@ -597,13 +580,7 @@ class MCPClient:
verbose_logger.debug(f"MCP client tool call traceback:\n{error_trace}")
# Log detailed error information
error_type = type(e).__name__
# When the caller opted into raise_on_error it owns the exception and logs it at the
# level that fits (an expected pass-through re-auth 401 is info, not an operator-actionable
# error), so log at debug here to avoid an error-level line that would trip error-rate
# alerts on that expected signal. The swallow path (raise_on_error=False) still logs at
# error because nothing downstream will surface the failure.
_log = verbose_logger.debug if raise_on_error else verbose_logger.error
_log(
verbose_logger.error(
f"MCP client call_tool failed - "
f"Error Type: {error_type}, "
f"Error: {str(e)}, "
@ -613,7 +590,7 @@ class MCPClient:
)
# Check if it's a stream/connection error
if "BrokenResourceError" in error_type or "Broken" in error_type:
_log(
verbose_logger.error(
"MCP client detected broken connection/stream - "
"the MCP server may have crashed, disconnected, or timed out."
)

69
litellm/integrations/custom_guardrail.py
View file

@ -1,4 +1,3 @@
import os
import secrets
from datetime import datetime
from typing import (
@ -18,7 +17,6 @@ from litellm._logging import verbose_logger
from litellm.litellm_core_utils.core_helpers import redact_nested_match_and_regex_keys
from litellm.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.secret_managers.main import str_to_bool
from litellm.types.guardrails import (
DynamicGuardrailParams,
GuardrailEventHooks,
@ -61,20 +59,6 @@ from litellm.exceptions import (
_PRE_CALL_EXECUTED_TOKEN = secrets.token_hex(16)
def _strict_guardrail_modes_enabled() -> bool:
"""Whether guardrail-mode validation raises (default) or logs a warning.
Set `LITELLM_STRICT_GUARDRAIL_MODES=false` to keep the pre-LIT-4226 behavior
for guardrails whose supported_event_hooks list newly includes their
configured mode: log the mismatch and continue instead of raising at boot.
"""
raw = os.environ.get("LITELLM_STRICT_GUARDRAIL_MODES")
if raw is None:
return True
parsed = str_to_bool(raw)
return True if parsed is None else parsed
def get_session_id_from_request_data(request_data: Dict[str, Any]) -> Optional[str]:
"""Extract session_id from request data (litellm_session_id or metadata)."""
session_id = request_data.get("litellm_session_id")
@ -148,17 +132,7 @@ class CustomGuardrail(CustomLogger):
if supported_event_hooks:
## validate event_hook is in supported_event_hooks
try:
self._validate_event_hook(event_hook, supported_event_hooks)
except ValueError as validation_error:
if _strict_guardrail_modes_enabled():
raise
verbose_logger.warning(
"%s. LITELLM_STRICT_GUARDRAIL_MODES=false; continuing "
"with unsupported event_hook. Set the env var to true "
"(default) to enforce validation and fail at startup.",
validation_error,
)
self._validate_event_hook(event_hook, supported_event_hooks)
super().__init__(**kwargs)
def render_violation_message(self, default: str, context: Optional[Dict[str, Any]] = None) -> str:
@ -329,18 +303,6 @@ class CustomGuardrail(CustomLogger):
"""
return None
@classmethod
def get_supported_event_hooks(cls) -> Optional[List[GuardrailEventHooks]]:
"""
Returns the event hooks this guardrail supports, for the UI to render.
Subclasses should override to return their supported hooks list. When a
subclass returns None, the endpoint omits it from the per-provider map
and the UI is expected to fall back to the global `supported_modes`
list client-side.
"""
return None
def _validate_event_hook(
self,
event_hook: Optional[Union[GuardrailEventHooks, List[GuardrailEventHooks], Mode]],
@ -515,22 +477,6 @@ class CustomGuardrail(CustomLogger):
return True
return False
def uses_apply_guardrail_interface(self) -> bool:
return type(self).apply_guardrail is not CustomGuardrail.apply_guardrail
def _deployment_pre_call_target(self) -> "CustomLogger":
if not self.uses_apply_guardrail_interface():
return self
try:
from litellm.proxy.utils import unified_guardrail
except ImportError as e:
raise ImportError(
f"Guardrail {self.guardrail_name or type(self).__name__} implements apply_guardrail, which needs "
"the litellm proxy dependencies to run at the deployment level. "
"Install them with: pip install 'litellm[proxy]'"
) from e
return unified_guardrail
async def async_pre_call_deployment_hook(
self, kwargs: Dict[str, Any], call_type: Optional[CallTypes]
) -> Optional[dict]:
@ -549,10 +495,7 @@ class CustomGuardrail(CustomLogger):
# CHECK IF GUARDRAIL REJECTS THE REQUEST
if call_type == CallTypes.completion or call_type == CallTypes.acompletion:
target = self._deployment_pre_call_target()
if target is not self:
kwargs["guardrail_to_apply"] = self
result = await target.async_pre_call_hook(
result = await self.async_pre_call_hook(
user_api_key_dict=UserAPIKeyAuth(
user_id=kwargs.get("user_api_key_user_id"),
team_id=kwargs.get("user_api_key_team_id"),
@ -562,7 +505,7 @@ class CustomGuardrail(CustomLogger):
),
cache=dc,
data=kwargs,
call_type="completion" if call_type == CallTypes.completion else "acompletion",
call_type=call_type.value or "acompletion", # type: ignore
)
if result is not None and isinstance(result, dict):
@ -814,12 +757,6 @@ class CustomGuardrail(CustomLogger):
# raw provider JSON so redaction is not duplicated upstream).
clean_guardrail_response = redact_nested_match_and_regex_keys(clean_guardrail_response)
from litellm.litellm_core_utils.sensitive_data_masker import (
mask_credentials_in_payload,
)
clean_guardrail_response = mask_credentials_in_payload(clean_guardrail_response)
slg = StandardLoggingGuardrailInformation(
guardrail_name=self.guardrail_name,
guardrail_provider=guardrail_provider,

29
litellm/integrations/datadog/datadog.py
View file

@ -19,7 +19,7 @@ import os
import time
import traceback
from datetime import datetime as datetimeObj
from typing import Any, Dict, List, Optional, Sequence, Union
from typing import Any, Dict, List, Optional, Union
import httpx
from httpx import Response
@ -50,7 +50,6 @@ from litellm.types.integrations.base_health_check import IntegrationHealthCheckS
from litellm.types.integrations.datadog import (
DD_ERRORS,
DD_MAX_BATCH_SIZE,
DD_MAX_PAYLOAD_SIZE_BYTES,
DataDogStatus,
DatadogInitParams,
DatadogPayload,
@ -385,10 +384,8 @@ class DataDogLogger(
async def _send_with_413_split(self, batch: List) -> List:
"""
Send a batch, halving any sub-batch that exceeds Datadog's intake limits before
sending, and halving again on a 413 (payload too large) response, since Datadog
enforces a 5MB uncompressed limit per request. The proactive split avoids paying
a serialize + gzip + round trip for a payload the intake is guaranteed to reject.
Send a batch, halving any sub-batch that 413s (payload too large) and retrying the
halves, since Datadog enforces a 5MB uncompressed limit per request.
A 413 surfaces as a raised MaskedHTTPStatusError (httpx raise_for_status), not a
returned response, so both paths are handled. A lone event that still 413s is
@ -401,11 +398,6 @@ class DataDogLogger(
chunk = pending.pop()
if not chunk:
continue
if len(chunk) > 1 and self._exceeds_intake_limits(chunk):
mid = len(chunk) // 2
pending.append(chunk[mid:])
pending.append(chunk[:mid])
continue
try:
response = await self.async_send_compressed_data(chunk)
except Exception as e:
@ -444,21 +436,6 @@ class DataDogLogger(
def _undelivered(chunk: List, pending: List[List]) -> List:
return chunk + [event for remaining in reversed(pending) for event in remaining]
@staticmethod
def _exceeds_intake_limits(chunk: Sequence[DatadogPayload]) -> bool:
"""
True when a chunk would breach Datadog's log intake limits: more than
DD_MAX_BATCH_SIZE events per payload, or a serialized size above
DD_MAX_PAYLOAD_SIZE_BYTES (held under Datadog's 5MB uncompressed cap so
the batch is split before the intake rejects it with a 413).
"""
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
if len(chunk) > DD_MAX_BATCH_SIZE:
return True
payload_size_bytes = len(safe_dumps(chunk).encode("utf-8"))
return payload_size_bytes > DD_MAX_PAYLOAD_SIZE_BYTES
async def flush_queue(self):
if self.flush_lock is None:
return

9
litellm/integrations/otel/README.md
View file

@ -223,15 +223,6 @@ lives in [`plumbing/`](./plumbing):
readers/exporters receive them alongside the server metrics, and one is built
and registered as the global only when none is set (mirroring how V2 owns trace
export).
- [`events.py`](./plumbing/events.py) — GenAI client events. Gated on
`enable_events` (`LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS`), a failed LLM call
records the semconv `gen_ai.client.operation.exception` log event at severity
WARN, carrying `exception.type` / `exception.message` / `exception.stacktrace`
and correlated to the failed span through the trace and span ids. The
`LoggerProvider` is resolved like the meter provider, except that an explicit
`NoOpLoggerProvider` global is an operator opt-out that builds no recorder at
all. The deprecated `error.*` span attributes and the `exception` span event
are still stamped by the emitter for backwards compatibility.
### Adapter

11
litellm/integrations/otel/emitter.py
View file

@ -18,7 +18,6 @@ from litellm.integrations.otel.model.payloads import (
ServiceSpanData,
SpanError,
)
from litellm.integrations.otel.plumbing.events import GenAIEventRecorder
from litellm.integrations.otel.plumbing.providers import to_otel_span_kind
from litellm.integrations.otel.model.semconv import Error, ExceptionEvent, LiteLLMError
from litellm.integrations.otel.model.spans import (
@ -78,11 +77,9 @@ class SpanEmitter:
tracer: Tracer,
config: OpenTelemetryV2Config,
mappers: Sequence[AttributeMapper] | None = None,
event_recorder: GenAIEventRecorder | None = None,
) -> None:
self._tracer = tracer
self._config = config
self._event_recorder = event_recorder
# The mapper chain is the sole source of span attributes. When not
# passed in, resolve it from the config so there's one source of truth.
self._mappers: list[AttributeMapper] = (
@ -226,14 +223,6 @@ class SpanEmitter:
ExceptionEvent.NAME,
{ExceptionEvent.TYPE: error_type, ExceptionEvent.MESSAGE: message},
)
if self._event_recorder is not None and role is SpanRole.LLM_CALL:
self._event_recorder.record_operation_exception(
span_context=span.get_span_context(),
error_type=error_type,
message=message,
stack_trace=error.stack_trace,
timestamp_ns=end_time_ns,
)
# On success leave the status UNSET (the semconv default) rather than
# forcing OK — that matches the FastAPI server span and avoids implying a
# span-level health signal litellm doesn't actually evaluate. Only a

29
litellm/integrations/otel/logger.py
View file

@ -6,7 +6,6 @@ from datetime import datetime
from typing import TYPE_CHECKING, Any, Callable, Iterator, Mapping, Sequence, cast
from opentelemetry.context import Context, attach, get_current
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import Span, Tracer, get_current_span, use_span
@ -41,17 +40,14 @@ from litellm.integrations.otel.model.payloads import (
is_mcp_list_tools,
is_mcp_tool_call,
)
from litellm.integrations.otel.plumbing.events import GenAIEventRecorder
from litellm.integrations.otel.plumbing.metrics import (
GenAIMetricRecorder,
create_genai_metrics,
)
from litellm.integrations.otel.plumbing.providers import (
build_tracer_provider,
get_event_logger,
get_meter,
get_tracer,
resolve_logger_provider,
resolve_meter_provider,
)
from litellm.integrations.otel.plumbing.routing import TenantTracerCache
@ -108,7 +104,7 @@ class OpenTelemetryV2(CustomLogger):
config: OpenTelemetryV2Config | None = None,
callback_name: str | None = None,
tracer_provider: TracerProvider | None = None,
logger_provider: LoggerProvider | None = None,
logger_provider: Any | None = None, # reserved for OTel logs
meter_provider: Any | None = None,
**kwargs: Any,
) -> None:
@ -121,12 +117,7 @@ class OpenTelemetryV2(CustomLogger):
self.tracer: Tracer = get_tracer(self._tracer_provider, LITELLM_TRACER_NAME)
self._metrics_recorder = self._init_metrics(meter_provider)
self._metric_filter_error_logged = False
self._emitter = SpanEmitter(
self.tracer,
self.config,
mappers=resolve_mappers(self.config.mapper_names),
event_recorder=self._init_events(logger_provider),
)
self._emitter = SpanEmitter(self.tracer, self.config, mappers=resolve_mappers(self.config.mapper_names))
self._tenant_tracers = TenantTracerCache(self.config, callback_name, LITELLM_TRACER_NAME)
self._open_llm_calls: "OrderedDict[str, _LLMCallSpan]" = OrderedDict()
self._init_otel_logger_on_litellm_proxy()
@ -145,22 +136,6 @@ class OpenTelemetryV2(CustomLogger):
meter = get_meter(provider, LITELLM_TRACER_NAME)
return GenAIMetricRecorder(create_genai_metrics(meter), self.callback_name)
def _init_events(self, logger_provider: LoggerProvider | None) -> "GenAIEventRecorder | None":
"""Create the GenAI event recorder when events are enabled, else ``None``.
``logger_provider`` is an explicit override (tests inject one); otherwise the
provider is resolved from the OTel global so an operator-configured logs
pipeline receives the events, building and registering one only when no
global provider is set. A ``None`` resolution means the operator opted out
of the logs signal, so no recorder is built.
"""
if not self.config.enable_events:
return None
provider = resolve_logger_provider(self.config, logger_provider)
if provider is None:
return None
return GenAIEventRecorder(get_event_logger(provider, LITELLM_TRACER_NAME))
# ====================================================================== #
# Proxy global registration
# ====================================================================== #

13
litellm/integrations/otel/model/semconv.py
View file

@ -177,19 +177,6 @@ class ExceptionEvent:
NAME: Final = "exception"
TYPE: Final = "exception.type"
MESSAGE: Final = "exception.message"
STACKTRACE: Final = "exception.stacktrace"
class GenAIEvent:
"""GenAI semconv event names, from the GenAI registry's *events* section.
``gen_ai.client.operation.exception`` is defined as a log-based event
(severity WARN) carrying the ``exception.*`` trio, correlated to the failed
span via the trace/span ids — the semconv-compliant home for GenAI failure
details, unlike the deprecated ``error.message`` span attribute.
"""
OPERATION_EXCEPTION: Final = "gen_ai.client.operation.exception"
class Server:

52
litellm/integrations/otel/plumbing/events.py
View file

@ -1,52 +0,0 @@
"""GenAI client events: the ``gen_ai.client.operation.exception`` log event.
The GenAI semantic conventions define exception recording for client
operations as a log-based event (severity WARN) carrying the ``exception.*``
attribute trio, correlated to the failed span through the trace/span ids —
not as a span attribute or span event. This module owns building and
emitting that event; the exporter pipeline it rides is built in
:mod:`litellm.integrations.otel.plumbing.providers`.
"""
from dataclasses import dataclass
from opentelemetry._events import Event, EventLogger
from opentelemetry._logs.severity import SeverityNumber
from opentelemetry.trace import SpanContext
from litellm.integrations.otel.model.semconv import ExceptionEvent, GenAIEvent
@dataclass(frozen=True, slots=True)
class GenAIEventRecorder:
event_logger: EventLogger
def record_operation_exception(
self,
span_context: SpanContext,
error_type: str,
message: str,
stack_trace: str | None,
timestamp_ns: int | None,
) -> None:
# ``exception.type`` and ``exception.message`` are the semconv-required
# pair and always ride the event; only the recommended stacktrace is
# conditional on the payload carrying one.
stacktrace = ((ExceptionEvent.STACKTRACE, stack_trace),) if stack_trace else ()
self.event_logger.emit(
Event(
name=GenAIEvent.OPERATION_EXCEPTION,
timestamp=timestamp_ns,
trace_id=span_context.trace_id,
span_id=span_context.span_id,
trace_flags=span_context.trace_flags,
severity_number=SeverityNumber.WARN,
attributes=dict(
(
(ExceptionEvent.TYPE, error_type),
(ExceptionEvent.MESSAGE, message),
*stacktrace,
)
),
)
)

119
litellm/integrations/otel/plumbing/providers.py
View file

@ -2,20 +2,9 @@
from typing import TYPE_CHECKING, Any, Callable, Iterable
from opentelemetry import _logs, baggage, metrics
from opentelemetry._events import EventLogger
from opentelemetry._logs import LoggerProvider, NoOpLoggerProvider
from opentelemetry import baggage, metrics
from opentelemetry.context import Context
from opentelemetry.metrics import MeterProvider, NoOpMeterProvider
from opentelemetry.sdk._events import EventLoggerProvider
from opentelemetry.sdk._logs import LoggerProvider as SDKLoggerProvider
from opentelemetry.sdk._logs.export import (
BatchLogRecordProcessor,
ConsoleLogExporter,
InMemoryLogExporter,
LogExporter,
SimpleLogRecordProcessor,
)
from opentelemetry.sdk.metrics import MeterProvider as SDKMeterProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import ReadableSpan, SpanProcessor, TracerProvider
@ -235,112 +224,6 @@ def build_metric_reader(config: OpenTelemetryV2Config) -> "MetricReader":
return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
def _otlp_logs_endpoint(endpoint: str | None) -> str | None:
"""Point an OTLP/HTTP base endpoint at the ``/v1/logs`` signal path.
The OTLP/HTTP exporter only appends ``/v1/logs`` when it reads
``OTEL_EXPORTER_OTLP_ENDPOINT`` itself; an explicitly passed endpoint is used
verbatim, so a base URL would POST to the root. Mirror ``_otlp_traces_endpoint``
for the logs signal (rewriting a sibling signal path when present).
"""
if not endpoint:
return endpoint
endpoint = endpoint.rstrip("/")
if endpoint.endswith("/v1/logs"):
return endpoint
for other_signal in ("/v1/traces", "/v1/metrics"):
if endpoint.endswith(other_signal):
return endpoint[: -len(other_signal)] + "/v1/logs"
return endpoint + "/v1/logs"
def build_log_exporter(config: OpenTelemetryV2Config) -> LogExporter:
"""Build a log exporter mirroring the exporter selection of the other signals.
``console`` (and any unrecognized kind) exports to the console; ``otlp_http``
and ``otlp_grpc`` export over OTLP with the configured endpoint/headers;
``in_memory`` buffers for tests. Like GenAI metrics, events ride the
single-destination shorthand fields, not the multi-exporter ``exporters`` list.
"""
kind = (config.exporter or "console").lower()
if kind in ("in_memory", "inmemory", "memory"):
return InMemoryLogExporter()
if kind in ("otlp_http", "http", "http/protobuf", "http/json"):
from opentelemetry.exporter.otlp.proto.http._log_exporter import (
OTLPLogExporter as HTTPLogExporter,
)
return HTTPLogExporter(
endpoint=_otlp_logs_endpoint(config.endpoint),
headers=parse_headers(config.headers),
)
if kind in ("otlp_grpc", "grpc"):
try:
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
OTLPLogExporter as GRPCLogExporter,
)
except ImportError as exc:
raise ImportError(
"OpenTelemetry OTLP gRPC log exporter is not available. Install "
"`opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`)."
) from exc
return GRPCLogExporter(endpoint=config.endpoint, headers=parse_headers(config.headers))
return ConsoleLogExporter()
def build_logger_provider(
config: OpenTelemetryV2Config,
log_exporter: LogExporter | None = None,
) -> SDKLoggerProvider:
"""Build the :class:`LoggerProvider` GenAI events export through.
``log_exporter`` is an explicit override (tests inject an
``InMemoryLogExporter``); otherwise the exporter is selected from the config's
exporter kind via :func:`build_log_exporter`. Console and in-memory exporters
get a Simple processor (synchronous export, which tests rely on), everything
else a Batch processor — the same split as span processing.
"""
exporter = log_exporter if log_exporter is not None else build_log_exporter(config)
provider = SDKLoggerProvider(resource=build_resource(config))
use_simple = isinstance(exporter, (ConsoleLogExporter, InMemoryLogExporter))
provider.add_log_record_processor(
SimpleLogRecordProcessor(exporter) if use_simple else BatchLogRecordProcessor(exporter)
)
return provider
def resolve_logger_provider(
config: OpenTelemetryV2Config,
logger_provider: SDKLoggerProvider | None = None,
) -> SDKLoggerProvider | None:
"""Resolve the :class:`LoggerProvider` GenAI events record through, or ``None``
when the operator has opted out of the logs signal.
Same resolution order as :func:`resolve_meter_provider`: an injected provider
wins (DI/tests); an operator-configured SDK global is reused so events ride
their pipeline; an explicit ``NoOpLoggerProvider`` global is an opt-out and
yields ``None``, so no event is ever built. Only the default placeholder
global makes V2 build a provider from the config and publish it as the global.
"""
if logger_provider is not None:
return logger_provider
existing: LoggerProvider = _logs.get_logger_provider()
if isinstance(existing, SDKLoggerProvider):
return existing
if isinstance(existing, NoOpLoggerProvider):
return None
provider = build_logger_provider(config)
_logs.set_logger_provider(provider)
return provider
def get_event_logger(provider: SDKLoggerProvider, name: str = "litellm") -> EventLogger:
return EventLoggerProvider(logger_provider=provider).get_event_logger(name, litellm_version)
def build_meter_provider(
config: OpenTelemetryV2Config,
metric_reader: "MetricReader | None" = None,

137
litellm/integrations/prometheus.py
View file

@ -239,18 +239,6 @@ class PrometheusLogger(CustomLogger):
labelnames=self.get_labels_for_metric("litellm_output_audio_tokens_metric"),
)
self.litellm_video_duration_seconds_metric = self._counter_factory(
"litellm_video_duration_seconds_metric",
"Seconds of video generated, from usage.duration_seconds on video generation calls",
labelnames=self.get_labels_for_metric("litellm_video_duration_seconds_metric"),
)
self.litellm_images_generated_metric = self._counter_factory(
"litellm_images_generated_metric",
"Number of images generated, from the image generation response",
labelnames=self.get_labels_for_metric("litellm_images_generated_metric"),
)
# Remaining Budget for Team
self.litellm_remaining_team_budget_metric = self._gauge_factory(
"litellm_remaining_team_budget_metric",
@ -1348,12 +1336,6 @@ class PrometheusLogger(CustomLogger):
label_context=label_context,
)
self._increment_media_generation_metrics(
standard_logging_payload=standard_logging_payload,
enum_values=enum_values,
label_context=label_context,
)
# MCP tool call metrics
self._increment_mcp_tool_call_metrics(
standard_logging_payload=standard_logging_payload,
@ -1477,65 +1459,8 @@ class PrometheusLogger(CustomLogger):
),
]
PrometheusLogger._inc_sparse_usage_counters(
self,
detail_metrics,
enum_values=enum_values,
label_context=label_context,
)
def _increment_media_generation_metrics(
self,
standard_logging_payload: StandardLoggingPayload,
enum_values: UserAPIKeyLabelValues,
label_context: PrometheusLabelFactoryContext | None = None,
) -> None:
"""
Increment video-seconds and images-generated counters from
``standard_logging_payload["metadata"]["usage_object"]``. Video
providers report ``duration_seconds`` there; image generation calls
report ``output_image_count``. Both are sparse: only emitted when the
value is present and > 0, so token-only call types are unaffected.
"""
metadata = standard_logging_payload.get("metadata") or {}
usage_object = metadata.get("usage_object") if isinstance(metadata, dict) else None
if not isinstance(usage_object, dict):
return
media_metrics: list[tuple[Any, DEFINED_PROMETHEUS_METRICS, Any]] = [
(
self.litellm_video_duration_seconds_metric,
"litellm_video_duration_seconds_metric",
usage_object.get("duration_seconds"),
),
(
self.litellm_images_generated_metric,
"litellm_images_generated_metric",
usage_object.get("output_image_count"),
),
]
PrometheusLogger._inc_sparse_usage_counters(
self,
media_metrics,
enum_values=enum_values,
label_context=label_context,
)
def _inc_sparse_usage_counters(
self,
counters_with_values: list[tuple[Any, DEFINED_PROMETHEUS_METRICS, Any]],
enum_values: UserAPIKeyLabelValues,
label_context: PrometheusLabelFactoryContext | None = None,
) -> None:
"""
Increment each ``(counter, metric_name, value)`` entry whose value is
a positive number. Non-numeric values (including booleans from
malformed provider usage dicts) and values <= 0 are skipped, keeping
scrape output sparse.
"""
for counter, metric_name, value in counters_with_values:
if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0:
for counter, metric_name, value in detail_metrics:
if not isinstance(value, (int, float)) or value <= 0:
continue
PrometheusLogger._inc_labeled_counter(
self,
@ -1693,14 +1618,6 @@ class PrometheusLogger(CustomLogger):
user_id: Optional[str] = None,
user_api_key_org_id: Optional[str] = None,
):
if (
isinstance(self.litellm_remaining_team_budget_metric, NoOpMetric)
and isinstance(self.litellm_remaining_api_key_budget_metric, NoOpMetric)
and isinstance(self.litellm_remaining_user_budget_metric, NoOpMetric)
and isinstance(self.litellm_remaining_org_budget_metric, NoOpMetric)
):
return
_metadata = litellm_params.get("metadata") or {}
_team_spend = _metadata.get("user_api_key_team_spend", None)
_team_max_budget = _metadata.get("user_api_key_team_max_budget", None)
@ -1791,35 +1708,6 @@ class PrometheusLogger(CustomLogger):
amount=float(response_cost),
)
@staticmethod
def _get_remaining_from_v3_rate_limit_headers(
standard_logging_payload: StandardLoggingPayload | None,
rate_limit_type: Literal["requests", "tokens"],
) -> int | None:
"""
Read the per-(key, model) remaining value emitted by the v3 rate
limiter (``parallel_request_limiter_v3.py``), which writes
``x-ratelimit-model_per_key-remaining-{requests,tokens}`` into
``standard_logging_object.hidden_params.additional_headers`` instead
of the ``litellm-key-remaining-*`` metadata keys the legacy limiter
sets. The header carries no model group; it always refers to this
request's model group, which is what the gauges are labeled with.
Values are written in-process as plain ints (never HTTP-serialized
strings), so anything else is rejected rather than coerced.
"""
if standard_logging_payload is None:
return None
hidden_params = standard_logging_payload.get("hidden_params")
if hidden_params is None:
return None
additional_headers = hidden_params.get("additional_headers")
if additional_headers is None:
return None
value = dict(additional_headers).get(f"x-ratelimit-model_per_key-remaining-{rate_limit_type}")
if isinstance(value, bool) or not isinstance(value, int):
return None
return value
def _set_virtual_key_rate_limit_metrics(
self,
user_api_key: Optional[str],
@ -1837,20 +1725,11 @@ class PrometheusLogger(CustomLogger):
model_group = get_model_group_from_litellm_kwargs(kwargs)
remaining_requests_variable_name = f"litellm-key-remaining-requests-{model_group}"
remaining_tokens_variable_name = f"litellm-key-remaining-tokens-{model_group}"
standard_logging_payload: StandardLoggingPayload | None = kwargs.get("standard_logging_object")
remaining_requests = metadata.get(remaining_requests_variable_name)
if remaining_requests is None:
remaining_requests = self._get_remaining_from_v3_rate_limit_headers(
standard_logging_payload=standard_logging_payload, rate_limit_type="requests"
)
if remaining_requests is None:
remaining_requests = sys.maxsize
remaining_tokens = metadata.get(remaining_tokens_variable_name)
if remaining_tokens is None:
remaining_tokens = self._get_remaining_from_v3_rate_limit_headers(
standard_logging_payload=standard_logging_payload, rate_limit_type="tokens"
)
if remaining_tokens is None:
remaining_tokens = sys.maxsize
@ -3453,9 +3332,6 @@ class PrometheusLogger(CustomLogger):
- looks up team info from db if not available in metadata
- Set team budget metrics
"""
if isinstance(self.litellm_remaining_team_budget_metric, NoOpMetric):
return
if user_api_team:
team_object = await self._assemble_team_object(
team_id=user_api_team,
@ -3577,9 +3453,6 @@ class PrometheusLogger(CustomLogger):
- Fetches org info via cache (get_org_object)
- Sets org budget metrics
"""
if isinstance(self.litellm_remaining_org_budget_metric, NoOpMetric):
return
if not org_id:
return
@ -3709,9 +3582,6 @@ class PrometheusLogger(CustomLogger):
key_max_budget: Optional[float],
key_spend: Optional[float],
):
if isinstance(self.litellm_remaining_api_key_budget_metric, NoOpMetric):
return
if user_api_key:
user_api_key_dict = await self._assemble_key_object(
user_api_key=user_api_key,
@ -3772,9 +3642,6 @@ class PrometheusLogger(CustomLogger):
- looks up user info from db if not available in metadata
- Set user budget metrics
"""
if isinstance(self.litellm_remaining_user_budget_metric, NoOpMetric):
return
if user_id:
user_object = await self._assemble_user_object(
user_id=user_id,

7
litellm/integrations/rubrik.py
View file

@ -7,7 +7,7 @@ import time
import urllib.parse
import uuid
from collections import Counter
from typing import TYPE_CHECKING, Any, List, Literal, Optional
from typing import TYPE_CHECKING, Any, Literal, Optional
import httpx
from litellm._logging import verbose_logger
@ -52,10 +52,6 @@ class _MalformedToolBlockingResponseError(Exception):
class RubrikLogger(CustomGuardrail, CustomBatchLogger):
@classmethod
def get_supported_event_hooks(cls) -> List[GuardrailEventHooks]:
return [GuardrailEventHooks.pre_call, GuardrailEventHooks.post_call]
def __init__(
self,
api_key: str | None = None,
@ -73,7 +69,6 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
kwargs["event_hook"] = kwargs.get("event_hook") or GuardrailEventHooks.post_call
if kwargs.get("default_on") is None:
kwargs["default_on"] = True
kwargs.setdefault("supported_event_hooks", list(self.get_supported_event_hooks()))
super().__init__(
flush_lock=self.flush_lock,
**kwargs,

7
litellm/integrations/s3.py
View file

@ -161,13 +161,8 @@ def get_s3_object_key(
start_time: datetime,
s3_file_name: str,
) -> str:
sanitized_s3_file_name = s3_file_name.replace("/", "_")
s3_object_key = (
(s3_path.rstrip("/") + "/" if s3_path else "")
+ prefix
+ start_time.strftime("%Y-%m-%d")
+ "/"
+ sanitized_s3_file_name
(s3_path.rstrip("/") + "/" if s3_path else "") + prefix + start_time.strftime("%Y-%m-%d") + "/" + s3_file_name
) # we need the s3 key to include the time, so we log cache hits too
s3_object_key += ".json"
return s3_object_key

15
litellm/litellm_core_utils/cli_token_utils.py
View file

@ -7,7 +7,6 @@ This module has no dependencies on proxy code and can be safely imported at the
import json
import os
import time
from pathlib import Path
from typing import Optional
@ -69,17 +68,3 @@ def get_litellm_gateway_api_key(
if stored_url != expected_base_url.rstrip("/"):
return None
return token_data["key"]
def is_cli_token_fresh(token_data: dict, buffer_hours: float = 0.1) -> bool:
"""Check whether a cached CLI token (as stored in token.json) is still
within its expiration window. Used by `lite auth print-token` to fail
fast, without a network round trip, once the cached token is past
`LITELLM_CLI_JWT_EXPIRATION_HOURS`."""
from litellm.constants import CLI_JWT_EXPIRATION_HOURS
timestamp = token_data.get("timestamp")
if not isinstance(timestamp, (int, float)):
return False
age_hours = (time.time() - timestamp) / 3600
return age_hours < (CLI_JWT_EXPIRATION_HOURS - buffer_hours)

3
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -95,9 +95,6 @@ class ExceptionCheckers:
if "current length is" in _error_str_lowercase and "while limit is" in _error_str_lowercase:
return True
if "maximum input length is" in _error_str_lowercase and "tokens" in _error_str_lowercase:
return True
return False
@staticmethod

225
litellm/litellm_core_utils/fallback_generalizations.py
View file

@ -3,69 +3,52 @@ Declarative fallback generalizations for unknown / newly-released models.
The ``fallback_generalizations`` block in ``model_prices_and_context_window.json``
holds an ordered list of rules. Each rule pairs a single case-insensitive regex
with a ``model_info`` dict, and the structure of ``model_info`` decides which of
two kinds the rule is.
with the metadata to apply when a model name has no exact entry in the cost map.
The metadata is a partial cost-map entry: ``litellm_provider`` drives provider
routing, and the remaining fields (``mode``, ``supports_*``, context window,
pricing, ...) drive ``get_model_info`` / ``supports_*``.
A ROUTING rule carries exactly one ``model_info`` key, ``litellm_provider``. It is
consumed only by ``get_llm_provider`` bare-id inference: the first routing rule
whose regex matches decides the provider. Routing rules never contribute to model
info.
A CAPABILITY rule carries any ``model_info`` keys except ``litellm_provider``
(``mode``, ``supports_*``, context window, pricing, ...). It is consumed by
``get_model_info`` fallback resolution: the ``model_info`` of ALL capability rules
whose regex matches is unioned in file order, with later rules overriding earlier
ones on key conflicts, and the caller backfills ``litellm_provider`` with the
provider it requested. If no capability rule matches, model-info resolution misses
as if no rules existed.
LEGACY-SCHEMA SHIM (temporary, until the new-schema JSON reaches main): released
proxies fetch this JSON remotely from main, whose block still ships the old schema
where a rule mixes ``litellm_provider`` with capability keys and may inherit a
parent's ``model_info`` via ``extends``. Such a legacy rule is tolerated rather
than skipped: ``extends`` is resolved once at install time (single level, against
raw parents), and the resolved rule acts as BOTH kinds, a routing rule (its
``litellm_provider`` participates in first-hit inference) and a capability rule
(its full ``model_info``, provider included, participates in the union). New-schema
rules never mix the two and never use ``extends``. A rule whose
``litellm_provider`` is not a string is invalid and is warned about and skipped
(a warning rather than a crash, for the same remote-fetch reason).
Rules are only consulted after exact and case-insensitive lookups miss, so an
exact cost-map entry always takes precedence over any rule.
Precedence: rules are evaluated in file order and the first match wins. They are
consulted only after exact and case-insensitive lookups miss, so an exact entry
always takes precedence over a rule.
Patterns are matched case-insensitively with ``re.search`` and are not implicitly
anchored: a rule must include ``^`` and ``$`` to bind to the whole model name,
otherwise it matches as a substring. Keeping anchoring in the regex makes the rule
the single, self-contained source of truth for what it matches.
anchored: a rule must include ``^`` and ``$`` (as the shipped rules do) to bind to
the whole model name, otherwise it matches as a substring. Keeping anchoring in the
regex makes the rule the single, self-contained source of truth for what it matches.
A rule may set ``extends`` to the ``name`` of another rule to inherit that rule's
``model_info``; the rule's own ``model_info`` overrides the inherited keys, so a
narrow rule (for example a version-gated capability flag) carries only its delta
instead of duplicating the parent's pricing block. Inheritance is resolved once,
at install time, against each rule's raw (unresolved) ``model_info``; it is a
single level (a parent that itself extends is not chained).
Any other keys on a rule (for example a free-text ``description`` documenting what
the regex matches) are ignored by the engine and exist only for the reader.
Rules are compiled and classified once, at install time. The match functions are
O(number of rules); callers must only invoke them on a cache miss.
The compiled-regex list is built once and cached. ``match_fallback_generalization``
is O(number of rules); callers must only invoke it on a cache miss.
"""
import re
from dataclasses import dataclass
from typing import Optional, Union
from typing import Optional
from litellm._logging import verbose_logger
NAME_FIELD = "name"
PATTERN_FIELD = "pattern"
MODEL_INFO_FIELD = "model_info"
PROVIDER_KEY = "litellm_provider"
LEGACY_EXTENDS_FIELD = "extends"
EXTENDS_FIELD = "extends"
def _resolve_legacy_extends(rules: list) -> list:
"""Expand legacy ``extends`` inheritance so each rule's ``model_info`` is self-contained.
def _resolve_extends(rules: list) -> list:
"""Expand ``extends`` inheritance so each rule's ``model_info`` is self-contained.
Compatibility shim for the old remote schema: single level, resolved against each
parent's raw ``model_info``, with the child's own keys winning on conflict. Non-dict
rules and dangling parents pass through unchanged; new-schema rules carry no
``extends`` and are untouched.
A rule with ``extends: <name>`` is rewritten with ``model_info`` set to the parent's
``model_info`` overlaid by its own. Resolution is single-level and uses each rule's
raw ``model_info`` as the parent source. Non-dict rules and dangling parents are
passed through unchanged.
"""
base_by_name = {
rule[NAME_FIELD]: rule[MODEL_INFO_FIELD]
@ -75,138 +58,84 @@ def _resolve_legacy_extends(rules: list) -> list:
and isinstance(rule.get(MODEL_INFO_FIELD), dict)
}
def resolved(rule: object) -> object:
if not isinstance(rule, dict):
return rule
parent_name = rule.get(LEGACY_EXTENDS_FIELD)
def resolved(rule: dict) -> dict:
parent_name = rule.get(EXTENDS_FIELD)
own_info = rule.get(MODEL_INFO_FIELD)
parent_info = base_by_name.get(parent_name) if isinstance(parent_name, str) else None
if parent_info is None or not isinstance(own_info, dict):
return rule
return {**rule, MODEL_INFO_FIELD: {**parent_info, **own_info}}
return [resolved(rule) for rule in rules]
@dataclass(frozen=True, slots=True)
class _RoutingRule:
pattern: re.Pattern
provider: str
@dataclass(frozen=True, slots=True)
class _CapabilityRule:
pattern: re.Pattern
model_info: dict
_CompiledRule = Union[_RoutingRule, _CapabilityRule]
def _compile_rule(rule: object) -> tuple[_CompiledRule, ...]:
if not isinstance(rule, dict):
return ()
pattern = rule.get(PATTERN_FIELD)
model_info = rule.get(MODEL_INFO_FIELD)
if not isinstance(pattern, str) or not isinstance(model_info, dict):
verbose_logger.warning(
"LiteLLM: skipping malformed fallback generalization rule %s (needs string '%s' and dict '%s').",
rule.get(NAME_FIELD, pattern),
PATTERN_FIELD,
MODEL_INFO_FIELD,
)
return ()
try:
compiled = re.compile(pattern, re.IGNORECASE)
except re.error as e:
verbose_logger.warning(
"LiteLLM: skipping fallback generalization rule with invalid regex %r: %s",
pattern,
e,
)
return ()
if PROVIDER_KEY not in model_info:
return (_CapabilityRule(pattern=compiled, model_info=model_info),)
provider = model_info[PROVIDER_KEY]
if not isinstance(provider, str):
verbose_logger.warning(
"LiteLLM: skipping invalid fallback generalization rule %s: '%s' in '%s' must be a string.",
rule.get(NAME_FIELD, pattern),
PROVIDER_KEY,
MODEL_INFO_FIELD,
)
return ()
if len(model_info) == 1:
return (_RoutingRule(pattern=compiled, provider=provider),)
return (
_RoutingRule(pattern=compiled, provider=provider),
_CapabilityRule(pattern=compiled, model_info=model_info),
)
return [resolved(rule) if isinstance(rule, dict) else rule for rule in rules]
class _FallbackGeneralizations:
"""Holds the raw rule list and its install-time-compiled routing and capability rules."""
"""Holds the active rule list and its lazily-compiled regex cache."""
def __init__(self) -> None:
self.rules: list = []
self.routing_rules: tuple = ()
self.capability_rules: tuple = ()
self.rules: list[dict] = []
self._compiled: Optional[list[tuple[re.Pattern, dict]]] = None
def set_rules(self, rules: Optional[list]) -> None:
installed = rules if isinstance(rules, list) else []
compiled = tuple(kind for rule in _resolve_legacy_extends(installed) for kind in _compile_rule(rule))
self.rules = installed
self.routing_rules = tuple(rule for rule in compiled if isinstance(rule, _RoutingRule))
self.capability_rules = tuple(rule for rule in compiled if isinstance(rule, _CapabilityRule))
def set_rules(self, rules: Optional[list[dict]]) -> None:
self.rules = rules if isinstance(rules, list) else []
self._compiled = None
def match_routing(self, model: str) -> Optional[str]:
def _compile(self) -> list[tuple[re.Pattern, dict]]:
compiled: list[tuple[re.Pattern, dict]] = []
for rule in self.rules:
if not isinstance(rule, dict):
continue
pattern = rule.get(PATTERN_FIELD)
model_info = rule.get(MODEL_INFO_FIELD)
if not isinstance(pattern, str) or not isinstance(model_info, dict):
verbose_logger.warning(
"LiteLLM: skipping malformed fallback generalization rule %s (needs string '%s' and dict '%s').",
rule.get("name", pattern),
PATTERN_FIELD,
MODEL_INFO_FIELD,
)
continue
try:
compiled.append((re.compile(pattern, re.IGNORECASE), model_info))
except re.error as e:
verbose_logger.warning(
"LiteLLM: skipping fallback generalization rule with invalid regex %r: %s",
pattern,
e,
)
return compiled
def match(self, model: str) -> Optional[dict]:
if not model:
return None
return next(
(rule.provider for rule in self.routing_rules if rule.pattern.search(model) is not None),
None,
)
def match_capabilities(self, model: str) -> Optional[dict]:
if not model:
return None
matched = tuple(rule.model_info for rule in self.capability_rules if rule.pattern.search(model) is not None)
if not matched:
return None
return {key: value for model_info in matched for key, value in model_info.items()}
if self._compiled is None:
self._compiled = self._compile()
for pattern, model_info in self._compiled:
if pattern.search(model) is not None:
return dict(model_info)
return None
_registry = _FallbackGeneralizations()
def set_fallback_generalizations(rules: Optional[list]) -> None:
"""Install the active rule list, compiling and classifying each rule.
def set_fallback_generalizations(rules: Optional[list[dict]]) -> None:
"""Install the active rule list and invalidate the compiled-regex cache.
Legacy ``extends`` inheritance is resolved here, once, before classification;
a legacy rule mixing ``litellm_provider`` with capability keys installs as both
kinds. Malformed and invalid-regex rules are warned about and skipped. Called
once when the model cost map is loaded (and again on any reload).
``extends`` inheritance is resolved here, once, before the rules are stored.
Called once when the model cost map is loaded (and again on any reload).
"""
_registry.set_rules(rules)
_registry.set_rules(_resolve_extends(rules) if isinstance(rules, list) else rules)
def get_fallback_generalization_rules() -> list:
def get_fallback_generalization_rules() -> list[dict]:
"""Return the raw rule list (read-only view for callers/tests)."""
return _registry.rules
def match_routing_generalization(model: str) -> Optional[str]:
"""Return the provider of the first routing rule whose regex matches ``model``.
def match_fallback_generalization(model: str) -> Optional[dict]:
"""Return the ``model_info`` of the first rule whose regex matches ``model``.
O(number of rules). Only call this once exact lookups have missed.
"""
return _registry.match_routing(model)
def match_capability_generalizations(model: str) -> Optional[dict]:
"""Return the union of the ``model_info`` of every capability rule matching ``model``.
Later rules override earlier ones on key conflicts. Returns ``None`` when no
capability rule matches. O(number of rules); only call once exact lookups have missed.
"""
return _registry.match_capabilities(model)
return _registry.match(model)

56
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -2,8 +2,26 @@ from typing import Optional
from litellm.llms.openai.data_residency import infer_openai_data_residency
AWS_CREDENTIAL_KWARGS_KEYS = frozenset(
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
OPTIONAL_KWARGS_KEYS = frozenset(
{
"azure_ad_token",
"tenant_id",
"client_id",
"client_secret",
"azure_username",
"azure_password",
"azure_scope",
"timeout",
"gcs_bucket_name",
"bucket_name",
"vertex_credentials",
"vertex_project",
"vertex_location",
"vertex_ai_project",
"vertex_ai_location",
"vertex_ai_credentials",
"aws_region_name",
"aws_access_key_id",
"aws_secret_access_key",
@ -16,40 +34,14 @@ AWS_CREDENTIAL_KWARGS_KEYS = frozenset(
"aws_external_id",
"aws_bedrock_runtime_endpoint",
"aws_bedrock_project_id",
"tpm",
"rpm",
"itpm",
"otpm",
"use_xai_oauth",
}
)
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
OPTIONAL_KWARGS_KEYS = (
frozenset(
{
"azure_ad_token",
"tenant_id",
"client_id",
"client_secret",
"azure_username",
"azure_password",
"azure_scope",
"timeout",
"gcs_bucket_name",
"bucket_name",
"vertex_credentials",
"vertex_project",
"vertex_location",
"vertex_ai_project",
"vertex_ai_location",
"vertex_ai_credentials",
"tpm",
"rpm",
"itpm",
"otpm",
"use_xai_oauth",
}
)
| AWS_CREDENTIAL_KWARGS_KEYS
)
# Backward-compatible alias for existing imports/tests.
_OPTIONAL_KWARGS_KEYS = OPTIONAL_KWARGS_KEYS

11
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -4,7 +4,7 @@ from urllib.parse import urlparse
import litellm
from litellm.constants import REPLICATE_MODEL_NAME_WITH_ID_LENGTH
from litellm.litellm_core_utils.fallback_generalizations import (
match_routing_generalization,
match_fallback_generalization,
)
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
from litellm.secret_managers.main import get_secret, get_secret_str
@ -346,9 +346,6 @@ def get_llm_provider(
elif endpoint == "https://pinstripes.io/v1":
custom_llm_provider = "pinstripes"
dynamic_api_key = get_secret_str("PINSTRIPES_API_KEY")
elif endpoint == "https://api.meta.ai/v1":
custom_llm_provider = "meta"
dynamic_api_key = get_secret_str("META_API_KEY")
if api_base is not None and not isinstance(api_base, str):
raise Exception("api base needs to be a string. api_base={}".format(api_base))
@ -474,10 +471,12 @@ def get_llm_provider(
custom_llm_provider = "sap"
# Last resort for an otherwise-unknown model: a declarative
# fallback-generalization routing rule (e.g. routes future claude-* to anthropic).
# fallback-generalization rule (e.g. routes future claude-* to anthropic).
# Exact provider matches above always win; this only runs on a miss.
if not custom_llm_provider:
custom_llm_provider = match_routing_generalization(model)
generalization = match_fallback_generalization(model)
if generalization is not None:
custom_llm_provider = generalization.get("litellm_provider") or None
if not custom_llm_provider:
if litellm.suppress_debug_info is False:

11
litellm/litellm_core_utils/litellm_logging.py
View file

@ -73,7 +73,6 @@ from litellm.litellm_core_utils.model_param_helper import ModelParamHelper
from litellm.litellm_core_utils.redact_messages import (
redact_message_input_output_from_custom_logger,
redact_message_input_output_from_logging,
redact_streaming_responses_for_custom_logger,
)
from litellm.llms.base_llm.ocr.transformation import OCRResponse
from litellm.llms.base_llm.search.transformation import SearchResponse
@ -2577,9 +2576,6 @@ class Logging(LiteLLMLoggingBaseClass):
model_call_details = callback.redact_standard_logging_payload_from_model_call_details(
model_call_details=model_call_details
)
model_call_details = redact_streaming_responses_for_custom_logger(
model_call_details=model_call_details, custom_logger=callback
)
##################################
if self.stream is True:
if "async_complete_streaming_response" in model_call_details:
@ -5212,15 +5208,10 @@ def get_standard_logging_object_payload(
call_type = kwargs.get("call_type")
cache_hit = kwargs.get("cache_hit", False)
# Extract usage as a plain dict, avoiding Pydantic round-trip
raw_usage_dict = StandardLoggingPayloadSetup.get_usage_as_dict(
usage_dict = StandardLoggingPayloadSetup.get_usage_as_dict(
response_obj=response_obj,
combined_usage_object=cast(Optional[Usage], kwargs.get("combined_usage_object")),
)
usage_dict = (
{**raw_usage_dict, "output_image_count": len(init_response_obj.data)}
if isinstance(init_response_obj, ImageResponse) and init_response_obj.data
else raw_usage_dict
)
id = response_obj.get("id", kwargs.get("litellm_call_id"))

139
litellm/litellm_core_utils/llm_cost_calc/tiered_pricing.py
View file

@ -1,139 +0,0 @@
"""
Provider-neutral graduated tiered pricing calculation.
Shared by provider cost calculators (e.g. Dashscope) and the proxy budget
reservation logic so neither has to depend on the other.
"""
from typing import List, Optional, Union
def _coerce_cost_per_token(value: Union[float, int, str, None]) -> float:
"""
Coerce a per-token cost into a float.
Model cost values loaded from YAML config may arrive as strings (e.g.
scientific notation like "4e-07"), which would break arithmetic.
"""
if value is None:
return 0.0
if isinstance(value, str):
try:
return float(value)
except ValueError:
return 0.0
return float(value)
def calculate_tiered_cost(
tokens: int,
tiered_pricing: List[dict],
cost_key: str,
fallback_cost_key: Optional[str] = None,
) -> float:
"""
Calculate cost for a given number of tokens based on a true tiered pricing structure.
This function iterates through sorted pricing tiers, calculates the cost for the
number of tokens that fall into each tier's range, and sums them up to get the total cost.
Args:
tokens (int): The total number of tokens to calculate the cost for.
tiered_pricing (List[dict]): A list of dictionaries, where each dictionary
represents a pricing tier.
cost_key (str): The key in the tier dictionary that holds the per-token cost
(e.g., 'input_cost_per_token').
fallback_cost_key (Optional[str], optional): A fallback key to use if the
primary `cost_key` is not found in a tier. Defaults to None.
Returns:
float: The total calculated cost for the given tokens.
Example:
>>> tiered_pricing = [
... {"range": [0, 100000], "input_cost_per_token": 0.0001},
... {"range": [100000, 500000], "input_cost_per_token": 0.00005},
... ]
Calculating cost for 150,000 tokens:
(100,000 * 0.0001) + (50,000 * 0.00005) = $12.5
"""
if not tiered_pricing or tokens <= 0:
return 0.0
total_cost = 0.0
tokens_processed = 0
sorted_tiers = sorted(tiered_pricing, key=lambda x: x.get("range", [0, 0])[0])
for tier in sorted_tiers:
if tokens_processed >= tokens:
break
tier_range = tier.get("range", [])
if len(tier_range) != 2:
continue
range_start, range_end = tier_range
if tokens <= range_start:
continue
tier_start = max(range_start, tokens_processed)
tier_end = min(range_end, tokens)
if tier_end > tier_start:
tokens_in_tier = tier_end - tier_start
cost_per_token = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
total_cost += tokens_in_tier * _coerce_cost_per_token(cost_per_token)
tokens_processed = tier_end
# After loop, check if any tokens remain (i.e., tokens > highest tier's end range)
# and charge them at the last tier's rate.
if tokens_processed < tokens and sorted_tiers:
last_tier = sorted_tiers[-1]
remaining_tokens = tokens - tokens_processed
cost_per_token = last_tier.get(cost_key) or last_tier.get(fallback_cost_key, 0)
total_cost += remaining_tokens * _coerce_cost_per_token(cost_per_token)
return total_cost
def select_tier_for_input(
tiered_pricing: List[dict],
input_tokens: int,
) -> Optional[dict]:
"""
Select the pricing tier for a request based on its total input token count.
Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is
chosen by the total input tokens of a single request and every token in the
request (input and output) is billed at that one tier's rate, rather than
graduated income-tax-style slicing. A tier matches when
``range_start < input_tokens <= range_end`` (so a request of exactly
``range_end`` tokens stays in the lower tier, matching the official
``0 < Token <= 32K`` phrasing). Requests above the highest declared range fall
back to the last (most expensive) tier.
"""
if not tiered_pricing or input_tokens <= 0:
return None
sorted_tiers = sorted(tiered_pricing, key=lambda t: t.get("range", [0, 0])[0])
valid_tiers = [tier for tier in sorted_tiers if len(tier.get("range", [])) == 2]
if not valid_tiers:
return None
matching = [tier for tier in valid_tiers if tier["range"][0] < input_tokens <= tier["range"][1]]
if matching:
return matching[0]
return valid_tiers[-1]
def tier_rate(
tier: dict,
cost_key: str,
fallback_cost_key: Optional[str] = None,
) -> float:
"""Read a per-token rate from a tier, coercing YAML string costs to float."""
raw = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
return _coerce_cost_per_token(raw)

35
litellm/litellm_core_utils/llm_cost_calc/utils.py
View file

@ -445,7 +445,6 @@ class PromptTokensDetailsResult(TypedDict):
text_tokens: int
audio_tokens: int
image_tokens: int
video_tokens: int
character_count: int
image_count: int
video_length_seconds: float
@ -474,7 +473,6 @@ def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
)
audio_tokens = cast(Optional[int], getattr(usage.prompt_tokens_details, "audio_tokens", 0)) or 0
image_tokens = cast(Optional[int], getattr(usage.prompt_tokens_details, "image_tokens", 0)) or 0
video_tokens = _coerce_token_count(getattr(usage.prompt_tokens_details, "video_tokens", 0))
character_count = (
cast(
Optional[int],
@ -505,7 +503,6 @@ def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
text_tokens=text_tokens,
audio_tokens=audio_tokens,
image_tokens=image_tokens,
video_tokens=video_tokens,
character_count=character_count,
image_count=image_count,
video_length_seconds=float(video_length_seconds),
@ -518,7 +515,6 @@ class CompletionTokensDetailsResult(TypedDict):
text_tokens: int
reasoning_tokens: int
image_tokens: int
video_tokens: int
def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsResult:
@ -550,14 +546,12 @@ def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsRes
)
or 0
)
video_tokens = _coerce_token_count(getattr(usage.completion_tokens_details, "video_tokens", 0))
return CompletionTokensDetailsResult(
audio_tokens=audio_tokens,
text_tokens=text_tokens,
reasoning_tokens=reasoning_tokens,
image_tokens=image_tokens,
video_tokens=video_tokens,
)
@ -592,13 +586,6 @@ def _calculate_input_cost(
image_token_cost_key = "input_cost_per_token"
prompt_cost += calculate_cost_component(model_info, image_token_cost_key, prompt_tokens_details["image_tokens"])
### VIDEO TOKEN COST
if prompt_tokens_details["video_tokens"]:
video_token_cost_key = "input_cost_per_video_token"
if model_info.get(video_token_cost_key) is None:
video_token_cost_key = "input_cost_per_token"
prompt_cost += calculate_cost_component(model_info, video_token_cost_key, prompt_tokens_details["video_tokens"])
### CACHE WRITING COST - Now uses tiered pricing
if (
prompt_tokens_details["cache_creation_tokens"]
@ -711,7 +698,6 @@ def generic_cost_per_token(
text_tokens=usage.prompt_tokens,
audio_tokens=0,
image_tokens=0,
video_tokens=0,
character_count=0,
image_count=0,
video_length_seconds=0.0,
@ -730,14 +716,13 @@ def generic_cost_per_token(
audio_tokens = prompt_tokens_details["audio_tokens"]
cache_creation = prompt_tokens_details["cache_creation_tokens"]
image_tokens = prompt_tokens_details["image_tokens"]
video_tokens = prompt_tokens_details["video_tokens"]
# Check for double-counting: sum of details > prompt_tokens means overlap
total_details = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens + video_tokens
total_details = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens
has_double_counting = cache_hit > 0 and total_details > usage.prompt_tokens
if (text_tokens == 0 and prompt_tokens_details["image_count"] == 0) or has_double_counting:
text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens - video_tokens
text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens
# Clamp to zero: inconsistent streaming usage
if text_tokens < 0:
text_tokens = 0
@ -766,7 +751,6 @@ def generic_cost_per_token(
audio_tokens = 0
reasoning_tokens = 0
image_tokens = 0
video_tokens = 0
is_text_tokens_total = False
if usage.completion_tokens_details is not None:
completion_tokens_details = _parse_completion_tokens_details(usage)
@ -774,20 +758,19 @@ def generic_cost_per_token(
text_tokens = completion_tokens_details["text_tokens"]
reasoning_tokens = completion_tokens_details["reasoning_tokens"]
image_tokens = completion_tokens_details["image_tokens"]
video_tokens = completion_tokens_details["video_tokens"]
# Handle text_tokens calculation:
# 1. If text_tokens is explicitly provided and > 0, use it
# 2. If there's a breakdown (reasoning/audio/image/video tokens), calculate text_tokens as the remainder
# 2. If there's a breakdown (reasoning/audio/image tokens), calculate text_tokens as the remainder
# 3. If no breakdown at all, assume all completion_tokens are text_tokens
has_token_breakdown = image_tokens > 0 or audio_tokens > 0 or reasoning_tokens > 0 or video_tokens > 0
has_token_breakdown = image_tokens > 0 or audio_tokens > 0 or reasoning_tokens > 0
if text_tokens == 0:
if has_token_breakdown:
# Calculate text tokens as remainder when we have a breakdown
# This handles cases like OpenAI's reasoning models where text_tokens isn't provided
text_tokens = max(
0,
usage.completion_tokens - reasoning_tokens - audio_tokens - image_tokens - video_tokens,
usage.completion_tokens - reasoning_tokens - audio_tokens - image_tokens,
)
else:
# No breakdown at all, all tokens are text tokens
@ -820,14 +803,6 @@ def generic_cost_per_token(
)
completion_cost += float(image_tokens) * _output_cost_per_image_token
## VIDEO COST
if not is_text_tokens_total and video_tokens and video_tokens > 0:
_output_cost_per_video_token = _get_cost_per_unit(model_info, "output_cost_per_video_token", None)
_output_cost_per_video_token = (
_output_cost_per_video_token if _output_cost_per_video_token is not None else completion_base_cost
)
completion_cost += float(video_tokens) * _output_cost_per_video_token
## REGIONAL DATA-RESIDENCY UPLIFT
# Applied as a flat multiplier across all token costs for the request
# when the upstream is a regionalized OpenAI host (eu./us.api.openai.com).

53
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -5494,56 +5494,3 @@ def has_tool_with_name(tools: Any, tool_name: str) -> bool:
elif tool.get("name") == tool_name:
return True
return False
def resolve_structured_messages(
messages: list[dict[str, Any]] | None,
request_kwargs: dict[str, Any],
) -> list[dict[str, Any]] | None:
"""
Normalize a request's messages to OpenAI-spec chat-completions shape,
regardless of which API surface produced them (chat completions,
Anthropic /v1/messages, Responses API ``input``, etc).
Returns ``messages`` unchanged if already present. Otherwise dispatches
through the guardrail translation handlers (the same per-surface
conversion logic guardrails use) to convert e.g. Responses API ``input``
into a message list. Returns ``None`` if no messages could be resolved.
"""
if messages:
return messages
from litellm.litellm_core_utils.api_route_to_call_types import (
get_call_types_for_route,
)
from litellm.llms import load_guardrail_translation_mappings
from litellm.types.utils import CallTypes
mappings = load_guardrail_translation_mappings()
call_type: CallTypes | None = None
# 1. Try route-based inference from proxy metadata
route = request_kwargs.get("litellm_metadata", {}).get("user_api_key_request_route")
if route:
call_types_list = get_call_types_for_route(route)
if call_types_list:
for ct in call_types_list:
if ct in mappings:
call_type = ct
break
# 2. Fallback: try each mapped handler until one produces messages
handlers_to_try: list[Any] = []
if call_type is not None and call_type in mappings:
handlers_to_try.append(mappings[call_type]())
else:
handlers_to_try.extend(handler_cls() for handler_cls in mappings.values())
for handler in handlers_to_try:
structured = handler.get_structured_messages(request_kwargs)
if structured:
return [
msg if isinstance(msg, dict) else msg.model_dump() # type: ignore
for msg in structured
]
return None

57
litellm/litellm_core_utils/redact_messages.py
View file

@ -38,45 +38,10 @@ def redact_message_input_output_from_custom_logger(
litellm_logging_obj: LiteLLMLoggingObject, result, custom_logger: CustomLogger
):
if hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True:
return perform_redaction(litellm_logging_obj.model_call_details, result, redact_streaming_responses=False)
return perform_redaction(litellm_logging_obj.model_call_details, result)
return result
def redact_streaming_responses_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
"""
Returns a copy of model_call_details whose streaming response entries are redacted deepcopies
when the custom logger has opted out of message logging. The shared model_call_details is left
untouched so other callbacks still receive the unredacted response.
"""
if not (hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True):
return model_call_details
redacted_entries = {
streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
if model_call_details.get(streaming_key) is not None
}
if not redacted_entries:
return model_call_details
return {**model_call_details, **redacted_entries}
def _redacted_streaming_response_copy(streaming_response):
redacted_response = copy.deepcopy(streaming_response)
_redact_streaming_response(redacted_response)
return redacted_response
def _redact_streaming_response(streaming_response):
if hasattr(streaming_response, "choices"):
for choice in streaming_response.choices:
_redact_choice_content(choice)
redact_vertex_ai_metadata_from_logged_object(streaming_response)
elif hasattr(streaming_response, "output"):
_redact_responses_api_output(streaming_response.output)
if hasattr(streaming_response, "reasoning") and streaming_response.reasoning is not None:
streaming_response.reasoning = None
def _redact_choice_content(choice):
"""Helper to redact content in a choice (message or delta)."""
if isinstance(choice, litellm.Choices):
@ -185,13 +150,9 @@ def _redact_model_response_dict_choices(choices, redacted_str: str):
_redact_choice_content(choice)
def perform_redaction(model_call_details: dict, result, redact_streaming_responses: bool = True):
def perform_redaction(model_call_details: dict, result):
"""
Performs the actual redaction on the logging object and result.
redact_streaming_responses=False skips the in-place redaction of the shared streaming
response entries; per-callback redaction hands each opted-out callback its own redacted
copy via redact_streaming_responses_for_custom_logger instead.
"""
# Redact model_call_details
model_call_details["messages"] = [{"role": "user", "content": "redacted-by-litellm"}]
@ -201,9 +162,17 @@ def perform_redaction(model_call_details: dict, result, redact_streaming_respons
redact_vertex_ai_metadata_from_litellm_params(model_call_details)
# Redact streaming response
if redact_streaming_responses and model_call_details.get("stream", False) is True:
for _streaming_key in ("complete_streaming_response", "async_complete_streaming_response"):
_redact_streaming_response(model_call_details.get(_streaming_key))
if model_call_details.get("stream", False) is True and "complete_streaming_response" in model_call_details:
_streaming_response = model_call_details["complete_streaming_response"]
if hasattr(_streaming_response, "choices"):
for choice in _streaming_response.choices:
_redact_choice_content(choice)
redact_vertex_ai_metadata_from_logged_object(_streaming_response)
elif hasattr(_streaming_response, "output"):
_redact_responses_api_output(_streaming_response.output)
# Redact reasoning field in ResponsesAPIResponse
if hasattr(_streaming_response, "reasoning") and _streaming_response.reasoning is not None:
_streaming_response.reasoning = None
# Redact result
if result is not None:

35
litellm/litellm_core_utils/sensitive_data_masker.py
View file

@ -1,8 +1,6 @@
from collections.abc import Mapping
from typing import Any, Dict, List, Optional, Set
from pydantic import BaseModel
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
@ -155,39 +153,6 @@ def mask_sensitive_structure(data: object) -> object:
return _error_masker.mask(data)
def mask_credentials_in_payload(data: object) -> object:
"""Return a copy of ``data`` where string values under sensitive-named keys
are masked but every other value (``None``, ``int``, ``float``, ``bool``,
``bytes``, ``datetime``, tuples, sets, typed objects) is preserved by
identity, and dicts/lists are rebuilt structurally.
Use this for logging payloads that carry response data through to
SpendLogs / OTel / Langfuse, where :meth:`SensitiveDataMasker.mask`'s
config-dump semantics (``None`` -> ``"None"``, tuples stringified,
objects flattened via ``__dict__``) would silently distort the record.
Sensitive-key detection is delegated to the shared
:class:`SensitiveDataMasker` so pattern updates stay in one place.
"""
return _walk_payload(data, key_is_sensitive=False, depth=0)
def _walk_payload(node: object, key_is_sensitive: bool, depth: int) -> object:
if depth >= DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER:
return node
if isinstance(node, Mapping):
return {k: _walk_payload(v, _default_masker.is_sensitive_key(k), depth + 1) for k, v in node.items()}
if isinstance(node, list):
return [_walk_payload(item, key_is_sensitive, depth + 1) for item in node]
if isinstance(node, tuple):
return tuple(_walk_payload(item, key_is_sensitive, depth + 1) for item in node)
if isinstance(node, BaseModel):
return _walk_payload(node.model_dump(), key_is_sensitive, depth)
if key_is_sensitive and isinstance(node, str) and node:
return _default_masker._mask_value(node)
return node
def mask_sensitive_keys(data: Dict[str, Any], sensitive_fields: Set[str]) -> Dict[str, Any]:
"""Return a new dict with values masked for keys listed in ``sensitive_fields``.

122
litellm/llms/anthropic/chat/transformation.py
View file

@ -227,10 +227,6 @@ DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING = (
"Sonnet 4.6+, and Mythos Preview."
)
DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING = (
"Dropping adaptive `thinking` for model=%s: max_tokens is too small to fit the minimum thinking budget."
)
DROP_UNSUPPORTED_SPEED_WARNING = (
"Dropping unsupported `speed` for model=%s (drop_params=True). Fast mode is only supported on select Opus models."
)
@ -270,10 +266,6 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
def custom_llm_provider(self) -> Optional[str]:
return "anthropic"
@property
def _resolved_provider(self) -> str:
return self.custom_llm_provider or "anthropic"
@classmethod
def get_config(cls, *, model: Optional[str] = None):
config = super().get_config()
@ -343,26 +335,23 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
return any(v in model_lower for v in ("opus-4-7", "opus_4_7", "opus-4.7", "opus_4.7"))
@staticmethod
def _supports_effort_level(model: str, level: str, custom_llm_provider: str) -> bool:
def _supports_effort_level(model: str, level: str) -> bool:
"""Check ``supports_{level}_reasoning_effort`` in the model map."""
return AnthropicConfig._supports_model_capability(
model, f"supports_{level}_reasoning_effort", custom_llm_provider
)
return AnthropicConfig._supports_model_capability(model, f"supports_{level}_reasoning_effort")
@staticmethod
def _validate_effort_for_model(model: str, effort: Optional[str], custom_llm_provider: str) -> Optional[str]:
def _validate_effort_for_model(model: str, effort: Optional[str]) -> Optional[str]:
"""Return ``None`` if ``effort`` is allowed on ``model``, else an error message."""
if effort == "max" and not (
AnthropicConfig._is_adaptive_thinking_model(model, custom_llm_provider)
or AnthropicConfig._supports_effort_level(model, "max", custom_llm_provider)
AnthropicConfig._is_adaptive_thinking_model(model) or AnthropicConfig._supports_effort_level(model, "max")
):
return f"effort='max' is not supported by this model. Got model: {model}"
if effort == "xhigh" and not AnthropicConfig._supports_effort_level(model, "xhigh", custom_llm_provider):
if effort == "xhigh" and not AnthropicConfig._supports_effort_level(model, "xhigh"):
return f"effort='xhigh' is not supported by this model. Got model: {model}"
return None
@staticmethod
def _model_supports_effort_param(model: str, custom_llm_provider: str) -> bool:
def _model_supports_effort_param(model: str) -> bool:
"""Whether the model accepts ``output_config.effort`` at all.
A model qualifies if its map entry advertises ``supports_output_config``
@ -370,10 +359,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
signals: e.g. Claude Opus 4.5 supports ``output_config`` without
advertising a non-default (max/xhigh) effort level.
"""
if AnthropicConfig._supports_model_capability(model, "supports_output_config", custom_llm_provider):
if AnthropicConfig._supports_model_capability(model, "supports_output_config"):
return True
return any(
AnthropicConfig._supports_effort_level(model, level, custom_llm_provider)
AnthropicConfig._supports_effort_level(model, level)
for level in ("low", "minimal", "medium", "high", "xhigh", "max")
)
@ -462,7 +451,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if (
"claude-3-7-sonnet" in model
or AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider)
or AnthropicConfig._is_adaptive_thinking_model(model)
or supports_reasoning(
model=model,
custom_llm_provider=self.custom_llm_provider,
@ -1170,13 +1159,11 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
def _map_reasoning_effort(
reasoning_effort: Optional[Union[REASONING_EFFORT, str]],
model: str,
custom_llm_provider: str,
llm_provider: str = "anthropic",
) -> Optional[AnthropicThinkingParam]:
"""Capability probes read the cost map under ``custom_llm_provider``; ``llm_provider`` only tags raised exceptions."""
if reasoning_effort is None or reasoning_effort == "none":
return None
if AnthropicConfig._is_adaptive_thinking_model(model, custom_llm_provider):
if AnthropicConfig._is_adaptive_thinking_model(model):
return AnthropicThinkingParam(
type="adaptive",
)
@ -1224,23 +1211,6 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
llm_provider=llm_provider,
)
@staticmethod
def _cap_thinking_budget_to_max_tokens(
thinking: AnthropicThinkingParam, max_tokens: Optional[int]
) -> Optional[AnthropicThinkingParam]:
"""Cap a legacy ``thinking.budget_tokens`` below ``max_tokens`` (Anthropic
requires ``max_tokens > budget_tokens``). Returns the (possibly capped)
thinking dict, or ``None`` when ``max_tokens`` is too small to fit even the
minimum thinking budget and thinking should be dropped."""
budget = thinking.get("budget_tokens")
if max_tokens is None or not isinstance(budget, int):
return thinking
if max_tokens <= ANTHROPIC_MIN_THINKING_BUDGET_TOKENS:
return None
if budget < max_tokens:
return thinking
return AnthropicThinkingParam(type=thinking.get("type", "enabled"), budget_tokens=max_tokens - 1)
def _extract_json_schema_from_response_format(self, value: Optional[dict]) -> Optional[dict]:
if value is None:
return None
@ -1441,10 +1411,24 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
output_key=param,
)
elif param == "response_format" and isinstance(value, dict):
if AnthropicConfig._supports_model_capability(
model,
"supports_native_structured_output",
self._resolved_provider,
if any(
substring in model
for substring in {
"sonnet-4.5",
"sonnet-4-5",
"opus-4.1",
"opus-4-1",
"opus-4.5",
"opus-4-5",
"opus-4.6",
"opus-4-6",
"opus-4.7",
"opus-4-7",
"sonnet-4.6",
"sonnet-4-6",
"sonnet_4.6",
"sonnet_4_6",
}
):
_output_format = self.map_response_format_to_anthropic_output_format(value)
if _output_format is not None:
@ -1470,38 +1454,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
):
optional_params["metadata"] = {"user_id": value}
elif param == "thinking":
if (
isinstance(value, dict)
and value.get("type") == "adaptive"
and not AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider)
):
# Callers (e.g. Claude Code) send adaptive thinking
# unconditionally; translate it down to the legacy
# `thinking={type: enabled, budget_tokens}` interface a
# pre-4.6 model actually supports instead of forwarding a
# shape the model will reject.
max_tokens = non_default_params.get("max_completion_tokens") or non_default_params.get("max_tokens")
legacy_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort="medium",
model=model,
custom_llm_provider=self._resolved_provider,
llm_provider=self._resolved_provider,
)
capped_thinking = (
AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
if legacy_thinking is not None
else None
)
if capped_thinking is not None:
optional_params["thinking"] = capped_thinking
else:
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING,
model,
)
optional_params.pop("thinking", None)
else:
optional_params["thinking"] = value
optional_params["thinking"] = value
elif param == "reasoning_effort":
# Accept both string ("low") and dict ({"effort": "low",
# "summary": "concise"}). The Responses->Chat parser keeps the
@ -1518,21 +1471,20 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
mapped_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort=effort_value,
model=model,
custom_llm_provider=self._resolved_provider,
llm_provider=self._resolved_provider,
llm_provider=self.custom_llm_provider or "anthropic",
)
if mapped_thinking is None:
optional_params.pop("thinking", None)
optional_params.pop("output_config", None)
else:
optional_params["thinking"] = mapped_thinking
if AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider):
if AnthropicConfig._is_adaptive_thinking_model(model):
mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(effort_value)
if mapped_effort is None:
AnthropicConfig._raise_invalid_reasoning_effort(
model=model,
value=effort_value,
llm_provider=self._resolved_provider,
llm_provider=self.custom_llm_provider or "anthropic",
)
optional_params["output_config"] = {"effort": mapped_effort}
elif param == "web_search_options" and isinstance(value, dict):
@ -1861,7 +1813,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
anthropic_messages = anthropic_messages_pt(
model=model,
messages=messages,
llm_provider=self._resolved_provider,
llm_provider=self.custom_llm_provider or "anthropic",
)
except Exception as e:
raise AnthropicError(
@ -1950,7 +1902,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
output_config = optional_params.get("output_config")
if not output_config or not isinstance(output_config, dict):
return
if litellm.drop_params is True and not self._model_supports_effort_param(model, self._resolved_provider):
if litellm.drop_params is True and not self._model_supports_effort_param(model):
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
model,
@ -1964,14 +1916,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
raise litellm.exceptions.BadRequestError(
message=(f"Invalid effort value: {effort!r}. Must be one of: 'high', 'medium', 'low', 'xhigh', 'max'"),
model=model,
llm_provider=self._resolved_provider,
llm_provider=self.custom_llm_provider or "anthropic",
)
gate_error = self._validate_effort_for_model(model, effort, self._resolved_provider)
gate_error = self._validate_effort_for_model(model, effort)
if gate_error is not None:
raise litellm.exceptions.BadRequestError(
message=gate_error,
model=model,
llm_provider=self._resolved_provider,
llm_provider=self.custom_llm_provider or "anthropic",
)
data["output_config"] = output_config

82
litellm/llms/anthropic/common_utils.py
View file

@ -289,13 +289,6 @@ class AnthropicModelInfo(BaseLLMModelInfo):
status_code=400,
)
@staticmethod
def _strip_version_suffix(model: str) -> str:
at = model.rfind("@")
if at > 0:
return model[:at]
return model
@staticmethod
def _model_map_lookup_candidates(model: str) -> List[str]:
"""Model-map keys to try for ``model``: the id itself, the same id with a
@ -331,7 +324,6 @@ class AnthropicModelInfo(BaseLLMModelInfo):
_DATED_RELEASE_SUFFIX_RE.sub("", cand),
_DOTTED_VERSION_RE.sub(r"\1-\2", cand),
_strip_bedrock_id_suffixes(cand),
AnthropicModelInfo._strip_version_suffix(cand),
)
)
return list(dict.fromkeys((*primary, *normalized)))
@ -340,15 +332,11 @@ class AnthropicModelInfo(BaseLLMModelInfo):
def _get_model_capability(model: str, key: str) -> Optional[bool]:
"""Read boolean capability ``key`` from the model map, or None when
no entry declares it."""
from litellm.utils import _get_bundled_model_cost_map
try:
candidates = AnthropicModelInfo._model_map_lookup_candidates(model)
for model_cost in (litellm.model_cost, _get_bundled_model_cost_map()):
for cand in candidates:
value = model_cost.get(cand, {}).get(key)
if isinstance(value, bool):
return value
for cand in AnthropicModelInfo._model_map_lookup_candidates(model):
value = litellm.model_cost.get(cand, {}).get(key)
if isinstance(value, bool):
return value
except Exception:
pass
return None
@ -364,43 +352,18 @@ class AnthropicModelInfo(BaseLLMModelInfo):
return value if isinstance(value, bool) else None
@staticmethod
def _get_provider_resolved_capability(model: str, key: str, custom_llm_provider: str) -> Optional[bool]:
"""Resolve boolean capability ``key`` for ``model`` under the caller's provider.
def _supports_model_capability(model: str, key: str) -> bool:
"""Check a boolean capability ``key`` in the model map.
Returns the flag when the provider-aware lookup resolves ``model`` to an
entry (or fallback rule) that sets it explicitly, and ``None`` when the
model does not resolve under that provider or the resolved entry has no
opinion on ``key``.
"""
from litellm.utils import _get_model_info_helper
try:
resolved_model, resolved_provider, _, _ = litellm.get_llm_provider(
model=model, custom_llm_provider=custom_llm_provider
)
value = _get_model_info_helper(model=resolved_model, custom_llm_provider=resolved_provider).get(key)
except Exception: # noqa: BLE001 # _get_model_info_helper raises bare Exception for unmapped models
return None
return value if isinstance(value, bool) else None
@staticmethod
def _supports_model_capability(model: str, key: str, custom_llm_provider: str) -> bool:
"""Check a boolean capability ``key`` in the model map under the caller's provider.
The provider-aware lookup is authoritative when it resolves an explicit flag,
so ``key: false`` on the provider-namespaced entry wins over every fallback.
Otherwise ``_supports_factory``'s provider-level fallbacks and the raw
model-map walk remain as backstops for alias forms the lookup misses.
Strips bedrock/vertex prefixes so a provider-routed Claude still
resolves to the Anthropic model-map entry.
"""
from litellm.utils import _supports_factory
resolved = AnthropicModelInfo._get_provider_resolved_capability(model, key, custom_llm_provider)
if resolved is not None:
return resolved
try:
if _supports_factory(
model=model,
custom_llm_provider=custom_llm_provider,
custom_llm_provider="anthropic",
key=key,
):
return True
@ -409,24 +372,17 @@ class AnthropicModelInfo(BaseLLMModelInfo):
return AnthropicModelInfo._get_model_capability(model, key) is True
@staticmethod
def _is_adaptive_thinking_model(model: str, custom_llm_provider: str) -> bool:
def _is_adaptive_thinking_model(model: str) -> bool:
"""Whether ``model`` uses adaptive thinking (``output_config.effort``).
The model cost map is authoritative: an explicit ``supports_adaptive_thinking``
entry resolved under ``custom_llm_provider``, or a ``fallback_generalizations``
rule for unknown Claude models. The version gate (>= 4.6, including
provider-prefixed Bedrock/Vertex ids that map to no exact entry) lives entirely
in that declarative rule, not here.
entry, or a ``fallback_generalizations`` rule for unknown Claude models. The
version gate (>= 4.6, including provider-prefixed Bedrock/Vertex ids that map to
no exact entry) lives entirely in that declarative rule, not here.
"""
return AnthropicModelInfo._supports_model_capability(model, "supports_adaptive_thinking", custom_llm_provider)
return AnthropicModelInfo._supports_model_capability(model, "supports_adaptive_thinking")
def is_effort_used(
self,
optional_params: Optional[dict],
model: Optional[str] = None,
*,
custom_llm_provider: str,
) -> bool:
def is_effort_used(self, optional_params: Optional[dict], model: Optional[str] = None) -> bool:
"""
Check if effort parameter is being used and requires a beta header.
@ -438,7 +394,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
return False
# Claude 4.6+ models use output_config as a stable API feature — no beta header needed
if model and self._is_adaptive_thinking_model(model, custom_llm_provider):
if model and self._is_adaptive_thinking_model(model):
return False
# Check if reasoning_effort is provided for Claude Opus 4.5
@ -519,8 +475,6 @@ class AnthropicModelInfo(BaseLLMModelInfo):
prompt_caching_set: bool = False,
file_id_used: bool = False,
mcp_server_used: bool = False,
*,
custom_llm_provider: str,
) -> List[str]:
"""
Get list of common beta headers based on the features that are active.
@ -533,7 +487,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
betas = []
# Detect features
effort_used = self.is_effort_used(optional_params, model, custom_llm_provider=custom_llm_provider)
effort_used = self.is_effort_used(optional_params, model)
if effort_used:
betas.append(ANTHROPIC_EFFORT_BETA_HEADER) # effort-2025-11-24
@ -689,7 +643,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
tool_search_used = self.is_tool_search_used(tools=tools)
programmatic_tool_calling_used = self.is_programmatic_tool_calling_used(tools=tools)
input_examples_used = self.is_input_examples_used(tools=tools)
effort_used = self.is_effort_used(optional_params=optional_params, model=model, custom_llm_provider="anthropic")
effort_used = self.is_effort_used(optional_params=optional_params, model=model)
code_execution_tool_used = self.is_code_execution_tool_used(tools=tools)
container_with_skills_used = self.is_container_with_skills_used(optional_params=optional_params)
user_anthropic_beta_headers = self._get_user_anthropic_beta_headers(

181
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
View file

@ -32,22 +32,8 @@ from ...common_utils import (
DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
DROP_UNSUPPORTED_ADAPTIVE_EFFORT_WARNING = (
"Dropping adaptive `thinking`/`output_config.effort` for model=%s: the model "
"does not support extended thinking, or max_tokens is too small to fit the "
"minimum thinking budget."
)
class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
@property
def custom_llm_provider(self) -> Optional[str]:
return "anthropic"
@property
def _resolved_provider(self) -> str:
return self.custom_llm_provider or "anthropic"
def get_supported_anthropic_messages_params(self, model: str) -> list:
return [
"messages",
@ -188,7 +174,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
return headers, api_base
@staticmethod
def _translate_reasoning_effort_to_anthropic(model: str, optional_params: Dict, custom_llm_provider: str) -> None:
def _translate_reasoning_effort_to_anthropic(model: str, optional_params: Dict) -> None:
"""Map OpenAI-style ``reasoning_effort`` to native Anthropic params.
Caller-supplied ``thinking`` / ``output_config`` win over the alias.
@ -205,11 +191,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
return
try:
mapped_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort=reasoning_effort,
model=model,
custom_llm_provider=custom_llm_provider,
)
mapped_thinking = AnthropicConfig._map_reasoning_effort(reasoning_effort=reasoning_effort, model=model)
except _BadRequestError as e:
raise AnthropicError(message=str(e.message), status_code=400)
@ -219,7 +201,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
return
optional_params.setdefault("thinking", mapped_thinking)
if AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
if AnthropicModelInfo._is_adaptive_thinking_model(model):
mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(reasoning_effort)
if mapped_effort is None:
raise AnthropicError(
@ -230,7 +212,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
),
status_code=400,
)
gate_error = AnthropicConfig._validate_effort_for_model(model, mapped_effort, custom_llm_provider)
gate_error = AnthropicConfig._validate_effort_for_model(model, mapped_effort)
if gate_error is not None:
raise AnthropicError(message=gate_error, status_code=400)
existing_output_config = optional_params.get("output_config")
@ -240,15 +222,13 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
optional_params["output_config"] = existing_output_config
@staticmethod
def _translate_legacy_thinking_for_adaptive_model(
model: str, optional_params: Dict, custom_llm_provider: str
) -> None:
def _translate_legacy_thinking_for_adaptive_model(model: str, optional_params: Dict) -> None:
"""Translate legacy ``thinking.type=enabled`` to adaptive for 4.6/4.7.
Caller-provided ``output_config.effort`` is never overridden.
"""
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
if not AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
if not AnthropicModelInfo._is_adaptive_thinking_model(model):
return
thinking = optional_params.get("thinking")
if not isinstance(thinking, dict) or thinking.get("type") != "enabled":
@ -256,7 +236,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
budget = int(thinking.get("budget_tokens") or 0)
if budget >= DEFAULT_REASONING_EFFORT_XHIGH_THINKING_BUDGET and (
AnthropicConfig._supports_effort_level(model, "xhigh", custom_llm_provider)
AnthropicConfig._supports_effort_level(model, "xhigh")
):
effort = "xhigh"
elif budget >= DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET:
@ -273,138 +253,6 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
existing_output_config.setdefault("effort", effort)
optional_params["output_config"] = existing_output_config
@staticmethod
def _translate_adaptive_effort_for_non_adaptive_model(
model: str, optional_params: Dict, max_tokens: Optional[int], custom_llm_provider: str
) -> None:
"""Translate the 4.6+ adaptive-thinking interface (``thinking.type=adaptive``
and/or ``output_config.effort``) down to what an older Anthropic model
supports. Clients like Claude Code send this interface unconditionally, so
without translation it reaches a pre-4.6 model and Anthropic rejects it with
"This model does not support the effort parameter".
The reshape is silent, matching how the messages path already strips
unsupported ``output_config`` for older models (bedrock invoke, issue
#22797): the goal is to keep the request working, not to fail it.
``thinking.type=adaptive`` and ``output_config.effort`` are independent
capabilities. Adaptive thinking needs ``supports_adaptive_thinking`` (4.6+);
``output_config.effort`` needs ``supports_output_config``, which some
non-adaptive models (e.g. Claude Opus 4.5) advertise on its own. So the two
are handled separately:
- Adaptive-thinking models (4.6+): both are native, left untouched.
- ``supports_output_config`` but non-adaptive (Opus 4.5): keep
``output_config.effort`` (native), only drop the unsupported adaptive
``thinking`` block. When adaptive thinking is being dropped and the
effort level itself isn't supported by the model (e.g. ``xhigh``/``max``
on Opus 4.5, which only accepts low/medium/high, while ``xhigh`` is
Claude Code's default), fall through to the legacy translation below
instead of forwarding a level Anthropic would reject. Effort-only
requests are always left untouched: provider subclasses own their level
normalization (bedrock clamps ``xhigh`` to the model's ceiling after
this base transform runs).
- Thinking-capable but neither (``supports_reasoning``, e.g. Haiku/Sonnet
4.5): map effort to legacy ``thinking={type: enabled, budget_tokens}`` via
``AnthropicConfig._map_reasoning_effort``, capped below ``max_tokens``
(Anthropic requires ``max_tokens > budget_tokens``) and dropped when
``max_tokens`` can't fit even the minimum budget.
- No reasoning support: ``thinking`` is dropped.
For the last two, only the consumed ``effort`` key is removed from
``output_config``; any residual (e.g. ``format``) is left for provider
subclasses to handle.
"""
from litellm.exceptions import BadRequestError as _BadRequestError
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
if AnthropicConfig._is_adaptive_thinking_model(model, custom_llm_provider):
return
output_config = optional_params.get("output_config")
thinking = optional_params.get("thinking")
effort = output_config.get("effort") if isinstance(output_config, dict) else None
adaptive_thinking = isinstance(thinking, dict) and thinking.get("type") == "adaptive"
if effort is None and not adaptive_thinking:
return
# Models that natively accept `output_config.effort` but are not adaptive (Claude Opus 4.5).
# Keep the native effort and only drop the adaptive `thinking` block, which these models
# reject. Effort-only requests pass through so provider subclasses (bedrock/vertex) keep
# owning level clamping; an adaptive request only stays here when its effort level is one
# the model supports, otherwise it falls through to the legacy budget translation below.
if AnthropicConfig._model_supports_effort_param(model, custom_llm_provider) and (
not adaptive_thinking
or AnthropicConfig._validate_effort_for_model(model, effort, custom_llm_provider) is None
):
if adaptive_thinking:
optional_params.pop("thinking", None)
return
supports_thinking = AnthropicModelInfo._supports_model_capability(
model, "supports_reasoning", custom_llm_provider
)
try:
legacy_thinking = (
AnthropicConfig._map_reasoning_effort(
reasoning_effort=effort or "medium",
model=model,
custom_llm_provider=custom_llm_provider,
)
if supports_thinking
else None
)
except _BadRequestError as e:
raise AnthropicError(message=str(e.message), status_code=400)
capped_thinking = (
AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
if legacy_thinking is not None
else None
)
if capped_thinking is not None:
optional_params["thinking"] = capped_thinking
else:
verbose_logger.warning(DROP_UNSUPPORTED_ADAPTIVE_EFFORT_WARNING, model)
optional_params.pop("thinking", None)
if isinstance(output_config, dict) and "effort" in output_config:
residual = {k: v for k, v in output_config.items() if k != "effort"}
if residual:
optional_params["output_config"] = residual
else:
optional_params.pop("output_config", None)
@staticmethod
def _drop_incompatible_temperature_for_thinking(
model: str, optional_params: dict, custom_llm_provider: str
) -> None:
"""Anthropic rejects any ``temperature`` other than 1 while extended thinking
is enabled ("temperature may only be set to 1 when thinking is enabled").
Clients like Claude Code send ``thinking``/``output_config.effort`` together
with a pinned ``temperature`` (e.g. the safety classifier uses ``temperature=0``
for determinism). When the request lands on a non-adaptive model, the effort
interface is reshaped above into legacy ``thinking={type: enabled}`` (or kept
as ``output_config.effort`` on Opus 4.5), and the leftover ``temperature`` would
400. Preserving the thinking the caller asked for wins over an unhonorable
sampling value (Anthropic forces ``temperature=1`` under thinking regardless),
so drop it and let the API default apply.
Adaptive models (4.6+) own this natively and are left untouched.
"""
if AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
return
temperature = optional_params.get("temperature")
if temperature is None or temperature == 1:
return
thinking = optional_params.get("thinking")
output_config = optional_params.get("output_config")
thinking_enabled = isinstance(thinking, dict) and thinking.get("type") == "enabled"
effort_enabled = isinstance(output_config, dict) and output_config.get("effort") is not None
if thinking_enabled or effort_enabled:
optional_params.pop("temperature", None)
def transform_anthropic_messages_request(
self,
model: str,
@ -429,26 +277,11 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
self._translate_reasoning_effort_to_anthropic(
model=model,
optional_params=anthropic_messages_optional_request_params,
custom_llm_provider=self._resolved_provider,
)
self._translate_legacy_thinking_for_adaptive_model(
model=model,
optional_params=anthropic_messages_optional_request_params,
custom_llm_provider=self._resolved_provider,
)
self._translate_adaptive_effort_for_non_adaptive_model(
model=model,
optional_params=anthropic_messages_optional_request_params,
max_tokens=max_tokens,
custom_llm_provider=self._resolved_provider,
)
self._drop_incompatible_temperature_for_thinking(
model=model,
optional_params=anthropic_messages_optional_request_params,
custom_llm_provider=self._resolved_provider,
)
system_param = anthropic_messages_optional_request_params.get("system")

8
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py
View file

@ -198,16 +198,14 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
@staticmethod
def translate_tool_choice_to_responses_api(
tool_choice: AnthropicMessagesToolChoice,
) -> Union[str, dict[str, Any]]:
) -> Dict[str, Any]:
"""Convert Anthropic tool_choice to Responses API tool_choice."""
tc_type = tool_choice.get("type")
if tc_type == "any":
return "required"
return {"type": "required"}
elif tc_type == "tool":
return {"type": "function", "name": tool_choice.get("name", "")}
elif tc_type == "none":
return "none"
return "auto"
return {"type": "auto"}
@staticmethod
def translate_context_management_to_responses_api(

4
litellm/llms/azure_ai/anthropic/messages_transformation.py
View file

@ -21,10 +21,6 @@ class AzureAnthropicMessagesConfig(AnthropicMessagesConfig):
and Azure endpoint format.
"""
@property
def custom_llm_provider(self) -> Optional[str]:
return "azure_ai"
def should_strip_billing_metadata(self) -> bool:
return True

38
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -1,5 +1,4 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Dict, List, Optional
if TYPE_CHECKING:
@ -12,30 +11,10 @@ if TYPE_CHECKING:
from litellm.types.llms.openai import AllMessageValues
@dataclass(slots=True)
class StreamTransformSink:
"""Out-parameter used by ``process_output_streaming_response`` to hand the
guardrailed streaming state back to the caller.
The streaming text-transform path must not mutate ``responses_so_far`` (it is
the raw accumulator the guardrail re-reads every round), so the guardrailed
accumulated text per choice (``mutated_text_per_choice``, keyed by
``StreamingChoices.index``) and the per-choice trailing holdback the guardrail
requested (``holdback_per_choice``, from ``stream_holdback_chars``) are
reported here instead of in place. Only the OpenAI chat handler populates this
today; the hook passes a fresh sink per round and reads it afterwards. A
mutable dataclass is deliberate: it is a write-once output parameter for a
single call, not shared state.
"""
mutated_text_per_choice: dict[int, str] = field(default_factory=dict)
holdback_per_choice: dict[int, int] = field(default_factory=dict)
class BaseTranslation(ABC):
@staticmethod
def transform_user_api_key_dict_to_metadata(
user_api_key_dict: Any | None,
user_api_key_dict: Optional[Any],
) -> Dict[str, Any]:
"""
Transform user_api_key_dict to a metadata dict with prefixed keys.
@ -94,7 +73,7 @@ class BaseTranslation(ABC):
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
request_data: dict | None = None,
request_data: Optional[dict] = None,
) -> Any:
"""
Process output response with guardrails.
@ -113,15 +92,12 @@ class BaseTranslation(ABC):
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
request_data: dict | None = None,
stream_transform_sink: StreamTransformSink | None = None,
request_data: Optional[dict] = None,
) -> Any:
"""
Process output streaming response with guardrails.
Optional to override in subclasses. ``stream_transform_sink`` is the
out-parameter used by handlers that support streaming text
transformations (see ``StreamTransformSink``); base handlers ignore it.
Optional to override in subclasses.
"""
return responses_so_far
@ -129,8 +105,8 @@ class BaseTranslation(ABC):
self,
exc: "ModifyResponseException",
stream_started: bool = False,
responses_so_far: list[Any] | None = None,
) -> list[bytes] | None:
responses_so_far: Optional[list[Any]] = None,
) -> Optional[list[bytes]]:
"""
Build the streaming chunks that deliver a guardrail block message and
cleanly terminate the stream in this provider's wire format.
@ -149,7 +125,7 @@ class BaseTranslation(ABC):
"""
return None
def get_structured_messages(self, data: dict) -> List["AllMessageValues"] | None:
def get_structured_messages(self, data: dict) -> Optional[List["AllMessageValues"]]:
"""
Convert request data to OpenAI-spec structured messages.

9
litellm/llms/bedrock/base_aws_llm.py
View file

@ -877,15 +877,6 @@ class BaseAWSLLM:
"Resource": "*",
"Condition": {"Bool": {"aws:SecureTransport": "true"}},
},
{
"Sid": "BedrockMantleLiteLLM",
"Effect": "Allow",
"Action": [
"bedrock-mantle:CreateInference",
],
"Resource": "*",
"Condition": {"Bool": {"aws:SecureTransport": "true"}},
},
],
}
assume_role_params = {

47
litellm/llms/bedrock/batches/transformation.py
View file

@ -5,9 +5,6 @@ from typing import Any, Dict, List, Literal, Optional, Union, cast
from httpx import Headers, Response
from litellm.litellm_core_utils.cloud_storage_security import (
BEDROCK_MANAGED_S3_BATCH_PREFIX,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.llms.base_llm.batches.transformation import BaseBatchesConfig
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@ -29,15 +26,6 @@ from litellm.types.utils import LiteLLMBatch, LlmProviders
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import CommonBatchFilesUtils
# Bedrock batch input files are uploaded as
# s3://bucket/litellm-bedrock-files-{model, ":" -> "-"}-{uuid4}.jsonl (see
# BedrockFilesTransformation._get_s3_object_name). A uuid4 is always 36 hex/dash
# characters, so it can be stripped off the end unambiguously even though the
# model name itself may contain dashes.
_S3_BATCH_FILE_UUID_SUFFIX_PATTERN = re.compile(
r"-[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\.jsonl$"
)
class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
"""
@ -52,41 +40,6 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
def custom_llm_provider(self) -> LlmProviders:
return LlmProviders.BEDROCK
@classmethod
def _get_bare_model_name_from_s3_key(cls, object_key: str) -> Optional[str]:
if not object_key.startswith(BEDROCK_MANAGED_S3_BATCH_PREFIX):
return None
model_part = object_key[len(BEDROCK_MANAGED_S3_BATCH_PREFIX) :]
match = _S3_BATCH_FILE_UUID_SUFFIX_PATTERN.search(model_part)
if not match or match.start() == 0:
return None
return model_part[: match.start()]
@classmethod
def is_unmanaged_s3_batch_input_file_id(cls, input_file_id: Optional[str]) -> bool:
"""
Returns True if `input_file_id` is a raw s3:// Bedrock batch input file (i.e. not a
LiteLLM-managed unified file id) whose object key embeds the model name in the
`litellm-bedrock-files-{model}-{uuid}.jsonl` layout.
"""
if input_file_id is None or not input_file_id.startswith("s3://"):
return False
object_key = input_file_id.rsplit("/", 1)[-1]
return cls._get_bare_model_name_from_s3_key(object_key) is not None
@classmethod
def get_bare_model_name_from_s3_file(cls, input_file_id: str) -> str:
"""
Extracts the bare model name (e.g. "us.anthropic.claude-sonnet-4-20250514-v1-0") from
an unmanaged batch's s3:// input file id. Note any ":" in the original model id was
replaced with "-" at upload time, so callers must fuzzy-match against configured
deployments rather than expect an exact string match.
"""
object_key = input_file_id.rsplit("/", 1)[-1]
bare_model_name = cls._get_bare_model_name_from_s3_key(object_key)
assert bare_model_name is not None # narrowed by is_unmanaged_s3_batch_input_file_id
return bare_model_name
def validate_environment(
self,
headers: dict,

33
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -33,7 +33,6 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
make_valid_bedrock_tool_name,
)
from litellm.llms.anthropic.chat.transformation import (
DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING,
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT,
AnthropicConfig,
@ -424,7 +423,6 @@ class AmazonConverseConfig(BaseConfig):
mapped_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort=reasoning_effort,
model=model,
custom_llm_provider="bedrock",
llm_provider="bedrock_converse",
)
if mapped_thinking is None:
@ -432,7 +430,7 @@ class AmazonConverseConfig(BaseConfig):
optional_params.pop("output_config", None)
else:
optional_params["thinking"] = mapped_thinking
if AnthropicConfig._is_adaptive_thinking_model(model, "bedrock"):
if AnthropicConfig._is_adaptive_thinking_model(model):
mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(reasoning_effort)
if mapped_effort is None:
AnthropicConfig._raise_invalid_reasoning_effort(
@ -467,7 +465,7 @@ class AmazonConverseConfig(BaseConfig):
model=model,
llm_provider="bedrock_converse",
)
error = AnthropicConfig._validate_effort_for_model(model=model, effort=effort, custom_llm_provider="bedrock")
error = AnthropicConfig._validate_effort_for_model(model=model, effort=effort)
if error is not None:
raise litellm.exceptions.BadRequestError(
message=error,
@ -900,28 +898,7 @@ class AmazonConverseConfig(BaseConfig):
"tool_choice": {"disable_parallel_tool_use": disable_parallel}
}
if param == "thinking":
if (
isinstance(value, dict)
and value.get("type") == "adaptive"
and not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock")
):
max_tokens = non_default_params.get("max_completion_tokens") or non_default_params.get("max_tokens")
legacy_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort="medium",
model=model,
custom_llm_provider="bedrock",
)
capped = (
AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
if legacy_thinking is not None
else None
)
if capped is not None:
optional_params["thinking"] = capped
else:
litellm.verbose_logger.warning(DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING, model)
else:
optional_params["thinking"] = value
optional_params["thinking"] = value
elif param == "reasoning_effort" and isinstance(value, str):
self._handle_reasoning_effort_parameter(
model=model, reasoning_effort=value, optional_params=optional_params
@ -1302,7 +1279,7 @@ class AmazonConverseConfig(BaseConfig):
if anthropic_output_config is not None and isinstance(anthropic_output_config, dict):
if base_model.startswith("anthropic"):
if litellm.drop_params is True and not AnthropicConfig._model_supports_effort_param(model, "bedrock"):
if litellm.drop_params is True and not AnthropicConfig._model_supports_effort_param(model):
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
model,
@ -1445,7 +1422,7 @@ class AmazonConverseConfig(BaseConfig):
if (
isinstance(output_config, dict)
and output_config.get("effort") is not None
and not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock")
and not AnthropicConfig._is_adaptive_thinking_model(model)
):
from litellm.types.llms.anthropic import (
ANTHROPIC_EFFORT_BETA_HEADER,

5
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -115,7 +115,7 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
keeps working. Non-adaptive models and models without a ceiling are
left untouched.
"""
if not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock"):
if not AnthropicConfig._is_adaptive_thinking_model(model):
return
effort = params.get("reasoning_effort")
if not isinstance(effort, str):
@ -228,7 +228,7 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
custom_llm_provider="bedrock",
key="supports_output_config",
)
or AnthropicConfig._model_supports_effort_param(model, "bedrock")
or AnthropicConfig._model_supports_effort_param(model)
):
if anthropic_request.pop("output_config", None) is not None:
verbose_logger.warning(
@ -269,7 +269,6 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
prompt_caching_set=False,
file_id_used=self.is_file_id_used(messages),
mcp_server_used=self.is_mcp_server_used(optional_params.get("mcp_servers")),
custom_llm_provider="bedrock",
)
beta_set.update(auto_betas)

4
litellm/llms/bedrock/claude_platform/transformation.py
View file

@ -54,9 +54,7 @@ class BedrockClaudePlatformConfig(BedrockClaudePlatformMixin, AnthropicConfig):
tool_search_used=self.is_tool_search_used(tools=optional_params.get("tools")),
programmatic_tool_calling_used=self.is_programmatic_tool_calling_used(tools=optional_params.get("tools")),
input_examples_used=self.is_input_examples_used(tools=optional_params.get("tools")),
effort_used=self.is_effort_used(
optional_params=optional_params, model=model, custom_llm_provider="anthropic"
),
effort_used=self.is_effort_used(optional_params=optional_params, model=model),
user_anthropic_beta_headers=self._get_user_anthropic_beta_headers(
anthropic_beta_header=headers.get("anthropic-beta")
),

65
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -77,10 +77,6 @@ class AmazonAnthropicClaudeMessagesConfig(
DEFAULT_BEDROCK_ANTHROPIC_API_VERSION = "bedrock-2023-05-31"
@property
def custom_llm_provider(self) -> Optional[str]:
return "bedrock"
BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS = frozenset(BedrockInvokeAnthropicMessagesRequest.__annotations__.keys())
def __init__(self, **kwargs):
@ -97,48 +93,26 @@ class AmazonAnthropicClaudeMessagesConfig(
return [{"type": "text", "text": value}]
return [value]
@staticmethod
def _is_system_role_message(message: Any) -> bool:
return isinstance(message, dict) and message.get("role") == "system"
def _normalize_system_role_messages_for_bedrock(self, anthropic_messages_request: dict, model: str) -> None:
"""Bedrock Invoke validates ``role: "system"`` entries inside ``messages``
per model. Models carrying ``supports_mid_conversation_system`` in the
cost map (the Opus 4.8 family) only reject a leading run ("messages.0:
use the top-level 'system' parameter for the initial system prompt") and
accept mid-conversation entries (e.g. Claude Code's
``mid-conversation-system-2026-04-07`` reminders) in place, where they
MUST stay: hoisting one mutates the ``system`` prefix and invalidates the
prompt cache for the entire message history. Older Claude models (Opus
4.7, Sonnet 4.6, Haiku 4.5, ...) reject the role in every position
("role 'system' is not supported on this model"), so without the flag
every system entry is hoisted into the top-level ``system`` field.
Billing-header system blocks are stripped from the top-level ``system``
field regardless of whether anything was hoisted."""
def _normalize_system_role_messages_for_bedrock(self, anthropic_messages_request: dict) -> None:
"""Bedrock Invoke rejects ``role: "system"`` entries inside ``messages`` on
some Claude aliases; Anthropic Messages carries that content in the
top-level ``system`` field. Move any such entries into ``system`` before
the Invoke request is built."""
messages = anthropic_messages_request.get("messages")
if not isinstance(messages, list):
return
if _supports_factory(
model=model,
custom_llm_provider="bedrock",
key="supports_mid_conversation_system",
):
leading_count = next(
(i for i, m in enumerate(messages) if not self._is_system_role_message(m)),
len(messages),
)
hoisted = messages[:leading_count]
remaining = messages[leading_count:]
else:
hoisted = [m for m in messages if self._is_system_role_message(m)]
remaining = [m for m in messages if not self._is_system_role_message(m)]
if hoisted:
anthropic_messages_request["messages"] = remaining
system_role_messages = [m for m in messages if isinstance(m, dict) and m.get("role") == "system"]
if not system_role_messages:
return
anthropic_messages_request["messages"] = [
m for m in messages if not (isinstance(m, dict) and m.get("role") == "system")
]
system_content = [
block
for source in (
anthropic_messages_request.get("system"),
*(m.get("content") for m in hoisted),
*(m.get("content") for m in system_role_messages),
)
for block in self._as_system_content_blocks(source)
]
@ -273,7 +247,7 @@ class AmazonAnthropicClaudeMessagesConfig(
Returns:
True if the model supports extended thinking on Bedrock
"""
if AnthropicModelInfo._is_adaptive_thinking_model(model, "bedrock"):
if AnthropicModelInfo._is_adaptive_thinking_model(model):
return True
model_lower = model.lower()
@ -323,7 +297,7 @@ class AmazonAnthropicClaudeMessagesConfig(
if not self._supports_extended_thinking_on_bedrock(model):
return False
is_adaptive_thinking_model = AnthropicModelInfo._is_adaptive_thinking_model(model, "bedrock")
is_adaptive_thinking_model = AnthropicModelInfo._is_adaptive_thinking_model(model)
thinking = anthropic_messages_request.get("thinking")
if isinstance(thinking, dict):
@ -600,7 +574,6 @@ class AmazonAnthropicClaudeMessagesConfig(
mcp_server_used=anthropic_model_info.is_mcp_server_used(
anthropic_messages_optional_request_params.get("mcp_servers")
),
custom_llm_provider="bedrock",
)
beta_set.update(auto_betas)
@ -667,7 +640,7 @@ class AmazonAnthropicClaudeMessagesConfig(
path degrades ``xhigh`` -> ``max`` rather than 400-ing. Non-adaptive models
and models without a ceiling are left untouched.
"""
if not AnthropicModelInfo._is_adaptive_thinking_model(model, "bedrock"):
if not AnthropicModelInfo._is_adaptive_thinking_model(model):
return
effort = optional_params.get("reasoning_effort")
if not isinstance(effort, str):
@ -696,7 +669,7 @@ class AmazonAnthropicClaudeMessagesConfig(
litellm_params=litellm_params,
headers=headers,
)
self._normalize_system_role_messages_for_bedrock(anthropic_messages_request, model=model)
self._normalize_system_role_messages_for_bedrock(anthropic_messages_request)
#########################################################
############## BEDROCK Invoke SPECIFIC TRANSFORMATION ###
#########################################################
@ -755,7 +728,7 @@ class AmazonAnthropicClaudeMessagesConfig(
custom_llm_provider="bedrock",
key="supports_output_config",
)
or AnthropicConfig._model_supports_effort_param(model, "bedrock")
or AnthropicConfig._model_supports_effort_param(model)
):
if anthropic_messages_request.pop("output_config", None) is not None:
verbose_logger.warning(
@ -792,7 +765,7 @@ class AmazonAnthropicClaudeMessagesConfig(
if (
litellm.drop_params is True
and "output_config" in anthropic_messages_request
and not AnthropicConfig._model_supports_effort_param(model, "bedrock")
and not AnthropicConfig._model_supports_effort_param(model)
):
verbose_logger.warning(
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,

83
litellm/llms/dashscope/cost_calculator.py
View file

@ -7,7 +7,6 @@ Handles tiered pricing and prompt caching scenarios.
from dataclasses import dataclass
from typing import List, Optional, Tuple
from litellm.litellm_core_utils.llm_cost_calc.tiered_pricing import calculate_tiered_cost
from litellm.types.utils import ModelInfo, Usage
from litellm.utils import get_model_info
@ -43,6 +42,80 @@ def _extract_token_breakdown(usage: Usage) -> TokenBreakdown:
return TokenBreakdown(text_tokens, cached_tokens, completion_tokens, reasoning_tokens)
def _calculate_tiered_cost(
tokens: int,
tiered_pricing: List[dict],
cost_key: str,
fallback_cost_key: Optional[str] = None,
) -> float:
"""
Calculate cost for a given number of tokens based on a true tiered pricing structure.
This function iterates through sorted pricing tiers, calculates the cost for the
number of tokens that fall into each tier's range, and sums them up to get the total cost.
Args:
tokens (int): The total number of tokens to calculate the cost for.
tiered_pricing (List[dict]): A list of dictionaries, where each dictionary
represents a pricing tier.
cost_key (str): The key in the tier dictionary that holds the per-token cost
(e.g., 'input_cost_per_token').
fallback_cost_key (Optional[str], optional): A fallback key to use if the
primary `cost_key` is not found in a tier. Defaults to None.
Returns:
float: The total calculated cost for the given tokens.
Example:
>>> tiered_pricing = [
... {"range": [0, 100000], "input_cost_per_token": 0.0001},
... {"range": [100000, 500000], "input_cost_per_token": 0.00005},
... ]
Calculating cost for 150,000 tokens:
(100,000 * 0.0001) + (50,000 * 0.00005) = $12.5
"""
if not tiered_pricing or tokens <= 0:
return 0.0
total_cost = 0.0
tokens_processed = 0
sorted_tiers = sorted(tiered_pricing, key=lambda x: x.get("range", [0, 0])[0])
for tier in sorted_tiers:
if tokens_processed >= tokens:
break
tier_range = tier.get("range", [])
if len(tier_range) != 2:
continue
range_start, range_end = tier_range
if tokens <= range_start:
continue
tier_start = max(range_start, tokens_processed)
tier_end = min(range_end, tokens)
if tier_end > tier_start:
tokens_in_tier = tier_end - tier_start
cost_per_token = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
total_cost += tokens_in_tier * cost_per_token
tokens_processed = tier_end
# After loop, check if any tokens remain (i.e., tokens > highest tier's end range)
# and charge them at the last tier's rate.
if tokens_processed < tokens and sorted_tiers:
last_tier = sorted_tiers[-1]
remaining_tokens = tokens - tokens_processed
cost_per_token = last_tier.get(cost_key) or last_tier.get(fallback_cost_key, 0)
total_cost += remaining_tokens * cost_per_token
return total_cost
def _calculate_prompt_cost(
breakdown: TokenBreakdown,
model_info: ModelInfo,
@ -50,12 +123,12 @@ def _calculate_prompt_cost(
) -> float:
"""Calculate total prompt cost including cached tokens."""
if tiered_pricing:
text_cost = calculate_tiered_cost(
text_cost = _calculate_tiered_cost(
tokens=breakdown.text_tokens,
tiered_pricing=tiered_pricing,
cost_key="input_cost_per_token",
)
cache_cost = calculate_tiered_cost(
cache_cost = _calculate_tiered_cost(
tokens=breakdown.cached_tokens,
tiered_pricing=tiered_pricing,
cost_key="cache_read_input_token_cost",
@ -82,12 +155,12 @@ def _calculate_completion_cost(
) -> float:
"""Calculate total completion cost including reasoning tokens."""
if tiered_pricing:
completion_cost = calculate_tiered_cost(
completion_cost = _calculate_tiered_cost(
tokens=breakdown.completion_tokens,
tiered_pricing=tiered_pricing,
cost_key="output_cost_per_token",
)
reasoning_cost = calculate_tiered_cost(
reasoning_cost = _calculate_tiered_cost(
tokens=breakdown.reasoning_tokens,
tiered_pricing=tiered_pricing,
cost_key="output_cost_per_reasoning_token",

7
litellm/llms/databricks/chat/transformation.py
View file

@ -181,10 +181,6 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@property
def custom_llm_provider(self) -> Optional[str]:
return "databricks"
@classmethod
def get_config(cls):
return super().get_config()
@ -376,7 +372,6 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
mapped_thinking = AnthropicConfig._map_reasoning_effort(
reasoning_effort=reasoning_effort_value,
model=model,
custom_llm_provider="databricks",
llm_provider="databricks",
)
if mapped_thinking is None:
@ -384,7 +379,7 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
optional_params.pop("output_config", None)
else:
optional_params["thinking"] = mapped_thinking
if AnthropicConfig._is_adaptive_thinking_model(model, "databricks"):
if AnthropicConfig._is_adaptive_thinking_model(model):
mapped_effort: Optional[str] = None
if isinstance(reasoning_effort_value, str):
mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(reasoning_effort_value)

4
litellm/llms/github_copilot/messages/transformation.py
View file

@ -25,10 +25,6 @@ class GithubCopilotAnthropicMessagesConfig(AnthropicMessagesConfig):
super().__init__()
self.authenticator = Authenticator()
@property
def custom_llm_provider(self) -> Optional[str]:
return "github_copilot"
def handles_web_search_natively(self) -> bool:
"""
Copilot's /v1/messages endpoint does not execute ``web_search`` tools, so

207
litellm/llms/openai/chat/guardrail_translation/handler.py
View file

@ -14,14 +14,11 @@ Pattern Overview:
This pattern can be replicated for other message formats (e.g., Anthropic).
"""
from typing import TYPE_CHECKING, Any, Dict, List, Tuple, Union, cast
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import (
BaseTranslation,
StreamTransformSink,
)
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.llms.base_llm.guardrail_translation.utils import (
effective_skip_system_message_for_guardrail,
effective_skip_tool_message_for_guardrail,
@ -30,9 +27,6 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
)
from litellm.main import stream_chunk_builder
from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
from litellm.types.proxy.guardrails.guardrail_hooks.generic_guardrail_api import (
coerce_stream_holdback_value,
)
from litellm.types.utils import (
Choices,
GenericGuardrailAPIInputs,
@ -56,7 +50,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Methods can be overridden to customize behavior for different message formats.
"""
def get_structured_messages(self, data: dict) -> List[AllMessageValues] | None:
def get_structured_messages(self, data: dict) -> Optional[List[AllMessageValues]]:
"""
Convert chat completions request data to OpenAI-spec structured messages.
@ -71,7 +65,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
data: dict,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None = None,
litellm_logging_obj: Optional[Any] = None,
) -> Any:
"""
Process input messages by applying guardrails to text content.
@ -86,7 +80,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str] = []
images_to_check: List[str] = []
tool_calls_to_check: List[ChatCompletionToolParam] = []
text_task_mappings: List[Tuple[int, int | None]] = []
text_task_mappings: List[Tuple[int, Optional[int]]] = []
tool_call_task_mappings: List[Tuple[int, int]] = []
# Step 1: Extract all text content, images, and tool calls
@ -190,7 +184,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str],
images_to_check: List[str],
tool_calls_to_check: List[ChatCompletionToolParam],
text_task_mappings: List[Tuple[int, int | None]],
text_task_mappings: List[Tuple[int, Optional[int]]],
tool_call_task_mappings: List[Tuple[int, int]],
skip_system_message: bool = False,
skip_tool_message: bool = False,
@ -245,7 +239,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
messages: List[Dict[str, Any]],
responses: List[str],
task_mappings: List[Tuple[int, int | None]],
task_mappings: List[Tuple[int, Optional[int]]],
) -> None:
"""
Apply guardrail responses back to input message text content.
@ -255,7 +249,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for task_idx, guardrail_response in enumerate(responses):
mapping = task_mappings[task_idx]
msg_idx = cast(int, mapping[0])
content_idx_optional = cast(int | None, mapping[1])
content_idx_optional = cast(Optional[int], mapping[1])
# Handle content
content = messages[msg_idx].get("content", None)
@ -297,9 +291,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
response: "ModelResponse",
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None = None,
user_api_key_dict: Any | None = None,
request_data: dict | None = None,
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
request_data: Optional[dict] = None,
) -> Any:
"""
Process output response by applying guardrails to text content.
@ -326,7 +320,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str] = []
images_to_check: List[str] = []
tool_calls_to_check: List[Dict[str, Any]] = []
text_task_mappings: List[Tuple[int, int | None]] = []
text_task_mappings: List[Tuple[int, Optional[int]]] = []
tool_call_task_mappings: List[Tuple[int, int]] = []
# text_task_mappings: Track (choice_index, content_index) for each text
# content_index is None for string content, int for list content
@ -408,10 +402,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
responses_so_far: List["ModelResponseStream"],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None = None,
user_api_key_dict: Any | None = None,
request_data: dict | None = None,
stream_transform_sink: StreamTransformSink | None = None,
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
request_data: Optional[dict] = None,
) -> List["ModelResponseStream"]:
"""
Process output streaming responses by applying guardrails to text content.
@ -421,50 +414,14 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
guardrail_to_apply: The guardrail instance to apply
litellm_logging_obj: Optional logging object
user_api_key_dict: User API key metadata to pass to guardrails
stream_transform_sink: Optional out-parameter for the streaming text
transformation path. When provided, the guardrail runs over the raw
accumulated text (``responses_so_far`` is left untouched so it stays
a correct raw accumulator across rounds) and the guardrailed text
plus requested holdback are reported per choice on the sink.
Returns:
The (unmodified) list of responses.
Modified list of responses with guardrail applied to content
Response Format Support:
- String content: choice.message.content = "text here"
- List content: choice.message.content = [{"type": "text", "text": "text here"}, ...]
"""
if stream_transform_sink is not None:
await self._process_streaming_transform(
responses_so_far=responses_so_far,
guardrail_to_apply=guardrail_to_apply,
litellm_logging_obj=litellm_logging_obj,
user_api_key_dict=user_api_key_dict,
request_data=request_data,
sink=stream_transform_sink,
)
return responses_so_far
return await self._process_streaming_block_only(
responses_so_far=responses_so_far,
guardrail_to_apply=guardrail_to_apply,
litellm_logging_obj=litellm_logging_obj,
user_api_key_dict=user_api_key_dict,
request_data=request_data,
)
async def _process_streaming_block_only(
self,
*,
responses_so_far: list["ModelResponseStream"],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None,
user_api_key_dict: Any | None,
request_data: dict | None,
) -> list["ModelResponseStream"]:
"""Block-only streaming path: run the guardrail so an in-flight BLOCK can
terminate the stream. Text rewrites are not propagated to the client here
(see ``_process_streaming_transform`` for the incremental_diff path)."""
# check if the stream has ended
has_stream_ended = False
for chunk in responses_so_far:
@ -510,7 +467,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# Step 2: Create lists for guardrail processing
texts_to_check: List[str] = []
images_to_check: List[str] = []
task_mappings: List[Tuple[int, int | None]] = []
task_mappings: List[Tuple[int, Optional[int]]] = []
# Track (choice_index, content_index) for each combined text
for (map_choice_idx, map_content_idx), combined_text in combined_texts.items():
@ -563,109 +520,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
return responses_so_far
@staticmethod
def _accumulate_string_content_by_choice_index(
responses_so_far: list["ModelResponseStream"],
) -> dict[int, str]:
"""Accumulate raw string ``delta.content`` per choice, keyed by
``StreamingChoices.index`` (not enumerate position, which collapses to 0
when each chunk carries a single non-zero-indexed choice for ``n > 1``).
Only string content participates; list-of-blocks content is out of scope
for the incremental transform path. Reads ``responses_so_far`` without
mutating it so it stays a correct raw accumulator across rounds.
"""
accumulated: dict[int, str] = {}
for response in responses_so_far:
for choice in response.choices:
if isinstance(choice, litellm.StreamingChoices):
content = choice.delta.content
elif isinstance(choice, litellm.Choices):
content = choice.message.content
else:
continue
if isinstance(content, str) and content:
idx = getattr(choice, "index", 0) or 0
accumulated[idx] = accumulated.get(idx, "") + content
return accumulated
async def _process_streaming_transform(
self,
*,
responses_so_far: list["ModelResponseStream"],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Any | None,
user_api_key_dict: Any | None,
request_data: dict | None,
sink: StreamTransformSink,
) -> None:
"""Run the guardrail over the raw accumulated text and report the
guardrailed text plus requested holdback per choice on ``sink``.
Unlike the block-only path this never mutates ``responses_so_far``: it
re-derives the raw accumulated text every round (so a rewrite guardrail
always sees consistent input) and hands the result back out of band.
"""
raw_by_index = self._accumulate_string_content_by_choice_index(responses_so_far)
if not raw_by_index:
sink.mutated_text_per_choice = {}
sink.holdback_per_choice = {}
return
# Fix #2 — sort by StreamingChoices.index so an n>1 stream that emits
# choice 1 before choice 0 still hands the guardrail texts in a
# deterministic index order. Without this, the guardrail's returned
# texts (aligned to the input order it received) would map back to the
# wrong choice indices when we rebuild the sink dicts by
# ``enumerate(indices)``.
indices = sorted(raw_by_index.keys())
texts_to_check = [raw_by_index[i] for i in indices]
if request_data is None:
request_data = {"responses": responses_so_far}
elif "responses" not in request_data:
request_data["responses"] = responses_so_far
if "litellm_metadata" not in request_data:
user_metadata = self.transform_user_api_key_dict_to_metadata(user_api_key_dict)
if user_metadata:
request_data["litellm_metadata"] = user_metadata
inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
if responses_so_far and getattr(responses_so_far[0], "model", None):
inputs["model"] = responses_so_far[0].model
guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=request_data,
input_type="response",
logging_obj=litellm_logging_obj,
)
returned_texts = guardrailed_inputs.get("texts")
# No "texts" key means the guardrail made no change (action NONE): the raw
# accumulated text is the guardrailed text. A present-but-shorter list is a
# guardrail contract violation; those choices are omitted below (withheld,
# not emitted raw) so a malformed response fails closed instead of leaking.
if returned_texts is None:
returned_texts = texts_to_check
elif len(returned_texts) < len(texts_to_check):
verbose_proxy_logger.warning(
"OpenAI Chat Completions: guardrail returned %s transformed texts for %s inputs on the "
"streaming transform path; withholding the unmatched choices to fail closed.",
len(returned_texts),
len(texts_to_check),
)
holdback = guardrailed_inputs.get("stream_holdback_chars") or []
sink.mutated_text_per_choice = {
idx: returned_texts[i] for i, idx in enumerate(indices) if i < len(returned_texts)
}
sink.holdback_per_choice = {
indices[i]: coerce_stream_holdback_value(holdback[i]) for i in range(len(indices)) if i < len(holdback)
}
def _combine_streaming_texts(
self, responses_so_far: List["ModelResponseStream"]
) -> Dict[Tuple[int, int | None], str]:
) -> Dict[Tuple[int, Optional[int]], str]:
"""
Combine all streaming chunks into complete text per choice.
@ -677,7 +534,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Returns:
Dict mapping (choice_idx, content_idx) to combined text string
"""
combined_texts: Dict[Tuple[int, int | None], str] = {}
combined_texts: Dict[Tuple[int, Optional[int]], str] = {}
for response_idx, response in enumerate(responses_so_far):
for choice_idx, choice in enumerate(response.choices):
@ -693,7 +550,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if isinstance(content, str):
# String content - accumulate for this choice
str_key: Tuple[int, int | None] = (choice_idx, None)
str_key: Tuple[int, Optional[int]] = (choice_idx, None)
if str_key not in combined_texts:
combined_texts[str_key] = ""
combined_texts[str_key] += content
@ -703,7 +560,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for content_idx, content_item in enumerate(content):
text_str = content_item.get("text")
if text_str:
list_key: Tuple[int, int | None] = (
list_key: Tuple[int, Optional[int]] = (
choice_idx,
content_idx,
)
@ -750,7 +607,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
texts_to_check: List[str],
images_to_check: List[str],
tool_calls_to_check: List[Dict[str, Any]],
text_task_mappings: List[Tuple[int, int | None]],
text_task_mappings: List[Tuple[int, Optional[int]]],
tool_call_task_mappings: List[Tuple[int, int]],
) -> None:
"""
@ -762,7 +619,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# Determine content source and tool calls based on choice type
content = None
tool_calls: List[Any] | None = None
tool_calls: Optional[List[Any]] = None
if isinstance(choice, litellm.Choices):
content = choice.message.content
tool_calls = choice.message.tool_calls
@ -805,7 +662,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
tool_calls_to_check.append(tool_call_dict)
tool_call_task_mappings.append((choice_idx, int(tool_call_idx)))
def _convert_tool_call_to_dict(self, tool_call: Union[Dict[str, Any], Any]) -> Dict[str, Any] | None:
def _convert_tool_call_to_dict(self, tool_call: Union[Dict[str, Any], Any]) -> Optional[Dict[str, Any]]:
"""
Convert a tool call object to dictionary format.
@ -834,7 +691,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
response: "ModelResponse",
responses: List[str],
task_mappings: List[Tuple[int, int | None]],
task_mappings: List[Tuple[int, Optional[int]]],
) -> None:
"""
Apply guardrail text responses back to output response.
@ -844,7 +701,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
for task_idx, guardrail_response in enumerate(responses):
mapping = task_mappings[task_idx]
choice_idx = cast(int, mapping[0])
content_idx_optional = cast(int | None, mapping[1])
content_idx_optional = cast(Optional[int], mapping[1])
choice = cast(Choices, response.choices[choice_idx])
@ -898,7 +755,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
self,
responses: List["ModelResponseStream"],
guardrailed_texts: List[str],
task_mappings: List[Tuple[int, int | None]],
task_mappings: List[Tuple[int, Optional[int]]],
) -> None:
"""
Apply guardrail responses back to output streaming responses.
@ -914,16 +771,16 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Override this method to customize how responses are applied to streaming responses.
"""
# Build a mapping of what guardrailed text to use for each (choice_idx, content_idx)
guardrail_map: Dict[Tuple[int, int | None], str] = {}
guardrail_map: Dict[Tuple[int, Optional[int]], str] = {}
for task_idx, guardrail_response in enumerate(guardrailed_texts):
mapping = task_mappings[task_idx]
choice_idx = cast(int, mapping[0])
content_idx_optional = cast(int | None, mapping[1])
content_idx_optional = cast(Optional[int], mapping[1])
guardrail_map[(choice_idx, content_idx_optional)] = guardrail_response
# Track which choices we've already set the guardrailed text for
# Key: (choice_idx, content_idx), Value: boolean (True if already set)
already_set: Dict[Tuple[int, int | None], bool] = {}
already_set: Dict[Tuple[int, Optional[int]], bool] = {}
# Iterate through all responses and update content
for response_idx, response in enumerate(responses):
@ -940,7 +797,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
if isinstance(content, str):
# String content
str_key: Tuple[int, int | None] = (choice_idx_in_response, None)
str_key: Tuple[int, Optional[int]] = (choice_idx_in_response, None)
if str_key in guardrail_map:
if str_key not in already_set:
# First chunk - set the complete guardrailed text
@ -960,7 +817,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
# List content - handle each content item
for content_idx, content_item in enumerate(content):
if "text" in content_item:
list_key: Tuple[int, int | None] = (
list_key: Tuple[int, Optional[int]] = (
choice_idx_in_response,
content_idx,
)

18
litellm/llms/openai/responses/transformation.py
View file

@ -20,8 +20,6 @@ from litellm.types.utils import LlmProviders
from ..common_utils import OpenAIError
OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS = 16
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -61,19 +59,6 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
key="supports_none_reasoning_effort",
)
@staticmethod
def _enforce_min_max_output_tokens(max_output_tokens: "int | None") -> "int | None":
"""Raise sub-minimum max_output_tokens up to the OpenAI Responses API minimum.
OpenAI's Responses API rejects max_output_tokens below 16 for every model
(not gpt-5 specific), so a client like Claude Code that sends a max_tokens=1
warmup probe on model switch would otherwise 400. Values that are None or
already at/above the minimum are returned unchanged.
"""
if isinstance(max_output_tokens, int) and max_output_tokens < OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS:
return OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS
return max_output_tokens
def get_supported_openai_params(self, model: str) -> list:
"""
All OpenAI Responses API params are supported
@ -107,9 +92,6 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
"""
params = dict(response_api_optional_params)
if "max_output_tokens" in params:
params["max_output_tokens"] = self._enforce_min_max_output_tokens(params.get("max_output_tokens"))
if self._is_gpt_5_model(model=model):
temperature = params.get("temperature")
if temperature is not None and temperature != 1:

6
litellm/llms/openai_like/dynamic_config.py
View file

@ -91,7 +91,7 @@ def create_config_class(provider: SimpleProviderConfig):
def get_supported_openai_params(self, model: str) -> list:
"""Get supported OpenAI params, excluding tool-related params for models
that don't support function calling."""
from litellm.utils import supports_function_calling, supports_reasoning
from litellm.utils import supports_function_calling
supported_params = super().get_supported_openai_params(model=model)
@ -113,10 +113,6 @@ def create_config_class(provider: SimpleProviderConfig):
f"function calling — removed tool-related params from supported params."
)
_supports_reasoning = supports_reasoning(model=model, custom_llm_provider=provider.slug)
if _supports_reasoning and "reasoning_effort" not in supported_params:
supported_params.append("reasoning_effort")
return supported_params
def map_openai_params(

69
litellm/llms/openai_like/messages/transformation.py
View file

@ -1,11 +1,8 @@
from typing import Any, Optional
import litellm
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
from litellm.llms.openai_like.json_loader import SimpleProviderConfig
from litellm.secret_managers.main import get_secret_str
DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
@ -70,69 +67,3 @@ class OpenAILikeAnthropicMessagesConfig(AnthropicMessagesConfig):
if base.endswith("/v1"):
base = base[: -len("/v1")]
return f"{base}/v1/messages"
class JSONProviderAnthropicMessagesConfig(OpenAILikeAnthropicMessagesConfig):
"""
Provider-level native Anthropic Messages passthrough for JSON-configured
OpenAI-compatible providers whose ``supported_endpoints`` in providers.json
includes ``"/v1/messages"``. Resolves the api key and api base from the
provider's configured env vars, then forwards the Anthropic payload
untranslated like ``OpenAILikeAnthropicMessagesConfig``.
"""
def __init__(self, provider: SimpleProviderConfig):
super().__init__()
self._provider = provider
@property
def custom_llm_provider(self) -> Optional[str]:
return self._provider.slug
def should_strip_billing_metadata(self) -> bool:
return True
def _resolve_api_key(self, api_key: Optional[str]) -> Optional[str]:
return api_key or get_secret_str(self._provider.api_key_env) or litellm.api_key
def _resolve_api_base(self, api_base: Optional[str]) -> str:
env_api_base = get_secret_str(self._provider.api_base_env) if self._provider.api_base_env else None
return api_base or env_api_base or self._provider.base_url
def validate_anthropic_messages_environment(
self,
headers: dict[str, str],
model: str,
messages: list[Any],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> tuple[dict[str, str], Optional[str]]:
return super().validate_anthropic_messages_environment(
headers=headers,
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
api_key=self._resolve_api_key(api_key),
api_base=api_base,
)
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
return super().get_complete_url(
api_base=self._resolve_api_base(api_base),
api_key=api_key,
model=model,
optional_params=optional_params,
litellm_params=litellm_params,
stream=stream,
)

7
litellm/llms/openai_like/providers.json
View file

@ -168,13 +168,6 @@
},
"supported_endpoints": ["/v1/chat/completions", "/v1/responses"]
},
"meta": {
"base_url": "https://api.meta.ai/v1",
"api_key_env": "META_API_KEY",
"api_base_env": "META_API_BASE",
"base_class": "openai_gpt",
"supported_endpoints": ["/v1/chat/completions", "/v1/responses", "/v1/messages"]
},
"pinstripes": {
"base_url": "https://pinstripes.io/v1",
"api_key_env": "PINSTRIPES_API_KEY",

2
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -998,8 +998,6 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
response_modalities.append("IMAGE")
elif modality == "audio":
response_modalities.append("AUDIO")
elif modality == "video":
response_modalities.append("VIDEO")
else:
response_modalities.append("MODALITY_UNSPECIFIED")
return response_modalities

4
litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py
View file

@ -17,10 +17,6 @@ from ..output_params_utils import sanitize_vertex_anthropic_output_params
class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, VertexBase):
@property
def custom_llm_provider(self) -> Optional[str]:
return "vertex_ai"
def should_strip_billing_metadata(self) -> bool:
return True

2
litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/output_params_utils.py
View file

@ -26,7 +26,7 @@ def _model_accepts_output_config_effort(model: str) -> bool:
"""
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
return AnthropicConfig._model_supports_effort_param(model, "vertex_ai")
return AnthropicConfig._model_supports_effort_param(model)
def sanitize_vertex_anthropic_output_params(data: dict, model: str) -> None:

1
litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py
View file

@ -112,7 +112,6 @@ class VertexAIAnthropicConfig(AnthropicConfig):
prompt_caching_set=self.is_cache_control_set(messages),
file_id_used=self.is_file_id_used(messages),
mcp_server_used=self.is_mcp_server_used(optional_params.get("mcp_servers")),
custom_llm_provider="vertex_ai",
)
beta_set = set(auto_betas)

7
litellm/main.py
View file

@ -92,10 +92,7 @@ from litellm.litellm_core_utils.completion_timeout import CompletionTimeout
from litellm.litellm_core_utils.request_timeout_resolver import (
get_configured_request_timeout,
)
from litellm.litellm_core_utils.get_litellm_params import (
AWS_CREDENTIAL_KWARGS_KEYS,
OPTIONAL_KWARGS_KEYS,
)
from litellm.litellm_core_utils.get_litellm_params import OPTIONAL_KWARGS_KEYS
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.get_provider_specific_headers import (
ProviderSpecificHeaderUtils,
@ -5325,7 +5322,7 @@ def completion( # type: ignore
tpm=kwargs.get("tpm"),
rpm=kwargs.get("rpm"),
use_xai_oauth=kwargs.get("use_xai_oauth", False),
**{key: kwargs[key] for key in AWS_CREDENTIAL_KWARGS_KEYS if key in kwargs},
aws_bedrock_project_id=kwargs.get("aws_bedrock_project_id"),
)
cast(LiteLLMLoggingObj, logging).update_environment_variables(
model=model,

839
litellm/model_prices_and_context_window_backup.json
View file

File diff suppressed because it is too large Load diff

1
litellm/models/mcp_server.py
View file

@ -96,7 +96,6 @@ class LiteLLM_MCPServerTable(LiteLLMPydanticObjectBase):
available_on_public_internet: bool = True
delegate_auth_to_upstream: bool = False
oauth_passthrough: bool = False
dcr_bridge: Optional[bool] = None
is_byok: bool = False
byok_description: List[str] = Field(default_factory=list)
byok_api_key_help_url: Optional[str] = None

1
litellm/models/verification_token.py
View file

@ -36,7 +36,6 @@ class LiteLLM_VerificationToken(LiteLLMPydanticObjectBase):
budget_reset_at: Optional[datetime] = None
allowed_cache_controls: Optional[list] = []
allowed_routes: Optional[list] = []
key_type: str | None = None
permissions: Dict = {}
model_spend: Dict = {}
model_max_budget: Dict = {}

372
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -1,26 +1,12 @@
import re
from datetime import datetime, timezone
from typing import Dict, List, Optional, Set, Tuple, cast
from fastapi import HTTPException
from starlette.datastructures import Headers
from starlette.requests import Request
from starlette.types import Scope
from typing_extensions import assert_never
import litellm
from litellm._logging import verbose_logger
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import (
BridgeEnvelopeAdmitted,
BridgeEnvelopeInvalid,
NotBridgeEnvelope,
envelope_keys_from_master_key,
is_bridge_envelope_shaped,
resolve_bridge_envelope,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (
EnvelopeIdentity,
)
from litellm.proxy._types import (
UI_TEAM_ID,
LiteLLM_TeamTable,
@ -31,17 +17,12 @@ from litellm.proxy._types import (
UserAPIKeyAuth,
)
from litellm.proxy.auth.ip_address_utils import IPAddressUtils
from litellm.proxy.auth.user_api_key_auth import (
_run_centralized_common_checks,
user_api_key_auth,
)
from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_utils.user_api_key_cache import get_management_object_ttl
from litellm.repositories.table_repositories import (
AgentsRepository,
MCPServerRepository,
)
from litellm.types.mcp_server.mcp_server_manager import MCPServer
def _parse_mcp_server_names_from_path(path: str, mcp_servers_header: Optional[List[str]] = None) -> Optional[List[str]]:
@ -245,29 +226,6 @@ class MCPRequestHandler:
client_ip=IPAddressUtils.get_mcp_client_ip(request),
):
validated_user_api_key_auth = UserAPIKeyAuth()
elif (
(
bridge_delegate_target := MCPRequestHandler._single_dcr_bridge_delegate_target(
path=request_route,
mcp_servers=mcp_servers,
client_ip=IPAddressUtils.get_mcp_client_ip(request),
)
)
is not None
and oauth2_headers
and is_bridge_envelope_shaped(oauth2_headers["Authorization"])
):
# A single DCR-bridge oauth_delegate target carrying an envelope-shaped
# Authorization: open the envelope, admit under its recovered identity, and
# inject the inner upstream token for egress. A non-envelope bearer on the same
# server is NOT admitted here — it falls through to the oauth2 arm, which 401s.
validated_user_api_key_auth, mcp_server_auth_headers = await MCPRequestHandler._admit_dcr_bridge_delegate(
server=bridge_delegate_target,
authorization_value=oauth2_headers["Authorization"],
mcp_server_auth_headers=mcp_server_auth_headers,
request=request,
route=request_route,
)
elif oauth2_headers:
# Authorization on a non-delegated server: the bearer must be a real
# LiteLLM credential, so a failed validation is a genuine 401/403 and
@ -474,334 +432,6 @@ class MCPRequestHandler:
return False
return True
@staticmethod
def _single_dcr_bridge_delegate_target(
path: str, mcp_servers: Optional[List[str]], client_ip: Optional[str]
) -> Optional[MCPServer]:
"""The one DCR-bridge ``oauth_delegate`` server this request targets, or ``None``.
Returns the server only when EXACTLY ONE target resolves and it is both
``is_oauth_delegate`` and ``is_dcr_bridge``. Fails closed (``None``) on a
multi-target request, an unresolved target, or a non-matching server, so the
envelope admission arm never fires for an aggregate scope or a server that did not
opt into the bridge. Mirrors :meth:`_target_servers_are_true_passthrough`.
"""
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
target_names = MCPRequestHandler._resolve_target_server_names(path=path, mcp_servers_header=mcp_servers)
if len(target_names) != 1:
return None
server = global_mcp_server_manager.get_mcp_server_by_name(target_names[0], client_ip=client_ip)
if server is None or not server.is_oauth_delegate or not server.is_dcr_bridge:
return None
# Egress resolves the injected per-server token only by alias / server_name; a server with
# neither cannot receive the forwarded token, so fail closed rather than admit-and-drop.
if not (server.server_name or server.alias):
return None
return server
@staticmethod
async def _admit_dcr_bridge_delegate(
server: MCPServer,
authorization_value: str,
mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
request: Request,
route: str,
) -> Tuple[UserAPIKeyAuth, Optional[Dict[str, Dict[str, str]]]]:
"""Open the bridge envelope and admit the caller under the live key it references.
The envelope's signature proves the user authenticated when it was minted, but
authorization is resolved fresh here rather than trusted from the envelope: the
sealed ``key_hash`` reloads the current ``UserAPIKeyAuth`` record, and the admitted
identity then runs through the standard pipeline's centralized policy gate, so the
key's present restrictions and revocation state gate the request instead of a
snapshot frozen at mint time. The inner upstream token is injected under the
server's per-server auth-header key so egress forwards it via the
``PassthroughConfig`` override; the envelope ``Authorization`` the leak-defense
strips never reaches the upstream. A new headers dict is returned rather than
mutating the input. Fails closed with a 401 on an invalid or expired envelope, or
when the referenced key is missing, blocked, or expired, its owner is
SCIM-deactivated, or the centralized policy gate rejects it (blocked team or
project, org or budget limits).
The sealed token is keyed alias-first, matching the order egress resolves
(``lookup_mcp_server_auth_in_headers`` tries ``alias`` before ``server_name``). Keying
under ``server_name`` would leave a caller-supplied ``x-mcp-{alias}-authorization`` at the
higher-priority alias slot, pairing the admitted identity with an attacker's upstream
credential; the alias-keyed injection overwrites any such caller value.
"""
from litellm.proxy.proxy_server import master_key
if not master_key:
raise HTTPException(status_code=500, detail="Server misconfigured: master_key is not set")
await MCPRequestHandler._run_pre_db_read_auth_checks(request=request, route=route)
keys = envelope_keys_from_master_key(master_key)
result = resolve_bridge_envelope(authorization_value, keys, datetime.now(timezone.utc), server.server_id)
match result:
case BridgeEnvelopeAdmitted():
header_key = server.alias or server.server_name
if header_key is None:
raise HTTPException(status_code=500, detail="Server misconfigured: MCP server has no routable name")
admitted = await MCPRequestHandler._reload_admitted_principal(result.identity)
await MCPRequestHandler._enforce_admitted_live_policy(admitted=admitted, request=request, route=route)
injected = {header_key: {"Authorization": result.upstream_authorization.get_secret_value()}}
new_headers = {**(mcp_server_auth_headers or {}), **injected}
return admitted, new_headers
case BridgeEnvelopeInvalid() | NotBridgeEnvelope():
raise HTTPException(status_code=401, detail="Invalid or expired credential")
case _:
assert_never(result)
@staticmethod
async def _run_pre_db_read_auth_checks(request: Request, route: str) -> None:
"""Run the proxy-wide gates ``user_api_key_auth`` applies before any key lookup: the
request-size and body-safety limits, the IP allowlist, and the ``general_settings``
route allowlist. The envelope arm bypasses ``user_api_key_auth`` (it opens the envelope
and reloads the identity itself), so without this a caller blocked by IP or hitting a
proxy route the allowlist forbids would be admitted through an envelope where the same
principal presented on the normal MCP admission path would be rejected. Runs before the
envelope crypto so a disallowed caller is turned away before any work, mirroring the
standard pipeline's pre-DB ordering. Violations raise the gate's own status (an IP or
route block is a 403, an oversized body its own limit error)."""
from litellm.proxy.auth.auth_utils import pre_db_read_auth_checks
await pre_db_read_auth_checks(
request=request,
request_data=await _read_request_body(request=request),
route=route,
)
@staticmethod
async def _reload_admitted_principal(identity: EnvelopeIdentity) -> UserAPIKeyAuth:
"""Reload the live litellm record the envelope's subject references.
Dispatches on the sealed subject type: a ``key_hash`` reloads the virtual key that
minted the envelope (the scripted two-header client that presents a litellm key at the
token endpoint), a ``user_id`` reloads the user that authenticated interactively (the
DCR client, whose SSO login at the bridged authorize yields a user, not a key). Both
return a ``UserAPIKeyAuth`` the caller runs through the centralized policy gate, so
team/project/org/budget/SCIM enforcement is identical to the principal presenting
itself directly."""
match identity.subject_type:
case "key_hash":
return await MCPRequestHandler._reload_admitted_key(identity.subject)
case "user_id":
return await MCPRequestHandler._reload_admitted_user(identity.subject)
case _:
assert_never(identity.subject_type)
@staticmethod
async def _reload_admitted_user(user_id: str) -> UserAPIKeyAuth:
"""Reload the live user an interactively-minted envelope references and admit them as
themselves.
The DCR client authenticates via SSO at the bridged authorize, which yields a user
subject rather than a virtual key, so the envelope admits under the user's own
identity: the reloaded ``user_id`` and the user's own MCP object permission ride on the
returned ``UserAPIKeyAuth``, and the SAME ``get_allowed_mcp_servers`` the key path uses then
computes which servers the user may reach, so the user's litellm MCP grants and access groups
gate the request exactly as a key's do. Only the user's OWN object permission is bound: a
``UserAPIKeyAuth`` carries a single ``team_id`` while a user may belong to many teams, so
team-inherited MCP grants for a user are a follow-up (they need a many-teams union
``get_allowed_mcp_servers`` does not do off one auth object). The caller's centralized policy
gate enforces the user's live budget and org state, and a SCIM-deactivated owner fails closed.
Error handling mirrors the key path's retryable-503 contract, but ``get_user_object`` defeats a
type-based check: where ``get_key_object`` raises a typed ``ProxyException`` for a missing key
and lets a DB outage propagate raw, ``get_user_object`` catches every DB failure and re-raises a
bare ``ValueError``, so a missing user and a real outage look identical and the original error
survives only as ``__context__``. ``_raise_503_if_db_unavailable`` therefore walks the cause
chain: a transient DB outage still surfaces as a retryable 503, while a missing user, or any
other non-outage resolution failure, fails closed as a 401 rather than an opaque 500. The
object-permission load shares this one boundary, so an outage there is classified the same
way (``get_object_permission`` itself swallows a failed load to ``None``, matching how
``get_key_object`` best-effort-loads a key's object permission)."""
from litellm.proxy.auth.auth_checks import get_object_permission, get_user_object
from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
if prisma_client is None:
raise HTTPException(status_code=500, detail="Server misconfigured: no database connection")
try:
user_object = await get_user_object(
user_id=user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
# Resolve the user's own MCP object permission (get_user_object does not load it) so the shared
# get_allowed_mcp_servers can grant the user their litellm-granted servers. Reuses the same
# get_object_permission resolver the key and team paths use; no permission logic is duplicated.
object_permission = user_object.object_permission if user_object is not None else None
if user_object is not None and object_permission is None and user_object.object_permission_id:
object_permission = await get_object_permission(
object_permission_id=user_object.object_permission_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
except (ProxyException, HTTPException):
raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
except Exception as e: # noqa: BLE001 # a DB outage anywhere in the resolution is a retryable 503, not an opaque 500; anything else fails closed as 401
MCPRequestHandler._raise_503_if_db_unavailable(e)
raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
if user_object is None:
raise HTTPException(status_code=401, detail="Invalid or expired credential")
if isinstance(user_object.metadata, dict) and user_object.metadata.get("scim_active") is False:
raise HTTPException(status_code=401, detail="Invalid or expired credential")
return UserAPIKeyAuth(
user_id=user_object.user_id,
user_role=user_object.user_role,
object_permission=object_permission,
object_permission_id=user_object.object_permission_id,
)
@staticmethod
async def _reload_admitted_key(key_hash: str) -> UserAPIKeyAuth:
"""Reload the live key record an admitted envelope references and re-check live policy.
Resolving the current ``UserAPIKeyAuth`` (cache first, then DB) is what stops the
envelope from carrying frozen authority: the key's present team/org/object-permission
restrictions ride on the returned object, and a key that has since been deleted,
blocked, or expired fails closed with a 401 here rather than being admitted as an
unrestricted identity. ``get_key_object`` raises for a hash with no key row; a
blocked or expired row is rejected explicitly because ``get_key_object`` resolves a
row without applying those checks (the main ``user_api_key_auth`` pipeline enforces
them downstream, which this admission path bypasses). The owner's SCIM state is the
other builder-inline check mirrored here, so IdP offboarding revokes every envelope
minted under the user's keys rather than leaving them live until expiry. Team,
project, org, and budget state are NOT re-checked here; the caller runs the admitted
identity through ``_enforce_admitted_live_policy`` for those.
"""
from litellm.proxy.auth.auth_checks import get_key_object
from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
if prisma_client is None:
raise HTTPException(status_code=500, detail="Server misconfigured: no database connection")
try:
key_object = await get_key_object(
hashed_token=key_hash,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
except (ProxyException, HTTPException):
raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
except Exception as e: # noqa: BLE001 # a DB outage during reload is a retryable 503, not an opaque 500
MCPRequestHandler._raise_503_if_db_unavailable(e)
raise
if not MCPRequestHandler._admitted_key_is_active(key_object):
raise HTTPException(status_code=401, detail="Invalid or expired credential")
await MCPRequestHandler._reject_if_admitted_owner_scim_deactivated(key_object)
return key_object
@staticmethod
def _raise_503_if_db_unavailable(e: Exception) -> None:
"""Raise a retryable 503 when ``e`` means the auth database is unreachable, else return so the
caller applies its own fail-closed mapping. A DB outage must not masquerade as an auth failure
(401) or surface as an opaque 500; the caller retries. Mirrors ``UserAPIKeyAuthExceptionHandler``,
which renders a service-unavailable database error as 503 on the standard pipeline.
Classifies across the ``__cause__``/``__context__`` chain, not just ``e`` itself: ``get_user_object``
re-raises every DB failure as a bare ``ValueError``, so a type-based check on the top exception
would miss a real outage wrapped inside it."""
from litellm.proxy.db.exception_handler import PrismaDBExceptionHandler
if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(e):
raise HTTPException(
status_code=503,
detail="Service Unavailable, the authentication database is temporarily unreachable. Please retry shortly.",
) from None
@staticmethod
async def _reject_if_admitted_owner_scim_deactivated(key_object: UserAPIKeyAuth) -> None:
"""Fail closed with a 401 when the key's owning user was deactivated via SCIM.
The standard pipeline enforces this inline in ``_user_api_key_auth_builder`` rather
than in ``common_checks``, so the centralized policy gate does not cover it; without
this mirror, IdP offboarding would leave the user's already-minted envelopes live
until expiry. A failed user lookup skips the gate (fail-open), matching the builder:
this is the one deliberately fail-open check in an otherwise fail-closed arm, so a
transient DB outage during this lookup admits the request rather than rejecting it,
keeping parity with how the standard pipeline treats the same lookup failure."""
if key_object.user_id is None:
return
from litellm.proxy.auth.auth_checks import get_user_object
from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
try:
user_object = await get_user_object(
user_id=key_object.user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
except Exception as e: # noqa: BLE001 # mirror the builder's fail-open user lookup; DB errors are of any type
verbose_logger.debug(f"bridge admission: user lookup failed, skipping SCIM gate: {e}")
user_object = None
if user_object is None or not isinstance(user_object.metadata, dict):
return
if user_object.metadata.get("scim_active") is False:
raise HTTPException(status_code=401, detail="Invalid or expired credential")
@staticmethod
async def _enforce_admitted_live_policy(admitted: UserAPIKeyAuth, request: Request, route: str) -> None:
"""Run the standard pipeline's authorization checks over the admitted identity.
Mirrors the ``user_api_key_auth`` wrapper between the builder and its return: clear the
request-scoped ``budget_reservation`` on the reloaded identity, run the route gate
(``RouteChecks.should_call_route``) to enforce the identity's ``allowed_routes`` and any
disabled/admin-only route, then run ``_run_centralized_common_checks`` (the same gate every
builder path funnels through) for team-block, project-block, org, and budget. The route gate
closes a bypass: a key barred from MCP routes could otherwise mint an envelope at the token
endpoint (not itself an MCP route) and replay it against MCP, because the centralized checks
treat MCP as an inference route and never re-check ``allowed_routes``.
Failures surface with the status the standard pipeline would give them, mirroring
``UserAPIKeyAuthExceptionHandler``: a disallowed route is the route gate's own 403, an
over-budget identity is a 429, a sub-check that raised its own ``HTTPException``/
``ProxyException`` keeps that status, a transient database outage is a retryable 503, and
only a genuinely unresolvable failure (a blocked team/project raises a bare ``Exception``,
same as the standard pipeline's fallback) becomes the fail-closed 401. Collapsing every
failure to 401 was misleading: it told an over-budget but validly-authenticated caller their
credential was invalid, which on a DCR client reads as broken auth and can trigger a
pointless re-authorize loop that cannot fix a budget problem, and it masked a DB outage as an
auth error."""
from litellm.proxy.auth.route_checks import RouteChecks
admitted.budget_reservation = None
try:
RouteChecks.should_call_route(route=route, valid_token=admitted, request=request)
await _run_centralized_common_checks(
user_api_key_auth_obj=admitted,
request=request,
request_data=await _read_request_body(request=request),
route=route,
)
except (HTTPException, ProxyException):
raise
except litellm.BudgetExceededError as e:
raise HTTPException(status_code=getattr(e, "status_code", 429), detail=str(e)) from None
except Exception as e: # noqa: BLE001 # untyped gate failure: retryable 503 for a DB outage, else fail closed 401
MCPRequestHandler._raise_503_if_db_unavailable(e)
raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
@staticmethod
def _admitted_key_is_active(key_object: UserAPIKeyAuth) -> bool:
"""False when the referenced key is blocked or past its expiry, so a revoked key
cannot be admitted through its still-unexpired envelope. Mirrors the active-key gate
the bridge token endpoint applies at mint time."""
if key_object.blocked is True:
return False
expires = key_object.expires
if expires is None:
return True
expiry = expires if isinstance(expires, datetime) else datetime.fromisoformat(expires)
if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
expiry = expiry.replace(tzinfo=timezone.utc)
return expiry >= datetime.now(timezone.utc)
@staticmethod
def _resolve_target_server_names(path: str, mcp_servers_header: Optional[List[str]]) -> List[str]:
"""

694
litellm/proxy/_experimental/mcp_server/bridge_token_flow.py
View file

@ -1,694 +0,0 @@
"""Bridge token flow: litellm identity resolution and the DCR-bridge oauth_delegate mint/refresh pipeline."""
import math
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Literal, Optional
from fastapi import HTTPException, Request
from fastapi.responses import JSONResponse
from pydantic import SecretStr
from typing_extensions import assert_never
from litellm._logging import verbose_logger
from litellm.proxy._experimental.mcp_server.oauth_utils import TOKEN_NO_CACHE_HEADERS
from litellm.types.mcp_server.mcp_server_manager import MCPServer
if TYPE_CHECKING:
from litellm.proxy._experimental.mcp_server.discoverable_endpoints import _BridgeAuthorizationCode
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (
EnvelopeIdentity,
EnvelopeKeys,
RefreshCredential,
UpstreamTokenGrant,
)
from litellm.proxy._types import UserAPIKeyAuth
def _litellm_key_from_request(request: Request) -> Optional[str]:
"""Return the LiteLLM API key presented on the request, or ``None``.
Accepts the key from ``x-litellm-api-key`` (what MCP clients such as Claude Desktop/Code
send) as well as ``Authorization``; either may carry a bare token or ``Bearer <token>``.
``x-litellm-api-key`` wins when both are present, since ``Authorization`` may instead carry
an OAuth/upstream bearer.
"""
for header_value in (
request.headers.get("x-litellm-api-key"),
request.headers.get("Authorization") or request.headers.get("authorization"),
):
if not header_value:
continue
value = header_value.strip()
if value.lower().startswith("bearer "):
value = value[7:].strip()
if value:
return value
return None
def _key_is_active(key_obj: "UserAPIKeyAuth") -> bool:
"""``True`` when the presented key is neither blocked nor past its expiry.
The OAuth token endpoint is unauthenticated, so the presented key is validated here before it is
trusted; a revoked or expired key must not mint a bridge envelope or write a stored credential.
``get_key_object`` resolves a row without these checks (the main ``user_api_key_auth`` pipeline
enforces them downstream, which this endpoint bypasses), so they are applied here. Deleted keys
are already rejected upstream, where ``get_key_object`` raises on a row that no longer exists.
This is an active-state gate only; it deliberately does not require a ``user_id``. A valid
team-scoped or service-account key has no ``user_id`` yet is a legitimate credential, so gating
on ``user_id`` presence would wrongly reject it. Callers that need the user (the per-user token
store) derive it separately via :func:`_active_key_user_id`.
Total by design: ``expires`` is typed ``str | datetime``, and an unparseable string would make
``datetime.fromisoformat`` raise. Since the callers run this outside their key-resolution
``try``, an uncaught parse error would surface as a 500 instead of the endpoint's fail-closed
behavior, so a malformed expiry is treated as inactive (return ``False``) rather than raising.
"""
if key_obj.blocked is True:
return False
expires = key_obj.expires
if expires is not None:
if isinstance(expires, datetime):
expiry = expires
else:
try:
expiry = datetime.fromisoformat(expires)
except (ValueError, TypeError):
return False
if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
expiry = expiry.replace(tzinfo=timezone.utc)
if expiry < datetime.now(timezone.utc):
return False
return True
def _active_key_user_id(key_obj: "UserAPIKeyAuth") -> str | None:
"""The active key's ``user_id``, or ``None`` when the key is blocked/expired or simply has no
``user_id`` (a team-scoped or service-account key). Used only by the per-user token store, which
needs a user to key the stored credential; the bridge mint uses the key hash and does not."""
return key_obj.user_id if _key_is_active(key_obj) else None
@dataclass(frozen=True, slots=True)
class _ResolvedKey:
"""An active litellm key resolved from the token request: its hash (the value ``get_key_object``
and the cache/DB layer key the record by) and the live record."""
key_hash: str
key: "UserAPIKeyAuth"
_KeyResolutionFailure = Literal["no_active_key", "unavailable", "unresolvable"]
"""Why a token request yielded no active litellm key, kept distinct so a caller statuses each truthfully
instead of blaming the client for a gateway problem:
- ``no_active_key``: none was presented, or the presented key is unknown / blocked / expired (the
caller's request is at fault)
- ``unavailable``: the auth database was transiently unreachable while resolving (retryable)
- ``unresolvable``: the gateway cannot resolve identity right now (no DB connection, or an unexpected
error) -- a gateway fault, not the caller's
The classification mirrors admission's ``_reload_admitted_key`` so the mint (ingress) and admission
(egress) never disagree on the status of the same outage."""
async def _resolve_active_litellm_key(request: Request) -> "_ResolvedKey | _KeyResolutionFailure":
"""Resolve the presented litellm key to an active key record, or say precisely why not.
Single resolution path the OAuth token endpoint reuses, resolving authoritatively via
``get_key_object`` (cache first, then DB). The failure is a value, not a bare ``None``, so a caller
can tell "the client sent no usable credential" (a request error) apart from "the gateway could not
check" (an infrastructure error) and status each truthfully; collapsing both to ``None`` is what let
a DB outage read as a 400. A resolved key is still gated by ``_key_is_active``, so a blocked or
expired key is ``no_active_key`` while a valid team-scoped or service-account key (no ``user_id``)
resolves. Classification mirrors admission's ``_reload_admitted_key``: no DB connection is a gateway
fault, a ``ProxyException`` / ``HTTPException`` from ``get_key_object`` is an unknown or invalid key,
a database-service-unavailable error is a retryable outage, and anything else is an unexpected
gateway fault."""
token = _litellm_key_from_request(request)
if not token:
return "no_active_key"
from litellm.proxy._types import hash_token # noqa: PLC0415 # inline import avoids a module-load circular import
return await _reload_active_key_by_hash(hash_token(token))
async def _reload_active_key_by_hash(key_hash: str) -> "_ResolvedKey | _KeyResolutionFailure":
"""Reload the live key record for ``key_hash`` (cache first, then DB) and gate it on active state,
returning the resolved key or a precise failure. Shared by the token request's presented-key
resolution (:func:`_resolve_active_litellm_key`, which hashes the presented key) and the refresh
path (which already holds the hash sealed in the refresh envelope), so both re-validate identity
through one active-key gate and one failure classification. Classification mirrors admission's
``_reload_admitted_key``: no DB connection is a gateway fault, a ``ProxyException`` / ``HTTPException``
from ``get_key_object`` is an unknown or invalid key, a database-service-unavailable error is a
retryable outage, and anything else is an unexpected gateway fault. A blocked or expired key is
``no_active_key``, so a revoked key can neither mint nor refresh a bridge envelope."""
from litellm.proxy._types import (
ProxyException, # noqa: PLC0415 # inline import avoids a module-load circular import
)
from litellm.proxy.auth.auth_checks import ( # noqa: PLC0415 # inline import avoids a module-load circular import
get_key_object,
)
from litellm.proxy.db.exception_handler import ( # noqa: PLC0415 # inline import avoids a module-load circular import
PrismaDBExceptionHandler,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
prisma_client,
user_api_key_cache,
)
if prisma_client is None:
return "unresolvable"
try:
key_obj = await get_key_object(
hashed_token=key_hash,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
except (ProxyException, HTTPException):
return "no_active_key"
except Exception as exc: # noqa: BLE001 # classify: a DB outage is retryable, anything else is an opaque gateway fault
if PrismaDBExceptionHandler.is_database_service_unavailable_error(exc):
return "unavailable"
verbose_logger.debug(
"_reload_active_key_by_hash: unexpected key-resolution error (%s)",
type(exc).__name__,
)
return "unresolvable"
if not _key_is_active(key_obj):
return "no_active_key"
return _ResolvedKey(key_hash=key_hash, key=key_obj)
async def _reload_active_user_by_id(user_id: str) -> "_KeyResolutionFailure | None":
"""Re-validate a live litellm user by id, returning ``None`` when the user is active or a precise
failure otherwise. The interactive DCR client authenticates via SSO, so its refresh envelope seals a
user subject; renewing it must re-check the user is still live (present and not SCIM-deactivated) so a
deactivated user cannot keep refreshing, mirroring how admission re-validates the same user subject on
the egress side. No DB connection is a gateway fault (``unresolvable``) and a
database-service-unavailable error is a retryable outage (``unavailable``). Everything else fails
closed as ``no_active_key`` (the caller maps it to invalid_grant): a ``ProxyException`` /
``HTTPException``, a SCIM-deactivated user, and, unlike the key path, a missing user. ``get_user_object``
catches every DB failure and re-raises a bare ``ValueError`` (a deleted user and a real outage look
identical, the original error surviving only as ``__context__``), so the outage check walks the cause
chain, and a missing user falls through to ``no_active_key`` rather than an opaque gateway fault."""
from litellm.proxy._types import (
ProxyException, # noqa: PLC0415 # inline import avoids a module-load circular import
)
from litellm.proxy.auth.auth_checks import ( # noqa: PLC0415 # inline import avoids a module-load circular import
get_user_object,
)
from litellm.proxy.db.exception_handler import ( # noqa: PLC0415 # inline import avoids a module-load circular import
PrismaDBExceptionHandler,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
prisma_client,
user_api_key_cache,
)
if prisma_client is None:
return "unresolvable"
try:
user_object = await get_user_object(
user_id=user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
except (ProxyException, HTTPException):
return "no_active_key"
except Exception as exc: # noqa: BLE001 # a DB outage is retryable; a missing user (get_user_object's wrapped ValueError) or any other resolution failure fails closed as no_active_key, never a 500
if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc):
return "unavailable"
verbose_logger.debug("_reload_active_user_by_id: user-resolution error (%s)", type(exc).__name__)
return "no_active_key"
if user_object is None:
return "no_active_key"
if isinstance(user_object.metadata, dict) and user_object.metadata.get("scim_active") is False:
return "no_active_key"
return None
async def _key_owner_scim_deactivated(key: "UserAPIKeyAuth") -> bool:
"""True only when the key's owning user was explicitly SCIM-deactivated, so a refresh revokes an
offboarded owner's key exactly as admission does via ``_reject_if_admitted_owner_scim_deactivated``.
A key with no owner, a missing owner record, or a failed lookup fails OPEN (returns ``False``),
matching admission and the standard builder: a key may outlive its owner record, and a transient DB
blip must not revoke a live key. Only an explicit ``scim_active`` of ``False`` gates renewal."""
if key.user_id is None:
return False
from litellm.proxy.auth.auth_checks import ( # noqa: PLC0415 # inline import avoids a module-load circular import
get_user_object,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
prisma_client,
user_api_key_cache,
)
if prisma_client is None:
return False
try:
owner = await get_user_object(
user_id=key.user_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
user_id_upsert=False,
)
except Exception as exc: # noqa: BLE001 # fail open: a missing owner (get_user_object's wrapped ValueError) or a DB blip must not revoke a live key
verbose_logger.debug("refresh: key-owner SCIM lookup failed, not revoking (%s)", type(exc).__name__)
return False
return owner is not None and isinstance(owner.metadata, dict) and owner.metadata.get("scim_active") is False
async def _revalidate_active_subject(identity: "EnvelopeIdentity") -> "_KeyResolutionFailure | None":
"""Re-validate that the subject sealed in a refresh envelope is still live, dispatching on its type:
a key_hash reloads the virtual key, a user_id reloads the user. Returns ``None`` when the subject is
active or a precise failure otherwise, so revocation gates renewal for either identity source the same
way admission gates the egress: a blocked or expired key, a SCIM-deactivated key owner (mirroring
admission's owner check, so an offboarded user cannot keep renewing a still-active key), and a
deactivated or deleted user all fail closed to ``no_active_key``."""
match identity.subject_type:
case "key_hash":
reloaded = await _reload_active_key_by_hash(identity.subject)
if not isinstance(reloaded, _ResolvedKey):
return reloaded
if await _key_owner_scim_deactivated(reloaded.key):
return "no_active_key"
return None
case "user_id":
return await _reload_active_user_by_id(identity.subject)
case _:
assert_never(identity.subject_type)
async def _extract_user_id_from_request(request: Request) -> str | None:
"""The litellm ``user_id`` for the token request, so a per-user token is stored under the same
identity the egress later reads it by. Storage is best-effort, so every non-resolved outcome
(including a transient DB outage) collapses to ``None`` here and the caller simply skips the store;
the bridge mint, which must status those outcomes differently, consumes
:func:`_resolve_active_litellm_key` directly."""
resolved = await _resolve_active_litellm_key(request)
if not isinstance(resolved, _ResolvedKey):
return None
return _active_key_user_id(resolved.key)
_UpstreamGrantRejection = Literal["no_access_token", "expired_lifetime"]
"""Why an upstream token response cannot back a bridge envelope:
- ``no_access_token``: the response carries no usable ``access_token``
- ``expired_lifetime``: the response reports a parseable, non-positive ``expires_in``, i.e. an upstream
token that is already dead, so sealing it would forward a bearer the edge cannot use
An absent or unparseable ``expires_in`` is NOT a rejection; the lifetime is merely unknown and the
envelope caps it, the by-design behaviour for an upstream that omits the field."""
def _classify_upstream_lifetime(raw_expires_in: object) -> "int | Literal['unspecified', 'expired']":
"""Classify an upstream ``expires_in`` into a positive number of seconds, ``"unspecified"`` (absent
or unparseable, so the envelope caps it), or ``"expired"`` (a non-positive value the upstream reports
as already elapsed). Telling "we do not know the lifetime" apart from "the upstream says it is
already dead" is what stops an explicitly-expired token from silently receiving the envelope's 1h
cap. The expired decision is made on the parsed numeric value, not on ``int(...)`` of it, so a
positive sub-second lifetime in ``(0, 1)`` is not truncated to ``0`` and misread as elapsed; the
envelope works in whole seconds, so such a lifetime clamps up to its 1s floor. ``bool`` is excluded
(an ``int`` subclass but never a real lifetime), and the conversions can raise on ``NaN`` /
``Infinity`` / oversized input, which reads as unparseable rather than surfacing as a 500."""
if raw_expires_in is None or isinstance(raw_expires_in, bool) or not isinstance(raw_expires_in, (int, float, str)):
return "unspecified"
try:
numeric = float(raw_expires_in)
seconds = int(numeric)
except (ValueError, TypeError, OverflowError):
return "unspecified"
if numeric <= 0:
return "expired"
return max(1, seconds)
def _bridge_grant_from_token_response(token_response: object) -> "UpstreamTokenGrant | _UpstreamGrantRejection":
"""Validate an upstream OAuth token response into a typed grant, or say why it cannot back an
envelope. Each field is isinstance-checked so nothing untyped from ``response.json()`` reaches the
grant. ``expires_in`` is read three ways (see :func:`_classify_upstream_lifetime`): an unknown
lifetime leaves the grant ``expires_in`` ``None`` for the envelope to cap, a positive value is
honoured, and an explicit already-elapsed value is a rejection rather than a silent fall-through to
the cap."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
UpstreamTokenGrant,
)
if not isinstance(token_response, dict):
return "no_access_token"
access = token_response.get("access_token")
if not isinstance(access, str) or not access:
return "no_access_token"
lifetime = _classify_upstream_lifetime(token_response.get("expires_in"))
if lifetime == "expired":
return "expired_lifetime"
token_type = token_response.get("token_type")
scope = token_response.get("scope")
return UpstreamTokenGrant(
access_token=SecretStr(access),
token_type=token_type if isinstance(token_type, str) and token_type else "Bearer",
# The upstream refresh_token is deliberately NOT sealed: the edge never consumes it (it forwards
# only token_type + access_token), so it would be dead weight embedding a long-lived upstream
# credential in the client-held bearer, and it enlarges the envelope. Refresh support is a
# follow-up (a dedicated refresh-envelope); the client re-runs authorization_code at the cap.
refresh_token=None,
scope=scope if isinstance(scope, str) and scope else None,
expires_in=lifetime if isinstance(lifetime, int) else None,
)
# ---------------------------------------------------------------------------
# DCR-bridge oauth_delegate mint: a three-phase pipeline whose failures are values.
#
# prepare (before the upstream exchange) -> validate every precondition and resolve identity+keys
# exchange (the single-use upstream code is consumed here, in exchange_token_with_server)
# finish (after the exchange) -> seal the upstream grant into the client-held envelope
#
# Every precondition lives in ``prepare``, which runs BEFORE the exchange, so no failure can burn the
# single-use code or rotate a refresh token, for either grant type -- that whole class of bug is gone
# by construction rather than guarded case by case. Failures are values mapped to an OAuth-shaped
# response in one place (``_bridge_mint_error_response``), so status codes and the RFC 6749 §5.2 body
# shape are uniform. Adding a failure mode is a new literal plus a match arm the type checker forces.
# ---------------------------------------------------------------------------
_BridgeMintError = Literal[
"no_identity",
"invalid_refresh",
"identity_unavailable",
"identity_unresolvable",
"not_configured",
"no_upstream_token",
"upstream_token_expired",
"too_large",
]
@dataclass(frozen=True, slots=True)
class _BridgeMintReady:
"""Everything the seal needs, resolved once before the exchange: the identity to bind the envelope
to and the master-key-derived envelope keys. The identity is a key_hash subject for the scripted
two-header client (resolved from the litellm key it presents) or a user_id subject for the
interactive SSO client (the user recovered from the gateway authorization code), so one phase-3 seal
serves both. Resolving identity here means ``_finish_bridge_mint`` has no preconditions left to
fail."""
identity: "EnvelopeIdentity"
keys: "EnvelopeKeys"
def _bridge_mint_error_response(error: _BridgeMintError) -> JSONResponse:
"""Map a bridge-mint failure value to its token-endpoint response: one place, RFC 6749 §5.2 shape
(top-level ``error``, no-store headers) for every case, with a status truthful about where the
failure is. The caller's request is 400, a transient gateway outage is 503, a gateway
misconfiguration is 500, and an upstream problem is 502. The identity-resolution statuses match how
admission statuses the same conditions on the egress side, so mint and admit never disagree under
one outage."""
match error:
case "no_identity":
status, code, desc = (
400,
"invalid_request",
"this server issues a gateway-bound credential; complete the interactive sign-in, or "
"send a litellm credential (x-litellm-api-key or Authorization) on the token request",
)
case "invalid_refresh":
status, code, desc = (
400,
"invalid_grant",
"the refresh credential is not a valid, live refresh envelope for this server; "
"re-run authorization_code to obtain a new one",
)
case "identity_unavailable":
status, code, desc = (
503,
"temporarily_unavailable",
"the authentication database is temporarily unreachable; retry shortly",
)
case "identity_unresolvable":
status, code, desc = (
500,
"server_error",
"the gateway could not resolve the litellm identity for this request",
)
case "not_configured":
status, code, desc = (
500,
"server_error",
"the gateway is not configured to mint a gateway-bound credential (master_key is not set)",
)
case "no_upstream_token":
status, code, desc = (
502,
"server_error",
"the upstream token response has no usable access_token",
)
case "upstream_token_expired":
status, code, desc = (
502,
"server_error",
"the upstream token response reports an already-expired lifetime",
)
case "too_large":
status, code, desc = (
502,
"server_error",
"the upstream token is too large to seal into a gateway-bound credential",
)
case _:
assert_never(error)
return JSONResponse(
status_code=status, content={"error": code, "error_description": desc}, headers=TOKEN_NO_CACHE_HEADERS
)
def _key_resolution_failure_to_mint_error(failure: _KeyResolutionFailure) -> _BridgeMintError:
"""Lift an identity-resolution failure into the mint taxonomy, preserving origin so the status stays
truthful: the caller's missing credential is 400, a transient DB outage is 503, and a gateway that
cannot resolve identity is 500."""
match failure:
case "no_active_key":
return "no_identity"
case "unavailable":
return "identity_unavailable"
case "unresolvable":
return "identity_unresolvable"
case _:
assert_never(failure)
def _upstream_rejection_to_mint_error(rejection: _UpstreamGrantRejection) -> _BridgeMintError:
"""Lift an upstream-response rejection into the mint taxonomy; both are upstream faults (502)."""
match rejection:
case "no_access_token":
return "no_upstream_token"
case "expired_lifetime":
return "upstream_token_expired"
case _:
assert_never(rejection)
async def _prepare_bridge_mint(
request: Request,
mcp_server: MCPServer,
bridge_identity: "_BridgeAuthorizationCode | None" = None,
) -> "_BridgeMintReady | _BridgeMintError":
"""Phase 1 for the authorization_code grant, BEFORE the upstream exchange: confirm the gateway can
mint (master_key set), resolve the litellm identity, and derive the envelope keys. Returns a ready
context or a precise failure value. Running before the exchange is what makes every failure here fail
closed without consuming the single-use code.
Two identity sources, one envelope. The interactive DCR client authenticates via SSO at the bridged
authorize, so its identity arrives as ``bridge_identity`` (the user recovered from the gateway
authorization code) and mints a user subject. The scripted two-header client presents a litellm key
on the token request instead, so its identity is the active key's hash and mints a key_hash subject.
A missing or invalid presented key keeps its resolution origin so the mapper statuses it truthfully;
neither source present is ``no_identity``. The refresh_token grant has its own phase-1
(:func:`_prepare_bridge_refresh`), which recovers identity from the presented refresh envelope."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
envelope_keys_from_master_key,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
key_hash_identity,
user_identity,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
master_key,
)
if not master_key:
return "not_configured"
keys = envelope_keys_from_master_key(master_key)
if bridge_identity is not None:
identity = user_identity(server_id=mcp_server.server_id, user_id=bridge_identity.litellm_user_id)
return _BridgeMintReady(identity=identity, keys=keys)
resolved = await _resolve_active_litellm_key(request)
if not isinstance(resolved, _ResolvedKey):
return _key_resolution_failure_to_mint_error(resolved)
identity = key_hash_identity(server_id=mcp_server.server_id, key_hash=resolved.key_hash)
return _BridgeMintReady(identity=identity, keys=keys)
@dataclass(frozen=True, slots=True)
class _BridgeRefreshReady:
"""A validated refresh request: the identity+keys to mint the renewed pair under, the upstream refresh
token (unwrapped from the client's refresh envelope) to exchange with the upstream IdP, and the scope
sealed alongside it at mint. The upstream refresh token is a ``SecretStr`` like every other credential
in this layer, so a repr or a traceback that captures this value never exposes the raw upstream refresh
token in plaintext. ``upstream_scope`` carries the originally-granted scope so the renewal re-requests
it when the client (a DCR/MCP client that typically omits scope on refresh) sends none, keeping the
renewed token's scope stable against an upstream that would otherwise narrow or drop it."""
ready: "_BridgeMintReady"
upstream_refresh_token: SecretStr
upstream_scope: str | None = None
def _refresh_key_failure_to_mint_error(failure: _KeyResolutionFailure) -> _BridgeMintError:
"""Lift an identity-resolution failure on the refresh path into the mint taxonomy. Unlike the mint
path, a resolved-but-inactive (or unknown) key is ``invalid_grant`` rather than ``invalid_request``:
the client did present an identity (sealed in the refresh envelope), but it is no longer live, so the
refresh is invalid and the client must re-authenticate. A transient outage is still 503 and a gateway
fault still 500, matching the mint path and admission."""
match failure:
case "no_active_key":
return "invalid_refresh"
case "unavailable":
return "identity_unavailable"
case "unresolvable":
return "identity_unresolvable"
case _:
assert_never(failure)
async def _prepare_bridge_refresh(
mcp_server: MCPServer, refresh_value: str | None
) -> "_BridgeRefreshReady | _BridgeMintError":
"""Phase 1 for the refresh_token grant, BEFORE the upstream exchange: open the client's refresh
envelope, re-validate the sealed litellm identity so a revoked key cannot keep refreshing, and
recover the upstream refresh token to exchange. Identity comes entirely from the sealed envelope, not
the HTTP request, so the request object is not needed here. The client presents a refresh envelope,
never a raw upstream refresh token, so a missing value, a non-envelope, an unopenable envelope, or one
minted for another server is ``invalid_grant``. Running before the exchange means a rejected refresh
never consumes or rotates the upstream refresh token."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
BridgeRefreshOpened,
envelope_keys_from_master_key,
open_bridge_refresh_envelope,
)
from litellm.proxy.proxy_server import ( # noqa: PLC0415 # inline import avoids a module-load circular import
master_key,
)
if not master_key:
return "not_configured"
if not refresh_value:
return "invalid_refresh"
keys = envelope_keys_from_master_key(master_key)
opened = open_bridge_refresh_envelope(refresh_value, keys, datetime.now(timezone.utc), mcp_server.server_id)
if not isinstance(opened, BridgeRefreshOpened):
return "invalid_refresh"
failure = await _revalidate_active_subject(opened.identity)
if failure is not None:
return _refresh_key_failure_to_mint_error(failure)
return _BridgeRefreshReady(
ready=_BridgeMintReady(identity=opened.identity, keys=keys),
upstream_refresh_token=opened.refresh.refresh_token,
upstream_scope=opened.refresh.scope,
)
def _finish_bridge_mint(
ready: "_BridgeMintReady", mcp_server: MCPServer, token_response: object, now: datetime
) -> "JSONResponse | _BridgeMintError":
"""Phase 3, AFTER the upstream exchange: seal the upstream grant into the client-held access envelope
using the pre-resolved identity and keys, and, when the upstream returned a refresh token, seal a
long-lived refresh envelope alongside it so the client can renew without re-authenticating. Shared by
the authorization_code and refresh_token paths, so a renewal that the upstream rotates re-issues a
fresh refresh envelope. The only hard failures here are properties of the upstream access token (no
usable token, an already-expired lifetime, or a token too large to seal); a refresh token that cannot
be sealed degrades to an access-only response rather than failing the whole exchange."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
build_bridge_token_response,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
SealedEnvelope,
UpstreamTokenGrant,
)
grant = _bridge_grant_from_token_response(token_response)
if not isinstance(grant, UpstreamTokenGrant):
return _upstream_rejection_to_mint_error(grant)
sealed = build_bridge_token_response(ready.identity, grant, ready.keys, now)
if not isinstance(sealed, SealedEnvelope):
return "too_large"
# Report expires_in from the JWT's own second-truncated exp, rounding the elapsed portion up, so the
# client is never told the bearer lives past the point admission (which uses that exp) rejects it.
expires_in = max(0, int(sealed.expires_at.timestamp()) - math.ceil(now.timestamp()))
refresh_envelope = _mint_refresh_envelope_value(ready.identity, token_response, ready.keys, now, mcp_server)
body = {
"access_token": sealed.token.get_secret_value(),
"token_type": "Bearer",
"expires_in": expires_in,
# A refresh envelope rides along only when the upstream returned a refresh token to seal; when it
# rotates on renewal, the client receives the new one and the old envelope's upstream token dies.
**({"refresh_token": refresh_envelope} if refresh_envelope is not None else {}),
}
return JSONResponse(body, headers=TOKEN_NO_CACHE_HEADERS)
def _upstream_refresh_credential(token_response: object) -> "RefreshCredential | None":
"""Extract the upstream refresh grant from a token response, or ``None`` when there is none to seal.
Each field is isinstance-checked so nothing untyped reaches the refresh envelope; ``refresh_expires_in``
(the refresh token's own lifetime, when the upstream reports it) is classified like ``expires_in`` and
bounds the refresh envelope's TTL. An upstream that reports the refresh token itself as already elapsed
(``refresh_expires_in`` non-positive) yields ``None`` rather than a refresh envelope: sealing a dead
token would hand the client a full-TTL-capped envelope the IdP will reject, so the exchange degrades to
an access-only response (the client re-authenticates at access expiry), mirroring how
:func:`_bridge_grant_from_token_response` refuses an already-elapsed access token instead of capping it."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
RefreshCredential,
)
if not isinstance(token_response, dict):
return None
refresh = token_response.get("refresh_token")
if not isinstance(refresh, str) or not refresh:
return None
lifetime = _classify_upstream_lifetime(token_response.get("refresh_expires_in"))
if lifetime == "expired":
return None
scope = token_response.get("scope")
return RefreshCredential(
refresh_token=SecretStr(refresh),
scope=scope if isinstance(scope, str) and scope else None,
expires_in=lifetime if isinstance(lifetime, int) else None,
)
def _mint_refresh_envelope_value(
identity: "EnvelopeIdentity", token_response: object, keys: "EnvelopeKeys", now: datetime, mcp_server: MCPServer
) -> str | None:
"""Seal the upstream refresh grant (if any) into a refresh envelope and return its bearer string, or
``None`` when the upstream returned no refresh token or the refresh token is too large to seal. A
too-large refresh token degrades to an access-only response (logged) rather than failing an exchange
that already succeeded upstream: the client simply re-authenticates when the access envelope expires."""
from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import ( # noqa: PLC0415 # inline import avoids a module-load circular import
build_bridge_refresh_token_response,
)
from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import ( # noqa: PLC0415 # inline import avoids a module-load circular import
SealedEnvelope,
)
refresh_credential = _upstream_refresh_credential(token_response)
if refresh_credential is None:
return None
sealed = build_bridge_refresh_token_response(identity, refresh_credential, keys, now)
if isinstance(sealed, SealedEnvelope):
return sealed.token.get_secret_value()
verbose_logger.warning(
"bridge mint: the upstream refresh token is too large to seal into a refresh envelope for "
"server=%s; issuing an access-only response, so the client re-authenticates at access expiry",
mcp_server.server_id,
)
return None

57
litellm/proxy/_experimental/mcp_server/db.py
View file

@ -52,7 +52,6 @@ _AUTH_FLOW_SCOPED_FIELDS: frozenset = frozenset(
"token_url",
"registration_url",
"oauth2_flow",
"dcr_bridge",
"token_exchange_endpoint",
"audience",
"subject_token_type",
@ -76,34 +75,6 @@ _TOKEN_EXCHANGE_COLUMN_FIELDS: frozenset = frozenset(
}
)
# The client-forwarded token modes share one stored-credential shape: the admin-declared upstream
# OAuth app (client_id/client_secret) plus the same authorize relay, and neither mints anything the
# gateway keeps. So a switch WITHIN this class must preserve the stored app, unlike a cross-class
# switch (e.g. an oauth2 row whose client may be DCR-minted and is not reusable elsewhere).
_CLIENT_FORWARDED_AUTH_TYPES: frozenset = frozenset({"true_passthrough", "oauth_delegate"})
# Minted token material that must never survive a client rotation on a persisted row.
_MINTED_TOKEN_CREDENTIAL_FIELDS: frozenset = frozenset({"access_token", "refresh_token", "expires_in"})
def _credential_auth_class(auth_type: Optional[str]) -> Optional[str]:
"""Collapse the client-forwarded modes to one credential class; every other auth_type is its own
class. Used so credential handling keys off whether the stored-credential shape actually changed,
not off a raw auth_type inequality that treats true_passthrough<->oauth_delegate as a full reset."""
if auth_type in _CLIENT_FORWARDED_AUTH_TYPES:
return "client_forwarded"
return auth_type
def _drop_stale_minted_on_client_rotation(merged: Dict[str, Any], new_creds: Dict[str, Any]) -> Dict[str, Any]:
"""When the update rotates the client, drop stale minted token keys it did not itself set, so an old
app's access/refresh token never rides forward under the new client. A no-op when no client key changed."""
if "client_id" not in new_creds and "client_secret" not in new_creds:
return merged
return {
key: value for key, value in merged.items() if key not in _MINTED_TOKEN_CREDENTIAL_FIELDS or key in new_creds
}
def _is_global_env_var_scope(scope: Any) -> bool:
"""``scope="user"`` entries are placeholders the user fills in; everything
@ -707,9 +678,7 @@ async def update_mcp_server(
existing = await MCPServerRepository(prisma_client).table.find_unique(where={"server_id": data.server_id})
auth_type_changed = bool(
data.auth_type
and existing
and _credential_auth_class(existing.auth_type) != _credential_auth_class(data.auth_type)
data.auth_type and existing and existing.auth_type is not None and existing.auth_type != data.auth_type
)
# Clear stale credentials when auth_type changes but no new credentials provided
@ -742,12 +711,11 @@ async def update_mcp_server(
# would wipe encrypted secrets that the UI cannot display back.
if "credentials" in data_dict and data_dict["credentials"] is not None:
if existing and existing.credentials:
# Only merge when the credential CLASS is unchanged. A cross-class switch
# (e.g. oauth2 → api_key, or oauth2 → true_passthrough) replaces credentials
# entirely to avoid stale secrets from the previous class lingering; a switch
# within the client-forwarded class (true_passthrough ↔ oauth_delegate) keeps
# the same declared app and so must merge, not replace.
if not auth_type_changed:
# Only merge when auth_type is unchanged. Switching auth types
# (e.g. oauth2 → api_key) should replace credentials entirely
# to avoid stale secrets from the previous auth type lingering.
auth_type_unchanged = data.auth_type is None or data.auth_type == existing.auth_type
if auth_type_unchanged:
existing_creds = (
json.loads(existing.credentials)
if isinstance(existing.credentials, str)
@ -758,9 +726,8 @@ async def update_mcp_server(
if isinstance(data_dict["credentials"], str)
else dict(data_dict["credentials"])
)
# New values override existing; existing keys not in update are preserved. A client
# rotation additionally drops the previous app's stale minted token keys.
merged = _drop_stale_minted_on_client_rotation({**existing_creds, **new_creds}, new_creds)
# New values override existing; existing keys not in update are preserved
merged = {**existing_creds, **new_creds}
# Migrate-on-write for legacy rows: token-exchange settings the
# old blob shape carried move to their dedicated columns (unless
# the caller set the column this update, or the row already has
@ -779,14 +746,6 @@ async def update_mcp_server(
# Add audit fields
data_dict["updated_by"] = touched_by
# prisma-python rejects a raw ``None`` for a ``Json?`` field ("value is required but not set"); the
# clear paths above use ``None`` as the merge-skip sentinel, so translate it here to ``Json(None)``,
# which writes SQL null and reads back as ``None``. Done at the edge so the merge guards stay simple.
if "credentials" in data_dict and data_dict["credentials"] is None:
from prisma import Json # noqa: PLC0415 # local import: prisma may be ungenerated at module load in some tools
data_dict["credentials"] = Json(None)
updated_mcp_server = await MCPServerRepository(prisma_client).table.update(
where={"server_id": data.server_id},
data=data_dict, # type: ignore

570
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -10,7 +10,7 @@ from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
import httpx
from fastapi import APIRouter, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response
from pydantic import BaseModel, ConfigDict, Field, SecretStr, ValidationError
from pydantic import BaseModel, ValidationError
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import (
@ -21,24 +21,6 @@ from litellm.proxy._experimental.mcp_server.auth.token_endpoint_auth import (
TokenEndpointAuthConfigError,
build_token_endpoint_client_auth,
)
from litellm.proxy._experimental.mcp_server.bridge_token_flow import (
_bridge_mint_error_response,
_BridgeMintReady,
_BridgeRefreshReady,
_extract_user_id_from_request,
_finish_bridge_mint,
_prepare_bridge_mint,
_prepare_bridge_refresh,
)
from litellm.proxy._experimental.mcp_server.faults import (
CallerRejected,
CredentialSource,
UpstreamProtocolFault,
classify_upstream_dcr_rejection,
classify_upstream_token_rejection,
dcr_fault_detail,
render_token_fault,
)
from litellm.proxy._experimental.mcp_server.oauth_utils import (
TOKEN_NO_CACHE_HEADERS,
get_request_base_url,
@ -55,7 +37,7 @@ from litellm.types.mcp import MCPAuth, MCPCredentials
from litellm.types.mcp_server.mcp_server_manager import MCPServer
if TYPE_CHECKING:
from litellm.proxy._types import LiteLLM_MCPServerTable
from litellm.proxy._types import LiteLLM_MCPServerTable, UserAPIKeyAuth
# TTL cache for upstream OAuth metadata fetched from pass-through MCP servers.
# Keeps us from hammering the upstream IdP on each discovery request.
@ -109,8 +91,6 @@ def encode_state_with_base_url(
code_challenge: Optional[str] = None,
code_challenge_method: Optional[str] = None,
client_redirect_uri: Optional[str] = None,
litellm_user_id: str | None = None,
mcp_server_id: str | None = None,
) -> str:
"""
Encode the base_url, original state, and PKCE parameters using encryption.
@ -121,11 +101,6 @@ def encode_state_with_base_url(
code_challenge: PKCE code challenge from client
code_challenge_method: PKCE code challenge method from client
client_redirect_uri: Original redirect_uri from client
litellm_user_id: The SSO-authenticated litellm user captured at the bridge authorize
(interactive dcr_bridge oauth_delegate only); the callback seals it into the gateway
authorization code so the token mint can bind the envelope to this user
mcp_server_id: The bridge server the interactive flow targets, sealed alongside
litellm_user_id so the gateway code cannot be replayed against another server
Returns:
An encrypted string that encodes all values
@ -136,8 +111,6 @@ def encode_state_with_base_url(
"code_challenge": code_challenge,
"code_challenge_method": code_challenge_method,
"client_redirect_uri": client_redirect_uri,
"litellm_user_id": litellm_user_id,
"mcp_server_id": mcp_server_id,
}
state_json = json.dumps(state_data, sort_keys=True)
encrypted_state = encrypt_value_helper(state_json)
@ -165,68 +138,6 @@ def decode_state_hash(encrypted_state: str) -> dict:
return state_data
_BRIDGE_AUTH_CODE_PREFIX = "llm_bcode_"
class _BridgeAuthorizationCode(BaseModel):
"""The identity and upstream code the gateway seals into the authorization code it hands a DCR
client for an interactive dcr_bridge oauth_delegate sign-in, recovered at the token endpoint."""
model_config = ConfigDict(frozen=True)
upstream_code: str = Field(min_length=1)
litellm_user_id: str = Field(min_length=1)
mcp_server_id: str = Field(min_length=1)
def is_bridge_authorization_code(code: str) -> bool:
"""Cheap prefix check that ``code`` is a gateway-sealed bridge authorization code rather than a
raw upstream code, so the token endpoint can route without decrypting."""
return code.startswith(_BRIDGE_AUTH_CODE_PREFIX)
def seal_bridge_authorization_code(upstream_code: str, litellm_user_id: str, mcp_server_id: str) -> str:
"""Seal the upstream authorization code and the SSO-captured litellm user into a gateway
authorization code. The DCR client only echoes this opaque value back at the token endpoint; the
gateway decrypts it there to recover the user (to bind the envelope) and the upstream code (to
exchange with the upstream), so a litellm identity captured in the browser at authorize survives
to the back-channel token call with nothing stored server-side. Encrypted with the repo's
authenticated symmetric helper (the same family the OAuth state uses), so the client can neither
read nor forge it."""
payload = json.dumps(
{"upstream_code": upstream_code, "litellm_user_id": litellm_user_id, "mcp_server_id": mcp_server_id},
sort_keys=True,
)
return _BRIDGE_AUTH_CODE_PREFIX + encrypt_value_helper(payload)
def open_bridge_authorization_code(code: str) -> _BridgeAuthorizationCode | None:
"""Recover the sealed identity and upstream code, or ``None`` when ``code`` is not a gateway
bridge code or does not decrypt / validate. Total over hostile input: a raw upstream code (the
scripted two-header path) returns ``None`` and the caller falls through to the existing
behavior."""
if not is_bridge_authorization_code(code):
return None
decrypted = decrypt_value_helper(
code[len(_BRIDGE_AUTH_CODE_PREFIX) :], "bridge_authorization_code", return_original_value=False
)
if not isinstance(decrypted, str):
return None
try:
return _BridgeAuthorizationCode.model_validate_json(decrypted)
except ValidationError:
return None
def _redirect_to_litellm_login(request: Request) -> RedirectResponse:
"""Send an unauthenticated browser through litellm login before the interactive bridge authorize
can capture its identity. The bridge oauth_delegate flow seals the SSO user into the gateway code,
so a session is required; without one there is nothing to bind. After login the user re-initiates
the connection, which then finds the session cookie (the seamless return-to round-trip, which is
origin-validated against the control-plane URL, is a follow-up)."""
base_url = get_request_base_url(request)
return RedirectResponse(f"{base_url}/sso/key/generate")
# LIT-4197: some upstream authorization servers reject an over-long ``state``
# (the encrypted OAuth session blob routinely exceeds their limit). The upstream
# only needs an opaque value it echoes back on ``/callback``, so we forward a
@ -393,6 +304,90 @@ def _validate_token_response(
)
def _litellm_key_from_request(request: Request) -> Optional[str]:
"""Return the LiteLLM API key presented on the request, or ``None``.
Accepts the key from ``x-litellm-api-key`` (what MCP clients such as Claude Desktop/Code
send) as well as ``Authorization``; either may carry a bare token or ``Bearer <token>``.
``x-litellm-api-key`` wins when both are present, since ``Authorization`` may instead carry
an OAuth/upstream bearer.
"""
for header_value in (
request.headers.get("x-litellm-api-key"),
request.headers.get("Authorization") or request.headers.get("authorization"),
):
if not header_value:
continue
value = header_value.strip()
if value.lower().startswith("bearer "):
value = value[7:].strip()
if value:
return value
return None
def _active_key_user_id(key_obj: "UserAPIKeyAuth") -> Optional[str]:
"""The key's ``user_id``, or ``None`` if the key is blocked or expired.
The OAuth token endpoint is unauthenticated, so the presented key is validated here before its
identity is trusted to key a stored credential; a revoked or expired key must not be able to
write or overwrite the per-user OAuth token. ``get_key_object`` resolves a row without these
checks (the main ``user_api_key_auth`` pipeline enforces them downstream, which this endpoint
bypasses), so they are applied here. Deleted keys are already rejected upstream, where
``get_key_object`` raises on a row that no longer exists.
"""
if key_obj.blocked is True:
return None
expires = key_obj.expires
if expires is not None:
expiry = expires if isinstance(expires, datetime) else datetime.fromisoformat(expires)
if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
expiry = expiry.replace(tzinfo=timezone.utc)
if expiry < datetime.now(timezone.utc):
return None
return key_obj.user_id
async def _extract_user_id_from_request(request: Request) -> Optional[str]:
"""Resolve the LiteLLM ``user_id`` at the OAuth token endpoint so a per-user token is stored
under the same identity the egress later reads it by (``user_api_key_auth.user_id``).
Resolves authoritatively via ``get_key_object`` (cache first, then DB) instead of a raw cache
peek. On a multi-replica gateway the token-exchange request can land on a worker whose in-memory
cache never saw the key, and a cross-replica Redis hit deserializes to a plain ``dict`` rather
than a ``UserAPIKeyAuth``; the previous code read only ``Authorization`` and did
``getattr(cached, "user_id")`` with no ``model_type`` rehydration and no DB fallback, so it
silently returned ``None`` and the token was never persisted, which makes the egress 401 on every
reconnect. The resolved key is validated (``_active_key_user_id``) before its identity is trusted,
so a blocked or expired key cannot write. Returns ``None`` when no key is present, the key cannot
be resolved, or it is blocked/expired.
"""
token = _litellm_key_from_request(request)
if not token:
return None
try:
from litellm.proxy._types import hash_token # noqa: PLC0415
from litellm.proxy.auth.auth_checks import get_key_object # noqa: PLC0415
from litellm.proxy.proxy_server import ( # noqa: PLC0415
prisma_client,
user_api_key_cache,
)
key_obj = await get_key_object(
hashed_token=hash_token(token),
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
return _active_key_user_id(key_obj)
except Exception as exc:
verbose_logger.debug(
"_extract_user_id_from_request: could not resolve a LiteLLM user_id for the presented "
"key (%s); per-user token will not be stored server-side.",
type(exc).__name__,
)
return None
async def _store_per_user_token_server_side(
server: MCPServer,
user_id: str,
@ -476,8 +471,7 @@ def _raise_if_not_oauth2(mcp_server: MCPServer) -> None:
through: the caller owns the upstream token, and this relayed flow is how a browser obtains
one against the upstream IdP (the admin UI's browser-only Authorize uses it). The minted
token is upstream-audienced and held by the caller; the gateway persists nothing for these
modes (``_persist_dcr_client_registration`` skips them unconditionally, so even the admin
Authorize path with ``persist_credentials`` enabled writes nothing to the server row).
modes (DCR persistence is opt-in and never enabled on this path).
"""
from litellm.proxy._experimental.mcp_server.mcp_server_manager import ( # noqa: PLC0415 # circular import with mcp_server_manager at module load
_UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES,
@ -504,86 +498,23 @@ def _raise_unless_oauth2_discovery_server(
mcp_server_name: Optional[str],
description: str,
) -> None:
"""404 a NAMED discovery request unless it resolves to an oauth2 or DCR-bridge server.
"""404 a NAMED discovery request unless it resolves to an oauth2 server.
A named server that is unknown (or hidden from the caller) and one that exists
but is non-oauth2 both return the same 404, so the well-known discovery paths
cannot be used to enumerate non-OAuth server names. Root discovery (no name) is
unaffected, and pass-through servers are resolved by the caller before this runs.
DCR-bridge servers are admitted because they serve the gateway's own authorization
server metadata (the register, authorize, and token relays).
"""
if mcp_server_name is None:
return
if mcp_server is not None and mcp_server.auth_type == MCPAuth.oauth2:
return
if mcp_server is not None and mcp_server.is_dcr_bridge:
return
raise HTTPException(
status_code=404,
detail=f"MCP server '{mcp_server_name}' is {description}",
)
def _dcr_bridge_relays_client_registration(mcp_server: MCPServer) -> bool:
"""True when a DCR-bridge server relays client registration to the upstream authorization
server instead of short-circuiting to an admin-configured OAuth client. In the relay arm the
upstream holds each client's own registration, so the authorize and token relays pass the
client's ``client_id`` and ``redirect_uri`` through verbatim and the authorization code
returns directly to the client's redirect URI without transiting the gateway. Gateway-side
redirect trust and the ``/callback`` state relay therefore only apply to the short-circuit
arm, where the upstream only knows the gateway's own callback."""
return mcp_server.is_dcr_bridge and bool(mcp_server.registration_url) and not mcp_server.client_id
def _require_s256_pkce(
code_challenge: Optional[str],
code_challenge_method: Optional[str],
) -> Tuple[str, str]:
"""DCR-bridge servers serve unauthenticated public OAuth clients, so the PKCE downgrade
paths (no challenge, or a non-S256 method; RFC 7636 defaults a missing method to ``plain``)
are rejected at the gateway instead of relying on upstream enforcement. Returns the
validated pair so callers get non-optional values."""
if code_challenge and code_challenge_method == "S256":
return code_challenge, code_challenge_method
raise HTTPException(
status_code=400,
detail=(
"This server requires PKCE: send code_challenge with "
"code_challenge_method=S256 on the authorization request"
),
)
def _redirect_to_upstream_authorize(
*,
mcp_server: MCPServer,
client_id: str,
redirect_uri: str,
state: str,
code_challenge: str,
code_challenge_method: str,
response_type: Optional[str],
scope: Optional[str],
) -> RedirectResponse:
"""The bridge relay arm's authorize redirect: every client-supplied parameter passes through
to the upstream authorize endpoint verbatim, no relay state cookie is set, and the upstream
enforces its own registered redirect binding for the client."""
scope_value = scope or (" ".join(mcp_server.scopes) if mcp_server.scopes else None)
passthrough_params = {
"client_id": client_id,
"redirect_uri": redirect_uri,
"state": state,
"response_type": response_type or "code",
"code_challenge": code_challenge,
"code_challenge_method": code_challenge_method,
**({"scope": scope_value} if scope_value else {}),
}
parsed_auth_url = urlparse(mcp_server.authorization_url or "")
merged_params = {**dict(parse_qsl(parsed_auth_url.query)), **passthrough_params}
return RedirectResponse(urlunparse(parsed_auth_url._replace(query=urlencode(merged_params))))
async def authorize_with_server(
request: Request,
mcp_server: MCPServer,
@ -599,24 +530,6 @@ async def authorize_with_server(
if mcp_server.authorization_url is None:
raise HTTPException(status_code=400, detail="MCP server authorization url is not set")
if mcp_server.is_dcr_bridge:
# Enforce S256 PKCE on both bridge arms. The relay arm forwards the validated,
# now-non-optional pair to the upstream authorize; the short-circuit arm keeps
# calling this for its enforcement side effect, then falls through to the gateway
# /callback flow below, which reads the original code_challenge names.
bridge_challenge, bridge_method = _require_s256_pkce(code_challenge, code_challenge_method)
if _dcr_bridge_relays_client_registration(mcp_server):
return _redirect_to_upstream_authorize(
mcp_server=mcp_server,
client_id=client_id,
redirect_uri=redirect_uri,
state=state,
code_challenge=bridge_challenge,
code_challenge_method=bridge_method,
response_type=response_type,
scope=scope,
)
# Trusted redirect_uri: same-origin, loopback, or ops-allowlisted.
# The URI is encrypted into the OAuth state and decoded on
# /callback to redirect the user back; a non-trusted URI would be
@ -625,31 +538,12 @@ async def authorize_with_server(
parsed = urlparse(redirect_uri)
base_url = urlunparse(parsed._replace(query=""))
request_base_url = get_request_base_url(request)
# Interactive dcr_bridge oauth_delegate sign-in: this arm runs the gateway /callback and /token in
# the loop, so the gateway can capture the litellm user here (from the browser's UI session) and
# carry it to the back-channel token mint. Seal the SSO user and the target server into the state;
# the callback reads them back to mint the gateway authorization code. A DCR client cannot present a
# litellm key, so the browser session is the only identity source; without one there is nothing to
# bind, so send the user through login first. Every other oauth2 server keeps the identity-less state.
litellm_user_id: str | None = None
if mcp_server.is_dcr_bridge and mcp_server.is_oauth_delegate:
from litellm.proxy._experimental.mcp_server.byok_oauth_endpoints import ( # noqa: PLC0415 # inline import avoids a module-load circular import
_user_id_from_session_cookie,
)
litellm_user_id = _user_id_from_session_cookie(request)
if litellm_user_id is None:
return _redirect_to_litellm_login(request)
encoded_state = encode_state_with_base_url(
base_url=base_url,
original_state=state,
code_challenge=code_challenge,
code_challenge_method=code_challenge_method,
client_redirect_uri=redirect_uri,
litellm_user_id=litellm_user_id,
mcp_server_id=mcp_server.server_id if litellm_user_id else None,
)
relay_state = secrets.token_urlsafe(_OAUTH_STATE_HANDLE_BYTES)
@ -678,13 +572,6 @@ async def authorize_with_server(
return response
def _token_credential_source(mcp_server: MCPServer) -> CredentialSource:
"""Mirrors the resolved-client rule in :func:`exchange_token_with_server`: when the server has a
stored client_id the gateway presents its own credentials upstream, so a credential rejection is
the operator's fault, not the caller's."""
return "gateway_stored" if mcp_server.client_id else "caller_supplied"
async def exchange_token_with_server(
request: Request,
mcp_server: MCPServer,
@ -719,123 +606,50 @@ async def exchange_token_with_server(
except TokenEndpointAuthConfigError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
bridge_identity: _BridgeAuthorizationCode | None = None
bridge_mint_ready: _BridgeMintReady | None = None
bridge_upstream_refresh: SecretStr | None = None
bridge_upstream_scope: str | None = None
refresh_request_scope: str | None = None
is_bridge = mcp_server.is_oauth_delegate and mcp_server.is_dcr_bridge
if grant_type == "refresh_token":
# Phase 1 for a bridge refresh: open the client's refresh envelope, re-validate the sealed
# identity, and unwrap the real upstream refresh token BEFORE building token_data, so the exchange
# sends the upstream token and never the envelope. A failure returns without touching the upstream.
if is_bridge:
prepared_refresh = await _prepare_bridge_refresh(mcp_server, refresh_token)
if not isinstance(prepared_refresh, _BridgeRefreshReady):
return _bridge_mint_error_response(prepared_refresh)
bridge_mint_ready = prepared_refresh.ready
bridge_upstream_refresh = prepared_refresh.upstream_refresh_token
bridge_upstream_scope = prepared_refresh.upstream_scope
# A bridge server sends the unwrapped upstream refresh token recovered from the client's refresh
# envelope above; every other server sends the client's own refresh token verbatim.
upstream_refresh_token = (
bridge_upstream_refresh.get_secret_value() if bridge_upstream_refresh is not None else refresh_token
)
if not upstream_refresh_token:
if not refresh_token:
raise HTTPException(
status_code=400,
detail="refresh_token is required for refresh_token grant",
)
token_data: dict = {
"grant_type": "refresh_token",
"refresh_token": upstream_refresh_token,
"refresh_token": refresh_token,
**client_auth.body,
}
refresh_request_scope = scope or bridge_upstream_scope
if refresh_request_scope:
token_data["scope"] = refresh_request_scope
if scope:
token_data["scope"] = scope
else:
if not code:
raise HTTPException(
status_code=400,
detail="code is required for authorization_code grant",
)
# Interactive dcr_bridge oauth_delegate: the client presents the gateway authorization code the
# callback sealed. Recover the SSO user and the real upstream code from it; the upstream exchange
# below uses the upstream code, and the mint binds the envelope to the recovered user. Bind the
# sealed server to this request so a code minted for one bridge server cannot be spent at another.
# A raw upstream code (scripted path) opens to None and the code is used as-is.
bridge_identity = open_bridge_authorization_code(code)
if bridge_identity is not None:
if bridge_identity.mcp_server_id != mcp_server.server_id:
raise HTTPException(
status_code=400,
detail="Authorization code was issued for a different MCP server",
)
code = bridge_identity.upstream_code
bridge_token_relay = _dcr_bridge_relays_client_registration(mcp_server)
if bridge_token_relay and not redirect_uri:
raise HTTPException(
status_code=400,
detail=(
"redirect_uri is required for the authorization_code grant on this server; "
"send the same redirect_uri used on the authorization request"
),
)
proxy_base_url = get_request_base_url(request)
resolved_redirect_uri = redirect_uri if bridge_token_relay else f"{proxy_base_url}/callback"
token_data = {
"grant_type": "authorization_code",
"code": code,
"redirect_uri": resolved_redirect_uri,
"redirect_uri": f"{proxy_base_url}/callback",
**client_auth.body,
}
if code_verifier:
token_data["code_verifier"] = code_verifier
# Phase 1 for a bridge authorization_code mint: resolve identity (the SSO user recovered above, or
# the presented litellm key) and the envelope keys BEFORE the exchange consumes the single-use code.
if is_bridge:
prepared = await _prepare_bridge_mint(request, mcp_server, bridge_identity)
if not isinstance(prepared, _BridgeMintReady):
return _bridge_mint_error_response(prepared)
bridge_mint_ready = prepared
async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Check)
try:
response = await async_client.post(
mcp_server.token_url,
headers={"Accept": "application/json", **client_auth.headers},
data=token_data,
)
if response is not None:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
fault = classify_upstream_token_rejection(
exc.response,
credential_source=_token_credential_source(mcp_server),
log_context=mcp_server.server_id,
)
upstream_rejected_bridge_refresh = (
is_bridge
and grant_type == "refresh_token"
and isinstance(fault, CallerRejected)
and fault.code == "invalid_grant"
)
if upstream_rejected_bridge_refresh:
verbose_logger.info(
"bridge refresh: the upstream rejected the sealed refresh token for server=%s with "
"invalid_grant (revoked or expired at the IdP); returning invalid_grant so the client "
"re-runs authorization_code rather than an opaque upstream error",
mcp_server.server_id,
)
return _bridge_mint_error_response("invalid_refresh")
return render_token_fault(fault)
response = await async_client.post(
mcp_server.token_url,
headers={"Accept": "application/json", **client_auth.headers},
data=token_data,
)
if response is None:
raise HTTPException(
status_code=502,
detail="MCP upstream token endpoint returned no response",
)
response.raise_for_status()
token_response = response.json()
access_token = token_response["access_token"]
# Validate token response against server-configured rules before any storage.
# This rejects tokens from wrong Slack workspaces, Atlassian orgs, etc.
@ -875,23 +689,8 @@ async def exchange_token_with_server(
mcp_server.server_id,
)
# A DCR-bridge oauth_delegate server hands the client a gateway-bound envelope (identity plus the
# upstream token) instead of the raw upstream token, so the one bearer both admits the caller and
# forwards the upstream credential. Only this mode mints; every other server returns the raw token.
if bridge_mint_ready is not None:
if refresh_request_scope and isinstance(token_response, dict) and not token_response.get("scope"):
token_response = {**token_response, "scope": refresh_request_scope}
# Phase 3: seal the upstream grant into the client-held envelope; failures map through the same
# OAuth-shaped response as the phase-1 preconditions.
minted = _finish_bridge_mint(bridge_mint_ready, mcp_server, token_response, datetime.now(timezone.utc))
return minted if isinstance(minted, JSONResponse) else _bridge_mint_error_response(minted)
raw_access_token = token_response.get("access_token") if isinstance(token_response, dict) else None
if not isinstance(raw_access_token, str) or not raw_access_token:
return render_token_fault(UpstreamProtocolFault(note="the upstream token response has no usable access_token"))
result = {
"access_token": raw_access_token,
"access_token": access_token,
"token_type": token_response.get("token_type", "Bearer"),
}
@ -919,22 +718,6 @@ class _PersistedDcrCredentials(BaseModel):
client_id: Optional[str] = None
client_secret: Optional[str] = None
token_endpoint_auth_method: Optional[str] = None
redirect_uris: Optional[list[str]] = None
def _redirect_uri_not_registered(credentials: _PersistedDcrCredentials, current_redirect_uri: str) -> bool:
"""Whether a persisted DCR client is positively known NOT to cover the current callback.
A DCR client is bound to the redirect_uris it was registered with; if the proxy's
resolved public origin has since changed, every authorize built for it will be
rejected by the IdP. Clients persisted before ``redirect_uris`` was recorded (and
admin-configured clients, which never get a recording) return False so they are
grandfathered rather than re-registered, because re-minting a client_id orphans
every user's refresh tokens for that server."""
recorded = credentials.redirect_uris
if not recorded:
return False
return current_redirect_uri not in recorded
def _get_persisted_dcr_credentials(credentials: object) -> Optional[_PersistedDcrCredentials]:
@ -1001,23 +784,11 @@ async def _get_persisted_mcp_server_with_dcr_client_id(
return persisted_mcp_server, credentials
async def _reuse_persisted_dcr_client_if_available(
mcp_server: MCPServer, current_redirect_uri: Optional[str] = None
) -> bool:
async def _reuse_persisted_dcr_client_if_available(mcp_server: MCPServer) -> bool:
persisted = await _get_persisted_mcp_server_with_dcr_client_id(mcp_server)
if persisted is None:
return False
persisted_mcp_server, credentials = persisted
if current_redirect_uri is not None and _redirect_uri_not_registered(credentials, current_redirect_uri):
verbose_logger.debug(
"register_client_with_server: not reusing persisted DCR client for server_id=%s; its registered "
"redirect_uris=%s do not include the current callback %s. The operator-facing warning for this "
"re-registration event is emitted once by _persisted_dcr_redirect_uri_is_stale.",
mcp_server.server_id,
credentials.redirect_uris,
current_redirect_uri,
)
return False
if not _apply_persisted_dcr_credentials(mcp_server, credentials):
return False
@ -1036,36 +807,11 @@ async def _reuse_persisted_dcr_client_if_available(
return bool(mcp_server.client_id)
async def _persisted_dcr_redirect_uri_is_stale(mcp_server: MCPServer, current_redirect_uri: str) -> bool:
"""Whether the server's persisted DCR client is bound to redirect_uris that no longer
cover the current proxy callback, meaning authorize is guaranteed to fail IdP-side.
Consulted when the in-memory server already carries a hydrated client_id, which
otherwise short-circuits registration before any redirect check can run. Servers
without a persisted DCR recording (admin-configured client_id, or registered before
redirect_uris were recorded) are never reported stale."""
persisted = await _get_persisted_mcp_server_with_dcr_client_id(mcp_server)
if persisted is None:
return False
_, credentials = persisted
if not _redirect_uri_not_registered(credentials, current_redirect_uri):
return False
verbose_logger.warning(
"register_client_with_server: persisted DCR client for server_id=%s is registered with redirect_uris=%s "
"which do not include the current callback %s (proxy origin changed); registering a replacement client. "
"Users previously signed in to this server will need to re-authenticate.",
mcp_server.server_id,
credentials.redirect_uris,
current_redirect_uri,
)
return True
DcrRegistrationPersistenceResult = Literal["persisted", "reused", "skipped", "failed"]
DcrRegistrationPersistenceResult = Literal["persisted", "reused", "failed"]
async def _persist_dcr_client_registration(
mcp_server: MCPServer, registration_response: object, current_redirect_uri: str
mcp_server: MCPServer, registration_response: object
) -> DcrRegistrationPersistenceResult:
"""Persist the dynamically registered OAuth client (RFC 7591) onto the MCP server row.
@ -1075,23 +821,7 @@ async def _persist_dcr_client_registration(
full re-authorization instead of a silent refresh. Mirrors the ``encrypt_credentials``
write that ``client_credentials`` and token exchange already use. Failures are logged,
never raised: registration still returns to the caller even when persistence fails.
The client-forwarded token modes (``true_passthrough`` / ``oauth_delegate``) are skipped
unconditionally: the caller holds the upstream token and the gateway must hold no OAuth
client identity for these servers. Persisting here would stamp ``oauth2_flow`` and a
``client_id`` onto a server whose mode promises the gateway stores nothing, making a
fresh pass-through server read as gateway-authorized.
``redirect_uris`` records what the client is bound to so a later origin change can be
detected as a positive mismatch and trigger re-registration instead of stranding the
server on IdP-side redirect_uri rejections. ``client_secret`` and
``token_endpoint_auth_method`` are written explicitly (None when absent) because
``update_mcp_server`` merges credential blobs: a re-registered public client must not
inherit the previous client's secret or auth method.
"""
if mcp_server.is_true_passthrough or mcp_server.is_oauth_delegate:
return "skipped"
try:
registration = _DcrClientRegistration.model_validate(registration_response)
except ValidationError as exc:
@ -1103,16 +833,17 @@ async def _persist_dcr_client_registration(
)
return "failed"
if await _reuse_persisted_dcr_client_if_available(mcp_server, current_redirect_uri=current_redirect_uri):
if await _reuse_persisted_dcr_client_if_available(mcp_server):
return "reused"
credentials: MCPCredentials = {
"client_id": registration.client_id,
"client_secret": registration.client_secret,
"token_endpoint_auth_method": (
"client_secret_basic" if registration.token_endpoint_auth_method == "client_secret_basic" else None
**({"client_secret": registration.client_secret} if registration.client_secret is not None else {}),
**(
{"token_endpoint_auth_method": "client_secret_basic"}
if registration.token_endpoint_auth_method == "client_secret_basic"
else {}
),
"redirect_uris": [current_redirect_uri],
}
from litellm.proxy._experimental.mcp_server.db import update_mcp_server # noqa: PLC0415
@ -1156,28 +887,19 @@ async def register_client_with_server(
token_endpoint_auth_method: Optional[str],
fallback_client_id: Optional[str] = None,
persist_credentials: bool = False,
client_redirect_uris: Optional[list] = None,
):
_raise_if_not_oauth2(mcp_server)
request_base_url = get_request_base_url(request)
current_redirect_uri = f"{request_base_url}/callback"
dummy_return = {
"client_id": fallback_client_id or mcp_server.server_name,
"client_secret": "dummy",
"redirect_uris": [current_redirect_uri],
"redirect_uris": [f"{request_base_url}/callback"],
}
if mcp_server.client_id and not (
persist_credentials
and mcp_server.registration_url
and await _persisted_dcr_redirect_uri_is_stale(mcp_server, current_redirect_uri)
):
if mcp_server.client_id:
return dummy_return
if await _reuse_persisted_dcr_client_if_available(
mcp_server,
current_redirect_uri=current_redirect_uri if persist_credentials else None,
):
if await _reuse_persisted_dcr_client_if_available(mcp_server):
return dummy_return
if mcp_server.authorization_url is None:
@ -1186,19 +908,12 @@ async def register_client_with_server(
if mcp_server.registration_url is None:
return dummy_return
bridge_relay = _dcr_bridge_relays_client_registration(mcp_server)
if bridge_relay and not client_redirect_uris:
raise HTTPException(
status_code=400,
detail="redirect_uris is required to register a client with this server",
)
register_data = {
"client_name": client_name,
"redirect_uris": client_redirect_uris if bridge_relay else [current_redirect_uri],
"grant_types": grant_types or (["authorization_code", "refresh_token"] if bridge_relay else []),
"response_types": response_types or (["code"] if bridge_relay else []),
"token_endpoint_auth_method": token_endpoint_auth_method or ("none" if bridge_relay else ""),
"redirect_uris": [f"{request_base_url}/callback"],
"grant_types": grant_types or [],
"response_types": response_types or [],
"token_endpoint_auth_method": token_endpoint_auth_method or "",
}
headers = {
"Content-Type": "application/json",
@ -1206,29 +921,22 @@ async def register_client_with_server(
}
async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Register)
try:
response = await async_client.post(
mcp_server.registration_url,
headers=headers,
json=register_data,
)
if response is not None:
response.raise_for_status()
except httpx.HTTPStatusError as exc:
status_code, detail = dcr_fault_detail(
classify_upstream_dcr_rejection(exc.response, log_context=mcp_server.server_id)
)
raise HTTPException(status_code=status_code, detail=detail) from exc
response = await async_client.post(
mcp_server.registration_url,
headers=headers,
json=register_data,
)
if response is None:
raise HTTPException(
status_code=502,
detail="MCP upstream registration endpoint returned no response",
)
response.raise_for_status()
token_response = response.json()
if persist_credentials and not bridge_relay:
persistence_result = await _persist_dcr_client_registration(mcp_server, token_response, current_redirect_uri)
if persist_credentials:
persistence_result = await _persist_dcr_client_registration(mcp_server, token_response)
if persistence_result == "reused":
return dummy_return
@ -1451,20 +1159,7 @@ async def callback(
# states while permitting same-origin / allowlisted clients.
redirect_uri = _get_validated_client_redirect_uri(request, state_data)
# Interactive dcr_bridge oauth_delegate: the state carries the litellm user the authorize step
# captured. Instead of forwarding the raw upstream code (which the client would present at the
# token endpoint with no way to prove who signed in), seal the user and the upstream code into a
# gateway authorization code and forward THAT. The token endpoint decrypts it to bind the
# envelope to this user. Every other flow forwards the raw code unchanged.
litellm_user_id = state_data.get("litellm_user_id")
mcp_server_id = state_data.get("mcp_server_id")
forwarded_code = code
if isinstance(litellm_user_id, str) and litellm_user_id and isinstance(mcp_server_id, str) and mcp_server_id:
forwarded_code = seal_bridge_authorization_code(
upstream_code=code, litellm_user_id=litellm_user_id, mcp_server_id=mcp_server_id
)
params = {"code": forwarded_code, "state": original_state}
params = {"code": code, "state": original_state}
complete_returned_url = _append_query_params(redirect_uri, params)
response = RedirectResponse(url=complete_returned_url, status_code=302)
_clear_oauth_state_cookie(response, request, state)
@ -1664,13 +1359,6 @@ async def _build_oauth_protected_resource_response(
else:
resource_url = f"{request_base_url}/mcp"
if mcp_server is not None and mcp_server_name and mcp_server.is_dcr_bridge:
return {
"authorization_servers": [f"{request_base_url}/{mcp_server_name}"],
"resource": resource_url,
"scopes_supported": (mcp_server.scopes if mcp_server.scopes else []),
}
# Pass-through branch: proxy the upstream's own metadata so discovery
# directs the client at the real IdP (Okta, Keycloak, …) instead of us.
if mcp_server is not None and (
@ -2000,7 +1688,6 @@ async def register_client(request: Request, mcp_server_name: Optional[str] = Non
response_types=data.get("response_types", []),
token_endpoint_auth_method=data.get("token_endpoint_auth_method", ""),
fallback_client_id=resolved.server_name or resolved.name,
client_redirect_uris=data.get("redirect_uris"),
)
return dummy_return
@ -2015,5 +1702,4 @@ async def register_client(request: Request, mcp_server_name: Optional[str] = Non
response_types=data.get("response_types", []),
token_endpoint_auth_method=data.get("token_endpoint_auth_method", ""),
fallback_client_id=mcp_server_name,
client_redirect_uris=data.get("redirect_uris"),
)

38
litellm/proxy/_experimental/mcp_server/faults/__init__.py
View file

@ -1,38 +0,0 @@
"""Typed fault values for upstream OAuth/DCR failures (phase 1 of the MCP error-handling framework).
The invariant this package exists to enforce: an upstream failure is classified ONCE into a single
fault value, and the response status, wire error code, and prose are all derived from that value.
Deriving all three from one classification makes contradictory pairings (a caller-fault error code on
a server-fault status) unrepresentable, and gives the trust-boundary rule one enforcement point:
spec-defined machine fields may cross to callers, upstream prose and raw bodies go to server logs.
"""
from litellm.proxy._experimental.mcp_server.faults.classify import (
classify_upstream_dcr_rejection,
classify_upstream_token_rejection,
)
from litellm.proxy._experimental.mcp_server.faults.render_oauth import (
dcr_fault_detail,
render_token_fault,
)
from litellm.proxy._experimental.mcp_server.faults.types import (
CallerRejected,
CredentialSource,
GatewayRejected,
UpstreamOAuthFault,
UpstreamProtocolFault,
UpstreamReportedFault,
)
__all__ = [
"CallerRejected",
"CredentialSource",
"GatewayRejected",
"UpstreamOAuthFault",
"UpstreamProtocolFault",
"UpstreamReportedFault",
"classify_upstream_dcr_rejection",
"classify_upstream_token_rejection",
"dcr_fault_detail",
"render_token_fault",
]

Some files were not shown because too many files have changed in this diff Show more