scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 36

chore: merge litellm_internal_staging into litellm_lit_4868_cache_write_split

Browse source
This commit is contained in:
mateo-berri 2026-08-14 13:57:47 -07:00
commit aaa619441e
979 changed files with 68673 additions and 30523 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
.circleci/config.yml
View file

@ -2744,84 +2744,6 @@ jobs:
file: ./coverage.xml
flags: circleci
ui_build:
docker:
- image: cimg/node:24.19@sha256:8966565f07189a67d64d6808a2b127f31dafae566508e3547f55640e1070bfad
auth:
username: ${DOCKERHUB_USERNAME}
password: ${DOCKERHUB_PASSWORD}
resource_class: medium+
working_directory: ~/project
steps:
- checkout
- skip_if_unrelated_changes:
category: client
- setup_google_dns
- restore_cache:
keys:
- ui-build-deps-v1-{{ checksum "ui/litellm-dashboard/package-lock.json" }}
- ui-build-deps-v1-
- restore_cache:
keys:
- ui-nextjs-cache-v1-{{ checksum "ui/litellm-dashboard/package-lock.json" }}
- ui-nextjs-cache-v1-
- run:
name: Install dependencies
command: |
cd ui/litellm-dashboard
npm ci
- save_cache:
key: ui-build-deps-v1-{{ checksum "ui/litellm-dashboard/package-lock.json" }}
paths:
- ui/litellm-dashboard/node_modules
- run:
name: Build UI
command: |
cd ui/litellm-dashboard
source ./build_ui.sh
- save_cache:
key: ui-nextjs-cache-v1-{{ checksum "ui/litellm-dashboard/package-lock.json" }}
paths:
- ui/litellm-dashboard/.next/cache
- persist_to_workspace:
root: .
paths:
- litellm/proxy/_experimental/out
ui_unit_tests:
docker:
- image: cimg/node:24.19@sha256:8966565f07189a67d64d6808a2b127f31dafae566508e3547f55640e1070bfad
auth:
username: ${DOCKERHUB_USERNAME}
password: ${DOCKERHUB_PASSWORD}
resource_class: xlarge
working_directory: ~/project
steps:
- checkout
- skip_if_unrelated_changes:
category: client
- setup_google_dns
- restore_cache:
keys:
- ui-unit-deps-v1-{{ checksum "ui/litellm-dashboard/package-lock.json" }}
- ui-unit-deps-v1-
- run:
name: Install dependencies
command: |
cd ui/litellm-dashboard
npm ci
- save_cache:
key: ui-unit-deps-v1-{{ checksum "ui/litellm-dashboard/package-lock.json" }}
paths:
- ui/litellm-dashboard/node_modules
- run:
name: Run UI unit tests (Vitest)
command: |
cd ui/litellm-dashboard
CI=true npm run test -- --run \
--pool forks --poolOptions.forks.maxForks=6
e2e_ui_testing:
docker:
- image: cimg/python:3.12-browsers@sha256:b432899af01c9a311bf74f4f22e9ada2e5306d4b1b4383f8d29e1228a5844ef2
@ -3181,12 +3103,6 @@ workflows:
filters: *main_branches
- litellm_router_unit_testing:
filters: *main_branches
- ui_build:
filters: *main_branches
- ui_unit_tests:
requires:
- ui_build
filters: *main_branches
- auth_ui_unit_tests:
filters: *main_branches
- proxy_behavior_tests:

56
.github/ISSUE_TEMPLATE/bug_report.yml vendored
View file

@ -23,30 +23,56 @@ body:
label: What happened?
description: Also tell us, what did you expect to happen?
placeholder: Tell us what you see!
value: "A bug happened!"
validations:
required: true
- type: textarea
id: steps-to-reproduce
id: user-flow
attributes:
label: Steps to Reproduce
description: Please provide a numbered list of the exact steps to reproduce this bug (include a curl/python snippet to reproduce it). Number each step (1., 2., 3., ...) in the order you performed them.
label: User Flow
description: |
Two ordered lists, "Before a (hypothetical) fix" and "After a (hypothetical) fix", walking the same end user through the same task, written strictly from that user's seat. Every rule below applies.
- Describe the real application and the routes its users actually hit, not a generic scenario
- Lead each list with one plain sentence saying where the flow fails (before) or would succeed (after), then number the steps
- Every step is something the user does or observes: the HTTP method and full URL they hit, what they sent, and what visibly came back (status code, error text, the shape of an ID). UI steps name the page URL and what is on screen
- No LiteLLM internals: never name functions, files, DB tables, config classes, hooks, callbacks, or code paths. "The upload hands back an ID that looks like OpenAI's own `file-abc123` instead of the scrambled one the gateway returned" is right, "no managed-file row was registered" is wrong
- Keep the two lists step-for-step identical until they diverge, so the broken step is obvious
- If the bug has a security or authorization consequence, end each list with what another user can do that they shouldn't be able to, and what they could no longer do after a fix
placeholder: |
1. config.yaml file/ .env file/ etc.
2. Run the following code...
3. Observe the error...
value: |
1.
2.
3.
Before a (hypothetical) fix: a developer whose app streams chat completions gets no token counts back, so their cost dashboard reads zero
1. They send POST https://litellm-domain/v1/chat/completions with "stream": true and no stream_options
2. The last SSE chunk arrives with "usage": null, so their app records 0 prompt and 0 completion tokens
3. They open https://litellm-domain/ui/?page=logs and see the request logged at $0 spend
After a (hypothetical) fix: the same request comes back with real token counts, so the dashboard shows real spend
1. The proxy admin sets always_include_stream_usage: true and restarts the proxy
2. The developer sends the same POST https://litellm-domain/v1/chat/completions with "stream": true and no stream_options
3. The last SSE chunk now carries a usage object with real prompt and completion token counts
4. https://litellm-domain/ui/?page=logs shows that request at non-zero spend
validations:
required: true
- type: textarea
id: logs
id: proof-of-bug
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
render: shell
label: Proof the bug occurs
description: |
The commands (e.g., curl) and their full output, screenshots, or a screen recording demonstrating that the bug happens. Every rule below applies.
- The proof must be completely e2e with no mocks, against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), hitting real LLM provider APIs, costing real $ if needed, where the bug involves a provider call. `pytest` commands are not enough
- Show exactly what the end user sees or does, matching the User Flow above step for step
- Start with the config.yaml (or SDK setup) and any env vars the proxy ran with, then the exact version or commit hash the proof was captured at, so a maintainer can stand up the same proxy before running your commands. Keep the real values for env vars that aren't sensitive, they are often the reason the bug happens, and redact only the secrets: never paste a real API key, virtual key, database URL, or other credential, here or anywhere else in the issue
- If the bug applies to more than one of the LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), include proof for every one of them, not just one
- For UI bugs: include screenshots and the page URLs you were on. Scrub keys and tokens out of screenshots too (for example, the virtual key is briefly shown in the panel right after you create a virtual key)
placeholder: |
Config / setup the proxy ran with:
Version or commit:
Commands and their full output:
validations:
required: true
- type: dropdown
id: component
attributes:

49
.github/ISSUE_TEMPLATE/feature_request.yml vendored
View file

@ -24,10 +24,53 @@ body:
validations:
required: true
- type: textarea
id: motivation
id: user-flow
attributes:
label: Motivation, pitch
description: Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., "I'm working on X and would like Y to be possible". If this is related to another GitHub issue, please link here too.
label: User Flow
description: |
Two ordered lists, "Before this feature (today)" and "After this feature (ideal user flow)", walking the same end user through the same task, written strictly from that user's seat. Every rule below applies.
- Describe the real application and the routes its users actually hit, not a generic scenario. Link any related GitHub issue or provider API docs
- Lead each list with one plain sentence saying where the flow dead-ends today and what it would let them do instead, then number the steps
- Every step is something the user does or observes: the HTTP method and full URL they hit, what they sent, and what visibly came back (status code, error text, the shape of an ID). UI steps name the page URL and what is on screen
- No LiteLLM internals: never name functions, files, DB tables, config classes, hooks, callbacks, or code paths. Ask for the behavior you need, not the implementation you imagine
- Keep the two lists step-for-step identical until they diverge, so the missing capability is obvious
- "Before this feature" is also where you show the workaround you're living with, which is what tells us how badly this is needed
placeholder: |
Before this feature (today): a developer batching nightly summaries has no way to mark those calls as low priority, so they compete with live traffic for the same rate limit
1. They send POST https://litellm-domain/v1/chat/completions for 500 documents in a loop
2. Around document 120 they start getting 429s naming the rpm limit, and their user-facing chat app starts getting them too
3. Their workaround is a hand-rolled sleep between calls, which stretches the batch to 3 hours and still collides at peak
After this feature (ideal user flow): the same batch runs as background work that yields to live traffic
1. The developer sends the same POST with "service_tier": "flex"
2. Batch calls queue behind interactive ones instead of 429ing, and the response comes back with the tier it was served at
3. The live chat app keeps returning 200s throughout the batch
4. https://litellm-domain/ui/?page=logs shows the batch requests tagged with that tier
validations:
required: true
- type: textarea
id: how-far-you-got
attributes:
label: How far you got
description: |
Run as many steps of the "After this feature (ideal user flow)" list as you can against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), then paste the commands (e.g., curl) and their full output, ending at the step that dead-ends. Every rule below applies.
- Say plainly what stopped you there, in user terms: the option you passed came back ignored, the response 400'd naming an unsupported field, there is no button on the page for it. This is what proves the feature is genuinely missing rather than undocumented
- No mocks. Where the flow involves a provider call, hit the real provider API, even if it costs real $. `pytest` commands are not enough
- Include the config.yaml (or SDK setup) and env vars the proxy ran with, plus the version or commit you were on. Keep the real values for env vars that aren't sensitive, and redact only the secrets: never paste a real API key, virtual key, database URL, or other credential, here or anywhere else in the issue
- If the provider already supports this, link their API docs and paste a direct call to them succeeding, so we can see the shape LiteLLM should be sending
- For UI asks: include screenshots of the page you got stuck on and its URL. Scrub keys and tokens out of screenshots too (for example, the virtual key is briefly shown in the panel right after you create a virtual key)
placeholder: |
Config / setup the proxy ran with:
Version or commit:
Commands and their full output, up to the step that dead-ends:
What stopped me there:
validations:
required: true
- type: dropdown

29
.github/scripts/triage_with_llm.py vendored
View file

@ -582,7 +582,9 @@ def build_issue_prompt(*, title: str, body: str) -> str:
Commands whose external dependencies (LLM provider, DB,
network) are mocked or stubbed do NOT count.
Prose-only "steps to reproduce" with no run output, video, or
screenshot do NOT satisfy (1).
screenshot do NOT satisfy (1). An unfilled template scaffold
(bare headings such as "Version or commit:" with nothing under
them, empty numbered lists) counts as absent, not as evidence.
(2) Expected vs. actual behavior (`has_expected_vs_actual`).
FAIL the bug report if either (1) or (2) is missing. Do not bias
@ -595,6 +597,13 @@ def build_issue_prompt(*, title: str, body: str) -> str:
that it does not today).
- Motivation / use case with a concrete example (config, API call,
UI flow, or scenario showing what's blocked today).
- END-TO-END EVIDENCE OF THE DEAD-END (set
`has_dead_end_evidence=true` only when this is present): a video,
a screenshot, or the exact command(s) actually run paired with
their real output, showing the point where the flow stops today.
Mocked or stubbed dependencies do NOT count, and an unfilled
template scaffold (bare headings, empty numbered lists) counts as
absent.
For an issue that is neither a bug report nor a feature request (a
question, support request, or discussion), PASS as long as it has a
@ -608,6 +617,7 @@ def build_issue_prompt(*, title: str, body: str) -> str:
"has_repro": boolean,
"has_expected_vs_actual": boolean,
"has_motivation_example": boolean,
"has_dead_end_evidence": boolean,
"missing": ["plain-english strings naming what is missing"],
"explanation": "1-2 sentence reasoning for the team to skim"
}}
@ -705,6 +715,10 @@ _ISSUE_BUG_LABELS: tuple[tuple[str, str], ...] = (
)
_ISSUE_FEATURE_LABELS: tuple[tuple[str, str], ...] = (
("has_motivation_example", "Motivation and concrete example"),
(
"has_dead_end_evidence",
"End-to-end evidence of the dead-end (video, screenshot, or command + real output)",
),
)
@ -836,8 +850,11 @@ def format_issue_close_comment(verdict: dict) -> str:
"video, a screenshot, or the exact commands you ran with their real output / "
"traceback) plus expected vs. actual behavior. Written steps with no run output, "
"video, or screenshot don't count, and mocked or stubbed runs don't count.\n"
" - For **feature requests**: a concrete description of what should change, plus a "
"use case and example (config / API call / UI flow).\n"
" - For **feature requests**: a concrete description of what should change, a "
"use case and example (config / API call / UI flow), plus end-to-end evidence of "
"the dead-end (a video, a screenshot, or the exact commands you ran with their "
"real output showing where the flow stops today). Mocked or stubbed runs don't "
"count.\n"
"2. Comment `@agent-shin reconsider`. I'll re-run triage and reopen the issue if it "
"now meets the bar. (GitHub doesn't let external authors reopen an issue a maintainer "
"or bot closed, so the comment-based reconsider is the reliable path.)\n"
@ -943,8 +960,10 @@ def format_grace_warning_issue_comment(verdict: dict) -> str:
"screenshot, or the exact commands you ran with their real output / traceback) plus "
"expected vs. actual behavior. Written steps with no run output don't count, and "
"mocked or stubbed runs don't count.\n"
"- For **feature requests**: a concrete description of what should change, plus a use "
"case and example (config / API call / UI flow).\n"
"- For **feature requests**: a concrete description of what should change, a use "
"case and example (config / API call / UI flow), plus end-to-end evidence of the "
"dead-end (a video, a screenshot, or the exact commands you ran with their real "
"output showing where the flow stops today). Mocked or stubbed runs don't count.\n"
"\n"
"**If the issue does get auto-closed in 2 hours**, comment `@agent-shin reconsider` "
"and I'll re-evaluate. If it now meets the bar, I'll reopen the issue.\n"

11
.github/workflows/test-linting.yml vendored
View file

@ -43,9 +43,10 @@ jobs:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
MERGE_BASE=$(gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
MERGE_BASE=$(retry gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
test -n "$MERGE_BASE"
git fetch --no-tags --depth=1 origin "$MERGE_BASE"
retry git fetch --no-tags --depth=1 origin "$MERGE_BASE"
echo "GATE_BASE_SHA=$MERGE_BASE" >> "$GITHUB_ENV"
- name: Set up Python
@ -161,7 +162,8 @@ jobs:
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
git fetch --no-tags --depth=1 origin "$BASE_SHA"
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
retry git fetch --no-tags --depth=1 origin "$BASE_SHA"
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@ -205,7 +207,8 @@ jobs:
GITGUARDIAN_API_KEY: ${{ secrets.GITGUARDIAN_API_KEY }}
run: |
if [ -n "$GITGUARDIAN_API_KEY" ]; then
git fetch --no-tags --unshallow origin
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
retry git fetch --no-tags --unshallow origin
uv tool run --from 'ggshield==1.48.0' ggshield secret scan repo .
else
echo "GITGUARDIAN_API_KEY not set, skipping ggshield scan"

54
.github/workflows/test-terraform-modules.yml vendored Normal file
View file

@ -0,0 +1,54 @@
name: Terraform Modules
on:
push:
paths:
- "terraform/litellm/aws/**"
- ".github/workflows/test-terraform-modules.yml"
pull_request:
branches:
- main
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
paths:
- "terraform/litellm/aws/**"
- ".github/workflows/test-terraform-modules.yml"
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
aws-module:
name: fmt, validate, test (aws)
runs-on: ubuntu-latest
timeout-minutes: 15
defaults:
run:
working-directory: terraform/litellm/aws
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3.1.2
with:
terraform_version: 1.13.3
terraform_wrapper: false
- name: fmt
run: terraform fmt -recursive -check -diff
- name: init
run: terraform init -backend=false -input=false
- name: validate
run: terraform validate
# Plan-only, mock_provider-backed: no AWS credentials, no API calls.
- name: test
run: terraform test

2
.github/workflows/test-unit-proxy-db.yml vendored
View file

@ -135,8 +135,6 @@ jobs:
test-path: >-
tests/proxy_unit_tests/test_proxy_server.py
tests/proxy_unit_tests/test_proxy_server_keys.py
tests/proxy_unit_tests/test_proxy_server_caching.py
tests/proxy_unit_tests/test_proxy_server_langfuse.py
tests/proxy_unit_tests/test_proxy_server_spend.py
tests/proxy_unit_tests/test_aproxy_startup.py
workers: 4

106
.github/workflows/test-unit-proxy-legacy.yml vendored
View file

@ -1,106 +0,0 @@
name: "Unit Tests: Proxy Legacy Tests"
on:
pull_request:
branches:
- main
- litellm_internal_staging
- litellm_oss_staging
- "litellm_**"
push:
branches:
- main
- litellm_internal_staging
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
test-group:
- name: "auth-and-jwt"
path: "tests/proxy_unit_tests/test_[a-j]*.py"
- name: "key-generation"
path: "tests/proxy_unit_tests/test_[k-o]*.py"
- name: "proxy-config"
path: "tests/proxy_unit_tests/test_prisma*.py tests/proxy_unit_tests/test_prompt*.py tests/proxy_unit_tests/test_proxy_[c-r]*.py"
- name: "proxy-server"
path: "tests/proxy_unit_tests/test_proxy_server.py"
- name: "proxy-server-extras"
path: "tests/proxy_unit_tests/test_proxy_server_*.py tests/proxy_unit_tests/test_proxy_setting_guardrails.py"
- name: "proxy-utils"
path: "tests/proxy_unit_tests/test_proxy_utils.py"
- name: "proxy-token-counter"
path: "tests/proxy_unit_tests/test_proxy_token_counter.py"
- name: "proxy-response-and-misc"
path: "tests/proxy_unit_tests/test_[r-t]*.py"
- name: "proxy-user-auth-and-spend"
path: "tests/proxy_unit_tests/test_[u-z]*.py"
name: ${{ matrix.test-group.name }}
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Detect backend-relevant changes
id: changes
uses: ./.github/actions/detect-backend-changes
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Cache uv dependencies
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
- name: Cache Prisma binaries
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-prisma-binaries
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
- name: Run tests - ${{ matrix.test-group.name }}
if: steps.changes.outputs.decision != 'skip'
env:
TEST_PATH: ${{ matrix.test-group.path }}
run: |
uv run --no-sync pytest ${TEST_PATH} \
--tb=short -vv \
--maxfail=10 \
-n 2 \
--reruns 1 \
--reruns-delay 1 \
--dist=loadscope \
--durations=20

4
CLAUDE.md
View file

@ -29,7 +29,9 @@ End-to-end tests belong in `tests/e2e/` and must follow the harness conventions
When creating PRs, don't set base to `main`. `litellm_internal_staging` is the default base branch and serves that purpose for both internal and external / OSS contributions
When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule
When writing a PR body, treat the comments and imperative instructions inside .github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule
Same applies for filing bug reports and feature requests, with .github/ISSUE_TEMPLATE/bug_report.yml and .github/ISSUE_TEMPLATE/feature_request.yml, respectively
If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank

36
basedpyright-code-budget.json
View file

@ -1,15 +1,15 @@
{
"reportAny": {
"limit": 26391
"limit": 22947
},
"reportArgumentType": {
"limit": 2614
"limit": 2579
},
"reportAssignmentType": {
"limit": 327
"limit": 323
},
"reportAttributeAccessIssue": {
"limit": 514
"limit": 488
},
"reportCallIssue": {
"limit": 114
@ -18,13 +18,13 @@
"limit": 40
},
"reportDeprecated": {
"limit": 215
"limit": 213
},
"reportDuplicateImport": {
"limit": 19
},
"reportExplicitAny": {
"limit": 8319
"limit": 7312
},
"reportFunctionMemberAccess": {
"limit": 7
@ -54,10 +54,10 @@
"limit": 0
},
"reportMissingParameterType": {
"limit": 5825
"limit": 5707
},
"reportMissingTypeArgument": {
"limit": 15695
"limit": 15642
},
"reportMissingTypeStubs": {
"limit": 40
@ -72,7 +72,7 @@
"limit": 0
},
"reportOptionalMemberAccess": {
"limit": 1077
"limit": 1069
},
"reportOptionalOperand": {
"limit": 0
@ -99,37 +99,37 @@
"limit": 0
},
"reportUnknownArgumentType": {
"limit": 44996
"limit": 44776
},
"reportUnknownLambdaType": {
"limit": 113
},
"reportUnknownMemberType": {
"limit": 39643
"limit": 39237
},
"reportUnknownParameterType": {
"limit": 20132
"limit": 19969
},
"reportUnknownVariableType": {
"limit": 31153
"limit": 30881
},
"reportUnnecessaryCast": {
"limit": 118
"limit": 117
},
"reportUnnecessaryComparison": {
"limit": 701
"limit": 699
},
"reportUnnecessaryContains": {
"limit": 5
},
"reportUnnecessaryIsInstance": {
"limit": 857
"limit": 853
},
"reportUntypedBaseClass": {
"limit": 0
},
"reportUntypedFunctionDecorator": {
"limit": 33
"limit": 27
},
"reportUnusedClass": {
"limit": 23
@ -138,7 +138,7 @@
"limit": 139
},
"reportUnusedImport": {
"limit": 555
"limit": 545
},
"reportUnusedVariable": {
"limit": 146

20
enterprise/litellm_enterprise/enterprise_callbacks/send_emails/base_email.py
View file

@ -99,6 +99,7 @@ class BaseEmailLogger(CustomLogger):
email_html_content = USER_INVITATION_EMAIL_TEMPLATE.format(
email_logo_url=email_params.logo_url,
recipient_email=email_params.recipient_email,
invitation_link=email_params.base_url,
base_url=email_params.base_url,
email_support_contact=email_params.support_contact,
email_footer=email_params.signature,
@ -826,10 +827,15 @@ class BaseEmailLogger(CustomLogger):
"""
# Early validation
if not user_id:
verbose_proxy_logger.debug("No user_id provided for invitation link")
verbose_proxy_logger.warning(
"No user_id provided for invitation link. Email will link to base URL instead of onboarding page"
)
return base_url
if not await self._is_prisma_client_available():
verbose_proxy_logger.warning(
"Prisma client not available. Email will link to base URL instead of onboarding page"
)
return base_url
# Wait for any concurrent invitation creation to complete
@ -839,11 +845,15 @@ class BaseEmailLogger(CustomLogger):
invitation = await self._get_or_create_invitation(user_id)
if not invitation:
verbose_proxy_logger.warning(
f"Failed to get/create invitation for user_id: {user_id}"
f"Failed to get/create invitation for user_id: {user_id}. Email will link to base URL instead of onboarding page"
)
return base_url
return self._construct_invitation_link(invitation.id, base_url)
invitation_link = self._construct_invitation_link(invitation.id, base_url)
verbose_proxy_logger.info(
f"Successfully created invitation link for user_id: {user_id}"
)
return invitation_link
async def _is_prisma_client_available(self) -> bool:
"""Check if Prisma client is available"""
@ -921,7 +931,9 @@ class BaseEmailLogger(CustomLogger):
# http://localhost:4000/ui/onboarding?invitation_id=7a096b3a-37c6-440f-9dd1-ba22e8043f6b
"""
return f"{base_url}/ui/onboarding?invitation_id={invitation_id}"
base_url = base_url.rstrip("/")
invitation_link = f"{base_url}/ui/onboarding?invitation_id={invitation_id}"
return invitation_link
async def send_email(
self,

103
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
View file

@ -3,7 +3,7 @@ Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if t
"""
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
from typing import TYPE_CHECKING, Any, Dict, Final, List, Optional, Tuple
from litellm._logging import verbose_proxy_logger
from litellm._uuid import uuid
@ -23,6 +23,15 @@ if TYPE_CHECKING:
CHECK_BATCH_COST_USER_AGENT = "LiteLLM Proxy/CheckBatchCost"
TERMINAL_MANAGED_OBJECT_STATUSES: Final[Tuple[str, ...]] = (
"completed",
"complete",
"failed",
"expired",
"cancelled",
"stale_expired",
)
class CheckBatchCost:
def __init__(
@ -132,11 +141,11 @@ class CheckBatchCost:
in non-terminal states as 'stale_expired'. These will never complete and
should not be polled.
"""
cutoff = datetime.now(timezone.utc) - timedelta(days=MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
result = await self.prisma_client.db.litellm_managedobjecttable.update_many(
cutoff: Final = datetime.now(timezone.utc) - timedelta(days=MANAGED_OBJECT_STALENESS_CUTOFF_DAYS)
result: Final = await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={
"file_purpose": "batch",
"status": {"not_in": ["completed", "complete", "failed", "expired", "cancelled", "stale_expired"]},
"status": {"not_in": list(TERMINAL_MANAGED_OBJECT_STATUSES)},
"created_at": {"lt": cutoff},
},
data={"status": "stale_expired"},
@ -147,6 +156,26 @@ class CheckBatchCost:
f"(older than {MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days) as stale_expired"
)
if not self._has_batch_processed_column:
return
# A row already in a terminal status is never rewritten by the sweep above, so
# without this it keeps a poll-page slot forever and starves newer batches.
retired: Final = await self.prisma_client.db.litellm_managedobjecttable.update_many(
where={
"file_purpose": "batch",
"batch_processed": False,
"status": {"in": ["complete", "completed"]},
"created_at": {"lt": cutoff},
},
data={"batch_processed": True},
)
if retired > 0:
verbose_proxy_logger.warning(
f"CheckBatchCost: gave up on {retired} completed managed objects older than "
f"{MANAGED_OBJECT_STALENESS_CUTOFF_DAYS} days that were never costed"
)
async def _fallback_find_jobs(self) -> list:
"""Query batch jobs without the batch_processed filter (for older schemas)."""
return await self.prisma_client.db.litellm_managedobjecttable.find_many(
@ -167,6 +196,68 @@ class CheckBatchCost:
order={"created_at": "asc"},
)
async def _retire_job(self, job: "LiteLLM_ManagedObjectTable", reason: str) -> None:
"""
Take a row that can never be costed out of the poll page. Leaving it selectable
would burn one of the MAX_OBJECTS_PER_POLL_CYCLE slots on every future cycle, and
once enough such rows accumulate no newer batch is ever reached. Older schemas
without batch_processed can only be excluded through the status filter.
"""
data: Final = (
{"batch_processed": True}
if self._has_batch_processed_column
else {"status": "stale_expired"}
)
try:
await self.prisma_client.db.litellm_managedobjecttable.update(
where={"id": job.id},
data=data,
)
except Exception as db_err:
verbose_proxy_logger.error(
f"CheckBatchCost: failed to retire uncostable job {job.id} ({reason}): {db_err}"
)
return
verbose_proxy_logger.warning(
f"CheckBatchCost: job {job.id} can never be costed ({reason}), "
"so it will no longer be polled"
)
@staticmethod
def _has_unified_id_without_model(job: "LiteLLM_ManagedObjectTable") -> bool:
"""A unified id that decodes but carries no model_id can never be routed."""
from litellm.proxy.openai_files_endpoints.common_utils import (
convert_b64_uid_to_unified_uid,
get_model_id_from_unified_batch_id,
)
decoded: Final = convert_b64_uid_to_unified_uid(job.unified_object_id)
return (
decoded != job.unified_object_id
and get_model_id_from_unified_batch_id(decoded) is None
)
@staticmethod
def _is_batch_gone_at_provider(error: Exception, batch_id: str) -> bool:
"""
A 404 naming the batch means the provider dropped its record of it, so no later
retrieve can ever succeed. A 404 about anything else, a renamed Azure deployment
or a fallback deployment that never saw this batch, is still fixable in config, so
it keeps retrying.
"""
import openai
from litellm.exceptions import NotFoundError
return isinstance(error, (NotFoundError, openai.NotFoundError)) and batch_id in str(error)
def _batch_deployment_exists(self, model_id: str) -> bool:
"""A 404 only proves the batch is gone when it came from the batch's own
deployment. Once that deployment leaves the router, default fallbacks can
silently send the retrieve to a provider that never saw the batch, so its
404 must not retire the row; the staleness sweep bounds it instead."""
return self.llm_router.get_deployment(model_id=model_id) is not None
@staticmethod
def _record_error(
prom_logger: Optional["PrometheusLogger"], error_type: str
@ -645,6 +736,8 @@ class CheckBatchCost:
for job in jobs:
routing = self._resolve_job_routing(job, prom_logger)
if routing is None:
if self._has_unified_id_without_model(job):
await self._retire_job(job, "unified object id has no model id")
continue
model_id, batch_id = routing
@ -667,6 +760,8 @@ class CheckBatchCost:
)
if prom_logger:
prom_logger.record_check_batch_cost_error("provider_retrieval_error")
if self._is_batch_gone_at_provider(e, batch_id) and self._batch_deployment_exists(model_id):
await self._retire_job(job, f"batch {batch_id} no longer exists at the provider")
continue
## RETRIEVE THE BATCH JOB OUTPUT FILE

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.54"
version = "0.1.55"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.54"
version = "0.1.55"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

4
helm/litellm-helm/templates/migrations-job.yaml
View file

@ -105,6 +105,10 @@ spec:
{{- toYaml . | nindent 8 }}
{{- end }}
restartPolicy: OnFailure
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}

24
helm/litellm-helm/tests/migrations-job_tests.yaml
View file

@ -290,3 +290,27 @@ tests:
value:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
- it: should schedule onto the same nodes as the gateway
template: migrations-job.yaml
set:
migrationJob:
enabled: true
nodeSelector:
karpenter.sh/nodepool: litellm-e2e
tolerations:
- key: workload
operator: Equal
value: litellm-e2e
effect: NoSchedule
asserts:
- equal:
path: spec.template.spec.nodeSelector
value:
karpenter.sh/nodepool: litellm-e2e
- equal:
path: spec.template.spec.tolerations
value:
- key: workload
operator: Equal
value: litellm-e2e
effect: NoSchedule

5
litellm-proxy-extras/litellm_proxy_extras/migrations/20260810000000_add_verificationtoken_settings_updated_at/migration.sql Normal file
View file

@ -0,0 +1,5 @@
-- AlterTable
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "settings_updated_at" TIMESTAMP(3);
-- AlterTable
ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN IF NOT EXISTS "settings_updated_at" TIMESTAMP(3);

49
litellm-proxy-extras/litellm_proxy_extras/migrations/20260811172448_add_shadow_eval/migration.sql Normal file
View file

@ -0,0 +1,49 @@
-- CreateTable
CREATE TABLE "LiteLLM_ShadowEvalJob" (
"id" TEXT NOT NULL,
"api_key_id" TEXT NOT NULL,
"router_name" TEXT NOT NULL,
"judge_model" TEXT NOT NULL,
"shadow_percentage" DOUBLE PRECISION NOT NULL,
"max_turns" INTEGER NOT NULL,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"created_by" TEXT,
"ends_at" TIMESTAMP(3) NOT NULL,
"stopped_at" TIMESTAMP(3),
CONSTRAINT "LiteLLM_ShadowEvalJob_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "LiteLLM_ShadowEvalAttempt" (
"id" TEXT NOT NULL,
"job_id" TEXT NOT NULL,
"request_id" TEXT NOT NULL,
"outcome" TEXT NOT NULL,
"tier" TEXT,
"real_model" TEXT,
"shadow_model" TEXT,
"confidence" DOUBLE PRECISION,
"judge_cost" DOUBLE PRECISION NOT NULL DEFAULT 0,
"error" TEXT,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "LiteLLM_ShadowEvalAttempt_pkey" PRIMARY KEY ("id")
);
-- CreateIndex
CREATE INDEX "LiteLLM_ShadowEvalJob_api_key_id_idx" ON "LiteLLM_ShadowEvalJob"("api_key_id");
-- CreateIndex
CREATE INDEX "LiteLLM_ShadowEvalJob_created_at_idx" ON "LiteLLM_ShadowEvalJob"("created_at");
-- CreateIndex
CREATE INDEX "LiteLLM_ShadowEvalAttempt_job_id_idx" ON "LiteLLM_ShadowEvalAttempt"("job_id");
-- One active job per key, enforced by the database rather than a read-then-create in the
-- start endpoint, which races against a concurrent start on another pod. Partial indexes
-- are not expressible in schema.prisma, so this lives here only. Active means not yet
-- stopped; the start endpoint stamps stopped_at on expired jobs before creating.
CREATE UNIQUE INDEX "LiteLLM_ShadowEvalJob_one_active_per_key"
ON "LiteLLM_ShadowEvalJob"("api_key_id") WHERE "stopped_at" IS NULL;

40
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -452,6 +452,7 @@ model LiteLLM_VerificationToken {
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")
updated_by String?
settings_updated_at DateTime? @map("settings_updated_at")
last_active DateTime? // When this key was last used
rotation_count Int? @default(0) // Number of times key has been rotated
auto_rotate Boolean? @default(false) // Whether this key should be auto-rotated
@ -548,6 +549,7 @@ model LiteLLM_DeletedVerificationToken {
created_by String? // Original creator
updated_at DateTime? // Last update timestamp before deletion
updated_by String? // Last user who updated before deletion
settings_updated_at DateTime? // Last configuration change before deletion
last_active DateTime? // When this key was last used before deletion
rotation_count Int? @default(0)
auto_rotate Boolean? @default(false)
@ -1448,6 +1450,44 @@ model LiteLLM_AutoRouterSession {
@@index([last_turn_at], map: "idx_autorouter_session_last_turn")
}
// Shadow eval: pre-adoption evaluation of an auto-router against a key's live traffic.
// A sampled slice of requests is duplicated through the router in a detached task and an
// LLM judge compares real vs shadow responses blind. The job row is immutable config plus
// stopped_at; every count, status, and spend figure is derived from the append-only
// attempt rows, so nothing can disagree across pods or stop races.
model LiteLLM_ShadowEvalJob {
id String @id @default(cuid())
api_key_id String // hashed virtual key whose traffic is shadowed
router_name String
judge_model String
shadow_percentage Float
max_turns Int // sample budget: judge at most this many turns
created_at DateTime @default(now())
created_by String?
ends_at DateTime
stopped_at DateTime?
@@index([api_key_id])
@@index([created_at])
}
// One row per sampled pipeline: a blind verdict (real | shadow | tie) or an error.
model LiteLLM_ShadowEvalAttempt {
id String @id @default(cuid())
job_id String
request_id String // the judged real request
outcome String // real | shadow | tie | error
tier String? // router's tier for the prompt, when classified
real_model String?
shadow_model String?
confidence Float?
judge_cost Float @default(0)
error String?
created_at DateTime @default(now())
@@index([job_id])
}
// ---------------------------------------------------------------------------
// Workflow Run Tracking
//

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.84"
version = "0.4.85"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.84"
version = "0.4.85"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

1
litellm/__init__.py
View file

@ -246,6 +246,7 @@ use_chat_completions_url_for_anthropic_messages: bool = bool(
# config.yaml.
strip_anthropic_total_tokens: bool = False
anthropic_sse_ping_interval_seconds: float = 15.0
sse_keepalive_ping_interval_seconds: float | None = None
route_all_chat_openai_to_responses: bool = (
os.getenv("LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES", "false").lower() == "true"
) # When True, routes all OpenAI /chat/completions requests through the Responses API bridge

23
litellm/a2a_protocol/litellm_completion_bridge/handler.py
View file

@ -10,7 +10,7 @@ A2A Streaming Events (in order):
4. Status update (kind: "status-update") - Final status "completed" with final=true
"""
from collections.abc import AsyncIterator, Mapping
from collections.abc import AsyncIterator, Callable, Coroutine, Mapping
from typing import Any, Final
import litellm
@ -54,7 +54,7 @@ class A2ACompletionBridgeHandler:
agent_extra_headers: Mapping[str, str] | None,
*,
stream: bool,
) -> Mapping[str, Any]:
) -> Mapping[str, object]:
# Extract message from params
message: Final = params.get("message", {})
@ -63,7 +63,7 @@ class A2ACompletionBridgeHandler:
# Get completion params
custom_llm_provider: Final = litellm_params.get("custom_llm_provider")
model: Final = litellm_params.get("model", "agent")
model: Final[str] = litellm_params.get("model", "agent")
# Build full model string if provider specified
# Skip prepending if model already starts with the provider prefix
@ -109,13 +109,16 @@ class A2ACompletionBridgeHandler:
return completion_params
@staticmethod
async def _acompletion(completion_params: Mapping[str, Any]) -> ModelResponse | CustomStreamWrapper:
return await litellm.acompletion(**completion_params)
async def _acompletion(completion_params: Mapping[str, object]) -> ModelResponse | CustomStreamWrapper:
acompletion_fn: Final[Callable[..., Coroutine[object, object, ModelResponse | CustomStreamWrapper]]] = vars(
litellm
)["acompletion"]
return await acompletion_fn(**completion_params)
@staticmethod
async def handle_non_streaming(
request_id: str,
params: dict[str, Any],
params: dict[str, object],
litellm_params: dict[str, Any],
api_base: str | None = None,
agent_extra_headers: dict[str, str] | None = None,
@ -296,8 +299,8 @@ class A2ACompletionBridgeHandler:
# Convenience functions that delegate to the class methods
async def handle_a2a_completion(
request_id: str,
params: dict[str, Any],
litellm_params: dict[str, Any],
params: dict[str, object],
litellm_params: dict[str, object],
api_base: str | None = None,
agent_extra_headers: dict[str, str] | None = None,
) -> dict[str, object]:
@ -313,8 +316,8 @@ async def handle_a2a_completion(
async def handle_a2a_completion_streaming(
request_id: str,
params: dict[str, Any],
litellm_params: dict[str, Any],
params: dict[str, object],
litellm_params: dict[str, object],
api_base: str | None = None,
agent_extra_headers: dict[str, str] | None = None,
) -> AsyncIterator[dict[str, object]]:

66
litellm/a2a_protocol/main.py
View file

@ -12,7 +12,8 @@ Provides standalone functions with @client decorator for LiteLLM logging integra
import asyncio
import datetime
import uuid
from collections.abc import AsyncIterator, Coroutine
from collections.abc import AsyncIterator, Coroutine, Mapping
from types import ModuleType
from typing import TYPE_CHECKING, Any, Final, Optional, cast
import litellm
@ -38,12 +39,15 @@ if TYPE_CHECKING:
SendMessageResponse,
SendStreamingMessageRequest,
SendStreamingMessageResponse,
SendStreamingMessageSuccessResponse,
Task,
)
from a2a.types.a2a_pb2 import SendMessageRequest as CoreSendMessageRequest
from a2a.types.a2a_pb2 import StreamResponse as CoreStreamResponse
# Runtime imports — requires a2a-sdk>=1.1.0
A2A_SDK_AVAILABLE = False
_a2a_conversions: Any = None
_a2a_conversions: ModuleType | None = None
try:
from a2a.client import Client, ClientCallContext, ClientConfig, create_client
@ -128,7 +132,7 @@ _A2A_COST_PARAM_KEYS: Final = ("cost_per_query", "input_cost_per_token", "output
def _set_litellm_params_on_logging_obj(
kwargs: dict[str, Any],
litellm_params: dict[str, Any],
litellm_params: Mapping[str, object],
) -> None:
"""
Merge the agent's pricing params into model_call_details["litellm_params"]
@ -150,7 +154,7 @@ def _set_litellm_params_on_logging_obj(
logging_obj.model_call_details["litellm_params"] = {**existing, **cost_params}
def _get_a2a_model_info(a2a_client: Any, kwargs: dict[str, Any]) -> str:
def _get_a2a_model_info(a2a_client: "A2AClientType", kwargs: dict[str, Any]) -> str:
"""
Extract agent info and set model/custom_llm_provider for cost tracking.
@ -179,7 +183,7 @@ def _get_a2a_model_info(a2a_client: Any, kwargs: dict[str, Any]) -> str:
return agent_name
def _get_a2a_client_agent_card(a2a_client: Any) -> Optional["AgentCard"]:
def _get_a2a_client_agent_card(a2a_client: "A2AClientType") -> Optional["AgentCard"]:
agent_card = cast(Optional["AgentCard"], getattr(a2a_client, "_litellm_agent_card", None))
if agent_card is not None:
return agent_card
@ -191,9 +195,9 @@ def _get_a2a_client_agent_card(a2a_client: Any) -> Optional["AgentCard"]:
async def _send_message_via_completion_bridge(
request: "SendMessageRequest",
custom_llm_provider: str,
custom_llm_provider: object,
api_base: str | None,
litellm_params: dict[str, Any],
litellm_params: dict[str, object],
agent_extra_headers: dict[str, str] | None = None,
) -> LiteLLMSendMessageResponse:
"""
@ -224,6 +228,20 @@ def _get_a2a_call_context(a2a_client: "A2AClientType") -> Optional["A2ACallConte
return getattr(a2a_client, "_litellm_call_context", None)
def _to_core_send_message_request(request: "SendMessageRequest") -> "CoreSendMessageRequest":
from a2a.compat.v0_3 import conversions
return conversions.to_core_send_message_request(request)
def _to_compat_stream_response(
event: "CoreStreamResponse", request_id: str | int
) -> "SendStreamingMessageSuccessResponse":
from a2a.compat.v0_3 import conversions
return conversions.to_compat_stream_response(event, request_id=request_id)
async def _send_message(a2a_client: "A2AClientType", request: "SendMessageRequest") -> "SendMessageResponse":
"""Send a non-streaming message via a2a-sdk 1.x and return JSON-RPC response."""
if _a2a_conversions is None:
@ -231,17 +249,14 @@ async def _send_message(a2a_client: "A2AClientType", request: "SendMessageReques
"The 'a2a' package is required for A2A agent invocation. Install it with: pip install a2a-sdk"
)
pb_request: Final = _a2a_conversions.to_core_send_message_request(request)
pb_request: Final = _to_core_send_message_request(request)
last_event = None
async for event in a2a_client.send_message(pb_request, context=_get_a2a_call_context(a2a_client)):
last_event = event
if last_event is None:
raise RuntimeError("A2A send_message failed: no response received from agent.")
stream_compat: Final = _a2a_conversions.to_compat_stream_response(
last_event,
request_id=request.id,
)
stream_compat: Final = _to_compat_stream_response(last_event, request_id=request.id)
result: Final = stream_compat.result
if not isinstance(result, (Message, Task)):
raise RuntimeError(
@ -306,12 +321,9 @@ async def _stream_messages(
"The 'a2a' package is required for A2A agent invocation. Install it with: pip install a2a-sdk"
)
pb_request: Final = _a2a_conversions.to_core_send_message_request(request)
pb_request: Final[CoreSendMessageRequest] = _a2a_conversions.to_core_send_message_request(request)
async for event in a2a_client.send_message(pb_request, context=_get_a2a_call_context(a2a_client)):
compat_chunk = _a2a_conversions.to_compat_stream_response(
event,
request_id=request.id,
)
compat_chunk = _to_compat_stream_response(event, request_id=request.id)
yield SendStreamingMessageResponse(root=compat_chunk)
@ -368,10 +380,10 @@ async def asend_message(
a2a_client: Optional["A2AClientType"] = None,
request: Optional["SendMessageRequest"] = None,
api_base: str | None = None,
litellm_params: dict[str, Any] | None = None,
litellm_params: dict[str, object] | None = None,
agent_id: str | None = None,
agent_extra_headers: dict[str, str] | None = None,
**kwargs: Any,
**kwargs: object,
) -> LiteLLMSendMessageResponse:
"""
Async: Send a message to an A2A agent.
@ -485,7 +497,7 @@ async def asend_message(
response: Final = LiteLLMSendMessageResponse.from_a2a_response(a2a_response, request_id=str(request.id))
# Calculate token usage from request and response
response_dict: Final = a2a_response.model_dump(mode="json", exclude_none=True)
response_dict: Final[dict[str, object]] = a2a_response.model_dump(mode="json", exclude_none=True)
(
prompt_tokens,
completion_tokens,
@ -516,7 +528,7 @@ def send_message(
a2a_client: "A2AClientType",
request: "SendMessageRequest",
**kwargs: Any,
) -> LiteLLMSendMessageResponse | Coroutine[Any, Any, LiteLLMSendMessageResponse]:
) -> LiteLLMSendMessageResponse | Coroutine[object, object, LiteLLMSendMessageResponse]:
"""
Sync: Send a message to an A2A agent.
@ -545,9 +557,9 @@ def _build_streaming_logging_obj(
request: "SendStreamingMessageRequest",
agent_name: str,
agent_id: str | None,
litellm_params: dict[str, Any] | None,
metadata: dict[str, Any] | None,
proxy_server_request: dict[str, Any] | None,
litellm_params: dict[str, object] | None,
metadata: dict[str, object] | None,
proxy_server_request: dict[str, object] | None,
) -> Logging:
"""Build logging object for streaming A2A requests."""
start_time: Final = datetime.datetime.now()
@ -588,10 +600,10 @@ async def asend_message_streaming(
a2a_client: Optional["A2AClientType"] = None,
request: Optional["SendStreamingMessageRequest"] = None,
api_base: str | None = None,
litellm_params: dict[str, Any] | None = None,
litellm_params: dict[str, object] | None = None,
agent_id: str | None = None,
metadata: dict[str, Any] | None = None,
proxy_server_request: dict[str, Any] | None = None,
metadata: dict[str, object] | None = None,
proxy_server_request: dict[str, object] | None = None,
agent_extra_headers: dict[str, str] | None = None,
**kwargs: object,
) -> AsyncIterator[Any]:

28
litellm/completion_extras/litellm_responses_transformation/handler.py
View file

@ -22,6 +22,7 @@ class ResponsesToCompletionBridgeHandlerInputKwargs(TypedDict):
model_response: "ModelResponse"
logging_obj: "LiteLLMLoggingObj"
custom_llm_provider: str
encoding: object
class ResponsesToCompletionBridgeHandler:
@ -102,35 +103,37 @@ class ResponsesToCompletionBridgeHandler:
from litellm import LiteLLMLoggingObj
from litellm.types.utils import ModelResponse
model: Final = kwargs.get("model")
typed_kwargs: Final[dict[str, object]] = kwargs
model: Final = typed_kwargs.get("model")
if model is None or not isinstance(model, str):
raise ValueError("model is required")
custom_llm_provider: Final = kwargs.get("custom_llm_provider")
custom_llm_provider: Final = typed_kwargs.get("custom_llm_provider")
if custom_llm_provider is None or not isinstance(custom_llm_provider, str):
raise ValueError("custom_llm_provider is required")
messages: Final = kwargs.get("messages")
messages: Final = typed_kwargs.get("messages")
if messages is None or not isinstance(messages, list):
raise ValueError("messages is required")
optional_params: Final = kwargs.get("optional_params")
optional_params: Final = typed_kwargs.get("optional_params")
if optional_params is None or not isinstance(optional_params, dict):
raise ValueError("optional_params is required")
litellm_params: Final = kwargs.get("litellm_params")
litellm_params: Final = typed_kwargs.get("litellm_params")
if litellm_params is None or not isinstance(litellm_params, dict):
raise ValueError("litellm_params is required")
headers: Final = kwargs.get("headers")
headers: Final = typed_kwargs.get("headers")
if headers is None or not isinstance(headers, dict):
raise ValueError("headers is required")
model_response: Final = kwargs.get("model_response")
model_response: Final = typed_kwargs.get("model_response")
if model_response is None or not isinstance(model_response, ModelResponse):
raise ValueError("model_response is required")
logging_obj: Final = kwargs.get("logging_obj")
logging_obj: Final = typed_kwargs.get("logging_obj")
if logging_obj is None or not isinstance(logging_obj, LiteLLMLoggingObj):
raise ValueError("logging_obj is required")
@ -143,6 +146,7 @@ class ResponsesToCompletionBridgeHandler:
model_response=model_response,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
encoding=typed_kwargs.get("encoding"),
)
def completion(
@ -205,7 +209,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)
@ -230,7 +234,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)
@ -303,7 +307,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)
@ -328,7 +332,7 @@ class ResponsesToCompletionBridgeHandler:
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=kwargs.get("encoding"),
encoding=validated_kwargs["encoding"],
api_key=kwargs.get("api_key"),
json_mode=kwargs.get("json_mode"),
)

101
litellm/completion_extras/litellm_responses_transformation/transformation.py
View file

@ -4,8 +4,8 @@ Handler for transforming /chat/completions api requests to litellm.responses req
import json
import os
from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Mapping
from typing import TYPE_CHECKING, Any, Final, Literal, Union, cast
from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, TypedDict, Union, cast
from openai.types.responses.custom_tool_param import CustomToolParam
from openai.types.responses.response_input_param import (
@ -45,6 +45,9 @@ from litellm.types.utils import GenericStreamingChunk, ModelResponseStream
if TYPE_CHECKING:
from openai.types.responses import ResponseInputImageParam
from openai.types.responses.response_text_config_param import (
ResponseTextConfigParam as ResponseText,
)
from pydantic import BaseModel
from litellm import LiteLLMLoggingObj, ModelResponse
@ -57,6 +60,19 @@ if TYPE_CHECKING:
ChatCompletionThinkingBlock,
OpenAIMessageContentListBlock,
)
from litellm.types.utils import Choices
class _ReasoningSummaryText(TypedDict):
type: str
text: str
class _BuiltReasoningItem(TypedDict):
type: Literal["reasoning"]
id: str
encrypted_content: str | None
summary: Sequence[_ReasoningSummaryText]
def _get_reasoning_items(
@ -72,13 +88,13 @@ def _get_reasoning_items(
def _build_reasoning_item(
item_id: str,
encrypted_content: str | None,
summary_raw: Any,
) -> dict[str, Any]:
summary_raw: Iterable[object] | None,
) -> _BuiltReasoningItem:
"""Build a ChatCompletionReasoningItem-shaped dict from raw response data.
Handles both pydantic objects (attribute access) and plain dicts.
"""
summary: Final[list[dict[str, Any]]] = []
summary: Final[list[_ReasoningSummaryText]] = []
for s in summary_raw or []:
if isinstance(s, dict):
summary.append({"type": s.get("type", "summary_text"), "text": s.get("text", "")})
@ -98,7 +114,7 @@ def _build_reasoning_item(
class _ChatToolCallDict(ChatCompletionToolCallChunk, total=False):
provider_specific_fields: Mapping[str, Any]
provider_specific_fields: Mapping[str, object]
def _tool_call_dict_from_output_item(item: Mapping[str, Any], index: int) -> _ChatToolCallDict:
@ -142,10 +158,10 @@ def _flat_responses_tool_choice(choice_type: str, name: str) -> ToolChoiceFuncti
def _reasoning_item_to_response_input(
r_item: ChatCompletionReasoningItem | dict[str, Any],
) -> dict[str, Any]:
r_item: ChatCompletionReasoningItem,
) -> dict[str, object]:
"""Convert a stored ChatCompletionReasoningItem back to a Responses API input item."""
r_input: Final[dict[str, Any]] = {
r_input: Final[dict[str, object]] = {
"type": "reasoning",
"id": r_item.get("id") or f"rs_{id(r_item)}",
# summary is always required by the Responses API, even when empty
@ -181,7 +197,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return _flat_responses_tool_choice(choice_type, nested_name)
return tool_choice
def _handle_raw_dict_response_item(self, item: dict[str, Any], index: int) -> tuple[Any | None, int]:
def _handle_raw_dict_response_item(self, item: dict[str, Any], index: int) -> tuple["Choices | None", int]:
"""
Handle raw dict response items from Responses API (e.g., GPT-5 Codex format).
@ -228,8 +244,8 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def convert_chat_completion_messages_to_responses_api(
self, messages: list["AllMessageValues"]
) -> tuple[list[Any], str | None]:
input_items: Final[list[Any]] = []
) -> tuple[list[object], str | None]:
input_items: Final[list[object]] = []
instructions: str | None = None
custom_tool_call_ids: Final = frozenset(
tool_call["id"]
@ -270,7 +286,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
# Convert tool message to function call output format
# The Responses API expects 'output' to be a list with input_text/input_image types
# Using list format for consistency across text and multimodal content
tool_output: list[dict[str, Any]]
tool_output: list[dict[str, object]]
if content is None:
tool_output = []
elif isinstance(content, str):
@ -308,7 +324,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
function = tool_call.get("function")
custom = tool_call.get("custom")
if function:
input_tool_call: dict[str, Any] = {
input_tool_call: dict[str, object] = {
"type": "function_call",
"call_id": tool_call["id"],
}
@ -376,15 +392,15 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
elif key == "web_search_options":
self._add_web_search_tool(responses_api_request, value)
def _build_sanitized_litellm_params(self, litellm_params: dict) -> dict[str, Any]:
def _build_sanitized_litellm_params(self, litellm_params: dict) -> dict[str, object]:
"""Build sanitized litellm_params with merged metadata."""
responses_optional_param_keys: Final = set(ResponsesAPIOptionalRequestParams.__annotations__.keys())
sanitized: Final[dict[str, Any]] = {
sanitized: Final[dict[str, object]] = {
key: value for key, value in litellm_params.items() if key not in responses_optional_param_keys
}
legacy_metadata: Final = litellm_params.get("metadata")
existing_litellm_metadata: Final = litellm_params.get("litellm_metadata")
merged_litellm_metadata: Final[dict[str, Any]] = {}
merged_litellm_metadata: Final[dict[str, object]] = {}
if isinstance(legacy_metadata, dict):
merged_litellm_metadata.update(legacy_metadata)
if isinstance(existing_litellm_metadata, dict):
@ -424,7 +440,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
litellm_params: dict,
headers: dict,
litellm_logging_obj: "LiteLLMLoggingObj",
client: Any | None = None,
client: object | None = None,
) -> dict:
(
input_items,
@ -498,9 +514,9 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
@staticmethod
def _convert_response_output_to_choices(
output_items: list[Any],
handle_raw_dict_callback: Callable | None = None,
) -> list[Any]:
output_items: Sequence[object],
handle_raw_dict_callback: Callable[..., tuple["Choices | None", int]] | None = None,
) -> list["Choices"]:
"""
Convert Responses API output items to chat completion choices.
@ -529,11 +545,11 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
choices: Final[list[Choices]] = []
index = 0
reasoning_content: str | None = None
pending_reasoning_item: dict[str, Any] | None = None
pending_reasoning_item: _BuiltReasoningItem | None = None
# Collect all tool calls to put them in a single choice
# (Chat Completions API expects all tool calls in one message)
accumulated_tool_calls: Final[list[dict[str, Any]]] = []
accumulated_tool_calls: Final[list[Mapping[str, object]]] = []
tool_call_index = 0
for item in output_items:
@ -640,7 +656,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return choices
@classmethod
def _extract_output_from_completed_event(cls, parsed_chunk: dict[str, Any]) -> list[dict[str, Any]] | None:
def _extract_output_from_completed_event(cls, parsed_chunk: Mapping[str, object]) -> list[dict[str, object]] | None:
response_payload: Final = parsed_chunk.get("response")
if not isinstance(response_payload, dict):
return None
@ -650,12 +666,12 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return cast(list[dict[str, Any]], response_output)
@classmethod
def _recover_output_items_from_raw_sse(cls, raw_sse: str | None) -> list[dict[str, Any]]:
def _recover_output_items_from_raw_sse(cls, raw_sse: str | None) -> list[dict[str, object]]:
if not raw_sse or not isinstance(raw_sse, str):
return []
recovered_output_items: Final[dict[int, dict[str, Any]]] = {}
recovered_text_only_items: Final[dict[int, dict[str, Any]]] = {}
recovered_output_items: Final[dict[int, dict[str, object]]] = {}
recovered_text_only_items: Final[dict[int, dict[str, object]]] = {}
for chunk in raw_sse.splitlines():
parsed_chunk = parse_sse_json_chunk(chunk)
@ -690,7 +706,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
# but text-only items at indices without a matching OUTPUT_ITEM_DONE
# must still be preserved (e.g. multi-output responses where some
# indices only emitted OUTPUT_TEXT_DONE).
merged_items: Final[dict[int, dict[str, Any]]] = {**recovered_text_only_items}
merged_items: Final[dict[int, dict[str, object]]] = {**recovered_text_only_items}
merged_items.update(recovered_output_items)
if merged_items:
@ -699,7 +715,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return []
@classmethod
def _recover_output_items_from_logging(cls, logging_obj: "LiteLLMLoggingObj") -> list[dict[str, Any]]:
def _recover_output_items_from_logging(cls, logging_obj: "LiteLLMLoggingObj") -> list[dict[str, object]]:
model_call_details: Final = getattr(logging_obj, "model_call_details", {}) or {}
original_response: Final = model_call_details.get("original_response")
return cls._recover_output_items_from_raw_sse(original_response)
@ -714,7 +730,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
messages: list["AllMessageValues"],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> "ModelResponse":
@ -788,7 +804,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
) -> BaseModelResponseIterator:
return OpenAiResponsesToChatCompletionStreamIterator(streaming_response, sync_stream, json_mode)
def _convert_content_str_to_input_text(self, content: str, role: str) -> dict[str, Any]:
def _convert_content_str_to_input_text(self, content: str, role: str) -> dict[str, object]:
if role == "user" or role == "system" or role == "tool":
return {"type": "input_text", "text": content}
else:
@ -825,13 +841,13 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def _convert_content_to_responses_format(
self,
content: str
| list[Any]
| list[object]
| Iterable[
Union["OpenAIMessageContentListBlock", "ChatCompletionThinkingBlock", "ChatCompletionRedactedThinkingBlock"]
]
| None,
role: str,
) -> list[dict[str, Any]]:
) -> list[dict[str, object]]:
"""Convert chat completion content to responses API format"""
from litellm.types.llms.openai import ChatCompletionImageObject
@ -973,7 +989,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return optional_params
def _map_reasoning_effort(self, reasoning_effort: str | dict[str, Any]) -> Reasoning | None:
def _map_reasoning_effort(self, reasoning_effort: str | Reasoning) -> Reasoning | None:
# If dict is passed, convert it directly to Reasoning object
if isinstance(reasoning_effort, dict):
return Reasoning(**reasoning_effort)
@ -1006,7 +1022,7 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def _add_web_search_tool(
self,
responses_api_request: ResponsesAPIOptionalRequestParams,
web_search_options: Any,
web_search_options: object,
) -> None:
"""
Add web search tool to responses API request.
@ -1024,14 +1040,14 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
tools = []
responses_api_request["tools"] = tools
web_search_tool: Final[dict[str, Any]] = {"type": "web_search"}
web_search_tool: Final[dict[str, object]] = {"type": "web_search"}
if isinstance(web_search_options, dict):
web_search_tool.update(web_search_options)
# Cast to Any to match the expected union type for tools list items
tools.append(cast(Any, web_search_tool))
def _transform_response_format_to_text_format(self, response_format: dict[str, Any] | Any) -> dict[str, Any] | None:
def _transform_response_format_to_text_format(self, response_format: object) -> "ResponseText | None":
"""
Transform Chat Completion response_format parameter to Responses API text.format parameter.
@ -1130,7 +1146,12 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
def __init__(self, streaming_response, sync_stream: bool, json_mode: bool | None = False):
def __init__(
self,
streaming_response: Union[Iterator[str], AsyncIterator[str], "ModelResponse", "BaseModel"],
sync_stream: bool,
json_mode: bool | None = False,
):
super().__init__(streaming_response, sync_stream, json_mode)
self._chat_completion_id: str | None = None
self._tool_call_index_map: dict[int, int] = {} # mutable-ok: per-stream accumulator state
@ -1387,7 +1408,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
finish_reason: Final = "tool_calls" if has_function_calls else "stop"
# Extract reasoning items with encrypted_content for round-tripping
completed_reasoning_items: list[dict[str, Any]] | None = None
completed_reasoning_items: list[_BuiltReasoningItem] | None = None
for item in output_items:
if not isinstance(item, dict) or item.get("type") != "reasoning":
continue
@ -1439,7 +1460,7 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
]
)
def chunk_parser(self, chunk: dict) -> "ModelResponseStream":
def chunk_parser(self, chunk: dict[str, object]) -> "ModelResponseStream":
"""
Parse a Responses API streaming chunk and convert to OpenAI format.

17
litellm/constants.py
View file

@ -472,6 +472,8 @@ EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE: Final = float(
### ANTHROPIC CONSTANTS ###
ANTHROPIC_TOKEN_COUNTING_BETA_VERSION = os.getenv("ANTHROPIC_TOKEN_COUNTING_BETA_VERSION", "token-counting-2024-11-01")
ANTHROPIC_SKILLS_API_BETA_VERSION: Final = "skills-2025-10-02"
ANTHROPIC_BATCHES_ROUTE: Final = "/v1/messages/batches"
VERTEX_BATCH_PREDICTION_JOBS_ROUTE: Final = "batchPredictionJobs"
ANTHROPIC_WEB_SEARCH_TOOL_MAX_USES: Final = {
"low": 1,
"medium": 5,
@ -1323,6 +1325,7 @@ LITELLM_METADATA_FIELD: Final = "litellm_metadata"
OLD_LITELLM_METADATA_FIELD: Final = "metadata"
RETURN_RAW_MODEL_NAME_METADATA_KEY: Final = "_complexity_router_return_raw_model_name"
SESSION_DEPLOYMENT_AFFINITY_TTL_METADATA_KEY: Final = "_session_deployment_affinity_ttl"
CONSUMED_REQUEST_TAGS_METADATA_KEY: Final = "_consumed_request_tags"
INTERNAL_CALL_ORIGIN_METADATA_KEY: Final = "internal_call_origin"
LITELLM_TRUNCATED_PAYLOAD_FIELD: Final = "litellm_truncated"
LITELLM_TRUNCATION_DB_SAFEGUARD_NOTE: Final = (
@ -1478,12 +1481,19 @@ CLOUDZERO_MAX_FETCHED_DATA_RECORDS: Final = int(os.getenv("CLOUDZERO_MAX_FETCHED
SPEND_LOG_CLEANUP_JOB_NAME: Final = "spend_log_cleanup"
KEY_ROTATION_JOB_NAME: Final = "litellm_key_rotation_job"
EXPIRED_UI_SESSION_KEY_CLEANUP_JOB_NAME: Final = "litellm_expired_ui_session_key_cleanup_job"
WEEKLY_SPEND_REPORT_JOB_ID: Final = "weekly_spend_report_job"
MONTHLY_SPEND_REPORT_JOB_ID: Final = "monthly_spend_report_job"
PROMETHEUS_FALLBACK_STATS_JOB_ID: Final = "prometheus_fallback_stats_job"
SLACK_DAILY_REPORT_LOCK_ID: Final = "slack_daily_report"
SPEND_LOG_RUN_LOOPS: Final = int(os.getenv("SPEND_LOG_RUN_LOOPS", 500))
SPEND_LOG_CLEANUP_BATCH_SIZE: Final = int(os.getenv("SPEND_LOG_CLEANUP_BATCH_SIZE", 1000))
SPEND_LOG_CLEANUP_MAX_CONSECUTIVE_BATCH_FAILURES = int(os.getenv("SPEND_LOG_CLEANUP_MAX_CONSECUTIVE_BATCH_FAILURES", 3))
SPEND_LOG_CLEANUP_BATCH_FAILURE_BACKOFF_SECONDS: Final = float(
os.getenv("SPEND_LOG_CLEANUP_BATCH_FAILURE_BACKOFF_SECONDS", 0.5)
)
SPEND_LOG_CLEANUP_RUN_BUDGET_SECONDS: Final = float(os.getenv("SPEND_LOG_CLEANUP_RUN_BUDGET_SECONDS", "300"))
SPEND_LOG_CLEANUP_BATCH_TIMEOUT_SECONDS: Final = float(os.getenv("SPEND_LOG_CLEANUP_BATCH_TIMEOUT_SECONDS", "30"))
SPEND_LOG_CLEANUP_REMAINING_COUNT_CAP: Final = int(os.getenv("SPEND_LOG_CLEANUP_REMAINING_COUNT_CAP", "100000"))
TOOL_SPEND_TOP_TOOLS: Final = 100
SPEND_LOG_PARTITION_INTERVAL: Final = os.getenv("SPEND_LOG_PARTITION_INTERVAL", "day")
SPEND_LOG_PARTITION_PRECREATE_AHEAD: Final = int(os.getenv("SPEND_LOG_PARTITION_PRECREATE_AHEAD", 7))
@ -1523,6 +1533,10 @@ APSCHEDULER_REPLACE_EXISTING: Final = os.getenv("APSCHEDULER_REPLACE_EXISTING",
"1",
] # always replace existing jobs
# Width of the window scheduled background jobs are spread across, so they do not all fire
# on one instant on every replica. Tunable per deployment via general_settings.
DEFAULT_STAGGER_WINDOW_SECONDS: Final = 300
# The number of tag entries are higher than number of user, team entries. This leads to a higher QPS.
# This will run tag spcific tasks at a later time to smooth QPS
DAILY_TAG_SPEND_BATCH_MULTIPLIER: Final = 2.3
@ -1731,6 +1745,9 @@ PTU_ROLLUP_LOCK_TTL_SECONDS: Final[int] = 900
# Furthest back the catch-up pass looks for unpriced PTU days when a deployment
# declares no ptu_effective_from, bounding the scan for an open-ended window.
PTU_ROLLUP_MAX_BACKFILL_DAYS: Final[int] = 90
# Deployments named in the lapsed-window alert before it is truncated, so a fleet-wide
# expiry cannot produce an alert too large for the channel delivering it.
PTU_LAPSED_ALERT_LIMIT: Final[int] = 10
# Slack allowed when deciding a sentinel row is stale. The row's updated_at and the
# run's cutoff are stamped by different hosts, so clock skew between them must not let
# one run delete a charge another just wrote. A stale row is hours old and a concurrent

46
litellm/experimental_mcp_client/client.py
View file

@ -6,10 +6,11 @@ import asyncio
import base64
import os
from collections.abc import Awaitable, Callable, Generator
from datetime import timedelta
from typing import Any, Final, TypeVar
import httpx
from mcp import ClientSession, ReadResourceResult, Resource, StdioServerParameters
from mcp import ClientSession, McpError, ReadResourceResult, Resource, StdioServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
@ -69,6 +70,29 @@ def _first_non_cancelled_cause(exc: BaseException) -> BaseException | None:
return None
_SDK_READ_TIMEOUT_CODE: Final = int(httpx.codes.REQUEST_TIMEOUT)
"""The code the MCP SDK puts on its own elapsed read timeout, an HTTP status in a field that
otherwise carries JSON-RPC error codes."""
def _as_read_timeout(exc: BaseException) -> TimeoutError | None:
"""The session read timeout elapsing, re-expressed as a ``TimeoutError``, or ``None``.
The SDK reports its own elapsed read timeout as ``McpError`` carrying an HTTP status code in a
field that otherwise holds JSON-RPC error codes, and it relays an upstream's JSON-RPC error
through that same class and field. The numeric code alone therefore cannot separate the two, and
an upstream answering with application code 408 would be reported as a gateway timeout it never
caused. The SDK raises its own from inside an ``except TimeoutError``, so the elapsed timeout is
on the context chain, while a relayed error is built from a received message and has no such
chain; that is the discriminator.
"""
if not isinstance(exc, McpError) or exc.error.code != _SDK_READ_TIMEOUT_CODE:
return None
if not isinstance(exc.__context__, TimeoutError):
return None
return TimeoutError(exc.error.message)
TSessionResult = TypeVar("TSessionResult")
@ -347,7 +371,14 @@ class MCPClient:
session_kwargs["elicitation_callback"] = self._elicitation_callback
if self._logging_callback is not None:
session_kwargs["logging_callback"] = self._logging_callback
session_ctx: Final = ClientSession(read_stream, write_stream, **session_kwargs)
# The SDK drops a response stream that ends without a JSON-RPC reply, so nothing else
# ever fails the request.
session_ctx: Final = ClientSession(
read_stream,
write_stream,
read_timeout_seconds=timedelta(seconds=self.timeout),
**session_kwargs,
)
session: Final = await session_ctx.__aenter__()
try:
init_result: Final = await session.initialize()
@ -390,7 +421,16 @@ class MCPClient:
self._last_initialize_instructions = None
transport_ctx, http_client = self._create_transport_context()
return await self._execute_session_operation(transport_ctx, operation)
except Exception:
except Exception as e:
read_timeout: Final = _as_read_timeout(e)
if read_timeout is not None:
verbose_logger.warning(
"MCP client timed out after %ss waiting for %s to answer; the server accepted the "
"request and ended its response stream without a JSON-RPC reply",
self.timeout,
self.server_url or "stdio",
)
raise read_timeout from e
_log: Final = verbose_logger.debug if quiet_on_error else verbose_logger.warning
_log("MCP client run_with_session failed for %s", self.server_url or "stdio")
raise

61
litellm/google_genai/adapters/transformation.py
View file

@ -1,6 +1,8 @@
import json
from collections.abc import AsyncIterator, Iterator
from typing import Any, Final, cast
from typing import Any, Final, TypedDict, cast
from typing_extensions import ReadOnly
from litellm import verbose_logger
from litellm.litellm_core_utils.json_validation_rule import normalize_tool_schema
@ -28,6 +30,19 @@ from litellm.types.utils import (
)
class _GenAITextPart(TypedDict, total=False):
text: ReadOnly[str]
class _GenAISystemInstruction(TypedDict, total=False):
parts: ReadOnly[list[_GenAITextPart]]
class _GenAIPart(TypedDict, total=False):
text: ReadOnly[str]
functionCall: ReadOnly[dict[str, object]]
class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
"""
Wrapper for streaming Google GenAI generate_content responses.
@ -36,9 +51,9 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
sent_first_chunk: bool = False
# State tracking for accumulating partial tool calls
accumulated_tool_calls: dict[str, dict[str, Any]]
accumulated_tool_calls: dict[str, dict[str, str]]
def __init__(self, completion_stream: Any):
def __init__(self, completion_stream: object):
self.sent_first_chunk = False
self.accumulated_tool_calls = {}
self._returned_response = False
@ -85,7 +100,7 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
# After the stream is exhausted, check for any remaining accumulated tool calls
if self.accumulated_tool_calls:
try:
parts: Final = []
parts: Final[list[_GenAIPart]] = []
for (
tool_call_index,
tool_call_data,
@ -94,7 +109,7 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
# For tool calls with no arguments, accumulated_args will be "", which is not valid JSON.
# We default to an empty JSON object in this case.
parsed_args = json.loads(tool_call_data["arguments"] or "{}")
function_call_part = {
function_call_part: _GenAIPart = {
"functionCall": {
"name": tool_call_data["name"] or "undefined_tool_name",
"args": parsed_args,
@ -110,7 +125,7 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
tool_call_data["arguments"],
)
if parts:
final_chunk: Final = {
final_chunk: Final[dict[str, object]] = {
"candidates": [
{
"content": {"parts": parts, "role": "model"},
@ -273,9 +288,9 @@ class GoogleGenAIAdapter:
def _add_generic_litellm_params_to_request(
self,
completion_request_dict: dict[str, Any],
completion_request_dict: dict[str, object],
litellm_params: GenericLiteLLMParams | None = None,
) -> dict:
) -> dict[str, object]:
"""Add generic litellm params to request. e.g add api_base, api_key, api_version, etc.
Args:
@ -295,7 +310,7 @@ class GoogleGenAIAdapter:
def translate_completion_output_params_streaming(
self,
completion_stream: Any,
completion_stream: object,
) -> AsyncIterator[bytes] | None:
"""Transform streaming completion output to Google GenAI format"""
google_genai_wrapper: Final = GoogleGenAIStreamWrapper(completion_stream=completion_stream)
@ -307,12 +322,12 @@ class GoogleGenAIAdapter:
tools: list[dict[str, Any]],
) -> list[ChatCompletionToolParam]:
"""Transform Google GenAI tools to OpenAI tools format"""
openai_tools: Final[list[dict[str, Any]]] = []
openai_tools: Final[list[dict[str, object]]] = []
for tool in tools:
if "functionDeclarations" in tool:
for func_decl in tool["functionDeclarations"]:
function_chunk: dict[str, Any] = {
function_chunk: dict[str, object] = {
"name": func_decl.get("name", ""),
}
@ -321,7 +336,7 @@ class GoogleGenAIAdapter:
if "parametersJsonSchema" in func_decl:
function_chunk["parameters"] = func_decl["parametersJsonSchema"]
openai_tool = {"type": "function", "function": function_chunk}
openai_tool: dict[str, object] = {"type": "function", "function": function_chunk}
openai_tools.append(openai_tool)
# normalize the tool schemas
@ -345,7 +360,7 @@ class GoogleGenAIAdapter:
def _transform_contents_to_messages(
self,
contents: list[dict[str, Any]],
system_instruction: dict[str, Any] | None = None,
system_instruction: _GenAISystemInstruction | None = None,
) -> list[AllMessageValues]:
"""Transform Google GenAI contents to OpenAI messages format"""
messages: Final[list[AllMessageValues]] = []
@ -461,7 +476,7 @@ class GoogleGenAIAdapter:
def translate_completion_to_generate_content(
self,
response: ModelResponse,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
Transform litellm completion response to Google GenAI generate_content format
@ -490,7 +505,7 @@ class GoogleGenAIAdapter:
parts = [{"text": message_content}] if message_content else []
# Create Google GenAI format response
generate_content_response: Final[dict[str, Any]] = {
generate_content_response: Final[dict[str, object]] = {
"candidates": [
{
"content": {"parts": parts, "role": "model"},
@ -524,7 +539,7 @@ class GoogleGenAIAdapter:
self,
response: ModelResponse | ModelResponseStream,
wrapper: GoogleGenAIStreamWrapper,
) -> dict[str, Any] | None:
) -> dict[str, object] | None:
"""
Transform streaming litellm completion chunk to Google GenAI generate_content format
@ -560,7 +575,7 @@ class GoogleGenAIAdapter:
return None
# Create Google GenAI streaming format response
streaming_chunk: Final[dict[str, Any]] = {
streaming_chunk: Final[dict[str, object]] = {
"candidates": [
{
"content": {"parts": parts, "role": "model"},
@ -597,9 +612,9 @@ class GoogleGenAIAdapter:
def _transform_openai_message_to_google_genai_parts(
self,
message: Any,
) -> list[dict[str, Any]]:
) -> list[_GenAIPart]:
"""Transform OpenAI message to Google GenAI parts format"""
parts: Final[list[dict[str, Any]]] = []
parts: Final[list[_GenAIPart]] = []
# Add text content if present
if hasattr(message, "content") and message.content:
@ -614,7 +629,7 @@ class GoogleGenAIAdapter:
except json.JSONDecodeError:
args = {}
function_call_part = {
function_call_part: _GenAIPart = {
"functionCall": {
"name": tool_call.function.name or "undefined_tool_name",
"args": args,
@ -626,14 +641,14 @@ class GoogleGenAIAdapter:
def _transform_openai_delta_to_google_genai_parts_with_accumulation(
self, delta: Any, wrapper: GoogleGenAIStreamWrapper
) -> list[dict[str, Any]]:
) -> list[_GenAIPart]:
"""Transforms OpenAI delta to Google GenAI parts, accumulating streaming tool calls."""
# 1. Initialize wrapper state if it doesn't exist
if not hasattr(wrapper, "accumulated_tool_calls"):
wrapper.accumulated_tool_calls = {}
parts: Final[list[dict[str, Any]]] = []
parts: Final[list[_GenAIPart]] = []
if hasattr(delta, "content") and delta.content:
parts.append({"text": delta.content})
@ -686,7 +701,7 @@ class GoogleGenAIAdapter:
# The part will be created by a later chunk that brings the name.
if accumulated_name:
# If successful, create the part and clean up
function_call_part = {"functionCall": {"name": accumulated_name, "args": parsed_args}}
function_call_part: _GenAIPart = {"functionCall": {"name": accumulated_name, "args": parsed_args}}
parts.append(function_call_part)
# Remove the completed tool call from the accumulator

7
litellm/images/main.py
View file

@ -315,7 +315,12 @@ def image_generation(
or get_secret_str("AZURE_API_KEY")
)
azure_ad_token: Final = optional_params.pop("azure_ad_token", None) or get_secret_str("AZURE_AD_TOKEN")
azure_ad_token_param: Final = optional_params.pop("azure_ad_token", None)
azure_ad_token: Final = (
azure_ad_token_param
if isinstance(azure_ad_token_param, str) and azure_ad_token_param
else get_secret_str("AZURE_AD_TOKEN")
)
# Create azure_ad_token_provider from tenant_id, client_id, client_secret if not already provided
if azure_ad_token_provider is None:

75
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -9,6 +9,7 @@ from datetime import timedelta
from typing import TYPE_CHECKING, Any, Final, Literal
from openai import APIError
from pydantic import TypeAdapter
import litellm
import litellm.litellm_core_utils
@ -16,7 +17,7 @@ import litellm.litellm_core_utils.litellm_logging
import litellm.types
from litellm._logging import verbose_logger, verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.constants import HOURS_IN_A_DAY
from litellm.constants import HOURS_IN_A_DAY, SLACK_DAILY_REPORT_LOCK_ID
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.integrations.SlackAlerting.budget_alert_types import get_budget_alert_type
from litellm.integrations.SlackAlerting.hanging_request_check import (
@ -33,10 +34,14 @@ from litellm.llms.custom_httpx.http_handler import (
from litellm.proxy._types import (
AlertType,
CallInfo,
InvitationModel,
InvitationNew,
Litellm_EntityType,
UserAPIKeyAuth,
VirtualKeyEvent,
WebhookEvent,
)
from litellm.repositories.table_repositories import InvitationLinkRepository
from litellm.repositories.team_repository import TeamRepository
from litellm.repositories.user_repository import UserRepository
from litellm.types.integrations.slack_alerting import *
@ -46,6 +51,7 @@ from .batching_handler import send_to_webhook, squash_payloads
from .utils import process_slack_alerting_variables
if TYPE_CHECKING:
from litellm.proxy.db.db_transaction_queue.pod_lock_manager import PodLockManager
from litellm.router import Router as _Router
Router = _Router
@ -1081,6 +1087,44 @@ Model Info:
if email_logo_url is not None or email_support_contact is not None:
raise ValueError(f"Trying to Customize Email Alerting\n {CommonProxyErrors.not_premium_user.value}")
async def _construct_user_invitation_link(self, recipient_user_id: str | None, base_url: str) -> str:
from litellm.proxy.management_helpers.user_invitation import (
create_invitation_for_user,
)
from litellm.proxy.proxy_server import prisma_client
if recipient_user_id is None or prisma_client is None:
return base_url
try:
existing_invitations: Final = TypeAdapter(list[InvitationModel]).validate_python(
await InvitationLinkRepository(prisma_client).table.find_many( # pyright: ignore[reportAny] # untyped prisma boundary (any-ok), result validated by TypeAdapter
where={"user_id": recipient_user_id}, # mutable-ok: prisma find_many requires a dict where filter
order={"created_at": "desc"}, # mutable-ok: prisma find_many requires a dict order arg
),
from_attributes=True,
)
invitation: Final = (
existing_invitations[0]
if existing_invitations
else TypeAdapter(InvitationModel).validate_python(
await create_invitation_for_user(
data=InvitationNew(user_id=recipient_user_id),
user_api_key_dict=UserAPIKeyAuth(user_id=recipient_user_id),
),
from_attributes=True,
)
)
except Exception as e: # noqa: BLE001 # best-effort link build; any DB/creation failure falls back to base_url
verbose_proxy_logger.error(
"Error creating invitation link for user_id %s: %s",
recipient_user_id,
str(e),
)
return base_url
return f"{base_url.rstrip('/')}/ui/onboarding?invitation_id={invitation.id}"
async def send_key_created_or_user_invited_email(self, webhook_event: WebhookEvent) -> bool:
try:
from litellm.proxy.utils import send_email
@ -1139,11 +1183,14 @@ Model Info:
team_row: Final = await TeamRepository(prisma_client).table.find_unique(where={"team_id": team_id})
if team_row is not None:
team_name = team_row.team_alias or "-"
invitation_link: Final = await self._construct_user_invitation_link(
recipient_user_id=recipient_user_id, base_url=base_url
)
email_html_content = USER_INVITED_EMAIL_TEMPLATE.format(
email_logo_url=email_logo_url,
recipient_email=recipient_email,
team_name=team_name,
base_url=base_url,
base_url=invitation_link,
email_support_contact=email_support_contact,
)
else:
@ -1530,7 +1577,11 @@ Model Info:
except Exception:
pass
async def _run_scheduler_helper(self, llm_router) -> bool:
async def _run_scheduler_helper(
self,
llm_router,
pod_lock_manager: "PodLockManager | None" = None,
) -> bool:
"""
Returns:
- True -> report sent
@ -1555,6 +1606,16 @@ Model Info:
interval_seconds: Final = self.alerting_args.daily_report_frequency
if current_time - report_sent >= interval_seconds:
if (
pod_lock_manager is not None
and (
await pod_lock_manager.acquire_lock(
cronjob_id=SLACK_DAILY_REPORT_LOCK_ID, ttl=interval_seconds, allow_reentrant=False
)
)
is False
):
return False
# Sneak in the reporting logic here
await self.send_daily_reports(router=llm_router)
# Also, don't forget to update the report_sent time after sending the report!
@ -1566,7 +1627,11 @@ Model Info:
return report_sent_bool
async def _run_scheduled_daily_report(self, llm_router: Any | None = None):
async def _run_scheduled_daily_report(
self,
llm_router: Any | None = None,
pod_lock_manager: "PodLockManager | None" = None,
):
"""
If 'daily_reports' enabled
@ -1579,7 +1644,7 @@ Model Info:
if "daily_reports" in self.alert_types:
while True:
await self._run_scheduler_helper(llm_router=llm_router)
await self._run_scheduler_helper(llm_router=llm_router, pod_lock_manager=pod_lock_manager)
interval = random.randint(
self.alerting_args.report_check_interval - 3,
self.alerting_args.report_check_interval + 3,

30
litellm/integrations/anthropic_cache_control_hook.py
View file

@ -382,19 +382,23 @@ class AnthropicCacheControlHook(CustomPromptManagement):
model: str,
custom_llm_provider: str | None,
tools: list | None = None,
enable_prompt_caching: bool | None = None,
) -> list[CacheControlInjectionPoint]:
"""Default breakpoints when ``litellm.enable_anthropic_prompt_caching`` is on.
Caches the system prompt and the trailing turn, so the stable prefix
(system + tools + history) is reused while the breakpoint advances with
the conversation. Returns [] (stand down) when the flag is off, the
provider does not consume cache_control breakpoints (only anthropic /
bedrock do), the model lacks prompt-caching support, or the request
already carries client-supplied cache_control.
``enable_prompt_caching`` is the per-request override (stamped from key
metadata by the proxy); True turns auto-injection on for this request
even when the global flag is off. Caches the system prompt and the
trailing turn, so the stable prefix (system + tools + history) is
reused while the breakpoint advances with the conversation. Returns []
(stand down) when neither flag is on, the provider does not consume
cache_control breakpoints (only anthropic / bedrock do), the model
lacks prompt-caching support, or the request already carries
client-supplied cache_control.
"""
import litellm
if litellm.enable_anthropic_prompt_caching is not True:
if litellm.enable_anthropic_prompt_caching is not True and enable_prompt_caching is not True:
return []
provider = custom_llm_provider
@ -433,6 +437,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
model: str,
custom_llm_provider: str | None,
tools: list | None = None,
enable_prompt_caching: bool | None = None,
) -> None:
"""For /chat/completions: resolve the injection points the request should carry.
@ -458,6 +463,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
model=model,
custom_llm_provider=custom_llm_provider,
tools=tools,
enable_prompt_caching=enable_prompt_caching,
)
if points:
non_default_params["cache_control_injection_points"] = points
@ -478,12 +484,17 @@ class AnthropicCacheControlHook(CustomPromptManagement):
judgment happens once per request; points a prior pass wrote back
carry the judged stamp and are never re-judged (see
``_should_stand_down``). When none are configured but
``litellm.enable_anthropic_prompt_caching`` is on, synthesize default
breakpoints for the native /v1/messages path. Pops the key from kwargs;
``litellm.enable_anthropic_prompt_caching`` or the per-request
``enable_prompt_caching`` kwarg (stamped from key metadata) is on,
synthesize default breakpoints for the native /v1/messages path. Pops
both keys from kwargs;
if remaining (non-message) points exist they are written back so
downstream transforms can handle them.
"""
typed_messages = cast(list[AllMessageValues], messages) # cast-ok: Anthropic-shaped dicts from v1/messages
enable_prompt_caching: Final = cast( # cast-ok: kwargs is untyped; key stamped as bool by the proxy
bool | None, kwargs.pop("enable_prompt_caching", None)
)
configured: Final = cast( # cast-ok: kwargs is untyped; this key only holds the documented injection-point list
list[CacheControlInjectionPoint] | None, kwargs.pop("cache_control_injection_points", None)
)
@ -497,6 +508,7 @@ class AnthropicCacheControlHook(CustomPromptManagement):
tools=tools,
model=model,
custom_llm_provider=custom_llm_provider,
enable_prompt_caching=enable_prompt_caching,
)
if not injection_points:
return messages, system

3
litellm/integrations/braintrust_mock_client.py
View file

@ -10,6 +10,7 @@ Usage:
import os
import time
from collections.abc import AsyncIterable, Iterable
from typing import Final
from urllib.parse import urlparse
@ -84,7 +85,7 @@ def _mock_http_handler_post(
timeout=None,
stream=False,
files=None,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
logging_obj=None,
):
"""Monkey-patched HTTPHandler.post that intercepts Braintrust calls with endpoint-specific responses."""

2
litellm/integrations/email_templates/templates.py
View file

@ -54,7 +54,7 @@ USER_INVITED_EMAIL_TEMPLATE: Final = """
You were invited to use OpenAI Proxy API for team {team_name} <br /> <br />
<a href="{base_url}" style="display: inline-block; padding: 10px 20px; background-color: #87ceeb; color: #fff; text-decoration: none; border-radius: 20px;">Get Started here</a> <br /> <br />
<a href="{base_url}" style="display: inline-block; padding: 10px 20px; background-color: #87ceeb; color: #fff; text-decoration: none; border-radius: 20px;">Accept Invitation</a> <br /> <br />
If you have any questions, please send an email to {email_support_contact} <br /> <br />

2
litellm/integrations/email_templates/user_invitation_email.py
View file

@ -131,7 +131,7 @@ USER_INVITATION_EMAIL_TEMPLATE: Final = """
</div>
<div class="btn-container">
<a href="{base_url}" class="btn">Accept Invitation</a>
<a href="{invitation_link}" class="btn">Accept Invitation</a>
</div>
<div class="quickstart">

3
litellm/integrations/gcs_bucket/gcs_bucket_mock_client.py
View file

@ -9,6 +9,7 @@ Usage:
"""
import asyncio
from collections.abc import AsyncIterable, Iterable
from typing import Final
from litellm._logging import verbose_logger
@ -113,7 +114,7 @@ async def _mock_async_handler_delete(
headers=None,
timeout=None,
stream=False,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
):
"""Monkey-patched AsyncHTTPHandler.delete that intercepts GCS calls."""
# Only mock GCS API calls

19
litellm/integrations/generic_api/generic_api_callback.py
View file

@ -11,7 +11,7 @@ import json
import os
import re
import traceback
from typing import Any, Final, Literal
from typing import Final, Literal
import httpx
@ -158,7 +158,7 @@ class GenericAPILogger(CustomBatchLogger):
"endpoint not set for GenericAPILogger, GENERIC_LOGGER_ENDPOINT not found in environment variables"
)
self.headers: dict = self._get_headers(headers)
self.headers: dict[str, str] = self._get_headers(headers)
self.endpoint: str = endpoint
self.event_types: list[API_EVENT_TYPES] | None = event_types
self.callback_name: str | None = callback_name
@ -248,18 +248,15 @@ class GenericAPILogger(CustomBatchLogger):
await asyncio.sleep(delay)
async def _post_with_retries(self, data: str) -> httpx.Response:
post_kwargs: Final[dict[str, Any]] = {
"url": self.endpoint,
"headers": self.headers,
"data": data,
}
if self.timeout is not None:
post_kwargs["timeout"] = self.timeout
total_attempts: Final = self.max_retries + 1
for attempt in range(total_attempts):
try:
return await self.async_httpx_client.post(**post_kwargs)
return await self.async_httpx_client.post(
url=self.endpoint,
headers=self.headers,
data=data,
timeout=self.timeout,
)
except Exception as e:
is_last_attempt = attempt == self.max_retries
should_retry = self._should_retry_exception(e)

112
litellm/integrations/langfuse/langfuse.py
View file

@ -2,8 +2,9 @@
# On success, logs events to Langfuse
import os
import traceback
from collections.abc import Callable
from collections.abc import Callable, Iterable, Mapping
from datetime import datetime
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, cast
from packaging.version import Version
@ -30,6 +31,7 @@ from litellm.types.utils import (
ImageResponse,
ModelResponse,
RerankResponse,
StandardLoggingMetadata,
StandardLoggingPayload,
StandardLoggingPromptManagementMetadata,
TextCompletionResponse,
@ -46,6 +48,11 @@ else:
Langfuse = Any
_DENIED_STEERING_KEYS: Final = frozenset({"headers", "endpoint", "caching_groups", "previous_models"})
_NO_METADATA: Final[Mapping[str, Any]] = MappingProxyType({})
_REDACTED_PROXY_HEADERS: Final[frozenset[str]] = frozenset({"authorization", "cookie", "referer"})
def _extract_cache_read_input_tokens(usage_obj) -> int:
"""
Extract cache_read_input_tokens from usage object.
@ -75,6 +82,22 @@ def _extract_cache_read_input_tokens(usage_obj) -> int:
return cache_read_input_tokens
def _as_steering_flag(value: object) -> bool:
"""A string ``str_to_bool`` does not recognise falls back to its truthiness."""
if isinstance(value, str):
parsed: Final = str_to_bool(value)
return bool(value) if parsed is None else parsed
return bool(value)
def _as_steering_key_sequence(value: object) -> tuple[str, ...]:
if isinstance(value, str):
return tuple(key.strip() for key in value.split(",") if key.strip())
if isinstance(value, Iterable):
return tuple(str(key) for key in value)
return ()
def resolve_langfuse_credentials(
langfuse_public_key=None,
langfuse_secret=None,
@ -496,16 +519,14 @@ class LangFuseLogger:
else []
)
if standard_logging_object is None:
end_user_id = None
prompt_management_metadata: StandardLoggingPromptManagementMetadata | None = None
else:
end_user_id = standard_logging_object["metadata"].get("user_api_key_end_user_id", None)
prompt_management_metadata = cast(
StandardLoggingPromptManagementMetadata | None,
standard_logging_object["metadata"].get("prompt_management_metadata", None),
)
allowlisted_metadata: Final[StandardLoggingMetadata | dict[str, Any]] = (
standard_logging_object["metadata"] if standard_logging_object is not None else _NO_METADATA
)
end_user_id: Final = allowlisted_metadata.get("user_api_key_end_user_id", None)
prompt_management_metadata: Final[StandardLoggingPromptManagementMetadata | None] = cast(
StandardLoggingPromptManagementMetadata | None,
allowlisted_metadata.get("prompt_management_metadata", None),
)
# Clean Metadata before logging - never log raw metadata
# the raw metadata can contain circular references which leads to infinite recursion
@ -524,12 +545,7 @@ class LangFuseLogger:
tags.append(f"{key}:{value}")
# clean litellm metadata before logging
if key in [
"headers",
"endpoint",
"caching_groups",
"previous_models",
]:
if key in _DENIED_STEERING_KEYS:
continue
else:
clean_metadata[key] = value
@ -552,10 +568,10 @@ class LangFuseLogger:
# This allows continuing an existing trace while still returning the correct trace_id
if existing_trace_id is not None:
trace_id = existing_trace_id
update_trace_keys: Final = cast(list, clean_metadata.pop("update_trace_keys", []))
update_trace_keys: Final = _as_steering_key_sequence(clean_metadata.pop("update_trace_keys", ()))
debug: Final = clean_metadata.pop("debug_langfuse", None)
mask_input: Final = clean_metadata.pop("mask_input", False)
mask_output: Final = clean_metadata.pop("mask_output", False)
mask_input: Final = _as_steering_flag(clean_metadata.pop("mask_input", False))
mask_output: Final = _as_steering_flag(clean_metadata.pop("mask_output", False))
# Look for masking function in the dedicated location first (set by scrub_sensitive_keys_in_metadata)
# Fall back to metadata for backwards compatibility
masking_function: Final = litellm_params.get("_langfuse_masking_function") or clean_metadata.pop(
@ -614,19 +630,18 @@ class LangFuseLogger:
trace_params["output"] = output if not mask_output else "redacted-by-litellm"
if debug is True or (isinstance(debug, str) and debug.lower() == "true"):
if "metadata" in trace_params:
# log the raw_metadata in the trace
trace_params["metadata"]["metadata_passed_to_litellm"] = metadata
else:
trace_params["metadata"] = {"metadata_passed_to_litellm": metadata}
debug_metadata: Final = {
key: value for key, value in metadata.items() if isinstance(value, (str, int, float, bool))
}
trace_params["metadata"] = {
**(trace_params.get("metadata") or _NO_METADATA),
"metadata_passed_to_litellm": debug_metadata,
}
cost: Final = kwargs.get("response_cost", None)
verbose_logger.debug("trace: %s", cost)
clean_metadata["litellm_response_cost"] = cost
if standard_logging_object is not None:
hidden_params: Final = standard_logging_object.get("hidden_params", {})
clean_metadata["hidden_params"] = filter_exceptions_from_params(hidden_params)
hidden_params: Final = standard_logging_object.get("hidden_params") if standard_logging_object else None
if (
litellm.langfuse_default_tags is not None
@ -638,22 +653,24 @@ class LangFuseLogger:
tags.append(f"proxy_base_url:{proxy_base_url}")
api_base: Final = litellm_params.get("api_base", None)
if api_base:
clean_metadata["api_base"] = api_base
vertex_location: Final = kwargs.get("vertex_location", None)
if vertex_location:
clean_metadata["vertex_location"] = vertex_location
aws_region_name: Final = kwargs.get("aws_region_name", None)
if aws_region_name:
clean_metadata["aws_region_name"] = aws_region_name
candidate_enrichments: Final = (
("litellm_response_cost", cost, True),
("hidden_params", filter_exceptions_from_params(hidden_params), hidden_params is not None),
("api_base", api_base, bool(api_base)),
("vertex_location", vertex_location, bool(vertex_location)),
("aws_region_name", aws_region_name, bool(aws_region_name)),
("cache_hit", kwargs.get("cache_hit") or False, self._supports_tags() and "cache_hit" in kwargs),
)
enrichments: Final[Mapping[str, Any]] = {
key: value for key, value, include in candidate_enrichments if include
}
if self._supports_tags():
if "cache_hit" in kwargs:
if kwargs["cache_hit"] is None:
kwargs["cache_hit"] = False
clean_metadata["cache_hit"] = kwargs["cache_hit"]
if "cache_hit" in kwargs and kwargs["cache_hit"] is None:
kwargs["cache_hit"] = False # rebind-ok: pre-existing normalization other integrations rely on
if existing_trace_id is None:
trace_params.update({"tags": tags})
@ -666,13 +683,13 @@ class LangFuseLogger:
if headers:
for key, value in headers.items():
# these headers can leak our API keys and/or JWT tokens
if key.lower() not in ["authorization", "cookie", "referer"]:
if key.lower() not in _REDACTED_PROXY_HEADERS:
clean_headers[key] = value
trace: Final[StatefulTraceClient] = self.Langfuse.trace(**trace_params)
# Log provider specific information as a span
log_provider_specific_information_as_span(trace, clean_metadata)
log_provider_specific_information_as_span(trace, enrichments)
# Log guardrail information as a span
self._log_guardrail_information_as_span(
@ -745,7 +762,10 @@ class LangFuseLogger:
"output": output if not mask_output else "redacted-by-litellm",
"usage": usage,
"usage_details": usage_details,
"metadata": log_requester_metadata(clean_metadata),
"metadata": {
**log_requester_metadata(redact_user_api_key_info(metadata=allowlisted_metadata)),
**enrichments,
},
"level": level,
"version": clean_metadata.pop("version", None),
}
@ -1042,7 +1062,7 @@ def _add_prompt_to_generation_params(
def log_provider_specific_information_as_span(
trace,
clean_metadata,
clean_metadata: Mapping[str, Any],
):
"""
Logs provider-specific information as spans.
@ -1082,7 +1102,7 @@ def log_provider_specific_information_as_span(
)
def log_requester_metadata(clean_metadata: dict):
def log_requester_metadata(clean_metadata: Mapping[str, Any]):
returned_metadata: Final = {}
requester_metadata: Final = clean_metadata.get("requester_metadata") or {}
for k, v in clean_metadata.items():

10
litellm/integrations/langfuse/langfuse_otel.py
View file

@ -90,7 +90,6 @@ class LangfuseOtelLogger(OpenTelemetry):
"generation_name": LangfuseSpanAttributes.GENERATION_NAME,
"generation_id": LangfuseSpanAttributes.GENERATION_ID,
"parent_observation_id": LangfuseSpanAttributes.PARENT_OBSERVATION_ID,
"version": LangfuseSpanAttributes.GENERATION_VERSION,
"mask_input": LangfuseSpanAttributes.MASK_INPUT,
"mask_output": LangfuseSpanAttributes.MASK_OUTPUT,
"trace_user_id": LangfuseSpanAttributes.TRACE_USER_ID,
@ -99,13 +98,18 @@ class LangfuseOtelLogger(OpenTelemetry):
"trace_name": LangfuseSpanAttributes.TRACE_NAME,
"trace_id": LangfuseSpanAttributes.TRACE_ID,
"trace_metadata": LangfuseSpanAttributes.TRACE_METADATA,
"trace_version": LangfuseSpanAttributes.TRACE_VERSION,
"trace_release": LangfuseSpanAttributes.TRACE_RELEASE,
"trace_release": LangfuseSpanAttributes.RELEASE,
"existing_trace_id": LangfuseSpanAttributes.EXISTING_TRACE_ID,
"update_trace_keys": LangfuseSpanAttributes.UPDATE_TRACE_KEYS,
"debug_langfuse": LangfuseSpanAttributes.DEBUG_LANGFUSE,
}
version: Final = (
metadata.get("trace_version") if metadata.get("trace_version") is not None else metadata.get("version")
)
if version is not None:
safe_set_attribute(span, LangfuseSpanAttributes.VERSION.value, version)
for key, enum_attr in mapping.items():
if key in metadata and metadata[key] is not None:
value = metadata[key]

5
litellm/integrations/mock_client_factory.py
View file

@ -8,6 +8,7 @@ making actual network calls.
import asyncio
import json
from collections.abc import AsyncIterable, Iterable
from dataclasses import dataclass
from datetime import timedelta
from typing import Final, cast
@ -140,7 +141,7 @@ def create_mock_client_factory(config: MockClientConfig):
stream=False,
logging_obj=None,
files=None,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
):
"""Monkey-patched AsyncHTTPHandler.post that intercepts API calls."""
if isinstance(url, str) and _is_mock_url(url):
@ -193,7 +194,7 @@ def create_mock_client_factory(config: MockClientConfig):
timeout=None,
stream=False,
files=None,
content=None,
content: str | bytes | Iterable[bytes] | AsyncIterable[bytes] | None = None,
logging_obj=None,
):
"""Monkey-patched HTTPHandler.post that intercepts API calls."""

64
litellm/integrations/opentelemetry.py
View file

@ -1,7 +1,8 @@
import os
from collections.abc import Mapping
from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final, cast
from typing import TYPE_CHECKING, Any, Final, TypedDict, cast
import litellm
from litellm._logging import verbose_logger
@ -37,9 +38,11 @@ from litellm.types.utils import (
# OpenTelemetry imports moved to individual functions to avoid import errors when not installed
if TYPE_CHECKING:
from opentelemetry.sdk.trace import TracerProvider as _SDKTracerProvider
from opentelemetry.sdk.trace.export import SpanExporter as _SpanExporter
from opentelemetry.trace import Context as _Context
from opentelemetry.trace import Span as _Span
from opentelemetry.trace import SpanKind as _SpanKind
from opentelemetry.trace import Tracer as _Tracer
from litellm.proxy._types import (
@ -61,6 +64,25 @@ else:
ManagementEndpointLoggingPayload = Any
Context = Any
class _StartSpanRequiredKwargs(TypedDict):
name: str
start_time: int
context: "Context | None"
class _StartSpanKwargs(_StartSpanRequiredKwargs, total=False):
kind: "_SpanKind"
class _UsageCompletionTokensView(TypedDict, total=False):
completion_tokens: int
class _ResponseWithUsageView(TypedDict, total=False):
usage: "_UsageCompletionTokensView | None"
LITELLM_TRACER_NAME: Final = os.getenv("OTEL_TRACER_NAME", "litellm")
LITELLM_METER_NAME: Final = os.getenv("LITELLM_METER_NAME", "litellm")
LITELLM_LOGGER_NAME: Final = os.getenv("LITELLM_LOGGER_NAME", "litellm")
@ -297,9 +319,9 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
config: OpenTelemetryConfig | None = None,
callback_name: str | None = None,
# injection points for testing
tracer_provider: Any | None = None,
logger_provider: Any | None = None,
meter_provider: Any | None = None,
tracer_provider: object | None = None,
logger_provider: object | None = None,
meter_provider: object | None = None,
**kwargs,
):
team_metadata_keys_override: Final = kwargs.pop("baggage_team_metadata_keys", None)
@ -325,7 +347,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
self.OTEL_EXPORTER = self.config.exporter
self.OTEL_ENDPOINT = self.config.endpoint
self.OTEL_HEADERS = self.config.headers
self._tracer_provider_cache: dict[str, Any] = {}
self._tracer_provider_cache: dict[str, _SDKTracerProvider] = {}
self._init_tracing(tracer_provider)
_debug_otel: Final = str(os.getenv("DEBUG_OTEL", "False")).lower()
@ -870,7 +892,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
def _emit_guardrail_spans_from_request_data(
self,
request_data: dict,
parent_span: Any | None,
parent_span: "Span | None",
) -> None:
"""Emit ``guardrail`` spans from the request's proxy-internal metadata bucket
(``standard_logging_guardrail_information``).
@ -896,7 +918,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
# kwargs["litellm_params"]["metadata"]["_otel_internal"]. Pass the
# SAME metadata dict the proxy populated so _handle_failure and
# this hook see the same dedupe markers.
kwargs: Final[dict[str, Any]] = {
kwargs: Final[dict[str, object]] = {
"litellm_params": {"metadata": metadata},
"standard_logging_object": {
"guardrail_information": guardrail_information,
@ -1257,13 +1279,13 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
response_obj,
start_time,
end_time,
context,
context: "Context | None",
):
from opentelemetry.trace import Status, StatusCode
otel_tracer: Final[Tracer] = self.get_tracer_to_use_for_request(kwargs)
span_kwargs: Final[dict[str, Any]] = {
span_kwargs: Final[_StartSpanKwargs] = {
"name": self._get_span_name(kwargs),
"start_time": self._to_ns(start_time),
"context": context,
@ -1454,7 +1476,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
) = _resolve_metric_attribute_filter(attributes)
self._metric_attr_filter_resolved = True
def _filter_metric_attributes(self, attrs: dict[str, Any]) -> dict[str, Any]:
def _filter_metric_attributes(self, attrs: dict[str, str]) -> dict[str, str]:
if not self._metric_attr_filter_resolved:
self._ensure_metric_attribute_filter()
if self._metric_attr_include is not None:
@ -1559,7 +1581,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
def _record_time_per_output_token_metric(
self,
kwargs: dict,
response_obj: Any | None,
response_obj: "_ResponseWithUsageView | None",
end_time: datetime,
duration_s: float,
common_attrs: dict,
@ -1775,10 +1797,10 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
@staticmethod
def _resolve_guardrail_context(
span: Any | None,
parent_span: Any | None,
fallback_ctx: Any | None,
) -> Any | None:
span: "Span | None",
parent_span: "Span | None",
fallback_ctx: "Context | None",
) -> "Context | None":
"""
Return a valid OTEL context for guardrail child spans so they are
never orphaned (Issue #5). Priority:
@ -1945,7 +1967,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
if should_create_primary_span:
# Span 1: Request sent to litellm SDK
otel_tracer: Final[Tracer] = self.get_tracer_to_use_for_request(kwargs)
span_kwargs: Final[dict[str, Any]] = {
span_kwargs: Final[_StartSpanKwargs] = {
"name": self._get_span_name(kwargs),
"start_time": self._to_ns(start_time),
"context": _parent_context,
@ -2131,10 +2153,10 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
@staticmethod
def _tool_calls_kv_pair(
tool_calls: list[ChatCompletionMessageToolCall],
) -> dict[str, Any]:
) -> dict[str, object]:
from litellm.proxy._types import SpanAttributes
kv_pairs: Final[dict[str, Any]] = {}
kv_pairs: Final[dict[str, object]] = {}
for idx, tool_call in enumerate(tool_calls):
_function = tool_call.get("function")
if not _function:
@ -2691,8 +2713,8 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
import json
try:
_raw_response = json.loads(_raw_response)
for param, val in _raw_response.items():
_parsed: Final[Mapping[str, object]] = json.loads(_raw_response)
for param, val in _parsed.items():
self.safe_set_attribute(
span=span,
key=f"llm.{custom_llm_provider}.{param}",
@ -2722,7 +2744,7 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger):
return int(dt * 1e9)
return int(dt.timestamp() * 1e9)
def _get_span_name(self, kwargs):
def _get_span_name(self, kwargs) -> str:
litellm_params: Final = kwargs.get("litellm_params", {})
metadata: Final = litellm_params.get("metadata") or {}
generation_name: Final = metadata.get("generation_name")

8
litellm/integrations/otel/presets/langfuse.py
View file

@ -42,9 +42,7 @@ def langfuse_dynamic_headers(params: StandardCallbackDynamicParams) -> dict[str,
public_key: Final = params.get("langfuse_public_key")
secret_key: Final = params.get("langfuse_secret_key")
if public_key and secret_key:
return {
"Authorization": _V1Langfuse._get_langfuse_authorization_header(
public_key=public_key, secret_key=secret_key
)
}
return _V1Langfuse._build_langfuse_otel_headers(
_V1Langfuse._get_langfuse_authorization_header(public_key=public_key, secret_key=secret_key)
)
return {}

85
litellm/integrations/rubrik.py
View file

@ -6,12 +6,13 @@ import random
import time
import uuid
from collections import Counter
from collections.abc import Mapping, Sequence
from collections.abc import Awaitable, Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal, Optional
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypedDict
import httpx
from typing_extensions import Never, ReadOnly
from litellm._logging import verbose_logger
from litellm.integrations.custom_batch_logger import CustomBatchLogger
@ -48,7 +49,20 @@ _WEBHOOK_PATH_PROMPT_MODERATION: Final = "/v1/before_prompt/openai/v1"
_WEBHOOK_PATH_LOGGING_BATCH: Final = "/v1/litellm/batch"
_MAX_QUEUE_SIZE: Final = 10_000
_DROP_WARNING_INTERVAL_SECONDS: Final = 60.0
_EMPTY_MAPPING: Final[Mapping[str, Any]] = MappingProxyType({})
_EMPTY_MAPPING: Final[Mapping[str, Never]] = MappingProxyType({})
class _ServiceToolCall(TypedDict):
id: ReadOnly[str]
class _ServiceMessage(TypedDict, total=False):
content: ReadOnly[str]
tool_calls: ReadOnly[Sequence[_ServiceToolCall]]
class _ServiceChoice(TypedDict, total=False):
message: ReadOnly[_ServiceMessage]
class _MalformedToolBlockingResponseError(Exception):
@ -143,7 +157,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
else {"Content-Type": "application/json"}
)
self._periodic_flush_task: asyncio.Task[Any] | None = self._start_periodic_flush_task()
self._periodic_flush_task: asyncio.Task[None] | None = self._start_periodic_flush_task()
@classmethod
def get_supported_event_hooks(cls) -> list[GuardrailEventHooks]:
@ -191,7 +205,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
params={"timeout": httpx.Timeout(5.0, connect=2.0)},
)
def _start_periodic_flush_task(self) -> asyncio.Task[Any] | None:
def _start_periodic_flush_task(self) -> asyncio.Task[None] | None:
"""Start the periodic flush task only when an event loop is already running."""
try:
loop: Final = asyncio.get_running_loop()
@ -212,7 +226,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
Closing them here would close the shared connection pool for every
other logger instance; let LiteLLM manage their lifecycle instead.
"""
task: Final = getattr(self, "_periodic_flush_task", None)
task: Final[asyncio.Task[None] | None] = getattr(self, "_periodic_flush_task", None)
if task is not None:
task.cancel()
@ -253,7 +267,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
@staticmethod
async def _guarded(
coro: Any,
coro: Awaitable[GenericGuardrailAPIInputs],
inputs: GenericGuardrailAPIInputs,
label: str,
) -> GenericGuardrailAPIInputs:
@ -400,7 +414,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
request_data["_rubrik_logging_obj"] = logging_obj
@staticmethod
def _normalize_tool_calls(tool_calls: Any) -> tuple[ChatCompletionMessageToolCall, ...]:
def _normalize_tool_calls(tool_calls: Sequence[object]) -> tuple[ChatCompletionMessageToolCall, ...]:
"""Convert tool_calls from inputs to ChatCompletionMessageToolCall objects."""
return tuple(RubrikLogger._normalize_tool_call(tc) for tc in tool_calls)
@ -427,7 +441,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
raise TypeError(f"Cannot normalize tool_call of type {type(tc).__name__}: {tc!r}")
@staticmethod
def _join_texts(texts: Any) -> str:
def _join_texts(texts: Sequence[str] | None) -> str:
"""Join response text segments into the single content string the
webhook evaluates. Empty when there is no assistant text."""
if not texts:
@ -439,14 +453,14 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
tool_calls: Sequence[ChatCompletionMessageToolCall],
content: str,
request_id: str | None,
) -> Mapping[str, Any]:
) -> Mapping[str, object]:
"""Build an OpenAI ChatCompletion-format dict (assistant text + tool
calls) for the after_completion webhook.
``content`` is sent so the webhook can moderate the response text;
``None`` when the assistant produced no text (tool-call-only response).
"""
message: Final[dict[str, Any]] = {
message: Final[dict[str, object]] = {
"role": "assistant",
"content": content or None,
}
@ -467,7 +481,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
}
@staticmethod
def _flatten_messages_for_moderation(messages: Any) -> tuple[Mapping[str, Any], ...]:
def _flatten_messages_for_moderation(messages: Sequence[object] | None) -> tuple[Mapping[str, Any], ...]:
"""Collapse each message's content to a plain string for the webhook.
litellm normalizes Anthropic ``/v1/messages`` requests to OpenAI shape,
@ -506,8 +520,8 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
@staticmethod
def _build_prompt_moderation_payload(
inputs: GenericGuardrailAPIInputs,
request_data: Mapping[str, Any],
) -> Mapping[str, Any]:
request_data: Mapping[str, object],
) -> Mapping[str, object]:
"""Build the bare OpenAI request the before_prompt webhook consumes.
Unlike the after_completion envelope, this endpoint takes a raw OpenAI
@ -516,7 +530,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
``/v1/messages`` requests too. Optional fields are sent only when
present so the payload stays clean.
"""
payload: Final[dict[str, Any]] = {
payload: Final[dict[str, object]] = {
"model": inputs.get("model") or request_data.get("model") or "",
"messages": RubrikLogger._flatten_messages_for_moderation(inputs.get("structured_messages")),
}
@ -540,8 +554,8 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
@staticmethod
def _extract_request_data(
call_details: Mapping[str, Any],
request_data: Mapping[str, Any] | None,
) -> Mapping[str, Any]:
request_data: Mapping[str, object] | None,
) -> Mapping[str, object]:
"""Extract original request data from model_call_details for the
response moderation service envelope.
@ -576,7 +590,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
}
@staticmethod
def _sanitize_proxy_server_request(proxy_server_request: Any) -> Any:
def _sanitize_proxy_server_request(proxy_server_request: object) -> object:
"""Allowlist only routing fields (``url``, ``method``) when forwarding
``proxy_server_request`` to an external webhook, dropping inbound
``headers`` (Authorization, Cookie, x-api-key, ...) and the raw
@ -586,17 +600,18 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
return {key: proxy_server_request[key] for key in ("url", "method") if key in proxy_server_request}
@staticmethod
def _resolve_model(request_data: Mapping[str, Any], call_details: Mapping[str, Any]) -> str:
def _resolve_model(request_data: Mapping[str, object], call_details: Mapping[str, str]) -> str:
"""Get the model name for the ModifyResponseException."""
response: Final = request_data.get("response")
if response and hasattr(response, "model"):
return response.model or "unknown"
response_model: Final[str | None] = getattr(response, "model", None)
return response_model or "unknown"
return call_details.get("model", "unknown")
# -- Logging hooks ---------------------------------------------------------
@staticmethod
def _correlation_id(call_details: Mapping[str, Any], request_data: Mapping[str, Any] | None = None) -> str | None:
def _correlation_id(call_details: Mapping[str, str], request_data: Mapping[str, str] | None = None) -> str | None:
"""The id that joins a blocked request's two S3 logs by filename: the
moderation (``_blocking``) log and the failure (response) log.
@ -610,7 +625,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
return call_details.get("litellm_call_id") or (request_data or _EMPTY_MAPPING).get("litellm_call_id")
@classmethod
def _apply_correlation_id(cls, payload: dict[str, Any], source: Mapping[str, Any]) -> None:
def _apply_correlation_id(cls, payload: dict[str, object], source: Mapping[str, str]) -> None:
"""Pin ``payload["id"]`` to ``litellm_call_id`` in place so this log
shares its S3 filename id with the moderation (``_blocking``) and
failure logs for the same request -- for every provider.
@ -630,7 +645,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
payload["id"] = correlated
@staticmethod
def _prepend_system_prompt(payload: dict[str, Any], source: Mapping[str, Any]) -> None:
def _prepend_system_prompt(payload: dict[str, object], source: Mapping[str, object]) -> None:
"""Prepend ``source["system"]`` onto ``payload["messages"]``.
Builds a NEW messages list rather than mutating ``payload["messages"]``
@ -658,7 +673,9 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
exc_info=True,
)
async def _prepare_log_payload(self, kwargs: Mapping[str, Any], event_type: str) -> StandardLoggingPayload | None:
async def _prepare_log_payload(
self, kwargs: Mapping[str, object], event_type: str
) -> StandardLoggingPayload | None:
"""Shared logic for success logging (sampled)."""
if random.random() > self.sampling_rate:
verbose_logger.debug("Skipping Rubrik %s logging (sampling_rate=%s)", event_type, self.sampling_rate)
@ -697,7 +714,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
self._dropped_since_warning = 0
self._last_drop_warning_time = now
async def _enqueue_log_event(self, kwargs: Mapping[str, Any], event_type: str):
async def _enqueue_log_event(self, kwargs: Mapping[str, object], event_type: str):
try:
payload: Final = await self._prepare_log_payload(kwargs, event_type)
if payload is None:
@ -862,7 +879,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
base: Final = call_details.get("standard_logging_object")
if base is not None:
payload: dict = safe_deep_copy(base)
payload: dict[str, object] = safe_deep_copy(base)
else:
verbose_logger.debug(
"Rubrik: standard_logging_object not yet on model_call_details "
@ -908,7 +925,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
cls,
call_details: Mapping[str, Any],
user_api_key_dict: "UserAPIKeyAuth",
) -> dict[str, Any]:
) -> dict[str, object]:
# Convert datetime to a Unix float so json.dumps can serialize it.
# httpx's json= parameter uses stdlib json.dumps with no custom encoder.
_raw_start: Final = call_details.get("start_time")
@ -996,7 +1013,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
# -- Webhook services ------------------------------------------------------
async def _post_json(self, endpoint: str, payload: Mapping[str, Any], service_name: str) -> Mapping[str, Any]:
async def _post_json(self, endpoint: str, payload: Mapping[str, object], service_name: str) -> Mapping[str, Any]:
"""POST ``payload`` to a Rubrik webhook and return its dict response.
Raises:
@ -1010,7 +1027,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
headers=self._headers,
)
http_response.raise_for_status()
result: Final = http_response.json()
result: Final[object] = http_response.json()
if not isinstance(result, dict):
raise TypeError(
f"{service_name} returned non-dict JSON "
@ -1021,8 +1038,8 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
async def _post_to_response_moderation_endpoint(
self,
response_data: Mapping[str, Any],
request_data: Mapping[str, Any],
response_data: Mapping[str, object],
request_data: Mapping[str, object],
) -> Mapping[str, Any]:
"""Post the ``{request, response}`` envelope to the after_completion
webhook and return its (possibly rewritten) response.
@ -1039,7 +1056,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
"Response moderation service",
)
async def _post_to_prompt_moderation_endpoint(self, payload: Mapping[str, Any]) -> Mapping[str, Any]:
async def _post_to_prompt_moderation_endpoint(self, payload: Mapping[str, object]) -> Mapping[str, Any]:
"""Post a bare OpenAI request to the before_prompt webhook.
Returns ``{}`` (passthrough) or a synthetic chat.completion (block).
@ -1054,7 +1071,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
chat.completion whose ``choices[0].message.content`` is the refusal
explanation.
"""
choices: Final = service_response.get("choices")
choices: Final[Sequence[_ServiceChoice] | None] = service_response.get("choices")
if not choices:
return None
message: Final = choices[0].get("message") or _EMPTY_MAPPING
@ -1086,7 +1103,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
Expects service_response in OpenAI chat completion format:
{"choices": [{"message": {"tool_calls": [...], "content": "..."}}]}
"""
choices: Final = service_response.get("choices") or ()
choices: Final[Sequence[_ServiceChoice]] = service_response.get("choices") or ()
if not choices:
raise _MalformedToolBlockingResponseError("Response moderation service returned empty response")

563
litellm/integrations/shadow_eval_logger.py Normal file
View file

@ -0,0 +1,563 @@
"""Shadow Eval Logger: samples a shadowed key's successful chat requests, duplicates each
through the auto-router in a detached task, blind-judges real vs shadow, and appends one
``LiteLLM_ShadowEvalAttempt`` row (verdict or error) as the feature's only hot-path write.
Counts, status, and spend derive from those rows at read time, so nothing can disagree
across pods or stop races; the hook reads active jobs through a short-TTL cache."""
import asyncio
import hashlib
import random
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from datetime import datetime, timezone
from types import MappingProxyType
from typing import TYPE_CHECKING, Final
from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm.caching.in_memory_cache import InMemoryCache
from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.core_helpers import get_litellm_metadata_from_kwargs
from litellm.litellm_core_utils.internal_call_metadata import sanitized_forwardable_call_metadata
from litellm.litellm_core_utils.llm_judge import (
default_router_provider,
extract_text_from_content,
judge_acompletion,
parse_json_verdict,
)
from litellm.litellm_core_utils.redact_messages import should_redact_message_logging
from litellm.types.utils import SHADOW_EVAL_JUDGE_CALL_ORIGIN, SHADOW_EVAL_ROUTER_CALL_ORIGIN
if TYPE_CHECKING:
from litellm.proxy.utils import PrismaClient
from litellm.router import Router
from litellm.types.utils import StandardLoggingPayload
# A job starting, stopping, or hitting its turn budget propagates to sampling within one
# TTL; the turn budget can overshoot by at most one TTL of in-flight samples per pod.
_JOBS_CACHE_TTL_SECONDS: Final = 10
# Concurrent shadow+judge pipelines per pod: a traffic spike turns into skipped samples
# rather than an unbounded task pileup.
_MAX_CONCURRENT_SHADOW_TASKS: Final = 16
# Total character budget for the judge's user prompt, however long the conversation and
# the two responses are, so the prompt can never overflow a judge model's context window.
_MAX_JUDGE_RESPONSE_CHARS: Final = 8_000
_MAX_JUDGE_PROMPT_CHARS: Final = 24_000
# The judge answers with a small JSON object; a tighter budget truncates the JSON
# mid-object and the attempt is lost to an error row.
JUDGE_MAX_OUTPUT_TOKENS: Final = 500
_MAX_ERROR_CHARS: Final = 500
_EMPTY_METADATA: Final[Mapping[str, object]] = MappingProxyType({})
_SAMPLED_CALL_TYPES: Final = frozenset({"completion", "acompletion"})
PAIRWISE_JUDGE_SYSTEM_PROMPT: Final = """You are an impartial quality judge comparing two responses to the same conversation.
The responses are labeled A and B in random order. You do not know which system produced which.
Criteria: correctness, completeness, clarity, conciseness.
Return ONLY valid JSON in this exact format, no other text:
{
"preference": "A" | "B" | "tie",
"confidence": <0.0 to 1.0>,
"reasoning": "<one sentence>"
}"""
class PairwiseVerdict(BaseModel):
"""The judge's blind A/B verdict, validated at the parse boundary."""
preference: str = "tie"
confidence: float = 0.0
def _sample_hits(request_id: str, job_id: str, percentage: float) -> bool:
"""Deterministically decide whether a request falls in the shadowed slice: hash-based
rather than random so retries sample the same way and pods agree without coordination."""
digest: Final = hashlib.sha256(f"{job_id}:{request_id}".encode()).digest()
bucket: Final = int.from_bytes(digest[:8], "big") / float(2**64)
return bucket * 100.0 < percentage
def _judge_call_cost(response: object) -> float:
"""Price a judge call, treating an unmapped judge model as free rather than fatal."""
import litellm
try:
return litellm.completion_cost(completion_response=response) or 0.0
except Exception: # noqa: BLE001 # unmapped judge model: the verdict still counts, cost stays 0
return 0.0
def _unmask_preference(raw_preference: str, real_is_a: bool) -> str:
"""Map the judge's blind A/B/tie verdict back to real/shadow/tie."""
normalized: Final = raw_preference.strip().lower()
if normalized == "a":
return "real" if real_is_a else "shadow"
if normalized == "b":
return "shadow" if real_is_a else "real"
return "tie"
def _judge_user_prompt(conversation: str, response_a: str, response_b: str) -> str:
"""The judge prompt under one total character budget: each response is capped, and
the conversation tail gets whatever budget the responses left over."""
a: Final = response_a[:_MAX_JUDGE_RESPONSE_CHARS]
b: Final = response_b[:_MAX_JUDGE_RESPONSE_CHARS]
conversation_budget: Final = _MAX_JUDGE_PROMPT_CHARS - len(a) - len(b)
return (
f"Conversation:\n{conversation[-conversation_budget:]}\n\n"
f"Response A:\n{a}\n\n"
f"Response B:\n{b}\n\n"
"Which response is better?"
)
async def _key_or_team_is_over_budget(metadata: Mapping[str, object]) -> bool:
"""Whether the shadowed key or its team is over budget, decided by the same owners
the request path uses, so counter keys and thresholds can never drift from auth's.
Advisory and fail-open: real traffic on an over-budget key is already rejected at
auth (so nothing reaches the success hook), and this gate only closes the race
where the key crosses its budget while a request is in flight.
"""
try:
from litellm.exceptions import BudgetExceededError
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.auth_checks import (
_team_max_budget_check,
_virtual_key_max_budget_check,
get_team_object,
)
from litellm.proxy.proxy_server import prisma_client, proxy_logging_obj, user_api_key_cache
except ImportError:
return False
auth: Final = metadata.get("user_api_key_auth")
if not isinstance(auth, UserAPIKeyAuth):
return False
try:
await _virtual_key_max_budget_check(valid_token=auth, proxy_logging_obj=proxy_logging_obj)
if auth.team_id:
team: Final = await get_team_object(
team_id=auth.team_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
check_cache_only=True,
)
await _team_max_budget_check(team_object=team, valid_token=auth, proxy_logging_obj=proxy_logging_obj)
except BudgetExceededError:
return True
except Exception as e: # noqa: BLE001 # advisory gate: a failed read must not block sampling
verbose_logger.debug("shadow_eval: budget read failed: %s", e)
return False
def _request_was_routed_by(request_metadata: Mapping[str, object], router_name: str) -> bool:
"""Duplicating a request the shadowed router already served compares the router to
itself: guaranteed ties, judge spend for zero information."""
decision: Final = request_metadata.get("routing_decision")
if not isinstance(decision, Mapping):
return False
return decision.get("router_model_name") == router_name
@dataclass(frozen=True, slots=True)
class _CallFailure:
"""A shadow or judge call that produced no usable response. cost carries any judge
spend the failed attempt still billed, so job-level judge_spend never undercounts."""
error: str
cost: float = 0.0
@dataclass(frozen=True, slots=True)
class _ShadowResponse:
"""A successful shadow call, with what the attempt row records."""
text: str
model: str
tier: str | None
@dataclass(frozen=True, slots=True)
class _JudgeVerdict:
"""A parsed judge verdict, unmasked back to real/shadow/tie."""
preference: str
confidence: float
cost: float
@dataclass(frozen=True, slots=True)
class ActiveShadowEvalJob:
"""One active job as the sampling path needs it: immutable config plus the attempt
count as of the cache fill (the turn budget's staleness is bounded by the cache TTL)."""
id: str
router_name: str
shadow_percentage: float
judge_model: str
max_turns: int
ends_at: datetime
attempts: int
def _as_utc(value: datetime) -> datetime:
return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value
_jobs_cache: Final = InMemoryCache(max_size_in_memory=4, default_ttl=_JOBS_CACHE_TTL_SECONDS)
_JOBS_CACHE_KEY: Final = "shadow_eval:active_jobs"
class ShadowEvalLogger(CustomLogger):
"""Fires blind pairwise shadow evaluations for keys with an active shadow-eval job."""
def __init__(
self,
router_provider: Callable[[], "Router | None"] | None = None,
prisma_provider: Callable[[], "PrismaClient | None"] | None = None,
jobs_cache: InMemoryCache | None = None,
) -> None:
"""Providers are callables so the proxy's lazily-initialized globals are resolved
at call time, not at logger construction."""
self._router_provider = router_provider or default_router_provider
self._prisma_provider = prisma_provider or _default_prisma_provider
self._jobs_cache = jobs_cache or _jobs_cache
self._inflight_shadow_tasks: int = 0
# Starts per job since the last cache fill, never decremented within a
# generation; the refill absorbs written rows and resets.
self._job_starts: dict[str, int] = {} # mutable-ok: per-generation counter
async def _active_jobs(self) -> Mapping[str, ActiveShadowEvalJob]:
"""Active jobs by api_key_id, cache-first. A DB fault returns empty without
caching, so sampling pauses for that request and the next one retries."""
cached: Final = await self._jobs_cache.async_get_cache(_JOBS_CACHE_KEY)
if cached is not None:
return cached # pyright: ignore[reportReturnType] # cache stores exactly this mapping shape
prisma: Final = self._prisma_provider()
if prisma is None:
return _EMPTY_JOBS
try:
records: Final = await prisma.db.litellm_shadowevaljob.find_many(
where={ # mutable-ok: Prisma filter
"stopped_at": None,
"ends_at": {"gt": datetime.now(timezone.utc)}, # mutable-ok: Prisma filter
},
)
grouped: Final = (
await prisma.db.litellm_shadowevalattempt.group_by(
by=["job_id"],
count=True,
where={"job_id": {"in": [str(record.id) for record in records]}}, # mutable-ok: Prisma filter
)
if records
else ()
)
attempt_counts: Final = {str(row["job_id"]): int(row["_count"]["_all"]) for row in grouped or []}
jobs: Final = {
str(record.api_key_id): ActiveShadowEvalJob(
id=str(record.id),
router_name=str(record.router_name),
shadow_percentage=float(record.shadow_percentage),
judge_model=str(record.judge_model),
max_turns=int(record.max_turns),
ends_at=_as_utc(record.ends_at),
attempts=attempt_counts.get(str(record.id), 0),
)
for record in records or []
}
await self._jobs_cache.async_set_cache(_JOBS_CACHE_KEY, jobs)
self._job_starts = {} # rebind-ok: new generation, counts absorbed into the fill
return jobs
except Exception as e: # noqa: BLE001 # a DB blip must never break request logging
verbose_logger.debug("shadow_eval: active-job read failed: %s", e)
return _EMPTY_JOBS
#### hook ####
async def async_log_success_event(
self,
kwargs: Mapping[str, object],
response_obj: object,
start_time: object,
end_time: object,
) -> None:
try:
payload: Final[StandardLoggingPayload | None] = kwargs.get("standard_logging_object") # pyright: ignore[reportAssignmentType] # untyped callback kwargs
if payload is None:
return
raw_meta: Final = get_litellm_metadata_from_kwargs(dict(kwargs)) # mutable-ok: helper needs dict
request_metadata: Final = raw_meta if isinstance(raw_meta, Mapping) else _EMPTY_METADATA
if request_metadata.get(INTERNAL_CALL_ORIGIN_METADATA_KEY):
return # internal sub-call (our own shadow/judge, a classifier), not user traffic
# redaction rewrites logged content before callbacks run, so this hook
# only ever sees placeholders for a redacted request
if should_redact_message_logging(dict(kwargs)): # mutable-ok: predicate takes a plain dict
return
metadata: Final = payload.get("metadata") or _EMPTY_METADATA
api_key_hash: Final = metadata.get("user_api_key_hash")
if not api_key_hash:
return
job: Final = (await self._active_jobs()).get(str(api_key_hash))
if job is None:
return
if datetime.now(timezone.utc) >= job.ends_at:
return
if job.attempts + self._job_starts.get(job.id, 0) >= job.max_turns:
return
request_id: Final = payload.get("id") or ""
if not request_id:
return
if not _sample_hits(request_id, job.id, job.shadow_percentage):
return
if payload.get("call_type") not in _SAMPLED_CALL_TYPES:
return # only known chat-shaped traffic is comparable; unknown or missing types fail closed
if _request_was_routed_by(request_metadata, job.router_name):
return
if self._inflight_shadow_tasks >= _MAX_CONCURRENT_SHADOW_TASKS:
return
raw_messages: Final = kwargs.get("messages")
self._job_starts[job.id] = self._job_starts.get(job.id, 0) + 1
self._inflight_shadow_tasks += 1
task: Final = asyncio.create_task(
self._run_shadow_eval(
job=job,
request_id=request_id,
messages=tuple(m for m in raw_messages if isinstance(m, Mapping))
if isinstance(raw_messages, Sequence)
else (),
response_obj=response_obj,
real_model=payload.get("model") or "",
model_parameters=MappingProxyType(
dict(payload.get("model_parameters") or {}) # mutable-ok: frozen snapshot
),
parent_metadata=MappingProxyType(dict(request_metadata)), # mutable-ok: frozen snapshot
)
)
task.add_done_callback(self._release_shadow_slot)
except Exception as e: # noqa: BLE001 # logging hooks must never fail the request
verbose_logger.debug("shadow_eval: failed to schedule task: %s", e)
def _release_shadow_slot(self, _task: "asyncio.Task[None]") -> None:
self._inflight_shadow_tasks -= 1
#### the detached pipeline: one attempt row per sampled request, verdict or error ####
async def _run_shadow_eval(
self,
job: ActiveShadowEvalJob,
request_id: str,
messages: Sequence[Mapping[str, object]],
response_obj: object,
real_model: str,
model_parameters: Mapping[str, object],
parent_metadata: Mapping[str, object],
) -> None:
"""Budget gate -> shadow call -> blind judge -> one attempt row. The prisma gate
sits above the dispatch so no provider spend happens without a place to record
the outcome, and the budget read lives here rather than in the success hook."""
prisma: Final = self._prisma_provider()
try:
if prisma is None:
return
real_text: Final = self._extract_response_text(response_obj)
if not real_text or not messages:
return
if await _key_or_team_is_over_budget(parent_metadata):
return
shadow: Final = await self._call_router_shadow(job.router_name, messages, model_parameters, parent_metadata)
if isinstance(shadow, _CallFailure):
await self._record_attempt(prisma, job, request_id, outcome="error", error=shadow.error)
return
verdict: Final = await self._call_judge(
judge_model=job.judge_model,
messages=messages,
real_text=real_text,
shadow_text=shadow.text,
parent_metadata=parent_metadata,
)
if isinstance(verdict, _CallFailure):
await self._record_attempt(
prisma,
job,
request_id,
outcome="error",
error=verdict.error,
shadow=shadow,
judge_cost=verdict.cost,
)
return
await self._record_attempt(
prisma,
job,
request_id,
outcome=verdict.preference,
shadow=shadow,
real_model=real_model,
confidence=verdict.confidence,
judge_cost=verdict.cost,
)
except Exception as e: # noqa: BLE001 # detached task: record what happened, never raise
verbose_logger.debug("shadow_eval: pipeline failed for %s: %s", request_id, e)
await self._record_attempt(prisma, job, request_id, outcome="error", error=f"pipeline error: {e}")
@staticmethod
async def _record_attempt(
prisma: "PrismaClient | None",
job: ActiveShadowEvalJob,
request_id: str,
*,
outcome: str,
shadow: _ShadowResponse | None = None,
real_model: str = "",
confidence: float | None = None,
judge_cost: float = 0.0,
error: str | None = None,
) -> None:
if prisma is None:
return
try:
await prisma.db.litellm_shadowevalattempt.create(
data={ # mutable-ok: Prisma payload
"job_id": job.id,
"request_id": request_id,
"outcome": outcome,
"tier": shadow.tier if shadow else None,
"real_model": real_model or None,
"shadow_model": shadow.model if shadow else None,
"confidence": confidence,
"judge_cost": judge_cost,
"error": error[:_MAX_ERROR_CHARS] if error else None,
}
)
except Exception as e: # noqa: BLE001 # a lost row degrades sample size, nothing can disagree with it
verbose_logger.debug("shadow_eval: attempt write failed for %s: %s", request_id, e)
async def _call_router_shadow(
self,
router_name: str,
messages: Sequence[Mapping[str, object]],
model_parameters: Mapping[str, object],
parent_metadata: Mapping[str, object],
) -> "_ShadowResponse | _CallFailure":
"""Send the prompt through the auto-router being evaluated. The metadata carries
the shadowed key's identity (spend attribution) and receives the router's routing
decision write-back, read back for tier attribution."""
router: Final = self._router_provider()
if router is None:
return _CallFailure("no router configured on this pod")
shadow_metadata: Final[dict[str, object]] = ( # mutable-ok: router writes its routing decision back
sanitized_forwardable_call_metadata(parent_metadata, SHADOW_EVAL_ROUTER_CALL_ORIGIN)
)
shadow_params: Final = { # mutable-ok: splatted as kwargs
k: v for k, v in model_parameters.items() if k not in ("stream", "metadata")
}
try:
response: Final = await router.acompletion(
model=router_name,
messages=messages, # pyright: ignore[reportArgumentType] # snapshot of the SDK's own message dicts
metadata=shadow_metadata,
num_retries=0,
fallbacks=[], # mutable-ok: SDK kwarg; a failed shadow is a recorded error, never a spend multiplier
**shadow_params,
)
except Exception as e: # noqa: BLE001 # provider errors become error rows, not crashes
verbose_logger.debug("shadow_eval: router call failed: %s", e)
return _CallFailure(f"shadow router call failed: {e}")
text: Final = self._extract_response_text(response)
if not text:
return _CallFailure("shadow router returned an empty response")
raw_decision: Final = shadow_metadata.get("routing_decision")
routing_decision: Final = raw_decision if isinstance(raw_decision, Mapping) else _EMPTY_METADATA
raw_tier: Final = routing_decision.get("tier_label") or routing_decision.get("tier")
return _ShadowResponse(
text=text,
model=str(getattr(response, "model", None) or routing_decision.get("routed_model") or ""),
tier=str(raw_tier) if raw_tier is not None else None,
)
async def _call_judge(
self,
judge_model: str,
messages: Sequence[Mapping[str, object]],
real_text: str,
shadow_text: str,
parent_metadata: Mapping[str, object],
) -> "_JudgeVerdict | _CallFailure":
"""Blind pairwise judge with A/B labels randomized to cancel position bias."""
real_is_a: Final = random.random() < 0.5
response_a: Final = real_text if real_is_a else shadow_text
response_b: Final = shadow_text if real_is_a else real_text
conversation: Final = "\n".join(
f"{str(m.get('role', 'user')).upper()}: {extract_text_from_content(m.get('content'))}"
for m in messages
if m.get("content") is not None
)
judge_metadata: Final = sanitized_forwardable_call_metadata(parent_metadata, SHADOW_EVAL_JUDGE_CALL_ORIGIN)
judge_messages: Final = [ # mutable-ok: SDK takes a list
{"role": "system", "content": PAIRWISE_JUDGE_SYSTEM_PROMPT}, # mutable-ok: SDK message
{
"role": "user",
"content": _judge_user_prompt(conversation, response_a, response_b),
}, # mutable-ok: SDK message
]
try:
response: Final = await judge_acompletion(
self._router_provider(),
judge_model,
judge_messages, # pyright: ignore[reportArgumentType] # plain SDK message dicts
temperature=0,
max_tokens=JUDGE_MAX_OUTPUT_TOKENS,
metadata=judge_metadata,
)
except Exception as e: # noqa: BLE001 # judge outages become error rows, not crashes
verbose_logger.debug("shadow_eval: judge call failed: %s", e)
return _CallFailure(f"judge call failed: {e}")
try:
raw: Final = response["choices"][0]["message"]["content"] or ""
verdict: Final = PairwiseVerdict.model_validate(parse_json_verdict(raw))
except Exception as e: # noqa: BLE001 # malformed verdicts become error rows
verbose_logger.debug("shadow_eval: unparseable judge verdict: %s", e)
return _CallFailure(f"unparseable judge verdict: {e}", cost=_judge_call_cost(response))
return _JudgeVerdict(
preference=_unmask_preference(verdict.preference, real_is_a),
confidence=max(0.0, min(1.0, verdict.confidence)),
cost=_judge_call_cost(response),
)
@staticmethod
def _extract_response_text(response_obj: object) -> str:
"""Extract the assistant's text from a ModelResponse-shaped object or dict."""
try:
content: Final = (
response_obj["choices"][0]["message"]["content"]
if isinstance(response_obj, Mapping)
else response_obj.choices[0].message.content # pyright: ignore[reportAttributeAccessIssue] # duck-typed ModelResponse
)
except (AttributeError, KeyError, IndexError, TypeError):
return ""
return extract_text_from_content(content)
_EMPTY_JOBS: Final[Mapping[str, ActiveShadowEvalJob]] = MappingProxyType({})
def _default_prisma_provider() -> "PrismaClient | None":
try:
from litellm.proxy.proxy_server import prisma_client
except ImportError:
return None
return prisma_client

13
litellm/integrations/websearch_interception/handler.py
View file

@ -85,6 +85,11 @@ class _WebSearchSettingsView(TypedDict):
websearch_interception_params: WebSearchInterceptionConfig
class _SearchToolConfig(TypedDict, total=False):
search_tool_name: str
litellm_params: Mapping[str, object] | None
class WebSearchInterceptionLogger(CustomLogger):
"""
CustomLogger that intercepts WebSearch tool calls for models that don't
@ -1487,7 +1492,7 @@ class WebSearchInterceptionLogger(CustomLogger):
return None
def _select_search_tool_from_router(self, llm_router: object) -> dict[str, Any] | None:
def _select_search_tool_from_router(self, llm_router: object) -> "_SearchToolConfig | None":
if llm_router is None or not hasattr(llm_router, "search_tools"):
return None
search_tools: Final = list(getattr(llm_router, "search_tools") or [])
@ -1495,9 +1500,9 @@ class WebSearchInterceptionLogger(CustomLogger):
def _select_search_tool_from_list(
self,
search_tools: list[dict[str, Any]],
search_tools: list[_SearchToolConfig],
source: str,
) -> dict[str, Any] | None:
) -> "_SearchToolConfig | None":
if self.search_tool_name:
matching_tools = [tool for tool in search_tools if tool.get("search_tool_name") == self.search_tool_name]
if matching_tools:
@ -1692,7 +1697,7 @@ class WebSearchInterceptionLogger(CustomLogger):
@staticmethod
def initialize_from_proxy_config(
litellm_settings: dict[str, Any],
litellm_settings: Mapping[str, WebSearchInterceptionConfig],
callback_specific_params: Mapping[str, object],
) -> "WebSearchInterceptionLogger":
"""

38
litellm/interactions/litellm_responses_transformation/handler.py
View file

@ -2,8 +2,8 @@
Handler for transforming interactions API requests to litellm.responses requests.
"""
from collections.abc import AsyncIterator, Coroutine, Iterator
from typing import Any, Final, cast
from collections.abc import AsyncIterator, Callable, Coroutine, Iterator
from typing import Any, Final
import litellm
from litellm.interactions.litellm_responses_transformation.streaming_iterator import (
@ -37,7 +37,7 @@ class LiteLLMResponsesInteractionsHandler:
) -> (
InteractionsAPIResponse
| Iterator[InteractionsAPIStreamingResponse]
| Coroutine[Any, Any, InteractionsAPIResponse | AsyncIterator[InteractionsAPIStreamingResponse]]
| Coroutine[object, object, InteractionsAPIResponse | AsyncIterator[InteractionsAPIStreamingResponse]]
):
"""
Handle Interactions API request by calling litellm.responses().
@ -55,13 +55,15 @@ class LiteLLMResponsesInteractionsHandler:
InteractionsAPIResponse or streaming iterator
"""
# Transform interactions request to responses request
responses_request = LiteLLMResponsesInteractionsConfig.transform_interactions_request_to_responses_request(
model=model,
input=input,
optional_params=optional_params,
custom_llm_provider=custom_llm_provider,
stream=stream,
**kwargs,
responses_request: Final = (
LiteLLMResponsesInteractionsConfig.transform_interactions_request_to_responses_request(
model=model,
input=input,
optional_params=optional_params,
custom_llm_provider=custom_llm_provider,
stream=stream,
**kwargs,
)
)
if _is_async:
@ -76,7 +78,10 @@ class LiteLLMResponsesInteractionsHandler:
# Call litellm.responses()
# Note: litellm.responses() returns Union[ResponsesAPIResponse, BaseResponsesAPIStreamingIterator]
# but the type checker may see it as a coroutine in some contexts
responses_response: Final = litellm.responses(
responses_fn: Final[Callable[..., ResponsesAPIResponse | BaseResponsesAPIStreamingIterator]] = vars(litellm)[
"responses"
]
responses_response: Final = responses_fn(
**responses_request,
)
@ -92,8 +97,7 @@ class LiteLLMResponsesInteractionsHandler:
)
# At this point, responses_response must be ResponsesAPIResponse (not streaming)
# Cast to satisfy type checker since we've already checked it's not a streaming iterator
responses_api_response: Final = cast(ResponsesAPIResponse, responses_response)
responses_api_response: Final = responses_response
# Transform responses response to interactions response
return LiteLLMResponsesInteractionsConfig.transform_responses_response_to_interactions_response(
@ -112,7 +116,10 @@ class LiteLLMResponsesInteractionsHandler:
"""Async handler for interactions API requests."""
# Call litellm.aresponses()
# Note: litellm.aresponses() returns Union[ResponsesAPIResponse, BaseResponsesAPIStreamingIterator]
responses_response: Final = await litellm.aresponses(
aresponses_fn: Final[
Callable[..., Coroutine[object, object, ResponsesAPIResponse | BaseResponsesAPIStreamingIterator]]
] = vars(litellm)["aresponses"]
responses_response: Final = await aresponses_fn(
**responses_request,
)
@ -128,8 +135,7 @@ class LiteLLMResponsesInteractionsHandler:
)
# At this point, responses_response must be ResponsesAPIResponse (not streaming)
# Cast to satisfy type checker since we've already checked it's not a streaming iterator
responses_api_response: Final = cast(ResponsesAPIResponse, responses_response)
responses_api_response: Final = responses_response
# Transform responses response to interactions response
return LiteLLMResponsesInteractionsConfig.transform_responses_response_to_interactions_response(

154
litellm/interactions/litellm_responses_transformation/transformation.py
View file

@ -2,12 +2,16 @@
Transformation utilities for bridging Interactions API to Responses API.
This module handles transforming between:
- Interactions API format (Google's format with Turn[], system_instruction, etc.)
- Interactions API format (Google's format with Step[]/Turn[], system_instruction, etc.)
- Responses API format (OpenAI's format with input[], instructions, etc.)
"""
from collections.abc import Mapping, Sequence
from types import MappingProxyType
from typing import Any, Final, cast
from pydantic import BaseModel
from litellm.types.interactions import (
InteractionInput,
InteractionsAPIOptionalRequestParams,
@ -19,6 +23,8 @@ from litellm.types.llms.openai import (
ResponsesAPIResponse,
)
_STEP_TYPE_ROLES: Final = MappingProxyType({"user_input": "user", "model_output": "assistant"})
class LiteLLMResponsesInteractionsConfig:
"""Configuration class for transforming between Interactions API and Responses API."""
@ -91,112 +97,94 @@ class LiteLLMResponsesInteractionsConfig:
Interactions API input can be:
- string: "Hello"
- Turn[]: [{"role": "user", "content": [...]}]
- Content object
- Step[]: [{"type": "user_input", "content": [...]}, {"type": "model_output", "content": [...]}]
- Turn[] (legacy): [{"role": "user", "content": [...]}]
- Content | Content[]: one user message worth of content parts
Responses API input is:
- string: "Hello"
- Message[]: [{"role": "user", "content": [...]}]
- Message[]: [{"role": "user", "content": [{"type": "input_text", ...}]}]
"""
if isinstance(input, str):
# ResponseInputParam accepts str
return cast(ResponseInputParam, input)
if isinstance(input, list):
# Turn[] format - convert to Responses API Message[] format
messages: Final = []
for turn in input:
if isinstance(turn, dict):
role = turn.get("role", "user")
content = turn.get("content", [])
transformed: Final = (
[
LiteLLMResponsesInteractionsConfig._transform_history_item(item)
for item in input
if LiteLLMResponsesInteractionsConfig._is_history_item(item)
]
if any(LiteLLMResponsesInteractionsConfig._is_history_item(item) for item in input)
else [
{
"role": "user",
"content": LiteLLMResponsesInteractionsConfig._transform_content_array(input, "user"),
}
]
)
return cast(ResponseInputParam, transformed)
# Transform content array
transformed_content = LiteLLMResponsesInteractionsConfig._transform_content_array(content)
messages.append(
{
"role": role,
"content": transformed_content,
}
)
elif isinstance(turn, Turn):
# Pydantic model
role = turn.role if hasattr(turn, "role") else "user"
content = turn.content if hasattr(turn, "content") else []
# Ensure content is a list for _transform_content_array
# Cast to List[Any] to handle various content types
if isinstance(content, list):
content_list: list[Any] = list(content)
elif content is not None:
content_list = [content]
else:
content_list = []
transformed_content = LiteLLMResponsesInteractionsConfig._transform_content_array(content_list)
messages.append(
{
"role": role,
"content": transformed_content,
}
)
return cast(ResponseInputParam, messages)
# Single content object - wrap in message
if isinstance(input, dict):
raw_content: Final = input.get("content")
content_items: Final = raw_content if isinstance(raw_content, list) else [input]
return cast(
ResponseInputParam,
[
{
"role": "user",
"content": LiteLLMResponsesInteractionsConfig._transform_content_array(
input.get("content", []) if isinstance(input.get("content"), list) else [input]
),
"content": LiteLLMResponsesInteractionsConfig._transform_content_array(content_items, "user"),
}
],
)
# Fallback: convert to string
return cast(ResponseInputParam, str(input))
@staticmethod
def _transform_content_array(content: list[Any]) -> list[dict[str, Any]]:
"""Transform Interactions API content array to Responses API format."""
if not isinstance(content, list):
# Single content item - wrap in array
content = [content]
def _is_history_item(item: object) -> bool:
if isinstance(item, Turn):
return True
return isinstance(item, dict) and ("role" in item or item.get("type") in _STEP_TYPE_ROLES)
transformed: Final[list[dict[str, Any]]] = []
for item in content:
if isinstance(item, dict):
# Already in dict format, pass through
transformed.append(item)
elif isinstance(item, str):
# Plain string - wrap in text format
transformed.append({"type": "text", "text": item})
else:
# Pydantic model or other - convert to dict
if hasattr(item, "model_dump"):
dumped = item.model_dump()
if isinstance(dumped, dict):
transformed.append(dumped)
else:
# Fallback: wrap in text format
transformed.append({"type": "text", "text": str(dumped)})
elif hasattr(item, "dict"):
dumped = item.dict()
if isinstance(dumped, dict):
transformed.append(dumped)
else:
# Fallback: wrap in text format
transformed.append({"type": "text", "text": str(dumped)})
else:
# Fallback: wrap in text format
transformed.append({"type": "text", "text": str(item)})
@staticmethod
def _transform_history_item(item: object) -> Mapping[str, object]:
raw: Final = item.model_dump(exclude_none=True) if isinstance(item, Turn) else item
fields: Final = raw if isinstance(raw, Mapping) else {}
role: Final = LiteLLMResponsesInteractionsConfig._responses_role(fields)
raw_content: Final = fields.get("content")
content_items: Final = (
raw_content if isinstance(raw_content, list) else [] if raw_content is None else [raw_content]
)
return {
"role": role,
"content": LiteLLMResponsesInteractionsConfig._transform_content_array(content_items, role),
}
return transformed
@staticmethod
def _responses_role(item: Mapping[str, object]) -> str:
step_role: Final = _STEP_TYPE_ROLES.get(str(item.get("type", "")))
if step_role is not None:
return step_role
raw_role: Final = str(item.get("role") or "user")
return "assistant" if raw_role == "model" else raw_role
@staticmethod
def _transform_content_array(content: Sequence[object], role: str) -> Sequence[Mapping[str, object]]:
"""Transform Interactions API content parts to Responses API parts for the given role."""
return [LiteLLMResponsesInteractionsConfig._transform_content_item(item, role) for item in content]
@staticmethod
def _transform_content_item(item: object, role: str) -> Mapping[str, object]:
text_type: Final = "output_text" if role == "assistant" else "input_text"
if isinstance(item, str):
return {"type": text_type, "text": item}
if isinstance(item, Mapping):
if item.get("type") == "text":
return {"type": text_type, "text": str(item.get("text", ""))}
return item
if isinstance(item, BaseModel):
return LiteLLMResponsesInteractionsConfig._transform_content_item(item.model_dump(exclude_none=True), role)
return {"type": text_type, "text": str(item)}
@staticmethod
def transform_responses_response_to_interactions_response(

4
litellm/litellm_core_utils/core_helpers.py
View file

@ -1,7 +1,7 @@
# What is this?
## Helper utilities
import copy
from collections.abc import Iterable
from collections.abc import Iterable, Mapping
from typing import TYPE_CHECKING, Any, Final, Literal
import httpx
@ -181,7 +181,7 @@ def add_missing_spend_metadata_to_litellm_metadata(litellm_metadata: dict, metad
def get_metadata_variable_name_from_kwargs(
kwargs: dict,
kwargs: Mapping[str, object],
) -> Literal["metadata", "litellm_metadata"]:
"""
Helper to return what the "metadata" field should be called in the request data

15
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -34,12 +34,16 @@ class ExceptionCheckers:
"""
@staticmethod
def is_error_str_rate_limit(error_str: str) -> bool:
def is_error_str_rate_limit(error_str: str, status_code: int | None = None) -> bool:
"""
Check if an error string indicates a rate limit error.
Args:
error_str: The error string to check
status_code: The HTTP status the provider returned, when known. Gates only the
bare-number branch: providers echo the request back in validation errors and
429 is an ordinary token id, so an echoed prompt can put a standalone 429 in
the body of a 400. The phrase branches stay ungated (#11455).
Returns:
True if the error indicates a rate limit, False otherwise
@ -47,8 +51,9 @@ class ExceptionCheckers:
if not isinstance(error_str, str):
return False
# Only treat 429 as a rate limit signal when it appears as a standalone token
if re.search(r"\b429\b", error_str):
# A standalone 429 counts unless the provider's own status says otherwise. The
# status is read off an arbitrary exception, so a non-integer means "unknown".
if re.search(r"\b429\b", error_str) and (not isinstance(status_code, int) or status_code == 429):
return True
_error_str_lower: Final = error_str.lower()
@ -280,7 +285,9 @@ def _map_openai_exception(
else:
exception_provider = custom_llm_provider[0].upper() + custom_llm_provider[1:] + "Exception"
if ExceptionCheckers.is_error_str_rate_limit(error_str):
if ExceptionCheckers.is_error_str_rate_limit(
error_str, status_code=getattr(original_exception, "status_code", None)
):
raise RateLimitError(
message=f"RateLimitError: {exception_provider} - {message}",
model=model,

94
litellm/litellm_core_utils/internal_call_metadata.py Normal file
View file

@ -0,0 +1,94 @@
"""Metadata a request forwards to the internal LLM sub-calls it triggers.
Internal features (the auto-router's classifier and embeddings, shadow eval's shadow and
judge calls) bill real provider spend that nobody typed a prompt for. That spend must land
on the same key/team/org/user as the request that caused it, so the sub-call carries the
caller's identity metadata, minus two things that must never be forwarded as-is:
* ``user_api_key_budget_reservation`` (and the reservation nested inside
``user_api_key_auth``) belongs to the parent completion. If a sub-call's cost callback
sees it, that callback finalizes the reservation and the parent's own callback then
skips incrementing the key/team budget counters, losing the parent's spend.
``user_api_key_auth`` itself is kept, sanitized, because model access-group filtering
needs it.
* The sub-call is stamped with ``INTERNAL_CALL_ORIGIN_METADATA_KEY`` so its spend log row
records that it is not traffic the caller sent.
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import Final
from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY
from litellm.types.utils import InternalCallOrigin
BUDGET_RESERVATION_METADATA_KEYS: Final = frozenset({"user_api_key_budget_reservation"})
_USER_API_KEY_AUTH_KEY: Final = "user_api_key_auth"
FORWARDABLE_IDENTITY_METADATA_KEYS: Final = frozenset(
{
"user_api_key",
"user_api_key_hash",
"user_api_key_alias",
"user_api_key_team_id",
"user_api_key_org_id",
"user_api_key_user_id",
"user_api_key_end_user_id",
_USER_API_KEY_AUTH_KEY,
}
)
"""The caller-identity subset a detached sub-call needs to be attributed and
budget-checked like the request that spawned it. Everything else on the parent's metadata
(routing decision, guardrail state, logging payload) describes the parent call and would
be a lie on a sub-call that runs after it returned."""
def sanitize_user_api_key_auth(auth: object) -> object:
"""Copy of the auth object with its budget reservation removed; the cost callback
falls back to reading the reservation from inside the auth object."""
if isinstance(auth, dict):
return {k: v for k, v in auth.items() if k != "budget_reservation"} # mutable-ok: SDK metadata value
reservation: Final[object] = getattr(auth, "budget_reservation", None)
model_copy: Final[object] = getattr(auth, "model_copy", None)
if reservation is not None and callable(model_copy):
return model_copy(update={"budget_reservation": None}) # mutable-ok: pydantic update payload
return auth
def _sanitized(parent_metadata: Mapping[str, object]) -> dict[str, object]: # mutable-ok: SDK metadata kwarg
return { # mutable-ok: SDK metadata kwarg
k: sanitize_user_api_key_auth(v) if k == _USER_API_KEY_AUTH_KEY else v
for k, v in parent_metadata.items()
if k not in BUDGET_RESERVATION_METADATA_KEYS
}
def forwarded_internal_call_metadata(
parent_metadata: Mapping[str, object] | None,
call_origin: InternalCallOrigin,
) -> dict[str, object]: # mutable-ok: SDK metadata kwarg
"""Parent metadata, minus its budget reservation, stamped with the sub-call's origin.
For sub-calls made inside the parent request (classifier, embeddings), where the
parent's full context still describes the call being made.
"""
if not parent_metadata:
return {} # mutable-ok: SDK metadata kwarg
return _sanitized(parent_metadata) | { # mutable-ok: SDK metadata kwarg
INTERNAL_CALL_ORIGIN_METADATA_KEY: call_origin
}
def sanitized_forwardable_call_metadata(
parent_metadata: Mapping[str, object],
call_origin: InternalCallOrigin,
) -> dict[str, object]: # mutable-ok: SDK metadata kwarg
"""Just the caller's identity, stamped with the sub-call's origin.
For sub-calls detached from the parent request (shadow eval), which outlive it and
must not inherit per-request state such as its routing decision or logging payload.
"""
identity: Final = {k: v for k, v in parent_metadata.items() if k in FORWARDABLE_IDENTITY_METADATA_KEYS}
return _sanitized(identity) | {INTERNAL_CALL_ORIGIN_METADATA_KEY: call_origin} # mutable-ok: SDK metadata kwarg

60
litellm/litellm_core_utils/litellm_logging.py
View file

@ -10,7 +10,7 @@ import subprocess
import sys
import time
import traceback
from collections.abc import Callable, Mapping
from collections.abc import Callable, Mapping, Sequence
from datetime import datetime as dt_object
from functools import lru_cache
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast
@ -176,6 +176,9 @@ from .initialize_dynamic_callback_params import (
from .specialty_caches.dynamic_logging_cache import DynamicLoggingCache
if TYPE_CHECKING:
from mcp.types import EmbeddedResource, ImageContent, TextContent
from litellm.integrations.otel.logger import OpenTelemetryV2
from litellm.llms.base_llm.passthrough.transformation import BasePassthroughConfig
try:
from litellm_enterprise.enterprise_callbacks.callback_controls import (
@ -211,14 +214,30 @@ except Exception as e:
PagerDutyAlerting = CustomLogger
EnterpriseCallbackControls = None
EnterpriseStandardLoggingPayloadSetupVAR = None
_in_memory_loggers: Final[list[Any]] = []
if TYPE_CHECKING:
from litellm.integrations.generic_api.generic_api_callback import (
GenericAPILogger as _GenericAPILoggerCls,
)
_STANDARD_LOGGING_METADATA_KEYS: Final[frozenset] = frozenset(StandardLoggingMetadata.__annotations__.keys())
_GENERIC_API_LOGGER_CLS: Final = _GenericAPILoggerCls
_RESEND_EMAIL_LOGGER_FACTORY: Final = CustomLogger
_SENDGRID_EMAIL_LOGGER_FACTORY: Final = CustomLogger
_SMTP_EMAIL_LOGGER_FACTORY: Final = CustomLogger
_PAGERDUTY_ALERTING_FACTORY: Final = CustomLogger
else:
_GENERIC_API_LOGGER_CLS: Final = GenericAPILogger
_RESEND_EMAIL_LOGGER_FACTORY: Final = ResendEmailLogger
_SENDGRID_EMAIL_LOGGER_FACTORY: Final = SendGridEmailLogger
_SMTP_EMAIL_LOGGER_FACTORY: Final = SMTPEmailLogger
_PAGERDUTY_ALERTING_FACTORY: Final = PagerDutyAlerting
_in_memory_loggers: Final[list[CustomLogger]] = []
_STANDARD_LOGGING_METADATA_KEYS: Final[frozenset[str]] = frozenset(StandardLoggingMetadata.__annotations__.keys())
### GLOBAL VARIABLES ###
# Cache custom pricing keys as frozenset for O(1) lookups instead of looping through 49 keys
_CUSTOM_PRICING_KEYS: Final[frozenset] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
_CUSTOM_PRICING_KEYS: Final[frozenset[str]] = frozenset(CustomPricingLiteLLMParams.model_fields.keys())
sentry_sdk_instance = None
capture_exception = None
@ -1285,7 +1304,9 @@ class Logging(LiteLLMLoggingBaseClass):
verbose_logger.exception("LiteLLM.LoggingError: [Non-Blocking] Exception occurred while logging %s", e)
return response_obj
def _parse_post_mcp_call_hook_response(self, response: MCPPostCallResponseObject | None) -> Any:
def _parse_post_mcp_call_hook_response(
self, response: MCPPostCallResponseObject | None
) -> "Sequence[TextContent | ImageContent | EmbeddedResource] | None":
"""
Parse the response from the post_mcp_tool_call_hook
@ -1729,7 +1750,7 @@ class Logging(LiteLLMLoggingBaseClass):
self.completion_start_time = completion_start_time
self.model_call_details["completion_start_time"] = self.completion_start_time
def normalize_logging_result(self, result: Any) -> Any:
def normalize_logging_result(self, result: Any) -> object:
"""
Some endpoints return a different type of result than what is expected by the logging system.
This function is used to normalize the result to the expected type.
@ -1765,7 +1786,7 @@ class Logging(LiteLLMLoggingBaseClass):
)
return logging_result
def _merge_hidden_params_from_response_into_metadata(self, logging_result: Any) -> None:
def _merge_hidden_params_from_response_into_metadata(self, logging_result: object) -> None:
"""
Copy response._hidden_params into litellm_params.metadata['hidden_params'].
@ -1826,7 +1847,9 @@ class Logging(LiteLLMLoggingBaseClass):
if (standard_logging_payload := self.model_call_details.get("standard_logging_object")) is not None:
emit_standard_logging_payload(standard_logging_payload)
def _build_standard_logging_payload(self, init_response_obj: Any, start_time: Any, end_time: Any) -> Any:
def _build_standard_logging_payload(
self, init_response_obj: object, start_time: Any, end_time: Any
) -> StandardLoggingPayload | None:
"""Build StandardLoggingPayload and accumulate its construction time."""
_start: Final = time.time()
payload: Final = get_standard_logging_object_payload(
@ -1947,7 +1970,7 @@ class Logging(LiteLLMLoggingBaseClass):
def _is_recognized_call_type_for_logging(
self,
logging_result: Any,
logging_result: object,
):
"""
Returns True if the call type is recognized for logging (eg. ModelResponse, ModelResponseStream, etc.)
@ -3439,7 +3462,7 @@ class Logging(LiteLLMLoggingBaseClass):
model=self.model,
messages=[],
logging_obj=self,
optional_params={},
optional_params=self.optional_params or {},
api_key="",
request_data={},
encoding=litellm.encoding,
@ -3460,6 +3483,7 @@ class Logging(LiteLLMLoggingBaseClass):
),
model_response=litellm.ModelResponse(),
json_mode=None,
speed=self.optional_params.get("speed") if self.optional_params else None,
)
return result
@ -4216,7 +4240,7 @@ def _init_custom_logger_compatible_class(
for callback in _in_memory_loggers:
if isinstance(callback, PagerDutyAlerting):
return callback
pagerduty_logger: Final = PagerDutyAlerting(**custom_logger_init_args)
pagerduty_logger: Final = _PAGERDUTY_ALERTING_FACTORY(**custom_logger_init_args)
_in_memory_loggers.append(pagerduty_logger)
return pagerduty_logger
elif logging_integration == "anthropic_cache_control_hook":
@ -4246,7 +4270,7 @@ def _init_custom_logger_compatible_class(
return _gcs_pubsub_logger
elif logging_integration == "generic_api":
for callback in _in_memory_loggers:
if isinstance(callback, GenericAPILogger):
if isinstance(callback, _GENERIC_API_LOGGER_CLS):
return callback
generic_api_logger: Final = GenericAPILogger()
_in_memory_loggers.append(generic_api_logger)
@ -4255,21 +4279,21 @@ def _init_custom_logger_compatible_class(
for callback in _in_memory_loggers:
if isinstance(callback, ResendEmailLogger):
return callback
resend_email_logger: Final = ResendEmailLogger()
resend_email_logger: Final = _RESEND_EMAIL_LOGGER_FACTORY()
_in_memory_loggers.append(resend_email_logger)
return resend_email_logger
elif logging_integration == "sendgrid_email":
for callback in _in_memory_loggers:
if isinstance(callback, SendGridEmailLogger):
return callback
sendgrid_email_logger: Final = SendGridEmailLogger()
sendgrid_email_logger: Final = _SENDGRID_EMAIL_LOGGER_FACTORY()
_in_memory_loggers.append(sendgrid_email_logger)
return sendgrid_email_logger
elif logging_integration == "smtp_email":
for callback in _in_memory_loggers:
if isinstance(callback, SMTPEmailLogger):
return callback
smtp_email_logger: Final = SMTPEmailLogger()
smtp_email_logger: Final = _SMTP_EMAIL_LOGGER_FACTORY()
_in_memory_loggers.append(smtp_email_logger)
return smtp_email_logger
elif logging_integration == "humanloop":
@ -4336,7 +4360,7 @@ def _init_custom_logger_compatible_class(
return None
def _maybe_construct_otel_v2(callback_name: str, _in_memory_loggers: list) -> Any | None:
def _maybe_construct_otel_v2(callback_name: str, _in_memory_loggers: list[CustomLogger]) -> "OpenTelemetryV2 | None":
"""If ``LITELLM_OTEL_V2`` is on, build (or reuse) a single ``OpenTelemetryV2``
instance configured via the preset for ``callback_name``.
@ -4367,7 +4391,7 @@ def _maybe_construct_otel_v2(callback_name: str, _in_memory_loggers: list) -> An
return v2_logger
def _maybe_auto_initialize_arize_phoenix(_in_memory_loggers: list) -> None:
def _maybe_auto_initialize_arize_phoenix(_in_memory_loggers: list[CustomLogger]) -> None:
"""
Auto-initialize ArizePhoenixLogger when Phoenix env vars are detected.
@ -4594,7 +4618,7 @@ def get_custom_logger_compatible_class(
return callback
elif logging_integration == "generic_api":
for callback in _in_memory_loggers:
if isinstance(callback, GenericAPILogger):
if isinstance(callback, _GENERIC_API_LOGGER_CLS):
return callback
elif logging_integration == "resend_email":
for callback in _in_memory_loggers:

19
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
View file

@ -2,6 +2,7 @@
Helper utilities for tracking the cost of built-in tools.
"""
from collections.abc import Mapping
from typing import Any, Final, Literal
import litellm
@ -23,6 +24,14 @@ from litellm.types.utils import (
)
def _usage_reports_server_side_web_search_calls(usage: Usage) -> bool:
details: Final = getattr(usage, "server_side_tool_usage_details", None)
if not isinstance(details, Mapping):
return False
calls: Final = details.get("web_search_calls")
return isinstance(calls, int) and calls > 0
class StandardBuiltInToolCostTracking:
"""
Helper class for tracking the cost of built-in tools
@ -351,6 +360,10 @@ class StandardBuiltInToolCostTracking:
# and _handle_web_search_cost() is never called.
if hasattr(usage, "server_tool_use") and _get_web_search_requests(usage.server_tool_use) is not None:
return True
# xAI reports usage.server_side_tool_usage_details.web_search_calls; a searched
# answer with no url_citation annotations has no other chat-path signal
if _usage_reports_server_side_web_search_calls(usage):
return True
return False
elif isinstance(response_object, ResponsesAPIResponse):
# response api explicitly includes web_search_call in the output
@ -370,6 +383,8 @@ class StandardBuiltInToolCostTracking:
)
):
return True
if _usage_reports_server_side_web_search_calls(usage):
return True
return False
@ -432,7 +447,9 @@ class StandardBuiltInToolCostTracking:
"""
output: Final = response_object.output
for output_item in output:
_output_type: str | None = getattr(output_item, "type", None)
_output_type: str | None = (
output_item.get("type") if isinstance(output_item, dict) else getattr(output_item, "type", None)
)
if _output_type == output_type:
return True
return False

87
litellm/litellm_core_utils/llm_judge.py Normal file
View file

@ -0,0 +1,87 @@
"""Shared primitives for LLM-judge features (llm_as_a_judge guardrail, shadow eval)."""
from __future__ import annotations
import json
import re
from typing import TYPE_CHECKING, Final
import litellm
if TYPE_CHECKING:
from litellm import Router
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
JSON_FENCE_RE: Final = re.compile(r"```(?:json)?\s*(.*?)\s*```", re.DOTALL | re.IGNORECASE)
def default_router_provider() -> Router | None:
try:
from litellm.proxy.proxy_server import llm_router
except ImportError:
return None
return llm_router
def parse_json_verdict(raw: str) -> dict[str, object]: # mutable-ok: plain parsed-JSON payload
"""Parse a judge's JSON verdict, tolerating markdown fences and surrounding prose."""
text = raw.strip() # rebind-ok: progressively narrowed to the JSON payload
fenced: Final = JSON_FENCE_RE.search(text)
if fenced is not None:
text = fenced.group(1).strip() # rebind-ok: progressively narrowed to the JSON payload
parsed: object
try:
parsed = json.loads(text)
except json.JSONDecodeError:
start: Final = text.find("{")
end: Final = text.rfind("}")
if start == -1 or end <= start:
raise
parsed = json.loads(text[start : end + 1])
if not isinstance(parsed, dict):
raise ValueError("judge response is not a JSON object")
return {str(k): v for k, v in parsed.items()} # mutable-ok: plain parsed-JSON payload
def extract_text_from_content(content: object) -> str:
"""Return plain text from a message content field (str or multimodal list)."""
if isinstance(content, str):
return content
if isinstance(content, list):
return " ".join(
str(part.get("text", "")) for part in content if isinstance(part, dict) and part.get("type") == "text"
)
return ""
def router_resolves_model(router: Router | None, model: str) -> bool:
"""Whether the model name resolves through the proxy's router (configured deployment
or model-group alias), the same check the judge dispatch itself makes, so start-time
validation cannot accept a name the call path then fails on."""
return router is not None and bool(model in router.model_group_alias or router.get_model_list(model_name=model))
async def judge_acompletion(
router: Router | None,
judge_model: str,
messages: list[AllMessageValues], # mutable-ok: the SDK acompletion signature takes a list
**params: object,
) -> ModelResponse:
"""Dispatch a judge call through the proxy's router when the judge model is a
configured deployment (DB-stored credentials work), through the SDK for
provider-qualified public names. The router path never retries or falls back:
a failed judge call is the caller's counted failure, not a spend multiplier.
Sampling preferences are advisory: models that removed sampling params (e.g.
claude-sonnet-5) drop them instead of rejecting the judge call."""
if router_resolves_model(router, judge_model):
return await router.acompletion( # pyright: ignore[reportOptionalMemberAccess] # router_resolves_model implies router is not None
model=judge_model,
messages=messages,
num_retries=0,
fallbacks=[],
drop_params=True,
**params,
)
return await litellm.acompletion(model=judge_model, messages=messages, num_retries=0, drop_params=True, **params)

84
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -549,7 +549,7 @@ def _fetch_and_extract_template(
return chat_template, bos_token, eos_token
async def ahf_chat_template(model: str, messages: list, chat_template: Any | None = None):
async def ahf_chat_template(model: str, messages: list, chat_template: str | None = None):
"""HuggingFace chat template (async version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_aget_chat_template_file,
@ -576,7 +576,7 @@ async def ahf_chat_template(model: str, messages: list, chat_template: Any | Non
)
def hf_chat_template(model: str, messages: list, chat_template: Any | None = None):
def hf_chat_template(model: str, messages: list, chat_template: str | None = None):
"""HuggingFace chat template (sync version)"""
from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
_get_chat_template_file,
@ -1130,7 +1130,7 @@ def convert_to_azure_openai_messages(
def infer_protocol_value(
value: Any,
value: object,
) -> Literal[
"string_value",
"number_value",
@ -1702,7 +1702,9 @@ def convert_function_to_anthropic_tool_invoke(
_name: Final = get_attribute_or_key(function_call, "name") or ""
_arguments: Final = get_attribute_or_key(function_call, "arguments")
tool_input = parse_tool_call_arguments(_arguments, tool_name=_name, context="Anthropic function to tool invoke")
tool_input: Final = parse_tool_call_arguments(
_arguments, tool_name=_name, context="Anthropic function to tool invoke"
)
anthropic_tool_invoke: Final = [
AnthropicMessagesToolUseParam(
@ -1764,7 +1766,7 @@ def convert_to_anthropic_tool_invoke(
Fixes: https://github.com/BerriAI/litellm/issues/17737
"""
anthropic_tool_invoke: Final[list[AnthropicMessagesToolUseParam | dict[str, Any]]] = []
anthropic_tool_invoke: Final[list[AnthropicMessagesToolUseParam | dict[str, object]]] = []
for tool in tool_calls:
if not get_attribute_or_key(tool, "type") == "function":
@ -1785,7 +1787,7 @@ def convert_to_anthropic_tool_invoke(
# Server tool IDs start with "srvtoolu_"
if tool_id.startswith("srvtoolu_"):
# Create server_tool_use block instead of tool_use
_anthropic_server_tool_use: dict[str, Any] = {
_anthropic_server_tool_use: dict[str, object] = {
"type": "server_tool_use",
"id": tool_id,
"name": tool_name,
@ -2177,7 +2179,7 @@ def _is_orphaned_tool_result(
return False
def _declared_tool_call_ids(message: Mapping[str, Any]) -> frozenset[str]:
def _declared_tool_call_ids(message: Mapping[str, object]) -> frozenset[str]:
tool_calls: Final = message.get("tool_calls")
if not isinstance(tool_calls, list):
return frozenset()
@ -2186,7 +2188,7 @@ def _declared_tool_call_ids(message: Mapping[str, Any]) -> frozenset[str]:
)
def group_tool_exchanges(messages: Sequence[Mapping[str, Any]]) -> tuple[tuple[int, ...], ...]:
def group_tool_exchanges(messages: Sequence[Mapping[str, object]]) -> tuple[tuple[int, ...], ...]:
"""Group message indices into tool exchanges: an assistant row that made
tool calls, together with the tool rows answering the ids it declared.
@ -2204,7 +2206,7 @@ def group_tool_exchanges(messages: Sequence[Mapping[str, Any]]) -> tuple[tuple[i
return tuple(_iter_tool_exchange_groups(messages))
def _iter_tool_exchange_groups(messages: Sequence[Mapping[str, Any]]) -> Iterator[tuple[int, ...]]:
def _iter_tool_exchange_groups(messages: Sequence[Mapping[str, object]]) -> Iterator[tuple[int, ...]]:
index = 0
while index < len(messages):
declared = _declared_tool_call_ids(messages[index])
@ -2409,7 +2411,7 @@ def anthropic_messages_pt(
# Convert ChatCompletionImageUrlObject to dict if needed
image_url_value = m["image_url"]
if isinstance(image_url_value, str):
image_url_input: str | dict[str, Any] = image_url_value
image_url_input: str | dict[str, object] = image_url_value
else:
# ChatCompletionImageUrlObject or dict case - convert to dict
image_url_input = {
@ -3179,7 +3181,7 @@ def _load_image_from_url(image_url):
try:
# Send a GET request to the image URL
client: Final = HTTPHandler(concurrent_limit=1)
response: Final = safe_get(client, image_url)
response: Final[httpx.Response] = safe_get(client, image_url)
response.raise_for_status() # Raise an exception for HTTP errors
# Check the response's content type to ensure it is an image
@ -3382,7 +3384,7 @@ class BedrockImageProcessor:
@staticmethod
def _post_call_image_processing(response: httpx.Response, image_url: str = "") -> tuple[str, str]:
# Check the response's content type to ensure it is an image
content_type = response.headers.get("content-type")
content_type: str | None = response.headers.get("content-type")
# Use helper function to infer content type with fallback logic
content_type = infer_content_type_from_url_and_content(
@ -3406,7 +3408,7 @@ class BedrockImageProcessor:
params={"concurrent_limit": 1},
)
# Send a GET request to the image URL
response: Final = await async_safe_get(client, image_url)
response: Final[httpx.Response] = await async_safe_get(client, image_url)
response.raise_for_status() # Raise an exception for HTTP errors
return BedrockImageProcessor._post_call_image_processing(response, image_url)
@ -3419,7 +3421,7 @@ class BedrockImageProcessor:
try:
client: Final = HTTPHandler(concurrent_limit=1)
# Send a GET request to the image URL
response: Final = safe_get(client, image_url)
response: Final[httpx.Response] = safe_get(client, image_url)
response.raise_for_status() # Raise an exception for HTTP errors
return BedrockImageProcessor._post_call_image_processing(response, image_url)
@ -3967,6 +3969,36 @@ def _rename_duplicate_bedrock_document_names(
return contents
BEDROCK_DOCUMENT_PLACEHOLDER_TEXT: Final = "."
def _with_text_when_document_only(message: BedrockMessageBlock) -> BedrockMessageBlock:
blocks: Final = message["content"]
needs_text: Final = (
message["role"] == "user"
and any("document" in block for block in blocks)
and all("text" not in block for block in blocks)
)
if not needs_text:
return message
placeholder: Final = BedrockContentBlock(text=BEDROCK_DOCUMENT_PLACEHOLDER_TEXT)
cut: Final = len(blocks) - 1 if "cachePoint" in blocks[-1] else len(blocks)
return BedrockMessageBlock(role="user", content=[*blocks[:cut], placeholder, *blocks[cut:]])
def _ensure_document_messages_have_text(
contents: list[BedrockMessageBlock],
) -> list[BedrockMessageBlock]:
"""
Bedrock Converse rejects any user message that carries a document block
without a sibling text block ("A text block must be included when using
documents"), e.g. Claude Code sends the PDF as a document-only user turn.
Inject a placeholder text block, kept ahead of a trailing cachePoint so
the caller's cache boundary stays the final block.
"""
return [_with_text_when_document_only(message) for message in contents]
def _sort_bedrock_assistant_content_blocks(
blocks: list[BedrockContentBlock],
) -> list[BedrockContentBlock]:
@ -4535,7 +4567,7 @@ class BedrockConverseMessagesProcessor:
llm_provider=llm_provider,
)
return _rename_duplicate_bedrock_document_names(contents)
return _ensure_document_messages_have_text(_rename_duplicate_bedrock_document_names(contents))
@staticmethod
def translate_thinking_blocks_to_reasoning_content_blocks(
@ -4911,7 +4943,7 @@ def _bedrock_converse_messages_pt(
llm_provider=llm_provider,
)
return _rename_duplicate_bedrock_document_names(contents)
return _ensure_document_messages_have_text(_rename_duplicate_bedrock_document_names(contents))
def make_valid_bedrock_tool_name(input_tool_name: str) -> str:
@ -5328,10 +5360,10 @@ def get_attribute_or_key(tool_or_function, attribute, default=None):
class NormalizedToolCall(TypedDict):
id: str | None
name: str | None
arguments: dict[str, Any]
arguments: dict[str, object]
def _parse_tool_call_arguments(raw: Any, tool_name: str | None, context: str) -> dict[str, Any]:
def _parse_tool_call_arguments(raw: Any, tool_name: str | None, context: str) -> dict[str, object]:
# Anthropic's tool_use blocks already carry a parsed dict in "input";
# chat completions and the Responses API carry a JSON string that may be
# truncated by the model, so route those through the repair-aware parser.
@ -5352,12 +5384,12 @@ def _parse_tool_call_arguments(raw: Any, tool_name: str | None, context: str) ->
def _tool_calls_from_chat_completion_response(
response: Any, include_all_choices: bool = False
response: object, include_all_choices: bool = False
) -> list[NormalizedToolCall]:
choices: Final = get_attribute_or_key(response, "choices", None)
if not (isinstance(choices, list) and choices):
return []
tool_calls: Final[list[Any]] = []
tool_calls: Final[list[object]] = []
for choice in choices if include_all_choices else choices[:1]:
message = get_attribute_or_key(choice, "message", None)
choice_tool_calls = get_attribute_or_key(message, "tool_calls", None) if message else None
@ -5383,7 +5415,7 @@ def _tool_calls_from_chat_completion_response(
return result
def _tool_calls_from_responses_api_response(response: Any) -> list[NormalizedToolCall]:
def _tool_calls_from_responses_api_response(response: object) -> list[NormalizedToolCall]:
output: Final = get_attribute_or_key(response, "output", None)
if not isinstance(output, list):
return []
@ -5406,7 +5438,7 @@ def _tool_calls_from_responses_api_response(response: Any) -> list[NormalizedToo
return result
def _tool_calls_from_anthropic_messages_response(response: Any) -> list[NormalizedToolCall]:
def _tool_calls_from_anthropic_messages_response(response: object) -> list[NormalizedToolCall]:
content: Final = get_attribute_or_key(response, "content", None)
if not isinstance(content, list):
return []
@ -5425,7 +5457,7 @@ def _tool_calls_from_anthropic_messages_response(response: Any) -> list[Normaliz
return result
def get_tool_calls_from_response(response: Any, include_all_choices: bool = False) -> list[NormalizedToolCall]:
def get_tool_calls_from_response(response: object, include_all_choices: bool = False) -> list[NormalizedToolCall]:
"""
Extract tool/function calls from a response object into a normalized
``{"id", "name", "arguments"}`` shape, regardless of which API surface
@ -5456,7 +5488,7 @@ def get_tool_calls_from_response(response: Any, include_all_choices: bool = Fals
return []
def has_tool_with_name(tools: Any, tool_name: str) -> bool:
def has_tool_with_name(tools: object, tool_name: str) -> bool:
"""
Check whether a tools list (as sent to an LLM) includes a tool with the
given name, regardless of shape: OpenAI-style function tools
@ -5482,9 +5514,9 @@ def has_tool_with_name(tools: Any, tool_name: str) -> bool:
def resolve_structured_messages(
messages: list[dict[str, Any]] | None,
messages: list[dict[str, object]] | None,
request_kwargs: dict[str, Any],
) -> list[dict[str, Any]] | None:
) -> list[dict[str, object]] | None:
"""
Normalize a request's messages to OpenAI-spec chat-completions shape,
regardless of which API surface produced them (chat completions,

41
litellm/litellm_core_utils/streaming_chunk_builder_utils.py
View file

@ -145,7 +145,7 @@ class ChunkProcessor:
if first_hidden_params.get("created_at"):
def _created_at(chunk: Any) -> int | float:
def _created_at(chunk: object) -> int | float:
if isinstance(chunk, dict):
params = chunk.get("_hidden_params", {})
else:
@ -158,7 +158,7 @@ class ChunkProcessor:
return chunks
def update_model_response_with_hidden_params(
self, model_response: ModelResponse, chunk: dict[str, Any] | None = None
self, model_response: ModelResponse, chunk: Mapping[str, dict[str, object]] | None = None
) -> ModelResponse:
if chunk is None:
return model_response
@ -176,7 +176,7 @@ class ChunkProcessor:
if not chunks:
return
model: Final = getattr(response, "model", None)
model: Final[str | None] = getattr(response, "model", None)
if not model:
return
@ -214,7 +214,7 @@ class ChunkProcessor:
)
@staticmethod
def _get_chunk_id(chunks: list[dict[str, Any]]) -> str:
def _get_chunk_id(chunks: Sequence[Mapping[str, str]]) -> str:
"""
Chunks:
[{"id": ""}, {"id": "1"}, {"id": "1"}]
@ -225,7 +225,7 @@ class ChunkProcessor:
return ""
@staticmethod
def _get_model_from_chunks(chunks: list[dict[str, Any]], first_chunk_model: str) -> str:
def _get_model_from_chunks(chunks: Sequence[Mapping[str, str]], first_chunk_model: str) -> str:
"""
Get the actual model from chunks, preferring a model that differs from the first chunk.
@ -803,8 +803,28 @@ class ChunkProcessor:
completion_tokens_details=completion_tokens_details,
prompt_tokens_details=prompt_tokens_details,
cost=cost,
inference_geo=self._last_provider_pricing_field(chunks, "inference_geo"),
speed=self._last_provider_pricing_field(chunks, "speed"),
)
def _last_provider_pricing_field(
self,
chunks: Sequence["_UsageBearingChunk | ModelResponse"],
field: str,
) -> str | None:
"""
Last value of a provider-specific usage field that changes pricing but is not a
declared ``Usage`` field, e.g. Anthropic's ``speed`` (fast mode multiplies
non-cache token cost) and ``inference_geo``.
"""
values: Final = [
value
for chunk in chunks
if (usage_chunk := self._extract_usage_chunk(chunk)) is not None
and isinstance(value := getattr(usage_chunk, field, None), str)
]
return values[-1] if values else None
@staticmethod
def _reset_anthropic_cursor_completion_tokens(
chunks: Sequence["_UsageBearingChunk | ModelResponse"],
@ -934,7 +954,16 @@ class ChunkProcessor:
# Return a new usage object with the new values
returned_usage = Usage(**returned_usage.model_dump())
provider_pricing_fields: Final = {
field: value
for field, value in (
("inference_geo", calculated_usage_per_chunk["inference_geo"]),
("speed", calculated_usage_per_chunk["speed"]),
)
if value is not None
}
returned_usage = Usage(**returned_usage.model_dump(), **provider_pricing_fields)
return returned_usage

100
litellm/litellm_core_utils/streaming_handler.py
View file

@ -6,13 +6,14 @@ import logging
import threading
import time
import traceback
from collections.abc import AsyncIterator, Callable, Iterator
from collections.abc import AsyncIterator, Callable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from typing import Any, Final, NoReturn, TypeVar, cast
from typing import Any, Final, NoReturn, Protocol, TypeVar, cast
import anyio
import httpx
from pydantic import BaseModel
from typing_extensions import NotRequired, TypedDict
import litellm
from litellm import verbose_logger
@ -54,7 +55,7 @@ _SYNC_ITER_EXHAUSTED: Final = object()
_GCHUNK_FIELDS: Final[frozenset] = frozenset(GChunk.__annotations__)
def _next_sync_or_exhausted(it: Any) -> Any:
def _next_sync_or_exhausted(it: Any) -> object:
"""
Call next(it) from a thread and return _SYNC_ITER_EXHAUSTED on StopIteration.
@ -68,7 +69,7 @@ def _next_sync_or_exhausted(it: Any) -> Any:
return _SYNC_ITER_EXHAUSTED
def is_async_iterable(obj: Any) -> bool:
def is_async_iterable(obj: object) -> bool:
"""
Check if an object is an async iterable (can be used with 'async for').
@ -81,7 +82,7 @@ def is_async_iterable(obj: Any) -> bool:
return isinstance(obj, collections.abc.AsyncIterable)
def print_verbose(print_statement):
def print_verbose(print_statement: object):
try:
if litellm.set_verbose:
print(print_statement) # noqa: T201
@ -96,18 +97,70 @@ class _ProviderChunkParsed:
@dataclass(frozen=True, slots=True)
class _ProviderChunkEarlyReturn:
value: Any
value: "ModelResponseStream | None"
_ProviderChunkResult = _ProviderChunkParsed | _ProviderChunkEarlyReturn
class _PredibaseStreamData(TypedDict):
token: NotRequired[Mapping[str, str]]
details: Mapping[str, str]
generated_text: str | None
error: str | None
class _Ai21StreamData(TypedDict):
completions: Sequence[Mapping[str, Mapping[str, str]]]
class _MaritalkStreamData(TypedDict):
answer: str
class _NlpCloudStreamData(TypedDict):
generated_text: str
class _AlephAlphaStreamData(TypedDict):
completions: Sequence[Mapping[str, str]]
class _AzureStreamChoice(TypedDict):
delta: Mapping[str, str] | None
finish_reason: str | None
class _AzureStreamData(TypedDict):
choices: Sequence[_AzureStreamChoice]
class _BasetenModelOutput(TypedDict):
data: NotRequired[Sequence[str]]
class _BasetenStreamData(TypedDict):
token: NotRequired[Mapping[str, str]]
model_output: NotRequired["_BasetenModelOutput | str"]
completion: NotRequired[object]
class _DeltaDumpDict(TypedDict):
role: NotRequired[str | None]
tool_calls: NotRequired[Sequence[Mapping[str, object]]]
class _TextCompletionChoiceLike(Protocol):
text: str
finish_reason: str | None
class CustomStreamWrapper:
def __init__(
self,
completion_stream,
model,
logging_obj: Any,
logging_obj: LiteLLMLoggingObject,
custom_llm_provider: str | None = None,
stream_options=None,
make_call: Callable | None = None,
@ -186,7 +239,7 @@ class CustomStreamWrapper:
# Snapshot assumes self._hidden_params is populated from litellm_params
# at init and never mutated during the stream. If that ever changes,
# this cache must be removed.
self._base_hidden_params: dict[str, Any] = {
self._base_hidden_params: dict[str, object] = {
**self._hidden_params,
"response_cost": None,
}
@ -416,7 +469,7 @@ class CustomStreamWrapper:
finish_reason = ""
print_verbose(f"chunk: {chunk}")
if chunk.startswith("data:"):
data_json: Final = json.loads(chunk[5:])
data_json: Final[_PredibaseStreamData] = json.loads(chunk[5:])
print_verbose(f"data json: {data_json}")
if "token" in data_json and "text" in data_json["token"]:
text = data_json["token"]["text"]
@ -446,7 +499,7 @@ class CustomStreamWrapper:
def handle_ai21_chunk(self, chunk): # fake streaming
chunk = chunk.decode("utf-8")
data_json: Final = json.loads(chunk)
data_json: Final[_Ai21StreamData] = json.loads(chunk)
try:
text: Final = data_json["completions"][0]["data"]["text"]
is_finished: Final = True
@ -461,7 +514,7 @@ class CustomStreamWrapper:
def handle_maritalk_chunk(self, chunk): # fake streaming
chunk = chunk.decode("utf-8")
data_json: Final = json.loads(chunk)
data_json: Final[_MaritalkStreamData] = json.loads(chunk)
try:
text: Final = data_json["answer"]
is_finished: Final = True
@ -482,7 +535,7 @@ class CustomStreamWrapper:
if self.model and "dolphin" in self.model:
chunk = self.process_chunk(chunk=chunk)
else:
data_json: Final = json.loads(chunk)
data_json: Final[_NlpCloudStreamData] = json.loads(chunk)
chunk = data_json["generated_text"]
text = chunk
if "[DONE]" in text:
@ -499,7 +552,7 @@ class CustomStreamWrapper:
def handle_aleph_alpha_chunk(self, chunk):
chunk = chunk.decode("utf-8")
data_json: Final = json.loads(chunk)
data_json: Final[_AlephAlphaStreamData] = json.loads(chunk)
try:
text: Final = data_json["completions"][0]["completion"]
is_finished: Final = True
@ -527,7 +580,7 @@ class CustomStreamWrapper:
"finish_reason": finish_reason,
}
elif chunk.startswith("data:"):
data_json: Final = json.loads(chunk[5:]) # chunk.startswith("data:"):
data_json: Final[_AzureStreamData] = json.loads(chunk[5:]) # chunk.startswith("data:"):
try:
if len(data_json["choices"]) > 0:
delta: Final = data_json["choices"][0]["delta"]
@ -616,7 +669,7 @@ class CustomStreamWrapper:
text = ""
is_finished = False
finish_reason = None
choices: Final = getattr(chunk, "choices", [])
choices: Final[Sequence[_TextCompletionChoiceLike]] = getattr(chunk, "choices", [])
if len(choices) > 0:
text = choices[0].text
if choices[0].finish_reason is not None:
@ -637,7 +690,7 @@ class CustomStreamWrapper:
is_finished = False
finish_reason = None
usage = None
choices: Final = getattr(chunk, "choices", [])
choices: Final[Sequence[_TextCompletionChoiceLike]] = getattr(chunk, "choices", [])
if len(choices) > 0:
text = choices[0].text
if choices[0].finish_reason is not None:
@ -654,12 +707,12 @@ class CustomStreamWrapper:
except Exception as e:
raise e
def handle_baseten_chunk(self, chunk):
def handle_baseten_chunk(self, chunk) -> str:
try:
chunk = chunk.decode("utf-8")
if len(chunk) > 0:
if chunk.startswith("data:"):
data_json = json.loads(chunk[5:])
data_json: _BasetenStreamData = json.loads(chunk[5:])
if "token" in data_json and "text" in data_json["token"]:
return data_json["token"]["text"]
else:
@ -1325,13 +1378,14 @@ class CustomStreamWrapper:
if response_obj["is_finished"]:
self.received_finish_reason = response_obj["finish_reason"]
if "usage" in response_obj is not None:
_codestral_usage: Final[Usage] = response_obj["usage"]
setattr(
model_response,
"usage",
litellm.Usage(
prompt_tokens=response_obj["usage"].prompt_tokens,
completion_tokens=response_obj["usage"].completion_tokens,
total_tokens=response_obj["usage"].total_tokens,
prompt_tokens=_codestral_usage.prompt_tokens,
completion_tokens=_codestral_usage.completion_tokens,
total_tokens=_codestral_usage.total_tokens,
),
)
elif self.custom_llm_provider == "azure_text":
@ -1474,7 +1528,7 @@ class CustomStreamWrapper:
is None
):
t.function.arguments = ""
_json_delta: Final = delta.model_dump()
_json_delta: Final[_DeltaDumpDict] = delta.model_dump()
if "role" not in _json_delta or _json_delta["role"] is None:
_json_delta["role"] = "assistant" # mistral's api returns role as None
if "tool_calls" in _json_delta and isinstance(_json_delta["tool_calls"], list):
@ -1744,7 +1798,7 @@ class CustomStreamWrapper:
usage.cost, copy it into _hidden_params so litellm's cost
calculator uses it instead of a token-based estimate.
"""
_usage: Final = getattr(response, "usage", None)
_usage: Final[Usage | None] = getattr(response, "usage", None)
if _usage is not None and hasattr(_usage, "cost") and _usage.cost is not None:
if "additional_headers" not in response._hidden_params:
response._hidden_params["additional_headers"] = {}

230
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -14,6 +14,7 @@ Pattern Overview:
import json
from collections.abc import Mapping
from copy import deepcopy
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Final, cast
@ -110,14 +111,10 @@ EMPTY_EXTRACTED_INPUT: Final = ExtractedInput(scanned=(), images=())
class AnthropicMessagesHandler(BaseTranslation):
"""
Handler for processing Anthropic messages with guardrails.
"""Process Anthropic messages with guardrails.
This class provides methods to:
1. Process input messages (pre-call hook)
2. Process output responses (post-call hook)
Methods can be overridden to customize behavior for different message formats.
In-sequence system entries are untrusted client input. This handler scans and preserves
them through guardrail rewrites; downstream provider handling is out of scope.
"""
def __init__(self):
@ -331,16 +328,30 @@ class AnthropicMessagesHandler(BaseTranslation):
skip_tool: Final = effective_skip_tool_message_for_guardrail(guardrail_to_apply)
scan_only_tool_results: Final = effective_scan_only_tool_results_for_guardrail(guardrail_to_apply)
chat_completion_compatible_request: Final = self._translate_to_openai(data)
# Exclude only the trusted top-level prompt. In-sequence system entries are untrusted
# and must stay aligned with texts_to_check for positional masking. When the top-level
# prompt is included, the pre-existing count mismatch disables positional masking.
translation_source: Final = { # mutable-ok: API message payload
key: value for key, value in data.items() if key != "system"
}
chat_completion_compatible_request: Final = self._translate_to_openai(translation_source)
full_structured_messages: Final = cast(
list[AllMessageValues],
chat_completion_compatible_request.get("messages", []),
)
has_midturn_system_message: Final = any(
str(message.get("role") or "").lower() == "system" for message in full_structured_messages
)
hoisted_system_message: Final = None if skip_system else self._hoisted_top_level_system_message(data)
if hoisted_system_message is not None:
full_structured_messages.insert(0, hoisted_system_message)
# skip_system already excluded the trusted top-level prompt (it is simply not hoisted);
# in-sequence system entries are untrusted and always stay in scope.
scoped_message_indices: Final = scoped_structured_message_indices(
full_structured_messages,
scan_only_tool_results=scan_only_tool_results,
skip_system=skip_system,
skip_system=False,
skip_tool=skip_tool,
)
structured_messages: Final = [full_structured_messages[index] for index in scoped_message_indices]
@ -422,6 +433,8 @@ class AnthropicMessagesHandler(BaseTranslation):
scoped_indices=scoped_message_indices,
guardrailed_scoped=guardrailed_structured_messages,
),
hoisted_system_message=hoisted_system_message,
preserve_system_messages=has_midturn_system_message,
)
else:
# Step 3: Map guardrail responses back to original message structure
@ -435,36 +448,150 @@ class AnthropicMessagesHandler(BaseTranslation):
return data
@staticmethod
def _write_back_structured_messages(data: dict, structured_messages: list) -> None:
"""Convert compressed structured_messages back to Anthropic format and write to data.
def _hoisted_top_level_system_message(
self, data: dict
) -> AllMessageValues | None: # mutable-ok: API message payload
"""Return the system message produced by translating the top-level prompt."""
system: Final = data.get("system")
if not system:
return None
probe: Final = self._translate_to_openai(
{ # mutable-ok: API message payload
"model": data.get("model") or "",
"messages": [], # mutable-ok: API message payload
"system": system,
}
)
hoisted: Final = probe.get("messages") or [] # mutable-ok: API message payload
return hoisted[0] if hoisted else None
``anthropic_messages_pt`` merges every run of consecutive user/tool rows
into a single message, so a turn carrying only tool results and the user
turn that follows it come back fused, and the request the model sees no
longer has the boundaries the client sent. Converting a row at a time
would keep them apart but breaks tool pairing: an assistant row whose
tool results sit outside its own call reads as an orphaned tool call,
and under ``modify_params`` the sanitizer answers it with a synthetic
"tool execution skipped" result and drops the real one. Converting each
assistant row together with the tool rows that answer it, and every
other row on its own, satisfies both.
"""
@staticmethod
def _openai_system_message_to_anthropic(
message: dict[str, Any],
) -> dict[str, Any] | None: # mutable-ok: API message payload
"""Convert an OpenAI system message to the client's Anthropic-shaped entry."""
content: Final = message.get("content")
if isinstance(content, str):
return (
{"role": "system", "content": content} if content else None # mutable-ok: API message payload
) # mutable-ok: API message payload
if not isinstance(content, list):
return None
blocks: Final[list[dict[str, Any]]] = [] # mutable-ok: API message payload
for block in content:
if not isinstance(block, dict) or block.get("type") != "text":
continue
text = block.get("text")
if not isinstance(text, str) or not text:
continue
anthropic_block: dict[str, Any] = { # mutable-ok: API message payload
"type": "text",
"text": text,
} # mutable-ok: API message payload
cache_control = block.get("cache_control")
if cache_control:
anthropic_block["cache_control"] = deepcopy(cache_control)
blocks.append(anthropic_block)
return (
{"role": "system", "content": blocks} if blocks else None # mutable-ok: API message payload
) # mutable-ok: API message payload
@staticmethod
def _is_hoisted_top_level_system(message: object, hoisted_system_message: object) -> bool:
"""Match the hoisted prompt by identity, or by value after serialization."""
if hoisted_system_message is None:
return False
if message is hoisted_system_message:
return True
return (
isinstance(message, dict) and isinstance(hoisted_system_message, dict) and message == hoisted_system_message
)
@staticmethod
def _is_system(message: object) -> bool:
"""Whether the row is an in-sequence system message."""
return isinstance(message, dict) and str(message.get("role") or "").lower() == "system"
@staticmethod
def _defer_systems_inside_tool_exchanges(
structured_messages: list, # mutable-ok: API message payload
) -> list:
"""Hold a system row until the tool exchange around it completes so the call/result pair converts together."""
from litellm.litellm_core_utils.prompt_templates.factory import group_tool_exchanges
non_system_positions: Final[list[int]] = [
index
for index, message in enumerate(structured_messages)
if not AnthropicMessagesHandler._is_system(message)
]
exchange_end_for_start: Final[dict[int, int]] = {
non_system_positions[group[0]]: non_system_positions[group[-1]]
for group in group_tool_exchanges([structured_messages[index] for index in non_system_positions])
if len(group) > 1
}
ordered: Final[list] = [] # mutable-ok: API message payload
deferred_systems: Final[list] = [] # mutable-ok: API message payload
open_exchange_end = -1 # rebind-ok: advances to the enclosing exchange's last index
for index, message in enumerate(structured_messages):
if AnthropicMessagesHandler._is_system(message) and index < open_exchange_end:
deferred_systems.append(message)
continue
open_exchange_end = exchange_end_for_start.get(index, open_exchange_end)
ordered.append(message)
if index >= open_exchange_end and deferred_systems:
ordered.extend(deferred_systems)
deferred_systems.clear()
ordered.extend(deferred_systems)
return ordered
@staticmethod
def _write_back_structured_messages(
data: dict, # mutable-ok: API message payload
structured_messages: list, # mutable-ok: API message payload
hoisted_system_message: object = None,
preserve_system_messages: bool = False,
) -> None:
"""Write a guardrail's structured-message rewrite back without losing corrections."""
from litellm.litellm_core_utils.prompt_templates.factory import (
anthropic_messages_pt,
group_tool_exchanges,
)
_is_system: Final = AnthropicMessagesHandler._is_system
model: Final = str(data.get("model") or "")
non_system: Final = [m for m in structured_messages if m.get("role") != "system"]
groups: Final = tuple([non_system[index] for index in group] for group in group_tool_exchanges(non_system)) or (
non_system,
)
converted: Final = [
message
for group in groups
for message in anthropic_messages_pt(messages=group, model=model, llm_provider="anthropic")
]
converted: Final[list] = [] # mutable-ok: API message payload
def _convert_run(run: list) -> None: # mutable-ok: API message payload
for group in group_tool_exchanges(run):
converted.extend(
anthropic_messages_pt(
messages=[run[index] for index in group], # mutable-ok: API message payload
model=model,
llm_provider="anthropic",
)
)
ordered: Final = AnthropicMessagesHandler._defer_systems_inside_tool_exchanges(structured_messages)
run: Final[list] = [] # mutable-ok: API message payload
hoisted_dropped = False # rebind-ok: flips once the hoisted prompt is dropped
for message in ordered:
if not _is_system(message):
run.append(message)
continue
_convert_run(run)
run.clear()
if not hoisted_dropped and AnthropicMessagesHandler._is_hoisted_top_level_system(
message, hoisted_system_message
):
hoisted_dropped = True
continue
if preserve_system_messages:
anthropic_system = AnthropicMessagesHandler._openai_system_message_to_anthropic(message)
if anthropic_system is not None:
converted.append(anthropic_system)
_convert_run(run)
if not any(not _is_system(message) for message in converted):
converted.extend(anthropic_messages_pt(messages=[], model=model, llm_provider="anthropic"))
for msg in converted:
content = msg.get("content")
if isinstance(content, list):
@ -473,6 +600,31 @@ class AnthropicMessagesHandler(BaseTranslation):
block.pop("cache_control", None)
data["messages"] = converted
@staticmethod
def _extract_midturn_system_text(
message: dict[str, Any], # mutable-ok: API message payload
msg_idx: int,
) -> ExtractedInput:
"""Match the adapter's filtering so positional guardrail write-back stays aligned."""
content: Final = message.get("content")
if isinstance(content, str):
if not content:
return EMPTY_EXTRACTED_INPUT
return ExtractedInput(scanned=(ScannedText(content, MessageContentTarget(msg_idx)),), images=())
if not isinstance(content, list):
return EMPTY_EXTRACTED_INPUT
return ExtractedInput(
scanned=tuple(
ScannedText(text_str, ContentBlockTextTarget(msg_idx, content_idx))
for content_idx, content_item in enumerate(content)
if isinstance(content_item, dict)
and content_item.get("type") == "text"
and isinstance(text_str := content_item.get("text"), str)
and text_str
),
images=(),
)
def extract_request_tool_names(self, data: dict) -> list[str]:
"""Extract tool names from Anthropic messages request (tools[].name)."""
names: Final[list[str]] = []
@ -490,11 +642,17 @@ class AnthropicMessagesHandler(BaseTranslation):
skip_tool_message: bool = False,
scan_only_tool_results: bool = False,
) -> ExtractedInput:
"""Extract text content and images from a message.
In-sequence system entries are scanned even when ``skip_system_message`` is set:
that flag covers only the trusted top-level prompt, which never appears here.
"""
Extract text content and images from a message.
"""
role: Final = str(message.get("role") or "").lower()
if (skip_system_message and role == "system") or (skip_tool_message and role == "tool"):
role: Final = str(message.get("role") or "")
if role == "system":
if scan_only_tool_results:
return EMPTY_EXTRACTED_INPUT
return cls._extract_midturn_system_text(message=message, msg_idx=msg_idx)
if skip_tool_message and role.lower() == "tool":
return EMPTY_EXTRACTED_INPUT
content: Final = message.get("content", None)

5
litellm/llms/anthropic/chat/handler.py
View file

@ -58,6 +58,7 @@ from ..common_utils import AnthropicError, process_anthropic_headers
from .transformation import ANTHROPIC_TOOL_NAME_REVERSE_MAP_KEY, AnthropicConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
from litellm.llms.base_llm.chat.transformation import BaseConfig
@ -206,7 +207,7 @@ class AnthropicChatCompletion(BaseLLM):
client: AsyncHTTPHandler | None,
encoding,
api_key,
logging_obj,
logging_obj: "LiteLLMLoggingObj",
stream,
_is_function_call,
data: dict,
@ -324,7 +325,7 @@ class AnthropicChatCompletion(BaseLLM):
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: "LiteLLMLoggingObj",
optional_params: dict,
timeout: float | httpx.Timeout,
litellm_params: dict,

39
litellm/llms/anthropic/common_utils.py
View file

@ -5,6 +5,7 @@ This file contains common utils for anthropic calls.
import copy
import re
from collections.abc import Mapping, Sequence
from datetime import datetime, timezone
from types import MappingProxyType
from typing import Any, Final, Literal
@ -12,6 +13,7 @@ import httpx
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
import litellm
from litellm.constants import DEFAULT_MODEL_CREATED_AT_TIME
from litellm.litellm_core_utils.prompt_templates.common_utils import (
get_file_ids_from_messages,
)
@ -28,6 +30,7 @@ from litellm.types.llms.anthropic import (
AnthropicMcpServerTool,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.proxy.model_listing import ModelInfoResponse
_BEDROCK_VERSION_SUFFIX_RE: Final = re.compile(r"-v\d+(?::\d+)?$")
_INFERENCE_PROFILE_MINOR_RE: Final = re.compile(r":\d+$")
@ -1221,3 +1224,39 @@ def process_anthropic_headers(headers: httpx.Headers | dict) -> dict:
additional_headers: Final = {**llm_response_headers, **openai_headers}
return additional_headers
def _anthropic_model_entry(model: ModelInfoResponse, created_at: str) -> Mapping[str, object]:
token_limits: Final = (
("max_input_tokens", model.get("max_input_tokens")),
("max_tokens", model.get("max_output_tokens")),
)
return { # mutable-ok: JSON response body, serialized by the route and never mutated
"type": "model",
"id": model["id"],
"display_name": model["id"],
"created_at": created_at,
**{name: limit for name, limit in token_limits if limit is not None}, # mutable-ok: merged into the body above
}
def create_anthropic_model_list_response(models: Sequence[ModelInfoResponse]) -> Mapping[str, object]:
"""Build the Anthropic-native /v1/models envelope.
Clients that send an anthropic-version header parse the Anthropic Models API
shape (type/display_name/created_at plus has_more/first_id/last_id) and filter
the list themselves, so every model is returned here. The token limits carry
over from the OpenAI-shaped listing, named as the Messages API names them
"""
created_at: Final = (
datetime.fromtimestamp(DEFAULT_MODEL_CREATED_AT_TIME, tz=timezone.utc).isoformat().replace("+00:00", "Z")
)
data: Final = [ # mutable-ok: JSON response body, serialized by the route and never mutated
_anthropic_model_entry(model, created_at) for model in models
]
return { # mutable-ok: JSON response body, serialized by the route and never mutated
"data": data,
"has_more": False,
"first_id": models[0]["id"] if models else None,
"last_id": models[-1]["id"] if models else None,
}

49
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
View file

@ -74,12 +74,12 @@ from litellm.llms.anthropic.experimental_pass_through.context_management import
)
from litellm.types.llms.anthropic import (
ANTHROPIC_HOSTED_TOOLS,
AllAnthropicPassThroughMessageValues,
AllAnthropicToolsValues,
AnthopicMessagesAssistantMessageParam,
AnthropicFinishReason,
AnthropicMessagesRequest,
AnthropicMessagesSystemMessageParam,
AnthropicMessagesToolChoice,
AnthropicMessagesUserMessageParam,
AnthropicResponseContentBlockRedactedThinking,
AnthropicResponseContentBlockText,
AnthropicResponseContentBlockThinking,
@ -343,7 +343,7 @@ class LiteLLMAnthropicMessagesAdapter:
def translate_anthropic_messages_to_openai(
self,
messages: list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam],
messages: list[AllAnthropicPassThroughMessageValues],
model: str | None = None,
) -> list:
new_messages: Final[list[AllMessageValues]] = []
@ -351,6 +351,11 @@ class LiteLLMAnthropicMessagesAdapter:
user_message: ChatCompletionUserMessage | None = None
tool_message_list: list[ChatCompletionToolMessage] = []
new_user_content_list: list[ChatCompletionTextObject | ChatCompletionImageObject] = []
if m["role"] == "system":
system_message = self._translate_midturn_system_message_to_openai(m, model)
if system_message is not None:
new_messages.append(system_message)
continue
## USER MESSAGE ##
if m["role"] == "user":
## translate user message
@ -848,6 +853,29 @@ class LiteLLMAnthropicMessagesAdapter:
for def_schema in schema[key].values():
LiteLLMAnthropicMessagesAdapter._add_additional_properties_false(def_schema)
def _translate_midturn_system_message_to_openai(
self,
message: AnthropicMessagesSystemMessageParam,
model: str | None,
) -> ChatCompletionSystemMessage | None:
"""Translate an in-sequence system entry without changing its role or position."""
content: Final = message.get("content")
if isinstance(content, str):
return ChatCompletionSystemMessage(role="system", content=content) if content else None
if not isinstance(content, list):
return None
text_parts: Final[list[ChatCompletionTextObject]] = [] # mutable-ok: API message payload
for block in content:
if not isinstance(block, dict) or block.get("type") != "text": # pyright: ignore[reportUnnecessaryIsInstance] # untrusted client payload
continue
text = block.get("text")
if not text:
continue
text_obj = ChatCompletionTextObject(type="text", text=text)
self._add_cache_control_if_applicable(block, text_obj, model)
text_parts.append(text_obj)
return ChatCompletionSystemMessage(role="system", content=text_parts) if text_parts else None
def _add_system_message_to_messages(
self,
new_messages: list[AllMessageValues],
@ -976,6 +1004,17 @@ class LiteLLMAnthropicMessagesAdapter:
model: Final = new_kwargs.get("model", "")
if self.is_anthropic_claude_model(model) or self.is_bedrock_arn_model(model):
new_kwargs["thinking"] = thinking
# Adaptive thinking without its effort tier makes Bedrock Converse
# return zero reasoning blocks, so forward output_config (minus
# `format`, already translated to response_format) for Bedrock
# targets only: other bridged providers reject the raw param, and
# get_llm_provider strips the `bedrock/` prefix before this runs.
if model.startswith(("bedrock/", "converse/", "invoke/")) or self.is_bedrock_arn_model(model):
claude_output_config: Final = anthropic_message_request.get("output_config")
if isinstance(claude_output_config, dict):
effort_config: Final = {k: v for k, v in claude_output_config.items() if k != "format"}
if effort_config:
new_kwargs["output_config"] = effort_config # rebind-ok: out-param store like thinking above
return
reasoning_effort = self.translate_anthropic_thinking_to_reasoning_effort(cast(dict[str, Any], thinking))
@ -1049,8 +1088,8 @@ class LiteLLMAnthropicMessagesAdapter:
tool_name_mapping: dict[str, str] = {}
## CONVERT ANTHROPIC MESSAGES TO OPENAI
messages_list: Final[list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam]] = cast(
list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam],
messages_list: Final[list[AllAnthropicPassThroughMessageValues]] = cast(
list[AllAnthropicPassThroughMessageValues],
anthropic_message_request["messages"],
)
new_messages = self.translate_anthropic_messages_to_openai(

66
litellm/llms/anthropic/experimental_pass_through/context_management/editors/compact.py
View file

@ -13,8 +13,10 @@ Mirrors Anthropic's native ``compact_20260112`` for non-Anthropic providers:
"""
import re
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast
from collections.abc import Mapping, Sequence
from typing import TYPE_CHECKING, Any, Final, Literal, NotRequired, Optional, TypedDict, Union, cast
from typing_extensions import ReadOnly
import litellm
from litellm._logging import verbose_logger
@ -29,9 +31,8 @@ if TYPE_CHECKING:
from litellm.proxy._types import UserAPIKeyAuth
from litellm.router import Router
from litellm.types.llms.anthropic import (
AllAnthropicPassThroughMessageValues,
AllAnthropicToolsValues,
AnthopicMessagesAssistantMessageParam,
AnthropicMessagesUserMessageParam,
)
from litellm.types.llms.openai import ChatCompletionToolParam
from litellm.types.utils import ModelResponse
@ -534,7 +535,7 @@ def _augment_system_with_summary(
return [{"type": "text", "text": prefix.rstrip()}, *system]
def _resolve_trigger_tokens(edit_spec: dict[str, object]) -> tuple[int, list[str]]:
def _resolve_trigger_tokens(edit_spec: Mapping[str, object]) -> tuple[int, list[str]]:
"""Validate and resolve ``trigger.value``.
Raises ``AnthropicContextManagementError`` if the explicitly-supplied value
@ -568,7 +569,7 @@ def _resolve_trigger_tokens(edit_spec: dict[str, object]) -> tuple[int, list[str
return value, warnings
def _build_summary_prompt(edit_spec: dict[str, object], tools: list[dict[str, object]] | None) -> str:
def _build_summary_prompt(edit_spec: Mapping[str, object], tools: Sequence[Mapping[str, object]] | None) -> str:
custom: Final = edit_spec.get("instructions")
if isinstance(custom, str) and custom.strip():
return custom
@ -623,7 +624,7 @@ def _count_effective_tokens(
try:
openai_shape = adapter.translate_anthropic_messages_to_openai(
messages=cast(
"list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam]",
"list[AllAnthropicPassThroughMessageValues]",
messages_without_compaction,
)
)
@ -736,7 +737,7 @@ def _extract_summary_text(raw: str | None) -> str | None:
def _system_to_openai_message(
system: str | list[dict[str, Any]] | None,
) -> dict[str, Any] | None:
) -> dict[str, object] | None:
"""Translate Anthropic-shaped ``system`` to an OpenAI system message.
Accepts a bare string or a list of Anthropic content blocks; returns
@ -773,7 +774,7 @@ def _build_summary_messages(
try:
openai_messages = LiteLLMAnthropicMessagesAdapter().translate_anthropic_messages_to_openai(
messages=cast(
"list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam]",
"list[AllAnthropicPassThroughMessageValues]",
stripped,
)
)
@ -809,7 +810,7 @@ def _is_user_message(msg: object) -> bool:
return isinstance(msg, dict) and msg.get("role") == "user"
def _append_text_to_content(content: Any, extra_text: str) -> Any:
def _append_text_to_content(content: object, extra_text: str) -> object:
"""Append ``extra_text`` to an OpenAI-shape message ``content`` field.
Handles the two common shapes: ``str`` and ``list`` of content parts.
@ -820,10 +821,29 @@ def _append_text_to_content(content: Any, extra_text: str) -> Any:
if isinstance(content, str):
return f"{content}\n\n{extra_text}"
if isinstance(content, list):
return [*content, {"type": "text", "text": extra_text}]
appended: Final[list[object]] = [*content, {"type": "text", "text": extra_text}]
return appended
return [content, {"type": "text", "text": extra_text}]
class _SummaryCallUserKwarg(TypedDict, total=False):
user: ReadOnly[object]
class _SummaryCallRegionKwarg(TypedDict, total=False):
allowed_model_region: ReadOnly[str]
class _SummaryCallKwargs(TypedDict):
model: ReadOnly[str]
messages: ReadOnly[list[dict[str, object]]]
max_tokens: ReadOnly[int]
timeout: ReadOnly[float]
litellm_metadata: ReadOnly[Mapping[str, object]]
user: NotRequired[ReadOnly[object]]
allowed_model_region: NotRequired[ReadOnly[str]]
async def _call_summary_model(
*,
summary_model: str,
@ -860,22 +880,24 @@ async def _call_summary_model(
# the parent ``/v1/messages`` request. On timeout the caller catches the
# exception and surfaces ``applied_edits[0].error = "summary_call_failed"``,
# forwarding the request without compaction rather than hanging.
call_kwargs: Final[dict[str, Any]] = {
"model": summary_model,
"messages": summary_messages,
"max_tokens": max_tokens,
"timeout": COMPACT_SUMMARY_TIMEOUT_SECONDS,
"litellm_metadata": metadata,
}
# The end-user id must also travel as the top-level ``user`` kwarg: legacy
# limiter hooks and prometheus end-user tracking read it from there rather
# than from ``litellm_metadata``, so without it the summary tokens would not
# debit the caller's end-user counters.
end_user_id: Final = metadata.get("user_api_key_end_user_id")
if end_user_id:
call_kwargs["user"] = end_user_id
if allowed_model_region is not None:
call_kwargs["allowed_model_region"] = allowed_model_region
call_kwargs: Final[_SummaryCallKwargs] = {
"model": summary_model,
"messages": summary_messages,
"max_tokens": max_tokens,
"timeout": COMPACT_SUMMARY_TIMEOUT_SECONDS,
"litellm_metadata": metadata,
**(_SummaryCallUserKwarg(user=end_user_id) if end_user_id else _SummaryCallUserKwarg()),
**(
_SummaryCallRegionKwarg(allowed_model_region=allowed_model_region)
if allowed_model_region is not None
else _SummaryCallRegionKwarg()
),
}
if llm_router is not None and hasattr(llm_router, "acompletion"):
return await llm_router.acompletion(**call_kwargs)
return await litellm.acompletion(**call_kwargs)

39
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py
View file

@ -6,6 +6,7 @@ path used for OpenAI and Azure models.
"""
import json
from collections.abc import Iterable
from typing import Any, Final, cast
from litellm.litellm_core_utils.reasoning_effort_utils import (
@ -15,15 +16,15 @@ from litellm.llms.anthropic.experimental_pass_through.utils import (
is_reasoning_auto_summary_enabled,
)
from litellm.types.llms.anthropic import (
AllAnthropicPassThroughMessageValues,
AllAnthropicToolsValues,
AnthopicMessagesAssistantMessageParam,
AnthropicFinishReason,
AnthropicMessagesRequest,
AnthropicMessagesToolChoice,
AnthropicMessagesUserMessageParam,
AnthropicResponseContentBlockText,
AnthropicResponseContentBlockThinking,
AnthropicResponseContentBlockToolUse,
AnthropicSystemMessageContent,
)
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
@ -72,14 +73,32 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
return source.get("url")
return None
@staticmethod
def _translate_midturn_system_content_to_responses(
content: str | Iterable[AnthropicSystemMessageContent],
) -> list[dict[str, str]]: # mutable-ok: API message payload
"""Convert in-sequence system content to Responses input-text parts."""
if isinstance(content, str):
return (
[{"type": "input_text", "text": content}] if content else [] # mutable-ok: API message payload
) # mutable-ok: API message payload
if not isinstance(content, list):
return [] # mutable-ok: API message payload
return [ # mutable-ok: API message payload
{"type": "input_text", "text": text} # mutable-ok: API message payload
for block in content
if isinstance(block, dict) and block.get("type") == "text" and (text := block.get("text")) # pyright: ignore[reportUnnecessaryIsInstance] # untrusted client payload
]
def translate_messages_to_responses_input(
self,
messages: list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam],
messages: list[AllAnthropicPassThroughMessageValues],
) -> list[dict[str, Any]]:
"""
Convert Anthropic messages list to Responses API `input` items.
Mapping:
system text -> message(role=system, input_text)
user text -> message(role=user, input_text)
user image -> message(role=user, input_image)
user tool_result -> function_call_output
@ -89,6 +108,18 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
input_items: Final[list[dict[str, Any]]] = []
for m in messages:
if m["role"] == "system":
system_parts = self._translate_midturn_system_content_to_responses(m.get("content"))
if system_parts:
input_items.append(
{ # mutable-ok: API message payload
"type": "message",
"role": "system",
"content": system_parts,
}
)
continue
role = m["role"]
content = m.get("content")
@ -300,7 +331,7 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
"""
model: Final[str] = anthropic_request["model"]
messages_list: Final = cast(
list[AnthropicMessagesUserMessageParam | AnthopicMessagesAssistantMessageParam],
list[AllAnthropicPassThroughMessageValues],
anthropic_request["messages"],
)

6
litellm/llms/azure/azure.py
View file

@ -228,7 +228,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
litellm_params=litellm_params,
)
data = {"model": None, "messages": messages, **optional_params}
data: dict[str, object] = {"model": None, "messages": messages, **optional_params}
elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=litellm_params.get("base_model") or model):
data = litellm.AzureOpenAIGPT5Config().transform_request(
model=model,
@ -482,12 +482,12 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
def streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_base: str,
api_key: str | None,
api_version: str,
dynamic_params: bool,
data: dict,
data: dict[str, object],
model: str,
timeout: Any,
max_retries: int,

5
litellm/llms/azure/chat/o_series_handler.py
View file

@ -5,10 +5,11 @@ Written separately to handle faking streaming for o1 and o3 models.
"""
from collections.abc import Callable
from typing import TYPE_CHECKING, Any, Optional
from typing import TYPE_CHECKING, Optional
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.utils import ModelResponse
from ...openai.openai import OpenAIChatCompletion
@ -25,7 +26,7 @@ class AzureOpenAIO1ChatCompletion(BaseAzureLLM, OpenAIChatCompletion):
timeout: float | httpx.Timeout,
optional_params: dict,
litellm_params: dict,
logging_obj: Any,
logging_obj: LiteLLMLoggingObj,
model: str | None = None,
messages: list | None = None,
print_verbose: Callable | None = None,

31
litellm/llms/azure/common_utils.py
View file

@ -2,11 +2,12 @@ import asyncio
import hashlib
import json
import os
from collections.abc import Callable
from collections.abc import Callable, Mapping
from typing import Any, Final, Literal, NamedTuple, cast
import httpx
from openai import AsyncAzureOpenAI, AsyncOpenAI, AzureOpenAI, OpenAI
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm._logging import verbose_logger
@ -23,6 +24,22 @@ from litellm.utils import _add_path_to_api_base
azure_ad_cache: Final = DualCache()
class _AzureAdTokenJson(TypedDict, total=False):
access_token: ReadOnly[str]
expires_in: ReadOnly[int]
class _AzureV1ClientParams(TypedDict, total=False, extra_items=object):
base_url: ReadOnly[str]
class _AzureGatewayClientParams(TypedDict, total=False, extra_items=object):
api_version: ReadOnly[str]
base_url: ReadOnly[str]
max_retries: ReadOnly[int]
timeout: ReadOnly[float | httpx.Timeout]
class AzureOpenAIError(BaseLLMException):
def __init__(
self,
@ -220,7 +237,7 @@ def get_azure_ad_token_from_oidc(
message=req_token.text,
)
azure_ad_token_json: Final = req_token.json()
azure_ad_token_json: Final[_AzureAdTokenJson] = req_token.json()
azure_ad_token_access_token = azure_ad_token_json.get("access_token", None)
azure_ad_token_expires_in: Final = azure_ad_token_json.get("expires_in", None)
@ -486,7 +503,7 @@ class BaseAzureLLM(BaseOpenAILLM):
v1_api_key = _async_v1_api_key
v1_params: Final[dict[str, Any]] = {
v1_params: Final[_AzureV1ClientParams] = {
"api_key": v1_api_key,
"base_url": f"{api_base}/openai/v1/",
}
@ -643,7 +660,7 @@ class BaseAzureLLM(BaseOpenAILLM):
api_base += "/"
api_base += f"{model}"
azure_client_params: Final[dict[str, Any]] = {
azure_client_params: Final[_AzureGatewayClientParams] = {
"api_version": api_version,
"base_url": f"{api_base}",
"http_client": litellm.client_session,
@ -702,7 +719,7 @@ class BaseAzureLLM(BaseOpenAILLM):
@staticmethod
def _get_base_azure_url(
api_base: str | None,
litellm_params: GenericLiteLLMParams | dict[str, Any] | None,
litellm_params: GenericLiteLLMParams | Mapping[str, object] | None,
route: Literal["/openai/responses", "/openai/vector_stores"] | str,
default_api_version: str | Literal["latest", "preview"] | None = None,
) -> str:
@ -757,7 +774,9 @@ class BaseAzureLLM(BaseOpenAILLM):
return False
return api_version in {"preview", "latest", "v1"}
def _resolve_env_var(self, litellm_params: dict[str, Any], param_key: str, env_var_key: str) -> str | None:
def _resolve_env_var(
self, litellm_params: Mapping[str, str | None], param_key: str, env_var_key: str
) -> str | None:
"""Resolve the environment variable for a given parameter key.
The logic here is different from `params.get(key, os.getenv(env_var))` because

9
litellm/llms/azure/completion/handler.py
View file

@ -3,6 +3,7 @@ from typing import Any, Final
from openai import AsyncAzureOpenAI, AzureOpenAI
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.prompt_templates.factory import prompt_factory
from litellm.utils import CustomStreamWrapper, ModelResponse, TextCompletionResponse
@ -39,9 +40,9 @@ class AzureTextCompletion(BaseAzureLLM):
azure_ad_token_provider: Callable | None,
print_verbose: Callable,
timeout,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params,
litellm_params,
litellm_params: dict[str, object],
logger_fn,
acompletion: bool = False,
headers: dict | None = None,
@ -246,7 +247,7 @@ class AzureTextCompletion(BaseAzureLLM):
def streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_base: str,
api_key: str | None,
api_version: str,
@ -299,7 +300,7 @@ class AzureTextCompletion(BaseAzureLLM):
async def async_streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_base: str,
api_key: str | None,
api_version: str,

3
litellm/llms/azure_ai/anthropic/handler.py
View file

@ -9,6 +9,7 @@ from typing import Final
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.anthropic.chat.handler import AnthropicChatCompletion
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
@ -40,7 +41,7 @@ class AzureAnthropicChatCompletion(AnthropicChatCompletion):
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
timeout: float | httpx.Timeout,
litellm_params: dict,

2
litellm/llms/azure_ai/chat/transformation.py
View file

@ -248,7 +248,7 @@ class AzureAIStudioConfig(OpenAIConfig):
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
encoding=encoding,
encoding=encoding if encoding is not None else None,
api_key=api_key,
json_mode=json_mode,
)

17
litellm/llms/bedrock/batches/transformation.py
View file

@ -28,7 +28,11 @@ from litellm.types.llms.openai import (
from litellm.types.utils import LiteLLMBatch, LlmProviders
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import CommonBatchFilesUtils, resolve_s3_encryption_key_id
from ..common_utils import (
CommonBatchFilesUtils,
merge_bedrock_aws_request_params,
resolve_s3_encryption_key_id,
)
# Bedrock batch input files are uploaded as
# s3://bucket/litellm-bedrock-files-{model, ":" -> "-"}-{uuid4}.jsonl (see
@ -130,7 +134,8 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
Get the complete URL for Bedrock batch creation.
Bedrock batch jobs are created via the model invocation job API.
"""
aws_region_name: Final = self._get_aws_region_name(optional_params, model)
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
aws_region_name: Final = self._get_aws_region_name(request_params, model)
# Bedrock model invocation job endpoint
# Format: https://bedrock.{region}.amazonaws.com/model-invocation-job
@ -232,14 +237,15 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
# For Bedrock, we need to return a pre-signed request with AWS auth headers
# Use common utility for AWS signing
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
endpoint_url: Final = (
f"https://bedrock.{self._get_aws_region_name(optional_params, model)}.amazonaws.com/model-invocation-job"
f"https://bedrock.{self._get_aws_region_name(request_params, model)}.amazonaws.com/model-invocation-job"
)
signed_headers, signed_data = self.common_utils.sign_aws_request(
service_name="bedrock",
data=bedrock_request,
endpoint_url=endpoint_url,
optional_params=optional_params,
optional_params=request_params,
method="POST",
)
@ -387,11 +393,12 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
endpoint_url: Final = f"https://bedrock.{region}.amazonaws.com/model-invocation-job/{encoded_arn}"
# Use common utility for AWS signing
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
signed_headers, _ = self.common_utils.sign_aws_request(
service_name="bedrock",
data={}, # GET request has no body
endpoint_url=endpoint_url,
optional_params=optional_params,
optional_params=request_params,
method="GET",
)

2
litellm/llms/bedrock/chat/converse_handler.py
View file

@ -89,7 +89,7 @@ class BedrockConverseLLM(BaseAWSLLM):
model_response: ModelResponse,
timeout: float | httpx.Timeout | None,
encoding,
logging_obj,
logging_obj: LiteLLMLoggingObject,
stream,
optional_params: dict,
litellm_params: dict,

14
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -80,6 +80,7 @@ from ..common_utils import (
bedrock_converse_supports_parallel_tool_use_config,
get_anthropic_beta_from_headers,
get_bedrock_tool_name,
is_bedrock_application_inference_profile_arn,
is_claude_4_5_on_bedrock,
normalize_bedrock_opus_output_config_effort,
)
@ -514,6 +515,7 @@ class AmazonConverseConfig(BaseConfig):
supported_params.append("tool_choice")
supported_params.append("thinking")
supported_params.append("reasoning_effort")
supported_params.append("output_config")
# For nova imported models, also add web_search_options
if "nova" in model.lower():
supported_params.append("web_search_options")
@ -564,6 +566,7 @@ class AmazonConverseConfig(BaseConfig):
):
supported_params.append("thinking")
supported_params.append("reasoning_effort")
supported_params.append("output_config")
if base_model.startswith("anthropic"):
supported_params.append("context_management")
@ -919,6 +922,10 @@ class AmazonConverseConfig(BaseConfig):
self._handle_reasoning_effort_parameter(
model=model, reasoning_effort=value, optional_params=optional_params
)
elif param == "output_config" and isinstance(value, dict):
mapped_output_config = dict(value)
normalize_bedrock_opus_output_config_effort(model=model, output_config=mapped_output_config)
optional_params["output_config"] = mapped_output_config # rebind-ok: out-param store like siblings
elif param == "context_management" and isinstance(value, (dict, list)):
self._map_context_management_param(value, optional_params)
if param == "requestMetadata":
@ -1312,7 +1319,12 @@ class AmazonConverseConfig(BaseConfig):
additional_request_params = filter_exceptions_from_params(additional_request_params)
if anthropic_output_config is not None and isinstance(anthropic_output_config, dict):
if base_model.startswith("anthropic"):
# Application inference profile ARNs hide the underlying model, so the
# effort ceiling and capability gates below cannot run; forward
# verbatim (like ``thinking``) and let Bedrock enforce.
if is_bedrock_application_inference_profile_arn(model):
additional_request_params["output_config"] = anthropic_output_config
elif base_model.startswith("anthropic"):
if litellm.drop_params is True and not AnthropicConfig._model_supports_effort_param(model, "bedrock"):
litellm.verbose_logger.warning(
DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,

6
litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -19,9 +19,9 @@ from litellm.llms.bedrock.common_utils import (
convert_bedrock_invoke_output_format_to_inline_schema,
get_anthropic_beta_from_headers,
normalize_bedrock_opus_output_config_effort,
normalize_custom_field_on_tools,
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
remove_custom_field_from_tools,
)
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
from litellm.types.llms.openai import AllMessageValues
@ -243,8 +243,8 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
if "anthropic_version" not in anthropic_request:
anthropic_request["anthropic_version"] = self.anthropic_version
# Remove `custom` field from tools (Bedrock doesn't support it)
remove_custom_field_from_tools(anthropic_request)
# Hoist `custom.defer_loading` then drop `custom` (Bedrock doesn't support it)
normalize_custom_field_on_tools(anthropic_request)
normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_request)
return anthropic_request

59
litellm/llms/bedrock/common_utils.py
View file

@ -36,6 +36,44 @@ class BedrockError(BaseLLMException):
pass
_BEDROCK_AWS_AUTH_PARAMETER_KEYS: Final[tuple[str, ...]] = (
"aws_access_key_id",
"aws_secret_access_key",
"aws_session_token",
"aws_region_name",
"aws_session_name",
"aws_profile_name",
"aws_role_name",
"aws_web_identity_token",
"aws_sts_endpoint",
"aws_external_id",
)
def merge_bedrock_aws_request_params(
litellm_params: Mapping[str, Any],
optional_params: Mapping[str, Any],
) -> dict[str, Any]:
"""Merge deployment and request parameters without allowing auth escalation.
Deployment configuration is authoritative for AWS authentication. When a
deployment supplies static credentials, caller-supplied profile/role/token
selectors must not redirect signing to another identity available on the
server. Requests may still provide AWS credentials when the deployment has
no static credentials configured.
"""
request_params: Final = {**optional_params, **litellm_params} # mutable-ok: AWS helpers require a plain dict
has_static_deployment_credentials: Final = all(
isinstance(litellm_params.get(key), str) and bool(litellm_params.get(key))
for key in ("aws_access_key_id", "aws_secret_access_key", "aws_region_name")
)
if has_static_deployment_credentials:
for key in _BEDROCK_AWS_AUTH_PARAMETER_KEYS:
if key not in litellm_params:
request_params.pop(key, None)
return request_params
# Lazy import cache to avoid circular imports and performance impact
_get_model_info = None
@ -138,13 +176,14 @@ def convert_bedrock_invoke_output_format_to_inline_schema(
request_body["messages"] = new_messages
def remove_custom_field_from_tools(request_body: dict) -> None:
def normalize_custom_field_on_tools(request_body: dict) -> None:
"""
Remove ``custom`` field from each tool in the request body.
Drop the ``custom`` field from each tool, first hoisting a boolean
``custom.defer_loading`` onto the top-level ``defer_loading`` flag that
Bedrock and Anthropic actually document, unless the tool already carries one.
Claude Code (v2.1.69+) sends ``custom: {defer_loading: true}`` on tool
definitions, which Anthropic's API accepts but Bedrock rejects with
``"Extra inputs are not permitted"``.
Claude Code (v2.1.69+) is reported to send ``custom: {defer_loading: true}`` on
tool definitions, which Bedrock rejects with ``"Extra inputs are not permitted"``.
Args:
request_body: The request dictionary to modify in-place.
@ -155,8 +194,14 @@ def remove_custom_field_from_tools(request_body: dict) -> None:
if not tools or not isinstance(tools, list):
return
for tool in tools:
if isinstance(tool, dict):
tool.pop("custom", None)
if not isinstance(tool, dict):
continue
custom: dict[str, object] | None = tool.pop("custom", None)
if not isinstance(custom, dict) or "defer_loading" in tool:
continue
deferred: object = custom.get("defer_loading")
if isinstance(deferred, bool):
tool["defer_loading"] = deferred
def normalize_json_schema_custom_types_to_object(schema: dict) -> None:

116
litellm/llms/bedrock/files/transformation.py
View file

@ -2,17 +2,18 @@ import base64
import json
import os
import time
from collections.abc import Iterable, Mapping, MutableMapping
from collections.abc import Iterable, Mapping, MutableMapping, Sequence
from functools import cache
from itertools import chain
from types import MappingProxyType
from typing import Any, Final
from typing import Any, Final, TypeAlias, TypedDict
from urllib.parse import unquote
import httpx
from httpx import Headers, Response
from openai.types.file_deleted import FileDeleted
from pydantic import BaseModel, ConfigDict, TypeAdapter
from typing_extensions import ReadOnly
from litellm._logging import verbose_logger
from litellm._uuid import uuid
@ -54,7 +55,7 @@ from litellm.types.utils import ExtractedFileData, LlmProviders, SpecialEnums
from litellm.utils import get_llm_provider
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import BedrockError, resolve_s3_encryption_key_id
from ..common_utils import BedrockError, merge_bedrock_aws_request_params, resolve_s3_encryption_key_id
# litellm_params key used to hand the SigV4-signed GET headers from
# `transform_file_content_request` to `validate_environment` (the only hook
@ -63,10 +64,39 @@ from ..common_utils import BedrockError, resolve_s3_encryption_key_id
S3_SIGNED_GET_HEADERS_PARAM: Final = "_s3_signed_get_headers"
def _frozen_mapping(items: Iterable[tuple[str, Any]]) -> Mapping[str, Any]:
def _frozen_mapping(items: Iterable[tuple[str, object]]) -> Mapping[str, object]:
return MappingProxyType(dict(items))
_EmbeddingBatchInput: TypeAlias = (
str | int | float | Sequence[str] | Sequence[int] | Sequence[Sequence[int]] | Mapping[str, object]
)
class _OpenAIBatchRecordBody(TypedDict, total=False):
model: ReadOnly[str]
prompt: ReadOnly[str | Sequence[str] | Sequence[int] | Sequence[Sequence[int]]]
input: ReadOnly[_EmbeddingBatchInput]
metadata: ReadOnly[Mapping[str, object]]
class _OpenAIBatchRecord(TypedDict, total=False):
custom_id: ReadOnly[str]
url: ReadOnly[str]
body: ReadOnly[_OpenAIBatchRecordBody]
class _BedrockBatchRecord(TypedDict):
recordId: ReadOnly[str]
modelInput: ReadOnly[Mapping[str, object]]
class _S3UploadResponse(TypedDict, total=False):
Key: ReadOnly[str]
Bucket: ReadOnly[str]
ContentLength: ReadOnly[int]
# JSONL batch records are untyped json, so the `/v1/responses` fields are
# validated into their concrete Responses API types before being handed to the
# Responses-to-Chat bridge. Both adapters drop keys the Responses API doesn't
@ -231,7 +261,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
def _get_s3_object_name_from_batch_jsonl(
self,
openai_jsonl_content: list[dict[str, Any]],
openai_jsonl_content: Sequence[_OpenAIBatchRecord],
) -> str:
"""
Gets a unique S3 object name for the Bedrock batch processing job
@ -285,6 +315,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
"""
Get the complete S3 URL for the file upload request
"""
request_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
bucket_name = litellm_params.get("s3_bucket_name") or os.getenv("AWS_S3_BUCKET_NAME")
if not bucket_name:
raise ValueError(
@ -293,7 +324,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
bucket_name, object_prefix = split_configured_cloud_bucket_name(bucket_name)
s3_region_name: Final = litellm_params.get("s3_region_name") or optional_params.get("s3_region_name")
aws_region_name: Final = s3_region_name or self._get_aws_region_name(optional_params, model)
aws_region_name: Final = s3_region_name or self._get_aws_region_name(request_params, model)
file_data: Final = data.get("file")
purpose: Final = data.get("purpose")
@ -309,7 +340,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
# S3 endpoint URL format
s3_endpoint_url: Final = (
optional_params.get("s3_endpoint_url") or f"https://s3.{aws_region_name}.amazonaws.com"
request_params.get("s3_endpoint_url") or f"https://s3.{aws_region_name}.amazonaws.com"
).rstrip("/")
return f"{s3_endpoint_url}/{bucket_name}/{encoded_object_name}"
@ -340,7 +371,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
OPENAI_RESPONSES_URL = "/v1/responses"
@staticmethod
def _classify_batch_record(openai_jsonl_record: Mapping[str, Any]) -> BedrockBatchRecordKind:
def _classify_batch_record(openai_jsonl_record: _OpenAIBatchRecord) -> BedrockBatchRecordKind:
"""
Decide which OpenAI endpoint shape an OpenAI batch JSONL line carries.
@ -483,7 +514,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
return value if isinstance(value, str) and value else None
@staticmethod
def _coerce_embedding_input_to_string(raw_input: Any, model: str = "") -> str:
def _coerce_embedding_input_to_string(raw_input: _EmbeddingBatchInput | None, model: str = "") -> str:
"""
Normalize an OpenAI /v1/embeddings `input` field into the single
string that Bedrock Titan v2 InvokeModel expects in `inputText`.
@ -540,8 +571,8 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
def _map_openai_embedding_to_bedrock_params(
self,
openai_request_body: dict[str, Any],
) -> dict[str, Any]:
openai_request_body: _OpenAIBatchRecordBody,
) -> dict[str, object]:
"""
Transform an OpenAI /v1/embeddings request body into the
Bedrock InvokeModel `modelInput` for embedding models that AWS
@ -587,7 +618,9 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
return dict(titan_config._transform_request(input=input_text, inference_params=inference_params))
@staticmethod
def _transform_text_completion_body_to_chat_body(openai_request_body: Mapping[str, Any]) -> Mapping[str, Any]:
def _transform_text_completion_body_to_chat_body(
openai_request_body: _OpenAIBatchRecordBody,
) -> Mapping[str, object]:
"""
Rewrite an OpenAI `/v1/completions` batch body as a Chat Completions body.
@ -609,7 +642,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
)
@staticmethod
def _transform_responses_body_to_chat_body(openai_request_body: Mapping[str, Any]) -> Mapping[str, Any]:
def _transform_responses_body_to_chat_body(openai_request_body: _OpenAIBatchRecordBody) -> Mapping[str, object]:
"""
Rewrite an OpenAI `/v1/responses` batch body as a Chat Completions body.
@ -630,23 +663,25 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
"Batch record for /v1/responses is missing required `input` field: "
f"model={openai_request_body.get('model', '')}"
)
chat_body: Final = LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request(
model=openai_request_body.get("model", ""),
input=_responses_input_adapter().validate_python(responses_input),
responses_api_request=_responses_request_adapter().validate_python(
_frozen_mapping(
(key, value) for key, value in openai_request_body.items() if key not in ("model", "input")
)
),
metadata=openai_request_body.get("metadata"),
chat_body: Final[Mapping[str, object]] = (
LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request(
model=openai_request_body.get("model", ""),
input=_responses_input_adapter().validate_python(responses_input),
responses_api_request=_responses_request_adapter().validate_python(
_frozen_mapping(
(key, value) for key, value in openai_request_body.items() if key not in ("model", "input")
)
),
metadata=openai_request_body.get("metadata"),
)
)
return _frozen_mapping((key, value) for key, value in chat_body.items() if key != "tools" or value)
@staticmethod
def _transform_batch_body_to_chat_body(
openai_request_body: Mapping[str, Any],
openai_request_body: _OpenAIBatchRecordBody,
record_kind: BedrockBatchRecordKind,
) -> Mapping[str, Any]:
) -> Mapping[str, object]:
"""
Normalize a non-embedding batch body to the Chat Completions shape the
per-provider Bedrock transformations expect.
@ -665,7 +700,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
self,
openai_request_body: Mapping[str, Any],
provider: str | None = None,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
Transform OpenAI request body to Bedrock-compatible modelInput
parameters using existing transformation logic.
@ -676,7 +711,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
"""
from litellm.types.utils import LlmProviders
_model: Final = openai_request_body.get("model", "")
_model: Final[str] = openai_request_body.get("model", "")
messages: Final = openai_request_body.get("messages", [])
optional_params: Final = {k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
@ -732,8 +767,8 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
}
def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
self, openai_jsonl_content: list[dict[str, Any]]
) -> list[dict[str, Any]]:
self, openai_jsonl_content: Sequence[_OpenAIBatchRecord]
) -> list[_BedrockBatchRecord]:
"""
Transforms OpenAI JSONL content to Bedrock batch format
@ -843,20 +878,23 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
)
# s3_region_name always wins for S3 operations (same priority as in
# get_complete_file_url above). Overwrite aws_region_name unconditionally
# so the SigV4 region matches the URL region, avoiding SignatureDoesNotMatch.
# get_complete_file_url above). Overwrite aws_region_name unconditionally,
# after the deployment-credential merge, so the SigV4 region matches the
# URL region, avoiding SignatureDoesNotMatch.
merged_params: Final = merge_bedrock_aws_request_params(litellm_params, optional_params)
s3_region_name: Final = litellm_params.get("s3_region_name") or optional_params.get("s3_region_name")
if s3_region_name:
optional_params = {**optional_params, "aws_region_name": s3_region_name}
request_params: Final = (
{**merged_params, "aws_region_name": s3_region_name} if s3_region_name else merged_params
)
# Sign the request and return a pre-signed request object
signed_headers, signed_body = self._sign_s3_request(
content=file_content,
api_base=api_base,
optional_params=optional_params,
optional_params=request_params,
s3_encryption_key_id=resolve_s3_encryption_key_id(
litellm_params=litellm_params,
optional_params=optional_params,
optional_params=request_params,
),
)
@ -1022,7 +1060,7 @@ class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
response_headers: Final = raw_response.headers
# Extract S3 object information from the response
# S3 PUT object returns ETag and other metadata in headers
content_length: Final = response_headers.get("Content-Length", "0")
content_length: Final[str] = response_headers.get("Content-Length", "0")
# Use the actual upload URL that was used for the S3 upload
upload_url: Final = litellm_params.get("upload_url")
@ -1220,7 +1258,9 @@ class BedrockJsonlFilesTransformation:
object_name: Final = self._get_s3_object_name(openai_jsonl_content=openai_jsonl_content)
return bedrock_jsonl_string, object_name
def _transform_openai_jsonl_content_to_bedrock_jsonl_content(self, openai_jsonl_content: list[dict[str, Any]]):
def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
self, openai_jsonl_content: Sequence[_OpenAIBatchRecord]
):
"""
Delegate to the main BedrockFilesConfig transformation method
"""
@ -1229,7 +1269,7 @@ class BedrockJsonlFilesTransformation:
def _get_s3_object_name(
self,
openai_jsonl_content: list[dict[str, Any]],
openai_jsonl_content: Sequence[_OpenAIBatchRecord],
) -> str:
"""
Gets a unique S3 object name for the Bedrock batch processing job
@ -1281,7 +1321,7 @@ class BedrockJsonlFilesTransformation:
return content
def transform_s3_bucket_response_to_openai_file_object(
self, create_file_data: CreateFileRequest, s3_upload_response: dict[str, Any]
self, create_file_data: CreateFileRequest, s3_upload_response: _S3UploadResponse
) -> OpenAIFileObject:
"""
Transforms S3 Bucket upload file response to OpenAI FileObject

41
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
View file

@ -33,9 +33,9 @@ from litellm.llms.bedrock.common_utils import (
get_anthropic_beta_from_headers,
is_claude_4_5_on_bedrock,
normalize_bedrock_opus_output_config_effort,
normalize_custom_field_on_tools,
normalize_tool_input_schema_types_for_bedrock_invoke,
pop_bedrock_invoke_output_config_format,
remove_custom_field_from_tools,
)
from litellm.types.llms.anthropic import (
ANTHROPIC_BETA_HEADER_VALUES,
@ -372,8 +372,9 @@ class AmazonAnthropicClaudeMessagesConfig(
"""
Check if the model supports tool search on Bedrock.
On Amazon Bedrock, server-side tool search is supported on Claude Opus 4.5
and Claude Sonnet 4.5 with the tool-search-tool-2025-10-19 beta header.
The model map's ``supports_tool_search`` flag is authoritative when
``model`` resolves to an entry that sets it; the name patterns below
cover ids the map cannot resolve (ARNs, unlisted regional variants).
Ref: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool
@ -383,9 +384,12 @@ class AmazonAnthropicClaudeMessagesConfig(
Returns:
True if the model supports tool search on Bedrock
"""
catalog: Final = AnthropicModelInfo._get_provider_resolved_capability(model, "supports_tool_search", "bedrock")
if catalog is not None:
return catalog
model_lower: Final = model.lower()
# Supported models for tool search on Bedrock
supported_patterns: Final = [
# Opus 4.5
"opus-4.5",
@ -407,10 +411,16 @@ class AmazonAnthropicClaudeMessagesConfig(
"sonnet_4.6",
"sonnet-4-6",
"sonnet_4_6",
# NOTE: Opus 4.7 on Bedrock does not support server-side tool search
# as of launch (2026-04-16). Bedrock rejects the tool type with:
# "tool type 'tool_search_tool_..._20251119' is not supported for this model".
# Re-add the opus-4.7 patterns here once AWS announces support.
# Opus 4.7
"opus-4.7",
"opus_4.7",
"opus-4-7",
"opus_4_7",
# Haiku 4.5
"haiku-4.5",
"haiku_4.5",
"haiku-4-5",
"haiku_4_5",
]
return any(pattern in model_lower for pattern in supported_patterns)
@ -426,11 +436,10 @@ class AmazonAnthropicClaudeMessagesConfig(
"""
Adjust tool search beta header for Bedrock.
Bedrock requires a different beta header for tool search on Opus 4 models
when tool search is used without programmatic tool calling or input examples.
Note: On Amazon Bedrock, server-side tool search is only supported on Claude Opus 4
with the `tool-search-tool-2025-10-19` beta header.
Bedrock requires a different beta header for tool search than the
Anthropic API when tool search is used without programmatic tool
calling or input examples: `tool-search-tool-2025-10-19`, and only on
the models listed in `_supports_tool_search_on_bedrock`.
Ref: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool
@ -740,11 +749,9 @@ class AmazonAnthropicClaudeMessagesConfig(
model,
)
# 5b. Remove `custom` field from tools (Bedrock doesn't support it)
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
# 5b. Hoist `custom.defer_loading` then drop `custom` (Bedrock doesn't support it)
# Ref: https://github.com/BerriAI/litellm/issues/22847
remove_custom_field_from_tools(anthropic_messages_request)
normalize_custom_field_on_tools(anthropic_messages_request)
normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_messages_request)
ensure_bedrock_anthropic_messages_tool_names(anthropic_messages_request)

4
litellm/llms/codestral/completion/handler.py
View file

@ -195,7 +195,7 @@ class CodestralTextCompletion:
print_verbose: Callable,
encoding,
api_key: str,
logging_obj,
logging_obj: LiteLLMLogging,
optional_params: dict,
timeout: float | httpx.Timeout,
acompletion=None,
@ -383,7 +383,7 @@ class CodestralTextCompletion:
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLogging,
data: dict,
timeout: float | httpx.Timeout,
optional_params=None,

2
litellm/llms/custom_httpx/aiohttp_handler.py
View file

@ -221,7 +221,7 @@ class BaseLLMAIOHTTPHandler:
timeout=timeout,
stream=stream,
files=files,
content=content,
content=content if content is not None else None,
params=params,
)
except httpx.HTTPStatusError as e:

61
litellm/llms/custom_httpx/http_handler.py
View file

@ -7,9 +7,9 @@ import ssl
import sys
import threading
import time
from collections.abc import Callable, Mapping
from collections.abc import AsyncIterable, Callable, Iterable, Mapping
from http.cookiejar import CookieJar, DefaultCookiePolicy
from typing import TYPE_CHECKING, Any, Final, Optional
from typing import TYPE_CHECKING, Any, Final, Optional, TypeAlias, TypedDict
import certifi
import httpx
@ -62,8 +62,23 @@ except Exception:
# https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.TCPConnector
_AIOHTTP_SUPPORTS_SOCKET_FACTORY: Final = "socket_factory" in inspect.signature(TCPConnector.__init__).parameters
_AddrInfo: TypeAlias = tuple[int | socket.AddressFamily, int | socket.SocketKind, int, str, tuple[object, ...]]
def _build_aiohttp_keepalive_socket_factory() -> Callable[[tuple[Any, ...]], socket.socket] | None:
_RequestContent: TypeAlias = str | bytes | Iterable[bytes] | AsyncIterable[bytes]
class _TCPConnectorKwargs(TypedDict, total=False):
local_addr: tuple[str, int] | None
ssl: "ssl.SSLContext | bool"
keepalive_timeout: float
ttl_dns_cache: int
enable_cleanup_closed: bool
limit: int
limit_per_host: int
socket_factory: Callable[[_AddrInfo], socket.socket]
def _build_aiohttp_keepalive_socket_factory() -> Callable[[_AddrInfo], socket.socket] | None:
"""
Build a socket_factory that enables SO_KEEPALIVE on aiohttp TCP sockets.
@ -78,7 +93,7 @@ def _build_aiohttp_keepalive_socket_factory() -> Callable[[tuple[Any, ...]], soc
if not AIOHTTP_SO_KEEPALIVE or not _AIOHTTP_SUPPORTS_SOCKET_FACTORY:
return None
def factory(addr_info: tuple[Any, ...]) -> socket.socket:
def factory(addr_info: _AddrInfo) -> socket.socket:
family, type_, proto = addr_info[0], addr_info[1], addr_info[2]
sock: Final = socket.socket(family=family, type=type_, proto=proto)
sock.setblocking(False)
@ -163,8 +178,8 @@ _STREAMING_ERROR_BODY_READ_EXECUTOR: Final = concurrent.futures.ThreadPoolExecut
def _prepare_request_data_and_content(
data: dict | str | bytes | None = None,
content: Any = None,
) -> tuple[dict | Mapping | None, Any]:
content: _RequestContent | None = None,
) -> tuple[dict | Mapping | None, _RequestContent | None]:
"""
Helper function to route data/content parameters correctly for httpx requests
@ -528,7 +543,7 @@ class AsyncHTTPHandler:
def __init__(
self,
timeout: float | httpx.Timeout | None = None,
event_hooks: Mapping[str, list[Callable[..., Any]]] | None = None,
event_hooks: Mapping[str, list[Callable[..., object]]] | None = None,
concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
client_alias: str | None = None, # name for client in logs
ssl_verify: VerifyTypes | None = None,
@ -566,7 +581,7 @@ class AsyncHTTPHandler:
def create_client(
self,
timeout: float | httpx.Timeout | None,
event_hooks: Mapping[str, list[Callable[..., Any]]] | None,
event_hooks: Mapping[str, list[Callable[..., object]]] | None,
ssl_verify: VerifyTypes | None = None,
shared_session: Optional["ClientSession"] = None,
) -> httpx.AsyncClient:
@ -648,7 +663,7 @@ class AsyncHTTPHandler:
stream: bool = False,
logging_obj: LiteLLMLoggingObject | None = None,
files: RequestFiles | None = None,
content: Any = None,
content: _RequestContent | None = None,
):
start_time: Final = time.time()
try:
@ -691,7 +706,7 @@ class AsyncHTTPHandler:
end_time: Final = time.time()
time_delta: Final = round(end_time - start_time, 3)
headers = {}
error_response: Final = getattr(e, "response", None)
error_response: Final[httpx.Response | None] = getattr(e, "response", None)
if error_response is not None:
for key, value in error_response.headers.items():
headers[f"response_headers-{key}"] = value
@ -716,7 +731,7 @@ class AsyncHTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
if timeout is None:
@ -755,7 +770,7 @@ class AsyncHTTPHandler:
await new_client.aclose()
except httpx.TimeoutException as e:
headers = {}
error_response: Final = getattr(e, "response", None)
error_response: Final[httpx.Response | None] = getattr(e, "response", None)
if error_response is not None:
for key, value in error_response.headers.items():
headers[f"response_headers-{key}"] = value
@ -780,7 +795,7 @@ class AsyncHTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
if timeout is None:
@ -819,7 +834,7 @@ class AsyncHTTPHandler:
await new_client.aclose()
except httpx.TimeoutException as e:
headers = {}
error_response: Final = getattr(e, "response", None)
error_response: Final[httpx.Response | None] = getattr(e, "response", None)
if error_response is not None:
for key, value in error_response.headers.items():
headers[f"response_headers-{key}"] = value
@ -844,7 +859,7 @@ class AsyncHTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
if timeout is None:
@ -895,7 +910,7 @@ class AsyncHTTPHandler:
params: dict | None = None,
headers: dict | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
"""
Making POST request for a single connection client.
@ -993,7 +1008,7 @@ class AsyncHTTPHandler:
def _get_ssl_connector_kwargs(
ssl_verify: bool | None = None,
ssl_context: ssl.SSLContext | None = None,
) -> dict[str, Any]:
) -> _TCPConnectorKwargs:
"""
Helper method to get SSL connector initialization arguments for aiohttp TCPConnector.
@ -1004,7 +1019,7 @@ class AsyncHTTPHandler:
Returns:
Dict with appropriate SSL configuration for TCPConnector
"""
connector_kwargs: Final[dict[str, Any]] = {
connector_kwargs: Final[_TCPConnectorKwargs] = {
"local_addr": ("0.0.0.0", 0) if litellm.force_ipv4 else None,
}
@ -1054,7 +1069,7 @@ class AsyncHTTPHandler:
verbose_logger.debug("Creating AiohttpTransport...")
transport_connector_kwargs: Final = {
transport_connector_kwargs: Final[_TCPConnectorKwargs] = {
"keepalive_timeout": AIOHTTP_KEEPALIVE_TIMEOUT,
"ttl_dns_cache": AIOHTTP_TTL_DNS_CACHE,
**connector_kwargs,
@ -1212,7 +1227,7 @@ class HTTPHandler:
stream: bool = False,
timeout: float | httpx.Timeout | None = None,
files: dict | RequestFiles | None = None,
content: Any = None,
content: _RequestContent | None = None,
logging_obj: LiteLLMLoggingObject | None = None,
):
try:
@ -1265,7 +1280,7 @@ class HTTPHandler:
headers: dict | None = None,
stream: bool = False,
timeout: float | httpx.Timeout | None = None,
content: Any = None,
content: _RequestContent | None = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
@ -1315,7 +1330,7 @@ class HTTPHandler:
headers: dict | None = None,
stream: bool = False,
timeout: float | httpx.Timeout | None = None,
content: Any = None,
content: _RequestContent | None = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
@ -1364,7 +1379,7 @@ class HTTPHandler:
headers: dict | None = None,
timeout: float | httpx.Timeout | None = None,
stream: bool = False,
content: Any = None,
content: _RequestContent | None = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)

64
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -5,7 +5,8 @@ import ssl
from collections.abc import AsyncIterator, Coroutine, Iterator, Mapping
from contextlib import asynccontextmanager
from functools import lru_cache
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypeVar, Union, cast, get_type_hints
from types import ModuleType
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, TypedDict, TypeVar, Union, cast, get_type_hints
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
import httpx
@ -148,6 +149,7 @@ from .http_handler import get_shared_realtime_ssl_context
if TYPE_CHECKING:
from aiohttp import ClientSession
from websockets.asyncio.client import ClientConnection
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@ -176,6 +178,19 @@ else:
_ResponseT = TypeVar("_ResponseT")
class _DeleteRequestKwargs(TypedDict, total=False):
url: str
headers: dict[str, str]
timeout: float | httpx.Timeout | None
json: dict[str, object]
class _MediaUploadKwargs(TypedDict, total=False):
headers: dict[str, str]
content: Iterator[bytes] | AsyncIterator[bytes]
timeout: float | httpx.Timeout
def _google_genai_streaming_hidden_params(
*,
api_base: str,
@ -1413,7 +1428,7 @@ class BaseLLMHTTPHandler:
headers: dict[str, object] | None,
provider_config: BaseOCRConfig,
litellm_params: dict,
) -> tuple[dict[str, Any], str, dict[str, Any], None]:
) -> tuple[dict[str, object], str, dict[str, object], None]:
"""
Shared logic for preparing OCR requests.
Returns: (headers, complete_url, data, files)
@ -1479,7 +1494,7 @@ class BaseLLMHTTPHandler:
headers: dict[str, object] | None,
provider_config: BaseOCRConfig,
litellm_params: dict,
) -> tuple[dict[str, Any], str, dict[str, Any], None]:
) -> tuple[dict[str, object], str, dict[str, object], None]:
"""
Async version of _prepare_ocr_request for providers that need async transforms.
Returns: (headers, complete_url, data, files)
@ -2361,14 +2376,14 @@ class BaseLLMHTTPHandler:
model: str,
input: str | ResponseInputParam,
custom_llm_provider: str,
response_api_optional_request_params: dict[str, Any],
response_api_optional_request_params: dict[str, object],
litellm_params: GenericLiteLLMParams,
logging_obj: LiteLLMLoggingObj,
) -> tuple[
str,
str | ResponseInputParam,
str,
dict[str, Any],
dict[str, object],
GenericLiteLLMParams,
]:
if not _has_pre_call_deployment_hook(logging_obj):
@ -2894,7 +2909,7 @@ class BaseLLMHTTPHandler:
},
)
delete_kwargs: Final[dict[str, Any]] = {
delete_kwargs: Final[_DeleteRequestKwargs] = {
"url": url,
"headers": headers,
"timeout": timeout,
@ -2984,7 +2999,7 @@ class BaseLLMHTTPHandler:
},
)
delete_kwargs: Final[dict[str, Any]] = {
delete_kwargs: Final[_DeleteRequestKwargs] = {
"url": url,
"headers": headers,
"timeout": timeout,
@ -3725,7 +3740,7 @@ class BaseLLMHTTPHandler:
timeout: float | httpx.Timeout | None,
) -> httpx.Response:
headers: Final = {**base_headers, "Content-Type": content_type}
kwargs: Final[dict[str, Any]] = {
kwargs: Final[_MediaUploadKwargs] = {
"headers": headers,
"content": self._iter_in_blocks(body_stream.iter_bytes(), self._MEDIA_UPLOAD_BLOCK_SIZE),
}
@ -3762,7 +3777,7 @@ class BaseLLMHTTPHandler:
break
yield cast(bytes, block)
kwargs: Final[dict[str, Any]] = {"headers": headers, "content": _abody()}
kwargs: Final[_MediaUploadKwargs] = {"headers": headers, "content": _abody()}
if timeout is not None:
kwargs["timeout"] = timeout
resp: Final = await client.client.post(url, **kwargs)
@ -5242,7 +5257,7 @@ class BaseLLMHTTPHandler:
def _wrap_responses_response_as_fake_stream(
self,
result: Any,
result: ResponsesAPIResponse,
model: str,
responses_api_provider_config: BaseResponsesAPIConfig,
logging_obj: "LiteLLMLoggingObj",
@ -5365,7 +5380,7 @@ class BaseLLMHTTPHandler:
async def _call_agentic_completion_hooks(
self,
response: Any,
response: object,
model: str,
messages: list[dict],
anthropic_messages_provider_config: "BaseAnthropicMessagesConfig",
@ -5536,7 +5551,7 @@ class BaseLLMHTTPHandler:
async def _call_agentic_chat_completion_hooks(
self,
response: Any,
response: ModelResponse,
model: str,
messages: list[dict],
optional_params: dict,
@ -5760,14 +5775,14 @@ class BaseLLMHTTPHandler:
@staticmethod
async def _open_realtime_backend_ws(
websockets_module: Any,
websockets_module: ModuleType,
url: str,
headers: dict,
ssl_context: Any,
ssl_context: bool | str | ssl.SSLContext,
*,
open_timeout: float = 8.0,
max_attempts: int = 3,
) -> Any:
) -> "ClientConnection":
"""Open the backend realtime websocket, retrying a hung open handshake.
The upstream Live handshake (e.g. Gemini Live) intermittently hangs on
@ -5826,7 +5841,6 @@ class BaseLLMHTTPHandler:
query_params: RealtimeQueryParams | None = None,
):
import websockets
from websockets.asyncio.client import ClientConnection
url: Final = provider_config.get_complete_url(api_base, model, api_key)
headers = provider_config.validate_environment(
@ -5844,12 +5858,12 @@ class BaseLLMHTTPHandler:
ssl_context.verify_mode = ssl.CERT_NONE
backend_ws: Final = await self._open_realtime_backend_ws(websockets, url, headers, ssl_context)
async with backend_ws:
_request_data: Final[dict[str, Any]] = {}
_request_data: Final[dict[str, object]] = {}
if litellm_metadata:
_request_data["litellm_metadata"] = litellm_metadata
realtime_streaming: Final = RealTimeStreaming(
websocket,
cast(ClientConnection, backend_ws),
backend_ws,
logging_obj,
provider_config,
model,
@ -6008,7 +6022,7 @@ class BaseLLMHTTPHandler:
)
else:
url = provider_config.get_complete_url(api_base=api_base, model=model or "", api_version=api_version)
headers: dict[str, Any] = provider_config.validate_environment(
headers: dict[str, object] = provider_config.validate_environment(
headers={}, model=model or "", api_key=api_key
)
else:
@ -6079,7 +6093,7 @@ class BaseLLMHTTPHandler:
if provider_config is not None:
url = provider_config.get_realtime_calls_url(api_base=api_base, model=model or "", api_version=api_version)
headers: dict[str, Any] = provider_config.get_realtime_calls_headers(ephemeral_key=openai_ephemeral_key)
headers: dict[str, object] = provider_config.get_realtime_calls_headers(ephemeral_key=openai_ephemeral_key)
else:
url = f"{api_base.rstrip('/')}/v1/realtime/calls"
headers = {
@ -6247,7 +6261,7 @@ class BaseLLMHTTPHandler:
yield backend
async with _backend_connection() as backend_ws:
_request_data: Final[dict[str, Any]] = {}
_request_data: Final[dict[str, object]] = {}
if litellm_metadata:
_request_data["litellm_metadata"] = litellm_metadata
@ -9444,7 +9458,7 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
extra_body=extra_body,
)
all_optional_params: Final[dict[str, Any]] = dict(litellm_params)
all_optional_params: Final[dict[str, object]] = dict(litellm_params)
all_optional_params.update(vector_store_search_optional_params or {})
headers, signed_json_body = vector_store_provider_config.sign_request(
headers=headers,
@ -9540,7 +9554,7 @@ class BaseLLMHTTPHandler:
extra_body=extra_body,
)
all_optional_params: Final[dict[str, Any]] = dict(litellm_params)
all_optional_params: Final[dict[str, object]] = dict(litellm_params)
all_optional_params.update(vector_store_search_optional_params or {})
headers, signed_json_body = vector_store_provider_config.sign_request(
@ -9860,7 +9874,7 @@ class BaseLLMHTTPHandler:
url: Final = api_base
params: Final[dict[str, Any]] = {}
params: Final[dict[str, object]] = {}
if after is not None:
params["after"] = after
if before is not None:
@ -9938,7 +9952,7 @@ class BaseLLMHTTPHandler:
url: Final = api_base
params: Final[dict[str, Any]] = {}
params: Final[dict[str, object]] = {}
if after is not None:
params["after"] = after
if before is not None:

2
litellm/llms/github_copilot/chat/transformation.py
View file

@ -277,7 +277,7 @@ class GithubCopilotConfig(OpenAIConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> "ModelResponse":

120
litellm/llms/nvidia_nim/rerank/ranking_transformation.py
View file

@ -8,13 +8,23 @@ Reference: https://build.nvidia.com/nvidia/llama-3_2-nv-rerankqa-1b-v2/deploy
from typing import Final
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.nvidia_nim.rerank.transformation import NvidiaNimRerankConfig
from litellm.types.rerank import RerankResponse
class NvidiaNimRankingConfig(NvidiaNimRerankConfig):
"""
Configuration for NVIDIA NIM models that use the /v1/ranking endpoint.
The native /v1/ranking request schema accepts only 'model', 'query',
'passages', and 'truncate' -- requests containing 'top_k' are rejected
with a 400 validation error. Cohere-compatible 'top_n' is therefore
applied client-side by truncating the converted response instead of
being forwarded to the endpoint.
Example:
curl -X "POST" 'https://ai.api.nvidia.com/v1/ranking' \
-H 'Accept: application/json' \
@ -27,6 +37,16 @@ class NvidiaNimRankingConfig(NvidiaNimRerankConfig):
}'
"""
SUPPORTED_PASSAGE_FIELDS: tuple[str, ...] = ("text", "image")
def __init__(self) -> None:
super().__init__()
# top_n captured in transform_rerank_request and applied in
# transform_rerank_response. The provider config is instantiated
# per-request (see ProviderConfigManager.get_provider_rerank_config),
# so this does not leak across requests.
self._client_side_top_n: int | None = None
def _get_clean_model_name(self, model: str) -> str:
"""Strip 'nvidia_nim/' and 'ranking/' prefixes from model name."""
# First strip nvidia_nim/ prefix if present
@ -58,6 +78,47 @@ class NvidiaNimRankingConfig(NvidiaNimRerankConfig):
return f"{api_base}/v1/ranking"
def map_cohere_rerank_params(
self,
non_default_params: dict | None, # mutable-ok: matches BaseRerankConfig's request contract
model: str,
drop_params: bool,
query: str,
documents: list[str | dict[str, object]], # mutable-ok: matches BaseRerankConfig's document contract
custom_llm_provider: str | None = None,
top_n: int | None = None,
rank_fields: list[str] | None = None, # mutable-ok: matches BaseRerankConfig's field contract
return_documents: bool | None = True,
max_chunks_per_doc: int | None = None,
max_tokens_per_doc: int | None = None,
instruction: str | None = None,
) -> dict: # mutable-ok: LiteLLM provider transforms return mutable request dictionaries
"""
Keep Cohere's top_n as-is instead of mapping it to top_k.
The native /v1/ranking endpoint rejects top_k, so top_n is applied
client-side after the response is converted.
"""
optional_params: Final = super().map_cohere_rerank_params(
non_default_params=non_default_params,
model=model,
drop_params=drop_params,
query=query,
documents=documents,
custom_llm_provider=custom_llm_provider,
top_n=None, # do not map top_n -> top_k for /v1/ranking
rank_fields=rank_fields,
return_documents=return_documents,
max_chunks_per_doc=max_chunks_per_doc,
max_tokens_per_doc=max_tokens_per_doc,
instruction=instruction,
)
# /v1/ranking rejects top_k even when passed as a provider-specific param
optional_params.pop("top_k", None)
if top_n is not None:
optional_params["top_n"] = top_n
return optional_params
def transform_rerank_request(
self,
model: str,
@ -67,11 +128,66 @@ class NvidiaNimRankingConfig(NvidiaNimRerankConfig):
) -> dict:
"""
Transform request, using clean model name without 'ranking/' prefix.
top_n / top_k are stripped from the outgoing request: the native
/v1/ranking endpoint accepts only model, query, passages, and
truncate. top_n is stashed and applied client-side in
transform_rerank_response.
"""
top_n: Final = optional_rerank_params.get("top_n")
if top_n is not None:
if isinstance(top_n, bool) or not isinstance(top_n, int) or top_n < 1:
raise ValueError(f"top_n must be a positive integer, got: {top_n!r}")
self._client_side_top_n = top_n
clean_model: Final = self._get_clean_model_name(model)
filtered_params: Final = { # mutable-ok: the base transformer requires a mutable request dictionary
k: v for k, v in optional_rerank_params.items() if k not in ("top_n", "top_k")
}
return super().transform_rerank_request(
model=clean_model,
optional_rerank_params=optional_rerank_params,
optional_rerank_params=filtered_params,
headers=headers,
litellm_params=litellm_params,
)
def transform_rerank_response(
self,
model: str,
raw_response: httpx.Response,
model_response: RerankResponse,
logging_obj: LiteLLMLoggingObj,
api_key: str | None = None,
request_data: dict | None = None, # mutable-ok: matches BaseRerankConfig's response contract
optional_params: dict | None = None, # mutable-ok: matches BaseRerankConfig's response contract
litellm_params: dict | None = None, # mutable-ok: matches BaseRerankConfig's response contract
) -> RerankResponse:
"""
Convert the native ranking response, then apply top_n client-side.
/v1/ranking returns rankings sorted by relevance, but sort before
truncating in case a server returns them unsorted.
"""
resolved_request_data: Final = request_data or {} # mutable-ok: the base transformer requires a dictionary
resolved_optional_params: Final = optional_params or {} # mutable-ok: response options are keyed lookups
resolved_litellm_params: Final = litellm_params or {} # mutable-ok: the base transformer requires a dictionary
response: Final = super().transform_rerank_response(
model=model,
raw_response=raw_response,
model_response=model_response,
logging_obj=logging_obj,
api_key=api_key,
request_data=resolved_request_data,
optional_params=resolved_optional_params,
litellm_params=resolved_litellm_params,
)
top_n: Final = resolved_optional_params.get("top_n") or self._client_side_top_n
if top_n is not None and response.results is not None and len(response.results) > top_n:
response.results = sorted(
response.results,
key=lambda result: result["relevance_score"],
reverse=True,
)[:top_n]
return response

28
litellm/llms/nvidia_nim/rerank/transformation.py
View file

@ -21,8 +21,9 @@ class NvidiaNimQueryObject(TypedDict):
text: Required[str]
class NvidiaNimPassageObject(TypedDict):
text: Required[str]
class NvidiaNimPassageObject(TypedDict, total=False):
text: str
image: str
class NvidiaNimRerankRequest(TypedDict, total=False):
@ -53,6 +54,10 @@ class NvidiaNimRerankConfig(BaseRerankConfig):
DEFAULT_NIM_RERANK_API_BASE = "https://ai.api.nvidia.com"
# The legacy retrieval rerank route accepts text passages only. The native
# ranking subclass expands this tuple for VL models that accept images.
SUPPORTED_PASSAGE_FIELDS: tuple[str, ...] = ("text",)
def __init__(self) -> None:
pass
@ -206,11 +211,17 @@ class NvidiaNimRerankConfig(BaseRerankConfig):
if isinstance(doc, str):
passages.append({"text": doc})
elif isinstance(doc, dict):
# If document is already a dict, check if it has 'text' field
if "text" in doc:
passages.append({"text": doc["text"]})
# Preserve only the structured passage fields supported by the
# selected rerank route.
supported_fields: NvidiaNimPassageObject = {} # mutable-ok: assembling a request TypedDict
if "text" in self.SUPPORTED_PASSAGE_FIELDS and "text" in doc:
supported_fields["text"] = doc["text"]
if "image" in self.SUPPORTED_PASSAGE_FIELDS and "image" in doc:
supported_fields["image"] = doc["image"]
if supported_fields:
passages.append(supported_fields)
else:
# Otherwise, stringify the dict
# No supported fields - stringify the dict
import json
passages.append({"text": json.dumps(doc)})
@ -304,9 +315,10 @@ class NvidiaNimRerankConfig(BaseRerankConfig):
"relevance_score": ranking["logit"],
}
# Include document if it was in the original request
# Include document if it was in the original request.
# Image-only passages carry no 'text' field, so guard the lookup.
index: int = ranking["index"]
if index < len(original_passages):
if index < len(original_passages) and "text" in original_passages[index]:
result_item["document"] = {"text": original_passages[index]["text"]}
results.append(result_item)

23
litellm/llms/oci/chat/transformation.py
View file

@ -10,7 +10,7 @@ implement the LiteLLM BaseConfig interface. Heavy-lifting lives in:
"""
import json
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Callable, Iterator
from typing import TYPE_CHECKING, Any, Final
import httpx
@ -713,8 +713,25 @@ class OCIChatConfig(BaseConfig):
class OCIStreamWrapper(CustomStreamWrapper):
"""Custom stream wrapper that dispatches OCI SSE chunks to the correct handler."""
def __init__(self, **kwargs: Any):
super().__init__(**kwargs)
def __init__(
self,
completion_stream: object,
model: str,
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: str | None = None,
stream_options: object = None,
make_call: Callable[..., object] | None = None,
_response_headers: dict[str, object] | None = None,
) -> None:
super().__init__(
completion_stream=completion_stream,
model=model,
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
stream_options=stream_options,
make_call=make_call,
_response_headers=_response_headers,
)
# Tracks whether any prior Cohere chunk in this stream has emitted
# tool calls. The Cohere handler uses this to decide whether the
# terminal consolidation chunk's tool calls are duplicates (suppress)

4
litellm/llms/openai/completion/handler.py
View file

@ -217,7 +217,7 @@ class OpenAITextCompletion(BaseLLM):
def streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_key: str,
data: dict,
headers: dict,
@ -274,7 +274,7 @@ class OpenAITextCompletion(BaseLLM):
async def async_streaming(
self,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_key: str,
data: dict,
headers: dict,

72
litellm/llms/openai/openai.py
View file

@ -1,6 +1,6 @@
import time
import types
from collections.abc import AsyncIterator, Callable, Coroutine, Iterable, Iterator
from collections.abc import AsyncIterator, Callable, Coroutine, Iterable, Iterator, Mapping
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, cast
from urllib.parse import urlparse
@ -61,16 +61,17 @@ class MistralEmbeddingConfig:
def __init__(
self,
) -> None:
locals_: Final = locals().copy()
locals_: Final[Mapping[str, object]] = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@classmethod
def get_config(cls):
config_attrs: Final[Mapping[str, object]] = cls.__dict__
return {
k: v
for k, v in cls.__dict__.items()
for k, v in config_attrs.items()
if not k.startswith("__")
and not isinstance(
v,
@ -153,7 +154,7 @@ class OpenAIConfig(BaseConfig):
top_p: int | None = None,
response_format: dict | None = None,
) -> None:
locals_: Final = locals().copy()
locals_: Final[Mapping[str, object]] = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@ -261,7 +262,7 @@ class OpenAIConfig(BaseConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> ModelResponse:
@ -299,7 +300,7 @@ class OpenAIConfig(BaseConfig):
streaming_response: Iterator[str] | AsyncIterator[str] | ModelResponse,
sync_stream: bool,
json_mode: bool | None = False,
) -> Any:
) -> "OpenAIChatCompletionResponseIterator":
return OpenAIChatCompletionResponseIterator(
streaming_response=streaming_response,
sync_stream=sync_stream,
@ -478,14 +479,14 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
async def _call_agentic_completion_hooks_openai(
self,
response: Any,
response: object,
model: str,
messages: list[dict],
optional_params: dict,
logging_obj: LiteLLMLoggingObj,
stream: bool,
litellm_params: dict,
) -> Any | None:
) -> object | None:
"""
Call agentic completion hooks for all custom loggers (OpenAI Chat Completions API).
@ -536,7 +537,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
kwargs_with_provider["custom_llm_provider"] = custom_llm_provider
# For OpenAI Chat Completions, use the chat completion agentic loop method
agentic_response = await callback.async_run_chat_completion_agentic_loop(
agentic_response: object = await callback.async_run_chat_completion_agentic_loop(
tools=tool_calls,
model=model,
messages=messages,
@ -580,7 +581,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
timeout: float | httpx.Timeout,
optional_params: dict,
litellm_params: dict,
logging_obj: Any,
logging_obj: LiteLLMLoggingObj,
model: str | None = None,
messages: list | None = None,
print_verbose: Callable | None = None,
@ -1590,7 +1591,7 @@ class OpenAIFilesAPI(BaseLLM):
client: OpenAI | AsyncOpenAI | None = None,
_is_async: bool = False,
) -> OpenAI | AsyncOpenAI | None:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
openai_client: OpenAI | AsyncOpenAI | None = None
if client is None:
data: Final = {}
@ -1628,7 +1629,7 @@ class OpenAIFilesAPI(BaseLLM):
max_retries: int | None,
organization: str | None,
client: OpenAI | AsyncOpenAI | None = None,
) -> OpenAIFileObject | Coroutine[Any, Any, OpenAIFileObject]:
) -> OpenAIFileObject | Coroutine[None, None, OpenAIFileObject]:
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
@ -1670,7 +1671,7 @@ class OpenAIFilesAPI(BaseLLM):
max_retries: int | None,
organization: str | None,
client: OpenAI | AsyncOpenAI | None = None,
) -> HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent]:
) -> HttpxBinaryResponseContent | Coroutine[None, None, HttpxBinaryResponseContent]:
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
@ -1948,7 +1949,7 @@ class OpenAIBatchesAPI(BaseLLM):
client: OpenAI | AsyncOpenAI | None = None,
_is_async: bool = False,
) -> OpenAI | AsyncOpenAI | None:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
openai_client: OpenAI | AsyncOpenAI | None = None
if client is None:
data: Final = {}
@ -1986,7 +1987,7 @@ class OpenAIBatchesAPI(BaseLLM):
max_retries: int | None,
organization: str | None,
client: OpenAI | AsyncOpenAI | None = None,
) -> LiteLLMBatch | Coroutine[Any, Any, LiteLLMBatch]:
) -> LiteLLMBatch | Coroutine[None, None, LiteLLMBatch]:
openai_client: Final[OpenAI | AsyncOpenAI | None] = self.get_openai_client(
api_key=api_key,
api_base=api_base,
@ -2160,7 +2161,7 @@ class OpenAIAssistantsAPI(BaseLLM):
organization: str | None,
client: OpenAI | None = None,
) -> OpenAI:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
if client is None:
data: Final = {}
for k, v in received_args.items():
@ -2185,7 +2186,7 @@ class OpenAIAssistantsAPI(BaseLLM):
organization: str | None,
client: AsyncOpenAI | None = None,
) -> AsyncOpenAI:
received_args: Final = locals()
received_args: Final[Mapping[str, object]] = locals()
if client is None:
data: Final = {}
for k, v in received_args.items():
@ -2848,7 +2849,7 @@ class OpenAIAssistantsAPI(BaseLLM):
assistant_id: str,
additional_instructions: str | None,
instructions: str | None,
metadata: dict | None,
metadata: dict[str, str] | None,
model: str | None,
stream: bool | None,
tools: Iterable[AssistantToolParam] | None,
@ -2912,23 +2913,32 @@ class OpenAIAssistantsAPI(BaseLLM):
assistant_id: str,
additional_instructions: str | None,
instructions: str | None,
metadata: dict | None,
metadata: dict[str, str] | None,
model: str | None,
tools: Iterable[AssistantToolParam] | None,
event_handler: AssistantEventHandler | None,
) -> AssistantStreamManager[AssistantEventHandler]:
data: Final[dict[str, Any]] = {
"thread_id": thread_id,
"assistant_id": assistant_id,
"additional_instructions": additional_instructions,
"instructions": instructions,
"metadata": metadata,
"model": model,
"tools": tools,
}
runs_stream: Final = client.beta.threads.runs.stream
if event_handler is not None:
data["event_handler"] = event_handler
return client.beta.threads.runs.stream(**data)
return runs_stream(
thread_id=thread_id,
assistant_id=assistant_id,
additional_instructions=additional_instructions,
instructions=instructions,
metadata=metadata,
model=model,
tools=tools,
event_handler=event_handler,
)
return runs_stream(
thread_id=thread_id,
assistant_id=assistant_id,
additional_instructions=additional_instructions,
instructions=instructions,
metadata=metadata,
model=model,
tools=tools,
)
# fmt: off
@ -2984,7 +2994,7 @@ class OpenAIAssistantsAPI(BaseLLM):
assistant_id: str,
additional_instructions: str | None,
instructions: str | None,
metadata: dict | None,
metadata: dict[str, str] | None,
model: str | None,
stream: bool | None,
tools: Iterable[AssistantToolParam] | None,

10
litellm/llms/openai/responses/transformation.py
View file

@ -353,6 +353,16 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
)
return event_pydantic_model.model_construct(**parsed_chunk)
@staticmethod
def parse_terminal_response_from_stream_chunks(all_chunks: list[str]) -> ResponsesAPIResponse | None:
for chunk_str in reversed(all_chunks):
for event_model in (ResponseCompletedEvent, ResponseIncompleteEvent, ResponseFailedEvent):
try:
return event_model.model_validate_json(chunk_str.removeprefix("data: ")).response
except ValueError:
continue
return None
@staticmethod
def get_event_model_class(event_type: str) -> Any:
"""

5
litellm/llms/openai_like/chat/handler.py
View file

@ -12,6 +12,7 @@ import httpx
import litellm
from litellm import LlmProviders
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.bedrock.chat.invoke_handler import MockResponseIterator
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.databricks.streaming_utils import ModelResponseIterator
@ -112,7 +113,7 @@ class OpenAILikeChatHandler(OpenAILikeBase):
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLoggingObj,
stream,
data: dict,
optional_params=None,
@ -214,7 +215,7 @@ class OpenAILikeChatHandler(OpenAILikeBase):
print_verbose: Callable,
encoding,
api_key: str | None,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
acompletion=None,
litellm_params: dict = {},

5
litellm/llms/predibase/chat/handler.py
View file

@ -9,6 +9,7 @@ from typing import Final
import httpx
import litellm
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
get_async_httpx_client,
@ -59,7 +60,7 @@ class PredibaseChatCompletion:
print_verbose: Callable,
encoding,
api_key: str,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
litellm_params: dict,
tenant_id: str,
@ -250,7 +251,7 @@ class PredibaseChatCompletion:
print_verbose: Callable,
encoding,
api_key,
logging_obj,
logging_obj: LiteLLMLoggingObj,
data: dict,
timeout: float | httpx.Timeout,
optional_params=None,

5
litellm/llms/replicate/chat/handler.py
View file

@ -6,6 +6,7 @@ from typing import Final
import litellm
from litellm.constants import REPLICATE_POLLING_DELAY_SECONDS
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,
@ -128,7 +129,7 @@ def completion(
print_verbose: Callable,
optional_params: dict,
litellm_params: dict,
logging_obj,
logging_obj: LiteLLMLoggingObj,
api_key,
encoding,
custom_prompt_dict={},
@ -246,7 +247,7 @@ async def async_completion(
input_data,
api_key,
api_base,
logging_obj,
logging_obj: LiteLLMLoggingObj,
print_verbose,
headers: dict,
) -> ModelResponse | CustomStreamWrapper:

55
litellm/llms/runwayml/videos/transformation.py
View file

@ -1,8 +1,10 @@
from collections.abc import Mapping, Sequence
from datetime import datetime
from typing import TYPE_CHECKING, Any, Final
from typing import TYPE_CHECKING, Any, Final, Literal
import httpx
from httpx._types import RequestFiles
from typing_extensions import ReadOnly, TypedDict
import litellm
from litellm.constants import RUNWAYML_DEFAULT_API_VERSION
@ -31,6 +33,29 @@ else:
LiteLLMLoggingObj = Any
class _RunwayTaskResponse(TypedDict, total=False):
id: ReadOnly[str]
status: ReadOnly[str]
createdAt: ReadOnly[str]
completedAt: ReadOnly[str]
output: ReadOnly[Sequence[str] | str]
failureCode: ReadOnly[str]
failure: ReadOnly[str]
progress: ReadOnly[int]
class _VideoObjectData(TypedDict, extra_items=object):
id: ReadOnly[str]
object: ReadOnly[Literal["video"]]
status: ReadOnly[str]
created_at: ReadOnly[int]
def _parse_runway_task_response(raw_response: httpx.Response) -> _RunwayTaskResponse:
response_data: Final[_RunwayTaskResponse] = raw_response.json()
return response_data
class RunwayMLVideoConfig(BaseVideoConfig):
"""
Configuration class for RunwayML video generation.
@ -78,7 +103,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
- size -> ratio (convert "WIDTHxHEIGHT" to "WIDTH:HEIGHT")
- seconds -> duration (convert to integer)
"""
mapped_params: Final[dict[str, Any]] = {}
mapped_params: Final[dict[str, object]] = {}
# Handle input_reference parameter - map to promptImage
if "input_reference" in video_create_optional_params:
@ -180,7 +205,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
}
"""
# Build the request data
request_data: Final[dict[str, Any]] = {
request_data: Final[dict[str, object]] = {
"model": model,
"promptText": prompt,
}
@ -189,7 +214,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
request_data.update(video_create_optional_request_params)
# RunwayML uses JSON body, no files multipart
files_list: Final[list[tuple[str, Any]]] = []
files_list: Final[RequestFiles] = []
# Append the specific endpoint for video generation
full_api_base: Final = f"{api_base}/image_to_video"
@ -216,10 +241,10 @@ class RunwayMLVideoConfig(BaseVideoConfig):
We map this to OpenAI VideoObject format.
"""
response_data: Final = raw_response.json()
response_data: Final = _parse_runway_task_response(raw_response)
# Map RunwayML task response to VideoObject format
video_data: Final[dict[str, Any]] = {
video_data: Final[_VideoObjectData] = {
"id": response_data.get("id", ""),
"object": "video",
"status": self._map_runway_status(response_data.get("status", "pending")),
@ -326,7 +351,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
# Get task status to retrieve video URL
url: Final = f"{api_base}/tasks/{encoded_video_id}"
params: Final[dict[str, Any]] = {}
params: Final[dict[str, str]] = {}
return url, params
@ -421,7 +446,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: dict[str, Any] | None = None,
extra_body: Mapping[str, object] | None = None,
) -> tuple[str, dict]:
"""
Transform the video remix request for RunwayML API.
@ -448,7 +473,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
after: str | None = None,
limit: int | None = None,
order: str | None = None,
extra_query: dict[str, Any] | None = None,
extra_query: Mapping[str, object] | None = None,
) -> tuple[str, dict]:
"""
Transform the video list request for RunwayML API.
@ -484,7 +509,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
# Construct the URL for task cancellation
url: Final = f"{api_base}/tasks/{encoded_video_id}/cancel"
data: Final[dict[str, Any]] = {}
data: Final[dict[str, str]] = {}
return url, data
@ -494,7 +519,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
logging_obj: LiteLLMLoggingObj,
) -> VideoObject:
"""Transform the RunwayML video delete/cancel response."""
response_data: Final = raw_response.json()
response_data: Final = _parse_runway_task_response(raw_response)
video_obj: Final = VideoObject(
id=response_data.get("id", ""),
@ -524,7 +549,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
url: Final = f"{api_base}/tasks/{encoded_video_id}"
# Empty dict for GET request (no body)
data: Final[dict[str, Any]] = {}
data: Final[dict[str, str]] = {}
return url, data
@ -537,10 +562,10 @@ class RunwayMLVideoConfig(BaseVideoConfig):
"""
Transform the RunwayML video status retrieve response.
"""
response_data: Final = raw_response.json()
response_data: Final = _parse_runway_task_response(raw_response)
# Map RunwayML task response to VideoObject format
video_data: Final[dict[str, Any]] = {
video_data: Final[_VideoObjectData] = {
"id": response_data.get("id", ""),
"object": "video",
"status": self._map_runway_status(response_data.get("status", "pending")),
@ -572,7 +597,7 @@ class RunwayMLVideoConfig(BaseVideoConfig):
return video_obj
def transform_video_create_character_request(self, name, video, api_base, litellm_params, headers):
def transform_video_create_character_request(self, name, video: object, api_base, litellm_params, headers):
raise NotImplementedError("video create character is not supported for RunwayML")
def transform_video_create_character_response(self, raw_response, logging_obj):

8
litellm/llms/sagemaker/completion/handler.py
View file

@ -8,6 +8,7 @@ import httpx
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.asyncify import asyncify
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.custom_httpx.http_handler import (
_get_httpx_client,
@ -138,7 +139,7 @@ class SagemakerLLM(BaseAWSLLM):
model_response: ModelResponse,
print_verbose: Callable,
encoding,
logging_obj,
logging_obj: LiteLLMLoggingObj,
optional_params: dict,
litellm_params: dict,
timeout: float | httpx.Timeout | None = None,
@ -431,17 +432,18 @@ class SagemakerLLM(BaseAWSLLM):
if not prepared_request.body:
raise ValueError("Prepared request body is empty")
stream_logging_obj: Final[LiteLLMLoggingObj] = logging_obj
completion_stream: Final = await self.make_async_call(
api_base=prepared_request.url,
headers=prepared_request.headers,
data=cast(str, prepared_request.body),
logging_obj=logging_obj,
logging_obj=stream_logging_obj,
)
streaming_response: Final = CustomStreamWrapper(
completion_stream=completion_stream,
model=model,
custom_llm_provider="sagemaker",
logging_obj=logging_obj,
logging_obj=stream_logging_obj,
)
# LOGGING

71
litellm/llms/vertex_ai/files/transformation.py
View file

@ -5,12 +5,13 @@ import json
import os
import re
import time
from collections.abc import Callable, Iterable, Iterator
from typing import Any, Final
from collections.abc import Callable, Iterable, Iterator, Mapping
from typing import Any, Final, TypedDict
import httpx
from httpx import Headers, Response
from openai.types.file_deleted import FileDeleted
from typing_extensions import ReadOnly
import litellm
from litellm._uuid import uuid
@ -50,6 +51,7 @@ from litellm.types.llms.openai import (
HttpxBinaryResponseContent,
OpenAICreateFileRequestOptionalParams,
OpenAIFileObject,
OpenAIFilesPurpose,
PathLike,
)
from litellm.types.llms.vertex_ai import GcsBucketResponse
@ -62,6 +64,46 @@ _GCP_LABEL_VALUE_MAX_LEN: Final = 63
_CUSTOM_ID_RAW_LABEL_PREFIX: Final = "b32_"
class _GcsObjectMetadataJson(TypedDict, total=False):
purpose: ReadOnly[OpenAIFilesPurpose]
class _GcsObjectJson(TypedDict, total=False):
id: ReadOnly[str]
name: ReadOnly[str]
size: ReadOnly[str]
timeCreated: ReadOnly[str]
metadata: ReadOnly[_GcsObjectMetadataJson]
class _VertexBatchRowRequest(TypedDict, total=False):
labels: ReadOnly[Mapping[str, object]]
class _VertexBatchRow(TypedDict, total=False):
request: ReadOnly[_VertexBatchRowRequest]
status: ReadOnly[str]
processed_time: ReadOnly[str]
class _OpenAIBatchOutputError(TypedDict):
code: ReadOnly[str]
message: ReadOnly[str]
class _OpenAIBatchOutputResponse(TypedDict):
status_code: ReadOnly[int]
request_id: ReadOnly[str]
body: ReadOnly[Mapping[str, object]]
class _OpenAIBatchOutputRow(TypedDict):
id: ReadOnly[str]
custom_id: ReadOnly[str]
response: ReadOnly[_OpenAIBatchOutputResponse | None]
error: ReadOnly[_OpenAIBatchOutputError | None]
def _sanitize_gcp_label_value(value: str) -> str:
"""
Sanitize a string to meet GCP label value constraints.
@ -106,7 +148,7 @@ def _decode_gcp_label_value_chunks(values: list[str]) -> str | None:
return None
def _set_litellm_batch_custom_id_labels(labels: dict[str, str], custom_id: Any) -> None:
def _set_litellm_batch_custom_id_labels(labels: dict[str, str], custom_id: object) -> None:
"""
Store OpenAI batch custom_id for Vertex batch correlation.
@ -122,7 +164,7 @@ def _set_litellm_batch_custom_id_labels(labels: dict[str, str], custom_id: Any)
labels[f"litellm_custom_id_raw_{index}"] = raw_label_chunk
def _get_litellm_batch_custom_id_from_labels(labels: dict[str, Any]) -> str:
def _get_litellm_batch_custom_id_from_labels(labels: Mapping[str, object]) -> str:
"""Prefer encoded custom_id when present (see _set_litellm_batch_custom_id_labels)."""
raw: Final = labels.get("litellm_custom_id_raw")
if raw:
@ -186,7 +228,7 @@ def _iter_openai_jsonl_lines(openai_file_content: FileTypes) -> Iterator[str]:
``str.splitlines()`` + ``line.strip()`` for ``\\n`` / ``\\r\\n`` delimited
JSONL.
"""
content: Any = openai_file_content
content: FileTypes | str = openai_file_content
if isinstance(content, tuple):
content = content[1]
@ -246,6 +288,11 @@ def _iter_openai_jsonl_entries(
yield json.loads(line)
def _parse_vertex_batch_output_row(line: str) -> _VertexBatchRow:
row: Final[_VertexBatchRow] = json.loads(line)
return row
class _OpenAIToVertexBatchUploadStream(BaseFileUploadStream):
"""Streams an OpenAI batch JSONL upload as Vertex-wrapped JSONL one row at a
time, so the transformed payload is never held in full.
@ -463,7 +510,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
"""
Transform VertexAI File upload response into OpenAI-style FileObject
"""
response_json: Final = raw_response.json()
response_json: Final[GcsBucketResponse] = raw_response.json()
try:
response_object: Final = GcsBucketResponse(**response_json)
@ -523,7 +570,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
logging_obj: LiteLLMLoggingObj,
litellm_params: dict,
) -> OpenAIFileObject:
response_json: Final = raw_response.json()
response_json: Final[_GcsObjectJson] = raw_response.json()
gcs_id = response_json.get("id", "")
gcs_id = "/".join(gcs_id.split("/")[:-1]) if gcs_id else ""
return OpenAIFileObject(
@ -682,7 +729,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
# discriminating fields. Anything else (e.g. a binary file whose
# first line is not valid UTF-8/JSON) raises and falls through to the
# passthrough below, leaving the content untouched.
first_row: Final = json.loads(first_line)
first_row: Final = _parse_vertex_batch_output_row(first_line)
is_vertex_batch_output: Final = (
"request" in first_row
and "response" in first_row
@ -723,7 +770,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
for line in itertools.chain([first_line], lines):
try:
openai_output = self._transform_single_vertex_batch_output_to_openai(
vertex_output=json.loads(line),
vertex_output=_parse_vertex_batch_output_row(line),
vertex_gemini_config=vertex_gemini_config,
logging_obj=batch_transform_logging_obj,
mock_httpx_response=mock_httpx_response,
@ -742,18 +789,18 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
def _transform_single_vertex_batch_output_to_openai(
self,
vertex_output: dict[str, Any],
vertex_output: _VertexBatchRow,
vertex_gemini_config: VertexGeminiConfig,
logging_obj: Logging,
mock_httpx_response: httpx.Response,
) -> dict[str, Any]:
) -> _OpenAIBatchOutputRow:
"""
Transform a single Vertex AI batch output line to OpenAI format.
Uses the existing VertexGeminiConfig transformation for the response.
"""
# Extract custom_id from request labels (prefer raw for OpenAI round-trip)
request_data: Final = vertex_output.get("request", {})
labels: Final = request_data.get("labels", {}) or {}
labels: Final[Mapping[str, object]] = request_data.get("labels", {}) or {}
custom_id: Final = _get_litellm_batch_custom_id_from_labels(labels)
# Check if there's an error

41
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -3,7 +3,7 @@
## Initial implementation - covers gemini + image gen calls
import json
import time
from collections.abc import Callable, Mapping
from collections.abc import Callable, Mapping, Sequence
from copy import deepcopy
from functools import partial
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast
@ -208,7 +208,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
presence_penalty: float | None = None,
seed: int | None = None,
) -> None:
locals_: Final = locals().copy()
locals_: Final[Mapping[str, object]] = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
setattr(self.__class__, key, value)
@ -1427,7 +1427,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
@staticmethod
def _extract_server_side_tool_invocations(
parts: list[HttpxPartType],
) -> list[dict[str, Any]] | None:
) -> list[dict[str, object]] | None:
"""Extract server-side tool invocations (toolCall/toolResponse) from parts.
These are returned by Gemini when context circulation is enabled
@ -1438,15 +1438,15 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
Returns:
List of server-side invocation dicts if any found, None otherwise.
"""
invocations: Final[list[dict[str, Any]]] = []
invocations: Final[list[dict[str, object]]] = []
# Index toolCalls by id so we can pair them with responses
tool_calls_by_id: Final[dict[str, dict[str, Any]]] = {}
tool_responses_by_id: Final[dict[str, dict[str, Any]]] = {}
tool_calls_by_id: Final[dict[str, dict[str, object]]] = {}
tool_responses_by_id: Final[dict[str, dict[str, object]]] = {}
for part in parts:
if "toolCall" in part:
tc = part["toolCall"]
entry: dict[str, Any] = {
entry: dict[str, object] = {
"tool_type": tc.get("toolType"),
"id": tc.get("id"),
"args": tc.get("args"),
@ -1753,7 +1753,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
response_tokens_details: CompletionTokensDetailsWrapper | None = None
usage_metadata: Final = completion_response["usageMetadata"]
def _get_token_count(detail: Mapping[str, Any]) -> int:
def _get_token_count(detail: Mapping[str, object]) -> int:
raw_token_count: Final = detail.get("tokenCount", detail.get("token_count", 0))
return raw_token_count if isinstance(raw_token_count, int) else 0
@ -2068,7 +2068,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
)
@staticmethod
def _get_stream_chunk_attr(chunk: Any, field_name: str) -> Any:
def _get_stream_chunk_attr(chunk: object, field_name: str) -> object:
if isinstance(chunk, dict):
value = chunk.get(field_name)
if value is not None:
@ -2110,10 +2110,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
def apply_assembled_streaming_response_metadata(
self,
response: ModelResponse,
chunks: list[Any],
chunks: list[object],
) -> None:
for field_name in VERTEX_AI_PROVIDER_METADATA_FIELDS:
merged: list[Any] = []
merged: list[object] = []
for chunk in chunks:
value = VertexGeminiConfig._get_stream_chunk_attr(chunk, field_name)
if not value:
@ -2214,8 +2214,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
functions: ChatCompletionToolCallFunctionChunk | None = None
thinking_blocks: list[ChatCompletionThinkingBlock] | None = None
reasoning_content: str | None = None
thought_signatures: Any | None = None
server_side_tool_invocations: list[dict[str, Any]] | None = None
thought_signatures: Sequence[str] | None = None
server_side_tool_invocations: list[dict[str, object]] | None = None
for idx, candidate in enumerate(_candidates):
if "content" not in candidate:
@ -2370,7 +2370,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
encoding: object,
api_key: str | None = None,
json_mode: bool | None = None,
) -> ModelResponse:
@ -2486,7 +2486,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
## ADD SERVICE TIER ##
if getattr(raw_response, "headers", None):
if service_tier := raw_response.headers.get("x-gemini-service-tier"):
service_tier: Final[str | None] = raw_response.headers.get("x-gemini-service-tier")
if service_tier:
if service_tier.lower() == "standard":
setattr(model_response, "service_tier", "default")
else:
@ -2660,7 +2661,7 @@ class VertexLLM(VertexBase):
print_verbose: Callable,
data: dict,
timeout: float | httpx.Timeout | None,
encoding,
encoding: object,
logging_obj,
stream,
optional_params: dict,
@ -2756,7 +2757,7 @@ class VertexLLM(VertexBase):
"vertex_ai", "vertex_ai_beta", "gemini"
], # if it's vertex_ai or gemini (google ai studio)
timeout: float | httpx.Timeout | None,
encoding,
encoding: object,
logging_obj,
stream,
optional_params: dict,
@ -2873,7 +2874,7 @@ class VertexLLM(VertexBase):
custom_llm_provider: Literal[
"vertex_ai", "vertex_ai_beta", "gemini"
], # if it's vertex_ai or gemini (google ai studio)
encoding,
encoding: object,
logging_obj,
optional_params: dict,
acompletion: bool,
@ -3122,7 +3123,7 @@ class ModelResponseIterator:
def _apply_stream_candidates(
self,
_candidates: list[Candidates],
model_response: Any,
model_response: "ModelResponseStream",
) -> tuple[list[dict], list[dict], list[dict], list[dict]]:
(
grounding_metadata,
@ -3200,7 +3201,7 @@ class ModelResponseIterator:
def _apply_stream_usage_metadata(
self,
processed_chunk: Any,
processed_chunk: GenerateContentResponseBody,
model_response: Any,
grounding_metadata: list[dict],
) -> Usage | None:

2
litellm/llms/vertex_ai/vertex_ai_non_gemini.py
View file

@ -28,7 +28,7 @@ class TextStreamer:
Fake streaming iterator for Vertex AI Model Garden calls
"""
def __init__(self, text):
def __init__(self, text: str):
self.text = text.split() # let's assume words as a streaming unit
self.index = 0

115
litellm/llms/vertex_ai/vertex_gemma_models/transformation.py
View file

@ -14,6 +14,11 @@ from typing import Any, Final, cast
import httpx
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,
_get_httpx_client,
)
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
@ -123,6 +128,79 @@ class VertexGemmaConfig(OpenAIGPTConfig):
return response_json["predictions"]
@staticmethod
def _sync_post(
client: HTTPHandler | httpx.Client | None,
api_base: str,
headers: dict[str, str], # mutable-ok: forwarded to post(headers: dict | None)
request_data: dict[str, Any], # mutable-ok: forwarded to post(json: dict | ...)
timeout: float | httpx.Timeout | None,
) -> httpx.Response:
if isinstance(client, HTTPHandler):
return client.post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
if isinstance(client, httpx.Client):
if timeout is None:
return client.post(
url=api_base,
headers=headers,
json=request_data,
)
return client.post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
return _get_httpx_client().post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
@staticmethod
async def _async_post(
client: AsyncHTTPHandler | httpx.AsyncClient | None,
api_base: str,
headers: dict[str, str], # mutable-ok: forwarded to post(headers: dict | None)
request_data: dict[str, Any], # mutable-ok: forwarded to post(json: dict | ...)
timeout: float | httpx.Timeout | None,
) -> httpx.Response:
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.types.utils import LlmProviders
if isinstance(client, AsyncHTTPHandler):
return await client.post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
if isinstance(client, httpx.AsyncClient):
if timeout is None:
return await client.post(
url=api_base,
headers=headers,
json=request_data,
)
return await client.post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
return await get_async_httpx_client(llm_provider=LlmProviders.VERTEX_AI).post(
url=api_base,
headers=headers,
json=request_data,
timeout=timeout,
)
def completion(
self,
model: str,
@ -137,7 +215,7 @@ class VertexGemmaConfig(OpenAIGPTConfig):
acompletion: bool,
litellm_params: dict,
logger_fn: Callable | None = None,
client: httpx.Client | None = None,
client: HTTPHandler | AsyncHTTPHandler | httpx.Client | httpx.AsyncClient | None = None,
timeout: float | httpx.Timeout | None = None,
encoding=None,
custom_llm_provider: str = "vertex_ai",
@ -147,6 +225,7 @@ class VertexGemmaConfig(OpenAIGPTConfig):
Supports both sync and async requests with fake streaming.
"""
if acompletion:
async_client = client if isinstance(client, (AsyncHTTPHandler, httpx.AsyncClient)) else None
return self._async_completion(
model=model,
messages=messages,
@ -157,10 +236,12 @@ class VertexGemmaConfig(OpenAIGPTConfig):
logging_obj=logging_obj,
optional_params=optional_params,
litellm_params=litellm_params,
client=async_client,
timeout=timeout,
encoding=encoding,
)
else:
sync_client = client if isinstance(client, (HTTPHandler, httpx.Client)) else None
return self._sync_completion(
model=model,
messages=messages,
@ -171,6 +252,7 @@ class VertexGemmaConfig(OpenAIGPTConfig):
logging_obj=logging_obj,
optional_params=optional_params,
litellm_params=litellm_params,
client=sync_client,
timeout=timeout,
encoding=encoding,
)
@ -186,11 +268,11 @@ class VertexGemmaConfig(OpenAIGPTConfig):
logging_obj: Any,
optional_params: dict,
litellm_params: dict,
timeout: float | httpx.Timeout | None,
encoding: Any,
client: HTTPHandler | httpx.Client | None = None,
timeout: float | httpx.Timeout | None = None,
encoding: Any = None,
):
"""Synchronous completion request"""
from litellm.llms.custom_httpx.http_handler import HTTPHandler
from litellm.utils import convert_to_model_response_object
# Check if streaming is requested (will be faked)
@ -222,11 +304,11 @@ class VertexGemmaConfig(OpenAIGPTConfig):
)
# Make the HTTP request
http_handler: Final = HTTPHandler(concurrent_limit=1)
response: Final = http_handler.post(
url=api_base,
response: Final = self._sync_post(
client=client,
api_base=api_base,
headers=headers,
json=request_data,
request_data=request_data,
timeout=timeout,
)
@ -276,12 +358,11 @@ class VertexGemmaConfig(OpenAIGPTConfig):
logging_obj: Any,
optional_params: dict,
litellm_params: dict,
timeout: float | httpx.Timeout | None,
encoding: Any,
client: AsyncHTTPHandler | httpx.AsyncClient | None = None,
timeout: float | httpx.Timeout | None = None,
encoding: Any = None,
):
"""Asynchronous completion request"""
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.types.utils import LlmProviders
from litellm.utils import convert_to_model_response_object
# Check if streaming is requested (will be faked)
@ -313,13 +394,11 @@ class VertexGemmaConfig(OpenAIGPTConfig):
)
# Make the HTTP request
http_handler: Final = get_async_httpx_client(
llm_provider=LlmProviders.VERTEX_AI,
)
response: Final = await http_handler.post(
url=api_base,
response: Final = await self._async_post(
client=client,
api_base=api_base,
headers=headers,
json=request_data,
request_data=request_data,
timeout=timeout,
)

71
litellm/llms/vertex_ai/videos/transformation.py
View file

@ -7,10 +7,12 @@ Based on: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-refer
import base64
import time
from typing import TYPE_CHECKING, Any, Final, cast
from collections.abc import Sequence
from typing import TYPE_CHECKING, Any, Final, TypedDict, cast
import httpx
from httpx._types import RequestFiles
from typing_extensions import ReadOnly
from litellm.constants import DEFAULT_GOOGLE_VIDEO_DURATION_SECONDS
from litellm.images.utils import ImageEditRequestUtils
@ -40,11 +42,37 @@ else:
BaseLLMException = Any
class _VeoVideo(TypedDict, total=False):
gcsUri: ReadOnly[str]
bytesBase64Encoded: ReadOnly[str]
mimeType: ReadOnly[str]
class _VeoOperationResponse(TypedDict, total=False):
videos: ReadOnly[Sequence[_VeoVideo]]
class _VeoOperationMetadata(TypedDict, total=False):
createTime: ReadOnly[str]
class _VeoOperation(TypedDict, total=False):
name: ReadOnly[str]
done: ReadOnly[bool]
metadata: ReadOnly[_VeoOperationMetadata]
response: ReadOnly[_VeoOperationResponse]
def _parse_veo_operation(raw_response: httpx.Response) -> _VeoOperation:
operation: Final[_VeoOperation] = raw_response.json()
return operation
def _build_vertex_video_usage_from_request_data(
request_data: dict[str, Any] | None,
) -> dict[str, Any]:
) -> dict[str, float | str]:
"""Build usage metadata (duration, resolution) for video cost calculation."""
usage_data: Final[dict[str, Any]] = {}
usage_data: Final[dict[str, float | str]] = {}
if not request_data:
return usage_data
@ -125,7 +153,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
video_create_optional_params: VideoCreateOptionalRequestParams,
model: str,
drop_params: bool,
) -> dict[str, Any]:
) -> dict[str, object]:
"""
Map OpenAI-style parameters to Veo format.
@ -135,7 +163,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
- size → aspectRatio (e.g., "1280x720" → "16:9")
- seconds → durationSeconds (defaults to 4 seconds if not provided)
"""
mapped_params: Final[dict[str, Any]] = {}
mapped_params: Final[dict[str, object]] = {}
# Map input_reference to image (will be processed in transform_video_create_request)
if "input_reference" in video_create_optional_params:
@ -289,7 +317,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
}
"""
# Build instance with prompt
instance_dict: Final[dict[str, Any]] = {"prompt": prompt}
instance_dict: Final[dict[str, object]] = {"prompt": prompt}
params_copy: Final = video_create_optional_request_params.copy()
# Check if user wants to provide full instance dict
@ -324,13 +352,13 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
# {"parameters": {"parameters": {...}}} ← wrong
# {"parameters": {...}} ← correct
nested_params: Final = params_copy.pop("parameters", None)
vertex_params: Final[dict[str, Any]] = {}
vertex_params: Final[dict[str, object]] = {}
if isinstance(nested_params, dict):
vertex_params.update(nested_params)
vertex_params.update(params_copy)
# Build request data directly (TypedDict doesn't have model_dump)
request_data: Final[dict[str, Any]] = {"instances": [instance_dict]}
request_data: Final[dict[str, object]] = {"instances": [instance_dict]}
# Only add parameters if there are any
if vertex_params:
@ -363,7 +391,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
- status: "processing"
- usage: includes duration_seconds and optional video_resolution for cost calculation
"""
response_data: Final = raw_response.json()
response_data: Final = _parse_veo_operation(raw_response)
operation_name: Final = response_data.get("name")
if not operation_name:
@ -441,7 +469,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
}
}
"""
response_data: Final = raw_response.json()
response_data: Final = _parse_veo_operation(raw_response)
operation_name: Final = response_data.get("name", "")
is_done: Final = response_data.get("done", False)
@ -513,7 +541,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
Extracts the base64 encoded video from the response and decodes it to bytes.
"""
response_data: Final = raw_response.json()
response_data: Final = _parse_veo_operation(raw_response)
if not response_data.get("done", False):
raise ValueError(
@ -548,7 +576,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: dict[str, Any] | None = None,
extra_body: dict[str, object] | None = None,
) -> tuple[str, dict]:
"""
Video remix is not supported by Veo API.
@ -574,7 +602,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
after: str | None = None,
limit: int | None = None,
order: str | None = None,
extra_query: dict[str, Any] | None = None,
extra_query: dict[str, object] | None = None,
) -> tuple[str, dict]:
"""
Video list is not supported by Veo API.
@ -615,7 +643,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
"""Video delete is not supported."""
raise NotImplementedError("Video delete is not supported by Vertex AI Veo.")
def transform_video_create_character_request(self, name, video, api_base, litellm_params, headers):
def transform_video_create_character_request(self, name, video: object, api_base, litellm_params, headers):
raise NotImplementedError("video create character is not supported for Vertex AI")
def transform_video_create_character_response(self, raw_response, logging_obj):
@ -649,7 +677,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict,
extra_body: dict[str, Any] | None = None,
extra_body: dict[str, object] | None = None,
prefetched_source_data: dict[str, Any] | None = None,
) -> tuple[str, dict]:
"""
@ -667,12 +695,13 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
if not prefetched_source_data.get("done", False):
raise ValueError("Source video generation is not complete yet. Check the video status before editing.")
videos: Final = prefetched_source_data.get("response", {}).get("videos", [])
source_response: Final[_VeoOperationResponse] = prefetched_source_data.get("response", {})
videos: Final = source_response.get("videos", [])
if not videos:
raise ValueError("No videos found in the completed operation. Cannot edit.")
source_video: Final = videos[0]
video_input: Final[dict[str, Any]] = {}
video_input: Final[dict[str, str]] = {}
if "gcsUri" in source_video:
video_input["gcsUri"] = source_video["gcsUri"]
elif "bytesBase64Encoded" in source_video:
@ -684,13 +713,13 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
operation_name: Final = extract_original_video_id(video_id)
model: Final = self.extract_model_from_operation_name(operation_name) or ""
instance_dict: Final[dict[str, Any]] = {"prompt": prompt, "video": video_input}
request_data: Final[dict[str, Any]] = {"instances": [instance_dict]}
instance_dict: Final[dict[str, object]] = {"prompt": prompt, "video": video_input}
request_data: Final[dict[str, object]] = {"instances": [instance_dict]}
if extra_body:
extra_body_copy: Final = dict(extra_body)
nested_params: Final = extra_body_copy.pop("parameters", None)
vertex_params: Final[dict[str, Any]] = {}
vertex_params: Final[dict[str, object]] = {}
if isinstance(nested_params, dict):
vertex_params.update(nested_params)
vertex_params.update(extra_body_copy)
@ -716,7 +745,7 @@ class VertexAIVideoConfig(BaseVideoConfig, VertexBase):
usage includes duration_seconds and optional video_resolution from the
edit request parameters for cost calculation.
"""
response_data: Final = raw_response.json()
response_data: Final = _parse_veo_operation(raw_response)
operation_name: Final = response_data.get("name")
if not operation_name:

31
litellm/llms/xai/chat/transformation.py
View file

@ -1,4 +1,4 @@
from collections.abc import AsyncIterator, Iterator
from collections.abc import AsyncIterator, Iterator, Mapping
from typing import Any, Final
import httpx
@ -12,13 +12,15 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
strip_name_from_messages,
)
from litellm.llms.xai.common_utils import XAIModelInfo
from litellm.llms.xai.cost_calculator import (
apply_server_side_tool_usage_details_to_usage,
)
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import (
Choices,
ModelResponse,
ModelResponseStream,
PromptTokensDetailsWrapper,
Usage,
)
@ -248,7 +250,7 @@ class XAIChatConfig(OpenAIGPTConfig):
XAI API returns empty string for finish_reason when using tools,
so we need to fix this after the standard OpenAI transformation.
Also handles X.AI web search usage tracking by extracting num_sources_used.
Also handles X.AI web search usage tracking.
"""
# First, let the parent class handle the standard transformation
@ -351,25 +353,20 @@ class XAIChatConfig(OpenAIGPTConfig):
def _enhance_usage_with_xai_web_search_fields(self, model_response: ModelResponse, raw_response_json: dict) -> None:
"""
Extract num_sources_used from X.AI response and map it to web_search_requests.
Copy usage.server_side_tool_usage_details from the provider usage block
onto model_response.usage for tool cost calculation.
"""
if not hasattr(model_response, "usage") or model_response.usage is None:
return
usage: Final[Usage] = model_response.usage
num_sources_used = None
response_usage: Final = raw_response_json.get("usage", {})
if isinstance(response_usage, dict) and "num_sources_used" in response_usage:
num_sources_used = response_usage.get("num_sources_used")
# Map num_sources_used to web_search_requests for cost detection
if num_sources_used is not None and num_sources_used > 0:
if usage.prompt_tokens_details is None:
usage.prompt_tokens_details = PromptTokensDetailsWrapper()
usage.prompt_tokens_details.web_search_requests = int(num_sources_used)
setattr(usage, "num_sources_used", int(num_sources_used))
verbose_logger.debug("X.AI web search sources used: %s", num_sources_used)
response_usage: Final = raw_response_json.get("usage")
if not isinstance(response_usage, dict):
return
details: Final = response_usage.get("server_side_tool_usage_details")
if isinstance(details, Mapping):
apply_server_side_tool_usage_details_to_usage(usage, details)
verbose_logger.debug("X.AI server_side_tool_usage_details: %s", details)
@staticmethod
def _normalize_openai_compatible_usage_totals(

98
litellm/llms/xai/cost_calculator.py
View file

@ -4,14 +4,37 @@ Helper util for handling XAI-specific cost calculation
- Handles XAI-specific reasoning token billing (billed as part of completion tokens)
"""
from collections.abc import Mapping
from typing import TYPE_CHECKING, Final
from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
from litellm.types.utils import Usage
from litellm.types.utils import PromptTokensDetailsWrapper, Usage
if TYPE_CHECKING:
from litellm.types.utils import ModelInfo
# https://docs.x.ai/developers/pricing#tools-pricing — default when unset in model map
_DEFAULT_WEB_SEARCH_COST_PER_CALL: Final = 5.0 / 1000.0
def apply_server_side_tool_usage_details_to_usage(usage: Usage, details: Mapping[str, object] | None) -> None:
"""
Attach server_side_tool_usage_details and mirror web_search_calls onto
prompt_tokens_details.web_search_requests for built-in tool cost gating.
"""
if details is None:
return
usage.server_side_tool_usage_details = details # pyright: ignore[reportAttributeAccessIssue] # extra # rebind-ok: extras
try:
web_search_calls: Final = int(details.get("web_search_calls") or 0)
except (TypeError, ValueError):
return
if web_search_calls <= 0:
return
prompt_tokens_details: Final = usage.prompt_tokens_details or PromptTokensDetailsWrapper()
prompt_tokens_details.web_search_requests = web_search_calls
usage.prompt_tokens_details = prompt_tokens_details # rebind-ok: write details onto caller usage
def cost_per_token(model: str, usage: Usage) -> tuple[float, float]:
"""
@ -32,9 +55,11 @@ def cost_per_token(model: str, usage: Usage) -> tuple[float, float]:
prompt_tokens: Final = int(getattr(usage, "prompt_tokens", 0) or 0)
completion_tokens: Final = int(getattr(usage, "completion_tokens", 0) or 0)
total_tokens: Final = int(getattr(usage, "total_tokens", 0) or 0)
reasoning_tokens = 0
if hasattr(usage, "completion_tokens_details") and usage.completion_tokens_details:
reasoning_tokens = int(getattr(usage.completion_tokens_details, "reasoning_tokens", 0) or 0)
reasoning_tokens: Final = (
int(getattr(usage.completion_tokens_details, "reasoning_tokens", 0) or 0)
if hasattr(usage, "completion_tokens_details") and usage.completion_tokens_details
else 0
)
already_normalised: Final = total_tokens == prompt_tokens + completion_tokens
total_completion_tokens: Final = completion_tokens if already_normalised else completion_tokens + reasoning_tokens
@ -52,33 +77,48 @@ def cost_per_token(model: str, usage: Usage) -> tuple[float, float]:
return prompt_cost, completion_cost
def _web_search_cost_per_call_from_model_info(model_info: "ModelInfo") -> float:
"""
Per-invocation web_search price from model_info when configured.
Prefer ``search_context_cost_per_query`` (same shape as Gemini/Anthropic web
search pricing in the model cost map). Fall back to current xAI list pricing.
"""
search_costs: Final = model_info.get("search_context_cost_per_query")
if not isinstance(search_costs, Mapping):
return _DEFAULT_WEB_SEARCH_COST_PER_CALL
for key in (
"search_context_size_medium",
"search_context_size_low",
"search_context_size_high",
):
value = search_costs.get(key)
if value is None:
continue
try:
cost = float(value)
except (TypeError, ValueError):
continue
if cost > 0:
return cost
return _DEFAULT_WEB_SEARCH_COST_PER_CALL
def cost_per_web_search_request(usage: "Usage", model_info: "ModelInfo") -> float:
"""
Calculate the cost of web search requests for X.AI models.
X.AI Live Search costs $25 per 1,000 sources used.
Each source costs $0.025.
The number of sources is stored in prompt_tokens_details.web_search_requests
by the transformation layer to be compatible with the existing detection system.
Counts invocations from usage.server_side_tool_usage_details.web_search_calls.
Per-call rate comes from model_info.search_context_cost_per_query when set,
otherwise the default xAI tools rate ($5 / 1k calls).
"""
# Cost per source used: $25 per 1,000 sources = $0.025 per source
cost_per_source: Final = 25.0 / 1000.0 # $0.025
num_sources_used = 0
if (
hasattr(usage, "prompt_tokens_details")
and usage.prompt_tokens_details is not None
and hasattr(usage.prompt_tokens_details, "web_search_requests")
and usage.prompt_tokens_details.web_search_requests is not None
):
num_sources_used = int(usage.prompt_tokens_details.web_search_requests)
# Fallback: try to get from num_sources_used if set directly
elif hasattr(usage, "num_sources_used") and usage.num_sources_used is not None:
num_sources_used = int(usage.num_sources_used)
total_cost: Final = cost_per_source * num_sources_used
return total_cost
details: Final = getattr(usage, "server_side_tool_usage_details", None)
if not isinstance(details, Mapping):
return 0.0
try:
web_search_calls: Final = int(details.get("web_search_calls") or 0)
except (TypeError, ValueError):
return 0.0
if web_search_calls <= 0:
return 0.0
return _web_search_cost_per_call_from_model_info(model_info) * web_search_calls

9
litellm/llms/xai/responses/transformation.py
View file

@ -1,4 +1,4 @@
from typing import TYPE_CHECKING, Any, Final
from typing import Any, Final
import litellm
from litellm._logging import verbose_logger
@ -12,13 +12,6 @@ from litellm.types.llms.xai import XAIWebSearchTool, XAIXSearchTool
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class XAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
"""

280
litellm/main.py
View file

@ -19,12 +19,12 @@ import random
import sys
import time
import traceback
from collections.abc import AsyncIterator, Coroutine, Iterable, Mapping
from collections.abc import AsyncIterator, Coroutine, Iterable, Mapping, Sequence
from concurrent import futures
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from copy import deepcopy
from functools import partial
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Union, cast, get_args
from typing import TYPE_CHECKING, Any, Final, Literal, Optional, Protocol, Union, cast, get_args
from litellm._logging import _redact_string
from litellm._uuid import uuid
@ -504,6 +504,7 @@ async def acompletion(
model=model,
custom_llm_provider=cast(str | None, custom_llm_provider), # cast-ok: read from untyped kwargs
tools=tools,
enable_prompt_caching=cast(bool | None, kwargs.get("enable_prompt_caching")), # cast-ok: untyped kwargs
)
if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (
@ -596,7 +597,7 @@ async def acompletion(
_, custom_llm_provider, _, _ = get_llm_provider(
model=model,
custom_llm_provider=custom_llm_provider,
api_base=completion_kwargs.get("base_url", None),
api_base=base_url,
)
fallbacks = fallbacks or litellm.model_fallbacks
@ -633,10 +634,10 @@ async def acompletion(
init_response: Final = await loop.run_in_executor(None, func_with_context)
if isinstance(init_response, dict) or isinstance(init_response, ModelResponse): ## CACHING SCENARIO
if isinstance(init_response, dict):
response = ModelResponse(**init_response)
response = _model_response_from_cached_dict(init_response)
response = init_response
elif asyncio.iscoroutine(init_response):
response = await init_response
response = await _resolve_dispatched_chat_response(init_response)
else:
response = init_response
@ -698,6 +699,20 @@ async def acompletion(
)
async def _resolve_dispatched_chat_response(
pending: Coroutine[object, object, ModelResponse | CustomStreamWrapper],
) -> ModelResponse | CustomStreamWrapper:
return await pending
def _model_response_from_cached_dict(cached_response_dict: Mapping[str, object]) -> ModelResponse:
return ModelResponse(**cached_response_dict)
def _transcription_response_from_cached_dict(cached_response_dict: Mapping[str, object]) -> TranscriptionResponse:
return TranscriptionResponse(**cached_response_dict)
async def _async_streaming(response, model, custom_llm_provider, args):
try:
print_verbose(f"received response in _async_streaming: {response}")
@ -983,12 +998,12 @@ def responses_api_bridge_check(
model: str,
custom_llm_provider: str,
web_search_options: OpenAIWebSearchOptions | None = None,
tools: list[Any] | None = None,
reasoning_effort: Any | None = None,
reasoning_summary: Any | None = None,
tools: Sequence[Mapping[str, object]] | None = None,
reasoning_effort: str | Mapping[str, object] | None = None,
reasoning_summary: object | None = None,
api_base: str | None = None,
) -> tuple[dict, str]:
model_info: dict[str, Any] = {}
model_info: dict[str, object] = {}
# Global flag: route ALL OpenAI chat completions through Responses API.
# Returns early with minimal model_info; callers only inspect the "mode" key.
@ -1110,6 +1125,22 @@ def _drop_input_examples_from_tools(
return cleaned_tools
class _ProxyAuthHeadersProvider(Protocol):
def get_auth_headers(self) -> Mapping[str, str]: ...
def _proxy_auth_headers(proxy_auth: _ProxyAuthHeadersProvider) -> Mapping[str, str]:
return proxy_auth.get_auth_headers()
def _provider_config_items(config: Mapping[str, object]) -> Iterable[tuple[str, object]]:
return config.items()
def _locals_snapshot(values: Mapping[str, object]) -> Mapping[str, object]:
return values
def _build_custom_pricing_entry(
custom_llm_provider: str,
kwargs: dict,
@ -1185,13 +1216,31 @@ def _register_custom_pricing_for_request(
)
def _dispatch_metadata(ctx: _CompletionDispatchContext) -> Mapping[str, object] | None:
return ctx.metadata
def _dispatch_client_http(ctx: _CompletionDispatchContext) -> HTTPHandler | AsyncHTTPHandler | None:
return ctx.client
def _dispatch_client_azure(
ctx: _CompletionDispatchContext,
) -> openai.AzureOpenAI | openai.AsyncAzureOpenAI | HTTPHandler | AsyncHTTPHandler | None:
return ctx.client
def _dispatch_client_openai(ctx: _CompletionDispatchContext) -> openai.OpenAI | openai.AsyncOpenAI | None:
return ctx.client
def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
_azure_detection_model: Final = ctx._azure_detection_model
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
api_version = ctx.api_version
client: Final = ctx.client
client: Final = _dispatch_client_azure(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
@ -1232,7 +1281,8 @@ def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
"AZURE_AD_TOKEN"
)
azure_ad_token_provider: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider_value: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider: Final = azure_ad_token_provider_value if callable(azure_ad_token_provider_value) else None
headers = headers or litellm.headers
@ -1244,7 +1294,7 @@ def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
if litellm.AzureOpenAIO1Config().is_o_series_model(model=_azure_detection_model):
## LOAD CONFIG - if set
config = litellm.AzureOpenAIO1Config.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > azure_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1273,7 +1323,7 @@ def _complete_azure(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
else:
## LOAD CONFIG - if set
config = litellm.AzureOpenAIConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > azure_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1323,7 +1373,7 @@ def _complete_azure_text(ctx: _CompletionDispatchContext) -> _CompletionDispatch
api_base = ctx.api_base
api_key = ctx.api_key
api_version = ctx.api_version
client: Final = ctx.client
client: Final = _dispatch_client_azure(ctx)
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1358,7 +1408,8 @@ def _complete_azure_text(ctx: _CompletionDispatchContext) -> _CompletionDispatch
"AZURE_AD_TOKEN"
)
azure_ad_token_provider: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider_value: Final = litellm_params.get("azure_ad_token_provider", None)
azure_ad_token_provider: Final = azure_ad_token_provider_value if callable(azure_ad_token_provider_value) else None
headers = headers or litellm.headers
@ -1367,7 +1418,7 @@ def _complete_azure_text(ctx: _CompletionDispatchContext) -> _CompletionDispatch
## LOAD CONFIG - if set
config: Final = litellm.AzureOpenAIConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > azure_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1415,7 +1466,7 @@ def _complete_deepseek(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1466,7 +1517,7 @@ def _complete_azure_ai(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
@ -1622,7 +1673,7 @@ def _complete_text_completion_openai(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_openai(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1654,7 +1705,7 @@ def _complete_text_completion_openai(
## LOAD CONFIG - if set
config: Final = litellm.OpenAITextCompletionConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_text_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1704,7 +1755,7 @@ def _complete_fireworks_ai(
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1755,7 +1806,7 @@ def _complete_heroku(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1805,7 +1856,7 @@ def _complete_ragflow(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1855,7 +1906,7 @@ def _complete_xai(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1906,7 +1957,7 @@ def _complete_groq(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1938,7 +1989,7 @@ def _complete_groq(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult
## LOAD CONFIG - if set
config: Final = litellm.GroqChatConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
@ -1970,7 +2021,7 @@ def _complete_bedrock_mantle(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -1987,7 +2038,7 @@ def _complete_bedrock_mantle(
api_key = api_key or litellm.api_key or get_secret("BEDROCK_MANTLE_API_KEY")
headers = headers or litellm.headers
config: Final = litellm.BedrockMantleChatConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if k not in optional_params:
optional_params[k] = v
return base_llm_http_handler.completion(
@ -2014,7 +2065,7 @@ def _complete_a2a(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2077,7 +2128,7 @@ def _complete_gigachat(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2139,7 +2190,7 @@ def _complete_sap(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2155,7 +2206,7 @@ def _complete_sap(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
headers = headers or litellm.headers
## LOAD CONFIG - if set
config: Final = litellm.GenAIHubOrchestrationConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
@ -2187,7 +2238,7 @@ def _complete_aiohttp_openai(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
extra_headers: Final = ctx.extra_headers
headers = ctx.headers
@ -2242,7 +2293,7 @@ def _complete_cometapi(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2291,7 +2342,7 @@ def _complete_minimax(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2337,7 +2388,7 @@ def _complete_hosted_vllm(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2383,7 +2434,7 @@ def _complete_custom_openai(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
custom_prompt_dict: Final = ctx.custom_prompt_dict
extra_headers = ctx.extra_headers
@ -2392,7 +2443,7 @@ def _complete_custom_openai(
logger_fn: Final = ctx.logger_fn
logging: Final = ctx.logging
messages: Final = ctx.messages
metadata: Final = ctx.metadata
metadata: Final = _dispatch_metadata(ctx)
model: Final = ctx.model
model_response: Final = ctx.model_response
optional_params: Final = ctx.optional_params
@ -2445,7 +2496,7 @@ def _complete_custom_openai(
## LOAD CONFIG - if set
config: Final = litellm.OpenAIConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if (
k not in optional_params
): # completion(top_k=3) > openai_config(top_k=3) <- allows for dynamic variables to be passed in
@ -2522,7 +2573,7 @@ def _complete_mistral(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -2673,7 +2724,7 @@ def _complete_anthropic(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
custom_prompt_dict = ctx.custom_prompt_dict
headers: Final = ctx.headers
@ -2972,7 +3023,7 @@ def _complete_huggingface(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3015,7 +3066,7 @@ def _complete_oci(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3050,7 +3101,7 @@ def _complete_compactifai(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3126,7 +3177,7 @@ def _complete_databricks(ctx: _CompletionDispatchContext) -> _CompletionDispatch
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -3198,7 +3249,7 @@ def _complete_datarobot(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3235,7 +3286,7 @@ def _complete_openrouter(ctx: _CompletionDispatchContext) -> _CompletionDispatch
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -3273,7 +3324,7 @@ def _complete_openrouter(ctx: _CompletionDispatchContext) -> _CompletionDispatch
## Load Config
config: Final = litellm.OpenrouterConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if k == "extra_body":
# we use openai 'extra_body' to pass openrouter specific params - transforms, route, models
if "extra_body" in optional_params:
@ -3314,7 +3365,7 @@ def _complete_vercel_ai_gateway(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -3351,7 +3402,7 @@ def _complete_vercel_ai_gateway(
## Load Config
config: Final = litellm.VercelAIGatewayConfig.get_config()
for k, v in config.items():
for k, v in _provider_config_items(config):
if k == "extra_body":
# we use openai 'extra_body' to pass vercel specific params - providerOptions
if "extra_body" in optional_params:
@ -3392,7 +3443,7 @@ def _complete_vertex_ai_beta(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3457,7 +3508,7 @@ def _complete_vertex_ai_beta(
def _complete_vertex_ai(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
custom_prompt_dict: Final = ctx.custom_prompt_dict
headers: Final = ctx.headers
@ -3754,7 +3805,7 @@ def _complete_text_completion_inception(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_openai(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logger_fn: Final = ctx.logger_fn
@ -3818,7 +3869,7 @@ def _complete_sagemaker_chat(
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -3881,7 +3932,7 @@ def _complete_bedrock(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_prompt_dict = ctx.custom_prompt_dict
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4005,7 +4056,7 @@ def _complete_watsonx(ctx: _CompletionDispatchContext) -> _CompletionDispatchRes
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_prompt_dict: Final = ctx.custom_prompt_dict
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4044,7 +4095,7 @@ def _complete_watsonx_text(
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -4156,7 +4207,7 @@ def _complete_ollama(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key: Final = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -4196,7 +4247,7 @@ def _complete_ollama_chat(ctx: _CompletionDispatchContext) -> _CompletionDispatc
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
logging: Final = ctx.logging
@ -4311,7 +4362,7 @@ def _complete_cloudflare(ctx: _CompletionDispatchContext) -> _CompletionDispatch
def _complete_petals(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
api_base = ctx.api_base
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
litellm_params: Final = ctx.litellm_params
logger_fn: Final = ctx.logger_fn
logging: Final = ctx.logging
@ -4353,7 +4404,7 @@ def _complete_snowflake(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key: Final = ctx.api_key
client = ctx.client
client = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4441,7 +4492,7 @@ def _complete_gdc(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4480,7 +4531,7 @@ def _complete_bytez(ctx: _CompletionDispatchContext) -> _CompletionDispatchResul
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4520,7 +4571,7 @@ def _complete_lemonade(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base: Final = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4560,7 +4611,7 @@ def _complete_ovhcloud(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers: Final = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4603,6 +4654,10 @@ def _complete_ovhcloud(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
return response
def _custom_api_first_output(resp: httpx.Response | None) -> str:
return resp.json()["data"][0]["output"][0]
def _complete_custom(ctx: _CompletionDispatchContext) -> _CompletionDispatchResult:
api_base: Final = ctx.api_base
headers: Final = ctx.headers
@ -4651,7 +4706,6 @@ def _complete_custom(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
**kwargs.get("extra_body", {}),
},
)
response_json: Final = resp.json()
"""
assume all responses from custom api_bases of this format:
{
@ -4665,7 +4719,7 @@ def _complete_custom(ctx: _CompletionDispatchContext) -> _CompletionDispatchResu
]
}
"""
string_response: Final = response_json["data"][0]["output"][0]
string_response: Final = _custom_api_first_output(resp)
## RESPONSE OBJECT
model_response.choices[0].message.content = string_response
model_response.created = int(time.time())
@ -4740,7 +4794,7 @@ def _complete_langgraph(ctx: _CompletionDispatchContext) -> _CompletionDispatchR
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4789,7 +4843,7 @@ def _complete_langflow(ctx: _CompletionDispatchContext) -> _CompletionDispatchRe
acompletion: Final = ctx.acompletion
api_base = ctx.api_base
api_key = ctx.api_key
client: Final = ctx.client
client: Final = _dispatch_client_http(ctx)
custom_llm_provider: Final = ctx.custom_llm_provider
headers = ctx.headers
litellm_params: Final = ctx.litellm_params
@ -4947,7 +5001,7 @@ def completion(
thinking = validate_and_fix_thinking_param(thinking=thinking)
######### unpacking kwargs #####################
args: Final = locals()
args: Final = _locals_snapshot(locals())
# Set by the responses->completion fallback so completion() does not bridge
# back to the Responses API: that round-trip mutually recurses forever for a
@ -5038,7 +5092,7 @@ def completion(
# Inject proxy auth headers if configured
if litellm.proxy_auth is not None:
try:
proxy_headers: Final = litellm.proxy_auth.get_auth_headers()
proxy_headers: Final = _proxy_auth_headers(litellm.proxy_auth)
headers.update(proxy_headers)
except Exception as e:
verbose_logger.warning("Failed to get proxy auth headers: %s", e)
@ -5091,7 +5145,7 @@ def completion(
)
######## end of unpacking kwargs ###########
non_default_params: Final = get_non_default_completion_params(kwargs=kwargs)
litellm_params = {} # used to prevent unbound var errors
litellm_params: dict[str, object] = {} # used to prevent unbound var errors
## PROMPT MANAGEMENT HOOKS ##
from litellm.integrations.anthropic_cache_control_hook import (
@ -5105,6 +5159,7 @@ def completion(
model=model,
custom_llm_provider=cast(str | None, kwargs.get("custom_llm_provider")), # cast-ok: untyped kwargs
tools=tools,
enable_prompt_caching=cast(bool | None, kwargs.get("enable_prompt_caching")), # cast-ok: untyped kwargs
)
if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (
@ -5561,7 +5616,12 @@ def completion(
elif custom_llm_provider == "hosted_vllm":
response = _complete_hosted_vllm(_dispatch_ctx)
elif (
model in litellm.open_ai_chat_completion_models
# A known OpenAI model name only decides the route when nothing else
# resolved a provider. get_llm_provider() already maps these names to
# "openai", so a different value here was asked for explicitly (or came
# from a register_model entry), and the provider config built for it
# would be handed to the OpenAI handler.
(model in litellm.open_ai_chat_completion_models and custom_llm_provider in (None, "openai"))
or custom_llm_provider == "custom_openai"
or custom_llm_provider == "deepinfra"
or custom_llm_provider == "perplexity"
@ -5913,7 +5973,7 @@ def embedding(
*,
aembedding: Literal[True],
**kwargs,
) -> Coroutine[Any, Any, EmbeddingResponse]:
) -> Coroutine[object, object, EmbeddingResponse]:
...
@ -5964,7 +6024,7 @@ def embedding(
litellm_call_id=None,
logger_fn=None,
**kwargs,
) -> EmbeddingResponse | Coroutine[Any, Any, EmbeddingResponse]:
) -> EmbeddingResponse | Coroutine[object, object, EmbeddingResponse]:
"""
Embedding function that calls an API to generate embeddings for the given input.
@ -6007,7 +6067,7 @@ def embedding(
# Inject proxy auth headers if configured
if litellm.proxy_auth is not None:
try:
proxy_headers: Final = litellm.proxy_auth.get_auth_headers()
proxy_headers: Final = _proxy_auth_headers(litellm.proxy_auth)
headers.update(proxy_headers)
except Exception as e:
verbose_logger.warning("Failed to get proxy auth headers: %s", e)
@ -6084,7 +6144,7 @@ def embedding(
if mock_response is not None:
return mock_embedding(model=model, mock_response=mock_response)
try:
response: EmbeddingResponse | Coroutine[Any, Any, EmbeddingResponse] | None = None
response: EmbeddingResponse | Coroutine[object, object, EmbeddingResponse] | None = None
if azure is True or custom_llm_provider == "azure":
# azure configs
@ -6387,7 +6447,7 @@ def embedding(
response = huggingface_embed.embedding(
model=model,
input=input,
encoding=_get_encoding(),
encoding=sys.modules[__name__].encoding,
api_key=api_key,
api_base=api_base,
logging_obj=logging,
@ -6990,6 +7050,20 @@ def embedding(
###### Text Completion ################
async def _resolve_dispatched_text_completion_response(
pending: Coroutine[
object,
object,
TextCompletionResponse | ModelResponse | CustomStreamWrapper | TextCompletionStreamWrapper,
],
) -> TextCompletionResponse | ModelResponse | CustomStreamWrapper | TextCompletionStreamWrapper:
return await pending
async def _resolve_pending_chat_response(pending: Coroutine[object, object, ModelResponse]) -> ModelResponse:
return await pending
@client
async def atext_completion(*args, **kwargs) -> TextCompletionResponse | TextCompletionStreamWrapper:
"""
@ -7015,7 +7089,7 @@ async def atext_completion(*args, **kwargs) -> TextCompletionResponse | TextComp
else:
response = init_response
elif asyncio.iscoroutine(init_response):
response = await init_response
response = await _resolve_dispatched_text_completion_response(init_response)
else:
response = init_response
@ -7040,7 +7114,7 @@ async def atext_completion(*args, **kwargs) -> TextCompletionResponse | TextComp
if isinstance(response, TextCompletionResponse):
return response
elif asyncio.iscoroutine(response):
response = await response
response = await _resolve_pending_chat_response(response)
text_completion_response = TextCompletionResponse()
text_completion_response = litellm.utils.LiteLLMResponseObjectHandler.convert_chat_to_text_completion(
@ -7330,11 +7404,11 @@ async def aadapter_completion(*, adapter_id: str, **kwargs) -> BaseModel | Adapt
async def aadapter_generate_content(
**kwargs,
) -> dict[str, Any] | AsyncIterator[bytes]:
) -> dict[str, object] | AsyncIterator[bytes]:
from litellm.google_genai.adapters.handler import GenerateContentToCompletionHandler
coro: Final = cast(
Coroutine[Any, Any, dict[str, Any] | AsyncIterator[bytes]],
Coroutine[object, object, dict[str, object] | AsyncIterator[bytes]],
GenerateContentToCompletionHandler.generate_content_handler(**kwargs, _is_async=True),
)
return await coro
@ -7486,7 +7560,7 @@ async def atranscription(*args, **kwargs) -> TranscriptionResponse:
# Await normally
init_response: Final = await loop.run_in_executor(None, func_with_context)
if isinstance(init_response, dict):
response = TranscriptionResponse(**init_response)
response = _transcription_response_from_cached_dict(init_response)
elif isinstance(init_response, TranscriptionResponse): ## CACHING SCENARIO
response = init_response
elif asyncio.iscoroutine(init_response):
@ -7541,7 +7615,7 @@ def transcription(
max_retries: int | None = None,
custom_llm_provider=None,
**kwargs,
) -> TranscriptionResponse | Coroutine[Any, Any, TranscriptionResponse]:
) -> TranscriptionResponse | Coroutine[object, object, TranscriptionResponse]:
"""
Calls openai + azure whisper endpoints.
@ -7608,7 +7682,7 @@ def transcription(
custom_llm_provider=custom_llm_provider,
)
response: TranscriptionResponse | Coroutine[Any, Any, TranscriptionResponse] | None = None
response: TranscriptionResponse | Coroutine[object, object, TranscriptionResponse] | None = None
provider_config: Final = ProviderConfigManager.get_provider_audio_transcription_config(
model=model,
@ -7842,7 +7916,7 @@ def speech(
custom_llm_provider: str | None = None,
aspeech: bool | None = None,
**kwargs,
) -> HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent]:
) -> HttpxBinaryResponseContent | Coroutine[object, object, HttpxBinaryResponseContent]:
user: Final = kwargs.get("user", None)
litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
proxy_server_request: Final = kwargs.get("proxy_server_request", None)
@ -7901,7 +7975,7 @@ def speech(
},
custom_llm_provider=custom_llm_provider,
)
response: HttpxBinaryResponseContent | Coroutine[Any, Any, HttpxBinaryResponseContent] | None = None
response: HttpxBinaryResponseContent | Coroutine[object, object, HttpxBinaryResponseContent] | None = None
if custom_llm_provider == "openai" or custom_llm_provider in litellm.openai_compatible_providers:
if voice is None or not (isinstance(voice, str)):
raise litellm.BadRequestError(
@ -8663,7 +8737,7 @@ def stream_chunk_builder(
]
if len(provider_specific_chunks) > 0:
combined_provider_fields: Final[dict[str, Any]] = {}
combined_provider_fields: Final[dict[str, object]] = {}
for chunk in provider_specific_chunks:
fields = chunk["choices"][0]["delta"]["provider_specific_fields"]
if isinstance(fields, dict):
@ -8728,7 +8802,7 @@ def stream_chunk_builder(
async def acount_tokens(
model: str,
messages: list[dict[str, Any]] | None = None,
messages: list[dict[str, object]] | None = None,
tools: list[dict[str, Any]] | None = None,
system: str | None = None,
api_key: str | None = None,
@ -8774,7 +8848,7 @@ async def acount_tokens(
api_base = dynamic_api_base
# Build deployment dict for the token counter
deployment: Final[dict[str, Any]] = {
deployment: Final[dict[str, object]] = {
"litellm_params": {
"model": model,
"api_key": api_key,
@ -8825,29 +8899,37 @@ async def acount_tokens(
# Cache for encoding to avoid repeated __getattr__ calls
_encoding_cache: Any | None = None
_encoding_cache: tiktoken.Encoding | None = None
def _get_encoding():
def _load_module_encoding() -> tiktoken.Encoding:
import sys
return sys.modules[__name__].encoding
def _get_encoding() -> tiktoken.Encoding:
"""Get encoding, loading it lazily if needed."""
global _encoding_cache
if _encoding_cache is None:
import sys
# Access via module to trigger __getattr__ if not cached
_encoding_cache = sys.modules[__name__].encoding
_encoding_cache = _load_module_encoding()
return _encoding_cache
def __getattr__(name: str) -> Any:
def _load_default_encoding() -> tiktoken.Encoding:
from litellm._lazy_imports import _get_default_encoding
return _get_default_encoding()
def __getattr__(name: str) -> tiktoken.Encoding:
"""Lazy import handler for main module"""
if name == "encoding":
# Use _get_default_encoding which properly sets TIKTOKEN_CACHE_DIR
# before loading tiktoken, ensuring the local cache is used
# instead of downloading from the internet
from litellm._lazy_imports import _get_default_encoding
_encoding: Final = _get_default_encoding()
_encoding: Final = _load_default_encoding()
# Cache it in the module's __dict__ for subsequent accesses
import sys

955
litellm/model_prices_and_context_window_backup.json
View file

File diff suppressed because it is too large Load diff

1
litellm/models/verification_token.py
View file

@ -49,6 +49,7 @@ class LiteLLM_VerificationToken(LiteLLMPydanticObjectBase):
created_by: str | None = None
updated_at: datetime | None = None
updated_by: str | None = None
settings_updated_at: datetime | None = None
last_active: datetime | None = None
object_permission_id: str | None = None
object_permission: LiteLLM_ObjectPermissionTable | None = None

Some files were not shown because too many files have changed in this diff Show more