scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 84

Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_ban_data_migrations

Browse source
This commit is contained in:
mateo-berri 2026-08-22 09:55:24 -07:00
commit cc5ff14d47
1123 changed files with 7899 additions and 8398 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

31
.github/actions/cache-cargo-build/action.yml vendored Normal file
View file

@ -0,0 +1,31 @@
name: "Cache the Rust build"
description: >-
Cache the Cargo registry and target directory the root package's build needs,
so only the first job on a given Cargo.lock compiles the bridge from scratch.
litellm builds through maturin, which compiles litellm-rust/crates/python-bridge
in release mode before it can produce a wheel. `uv sync` therefore pays a full
build in every job that installs the workspace: measured at 2m40s per unit shard
on 2026-08-21, more than the whole unit tier spends running tests. Nothing caught
it, because the uv cache holds wheels uv downloads rather than wheels it builds,
and a path dependency whose source moves every commit could never hit that cache
anyway. Cargo rebuilds only what changed when its target directory survives, so a
warm job pays for the bridge crate alone.
The key namespace is separate from test-rust.yml's. Both cache the same directory,
but that workflow fills it with debug and clippy artifacts, which a release build
cannot reuse, and a shared key would let whichever ran first deny the other a save.
runs:
using: composite
steps:
- name: Restore the Cargo registry and target directory
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cargo/registry
~/.cargo/git
litellm-rust/target
key: ${{ runner.os }}-cargo-release-${{ hashFiles('litellm-rust/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-release-

10
.github/ci-coverage-allowlist.yml vendored
View file

@ -48,16 +48,6 @@ test_paths:
choice it informed is settled
paths:
- tests/code_coverage_tests/test_aio_http_image_conversion.py
- reason: >-
The last file of a second mirror that sat beside tests/test_litellm and ran nowhere. Its
other 33 files landed in the real mirror during August 2026, 30 as moves and 3 by merging
their bodies into the live file of the same name. This one cannot follow either route yet:
its live twin was rewritten from 1268 lines to 9434, and of the 19 tests here 5 have no
counterpart while 25 assertions fail against today's code, so what survives that rewrite
is a judgement about the endpoints, not a merge. Revisit by deciding which of the five
behaviours still hold
paths:
- tests/litellm/proxy/_experimental/mcp_server/test_discoverable_endpoints.py
- reason: >-
No job invokes this suite and its files mix pure transformation tests with ones driving live
vendor vector stores, so assigning them needs a per-file decision

9
.github/workflows/_test-unit-base.yml vendored
View file

@ -27,7 +27,7 @@ on:
default: 20
job-timeout-minutes:
description: >-
Backstop for the whole job. Keep it >= `timeout-minutes` plus 35: 30 for
Backstop for the whole job. Keep it >= `timeout-minutes` plus 40: 35 for
the per-step ceilings on the setup steps below, and 5 for the runner
overhead the job clock charges but no step owns (job init, step
transitions, post-job cleanup). That headroom is what makes the test
@ -36,7 +36,7 @@ on:
arithmetic, so the sum is passed in rather than computed.
required: false
type: number
default: 55
default: 60
max-failures:
description: "Stop after this many failures"
required: false
@ -103,6 +103,11 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
timeout-minutes: 5
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
timeout-minutes: 8

4
.github/workflows/check-ui-api-types.yml vendored
View file

@ -67,6 +67,10 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
if: steps.changes.outputs.relevant == 'true'
uses: ./.github/actions/cache-cargo-build
- name: Install backend dependencies
if: steps.changes.outputs.relevant == 'true'
run: .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router

3
.github/workflows/mutation-test.yml vendored
View file

@ -53,6 +53,9 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml

3
.github/workflows/publish-basedpyright-base-counts.yml vendored
View file

@ -43,6 +43,9 @@ jobs:
with:
version: "0.10.9"
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build
- name: Cache Prisma binaries
uses: ./.github/actions/cache-prisma-binaries

3
.github/workflows/test-code-quality.yml vendored
View file

@ -56,6 +56,9 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
run: uv sync --frozen --all-groups --all-extras

4
.github/workflows/test-linting.yml vendored
View file

@ -78,6 +78,10 @@ jobs:
run: |
uv lock --check || (echo "❌ uv.lock is out of sync with pyproject.toml. Run 'uv lock' locally and commit the result." && exit 1)
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |

4
.github/workflows/test-mcp.yml vendored
View file

@ -47,6 +47,10 @@ jobs:
with:
version: "0.10.9"
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |

3
.github/workflows/test-terraform-provider.yml vendored
View file

@ -88,6 +88,9 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router

4
.github/workflows/test-unit-documentation.yml vendored
View file

@ -67,6 +67,10 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |

22
.github/workflows/test-unit.yml vendored
View file

@ -55,7 +55,7 @@ jobs:
workers: 2
reruns: 1
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: enterprise-routing
artifact-name: enterprise-routing
@ -67,7 +67,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: integrations
artifact-name: integrations
@ -75,7 +75,7 @@ jobs:
workers: 2
reruns: 3
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: Vertex AI
artifact-name: llm-vertex-ai
@ -83,7 +83,7 @@ jobs:
workers: 1
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: All Other Providers
artifact-name: llm-other-providers
@ -91,7 +91,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: misc
artifact-name: misc
@ -122,7 +122,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: proxy-auth
artifact-name: proxy-auth
@ -134,7 +134,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: proxy-endpoints
artifact-name: proxy-endpoints
@ -171,7 +171,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: proxy-server
artifact-name: proxy-server
@ -179,7 +179,7 @@ jobs:
workers: 4
reruns: 2
timeout-minutes: 60
job-timeout-minutes: 95
job-timeout-minutes: 100
- shard: proxy-infra
artifact-name: proxy-infra
@ -198,7 +198,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
- shard: responses-caching-types
artifact-name: responses-caching-types
@ -209,7 +209,7 @@ jobs:
workers: 2
reruns: 2
timeout-minutes: 20
job-timeout-minutes: 55
job-timeout-minutes: 60
uses: ./.github/workflows/_test-unit-base.yml
with:
test-path: ${{ matrix.test-path }}

3
.github/workflows/weekly_load_anomaly.yml vendored
View file

@ -47,6 +47,9 @@ jobs:
with:
version: "0.10.9"
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra proxy

4
enterprise/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-enterprise"
version = "0.1.58"
version = "0.1.59"
description = "Package for LiteLLM Enterprise features"
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.1.58"
version = "0.1.59"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-enterprise==",

4
litellm-proxy-extras/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "litellm-proxy-extras"
version = "0.4.88"
version = "0.4.89"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
readme = "README.md"
requires-python = ">=3.9"
@ -26,7 +26,7 @@ required-version = ">=0.10.9"
module-root = ""
[tool.commitizen]
version = "0.4.88"
version = "0.4.89"
version_files = [
"pyproject.toml:^version",
"../pyproject.toml:litellm-proxy-extras==",

24
litellm/_logging.py
View file

@ -88,6 +88,24 @@ def redact_secrets(value: str) -> str:
return _redact_string(value)
def _substituted_color_message(record: logging.LogRecord) -> str | None:
"""Render a record's ``color_message`` against its args, or None if absent.
uvicorn's colorized formatter re-renders `color_message` against
record.args at emit time (see uvicorn.logging.ColourizedFormatter) instead
of using the already-formatted record.msg, so it has to be substituted
before args are cleared or it is later formatted with no args and prints
the raw "%s://%s:%d" placeholders instead of the URL.
"""
color_message: Final = record.__dict__.get("color_message")
if not isinstance(color_message, str) or not record.args:
return None
try:
return color_message % record.args
except TypeError:
return color_message
class SecretRedactionFilter(logging.Filter):
"""Scrubs known secret/credential patterns from log records."""
@ -97,6 +115,12 @@ class SecretRedactionFilter(logging.Filter):
if not _ENABLE_SECRET_REDACTION:
return True
# Runs before args are cleared, and before the extra-field loop below
# that redacts the substituted result.
substituted_color_message: Final = _substituted_color_message(record)
if substituted_color_message is not None:
record.color_message = substituted_color_message # rebind-ok: a Filter scrubs records in place
try:
record.msg = _redact_string(record.getMessage())
record.args = None

40
litellm/batches/batch_utils.py
View file

@ -296,6 +296,32 @@ def calculate_vertex_ai_batch_cost_and_usage(
)
def _provider_output_file_id(output_file_id: str) -> str:
"""
Resolve the file id the provider actually knows: unified ids yield their embedded
llm_output_file_id, model-encoded ids decode to the raw provider id, raw ids pass through.
"""
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
get_original_file_id,
)
unified_file_id: Final = _is_base64_encoded_unified_file_id(output_file_id)
if not unified_file_id:
return get_original_file_id(output_file_id)
try:
extracted: Final = unified_file_id.split("llm_output_file_id,")[1].split(";")[0]
except (IndexError, AttributeError) as e:
verbose_logger.error(
"Failed to extract LLM output file ID from unified file ID: %s, error: %s",
output_file_id,
e,
)
return output_file_id
verbose_logger.debug("Extracted LLM output file ID from unified file ID: %s", extracted)
return extracted
async def _fetch_batch_output_file_content(
batch: Batch,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic"] = "openai",
@ -311,23 +337,11 @@ async def _fetch_batch_output_file_content(
Required for Azure and other providers that need authentication
"""
from litellm.files.main import afile_content
from litellm.proxy.openai_files_endpoints.common_utils import (
_is_base64_encoded_unified_file_id,
)
if batch.output_file_id is None:
raise ValueError("Output file id is None cannot retrieve file content")
file_id = batch.output_file_id
is_base64_unified_file_id: Final = _is_base64_encoded_unified_file_id(file_id)
if is_base64_unified_file_id:
try:
file_id = is_base64_unified_file_id.split("llm_output_file_id,")[1].split(";")[0]
verbose_logger.debug("Extracted LLM output file ID from unified file ID: %s", file_id)
except (IndexError, AttributeError) as e:
verbose_logger.error(
"Failed to extract LLM output file ID from unified file ID: %s, error: %s", batch.output_file_id, e
)
file_id: Final = _provider_output_file_id(batch.output_file_id)
# Build kwargs for afile_content with credentials from litellm_params
file_content_kwargs: Final = {

2
litellm/constants.py
View file

@ -783,6 +783,7 @@ openai_compatible_endpoints: Final[list] = [
"https://pinstripes.io/v1",
"https://api.meta.ai/v1",
"https://api.cognition.ai/v1",
"https://api.scx.ai/v1",
]
@ -851,6 +852,7 @@ openai_compatible_providers: Final[list] = [
"darkbloom",
"meta", # Meta Model API (Muse Spark) - JSON-configured provider
"cognition",
"scx-ai",
]
openai_text_completion_compatible_providers: Final[list] = [ # providers that support `/v1/completions`
"together_ai",

24
litellm/litellm_core_utils/get_provider_specific_headers.py
View file

@ -1,3 +1,4 @@
from collections.abc import Sequence
from typing import Final
from litellm.types.utils import ProviderSpecificHeader
@ -6,13 +7,17 @@ from litellm.types.utils import ProviderSpecificHeader
class ProviderSpecificHeaderUtils:
@staticmethod
def get_provider_specific_headers(
provider_specific_header: ProviderSpecificHeader | None,
provider_specific_header: ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None,
custom_llm_provider: str | None,
) -> dict:
"""
Get the provider specific headers for the given custom llm provider.
Supports comma-separated provider lists for headers that work across multiple providers.
Accepts either a single ProviderSpecificHeader or a sequence of them. Each entry
carries its own comma-separated provider list, so headers that are safe for several
providers and headers that are safe for exactly one can travel on the same request
without sharing a scope. Entries whose provider list does not contain
`custom_llm_provider` contribute nothing.
Returns:
Dict: The provider specific headers for the given custom llm provider
@ -20,10 +25,15 @@ class ProviderSpecificHeaderUtils:
if provider_specific_header is None or custom_llm_provider is None:
return {}
stored_providers: Final = provider_specific_header.get("custom_llm_provider", "")
provider_list: Final = [p.strip() for p in stored_providers.split(",")]
scoped_headers: Final = (
(provider_specific_header,) if isinstance(provider_specific_header, dict) else provider_specific_header
)
if custom_llm_provider in provider_list:
return provider_specific_header.get("extra_headers", {})
matched_headers: Final = {}
for scoped_header in scoped_headers:
stored_providers = scoped_header.get("custom_llm_provider", "")
provider_list = [p.strip() for p in stored_providers.split(",")]
if custom_llm_provider in provider_list:
matched_headers.update(scoped_header.get("extra_headers", {}))
return {}
return matched_headers

45
litellm/litellm_core_utils/litellm_logging.py
View file

@ -5615,6 +5615,37 @@ def _extract_response_obj_and_hidden_params(
return response_obj, hidden_params
def _autorouter_savings_for_payload(
request_metadata: Mapping[str, object],
model: str | None,
custom_llm_provider: str | None,
model_id: str | None,
usage_object: Mapping[str, object] | None,
cost_breakdown: Mapping[str, object] | None,
) -> float | None:
"""The auto-router savings figure for the payload, or ``None`` when there is none.
Lazy proxy import: the savings module lives with the spend trackers that own the
math, and SDK-only installs have no proxy package to import.
"""
try:
from litellm.proxy.spend_tracking.savings import autorouter_savings_for_logging_payload
except Exception: # noqa: BLE001 # SDK-only install: no savings driver to run
return None
try:
return autorouter_savings_for_logging_payload(
request_metadata=request_metadata,
model=model,
custom_llm_provider=custom_llm_provider,
model_id=model_id,
usage_object=usage_object,
cost_breakdown=cost_breakdown,
)
except Exception as e: # noqa: BLE001 # a savings figure must never fail request logging
verbose_logger.debug("autorouter savings skipped on logging payload: %s", e)
return None
def get_standard_logging_object_payload(
kwargs: dict | None,
init_response_obj: Any | BaseModel | dict,
@ -5772,6 +5803,16 @@ def get_standard_logging_object_payload(
):
model_name = response_model_name
request_cost_breakdown: Final = cost_breakdown_with_guardrail(logging_obj.cost_breakdown, guardrail_cost)
autorouter_savings: Final = _autorouter_savings_for_payload(
request_metadata=metadata,
model=model_name,
custom_llm_provider=custom_llm_provider,
model_id=_model_id,
usage_object=usage_dict,
cost_breakdown=request_cost_breakdown,
)
payload: Final[StandardLoggingPayload] = StandardLoggingPayload(
id=str(id),
litellm_call_id=kwargs.get("litellm_call_id") or litellm_params.get("litellm_call_id"),
@ -5802,7 +5843,8 @@ def get_standard_logging_object_payload(
metadata=clean_metadata,
cache_key=clean_hidden_params["cache_key"],
response_cost=response_cost,
cost_breakdown=cost_breakdown_with_guardrail(logging_obj.cost_breakdown, guardrail_cost),
cost_breakdown=request_cost_breakdown,
autorouter_savings=autorouter_savings,
total_tokens=usage_dict.get("total_tokens", 0),
prompt_tokens=usage_dict.get("prompt_tokens", 0),
completion_tokens=usage_dict.get("completion_tokens", 0),
@ -5998,6 +6040,7 @@ def create_dummy_standard_logging_payload() -> StandardLoggingPayload:
call_type="completion",
stream=False,
response_cost=response_cost,
autorouter_savings=None,
response_cost_failure_debug_info=None,
status="success",
total_tokens=int(DEFAULT_MOCK_RESPONSE_PROMPT_TOKEN_COUNT + DEFAULT_MOCK_RESPONSE_COMPLETION_TOKEN_COUNT),

8
litellm/litellm_core_utils/ptu_pricing.py
View file

@ -124,6 +124,14 @@ def ptu_identity_error(
return None
PTU_MODEL_INFO_FIELDS: Final = ("ptu_count", "cost_per_ptu_per_hour", "ptu_effective_from", "ptu_effective_to")
def declares_ptu(model_info: Mapping[str, object]) -> bool:
"""Whether any PTU field is set here, including one too malformed to charge."""
return any(model_info.get(field) is not None for field in PTU_MODEL_INFO_FIELDS)
def ptu_config_error(model_info: Mapping[str, object], *, model_name: str | None = None) -> str | None:
"""Why this PTU configuration cannot be honoured, else None.

11
litellm/llms/anthropic/experimental_pass_through/adapters/handler.py
View file

@ -21,6 +21,7 @@ from litellm.llms.anthropic.experimental_pass_through.context_management import
)
from litellm.llms.anthropic.experimental_pass_through.utils import (
is_reasoning_auto_summary_enabled,
local_model_name,
)
from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
@ -358,9 +359,9 @@ class LiteLLMMessagesToCompletionTransformationHandler:
except Exception:
pass
if isinstance(model, str) and model and not model.startswith("responses/"):
# Prefix model with "responses/" to route to OpenAI Responses API
completion_kwargs["model"] = f"responses/{model}"
if isinstance(model, str) and model and "responses/" not in model:
local_model: Final = model.removeprefix(f"{custom_llm_provider}/")
completion_kwargs["model"] = f"{custom_llm_provider}/responses/{local_model}"
auto_summary: Final = is_reasoning_auto_summary_enabled()
@ -616,7 +617,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
if stream:
transformed_stream: Final = ANTHROPIC_ADAPTER.translate_completion_output_params_streaming(
completion_response,
model=model,
model=local_model_name(model, kwargs.get("custom_llm_provider")),
tool_name_mapping=tool_name_mapping,
polyfill_result=polyfill_result,
is_async=True,
@ -750,7 +751,7 @@ class LiteLLMMessagesToCompletionTransformationHandler:
if stream:
transformed_stream: Final = ANTHROPIC_ADAPTER.translate_completion_output_params_streaming(
completion_response,
model=model,
model=local_model_name(model, kwargs.get("custom_llm_provider")),
tool_name_mapping=tool_name_mapping,
polyfill_result=polyfill_result,
is_async=False,

41
litellm/llms/anthropic/experimental_pass_through/messages/handler.py
View file

@ -42,15 +42,46 @@ from .utils import AnthropicMessagesRequestUtils, mock_response
_RESPONSES_API_PROVIDERS: Final = frozenset({"openai"})
def _should_route_to_responses_api(custom_llm_provider: str | None) -> bool:
"""Return True when the provider should use the Responses API path.
def _bridges_to_responses_api(model: str, custom_llm_provider: str) -> bool:
from litellm.main import responses_api_bridge_check
model_info, _ = responses_api_bridge_check(model=model, custom_llm_provider=custom_llm_provider)
return model_info.get("mode") == "responses"
def _responses_mode_is_lost_by_prefix_strip(
requested_model: str, resolved_model: str, custom_llm_provider: str
) -> bool:
"""Whether a Responses-only deployment stops looking like one once its provider prefix is stripped.
``litellm.completion`` re-derives the Responses bridge from the stripped id alone, so a
deployment id such as ``perplexity/perplexity/sonar`` (mode ``responses``) is shadowed by the
chat entry ``perplexity/sonar`` and would otherwise be sent to chat/completions.
"""
if requested_model == resolved_model:
return False
return _bridges_to_responses_api(requested_model, custom_llm_provider) and not _bridges_to_responses_api(
resolved_model, custom_llm_provider
)
def _should_route_to_responses_api(
custom_llm_provider: str | None,
requested_model: str | None = None,
resolved_model: str | None = None,
) -> bool:
"""Return True when the request should use the Responses API path.
Set ``litellm.use_chat_completions_url_for_anthropic_messages = True`` to
opt out and route OpenAI/Azure requests through chat/completions instead.
"""
if litellm.use_chat_completions_url_for_anthropic_messages:
return False
return custom_llm_provider in _RESPONSES_API_PROVIDERS
if custom_llm_provider in _RESPONSES_API_PROVIDERS:
return True
if custom_llm_provider is None or requested_model is None or resolved_model is None:
return False
return _responses_mode_is_lost_by_prefix_strip(requested_model, resolved_model, custom_llm_provider)
def _deployment_passes_through_anthropic_messages(model_info: object) -> bool:
@ -533,7 +564,7 @@ def anthropic_messages_handler(
_shared_kwargs: Final = dict(
max_tokens=max_tokens,
messages=messages,
model=model,
model=original_model,
metadata=metadata,
stop_sequences=stop_sequences,
stream=stream,
@ -551,7 +582,7 @@ def anthropic_messages_handler(
custom_llm_provider=custom_llm_provider,
**kwargs,
)
if _should_route_to_responses_api(custom_llm_provider):
if _should_route_to_responses_api(custom_llm_provider, original_model, model):
return LiteLLMMessagesToResponsesAPIHandler.anthropic_messages_handler(**_shared_kwargs)
# The in-gateway context_management polyfill runs inside

11
litellm/llms/anthropic/experimental_pass_through/messages/utils.py
View file

@ -44,6 +44,7 @@ class AnthropicMessagesRequestUtils:
filtered_params: Final = {k: v for k, v in params.items() if k in valid_keys and v is not None}
if model is not None:
from litellm.llms.anthropic.chat.transformation import AnthropicConfig
from litellm.llms.anthropic.common_utils import AnthropicModelInfo
AnthropicConfig._maybe_drop_speed_param(
model=model,
@ -51,6 +52,16 @@ class AnthropicMessagesRequestUtils:
drop_params=drop_params,
custom_llm_provider=custom_llm_provider,
)
for param in ("temperature", "top_p", "top_k"):
if param in filtered_params:
AnthropicModelInfo._apply_sampling_param( # pyright: ignore[reportPrivateUsage] # same gating the /chat/completions path applies; forking it would drift
optional_params=filtered_params,
model=model,
param=param,
value=filtered_params.pop(param),
drop_params=drop_params,
output_key=param,
)
return cast(AnthropicMessagesRequestOptionalParams, filtered_params)

9
litellm/llms/anthropic/experimental_pass_through/responses_adapters/handler.py
View file

@ -19,6 +19,7 @@ from litellm.types.llms.anthropic_messages.anthropic_response import (
)
from litellm.types.llms.openai import ResponsesAPIResponse
from ..utils import local_model_name
from .streaming_iterator import AnthropicResponsesStreamWrapper
from .transformation import LiteLLMAnthropicToResponsesAPIAdapter
@ -179,7 +180,9 @@ class LiteLLMMessagesToResponsesAPIHandler:
result: Final = await litellm.aresponses(**responses_kwargs)
if stream:
wrapper: Final = AnthropicResponsesStreamWrapper(responses_stream=result, model=model)
wrapper: Final = AnthropicResponsesStreamWrapper(
responses_stream=result, model=local_model_name(model, kwargs.get("custom_llm_provider"))
)
return wrapper.async_anthropic_sse_wrapper()
if not isinstance(result, ResponsesAPIResponse):
@ -257,7 +260,9 @@ class LiteLLMMessagesToResponsesAPIHandler:
result: Final = litellm.responses(**responses_kwargs)
if stream:
wrapper: Final = AnthropicResponsesStreamWrapper(responses_stream=result, model=model)
wrapper: Final = AnthropicResponsesStreamWrapper(
responses_stream=result, model=local_model_name(model, kwargs.get("custom_llm_provider"))
)
return wrapper.async_anthropic_sse_wrapper()
if not isinstance(result, ResponsesAPIResponse):

5
litellm/llms/anthropic/experimental_pass_through/utils.py
View file

@ -13,6 +13,11 @@ def prompt_cache_key_from_user_id(user_id: object) -> str | None:
return str(user_id)[:OPENAI_MAX_PROMPT_CACHE_KEY_LENGTH] or None
def local_model_name(model: str, custom_llm_provider: object) -> str:
"""The id the provider itself knows, for reporting back to the caller in ``message_start``."""
return model.removeprefix(f"{custom_llm_provider}/") if isinstance(custom_llm_provider, str) else model
def is_reasoning_auto_summary_enabled() -> bool:
"""Check whether the default 'summary: detailed' injection is enabled (opt-in)."""
return litellm.reasoning_auto_summary or os.getenv("LITELLM_REASONING_AUTO_SUMMARY", "false").lower() == "true"

2
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -2056,7 +2056,7 @@ class BaseLLMHTTPHandler:
# Prepare headers
kwargs = kwargs or {}
provider_specific_header: Final = cast(
litellm.types.utils.ProviderSpecificHeader | None,
litellm.types.utils.ProviderSpecificHeader | Sequence[litellm.types.utils.ProviderSpecificHeader] | None,
kwargs.get("provider_specific_header", None),
)
provider_specific_headers: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(

12
litellm/llms/openai_like/providers.json
View file

@ -188,5 +188,17 @@
"max_completion_tokens": "max_tokens"
},
"supported_endpoints": ["/v1/chat/completions", "/v1/responses", "/v1/embeddings"]
},
"scx-ai": {
"base_url": "https://api.scx.ai/v1",
"api_key_env": "SCX_API_KEY",
"api_base_env": "SCX_API_BASE",
"param_mappings": {
"max_completion_tokens": "max_tokens"
},
"constraints": {
"temperature_max": 1.99
},
"supported_endpoints": ["/v1/chat/completions"]
}
}

8
litellm/main.py
View file

@ -5091,14 +5091,16 @@ def completion(
model_info: Final = kwargs.get("model_info", None)
proxy_server_request: Final = kwargs.get("proxy_server_request", None)
fallbacks = kwargs.get("fallbacks", None)
provider_specific_header: Final = cast(ProviderSpecificHeader | None, kwargs.get("provider_specific_header", None))
provider_specific_header: Final = cast(
ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None,
kwargs.get("provider_specific_header", None),
)
headers = kwargs.get("headers", None) or extra_headers
ensure_alternating_roles: Final[bool | None] = kwargs.get("ensure_alternating_roles", None)
user_continue_message: Final[ChatCompletionUserMessage | None] = kwargs.get("user_continue_message", None)
assistant_continue_message: ChatCompletionAssistantMessage | None = kwargs.get("assistant_continue_message", None)
if headers is None:
headers = {}
headers = {} if headers is None else dict(headers)
if extra_headers is not None:
headers.update(extra_headers)
# Inject proxy auth headers if configured

581
litellm/model_prices_and_context_window_backup.json
View file

@ -4881,6 +4881,38 @@
"supports_tool_choice": true,
"supports_vision": false
},
"azure/gpt-audio-mini": {
"deprecation_date": "2027-04-06",
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"azure/gpt-audio-mini-2025-10-06": {
"deprecation_date": "2027-04-06",
"input_cost_per_audio_token": 1e-05,
@ -5094,6 +5126,38 @@
"supports_system_messages": true,
"supports_tool_choice": true
},
"azure/gpt-realtime-mini": {
"cache_creation_input_audio_token_cost": 3e-07,
"cache_read_input_token_cost": 6e-08,
"input_cost_per_audio_token": 1e-05,
"input_cost_per_image": 8e-07,
"input_cost_per_token": 6e-07,
"litellm_provider": "azure",
"max_input_tokens": 32000,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "realtime",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/realtime"
],
"supported_modalities": [
"text",
"image",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"azure/gpt-realtime-mini-2025-10-06": {
"cache_creation_input_audio_token_cost": 3e-07,
"cache_read_input_token_cost": 6e-08,
@ -19498,106 +19562,6 @@
},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-lite-image": {
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_response_schema": false,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true
},
"gemini/gemini-3.1-flash-lite-image": {
"rpm": 1000,
"tpm": 4000000,
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "gemini",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite-image",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": true,
"supports_prompt_caching": false,
"supports_response_schema": false,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true
},
"vertex_ai/gemini-3.1-flash-lite-image": {
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_response_schema": false,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true
},
"gemini-3.1-flash-image": {
"deprecation_date": "2027-05-28",
"input_cost_per_image": 0.00056,
@ -19675,6 +19639,44 @@
},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-lite-image": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"video"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_video_input": true,
"supports_vision": true
},
"gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_audio_token": 5e-07,
@ -21505,6 +21507,42 @@
},
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3.1-flash-lite-image": {
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "gemini",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"rpm": 1000,
"source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite-image",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_vision": true,
"tpm": 4000000
},
"gemini/deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -26041,33 +26079,33 @@
"supports_minimal_reasoning_effort": true
},
"gpt-5.6": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 6.25e-06,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_flex": 5e-07,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_above_272k_tokens_flex": 5e-06,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"cache_creation_input_token_cost": 5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 5e-06,
"cache_creation_input_token_cost_flex": 2.5e-06,
"cache_creation_input_token_cost_priority": 1e-05,
"cache_read_input_token_cost": 4e-07,
"cache_read_input_token_cost_above_272k_tokens": 8e-07,
"cache_read_input_token_cost_above_272k_tokens_flex": 4e-07,
"cache_read_input_token_cost_flex": 2e-07,
"cache_read_input_token_cost_priority": 8e-07,
"input_cost_per_token": 4e-06,
"input_cost_per_token_above_272k_tokens": 8e-06,
"input_cost_per_token_above_272k_tokens_flex": 4e-06,
"input_cost_per_token_batches": 2e-06,
"input_cost_per_token_flex": 2e-06,
"input_cost_per_token_priority": 8e-06,
"litellm_provider": "openai",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_above_272k_tokens_flex": 2.25e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_272k_tokens": 3e-05,
"output_cost_per_token_above_272k_tokens_flex": 1.5e-05,
"output_cost_per_token_batches": 1e-05,
"output_cost_per_token_flex": 1e-05,
"output_cost_per_token_priority": 4e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"search_context_cost_per_query": {
@ -26104,33 +26142,33 @@
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-sol": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 6.25e-06,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_flex": 5e-07,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_above_272k_tokens_flex": 5e-06,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"cache_creation_input_token_cost": 5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 5e-06,
"cache_creation_input_token_cost_flex": 2.5e-06,
"cache_creation_input_token_cost_priority": 1e-05,
"cache_read_input_token_cost": 4e-07,
"cache_read_input_token_cost_above_272k_tokens": 8e-07,
"cache_read_input_token_cost_above_272k_tokens_flex": 4e-07,
"cache_read_input_token_cost_flex": 2e-07,
"cache_read_input_token_cost_priority": 8e-07,
"input_cost_per_token": 4e-06,
"input_cost_per_token_above_272k_tokens": 8e-06,
"input_cost_per_token_above_272k_tokens_flex": 4e-06,
"input_cost_per_token_batches": 2e-06,
"input_cost_per_token_flex": 2e-06,
"input_cost_per_token_priority": 8e-06,
"litellm_provider": "openai",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_above_272k_tokens_flex": 2.25e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_272k_tokens": 3e-05,
"output_cost_per_token_above_272k_tokens_flex": 1.5e-05,
"output_cost_per_token_batches": 1e-05,
"output_cost_per_token_flex": 1e-05,
"output_cost_per_token_priority": 4e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"search_context_cost_per_query": {
@ -26372,19 +26410,19 @@
"supports_parallel_function_calling": true
},
"daybreak-blue-latest": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost": 5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1e-05,
"cache_read_input_token_cost": 4e-07,
"cache_read_input_token_cost_above_272k_tokens": 8e-07,
"input_cost_per_token": 4e-06,
"input_cost_per_token_above_272k_tokens": 8e-06,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_272k_tokens": 3e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
@ -31147,6 +31185,23 @@
"supports_video_input": true,
"supports_vision": true
},
"moonshot/kimi-k3": {
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "moonshot",
"max_input_tokens": 1048576,
"max_output_tokens": 1048576,
"max_tokens": 1048576,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"source": "https://platform.kimi.ai/docs/pricing/chat-k3",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_video_input": true,
"supports_vision": true
},
"moonshot/kimi-latest": {
"cache_read_input_token_cost": 1.5e-07,
"deprecation_date": "2026-01-28",
@ -36731,6 +36786,40 @@
"supports_vision": true,
"source": "https://cloud.sambanova.ai/plans/pricing"
},
"scx-ai/GLM-5.2": {
"cache_read_input_token_cost": 2.2e-07,
"input_cost_per_token": 6.1e-07,
"litellm_provider": "scx-ai",
"max_input_tokens": 1048576,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
"output_cost_per_token": 1.98e-06,
"source": "https://scx.ai/pricing",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": false
},
"scx-ai/Qwen3.8-Max": {
"cache_read_input_token_cost": 2.1e-07,
"input_cost_per_token": 1.65e-06,
"litellm_provider": "scx-ai",
"max_input_tokens": 1000000,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
"output_cost_per_token": 4.99e-06,
"source": "https://scx.ai/pricing",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"snowflake/claude-3-5-sonnet": {
"litellm_provider": "snowflake",
"max_input_tokens": 200000,
@ -41015,6 +41104,44 @@
"supports_reasoning": false,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models"
},
"vertex_ai/gemini-3.1-flash-lite-image": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"video"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_video_input": true,
"supports_vision": true
},
"vertex_ai/gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_audio_token": 5e-07,
@ -48554,6 +48681,156 @@
"supports_tool_choice": true,
"supports_vision": true
},
"us.openai.gpt-5.6-sol": {
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.375e-05,
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"global.openai.gpt-5.6-sol": {
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"us.openai.gpt-5.6-terra": {
"input_cost_per_token": 2.2e-06,
"input_cost_per_token_above_272k_tokens": 4.4e-06,
"cache_creation_input_token_cost": 2.75e-06,
"cache_creation_input_token_cost_above_272k_tokens": 5.5e-06,
"cache_read_input_token_cost": 2.2e-07,
"cache_read_input_token_cost_above_272k_tokens": 4.4e-07,
"output_cost_per_token": 1.32e-05,
"output_cost_per_token_above_272k_tokens": 1.98e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"global.openai.gpt-5.6-terra": {
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_272k_tokens": 4e-06,
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 5e-06,
"cache_read_input_token_cost": 2e-07,
"cache_read_input_token_cost_above_272k_tokens": 4e-07,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_above_272k_tokens": 1.8e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"us.openai.gpt-5.6-luna": {
"input_cost_per_token": 2.2e-07,
"input_cost_per_token_above_272k_tokens": 4.4e-07,
"cache_creation_input_token_cost": 2.75e-07,
"cache_creation_input_token_cost_above_272k_tokens": 5.5e-07,
"cache_read_input_token_cost": 2.2e-08,
"cache_read_input_token_cost_above_272k_tokens": 4.4e-08,
"output_cost_per_token": 1.32e-06,
"output_cost_per_token_above_272k_tokens": 1.98e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"global.openai.gpt-5.6-luna": {
"input_cost_per_token": 2e-07,
"input_cost_per_token_above_272k_tokens": 4e-07,
"cache_creation_input_token_cost": 2.5e-07,
"cache_creation_input_token_cost_above_272k_tokens": 5e-07,
"cache_read_input_token_cost": 2e-08,
"cache_read_input_token_cost_above_272k_tokens": 4e-08,
"output_cost_per_token": 1.2e-06,
"output_cost_per_token_above_272k_tokens": 1.8e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"bedrock_mantle/openai.gpt-5.5": {
"input_cost_per_token": 5.5e-06,
"cache_read_input_token_cost": 5.5e-07,

17
litellm/provider_endpoints_support_backup.json
View file

@ -2027,6 +2027,23 @@
"interactions": true
}
},
"scx-ai": {
"display_name": "SCX.ai (`scx-ai`)",
"url": "https://docs.litellm.ai/docs/providers/scx_ai",
"endpoints": {
"chat_completions": true,
"messages": false,
"responses": false,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": false,
"moderations": false,
"batches": false,
"rerank": false,
"a2a": false
}
},
"snowflake": {
"display_name": "Snowflake (`snowflake`)",
"url": "https://docs.litellm.ai/docs/providers/snowflake",

40
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
View file

@ -39,6 +39,7 @@ from litellm.proxy._types import (
SpecialMCPServerName,
SpecialMCPServerNames,
UserAPIKeyAuth,
user_api_key_has_admin_view,
)
from litellm.proxy.auth.ip_address_utils import IPAddressUtils
from litellm.proxy.auth.user_api_key_auth import (
@ -1785,11 +1786,14 @@ class MCPRequestHandler:
global_mcp_server_manager,
)
# An OPEN channel (allow_all_keys, the user's own BYOM) makes the server REACHABLE through the
# user, though no grant source names it — without this the union returns [], listable but
# uninvokable. Reachability is ALL it confers, NOT a ceiling waiver: the user's own
# mcp_tool_permissions and org tool ceiling still bind, exactly as a key's do on an allow_all server.
reachable_via_open_channel: Final = server_id in await global_mcp_server_manager.operator_open_server_ids(auth)
# An OPEN channel (allow_all_keys, the user's own BYOM, an unscoped admin-view role) makes the
# server REACHABLE through the user, though no grant source names it — without this the union
# returns [], listable but uninvokable. Reachability is ALL it confers, NOT a ceiling waiver:
# the user's own mcp_tool_permissions and org tool ceiling still bind, exactly as a key's do
# on an allow_all server or an admin key's do on any server.
reachable_via_open_channel: Final = server_id in await global_mcp_server_manager.operator_open_server_ids(
auth
) or await MCPRequestHandler.admin_view_unscoped(auth)
allowed: Final[set[str]] = set()
for source, granted in await MCPRequestHandler.admitted_source_grants(auth):
@ -2723,6 +2727,32 @@ class MCPRequestHandler:
entitled_servers: Final = await MCPRequestHandler._get_allowed_mcp_servers_for_user(user_api_key_auth)
return entitled_servers is None or len(entitled_servers) > 0
@staticmethod
async def admin_view_unscoped(user_api_key_auth: UserAPIKeyAuth | None = None) -> bool:
"""Whether this principal's admin-view role grants the unscoped MCP resolution, whatever
credential carries it (admin key, dashboard session, or OAuth-admitted session subject).
Two bounds disqualify, one per ownership of the row. A CREDENTIAL's explicit
``object_permission.mcp_servers`` scope wins even for admins, including the empty list. An
admitted subject's object_permission is the user's own row, whose ``mcp_servers`` column is
[] by DB default, so for that shape the row binds through the entitlement ceiling instead
(any non-empty entitlement, or an unresolved one, disqualifies), exactly as
``operator_open_server_ids`` reads the same row. The one owner of this predicate: the
server-axis registry resolution in ``get_allowed_mcp_servers`` and the tools-axis open
channel in ``_resolve_admitted_subject_tools`` both consult it, so the two axes cannot
disagree."""
if user_api_key_auth is None or not user_api_key_has_admin_view(user_api_key_auth):
return False
object_permission: Final = user_api_key_auth.object_permission
credential_scoped: Final = (
not _is_mcp_admitted_user_subject(user_api_key_auth)
and object_permission is not None
and object_permission.mcp_servers is not None
)
if credential_scoped:
return False
return not await MCPRequestHandler._user_places_mcp_ceiling(user_api_key_auth)
@staticmethod
async def _apply_user_tool_ceiling(
allowed_tools: Sequence[str] | None,

41
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
View file

@ -2943,17 +2943,14 @@ class MCPServerManager:
2. If admin and no object_permission, return all servers
3. Otherwise, use standard permission checks
"""
from litellm.proxy.management_endpoints.common_utils import _user_has_admin_view
allow_all_server_ids: Final = self.get_allow_all_keys_server_ids()
# A keyless admitted subject is resolved per grant source, and channel decisions that are
# absolute for a scoped KEY credential are not absolute for it: its own opt-out silences its
# own source (handled per source in the resolver), never its teams' grants, and its admin
# role does not swallow the grant model — a session bearer is a third-party client
# credential, not the dashboard, so an admin signing in through the connect flow gets their
# grants like anyone else rather than handing the client the full registry ahead of every
# per-team org ceiling.
# own source (handled per source in the resolver), never its teams' grants. Its admin role
# rides the HUMAN, not the credential: an admin's session resolves the same registry their
# dashboard shows (connect-page parity), bounded like an admin key by explicit
# object_permission scope, the entitlement ceiling, and the session resource scope below.
is_admitted_subject: Final = _is_mcp_admitted_user_subject(user_api_key_auth)
# The key explicitly opted out of every MCP server. Return zero before
@ -2982,26 +2979,16 @@ class MCPServerManager:
)
try:
# If admin but NO explicit object permission, get all servers (never for an admitted
# subject — see is_admitted_subject above)
if (
user_api_key_auth
and not is_admitted_subject
and _user_has_admin_view(user_api_key_auth)
and not has_explicit_object_permission
# An entitlement attached to the HUMAN binds them whatever their role: it is the
# person's scope, not the credential's, so an admin role is not a waiver of it. An
# UNRESOLVED entitlement also skips the shortcut, so the resolver denies rather than
# handing over the whole registry on a transient fault.
and not await MCPRequestHandler._user_places_mcp_ceiling(user_api_key_auth)
):
verbose_logger.debug("Admin user without explicit object_permission - returning all servers")
return list(self.get_registry().keys())
# Get allowed servers from object permissions (respects object_permission even for admins)
allowed_mcp_servers: Final = await MCPRequestHandler.get_allowed_mcp_servers(user_api_key_auth)
verbose_logger.debug("Allowed MCP Servers for user api key auth: %s", allowed_mcp_servers)
combined_servers: Final = set(allowed_mcp_servers)
# Admin view with no explicit object permission and no entitlement ceiling resolves the
# whole registry, for keys AND admitted session subjects alike (one predicate owns the
# question). Seeded into the union rather than returned early so the session resource
# scope below still bounds a per-server envelope held by an admin.
combined_servers: Final = (
set(self.get_registry().keys())
if await MCPRequestHandler.admin_view_unscoped(user_api_key_auth)
else set(await MCPRequestHandler.get_allowed_mcp_servers(user_api_key_auth))
)
verbose_logger.debug("Allowed MCP Servers for user api key auth: %s", combined_servers)
combined_servers.update(
await self.operator_open_server_ids(
user_api_key_auth,

18
litellm/proxy/_experimental/mcp_server/oauth_utils.py
View file

@ -180,6 +180,22 @@ def well_known_root_suffix() -> str:
return "" if root == "/" else root
def get_route_relative_request_path(scope: Scope) -> str:
"""The request path the MCP route shapes are written against: the raw ASGI path with the
deployment's ``root_path`` removed.
``scope["path"]`` and ``_original_path`` are both raw request-line paths, so on a sub-path
deployment they still carry the ``SERVER_ROOT_PATH`` prefix (``/litellm/{server}/mcp``) while
every route shape compared against them is root-relative. Mirrors the segment-boundary strip in
:func:`litellm.proxy.auth.auth_utils.get_request_route`, which the rest of the MCP auth path
already routes through, so ``/litellmfoo`` is not truncated under ``root_path=/litellm``."""
raw_path = str(scope.get("_original_path") or scope.get("path", "") or "")
root_path = str(scope.get("app_root_path") or scope.get("root_path") or "").rstrip("/")
if root_path and (raw_path == root_path or raw_path.startswith(f"{root_path}/")):
return raw_path[len(root_path) :]
return raw_path
def get_passthrough_resource_metadata_url(scope: Scope, server_name: str) -> str:
"""The per-server protected-resource metadata URL matching the spelling the request
arrived on, so a strict RFC 9728 client resolves the same route the proxy registered.
@ -188,7 +204,7 @@ def get_passthrough_resource_metadata_url(scope: Scope, server_name: str) -> str
the route decorators insert it (see :func:`well_known_root_suffix`)."""
request: Final = Request(scope)
base_url: Final = get_request_base_url(request)
_path: Final = scope.get("_original_path") or scope.get("path", "") or ""
_path: Final = get_route_relative_request_path(scope)
if _path.startswith(f"/{server_name}/mcp"):
return f"{base_url}/.well-known/oauth-protected-resource{well_known_root_suffix()}/{server_name}/mcp"

9
litellm/proxy/_experimental/mcp_server/server.py
View file

@ -51,6 +51,8 @@ from litellm.proxy._experimental.mcp_server.mcp_debug import MCPDebug
from litellm.proxy._experimental.mcp_server.oauth_utils import (
_redact_mcp_resource_url,
get_passthrough_www_authenticate,
get_route_relative_request_path,
well_known_root_suffix,
)
from litellm.proxy._experimental.mcp_server.utils import (
LITELLM_MCP_SERVER_DESCRIPTION,
@ -3782,14 +3784,15 @@ if MCP_AVAILABLE:
request = StarletteRequest(scope)
base_url = get_request_base_url(request)
_path = scope.get("_original_path") or scope.get("path", "") or ""
_path = get_route_relative_request_path(scope)
# Pick the well-known AS-metadata form that matches the inbound route
# so strict RFC 9728 §3.2 clients can resolve it correctly.
as_metadata_root = f"{base_url}/.well-known/oauth-authorization-server{well_known_root_suffix()}"
if _path.startswith(f"/mcp/{server_name}"):
_as_url = f"{base_url}/.well-known/oauth-authorization-server/mcp/{server_name}"
_as_url = f"{as_metadata_root}/mcp/{server_name}"
else:
_as_url = f"{base_url}/.well-known/oauth-authorization-server/{server_name}"
_as_url = f"{as_metadata_root}/{server_name}"
authorization_uri = f'Bearer authorization_uri="{_as_url}"'
raise HTTPException(

3
litellm/proxy/_types.py
View file

@ -15,7 +15,7 @@ from pydantic import (
field_validator,
model_validator,
)
from typing_extensions import NotRequired, Required, TypedDict
from typing_extensions import NotRequired, ReadOnly, Required, TypedDict
from litellm._uuid import uuid
from litellm.constants import DEFAULT_STAGGER_WINDOW_SECONDS, MCP_STDIO_ALLOWED_COMMANDS
@ -3537,6 +3537,7 @@ class SpendLogsMetadata(TypedDict):
max_retries: int | None # Max retries configured for this request
cost_breakdown: CostBreakdown | None # Detailed cost breakdown (input_cost, output_cost, margin, discount, etc.)
compression_savings: CompressionSavingsMetadata | None
autorouter_savings: ReadOnly[float | None] # stamped by the logging payload; None = not auto-routed
class SpendLogsPayload(TypedDict):

2
litellm/proxy/db/db_spend_update_writer.py
View file

@ -316,6 +316,7 @@ class DBSpendUpdateWriter:
model_id=payload.get("model_id"),
llm_router=_get_llm_router,
cost_breakdown=metadata.get("cost_breakdown"),
recorded_autorouter_savings=metadata.get("autorouter_savings"),
)
transaction: Final = build_autorouter_turn_transaction(
payload=payload,
@ -1877,6 +1878,7 @@ class DBSpendUpdateWriter:
llm_router=_get_llm_router,
usage_object=usage_obj,
cost_breakdown=_metadata.get("cost_breakdown"),
recorded_autorouter_savings=_metadata.get("autorouter_savings"),
)
daily_transaction: Final = BaseDailySpendTransaction(

95
litellm/proxy/litellm_pre_call_utils.py
View file

@ -64,6 +64,15 @@ _TRANSPORT_ONLY_CREDENTIAL_KEYS: Final = frozenset({"provider_specific_header",
# Excludes the two explicit litellm headers which are handled with higher priority.
_GENERIC_SESSION_ID_HEADER_RE: Final = re.compile(r"^x-.+-session-id$", re.IGNORECASE)
_EXPLICIT_SESSION_HEADERS: Final = frozenset({"x-litellm-trace-id", "x-litellm-session-id"})
# Codex carries its conversation uuid in unprefixed headers, so the
# x-<vendor>-session-id convention above never matches it. Current builds send
# ``session-id``/``thread-id``; builds before the codex-api split sent
# ``session_id``/``conversation_id``. Ordered session before thread.
_CODEX_SESSION_ID_HEADERS: Final = ("session-id", "session_id", "thread-id", "conversation_id")
# Matches every first-party Codex originator: codex-tui, codex_cli_rs, codex_exec,
# codex_vscode, "Codex ...". A separator is required so an unrelated "codexfoo" client
# does not read as Codex.
_CODEX_CLIENT_PREFIX_RE: Final = re.compile(r"^codex[-_ /]", re.IGNORECASE)
# Session-id values must be non-empty strings of alphanumerics, hyphens, or underscores
# (covers UUIDs and most common session-id formats).
_SESSION_ID_VALUE_RE: Final = re.compile(r"^[a-zA-Z0-9_\-]{8,}$")
@ -583,6 +592,35 @@ def _extract_generic_session_id_from_headers(
return None
def _extract_codex_session_id_from_headers(
normalized: Mapping[str, str],
) -> str | None:
"""
Read Codex's conversation uuid off one of ``_CODEX_SESSION_ID_HEADERS``.
Codex sends no request metadata the Anthropic path could parse and no
``x-``-prefixed session header, so without this every turn of a Codex session
falls through to a freshly generated per-call trace id and lands as its own
row in the logs instead of grouping.
Unprefixed names like ``session-id`` are generic enough that another client
could send one meaning something unrelated, and colliding values across
callers would merge their traces, so this only applies to callers that
identify as Codex.
"""
user_agent: Final = normalized.get("user-agent")
if not isinstance(user_agent, str) or not is_codex_user_agent(user_agent):
return None
return next(
(
value
for value in (normalized.get(header) for header in _CODEX_SESSION_ID_HEADERS)
if isinstance(value, str) and _SESSION_ID_VALUE_RE.match(value)
),
None,
)
def get_chain_id_from_headers(headers: dict[str, str] | None) -> str | None:
"""
Extract chain id for call chaining from request headers.
@ -592,6 +630,7 @@ def get_chain_id_from_headers(headers: dict[str, str] | None) -> str | None:
2. ``x-litellm-session-id`` (explicit)
3. Any ``x-<vendor>-session-id`` header whose value looks like a session id
(alphanumeric / UUID, at least 8 chars). E.g. ``x-claude-code-session-id``.
4. Codex's unprefixed ``session-id`` / ``thread-id``, for Codex callers only.
Header keys are matched case-insensitively so this works with raw header
dicts from any transport.
@ -606,6 +645,7 @@ def get_chain_id_from_headers(headers: dict[str, str] | None) -> str | None:
normalized.get("x-litellm-trace-id")
or normalized.get("x-litellm-session-id")
or _extract_generic_session_id_from_headers(normalized)
or _extract_codex_session_id_from_headers(normalized)
)
@ -640,10 +680,13 @@ def is_claude_code_user_agent(user_agent: str) -> bool:
def is_codex_user_agent(user_agent: str) -> bool:
"""Codex identifies itself as ``codex_cli_rs/<version> ...`` (TUI),
``codex_exec/<version> ...`` (exec mode), or ``codex_vscode/<version> ...``
(IDE extension); all share the ``codex_`` prefix."""
return user_agent.startswith("codex_")
"""Codex builds its user agent as ``<originator>/<version> ...`` and ships
several first-party originators: ``codex-tui``, ``codex_cli_rs``,
``codex_exec`` (exec mode), ``codex_vscode`` (IDE extension) and ``Codex ...``
(see ``is_first_party_originator`` in codex-rs). They agree only on the
``codex`` stem, and the TUI sends a bare ``codex-tui`` with no version at all,
so match the stem plus a separator rather than any one spelling."""
return bool(_CODEX_CLIENT_PREFIX_RE.match(user_agent))
def should_auto_drop_params_for_agentic_cli(user_agent: str, data: dict, proxy_config: ProxyConfig) -> bool:
@ -3001,36 +3044,36 @@ async def add_guardrails_from_policy_engine(
)
_ANTHROPIC_API_HEADER_PROVIDERS: Final = ",".join(
(LlmProviders.ANTHROPIC.value, LlmProviders.BEDROCK.value, LlmProviders.VERTEX_AI.value)
)
_ANTHROPIC_OAUTH_CREDENTIAL_PROVIDERS: Final = LlmProviders.ANTHROPIC.value
def add_provider_specific_headers_to_request(
data: dict,
headers: dict,
):
from litellm.llms.anthropic.common_utils import is_anthropic_oauth_key
anthropic_headers: Final = {}
# boolean to indicate if a header was added
added_header = False
for header in ANTHROPIC_API_HEADERS:
if header in headers:
header_value = headers[header]
anthropic_headers[header] = header_value
added_header = True
anthropic_api_headers: Final = {header: headers[header] for header in ANTHROPIC_API_HEADERS if header in headers}
anthropic_oauth_credential_headers: Final = {
header: value
for header, value in headers.items()
if header.lower() == "authorization" and is_anthropic_oauth_key(value)
}
# Check for Authorization header with Anthropic OAuth token (sk-ant-oat*)
# This needs to be handled via provider-specific headers to ensure it only
# goes to Anthropic-compatible providers, not all providers in the router
for header, value in headers.items():
if header.lower() == "authorization" and is_anthropic_oauth_key(value):
anthropic_headers[header] = value
added_header = True
break
if added_header is True:
# Anthropic headers work across multiple providers
# Store as comma-separated list so retrieval can match any of them
data["provider_specific_header"] = ProviderSpecificHeader(
custom_llm_provider=f"{LlmProviders.ANTHROPIC.value},{LlmProviders.BEDROCK.value},{LlmProviders.VERTEX_AI.value}",
extra_headers=anthropic_headers,
scoped_headers: Final = [
ProviderSpecificHeader(custom_llm_provider=providers, extra_headers=extra_headers)
for providers, extra_headers in (
(_ANTHROPIC_API_HEADER_PROVIDERS, anthropic_api_headers),
(_ANTHROPIC_OAUTH_CREDENTIAL_PROVIDERS, anthropic_oauth_credential_headers),
)
if extra_headers
]
if scoped_headers:
data["provider_specific_header"] = scoped_headers[0] if len(scoped_headers) == 1 else scoped_headers
def _add_otel_traceparent_to_data(data: dict, request: Request):

6
litellm/proxy/management_endpoints/model_management_endpoints.py
View file

@ -27,6 +27,7 @@ from litellm.constants import LITELLM_PROXY_ADMIN_NAME
from litellm.litellm_core_utils.ptu_pricing import (
CUSTOM_PRICING_FIELDS,
PTU_EMPTIED_PRICING_FIELDS,
PTU_MODEL_INFO_FIELDS,
PTU_ZEROED_PRICING_FIELDS,
PTU_ZEROED_TABLE_FIELDS,
SEARCH_CONTEXT_SIZES,
@ -247,7 +248,6 @@ def _raise_on_strategy_router_write_violation(
)
_PTU_MODEL_INFO_FIELDS: Final = ("ptu_count", "cost_per_ptu_per_hour", "ptu_effective_from", "ptu_effective_to")
_PTU_PRICED_PAIR: Final = frozenset({"ptu_count", "cost_per_ptu_per_hour"})
@ -261,7 +261,7 @@ def _explicitly_cleared_ptu_fields(model_info: ModelInfo | None) -> frozenset[st
return frozenset()
return frozenset(
field
for field in _PTU_MODEL_INFO_FIELDS
for field in PTU_MODEL_INFO_FIELDS
if field in model_info.model_fields_set and getattr(model_info, field) is None
)
@ -294,7 +294,7 @@ def _raise_if_ptu_cost_attribution_disabled(incoming_model_info: Mapping[str, ob
"""
if is_ptu_cost_attribution_enabled():
return
supplied: Final = tuple(field for field in _PTU_MODEL_INFO_FIELDS if incoming_model_info.get(field) is not None)
supplied: Final = tuple(field for field in PTU_MODEL_INFO_FIELDS if incoming_model_info.get(field) is not None)
if not supplied:
return
raise HTTPException(

28
litellm/proxy/public_endpoints/provider_create_fields.json
View file

@ -2726,6 +2726,34 @@
],
"default_model_placeholder": "sap/gpt-4"
},
{
"provider": "SCX_AI",
"provider_display_name": "SCX.ai",
"litellm_provider": "scx-ai",
"credential_fields": [
{
"key": "api_base",
"label": "API Base",
"placeholder": "https://api.scx.ai/v1",
"tooltip": null,
"required": false,
"field_type": "text",
"options": null,
"default_value": null
},
{
"key": "api_key",
"label": "API Key",
"placeholder": null,
"tooltip": null,
"required": true,
"field_type": "password",
"options": null,
"default_value": null
}
],
"default_model_placeholder": "scx-ai/GLM-5.2"
},
{
"provider": "Snowflake",
"provider_display_name": "Snowflake",

140
litellm/proxy/spend_tracking/savings.py
View file

@ -13,6 +13,7 @@ from typing import TYPE_CHECKING, Final, NamedTuple
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY
from litellm.litellm_core_utils.llm_cost_calc.utils import _get_cost_per_unit, generic_cost_per_token
if TYPE_CHECKING:
@ -437,6 +438,97 @@ def extract_cache_creation_tokens(usage_object: Mapping[str, object] | None) ->
return int(written)
def _proxy_llm_router() -> "Router | None":
"""The running proxy's router, or ``None`` outside a proxy (public rates only)."""
try:
from litellm.proxy.proxy_server import llm_router
except Exception: # noqa: BLE001 # SDK-only usage has no proxy module to import
return None
return llm_router
def _numeric_savings(value: object) -> float | None:
"""``value`` as a recorded savings figure, or ``None`` when it is not one."""
if isinstance(value, bool) or not isinstance(value, (int, float)):
return None
return float(value)
def autorouter_savings_for_request(
model: str | None,
custom_llm_provider: str | None,
routing_decision: Mapping[str, object] | None,
usage_object: Mapping[str, object] | None,
model_id: str | None = None,
llm_router: "Callable[[], Router | None] | None" = None,
cost_breakdown: Mapping[str, object] | None = None,
) -> float | None:
"""Auto-router savings for one request, or ``None`` when the driver is off.
``None`` and ``0.0`` are different facts: ``None`` means this request cannot carry a
figure at all (no routing decision, no baseline, unusable usage), while ``0.0`` is a
real figure for a routed request whose baseline resolved to the served deployment.
Never raises: pricing failures inside degrade to zero, and the driver-off cases
return ``None``, so this is safe on the logging path where a raise would fail the
request's logging.
"""
usage: Final = _usage_from_spend_log(usage_object)
if usage is None or not model:
return None
# The configured `autorouter_savings_baseline_model` wins; otherwise the baseline
# the deciding router recorded on its decision; neither means the driver is off.
decision: Final = routing_decision if isinstance(routing_decision, Mapping) else {}
recorded: Final = decision.get("savings_baseline_model")
recorded_id: Final = decision.get("savings_baseline_deployment_id")
configured: Final = litellm.autorouter_savings_baseline_model
baseline_model: Final = configured or (recorded if isinstance(recorded, str) else None)
baseline_id: Final = recorded_id if configured is None and isinstance(recorded_id, str) else None
if not decision or not baseline_model:
return None
router_instance: Final = llm_router() if llm_router else None
return compute_autorouter_savings(
baseline_model=baseline_model,
selected_model=model,
selected_provider=custom_llm_provider,
usage=usage,
# Absent means the router never recorded a shape, which is the conservative
# reading: charge the cache write rather than claim a first turn's saving.
conversation_continuing=decision.get("conversation_continuing") is not False,
selected_info=_effective_model_info(router_instance, model_id, model or ""),
baseline_info=_effective_model_info(router_instance, baseline_id, baseline_model or ""),
cost_breakdown=cost_breakdown,
)
def autorouter_savings_for_logging_payload(
request_metadata: Mapping[str, object],
model: str | None,
custom_llm_provider: str | None,
model_id: str | None,
usage_object: Mapping[str, object] | None,
cost_breakdown: Mapping[str, object] | None,
) -> float | None:
"""The figure the logging payload records for a request, or ``None`` when none should be.
Internal sub-calls (the auto-router classifier, shadow eval's shadow and judge legs)
are excluded here for the same reason the spend writer zeroes them: they can carry a
real routing decision, but they are not requests the caller made, so a figure stamped
on them would report savings for traffic no user sent.
"""
if request_metadata.get(INTERNAL_CALL_ORIGIN_METADATA_KEY):
return None
routing_decision: Final = request_metadata.get("routing_decision")
return autorouter_savings_for_request(
model=model,
custom_llm_provider=custom_llm_provider,
routing_decision=routing_decision if isinstance(routing_decision, Mapping) else None,
usage_object=usage_object,
model_id=model_id,
llm_router=_proxy_llm_router,
cost_breakdown=cost_breakdown,
)
def compute_savings_spend(
model: str | None,
custom_llm_provider: str | None,
@ -446,6 +538,7 @@ def compute_savings_spend(
model_id: str | None = None,
llm_router: "Callable[[], Router | None] | None" = None,
cost_breakdown: Mapping[str, object] | None = None,
recorded_autorouter_savings: object = None,
) -> SavingsSpend:
"""
Dollar savings for one request, split by optimization driver.
@ -488,6 +581,11 @@ def compute_savings_spend(
hypothetical token delta off flat rate keys, so they are blind to tiered pricing in
the same way; that is pre-existing behaviour on two shipped drivers rather than
something introduced here, and moving those numbers is its own change.
``recorded_autorouter_savings`` is the figure the logging path stamped on the spend
log's metadata, honoured over recomputation so the rollup, the turn table and the
per-request record cannot disagree; rows written before the field shipped carry
nothing and recompute, mirroring ``_recorded_token_cost``.
"""
# Deployment rates when the request came through one, public rates otherwise --
# `_effective_model_info` merges a deployment's configured prices over the built-in
@ -505,32 +603,24 @@ def compute_savings_spend(
write_premium: Final = max(cache_creation_input_tokens, 0) * (cache_write_cost - input_cost)
prompt_caching: Final = read_discount - write_premium
usage: Final = _usage_from_spend_log(usage_object)
if usage is None or not model:
return SavingsSpend(compression=compression, prompt_caching=prompt_caching)
# The configured `autorouter_savings_baseline_model` wins; otherwise the baseline
# the deciding router recorded on its decision; neither means the driver is off.
decision: Final = routing_decision if isinstance(routing_decision, Mapping) else {}
recorded: Final = decision.get("savings_baseline_model")
recorded_id: Final = decision.get("savings_baseline_deployment_id")
configured: Final = litellm.autorouter_savings_baseline_model
baseline_model: Final = configured or (recorded if isinstance(recorded, str) else None)
baseline_id: Final = recorded_id if configured is None and isinstance(recorded_id, str) else None
# The figure the logging path recorded wins, before the usage gate on purpose: a row
# whose usage no longer parses still carries the number computed when it did.
recorded_savings: Final = _numeric_savings(recorded_autorouter_savings)
autorouter: Final = (
compute_autorouter_savings(
baseline_model=baseline_model,
selected_model=model,
selected_provider=custom_llm_provider,
usage=usage,
# Absent means the router never recorded a shape, which is the conservative
# reading: charge the cache write rather than claim a first turn's saving.
conversation_continuing=decision.get("conversation_continuing") is not False,
selected_info=_effective_model_info(router_instance, model_id, model or ""),
baseline_info=_effective_model_info(router_instance, baseline_id, baseline_model or ""),
recorded_savings
if recorded_savings is not None
else autorouter_savings_for_request(
model=model,
custom_llm_provider=custom_llm_provider,
routing_decision=routing_decision,
usage_object=usage_object,
model_id=model_id,
llm_router=llm_router,
cost_breakdown=cost_breakdown,
)
if decision and baseline_model
else 0.0
)
return SavingsSpend(compression=compression, prompt_caching=prompt_caching, autorouter=autorouter)
return SavingsSpend(
compression=compression,
prompt_caching=prompt_caching,
autorouter=0.0 if autorouter is None else autorouter,
)

6
litellm/proxy/spend_tracking/spend_tracking_utils.py
View file

@ -100,6 +100,7 @@ def _get_spend_logs_metadata(
litellm_overhead_time_ms: float | None = None,
cost_breakdown: CostBreakdown | None = None,
litellm_call_id: str | None = None,
autorouter_savings: float | None = None,
) -> SpendLogsMetadata:
if metadata is None:
return SpendLogsMetadata(
@ -132,6 +133,7 @@ def _get_spend_logs_metadata(
max_retries=None,
cost_breakdown=None,
compression_savings=None,
autorouter_savings=autorouter_savings,
litellm_call_id=litellm_call_id,
)
verbose_proxy_logger.debug(
@ -158,6 +160,7 @@ def _get_spend_logs_metadata(
clean_metadata["cold_storage_object_key"] = cold_storage_object_key
clean_metadata["litellm_overhead_time_ms"] = litellm_overhead_time_ms
clean_metadata["cost_breakdown"] = cost_breakdown
clean_metadata["autorouter_savings"] = autorouter_savings
clean_metadata["litellm_call_id"] = litellm_call_id
return clean_metadata
@ -385,6 +388,9 @@ def get_logging_payload(kwargs, response_obj, start_time, end_time) -> SpendLogs
cost_breakdown=(
standard_logging_payload.get("cost_breakdown", None) if standard_logging_payload is not None else None
),
autorouter_savings=(
standard_logging_payload.get("autorouter_savings", None) if standard_logging_payload is not None else None
),
litellm_call_id=cast(
str | None,
kwargs.get("litellm_call_id") or litellm_params.get("litellm_call_id"),

17
litellm/router.py
View file

@ -66,6 +66,8 @@ from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.litellm_core_utils.ptu_pricing import (
PTU_COST_ATTRIBUTION_ENV_VAR,
declares_ptu,
is_ptu_cost_attribution_enabled,
ptu_config_error,
ptu_identity_error,
@ -8234,6 +8236,21 @@ class Router:
)
duplicate_ids: Final = frozenset(model_id for model_id in declared_ids if declared_ids.count(model_id) > 1)
ptu_declared: Final = tuple(
str(entry.get("model_name"))
for entry in original_model_list
if isinstance(entry.get("model_info"), dict)
and entry["model_info"].get("db_model") is not True
and declares_ptu(entry["model_info"])
)
if ptu_declared and not is_ptu_cost_attribution_enabled():
verbose_router_logger.warning(
"PTU fields are set on config.yaml deployment(s) %s, but PTU cost attribution is disabled, so no "
"flat cost accrues and this traffic is billed per token. Set %s=True to enable it",
", ".join(ptu_declared),
PTU_COST_ATTRIBUTION_ENV_VAR,
)
for model in original_model_list:
_model_name = model.pop("model_name")
_litellm_params = model.pop("litellm_params")

2
litellm/types/utils.py
View file

@ -3193,6 +3193,7 @@ class StandardLoggingPayload(TypedDict):
stream: bool | None
response_cost: float
cost_breakdown: CostBreakdown | None # Detailed cost breakdown
autorouter_savings: ReadOnly[float | None] # None = not an auto-routed caller request; 0.0 is a real figure
response_cost_failure_debug_info: StandardLoggingModelCostFailureDebugInformation | None
status: StandardLoggingPayloadStatus
status_fields: StandardLoggingPayloadStatusFields
@ -3789,6 +3790,7 @@ class LlmProviders(str, Enum):
LIBERTAI = "libertai"
PINSTRIPES = "pinstripes"
COGNITION = "cognition"
SCX_AI = "scx-ai"
DARKBLOOM = "darkbloom"
META = "meta"
LITELLM_AGENT = "litellm_agent"

581
model_prices_and_context_window.json
View file

@ -4881,6 +4881,38 @@
"supports_tool_choice": true,
"supports_vision": false
},
"azure/gpt-audio-mini": {
"deprecation_date": "2027-04-06",
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
"litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"max_tokens": 16384,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/chat/completions"
],
"supported_modalities": [
"text",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": false
},
"azure/gpt-audio-mini-2025-10-06": {
"deprecation_date": "2027-04-06",
"input_cost_per_audio_token": 1e-05,
@ -5094,6 +5126,38 @@
"supports_system_messages": true,
"supports_tool_choice": true
},
"azure/gpt-realtime-mini": {
"cache_creation_input_audio_token_cost": 3e-07,
"cache_read_input_token_cost": 6e-08,
"input_cost_per_audio_token": 1e-05,
"input_cost_per_image": 8e-07,
"input_cost_per_token": 6e-07,
"litellm_provider": "azure",
"max_input_tokens": 32000,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "realtime",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_token": 2.4e-06,
"supported_endpoints": [
"/v1/realtime"
],
"supported_modalities": [
"text",
"image",
"audio"
],
"supported_output_modalities": [
"text",
"audio"
],
"supports_audio_input": true,
"supports_audio_output": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"azure/gpt-realtime-mini-2025-10-06": {
"cache_creation_input_audio_token_cost": 3e-07,
"cache_read_input_token_cost": 6e-08,
@ -19498,106 +19562,6 @@
},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-lite-image": {
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_response_schema": false,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true
},
"gemini/gemini-3.1-flash-lite-image": {
"rpm": 1000,
"tpm": 4000000,
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "gemini",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite-image",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": true,
"supports_prompt_caching": false,
"supports_response_schema": false,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true
},
"vertex_ai/gemini-3.1-flash-lite-image": {
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_prompt_caching": true,
"supports_response_schema": false,
"supports_reasoning": true,
"supports_system_messages": true,
"supports_vision": true
},
"gemini-3.1-flash-image": {
"deprecation_date": "2027-05-28",
"input_cost_per_image": 0.00056,
@ -19675,6 +19639,44 @@
},
"web_search_billing_unit": "per_query"
},
"gemini-3.1-flash-lite-image": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"video"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_video_input": true,
"supports_vision": true
},
"gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_audio_token": 5e-07,
@ -21505,6 +21507,42 @@
},
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3.1-flash-lite-image": {
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "gemini",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"rpm": 1000,
"source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite-image",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": true,
"supports_prompt_caching": false,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_vision": true,
"tpm": 4000000
},
"gemini/deep-research-pro-preview-12-2025": {
"input_cost_per_image": 0.0011,
"input_cost_per_token": 2e-06,
@ -26041,33 +26079,33 @@
"supports_minimal_reasoning_effort": true
},
"gpt-5.6": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 6.25e-06,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_flex": 5e-07,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_above_272k_tokens_flex": 5e-06,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"cache_creation_input_token_cost": 5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 5e-06,
"cache_creation_input_token_cost_flex": 2.5e-06,
"cache_creation_input_token_cost_priority": 1e-05,
"cache_read_input_token_cost": 4e-07,
"cache_read_input_token_cost_above_272k_tokens": 8e-07,
"cache_read_input_token_cost_above_272k_tokens_flex": 4e-07,
"cache_read_input_token_cost_flex": 2e-07,
"cache_read_input_token_cost_priority": 8e-07,
"input_cost_per_token": 4e-06,
"input_cost_per_token_above_272k_tokens": 8e-06,
"input_cost_per_token_above_272k_tokens_flex": 4e-06,
"input_cost_per_token_batches": 2e-06,
"input_cost_per_token_flex": 2e-06,
"input_cost_per_token_priority": 8e-06,
"litellm_provider": "openai",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_above_272k_tokens_flex": 2.25e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_272k_tokens": 3e-05,
"output_cost_per_token_above_272k_tokens_flex": 1.5e-05,
"output_cost_per_token_batches": 1e-05,
"output_cost_per_token_flex": 1e-05,
"output_cost_per_token_priority": 4e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"search_context_cost_per_query": {
@ -26104,33 +26142,33 @@
"supports_xhigh_reasoning_effort": true
},
"gpt-5.6-sol": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 6.25e-06,
"cache_creation_input_token_cost_flex": 3.125e-06,
"cache_creation_input_token_cost_priority": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"cache_read_input_token_cost_above_272k_tokens_flex": 5e-07,
"cache_read_input_token_cost_flex": 2.5e-07,
"cache_read_input_token_cost_priority": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"input_cost_per_token_above_272k_tokens_flex": 5e-06,
"input_cost_per_token_batches": 2.5e-06,
"input_cost_per_token_flex": 2.5e-06,
"input_cost_per_token_priority": 1e-05,
"cache_creation_input_token_cost": 5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost_above_272k_tokens_flex": 5e-06,
"cache_creation_input_token_cost_flex": 2.5e-06,
"cache_creation_input_token_cost_priority": 1e-05,
"cache_read_input_token_cost": 4e-07,
"cache_read_input_token_cost_above_272k_tokens": 8e-07,
"cache_read_input_token_cost_above_272k_tokens_flex": 4e-07,
"cache_read_input_token_cost_flex": 2e-07,
"cache_read_input_token_cost_priority": 8e-07,
"input_cost_per_token": 4e-06,
"input_cost_per_token_above_272k_tokens": 8e-06,
"input_cost_per_token_above_272k_tokens_flex": 4e-06,
"input_cost_per_token_batches": 2e-06,
"input_cost_per_token_flex": 2e-06,
"input_cost_per_token_priority": 8e-06,
"litellm_provider": "openai",
"max_input_tokens": 922000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token_above_272k_tokens_flex": 2.25e-05,
"output_cost_per_token_batches": 1.5e-05,
"output_cost_per_token_flex": 1.5e-05,
"output_cost_per_token_priority": 6e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_272k_tokens": 3e-05,
"output_cost_per_token_above_272k_tokens_flex": 1.5e-05,
"output_cost_per_token_batches": 1e-05,
"output_cost_per_token_flex": 1e-05,
"output_cost_per_token_priority": 4e-05,
"regional_processing_uplift_multiplier_eu": 1.1,
"regional_processing_uplift_multiplier_us": 1.1,
"search_context_cost_per_query": {
@ -26372,19 +26410,19 @@
"supports_parallel_function_calling": true
},
"daybreak-blue-latest": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost": 5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1e-05,
"cache_read_input_token_cost": 4e-07,
"cache_read_input_token_cost_above_272k_tokens": 8e-07,
"input_cost_per_token": 4e-06,
"input_cost_per_token_above_272k_tokens": 8e-06,
"litellm_provider": "openai",
"max_input_tokens": 1050000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_272k_tokens": 3e-05,
"supported_endpoints": [
"/v1/chat/completions",
"/v1/responses"
@ -31147,6 +31185,23 @@
"supports_video_input": true,
"supports_vision": true
},
"moonshot/kimi-k3": {
"cache_read_input_token_cost": 3e-07,
"input_cost_per_token": 3e-06,
"litellm_provider": "moonshot",
"max_input_tokens": 1048576,
"max_output_tokens": 1048576,
"max_tokens": 1048576,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"source": "https://platform.kimi.ai/docs/pricing/chat-k3",
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_video_input": true,
"supports_vision": true
},
"moonshot/kimi-latest": {
"cache_read_input_token_cost": 1.5e-07,
"deprecation_date": "2026-01-28",
@ -36731,6 +36786,40 @@
"supports_vision": true,
"source": "https://cloud.sambanova.ai/plans/pricing"
},
"scx-ai/GLM-5.2": {
"cache_read_input_token_cost": 2.2e-07,
"input_cost_per_token": 6.1e-07,
"litellm_provider": "scx-ai",
"max_input_tokens": 1048576,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
"output_cost_per_token": 1.98e-06,
"source": "https://scx.ai/pricing",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": false
},
"scx-ai/Qwen3.8-Max": {
"cache_read_input_token_cost": 2.1e-07,
"input_cost_per_token": 1.65e-06,
"litellm_provider": "scx-ai",
"max_input_tokens": 1000000,
"max_output_tokens": 131072,
"max_tokens": 131072,
"mode": "chat",
"output_cost_per_token": 4.99e-06,
"source": "https://scx.ai/pricing",
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true
},
"snowflake/claude-3-5-sonnet": {
"litellm_provider": "snowflake",
"max_input_tokens": 200000,
@ -41015,6 +41104,44 @@
"supports_reasoning": false,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models"
},
"vertex_ai/gemini-3.1-flash-lite-image": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_image": 0.00028,
"input_cost_per_token": 2.5e-07,
"input_cost_per_token_batches": 1.25e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 65536,
"max_output_tokens": 4096,
"max_tokens": 4096,
"mode": "image_generation",
"output_cost_per_image": 0.0336,
"output_cost_per_image_token": 3e-05,
"output_cost_per_token": 1.5e-06,
"output_cost_per_token_batches": 7.5e-07,
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
"/v1/batch"
],
"supported_modalities": [
"text",
"image",
"video"
],
"supported_output_modalities": [
"text",
"image"
],
"supports_function_calling": false,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": false,
"supports_response_schema": false,
"supports_system_messages": true,
"supports_video_input": true,
"supports_vision": true
},
"vertex_ai/gemini-3.1-flash-lite-preview": {
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_audio_token": 5e-07,
@ -48554,6 +48681,156 @@
"supports_tool_choice": true,
"supports_vision": true
},
"us.openai.gpt-5.6-sol": {
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_272k_tokens": 1.1e-05,
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.375e-05,
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1.1e-06,
"output_cost_per_token": 3.3e-05,
"output_cost_per_token_above_272k_tokens": 4.95e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"global.openai.gpt-5.6-sol": {
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_272k_tokens": 1e-05,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_272k_tokens": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_272k_tokens": 1e-06,
"output_cost_per_token": 3e-05,
"output_cost_per_token_above_272k_tokens": 4.5e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"us.openai.gpt-5.6-terra": {
"input_cost_per_token": 2.2e-06,
"input_cost_per_token_above_272k_tokens": 4.4e-06,
"cache_creation_input_token_cost": 2.75e-06,
"cache_creation_input_token_cost_above_272k_tokens": 5.5e-06,
"cache_read_input_token_cost": 2.2e-07,
"cache_read_input_token_cost_above_272k_tokens": 4.4e-07,
"output_cost_per_token": 1.32e-05,
"output_cost_per_token_above_272k_tokens": 1.98e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"global.openai.gpt-5.6-terra": {
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_272k_tokens": 4e-06,
"cache_creation_input_token_cost": 2.5e-06,
"cache_creation_input_token_cost_above_272k_tokens": 5e-06,
"cache_read_input_token_cost": 2e-07,
"cache_read_input_token_cost_above_272k_tokens": 4e-07,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_above_272k_tokens": 1.8e-05,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"us.openai.gpt-5.6-luna": {
"input_cost_per_token": 2.2e-07,
"input_cost_per_token_above_272k_tokens": 4.4e-07,
"cache_creation_input_token_cost": 2.75e-07,
"cache_creation_input_token_cost_above_272k_tokens": 5.5e-07,
"cache_read_input_token_cost": 2.2e-08,
"cache_read_input_token_cost_above_272k_tokens": 4.4e-08,
"output_cost_per_token": 1.32e-06,
"output_cost_per_token_above_272k_tokens": 1.98e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"global.openai.gpt-5.6-luna": {
"input_cost_per_token": 2e-07,
"input_cost_per_token_above_272k_tokens": 4e-07,
"cache_creation_input_token_cost": 2.5e-07,
"cache_creation_input_token_cost_above_272k_tokens": 5e-07,
"cache_read_input_token_cost": 2e-08,
"cache_read_input_token_cost_above_272k_tokens": 4e-08,
"output_cost_per_token": 1.2e-06,
"output_cost_per_token_above_272k_tokens": 1.8e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supported_modalities": [
"text",
"image"
],
"supported_output_modalities": [
"text"
],
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_vision": true
},
"bedrock_mantle/openai.gpt-5.5": {
"input_cost_per_token": 5.5e-06,
"cache_read_input_token_cost": 5.5e-07,

17
provider_endpoints_support.json
View file

@ -2261,6 +2261,23 @@
"interactions": true
}
},
"scx-ai": {
"display_name": "SCX.ai (`scx-ai`)",
"url": "https://docs.litellm.ai/docs/providers/scx_ai",
"endpoints": {
"chat_completions": true,
"messages": false,
"responses": false,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": false,
"moderations": false,
"batches": false,
"rerank": false,
"a2a": false
}
},
"snowflake": {
"display_name": "Snowflake (`snowflake`)",
"url": "https://docs.litellm.ai/docs/providers/snowflake",

16
pyproject.toml
View file

@ -67,8 +67,8 @@ proxy = [
"azure-identity>=1.25.2,<2.0",
"azure-storage-blob>=12.28.0,<13.0",
"mcp>=1.28.1,<2.0",
"litellm-proxy-extras==0.4.88",
"litellm-enterprise==0.1.58",
"litellm-proxy-extras==0.4.89",
"litellm-enterprise==0.1.59",
"RestrictedPython>=8.1,<9.0",
"rich>=13.9.4,<14.0",
"InquirerPy>=0.3.4,<1.0",
@ -341,9 +341,13 @@ filterwarnings = [
paths_to_mutate = [
"litellm/proxy/management_endpoints/",
]
# Only the unit tier that maps to paths_to_mutate. mutmut times and
# coverage-maps this whole set once before mutating, so a tier that needs a
# seeded database (tests/proxy_behavior/) kills the run before it starts, and
# a mutation score is only meaningful against the tests that claim to cover
# the mutated code anyway.
tests_dir = [
"tests/test_litellm/proxy/management_endpoints/",
"tests/proxy_behavior/management/",
]
also_copy = [
"litellm/",
@ -360,10 +364,16 @@ mutate_only_covered_lines = true
# - rerunning a "failed" test on a mutant would mask which mutants are killed
# vs. survive, so reruns are wrong for mutation testing regardless.
# - xdist is unnecessary inside mutmut (mutmut handles its own parallelism).
# test_saml_sso.py cannot run inside mutmut's mutants/ sandbox: the copied tree
# re-imports cryptography's hash classes under a second identity, so x509 .sign()
# rejects the SHA256 instance the fixture builds with "Algorithm must be a
# registered hash algorithm". Nothing to do with mutation coverage, and one
# erroring test is enough to end the stats phase before any mutant runs.
pytest_add_cli_args = [
"-p", "no:retry",
"-p", "no:rerunfailures",
"-p", "no:xdist",
"--ignore=tests/test_litellm/proxy/management_endpoints/test_saml_sso.py",
]
[tool.coverage.run]

10
ruff-tests.toml
View file

@ -32,6 +32,14 @@
# PT017 an `assert` on the caught error inside `except`. Nothing runs the handler when
# the call stops raising, so the test goes green on the exact regression it was
# written to catch. `pytest.raises` fails when the call succeeds
# RUF043 a `match=` pattern carrying regex metacharacters in a plain string. `match=` is
# `re.search`, so a `.` copied out of an error message is a wildcard and the block
# accepts messages the author never meant to accept. Mark a real regex raw, wrap a
# literal message in `re.escape`, and the pattern says which one it is
# F823 a module-level name read inside a function that also binds it lower down. The
# later binding makes the name local for the whole body, so the read raises
# UnboundLocalError, and in an autouse fixture that takes every test in the
# directory down with it
#
# No target-version here on purpose: it resolves from requires-python (>=3.10), so
# 3.11-only builtins like BaseExceptionGroup are correctly flagged in a tree that
@ -53,4 +61,6 @@ lint.select = [
"PT017",
"PLR0133",
"PLW0127",
"RUF043",
"F823",
]

106
scripts/mutation_report.py
View file

@ -22,6 +22,7 @@ import tomllib
from collections import defaultdict
from difflib import SequenceMatcher
from pathlib import Path
from typing import Final, NamedTuple
from textwrap import dedent
ROOT = Path(__file__).resolve().parent.parent
@ -33,16 +34,24 @@ def load_mutmut_config() -> dict:
return tomllib.load(f)["tool"]["mutmut"]
def get_survivors() -> list[str]:
class MutmutResults(NamedTuple):
survivors: tuple[str, ...]
reported: int
def get_survivors() -> MutmutResults:
proc = subprocess.run(
[*MUTMUT_INVOCATION, "results"], capture_output=True, text=True, check=False
)
survivors = []
for line in proc.stdout.splitlines():
m = re.match(r"\s*(\S+):\s*survived\s*$", line)
if m:
survivors.append(m.group(1))
return survivors
verdicts = tuple(
m.groups()
for line in proc.stdout.splitlines()
if (m := re.match(r"\s*(\S+):\s*(\S.*?)\s*$", line))
)
return MutmutResults(
survivors=tuple(name for name, verdict in verdicts if verdict == "survived"),
reported=len(verdicts),
)
def get_mutmut_show(mutant_name: str) -> str:
@ -222,7 +231,52 @@ def render_meta_style_mutant(
return "\n".join(out)
def render(config: dict, survivors: list[str], stats: dict | None) -> str:
RESOLVED_KEYS: Final = frozenset({"killed", "survived", "total"})
def unresolved_counts(stats: dict) -> dict[str, int]:
"""Every non-zero count that is neither a kill nor a survivor means a mutant did not
reach the tests. Reading it as "anything else" rather than as a list of known statuses
keeps a status this reporter has never met from passing as a clean sweep."""
return {k: v for k, v in sorted(stats.items()) if k not in RESOLVED_KEYS and isinstance(v, int) and v > 0}
def clean_sweep_is_provable(stats: dict | None) -> bool:
"""`mutmut results` omits killed mutants, so its silence is equally consistent with a
perfect run and with a run that never started. Only the stats file can tell them apart,
and only when it agrees that nothing survived and every mutant reached the tests."""
if not stats or stats.get("killed", 0) <= 0 or stats.get("survived", 0) != 0:
return False
return not unresolved_counts(stats)
def no_survivors_verdict(results: MutmutResults, stats: dict | None) -> str:
if clean_sweep_is_provable(stats):
return "**No surviving mutants, and the run killed some, so the test suite caught every mutation.**"
if stats and stats.get("survived", 0) > 0:
return (
f"**mutmut-cicd-stats.json counts {stats['survived']} surviving mutant(s) that "
"`mutmut results` did not list, so the two disagree and neither can be trusted. "
"This is not a passing score.**"
)
if stats and unresolved_counts(stats):
unresolved = ", ".join(f"{v} {k.replace('_', ' ')}" for k, v in unresolved_counts(stats).items())
return (
f"**No survivors, but {unresolved}, so those mutants never reached the tests "
"and the suite was not shown to catch them. This is not a passing score.**"
)
if stats:
return "**Not one mutant was killed. This is not a passing score.**"
return (
f"**mutmut-cicd-stats.json is missing and `mutmut results` printed {results.reported} "
"verdict(s), none of them a survivor. Since that command never lists killed mutants, a "
"clean sweep and a run that mutated nothing look identical from here. This is not a "
"passing score.**"
)
def render(config: dict, results: MutmutResults, stats: dict | None) -> str:
survivors = list(results.survivors)
by_function: dict[tuple[str, str], list[tuple[str, str]]] = defaultdict(list)
for survivor in survivors:
module_path, function_name, mutant_num = parse_mutant_name(survivor)
@ -235,17 +289,8 @@ def render(config: dict, survivors: list[str], stats: dict | None) -> str:
out.append("## Summary")
out.append("")
if stats:
total = stats.get("total", 0) or sum(
stats.get(k, 0)
for k in (
"killed",
"survived",
"no_tests",
"skipped",
"suspicious",
"timeout",
"segfault",
)
total = stats.get("total", 0) or (
stats.get("killed", 0) + stats.get("survived", 0) + sum(unresolved_counts(stats).values())
)
killed = stats.get("killed", 0)
survived = stats.get("survived", 0)
@ -254,17 +299,15 @@ def render(config: dict, survivors: list[str], stats: dict | None) -> str:
out.append(f"- Killed: **{killed}**")
out.append(f"- Survived: **{survived}**")
out.append(f"- Mutation score: **{score:.1f}%**")
for k in ("no_tests", "skipped", "suspicious", "timeout", "segfault"):
v = stats.get(k, 0)
if v:
out.append(f"- {k.replace('_', ' ').title()}: {v}")
for k, v in unresolved_counts(stats).items():
out.append(f"- {k.replace('_', ' ').title()}: {v}")
else:
out.append(f"- Survivors found: **{len(survivors)}**")
out.append("- (mutmut-cicd-stats.json not available — full counts unavailable)")
out.append("")
if not survivors:
out.append("**No surviving mutants — the test suite caught every mutation.**")
out.append(no_survivors_verdict(results, stats))
out.append("")
return "\n".join(out)
@ -407,15 +450,22 @@ def main() -> int:
except json.JSONDecodeError as exc:
print(f"warning: could not parse {stats_file}: {exc}", file=sys.stderr)
survivors = get_survivors()
report = render(config, survivors, stats)
results = get_survivors()
report = render(config, results, stats)
out_path = ROOT / "mutation-report.md"
out_path.write_text(report)
print(
f"Wrote {out_path} ({len(survivors)} survivor"
f"{'s' if len(survivors) != 1 else ''}, {len(report)} chars)"
f"Wrote {out_path} ({len(results.survivors)} survivor"
f"{'s' if len(results.survivors) != 1 else ''}, {len(report)} chars)"
)
if not results.survivors and not clean_sweep_is_provable(stats):
print(
"error: nothing was shown to have been killed, so the report cannot say "
"anything about the suite",
file=sys.stderr,
)
return 1
return 0

1
terraform/provider/.goreleaser.yml
View file

@ -72,6 +72,7 @@ signs:
- "--detach-sign"
- "${artifact}"
release:
prerelease: auto
extra_files:
- glob: 'terraform-registry-manifest.json'
name_template: '{{ .ProjectName }}_{{ .Version }}_manifest.json'

15
terraform/provider/CHANGELOG.md
View file

@ -2,11 +2,22 @@
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
Up to `0.4.0` the provider had its own version line, cut from the headings in
this file. It now ships at the **LiteLLM version**, on every LiteLLM release
channel, built from the same commit as the proxy (see `RELEASING.md`). The
headings below no longer drive a release; they record what changed and which
LiteLLM line first carried it. A change that breaks existing configurations
or state must be called out loudly here, because the version number can no
longer signal it.
## [Unreleased]
### Changed
- **Versioning**: the provider is now published at the LiteLLM version, from the same commit as the proxy, on every LiteLLM release (dev, rc, stable). The `0.x` line ends at `0.4.0`; a `~> 0.4` constraint will not receive further releases, so re-pin to the LiteLLM version your proxy runs (for example `~> 1.99.0`). Existing `0.x` versions remain in the registry and keep verifying
## [0.4.0] - 2026-08-06
### Fixed

16
terraform/provider/README.md
View file

@ -6,6 +6,18 @@ This Terraform provider allows you to manage LiteLLM resources through Infrastru
This directory (`terraform/provider/` in [BerriAI/litellm](https://github.com/BerriAI/litellm)) is the source of truth for the provider. [BerriAI/terraform-provider-litellm](https://github.com/BerriAI/terraform-provider-litellm) is a thin release mirror that the public Terraform Registry ingests from; do not open PRs there. Changes land here, where CI builds the provider, runs its tests, and statically audits every endpoint the provider calls against the proxy's generated OpenAPI schema (`tools/endpointaudit/`), so the provider cannot drift from the LiteLLM API silently. Releases are published by mirroring this directory into the split repo and tagging it, which triggers the goreleaser workflow there (see `RELEASING.md`)
## Versioning
The provider version **is the LiteLLM version**. Every LiteLLM release (dev, rc and stable) publishes the provider at the same version as the proxy, built from the same commit, so `1.99.0` of the provider is the one that shipped with `1.99.0` of the proxy and was audited against that proxy's API. Pin the provider to the line your proxy runs:
```hcl
version = "~> 1.99.0"
```
Pre-release versions (`1.99.0-rc.1`, `1.99.0-dev.1`) are published too; Terraform only selects one when it is pinned exactly.
Versions `0.1.0` through `0.4.0` predate this scheme and sit on their own line. They stay in the registry, but **a `~> 0.4` constraint will never pick up another release**: re-pin to the LiteLLM version to keep receiving updates.
## Features
- Manage LiteLLM model configurations
@ -32,7 +44,7 @@ terraform {
required_providers {
litellm = {
source = "BerriAI/litellm"
version = "~> 0.1.1" #HERE UPDATE VERSION ACCORDINGLY
version = "~> 1.99.0" # the LiteLLM version your proxy runs
}
}
}
@ -218,6 +230,6 @@ This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENS
- Always use environment variables or secure secret management solutions to handle sensitive information like API keys and AWS credentials.
- Refer to the comprehensive documentation in the `docs/` directory for detailed usage examples and configuration options.
- Make sure to keep your provider version updated for the latest features and bug fixes.
- Keep the provider version in step with the LiteLLM version your proxy runs; see [Versioning](#versioning).
- The provider now supports AWS cross-account access with `aws_session_name` and `aws_role_name` parameters in the model resource.
- All example configurations have been consolidated into the documentation for better organization and maintenance.

139
terraform/provider/RELEASING.md
View file

@ -4,7 +4,16 @@ This document describes the release process for the LiteLLM Terraform Provider.
## Overview
Releases are automated via GitHub Actions when a version tag is pushed. The workflow builds the provider for multiple platforms, signs the artifacts with GPG, and publishes them to GitHub Releases.
The provider is released **in lockstep with LiteLLM**: every LiteLLM release (dev, rc and stable) publishes the provider at the LiteLLM version, built from the same commit as the proxy. There is no separate provider release to cut.
The flow, end to end:
1. `BerriAI/project-releaser`'s release pipeline resolves the commit to release (`main` HEAD for dev; `main` HEAD or an operator-supplied SHA for rc/stable) and passes the release approval gate
2. Its componentized terraform job rsyncs `terraform/provider/` from that commit into `BerriAI/terraform-provider-litellm`, commits, and pushes the tag `v<litellm version>` (for example `v1.99.0`, `v1.99.0-rc.1`, `v1.99.0-dev.1`), alongside the `terraform-aws-litellm` / `terraform-google-litellm` module mirrors which get the same tag
3. The tag push triggers the mirror's own `Release` workflow (goreleaser): multi-platform build, GPG-signed checksums, GitHub release. It runs unattended; project-releaser does not wait for it
4. The public Terraform Registry ingests the GitHub release as provider version `<litellm version>`
`terraform/provider/` only exists from LiteLLM ~1.95, so a stable patch cut from an older line skips the provider and publishes only the modules.
## Prerequisites
@ -68,113 +77,26 @@ Before publishing to the Terraform Registry:
**Note**: The public key fingerprint must match the key used to sign the provider releases.
## Release Steps
## What a change needs
### 1. Prepare the Release
1. **Land it in `BerriAI/litellm`.** Open a PR against `litellm_internal_staging` with the source change and a `CHANGELOG.md` entry under `[Unreleased]`. CI runs `gofmt`, `go vet`, build, tests and the endpoint-drift audit. A change that breaks existing configurations or state must say so in the changelog: the version number cannot signal it any more
2. **Wait for the next LiteLLM release.** The nightly dev release carries it within a day; it reaches a stable version on the next stable cut
3. **Verify** (optional): the version appears at https://registry.terraform.io/providers/BerriAI/litellm and https://github.com/BerriAI/terraform-provider-litellm/releases. If the tag is on the mirror but there is no release, the goreleaser run failed: https://github.com/BerriAI/terraform-provider-litellm/actions
Before creating a release:
Locally, before opening the PR:
1. **Update CHANGELOG.md**
- Move items from `[Unreleased]` section to a new version section
- Follow [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format
- Use [Semantic Versioning](https://semver.org/spec/v2.0.0.html) for version numbers
- Include all notable changes since the last release
```bash
make test
make build
```
Example:
```markdown
## [0.1.2] - 2026-02-20
## Out-of-band publish or recovery
### Added
- New feature description
Dispatch `Build and Publish Componentized Images + Chart` in `BerriAI/project-releaser` by hand with only `publish_terraform` enabled and the `git_ref` / `tag` of the release to (re)publish. The run waits on project-releaser's release approval, then mirrors and tags exactly as the pipeline does.
### Fixed
- Bug fix description
The mirror is push-only: do not commit or tag `BerriAI/terraform-provider-litellm` directly. The publish refuses to overwrite an existing tag; a version that failed in goreleaser is recovered by re-running the mirror's `Release` workflow for that tag, not by re-tagging.
### Changed
- Changed behavior description
```
2. **Verify tests pass**
```bash
make test
```
3. **Verify the build works locally**
```bash
make build
```
4. **Land the changes in BerriAI/litellm**
Open a PR to `BerriAI/litellm` updating `terraform/provider/CHANGELOG.md` (and any source changes) and merge it
### 2. Mirror and Tag via project-releaser
The provider source lives at `terraform/provider/` in `BerriAI/litellm`; `BerriAI/terraform-provider-litellm` is a thin release mirror. Do not commit or tag the mirror directly
Normally there is nothing to do here. `BerriAI/project-releaser`'s release pipeline runs the same check on every release except `adhoc`, nightly included: it reads the topmost released heading in `terraform/provider/CHANGELOG.md`, probes the mirror for `v<version>`, and dispatches `Publish Terraform provider` only when the changelog has moved ahead of what the mirror carries. Cutting the version heading in step 1 is therefore what releases the provider, and the next release picks it up, so the wait is a day rather than a week
Dispatch by hand only for an out-of-band release, or to recover a run that failed:
1. Go to `BerriAI/project-releaser` > **Actions** > `Publish Terraform provider`
2. Click **Run workflow**:
- `git_ref`: full 40-char commit SHA from `BerriAI/litellm` to release from
- `provider_version`: the new version without the `v` prefix (e.g. `0.3.0`)
- `dry_run`: optional; validates without pushing
Automatic or manual, the run waits on the `production-release` approval in `project-releaser`, then rsyncs `terraform/provider/` into the mirror repo, commits, and pushes tag `v<provider_version>`. That approval is the only one in the flow. The tag push triggers the mirror's `Release` workflow (goreleaser), which runs unattended
**Important**:
- Tags must follow the format: `v<MAJOR>.<MINOR>.<PATCH>` (e.g., `v0.1.2`, `v1.0.0`)
- The workflow refuses to overwrite an existing tag; publish a new version instead
### 3. Monitor the Release Workflow
1. Go to: https://github.com/BerriAI/terraform-provider-litellm/actions
2. Find the "Release" workflow run for your tag
3. Monitor the progress and check for any errors
The workflow will:
- Check out the code
- Set up Go
- Import the GPG key
- Run `go mod tidy`
- Build binaries for multiple platforms (Linux, macOS, Windows, FreeBSD)
- Create archives and checksums
- Sign the checksums with GPG
- Create a GitHub release
- Upload all artifacts
### 4. Verify the Release
After the workflow completes successfully:
1. **Check the GitHub Release**
- Go to: https://github.com/BerriAI/terraform-provider-litellm/releases
- Verify the release was created with the correct version
- Confirm all artifacts are present:
- Binary archives for each platform
- SHA256SUMS file
- SHA256SUMS.sig (GPG signature)
- terraform-registry-manifest.json
2. **Verify the signature** (optional)
```bash
# Download the checksums and signature
wget https://github.com/BerriAI/terraform-provider-litellm/releases/download/v0.1.2/terraform-provider-litellm_0.1.2_SHA256SUMS
wget https://github.com/BerriAI/terraform-provider-litellm/releases/download/v0.1.2/terraform-provider-litellm_0.1.2_SHA256SUMS.sig
# Verify the signature
gpg --verify terraform-provider-litellm_0.1.2_SHA256SUMS.sig terraform-provider-litellm_0.1.2_SHA256SUMS
```
### 5. Publish to Terraform Registry (Optional)
If this provider is published to the Terraform Registry:
1. The registry should automatically detect the new release via the GitHub webhook
2. If not, you may need to manually trigger a sync on the Terraform Registry dashboard
3. Verify the new version appears at: https://registry.terraform.io/providers/BerriAI/litellm/latest
The mirror's `.github/` directory (the `Release` workflow) is the one thing the rsync preserves, so a change to the goreleaser *workflow* is a direct PR on the mirror; a change to `.goreleaser.yml` itself lands here like any other source change.
## Troubleshooting
@ -207,21 +129,15 @@ If this provider is published to the Terraform Registry:
### Tag Already Exists
**Error**: The publish workflow refuses to push because the tag already exists on the mirror
**Error**: The publish job refuses to push because the tag already exists on the mirror
**Solution**: Tags are immutable by design. Re-run the workflow with a new patch version instead of deleting or moving an existing tag
**Solution**: Tags are immutable by design and the version is the LiteLLM version, so this means the provider was already mirrored for this release. If the registry is missing the version, re-run the mirror's `Release` workflow for the existing tag rather than re-tagging
## Version Numbering
This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html):
The provider version is the LiteLLM version, verbatim: `X.Y.Z` for a stable release, `X.Y.Z-rc.N` for a release candidate and `X.Y.Z-dev.N` for a nightly. It says which proxy the provider shipped with and was audited against; it does not follow SemVer's break-signalling, so breaking changes are announced in `CHANGELOG.md` and the registry docs instead.
- **MAJOR** version (1.0.0): Incompatible API changes
- **MINOR** version (0.1.0): New functionality in a backward-compatible manner
- **PATCH** version (0.0.1): Backward-compatible bug fixes
For pre-1.0 releases:
- Breaking changes may occur in minor versions
- Patch versions should only contain bug fixes
Versions `0.1.0` to `0.4.0` predate this and remain in the registry on their own line. A `~> 0.4` constraint never receives another release.
## Security Considerations
@ -237,5 +153,4 @@ For pre-1.0 releases:
- [Terraform Provider Publishing](https://www.terraform.io/docs/registry/providers/publishing.html)
- [HashiCorp GPG Signing Requirements](https://www.terraform.io/docs/registry/providers/publishing.html#signing-releases)
- [GitHub Actions Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets)
- [Semantic Versioning](https://semver.org/)
- [Keep a Changelog](https://keepachangelog.com/)

8
test-quality-budget.json
View file

@ -1,18 +1,18 @@
{
"TQ001": {
"limit": 750
"limit": 744
},
"TQ002": {
"limit": 742
},
"TQ003": {
"limit": 1078
"limit": 62
},
"TQ004": {
"limit": 757
"limit": 469
},
"TQ005": {
"limit": 2810
"limit": 2405
},
"TQ006": {
"limit": 34

5
tests/agent_tests/local_only_agent_tests/test_a2a.py
View file

@ -6,8 +6,6 @@ Run with:
"""
import asyncio
import os
import sys
import json
from typing import Optional
from uuid import uuid4
@ -18,9 +16,6 @@ import litellm
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.utils import StandardLoggingPayload
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
from a2a.types import MessageSendParams, SendMessageRequest

3
tests/agent_tests/local_only_agent_tests/test_a2a_completion_bridge.py
View file

@ -10,13 +10,10 @@ Prerequisites:
- LangGraph server running on localhost:2024
"""
import os
import sys
from uuid import uuid4
import pytest
sys.path.insert(0, os.path.abspath("../.."))
import litellm
from a2a.types import MessageSendParams, SendMessageRequest, SendStreamingMessageRequest

3
tests/audio_tests/conftest.py
View file

@ -1,9 +1,6 @@
import os
import sys
import pytest
sys.path.insert(0, os.path.abspath("../.."))
from tests._vcr_conftest_common import ( # noqa: E402,F401
VerboseReporterState,

4
tests/audio_tests/test_audio_speech.py
View file

@ -4,7 +4,6 @@
import asyncio
import os
import random
import sys
import time
import traceback
from litellm._uuid import uuid
@ -13,9 +12,6 @@ from dotenv import load_dotenv
load_dotenv()
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch

4
tests/audio_tests/test_whisper.py
View file

@ -4,7 +4,6 @@
import asyncio
import logging
import os
import sys
import time
import traceback
from typing import Optional
@ -41,9 +40,6 @@ def _audio_file2():
load_dotenv()
sys.path.insert(
0, os.path.abspath("../")
) # Adds the parent directory to the system path
from litellm import Router

5
tests/batches_tests/conftest.py
View file

@ -1,12 +1,7 @@
import asyncio
import os
import sys
import pytest
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm # noqa: E402,F401
from tests._vcr_conftest_common import ( # noqa: E402,F401

4
tests/batches_tests/test_batch_rate_limits.py
View file

@ -5,14 +5,10 @@ Integration Tests for Batch Rate Limits
import asyncio
import json
import os
import sys
import pytest
from fastapi import HTTPException
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm
from litellm import DualCache

5
tests/batches_tests/test_batches_logging_unit_tests.py
View file

@ -1,15 +1,10 @@
import asyncio
import json
import os
import sys
import traceback
from unittest.mock import AsyncMock, MagicMock, patch
from dotenv import load_dotenv
load_dotenv()
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system-path
import logging
import time

4
tests/batches_tests/test_bedrock_files_and_batches.py
View file

@ -3,15 +3,11 @@
import asyncio
import json as json_module
import os
import sys
import traceback
import tempfile
from dotenv import load_dotenv
load_dotenv()
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system-path
import pytest

5
tests/batches_tests/test_fine_tuning_api.py
View file

@ -1,12 +1,7 @@
import os
import sys
import traceback
import json
import pytest
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
from openai import APITimeoutError as Timeout
import litellm

61
tests/batches_tests/test_openai_batches_and_files.py
View file

@ -3,14 +3,10 @@
import asyncio
import json
import os
import sys
import tempfile
from dotenv import load_dotenv
load_dotenv()
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system-path
import logging
import time
@ -103,6 +99,25 @@ def load_vertex_ai_credentials():
print("created gcs path service account=", os.environ["GCS_PATH_SERVICE_ACCOUNT"])
async def cancel_batch_unless_already_terminal(batch_id: str, provider: str) -> None:
try:
cancel_batch_response = await litellm.acancel_batch(batch_id=batch_id, custom_llm_provider=provider)
except openai.ConflictError as e:
if "Cannot cancel a batch with status 'completed'" in str(e):
print(f"Batch already completed, cannot cancel: {e}")
return
if "Cannot cancel a batch with status 'failed'" not in str(e):
raise
failed_batch = await litellm.aretrieve_batch(batch_id=batch_id, custom_llm_provider=provider)
print(f"Batch failed before cancel, errors={failed_batch.errors}")
failure_codes = {err.code for err in (failed_batch.errors.data if failed_batch.errors else None) or []}
assert failure_codes == {"token_limit_exceeded"}, (
f"batch failed for a reason other than the org's enqueued token limit: {failed_batch.errors}"
)
return
print("cancel_batch_response=", cancel_batch_response)
@pytest.mark.parametrize("provider", ["openai"]) # , "azure"
@pytest.mark.asyncio
@skip_if_no_openai_network
@ -176,24 +191,7 @@ async def test_create_batch(provider, tmp_path):
result_file_path = tmp_path / "batch_job_results_furniture.jsonl"
result_file_path.write_bytes(result)
# Cancel Batch - handle race condition where batch may already be completed
try:
cancel_batch_response = await litellm.acancel_batch(
batch_id=create_batch_response.id,
custom_llm_provider=provider,
)
print("cancel_batch_response=", cancel_batch_response)
except openai.ConflictError as e:
# Only allow to pass if it's specifically the "batch already completed" error
if "Cannot cancel a batch with status 'completed'" in str(e):
print(f"Batch already completed, cannot cancel: {e}")
else:
# Re-raise other ConflictError types
raise
except Exception as e:
# Re-raise any other unexpected errors
print(f"Unexpected error during batch cancellation: {e}")
raise
await cancel_batch_unless_already_terminal(batch_id=create_batch_response.id, provider=provider)
pass
@ -395,24 +393,7 @@ async def test_async_create_batch(provider, tmp_path):
result_file_path = tmp_path / "batch_job_results_furniture.jsonl"
result_file_path.write_bytes(file_content.content)
# Cancel Batch - handle race condition where batch may already be completed
try:
cancel_batch_response = await litellm.acancel_batch(
batch_id=create_batch_response.id,
custom_llm_provider=provider,
)
print("cancel_batch_response=", cancel_batch_response)
except openai.ConflictError as e:
# Only allow to pass if it's specifically the "batch already completed" error
if "Cannot cancel a batch with status 'completed'" in str(e):
print(f"Batch already completed, cannot cancel: {e}")
else:
# Re-raise other ConflictError types
raise
except Exception as e:
# Re-raise any other unexpected errors
print(f"Unexpected error during batch cancellation: {e}")
raise
await cancel_batch_unless_already_terminal(batch_id=create_batch_response.id, provider=provider)
mock_file_response = {

2
tests/code_coverage_tests/bedrock_pricing.py
View file

@ -1,7 +1,5 @@
import os
import sys
sys.path.insert(0, os.path.abspath("../.."))
import litellm
import requests
from bs4 import BeautifulSoup

2
tests/code_coverage_tests/check_spanattributes_value_usage.py
View file

@ -27,10 +27,8 @@ import ast
import os
import re
from typing import List, Tuple
import sys
# Add parent directory to path so we can import litellm
sys.path.insert(0, os.path.abspath("../.."))
import litellm

2
tests/code_coverage_tests/enforce_llms_folder_style.py
View file

@ -1,8 +1,6 @@
import ast
import os
import sys
sys.path.insert(0, os.path.abspath("../.."))
import litellm

5
tests/code_coverage_tests/test_router_strategy_async.py
View file

@ -4,14 +4,9 @@ Test that all cache calls in async functions in router_strategy/ are async
"""
import os
import sys
from typing import Dict, List, Tuple
import ast
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import os
class AsyncCacheCallVisitor(ast.NodeVisitor):

4
tests/documentation_tests/test_api_docs.py
View file

@ -4,11 +4,7 @@ import os
from dataclasses import dataclass
import argparse
import re
import sys
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm

3
tests/documentation_tests/test_exception_types.py
View file

@ -11,9 +11,6 @@ import re
# Backup the original sys.path
original_sys_path = sys.path.copy()
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm
public_exceptions = litellm.LITELLM_EXCEPTION_TYPES

4
tests/documentation_tests/test_router_settings.py
View file

@ -2,11 +2,7 @@ import os
import re
import inspect
from typing import Type
import sys
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm

5
tests/documentation_tests/test_standard_logging_payload.py
View file

@ -1,12 +1,7 @@
import os
import re
import sys
from typing import get_type_hints
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
from litellm.types.utils import StandardLoggingPayload

17
tests/e2e/CLAUDE.md
View file

@ -77,13 +77,26 @@ Mark live tests with `@pytest.mark.e2e` (on the class or the module). Pure cover
The seam is `provider_edge.py`: `start_provider_edge` boots an in-process HTTP server (one shared instance per pytest process, `e2e_config.provider_edge_base` is the accessor) that mounts each supported provider under a path prefix (`EDGE_MOUNTS`: `/openai` -> `https://api.openai.com`, `/anthropic` -> `https://api.anthropic.com`). A test participates by registering its deployment with `api_base=provider_edge_base("openai")` plus the provider's path suffix; `quota_management/spend_tracking/test_provider_edge_spend_e2e.py` is the reference. In live mode the accessor returns None and the deployment defaults to the real provider, so an edge-wired test runs in all three modes unchanged. Non-wired tests hit their providers live in every mode. The edge binds `E2E_PROVIDER_EDGE_BIND_HOST` (default 127.0.0.1) and advertises `E2E_PROVIDER_EDGE_ADVERTISE_HOST` in the api_base it hands out, for proxies running in containers
A bundle (default `tests/e2e/.fixtures`, override with `E2E_FIXTURE_DIR`) is a directory: `manifest.json` carries the record timestamp, harness git version, and format version, and each test gets a subdirectory holding one JSON file per provider call in call order (`0000-post-openai-v1-chat-completions.json`). Request headers are never stored (provider credentials never touch disk), non-JSON request bodies store a canonicalized sha256 digest instead of the bytes, and responses store status, filtered headers, and the verbatim body base64-encoded, which is part of why bundles are gitignored. `fixture_bundle.py` owns the format. Record serves the proxy the same filtered stored response replay will serve later, so the two modes are byte-identical from the proxy's side of the socket
A bundle (default `tests/e2e/.fixtures`, override with `E2E_FIXTURE_DIR`) is a directory: `manifest.json` carries the record timestamp, harness git version, and format version, and each test gets a subdirectory holding one JSON file per provider call in call order (`0000-post-openai-v1-chat-completions.json`). Request headers are never stored (provider credentials never touch disk), non-JSON request bodies store a canonicalized sha256 digest instead of the bytes, `multipart/form-data` bodies store their ordinary fields plus a JSON list of the uploaded parts' `[field, filename, content-type]` triples and a digest of their content, so the per-request random boundary and the envelope never reach the key, and responses store status, filtered headers, and the verbatim body base64-encoded, which is part of why bundles are gitignored. `fixture_bundle.py` owns the format. Record serves the proxy the same filtered stored response replay will serve later, so the two modes are byte-identical from the proxy's side of the socket
Multipart identity is the fiddly corner, and the rules exist because each one had a collision behind it. A part counts as an upload when it carries a filename or declares its own content type, and everything else is an ordinary field. Field names get a `name[n]` suffix on repeats, with a literal `[` doubled first, so a form that repeats `purpose` never keys the same as one that literally sends `purpose[1]`. A field whose name reads as a credential is stored as `<secret>`, which stays key-preserving because the key is recomputed from the stored request rather than saved alongside it, so the live request carrying the real value still matches its redacted fixture. A field value that is not UTF-8 is stored as a base64 sha256 digest, base64 and not hex because the canonicalizer rewrites any 64-character hex run to `<sha256>` and would fold every binary value onto one key. The uploaded parts contribute a JSON list rather than a `field:filename` string, so a separator inside a filename cannot impersonate a field boundary, and their byte length is stored for a reader's benefit but deliberately left out of the key, since the canonicalizer absorbs timestamp and id drift inside a file that changes its length
Replay matches calls per test by canonical key: `fixture_canonical.py` canonicalizes the recorded request (volatile headers and credential fields out, unique markers, generated ids, uuids, and timestamps replaced with fixed placeholders, object keys sorted) and the key is the method, edge path, and a content hash, so identity survives re-records and machine changes while any real content drift comes back as an HTTP 599 naming the computed key, the closest recorded key with its file, and a content diff, and never falls through to a live call. Matching is order-independent across distinct keys (concurrent calls may interleave) and FIFO within one key (a retry loop replays its responses in recorded order); a passed test must also consume its whole recording, or teardown fails it naming a leftover key. Either way the fix is always to re-record with `E2E_FIXTURE_MODE=record`. Every rewrite rule lives in `fixture_canonical.py`, so a new volatile header, credential field name, or generated-id shape is one edit there. Record starts fresh every time: it wipes the previous bundle (refusing to wipe a directory that is not a bundle) and never reads it. A replay bundle whose manifest is older than seven days hard-fails at collection time naming the bundle's age, so replay can never certify against fixtures that have drifted more than a week from the live providers
A replayed response carries the recorded provider response id, and `LiteLLM_SpendLogs.request_id` (the table's primary key) is that id, so a replay against a database that still holds the record run's rows silently dedupes its spend inserts and any spend assertion goes red with zero matching rows and nothing in the proxy log. Run both modes with `E2E_RESET_SPEND_LOGS=1` (plus `DATABASE_URL` in the runner env) so each session truncates the table after itself, or replay against a fresh database, which is the CI shape
Current limits: streaming chunk fidelity is LIT-5742 (a streamed response records as one buffered body), CI wiring is LIT-5748, Bedrock cannot be mounted (SigV4 signs the Host header, so a rewritten api_base fails signature verification), multipart uploads have per-run random boundaries (the digest changes every run, so they always miss), and deployments baked into the proxy's config file cannot be edge-wired (only `/model/new` registrations can carry the edge api_base)
The same id reuse reaches the managed-object tables. A replayed `/v1/files` or `/v1/batches` response carries the recorded provider object id, and `LiteLLM_ManagedObjectTable.model_object_id` is unique, so a unified batch create replayed against a database that still holds the record run's row fails on a Prisma unique-constraint violation, which surfaces as a 500, makes the router retry, and exhausts the recording. Replay the batches suite against a fresh database, or truncate `LiteLLM_ManagedObjectTable` and `LiteLLM_ManagedFileTable` before the run
Edge-wired today: `quota_management/spend_tracking/test_provider_edge_spend_e2e.py` (the reference), `llm_translation/test_chat_completions_contract_e2e.py`, the OpenAI registrations in `llm_translation/test_embeddings_endpoint_e2e.py`, the Anthropic deployments in `llm_translation/test_messages_e2e.py` except the streaming test, and the OpenAI batch deployment behind `batches/` (`capabilities.openai_batch_params`). The mount base is not the same for both providers: OpenAI deployments register `f"{base}/v1"`, Anthropic deployments register `base` on its own, because litellm's Anthropic handler appends `/v1/messages` to `api_base` itself where the OpenAI handler appends only `/chat/completions`. Recording one suite locally is two runs against a proxy you already have up:
```bash
E2E_FIXTURE_MODE=record E2E_FIXTURE_DIR=/tmp/e2e-fixtures E2E_RESET_SPEND_LOGS=1 uv run pytest tests/e2e/llm_translation/test_chat_completions_contract_e2e.py
E2E_FIXTURE_MODE=replay E2E_FIXTURE_DIR=/tmp/e2e-fixtures E2E_RESET_SPEND_LOGS=1 uv run pytest tests/e2e/llm_translation/test_chat_completions_contract_e2e.py
```
Point the proxy at bogus provider credentials for the replay run and it still has to pass: that is the whole proof that nothing left the process. Bundles are never committed. `tests/e2e/.fixtures` is gitignored because a bundle holds verbatim provider response bodies and hard-fails after seven days, and publishing one for CI is LIT-5748
Current limits: streaming chunk fidelity is LIT-5742 (a streamed response records as one buffered body), CI wiring is LIT-5748, Bedrock cannot be mounted (SigV4 signs the Host header, so a rewritten api_base fails signature verification), deployments baked into the proxy's config file cannot be edge-wired (only `/model/new` registrations can carry the edge api_base), and a file upload routed by `custom_llm_provider` through the proxy's `files_settings` block never passes a deployment at all, so the batches `model_param` and `provider_fallback` scenarios keep uploading live in every mode
## Typing

8
tests/e2e/CONTRIBUTING.md
View file

@ -57,13 +57,15 @@ Some suites need extra services the bare proxy does not start. The `logging/` OT
Record/replay scopes to the proxy's provider-bound traffic only. In `E2E_FIXTURE_MODE=record` the harness boots a local provider-edge server, edge-wired tests register their deployments with an `api_base` pointing at it, and every provider call the proxy makes is forwarded verbatim and written to a fixture bundle (default `tests/e2e/.fixtures`, override with `E2E_FIXTURE_DIR`). `E2E_FIXTURE_MODE=replay` runs the same tests against the same live proxy and database, but the edge answers the proxy's provider calls from the bundle instead of the provider, so the run makes zero provider calls and spends nothing while key auth, routing, cost calculation, and spend-log writes all still execute for real. Unset (or `live`) behaves exactly as before the knob existed. Both record and replay need the proxy up; only the provider is taken out of the loop
```bash
E2E_FIXTURE_MODE=record uv run pytest tests/e2e/quota_management/spend_tracking/test_provider_edge_spend_e2e.py -v
E2E_FIXTURE_MODE=replay uv run pytest tests/e2e/quota_management/spend_tracking/test_provider_edge_spend_e2e.py -v
E2E_FIXTURE_MODE=record E2E_FIXTURE_DIR=/tmp/e2e-fixtures uv run pytest tests/e2e/quota_management/spend_tracking/test_provider_edge_spend_e2e.py -v
E2E_FIXTURE_MODE=replay E2E_FIXTURE_DIR=/tmp/e2e-fixtures uv run pytest tests/e2e/quota_management/spend_tracking/test_provider_edge_spend_e2e.py -v
```
Bundles stay local. `tests/e2e/.fixtures` is gitignored because a bundle holds verbatim provider response bodies and expires seven days after it was recorded, so record the suite you want before you replay it and never commit the result; publishing bundles for CI is LIT-5748
One sharp edge: a replayed response reuses the recorded provider response id, and that id is the primary key of `LiteLLM_SpendLogs`, so replaying against a database that still holds the record run's rows silently dedupes the spend writes and a spend assertion fails with zero rows. Run both commands above with `E2E_RESET_SPEND_LOGS=1` (and `DATABASE_URL` set in the pytest env) so each session truncates the spend log table after itself, or point replay at a fresh database
Replay answers any provider call that drifted from the recording with an HTTP 599 whose body names the computed and closest recorded keys, so the test fails loudly instead of silently going live, and a bundle older than seven days fails at collection time naming its age; either way the fix is to re-record. Only tests that register edge-wired deployments participate: everything else hits its provider live in every mode, so record exactly the suite you replay. If the proxy runs in a container, set `E2E_PROVIDER_EDGE_ADVERTISE_HOST` (e.g. `host.docker.internal`) so the api_base the proxy stores can reach the edge on the pytest host, and `E2E_PROVIDER_EDGE_BIND_HOST=0.0.0.0` so the edge accepts it. See `CLAUDE.md` in this directory for the bundle format, the edge design, and the current limits (streaming, Bedrock, multipart)
Replay answers any provider call that drifted from the recording with an HTTP 599 whose body names the computed and closest recorded keys, so the test fails loudly instead of silently going live, and a bundle older than seven days fails at collection time naming its age; either way the fix is to re-record. Only tests that register edge-wired deployments participate: everything else hits its provider live in every mode, so record exactly the suite you replay. If the proxy runs in a container, set `E2E_PROVIDER_EDGE_ADVERTISE_HOST` (e.g. `host.docker.internal`) so the api_base the proxy stores can reach the edge on the pytest host, and `E2E_PROVIDER_EDGE_BIND_HOST=0.0.0.0` so the edge accepts it. The suites wired to the edge today are `quota_management/spend_tracking/test_provider_edge_spend_e2e.py`, `llm_translation/test_chat_completions_contract_e2e.py`, the OpenAI registrations in `llm_translation/test_embeddings_endpoint_e2e.py`, the non-streaming Anthropic tests in `llm_translation/test_messages_e2e.py`, and the OpenAI batch deployment behind `batches/`. See `CLAUDE.md` in this directory for the bundle format, the edge design, and the current limits (streaming, Bedrock)
Tests marked `@pytest.mark.e2e` hard-fail when no proxy answers `/health/liveliness`, so a run that goes red with `No live proxy` at setup means the proxy isn't up; they never skip for a missing proxy, so an absent proxy can't be mistaken for a pass

87
tests/e2e/batches/COVERAGE.md
View file

@ -1,9 +1,11 @@
# Batches Test Coverage Matrix
Live e2e coverage of the Batches API over a real proxy, real provider keys, and
real cost. Synchronous tier only: a batch's completion window is 24h, so these
tests never wait for `completed`. They assert the proxy accepts, routes, retrieves,
cancels, and lists a batch; everything created is deleted on teardown.
real cost. Mostly synchronous tier: a batch's completion window is 24h, so the
lifecycle matrix never waits for `completed`. It asserts the proxy accepts, routes,
retrieves, cancels, and lists a batch; everything created is deleted on teardown.
The exception is `TestBatchTerminalState`, which covers the completed state and
cost write-back via a cross-run marker baton (design below).
## Provider x operation
@ -12,19 +14,26 @@ row per supported (provider, scenario) pair, so there are no skipped cells in th
parametrized run. The batches suite never skips: missing provider creds or upstream
failures are hard test failures (see `tests/e2e/CLAUDE.md`).
| Provider | create | retrieve | cancel | list | file backing |
|-----------|--------|----------|--------|------|--------------|
| OpenAI | yes | yes | yes | yes | OpenAI Files |
| Azure | yes | yes | yes | yes | Azure Files |
| Vertex AI | yes | yes | yes | yes | GCS (`gcs_bucket_name` / `GCS_BUCKET_NAME` on model) |
| Bedrock | yes (unified only) | yes | no (limited upstream) | no | S3 (`s3_bucket_name` + `aws_*` + `AWS_BATCH_ROLE_ARN` on model) |
| Provider | create | retrieve | cancel | list | content download | file backing |
|-----------|--------|----------|--------|------|------------------|--------------|
| OpenAI | yes | yes | yes | yes | yes (lifecycle + terminal output) | OpenAI Files |
| Azure | yes | yes | yes | yes | yes (byte-verbatim) | Azure Files |
| Vertex AI | yes | yes | yes | yes | yes (provider-transformed) | GCS (`gcs_bucket_name` / `GCS_BUCKET_NAME` on model) |
| Bedrock | yes (unified only) | yes | no (limited upstream) | no | yes (provider-transformed) | S3 (`s3_bucket_name` + `aws_*` + `AWS_BATCH_ROLE_ARN` on model) |
Bedrock cancel is unreliable upstream and list is unsupported, so both are gated off
(`can_cancel=False`, `can_list=False`) when that provider is enabled in the matrix.
(`can_cancel=False`, `can_list=False`) when that provider is enabled in the matrix;
flipping those gates is tracked in LIT-4774 and deliberately not part of this suite.
Bedrock file upload requires a model on the request (`encoded` / `unified` scenarios only);
`model_param` and `provider_fallback` are omitted because `POST /bedrock/v1/files` has no
model-less passthrough path.
`GET /v1/files/{id}/content` is exercised for the unified upload path per backend in
`test_unified_file_content_downloads`. Azure stores the JSONL verbatim, so its download
is asserted byte-equal to the upload. Vertex (GCS) and Bedrock (S3) transform lines at
upload time, so those assert a 200 with non-empty parseable JSON lines instead. Gemini
(non-Vertex) raises `NotImplementedError` for file content and has no cell here.
## Routing scenarios (per `litellm/proxy/batches_endpoints/endpoints.py`)
Each create-capable provider runs all four. The test asserts the returned file id
@ -71,11 +80,59 @@ File delete asserts `object=="file"` and `deleted==True`.
| `batch_client.py` | typed file upload/download + batch create/retrieve/cancel/list/delete over the shared ProxyClient; runtime batch model registration via /model/new; denial helpers |
| `capabilities.py` | the provider x scenario matrix + per-provider /model/new params + id-shape classifiers + per-provider raw-id assertion |
| `conftest.py` | session-scoped batch deployment registration and teardown |
| `test_batches_e2e.py` | parametrized lifecycle with per-endpoint output assertions, file upload/delete outputs, key-model-access denial |
| `test_batches_e2e.py` | parametrized lifecycle with per-endpoint output assertions, file upload/delete outputs, key-model-access denial, per-backend content download, failure paths, second-hop routing, terminal state + cost |
## Failure paths
`TestBatchFailurePaths` pins the customer-facing error contracts. A malformed input
file is a 400 at upload naming the bad content. A JSONL line whose url contradicts
the batch endpoint passes create (providers validate asynchronously) and drives the
batch to `failed` with structured `errors.data` (code/line/message), a null
`output_file_id`, and a $0 spend row keyed `{batch_id}_batch_cost` (LIT-4852: a
failed batch books $0 instead of crashing cost tracking). Cancelling that failed
batch is a 409 naming the terminal status. A file id encoded for one deployment wins
over a conflicting `model` param on create: the batch routes and re-encodes by the
file's embedded model (foreign-id precedence).
## Second hop (two chained gateways)
`TestBatchSecondHop` registers a `litellm_proxy/<inner model>` deployment pointing at
the proxy's own base URL with a freshly minted virtual key, so unified upload and
create traverse gateway -> gateway -> OpenAI (LIT-5347, PR #36240). The pin:
`target_model_names` is rewritten to the inner deployment on the second hop and the
nested managed ids round-trip retrieve. This self-chaining only needs the proxy to
reach its own `PROXY_BASE_URL`, which holds both locally and on the e2e stage.
## Terminal state + cost write-back (cross-run marker baton)
The 24h completion window rules out submit-and-wait inside one run, so
`TestBatchTerminalState` amortizes across runs. Each run submits a 1-line marker
batch (stable metadata key/value plus a per-run field) and deliberately never
cancels or deletes it or its input file: the marker is the baton the next run picks
up (OpenAI files expire on their own after ~30 days). Polling is list-only, up to 5
minutes, because retrieving a non-terminal batch books a $0 spend row whose
request_id then blocks the later real-cost row (`skip_duplicates`); the single
retrieve happens only once a completed marker exists. The assertion target is the
newest completed marker from ANY run: run-scoped deployment names mean the list
re-encodes prior-run batches under new encoded ids, so their spend keys are fresh
and a prior-run marker is billable by this run. On the 6h stage cadence the full
assertions are therefore deterministic from run 2 onward. On a cold start (no
completed marker within the poll budget) the test passes on the submission
assertions alone: a documented vacuous pass, not a skip. Markers aged past the 24h
window (25h-73h band, within the newest 100-item list page) must be terminal.
The cost assertion is the LIT-5730 headline: retrieving a completed model-encoded
batch must write a positive spend row with call_type `aretrieve_batch` and token
usage. Before the fix in `litellm/batches/batch_utils.py`, the retrieve endpoint
re-encoded the response's `output_file_id` in place before the queued logging
worker ran, the worker sent that encoded id to OpenAI, got a 404, and the spend row
never landed.
## Out of scope (intentionally)
Driving a batch to `completed`, cost tracking on completion, and the DB write-back
are not covered here; the 24h window makes them unfit for a synchronous gate. That
logic belongs in a DI-stubbed proxy integration test under `tests/test_litellm/proxy/`
where the provider client is injected to return `completed` deterministically.
Unified (managed) batch cost is owned by the hourly `CheckBatchCost` poller, and a
terminal DB status short-circuits retrieve for those ids, so the terminal-state cell
uses the encoded path; poller timing does not fit an e2e gate and belongs in a
DI-stubbed proxy integration test under `tests/test_litellm/proxy/`. Bedrock
cancel/list stay gated pending LIT-4774. Gemini (non-Vertex) file content raises
`NotImplementedError` upstream and is not a coverage cell.

29
tests/e2e/batches/batch_client.py
View file

@ -51,6 +51,17 @@ class FileList(BaseModel):
has_more: bool | None = None
class BatchErrorItem(BaseModel):
code: str | None = None
line: int | None = None
message: str | None = None
class BatchErrorList(BaseModel):
object: str | None = None
data: list[BatchErrorItem] = []
class BatchObject(BaseModel):
id: str
object: str | None = None
@ -58,6 +69,9 @@ class BatchObject(BaseModel):
endpoint: str | None = None
input_file_id: str | None = None
output_file_id: str | None = None
error_file_id: str | None = None
errors: BatchErrorList | None = None
metadata: dict[str, str] | None = None
completion_window: str | None = None
created_at: int | None = None
model: str | None = None
@ -79,12 +93,18 @@ class BatchCreateBody(BaseModel):
endpoint: str = "/v1/chat/completions"
completion_window: str = "24h"
model: str | None = None
metadata: dict[str, str] | None = None
class ModelQuery(BaseModel):
model: str | None = None
class BatchListQuery(BaseModel):
model: str | None = None
limit: int | None = None
def is_model_access_denied(resp: StreamingResponse) -> bool:
"""True if the proxy rejected the call because the key may not access the model."""
return resp.status_code == 403 and "key_model_access_denied" in resp.body
@ -175,12 +195,17 @@ class BatchClient:
)
def list_batches(
self, *, key: str, provider: str | None = None
self,
*,
key: str,
provider: str | None = None,
model: str | None = None,
limit: int | None = None,
) -> Result[BatchList]:
return self.proxy.transport.get(
_batches_path(provider),
headers=self.proxy.transport.bearer(key),
params=NoBody(),
params=BatchListQuery(model=model, limit=limit),
response_type=BatchList,
)

40
tests/e2e/batches/capabilities.py
View file

@ -5,9 +5,9 @@ from __future__ import annotations
import base64
import os
from dataclasses import dataclass
from typing import Literal
from typing import Final, Literal
from e2e_config import unique_marker
from e2e_config import provider_edge_base, unique_marker
from models import LiteLLMParamsBody
_BATCH_RUN = unique_marker()
@ -17,6 +17,21 @@ def batch_model_name(base: str) -> str:
return f"{base}-{_BATCH_RUN}"
OPENAI_BATCH_BACKEND: Final = "gpt-4o-mini"
def openai_batch_params() -> LiteLLMParamsBody:
"""The OpenAI batch deployment, wired through the record/replay edge when a fixture
mode is active and straight at OpenAI otherwise (LIT-5974). Azure, Vertex, and
Bedrock stay live: none of them has an edge mount."""
base = provider_edge_base("openai")
return LiteLLMParamsBody(
model=f"openai/{OPENAI_BATCH_BACKEND}",
api_key="os.environ/OPENAI_API_KEY",
api_base=None if base is None else f"{base}/v1",
)
def _env_ref(*names: str) -> str:
for name in names:
value = os.environ.get(name)
@ -47,10 +62,7 @@ class Provider:
def litellm_params(self) -> LiteLLMParamsBody:
match self.name:
case "openai":
return LiteLLMParamsBody(
model="openai/gpt-4o-mini",
api_key="os.environ/OPENAI_API_KEY",
)
return openai_batch_params()
case "azure":
return LiteLLMParamsBody(
model="azure/gpt-5.4-mini-batch",
@ -107,7 +119,11 @@ class Capability:
PROVIDERS: tuple[Provider, ...] = (
Provider(
"openai", batch_model_name("openai-batch"), "gpt-4o-mini", can_cancel=True, can_list=True
"openai",
batch_model_name("openai-batch"),
OPENAI_BATCH_BACKEND,
can_cancel=True,
can_list=True,
),
Provider(
"azure",
@ -210,6 +226,16 @@ def is_model_encoded_id(id_str: str) -> bool:
return False
def decoded_model_from_id(id_str: str) -> str | None:
"""Deployment name embedded in a model-encoded file/batch id, or None."""
for prefix in ("file-", "batch_"):
if id_str.startswith(prefix):
decoded = _b64_decode(id_str[len(prefix) :])
if decoded.startswith("litellm:") and ";model," in decoded:
return decoded.split(";model,", 1)[1].split(";")[0]
return None
def matches_id_shape(shape: IdShape, id_str: str) -> bool:
if shape == "managed":
return is_managed_id(id_str)

483
tests/e2e/batches/test_batches_e2e.py
View file

@ -1,11 +1,12 @@
"""Live e2e for the Batches API across every provider LiteLLM supports.
Synchronous tier only: a batch's completion window is 24h, so these never wait for
"completed". Each case uploads a tiny JSONL, creates the batch through one of the
four routing scenarios, asserts it was accepted (non-terminal status) and routed to
the right provider, then retrieves / cancels / lists where the provider supports it.
Everything created is deleted on teardown. Completion + cost tracking are out of
scope here (see COVERAGE.md).
Mostly synchronous tier: a batch's completion window is 24h, so the lifecycle
matrix never waits for "completed". Each case uploads a tiny JSONL, creates the
batch through one of the four routing scenarios, asserts it was accepted
(non-terminal status) and routed to the right provider, then retrieves / cancels /
lists where the provider supports it. Everything created is deleted on teardown.
The exception is TestBatchTerminalState, which carries completed-state + cost
write-back coverage via a cross-run marker baton (design in COVERAGE.md).
Routing signal: for provider_fallback the raw batch id discriminates the provider;
for the encoded/unified/model_param scenarios the proxy re-encodes the id, so the
@ -23,8 +24,9 @@ from datetime import datetime, timedelta, timezone
from typing import Callable
import pytest
from pydantic import BaseModel
from e2e_config import unique_marker
from e2e_config import PROXY_BASE_URL, unique_marker
from batch_client import (
UPLOAD_FILENAME,
@ -40,12 +42,17 @@ from capabilities import (
BATCH_ID_SHAPE,
CAPABILITIES,
FILE_ID_SHAPE,
OPENAI_BATCH_BACKEND,
OPENAI_BATCH_MODEL,
PROVIDERS,
Capability,
Provider,
batch_model_name,
coverage_cells_for_lifecycle,
decoded_model_from_id,
is_managed_id,
matches_id_shape,
openai_batch_params,
raw_id_matches_provider,
)
from e2e_http import (
@ -474,11 +481,22 @@ def test_rate_limited_batch_create_leaves_no_unattributed_spend_row(
)
OPENAI_FILE_CONTENT_BACKEND = "gpt-4o-mini"
FILE_CONTENT_CELLS = {
"azure": "llm.files.azure_openai.content.nonstream.works",
"vertex_ai": "llm.files.vertex.content.nonstream.works",
"bedrock": "llm.files.bedrock.content.nonstream.works",
}
BYTE_FIDELITY_CONTENT_PROVIDERS = frozenset({"azure"})
class TestBatchFileContent:
"""GET /v1/files/{id}/content returns the uploaded batch JSONL bytes."""
"""GET /v1/files/{id}/content returns the uploaded batch JSONL bytes.
Azure stores the upload verbatim, so its download is asserted byte-equal.
Vertex (GCS) and Bedrock (S3) transform each JSONL line into the provider's
request format at upload time, so their downloads assert 200 plus non-empty
parseable JSON lines instead of byte equality.
"""
@pytest.mark.covers(
"llm.files.openai.content.nonstream.works",
@ -488,17 +506,11 @@ class TestBatchFileContent:
self, client: BatchClient, resources: ResourceManager
) -> None:
proxy_name = f"e2e-file-content-{unique_marker()}"
model_id = client.create_model(
proxy_name,
LiteLLMParamsBody(
model=f"openai/{OPENAI_FILE_CONTENT_BACKEND}",
api_key="os.environ/OPENAI_API_KEY",
),
)
model_id = client.create_model(proxy_name, openai_batch_params())
resources.defer(lambda: client.delete_model(model_id))
key = resources.key()
payload = render_jsonl(OPENAI_FILE_CONTENT_BACKEND)
payload = render_jsonl(OPENAI_BATCH_BACKEND)
file = unwrap(
client.upload_file(
content=payload,
@ -522,6 +534,62 @@ class TestBatchFileContent:
"downloaded file content must match the uploaded JSONL bytes"
)
@pytest.mark.parametrize(
"provider",
[
pytest.param(
p,
id=p.name,
marks=pytest.mark.covers(
FILE_CONTENT_CELLS[p.name], exercised_on=["files"]
),
)
for p in PROVIDERS
if p.name in FILE_CONTENT_CELLS
],
)
def test_unified_file_content_downloads(
self,
provider: Provider,
client: BatchClient,
resources: ResourceManager,
batch_deployments: None,
) -> None:
key = resources.key()
payload = render_jsonl(provider.raw_model)
file = unwrap(
client.upload_file(
content=payload,
form=FileUploadForm(purpose="batch", target_model_names=provider.model),
key=key,
)
)
resources.defer(quietly(lambda: client.delete_file(file.id, key=key)))
assert_file_object(file, provider=provider.name)
assert is_managed_id(file.id), (
f"{provider.name}: unified upload must return a managed file id, got {file.id!r}"
)
downloaded = client.proxy.transport.download(
f"/v1/files/{file.id}/content",
headers=client.proxy.transport.bearer(key),
)
assert downloaded.status_code == 200, (
f"{provider.name}: file content must be 200, "
f"got {downloaded.status_code}: {downloaded.body[:300]}"
)
body = downloaded.body.strip()
assert body, f"{provider.name}: file content download returned an empty body"
if provider.name in BYTE_FIDELITY_CONTENT_PROVIDERS:
assert body == payload.decode().strip(), (
f"{provider.name}: downloaded content must match the uploaded JSONL bytes"
)
else:
for line in body.splitlines():
assert json.loads(line), (
f"{provider.name}: content line is not JSON: {line[:200]}"
)
class TestOpenAIFiles:
"""GET /v1/files (list) and GET /v1/files/{id} (retrieve) over the OpenAI route.
@ -1045,3 +1113,384 @@ class TestHostedVllmBatch:
f"hosted_vllm batch has non-transitional status {batch.status!r}"
)
assert_batch_object(batch)
BATCH_TERMINAL_STATUSES = frozenset({"completed", "failed", "expired", "cancelled"})
FAILED_BATCH_POLL_SECONDS = 120.0
FAILED_BATCH_POLL_INTERVAL_SECONDS = 5.0
AZURE_BATCH_RAW_MODEL = next(p.raw_model for p in PROVIDERS if p.name == "azure")
def _mismatched_endpoint_jsonl(model: str) -> bytes:
line = {
"custom_id": "req-1",
"method": "POST",
"url": "/v1/embeddings",
"body": {"model": model, "input": "ping"},
}
return (json.dumps(line) + "\n").encode()
def _poll_until_terminal(client: BatchClient, batch_id: str, key: str) -> BatchObject:
deadline = time.monotonic() + FAILED_BATCH_POLL_SECONDS
fetched = retrieve_batch(client, batch_id, key=key, provider=None)
while fetched.status not in BATCH_TERMINAL_STATUSES and time.monotonic() < deadline:
time.sleep(FAILED_BATCH_POLL_INTERVAL_SECONDS)
fetched = retrieve_batch(client, batch_id, key=key, provider=None)
return fetched
class TestBatchFailurePaths:
"""Customer-facing failure contracts for /v1/batches.
A malformed input file is rejected at upload with a 400 naming the bad
content. A JSONL line whose url contradicts the batch endpoint is accepted
at create (providers validate asynchronously) and drives the batch to
"failed" with structured per-line errors, a null output_file_id, and a
zero-cost spend row (LIT-4852: a failed batch must book $0, not crash cost
tracking). Cancelling that already-failed batch returns a 409 naming the
terminal status. A file id encoded for one deployment wins over a
conflicting model param on create: the batch routes (and re-encodes) by the
file's embedded model, pinning that precedence.
"""
@pytest.mark.covers(
"llm.batches.openai.malformed_jsonl.nonstream.works",
exercised_on=["files"],
)
def test_malformed_jsonl_upload_rejected(
self, client: BatchClient, resources: ResourceManager, batch_deployments: None
) -> None:
result = client.upload_file(
content=b"this is not json\n",
form=FileUploadForm(purpose="batch"),
model=OPENAI_BATCH_MODEL,
key=resources.key(),
)
match result:
case UnknownApiError(status_code=400, body=body):
assert "json" in body.lower(), (
f"400 must name the malformed JSONL so users can fix the file, got: {body[:300]}"
)
case _:
pytest.fail(f"malformed JSONL upload must be rejected with a 400, got: {result}")
@pytest.mark.covers(
"llm.batches.openai.jsonl_endpoint_mismatch.nonstream.works",
"llm.batches.openai.cancel_terminal.nonstream.works",
exercised_on=["batches", "files"],
)
def test_endpoint_mismatch_fails_batch_and_cancel_conflicts(
self, client: BatchClient, resources: ResourceManager, batch_deployments: None
) -> None:
key = resources.key()
file = unwrap(
client.upload_file(
content=_mismatched_endpoint_jsonl("gpt-4o-mini"),
form=FileUploadForm(purpose="batch"),
model=OPENAI_BATCH_MODEL,
key=key,
)
)
resources.defer(quietly(lambda: client.delete_file(file.id, key=key)))
created = client.create_batch(body=BatchCreateBody(input_file_id=file.id), key=key)
require_successful_call(created)
batch = BatchObject.model_validate_json(created.body)
fetched = _poll_until_terminal(client, batch.id, key)
assert fetched.status == "failed", (
f"endpoint-mismatched batch must fail, got {fetched.status!r}"
)
assert fetched.output_file_id is None, (
f"failed batch must have no output file, got {fetched.output_file_id!r}"
)
assert fetched.errors is not None and fetched.errors.data, (
"failed batch must surface structured errors so users can fix the JSONL"
)
first_error = fetched.errors.data[0]
assert first_error.message, "batch error item has no message"
assert first_error.code, "batch error item has no code"
rows = client.proxy.poll_logs_for_request_id(f"{fetched.id}_batch_cost")
assert rows, (
f"failed batch {fetched.id} wrote no spend row; retrieve must book $0 (LIT-4852)"
)
assert all((row.spend or 0) == 0 for row in rows), (
f"failed batch must cost $0, got {[(r.request_id, r.spend) for r in rows]}"
)
assert rows[0].call_type == "aretrieve_batch", (
f"batch cost row call_type={rows[0].call_type!r}"
)
conflict = client.cancel_batch(batch.id, key=key)
match conflict:
case UnknownApiError(status_code=409, body=body):
assert "failed" in body.lower(), (
f"409 must name the terminal status blocking the cancel, got: {body[:300]}"
)
case _:
pytest.fail(f"cancel of a failed batch must return a 409 conflict, got: {conflict}")
@pytest.mark.covers(
"llm.batches.openai.foreign_file_id.nonstream.works",
exercised_on=["batches", "files"],
)
def test_foreign_encoded_file_id_routes_by_file_model(
self, client: BatchClient, resources: ResourceManager, batch_deployments: None
) -> None:
key = resources.key()
file = unwrap(
client.upload_file(
content=render_jsonl(AZURE_BATCH_RAW_MODEL),
form=FileUploadForm(purpose="batch"),
model=AZURE_BATCH_MODEL,
key=key,
)
)
resources.defer(quietly(lambda: client.delete_file(file.id, key=key)))
assert decoded_model_from_id(file.id) == AZURE_BATCH_MODEL, (
f"upload did not encode the azure deployment into the file id: {file.id!r}"
)
created = client.create_batch(
body=BatchCreateBody(input_file_id=file.id, model=OPENAI_BATCH_MODEL), key=key
)
require_successful_call(created)
batch = BatchObject.model_validate_json(created.body)
resources.defer(quietly(lambda: client.cancel_batch(batch.id, key=key)))
assert decoded_model_from_id(batch.id) == AZURE_BATCH_MODEL, (
"create with a foreign encoded file id must route by the file's embedded model, "
f"but the batch id encodes {decoded_model_from_id(batch.id)!r} "
f"(model param was {OPENAI_BATCH_MODEL!r})"
)
fetched = retrieve_batch(client, batch.id, key=key, provider=None)
assert fetched.id == batch.id
assert fetched.status, "retrieved foreign-file batch has no status"
class TestBatchSecondHop:
"""Two-proxy batch routing: a litellm_proxy deployment chained to the gateway
itself (LIT-5347, PR #36240).
The hop deployment's litellm_params point litellm_proxy/<inner model> at this
gateway's own base URL with a freshly minted virtual key, so the unified
upload and batch create traverse gateway -> gateway -> OpenAI. The regression
this pins: target_model_names must be rewritten to the inner deployment on
the second hop and the nested managed ids must round-trip retrieve.
"""
@pytest.mark.covers(
"llm.batches.openai.second_hop.nonstream.works",
exercised_on=["batches", "files"],
)
def test_unified_create_and_retrieve_via_chained_gateway(
self, client: BatchClient, resources: ResourceManager, batch_deployments: None
) -> None:
key = resources.key()
hop_name = batch_model_name("openai-batch-hop")
model_id = client.create_model(
hop_name,
LiteLLMParamsBody(
model=f"litellm_proxy/{OPENAI_BATCH_MODEL}",
api_base=PROXY_BASE_URL,
api_key=key,
),
)
resources.defer(lambda: client.delete_model(model_id))
file = unwrap(
client.upload_file(
content=render_jsonl("gpt-4o-mini"),
form=FileUploadForm(purpose="batch", target_model_names=hop_name),
key=key,
)
)
resources.defer(quietly(lambda: client.delete_file(file.id, key=key)))
assert is_managed_id(file.id), (
f"second-hop unified upload must return a managed file id, got {file.id!r}"
)
created = client.create_batch(body=BatchCreateBody(input_file_id=file.id), key=key)
require_successful_call(created)
batch = BatchObject.model_validate_json(created.body)
resources.defer(quietly(lambda: client.cancel_batch(batch.id, key=key)))
assert is_managed_id(batch.id), (
f"second-hop create must return a managed batch id, got {batch.id!r}"
)
assert batch.status in CREATED_BATCH_STATUSES, (
f"second-hop batch has non-transitional status {batch.status!r}"
)
assert_batch_object(batch)
fetched = retrieve_batch(client, batch.id, key=key, provider=None)
assert fetched.id == batch.id
assert fetched.status, "second-hop retrieve returned no status"
class BatchOutputBody(BaseModel):
choices: list[object] = []
class BatchOutputResponse(BaseModel):
status_code: int | None = None
body: BatchOutputBody | None = None
class BatchOutputLine(BaseModel):
response: BatchOutputResponse
TERMINAL_MARKER_KEY = "litellm_e2e_suite"
TERMINAL_MARKER_VALUE = "batches-terminal-baton"
TERMINAL_POLL_SECONDS = 300.0
TERMINAL_POLL_INTERVAL_SECONDS = 10.0
TERMINAL_LIST_LIMIT = 100
TERMINAL_BAND_MIN_AGE_SECONDS = 25 * 3600
TERMINAL_BAND_MAX_AGE_SECONDS = 73 * 3600
def _marker_batches(client: BatchClient, key: str) -> list[BatchObject]:
listed = unwrap(
client.list_batches(key=key, model=OPENAI_BATCH_MODEL, limit=TERMINAL_LIST_LIMIT)
)
return [
b
for b in listed.data
if (b.metadata or {}).get(TERMINAL_MARKER_KEY) == TERMINAL_MARKER_VALUE
]
def _await_completed_marker(
client: BatchClient, key: str
) -> tuple[BatchObject | None, list[BatchObject]]:
deadline = time.monotonic() + TERMINAL_POLL_SECONDS
while True:
markers = _marker_batches(client, key)
completed = max(
(b for b in markers if b.status == "completed"),
key=lambda b: b.created_at or 0,
default=None,
)
if completed is not None or time.monotonic() >= deadline:
return completed, markers
time.sleep(TERMINAL_POLL_INTERVAL_SECONDS)
def _assert_aged_markers_terminal(markers: list[BatchObject]) -> None:
now = time.time()
stuck = [
b
for b in markers
if b.created_at is not None
and TERMINAL_BAND_MIN_AGE_SECONDS <= now - b.created_at <= TERMINAL_BAND_MAX_AGE_SECONDS
and b.status not in BATCH_TERMINAL_STATUSES
]
assert not stuck, (
"marker batches past their 24h completion window must be terminal; stuck: "
f"{[(b.id, b.status, b.created_at) for b in stuck]}"
)
class TestBatchTerminalState:
"""Terminal state + cost write-back via a cross-run marker baton.
Each run submits a 1-line marker batch (stable metadata key/value plus a
per-run field) and never cancels or deletes it: the marker is the baton the
next run picks up. Polling is list-only for up to 5 minutes because a
retrieve of a non-terminal batch books a $0 spend row whose request_id then
blocks the real-cost row (skip_duplicates); the single retrieve happens only
once a completed marker exists. The assertion target is the newest completed
marker from ANY run, so on the 6h stage cadence the full assertions are
deterministic from run 2 onward. On a cold start (no marker has ever
completed within the poll budget) the test passes on the submission
assertions alone: that is a documented vacuous pass, not a skip, and this
run's marker becomes the next run's target. Markers aged past OpenAI's 24h
completion window (25h-73h band, within the newest list page) must be
terminal. The cost assertion is the LIT-5730 headline: retrieving a
completed model-encoded batch must write a positive spend row keyed
{batch_id}_batch_cost; before the fix the logging worker fetched the
re-encoded output_file_id, 404d, and the row never landed.
"""
@pytest.mark.covers(
"llm.batches.openai.terminal_state.nonstream.works",
"llm.batches.openai.terminal_state.nonstream.cost_logged",
exercised_on=["batches", "files"],
)
def test_completed_batch_downloads_output_and_books_cost(
self, client: BatchClient, resources: ResourceManager, batch_deployments: None
) -> None:
key = resources.key()
file = unwrap(
client.upload_file(
content=render_jsonl("gpt-4o-mini"),
form=FileUploadForm(purpose="batch"),
model=OPENAI_BATCH_MODEL,
key=key,
)
)
created = client.create_batch(
body=BatchCreateBody(
input_file_id=file.id,
metadata={
TERMINAL_MARKER_KEY: TERMINAL_MARKER_VALUE,
"run": unique_marker(),
},
),
key=key,
)
require_successful_call(created)
submitted = BatchObject.model_validate_json(created.body)
assert submitted.status in CREATED_BATCH_STATUSES, (
f"marker batch has non-transitional status {submitted.status!r}"
)
assert (submitted.metadata or {}).get(TERMINAL_MARKER_KEY) == TERMINAL_MARKER_VALUE, (
f"create dropped the marker metadata: {submitted.metadata!r}"
)
completed, markers = _await_completed_marker(client, key)
_assert_aged_markers_terminal(markers)
if completed is None:
return
fetched = retrieve_batch(client, completed.id, key=key, provider=None)
assert fetched.status == "completed", (
f"listed-completed marker retrieved as {fetched.status!r}"
)
assert fetched.output_file_id, "completed batch has no output_file_id"
downloaded = client.proxy.transport.download(
f"/v1/files/{fetched.output_file_id}/content",
headers=client.proxy.transport.bearer(key),
)
assert downloaded.status_code == 200, (
f"output content must be 200, got {downloaded.status_code}: {downloaded.body[:300]}"
)
first_line = BatchOutputLine.model_validate_json(downloaded.body.strip().splitlines()[0])
assert first_line.response.status_code == 200, (
f"batch output line reports failure: {downloaded.body[:400]}"
)
assert first_line.response.body is not None and first_line.response.body.choices, (
"batch output line has no choices"
)
rows = client.proxy.poll_logs_for_request_id(
f"{fetched.id}_batch_cost",
predicate=lambda found: any((row.spend or 0) > 0 for row in found),
)
priced = [row for row in rows if (row.spend or 0) > 0]
assert priced, (
f"completed batch {fetched.id} wrote no positive-cost spend row under "
f"request_id {fetched.id}_batch_cost; cost write-back is broken (LIT-5730)"
)
cost_row = priced[0]
assert cost_row.call_type == "aretrieve_batch", (
f"batch cost row call_type={cost_row.call_type!r}"
)
assert (cost_row.total_tokens or 0) > 0, (
f"batch cost row has no token usage: {cost_row.total_tokens!r}"
)

1
tests/e2e/coverage_registry/llm_conversational.yaml
View file

@ -64,6 +64,7 @@
- {id: llm.responses.openai.basic.stream.works, module: llm, tier: P0, subject_endpoint: responses, route: openai, capability: basic, streaming: stream, assertions: [works], source: "response_api_endpoints/endpoints.py:26", rationale: "Streaming via /v1/responses"}
- {id: llm.responses.openai.basic.nonstream.cost_logged, module: llm, tier: P0, subject_endpoint: responses, route: openai, capability: basic, streaming: nonstream, assertions: [works, cost_logged], source: "response_api_endpoints/endpoints.py:26", rationale: "Cost logged on responses"}
- {id: llm.responses.openai.passthrough.stream.cost_logged, module: llm, tier: P0, subject_endpoint: responses, route: openai, capability: basic, streaming: stream, assertions: [cost_logged], source: "test_passthrough_e2e.py", rationale: "A streamed POST /openai_passthrough/v1/responses is costed and keyed by the provider response id; it used to log a zero-cost row under a random id (GitHub issue #36523)"}
- {id: llm.responses.openai.passthrough_websocket.stream.works, module: llm, tier: P1, subject_endpoint: responses, route: openai, capability: basic, streaming: stream, assertions: [works], fail_before_fix: proven, source: "test_passthrough_e2e.py", rationale: "A websocket upgrade on /openai/v1/responses is accepted, so a responses.connect client reaches OpenAI through the same prefix its HTTP traffic uses; the prefix carried no websocket route and refused the upgrade with a 403 (GitHub issue #36088)"}
- {id: llm.responses.openai.tool_use.nonstream.works, module: llm, tier: P0, subject_endpoint: responses, route: openai, capability: tool_use, streaming: nonstream, assertions: [works], source: "model_prices json", rationale: "Tool calls via Responses API"}
- {id: llm.responses.openai.vision.nonstream.works, module: llm, tier: P0, subject_endpoint: responses, route: openai, capability: vision, streaming: nonstream, assertions: [works], source: "model_prices json", rationale: "Vision via Responses API"}
- {id: llm.responses.anthropic.basic.nonstream.works, module: llm, tier: P1, subject_endpoint: responses, route: anthropic, capability: basic, streaming: nonstream, assertions: [works], source: "response_api_endpoints/endpoints.py:26", rationale: "Responses w/ Anthropic translation (smoke)"}

11
tests/e2e/coverage_registry/llm_nonconversational.yaml
View file

@ -26,6 +26,13 @@
- {id: llm.batches.hosted_vllm.basic.nonstream.works, module: llm, tier: P1, subject_endpoint: batches, route: hosted_vllm, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py", rationale: "hosted_vllm OpenAI-compatible batch create"}
- {id: llm.batches.openai.key_model_access_denied.nonstream.works, module: llm, tier: P0, subject_endpoint: batches, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py", rationale: "Key model restriction 403 on upload/create"}
- {id: llm.batches.openai.input_validation.nonstream.works, module: llm, tier: P1, subject_endpoint: batches, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "vendor strategy §9.18 / LIT-4778", rationale: "Missing input_file_id and invalid batch id rejected"}
- {id: llm.batches.openai.terminal_state.nonstream.works, module: llm, tier: P0, subject_endpoint: batches, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "A batch actually reaches completed and its output file downloads through GET /v1/files/{id}/content with per-line provider responses"}
- {id: llm.batches.openai.terminal_state.nonstream.cost_logged, module: llm, tier: P0, subject_endpoint: batches, route: openai, capability: basic, streaming: nonstream, assertions: [cost_logged], source: "test_batches_e2e.py / LIT-5730", fail_before_fix: proven, rationale: "Retrieving a completed model-encoded batch writes a positive spend row keyed {batch_id}_batch_cost (pins LIT-4852/LIT-5666; before the fix the logging worker 404d fetching the re-encoded output_file_id and the row was never written)"}
- {id: llm.batches.openai.malformed_jsonl.nonstream.works, module: llm, tier: P1, subject_endpoint: batches, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "Uploading a non-JSON batch file is rejected with a 400 naming the bad line"}
- {id: llm.batches.openai.jsonl_endpoint_mismatch.nonstream.works, module: llm, tier: P1, subject_endpoint: batches, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "JSONL line url that contradicts the batch endpoint drives the batch to failed with structured errors, retrieve stays clean, and the terminal retrieve books a zero-cost spend row (LIT-4852)"}
- {id: llm.batches.openai.cancel_terminal.nonstream.works, module: llm, tier: P1, subject_endpoint: batches, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "Cancelling an already-terminal batch returns a 409 conflict naming the terminal status"}
- {id: llm.batches.openai.foreign_file_id.nonstream.works, module: llm, tier: P1, subject_endpoint: batches, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "Create with one deployment's encoded file id and a conflicting model param routes by the file's embedded model; the returned batch id pins that precedence"}
- {id: llm.batches.openai.second_hop.nonstream.works, module: llm, tier: P0, subject_endpoint: batches, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5347", rationale: "A litellm_proxy deployment chained to the gateway itself preserves target_model_names through nested unified ids; upload, create, and retrieve work over the two-hop chain (PR #36240)"}
- {id: llm.files.openai.upload.nonstream.works, module: llm, tier: P0, subject_endpoint: files, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "openai_files_endpoints/files_endpoints.py:46", rationale: "File upload returns OpenAIFileObject"}
- {id: llm.files.openai.input_validation.nonstream.works, module: llm, tier: P1, subject_endpoint: files, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "vendor strategy §9.16 / LIT-4778", rationale: "File upload without purpose rejected"}
- {id: llm.files.openai.retrieve.nonstream.works, module: llm, tier: P0, subject_endpoint: files, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "files_endpoints.py", rationale: "File retrieve by id"}
@ -40,10 +47,14 @@
- {id: llm.files.hosted_vllm.upload.nonstream.works, module: llm, tier: P1, subject_endpoint: files, route: hosted_vllm, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py", rationale: "hosted_vllm OpenAI-compatible file upload"}
- {id: llm.rerank.cohere.basic.nonstream.works, module: llm, tier: P1, subject_endpoint: rerank, route: cohere, capability: basic, streaming: nonstream, assertions: [works], source: "test_rerank_e2e.py:29", rationale: "Cohere rerank, top_n + relevance_score"}
- {id: llm.files.openai.content.nonstream.works, module: llm, tier: P0, subject_endpoint: files, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py", rationale: "GET /v1/files/{id}/content returns uploaded batch JSONL bytes"}
- {id: llm.files.azure_openai.content.nonstream.works, module: llm, tier: P0, subject_endpoint: files, route: azure_openai, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "GET /v1/files/{id}/content on an Azure unified file returns the uploaded JSONL bytes verbatim"}
- {id: llm.files.vertex.content.nonstream.works, module: llm, tier: P0, subject_endpoint: files, route: vertex, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "GET /v1/files/{id}/content on a Vertex unified file streams the GCS object back (provider-transformed JSONL, so asserts non-empty JSON lines rather than byte equality)"}
- {id: llm.files.bedrock.content.nonstream.works, module: llm, tier: P0, subject_endpoint: files, route: bedrock_converse, capability: basic, streaming: nonstream, assertions: [works], source: "test_batches_e2e.py / LIT-5730", rationale: "GET /v1/files/{id}/content on a Bedrock unified file streams the S3 object back (provider-transformed JSONL, so asserts non-empty JSON lines rather than byte equality)"}
- {id: llm.realtime.bedrock_converse.basic.stream.works, module: llm, tier: P0, subject_endpoint: realtime, route: bedrock_converse, capability: basic, streaming: stream, assertions: [works], source: "test_realtime_bedrock_e2e.py", rationale: "Nova Sonic realtime session emits response.done (LIT-2239)"}
- {id: llm.google_native.gemini.basic.nonstream.cost_logged, module: llm, tier: P0, subject_endpoint: google_native, route: gemini, capability: basic, streaming: nonstream, assertions: [cost_logged], source: "LIT-4076 / proxy/google_endpoints/endpoints.py", fail_before_fix: proven, rationale: "google-native generateContent must stamp x-litellm-response-cost so SDK traffic reconciles against spend"}
- {id: llm.google_native.gemini.basic.stream.works, module: llm, tier: P0, subject_endpoint: google_native, route: gemini, capability: basic, streaming: stream, assertions: [works], source: "PR #28213 / proxy/proxy_server.py async_data_generator", fail_before_fix: proven, rationale: "streamGenerateContent must relay single-prefixed SSE frames with no [DONE] sentinel; doubled data: prefixes and the OpenAI terminator both break the Vertex Java SDK"}
- {id: llm.realtime.openai.basic.nonstream.works, module: llm, tier: P1, subject_endpoint: realtime, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "vendor strategy §9.19 / LIT-4778", rationale: "HTTP /v1/realtime/client_secrets returns an ephemeral credential"}
- {id: llm.realtime.openai.passthrough.stream.works, module: llm, tier: P0, subject_endpoint: realtime, route: openai, capability: basic, streaming: stream, assertions: [works], fail_before_fix: proven, source: "test_passthrough_e2e.py", rationale: "A websocket upgrade on /openai_passthrough/v1/realtime is accepted and relayed to OpenAI; only HTTP routes were registered under the prefix, so realtime clients were refused with a 403 before a socket existed (GitHub issue #36088)"}
- {id: llm.vector_stores.openai.basic.nonstream.works, module: llm, tier: P1, subject_endpoint: vector_stores, route: openai, capability: basic, streaming: nonstream, assertions: [works], source: "vendor strategy §9.17 / LIT-4778", rationale: "Vector store create/list/retrieve/delete lifecycle"}
- {id: llm.vector_stores.openai.input_validation.nonstream.works, module: llm, tier: P1, subject_endpoint: vector_stores, route: openai, capability: input_validation, streaming: nonstream, assertions: [works], source: "vendor strategy §9.17 / LIT-4778", rationale: "Vector store search and invalid id errors"}
- {id: llm.bedrock_native.bedrock_converse.basic.nonstream.works, module: llm, tier: P1, subject_endpoint: bedrock_native, route: bedrock_converse, capability: basic, streaming: nonstream, assertions: [works], source: "vendor strategy §9.12 / LIT-4778", rationale: "Bedrock native converse happy path"}

9
tests/e2e/e2e_config.py
View file

@ -150,6 +150,15 @@ ANOMALY_SPEND_SETTLE_SECONDS = float(
)
def ws_base_url() -> str:
"""PROXY_BASE_URL with its scheme swapped for the websocket one, so a suite
opening a socket points at the same proxy every HTTP suite uses."""
for scheme, ws_scheme in (("https://", "wss://"), ("http://", "ws://")):
if PROXY_BASE_URL.startswith(scheme):
return ws_scheme + PROXY_BASE_URL[len(scheme) :]
return PROXY_BASE_URL
def datadog_mcp_url(*, toolsets: str = "core") -> str:
"""Regional Datadog remote MCP endpoint for this process's DD_SITE.

15
tests/e2e/fixture_bundle.py
View file

@ -5,7 +5,9 @@ version + format version) plus one subdirectory per test, holding one JSON file
per provider-bound interaction in call order. Bundles older than
``MAX_BUNDLE_AGE`` hard-fail replay at collection time (see conftest), so a
green replay run can never certify against fixtures that have drifted more than
a week from the live providers.
a week from the live providers. Bump ``BUNDLE_FORMAT_VERSION`` whenever a change
moves recorded keys: a bundle recorded under the old rules then fails naming
both versions instead of quietly missing on every call.
This module owns the format only. The provider-edge server that produces and
consumes it lives in provider_edge.py (LIT-5745) and the canonical match keys
@ -28,7 +30,7 @@ from typing import Final
from pydantic import BaseModel, JsonValue
BUNDLE_FORMAT_VERSION: Final = 2
BUNDLE_FORMAT_VERSION: Final = 3
MAX_BUNDLE_AGE: Final = timedelta(days=7)
MANIFEST_FILENAME: Final = "manifest.json"
@ -47,7 +49,14 @@ class RecordedRequest(BaseModel):
over ``method``, ``path`` (the edge path including the provider mount,
query string excluded), and the canonicalized headers, params, body, form,
and file identity. Non-JSON bodies store a canonicalized content digest
instead of the bytes."""
instead of the bytes.
``file_name`` is a JSON list of the uploaded parts' ``[field, filename,
content-type]`` triples rather than a flat label, so a separator inside a
filename cannot impersonate a field boundary. ``file_bytes`` is recorded for
a reader's benefit and stays out of the key: the canonicalizer absorbs
timestamp and id drift inside an uploaded file, and that drift moves the
byte count."""
method: str
path: str

1
tests/e2e/fixture_canonical.py
View file

@ -129,7 +129,6 @@ def canonicalize(request: RecordedRequest) -> CanonicalRequest:
else {
"name": None if request.file_name is None else canonical_string(request.file_name),
"sha256": request.file_sha256,
"bytes": request.file_bytes,
}
)
content: Final[dict[str, JsonValue]] = {

1
tests/e2e/llm_translation/endpoints_client.py
View file

@ -87,6 +87,7 @@ class RichMessagesRequest(BaseModel):
max_tokens: int = 64
system: list[TextBlock]
messages: list[RichMessage]
cache: dict[str, bool] = {"no-cache": True}
class CompletionsRequest(BaseModel):

58
tests/e2e/llm_translation/passthrough_client.py
View file

@ -11,9 +11,13 @@ native request models are co-located here because only this suite uses them.
from __future__ import annotations
from dataclasses import dataclass
from urllib.parse import urlencode
from pydantic import BaseModel, Field
from websockets.exceptions import InvalidStatus
from websockets.sync.client import connect
from e2e_config import ws_base_url
from proxy_client import ProxyClient
from e2e_http import FileUploadForm, Headers, NoBody, Result, StreamingResponse
from models import ChatMessage
@ -175,6 +179,26 @@ class OpenAIEmbeddingBody(BaseModel):
input: str
class WebsocketEnvelope(BaseModel):
"""The one field every provider event carries, so the first frame off a
passthrough socket identifies itself without the suite parsing raw dicts."""
type: str
class WebsocketHandshake(BaseModel):
"""What the proxy did with a websocket upgrade on a passthrough prefix.
`rejected_status` is the HTTP status of a refused upgrade: a prefix carrying no
websocket route answers 403, before any socket exists. `first_event_type` is the
type of the first frame an accepted socket delivered, which is None when the
provider waits for the client to speak first.
"""
rejected_status: int | None = None
first_event_type: str | None = None
class PassthroughBatchList(BaseModel):
"""OpenAI's own batch page, relayed verbatim. `object` is required so a body
that is not an OpenAI list fails validation instead of passing vacuously."""
@ -339,5 +363,39 @@ class PassthroughClient:
),
)
# ---- OpenAI websocket passthrough ----------------------------------
#
# The same prefixes over an upgrade instead of a POST, for the provider APIs
# that only speak websocket (realtime, responses.connect).
def openai_passthrough_websocket(
self,
key: str,
path: str,
*,
model: str | None = None,
open_timeout: float = 30.0,
first_event_timeout: float = 30.0,
) -> WebsocketHandshake:
query = f"?{urlencode({'model': model})}" if model is not None else ""
try:
connection = connect(
f"{ws_base_url()}{path}{query}",
additional_headers={"Authorization": f"Bearer {key}"},
open_timeout=open_timeout,
)
except InvalidStatus as rejected:
return WebsocketHandshake(rejected_status=rejected.response.status_code)
with connection:
try:
frame = connection.recv(timeout=first_event_timeout)
except TimeoutError:
return WebsocketHandshake()
text = frame.decode("utf-8") if isinstance(frame, bytes) else frame
return WebsocketHandshake(
first_event_type=WebsocketEnvelope.model_validate_json(text).type
)
def build_client(proxy: ProxyClient) -> PassthroughClient:
return PassthroughClient(proxy=proxy)

9
tests/e2e/llm_translation/realtime/realtime_client.py
View file

@ -21,20 +21,13 @@ from pydantic import BaseModel, ConfigDict
from websockets.sync.client import connect
from websockets.sync.connection import Connection
from e2e_config import PROXY_BASE_URL, unique_marker
from e2e_config import unique_marker, ws_base_url
from proxy_client import ProxyClient
from models import LiteLLMParamsBody
_M = TypeVar("_M", bound=BaseModel)
def ws_base_url() -> str:
for scheme, ws_scheme in (("https://", "wss://"), ("http://", "ws://")):
if PROXY_BASE_URL.startswith(scheme):
return ws_scheme + PROXY_BASE_URL[len(scheme) :]
return PROXY_BASE_URL
def realtime_ws_url(model: str) -> str:
return f"{ws_base_url()}/v1/realtime?{urlencode({'model': model})}"

2
tests/e2e/llm_translation/realtime/test_realtime_pipecat_audio_e2e.py
View file

@ -27,10 +27,10 @@ from pathlib import Path
import pytest
from e2e_config import ws_base_url
from realtime_client import (
PROVIDERS,
RealtimeProvider,
ws_base_url,
realtime_model,
)

2
tests/e2e/llm_translation/realtime/test_realtime_pipecat_e2e.py
View file

@ -25,10 +25,10 @@ import asyncio
import pytest
from e2e_config import ws_base_url
from realtime_client import (
PROVIDERS,
RealtimeProvider,
ws_base_url,
realtime_model,
)

9
tests/e2e/llm_translation/test_chat_completions_contract_e2e.py
View file

@ -6,7 +6,7 @@ Exercises the gateway against a live OpenAI deployment using customer request sh
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_config import provider_edge_base, unique_marker
from e2e_http import StreamingResponse, assert_client_error, require_successful_call, unwrap
from lifecycle import ResourceManager
from models import ChatBody, ChatMessage, ChatResponse, LiteLLMParamsBody
@ -38,10 +38,15 @@ class ChatErrorEnvelope(BaseModel):
def _register_chat_model(proxy: ProxyClient, resources: ResourceManager) -> tuple[str, str]:
base = provider_edge_base("openai")
model = f"e2e-chat-sec-{unique_marker()}"
model_id = proxy.create_model(
model,
LiteLLMParamsBody(model=OPENAI_BACKEND, api_key="os.environ/OPENAI_API_KEY"),
LiteLLMParamsBody(
model=OPENAI_BACKEND,
api_key="os.environ/OPENAI_API_KEY",
api_base=None if base is None else f"{base}/v1",
),
)
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()

26
tests/e2e/llm_translation/test_embeddings_endpoint_e2e.py
View file

@ -9,7 +9,7 @@ covered by tests/e2e/quota_management/spend_tracking/.
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_config import provider_edge_base, unique_marker
from e2e_http import (
assert_client_error,
require_successful_call,
@ -27,6 +27,18 @@ class _OptionalEmbeddingsBody(BaseModel):
input: str | list[str] | None = None
def _openai_embeddings_params() -> LiteLLMParamsBody:
"""The OpenAI embeddings deployment, wired through the record/replay edge when a
fixture mode is active and straight at OpenAI otherwise (LIT-5974). Bedrock and
Vertex stay live: SigV4 signs the Host header, and neither has an edge mount."""
base = provider_edge_base("openai")
return LiteLLMParamsBody(
model="openai/text-embedding-3-small",
api_key="os.environ/OPENAI_API_KEY",
api_base=None if base is None else f"{base}/v1",
)
class TestEmbeddingsEndpoint:
@pytest.mark.covers("llm.embeddings.openai.basic.nonstream.works")
def test_embeddings_returns_vector(
@ -35,9 +47,7 @@ class TestEmbeddingsEndpoint:
model = f"e2e-embeddings-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model="openai/text-embedding-3-small", api_key="os.environ/OPENAI_API_KEY"
),
_openai_embeddings_params(),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
@ -106,9 +116,7 @@ class TestEmbeddingsEndpoint:
model = f"e2e-embeddings-array-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model="openai/text-embedding-3-small", api_key="os.environ/OPENAI_API_KEY"
),
_openai_embeddings_params(),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
@ -140,9 +148,7 @@ class TestEmbeddingsEndpoint:
model = f"e2e-embeddings-missin-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model="openai/text-embedding-3-small", api_key="os.environ/OPENAI_API_KEY"
),
_openai_embeddings_params(),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()

38
tests/e2e/llm_translation/test_messages_e2e.py
View file

@ -9,7 +9,7 @@ litellm-regression-tests/tests/test_inference_endpoints.py.
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_config import provider_edge_base, unique_marker
from e2e_http import assert_client_error, require_successful_call, unwrap
from endpoints_client import EndpointsClient, MessagesResult
from lifecycle import ResourceManager
@ -50,16 +50,27 @@ def _approx_equal(actual: float, expected: float) -> bool:
return abs(actual - expected) <= max(1e-9, abs(expected) * 1e-2)
def _anthropic_params() -> LiteLLMParamsBody:
"""The Anthropic deployment, wired through the record/replay edge when a fixture
mode is active (LIT-5974). The mount base carries no ``/v1``: litellm's Anthropic
handler appends ``/v1/messages`` to ``api_base`` itself, where the OpenAI handler
appends only ``/chat/completions``."""
base = provider_edge_base("anthropic")
return LiteLLMParamsBody(
model=ANTHROPIC_BACKEND, api_key="os.environ/ANTHROPIC_API_KEY", api_base=base
)
class TestAnthropicMessages:
def _register(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self,
endpoints_client: EndpointsClient,
resources: ResourceManager,
params: LiteLLMParamsBody | None = None,
) -> tuple[str, str]:
model = f"e2e-messages-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model=ANTHROPIC_BACKEND, api_key="os.environ/ANTHROPIC_API_KEY"
),
model, _anthropic_params() if params is None else params
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
return model, resources.key()
@ -81,12 +92,7 @@ class TestAnthropicMessages:
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-messages-cost-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model=ANTHROPIC_BACKEND, api_key="os.environ/ANTHROPIC_API_KEY"
),
)
model_id = endpoints_client.create_model(model, _anthropic_params())
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
@ -131,7 +137,13 @@ class TestAnthropicMessages:
def test_messages_streams_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
"""Stays on a live Anthropic deployment in every mode: the edge buffers a
streamed response into one body, so chunk fidelity waits on LIT-5742."""
model, key = self._register(
endpoints_client,
resources,
LiteLLMParamsBody(model=ANTHROPIC_BACKEND, api_key="os.environ/ANTHROPIC_API_KEY"),
)
result = endpoints_client.proxy.messages_stream(
key,

50
tests/e2e/llm_translation/test_passthrough_e2e.py
View file

@ -28,6 +28,7 @@ from passthrough_client import (
)
EMBEDDING_MODEL = "text-embedding-3-small"
REALTIME_MODEL = "gpt-realtime-2"
pytestmark = pytest.mark.e2e
@ -339,3 +340,52 @@ class TestOpenAIPassthroughSpend:
f"the embeddings row logged no prompt tokens, so whatever cost it carries "
f"was not computed from the real usage: {row}"
)
class TestOpenAIPassthroughWebsocket:
"""The OpenAI passthrough prefixes must answer a websocket upgrade, not only a POST.
The customer points realtime and responses.connect clients at the same prefixes
their HTTP traffic already uses. Only HTTP routes were registered under those
prefixes, so every upgrade was refused before a socket existed and those clients
could not reach the gateway at all. A refused upgrade is an HTTP response, not a
close frame, which is why these assert on the handshake rather than a close code.
"""
@pytest.mark.covers("llm.realtime.openai.passthrough.stream.works")
def test_realtime_upgrade_reaches_openai_through_the_passthrough_prefix(
self, client: PassthroughClient, scoped_key: str
) -> None:
"""Pins GitHub issue #36088: /openai_passthrough/v1/realtime accepts the
upgrade and relays OpenAI's own session, instead of rejecting it with a 403."""
handshake = client.openai_passthrough_websocket(
scoped_key, "/openai_passthrough/v1/realtime", model=REALTIME_MODEL
)
assert handshake.rejected_status is None, (
f"/openai_passthrough/v1/realtime refused the websocket upgrade with HTTP "
f"{handshake.rejected_status}, so a realtime client cannot connect through "
"the gateway at all"
)
assert handshake.first_event_type == "session.created", (
"the accepted socket never carried OpenAI's opening session event, so the "
f"upgrade was not relayed upstream; the first frame was "
f"{handshake.first_event_type}"
)
@pytest.mark.covers("llm.responses.openai.passthrough_websocket.stream.works")
def test_responses_upgrade_is_accepted_on_the_openai_prefix(
self, client: PassthroughClient, scoped_key: str
) -> None:
"""Pins GitHub issue #36088 on the second prefix: /openai/v1/responses upgrades
as well. A responses.connect socket waits for the client to speak first, so the
accepted handshake is the whole signal here."""
handshake = client.openai_passthrough_websocket(
scoped_key, "/openai/v1/responses", first_event_timeout=2.0
)
assert handshake.rejected_status is None, (
f"/openai/v1/responses refused the websocket upgrade with HTTP "
f"{handshake.rejected_status}; the prefix relays this route over HTTP but "
"drops a responses.connect client before the socket opens"
)

274
tests/e2e/provider_edge.py
View file

@ -20,10 +20,9 @@ headers must never touch disk. An unmatched replay call returns HTTP
proxy relays as a provider error the failing test surfaces.
v1 limits: only the mounts in ``EDGE_MOUNTS`` (SigV4 providers like Bedrock
sign the Host header, so a forwarding edge breaks their signatures), JSON and
opaque single-part bodies (multipart boundaries are random per request),
streaming fidelity is LIT-5742, and CI wiring is LIT-5748. Suites that do not
wire the edge keep hitting providers live in every mode.
sign the Host header, so a forwarding edge breaks their signatures), streaming
fidelity is LIT-5742, and CI wiring is LIT-5748. Suites that do not wire the
edge keep hitting providers live in every mode.
"""
from __future__ import annotations
@ -32,6 +31,7 @@ import base64
import difflib
import functools
import hashlib
import re
import threading
from collections import deque
from collections.abc import Mapping
@ -59,7 +59,13 @@ from fixture_bundle import (
prepare_bundle,
slug_for_test,
)
from fixture_canonical import CanonicalRequest, canonical_string, canonicalize
from fixture_canonical import (
SECRET_PLACEHOLDER,
CanonicalRequest,
canonical_string,
canonicalize,
is_secret_field,
)
from fixture_mode import (
FIXTURE_MODES,
InvalidFixtureMode,
@ -103,26 +109,244 @@ _RESPONSE_DROPPED_HEADERS: Final[frozenset[str]] = _HOP_BY_HOP_HEADERS | {
_JSON: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
def _edge_request(method: str, path: str, query: str, body: bytes | None) -> RecordedRequest:
"""The identity replay matches on: the edge path (mount included), the query
as params, and the body as parsed JSON, or as a canonicalized content digest
when it is not JSON so opaque uploads still match across runs."""
params: Final = dict(parse_qsl(query, keep_blank_values=True))
if not body:
return RecordedRequest(method=method.lower(), path=path, headers={}, params=params)
decoded: Final = body.decode("utf-8", errors="replace")
_BOUNDARY_PATTERN: Final = re.compile(
r'(?:^|;)\s*boundary\s*=\s*(?:"([^"]*)"|([^;,\s]+))', re.IGNORECASE
)
_DISPOSITION_NAME_PATTERN: Final = re.compile(r'(?:^|;)\s*name="([^"]*)"', re.IGNORECASE)
_DISPOSITION_FILENAME_PATTERN: Final = re.compile(
r'(?:^|;)\s*filename="([^"]*)"', re.IGNORECASE
)
_UNPARSED_MULTIPART: Final = "<unparsed-multipart>"
_BOUNDARY_PLACEHOLDER: Final = b"--<boundary>"
_BINARY_FIELD_PREFIX: Final = "<binary:sha256:"
@dataclass(frozen=True)
class _MultipartPart:
field_name: str
filename: str | None
content: bytes
content_type: str = ""
def _header_value(headers: Mapping[str, str], name: str) -> str:
wanted: Final = name.lower()
return next((value for key, value in headers.items() if key.lower() == wanted), "")
def _multipart_boundary(content_type: str) -> str | None:
"""The declared boundary, or None when the envelope is not multipart or names no
usable boundary. ``boundary`` is matched only as a parameter in its own right, so a
longer name ending in it (``myboundary=``) is not mistaken for one, and an empty
boundary is refused rather than splitting the body on a bare ``--``."""
if "multipart/form-data" not in content_type.lower():
return None
match: Final = _BOUNDARY_PATTERN.search(content_type)
if match is None:
return None
quoted, bare = match.group(1), match.group(2)
return (quoted if quoted is not None else bare) or None
def _part_headers(head: bytes) -> dict[str, str]:
return {
name.strip().lower(): value.strip()
for line in head.decode("utf-8", errors="replace").split("\r\n")
for name, separator, value in [line.partition(":")]
if separator
}
def _parse_multipart_part(segment: bytes) -> _MultipartPart | None:
head, separator, content = segment.partition(b"\r\n\r\n")
if not separator:
return None
headers: Final = _part_headers(head)
disposition: Final = headers.get("content-disposition", "")
name_match: Final = _DISPOSITION_NAME_PATTERN.search(disposition)
if name_match is None:
return None
filename_match: Final = _DISPOSITION_FILENAME_PATTERN.search(disposition)
return _MultipartPart(
field_name=name_match.group(1),
filename=None if filename_match is None else filename_match.group(1),
content=content,
content_type=headers.get("content-type", ""),
)
def _multipart_parts(body: bytes, boundary: str) -> tuple[_MultipartPart, ...] | None:
"""The wire body split back into its parts, or None when it does not parse as the
declared envelope so the caller can fall back to the opaque content digest."""
segments: Final = body.split(b"--" + boundary.encode())
if len(segments) < 3 or not segments[-1].startswith(b"--"):
return None
parsed: Final = tuple(
_parse_multipart_part(segment.removeprefix(b"\r\n").removesuffix(b"\r\n"))
for segment in segments[1:-1]
)
if any(part is None for part in parsed):
return None
return tuple(part for part in parsed if part is not None)
def _content_digest(content: bytes) -> str:
"""Text is canonicalized before hashing so a per-run marker inside an uploaded JSONL
does not move the key; anything that is not UTF-8 is hashed byte for byte, since a
lossy decode collapses every binary payload of one length onto one digest."""
try:
parsed: Final[JsonValue] = _JSON.validate_json(decoded)
except ValueError:
return RecordedRequest(
method=method.lower(),
path=path,
headers={},
params=params,
file_sha256=hashlib.sha256(canonical_string(decoded).encode()).hexdigest(),
file_bytes=len(body),
text: Final = content.decode("utf-8")
except UnicodeDecodeError:
return hashlib.sha256(content).hexdigest()
return hashlib.sha256(canonical_string(text).encode()).hexdigest()
def _is_file_part(part: _MultipartPart) -> bool:
"""Whether a part is an upload rather than an ordinary field. A filename says so
outright, and so does a declared content type: clients attach one per part only for
a file, and a client that omits the filename (httpx drops the parameter when it is
empty) would otherwise have the file's bytes stored inline as a field value and key
identically to a plain field of the same name."""
return part.filename is not None or bool(part.content_type)
def _field_value(part: _MultipartPart) -> str:
"""What a field part contributes to the stored form. A secret-named field never has
its value written out, since the bundle is a file on disk and the key redacts that
field to the same placeholder either way, so replay still matches. A value that is
not UTF-8 is carried as a digest rather than decoded lossily, because a replacing
decode collapses every binary value of one length onto one string. That digest is
base64 rather than hex, since the canonicalizer rewrites any long hex run to a
``<sha256>`` placeholder and would collapse the values right back together."""
if is_secret_field(part.field_name):
return SECRET_PLACEHOLDER
try:
return part.content.decode("utf-8")
except UnicodeDecodeError:
digest: Final = base64.b64encode(hashlib.sha256(part.content).digest()).decode()
return f"{_BINARY_FIELD_PREFIX}{digest}>"
def _form_fields(fields: tuple[_MultipartPart, ...]) -> dict[str, str]:
"""The ordinary field parts, flattened into the mapping the bundle format stores. A
name sent more than once takes an occurrence suffix instead of overwriting the
earlier value, so nothing an upload said is dropped from its key. The suffix is
escaped so a field literally named ``x[1]`` cannot collide with a second ``x``."""
form: dict[str, str] = {}
for part in fields:
name = part.field_name.replace("[", "[[")
occurrence = 1
while name in form:
name = f"{part.field_name.replace('[', '[[')}[{occurrence}]"
occurrence += 1
form[name] = _field_value(part)
return form
def _file_identity(files: tuple[_MultipartPart, ...]) -> tuple[str | None, str | None, int | None]:
"""Name, content digest, and total length for the uploaded file parts.
The name is a structured list of every part's field name, filename, and declared
content type rather than a joined string, so a filename containing the separator
cannot be confused for a different split, and two parts that differ only in the type
they declare stay apart. It goes through the canonicalizer as one string, which is
why per-run markers inside a filename do not move the key in the multi-file case any
more than they do in the single-file one.
The digest covers content only. A lone file keeps its own canonicalized digest;
several fold into one ordered digest, so parts arriving in a different order key
differently. Total length is recorded for a reader but deliberately kept out of the
key: it is the raw byte count, and keying on it would undo exactly the drift the
canonicalized digest exists to absorb."""
if not files:
return None, None, None
names: Final = _JSON.dump_json(
[[part.field_name, part.filename, part.content_type] for part in files]
).decode()
total: Final = sum(len(part.content) for part in files)
if len(files) == 1:
return names, _content_digest(files[0].content), total
folded: Final = _JSON.dump_json([_content_digest(part.content) for part in files])
return names, hashlib.sha256(folded).hexdigest(), total
def _multipart_request(
method: str, path: str, params: dict[str, str], parts: tuple[_MultipartPart, ...]
) -> RecordedRequest:
"""A multipart upload keyed by what it says rather than by its wire bytes: every
ordinary field, plus the identity of the uploaded file. The random per-request
boundary is envelope, never content, so it never reaches the digest."""
form: Final = _form_fields(tuple(part for part in parts if not _is_file_part(part)))
file_name, file_sha256, file_bytes = _file_identity(
tuple(part for part in parts if _is_file_part(part))
)
return RecordedRequest(
method=method,
path=path,
headers={},
params=params,
form=form,
file_name=file_name,
file_sha256=file_sha256,
file_bytes=file_bytes,
)
def _opaque_request(
method: str,
path: str,
params: dict[str, str],
body: bytes,
digested: bytes,
file_name: str | None = None,
) -> RecordedRequest:
"""A body kept out of the bundle and matched on its digest alone. ``digested`` is
what the digest runs over, which is the body itself unless something in it has to be
normalized away first."""
return RecordedRequest(
method=method,
path=path,
headers={},
params=params,
file_name=file_name,
file_sha256=_content_digest(digested),
file_bytes=len(body),
)
def edge_request(
method: str, path: str, query: str, body: bytes | None, content_type: str = ""
) -> RecordedRequest:
"""The identity replay matches on: the edge path (mount included), the query as
params, and the body as parsed JSON, as parsed multipart fields and file identity
when the content type declares an envelope, or as a content digest otherwise so
opaque uploads still match across runs. A multipart body that does not parse still
has its boundary normalized away, because that boundary is fresh every request and
would otherwise guarantee a miss."""
params: Final = dict(parse_qsl(query, keep_blank_values=True))
lowered_method: Final = method.lower()
if not body:
return RecordedRequest(method=lowered_method, path=path, headers={}, params=params)
boundary: Final = _multipart_boundary(content_type)
if boundary is not None:
parts = _multipart_parts(body, boundary)
if parts is not None:
return _multipart_request(lowered_method, path, params, parts)
return _opaque_request(
lowered_method,
path,
params,
body,
body.replace(b"--" + boundary.encode(), _BOUNDARY_PLACEHOLDER),
_UNPARSED_MULTIPART,
)
return RecordedRequest(method=method.lower(), path=path, headers={}, params=params, body=parsed)
try:
parsed: Final[JsonValue] = _JSON.validate_json(body)
except ValueError:
return _opaque_request(lowered_method, path, params, body, body)
return RecordedRequest(
method=lowered_method, path=path, headers={}, params=params, body=parsed
)
def _build_pool(recorded: tuple[Interaction, ...]) -> dict[str, deque[Interaction]]:
@ -351,7 +575,9 @@ def handle_edge_request(
return _text_reply(
404, f"unknown provider mount {mount!r}; known mounts: {', '.join(sorted(mounts))}"
)
request: Final = _edge_request(method, split.path, split.query, body)
request: Final = edge_request(
method, split.path, split.query, body, _header_value(headers, "content-type")
)
match backend:
case RecordEdge():
return _handle_record(

2
tests/e2e/router/reliability_support.py
View file

@ -56,7 +56,7 @@ def chat_override(
json=ReliabilityChatBody(
model=model,
messages=[ChatMessage(role="user", content=content)],
max_tokens=16,
max_tokens=64,
stream=stream,
router_settings_override=override,
),

4
tests/e2e/router/test_reliability_fallbacks_e2e.py
View file

@ -49,7 +49,7 @@ class TestReliabilityFallbacks:
resources.defer(lambda: client.proxy.delete_model(model_id))
resp = chat_override(
client.proxy, scoped_key, primary, "say hi",
client.proxy, scoped_key, primary, f"say hi {unique_marker()}",
override=RouterSettingsOverride(fallbacks=[{primary: ["gpt-5.5"]}]),
)
_assert_served_by_fallback(resp)
@ -63,7 +63,7 @@ class TestReliabilityFallbacks:
resources.defer(lambda: client.proxy.delete_model(model_id))
resp = chat_override(
client.proxy, scoped_key, primary, "say hi",
client.proxy, scoped_key, primary, f"say hi {unique_marker()}",
override=RouterSettingsOverride(fallbacks=[{primary: ["gpt-5.5"]}]),
)
_assert_served_by_fallback(resp)

386
tests/e2e/test_provider_edge.py
View file

@ -23,11 +23,13 @@ from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Final
import pytest
from pydantic import TypeAdapter
from e2e_http import RawResponse, forward
from fixture_canonical import canonicalize
from fixture_bundle import (
BundleRecorder,
Interaction,
@ -46,6 +48,7 @@ from provider_edge import (
RecordEdge,
ReplayEdge,
ReplaySource,
edge_request,
handle_edge_request,
provider_edge_api_base,
replay_leftover_error,
@ -53,8 +56,10 @@ from provider_edge import (
)
CHAT_PATH = "/openai/v1/chat/completions"
UPLOAD_PATH = "/openai/v1/files"
REPLAY_MOUNTS = {"openai": "https://replay.invalid"}
JSON_OBJECT = TypeAdapter(dict[str, object])
BATCH_JSONL = b'{"custom_id":"one"}\n{"custom_id":"two"}\n'
def json_object(body: bytes) -> dict[str, object]:
@ -164,6 +169,46 @@ def chat_body(prompt: str) -> bytes:
return json.dumps({"model": "gpt", "messages": [{"role": "user", "content": prompt}]}).encode()
def multipart_body(
boundary: str,
fields: tuple[tuple[str, str], ...] = (),
files: tuple[tuple[str, str, bytes], ...] = (),
) -> bytes:
"""One multipart/form-data body on the wire, exactly as ``requests`` writes it, with
the boundary under the caller's control instead of randomly generated."""
parts = [
f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"\r\n\r\n'.encode()
+ value.encode()
for name, value in fields
] + [
(
f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"; '
f'filename="{filename}"\r\nContent-Type: application/octet-stream\r\n\r\n'
).encode()
+ content
for name, filename, content in files
]
return b"\r\n".join(parts) + f"\r\n--{boundary}--\r\n".encode()
def upload_headers(boundary: str) -> dict[str, str]:
return {
"content-type": f"multipart/form-data; boundary={boundary}",
"authorization": "Bearer sk-upload-secret",
}
def record_upload(root: Path, body: bytes, boundary: str) -> None:
with fake_provider() as provider:
with running_edge(record_backend(root), {"openai": provider_url(provider)}) as edge:
call_edge(edge, "POST", UPLOAD_PATH, body=body, headers=upload_headers(boundary))
def replay_upload(root: Path, body: bytes, boundary: str) -> RawResponse:
with running_edge(ReplayEdge(source=replay_source(root)), REPLAY_MOUNTS) as edge:
return call_edge(edge, "POST", UPLOAD_PATH, body=body, headers=upload_headers(boundary))
class TestRecordMode:
def test_forwards_to_the_provider_and_writes_one_interaction_file(self, tmp_path: Path) -> None:
root = tmp_path / "bundle"
@ -328,6 +373,347 @@ class TestReplayMode:
assert replayed.status_code == 200
class TestMultipartIdentity:
"""LIT-5974: a multipart upload is keyed by its parsed fields and file identity.
``requests`` picks a fresh random boundary per request, so hashing the wire body
made every upload miss on replay; parsing the envelope keys the upload on what it
actually says, which is stable across runs and still separates real drift."""
def test_a_fresh_boundary_replays_the_same_upload(self, tmp_path: Path) -> None:
root = tmp_path / "bundle"
recorded = multipart_body(
"d0a1b2c3d4e5f60718293a4b5c6d7e8f",
fields=(("purpose", "batch"),),
files=(("file", "batch.jsonl", BATCH_JSONL),),
)
record_upload(root, recorded, "d0a1b2c3d4e5f60718293a4b5c6d7e8f")
rerun = multipart_body(
"ffffeeeeddddccccbbbbaaaa99998888",
fields=(("purpose", "batch"),),
files=(("file", "batch.jsonl", BATCH_JSONL),),
)
assert rerun != recorded
replayed = replay_upload(root, rerun, "ffffeeeeddddccccbbbbaaaa99998888")
assert replayed.status_code == 200, replayed.body[:400]
def test_the_stored_request_carries_fields_and_file_identity_but_no_secrets(
self, tmp_path: Path
) -> None:
root = tmp_path / "bundle"
boundary = "0123456789abcdef0123456789abcdef"
record_upload(
root,
multipart_body(
boundary,
fields=(("purpose", "batch"),),
files=(("file", "batch.jsonl", BATCH_JSONL),),
),
boundary,
)
raw = this_tests_files(root)[0].read_text(encoding="utf-8")
interaction = Interaction.model_validate_json(raw)
assert interaction.request.form == {"purpose": "batch"}
assert interaction.request.file_name == json.dumps(
[["file", "batch.jsonl", "application/octet-stream"]], separators=(",", ":")
)
assert interaction.request.file_bytes == len(BATCH_JSONL)
stored = interaction.request.model_dump_json()
assert boundary not in stored
assert "sk-upload-secret" not in stored
assert "custom_id" not in stored
@pytest.mark.parametrize(
("fields", "files"),
[
pytest.param(
(("purpose", "batch"),),
(("file", "batch.jsonl", b'{"custom_id":"three"}\n'),),
id="file-content",
),
pytest.param(
(("purpose", "batch"),),
(("file", "other.jsonl", BATCH_JSONL),),
id="file-name",
),
pytest.param(
(("purpose", "fine-tune"),),
(("file", "batch.jsonl", BATCH_JSONL),),
id="form-field",
),
pytest.param(
(("purpose", "batch"), ("purpose", "batch")),
(("file", "batch.jsonl", BATCH_JSONL),),
id="repeated-form-field",
),
pytest.param(
(("purpose", "batch"),),
(
("file", "batch.jsonl", BATCH_JSONL),
("mask", "mask.jsonl", BATCH_JSONL),
),
id="extra-file-part",
),
],
)
def test_a_structurally_different_upload_misses(
self,
tmp_path: Path,
fields: tuple[tuple[str, str], ...],
files: tuple[tuple[str, str, bytes], ...],
) -> None:
root = tmp_path / "bundle"
record_upload(
root,
multipart_body(
"aaaaaaaabbbbbbbbccccccccdddddddd",
fields=(("purpose", "batch"),),
files=(("file", "batch.jsonl", BATCH_JSONL),),
),
"aaaaaaaabbbbbbbbccccccccdddddddd",
)
drifted = replay_upload(
root,
multipart_body("11112222333344445555666677778888", fields=fields, files=files),
"11112222333344445555666677778888",
)
assert drifted.status_code == REPLAY_MISS_STATUS
def test_several_file_parts_separate_when_their_contents_swap(self, tmp_path: Path) -> None:
root = tmp_path / "bundle"
image, mask = b"image-bytes", b"mask-bytes"
record_upload(
root,
multipart_body(
"1a1a1a1a2b2b2b2b3c3c3c3c4d4d4d4d",
fields=(("prompt", "a cat"),),
files=(("image", "a.png", image), ("mask", "b.png", mask)),
),
"1a1a1a1a2b2b2b2b3c3c3c3c4d4d4d4d",
)
swapped = replay_upload(
root,
multipart_body(
"5e5e5e5e6f6f6f6f7070707081818181",
fields=(("prompt", "a cat"),),
files=(("image", "a.png", mask), ("mask", "b.png", image)),
),
"5e5e5e5e6f6f6f6f7070707081818181",
)
assert swapped.status_code == REPLAY_MISS_STATUS
same = replay_upload(
root,
multipart_body(
"9292929203030303a4a4a4a4b5b5b5b5",
fields=(("prompt", "a cat"),),
files=(("image", "a.png", image), ("mask", "b.png", mask)),
),
"9292929203030303a4a4a4a4b5b5b5b5",
)
assert same.status_code == 200, same.body[:400]
def test_a_body_that_does_not_match_its_declared_boundary_stays_opaque(
self, tmp_path: Path
) -> None:
root = tmp_path / "bundle"
opaque = b"custom_id one\ncustom_id two\n"
absent = "boundary-that-is-absent-from-the-body"
record_upload(root, opaque, absent)
raw = this_tests_files(root)[0].read_text(encoding="utf-8")
interaction = Interaction.model_validate_json(raw)
assert interaction.request.form is None
assert interaction.request.file_name == "<unparsed-multipart>"
assert interaction.request.file_bytes == len(opaque)
assert "custom_id" not in interaction.request.model_dump_json()
assert replay_upload(root, opaque, absent).status_code == 200
def raw_multipart(boundary: str, *parts: tuple[str, bytes]) -> bytes:
"""A body assembled from literal part headers, so a test can send the shapes a
well-formed helper cannot: a file part with no filename, a declared per-part content
type, a repeated or bracketed field name, or a non-UTF-8 value."""
return (
b"".join(
f"--{boundary}\r\n{head}\r\n\r\n".encode() + content + b"\r\n"
for head, content in parts
)
+ f"--{boundary}--\r\n".encode()
)
def upload_key(body: bytes, boundary: str) -> str:
content_type: Final = f"multipart/form-data; boundary={boundary}"
return canonicalize(edge_request("POST", UPLOAD_PATH, "", body, content_type)).key
DISPOSITION = 'Content-Disposition: form-data; name="{name}"'
FILE_DISPOSITION = DISPOSITION + '; filename="{filename}"'
class TestMultipartIdentityEdges:
"""The identity a multipart upload keys on, pinned against the ways two materially
different uploads could otherwise collapse onto one key. A collision here is the
dangerous failure: replay would answer one request with another's response."""
def test_a_declared_part_content_type_separates_otherwise_identical_uploads(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
as_json = raw_multipart(
boundary,
(FILE_DISPOSITION.format(name="file", filename="a") + "\r\nContent-Type: application/json", b"xy"),
)
as_csv = raw_multipart(
boundary,
(FILE_DISPOSITION.format(name="file", filename="a") + "\r\nContent-Type: text/csv", b"xy"),
)
assert upload_key(as_json, boundary) != upload_key(as_csv, boundary)
def test_a_file_part_without_a_filename_is_not_mistaken_for_a_plain_field(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
upload = raw_multipart(
boundary,
(DISPOSITION.format(name="file") + "\r\nContent-Type: application/octet-stream", b"CONTENT"),
)
plain_field = raw_multipart(boundary, (DISPOSITION.format(name="file"), b"CONTENT"))
request = edge_request(
"POST", UPLOAD_PATH, "", upload, f"multipart/form-data; boundary={boundary}"
)
assert upload_key(upload, boundary) != upload_key(plain_field, boundary)
assert request.form == {}
assert b"CONTENT".decode() not in request.model_dump_json()
def test_a_filename_carrying_a_per_run_marker_keys_the_same_next_run(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
def upload(marker: str) -> str:
body = raw_multipart(
boundary,
(FILE_DISPOSITION.format(name="one", filename=f"{marker}.jsonl"), b"first"),
(FILE_DISPOSITION.format(name="two", filename="steady.jsonl"), b"second"),
)
return upload_key(body, boundary)
assert upload("a1b2c3d4e5f6") == upload("0f9e8d7c6b5a")
def test_a_separator_inside_a_filename_cannot_forge_a_different_split(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
colon_in_filename = raw_multipart(
boundary, (FILE_DISPOSITION.format(name="file", filename="a:b.jsonl"), b"same")
)
colon_in_field = raw_multipart(
boundary, (FILE_DISPOSITION.format(name="file:a", filename="b.jsonl"), b"same")
)
assert upload_key(colon_in_filename, boundary) != upload_key(colon_in_field, boundary)
def test_a_repeated_field_cannot_collide_with_a_literal_indexed_name(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
repeated = raw_multipart(
boundary,
(DISPOSITION.format(name="purpose"), b"x"),
(DISPOSITION.format(name="purpose"), b"y"),
)
literal_index = raw_multipart(
boundary,
(DISPOSITION.format(name="purpose"), b"x"),
(DISPOSITION.format(name="purpose[1]"), b"y"),
)
assert upload_key(repeated, boundary) != upload_key(literal_index, boundary)
def test_two_binary_field_values_of_one_length_stay_apart(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
first = raw_multipart(boundary, (DISPOSITION.format(name="blob"), b"\xff\xfe\xfd"))
second = raw_multipart(boundary, (DISPOSITION.format(name="blob"), b"\xf0\xf1\xf2"))
assert upload_key(first, boundary) != upload_key(second, boundary)
def test_a_secret_named_field_never_reaches_the_stored_request(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
body = raw_multipart(
boundary,
(DISPOSITION.format(name="openai_api_key"), b"sk-live-DEADBEEF-0123456789abcd"),
(DISPOSITION.format(name="purpose"), b"batch"),
)
request = edge_request(
"POST", UPLOAD_PATH, "", body, f"multipart/form-data; boundary={boundary}"
)
assert "sk-live-DEADBEEF-0123456789abcd" not in request.model_dump_json()
assert request.form == {"openai_api_key": "<secret>", "purpose": "batch"}
def test_a_redacted_field_still_matches_the_live_request_that_carried_the_secret(
self,
) -> None:
boundary = "0123456789abcdef0123456789abcdef"
def upload(secret: str) -> str:
body = raw_multipart(
boundary,
(DISPOSITION.format(name="openai_api_key"), secret.encode()),
(DISPOSITION.format(name="purpose"), b"batch"),
)
return upload_key(body, boundary)
assert upload("sk-live-DEADBEEF-0123456789abcd") == upload("<secret>")
def test_a_length_change_the_canonicalizer_absorbs_does_not_move_the_key(self) -> None:
boundary = "0123456789abcdef0123456789abcdef"
def upload(created: str) -> str:
body = raw_multipart(
boundary,
(
FILE_DISPOSITION.format(name="file", filename="batch.jsonl"),
b'{"created_at":"' + created.encode() + b'"}',
),
)
return upload_key(body, boundary)
assert upload("2026-08-21T02:08:19Z") == upload("2026-08-21T02:08:19.123456Z")
@pytest.mark.parametrize(
"content_type",
[
pytest.param("multipart/form-data; myboundary=zzz; boundary={boundary}", id="lookalike-parameter"),
pytest.param("multipart/form-data; BOUNDARY={boundary}", id="uppercase-parameter"),
],
)
def test_the_boundary_parameter_is_read_the_way_the_client_meant_it(
self, content_type: str
) -> None:
boundary = "0123456789abcdef0123456789abcdef"
body = raw_multipart(
boundary, (FILE_DISPOSITION.format(name="file", filename="batch.jsonl"), BATCH_JSONL)
)
request = edge_request(
"POST", UPLOAD_PATH, "", body, content_type.format(boundary=boundary)
)
assert request.form == {}
assert request.file_name is not None
assert "batch.jsonl" in request.file_name
def test_an_empty_declared_boundary_falls_back_instead_of_splitting_on_dashes(self) -> None:
body = b'--\r\nContent-Disposition: form-data; name="a"\r\n\r\nvalue\r\n----\r\n'
request = edge_request(
"POST", UPLOAD_PATH, "", body, 'multipart/form-data; boundary=""'
)
assert request.form is None
assert request.file_sha256 is not None
class TestReplayLeftover:
def test_partially_consumed_recording_names_the_leftover(self, tmp_path: Path) -> None:
root = tmp_path / "bundle"

9
tests/enterprise/conftest.py
View file

@ -3,13 +3,9 @@
import asyncio
import importlib
import os
import sys
import pytest
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm
@ -31,9 +27,6 @@ def setup_and_teardown():
This fixture reloads litellm before every function. To speed up testing by removing callbacks being chained.
"""
curr_dir = os.getcwd() # Get the current working directory
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the project directory to the system path
from litellm import Router
@ -41,8 +34,6 @@ def setup_and_teardown():
try:
if hasattr(litellm, "proxy") and hasattr(litellm.proxy, "proxy_server"):
import litellm.proxy.proxy_server
importlib.reload(litellm.proxy.proxy_server)
except Exception as e:
print(f"Error reloading litellm.proxy.proxy_server: {e}")

3
tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py
View file

@ -1,7 +1,4 @@
import os
import sys
sys.path.insert(0, os.path.abspath("../.."))
import asyncio
import logging

5
tests/enterprise/litellm_enterprise/integrations/test_custom_guardrail.py
View file

@ -1,9 +1,4 @@
import os
import sys
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system-path
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.types.guardrails import GuardrailEventHooks, Mode

Some files were not shown because too many files have changed in this diff Show more