scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102

chore(release): backport 11 staging PRs onto patch-1.93.0rc2 for the 1.93.0 stable cut (#33847)

Browse source 
* fix(ci): bump pillow to 12.3.0 to resolve osv-scan CVEs (#33093)

(cherry picked from commit 20e646c49a)

* chore(deps): pin httplib2 and setuptools transitive floors (#33233)

Raise the constraint floors for two transitive dependencies so resolution moves them to their latest maintenance releases: httplib2 0.31.2 -> 0.32.0 and setuptools 82.0.1 -> 83.0.0. Both are pulled in only by optional integrations (Google API client, grpc tooling, lunary observability, the nvidia-riva extra), all lower-bound only, so the floors stay inside every requirer's allowed range and a default install is unaffected

(cherry picked from commit 8b323202ec)

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244)

* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models

* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models

Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 71dffc1e9a)

* build: raise requires-python cap to <3.15 so Python 3.14 installs current releases (#33438)

* build: drop requires-python upper cap so Python 3.14 resolves to current releases

The <3.14 cap made pip on Python 3.14 fall back to litellm 1.83.7, a
pre-April release whose old auth flow fails with 400s. The cap was added
in d9a460277a because deps lacked 3.14 wheels and uv could not resolve
the 3.14 split; both are fixed now via the existing python_version
markers plus a ddtrace version split (2.x has no cp314 wheels, 3.16+
does). Verified on 3.14.5: uv sync --all-extras installs, litellm and
proxy_server import (rust bridge falls back to pure python), real
provider calls succeed sync/async/streaming, and the core-utils test
suite passes.

* build: cap requires-python at <3.15 and keep ddtrace on one major per python band

Reviewer preference to bound the supported window at the newest tested
minor rather than leaving it open-ended, and Greptile flagged the
ddtrace 3.14+ range spanning two majors; every ddtrace 4.x ships cp314
wheels so the band is now >=4.0,<5.0, matching the single-major
convention of the 2.x band.

(cherry picked from commit c6d49a85b2)

* build(deps): update ddtrace to the 4.x line

A single ddtrace constraint now covers every supported Python version, so this collapses the version split introduced in #33438. Also aligns the build_from_pip image pin and updates the type-only Tracer import to its current module path

(cherry picked from commit edc38eab34)

* fix(docker): restore litellm-proxy-extras source dir in runtime images (#33592)

* fix(docker): restore litellm-proxy-extras source dir in runtime images

#30243 narrowed the runtime stage to an allowlist COPY, which dropped
/app/litellm-proxy-extras from the published images. Downstream
migration jobs point prisma migrate deploy at that path; with the
schema gone (or a schema with no adjacent migrations dir, where prisma
exits 0 without applying anything) those jobs went green while never
migrating the database. Restore the folder in all three runtime stages
and assert in image-scan that the schema and a non-empty migrations dir
ship at the source path

* chore(ci): drop image-scan migration-assets assertion

(cherry picked from commit 111d447e1b)

* fix(model_armor): restore reference attachments via skip_unscannable_attachments and remove the attachment count cap (#33554)

* fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through

* fix(model_armor): wire skip_unscannable_attachments through guardrail config

* fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping

* fix(model_armor): remove the per-request attachment count cap and scan all attachments

---------

Co-authored-by: yucheng <yucheng@berri.ai>
(cherry picked from commit 0d7b0f708b)

* build(rust): raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 (#33798)

pyo3 0.23.5 hard-caps the interpreter at Python 3.13, so building the
native bridge against a 3.14 interpreter aborts inside pyo3-ffi's build
script before anything links. This raises pyo3 and pyo3-async-runtimes
to 0.29 (currently the newest line, and the range starting at 0.26 that
supports 3.14) and migrates the three call sites whose APIs were renamed
across that range: Python::with_gil is now Python::attach and
Python::allow_threads is now Python::detach. On a GIL-enabled interpreter
those are pure renames with identical semantics, so behavior on 3.10
through 3.13 is unchanged

Verified by compiling the native module for cp313 and cp314 and driving
it directly on both interpreters: gil_stats reports exactly one GIL
release per sync OCR call and the async path completes, matching the
0.23.5 baseline. cargo fmt, clippy, and the workspace tests pass on both
3.13 and 3.14 with the lockfile locked, and the lock churn is confined to
the pyo3 crates

Part of #26343; addresses the pyo3 build failure reported in #33116

(cherry picked from commit f3d20153b3)

* build(deps): allow redisvl, pypdf, and openapi-core on Python 3.14 (#33801)

Remove the python_version < '3.14' environment markers from redisvl,
pypdf, and openapi-core now that all three install and import cleanly
on 3.14. The relock is marker-only: no package version changed for any
Python branch, and the locked versions (redisvl 0.4.1, pypdf 6.13.3,
openapi-core 0.22.0) now serve 3.14 as well. semantic-router and
aurelio-sdk stay gated because every published release caps
python_requires below 3.14

(cherry picked from commit 967d934484)

* build(deps): bump mcp lock to 1.28.1 to clear image-scan findings (#33803)

* build(deps): bump mcp lock to 1.28.1 to clear image-scan findings

* build(deps): require mcp>=1.28.1

(cherry picked from commit 40e914cfa7)

* fix(proxy): source /v1/models token limits from the cost map instead of Router.get_model_group_info (#33721)

* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info

Resolves the per-model get_model_group_info fan-out on GET /v1/models
(and /models) that pegged the event loop on wildcard listings (#33636).
create_model_info_response now reads max_input_tokens/max_output_tokens
from litellm.get_model_info (the static cost map) rather than the router,
which aggregated and deepcopied every deployment in a group per listed
model.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage

Inject the cost-map lookup (defaulting to litellm.get_model_info) so the
except and max_output_tokens branches are exercised deterministically and
the token-limit tests no longer hardcode mutable cost-map values.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup

Add Router.get_configured_token_limits, an O(1) model-name index lookup that
reads a concrete deployment's configured max_input_tokens/max_output_tokens
without triggering pattern matching or deep copies. create_model_info_response
layers this over the cost map so custom deployments absent from the cost map
still surface their limits, and admin-configured limits override cost-map
defaults, while wildcard-expanded names stay on the fast path.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 8536e3b80e)

---------

Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
This commit is contained in:
yuneng-jiang 2026-07-18 14:19:56 -07:00 • committed by GitHub
parent b5cd431c1e
commit 2530e32727
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
24 changed files with 1735 additions and 439 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
Dockerfile
View file

@ -114,6 +114,7 @@ COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/pr
# working directory on sys.path; litellm/proxy/hooks resolves
# enterprise.enterprise_hooks from it)
COPY --from=builder /app/enterprise /app/enterprise
COPY --from=builder /app/litellm-proxy-extras /app/litellm-proxy-extras
# Prisma binaries live in $HOME/.cache (default prisma-python location),
# which is /root/.cache here. Copy only the Prisma subdirs — copying the
# whole /root/.cache drags in the uv build cache (~660 MB, includes a

1
docker/Dockerfile.database
View file

@ -111,6 +111,7 @@ COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/pr
# working directory on sys.path; litellm/proxy/hooks resolves
# enterprise.enterprise_hooks from it)
COPY --from=builder /app/enterprise /app/enterprise
COPY --from=builder /app/litellm-proxy-extras /app/litellm-proxy-extras
# Prisma binaries live in $HOME/.cache (default prisma-python location),
# which is /root/.cache here. Copy them from the builder so they survive
# deployments that volume-mount /app/.cache (e.g. readOnlyRootFilesystem

1
docker/Dockerfile.non_root
View file

@ -137,6 +137,7 @@ COPY --from=builder /app/litellm/proxy/prisma_migration.py /app/litellm/proxy/pr
# working directory on sys.path; litellm/proxy/hooks resolves
# enterprise.enterprise_hooks from it)
COPY --from=builder /app/enterprise /app/enterprise
COPY --from=builder /app/litellm-proxy-extras /app/litellm-proxy-extras
COPY --from=builder /app/.cache /app/.cache
COPY --from=builder /var/lib/litellm/ui /var/lib/litellm/ui
COPY --from=builder /var/lib/litellm/assets /var/lib/litellm/assets

2
docker/build_from_pip/Dockerfile.build_from_pip
View file

@ -36,7 +36,7 @@ RUN uv venv --python python && \
"opentelemetry-api==1.28.0" \
"opentelemetry-sdk==1.28.0" \
"opentelemetry-exporter-otlp==1.28.0" \
"ddtrace==2.19.0" \
"ddtrace==4.11.0" \
"sentry-sdk==2.21.0" \
"mangum==0.17.0" \
"azure-ai-contentsafety==1.0.0" \

94
litellm-rust/Cargo.lock generated
View file

@ -19,12 +19,6 @@ version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "axum"
version = "0.7.9"
@ -233,21 +227,6 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "futures"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d"
dependencies = [
"futures-channel",
"futures-core",
"futures-executor",
"futures-io",
"futures-sink",
"futures-task",
"futures-util",
]
[[package]]
name = "futures-channel"
version = "0.3.32"
@ -264,17 +243,6 @@ version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d"
[[package]]
name = "futures-executor"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d"
dependencies = [
"futures-core",
"futures-task",
"futures-util",
]
[[package]]
name = "futures-io"
version = "0.3.32"
@ -310,7 +278,6 @@ version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6"
dependencies = [
"futures-channel",
"futures-core",
"futures-io",
"futures-macro",
@ -608,15 +575,6 @@ dependencies = [
"hashbrown",
]
[[package]]
name = "indoc"
version = "2.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706"
dependencies = [
"rustversion",
]
[[package]]
name = "ipnet"
version = "2.12.0"
@ -718,15 +676,6 @@ version = "2.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
[[package]]
name = "memoffset"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a"
dependencies = [
"autocfg",
]
[[package]]
name = "mime"
version = "0.3.17"
@ -803,29 +752,26 @@ dependencies = [
[[package]]
name = "pyo3"
version = "0.23.5"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7778bffd85cf38175ac1f545509665d0b9b92a198ca7941f131f85f7a4f9a872"
checksum = "cd274650b21d4bfc26a0a47587962c1edb425f69287324355cd040c3ea66071c"
dependencies = [
"cfg-if",
"indoc",
"libc",
"memoffset",
"once_cell",
"portable-atomic",
"pyo3-build-config",
"pyo3-ffi",
"pyo3-macros",
"unindent",
]
[[package]]
name = "pyo3-async-runtimes"
version = "0.23.0"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "977dc837525cfd22919ba6a831413854beb7c99a256c03bf8624ad707e45810e"
checksum = "b3ef68daa7316a3fac65e5e18b2203f010346de1c1c53456811a2624673ab046"
dependencies = [
"futures",
"futures-channel",
"futures-util",
"once_cell",
"pin-project-lite",
"pyo3",
@ -834,19 +780,18 @@ dependencies = [
[[package]]
name = "pyo3-build-config"
version = "0.23.5"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94f6cbe86ef3bf18998d9df6e0f3fc1050a8c5efa409bf712e661a4366e010fb"
checksum = "c5e2a7d2f0d013342f295c048ad19237add5154a55b1c5a254c0ec93d4109078"
dependencies = [
"once_cell",
"target-lexicon",
]
[[package]]
name = "pyo3-ffi"
version = "0.23.5"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9f1b4c431c0bb1c8fb0a338709859eed0d030ff6daa34368d3b152a63dfdd8d"
checksum = "ca85c467da1bbc8d866eea5deff9cf29ea5f7785054a17da36e65bda9c05845b"
dependencies = [
"libc",
"pyo3-build-config",
@ -854,9 +799,9 @@ dependencies = [
[[package]]
name = "pyo3-macros"
version = "0.23.5"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fbc2201328f63c4710f68abdf653c89d8dbc2858b88c5d88b0ff38a75288a9da"
checksum = "9ac53762fd065daa3194dd09337a38bd793a188100fd1a9304c4ab312d901771"
dependencies = [
"proc-macro2",
"pyo3-macros-backend",
@ -866,13 +811,12 @@ dependencies = [
[[package]]
name = "pyo3-macros-backend"
version = "0.23.5"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fca6726ad0f3da9c9de093d6f116a93c1a38e417ed73bf138472cf4064f72028"
checksum = "4ca3a1557399783172dc5bf39cfca835157732532cba56b71d2292161e53b362"
dependencies = [
"heck",
"proc-macro2",
"pyo3-build-config",
"quote",
"syn",
]
@ -1321,9 +1265,9 @@ dependencies = [
[[package]]
name = "target-lexicon"
version = "0.12.16"
version = "0.13.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c41af27dd6d1e27b1b16b489db798443478cef1f06a660c96db617ba5de3b1"
checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
[[package]]
name = "thiserror"
@ -1559,12 +1503,6 @@ version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "unindent"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7264e107f553ccae879d21fbea1d6724ac785e8c3bfc762137959b5802826ef3"
[[package]]
name = "untrusted"
version = "0.9.0"

4
litellm-rust/Cargo.toml
View file

@ -15,8 +15,8 @@ repository = "https://github.com/BerriAI/litellm"
litellm-core = { path = "crates/core" }
litellm-ai-gateway = { path = "crates/ai-gateway", default-features = false }
axum = "0.7"
pyo3 = "0.23.5"
pyo3-async-runtimes = { version = "0.23.0", features = ["tokio-runtime"] }
pyo3 = "0.29.0"
pyo3-async-runtimes = { version = "0.29.0", features = ["tokio-runtime"] }
rand = "0.8"
reqwest = { version = "0.12", default-features = false, features = ["blocking", "json", "rustls-tls", "http2", "stream"] }
serde = { version = "1.0", features = ["derive"] }

2
litellm-rust/crates/ai-gateway/src/python/config.rs
View file

@ -17,7 +17,7 @@ use crate::gil;
/// Load the router's `model_list` from `config_path` via the Python reader.
pub fn load_router_from_config(config_path: &str) -> CoreResult<Router> {
gil::record_acquisition();
Python::with_gil(|py| {
Python::attach(|py| {
let model_list = py
.import("litellm.proxy.read_model_list")
.and_then(|module| module.getattr("read_model_list"))

4
litellm-rust/crates/python-bridge/src/gil.rs
View file

@ -2,7 +2,7 @@
//!
//! A single chokepoint for releasing the GIL around blocking work. Every
//! blocking call in the bridge goes through [`release_gil`] instead of calling
//! `Python::allow_threads` directly, so the release count stays accurate and we
//! `Python::detach` directly, so the release count stays accurate and we
//! have one place to extend later (timing histograms, per-call labels, etc.).
use std::sync::atomic::{AtomicU64, Ordering};
@ -23,7 +23,7 @@ where
T: Send,
{
GIL_RELEASES.fetch_add(1, Ordering::Relaxed);
py.allow_threads(f)
py.detach(f)
}
/// Total GIL releases performed by the bridge so far.

2
litellm-rust/crates/python-bridge/src/lib.rs
View file

@ -167,7 +167,7 @@ fn aocr(
.await
.map_err(core_error_to_pyerr)?;
Python::with_gil(|py| json_to_py(py, value))
Python::attach(|py| json_to_py(py, value))
})
}

2
litellm/litellm_core_utils/dd_tracing.py
View file

@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any, Optional, Union
from litellm.secret_managers.main import get_secret_bool
if TYPE_CHECKING:
from ddtrace.tracer import Tracer as DD_TRACER
from ddtrace.trace import Tracer as DD_TRACER
else:
DD_TRACER = Any

36
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py
View file

@ -391,6 +391,36 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
return thinking
return {**thinking, "budget_tokens": max_tokens - 1}
@staticmethod
def _drop_incompatible_temperature_for_thinking(
model: str, optional_params: dict, custom_llm_provider: str
) -> None:
"""Anthropic rejects any ``temperature`` other than 1 while extended thinking
is enabled ("temperature may only be set to 1 when thinking is enabled").
Clients like Claude Code send ``thinking``/``output_config.effort`` together
with a pinned ``temperature`` (e.g. the safety classifier uses ``temperature=0``
for determinism). When the request lands on a non-adaptive model, the effort
interface is reshaped above into legacy ``thinking={type: enabled}`` (or kept
as ``output_config.effort`` on Opus 4.5), and the leftover ``temperature`` would
400. Preserving the thinking the caller asked for wins over an unhonorable
sampling value (Anthropic forces ``temperature=1`` under thinking regardless),
so drop it and let the API default apply.
Adaptive models (4.6+) own this natively and are left untouched.
"""
if AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
return
temperature = optional_params.get("temperature")
if temperature is None or temperature == 1:
return
thinking = optional_params.get("thinking")
output_config = optional_params.get("output_config")
thinking_enabled = isinstance(thinking, dict) and thinking.get("type") == "enabled"
effort_enabled = isinstance(output_config, dict) and output_config.get("effort") is not None
if thinking_enabled or effort_enabled:
optional_params.pop("temperature", None)
def transform_anthropic_messages_request(
self,
model: str,
@ -431,6 +461,12 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
custom_llm_provider=self._resolved_provider,
)
self._drop_incompatible_temperature_for_thinking(
model=model,
optional_params=anthropic_messages_optional_request_params,
custom_llm_provider=self._resolved_provider,
)
system_param = anthropic_messages_optional_request_params.get("system")
if self.should_strip_billing_metadata() and system_param is not None:
filtered_system = self._filter_billing_headers_from_system(system_param)

1
litellm/proxy/guardrails/guardrail_hooks/model_armor/__init__.py
View file

@ -26,6 +26,7 @@ def initialize_guardrail(litellm_params: "LitellmParams", guardrail: "Guardrail"
mask_request_content=litellm_params.mask_request_content,
mask_response_content=litellm_params.mask_response_content,
fail_on_error=litellm_params.fail_on_error,
skip_unscannable_attachments=litellm_params.skip_unscannable_attachments,
)
litellm.logging_callback_manager.add_litellm_callback(_model_armor_callback)

4
litellm/proxy/guardrails/guardrail_hooks/model_armor/file_scanning.py
View file

@ -25,10 +25,6 @@ from litellm.types.llms.openai import AllMessageValues
MODEL_ARMOR_MAX_FILE_SIZE_BYTES = 4 * 1024 * 1024
# Hard cap on how many attachments a single request may submit to Model Armor, to bound
# per-request fan-out (latency and quota).
MAX_FILE_ATTACHMENTS_PER_REQUEST = 10
_REMOTE_URI_SCHEMES = ("gs://", "http://", "https://")
ModelArmorByteDataType = Literal["PDF", "WORD_DOCUMENT", "EXCEL_DOCUMENT", "POWERPOINT_DOCUMENT", "CSV", "TXT"]

30
litellm/proxy/guardrails/guardrail_hooks/model_armor/model_armor.py
View file

@ -32,7 +32,6 @@ from litellm.llms.custom_httpx.http_handler import (
from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.guardrails.guardrail_hooks.model_armor.file_scanning import (
MAX_FILE_ATTACHMENTS_PER_REQUEST,
MODEL_ARMOR_MAX_FILE_SIZE_BYTES,
plan_file_scans,
)
@ -383,10 +382,14 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
Each attachment is sent through the byte API and a MATCH_FOUND raises a 400 before the
request reaches the LLM. File scanning does not support masking (Model Armor returns
findings, not a sanitized document), so it only blocks. Anything the guardrail cannot
scan - a file_id or remote URL reference with no inline bytes, a document over the 4 MB
byte limit, or more attachments than the per-request cap - is a guardrail failure and
blocks unless the operator has opted into fail-open via fail_on_error=False.
findings, not a sanitized document), so it only blocks. A file_id or remote URL reference
with no inline bytes and a document over the 4 MB byte limit are guardrail failures that
block unless the operator has opted into fail-open via fail_on_error=False.
skip_unscannable_attachments decouples reference-only attachments from fail_on_error: when
enabled, attachments Model Armor cannot scan (file_id, gs://, or http(s) references with no
inline bytes, and inline content whose base64 will not decode) pass through instead of
blocking, while fail_on_error still governs real Model Armor API errors.
"""
from litellm.proxy.common_utils.callback_utils import (
_get_or_create_proxy_metadata_bucket,
@ -395,7 +398,14 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
plan = plan_file_scans(messages)
attachments = plan.attachments
unscannable_references = plan.unscannable_count
skip_unscannable = bool(self.optional_params.get("skip_unscannable_attachments", False))
if skip_unscannable and plan.unscannable_count > 0:
verbose_proxy_logger.warning(
"Model Armor: allowing %d unscannable attachment(s) through because "
"skip_unscannable_attachments is enabled",
plan.unscannable_count,
)
unscannable_references = 0 if skip_unscannable else plan.unscannable_count
if not attachments and unscannable_references == 0:
return
@ -415,14 +425,6 @@ class ModelArmorGuardrail(CustomGuardrail, VertexBase):
metadata["_model_armor_status"] = "blocked"
raise self._unscannable_block_error(reason)
if len(attachments) > MAX_FILE_ATTACHMENTS_PER_REQUEST:
reason = f"{len(attachments)} attachments exceed the per-request scan limit of {MAX_FILE_ATTACHMENTS_PER_REQUEST}"
verbose_proxy_logger.warning("Model Armor: %s", reason)
if fail_on_error:
metadata["_model_armor_status"] = "blocked"
raise self._unscannable_block_error(reason)
attachments = attachments[:MAX_FILE_ATTACHMENTS_PER_REQUEST]
for attachment in attachments:
if len(attachment.file_bytes) > MODEL_ARMOR_MAX_FILE_SIZE_BYTES:
reason = (

52
litellm/proxy/utils.py
View file

@ -19,6 +19,7 @@ from typing import (
Any,
AsyncGenerator,
Awaitable,
Callable,
ClassVar,
Dict,
List,
@ -49,7 +50,7 @@ from litellm.proxy._types import (
from litellm.proxy.spend_tracking.spend_log_error_logger import spend_log_error
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.proxy.model_listing import ModelInfoResponse
from litellm.types.utils import CallTypes, CallTypesLiteral
from litellm.types.utils import CallTypes, CallTypesLiteral, ModelInfo
try:
from litellm_enterprise.enterprise_callbacks.send_emails.base_email import (
@ -6098,6 +6099,7 @@ def create_model_info_response(
include_metadata: bool = False,
fallback_type: Optional[str] = None,
llm_router: Optional["Router"] = None,
get_model_info: Callable[[str], ModelInfo] = litellm.get_model_info,
) -> ModelInfoResponse:
"""
Create a standardized OpenAI-compatible model object.
@ -6115,25 +6117,37 @@ def create_model_info_response(
"owned_by": provider,
}
# Surface context-window limits for OpenAI-compatible discovery clients.
# Only emitted when known, so wildcard routes and limitless backends stay clean.
# Limits are best-effort enrichment, so a single malformed deployment degrades
# to the base response rather than 500-ing the whole listing.
try:
model_cost_info: ModelInfo | None = get_model_info(model_id)
except Exception as e:
verbose_proxy_logger.debug(
"create_model_info_response: cost map lookup failed for %s: %s",
model_id,
e,
)
model_cost_info = None
max_input_tokens: int | None = None
max_output_tokens: int | None = None
if model_cost_info is not None:
cost_map_input = model_cost_info.get("max_input_tokens")
if cost_map_input is not None:
max_input_tokens = int(cost_map_input)
cost_map_output = model_cost_info.get("max_output_tokens")
if cost_map_output is not None:
max_output_tokens = int(cost_map_output)
if llm_router is not None:
try:
model_group_info = llm_router.get_model_group_info(model_id)
except Exception as e:
verbose_proxy_logger.debug(
"create_model_info_response: get_model_group_info failed for %s: %s",
model_id,
e,
)
model_group_info = None
if model_group_info is not None:
if model_group_info.max_input_tokens is not None:
base["max_input_tokens"] = int(model_group_info.max_input_tokens)
if model_group_info.max_output_tokens is not None:
base["max_output_tokens"] = int(model_group_info.max_output_tokens)
configured_input, configured_output = llm_router.get_configured_token_limits(model_id)
if configured_input is not None:
max_input_tokens = configured_input
if configured_output is not None:
max_output_tokens = configured_output
if max_input_tokens is not None:
base["max_input_tokens"] = max_input_tokens
if max_output_tokens is not None:
base["max_output_tokens"] = max_output_tokens
if not include_metadata:
return base

21
litellm/router.py
View file

@ -8357,6 +8357,27 @@ class Router:
raise Exception("Model Name invalid - {}".format(type(model)))
return None
def get_configured_token_limits(self, model_name: str) -> "tuple[int | None, int | None]":
"""
Return (max_input_tokens, max_output_tokens) explicitly configured in a concrete
deployment's model_info for model_name, via O(1) index lookup.
Returns (None, None) for wildcard-expanded or unknown names. Unlike
get_model_group_info, this never triggers pattern matching or deep copies, so it
is safe to call per listed model on the /v1/models hot path.
"""
deployment = self.get_deployment_by_model_group_name(model_group_name=model_name)
if deployment is None:
return (None, None)
model_info = deployment.model_info
max_input = model_info.get("max_input_tokens")
max_output = model_info.get("max_output_tokens")
return (
int(max_input) if max_input is not None else None,
int(max_output) if max_output is not None else None,
)
def get_deployment_credentials_with_provider(self, model_id: str) -> Optional[Dict[str, Any]]:
"""
Get API credentials and provider info from a model name in model_list.

8
litellm/types/guardrails.py
View file

@ -687,6 +687,14 @@ class BaseLitellmParams(ContentFilterConfigModel): # works for new and patch up
"so only a valid guardrail response can block or modify it."
),
)
skip_unscannable_attachments: Optional[bool] = Field(
default=False,
description=(
"Implemented by guardrail='model_armor'. When True, attachment references that carry no "
"inline bytes (file_id, gs://, or http(s) URLs) pass through unscanned instead of blocking, "
"while fail_on_error still governs real Model Armor API errors. Default False blocks them."
),
)
additional_provider_specific_params: Optional[Dict[str, Any]] = Field(
default=None,

16
pyproject.toml
View file

@ -3,7 +3,7 @@ name = "litellm"
version = "1.93.0"
description = "Library to easily interface with LLM API providers"
readme = "README.md"
requires-python = ">=3.10, <3.14"
requires-python = ">=3.10, <3.15"
license = "MIT"
license-files = ["LICENSE"]
authors = [
@ -61,7 +61,7 @@ proxy = [
"boto3>=1.43.1,<2.0",
"azure-identity>=1.25.2,<2.0",
"azure-storage-blob>=12.28.0,<13.0",
"mcp>=1.26.0,<2.0",
"mcp>=1.28.1,<2.0",
"litellm-proxy-extras==0.4.76",
"litellm-enterprise==0.1.49",
"RestrictedPython>=8.1,<9.0",
@ -89,7 +89,7 @@ extra_proxy = [
"google-cloud-iam>=2.19.1,<3.0",
# Not in PyPI proxy extra.
"resend>=2.23.0,<3.0",
"redisvl>=0.4.1,<1.0; python_version < '3.14'",
"redisvl>=0.4.1,<1.0",
"a2a-sdk>=1.1.0,<2.0",
]
utils = [
@ -129,12 +129,12 @@ proxy-runtime = [
"opentelemetry-sdk==1.28.0",
"opentelemetry-exporter-otlp==1.28.0",
"opentelemetry-instrumentation-fastapi==0.49b0",
"ddtrace>=2.19.0,<3.0",
"ddtrace>=4.8.2,<5.0",
"sentry-sdk>=2.21.0,<3.0",
"mangum>=0.17.0,<1.0",
"azure-ai-contentsafety>=1.0.0,<2.0",
"azure-storage-file-datalake>=12.20.0,<13.0",
"pypdf>=6.12.0,<7.0; python_version < '3.14'",
"pypdf>=6.12.0,<7.0",
"llm-sandbox>=0.3.39,<1.0",
"detect-secrets>=1.5.0,<2.0",
]
@ -179,7 +179,7 @@ dev = [
"pytest-rerunfailures==15.1",
"pytest-cov==5.0.0",
"parameterized==0.9.0",
"openapi-core==0.22.0; python_version < '3.14'",
"openapi-core==0.22.0",
"pytest-timeout==2.4.0",
"vcrpy==8.2.1",
"pytest-recording==0.13.4",
@ -205,7 +205,7 @@ ci = [
# protobuf, Pillow is a compiled C extension).
"tenacity==8.5.0",
"google-generativeai==0.8.6",
"Pillow==12.2.0",
"Pillow==12.3.0",
# Azure batch E2E tests still import psycopg2 directly.
"psycopg2-binary==2.9.11",
"pytest-codspeed==4.3.0",
@ -264,6 +264,8 @@ constraint-dependencies = [
"aiohttp>=3.14.1,<4.0",
"packaging>=24.0",
"soupsieve>=2.8.4",
"httplib2>=0.32.0",
"setuptools>=83.0.0",
]
override-dependencies = [
# a2a-sdk 1.x requires packaging>=24.0; lunary 1.4.x still caps at <24.0.

76
tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_effort.py
View file

@ -2,6 +2,7 @@ import pytest
from litellm.constants import (
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_XHIGH_THINKING_BUDGET,
)
@ -174,6 +175,81 @@ def test_unrecognized_effort_raises_clean_400():
assert exc_info.value.status_code == 400
def test_pinned_temperature_dropped_when_adaptive_downgraded_to_enabled():
"""Regression (#33203): Claude Code's safety classifier sends adaptive thinking +
temperature=0 to Haiku 4.5. The adaptive interface is downgraded to legacy enabled
thinking, but Anthropic rejects "temperature may only be set to 1 when thinking is
enabled". The pinned temperature must be dropped so the request succeeds while the
downgraded thinking is preserved."""
params = _claude_code_payload(effort="medium")
params["temperature"] = 0
result = _transform("claude-haiku-4-5", params)
assert result["thinking"] == {
"type": "enabled",
"budget_tokens": DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
}
assert "temperature" not in result
def test_temperature_one_preserved_with_enabled_thinking():
"""temperature=1 is compatible with extended thinking, so it must be kept."""
params = _claude_code_payload(effort="medium")
params["temperature"] = 1
result = _transform("claude-haiku-4-5", params)
assert result["thinking"]["type"] == "enabled"
assert result["temperature"] == 1
def test_pinned_temperature_preserved_when_thinking_dropped():
"""When thinking is dropped entirely (non-reasoning model), there is no thinking
conflict, so a pinned temperature must survive untouched."""
params = _claude_code_payload(effort="medium")
params["temperature"] = 0
result = _transform("claude-3-5-haiku-latest", params)
assert "thinking" not in result
assert result["temperature"] == 0
def test_pinned_temperature_preserved_for_adaptive_model():
"""Adaptive models (4.6+) own the thinking/temperature relationship natively, so
the passthrough must not strip a pinned temperature for them."""
params = _claude_code_payload(effort="high")
params["temperature"] = 0
result = _transform("claude-sonnet-4-6", params)
assert result["thinking"] == {"type": "adaptive"}
assert result["temperature"] == 0
def test_pinned_temperature_dropped_for_opus_4_5_effort():
"""Opus 4.5 keeps native output_config.effort (extended thinking), which is equally
incompatible with a pinned non-1 temperature, so the temperature must be dropped."""
params = _claude_code_payload(effort="medium")
params["temperature"] = 0
result = _transform("claude-opus-4-5", params)
assert result["output_config"] == {"effort": "medium"}
assert "temperature" not in result
def test_reasoning_effort_with_pinned_temperature_drops_temperature():
"""The reasoning_effort alias synthesizes legacy enabled thinking on a non-adaptive
model; a co-pinned non-1 temperature must be dropped to avoid the Anthropic 400."""
result = _transform(
"claude-haiku-4-5",
{"max_tokens": 8192, "reasoning_effort": "low", "temperature": 0},
)
assert result["thinking"] == {
"type": "enabled",
"budget_tokens": DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
}
assert "temperature" not in result
def test_non_adaptive_request_without_effort_is_untouched():
"""A non-adaptive model receiving a request with no adaptive interface (no
effort, no adaptive thinking) must pass through untouched."""

159
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_model_armor.py
View file

@ -2205,21 +2205,20 @@ async def test_pre_call_file_id_reference_skipped_when_fail_open():
@pytest.mark.asyncio
async def test_pre_call_blocks_when_attachment_count_exceeds_cap():
"""More attachments than the per-request cap fail closed by default to bound scan fan-out."""
from litellm.proxy.guardrails.guardrail_hooks.model_armor.file_scanning import (
MAX_FILE_ATTACHMENTS_PER_REQUEST,
)
guardrail = _make_guardrail()
pdf_b64 = base64.b64encode(PDF_BYTES).decode("utf-8")
block = {
"type": "file",
"file": {"file_data": f"data:application/pdf;base64,{pdf_b64}"},
}
async def test_pre_call_file_id_reference_passthrough_when_skip_unscannable_enabled():
"""skip_unscannable_attachments lets a file_id reference through even with fail_on_error=True."""
guardrail = _make_guardrail(skip_unscannable_attachments=True)
request_data = {
"model": "gpt-4",
"messages": [{"role": "user", "content": [block] * (MAX_FILE_ATTACHMENTS_PER_REQUEST + 1)}],
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "summarize this"},
{"type": "file", "file": {"file_id": "file-abc123"}},
],
}
],
"metadata": {"guardrails": ["model-armor-test"]},
}
@ -2227,8 +2226,109 @@ async def test_pre_call_blocks_when_attachment_count_exceeds_cap():
guardrail.async_handler,
"post",
AsyncMock(return_value=_armor_response(blocked=False)),
) as mock_post:
await guardrail.async_pre_call_hook(
user_api_key_dict=UserAPIKeyAuth(),
cache=MagicMock(spec=DualCache),
data=request_data,
call_type="completion",
)
assert _byte_items_sent(mock_post) == []
assert _text_payloads_sent(mock_post) == ["summarize this"]
@pytest.mark.asyncio
async def test_pre_call_gs_uri_reference_passthrough_when_skip_unscannable_enabled():
"""A gs:// document reference passes through when skip_unscannable_attachments is enabled."""
guardrail = _make_guardrail(skip_unscannable_attachments=True)
request_data = {
"model": "gpt-4",
"messages": [
{
"role": "user",
"content": [
{
"type": "file",
"file": {"file_data": "gs://my-bucket/report.pdf", "filename": "report.pdf"},
}
],
}
],
"metadata": {"guardrails": ["model-armor-test"]},
}
with patch.object(
guardrail.async_handler,
"post",
AsyncMock(return_value=_armor_response(blocked=False)),
) as mock_post:
await guardrail.async_pre_call_hook(
user_api_key_dict=UserAPIKeyAuth(),
cache=MagicMock(spec=DualCache),
data=request_data,
call_type="completion",
)
assert _byte_items_sent(mock_post) == []
def test_initialize_guardrail_forwards_skip_unscannable_attachments():
"""skip_unscannable_attachments configured in litellm_params reaches the guardrail instance."""
from litellm.proxy.guardrails.guardrail_hooks.model_armor import initialize_guardrail
from litellm.types.guardrails import Guardrail, LitellmParams
litellm_params = LitellmParams(
guardrail="model_armor",
mode="pre_call",
template_id="demo-template",
project_id="demo-project",
skip_unscannable_attachments=True,
)
guardrail = initialize_guardrail(
litellm_params=litellm_params,
guardrail=Guardrail(guardrail_name="model-armor-config-test"),
)
assert guardrail.optional_params.get("skip_unscannable_attachments") is True
def test_initialize_guardrail_skip_unscannable_defaults_false():
"""A config that omits skip_unscannable_attachments keeps the secure default (block)."""
from litellm.proxy.guardrails.guardrail_hooks.model_armor import initialize_guardrail
from litellm.types.guardrails import Guardrail, LitellmParams
litellm_params = LitellmParams(
guardrail="model_armor",
mode="pre_call",
template_id="demo-template",
project_id="demo-project",
)
guardrail = initialize_guardrail(
litellm_params=litellm_params,
guardrail=Guardrail(guardrail_name="model-armor-config-default"),
)
assert guardrail.optional_params.get("skip_unscannable_attachments") is False
@pytest.mark.asyncio
async def test_skip_unscannable_still_fails_closed_on_api_error():
"""skip_unscannable_attachments only affects references; a real API error still fails closed."""
guardrail = _make_guardrail(skip_unscannable_attachments=True, fail_on_error=True)
pdf_b64 = base64.b64encode(PDF_BYTES).decode("utf-8")
request_data = {
"model": "gpt-4",
"messages": [_file_message(pdf_b64)],
"metadata": {"guardrails": ["model-armor-test"]},
}
with patch.object(
guardrail.async_handler,
"post",
AsyncMock(side_effect=Exception("model armor upstream 500")),
):
with pytest.raises(HTTPException) as exc_info:
with pytest.raises(Exception) as exc_info:
await guardrail.async_pre_call_hook(
user_api_key_dict=UserAPIKeyAuth(),
cache=MagicMock(spec=DualCache),
@ -2236,8 +2336,35 @@ async def test_pre_call_blocks_when_attachment_count_exceeds_cap():
call_type="completion",
)
assert exc_info.value.status_code == 400
assert "per-request scan limit" in str(exc_info.value.detail)
assert "model armor upstream 500" in str(exc_info.value)
@pytest.mark.asyncio
async def test_pre_call_scans_every_attachment_without_a_count_cap():
"""There is no per-request attachment cap: every scannable attachment is submitted to Model Armor."""
guardrail = _make_guardrail()
pdf_b64 = base64.b64encode(PDF_BYTES).decode("utf-8")
block = {
"type": "file",
"file": {"file_data": f"data:application/pdf;base64,{pdf_b64}"},
}
count = 25
request_data = {
"model": "gpt-4",
"messages": [{"role": "user", "content": [block] * count}],
"metadata": {"guardrails": ["model-armor-test"]},
}
mock_post = AsyncMock(return_value=_armor_response(blocked=False))
with patch.object(guardrail.async_handler, "post", mock_post):
await guardrail.async_pre_call_hook(
user_api_key_dict=UserAPIKeyAuth(),
cache=MagicMock(spec=DualCache),
data=request_data,
call_type="completion",
)
assert len(_byte_items_sent(mock_post)) == count
@pytest.mark.asyncio

186
tests/test_litellm/proxy/test_proxy_utils.py
View file

@ -476,101 +476,118 @@ class TestPostCallFailureHookLiftsRecoveredPartialSpend:
assert "response_cost" not in request_data
from typing import cast
from litellm.proxy.utils import create_model_info_response
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import ModelInfo
def _router_returning(model_group_info):
router = MagicMock()
router.get_model_group_info = MagicMock(return_value=model_group_info)
return router
def _fake_model_info(**fields: int) -> ModelInfo:
return cast(ModelInfo, dict(fields))
def test_create_model_info_response_includes_max_tokens_when_available():
router = _router_returning(
ModelGroupInfo(
model_group="qwen-vllm",
providers=["hosted_vllm"],
max_input_tokens=32768,
max_output_tokens=8192,
)
def _raise_unmapped(model_id: str) -> ModelInfo:
raise ValueError(f"This model isn't mapped yet: {model_id}")
def test_create_model_info_response_includes_max_tokens_from_lookup():
response = create_model_info_response(
model_id="some-model",
provider="openai",
llm_router=None,
get_model_info=lambda _model: _fake_model_info(
max_input_tokens=128000, max_output_tokens=16384
),
)
assert response["id"] == "some-model"
assert response["object"] == "model"
assert response["max_input_tokens"] == 128000
assert response["max_output_tokens"] == 16384
def test_create_model_info_response_does_not_call_router_group_info():
router = MagicMock()
router.get_configured_token_limits.return_value = (None, None)
response = create_model_info_response(
model_id="qwen-vllm", provider="openai", llm_router=router
model_id="some-model",
provider="openai",
llm_router=router,
get_model_info=lambda _model: _fake_model_info(
max_input_tokens=128000, max_output_tokens=16384
),
)
router.get_model_group_info.assert_called_once_with("qwen-vllm")
assert response["id"] == "qwen-vllm"
assert response["object"] == "model"
assert response["max_input_tokens"] == 32768
assert response["max_output_tokens"] == 8192
router.get_model_group_info.assert_not_called()
assert response["max_input_tokens"] == 128000
def test_create_model_info_response_uses_deployment_limits_when_not_in_cost_map():
router = MagicMock()
router.get_configured_token_limits.return_value = (32000, 8000)
response = create_model_info_response(
model_id="my-custom-deployment",
provider="openai",
llm_router=router,
get_model_info=_raise_unmapped,
)
router.get_model_group_info.assert_not_called()
assert response["max_input_tokens"] == 32000
assert response["max_output_tokens"] == 8000
def test_create_model_info_response_deployment_limits_override_cost_map():
router = MagicMock()
router.get_configured_token_limits.return_value = (200000, None)
response = create_model_info_response(
model_id="gpt-4o",
provider="openai",
llm_router=router,
get_model_info=lambda _model: _fake_model_info(
max_input_tokens=128000, max_output_tokens=16384
),
)
assert response["max_input_tokens"] == 200000
assert response["max_output_tokens"] == 16384
def test_create_model_info_response_emits_integer_token_counts():
# ModelGroupInfo types the limits as float; OpenAI-compatible clients expect
# plain integers, so the response must not leak 128000.0.
router = _router_returning(
ModelGroupInfo(
model_group="gpt-4o",
providers=["openai"],
max_input_tokens=128000.0,
max_output_tokens=16384.0,
)
)
response = create_model_info_response(
model_id="gpt-4o", provider="openai", llm_router=router
model_id="some-model",
provider="openai",
llm_router=None,
get_model_info=lambda _model: _fake_model_info(
max_input_tokens=128000, max_output_tokens=16384
),
)
assert response["max_input_tokens"] == 128000
assert isinstance(response["max_input_tokens"], int)
assert response["max_output_tokens"] == 16384
assert isinstance(response["max_output_tokens"], int)
def test_create_model_info_response_omits_unknown_individual_limit():
router = _router_returning(
ModelGroupInfo(
model_group="partial",
providers=["openai"],
max_input_tokens=4096,
max_output_tokens=None,
)
)
response = create_model_info_response(
model_id="partial", provider="openai", llm_router=router
model_id="some-embedding",
provider="openai",
llm_router=None,
get_model_info=lambda _model: _fake_model_info(max_input_tokens=8191),
)
assert response["max_input_tokens"] == 4096
assert response["max_input_tokens"] == 8191
assert "max_output_tokens" not in response
def test_create_model_info_response_omits_limits_when_both_none():
router = _router_returning(
ModelGroupInfo(
model_group="no-limits",
providers=["openai"],
max_input_tokens=None,
max_output_tokens=None,
)
)
def test_create_model_info_response_omits_limits_when_lookup_raises():
response = create_model_info_response(
model_id="no-limits", provider="openai", llm_router=router
)
assert "max_input_tokens" not in response
assert "max_output_tokens" not in response
def test_create_model_info_response_omits_limits_when_group_unknown():
# Wildcard routes / access groups have no ModelGroupInfo.
router = _router_returning(None)
response = create_model_info_response(
model_id="openai/*", provider="openai", llm_router=router
model_id="openai/*",
provider="openai",
llm_router=None,
get_model_info=_raise_unmapped,
)
assert response["id"] == "openai/*"
@ -578,32 +595,33 @@ def test_create_model_info_response_omits_limits_when_group_unknown():
assert "max_output_tokens" not in response
def test_create_model_info_response_degrades_when_group_info_raises():
# A malformed deployment must not turn the listing into a 500; the entry
# falls back to the base fields without limits.
router = MagicMock()
router.get_model_group_info = MagicMock(side_effect=ValueError("bad deployment"))
response = create_model_info_response(
model_id="broken", provider="openai", llm_router=router
)
assert response["id"] == "broken"
assert "max_input_tokens" not in response
assert "max_output_tokens" not in response
def test_create_model_info_response_no_router_keeps_base_fields():
response = create_model_info_response(
model_id="some-model", provider="openai", llm_router=None
model_id="totally-unknown-model-xyz",
provider="openai",
llm_router=None,
get_model_info=_raise_unmapped,
)
assert response == {
"id": "some-model",
"id": "totally-unknown-model-xyz",
"object": "model",
"created": response["created"],
"owned_by": "openai",
}
def test_create_model_info_response_reads_real_cost_map():
response = create_model_info_response(
model_id="gpt-4o", provider="openai", llm_router=None
)
assert isinstance(response["max_input_tokens"], int)
assert response["max_input_tokens"] > 0
assert isinstance(response["max_output_tokens"], int)
assert response["max_output_tokens"] > 0
class TestPostCallFailureHookLLMExceptionAlerting:
"""The llm_exceptions alert is for infra / LLM-API failures, not user
errors (https://github.com/BerriAI/litellm/issues/3395). Already-normalized

12
tests/test_litellm/proxy/utils/helpers/test_model_access.py
View file

@ -103,18 +103,16 @@ def test_is_known_vector_store_index_error_path_no_registry(monkeypatch):
def test_create_model_info_response_happy_path_no_metadata():
result = create_model_info_response(model_id="gpt-4o", provider="openai")
assert result == {
"id": "gpt-4o",
"object": "model",
"created": result["created"],
"owned_by": "openai",
}
snapshot = {
"id": result["id"],
"object": result["object"],
"owned_by": result["owned_by"],
"created_is_int": isinstance(result["created"], int),
"metadata_absent": "metadata" not in result,
"max_input_tokens_positive_int": isinstance(result["max_input_tokens"], int)
and result["max_input_tokens"] > 0,
"max_output_tokens_positive_int": isinstance(result["max_output_tokens"], int)
and result["max_output_tokens"] > 0,
}
assert snapshot == {
"id": "gpt-4o",
@ -122,6 +120,8 @@ def test_create_model_info_response_happy_path_no_metadata():
"owned_by": "openai",
"created_is_int": True,
"metadata_absent": True,
"max_input_tokens_positive_int": True,
"max_output_tokens_positive_int": True,
}

47
tests/test_litellm/test_router.py
View file

@ -5304,3 +5304,50 @@ class TestRouterRequestTimeoutPropagation:
)
== 60
)
def test_get_configured_token_limits_reads_deployment_model_info():
router = litellm.Router(
model_list=[
{
"model_name": "my-custom-model",
"litellm_params": {"model": "openai/some-unmapped-model"},
"model_info": {"max_input_tokens": 32000, "max_output_tokens": 8000},
}
]
)
assert router.get_configured_token_limits("my-custom-model") == (32000, 8000)
def test_get_configured_token_limits_returns_none_for_unset_or_unknown():
router = litellm.Router(
model_list=[
{
"model_name": "no-limits-model",
"litellm_params": {"model": "openai/some-unmapped-model"},
}
]
)
assert router.get_configured_token_limits("no-limits-model") == (None, None)
assert router.get_configured_token_limits("not-a-real-model") == (None, None)
def test_get_configured_token_limits_skips_wildcard_pattern_matching():
router = litellm.Router(
model_list=[
{
"model_name": "bedrock/*",
"litellm_params": {"model": "bedrock/*"},
"model_info": {"max_input_tokens": 12345},
}
]
)
with patch.object(
router.pattern_router, "route", side_effect=AssertionError("pattern route called")
):
assert router.get_configured_token_limits(
"bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0"
) == (None, None)

1413
uv.lock generated
View file

File diff suppressed because it is too large Load diff