scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-03 02:22:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 81

feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to #27687) (#27706)

Browse source 
* feat(prometheus): add rate_limit_category and rate_limit_type labels

Adds two new labels to litellm_proxy_failed_requests_metric so dashboards
can split 429s by rate-limit source (vendor vs. litellm-internal) and by
the dimension that was exceeded (requests/tokens/concurrent_requests/
budget/max_iterations) without parsing free-text error messages.

Closes the Prometheus side of LIT-2718. The unified RateLimitError.category
and .rate_limit_type fields landed in PR #27687 but were only surfaced on
StandardLoggingPayload (custom-callback channel); this exposes them on
the metric label set as well.

Both labels are populated only when the underlying exception is a
litellm.RateLimitError; non-rate-limit failures keep them empty.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* feat(prometheus): populate rate-limit labels + preserve exception_class back-compat

Two coupled changes in the Prometheus integration:

1. async_post_call_failure_hook now extracts the new RateLimitError
   .category / .rate_limit_type fields (added in PR #27687) via a
   _extract_rate_limit_labels helper and forwards them through
   UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric.
   Empty for non-rate-limit failures.

2. _get_exception_class_name special-cases ProxyRateLimitError and
   keeps emitting 'HTTPException' for the exception_class label.
   Without this shim, ProxyRateLimitError (which multi-inherits from
   HTTPException + RateLimitError) would silently flip the label
   from 'HTTPException' (the historical value for proxy-side 429s)
   to 'ProxyRateLimitError', breaking existing dashboards / alerts
   that key off exception_class='HTTPException'. Distinguishing
   vendor vs. litellm 429s is now the job of the new
   rate_limit_category label.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(prometheus): cover rate-limit labels and exception_class back-compat

Adds 19 tests across:
- enum / label-list registration
- _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError,
  non-rate-limit and None inputs (incl. parametrized over every
  RateLimitErrorCategory x RateLimitType combo)
- _get_exception_class_name back-compat: ProxyRateLimitError keeps the
  legacy 'HTTPException' string while vendor RateLimitError keeps the
  historical 'Provider.ClassName' format
- end-to-end through async_post_call_failure_hook with both
  ProxyRateLimitError and vendor RateLimitError, asserting both new
  labels populate and exception_class stays back-compat

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import

Address greptile feedback:
- async_post_call_failure_hook docstring: drop the stale labelnames listing
  and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric
  as the source of truth so the doc cannot drift from the actual labelset.
- _get_exception_class_name: guard the lazy ProxyRateLimitError import with
  ImportError so router-side fallback callsites don't blow up in non-proxy
  installs that don't have fastapi (a transitive dep of
  proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when
  fastapi is available.

Also fix the existing enterprise callback test that asserted the old
labelset on litellm_proxy_failed_requests_metric — it now expects the new
rate_limit_category / rate_limit_type labels populated for vendor 429s.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
This commit is contained in:
Mateo Wang 2026-05-13 15:57:02 -07:00 • committed by GitHub
parent af49dbebd6
commit 26abed5cc3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 368 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

72
litellm/integrations/prometheus.py
View file

@ -1814,14 +1814,8 @@ class PrometheusLogger(CustomLogger):
Proxy level tracking - failed client side requests
labelnames=[
"end_user",
"hashed_api_key",
"api_key_alias",
REQUESTED_MODEL,
"team",
"team_alias",
] + EXCEPTION_LABELS,
See :attr:`PrometheusMetricLabels.litellm_proxy_failed_requests_metric`
for the authoritative list of labels emitted on this metric.
"""
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
@ -1844,6 +1838,9 @@ class PrometheusLogger(CustomLogger):
model_id = _metadata.get("model_info", {}).get("id") or request_data.get(
"model_info", {}
).get("id")
rate_limit_category, rate_limit_type = self._extract_rate_limit_labels(
original_exception
)
enum_values = UserAPIKeyLabelValues(
end_user=user_api_key_dict.end_user_id,
user=user_api_key_dict.user_id,
@ -1858,6 +1855,8 @@ class PrometheusLogger(CustomLogger):
status_code=str(status_code),
exception_status=str(status_code),
exception_class=self._get_exception_class_name(original_exception),
rate_limit_category=rate_limit_category,
rate_limit_type=rate_limit_type,
tags=_tags,
route=user_api_key_dict.request_route,
client_ip=_metadata.get("requester_ip_address"),
@ -2535,6 +2534,33 @@ class PrometheusLogger(CustomLogger):
@staticmethod
def _get_exception_class_name(exception: Exception) -> str:
# Back-compat: ProxyRateLimitError multi-inherits from
# fastapi.HTTPException + litellm.RateLimitError. Before the unified
# rate-limit error class landed, every proxy-side 429 surfaced on this
# label as the literal string "HTTPException", and existing dashboards
# / alerts (e.g. trho's HubSpot dashboard) key off that exact value.
# Renaming the class would silently break those queries, so we keep
# emitting "HTTPException" here. Callers that need to distinguish
# vendor vs. litellm rate limits should use the new
# ``rate_limit_category`` / ``rate_limit_type`` labels instead.
#
# The import is intentionally lazy + ImportError-tolerant: this method
# is also called from router-side fallback events
# (``log_success_fallback_event`` / ``log_failure_fallback_event``)
# which can run in non-proxy installs where ``fastapi`` (a transitive
# dep of ``proxy_rate_limit_error``) is not installed.
try:
from litellm.proxy.common_utils.proxy_rate_limit_error import (
ProxyRateLimitError,
)
except ImportError:
ProxyRateLimitError = None # type: ignore[assignment,misc]
if ProxyRateLimitError is not None and isinstance(
exception, ProxyRateLimitError
):
return "HTTPException"
exception_class_name = ""
if hasattr(exception, "llm_provider"):
exception_class_name = getattr(exception, "llm_provider") or ""
@ -2549,6 +2575,36 @@ class PrometheusLogger(CustomLogger):
exception_class_name += exception.__class__.__name__
return exception_class_name
@staticmethod
def _extract_rate_limit_labels(
exception: Optional[Exception],
) -> Tuple[Optional[str], Optional[str]]:
"""
Pull the unified ``category`` / ``rate_limit_type`` fields off any
:class:`litellm.RateLimitError` (vendor-side or litellm-internal) so
Prometheus can split 429s by source + dimension without the consumer
parsing free-text error messages.
Returns ``(None, None)`` for non-rate-limit exceptions. Both values are
sanitized to plain ``str`` so str-enum subclasses don't leak their
repr into the label.
"""
if exception is None or not isinstance(exception, litellm.RateLimitError):
return None, None
def _coerce(value: Any) -> Optional[str]:
if value is None:
return None
inner = getattr(value, "value", None)
if isinstance(inner, str):
return inner
return str(value)
return (
_coerce(getattr(exception, "category", None)),
_coerce(getattr(exception, "rate_limit_type", None)),
)
async def log_success_fallback_event(
self, original_model_group: str, kwargs: dict, original_exception: Exception
):

12
litellm/types/integrations/prometheus.py
View file

@ -115,6 +115,8 @@ class ValidationResults:
REQUESTED_MODEL = "requested_model"
EXCEPTION_STATUS = "exception_status"
EXCEPTION_CLASS = "exception_class"
RATE_LIMIT_CATEGORY = "rate_limit_category"
RATE_LIMIT_TYPE = "rate_limit_type"
STATUS_CODE = "status_code"
EXCEPTION_LABELS = [EXCEPTION_STATUS, EXCEPTION_CLASS]
LATENCY_BUCKETS = (
@ -173,6 +175,8 @@ class UserAPIKeyLabelNames(Enum):
API_PROVIDER = "api_provider"
EXCEPTION_STATUS = EXCEPTION_STATUS
EXCEPTION_CLASS = EXCEPTION_CLASS
RATE_LIMIT_CATEGORY = RATE_LIMIT_CATEGORY
RATE_LIMIT_TYPE = RATE_LIMIT_TYPE
STATUS_CODE = "status_code"
FALLBACK_MODEL = "fallback_model"
ROUTE = "route"
@ -334,6 +338,12 @@ class PrometheusMetricLabels:
UserAPIKeyLabelNames.USER_EMAIL.value,
UserAPIKeyLabelNames.EXCEPTION_STATUS.value,
UserAPIKeyLabelNames.EXCEPTION_CLASS.value,
# Surfaced from RateLimitError.category / .rate_limit_type when the
# underlying exception is a rate-limit error; ``None`` otherwise. Lets
# dashboards split 429s into vendor vs. litellm and by exceeded
# dimension (RPM/TPM/concurrent/budget) without parsing error text.
UserAPIKeyLabelNames.RATE_LIMIT_CATEGORY.value,
UserAPIKeyLabelNames.RATE_LIMIT_TYPE.value,
UserAPIKeyLabelNames.ROUTE.value,
UserAPIKeyLabelNames.CLIENT_IP.value,
UserAPIKeyLabelNames.USER_AGENT.value,
@ -775,6 +785,8 @@ class UserAPIKeyLabelValues:
api_provider: Optional[str] = None
exception_status: Optional[str] = None
exception_class: Optional[str] = None
rate_limit_category: Optional[str] = None
rate_limit_type: Optional[str] = None
status_code: Optional[str] = None
fallback_model: Optional[str] = None
route: Optional[str] = None

2
tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py
View file

@ -825,6 +825,8 @@ async def test_async_post_call_failure_hook(prometheus_logger):
requested_model="gpt-3.5-turbo",
exception_status="429",
exception_class="Openai.RateLimitError",
rate_limit_category="vendor_rate_limit",
rate_limit_type=None,
route=user_api_key_dict.request_route,
model_id=None,
client_ip=None,

290
tests/test_litellm/integrations/test_prometheus_rate_limit_labels.py Normal file
View file

@ -0,0 +1,290 @@
"""
Tests for the Prometheus rate-limit labels added on top of PR #27687.
Covers two follow-up gaps to the unified rate-limit error work:
1. ``litellm_proxy_failed_requests_metric`` now carries
``rate_limit_category`` and ``rate_limit_type`` labels populated from
:class:`litellm.RateLimitError` (vendor + ``ProxyRateLimitError``
subclass). Closes the Prometheus side of LIT-2718.
2. ``_get_exception_class_name`` keeps emitting the literal string
``"HTTPException"`` for ``ProxyRateLimitError`` so existing dashboards
that key off ``exception_class="HTTPException"`` for litellm-internal
429s don't silently break when the new class lands.
"""
from unittest.mock import MagicMock, patch
import pytest
from litellm.exceptions import (
RateLimitError,
RateLimitErrorCategory,
RateLimitType,
)
from litellm.integrations.prometheus import PrometheusLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
from litellm.types.integrations.prometheus import (
PrometheusMetricLabels,
UserAPIKeyLabelNames,
UserAPIKeyLabelValues,
)
# ---------------------------------------------------------------------------
# Label / enum wiring
# ---------------------------------------------------------------------------
def test_should_register_rate_limit_label_names_on_enum():
assert UserAPIKeyLabelNames.RATE_LIMIT_CATEGORY.value == "rate_limit_category"
assert UserAPIKeyLabelNames.RATE_LIMIT_TYPE.value == "rate_limit_type"
def test_should_include_rate_limit_labels_on_failed_requests_metric():
labels = PrometheusMetricLabels.get_labels("litellm_proxy_failed_requests_metric")
assert "rate_limit_category" in labels
assert "rate_limit_type" in labels
# These must coexist with the legacy exception labels (back-compat).
assert "exception_class" in labels
assert "exception_status" in labels
def test_should_accept_rate_limit_fields_on_user_api_key_label_values():
enum_values = UserAPIKeyLabelValues(
rate_limit_category="litellm_rate_limit",
rate_limit_type="requests",
)
assert enum_values.rate_limit_category == "litellm_rate_limit"
assert enum_values.rate_limit_type == "requests"
# ---------------------------------------------------------------------------
# _extract_rate_limit_labels helper
# ---------------------------------------------------------------------------
def test_should_extract_vendor_category_for_vanilla_rate_limit_error():
err = RateLimitError(message="vendor 429", llm_provider="openai", model="gpt-4o")
category, rate_limit_type = PrometheusLogger._extract_rate_limit_labels(err)
assert category == "vendor_rate_limit"
assert rate_limit_type is None
def test_should_extract_litellm_category_and_type_for_proxy_rate_limit_error():
err = ProxyRateLimitError(
detail={"error": "tpm exceeded"},
category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
rate_limit_type=RateLimitType.TOKENS,
)
category, rate_limit_type = PrometheusLogger._extract_rate_limit_labels(err)
assert category == "litellm_rate_limit"
assert rate_limit_type == "tokens"
def test_should_return_none_for_non_rate_limit_exception():
assert PrometheusLogger._extract_rate_limit_labels(ValueError("nope")) == (
None,
None,
)
def test_should_return_none_for_none_exception():
assert PrometheusLogger._extract_rate_limit_labels(None) == (None, None)
@pytest.mark.parametrize(
"category_enum,rate_limit_enum,expected_category,expected_type",
[
(
RateLimitErrorCategory.LITELLM_RATE_LIMIT,
RateLimitType.REQUESTS,
"litellm_rate_limit",
"requests",
),
(
RateLimitErrorCategory.LITELLM_RATE_LIMIT,
RateLimitType.TOKENS,
"litellm_rate_limit",
"tokens",
),
(
RateLimitErrorCategory.LITELLM_RATE_LIMIT,
RateLimitType.CONCURRENT_REQUESTS,
"litellm_rate_limit",
"concurrent_requests",
),
(
RateLimitErrorCategory.LITELLM_RATE_LIMIT,
RateLimitType.BUDGET,
"litellm_rate_limit",
"budget",
),
(
RateLimitErrorCategory.LITELLM_RATE_LIMIT,
RateLimitType.MAX_ITERATIONS,
"litellm_rate_limit",
"max_iterations",
),
(
RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT,
RateLimitType.REQUESTS,
"litellm_batch_rate_limit",
"requests",
),
],
)
def test_should_serialize_rate_limit_enums_as_underlying_string_values(
category_enum, rate_limit_enum, expected_category, expected_type
):
err = ProxyRateLimitError(
detail="boom", category=category_enum, rate_limit_type=rate_limit_enum
)
category, rate_limit_type = PrometheusLogger._extract_rate_limit_labels(err)
assert category == expected_category
assert rate_limit_type == expected_type
# ---------------------------------------------------------------------------
# _get_exception_class_name back-compat
# ---------------------------------------------------------------------------
def test_should_emit_legacy_http_exception_label_for_proxy_rate_limit_error():
"""
``ProxyRateLimitError`` multi-inherits from ``HTTPException`` +
``RateLimitError``. The ``exception_class`` label MUST keep emitting
"HTTPException" for back-compat with existing dashboards (see Slack
thread + PR #27687 review). Distinguishing vendor vs. litellm 429s
is now the job of the new ``rate_limit_category`` label.
"""
err = ProxyRateLimitError(detail={"error": "boom"})
assert PrometheusLogger._get_exception_class_name(err) == "HTTPException"
def test_should_keep_provider_prefixed_exception_class_for_vendor_rate_limit_errors():
err = RateLimitError(message="vendor 429", llm_provider="openai", model="gpt-4o")
# Vendor-side errors keep the historical "Provider.ClassName" formatting.
assert PrometheusLogger._get_exception_class_name(err) == "Openai.RateLimitError"
def test_should_preserve_exception_class_name_for_unrelated_exceptions():
assert PrometheusLogger._get_exception_class_name(ValueError("nope")) == (
"ValueError"
)
# ---------------------------------------------------------------------------
# End-to-end wiring through async_post_call_failure_hook
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_should_populate_rate_limit_labels_for_proxy_rate_limit_error_on_failure_hook():
"""
When a proxy hook raises ``ProxyRateLimitError`` and the failure flows
through ``async_post_call_failure_hook``, the resulting
``UserAPIKeyLabelValues`` must carry both new labels AND keep
``exception_class="HTTPException"`` for back-compat.
"""
with patch(
"litellm.integrations.prometheus.PrometheusLogger.__init__", return_value=None
):
logger = PrometheusLogger()
logger.litellm_proxy_failed_requests_metric = MagicMock()
logger.litellm_proxy_total_requests_metric = MagicMock()
logger.get_labels_for_metric = MagicMock(
return_value=PrometheusMetricLabels.get_labels(
"litellm_proxy_failed_requests_metric"
)
)
err = ProxyRateLimitError(
detail={"error": "rpm exceeded"},
category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
rate_limit_type=RateLimitType.REQUESTS,
)
with patch(
"litellm.integrations.prometheus.prometheus_label_factory"
) as mock_label_factory:
mock_label_factory.return_value = {}
await logger.async_post_call_failure_hook(
request_data={"model": "gpt-4o-mini", "metadata": {}},
original_exception=err,
user_api_key_dict=UserAPIKeyAuth(token="t"),
)
enum_values = mock_label_factory.call_args_list[0].kwargs["enum_values"]
assert isinstance(enum_values, UserAPIKeyLabelValues)
assert enum_values.rate_limit_category == "litellm_rate_limit"
assert enum_values.rate_limit_type == "requests"
# Back-compat: exception_class on a ProxyRateLimitError stays "HTTPException".
assert enum_values.exception_class == "HTTPException"
assert enum_values.exception_status == "429"
@pytest.mark.asyncio
async def test_should_populate_rate_limit_labels_for_vendor_rate_limit_error_on_failure_hook():
with patch(
"litellm.integrations.prometheus.PrometheusLogger.__init__", return_value=None
):
logger = PrometheusLogger()
logger.litellm_proxy_failed_requests_metric = MagicMock()
logger.litellm_proxy_total_requests_metric = MagicMock()
logger.get_labels_for_metric = MagicMock(
return_value=PrometheusMetricLabels.get_labels(
"litellm_proxy_failed_requests_metric"
)
)
err = RateLimitError(message="upstream 429", llm_provider="openai", model="gpt-4o")
with patch(
"litellm.integrations.prometheus.prometheus_label_factory"
) as mock_label_factory:
mock_label_factory.return_value = {}
await logger.async_post_call_failure_hook(
request_data={"model": "gpt-4o", "metadata": {}},
original_exception=err,
user_api_key_dict=UserAPIKeyAuth(token="t"),
)
enum_values = mock_label_factory.call_args_list[0].kwargs["enum_values"]
assert isinstance(enum_values, UserAPIKeyLabelValues)
assert enum_values.rate_limit_category == "vendor_rate_limit"
assert enum_values.rate_limit_type is None
# Vendor errors keep the historical Provider.ClassName label.
assert enum_values.exception_class == "Openai.RateLimitError"
assert enum_values.exception_status == "429"
@pytest.mark.asyncio
async def test_should_leave_rate_limit_labels_blank_for_non_rate_limit_failure():
with patch(
"litellm.integrations.prometheus.PrometheusLogger.__init__", return_value=None
):
logger = PrometheusLogger()
logger.litellm_proxy_failed_requests_metric = MagicMock()
logger.litellm_proxy_total_requests_metric = MagicMock()
logger.get_labels_for_metric = MagicMock(
return_value=PrometheusMetricLabels.get_labels(
"litellm_proxy_failed_requests_metric"
)
)
with patch(
"litellm.integrations.prometheus.prometheus_label_factory"
) as mock_label_factory:
mock_label_factory.return_value = {}
await logger.async_post_call_failure_hook(
request_data={"model": "gpt-4o", "metadata": {}},
original_exception=RuntimeError("boom"),
user_api_key_dict=UserAPIKeyAuth(token="t"),
)
enum_values = mock_label_factory.call_args_list[0].kwargs["enum_values"]
assert isinstance(enum_values, UserAPIKeyLabelValues)
assert enum_values.rate_limit_category is None
assert enum_values.rate_limit_type is None