mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
f028a622e2
* fix(prometheus): emit remaining_tokens/requests gauges for bedrock + vertex (LIT-2719)
Bedrock and Vertex AI never return x-ratelimit-remaining-* response headers,
so litellm_remaining_tokens_metric / litellm_remaining_requests_metric only
fired for OpenAI / Azure / Anthropic deployments even when tpm/rpm was
configured on the router.
Add a provider-agnostic fallback in PrometheusLogger.async_log_success_event
that asks Router.get_remaining_model_group_usage() for the same model_group
and emits the gauges with configured_limit - current_usage when the upstream
provider didn't populate the headers itself. Existing OpenAI / Azure /
Anthropic flows are unchanged because the fallback short-circuits when both
header values are already present.
Tests: 8 new tests covering bedrock + vertex emission, header short-circuit,
partial-header fill, llm_router=None, missing model_group, empty router
result, and router exception swallowing.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): narrow except to ImportError, log router lookup failures via verbose_logger.exception
Address greptile review:
- The optional 'from litellm.proxy.proxy_server import llm_router' should
guard against ImportError specifically, not all exceptions, so that
unexpected errors (e.g. AttributeError from partially-initialized state)
stay visible.
- get_remaining_model_group_usage failures are now logged via
verbose_logger.exception (with traceback) instead of debug, matching the
PR description's intent and avoiding silent loss of router-cache errors
in production.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): subtract in-flight delta in router-remaining fallback
The router's TPM/RPM counter is incremented by
Router.deployment_callback_on_success, which fires alongside this
prometheus callback in the success-log fan-out. Prometheus wins the
race, so get_remaining_model_group_usage returns the pre-decrement
counter for the current request — while vendor headers
(OpenAI/Anthropic/Azure) are already post-decrement.
That broke parity between providers on the same gauge: dashboards
plotting litellm_remaining_requests_metric showed Bedrock/Vertex
perpetually one request behind Anthropic for the same throughput.
Replay the in-flight increment before emit: subtract total_tokens
from remaining_tokens and 1 from remaining_requests.
* Revert "fix(prometheus): subtract in-flight delta in router-remaining fallback"
This reverts commit 001ce95ecdd952b4b5a23dd2b1e62c4562c932bc.
* fix(router): post-decrement router-derived ratelimit headers
Router.set_response_headers injects x-ratelimit-remaining-{tokens,
requests} for providers that don't return them natively (Bedrock,
Vertex). The values come from get_remaining_model_group_usage, which
reads the router's TPM/RPM counter — incremented post-response by
deployment_callback_on_success. So the headers reflected the counter
state before the current request was counted: pre-decrement.
Vendor headers from OpenAI/Anthropic/Azure are post-decrement (the
vendor counted the request before responding). Same metric name, two
semantics — dashboards plotting litellm_remaining_requests_metric
showed Bedrock/Vertex perpetually one request behind for the same
throughput, and the HTTP response headers exposed the same skew to
clients.
Subtract the in-flight delta before writing: 1 from
remaining-requests, response.usage.total_tokens from remaining-tokens.
Fixes both the response headers and (transitively) the prometheus
gauges that read from standard_logging_payload.additional_headers.
---------
Co-authored-by: cursor <cursor@example.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| conftest.py | ||
| create_mock_standard_logging_payload.py | ||
| gettysburg.wav | ||
| README.md | ||
| test_completion_no_copy.py | ||
| test_default_deployment_copy.py | ||
| test_get_model_list_alias_optimization.py | ||
| test_pre_call_checks_optimization.py | ||
| test_prompt_management_check.py | ||
| test_router_acancel_batch.py | ||
| test_router_adding_deployments.py | ||
| test_router_batch_utils.py | ||
| test_router_cooldown_utils.py | ||
| test_router_embedding_headers.py | ||
| test_router_embedding_integration.py | ||
| test_router_endpoints.py | ||
| test_router_handle_error.py | ||
| test_router_helper_utils.py | ||
| test_router_index_management.py | ||
| test_router_prompt_caching.py | ||
Router component unit tests.
Please name all files with the word 'router' in them.
This is used to ensure all functions in the router are tested.