mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-29 01:42:19 +00:00
f028a622e2
* fix(prometheus): emit remaining_tokens/requests gauges for bedrock + vertex (LIT-2719)
Bedrock and Vertex AI never return x-ratelimit-remaining-* response headers,
so litellm_remaining_tokens_metric / litellm_remaining_requests_metric only
fired for OpenAI / Azure / Anthropic deployments even when tpm/rpm was
configured on the router.
Add a provider-agnostic fallback in PrometheusLogger.async_log_success_event
that asks Router.get_remaining_model_group_usage() for the same model_group
and emits the gauges with configured_limit - current_usage when the upstream
provider didn't populate the headers itself. Existing OpenAI / Azure /
Anthropic flows are unchanged because the fallback short-circuits when both
header values are already present.
Tests: 8 new tests covering bedrock + vertex emission, header short-circuit,
partial-header fill, llm_router=None, missing model_group, empty router
result, and router exception swallowing.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): narrow except to ImportError, log router lookup failures via verbose_logger.exception
Address greptile review:
- The optional 'from litellm.proxy.proxy_server import llm_router' should
guard against ImportError specifically, not all exceptions, so that
unexpected errors (e.g. AttributeError from partially-initialized state)
stay visible.
- get_remaining_model_group_usage failures are now logged via
verbose_logger.exception (with traceback) instead of debug, matching the
PR description's intent and avoiding silent loss of router-cache errors
in production.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): subtract in-flight delta in router-remaining fallback
The router's TPM/RPM counter is incremented by
Router.deployment_callback_on_success, which fires alongside this
prometheus callback in the success-log fan-out. Prometheus wins the
race, so get_remaining_model_group_usage returns the pre-decrement
counter for the current request — while vendor headers
(OpenAI/Anthropic/Azure) are already post-decrement.
That broke parity between providers on the same gauge: dashboards
plotting litellm_remaining_requests_metric showed Bedrock/Vertex
perpetually one request behind Anthropic for the same throughput.
Replay the in-flight increment before emit: subtract total_tokens
from remaining_tokens and 1 from remaining_requests.
* Revert "fix(prometheus): subtract in-flight delta in router-remaining fallback"
This reverts commit 001ce95ecdd952b4b5a23dd2b1e62c4562c932bc.
* fix(router): post-decrement router-derived ratelimit headers
Router.set_response_headers injects x-ratelimit-remaining-{tokens,
requests} for providers that don't return them natively (Bedrock,
Vertex). The values come from get_remaining_model_group_usage, which
reads the router's TPM/RPM counter — incremented post-response by
deployment_callback_on_success. So the headers reflected the counter
state before the current request was counted: pre-decrement.
Vendor headers from OpenAI/Anthropic/Azure are post-decrement (the
vendor counted the request before responding). Same metric name, two
semantics — dashboards plotting litellm_remaining_requests_metric
showed Bedrock/Vertex perpetually one request behind for the same
throughput, and the HTTP response headers exposed the same skew to
clients.
Subtract the in-flight delta before writing: 1 from
remaining-requests, response.usage.total_tokens from remaining-tokens.
Fixes both the response headers and (transitively) the prometheus
gauges that read from standard_logging_payload.additional_headers.
---------
Co-authored-by: cursor <cursor@example.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| _types | ||
| agentops | ||
| arize | ||
| azure_sentinel | ||
| azure_storage | ||
| bitbucket | ||
| cloudzero | ||
| compression_interception | ||
| datadog | ||
| deepeval | ||
| dotprompt | ||
| email_templates | ||
| focus | ||
| gcs_bucket | ||
| gcs_pubsub | ||
| generic_api | ||
| generic_prompt_management | ||
| gitlab | ||
| langfuse | ||
| levo | ||
| litellm_agent | ||
| opentelemetry_utils | ||
| opik | ||
| prometheus_helpers | ||
| SlackAlerting | ||
| vantage | ||
| vector_store_integrations | ||
| weave | ||
| websearch_interception | ||
| __init__.py | ||
| additional_logging_utils.py | ||
| anthropic_cache_control_hook.py | ||
| argilla.py | ||
| athina.py | ||
| braintrust_logging.py | ||
| braintrust_mock_client.py | ||
| callback_configs.json | ||
| custom_batch_logger.py | ||
| custom_guardrail.py | ||
| custom_logger.py | ||
| custom_prompt_management.py | ||
| custom_secret_manager.py | ||
| custom_sso_handler.py | ||
| dynamodb.py | ||
| email_alerting.py | ||
| galileo.py | ||
| greenscale.py | ||
| helicone.py | ||
| helicone_mock_client.py | ||
| humanloop.py | ||
| lago.py | ||
| langsmith.py | ||
| langsmith_mock_client.py | ||
| langtrace.py | ||
| literal_ai.py | ||
| logfire_logger.py | ||
| lunary.py | ||
| mlflow.py | ||
| mock_client_factory.py | ||
| openmeter.py | ||
| opentelemetry.py | ||
| posthog.py | ||
| posthog_mock_client.py | ||
| prometheus.py | ||
| prometheus_services.py | ||
| prompt_layer.py | ||
| prompt_management_base.py | ||
| Readme.md | ||
| s3.py | ||
| s3_v2.py | ||
| sqs.py | ||
| supabase.py | ||
| test_httpx.py | ||
| traceloop.py | ||
| weights_biases.py | ||
Integrations
This folder contains logging integrations for litellm
eg. logging to Datadog, Langfuse, Prometheus, s3, GCS Bucket, etc.