docs: add in_flight_requests to prometheus metrics and latency troubleshooting

2026-09-09 22:31:41 +00:00 · 2026-02-27 17:57:50 -08:00 · 2026-02-27 17:57:50 -08:00 · 539e9b9524
commit 539e9b9524
parent 03500799be
3 changed files with 151 additions and 72 deletions
--- a/docs/my-website/docs/proxy/prometheus.md
+++ b/docs/my-website/docs/proxy/prometheus.md
@ -113,6 +113,31 @@ litellm_settings:
 ```


+## Pod Health Metrics
+
+Use these to measure per-pod queue depth and diagnose latency that occurs **before** LiteLLM starts processing a request.
+
+| Metric Name | Type | Description |
+|---|---|---|
+| `litellm_in_flight_requests` | Gauge | Number of HTTP requests currently in-flight on this uvicorn worker. Tracks the pod's queue depth in real time. With multiple workers, values are summed across all live workers (`livesum`). |
+
+### When to use this
+
+LiteLLM measures latency from when its handler starts. If a request waits in uvicorn's event loop before the handler runs, that wait is invisible to LiteLLM's own logs. `litellm_in_flight_requests` shows how loaded the pod was at any point in time.
+
+```
+high in_flight_requests + high ALB TargetResponseTime → pod overloaded, scale out
+low  in_flight_requests + high ALB TargetResponseTime → delay is pre-ASGI (event loop blocking)
+```
+
+You can also check the current value directly without Prometheus:
+
+```bash
+curl http://localhost:4000/health/backlog \
+  -H "Authorization: Bearer sk-..."
+# {"in_flight_requests": 47}
+```
+
 ## Proxy Level Tracking Metrics

 Use this to track overall LiteLLM Proxy usage.
--- a/docs/my-website/docs/troubleshoot/latency_overhead.md
+++ b/docs/my-website/docs/troubleshoot/latency_overhead.md
@ -0,0 +1,122 @@
+# Latency Overhead Troubleshooting
+
+Use this guide when you see unexpected latency overhead between LiteLLM proxy and the LLM provider.
+
+## The Invisible Latency Gap
+
+LiteLLM measures latency from when its handler starts. If a request waits in uvicorn's event loop **before** the handler runs, that wait is invisible to LiteLLM's own logs.
+
+```
+T=0   Request arrives at load balancer
+      [queue wait — LiteLLM never logs this]
+T=10  LiteLLM handler starts → timer begins
+T=20  Response sent
+
+LiteLLM logs: 10s    User experiences: 20s
+```
+
+To measure the pre-handler wait, poll `/health/backlog` on each pod:
+
+```bash
+curl http://localhost:4000/health/backlog \
+  -H "Authorization: Bearer sk-..."
+# {"in_flight_requests": 47}
+```
+
+Or scrape the `litellm_in_flight_requests` Prometheus gauge at `/metrics`.
+
+| `in_flight_requests` | ALB `TargetResponseTime` | Diagnosis |
+|---|---|---|
+| High | High | Pod overloaded → scale out |
+| Low | High | Delay is pre-ASGI — check for sync blocking code or event loop saturation |
+| High | Normal | Pod is busy but healthy, no queue buildup |
+
+If you're on **AWS ALB**, correlate `litellm_in_flight_requests` spikes with ALB's `TargetResponseTime` CloudWatch metric. The gap between what ALB reports and what LiteLLM logs is the invisible wait.
+
+## Quick Checklist
+
+1. **Check `in_flight_requests` on each pod** via `/health/backlog` or the `litellm_in_flight_requests` Prometheus gauge — this tells you if requests are queuing before LiteLLM starts processing. Start here for unexplained latency.
+2. **Collect the `x-litellm-overhead-duration-ms` response header** — this tells you LiteLLM's total overhead on every request.
+2. **Is DEBUG logging enabled?** This is the #1 cause of latency with large payloads.
+3. **Are you sending large base64 payloads?** (images, PDFs) — see [Large Payload Overhead](#large-payload-overhead).
+4. **Enable detailed timing headers** to pinpoint where time is spent.
+
+## Diagnostic Headers
+
+### `x-litellm-overhead-duration-ms` (always on)
+
+Every response from LiteLLM includes this header. It shows the total latency overhead in milliseconds added by LiteLLM proxy (i.e. total response time minus the LLM API call time). Collect this on every request to understand your baseline overhead.
+
+```bash
+curl -s -D - http://localhost:4000/v1/chat/completions \
+  -H "Authorization: Bearer sk-..." \
+  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}' \
+  2>&1 | grep x-litellm-overhead-duration-ms
+```
+
+### `x-litellm-callback-duration-ms` (always on)
+
+Shows time spent building callback/logging payloads (ms). If this is high (>100ms), your payloads may be too large for efficient logging.
+
+```bash
+curl -s -D - http://localhost:4000/v1/chat/completions \
+  -H "Authorization: Bearer sk-..." \
+  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}' \
+  2>&1 | grep x-litellm
+```
+
+### Detailed Timing Breakdown (opt-in)
+
+Set `LITELLM_DETAILED_TIMING=true` to get per-phase timing in response headers:
+
+| Header | What it measures |
+|--------|-----------------|
+| `x-litellm-timing-pre-processing-ms` | Auth, routing, request processing (before LLM call) |
+| `x-litellm-timing-llm-api-ms` | Actual LLM API call duration |
+| `x-litellm-timing-post-processing-ms` | Response processing (after LLM returns) |
+| `x-litellm-timing-message-copy-ms` | Message copy time in logging layer |
+
+```bash
+# Enable detailed timing
+export LITELLM_DETAILED_TIMING=true
+```
+
+## Large Payload Overhead
+
+When sending large payloads (>1MB, e.g. base64-encoded images/PDFs), three things can add overhead:
+
+### 1. DEBUG Logging (most common)
+
+When `LITELLM_LOG=DEBUG` or `set_verbose=True` is enabled, every request payload is serialized with `json.dumps(indent=4)` synchronously. For a 2MB+ payload, this alone can take **2-5 seconds**.
+
+**Fix:** Don't use DEBUG logging in production. Use `INFO` level instead:
+
+```bash
+export LITELLM_LOG=INFO
+```
+
+If you need DEBUG logging but have large payloads, you can increase the size threshold for full payload logging:
+
+```bash
+# Only fully serialize payloads under 100KB for DEBUG logs (default)
+export MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG=102400
+```
+
+### 2. Base64 in Logging Payloads
+
+Callback payloads (sent to Langfuse, etc.) include message content. Large base64 strings are automatically truncated to size placeholders in logging payloads.
+
+You can control the truncation threshold:
+
+```bash
+# Max base64 characters before truncation (default: 64)
+export MAX_BASE64_LENGTH_FOR_LOGGING=64
+```
+
+## Environment Variables Reference
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LITELLM_DETAILED_TIMING` | `false` | Enable per-phase timing headers |
+| `MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG` | `102400` | Max payload bytes for full DEBUG serialization |
+| `MAX_BASE64_LENGTH_FOR_LOGGING` | `64` | Max base64 chars before truncation in logging |
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@ -17357,39 +17357,6 @@
        "supports_tool_choice": true,
        "supports_vision": false
    },
-    "gpt-audio-1.5": {
-        "input_cost_per_audio_token": 3.2e-05,
-        "input_cost_per_token": 2.5e-06,
-        "litellm_provider": "openai",
-        "max_input_tokens": 128000,
-        "max_output_tokens": 16384,
-        "max_tokens": 16384,
-        "mode": "chat",
-        "output_cost_per_audio_token": 6.4e-05,
-        "output_cost_per_token": 1e-05,
-        "supported_endpoints": [
-            "/v1/chat/completions"
-        ],
-        "supported_modalities": [
-            "text",
-            "audio"
-        ],
-        "supported_output_modalities": [
-            "text",
-            "audio"
-        ],
-        "supports_audio_input": true,
-        "supports_audio_output": true,
-        "supports_function_calling": true,
-        "supports_native_streaming": true,
-        "supports_parallel_function_calling": true,
-        "supports_prompt_caching": false,
-        "supports_reasoning": false,
-        "supports_response_schema": false,
-        "supports_system_messages": true,
-        "supports_tool_choice": true,
-        "supports_vision": false
-    },
    "gpt-audio-2025-08-28": {
        "input_cost_per_audio_token": 3.2e-05,
        "input_cost_per_token": 2.5e-06,
@ -19074,38 +19041,6 @@
        "supports_system_messages": true,
        "supports_tool_choice": true
    },
-    "gpt-realtime-1.5": {
-        "cache_creation_input_audio_token_cost": 4e-07,
-        "cache_read_input_token_cost": 4e-07,
-        "input_cost_per_audio_token": 3.2e-05,
-        "input_cost_per_image": 5e-06,
-        "input_cost_per_token": 4e-06,
-        "litellm_provider": "openai",
-        "max_input_tokens": 32000,
-        "max_output_tokens": 4096,
-        "max_tokens": 4096,
-        "mode": "chat",
-        "output_cost_per_audio_token": 6.4e-05,
-        "output_cost_per_token": 1.6e-05,
-        "supported_endpoints": [
-            "/v1/realtime"
-        ],
-        "supported_modalities": [
-            "text",
-            "image",
-            "audio"
-        ],
-        "supported_output_modalities": [
-            "text",
-            "audio"
-        ],
-        "supports_audio_input": true,
-        "supports_audio_output": true,
-        "supports_function_calling": true,
-        "supports_parallel_function_calling": true,
-        "supports_system_messages": true,
-        "supports_tool_choice": true
-    },
    "gpt-realtime-mini": {
        "cache_creation_input_audio_token_cost": 3e-07,
        "cache_read_input_audio_token_cost": 3e-07,
@ -25066,8 +25001,8 @@
        "mode": "chat",
        "output_cost_per_token": 0.0,
        "source": "https://platform.publicai.co/docs",
-        "supports_function_calling": false,
-        "supports_tool_choice": false
+        "supports_function_calling": true,
+        "supports_tool_choice": true
    },
    "publicai/swiss-ai/apertus-70b-instruct": {
        "input_cost_per_token": 0.0,
@ -25078,8 +25013,8 @@
        "mode": "chat",
        "output_cost_per_token": 0.0,
        "source": "https://platform.publicai.co/docs",
-        "supports_function_calling": false,
-        "supports_tool_choice": false
+        "supports_function_calling": true,
+        "supports_tool_choice": true
    },
    "publicai/aisingapore/Gemma-SEA-LION-v4-27B-IT": {
        "input_cost_per_token": 0.0,
@ -30705,7 +30640,6 @@
        "supports_web_search": true
    },
    "xai/grok-2-vision-1212": {
-        "deprecation_date": "2026-02-28",
        "input_cost_per_image": 2e-06,
        "input_cost_per_token": 2e-06,
        "litellm_provider": "xai",
@ -30810,7 +30744,6 @@
    },
    "xai/grok-3-mini": {
        "cache_read_input_token_cost": 7.5e-08,
-        "deprecation_date": "2026-02-28",
        "input_cost_per_token": 3e-07,
        "litellm_provider": "xai",
        "max_input_tokens": 131072,
@ -30827,7 +30760,6 @@
    },
    "xai/grok-3-mini-beta": {
        "cache_read_input_token_cost": 7.5e-08,
-        "deprecation_date": "2026-02-28",
        "input_cost_per_token": 3e-07,
        "litellm_provider": "xai",
        "max_input_tokens": 131072,