diff --git a/docs/my-website/docs/troubleshoot/latency_overhead.md b/docs/my-website/docs/troubleshoot/latency_overhead.md new file mode 100644 index 00000000000..1dea11c0d57 --- /dev/null +++ b/docs/my-website/docs/troubleshoot/latency_overhead.md @@ -0,0 +1,78 @@ +# Latency Overhead Troubleshooting + +Use this guide when you see unexpected latency overhead between LiteLLM proxy and the LLM provider. + +## Quick Checklist + +1. **Is DEBUG logging enabled?** This is the #1 cause of latency with large payloads. +2. **Are you sending large base64 payloads?** (images, PDFs) — see [Large Payload Overhead](#large-payload-overhead). +3. **Enable detailed timing headers** to pinpoint where time is spent. + +## Diagnostic Headers + +### `x-litellm-callback-duration-ms` (always on) + +Shows time spent building callback/logging payloads (ms). If this is high (>100ms), your payloads may be too large for efficient logging. + +```bash +curl -s -D - http://localhost:4000/v1/chat/completions \ + -H "Authorization: Bearer sk-..." \ + -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}' \ + 2>&1 | grep x-litellm +``` + +### Detailed Timing Breakdown (opt-in) + +Set `LITELLM_DETAILED_TIMING=true` to get per-phase timing in response headers: + +| Header | What it measures | +|--------|-----------------| +| `x-litellm-timing-pre-processing-ms` | Auth, routing, request processing (before LLM call) | +| `x-litellm-timing-llm-api-ms` | Actual LLM API call duration | +| `x-litellm-timing-post-processing-ms` | Response processing (after LLM returns) | +| `x-litellm-timing-message-copy-ms` | Message copy time in logging layer | + +```bash +# Enable detailed timing +export LITELLM_DETAILED_TIMING=true +``` + +## Large Payload Overhead + +When sending large payloads (>1MB, e.g. base64-encoded images/PDFs), three things can add overhead: + +### 1. DEBUG Logging (most common) + +When `LITELLM_LOG=DEBUG` or `set_verbose=True` is enabled, every request payload is serialized with `json.dumps(indent=4)` synchronously. For a 2MB+ payload, this alone can take **2-5 seconds**. + +**Fix:** Don't use DEBUG logging in production. Use `INFO` level instead: + +```bash +export LITELLM_LOG=INFO +``` + +If you need DEBUG logging but have large payloads, you can increase the size threshold for full payload logging: + +```bash +# Only fully serialize payloads under 100KB for DEBUG logs (default) +export MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG=102400 +``` + +### 2. Base64 in Logging Payloads + +Callback payloads (sent to Langfuse, etc.) include message content. Large base64 strings are automatically truncated to size placeholders in logging payloads. + +You can control the truncation threshold: + +```bash +# Max base64 characters before truncation (default: 64) +export MAX_BASE64_LENGTH_FOR_LOGGING=64 +``` + +## Environment Variables Reference + +| Variable | Default | Description | +|----------|---------|-------------| +| `LITELLM_DETAILED_TIMING` | `false` | Enable per-phase timing headers | +| `MAX_PAYLOAD_SIZE_FOR_DEBUG_LOG` | `102400` | Max payload bytes for full DEBUG serialization | +| `MAX_BASE64_LENGTH_FOR_LOGGING` | `64` | Max base64 chars before truncation in logging |