litellm/problem_tracker.md

Zurich Performance Investigation

This document tracks two independent but compounding issues discovered during performance testing:

1. Prometheus callback overhead — latency added only when Prometheus is enabled
2. Baseline latency spikes — large spikes that occur even when callbacks are disabled

---

Problem 1: Prometheus Callback Overhead

Summary

Issue: When Prometheus is configured as a callback, request latency increases significantly. In reproducible scenarios, Prometheus increases:

• mean latency by ~27%
• p95 latency by ~42%
• requests >1s by ~2.6×

This issue was intermittent at first but became fully reproducible with random user IDs.

---

Reproduction Strategy

Baseline: Callbacks On vs Off

Multiple test batches comparing Prometheus enabled vs disabled:

Key finding

• Early batches showed severe impact
• Later batches showed smaller overhead
• Hypothesis formed: provider speed + cache behavior affects reproducibility

Batch	Callbacks	avg	p95	>1s
Worst cases	on	~0.45–0.54s	~1.2s	5–8%
Control	off	~0.30–0.38s	~0.4s	~1%

---

Making the Issue Reproducible

Random vs Sequential User IDs

Random user IDs reliably triggered the issue:

User IDs	Callbacks	p95	>1s
random	on	~1.05s	~5%
random	off	~0.50s	~1.5%
sequential	on/off	~0.39s	~0.8%

Conclusion: Prometheus overhead becomes reproducible when user lookups cause cache misses.

---

Root Cause Analysis

Suspected Areas

1. _increment_remaining_budget_metrics

   • Performs DB/cache lookups for key, team, and user
   • Executed on every request

2. Callback execution model

   • Callbacks run sequentially and block response completion

3. Prometheus label cardinality

   • High-cardinality labels amplify contention

---

Isolation & Bisection

Early Return Tests

Configuration	>1s
Full Prometheus	5.6%
Early return	2.1%

➡️ Confirms Prometheus callback is on the critical path.

---

Async vs Sync Breakdown

• Skipping sync metric updates → no improvement
• Skipping budget metrics DB lookups → major improvement

➡️ Root cause isolated to _increment_remaining_budget_metrics

---

Fix Validation

Confirming the Cause

• Commenting out _increment_remaining_budget_metrics eliminated the latency spike
• Moving it off the critical path with create_task() showed mixed results due to variance

---

Baseline Comparison (6 runs, 30k requests)

Prometheus ON vs OFF

Metric	On	Off	Δ
Mean avg	0.408s	0.322s	+27%
Mean p95	0.911s	0.641s	+42%
>1s	4.4%	1.7%	2.6×

Target after fix: Prometheus-on within ~10–15% of Prometheus-off.

---

Final Resolution

• Root cause: DB lookups on every request inside Prometheus budget metrics

• Cache was silently failing due to suppressed DB errors

• Fixes:

  • Disable check_db_only
  • Surface budget lookup failures in logs
  • Parallelize budget lookups

Fix commits

• 30534d7e82 — cache behavior fix
• d37796662 — error visibility

---

Problem 2: Baseline Latency Spikes (Independent of Prometheus)

Summary

Issue: Even with Prometheus fully disabled, Zurich sees extreme latency spikes (p95 >6s, 100% >1s).

This behavior:

• Affects callbacks on and off equally
• Is unrelated to the Prometheus fix
• Scales with end_user lookup patterns

---

Reference Baseline (Callbacks Off)

avg	p95	>1s	>5s
~4.0s	~6.3s	100%	~33%

---

Hypothesis

Baseline spikes are driven by:

• end_user DB lookups
• cache misses (especially for non-existent users)
• provider-side queuing under concurrency

---

User Mode Experiments

1000 requests / 100 users

User mode	avg	p95	>1s
none	~1.0s	~7.9s	10%
sequential	~1.5s	~8.6s	~32%
random	~1.6s	~8.0s	~34%

Key insight: Passing user dramatically increases latency.

---

One Request vs Many Requests per User

Callbacks Off, Random Users

Req/user	avg	>1s
1	~8.6s	100%
10	~1.5s	33%

➡️ Cache hits on subsequent requests reduce latency by ~6×

---

Created Users vs Random Users

Mode	avg (10/user)	>1s
random	~1.5s	33%
created	~1.2s	10%

---

Full Test Comparison (callbacks off)

1 request per user (100 requests, 100 users):

User mode	File	avg	p95	max	Above 1s
none	user_none_1_requests_per_user.txt	11.836s	20.736s	22.443s	100%
created	callbacks_off_1_per_user_created.txt	8.923s	14.962s	15.774s	100%
random	callbacks_off_1_per_user.txt	8.647s	14.909s	15.668s	100%

10 requests per user (1000 requests, 100 users):

User mode	File	avg	p95	max	Above 1s
none (callbacks off)	callbacks_off_user_none.txt	0.998s	7.966s	14.698s	10.0%
none (callbacks on)	callbacks_on_user_none.txt	1.099s	7.767s	14.873s	10.0%
created	callbacks_off_10_per_user_created.txt	1.198s	8.839s	15.483s	10.0%
random	callbacks_off_10_per_user.txt	1.538s	8.275s	15.499s	33.2%

Findings

The latency spike is high regardless of whether a user is passed to the payload or not.

---

Use a local LLM mock provider

• User none + caching true → user_none_1_request_per_user_local_llm_cache_on.txt

Config	File	avg	p95	max	Above 1s
User none, local LLM (localhost:8090), cache ON	user_none_1_request_per_user_local_llm_cache_on.txt	15.204s	26.543s	27.706s	100%

Note: With cache ON and user none, all 100 requests share the same cache key (same model + messages). Most requests are served from cache; only 1–2 requests reach the local LLM mock. Latency remains high (~15s avg) due to cache lookup and the few requests that hit the LLM.