scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-11 08:38:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 280

docs(otel): justify the retired-provider cap

Browse source
This commit is contained in:
mubashir1osmani 2026-08-19 21:14:51 -04:00
parent 9063c39b9d
commit 503879d83d
1 changed files with 6 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
litellm/integrations/otel/plumbing/routing.py
View file

@ -53,9 +53,12 @@ _MAX_CACHED_PROVIDERS: Final = 256
# other bound is the logger's open-call map (10k), so without this a caller
# cycling unique credential sets across long-lived calls could pin far more
# live providers, and exporter threads, than the cache cap allows. Past this
# many, the stalest retiree is shut down, dropping whatever it was draining: by
# then a span from a route evicted long ago that never closed. Total live
# providers is therefore ``_MAX_CACHED_PROVIDERS + _MAX_RETIRED_PROVIDERS``.
# many, the stalest retiree is shut down and whatever it was draining is
# dropped (a shut-down ``BatchSpanProcessor`` discards spans handed to it after
# the fact), which by then means a span on a route evicted long ago. A quarter
# of the cache cap: enough that a burst of tenant churn during long-lived calls
# still drains normally, small enough that the worst case is a bounded 320
# providers rather than one per concurrent call.
_MAX_RETIRED_PROVIDERS: Final = 64
_HeaderItems: TypeAlias = tuple[tuple[str, str], ...]