litellm

mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00

History

Yassin Kortam 06e8013e6c fix(logging): stop pinning large request payloads past request end (#33455 ) Three process-lifetime retention points kept full request payloads (messages included) alive after the request finished. Under bursts of large-token traffic (~73K tokens/request mean) this presented as stepwise RSS growth that never returned to baseline, ending in OOM: 1. Logging.pre_call/post_call stored their entire locals() (messages, the Logging object, complete_input_dict) in the module-level litellm.error_logs dict, pinning the most recent request's payload per worker forever. Nothing reads that dict; the writes are removed. 2. LLMCachingHandler.request_kwargs kept litellm_logging_obj inside the stored kwargs while the handler itself hangs off logging_obj._llm_caching_handler, closing a reference cycle (Logging -> LLMCachingHandler -> kwargs -> Logging). Cyclic payloads are only reclaimed by generational GC, so megabytes of dead request data lingered until a rare gen-2 pass, and the transient copies fragment the allocator into a permanent RSS high-water mark. The handler now drops litellm_logging_obj from its stored kwargs; the caching layer never reads it. 3. The router stored every request's kwargs in the ITPM/OTPM contextvar even when no deployment configures itpm/otpm. Pooled resources created mid-request (e.g. redis connections) capture the asyncio context, extending that pin far past the request. The slot is now populated only for deployments with io token limits and overwritten with None otherwise. Live-proxy verification (bursts of 30 x ~300KB requests, PII guardrail + prometheus + redis cache): unfixed grows 16-29MB per burst without release; fixed grows under 1MB per burst after warmup and flattens. Resolves LIT-4434	2026-07-15 15:28:37 -07:00
..
test_enforce_model_rate_limits.py	fix: use atomic increment-first pattern for model RPM rate limiting	2026-02-24 09:55:07 -03:00
test_io_token_rate_limits.py	fix(logging): stop pinning large request payloads past request end (#33455 )	2026-07-15 15:28:37 -07:00

fix(logging): stop pinning large request payloads past request end (#33455 )

Three process-lifetime retention points kept full request payloads
(messages included) alive after the request finished. Under bursts of
large-token traffic (~73K tokens/request mean) this presented as
stepwise RSS growth that never returned to baseline, ending in OOM:

1. Logging.pre_call/post_call stored their entire locals() (messages,
   the Logging object, complete_input_dict) in the module-level
   litellm.error_logs dict, pinning the most recent request's payload
   per worker forever. Nothing reads that dict; the writes are removed.

2. LLMCachingHandler.request_kwargs kept litellm_logging_obj inside the
   stored kwargs while the handler itself hangs off
   logging_obj._llm_caching_handler, closing a reference cycle
   (Logging -> LLMCachingHandler -> kwargs -> Logging). Cyclic payloads
   are only reclaimed by generational GC, so megabytes of dead request
   data lingered until a rare gen-2 pass, and the transient copies
   fragment the allocator into a permanent RSS high-water mark. The
   handler now drops litellm_logging_obj from its stored kwargs; the
   caching layer never reads it.

3. The router stored every request's kwargs in the ITPM/OTPM contextvar
   even when no deployment configures itpm/otpm. Pooled resources
   created mid-request (e.g. redis connections) capture the asyncio
   context, extending that pin far past the request. The slot is now
   populated only for deployments with io token limits and overwritten
   with None otherwise.

Live-proxy verification (bursts of 30 x ~300KB requests, PII guardrail
+ prometheus + redis cache): unfixed grows 16-29MB per burst without
release; fixed grows under 1MB per burst after warmup and flattens.

Resolves LIT-4434

2026-07-15 15:28:37 -07:00

test_enforce_model_rate_limits.py

fix: use atomic increment-first pattern for model RPM rate limiting

2026-02-24 09:55:07 -03:00

test_io_token_rate_limits.py

fix(logging): stop pinning large request payloads past request end (#33455 )

2026-07-15 15:28:37 -07:00