scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-06 02:48:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103
litellm/litellm
History
Yuneng Jiang e71e6fd726
fix(redis): reuse the async connection pool across client-cache rotations 
RedisCache caches its async client in in_memory_llm_clients_cache, which
expires entries on a 600s TTL. Every rotation took the cache-miss branch and
built a brand new BlockingConnectionPool alongside the replacement client.

The outgoing pool is never disconnected; its sockets stay open until the pool
is garbage collected and each connection's __del__ happens to run under a live
event loop. So while the new pool opens its own connections, the old set is
still established, and a single RedisCache transiently holds roughly twice its
working set against the server every 10 minutes. A proxy runs several of these
(the LLM cache, the router cache and the coordination cache) per worker, and
each pool defaults to redis-py's max_connections of 50, so the aggregate is
what walks into "max number of clients reached".

Build the pool once per event loop instead, and hand the same pool to each
replacement client. redis-py sets auto_close_connection_pool=False whenever a
pool is passed in, so an expiring client cannot disconnect a pool that a newer
client is still serving traffic from. The pool is still rebuilt when the
running loop changes, since its connections are bound to the loop that opened
them.

Measured against a local Redis, 40 concurrent ops over 8 rotations: peak
server-side connections drop from 83 to 43 and pools built from 8 to 1.
2026-07-27 18:09:06 -07:00
..
a2a_protocol fix(responses): stop scheduling sync success_handler concurrently with async_success_handler (#32239) 2026-07-07 09:13:50 -07:00
anthropic_interface style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
assistants style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
batch_completion style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
batches Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
caching fix(redis): reuse the async connection pool across client-cache rotations 2026-07-27 18:09:06 -07:00
completion_extras fix(responses-bridge): return CustomStreamWrapper from the completed-response stream helper 2026-07-27 14:48:18 -07:00
compression style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
containers style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
endpoints/speech/speech_to_completion_bridge style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
evals style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
experimental_mcp_client fix(mcp): sanitize Anthropic tool schemas and stop encoding gateway names 2026-07-17 10:33:28 -07:00
files style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
fine_tuning chore(lint): zero out crash-class pyright rules and ban new type: ignore comments (#32152) 2026-07-04 16:56:12 -07:00
google_genai fix(logging): classify async anthropic_messages and generate_content as async (#33589) 2026-07-16 20:56:47 -07:00
images style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
integrations fix(guardrails): classify all 4xx HTTPException guardrail blocks as intervened (#33821) 2026-07-27 16:50:13 -07:00
interactions fix(responses): stop scheduling sync success_handler concurrently with async_success_handler (#32239) 2026-07-07 09:13:50 -07:00
litellm_core_utils Merge pull request #34675 from BerriAI/litellm_tool_spend_rollup 2026-07-27 15:31:19 -07:00
llms chore(typing): clear 2.7k basedpyright Any errors across 15 hotspot files 2026-07-26 18:39:26 -07:00
models feat(mcp): issuer-anchored OAuth discovery (RFC 8414 §3.3) as the trust anchor 2026-07-15 16:59:01 -07:00
ocr chore(typing): clear 2.7k basedpyright Any errors across 15 hotspot files 2026-07-26 18:39:26 -07:00
passthrough fix(logging): classify allm_passthrough_route as async to prevent duplicate success callbacks (#32265) 2026-07-06 15:04:05 -07:00
proxy chore: update Next.js build artifacts (2026-07-28 00:06 UTC, node v20.20.2) (#34859) 2026-07-27 17:20:51 -07:00
proxy_auth style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
rag fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_api fix(vertex): forward realtime health check params (#32550) 2026-07-08 17:30:46 -07:00
repositories Merge pull request #34675 from BerriAI/litellm_tool_spend_rollup 2026-07-27 15:31:19 -07:00
rerank_api fix(rerank): log optional_rerank_params at debug to stop leaking request content (#32533) 2026-07-08 18:48:03 -07:00
responses fix(responses): strip include_usage from stream_options instead of dropping the param 2026-07-24 16:46:49 -07:00
router_strategy feat(complexity-router): add return_raw_model_name toggle for response model field (#33875) 2026-07-18 19:24:56 -07:00
router_utils fix(router): don't cool down parent deployment on advisor sub-call failure (#33792) 2026-07-25 10:17:13 -07:00
rust_bridge perf(bedrock): audio transcription via rust core (py->rust bridge) (#33990) 2026-07-20 14:09:41 -07:00
sandbox style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
search style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
secret_managers fix(bedrock): stop replaying expired Google OIDC tokens to STS on guardrail auth (#34637) 2026-07-25 16:27:54 -07:00
skills style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
types Merge pull request #34675 from BerriAI/litellm_tool_spend_rollup 2026-07-27 15:31:19 -07:00
vector_store_files style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
vector_stores style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
videos style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
__init__.py feat(proxy): add overwrite_user_with_key_hash to stamp outgoing user param with key hash (#34417) 2026-07-23 16:38:01 -07:00
_internal_context.py fix(proxy): harden request parameter handling 2026-04-16 01:38:12 +00:00
_lazy_imports.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
_lazy_imports_registry.py feat(tencent): add Tencent TokenHub as a provider (#31903) 2026-07-02 18:31:59 -07:00
_logging.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
_redis.py fix(redis): honor ssl value instead of key presence when building async connection pool (#32590) 2026-07-16 13:38:01 -07:00
_redis_credential_provider.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
_service_logger.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
_uuid.py build(deps-dev): bump black to 26.3.1 and apply formatting (#28525) 2026-05-21 17:24:18 -07:00
_version.py
anthropic_beta_headers_config.json fix(bedrock-invoke): retain clear_tool_uses_20250919 context_management edits and emit context-management-2025-06-27 beta (LIT-3393) (#32658) 2026-07-09 14:31:27 -07:00
anthropic_beta_headers_manager.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
blog_posts.json docs(blog): add WebRTC blog post link 2026-03-13 17:54:33 +05:30
budget_manager.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
constants.py fix(proxy): roll up tool spend daily instead of scanning SpendLogs 2026-07-25 21:52:58 -07:00
cost.json
cost_calculator.py fix(cost_calculator): sum mirrored cache token fields once in combine_usage_objects 2026-07-23 19:07:09 -07:00
exceptions.py feat(logging): add structured budget fields to budget rejection failure logs (#33460) 2026-07-16 12:39:04 -07:00
main.py fix(router): stop per-deployment num_retries from double-counting as provider max_retries (#34129) 2026-07-21 21:13:32 +00:00
model_prices_and_context_window_backup.json Merge pull request #34556 from BerriAI/litellm_azure_claude_1m_context 2026-07-25 18:32:28 -07:00
policy_templates_backup.json feat(add-new-block_code_execution-guardrail): prevent agent from executing code (#22154) 2026-02-25 22:02:14 -08:00
provider_endpoints_support_backup.json chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
py.typed
router.py fix(router): repair deployment indices before releasing strategies on delete 2026-07-27 15:32:27 -07:00
scheduler.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
setup_wizard.py feat(anthropic): add Claude Opus 5 2026-07-24 10:43:49 -07:00
timeout.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
utils.py fix: match exact class in callback dedup so a custom subclass does not block a built-in logger (#34804) 2026-07-27 12:18:42 -07:00