Krrish Dholakia dd57ae6691 feat(rate-limit): atomic check-and-increment-by-N for multi-process safety ... The previous fix for the TOCTOU bypass relied on a per-instance asyncio.Lock, which closed the window only within a single proxy worker. Multi-replica deployments still raced across processes — A and B both read counter=99, both passed validation, both incremented to 100/100 → effective limit doubled. Add `CHECK_AND_INCREMENT_BY_N_SCRIPT` Lua script that processes any number of (window_key, counter_key, limit, increment, ttl) descriptors atomically with all-or-nothing semantics: if any descriptor would exceed its limit, no counter is modified and the script returns OVER_LIMIT with the offending descriptor's state. When Redis isn't configured, the in-memory fallback uses the existing asyncio.Lock for single-process atomicity. Expose this as `_PROXY_MaxParallelRequestsHandler_v3.atomic_check_and_increment_by_n` and rewire both call sites: - batch_rate_limiter._check_and_increment_batch_counters: replace the read_only=True check + separate async_increment_tokens_with_ttl_preservation with a single atomic call passing the batch's (request_count, total_tokens) as the increment. - dynamic_rate_limiter_v3._check_rate_limits: bundle model_saturation_check (always enforced) and priority_model (enforced only when saturated) into one atomic call. When priority is unenforced, increment its counter via the existing should_rate_limit(read_only=False) path for tracking only. Update structural regression tests to assert the new atomic path is used rather than the legacy two-phase pattern. Tests: 4/4 TOCTOU tests pass, 59 existing rate-limiter tests pass, no regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>		2026-04-30 18:54:36 -07:00
..
test_async_post_call_streaming_iterator_hook.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_dynamic_rate_limiter_v3.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_image_generation_guardrails.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_key_management_event_hooks.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_max_budget_per_session_limiter.py	Agents - add max budget + tpm/rpm limiting per agent AND per agent session (#22849)	2026-03-07 19:12:42 -08:00
test_max_iterations_limiter.py	Agents - add max budget + tpm/rpm limiting per agent AND per agent session (#22849)	2026-03-07 19:12:42 -08:00
test_parallel_request_limiter_v3.py	Merge branch 'litellm_internal_staging' into litellm_project_rate_limiting	2026-04-17 19:10:06 -07:00
test_post_call_failure_hook_integration.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_post_call_response_headers_hook.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_post_call_streaming_hook_integration.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_post_call_success_hook_integration.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_prompt_injection_detection.py	fix: prompt injection not working (#16701)	2025-11-17 20:04:57 -08:00
test_proxy_track_cost_callback.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_rate_limiter_toctou.py	feat(rate-limit): atomic check-and-increment-by-N for multi-process safety	2026-04-30 18:54:36 -07:00
test_send_invite_email.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00