mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
14d50599fa
The limiter's keys ({descriptor:value}:requests, :tokens, :window and the
:max_parallel_requests gauge), its eight Lua scripts and its memory tier now
live in litellm-redis-store (rate_limits.rs). The scripts are byte for byte
main's, so both stacks share one script SHA during a rolling deploy.
The Python limiter keeps its admission logic and talks to a thin store in
litellm/redis/stores/rate_limits.py with domain ids (RateLimitScope,
RateLimitCounter) instead of key strings. Inside a request scope its
scripts ride the request's native pre-call batch, so the first admission
script and the armed cooldown read go out in one pipeline. Post-call token
scripts, window-guarded settlements, slot releases and refund increments
ride the native post-call batch, and their fallbacks run once it settles.
The batch and dynamic limiters read and write the keyspace through the
same store.
Rust tests run every script against real Redis, standalone and cluster,
including over-limit, refunds, gauge slot expiry and cross-slot behavior.
Matching inline wire cases are checked from both sides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The limiter sends the commands main does. Counter reads, slot acquire, count
and the pre-response release are direct EVALSHA calls with the counter-read
groups one after another, window-guarded token adjusts run one by one on their
own trips, plain increments are INCRBYFLOAT with EXPIRE, and an increment
without a TTL takes RedisCache.default_ttl like async_increment.
|
||
|---|---|---|
| .. | ||
| _support | ||
| audio_tests | ||
| base_sdk_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| e2e_harness | ||
| fixtures | ||
| harness_e2e | ||
| integration | ||
| rust-python-harness | ||
| test_litellm_rust | ||
| typing | ||
| unit | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _master_key.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_logging.conf | ||
| test_rust_python_harness.py | ||
| VCR_TESTING.md | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.