diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 512a7b96552..c84677511bd 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -103,6 +103,7 @@ graph TD
Auth --> PreCall
PreCall --> RouteRequest
RouteRequest --> Router
+ Router --> DualCache
Router --> Main
Main --> Client
```
@@ -158,6 +159,7 @@ graph LR
end
subgraph "SDK (litellm/)"
+ Router["router.py
Router.cache (DualCache)"]
LLMCache["caching/caching_handler.py
LLMCachingHandler"]
CacheClass["caching/caching.py
Cache"]
end
@@ -166,6 +168,8 @@ graph LR
RateLimit["Rate Limit Counters"]
SpendQueue["Spend Increment Queue"]
KeyCache["API Key Cache"]
+ TPM_RPM["TPM/RPM Tracking"]
+ Cooldowns["Deployment Cooldowns"]
LLMResponseCache["LLM Response Cache"]
end
@@ -180,6 +184,8 @@ graph LR
InternalCache --> KeyCache
InternalCache -.->|cache miss| Keys
InternalCache --> RateLimit
+ Router --> TPM_RPM
+ Router --> Cooldowns
LLMCache --> CacheClass
CacheClass --> LLMResponseCache
CostCallback --> DBWriter
@@ -191,9 +197,10 @@ graph LR
| Component | Purpose | Key Files/Classes |
|-----------|---------|-------------------|
-| **Redis** | Rate limiting, API key caching, LLM response caching, spend queuing | `caching/redis_cache.py` (`RedisCache`), `caching/dual_cache.py` (`DualCache`) |
+| **Redis** | Rate limiting, API key caching, TPM/RPM tracking, cooldowns, LLM response caching, spend queuing | `caching/redis_cache.py` (`RedisCache`), `caching/dual_cache.py` (`DualCache`) |
| **PostgreSQL** | API keys, teams, users, spend logs | `proxy/utils.py` (`PrismaClient`), `proxy/schema.prisma` |
| **InternalUsageCache** | Proxy-level cache for rate limits + API keys (in-memory + Redis) | `proxy/utils.py` (`InternalUsageCache`) |
+| **Router.cache** | TPM/RPM tracking, deployment cooldowns, client caching (in-memory + Redis) | `router.py` (`Router.cache: DualCache`) |
| **LLMCachingHandler** | SDK-level LLM response/embedding caching | `caching/caching_handler.py` (`LLMCachingHandler`), `caching/caching.py` (`Cache`) |
| **DBSpendUpdateWriter** | Batches spend updates to reduce DB writes | `proxy/db/db_spend_update_writer.py` (`DBSpendUpdateWriter`) |
| **Cost Tracking** | Calculates and logs response costs | `proxy/hooks/proxy_track_cost_callback.py` (`_ProxyDBLogger`) |