mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-11 08:38:24 +00:00
bdfe7ae195
* feat(gateway): connect to Redis from config.yaml * feat(config): own Redis settings resolution * fix(config): preserve Redis input precedence and presence * fix(config): box Redis cache settings * fix(gateway): preserve borrowed redis namespace --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
12 lines
1.8 KiB
Markdown
12 lines
1.8 KiB
Markdown
- Keep this crate a thin composition layer: mount endpoint routers and serve the supplied listener
|
|
- Server lifecycle and shared inbound middleware belong here, including client authentication, rate limiting, and request logging
|
|
- Endpoint paths, request handling, model resolution, and response encoding belong to the mounted crates. Provider execution belongs to the `inference-<fmt>` crates and `llms`
|
|
- Inject shared state and infrastructure. Avoid global runtimes, duplicate client pools, and abstractions for hypothetical endpoint groups
|
|
- Test mounting and server lifecycle through public HTTP behavior. Test endpoint semantics in the owning crate
|
|
|
|
## Redis
|
|
- `build_redis` builds the Redis clients once at boot (`src/redis.rs`). Components get a clone as a field, never a per-request extractor or a pool. redis-rs multiplexes one connection per node, so there is no connection pool and pool settings like `max_connections` are accepted and ignored
|
|
- Which Redis backs which role mirrors the proxy: coordination is `general_settings.coordination_redis`, else the `cache_params` Redis when `cache: true` and `type: redis`, else `REDIS_*` env. The transaction buffer reuses coordination, else env when `use_redis_transaction_buffer` is on. The response cache gets its own client so large values never queue in front of coordination commands
|
|
- `litellm-config` follows `litellm/_redis.py`'s merge of config blocks and `REDIS_*` env. A redis-py setting we can't honor (sentinel, IAM, custom TLS files, ...) fails boot with its name, never silently dropped
|
|
- Boot never waits for a configured Redis (clients connect lazily). Only the env-only coordination fallback is pinged, for 2s, and treated as absent when unreachable, as in Python
|
|
- `/health/readiness` never touches Redis, matching the proxy
|