mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-27 06:13:07 +00:00
57ea9f5ce4
Deployments sharing a model name now form a model group instead of overwriting each other. router_settings drive selection (simple-shuffle weighted by weight, rpm or tpm, or least-busy), num_retries on a different deployment for connect errors, 408, 429 and 5xx received before any byte reaches the client, and allowed_fails / cooldown_time cooldown with fallback to cooling deployments when no other is left. Per-deployment health is lock-free atomics. /v1/messages (Anthropic and Bedrock deployments) and /v1/chat/completions (native OpenAI passthrough and the core chat route) route through it, and the metrics label records the deployment each attempt used. Other endpoints pick one deployment per request without retries. |
||
|---|---|---|
| .. | ||
| .agents/skills | ||
| crates | ||
| docs | ||
| loadtest | ||
| .dockerignore | ||
| .gitignore | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| clippy.toml | ||