mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
16dacb02b7
Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3 (or model_capacity) in litellm_settings.callbacks. It caps each model group at the sum of its deployments' rpm and tpm across every replica, and with priority_reservation holds each priority's pool to its share once the group's saturation reaches the threshold, counting it below. It uses the proxy's model_saturation_check and priority_model keys and scripts, its refusal messages and headers, and adds a successful response's tokens to the group and the pool admission chose. The rate limit counters gain a general check-and-add over request and token pairs, and the throttling refusal shape is shared. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| .agents/skills | ||
| crates | ||
| docs | ||
| loadtest | ||
| .dockerignore | ||
| .gitignore | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| clippy.toml | ||