mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-30 06:43:27 +00:00
16dacb02b7
Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3 (or model_capacity) in litellm_settings.callbacks. It caps each model group at the sum of its deployments' rpm and tpm across every replica, and with priority_reservation holds each priority's pool to its share once the group's saturation reaches the threshold, counting it below. It uses the proxy's model_saturation_check and priority_model keys and scripts, its refusal messages and headers, and adds a successful response's tokens to the group and the pool admission chose. The rate limit counters gain a general check-and-add over request and token pairs, and the throttling refusal shape is shared. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| bedrock_messages-flow.svg | ||
| budgets.svg | ||
| hooks-pipeline.svg | ||
| loadtest-rig.svg | ||
| model-capacity.svg | ||
| openai_chat-flow.svg | ||
| ops-drain.svg | ||
| ops-metrics.svg | ||
| rate-limits.svg | ||
| request-hooks.svg | ||
| routing-cooldown.svg | ||
| routing-request-flow.svg | ||
| shared_state-redis.svg | ||
| spend-writer.svg | ||
| virtual_keys-auth-flow.svg | ||
