mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-28 06:23:07 +00:00
aa130f4e4c
We already support model_tpm_limit/model_rpm_limit per project, but that treats input and output tokens as one combined bucket, which doesn't match how Bedrock Mantle actually meters usage. This adds model_itpm_limit/model_otpm_limit so a project can cap input and output tokens separately, same shape as the existing fields. There's already deployment-level itpm/otpm support in router_utils/pre_call_checks/io_token_rate_limit_check.py, but that code only ever deals with one deployment at a time, whereas the proxy hook (parallel_request_limiter_v3.py) has to juggle key/team/org/project limits together on the same request. Rather than bolt a second copy of the reservation logic on, this just adds project-scoped ITPM/OTPM as two more descriptors into the same reservation path the hook already uses for TPM/RPM. If a project sets both the combined TPM limit and the new itpm/otpm limits on the same model, both get enforced, not one overriding the other, matching what the deployment-level check already does. Input and output are reserved separately up front, and if one side reserves fine but the other is over limit, we roll back the side that already went through so nothing gets left over-counted. On the way back, cached prompt tokens get excluded from the ITPM count since that's how Bedrock bills it, but that's purely for the rate limit math, cost and usage logging still see the full token count. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||