mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
5b93ba0ada
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(router): add separate ITPM/OTPM deployment rate limits Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): keep ITPM/OTPM diff minimal in router.py Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): make ITPM/OTPM limits separate and atomic Address Greptile review on separate ITPM/OTPM deployment rate limits. - OTPM is now reserved atomically pre-call with rollback, matching the ITPM path, so concurrent requests can no longer overshoot the configured output limit before reconciliation - ITPM counts input tokens only; it no longer accumulates completion tokens, so the input-token limit and x-ratelimit-limit-input-tokens header describe input usage as their names imply - _read_reservation_from_kwargs only falls back to litellm_params.metadata when the top-level metadata channel is absent, so production requests carrying a litellm_params.metadata dict still reconcile and refund their reservation Adds regression tests for OTPM atomicity under concurrency, input-only ITPM enforcement, and reservation lookup when litellm_params.metadata is present. * fix(router): subtract input tokens only from remaining-input-tokens header The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total tokens (input + output) instead of input tokens only, so clients saw remaining input quota understated by the completion token count on every response. Now consistent with the input-only ITPM counter. * fix(router): make itpm/otpm vs tpm/rpm precedence explicit When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path takes over and the tpm/rpm limits are not enforced. Log a warning the first time such a conflicting deployment is seen so the supersession is not silent, and document the mutual exclusivity. Post-call reconciliation now only trues up a counter that was actually reserved against, so the itpm/otpm keys are no longer incremented for deployments that never configured that limit. * fix(router): track actual io-token usage on the reservation-minute key Post-call reconciliation now keys off the exact cache key stashed at pre-call time rather than one recomputed from the response-time minute. This fixes two issues: a request whose pre-call estimate was 0 now still writes its actual billable input to the ITPM counter (previously it was skipped, leaving the limit unenforceable for that request), and a call that finishes in a later minute reconciles against the minute it reserved against instead of pushing a negative delta into the next minute. Counters are only touched when their limit is configured. * fix(router): run io-token reconciliation before the model_id guard async_log_success_event gated IO reconciliation behind the model_id guard that only the TPM tracking path needs. Since reconciliation works entirely from the cache keys stashed in kwargs, a success event whose standard_logging_object lacks model_id would skip reconciliation and leave the reservation on the counter until the TTL expired, wasting quota. Route the IO path first. * fix(router): don't replay in-flight delta for itpm/otpm headers For ITPM/OTPM model groups the counter is incremented at reservation time (pre-call), so the remaining values returned by get_remaining_model_group_usage already account for the current request. Replaying the in-flight delta on top double-counted it and understated x-ratelimit-remaining-input/output-tokens by up to max_tokens on every response. Skip the delta for io-token groups; the legacy TPM/RPM replay path is unchanged. * fix(router): clear io-token reservation after reconcile/refund async_io_token_refund_failure and async_io_token_reconcile_success now clear the stashed reservation keys from the request metadata once done. Otherwise, on a model group mixing IO-limited and non-IO deployments, a failed IO call that retries on a non-IO fallback left the stale sentinel in the shared request metadata; the fallback's success handler would divert into IO reconciliation against the already-refunded key, driving the ITPM counter negative and skipping the non-IO deployment's TPM tracking. * fix(router): tidy reservation channel lookup and header guard Consolidate the reservation channel lookup into a single ordered helper shared by read and clear, so top-level metadata always wins over litellm_params metadata without the tangled per-iteration fallback. Also stop gating the router rate-limit header block on the presence of x-ratelimit-remaining-input/output-tokens. That block only emits those headers for ITPM/OTPM groups; for a non-IO group backed by a provider that natively returns input/output token headers, the extra conditions suppressed the router's own remaining-tokens/requests headers. * fix(router): strip client-supplied io-token reservation keys The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key, and the otpm equivalents) are server-only, but metadata is caller-controlled on proxy requests. An authenticated caller could forge these fields with an arbitrary cache key so the post-call reconcile/refund path would decrement any deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs, which runs before the router stashes its own reservation, so only a genuine server-side reservation is ever read post-call. * fix(router): track TPM routing load for io-limited deployments deployment_callback_on_success early-returned for any deployment with itpm/otpm set, so its total-token usage never landed in the router's TPM routing counter. TPM-aware routing strategies then saw 0 load for IO deployments and over-routed to them in mixed model groups. Only skip tracking when neither tpm/rpm nor itpm/otpm are configured; itpm/otpm enforcement still runs separately in ModelRateLimitingCheck, so the routing counter and the enforcement counters stay independent. * fix(router): expose standard tpm/rpm headers for io-limited groups get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests; clients and prometheus gauges reading those saw no data. Build both header sets instead of returning early. Also simplify the in-flight header replay: only the tpm/rpm counters are incremented post-response, so the delta now adjusts just those. The itpm/otpm counters are incremented at reservation time (pre-call), so the input/output token headers already reflect the request and are left untouched - which removes the need for the separate io-group special case. * fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance Two follow-ups from review. The pre-call OTPM reservation only rolled back the ITPM reservation on a RateLimitError, so a transient cache error while reserving OTPM left the ITPM counter inflated until the TTL expired; catch any exception, release the ITPM reservation, then re-raise. Replace the module-level lru_cache warn-once (caching a logging side effect, which never re-warns in a long-lived process) with an instance-scoped set of already-warned deployment ids on ModelRateLimitingCheck. * fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup Clear the reservation in a finally block so a mid-reconciliation cache error still removes the stash and a duplicate success event can't re-process it. Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a deployment with no id no longer collapses every id-less deployment onto the str(None) key (which would suppress all but the first warning). * fix(router): skip io reservation when deployment can't be keyed _get_cache_keys returned a shared 'global_router:None:None:...' key when a deployment was missing model_info.id or litellm_params.model, so misconfigured deployments could share one rate-limit bucket. Return None in that case and skip io reservation for the request. * fix(router): honor explicit max_tokens=0 in io reservation _resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit max_tokens=0 fell through to the model default. Only fall back to max_completion_tokens when max_tokens is absent. * fix(ci): satisfy lint budget, router coverage, and dashboard schema sync - Modernize the new itpm/otpm module's type hints to PEP 585 lowercase generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006 violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match. - Replace three try/except Exception blocks that must stay broad by design (token_counter and litellm.get_model_info raise untyped exceptions, and an io-token refund failure must never break the logging pipeline) with contextlib.suppress(Exception), matching the codebase's existing resolution for this exact BLE001 pattern. - Add direct unit tests for get_model_group_io_token_usage (multi-deployment aggregation and the empty-model-list case) in test_router_helper_utils.py, satisfying the router function-coverage check. - Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types. * fix: enforce io token rate limits consistently * fix: honor zero max tokens in otpm reservation * fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10 floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span alias. The previously committed ruff-strict-budget.json ratcheted UP006 down from a stale base; litellm_internal_staging has since tightened that same ceiling further on its own. Reset the file to the current base's committed values and re-ratchet from there so the budget only ever moves down relative to the actual merge-base, never against a stale snapshot. * fix(router): attach ITPM/OTPM headers on dict responses and harden reservation Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit headers through /v1/messages dict responses via _hidden_params. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so set_response_headers can attach rate-limit headers to streaming Anthropic messages responses that lack a _hidden_params slot. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format add_retry_fallback_headers.py Fix CI ruff format check failure on get_hidden_params_dict call site. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): extract set_response_headers helpers to fix C901 budget Move header-attachment logic into add_retry_fallback_headers helpers so set_response_headers stays under the strict complexity ceiling. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: keep IO token reservation when response usage is missing Missing usage was reconciled as zero and fully refunded the pre-call reservation, allowing limit bypass on repeated successful calls. Only adjust counters when usage is resolved from the response or standard logging fields; otherwise keep the reservation until TTL expires. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: enforce RPM/TPM alongside IO-token limits on mixed deployments Deployments with both itpm/otpm and tpm/rpm previously returned after the IO reservation and skipped RPM/TPM checks. Run both paths and refund the IO reservation only when RPM/TPM rejects after a successful reservation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: track TPM usage on success for mixed IO+TPM deployments The early return after IO-token reconciliation in log_success_event and async_log_success_event skipped the TPM counter increment, so the tpm_key the pre-call check reads was never written and tpm_limit was never actually enforced on deployments that also configure itpm/otpm. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: treat total-only usage as unresolved in IO-token reconcile usage/standard_logging_object entries carrying only total_tokens (no prompt/completion or input/output breakdown) were treated as resolved usage, resolving to (0, 0) and refunding the full reservation. Both _usage_is_present and the standard_logging_object fallback now require an actual input/output breakdown before reconciling, keeping the reservation otherwise. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: reserve minimal token when input/output estimation fails _reservation_value(0, limit) reserved the entire limit whenever token estimation failed (empty/unsupported input, tokenizer error), letting one such request claim the whole bucket and 429 every concurrent request to the deployment until it completed. Reserve 1 token instead so estimation failures no longer serialize traffic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: refund IO reservation synchronously before retry deployment pick On retry, set_io_token_rate_limit_request_kwargs clears reservation sentinels from the shared kwargs dict before a background failure handler can refund them, stranding the counter until TTL. Refund and clear any stale reservation in _update_kwargs_with_deployment before stripping sentinels for the next attempt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling Pass the deployment litellm_params.model to token_counter so it uses the model's native tokenizer instead of the generic fallback, narrowing the reservation over/under-estimate window between pre-call and post-call reconcile. Add a ponytail: comment to refund_stale_reservation_before_retry explaining the known ceiling: the synchronous DualCache.increment_cache issues a blocking Redis INCR when a Redis backend is configured. This only fires on streaming mid-stream retries (non-streaming failures await their failure handler before the retry picks a new deployment, leaving no sentinels to refund). Upgrade path: make _update_kwargs_with_deployment async. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
368 lines
4.1 KiB
JSON
368 lines
4.1 KiB
JSON
{
|
|
"ANN001": {
|
|
"limit": 3152
|
|
},
|
|
"ANN002": {
|
|
"limit": 69
|
|
},
|
|
"ANN003": {
|
|
"limit": 835
|
|
},
|
|
"ANN201": {
|
|
"limit": 2138
|
|
},
|
|
"ANN202": {
|
|
"limit": 944
|
|
},
|
|
"ANN204": {
|
|
"limit": 724
|
|
},
|
|
"ANN205": {
|
|
"limit": 127
|
|
},
|
|
"ANN206": {
|
|
"limit": 130
|
|
},
|
|
"ANN401": {
|
|
"limit": 2075
|
|
},
|
|
"ASYNC230": {
|
|
"limit": 14
|
|
},
|
|
"B004": {
|
|
"limit": 4
|
|
},
|
|
"B006": {
|
|
"limit": 190
|
|
},
|
|
"B008": {
|
|
"limit": 505
|
|
},
|
|
"B009": {
|
|
"limit": 84
|
|
},
|
|
"B010": {
|
|
"limit": 197
|
|
},
|
|
"B018": {
|
|
"limit": 5
|
|
},
|
|
"B019": {
|
|
"limit": 4
|
|
},
|
|
"B021": {
|
|
"limit": 4
|
|
},
|
|
"B026": {
|
|
"limit": 6
|
|
},
|
|
"B033": {
|
|
"limit": 4
|
|
},
|
|
"BLE001": {
|
|
"limit": 2903
|
|
},
|
|
"C401": {
|
|
"limit": 11
|
|
},
|
|
"C404": {
|
|
"limit": 4
|
|
},
|
|
"C405": {
|
|
"limit": 23
|
|
},
|
|
"C408": {
|
|
"limit": 14
|
|
},
|
|
"C414": {
|
|
"limit": 7
|
|
},
|
|
"C419": {
|
|
"limit": 4
|
|
},
|
|
"C901": {
|
|
"limit": 316
|
|
},
|
|
"D419": {
|
|
"limit": 9
|
|
},
|
|
"DTZ001": {
|
|
"limit": 5
|
|
},
|
|
"DTZ003": {
|
|
"limit": 33
|
|
},
|
|
"DTZ005": {
|
|
"limit": 244
|
|
},
|
|
"DTZ006": {
|
|
"limit": 13
|
|
},
|
|
"DTZ007": {
|
|
"limit": 23
|
|
},
|
|
"DTZ011": {
|
|
"limit": 6
|
|
},
|
|
"EXE001": {
|
|
"limit": 7
|
|
},
|
|
"EXE002": {
|
|
"limit": 6
|
|
},
|
|
"F401": {
|
|
"limit": 23
|
|
},
|
|
"FURB136": {
|
|
"limit": 4
|
|
},
|
|
"FURB168": {
|
|
"limit": 4
|
|
},
|
|
"FURB188": {
|
|
"limit": 52
|
|
},
|
|
"I001": {
|
|
"limit": 273
|
|
},
|
|
"LOG015": {
|
|
"limit": 8
|
|
},
|
|
"N999": {
|
|
"limit": 4
|
|
},
|
|
"PERF102": {
|
|
"limit": 30
|
|
},
|
|
"PERF401": {
|
|
"limit": 146
|
|
},
|
|
"PERF402": {
|
|
"limit": 9
|
|
},
|
|
"PERF403": {
|
|
"limit": 74
|
|
},
|
|
"PIE790": {
|
|
"limit": 278
|
|
},
|
|
"PIE800": {
|
|
"limit": 4
|
|
},
|
|
"PIE804": {
|
|
"limit": 24
|
|
},
|
|
"PIE810": {
|
|
"limit": 44
|
|
},
|
|
"PLC0206": {
|
|
"limit": 31
|
|
},
|
|
"PLC0208": {
|
|
"limit": 4
|
|
},
|
|
"PLC0414": {
|
|
"limit": 38
|
|
},
|
|
"PLR0124": {
|
|
"limit": 4
|
|
},
|
|
"PLR0206": {
|
|
"limit": 4
|
|
},
|
|
"PLR0402": {
|
|
"limit": 9
|
|
},
|
|
"PLR1704": {
|
|
"limit": 6
|
|
},
|
|
"PLR1711": {
|
|
"limit": 34
|
|
},
|
|
"PLR1714": {
|
|
"limit": 265
|
|
},
|
|
"PLR1730": {
|
|
"limit": 10
|
|
},
|
|
"PLR2044": {
|
|
"limit": 4
|
|
},
|
|
"PLW0127": {
|
|
"limit": 44
|
|
},
|
|
"PLW0133": {
|
|
"limit": 4
|
|
},
|
|
"PLW0602": {
|
|
"limit": 230
|
|
},
|
|
"PLW0603": {
|
|
"limit": 193
|
|
},
|
|
"PLW1508": {
|
|
"limit": 198
|
|
},
|
|
"PLW1510": {
|
|
"limit": 5
|
|
},
|
|
"PYI030": {
|
|
"limit": 5
|
|
},
|
|
"PYI036": {
|
|
"limit": 5
|
|
},
|
|
"PYI041": {
|
|
"limit": 12
|
|
},
|
|
"PYI064": {
|
|
"limit": 5
|
|
},
|
|
"RET501": {
|
|
"limit": 38
|
|
},
|
|
"RET504": {
|
|
"limit": 721
|
|
},
|
|
"RUF010": {
|
|
"limit": 874
|
|
},
|
|
"RUF012": {
|
|
"limit": 168
|
|
},
|
|
"RUF015": {
|
|
"limit": 11
|
|
},
|
|
"RUF019": {
|
|
"limit": 41
|
|
},
|
|
"RUF022": {
|
|
"limit": 85
|
|
},
|
|
"RUF023": {
|
|
"limit": 5
|
|
},
|
|
"RUF046": {
|
|
"limit": 8
|
|
},
|
|
"RUF051": {
|
|
"limit": 6
|
|
},
|
|
"RUF059": {
|
|
"limit": 73
|
|
},
|
|
"RUF100": {
|
|
"limit": 480
|
|
},
|
|
"S110": {
|
|
"limit": 236
|
|
},
|
|
"S112": {
|
|
"limit": 24
|
|
},
|
|
"SIM101": {
|
|
"limit": 63
|
|
},
|
|
"SIM102": {
|
|
"limit": 324
|
|
},
|
|
"SIM103": {
|
|
"limit": 129
|
|
},
|
|
"SIM113": {
|
|
"limit": 6
|
|
},
|
|
"SIM114": {
|
|
"limit": 113
|
|
},
|
|
"SIM115": {
|
|
"limit": 5
|
|
},
|
|
"SIM117": {
|
|
"limit": 10
|
|
},
|
|
"SIM118": {
|
|
"limit": 114
|
|
},
|
|
"SIM201": {
|
|
"limit": 4
|
|
},
|
|
"SIM210": {
|
|
"limit": 12
|
|
},
|
|
"SIM211": {
|
|
"limit": 4
|
|
},
|
|
"SIM222": {
|
|
"limit": 4
|
|
},
|
|
"SIM401": {
|
|
"limit": 12
|
|
},
|
|
"TC004": {
|
|
"limit": 8
|
|
},
|
|
"TC005": {
|
|
"limit": 9
|
|
},
|
|
"TID251": {
|
|
"limit": 2710
|
|
},
|
|
"TRY002": {
|
|
"limit": 548
|
|
},
|
|
"TRY004": {
|
|
"limit": 98
|
|
},
|
|
"TRY201": {
|
|
"limit": 424
|
|
},
|
|
"TRY203": {
|
|
"limit": 123
|
|
},
|
|
"TRY300": {
|
|
"limit": 883
|
|
},
|
|
"UP006": {
|
|
"limit": 12869
|
|
},
|
|
"UP007": {
|
|
"limit": 2570
|
|
},
|
|
"UP008": {
|
|
"limit": 5
|
|
},
|
|
"UP012": {
|
|
"limit": 7
|
|
},
|
|
"UP018": {
|
|
"limit": 21
|
|
},
|
|
"UP024": {
|
|
"limit": 15
|
|
},
|
|
"UP028": {
|
|
"limit": 5
|
|
},
|
|
"UP031": {
|
|
"limit": 5
|
|
},
|
|
"UP032": {
|
|
"limit": 629
|
|
},
|
|
"UP034": {
|
|
"limit": 4
|
|
},
|
|
"UP035": {
|
|
"limit": 2295
|
|
},
|
|
"UP036": {
|
|
"limit": 4
|
|
},
|
|
"UP037": {
|
|
"limit": 105
|
|
},
|
|
"UP045": {
|
|
"limit": 18517
|
|
}
|
|
}
|