mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
73e17cba74
* feat(proxy): add per-model project batch enqueued token limits Enqueued batch token limits only read key and team metadata, so a deployment that gives each application its own project under a shared team cannot cap one application's queued batch tokens for a given model. The team limit is shared by every project in it, and JWT or custom-auth callers get a new key scope with every token. Read model_batch_enqueued_token_limit from project metadata, keyed by model like model_tpm_limit. Each limited model group the batch runs is reserved against only its own rows' tokens, rows that name no model count toward the routing model, and the reservation records each scope's charge so the terminal-state refund releases exactly that. Only proxy admins may set it, on project create and update as well as on keys and teams, so a team admin cannot lift their own project's cap. * fix(proxy): charge the routing model and keep daily caps for project batch limits Rows can name any model while the batch runs on the routing deployment, for example LiteLLM-executed batches replace every row's model with the run model, so charging only row models let a batch skip a capped deployment's per-model limit. Charge the routing model the whole batch as well as each other limited row model its own rows' tokens. An enqueued-token reservation replaced every batch counter, so a project limit also skipped the key, team and end-user daily token caps. Keep checking those daily caps after reserving, and refund the reservation if one rejects the batch. * fix(proxy): keep per-minute checks for batches a project allowance does not cover A per-model project allowance replaced every per-minute batch check as soon as any row's model had one. A batch routed to an unlimited model could add a single tiny row naming a limited model and skip the routing model's RPM/TPM and project ITPM/OTPM quotas, although every row runs on the routing model. Waive those checks only when the reservation covers the whole batch, through a key or team allowance or a project allowance on the routing model; otherwise reserve the covered models and run the normal checks too. * test(proxy): cover the project per-model batch cap through custom auth on a live proxy * fix(proxy): keep key and team batch allowances authoritative and harden project limits Key and team enqueued allowances again replace the other batch checks. A project allowance on the routing model keeps only the daily caps. Team admins can only be blocked from the project key, invalid project maps answer 400, the edit modal keeps stored limits, aliases share their tightest limit, file-bound batches charge only the routing model, deployment ids resolve to their model group, and reservation records keep project scopes apart so older pods still refund key and team scopes. * fix(proxy): charge row models for model-bound batch uploads Only a managed-file upload rewrites rows to the deployment model. A model-bound upload keeps each row's model, so its rows still count toward the project caps of the models they name. * fix(proxy): skip per-minute batch limits only when the project caps every model in the batch A project cap on the routing model alone used to switch off the project's per-minute input and output limits for the other models a batch file names. The skip now needs a project cap on every one of those models. -latest names are charged under the cap of the model they point to, the limit map only takes JSON integers, and a /project/update that sends only top-level fields keeps the stored metadata instead of replacing it. * fix(proxy): lift per-minute limits off managed batches under a project cap and edit the cap in the Admin UI * fix(ui): keep the project batch cap read-only for team admins and require a whole number above 0 * fix(ui): narrow project metadata directly when keeping the stored batch cap * fix(proxy): charge dated snapshot rows and header or query routed batches to project caps --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| litellm_skills | ||
| __init__.py | ||
| test_async_post_call_streaming_iterator_hook.py | ||
| test_autorouter_baseline_cache.py | ||
| test_banned_keyword_list.py | ||
| test_batch_enqueued_tokens.py | ||
| test_batch_file_validation.py | ||
| test_batch_rate_limiter.py | ||
| test_dynamic_rate_limiter.py | ||
| test_dynamic_rate_limiter_v3.py | ||
| test_image_generation_guardrails.py | ||
| test_key_management_event_hooks.py | ||
| test_max_budget_per_session_limiter.py | ||
| test_max_iterations_limiter.py | ||
| test_message_thread_ownership.py | ||
| test_model_budget_shared_windows.py | ||
| test_model_max_budget_limiter.py | ||
| test_parallel_request_limiter.py | ||
| test_parallel_request_limiter_v3.py | ||
| test_post_call_failure_hook_integration.py | ||
| test_post_call_response_headers_hook.py | ||
| test_post_call_streaming_hook_integration.py | ||
| test_post_call_success_hook_integration.py | ||
| test_priority_class_share.py | ||
| test_prompt_cache_observer.py | ||
| test_prompt_injection_detection.py | ||
| test_proxy_hooks_init.py | ||
| test_proxy_rate_limit_provider_field.py | ||
| test_proxy_track_cost_callback.py | ||
| test_rate_limiter_toctou.py | ||
| test_search_free_tier.py | ||
| test_send_invite_email.py | ||
| test_sensitive_data_routing.py | ||
| test_tpm_concurrent.py | ||
| test_unit_test_max_model_budget_limiter.py | ||
| test_user_management_event_hooks.py | ||