scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-15 09:12:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 311
litellm/tests/integration/spend
History
Exact
kirti22jain 73e17cba74
feat(proxy): add per-model project batch enqueued token limits (#51608) 
* feat(proxy): add per-model project batch enqueued token limits

Enqueued batch token limits only read key and team metadata, so a deployment
that gives each application its own project under a shared team cannot cap one
application's queued batch tokens for a given model. The team limit is shared
by every project in it, and JWT or custom-auth callers get a new key scope with
every token.

Read model_batch_enqueued_token_limit from project metadata, keyed by model like
model_tpm_limit. Each limited model group the batch runs is reserved against
only its own rows' tokens, rows that name no model count toward the routing
model, and the reservation records each scope's charge so the terminal-state
refund releases exactly that. Only proxy admins may set it, on project create
and update as well as on keys and teams, so a team admin cannot lift their own
project's cap.

* fix(proxy): charge the routing model and keep daily caps for project batch limits

Rows can name any model while the batch runs on the routing deployment, for
example LiteLLM-executed batches replace every row's model with the run model,
so charging only row models let a batch skip a capped deployment's per-model
limit. Charge the routing model the whole batch as well as each other limited
row model its own rows' tokens.

An enqueued-token reservation replaced every batch counter, so a project limit
also skipped the key, team and end-user daily token caps. Keep checking those
daily caps after reserving, and refund the reservation if one rejects the batch.

* fix(proxy): keep per-minute checks for batches a project allowance does not cover

A per-model project allowance replaced every per-minute batch check as soon as
any row's model had one. A batch routed to an unlimited model could add a
single tiny row naming a limited model and skip the routing model's RPM/TPM and
project ITPM/OTPM quotas, although every row runs on the routing model. Waive
those checks only when the reservation covers the whole batch, through a key or
team allowance or a project allowance on the routing model; otherwise reserve
the covered models and run the normal checks too.

* test(proxy): cover the project per-model batch cap through custom auth on a live proxy

* fix(proxy): keep key and team batch allowances authoritative and harden project limits

Key and team enqueued allowances again replace the other batch checks. A project
allowance on the routing model keeps only the daily caps. Team admins can only be
blocked from the project key, invalid project maps answer 400, the edit modal keeps
stored limits, aliases share their tightest limit, file-bound batches charge only
the routing model, deployment ids resolve to their model group, and reservation
records keep project scopes apart so older pods still refund key and team scopes.

* fix(proxy): charge row models for model-bound batch uploads

Only a managed-file upload rewrites rows to the deployment model. A model-bound
upload keeps each row's model, so its rows still count toward the project caps of
the models they name.

* fix(proxy): skip per-minute batch limits only when the project caps every model in the batch

A project cap on the routing model alone used to switch off the project's
per-minute input and output limits for the other models a batch file names.
The skip now needs a project cap on every one of those models.

-latest names are charged under the cap of the model they point to, the
limit map only takes JSON integers, and a /project/update that sends only
top-level fields keeps the stored metadata instead of replacing it.

* fix(proxy): lift per-minute limits off managed batches under a project cap and edit the cap in the Admin UI

* fix(ui): keep the project batch cap read-only for team admins and require a whole number above 0

* fix(ui): narrow project metadata directly when keeping the stored batch cap

* fix(proxy): charge dated snapshot rows and header or query routed batches to project caps

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 14:36:53 -05:00
..
fixtures feat(spend): report guardrail spend separately in daily activity (#51010) 2026-11-10 13:02:26 -08:00
golden feat(spend): report guardrail spend separately in daily activity (#51010) 2026-11-10 13:02:26 -08:00
__init__.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
_background_responses.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
_daily_activity_fixtures.py feat(usage): add top users by spend and user activity views (#45310) 2026-10-10 12:21:43 -07:00
_partial_stream.py fix(proxy): raise live budget counters for failure spend rows (#47741) 2026-10-30 13:05:12 -07:00
test_agent_spend.py fix(a2a)!: bill completion-bridge agents at their own prices on /a2a send and stream (#49150) 2026-10-30 19:43:45 -07:00
test_autorouter_session_rollup.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_background_interaction_settlement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_background_response_billed_once.py fix(responses): resume background response streams with stream and starting_after (#48819) 2026-10-29 02:31:22 -07:00
test_background_response_billing_proxies.py fix(proxy): report requested model in streamed Responses response.model (#46662) 2026-10-17 19:52:10 +00:00
test_background_response_poll_retirement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_baggage_tag_mappings.py test(spend): align queued baggage mapping fixtures (#51847) 2026-11-14 19:13:36 +00:00
test_baseline_accounting.py test(integration): fix the baseline accounting and series cap flakes (#51096) 2026-11-11 12:36:02 -08:00
test_batch_completion_accounting.py fix(logging): deliver completed batch success callbacks consistently (#47192) 2026-10-18 09:19:04 -07:00
test_batch_enqueued_tokens_redis_lua.py test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996) 2026-10-01 09:23:36 -07:00
test_batch_observability.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_batch_poll_starvation.py fix(batches): stop unbillable batches from blocking file deletes and new-batch polling (#48752) 2026-10-28 12:19:36 -07:00
test_budget_limit_envelopes.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_cache_activity.py fix(proxy): scope /global/activity/cache_hits analytics to the caller's own and permitted-team logs (#46210) 2026-10-14 10:01:21 +02:00
test_cache_and_quota.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_chaos_burst_spend_once.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_chatgpt_subscription_usage.py feat(usage): show subscription-covered providers and account fees separately (#44772) 2026-10-30 03:58:36 -07:00
test_container_spend.py test(integration): cover videos_containers endpoint contracts (#44691) 2026-10-14 03:49:01 +00:00
test_customer_spend_by_team.py feat(usage): split customer spend by the key's current team (#50764) 2026-11-09 19:35:31 -08:00
test_daily_activity_aggregated_breakdowns.py fix!: retain unattributed daily activity across reports and exports (#48925) 2026-10-29 09:39:49 -07:00
test_daily_activity_key_alias_probes.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_daily_activity_key_metadata_query_timeout.py feat(ui): unify gateway and usage assistance as Aria Agent (#48082) 2026-10-23 20:24:19 +00:00
test_daily_activity_key_owner.py test(usage): assert per-user entity spend and metadata (#50251) 2026-11-05 20:16:40 +00:00
test_daily_activity_key_owner_faults.py test(usage): assert per-user entity spend and metadata (#50251) 2026-11-05 20:16:40 +00:00
test_daily_activity_key_owner_traffic.py feat(ui): unify gateway and usage assistance as Aria Agent (#48082) 2026-10-23 20:24:19 +00:00
test_daily_activity_repository.py feat(usage): add top users by spend and user activity views (#45310) 2026-10-10 12:21:43 -07:00
test_daily_activity_routes.py feat(spend): report guardrail spend separately in daily activity (#51010) 2026-11-10 13:02:26 -08:00
test_daily_rollup_retry.py feat(spend): report guardrail spend separately in daily activity (#51010) 2026-11-10 13:02:26 -08:00
test_daily_tag_spend_retention.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_disable_end_user_cost_tracking.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_disconnected_bedrock_messages_stream_billing.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_end_user_budget_enforcement.py fix(anthropic_endpoints)!: return Anthropic error envelopes on /v1/messages auth failures (#48466) 2026-11-03 01:53:29 +00:00
test_end_user_spend_without_proxy_user.py fix(anthropic_endpoints)!: return Anthropic error envelopes on /v1/messages auth failures (#48466) 2026-11-03 01:53:29 +00:00
test_expired_ui_session_failure_rows.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_fail_closed_team_member_budget.py fix(proxy): fail-closed budget enforcement returns 403 naming /team/member_add for a team user with no membership row (#50603) 2026-11-09 15:38:59 -08:00
test_failed_dispatch_tokens.py test(spend): audit #47815 (#48774) 2026-10-28 03:04:11 -07:00
test_failing_request_stored_snapshot.py test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) 2026-11-04 09:55:53 -08:00
test_failure_spend_live_counters_chaos.py fix(proxy): raise live budget counters for failure spend rows (#47741) 2026-10-30 13:05:12 -07:00
test_filtered_ledger.py test: add extension and browser integration contracts 2026-09-16 13:15:53 -07:00
test_gemini_tts_speech_billing.py test(integration): repair seven stale integration tests on main (#47600) 2026-10-20 11:21:04 -07:00
test_global_proxy_spend_counter.py test(proxy): audit #46997 (#48918) 2026-10-28 18:05:03 -07:00
test_global_spend_report.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_global_spend_reset.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_google_genai_native_spend.py test(integration): cover google_genai endpoint contracts (#44680) 2026-10-14 04:01:35 +00:00
test_image_generation_key_spend.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_key_budget_alerts.py feat(ui): set per-threshold budget alert recipients on virtual keys (#51061) 2026-11-12 15:39:44 -08:00
test_key_budget_lockout.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_key_metadata_recovery_dropped_connection.py fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560) 2026-10-05 22:36:25 +00:00
test_key_metadata_recovery_probe_bounds.py fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948) 2026-10-01 14:12:22 -07:00
test_legacy_spend_logs_row_cap.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_messages_stream_usage_cost.py feat(proxy): per-request header opts a stream into usage.cost without the global include_cost_in_streaming_usage flag (#50574) 2026-11-07 16:57:20 -08:00
test_model_router_selected_model.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_non_finite_cost_spend_writes.py fix(spend): keep a NaN or infinite cost from wedging daily spend writes (#48829) 2026-10-30 08:27:35 -07:00
test_normalized_error_long_message.py test(integration): give each kill-burst its own prompts so cached control replies cannot stand in for late traffic (#50569) 2026-11-07 23:08:23 +00:00
test_openrouter_upstream_provider.py feat(openrouter): record upstream provider in logging payload and spend logs (#49287) 2026-10-30 12:51:48 -07:00
test_org_budget_cli_session_token.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_partial_stream_live_budget.py test(spend): assert Anthropic envelope for Messages budget refusals (#49800) 2026-11-03 06:10:08 +00:00
test_passthrough_budget_reservation.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_failure.py fix(proxy): write SpendLogs failure rows for config-defined pass-through endpoints (#41831) 2026-11-09 17:05:10 -08:00
test_passthrough_request_tags.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_stream_pricing_paths_spend.py fix(proxy): bill an OpenAI pass-through background response once across repeated reads (#50841) 2026-11-10 07:33:25 -08:00
test_passthrough_upstream_model.py feat(passthrough): let upstreams report the model name via x-litellm-upstream-model (#48275) 2026-10-27 18:28:54 -07:00
test_project_batch_enqueued_token_limit.py feat(proxy): add per-model project batch enqueued token limits (#51608) 2026-11-14 14:36:53 -05:00
test_prompt_caching_requests_pagination.py test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996) 2026-10-01 09:23:36 -07:00
test_ptu_share_attribution.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_redis_ttl_preserving_token_increment.py test(spend): exercise TTL-preserving increments through rate limit store (#50418) 2026-11-06 17:53:25 +00:00
test_requester_ip_address_logging.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_reset_budget_leader_election.py fix: make scheduled key, user, and team spend resets idempotent (#50298) 2026-11-09 22:21:43 -08:00
test_responses_bridge_spend_once.py fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API (#49710) 2026-11-03 10:02:54 +00:00
test_responses_cache_write_itemization.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_responses_stream_spend_row_id.py fix(spend): keep free response reads and cancels off the creation spend row (#51110) 2026-11-11 07:02:00 +00:00
test_responses_websocket_turn_rows.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_roi_branch_spend.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_router_cooldown_failure_row.py test(integration): read router cooldown failure rows from the proxy that served them (#47457) 2026-10-19 18:39:16 -07:00
test_scoped_spend_reports.py test(integration): cover budgets_spend endpoint contracts (#44736) 2026-10-14 06:58:22 +00:00
test_service_tier_stream_billing.py fix(streaming): keep the served service_tier on streamed chunks and spend rows (#42870) 2026-09-29 12:54:17 -07:00
test_session_grouped_logs_pages.py fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id (#50789) 2026-11-10 17:03:21 +00:00
test_session_total_spend.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_shutdown_flush.py test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
test_spend_attribution_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_spend_calculate.py fix(pricing): drop the unpublished cached rate from the Gemini Live preview entries (#42651) 2026-09-22 20:35:34 -07:00
test_spend_capture_rate_captured_spend.py test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996) 2026-10-01 09:23:36 -07:00
test_spend_log_caller_session.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_spend_log_latest_turn.py feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests (#49544) 2026-11-05 19:41:15 -08:00
test_spend_log_read_scope.py fix(proxy): open the Logs drawer by request_id without scanning for litellm_call_id (#50789) 2026-11-10 17:03:21 +00:00
test_spend_log_tool_payload_content.py fix(spend-logs): make token redaction opt-in (#51729) 2026-11-13 14:41:12 -08:00
test_spend_log_write_batching.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_spend_logs_metadata_fields.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_spend_rollup_accuracy.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_stream_alias_billing.py fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model (#44341) 2026-10-03 06:44:52 +00:00
test_streamed_usage_estimate.py test(spend): audit #47353 (#48785) 2026-10-28 03:50:50 -07:00
test_tag_budget_enforcement.py fix(budgets): invalidate customer hierarchy caches through stored team IDs (#51654) 2026-11-13 18:43:38 +00:00
test_tag_spend_reset.py feat(tags): reset a tag's spend through /tag/update (#50323) 2026-11-06 10:41:22 -08:00
test_team_budget_enforcement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_team_daily_activity_aggregated.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_team_member_budget_alerts.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_member_spend.py test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
test_team_member_spend_flush.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_member_total_spend_reconciliation.py test(spend): synchronize concurrent member resets before validation completes (#51655) 2026-11-13 18:23:36 +00:00
test_text_completion_stream_usage_cost.py fix(proxy)!: bill streamed /v1/completions without stream_options like chat (#45366) 2026-10-29 17:18:51 -07:00
test_unmanaged_vertex_batch_deployment_routing_wire.py fix(batches): cost an unmanaged gs:// batch through the deployment that created it (#49301) 2026-10-30 15:25:47 -07:00
test_user_budget_on_team_keys.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_vertex_gemini_image_token_spend.py fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates (#51017) 2026-11-11 02:50:55 +00:00
test_vertex_image_regional_uplift.py test(spend): expect token billing for Vertex image edits with usage (#51109) 2026-11-11 03:32:19 +00:00
test_vertex_passthrough_regional_uplift_spend.py fix(tests): read owned proxy rows from the owned proxy's database (#47333) 2026-10-20 10:07:06 -07:00
test_vertex_regional_uplift_spend.py test(spend): expect token billing for Vertex image edits (#51148) 2026-11-11 06:27:58 +00:00