litellm/tests/integration/management
Ankit Jha 5f381ad472
fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)
* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:03:19 +00:00
..
__init__.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
_batch_file_caps.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
cost_center_service.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
team_metadata_validation_proxy_config.yaml test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_access_group_team_sync.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_added_backup_deployment_serves_peer_fallbacks.py fix(proxy): apply new model rows on every worker as soon as they are written (#49034) 2026-10-30 03:19:25 -07:00
test_agent_management.py test(integration): repair agent reads and bulk deletion setup (#49719) 2026-11-02 13:44:45 -08:00
test_auto_router_databricks_classifier_writes.py test(integration): match Databricks classifier budget telemetry (#49382) 2026-10-30 20:13:51 -07:00
test_batch_file_usage_caps.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_batch_file_usage_caps_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_batch_limit_metadata_admin_only.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_batch_output_file_listing.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_budget_list_sorting.py feat(ui): sort budgets by reset duration (#48107) 2026-10-23 16:21:53 -07:00
test_budget_model_limits.py feat(budgets): add per-model tpm and rpm limit maps to budgets and customers (#50797) 2026-11-09 22:15:18 -08:00
test_budget_reset_windows.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_budget_updates.py fix(proxy): keep carried end-user spend when a budget reset is retried (#51143) 2026-11-14 01:03:19 +00:00
test_claude_code_plugin_installation_preference.py test(skills): assert patch publication and serialize preference writes (#50252) 2026-11-05 20:16:15 +00:00
test_credential_display_name.py fix(credentials): preserve stored info when patching credentials (#49919) 2026-11-03 16:15:09 -08:00
test_credential_federation_values.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_credential_migration_endpoint.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_credential_writes_survive_reload.py fix(proxy): hold DB_RECONCILE_LOCK around credential create, update and delete (#50551) 2026-11-07 16:02:44 -08:00
test_customer_budget_reset.py fix(proxy): /customer/new budgets reset on the same schedule as other budgets (#51689) 2026-11-13 19:22:14 +00:00
test_customer_model_limits.py feat(rate-limit): name the limit that rejected a request in a rate_limited_by header on the 429 (#51228) 2026-11-11 14:39:53 -08:00
test_customer_override_expiry.py fix(proxy): drop a customer override expiry sent without an override (#51667) 2026-11-13 11:24:05 -08:00
test_customer_team_delete_budgets.py fix(proxy): delete a customer team's override budget when the team has no overall budget (#51703) 2026-11-13 13:05:19 -08:00
test_customer_unknown_fields.py feat(customers): reject unknown fields on /customer/new and /customer/update instead of dropping them (#50965) 2026-11-10 18:08:21 +00:00
test_f7_coverage_closeout.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_f7_key_coverage_push.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_fallback_management_public_names.py fix(fallback): preserve concurrent writes and serve saved configuration (#51245) 2026-11-11 15:26:23 -08:00
test_guardrail_usage_config_guardrail.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_health_routing.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_key_aliases.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_block_unblock.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_budget_limits.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_bulk_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_default_allowed_routes.py feat(key_management)!: apply default_key_generate_params.allowed_routes on key generation (#41236) 2026-10-19 18:53:37 -07:00
test_key_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_delete_revocation.py fix(proxy): evict a deleted key from every worker's auth cache (#47609) 2026-10-30 10:20:00 -07:00
test_key_generate.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_generation_permissions.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_key_health.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_info.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_info_v2.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_list.py feat(ui): sort virtual keys by expiry, activity, spend and limits (#48074) 2026-10-23 15:45:17 -07:00
test_key_regenerate.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_reset_spend.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_route_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_key_service_account_generate.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_team_change.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_updates.py fix(tests): resolve the integration support package without run.py's PYTHONPATH 2026-09-15 21:10:08 -07:00
test_listing_and_health_contracts.py fix(budgets): reset 12months and other long unit spellings on their short form's schedule (#51672) 2026-11-13 11:08:55 -08:00
test_managed_file_create_ownership.py fix(managed_files): enforce input file ownership on batch and fine-tuning create (#45855) 2026-10-12 13:58:25 -04:00
test_managed_vector_store_unified_id.py test(integration): cover vector_stores endpoint contracts (#44739) 2026-10-14 03:59:08 +00:00
test_model_credential_name_updates.py fix(proxy): validate model credential name only when it changes (#42701) 2026-09-23 07:34:32 -07:00
test_model_deployment_lifecycle.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_model_health_check.py test(extraction): audit #47473 (#48898) 2026-10-28 17:12:07 -07:00
test_model_hub_custom_metadata.py feat(model hub): show admin-defined custom tags and metadata from model_info (#46064) 2026-10-20 17:45:06 -07:00
test_model_rename_propagation.py test(integration): cover model_management endpoint contracts (#44684) 2026-10-14 03:36:47 +00:00
test_organization_budget_clear.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_organization_lifecycle.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_partial_update_sequences.py test: expect 422 for budget refusals in unification, e2e and integration suites 2026-09-20 06:33:57 +00:00
test_pass_through_model_hub.py feat(proxy): publish opted-in pass-through endpoints on the Model Hub (#43905) 2026-11-12 17:46:38 +00:00
test_policy_attachment_runtime.py test(integration): cover guardrails_policies endpoint contracts (#44722) 2026-10-14 04:07:26 +00:00
test_project_lifecycle.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_prompt_management.py test(integration): cover prompts_tags_caching endpoint contracts (#44686) 2026-10-14 04:10:31 +00:00
test_ptu_share_guards.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_scim_admin_group.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_scim_group_bulk_roster.py fix(scim)!: delete a group through the team delete's bulk path instead of one member at a time (#47544) 2026-10-21 00:59:23 +00:00
test_scim_group_delete.py test(scim): audit #47544 (#48791) 2026-10-28 06:08:30 -07:00
test_scim_group_member_not_yet_provisioned.py test(integration): cover customer reported cache key, cache_control, bedrock request id, responses schema, scim and tag budget contracts (#42785) 2026-09-23 13:19:38 -07:00
test_scim_group_pathless_patch.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_scim_user_lifecycle.py fix(auth): return a generic 401 for an unknown key (#39787) 2026-10-15 01:05:25 -07:00
test_search_tool_crud.py test(integration): expect the Exa search envelope's provider_metadata after #47423 (#47572) 2026-10-20 15:29:30 +00:00
test_skill_download_history.py chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
test_smoke.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_sso_sign_in.py feat(sso): create the team_id_jwt_field team on UI SSO login when team_id_upsert is set (#49256) 2026-10-30 14:39:25 -07:00
test_tag_budget_update.py fix(proxy): stop /tag/update 500 when the body carries the tag's budget_id (#50324) 2026-11-06 08:55:44 -08:00
test_team_available.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_block.py fix(proxy): refresh cached team on /team/block and /team/unblock (#35643) 2026-10-19 22:50:33 +00:00
test_team_block_unblock.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_blocking_multi_instance.py fix(proxy): broadcast team cache invalidation on team writes (#46113) 2026-10-13 16:25:22 -04:00
test_team_budget_duration_defaults.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_budget_limits.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_bulk_member_add.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_bulk_member_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_daily_activity.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_delete_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_team_delete_inputs.py fix(proxy): delete large teams without per-member transaction fan-out (#42998) 2026-09-30 13:49:18 -07:00
test_team_delete_large_membership.py fix(proxy): let /team/delete transactions wait for a pooled connection (#51099) 2026-11-11 12:37:28 -08:00
test_team_delete_member_add_race.py test(integration): repair release qualification timing and labels (#49737) 2026-11-02 18:02:28 -08:00
test_team_delete_member_cache_eviction.py fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) 2026-10-15 10:57:00 -07:00
test_team_delete_prometheus.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_filter_ui.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_info.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_key_bulk_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_list.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_list_sorting.py feat(ui): sort every teams table data column (#48109) 2026-10-23 16:21:36 -07:00
test_team_list_v2.py feat(teams): show caller member spend and budget on Your Teams (#45674) 2026-10-24 17:09:55 -07:00
test_team_member_add.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_budget_cache.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_team_member_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_info_validation.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_lifecycle.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_team_member_me.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_reset_budget.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_reset_spend.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_routes.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_team_member_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_metadata_schema.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_metadata_validation.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_team_model.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_new.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_permissions.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_permissions_bulk_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_spend_by_user.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_update.py feat(proxy): let proxy admins open team metadata to team admins, keeping proxy settings with proxy admins (#46506) 2026-10-14 23:51:44 +00:00
test_team_update_multi_instance.py fix(proxy): broadcast team cache invalidation on team writes (#46113) 2026-10-13 16:25:22 -04:00
test_tool_policy_user.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_user_delete_revokes_keys.py fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555) 2026-10-30 06:58:05 -07:00
test_user_list_sorting.py feat(ui): sort every users table data column (#48105) 2026-10-23 16:12:51 -07:00
test_user_rate_limit_updates.py fix(proxy): evict cached user on every proxy for tpm/rpm updates and edit limits in the users UI (#44130) 2026-10-07 16:02:32 -07:00
test_user_routes.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_user_updates_wedged_coordination_redis.py test(integration): fix four flaky integration tests (#49554) 2026-11-02 09:23:17 -08:00
test_users_bulk_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_vector_store_config_ownership.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_vector_store_file_list_managed_ids.py fix(managed_files): keep the upload the managed id encodes when two target models share a deployment (#49420) 2026-11-03 00:58:41 +00:00