litellm/tests/integration/management
tessellark 080e643edc
fix(proxy): apply new model rows on every worker as soon as they are written (#49034)
* fix(proxy): apply published model rows on every worker before the debounced resync

Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path

* fix(proxy): load published model rows from the primary under the reconcile lock

Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s.

* test(proxy): cover a published model row that names no credential

* fix(proxy): keep the unknown model name read-through on the replica and outside the lock

Only the by-id load that follows a published write reads the primary inside
the reconcile lock. A request for a model name a worker does not know reads
through the replica routing before taking the lock again, as on main, so
unknown names put no load on the primary and do not wait behind a reload.

* fix(proxy): keep access group edits from dropping staying deployments and batch published config ids

* fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica

* fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing

A batch of published model ids that held one auto router skipped every row in it, so ordinary
rows written at the same moment waited for the full reload. The by-id load now leaves only the
auto router rows to that reload.

An access group update now reads every deployment from the primary, answers 404 for an unknown
group and 400 for an unknown model id or name before it writes anything, then removes the group
from the deployments that leave before it adds it to the ones that join, so a failed write leaves
the group narrower than asked, never wider.

* fix(proxy): write an access group update in one transaction after a fresh read

The update now checks model names first, reads every deployment from the
primary, and commits every join and leave in one batch right after that
read, so a failed write changes nothing and a field another admin set
during validation is kept. Other workers hear about the rows only after
the commit.

* fix(proxy): lock the model rows while an access group update reads and rewrites them

* test(proxy): run the model resync tests on the real installer and a real router

The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen.

* fix(proxy): lock only the rows an access group update touches and answer 404 before 400

An access group update locked every model row, so a lock on any unrelated
row stalled it for 30 s and then failed it with a 500, and unrelated model
edits on other proxies waited behind it. It now takes a per-group advisory
lock, locks only the rows that carry the group or that it selects, and
reads them again once locked. The group existence check runs before the
model name check again, so a missing group answers 404 as on main.

* fix(proxy): never wait on a locked row in an access group update

The update now takes its row locks with NOWAIT. When another edit holds one
of the group's rows, it rolls back, holding nothing, and tries again for about
10 s before answering 409. Holding rows while waiting deadlocked against team
deletes and let a concurrent model edit undo the update's change. The
per-group lock also moves to the two-key advisory space, apart from team ids.

* fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select

* fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-30 03:19:25 -07:00
..
__init__.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
_batch_file_caps.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
cost_center_service.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
team_metadata_validation_proxy_config.yaml test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_access_group_team_sync.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_added_backup_deployment_serves_peer_fallbacks.py fix(proxy): apply new model rows on every worker as soon as they are written (#49034) 2026-10-30 03:19:25 -07:00
test_agent_management.py test(integration): cover agents_a2a endpoint contracts (#44650) 2026-10-14 03:40:07 +00:00
test_auto_router_databricks_classifier_writes.py test: update integration tests left stale by recent product changes (#49143) 2026-10-29 18:06:28 -07:00
test_batch_file_usage_caps.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_batch_file_usage_caps_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_batch_limit_metadata_admin_only.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_batch_output_file_listing.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_budget_list_sorting.py feat(ui): sort budgets by reset duration (#48107) 2026-10-23 16:21:53 -07:00
test_budget_reset_windows.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_budget_updates.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_claude_code_plugin_installation_preference.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_credential_display_name.py feat(credentials): add display_name and make credential_name immutable on PATCH (#43148) 2026-10-08 23:49:56 -07:00
test_credential_federation_values.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_credential_migration_endpoint.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_f7_coverage_closeout.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_f7_key_coverage_push.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_fallback_management_public_names.py fix(fallback): read stored router_settings from the database on fallback writes (#48999) 2026-10-29 07:41:54 -07:00
test_guardrail_usage_config_guardrail.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_health_routing.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_key_aliases.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_block_unblock.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_budget_limits.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_bulk_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_default_allowed_routes.py feat(key_management)!: apply default_key_generate_params.allowed_routes on key generation (#41236) 2026-10-19 18:53:37 -07:00
test_key_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_delete_revocation.py test(integration): cover keys_auth endpoint contracts (#44674) 2026-10-14 07:12:28 +00:00
test_key_generate.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_generation_permissions.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_key_health.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_info.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_info_v2.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_list.py feat(ui): sort virtual keys by expiry, activity, spend and limits (#48074) 2026-10-23 15:45:17 -07:00
test_key_regenerate.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_reset_spend.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_route_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_key_service_account_generate.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_team_change.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_key_updates.py fix(tests): resolve the integration support package without run.py's PYTHONPATH 2026-09-15 21:10:08 -07:00
test_listing_and_health_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_managed_file_create_ownership.py fix(managed_files): enforce input file ownership on batch and fine-tuning create (#45855) 2026-10-12 13:58:25 -04:00
test_managed_vector_store_unified_id.py test(integration): cover vector_stores endpoint contracts (#44739) 2026-10-14 03:59:08 +00:00
test_model_credential_name_updates.py fix(proxy): validate model credential name only when it changes (#42701) 2026-09-23 07:34:32 -07:00
test_model_deployment_lifecycle.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_model_health_check.py test(extraction): audit #47473 (#48898) 2026-10-28 17:12:07 -07:00
test_model_hub_custom_metadata.py feat(model hub): show admin-defined custom tags and metadata from model_info (#46064) 2026-10-20 17:45:06 -07:00
test_model_rename_propagation.py test(integration): cover model_management endpoint contracts (#44684) 2026-10-14 03:36:47 +00:00
test_organization_budget_clear.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_organization_lifecycle.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_partial_update_sequences.py test: expect 422 for budget refusals in unification, e2e and integration suites 2026-09-20 06:33:57 +00:00
test_policy_attachment_runtime.py test(integration): cover guardrails_policies endpoint contracts (#44722) 2026-10-14 04:07:26 +00:00
test_project_lifecycle.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_prompt_management.py test(integration): cover prompts_tags_caching endpoint contracts (#44686) 2026-10-14 04:10:31 +00:00
test_ptu_share_guards.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_scim_admin_group.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_scim_group_bulk_roster.py fix(scim)!: delete a group through the team delete's bulk path instead of one member at a time (#47544) 2026-10-21 00:59:23 +00:00
test_scim_group_delete.py test(scim): audit #47544 (#48791) 2026-10-28 06:08:30 -07:00
test_scim_group_member_not_yet_provisioned.py test(integration): cover customer reported cache key, cache_control, bedrock request id, responses schema, scim and tag budget contracts (#42785) 2026-09-23 13:19:38 -07:00
test_scim_group_pathless_patch.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_scim_user_lifecycle.py fix(auth): return a generic 401 for an unknown key (#39787) 2026-10-15 01:05:25 -07:00
test_search_tool_crud.py test(integration): expect the Exa search envelope's provider_metadata after #47423 (#47572) 2026-10-20 15:29:30 +00:00
test_smoke.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_sso_sign_in.py fix(sso): enforce free user admission without locking out existing users (#48097) 2026-10-23 14:48:29 -07:00
test_team_available.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_block.py fix(proxy): refresh cached team on /team/block and /team/unblock (#35643) 2026-10-19 22:50:33 +00:00
test_team_block_unblock.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_blocking_multi_instance.py fix(proxy): broadcast team cache invalidation on team writes (#46113) 2026-10-13 16:25:22 -04:00
test_team_budget_duration_defaults.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_budget_limits.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_bulk_member_add.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_bulk_member_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_daily_activity.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_delete_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_team_delete_inputs.py fix(proxy): delete large teams without per-member transaction fan-out (#42998) 2026-09-30 13:49:18 -07:00
test_team_delete_large_membership.py fix(proxy)!: stop labelling management and health rejections as auth failures (#48739) 2026-10-28 21:49:03 -07:00
test_team_delete_member_add_race.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_team_delete_member_cache_eviction.py fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) 2026-10-15 10:57:00 -07:00
test_team_delete_prometheus.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_filter_ui.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_info.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_key_bulk_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_list.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_list_sorting.py feat(ui): sort every teams table data column (#48109) 2026-10-23 16:21:36 -07:00
test_team_list_v2.py feat(teams): show caller member spend and budget on Your Teams (#45674) 2026-10-24 17:09:55 -07:00
test_team_member_add.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_budget_cache.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_team_member_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_info_validation.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_lifecycle.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_team_member_me.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_reset_budget.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_reset_spend.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_member_routes.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_team_member_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_metadata_schema.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_metadata_validation.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_team_model.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_new.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_permissions.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_permissions_bulk_update.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_spend_by_user.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_team_update.py feat(proxy): let proxy admins open team metadata to team admins, keeping proxy settings with proxy admins (#46506) 2026-10-14 23:51:44 +00:00
test_team_update_multi_instance.py fix(proxy): broadcast team cache invalidation on team writes (#46113) 2026-10-13 16:25:22 -04:00
test_tool_policy_user.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_user_delete_revokes_keys.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_user_list_sorting.py feat(ui): sort every users table data column (#48105) 2026-10-23 16:12:51 -07:00
test_user_rate_limit_updates.py fix(proxy): evict cached user on every proxy for tpm/rpm updates and edit limits in the users UI (#44130) 2026-10-07 16:02:32 -07:00
test_user_routes.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_user_updates_wedged_coordination_redis.py test: fix flaky spend-log cleanup budget, login lockout, and wedged Redis tests (#47108) 2026-10-29 03:46:32 -07:00
test_users_bulk_delete.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_vector_store_config_ownership.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_vector_store_file_list_managed_ids.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00