scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-31 06:56:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 245
litellm/tests/integration/management/test_scim_group_delete.py
Mateo Wang e333ccabdb
fix(scim)!: delete a group through the team delete's bulk path instead of one member at a time (#47544) 
* fix(scim): write a group's roster in one locked transaction instead of one member at a time

POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote

* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers

* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write

A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock

Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize

The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock

* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer

A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.

sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.

* fix(scim): bump detached members' updated_at in UTC

The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.

* fix(scim): delete a group through the team delete's bulk path instead of one member at a time

* fix(scim): chunk the group write's member lookups under the bind cap

Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.

* fix(scim): demote a deleted admin group's members before the team delete so a failed delete stays retryable

* fix(scim): recompute member roles again after the team delete so a concurrent group write cannot regrant admin

* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none

Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.

Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.

* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts

Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.

Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.

* fix(scim): write recomputed member roles only where the membership still backs them

* fix(team): report the users a team delete detaches so SCIM demotes late-added members

* test(scim): audit cells for the bulk group delete

Move the SCIM group helpers into tests/integration/_support/scim.py and add the delete cells: a member's other
group survives, the admin group's members are demoted and the audit row names the SCIM caller, a member a
concurrent PUT adds while the delete waits on the team lock is demoted too, unknown, repeated, non-admin and
hostile ids, an empty group, the peer proxy, a 20-delete burst across both proxies, and a delete after a worker
was SIGKILLed

* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts

* docs(proxy): drop the module docstring that repeated the PR description

* fix(team): sweep deleted team references only under the lock so a late-added member is demoted

The unlocked sweep that ran before the team's advisory lock stripped a member a
concurrent group write added after the roster was read, so neither the roster
snapshot nor the locked sweep's RETURNING named them and the SCIM recompute
left them proxy_admin after the admin group was gone. The locked sweep does the
same strip and reports whom it stripped, so it is the only sweep now.

The integration cell holds the delete on a member's user row so a PATCH lands
between the roster read and the lock, then checks the late member is demoted.
Both waiter queries in the SCIM support module return their count as text so
the first poll parses.

* chore(team): carry the unbounded-IN baseline entries to delete_validated_teams

The delete pass moved from delete_team and the unlocked sweep into delete_validated_teams, so the checker's baseline names the function the two team_id IN lists now live in

* test(scim): expect the master key alias in the audit row and send post-kill requests on a fresh connection

* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps

Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.

User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.

* test(team): observe delete broadcasts at the Redis layer

* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently

* fix(team): chunk the email lookup behind the member row lock

* fix(scim): create a member deleted mid-push again instead of failing the group write

* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch

A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.

* fix(scim): lock member rows while recomputing admin roles after a group push

* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member

* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member

* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first

A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.

The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.

A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.

* fix(user-delete): keep the team alias label on the team members gauge after removals

* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them

* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404

* fix(scim): answer 404 and roll back when the team is deleted during a roster write

* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast

* test(scim): cover deleting a SCIM group through the team delete's bulk path

* fix(scim): evict a deleted group's member caches in bulk and drop cached roles after a recompute

* fix(scim): answer 404 to a group delete that lost a race to another delete

* fix(scim): chunk the locked team lookup and delete in the team delete transaction

* fix(scim): keep service-account keys and team models on a group delete and write its records once under the lock

* fix(scim): demote a deleted group's members inside the delete transaction

The admin-group role write now runs on the team delete's transaction, over the
roster read under the team lock, so a failed delete leaves roles untouched and
a member added while the delete waited is demoted with the rest. The delete
transaction also waits for a free connection as long as it may run, so a burst
of deletes queues instead of failing after Prisma's two-second default. Every
member in the 5-vs-500 statement test now holds a key.

* fix(audit): write deleted keys' audit rows in one insert

A delete that removes many keys wrote one audit insert task per key, and those tasks held pool connections the delete's follow-up writes waited on. The race test now waits on any lock wait behind the delete.
2026-10-21 00:59:23 +00:00

431 lines
20 KiB
Python
Raw Blame History

"""DELETE /scim/v2/Groups/{id} deletes the group's team the way /team/delete does.
The members' group entries, their membership rows and the keys the members hold on the team go in one locked
pass whatever the roster size, and the configured admin group's members are demoted in that same pass, so they stay
admins until the group is gone. A member a concurrent group write added while the delete waited on the team lock is
demoted with the rest, and a group write that arrives once the delete holds the lock finds the group gone. The group
is 404 after, on every worker and on the peer proxy, and an unrelated key keeps serving.
A service-account key and a model on the team stay, so re-creating the group with the same id brings them back.
"""
import os
import re
import uuid
from collections.abc import Iterator
from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager
from functools import partial
from hashlib import sha256
from pathlib import Path
from typing import Final
import httpx
import pytest
import yaml
from integration._support.client import GATEWAY_LIMITS, JSON_OBJECT, Gateway, eventually, object_value, string_value
from integration._support.database import read_rows
from integration._support.process import graceful_stop_seconds, owned_proxy_process
from integration._support.scim import (
GROUP_SCHEMA,
add_to_group,
assert_gone,
assert_landed,
body,
create_group,
created_team,
delete_group,
groups_of,
held_team_lock,
held_user_row,
keys_of,
member_ids,
membership_user_ids,
replace_group,
seed_users,
service_account_key,
team_key,
user_role,
waiters_on_lock_held_by,
waiters_on_rows_locked_by,
)
from pydantic import JsonValue, TypeAdapter
from litellm.constants import LITELLM_PROXY_MASTER_KEY_ALIAS
pytestmark: Final = pytest.mark.shares_worker_state
PROXY_ADMIN: Final = "proxy_admin"
DEMOTED_ROLE: Final = "internal_user_viewer"
MASTER_KEY_USER: Final = "default_user_id"
BURST_GROUPS: Final = 20
OWNED_PROXY_TIMEOUT: Final = int(2 * graceful_stop_seconds() + 120)
DELETED_TEAM_RECORDS_SQL: Final = 'SELECT team_id FROM "LiteLLM_DeletedTeamTable" WHERE team_id = %s'
DELETED_KEY_RECORDS_SQL: Final = 'SELECT token FROM "LiteLLM_DeletedVerificationToken" WHERE token = %s'
TEAM_MODEL_IDS_SQL: Final = """SELECT model_id FROM "LiteLLM_ProxyModelTable" WHERE model_info->>'team_id' = %s"""
BACKENDS_BLOCKED_BY_SQL: Final = "SELECT pid FROM pg_stat_activity WHERE %s::int = ANY(pg_blocking_pids(pid))"
USER_ROLE_SQL: Final = 'SELECT user_role FROM "LiteLLM_UserTable" WHERE user_id = %s'
DELETED_AUDIT_ROWS_SQL: Final = """
SELECT table_name, action, object_id, changed_by, changed_by_api_key
FROM "LiteLLM_AuditLog"
WHERE object_id IN (%s, %s) AND action = 'deleted'
ORDER BY table_name
"""
_STARTED_WORKER: Final = re.compile(r"Started server process \[(\d+)\]")
_PIDS: Final = TypeAdapter(tuple[int, ...])
def _hashed(key: str) -> str:
return sha256(key.encode()).hexdigest()
def _admin_group_config(tmp_path: Path, admin_alias: str) -> Path:
config: Final = JSON_OBJECT.validate_python(yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text()))
settings: Final = {**object_value(config["litellm_settings"]), "scim_admin_group": admin_alias}
path: Final = tmp_path / "scim-admin-group.yaml"
path.write_text(yaml.safe_dump({**config, "litellm_settings": settings}))
return path
def _roles_of(candidate: Gateway, users: tuple[str, ...]) -> frozenset[str]:
return frozenset(user_role(candidate, user) for user in users)
def _backends_blocked_by(holder: int) -> tuple[int, ...]:
return _PIDS.validate_python([row["pid"] for row in read_rows(BACKENDS_BLOCKED_BY_SQL, (str(holder),))])
def _stored_roles(users: tuple[str, ...]) -> frozenset[str]:
return frozenset(string_value(read_rows(USER_ROLE_SQL, (user,))[0]["user_role"]) for user in users)
def _models_status(candidate: Gateway, key: str) -> int:
return candidate.request("GET", "/v1/models", key=key).status_code
def _team_model_ids(team: str) -> frozenset[str]:
return frozenset(string_value(row["model_id"]) for row in read_rows(TEAM_MODEL_IDS_SQL, (team,)))
def _assert_revoked_everywhere(proxies: tuple[Gateway, ...], key: str) -> None:
for candidate in proxies:
eventually(partial(_models_status, candidate, key), lambda code: code == 401, seconds=30)
def _deleted_audit_rows(team: str, hashed_key: str) -> list[dict[str, JsonValue]]:
return read_rows(DELETED_AUDIT_ROWS_SQL, (team, hashed_key))
def _worker_startups(log: Path) -> tuple[tuple[int, ...], int]:
text: Final = log.read_text()
return _PIDS.validate_python(_STARTED_WORKER.findall(text)), text.count("Application startup complete.")
@contextmanager
def _fresh_connection(candidate: Gateway) -> Iterator[Gateway]:
with httpx.Client(
base_url=candidate.client.base_url, timeout=candidate.client.timeout, trust_env=False, limits=GATEWAY_LIMITS
) as client:
yield Gateway(client, candidate.key, candidate.upstream_url)
def test_deleting_one_group_leaves_the_members_other_group_intact(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
users: Final = seed_users(scenario, 3)
doomed: Final = created_team(scenario, create_group(gateway, users))
kept: Final = created_team(scenario, create_group(gateway, users))
kept_key: Final = team_key(gateway, kept, users[0])
deleted: Final = delete_group(gateway, doomed)
assert deleted.status_code == 204, deleted.text
assert gateway.request("GET", f"/scim/v2/Groups/{doomed}").status_code == 404
assert_landed(gateway, kept, users)
assert groups_of(gateway, users[0]) == frozenset({kept})
assert keys_of(kept) == frozenset({_hashed(kept_key)})
assert _models_status(gateway, kept_key) == 200
@pytest.mark.timeout(OWNED_PROXY_TIMEOUT)
def test_deleting_the_admin_group_demotes_its_members_and_writes_an_audit_row_naming_the_scim_caller(
gateway: Gateway, tmp_path: Path
) -> None:
admin_alias: Final = f"integration-admins-{uuid.uuid4().hex}"
with (
owned_proxy_process(
gateway,
tmp_path,
{"DATABASE_URL": os.environ["DATABASE_URL"], "LITELLM_STORE_AUDIT_LOGS": "true"},
config=_admin_group_config(tmp_path, admin_alias),
remove_environment=("DATABASE_URL_READ_REPLICA",),
workers=2,
) as owned,
owned.gateway.scenario() as scenario,
):
candidate: Final = owned.gateway
users: Final = seed_users(scenario, 3)
team: Final = created_team(scenario, create_group(candidate, users, display_name=admin_alias))
assert _roles_of(candidate, users) == frozenset({PROXY_ADMIN})
key: Final = team_key(candidate, team, users[0])
deleted: Final = delete_group(candidate, team)
assert deleted.status_code == 204, deleted.text
assert_gone(candidate, team, users, key)
assert _roles_of(candidate, users) == frozenset({DEMOTED_ROLE})
rows: Final = eventually(lambda: _deleted_audit_rows(team, _hashed(key)), lambda found: len(found) >= 2, 30)
caller: Final = {"changed_by": MASTER_KEY_USER, "changed_by_api_key": LITELLM_PROXY_MASTER_KEY_ALIAS}
assert rows == [
{"table_name": "LiteLLM_TeamTable", "action": "deleted", "object_id": team, **caller},
{"table_name": "LiteLLM_VerificationToken", "action": "deleted", "object_id": _hashed(key), **caller},
]
@pytest.mark.timeout(OWNED_PROXY_TIMEOUT)
def test_a_member_added_while_the_admin_group_delete_waits_on_the_team_lock_is_demoted_too(
gateway: Gateway, tmp_path: Path
) -> None:
admin_alias: Final = f"integration-admins-{uuid.uuid4().hex}"
with (
owned_proxy_process(
gateway,
tmp_path,
{"DATABASE_URL": os.environ["DATABASE_URL"]},
config=_admin_group_config(tmp_path, admin_alias),
remove_environment=("DATABASE_URL_READ_REPLICA",),
workers=2,
client_timeout=60,
) as owned,
owned.gateway.scenario() as scenario,
ThreadPoolExecutor(max_workers=2) as pool,
):
candidate: Final = owned.gateway
users: Final = seed_users(scenario, 5)
(late,) = seed_users(scenario, 1)
team: Final = created_team(scenario, create_group(candidate, users, display_name=admin_alias))
assert _roles_of(candidate, users) == frozenset({PROXY_ADMIN})
key: Final = team_key(candidate, team, users[0])
with held_team_lock(team) as holder:
put: Final = pool.submit(replace_group, candidate, team, (*users, late), display_name=admin_alias)
eventually(lambda: waiters_on_lock_held_by(holder), lambda waiting: waiting == 1, seconds=30)
delete: Final = pool.submit(delete_group, candidate, team)
eventually(lambda: waiters_on_lock_held_by(holder), lambda waiting: waiting == 2, seconds=30)
assert _stored_roles(users) == frozenset({PROXY_ADMIN})
replaced: Final = put.result()
deleted: Final = delete.result()
assert replaced.status_code == 200, replaced.text
assert member_ids(body(replaced)) == frozenset((*users, late))
assert deleted.status_code == 204, deleted.text
assert_gone(candidate, team, (*users, late), key)
assert _roles_of(candidate, (*users, late)) == frozenset({DEMOTED_ROLE})
@pytest.mark.timeout(OWNED_PROXY_TIMEOUT)
def test_a_member_added_while_the_admin_group_delete_holds_the_team_lock_is_refused_and_never_made_admin(
gateway: Gateway, tmp_path: Path
) -> None:
admin_alias: Final = f"integration-admins-{uuid.uuid4().hex}"
with (
owned_proxy_process(
gateway,
tmp_path,
{"DATABASE_URL": os.environ["DATABASE_URL"]},
config=_admin_group_config(tmp_path, admin_alias),
remove_environment=("DATABASE_URL_READ_REPLICA",),
workers=2,
client_timeout=60,
) as owned,
owned.gateway.scenario() as scenario,
ThreadPoolExecutor(max_workers=2) as pool,
):
candidate: Final = owned.gateway
users: Final = seed_users(scenario, 5)
(late,) = seed_users(scenario, 1)
team: Final = created_team(scenario, create_group(candidate, users, display_name=admin_alias))
assert _roles_of(candidate, users) == frozenset({PROXY_ADMIN})
key: Final = team_key(candidate, team, users[0])
with held_user_row(users[0]) as holder:
delete: Final = pool.submit(delete_group, candidate, team)
eventually(lambda: waiters_on_rows_locked_by(holder), lambda waiting: waiting == 1, seconds=30)
(deleting,) = _backends_blocked_by(holder)
added: Final = pool.submit(add_to_group, candidate, team, (late,))
eventually(lambda: _backends_blocked_by(deleting), lambda blocked: len(blocked) == 1, seconds=30)
assert _stored_roles(users) == frozenset({PROXY_ADMIN})
deleted: Final = delete.result()
assert deleted.status_code == 204, deleted.text
assert added.result().status_code == 404, added.result().text
assert_gone(candidate, team, users, key)
assert _roles_of(candidate, users) == frozenset({DEMOTED_ROLE})
assert user_role(candidate, late) != PROXY_ADMIN
def test_a_group_delete_revokes_member_keys_and_a_re_created_group_gets_its_service_account_key_and_model_back(
gateway: Gateway,
) -> None:
with gateway.scenario() as scenario:
users: Final = seed_users(scenario, 2)
team: Final = created_team(scenario, create_group(gateway, users))
member_key: Final = team_key(gateway, team, users[0])
service_key: Final = service_account_key(gateway, scenario, team)
scenario.model(model_info={"team_id": team})
team_models: Final = _team_model_ids(team)
assert team_models != frozenset()
deleted: Final = delete_group(gateway, team)
assert deleted.status_code == 204, deleted.text
assert gateway.request("GET", f"/scim/v2/Groups/{team}").status_code == 404
assert membership_user_ids(team) == frozenset()
assert keys_of(team) == frozenset({_hashed(service_key)})
assert _team_model_ids(team) == team_models
_assert_revoked_everywhere((gateway,), member_key)
recreated: Final = gateway.request(
"POST",
"/scim/v2/Groups",
{"schemas": [GROUP_SCHEMA], "externalId": team, "displayName": f"integration-{uuid.uuid4().hex}"},
)
assert recreated.status_code == 201, recreated.text
assert string_value(body(recreated)["id"]) == team
eventually(partial(_models_status, gateway, service_key), lambda code: code == 200, seconds=90)
assert _models_status(gateway, member_key) == 401
def test_deleting_an_unknown_group_is_404(gateway: Gateway) -> None:
missing: Final = f"missing-{uuid.uuid4().hex}"
response: Final = delete_group(gateway, missing)
assert response.status_code == 404, response.text
assert "error" in body(response), response.text
assert missing in response.text
def test_deleting_a_group_twice_is_404_the_second_time(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
users: Final = seed_users(scenario, 2)
team: Final = created_team(scenario, create_group(gateway, users))
first: Final = delete_group(gateway, team)
assert first.status_code == 204, first.text
second: Final = delete_group(gateway, team)
assert second.status_code == 404, second.text
assert "error" in body(second), second.text
assert gateway.request("GET", f"/scim/v2/Groups/{team}").status_code == 404
assert membership_user_ids(team) == frozenset()
def test_two_deletes_racing_on_one_group_answer_204_then_404_and_record_the_team_once(gateway: Gateway) -> None:
with gateway.scenario() as scenario, ThreadPoolExecutor(max_workers=2) as pool:
users: Final = seed_users(scenario, 2)
team: Final = created_team(scenario, create_group(gateway, users))
key: Final = team_key(gateway, team, users[0])
with held_team_lock(team) as holder:
deletes: Final = (pool.submit(delete_group, gateway, team), pool.submit(delete_group, gateway, team))
eventually(lambda: waiters_on_lock_held_by(holder), lambda waiting: waiting == 2, seconds=30)
assert sorted(delete.result().status_code for delete in deletes) == [204, 404]
assert len(read_rows(DELETED_TEAM_RECORDS_SQL, (team,))) == 1
assert len(read_rows(DELETED_KEY_RECORDS_SQL, (_hashed(key),))) == 1
assert gateway.request("GET", f"/scim/v2/Groups/{team}").status_code == 404
def test_deleting_a_group_with_a_non_admin_key_is_refused(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
users: Final = seed_users(scenario, 2)
team: Final = created_team(scenario, create_group(gateway, users))
outsider: Final = scenario.key()
response: Final = delete_group(gateway, team, key=outsider)
assert response.status_code in (401, 403), response.text
assert "error" in body(response), response.text
assert_landed(gateway, team, users)
@pytest.mark.parametrize("group_id", ("123", "x" * 5000, "a%2Fb"), ids=("int_like", "five_kb", "encoded_slash"))
def test_deleting_a_group_with_a_hostile_id_is_404(gateway: Gateway, group_id: str) -> None:
response: Final = delete_group(gateway, group_id)
assert response.status_code == 404, response.text
assert isinstance(response.json(), dict), response.text
assert gateway.request("GET", "/health/liveliness").status_code == 200
def test_deleting_an_empty_group_is_204_then_404(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
team: Final = created_team(scenario, create_group(gateway, ()))
assert member_ids(gateway.get(f"/scim/v2/Groups/{team}")) == frozenset()
deleted: Final = delete_group(gateway, team)
assert deleted.status_code == 204, deleted.text
assert gateway.request("GET", f"/scim/v2/Groups/{team}").status_code == 404
assert membership_user_ids(team) == frozenset()
def test_group_delete_on_one_proxy_revokes_the_team_key_and_the_group_on_the_peer(
gateway: Gateway, peer: Gateway
) -> None:
with gateway.scenario() as scenario:
users: Final = seed_users(scenario, 3)
team: Final = created_team(scenario, create_group(gateway, users))
key: Final = team_key(gateway, team, users[0])
assert _models_status(peer, key) == 200
assert member_ids(peer.get(f"/scim/v2/Groups/{team}")) == frozenset(users)
deleted: Final = delete_group(gateway, team)
assert deleted.status_code == 204, deleted.text
assert peer.request("GET", f"/scim/v2/Groups/{team}").status_code == 404
_assert_revoked_everywhere((peer, gateway), key)
assert_gone(peer, team, users, key)
def test_twenty_group_deletes_split_across_two_proxies_all_land(gateway: Gateway, peer: Gateway) -> None:
proxies: Final = (gateway, peer)
with gateway.scenario() as scenario, ThreadPoolExecutor(max_workers=10) as pool:
bystander: Final = scenario.key()
rosters: Final = tuple(seed_users(scenario, 3) for _ in range(BURST_GROUPS))
teams: Final = tuple(created_team(scenario, create_group(gateway, users)) for users in rosters)
keys: Final = tuple(team_key(gateway, team, users[0]) for team, users in zip(teams, rosters))
assigned: Final = tuple(proxies[index % 2] for index in range(BURST_GROUPS))
responses: Final = tuple(pool.map(delete_group, assigned, teams))
for response in responses:
assert response.status_code == 204, response.text
for team, users, key in zip(teams, rosters, keys):
_assert_revoked_everywhere(proxies, key)
assert_gone(gateway, team, users, key)
assert peer.request("GET", f"/scim/v2/Groups/{team}").status_code == 404
assert _models_status(gateway, bystander) == 200
assert _models_status(peer, bystander) == 200
@pytest.mark.timeout(OWNED_PROXY_TIMEOUT)
def test_group_delete_after_a_worker_was_killed_still_completes(gateway: Gateway, tmp_path: Path) -> None:
with (
owned_proxy_process(
gateway, tmp_path, {}, remove_environment=("DATABASE_URL_READ_REPLICA",), workers=2
) as owned,
owned.gateway.scenario() as scenario,
):
candidate: Final = owned.gateway
workers, _ = eventually(
lambda: _worker_startups(owned.log), lambda found: len(found[0]) == 2 and found[1] == 2, seconds=120
)
first_users: Final = seed_users(scenario, 3)
second_users: Final = seed_users(scenario, 3)
first: Final = created_team(scenario, create_group(candidate, first_users))
second: Final = created_team(scenario, create_group(candidate, second_users))
first_key: Final = team_key(candidate, first, first_users[0])
second_key: Final = team_key(candidate, second, second_users[0])
with _fresh_connection(candidate) as survivor:
with owned.replacing_worker(workers[0]) as kill:
kill.freeze()
kill()
deleted_first: Final = delete_group(survivor, first)
assert deleted_first.status_code == 204, deleted_first.text
assert_gone(survivor, first, first_users, first_key)
deleted_second: Final = delete_group(survivor, second)
assert deleted_second.status_code == 204, deleted_second.text
assert_gone(survivor, second, second_users, second_key)
Reference in a new issue View git blame Copy permalink