2 commits

Author	SHA1	Message	Date
Mateo Wang	622daf3cdf	fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) ... * fix(scim): write a group's roster in one locked transaction instead of one member at a time POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the membership rows and the users' team tags in one transaction under the team's advisory lock, so a 500-member push runs a fixed number of statements instead of thousands. PUT now answers with the roster it just wrote * test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers * fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock * fix(scim): evict the cached team on a group rename and read a no-op roster from the writer A rename-only PUT or PATCH wrote the new alias and left the team cached under its id and old alias on every worker, so keys kept logging spend against the old team alias until the TTL expired. The group write now evicts the cached team (and broadcasts the eviction) right after the alias or metadata update. sync_team_roster reads the team from the writer when the plan names nobody, so a replica cannot hand back the pre-rename row; a plan already matching the roster returns inside the lock without writing, evicting, or emitting the members metric; the response is built from the written row, so lastModified reflects the roster write; and detaching a removed member bumps the user's updated_at like every other user write. * fix(scim): bump detached members' updated_at in UTC The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts into Prisma's naive timestamp column in the session timezone, so on a database whose session is not UTC the removed member's row landed hours away from the UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and have the bulk roster cell assert the bumped row sits between two reads of the database's own UTC clock taken around the replace. * fix(scim): chunk the group write's member lookups under the bind cap Every Prisma "in" filter the group write issues bound one parameter per member with nothing bounding the list, which check_unbounded_in_lists reports as 11 new findings. The single-field lookups now go through litellm.repositories.chunked_in (caller_transaction inside the roster transaction, per_chunk_ok for the role recompute), and the one cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two slices, and a member named only in the last slice still resolves. * fix(proxy): send bulk cache evictions in slices so the broadcaster drops none Veria pointed out that the invalidation publisher caps its pending tasks at 1,024 and drops the excess with a warning, and that a group push now detaches every removed member's keys in one call instead of one member at a time, so a removal of more than roughly 340 members (three cache keys each) silently left stale key objects on the other replicas. The merge base never hit the cap because each member was a separate request. Bulk evictions (team, user, organization, and session key deletes, plus the roster settle) now publish in slices of 256 and wait, bounded at two seconds, for the backlog already handed to the event loop before each slice. A Redis that never answers costs a bulk caller at most that wait per slice instead of wedging the request, and single-key callers are untouched. Regression tests push 1,500 keys and a 500-member removal through a slow publisher and assert every eviction reaches Redis; both fail against the previous tip. * fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts Greptile pointed out that pacing the broadcasts in slices also paced the local eviction: a bulk delete waited for the publish backlog before dropping the next slice from this worker's own memory, so on a Redis that accepts connections but never answers a key deleted from the database kept authenticating on the handling worker for up to two seconds per slice. The merge base cleared every key locally at once. Bulk evictions now drop every affected entry from this worker's memory first, through the cache's own partition lookup since key objects live apart from the other auth objects, and only then publish in paced slices. The roster settle does the same for the deleted tokens, the JWT mapping keys and every changed member's user and membership entries before any step that can wait. Regression tests run both paths against a publisher that never answers and check that nothing is still cached on this worker before the first bound elapses; both fail with the local eviction removed. * fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts * docs(proxy): drop the module docstring that repeated the PR description * fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps Bulk key evictions now drop entries from memory, delete them from Redis in one chunked DEL, and broadcast every key in one pipelined publish, so a large roster removal no longer overflows the publisher's pending cap or costs a Redis round trip per member. A push waits for in-flight membership loads before evicting. User rows are locked in byte order to match the spend writer, a bulk user delete takes every team lock before its first user write, and a POST whose roster write fails deletes the empty team so the IdP retry creates it again. * test(team): observe delete broadcasts at the Redis layer * fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently * fix(team): chunk the email lookup behind the member row lock * fix(scim): create a member deleted mid-push again instead of failing the group write * fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch A member deleted mid-push is created again only when scim_upsert_user allows it, under the verb's own created_via. New group members are checked against the license seat limit as one batch before any is created. Auth cache publish drops log the key count instead of the key. * fix(scim): lock member rows while recomputing admin roles after a group push * fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member * fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member * fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time. The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team. A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup. * fix(user-delete): keep the team alias label on the team members gauge after removals * test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them * fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404 * fix(scim): answer 404 and roll back when the team is deleted during a roster write * test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast	2026-10-15 10:57:00 -07:00
devin-ai-integration[bot]	f285229b51	fix(proxy): delete large teams without per-member transaction fan-out (#42998) ... * fix(proxy): delete large teams without per-member transaction fan-out Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): evict email-only member caches and reset team members metric on delete Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep new delete-team literals within the LIT002 ceiling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve deleted-team member ids before the locked delete Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve email-only deleted-team members with one case-insensitive lookup `_deleted_team_member_user_ids` looked each email-only roster entry up with its own `find_users_by_email` call inside an unbounded `asyncio.gather`: one exact-match query per email, so a large roster fanned out against the pool again and a roster email that differed in case from its user row was missed. Add `UserRepository.find_by_emails`, a single case-insensitive `in` query, and call it once before the locked delete. `management_helpers/utils.py` goes back to its main-branch shape since the single-email helper no longer needs exporting. * fix(repositories): slice find_by_emails into bounded IN statements The unbounded-IN lint flagged the case-insensitive email lookup added for /team/delete cache eviction. chunked_in.find_many_in cannot carry Prisma's insensitive mode, so the repository slices the deduplicated list into IN_LIST_CHUNK_SIZE statements itself and concatenates the pages. Empty input still returns () without a query. * fix(proxy): delete a team once when /team/delete repeats its id The audit sent {"team_ids": [T, T]}: main answered 400 "User not found in team" after deleting the keys and memberships and writing two tombstones, leaving the team row behind; this branch answered 200 but still wrote the tombstone, audit row and eviction twice. DeleteTeamRequest now collapses repeated ids in order, so every later step sees each team once and the response lists each deleted team once. * test(integration): audit cells for /team/delete on large, legacy and concurrent teams Thirty-eight deterministic cells in tests/integration/management/ (the CircleCI integration-management group) covering the /team/delete happy, sad, edge and chaos rows: 250 members against a pool limit of five on two workers, the advisory-lock wait, email-only legacy roster entries in every casing, member and team cache eviction on both proxies for every client and endpoint, the Prometheus gauge, audit rows, malformed and duplicate input, the route gate, and a worker kill, a Redis outage and a proxy restart mid-burst. Every cell runs against the real proxy, Postgres and Redis with the scripted upstream; no component is mocked. On the merge base the rows this fix changes are red (P2028 on the 250-member team, two lock waiters, case-mismatched email lookups, duplicate ids, orphaned LiteLLM_UserTable.teams references under a concurrent burst); on the tip every cell is green twice with identical selections. Two pre-existing behaviours are pinned as observed rather than fixed here: a roster entry with neither user_id nor user_email answers 500, and the LiteLLM_DeletedTeamTable row is committed before the locked transaction, so a delete that dies in between leaves a tombstone for a live team and the retry adds a second. * test(integration): pin each chaos outage to a live /team/delete The three chaos cells applied the outage once three deletes had answered, which on a fast run let the whole burst finish before the worker kill, Redis stop or SIGTERM landed, so the cells passed without exercising the failure. Each cell now holds the first team's advisory lock from a test-owned transaction, waits until that team's delete is queued behind it in Postgres with its request unanswered, applies the outage, and only then releases the lock, so an in-flight delete meets the failure on every run and both legs. The pinned team's outcome and the number of deletes answered before the outage are recorded as junit properties (pinned_delete, answered_before_outage). --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-09-30 13:49:18 -07:00