scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-11 08:38:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 280
litellm/tests/integration/management/test_health_routing.py
Mateo Wang 622daf3cdf
fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) 
* fix(scim): write a group's roster in one locked transaction instead of one member at a time

POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote

* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers

* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write

A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock

Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize

The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock

* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer

A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.

sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.

* fix(scim): bump detached members' updated_at in UTC

The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.

* fix(scim): chunk the group write's member lookups under the bind cap

Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.

* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none

Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.

Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.

* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts

Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.

Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.

* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts

* docs(proxy): drop the module docstring that repeated the PR description

* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps

Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.

User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.

* test(team): observe delete broadcasts at the Redis layer

* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently

* fix(team): chunk the email lookup behind the member row lock

* fix(scim): create a member deleted mid-push again instead of failing the group write

* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch

A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.

* fix(scim): lock member rows while recomputing admin roles after a group push

* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member

* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member

* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first

A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.

The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.

A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.

* fix(user-delete): keep the team alias label on the team members gauge after removals

* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them

* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404

* fix(scim): answer 404 and roll back when the team is deleted during a roster write

* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
2026-10-15 10:57:00 -07:00

258 lines
12 KiB
Python
Raw Blame History

import hashlib
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Final
import pytest
import yaml
from integration._support.client import JSON_OBJECT, Gateway, eventually
from integration._support.database import scratch_database
from integration._support.database_relay import database_relay, held_statement_relay, plaintext_database_url
from integration._support.process import owned_proxy
_READINESS_PROBE_QUERY: Final = b"SELECT 1"
def _health_routing_config(directory: Path, general_settings: dict[str, object]) -> Path:
config: Final = yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text())
config["general_settings"] = {**config.get("general_settings", {}), **general_settings}
path: Final = directory / "health-routing.yaml"
path.write_text(yaml.safe_dump(config))
return path
def _assert_liveness_and_readiness(gateway: Gateway, *, status_code: int, body: dict[str, str]) -> None:
for path in ("/health/liveliness", "/health/liveness"):
response: Final = gateway.client.get(path)
assert response.status_code == status_code, response.text
assert response.json() == ("I'm alive!" if status_code == 200 else body), response.text
readiness: Final = gateway.client.get("/health/readiness")
assert readiness.status_code == status_code, readiness.text
assert readiness.json() == body, readiness.text
@pytest.mark.timeout(180)
def test_public_readiness_reports_database_outage_and_allow_unavailable_setting(
gateway: Gateway, tmp_path: Path
) -> None:
with scratch_database() as database_url:
with database_relay(database_url, _READINESS_PROBE_QUERY) as (relay, relayed_url):
with owned_proxy(
gateway,
tmp_path,
{
"DATABASE_URL": plaintext_database_url(relayed_url),
"PRISMA_HEALTH_WATCHDOG_ENABLED": "false",
},
workers=1,
) as candidate:
ready_before_outage: Final = candidate.client.get("/health/readiness")
assert ready_before_outage.status_code == 200, ready_before_outage.text
assert ready_before_outage.json() == {"status": "healthy", "db": "connected"}, ready_before_outage.text
relay.arm()
disconnected_after_outage: Final = eventually(
lambda: candidate.client.get("/health/readiness"),
lambda response: (
relay.tripped.is_set() and response.json() == {"status": "healthy", "db": "disconnected"}
),
seconds=45,
return_last_on_timeout=True,
)
assert relay.tripped.is_set(), "readiness database probe did not reach the relay"
assert disconnected_after_outage.status_code == 503, disconnected_after_outage.text
assert disconnected_after_outage.json() == {
"status": "healthy",
"db": "disconnected",
}, disconnected_after_outage.text
allow_unavailable_config: Final = _health_routing_config(
tmp_path,
{"allow_requests_on_db_unavailable": True},
)
with database_relay(database_url, _READINESS_PROBE_QUERY) as (relay, relayed_url):
with owned_proxy(
gateway,
tmp_path,
{
"DATABASE_URL": plaintext_database_url(relayed_url),
"PRISMA_HEALTH_WATCHDOG_ENABLED": "false",
},
config=allow_unavailable_config,
workers=1,
) as candidate:
ready_before_outage_with_allow: Final = candidate.client.get("/health/readiness")
assert ready_before_outage_with_allow.status_code == 200, ready_before_outage_with_allow.text
assert ready_before_outage_with_allow.json() == {"status": "healthy", "db": "connected"}, (
ready_before_outage_with_allow.text
)
relay.arm()
disconnected_with_allow: Final = eventually(
lambda: candidate.client.get("/health/readiness"),
lambda response: (
relay.tripped.is_set() and response.json() == {"status": "healthy", "db": "disconnected"}
),
seconds=45,
return_last_on_timeout=True,
)
assert relay.tripped.is_set(), "allow-unavailable readiness probe did not reach the relay"
assert disconnected_with_allow.status_code == 200, disconnected_with_allow.text
assert disconnected_with_allow.json() == {
"status": "healthy",
"db": "disconnected",
}, disconnected_with_allow.text
@pytest.mark.timeout(300)
def test_public_readiness_reports_stalled_database_lookups(gateway: Gateway, tmp_path: Path) -> None:
with scratch_database() as database_url:
for allow_unavailable in (False, True):
unknown_key = "sk-" + uuid.uuid4().hex
lookup_hash = hashlib.sha256(unknown_key.encode()).hexdigest().encode()
config = (
_health_routing_config(tmp_path, {"allow_requests_on_db_unavailable": True})
if allow_unavailable
else None
)
with held_statement_relay(database_url, lookup_hash) as (relay, relayed_url):
overrides = {
"DATABASE_URL": plaintext_database_url(relayed_url),
"PRISMA_HEALTH_WATCHDOG_ENABLED": "false",
"PROXY_DB_LOOKUP_DEADLINE_SECONDS": "1",
"PROXY_DB_LOOKUP_STALL_WINDOW_SECONDS": "15",
}
with owned_proxy(
gateway,
tmp_path,
overrides,
config=config,
workers=1,
) as candidate:
expected_request_status = 400 if allow_unavailable else 503
unknown_model_message = (
"/chat/completions: Invalid model name passed in model=unconfigured-stalled-readiness-model. "
"Call `/v1/models` to view available models for your key."
)
expected_request_body = (
{
"error": {
"message": unknown_model_message,
"type": "invalid_request_error",
"param": None,
"code": "400",
"provider_specific_fields": {"error": unknown_model_message},
}
}
if allow_unavailable
else {
"error": {
"message": (
"Service Unavailable, the authentication database is temporarily unreachable. "
"Please retry shortly."
),
"type": "no_db_connection",
"param": "None",
"code": "503",
}
}
)
with ThreadPoolExecutor(max_workers=1) as executor:
pending = executor.submit(
candidate.request,
"POST",
"/v1/chat/completions",
{
"model": "unconfigured-stalled-readiness-model",
"messages": [{"role": "user", "content": "stalled lookup control"}],
},
key=unknown_key,
)
assert relay.held.wait(30), "unknown-key lookup did not reach the held statement relay"
expected_status = 200 if allow_unavailable else 503
stalled = eventually(
lambda: candidate.client.get("/health/readiness"),
lambda response, expected_status=expected_status: (
response.status_code == expected_status
and response.json() == {"status": "healthy", "db": "stalled"}
),
seconds=5,
return_last_on_timeout=True,
)
assert stalled.status_code == expected_status, stalled.text
assert stalled.json() == {"status": "healthy", "db": "stalled"}, stalled.text
relay.release()
recovered = eventually(
lambda: candidate.client.get("/health/readiness"),
lambda response: (
response.status_code == 200
and response.json() == {"status": "healthy", "db": "connected"}
),
seconds=30,
return_last_on_timeout=True,
)
assert recovered.status_code == 200, recovered.text
assert recovered.json() == {"status": "healthy", "db": "connected"}, recovered.text
rejected = pending.result(timeout=60)
assert (rejected.status_code, JSON_OBJECT.validate_json(rejected.content)) == (
expected_request_status,
expected_request_body,
), rejected.text
@pytest.mark.parametrize("token_source", ("config", "environment"))
def test_drain_endpoint_requires_token_and_changes_health_state_only_after_authorization(
token_source: str, gateway: Gateway, tmp_path: Path
) -> None:
disabled: Final = gateway.client.get("/health/drain")
assert disabled.status_code == 404, disabled.text
assert disabled.json() == {"detail": "Not Found"}, disabled.text
token: Final = "synthetic-health-drain-token"
general_settings: Final = {
"enable_drain_endpoint": True,
**({"drain_endpoint_token": token} if token_source == "config" else {}),
}
config: Final = _health_routing_config(tmp_path, general_settings)
overrides: Final = {"DRAIN_ENDPOINT_TOKEN": token} if token_source == "environment" else {}
remove_environment: Final = ("DRAIN_ENDPOINT_TOKEN",) if token_source == "config" else ()
with owned_proxy(
gateway,
tmp_path,
overrides,
config=config,
workers=1,
remove_environment=remove_environment,
) as candidate:
_assert_liveness_and_readiness(
candidate,
status_code=200,
body={"status": "healthy", "db": "connected"},
)
missing: Final = candidate.client.get("/health/drain")
assert missing.status_code == 401, missing.text
assert missing.json() == {"detail": "Invalid or missing X-Drain-Token"}, missing.text
wrong: Final = candidate.client.get(
"/health/drain",
headers={"X-Drain-Token": "synthetic-wrong-token"},
)
assert wrong.status_code == 401, wrong.text
assert wrong.json() == {"detail": "Invalid or missing X-Drain-Token"}, wrong.text
_assert_liveness_and_readiness(
candidate,
status_code=200,
body={"status": "healthy", "db": "connected"},
)
drained: Final = candidate.client.get(
"/health/drain",
headers={"X-Drain-Token": token},
)
assert drained.status_code == 200, drained.text
assert drained.json() == {"status": "drained", "drained_requests": 0}, drained.text
for path in ("/health/liveliness", "/health/liveness", "/health/readiness"):
response: Final = candidate.client.get(path)
assert response.status_code == 503, response.text
assert response.json() == {"status": "shutting_down"}, response.text
Reference in a new issue View git blame Copy permalink