From f25173b5f940014aeff87c3f43a3ec0eef613c32 Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Tue, 19 May 2026 21:58:34 -0700 Subject: [PATCH] docs(proxy_behavior): suite README with local-repro + conventions + gates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slice 16 of the management-endpoints behavior-pinning effort. The README documents: * The same three commands the CI workflow runs locally (BYO-DATABASE_URL, no new tooling). * Suite layout — what each test file covers, which slice it lands. * The asyncio loop_scope convention required for session fixtures (httpx AsyncClient + prisma connection) to share a loop with each test body. * G3 strict-import convention + the test that enforces it. * Read-world vs scratch-world fixture conventions. * Behavior-pinning philosophy: pin observed codes; flag, don't judge. * Where each G1–G5 + PR1.M1–M3 gate's evidence lives. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d --- tests/proxy_behavior/management/README.md | 102 ++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 tests/proxy_behavior/management/README.md diff --git a/tests/proxy_behavior/management/README.md b/tests/proxy_behavior/management/README.md new file mode 100644 index 00000000000..20ddc86b4a7 --- /dev/null +++ b/tests/proxy_behavior/management/README.md @@ -0,0 +1,102 @@ +# Management-endpoint behavior-pinning suite + +HTTP-boundary regression tests for +`litellm/proxy/management_endpoints/key_management_endpoints.py` (PR1 — Key +Tier-1). Runs against the real proxy app via in-process `httpx.ASGITransport`, +connected to a real Postgres pointed at by `DATABASE_URL`. **No mocks** — +auth runs, prisma runs, integrations run. Test bodies make HTTP calls and +assert at the API boundary. + +The eventual goal (across PR1–PR3) is to pin every authorization / +cross-tenant / budget-bypass boundary on the key + team management +surfaces. PR1 covers six Tier-1 key endpoints (`/key/generate`, `/key/info`, +`/key/list`, `/key/update`, `/key/regenerate`, `/key/delete`). See the +[Notion plan](https://www.notion.so/36643b8acdab8128a581ced0f6a4744d) for +the full scope. + +## Local repro + +Identical to the three commands the CI workflow +(`.github/workflows/test-unit-proxy-mgmt-behavior.yml`, which delegates +to `_test-unit-services-base.yml`) runs: + +```bash +# 1. Bring up Postgres +docker run --rm -d --name litellm-test-pg \ + -e POSTGRES_USER=litellm -e POSTGRES_PASSWORD=litellm -e POSTGRES_DB=litellm_test \ + -p 5432:5432 postgres:14 + +# 2. Migrate the schema (one-time per fresh DB) +export DATABASE_URL=postgresql://litellm:litellm@localhost:5432/litellm_test +uv run prisma generate --schema litellm/proxy/schema.prisma +uv run prisma db push --schema litellm/proxy/schema.prisma --accept-data-loss + +# 3. Run the suite +uv run pytest tests/proxy_behavior/management/ +``` + +Whole-suite wall-time is ~6s on a warm cache (one ~1.5s session setup + +~0.01–0.04s per test). Re-running back-to-back produces identical pass +counts — the scratch-namespace teardown leaves no rows behind. + +### Single scenario / inner loop + +```bash +uv run pytest tests/proxy_behavior/management/test_key_update.py -k self/owner -v +``` + +## Layout + +``` +tests/proxy_behavior/management/ +├── conftest.py # session ASGI client, world seed, scratch fixture +├── actors.py # 8-actor enum + seed_world() helper +├── test_smoke.py # liveness + key/generate de-risk smoke +├── test_world_seed.py # every seeded actor key authenticates +├── test_scratch_teardown.py # scratch namespace cleanup invariants +├── test_no_management_imports.py # G3 — strict-import grep as a test +├── test_key_generate.py # Slice 7 — actor × target matrix +├── test_key_info.py # Slice 8 +├── test_key_list.py # Slice 9 +├── test_key_update.py # Slice 10 +├── test_key_regenerate.py # Slice 11 +├── test_key_delete.py # Slice 12 +├── regression_replay/README.md # G4 — fix-PR → catching-scenario mapping +└── mutmut_triage/pr1.md # G5 — survivor classification protocol +``` + +## Conventions + +- **Async fixture / loop scope.** `pyproject.toml` sets + `asyncio_default_fixture_loop_scope = "session"`, but the default + *test* loop scope is per-function. Add + `pytestmark = pytest.mark.asyncio(loop_scope="session")` at the top + of every test file so the AsyncClient and prisma connection (both + session-scoped) share a loop with the test body. +- **Forbidden imports (G3).** No `from litellm.proxy.management_endpoints`, + no `mock`/`patch` on `user_api_key_auth`. Enforced by + `test_no_management_imports.py` as a pytest item. +- **Read-world vs scratch-world.** The `world` fixture seeds an immutable + read-world under the `behavior-pin-` prefix; tests must not mutate + those rows. The `scratch` fixture gives a per-test + `scratch-` prefix and tears down any row tagged with it. + Write scenarios always tag their creates with `scratch.prefix`. +- **Behavior pinning, not behavior judging.** Expected status codes are + pinned against current handler behavior. The suite's job is to make + *changes* to that behavior visible — not to assert what the codes + *should* be. Comments above each `_SCENARIOS` block call out + surprising or potentially-buggy behaviors for human review. + +## Gate evidence + +PR1's evidence for each G1–G5 + PR1.M1–M3 gate lives in: + +- **G1** — CI run on the PR's workflow `test-unit-proxy-mgmt-behavior` (green). +- **G2** — `pytest --durations=…` summary in the PR description (≤ 10 min). +- **G3** — `test_no_management_imports.py` is part of the suite itself. +- **G4** — `regression_replay/README.md`. +- **G5** — `mutmut_triage/pr1.md`, filled in after the first manually-triggered + `mutation-test.yml` run. +- **PR1.M1** — total scenario count, this README's "Layout" section. +- **PR1.M2** — this README + the workflow YAML are the local-repro contract. +- **PR1.M3** — `mutmut_triage/pr1.md` "Baseline metrics" table.