mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
docs(proxy_behavior): suite README with local-repro + conventions + gates
Slice 16 of the management-endpoints behavior-pinning effort. The README
documents:
* The same three commands the CI workflow runs locally (BYO-DATABASE_URL,
no new tooling).
* Suite layout — what each test file covers, which slice it lands.
* The asyncio loop_scope convention required for session fixtures
(httpx AsyncClient + prisma connection) to share a loop with each
test body.
* G3 strict-import convention + the test that enforces it.
* Read-world vs scratch-world fixture conventions.
* Behavior-pinning philosophy: pin observed codes; flag, don't judge.
* Where each G1–G5 + PR1.M1–M3 gate's evidence lives.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
This commit is contained in:
parent
8753969a73
commit
f25173b5f9
1 changed files with 102 additions and 0 deletions
102
tests/proxy_behavior/management/README.md
Normal file
102
tests/proxy_behavior/management/README.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
# Management-endpoint behavior-pinning suite
|
||||
|
||||
HTTP-boundary regression tests for
|
||||
`litellm/proxy/management_endpoints/key_management_endpoints.py` (PR1 — Key
|
||||
Tier-1). Runs against the real proxy app via in-process `httpx.ASGITransport`,
|
||||
connected to a real Postgres pointed at by `DATABASE_URL`. **No mocks** —
|
||||
auth runs, prisma runs, integrations run. Test bodies make HTTP calls and
|
||||
assert at the API boundary.
|
||||
|
||||
The eventual goal (across PR1–PR3) is to pin every authorization /
|
||||
cross-tenant / budget-bypass boundary on the key + team management
|
||||
surfaces. PR1 covers six Tier-1 key endpoints (`/key/generate`, `/key/info`,
|
||||
`/key/list`, `/key/update`, `/key/regenerate`, `/key/delete`). See the
|
||||
[Notion plan](https://www.notion.so/36643b8acdab8128a581ced0f6a4744d) for
|
||||
the full scope.
|
||||
|
||||
## Local repro
|
||||
|
||||
Identical to the three commands the CI workflow
|
||||
(`.github/workflows/test-unit-proxy-mgmt-behavior.yml`, which delegates
|
||||
to `_test-unit-services-base.yml`) runs:
|
||||
|
||||
```bash
|
||||
# 1. Bring up Postgres
|
||||
docker run --rm -d --name litellm-test-pg \
|
||||
-e POSTGRES_USER=litellm -e POSTGRES_PASSWORD=litellm -e POSTGRES_DB=litellm_test \
|
||||
-p 5432:5432 postgres:14
|
||||
|
||||
# 2. Migrate the schema (one-time per fresh DB)
|
||||
export DATABASE_URL=postgresql://litellm:litellm@localhost:5432/litellm_test
|
||||
uv run prisma generate --schema litellm/proxy/schema.prisma
|
||||
uv run prisma db push --schema litellm/proxy/schema.prisma --accept-data-loss
|
||||
|
||||
# 3. Run the suite
|
||||
uv run pytest tests/proxy_behavior/management/
|
||||
```
|
||||
|
||||
Whole-suite wall-time is ~6s on a warm cache (one ~1.5s session setup +
|
||||
~0.01–0.04s per test). Re-running back-to-back produces identical pass
|
||||
counts — the scratch-namespace teardown leaves no rows behind.
|
||||
|
||||
### Single scenario / inner loop
|
||||
|
||||
```bash
|
||||
uv run pytest tests/proxy_behavior/management/test_key_update.py -k self/owner -v
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
tests/proxy_behavior/management/
|
||||
├── conftest.py # session ASGI client, world seed, scratch fixture
|
||||
├── actors.py # 8-actor enum + seed_world() helper
|
||||
├── test_smoke.py # liveness + key/generate de-risk smoke
|
||||
├── test_world_seed.py # every seeded actor key authenticates
|
||||
├── test_scratch_teardown.py # scratch namespace cleanup invariants
|
||||
├── test_no_management_imports.py # G3 — strict-import grep as a test
|
||||
├── test_key_generate.py # Slice 7 — actor × target matrix
|
||||
├── test_key_info.py # Slice 8
|
||||
├── test_key_list.py # Slice 9
|
||||
├── test_key_update.py # Slice 10
|
||||
├── test_key_regenerate.py # Slice 11
|
||||
├── test_key_delete.py # Slice 12
|
||||
├── regression_replay/README.md # G4 — fix-PR → catching-scenario mapping
|
||||
└── mutmut_triage/pr1.md # G5 — survivor classification protocol
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Async fixture / loop scope.** `pyproject.toml` sets
|
||||
`asyncio_default_fixture_loop_scope = "session"`, but the default
|
||||
*test* loop scope is per-function. Add
|
||||
`pytestmark = pytest.mark.asyncio(loop_scope="session")` at the top
|
||||
of every test file so the AsyncClient and prisma connection (both
|
||||
session-scoped) share a loop with the test body.
|
||||
- **Forbidden imports (G3).** No `from litellm.proxy.management_endpoints`,
|
||||
no `mock`/`patch` on `user_api_key_auth`. Enforced by
|
||||
`test_no_management_imports.py` as a pytest item.
|
||||
- **Read-world vs scratch-world.** The `world` fixture seeds an immutable
|
||||
read-world under the `behavior-pin-` prefix; tests must not mutate
|
||||
those rows. The `scratch` fixture gives a per-test
|
||||
`scratch-<uuid>` prefix and tears down any row tagged with it.
|
||||
Write scenarios always tag their creates with `scratch.prefix`.
|
||||
- **Behavior pinning, not behavior judging.** Expected status codes are
|
||||
pinned against current handler behavior. The suite's job is to make
|
||||
*changes* to that behavior visible — not to assert what the codes
|
||||
*should* be. Comments above each `_SCENARIOS` block call out
|
||||
surprising or potentially-buggy behaviors for human review.
|
||||
|
||||
## Gate evidence
|
||||
|
||||
PR1's evidence for each G1–G5 + PR1.M1–M3 gate lives in:
|
||||
|
||||
- **G1** — CI run on the PR's workflow `test-unit-proxy-mgmt-behavior` (green).
|
||||
- **G2** — `pytest --durations=…` summary in the PR description (≤ 10 min).
|
||||
- **G3** — `test_no_management_imports.py` is part of the suite itself.
|
||||
- **G4** — `regression_replay/README.md`.
|
||||
- **G5** — `mutmut_triage/pr1.md`, filled in after the first manually-triggered
|
||||
`mutation-test.yml` run.
|
||||
- **PR1.M1** — total scenario count, this README's "Layout" section.
|
||||
- **PR1.M2** — this README + the workflow YAML are the local-repro contract.
|
||||
- **PR1.M3** — `mutmut_triage/pr1.md` "Baseline metrics" table.
|
||||
Loading…
Add table
Reference in a new issue