Slice 15 of the management-endpoints behavior-pinning effort. Appends
``tests/proxy_behavior/management/`` to ``[tool.mutmut].tests_dir`` so
the existing mutation-test workflow runs against both the legacy mock
suite AND the new behavior suite — the latter is where the regression
signal will actually surface.
Adds a stub at ``tests/proxy_behavior/management/mutmut_triage/pr1.md``
documenting the G5 triage protocol (zero unreviewed survivors in the 6
Tier-1 handler functions) and a placeholder baseline-metrics table to
fill in after the first manually-triggered mutmut run completes — runs
take hours and run on a manual cadence, so PR1 ships with the wiring +
protocol, not the numbers. The actual baseline is recorded in a
follow-up once ``gh workflow run mutation-test.yml`` finishes.
The kill rate stays telemetry-only, never a gate. G5 (per-survivor
classification) is the binding mutation gate.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 14 of the management-endpoints behavior-pinning effort. Documents
the regression-replay verification methodology + a 12-row table mapping
recent fix-PRs touching key_management_endpoints.py to the catching
scenarios in the PR1 matrix.
One canonical RED→GREEN cycle is captured verbatim — c7c3df2b02
"extend /key/update admin check to non-budget fields". Under the
parent-of-fix code, 6 scenarios in test_key_update.py flip from 200 to
403; under HEAD code, all 21 pass. The handler swap is the only change
between the two runs, confirming the matrix catches the behavior shift
the fix introduced.
The table also calls out 4 genuine coverage gaps deferred to PR2/PR3:
404-on-missing-key, budget-limit counter assertions, /key/regenerate
upperbound enforcement, and /key/list filter-param views.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 12 of the management-endpoints behavior-pinning effort. Mirrors
slices 10/11. On success: cleartext can no longer authenticate
(handles both hard-delete and soft-delete to LiteLLM_DeletedVerificationToken).
On denial: row survives and cleartext still authenticates.
Notable behavior gap with /key/update: same-team peers (internal_user,
unrelated_same_org, etc.) get 403 on /key/delete for OWNER's key — i.e.
cannot delete each other's keys — whereas they CAN read each other's
keys (Slice 8). Delete is stricter than read. Pinned as-is.
Cumulative whole-suite wall-time is 5.9s for all 128 tests on the local
runner — well under the 10-min G2 budget for the CI job in Slice 13.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 11 of the management-endpoints behavior-pinning effort. 21 matrix
scenarios (8 actors × 3 target shapes, minus the cross_org/owner combo
that exists in the seed but isn't applicable) plus one smoke for the
``/key/{key:path}/regenerate`` route registration.
On 200 outcomes the test verifies the full rotation contract:
* the regenerate response key differs from the old cleartext,
* the OLD cleartext returns 401 on a follow-up ``/key/info``,
* the NEW cleartext returns 200 on a follow-up ``/key/info``.
On denied outcomes the test verifies the OLD cleartext still works —
catching any handler that mutates the token row on a failed call.
Pinned authz divergence vs /key/update: regenerate routes most denials
through the team-member-perm 401 path rather than the role-gate 403
path. The matrices for both endpoints are now in tree side-by-side, so
any future refactor that "harmonises" the codes will turn one of the two
red.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 10 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target shapes (self-owned, OWNER-scoped in org_a/team_alpha,
CROSS_ORG_USER-scoped in org_b/team_beta) = 21 applicable scenarios.
Each test:
1. Master-key-seeds a fresh scratch key with the target's (user_id,
team_id) scope (so the read-world stays untouched).
2. Has the actor under test POST /key/update flipping ``models`` to
a known marker list.
3. Asserts the status code AND the DB row's ``models`` field — present
when 200, unchanged otherwise — so a handler that silently mutates
on a denied response surfaces red.
Observed gating (pinned, not endorsed):
* PROXY_ADMIN bypasses every check.
* ORG_ADMIN is blocked by an early role gate, always 401.
* Every other (INTERNAL_USER-rolesed) actor hits one of three failure
modes — 403 "user can only create keys for themselves", 403
"only proxy admins, team admins, or org admins", or 401
"team_member_permission_error" — depending on whether they own the
target and whether they're a team admin / member of its team.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 9 of the management-endpoints behavior-pinning effort. For /key/list
the response IS the matrix: each of the 8 seeded actors calls the endpoint
with default filters and the test asserts set-equality between the returned
visible-token set (filtered to seeded tokens only, so unrelated rows can't
flap the assertion) and a pinned expected actor-set.
Pinned default visibility:
* PROXY_ADMIN sees all 8 actors' keys.
* Every other actor sees only their own key — including ORG_ADMIN
(which had broader expectations going in but currently behaves
same-as-internal-user for /key/list defaults) and TEAM_ADMIN (no
team-aggregation without include_team_keys=true).
Future changes that broaden or narrow any single actor's default
visibility will turn this matrix red — exactly the regression signal we
want. Parameter-driven views (include_team_keys, filters) are deferred to
Slice 13 / PR2 follow-up.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 8 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target keys (own, OWNER's key in org_a, CROSS_ORG_USER's key in org_b)
covering self-read, same-team-peer read, and cross-org read.
Notable pinned behaviors (intentionally surfaced for review, not "fixed"):
* ORG_ADMIN gets 403 on individual key info even within their own org
— visibility is scoped to "your own keys" + "your team's keys", not
"your org's keys".
* Same-team peers (INTERNAL_USER, UNRELATED_SAME_ORG, SERVICE_ACCOUNT)
DO see each other's keys. Whether that is desired is for the team
to decide; this PR only pins the existing behavior so unintentional
changes flip the matrix red.
Wall-time is unchanged (~4.3s for the slice on its own).
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 7 of the management-endpoints behavior-pinning effort. Parametrized
matrix across two axes: actor (8 seeded) × target scope (self, team_alpha
in org_a, team_beta in org_b). 18 scenarios after dropping non-applicable
combos. Whole-suite wall-time stays at ~4.7s (well under the 10-min G2
budget for the eventual CI job).
While pinning, the test surfaced one seed gap: ``_get_user_in_team`` reads
``members_with_roles`` (a JSON list of ``{user_id, role}``), not the plain
``members`` String[]. Both columns are now populated in the seed to match
what the real ``/team/new`` handler would produce.
Expected status codes are intentionally heterogeneous (200, 400, 401)
because the current handler emits different statuses depending on which
check fails first (role gate, team-member-perm gate, "not assigned"
check). Pinning the *observed* codes — not what they "should" be — is
exactly the regression signal we want.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 6 of the management-endpoints behavior-pinning effort. Two new tests
walk every .py file under tests/proxy_behavior/ and assert:
* no ``from litellm.proxy.management_endpoints`` import — the suite is
deliberately constrained to the HTTP boundary so it survives handler
refactors;
* no ``mock``/``patch`` on ``user_api_key_auth`` — mocking auth is the
structural failure mode of the existing 11k-line mock suite, and the
point of this harness is that the real auth layer runs.
Codifying G3 as a CI test removes the "did someone forget to check the
PR-description checklist" failure mode.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 5 of the management-endpoints behavior-pinning effort. Adds the
``scratch`` function-scoped fixture: each test gets a uuid4-derived
namespace prefix, tags writes with it (``key_alias``, ``team_alias``,
``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s
any row whose namespace column starts with that prefix.
Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and
orders deletes children-before-parents to avoid FK conflicts. The Slice 3
de-risk smoke is migrated onto the same fixture so it stops accumulating
untagged tokens across repeated local runs.
Smoke proves both halves of the contract: one test writes a scratch-tagged
key and asserts it lands; a second test runs after the first's teardown
and asserts no rows in the scratch namespace survived.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 4 of the management-endpoints behavior-pinning effort. New
``actors.py`` defines the actor enum + seeds an immutable world (2 orgs,
2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-``
prefix so the rows are identifiable in psql and ``_wipe_world`` is
targeted.
Each actor key is created with its cleartext form generated locally and
its hashed form (via ``litellm.proxy.utils.hash_token``) stored in
``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts
the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and
the service-account metadata flag are all set on the seeded rows so the
auth layer resolves the same scopes a real proxy would.
The session-scoped ``world`` fixture re-seeds at session start (idempotent
via wipe-then-create), and the smoke test confirms each of the 8 actor
keys can call ``/key/info`` on itself and receive its own row back.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 3 of the management-endpoints behavior-pinning effort. The fixture now
enters the real FastAPI lifespan (proxy_startup_event) instead of just calling
initialize() — that is where prisma_client is connected, password migration is
kicked off, and the rest of the startup wiring runs.
Tests pin the loop to the session scope so the AsyncClient created in the
session fixture and the prisma connection opened in the lifespan share the
same loop as the test bodies.
New de-risk smoke: POST /key/generate with the master key returns 200, the
returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and
the cleartext token is never stored. Proves auth + handler + helper + prisma
all wire together end-to-end against a real Postgres.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir
tests/proxy_behavior/management/ outside every existing pytest glob.
conftest.py initialises the proxy app once per session against the DATABASE_URL
the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process
ASGITransport. The one smoke test asserts /health/liveliness returns 200, which
exercises the full FastAPI middleware stack against a real app — no mocks.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
