* test(proxy_behavior): scaffold session-scoped async ASGI client + liveness smoke
Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir
tests/proxy_behavior/management/ outside every existing pytest glob.
conftest.py initialises the proxy app once per session against the DATABASE_URL
the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process
ASGITransport. The one smoke test asserts /health/liveliness returns 200, which
exercises the full FastAPI middleware stack against a real app — no mocks.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): connect prisma via real lifespan; key/generate de-risk
Slice 3 of the management-endpoints behavior-pinning effort. The fixture now
enters the real FastAPI lifespan (proxy_startup_event) instead of just calling
initialize() — that is where prisma_client is connected, password migration is
kicked off, and the rest of the startup wiring runs.
Tests pin the loop to the session scope so the AsyncClient created in the
session fixture and the prisma connection opened in the lifespan share the
same loop as the test bodies.
New de-risk smoke: POST /key/generate with the master key returns 200, the
returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and
the cleartext token is never stored. Proves auth + handler + helper + prisma
all wire together end-to-end against a real Postgres.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): seed 8-actor read-world for the authz matrix
Slice 4 of the management-endpoints behavior-pinning effort. New
``actors.py`` defines the actor enum + seeds an immutable world (2 orgs,
2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-``
prefix so the rows are identifiable in psql and ``_wipe_world`` is
targeted.
Each actor key is created with its cleartext form generated locally and
its hashed form (via ``litellm.proxy.utils.hash_token``) stored in
``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts
the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and
the service-account metadata flag are all set on the seeded rows so the
auth layer resolves the same scopes a real proxy would.
The session-scoped ``world`` fixture re-seeds at session start (idempotent
via wipe-then-create), and the smoke test confirms each of the 8 actor
keys can call ``/key/info`` on itself and receive its own row back.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): per-test scratch namespace + targeted delete_many teardown
Slice 5 of the management-endpoints behavior-pinning effort. Adds the
``scratch`` function-scoped fixture: each test gets a uuid4-derived
namespace prefix, tags writes with it (``key_alias``, ``team_alias``,
``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s
any row whose namespace column starts with that prefix.
Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and
orders deletes children-before-parents to avoid FK conflicts. The Slice 3
de-risk smoke is migrated onto the same fixture so it stops accumulating
untagged tokens across repeated local runs.
Smoke proves both halves of the contract: one test writes a scratch-tagged
key and asserts it lands; a second test runs after the first's teardown
and asserts no rows in the scratch namespace survived.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): codify G3 (strict-import grep) as a pytest item
Slice 6 of the management-endpoints behavior-pinning effort. Two new tests
walk every .py file under tests/proxy_behavior/ and assert:
* no ``from litellm.proxy.management_endpoints`` import — the suite is
deliberately constrained to the HTTP boundary so it survives handler
refactors;
* no ``mock``/``patch`` on ``user_api_key_auth`` — mocking auth is the
structural failure mode of the existing 11k-line mock suite, and the
point of this harness is that the real auth layer runs.
Codifying G3 as a CI test removes the "did someone forget to check the
PR-description checklist" failure mode.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* style(proxy_behavior): apply black to G3 grep test
Follow-up to 6f588c753b — line-length fixes only, no behavior change.
* test(proxy_behavior): pin /key/generate authz matrix (18 scenarios)
Slice 7 of the management-endpoints behavior-pinning effort. Parametrized
matrix across two axes: actor (8 seeded) × target scope (self, team_alpha
in org_a, team_beta in org_b). 18 scenarios after dropping non-applicable
combos. Whole-suite wall-time stays at ~4.7s (well under the 10-min G2
budget for the eventual CI job).
While pinning, the test surfaced one seed gap: ``_get_user_in_team`` reads
``members_with_roles`` (a JSON list of ``{user_id, role}``), not the plain
``members`` String[]. Both columns are now populated in the seed to match
what the real ``/team/new`` handler would produce.
Expected status codes are intentionally heterogeneous (200, 400, 401)
because the current handler emits different statuses depending on which
check fails first (role gate, team-member-perm gate, "not assigned"
check). Pinning the *observed* codes — not what they "should" be — is
exactly the regression signal we want.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/info authz matrix (24 scenarios)
Slice 8 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target keys (own, OWNER's key in org_a, CROSS_ORG_USER's key in org_b)
covering self-read, same-team-peer read, and cross-org read.
Notable pinned behaviors (intentionally surfaced for review, not "fixed"):
* ORG_ADMIN gets 403 on individual key info even within their own org
— visibility is scoped to "your own keys" + "your team's keys", not
"your org's keys".
* Same-team peers (INTERNAL_USER, UNRELATED_SAME_ORG, SERVICE_ACCOUNT)
DO see each other's keys. Whether that is desired is for the team
to decide; this PR only pins the existing behavior so unintentional
changes flip the matrix red.
Wall-time is unchanged (~4.3s for the slice on its own).
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/list default-visibility matrix (8 scenarios)
Slice 9 of the management-endpoints behavior-pinning effort. For /key/list
the response IS the matrix: each of the 8 seeded actors calls the endpoint
with default filters and the test asserts set-equality between the returned
visible-token set (filtered to seeded tokens only, so unrelated rows can't
flap the assertion) and a pinned expected actor-set.
Pinned default visibility:
* PROXY_ADMIN sees all 8 actors' keys.
* Every other actor sees only their own key — including ORG_ADMIN
(which had broader expectations going in but currently behaves
same-as-internal-user for /key/list defaults) and TEAM_ADMIN (no
team-aggregation without include_team_keys=true).
Future changes that broaden or narrow any single actor's default
visibility will turn this matrix red — exactly the regression signal we
want. Parameter-driven views (include_team_keys, filters) are deferred to
Slice 13 / PR2 follow-up.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/update authz matrix + mutation re-read (21 scenarios)
Slice 10 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target shapes (self-owned, OWNER-scoped in org_a/team_alpha,
CROSS_ORG_USER-scoped in org_b/team_beta) = 21 applicable scenarios.
Each test:
1. Master-key-seeds a fresh scratch key with the target's (user_id,
team_id) scope (so the read-world stays untouched).
2. Has the actor under test POST /key/update flipping ``models`` to
a known marker list.
3. Asserts the status code AND the DB row's ``models`` field — present
when 200, unchanged otherwise — so a handler that silently mutates
on a denied response surfaces red.
Observed gating (pinned, not endorsed):
* PROXY_ADMIN bypasses every check.
* ORG_ADMIN is blocked by an early role gate, always 401.
* Every other (INTERNAL_USER-rolesed) actor hits one of three failure
modes — 403 "user can only create keys for themselves", 403
"only proxy admins, team admins, or org admins", or 401
"team_member_permission_error" — depending on whether they own the
target and whether they're a team admin / member of its team.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/regenerate authz matrix + rotation contract (22 scenarios)
Slice 11 of the management-endpoints behavior-pinning effort. 21 matrix
scenarios (8 actors × 3 target shapes, minus the cross_org/owner combo
that exists in the seed but isn't applicable) plus one smoke for the
``/key/{key:path}/regenerate`` route registration.
On 200 outcomes the test verifies the full rotation contract:
* the regenerate response key differs from the old cleartext,
* the OLD cleartext returns 401 on a follow-up ``/key/info``,
* the NEW cleartext returns 200 on a follow-up ``/key/info``.
On denied outcomes the test verifies the OLD cleartext still works —
catching any handler that mutates the token row on a failed call.
Pinned authz divergence vs /key/update: regenerate routes most denials
through the team-member-perm 401 path rather than the role-gate 403
path. The matrices for both endpoints are now in tree side-by-side, so
any future refactor that "harmonises" the codes will turn one of the two
red.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/delete authz matrix + post-delete contract (21 scenarios)
Slice 12 of the management-endpoints behavior-pinning effort. Mirrors
slices 10/11. On success: cleartext can no longer authenticate
(handles both hard-delete and soft-delete to LiteLLM_DeletedVerificationToken).
On denial: row survives and cleartext still authenticates.
Notable behavior gap with /key/update: same-team peers (internal_user,
unrelated_same_org, etc.) get 403 on /key/delete for OWNER's key — i.e.
cannot delete each other's keys — whereas they CAN read each other's
keys (Slice 8). Delete is stricter than read. Pinned as-is.
Cumulative whole-suite wall-time is 5.9s for all 128 tests on the local
runner — well under the 10-min G2 budget for the CI job in Slice 13.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* ci(proxy-mgmt-behavior): add PR-triggered workflow for the behavior suite
Slice 13 of the management-endpoints behavior-pinning effort. New
workflow ``test-unit-proxy-mgmt-behavior.yml`` fires ``on: pull_request``
for the same branch set every other proxy unit-test workflow watches
(main, litellm_internal_staging, litellm_oss_branch, litellm_**).
It delegates to the existing reusable ``_test-unit-services-base.yml``
with ``enable-postgres: true``, which already provisions a postgres:14
service container and runs ``prisma db push`` against it before pytest
collects. ``reruns: 0`` because a behavior-pinning matrix that needs
reruns is itself a regression — flakes are signal.
``timeout-minutes: 15`` gives generous headroom over the local 5.9s
whole-suite wall-time; the binding G2 budget is 10 min.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* docs(proxy_behavior): G4 regression-replay table for Key Tier-1
Slice 14 of the management-endpoints behavior-pinning effort. Documents
the regression-replay verification methodology + a 12-row table mapping
recent fix-PRs touching key_management_endpoints.py to the catching
scenarios in the PR1 matrix.
One canonical RED→GREEN cycle is captured verbatim — c7c3df2b02
"extend /key/update admin check to non-budget fields". Under the
parent-of-fix code, 6 scenarios in test_key_update.py flip from 200 to
403; under HEAD code, all 21 pass. The handler swap is the only change
between the two runs, confirming the matrix catches the behavior shift
the fix introduced.
The table also calls out 4 genuine coverage gaps deferred to PR2/PR3:
404-on-missing-key, budget-limit counter assertions, /key/regenerate
upperbound enforcement, and /key/list filter-param views.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* chore(mutmut): include the behavior suite in tests_dir + G5 triage stub
Slice 15 of the management-endpoints behavior-pinning effort. Appends
``tests/proxy_behavior/management/`` to ``[tool.mutmut].tests_dir`` so
the existing mutation-test workflow runs against both the legacy mock
suite AND the new behavior suite — the latter is where the regression
signal will actually surface.
Adds a stub at ``tests/proxy_behavior/management/mutmut_triage/pr1.md``
documenting the G5 triage protocol (zero unreviewed survivors in the 6
Tier-1 handler functions) and a placeholder baseline-metrics table to
fill in after the first manually-triggered mutmut run completes — runs
take hours and run on a manual cadence, so PR1 ships with the wiring +
protocol, not the numbers. The actual baseline is recorded in a
follow-up once ``gh workflow run mutation-test.yml`` finishes.
The kill rate stays telemetry-only, never a gate. G5 (per-survivor
classification) is the binding mutation gate.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* docs(proxy_behavior): suite README with local-repro + conventions + gates
Slice 16 of the management-endpoints behavior-pinning effort. The README
documents:
* The same three commands the CI workflow runs locally (BYO-DATABASE_URL,
no new tooling).
* Suite layout — what each test file covers, which slice it lands.
* The asyncio loop_scope convention required for session fixtures
(httpx AsyncClient + prisma connection) to share a loop with each
test body.
* G3 strict-import convention + the test that enforces it.
* Read-world vs scratch-world fixture conventions.
* Behavior-pinning philosophy: pin observed codes; flag, don't judge.
* Where each G1–G5 + PR1.M1–M3 gate's evidence lives.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* ci(proxy-mgmt-behavior): drop xdist (workers=0) to fix seed race
First run on PR #28321 failed with UniqueViolation on
``behavior-pin-budget`` plus cascading missing-membership FK errors. Both
xdist workers entered ``seed_world()`` concurrently against the shared
Postgres service container; whichever lost the race left the world in a
half-seeded state and downstream tests ran against missing
team_membership rows.
Whole-suite wall-time is ~7s sequentially, so disabling xdist here costs
nothing — and the seed itself is the wrong place to add per-worker
isolation (the world is intentionally shared so set-equality assertions
in /key/list have a deterministic expected set).
* ci(proxy-mgmt-behavior): seed scratch keys via proxy_admin actor, not master
Second CI run failed: ``/key/generate`` with explicit ``user_id`` returned
403 "User can only create keys for themselves. Got user_id=X, Your ID=None"
in every test that called ``_create_scratch_key`` with a per-actor user_id.
The bare master key's auth path was producing ``user_id=None`` in the
fresh CI Postgres, which doesn't trigger the PROXY_ADMIN bypass in
``_user_can_only_create_keys_for_themselves`` reliably. Locally the same
master key path worked, masking the issue.
Fix: every ``_create_scratch_key`` helper now takes a seeder cleartext
and the test bodies pass ``world.keys[Actor.PROXY_ADMIN].cleartext``.
That actor was seeded with ``user_role=PROXY_ADMIN`` AND a concrete
``user_id``, so the bypass fires deterministically in both environments.
No behavior shift in the matrices themselves — all 128 scenarios still
pass locally; only the setup helper's auth identity changed.
The bare-master smoke (test_smoke + test_scratch_teardown) is intentionally
left on the master key path: those tests don't pass ``user_id`` in the
body so they don't hit the user_id-mismatch gate.
* ci(proxy-mgmt-behavior): diag — run world-seed test first + bump max-failures
Third CI run failed identically: seeded PROXY_ADMIN actor's auth resolves
to ``user_id=None`` even though the DB row has the right ``user_id``. The
suite was aborting at maxfail=10 inside test_key_delete, so test_world_seed
(which would tell us whether the seed itself is reachable) never ran in CI.
Two diagnostic moves on this push, no behavior change:
* Rename ``test_world_seed.py`` → ``test_aaa_world_seed.py`` so it's
the first collected file. If it passes in CI we know the seed is
fine and the bug lives downstream; if it fails the same way the
bug is in the auth resolution path.
* Bump ``max-failures`` to 200 for this workflow so we see the full
failure surface instead of stopping at the first cascading setup
error. Will tighten back down once the suite is green.
Adds one new test ``test_proxy_admin_actor_can_create_keys_for_others``
that explicitly exercises the PROXY_ADMIN bypass via /key/generate with
an explicit user_id — the same shape the matrix setup helper uses but
without the matrix machinery muddying the diagnostic.
* ci(proxy-mgmt-behavior): await LiteLLM_VerificationTokenView creation in fixture
Fourth CI run still failed because the proxy's lifespan kicks off
``prisma_client.check_view_exists()`` as a fire-and-forget background
task — that task is what creates ``LiteLLM_VerificationTokenView``, the
SQL view ``user_api_key_auth`` queries to resolve a token to its
user_id / user_role / team.
On a fresh Postgres (CI), the first test races the background task. The
view doesn't exist when the first auth call runs, the resolver falls
through to a degraded path that returns ``user_id=None``, and every
matrix test that depends on the seeded actor's identity then fails
confusingly with "Got user_id=X, Your ID=None" 403s. Locally the view
persists across pytest runs so the race is invisible.
Fix: await ``prisma_client.check_view_exists()`` explicitly inside the
session ``proxy_app`` fixture, after the lifespan enters but before the
fixture yields. Deterministic regardless of whether the underlying DB is
fresh (CI) or warm (local).
* ci(proxy-mgmt-behavior): widen diagnostic to dump token / user / view shape
The fifth CI run isolated the failure to ``/key/generate`` with explicit
user_id while ``/key/info`` works for the same seeded PROXY_ADMIN actor.
The auth context's user_id is None even though the DB row has it set.
This commit widens the diagnostic test: on failure, dump the raw token
row's user_id, the user row's user_role, and what
``LiteLLM_VerificationTokenView`` actually returns for the seeded token.
If the view returns user_id=None we know the view shape is the problem;
if the view returns the right user_id we know it's a downstream code
path stripping it.
* ci(proxy-mgmt-behavior): unambiguous diagnostic view query
Previous diagnostic's raw SQL had an ambiguous user_id column from
joining the view with the user table, so the diagnostic itself crashed
before printing useful state. Simplified to query just the view's columns.
* ci(proxy-mgmt-behavior): add auth-resolver chain diagnostic
Six runs and the underlying data (token row, user row, view row) all
verified correct in CI, but auth still returns user_id=None. This
diagnostic calls the resolver primitives directly:
1. ``prisma.get_data(table_name="combined_view")`` → raw view object
2. ``get_key_object(...)`` → cached/DB UserAPIKeyAuth
3. ``get_user_object(...)`` → LiteLLM_UserTable row
4. ``_is_user_proxy_admin`` / ``_get_user_role``
and prints each intermediate via captured stdout (-s). Whichever step
returns None/False in CI is where the chain breaks. Imports come from
``litellm.proxy.auth`` (not management_endpoints), so G3 still passes.
* ci(proxy-mgmt-behavior): set LITELLM_MASTER_KEY env so lifespan doesn't wipe it
Real root cause of every CI run that returned ``Your ID=None`` for the
seeded actors:
* In ``initialize()``, ``master_key`` is set from the config YAML's
``general_settings.master_key`` (load_config code path at
proxy_server.py:4174).
* Then the FastAPI lifespan (``proxy_startup_event``) runs and at line
776 does ``master_key = get_secret_str("LITELLM_MASTER_KEY")``,
which UNCONDITIONALLY overwrites the global.
* In CI the env var is unset, so the post-lifespan ``master_key`` is
None.
Downstream every auth path degrades: master-key requests don't bypass
because ``secrets.compare_digest(api_key, None)`` raises and is caught
to ``is_master_key_valid=False``; seeded-actor requests cache a
``UserAPIKeyAuth`` whose ``user_role`` never resolves through the
PROXY_ADMIN bypass; ``_is_allowed_to_make_key_request`` then hits the
``user_id`` mismatch path with ``Your ID=None``.
Locally my shell happened to have ``LITELLM_MASTER_KEY`` set from a prior
session, which is why every local run was green and CI red — exactly the
"don't generalize from your environment to CI" memory.
Fix: ``os.environ.setdefault("LITELLM_MASTER_KEY", MASTER_KEY)`` and
``os.environ.setdefault("CONFIG_FILE_PATH", config_path)`` before
entering the lifespan, so its re-read produces the same value as
``initialize()``.
Whole-suite still green locally (130 tests, ~6.4s).
* ci(proxy-mgmt-behavior): force premium_user=True so /key/regenerate isn't gated
Ninth CI run cleared every ``Your ID=None`` failure (the master_key env
fix worked end-to-end) and exposed the next thin layer of failures:
``/key/regenerate`` returns 500 "Regenerating Virtual Keys is an
Enterprise feature" in CI because the proxy can't see a
``LITELLM_LICENSE``. Locally my license is set, so the matrix passes.
The behavior matrix is supposed to pin authz, not licensing — so flip
``proxy_server.premium_user = True`` directly, both before and after the
lifespan (the lifespan re-runs ``_license_check.is_premium()`` and would
otherwise reset it). With premium gating disabled, the regenerate matrix
exercises the same authz path /key/update does.
Whole-suite still green locally (130 tests, ~6.3s).
* test(proxy_behavior): trim debug diagnostics, restore default max-failures
Followup to the CI-bring-up sequence: now that the suite is green in CI
(130 → 129 tests after this trim; 156s wall-time on ubuntu-latest), drop
the diagnostic noise left over from debugging the master_key wipe:
* Rename ``test_aaa_world_seed.py`` back to ``test_world_seed.py`` —
no longer needs to run first.
* Remove ``test_auth_resolver_returns_correct_user_id_and_role`` —
that test reached into private auth helpers to localize the bug
between the DB and ``UserAPIKeyAuth``; it has served its purpose
and isn't HTTP-boundary.
* Keep ``test_proxy_admin_actor_can_create_keys_for_others`` (without
the failure-time dump) — it's a real authz contract that pins the
PROXY_ADMIN bypass on /key/generate, and would catch a regression
of the same conftest interaction this sequence revealed.
* Drop the workflow's ``max-failures: 200`` override — that was a
debug aid for seeing the full failure surface in CI. Default of 10
is right for a stable suite.
* chore(proxy_behavior): drop empty mutmut triage stub, fold protocol into README
The mutmut_triage/pr1.md file was a placeholder for numbers and
classifications that don't exist yet — the first mutmut run is a manual
follow-up. Empty stubs aren't evidence; deleting it.
The G5 protocol (run the workflow, triage survivors in the six Tier-1
handler functions, kill-or-accept-with-reason, zero unreviewed) moves
into the suite README's "Gate evidence" block. The real triage file
will land alongside the first mutmut follow-up.
pyproject.toml's [tool.mutmut].tests_dir entry stays — that's the
one-line wiring that makes the existing (manual-trigger) mutation-test
workflow include our suite next time someone runs it. Comment updated
to drop the dead file reference.
* chore(proxy_behavior): drop README + trim comments
Removes the suite README — its contents (local repro, layout, conventions)
were either restated by the file structure or already covered by the
workflow YAML and pyproject.toml. Trims docstrings and inline comments
across every test file to keep only non-obvious WHY (the masking
``_get_user_in_team`` reads, the LiteLLM_VerificationTokenView models-can't-
be-NULL gotcha, the org_admin/peer-visibility surprise, the rotation
contract).
Suite still 129 green locally.
* test(proxy_behavior): address Greptile review — env force, pagination, dedup
- conftest: force LITELLM_MASTER_KEY / CONFIG_FILE_PATH unconditionally
instead of setdefault. An ambient LITELLM_MASTER_KEY with a different
value would make the proxy authenticate on that key while the tests
still send MASTER_KEY → silent 401s.
- test_key_list: paginate /key/list instead of a single size=100 request.
size is capped at 100 by the endpoint, so on a non-fresh DB a single
page could truncate PROXY_ADMIN's view and a seeded key could fall off
the page. Walk total_pages.
- conftest: hoist the duplicated _create_scratch_key helper (copy-pasted
and already diverged across test_key_{update,regenerate,delete}.py)
into a single shared create_scratch_key.
- Delete regression_replay/README.md — G4 regression-replay evidence
belongs in the PR description, not a committed doc file (repo docs
policy + the effort's own plan both say so). Content moved to the PR.
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
The mutation-test workflow timed out at the 350-minute job cap when
running whole-folder mutation against litellm/proxy/management_endpoints/
(~30 files, ~1.5 MB of source). Every mutant was running the full
test suite, and mutants were generated for lines no test covers — which
would survive regardless, just wasting compute.
mutmut 3.x's mutate_only_covered_lines setting runs the suite once up
front to compute coverage, then skips mutating uncovered lines. This
cuts the mutant count dramatically and is the right semantic for the
score (no test → no kill possible → uncountable). Per-mutant test
filtering by function name is already automatic in mutmut 3.x; no
external coverage step is needed.
* ci: add manually-triggered mutation testing smoke workflow
Adds a workflow_dispatch-only GitHub Actions workflow that runs mutmut
against a single source/test pair (router_settings_endpoints) to validate
the tooling end-to-end before scaling.
The workflow reinstalls litellm non-editable so the mutants/ sandbox is
not shadowed by the editable .pth on sys.path, and sets PYTHONPATH so
the trampolined sandbox copy wins over site-packages.
mutmut itself is pulled in via uv run --with so it does not appear in
uv.lock or affect the shared dev environment.
Includes a temporary push: trigger scoped to this branch so we can
iterate before the workflow file lands on the default branch — to be
removed before merging (workflow_dispatch only requires the file on the
default branch to surface the manual trigger button).
* ci(mutation): disable rerun and xdist plugins for mutmut runs
mutmut's in-process pytest.main() call hits
`INTERNALERROR: no option named 'filtered_exceptions'` from
pytest-retry's pytest_configure hook. Reruns are also wrong for
mutation testing — a "failed" mutant test that gets retried would
mask which mutants are killed vs. survive. Disable retry,
rerunfailures, and xdist via pytest_add_cli_args in [tool.mutmut].
* ci(mutation): uninstall pytest-retry before mutmut runs
`-p no:retry` (and similar names) didn't match pytest-retry's
entry-point name, so the plugin still loaded and crashed during
mutmut's "Running clean tests" phase. Uninstalling the package is
surgical and doesn't depend on guessing the entry-point name.
* ci(mutation): emit per-survivor diffs to run-page summary + artifact
The previous artifact only contained `mutmut results` text (which in
mutmut 3.x lists survivor names but not the actual mutations). Adds:
- `mutmut export-cicd-stats` to produce mutmut-cicd-stats.json with the
killed/survived/total scoreboard.
- `mutmut show <name>` per surviving mutant to capture each mutation as
a unified diff.
- A `mutmut-report.md` that combines summary + run-progress tail +
per-survivor diffs, written to both the artifact and
$GITHUB_STEP_SUMMARY (visible on the run page, no download needed).
- Corrected artifact paths: stats files live under mutants/, not the
project root.
- The trampolined source file from the sandbox so survivors can be
inspected even outside `mutmut show`.
* ci(mutation): document intended manual weekly cadence in trigger comment
* ci(mutation): generate ACH-style report with embedded function bodies
Replaces the inline bash markdown generation with a Python script that:
- Groups survivors by function (one section per function, function body
shown once per section, surviving mutants nested as subsections)
- Embeds each enclosing function's source via Python AST (so the agent
has full context, not just a 3-line `mutmut show` diff)
- Inlines the existing test file(s) listed in [tool.mutmut].tests_dir
- Writes an ACH-style task description at the bottom following the
prompt template from arXiv 2501.12862
Output goes to mutation-report.md (artifact) and the head of the file
is appended to $GITHUB_STEP_SUMMARY for at-a-glance visibility.
* fix(mutation report): correctly parse function names with leading underscores
mutmut's mutant-name prefix is x_ (single underscore), so a function
named _foo produces mutants x__foo__mutmut_N. The previous regex
\.x__(.+)__mutmut_ ate the function's leading underscore as part of
the prefix. Changed to \.x_(.+)__mutmut_ so leading underscores are
preserved in the captured function name; verified for normal, leading-
underscore, and dunder-method names.
* feat(mutation report): full Meta ACH-style rendering with MUTANT delimiters
For each surviving mutant, parse the mutmut sandbox trampoline file and
render the mutated function as it appears in the source — with the
differing lines wrapped in `# MUTANT START` / `# MUTANT END` comments,
matching the format from Meta's ACH paper (arXiv 2501.12862, Table 1).
Renames the function header back to its original name so the agent sees
the function as it would appear in the file. Falls back to the unified
diff if the trampoline lookup fails.
Handles replace, insert, and delete diff ops; uses difflib's
SequenceMatcher to find the differing line ranges.
The unified diff is preserved in a collapsible <details> block as
secondary context.
* ci(mutation): scope to whole management_endpoints folder, drop temp push trigger
Final scope before merge:
- paths_to_mutate / tests_dir broadened from one file to the entire
management_endpoints source/test folders
- Trigger is now `workflow_dispatch` only — the temporary push: block
used during workflow iteration is removed
- timeout-minutes bumped from 60 to 350 (just under the GH-hosted job
cap of 360); whole-folder mutation against ~15 files / ~7.5k LOC can
take a few hours
- Artifact path for the trampoline files glob-expanded to cover all
files under mutants/litellm/proxy/management_endpoints/
* fix(mutation report): warn when multiple functions in a file share a name
Addresses the Greptile review concern: ast.walk's first-match-wins
behavior could embed the wrong function body when a file defines the
same name in multiple places (e.g., a module-level helper and a class
method). mutmut's mutant identifier does not carry class context, so
we can't always determine which definition was mutated.
find_function_in_file now returns the start line of every matching
definition; render() surfaces a "Note: N functions named X" warning
in the report when there is more than one match. The first match is
still embedded as the body — the warning tells the reader to verify
manually instead of silently using the wrong context.
Smoke-tested against the existing artifact: single-match files render
unchanged.
* Fix mutation report anchors
* Fix mutation report TOC anchors
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Our `uv.lock` already resolves jinja2 to 3.1.6, so Docker / CI installs
get that version. The `pyproject.toml` floor was lagging at 3.1.0,
which means downstream consumers using `--resolution=lowest-direct` or
older constraint files can land on 3.1.0-3.1.5 instead of the version
we actually test against.
Aligns the declared floor with the resolved version so external
installers see the same baseline our test matrix exercises.
`uv lock` diff is metadata-only (no resolved-version drift).
* feat(audio_transcription): add NVIDIA Riva STT provider
Adds nvidia_riva as a new audio transcription provider, supporting both
NVCF-hosted and self-hosted Riva ASR deployments via gRPC streaming.
- Auto-resamples input audio to 16 kHz mono LINEAR_PCM (soundfile + numpy,
audioread fallback) so callers can send any common format.
- Maps OpenAI params: language (en -> en-US), response_format (text/json/
verbose_json), timestamp_granularities=["word"] -> enable_word_time_offsets,
word offsets converted ms -> s for verbose_json.
- Auth: NVCF when nvcf_function_id is set (SSL on by default), self-hosted
otherwise (SSL off by default), with explicit use_ssl override.
- gRPC errors wrapped via NvidiaRivaException -> litellm exception classes.
- Optional deps gated behind [stt-nvidia-riva] extra (nvidia-riva-client,
soundfile, audioread, numpy).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(nvidia_riva): address PR review feedback
- handler: forward call-level `timeout` to streaming_response_generator
(kwarg-detected via inspect for older riva-client compat) so a stalled
Riva server cannot block the caller indefinitely.
- audio_utils: spill bytes to a tempfile before audioread.audio_open;
most audioread backends (FFmpeg, GStreamer) require a real filesystem
path and previously raised TypeError on BytesIO, breaking the mp3/m4a
fallback path.
- audio_utils: prefer soxr / scipy.signal.resample_poly for resampling
(anti-aliased polyphase) when installed, falling back to linear only
as a last resort. Avoids aliasing on 44.1/48 kHz -> 16 kHz downsamples.
- transformation: bare `es` now maps to es-ES (Castilian) instead of
es-US, matching BCP-47 conventions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: trigger CI re-run [stabilize loop 1/3]
* Update litellm/llms/nvidia_riva/audio_transcription/transformation.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* chore: trigger CI re-run [stabilize loop 1/3]
* fix code qa
* fix lint
* fix mypy
* fix mypy
* Fix NVIDIA Riva ASR service lookup
* Fix NVIDIA Riva transcription payload logging
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: oss-pr-review-agent-shin[bot] <281797381+oss-pr-review-agent-shin[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Removing the new check-dependency-floors.yml workflow. It only fires when
pyproject.toml changes, which is rare; for those PRs, a maintainer can
run the same check by hand with one command. Documented that command in
a pyproject.toml comment next to the deps.
Also adds the missing upper bound on importlib-metadata (>=8.0.0,<9.0)
for consistency with every other entry in the list.
CI matrix on Python 3.13 caught three floors that predate cp313 prebuilt
wheels and would force users into a Rust/C build:
- tiktoken: 0.7.0 -> 0.8.0 (cp313 wheels start at 0.8)
- tokenizers: 0.20.0 -> 0.21.0 (cp313 wheels start at 0.21; sdist's
pyproject.toml pre-0.21 is also malformed for modern build backends)
- pydantic: 2.5.0 -> 2.10.0 (pydantic-core cp313 wheels start at 2.27,
shipped with pydantic 2.10)
Verified locally on Python 3.10 and 3.13: install at lowest-direct +
import litellm + import every openai-namespace symbol the codebase uses
all pass.
The 12 core `[project.dependencies]` entries in pyproject.toml were exact
`==` pins, a side effect of the Poetry → uv migration. This forces every
downstream package that lists litellm as a dependency to downgrade common
runtime libraries (openai, pydantic, aiohttp, click, jsonschema, ...) to
the exact versions we ship. Customers have flagged this as a coexistence
blocker.
Switch to lower-bounded ranges with upper bounds where the upstream
package is pre-1.0 or has a known breaking-major-version policy.
Reproducibility for our Docker proxy and CI continues to come from
`uv.lock`, which is regenerated here as a metadata-only diff (no
resolved versions or hashes change).
Inspired by #26157 (which got stranded on `litellm_oss_staging_04_21_2026`
when the forward-merge to internal staging in #26216 was closed). Floors
in this PR are tighter than #26157's: they were validated by installing
litellm at `--resolution=lowest-direct` and importing the openai-namespace
symbols the codebase actually uses.
Floor highlights vs #26157:
- openai >= 2.20 (was 2.0) — Responses API symbols + `Omit` need a 2.x mid-range floor
- httpx >= 0.28, < 1.0 (was no upper) — pre-1.0
- importlib-metadata >= 8.0 (was 6.0) — stay in tested major
- tokenizers >= 0.20, < 1.0 (was 0.19, no upper) — pre-1.0
- aiohttp >= 3.10, < 4.0 (was no upper) — bound major
- pydantic >= 2.5, < 3.0 — kept
- All other floors: keep tested major, add upper bound
Adds a `check-dependency-floors.yml` GitHub Actions workflow that
installs litellm at `--resolution=lowest-direct` on Python 3.10 and 3.13
and import-checks every openai symbol the codebase uses, so a future
floor regression fails fast in CI rather than silently in the field.
Per Yuneng's feedback, use a single @pytest.mark.vcr marker so one record
sweep populates cassettes for every marked test across all providers,
instead of forcing each test to bind to a hard-coded cassette path.
Changes vs. the initial scaffolding:
- Add 'pytest-recording==0.13.4' on top of vcrpy. Adopt its layout:
cassettes live at 'cassettes/<test_module>/<test_name>.yaml', resolved
automatically. New tests just decorate with '@pytest.mark.vcr' — no
imports or path bookkeeping.
- Move the shared filter/match config into a 'vcr_config' fixture in
'tests/llm_translation/conftest.py' (consumed by pytest-recording for
every marked test in the dir). Drop the standalone 'vcr_config.py'.
- Bulk record / replay via the standard '--record-mode' CLI flag:
'make test-llm-translation-record' now sweeps every '@pytest.mark.vcr'
test under tests/llm_translation in one shot. Optional 'TARGET=' var
scopes to a single file.
- Move existing cassettes to the per-test paths and update the local
in-process Anthropic regenerator to write to the same paths.
- Refresh README + Makefile target docs to match the sweep workflow.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Live LLM e2e tests have been draining provider billing accounts and going
flaky on outages (LIT-2683). This change introduces vcrpy-backed cassette
replay so CI can exercise the same end-to-end LiteLLM transformation paths
without hitting the live provider:
- Add 'vcrpy==8.1.1' to the dev dependency group.
- New 'tests/llm_translation/vcr_config.py' centralises the VCR config:
filters auth/secret headers and per-request response headers, matches on
method+URI+body, and exposes 'LITELLM_VCR_RECORD_MODE' for re-recording.
- New 'tests/llm_translation/test_anthropic_completion_vcr.py' demonstrates
the pattern with one non-streaming and one streaming Anthropic test that
replay from cassettes shipped under 'cassettes/'.
- New 'tests/llm_translation/cassettes/_record_anthropic_fixtures.py' lets
contributors regenerate the canned Anthropic cassettes against a local
in-process mock (no API key required), and 'cassettes/README.md' documents
the full record/replay/refresh workflow.
- New 'make test-llm-translation-record FILE=...' Makefile target to refresh
cassettes against the live API.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Dockerfile: pin the unscoped `brace-expansion@5.0.5` alongside
`@isaacs/brace-expansion@5.0.1`. The scoped package only has 5.0.0
and 5.0.1 published; CVE-2026-33750's fix (5.0.5) is on the unscoped
package which npm also vendors. The override loop now swaps both.
- Revert `black` 26.3.1 -> 24.10.0, `pytest` 9.0.3 -> 8.3.5, and
`pytest-asyncio` 1.3.0 -> 1.2.0. The major-version bumps cause CI
lint (black reformats hundreds of files) and code-quality
(liccheck.ini has no entry for the new versions) failures. Both
CVEs are dev-only; skipping leaves no runtime exposure.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
* bump litellm-proxy-extras version to 0.4.67
* bump litellm-proxy-extras pin to 0.4.67 in litellm pyproject
* regenerate uv.lock for litellm-proxy-extras 0.4.67
* bump litellm-enterprise version to 0.1.38
* bump litellm-enterprise pin to 0.1.38 in litellm pyproject
* regenerate uv.lock for litellm-enterprise 0.1.38
All three dependency bumps in this PR resolve on Python 3.10, so there
is no need to jump the floor all the way to 3.11. Also restore the
py3.10-specific lunary==1.4.36 pin that was collapsed when the floor
was temporarily at 3.11.
Now that requires-python starts at 3.11, the "python_version >= '3.9'"
and ">= '3.10'" markers are unconditionally true, and the "< '3.10'"
entries for psycopg, Pillow, pyarrow, langchain, lunary, and pylint can
never resolve. Drop the dead markers and remove the unreachable pins so
the dependency list reflects what actually gets installed.
Bumps orjson, fastapi-sso, and python-multipart to their latest releases
in the proxy extra, and raises the project python floor to 3.11 so the
updated pins can resolve. CI already runs on 3.11 / 3.12 / 3.13 and the
Docker images ship python 3.13, so the floor change aligns the declared
support range with what is actually tested and shipped.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
langgraph-prebuilt 1.0.9 imports ExecutionInfo and ServerInfo from
langgraph.runtime, but those symbols are not exported until
langgraph 1.1.0. Our pin of langgraph==1.0.10 allows
langgraph-prebuilt<1.1.0,>=1.0.8, and uv resolves to 1.0.9 (the
latest in range), which breaks at import time in every test that
touches langgraph.prebuilt (e.g. tests/pass_through_tests/test_mcp_routes.py):
ImportError: cannot import name 'ExecutionInfo' from 'langgraph.runtime'
Pinning langgraph-prebuilt to 1.0.8 pairs correctly with
langgraph==1.0.10 and restores the import path.
1. exclude-newer: change from absolute "2026-04-10" to relative "3 days".
All pinned deps were published before the 3-day cutoff. Re-locked so
uv lock --check passes in test-mcp.yml and test-linting.yml.
2. test_eager_tiktoken_load: run all 10 env var values in a single
subprocess instead of spawning 10 separate processes. Each cold
import litellm takes ~78s on CI, so the old loop took ~13 min on a
single xdist worker. Now takes ~78s total.
3. proxy-db remaining timeout: increase from 20 to 30 minutes. The
remaining group has 51 test files and was consistently timing out at
71% across all branches (pre-existing issue, not migration-related).
1. Cap requires-python to <3.14 — no deps ship 3.14 wheels yet, and
uv's cross-version resolver fails on the Python 3.14 split.
2. Change exclude-newer from relative "30 days" to absolute "2026-04-10"
so the lockfile stays reproducible. The relative date caused
cryptography==46.0.7 (published April 8) to fall outside the window.
3. Parametrize test_eager_loading_env_var_values instead of looping —
with xdist the 6 subprocess cases can run in parallel instead of all
running sequentially on one worker (~13 min → ~2 min).
Also removed redundant case variants (Yes/YES/On/ON) that test the
same str_to_bool code path.
