Ninth CI run cleared every ``Your ID=None`` failure (the master_key env
fix worked end-to-end) and exposed the next thin layer of failures:
``/key/regenerate`` returns 500 "Regenerating Virtual Keys is an
Enterprise feature" in CI because the proxy can't see a
``LITELLM_LICENSE``. Locally my license is set, so the matrix passes.
The behavior matrix is supposed to pin authz, not licensing — so flip
``proxy_server.premium_user = True`` directly, both before and after the
lifespan (the lifespan re-runs ``_license_check.is_premium()`` and would
otherwise reset it). With premium gating disabled, the regenerate matrix
exercises the same authz path /key/update does.
Whole-suite still green locally (130 tests, ~6.3s).
Real root cause of every CI run that returned ``Your ID=None`` for the
seeded actors:
* In ``initialize()``, ``master_key`` is set from the config YAML's
``general_settings.master_key`` (load_config code path at
proxy_server.py:4174).
* Then the FastAPI lifespan (``proxy_startup_event``) runs and at line
776 does ``master_key = get_secret_str("LITELLM_MASTER_KEY")``,
which UNCONDITIONALLY overwrites the global.
* In CI the env var is unset, so the post-lifespan ``master_key`` is
None.
Downstream every auth path degrades: master-key requests don't bypass
because ``secrets.compare_digest(api_key, None)`` raises and is caught
to ``is_master_key_valid=False``; seeded-actor requests cache a
``UserAPIKeyAuth`` whose ``user_role`` never resolves through the
PROXY_ADMIN bypass; ``_is_allowed_to_make_key_request`` then hits the
``user_id`` mismatch path with ``Your ID=None``.
Locally my shell happened to have ``LITELLM_MASTER_KEY`` set from a prior
session, which is why every local run was green and CI red — exactly the
"don't generalize from your environment to CI" memory.
Fix: ``os.environ.setdefault("LITELLM_MASTER_KEY", MASTER_KEY)`` and
``os.environ.setdefault("CONFIG_FILE_PATH", config_path)`` before
entering the lifespan, so its re-read produces the same value as
``initialize()``.
Whole-suite still green locally (130 tests, ~6.4s).
Fourth CI run still failed because the proxy's lifespan kicks off
``prisma_client.check_view_exists()`` as a fire-and-forget background
task — that task is what creates ``LiteLLM_VerificationTokenView``, the
SQL view ``user_api_key_auth`` queries to resolve a token to its
user_id / user_role / team.
On a fresh Postgres (CI), the first test races the background task. The
view doesn't exist when the first auth call runs, the resolver falls
through to a degraded path that returns ``user_id=None``, and every
matrix test that depends on the seeded actor's identity then fails
confusingly with "Got user_id=X, Your ID=None" 403s. Locally the view
persists across pytest runs so the race is invisible.
Fix: await ``prisma_client.check_view_exists()`` explicitly inside the
session ``proxy_app`` fixture, after the lifespan enters but before the
fixture yields. Deterministic regardless of whether the underlying DB is
fresh (CI) or warm (local).
Slice 5 of the management-endpoints behavior-pinning effort. Adds the
``scratch`` function-scoped fixture: each test gets a uuid4-derived
namespace prefix, tags writes with it (``key_alias``, ``team_alias``,
``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s
any row whose namespace column starts with that prefix.
Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and
orders deletes children-before-parents to avoid FK conflicts. The Slice 3
de-risk smoke is migrated onto the same fixture so it stops accumulating
untagged tokens across repeated local runs.
Smoke proves both halves of the contract: one test writes a scratch-tagged
key and asserts it lands; a second test runs after the first's teardown
and asserts no rows in the scratch namespace survived.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 4 of the management-endpoints behavior-pinning effort. New
``actors.py`` defines the actor enum + seeds an immutable world (2 orgs,
2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-``
prefix so the rows are identifiable in psql and ``_wipe_world`` is
targeted.
Each actor key is created with its cleartext form generated locally and
its hashed form (via ``litellm.proxy.utils.hash_token``) stored in
``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts
the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and
the service-account metadata flag are all set on the seeded rows so the
auth layer resolves the same scopes a real proxy would.
The session-scoped ``world`` fixture re-seeds at session start (idempotent
via wipe-then-create), and the smoke test confirms each of the 8 actor
keys can call ``/key/info`` on itself and receive its own row back.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 3 of the management-endpoints behavior-pinning effort. The fixture now
enters the real FastAPI lifespan (proxy_startup_event) instead of just calling
initialize() — that is where prisma_client is connected, password migration is
kicked off, and the rest of the startup wiring runs.
Tests pin the loop to the session scope so the AsyncClient created in the
session fixture and the prisma connection opened in the lifespan share the
same loop as the test bodies.
New de-risk smoke: POST /key/generate with the master key returns 200, the
returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and
the cleartext token is never stored. Proves auth + handler + helper + prisma
all wire together end-to-end against a real Postgres.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir
tests/proxy_behavior/management/ outside every existing pytest glob.
conftest.py initialises the proxy app once per session against the DATABASE_URL
the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process
ASGITransport. The one smoke test asserts /health/liveliness returns 200, which
exercises the full FastAPI middleware stack against a real app — no mocks.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
