Add Playwright E2E tests covering proxy admin team and key management
workflows, with a self-contained test runner and CircleCI integration.
Tests cover: create team, invite user, edit/delete team members, create
key in team, regenerate key, update TPM/RPM limits, delete key, and
verify internal user keys are visible.
Infrastructure: run_e2e.sh builds the UI from source before starting
the proxy, ensuring tests always run against the latest UI changes.
Added data-testid attributes to key UI components for reliable selectors.
Redis caching unit tests (test_dual_cache, test_redis_batch_optimizations,
test_router_utils) required Redis secrets that should live in CircleCI.
- Add redis_caching_unit_tests job to CircleCI config
- Delete test-unit-caching-redis.yml GHA workflow
- Remove all Redis plumbing (inputs, secrets, env vars) from
_test-unit-services-base.yml and its callers
- Move os and MCP_STDIO_ALLOWED_COMMANDS imports to module level in mcp_server_manager.py
- Move MCP_STDIO_ALLOWED_COMMANDS import to module level in _types.py
- Change defense-in-depth warning to HTTPException 403 for legacy non-allowlisted commands
- Ensures arbitrary command execution is blocked for both new and legacy MCP servers
Addresses Greptile review comments:
- P2: Inline imports violate CLAUDE.md style guide
- P1 security: Defense-in-depth should block, not warn, for legacy commands
Made-with: Cursor
- Defense-in-depth: warn instead of hard-fail for legacy servers
- Move os import to module level in _types.py
- Document args residual risk in allowlist comment
- Add UpdateMCPServerRequest allowlist test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add command allowlist for MCP stdio transport to prevent RCE via
/mcp-rest/test/* endpoints. Restrict test endpoints to PROXY_ADMIN
role. Fix docker/README.md MASTER_KEY -> LITELLM_MASTER_KEY.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace static Modal.confirm with DeleteResourceModal so attachment delete reliably triggers the API call. Add a regression test covering the confirm->delete flow.
Made-with: Cursor
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
The /v2/key/info endpoint was missing response filtering that
the v1 /key/info endpoint already had. This aligns the two
endpoints so v2 applies the same per-key permission checks and
strips internal fields from the response. Also fixes the
key_aliases query path to resolve aliases before querying.
Allow JWT tokens matching routing_overrides to use OAuth2 introspection without enabling global OAuth2 while keeping OAuth2 routing limited to LLM/info routes. Add regression coverage for management-route boundary and tighten opaque-token assertions; update docs to reflect selective-mode route scope.
Made-with: Cursor
The .npmrc file (ignore-scripts=true, min-release-age=3d) is temporarily
removed during the Docker build since lifecycle scripts are needed by
npm ci. However, the unconditional `mv` fails when the build context
doesn't include .npmrc (e.g. when LiteLLM is vendored in a subdirectory).
Make all .npmrc mv operations conditional. This is safe because npm ci
already installs from package-lock.json with pinned versions and
integrity hashes.
PR #25258 changed _cleanup_stale_managed_objects from update_many to
execute_raw via _expire_stale_rows, but the tests were not updated.
The tests now mock _expire_stale_rows on the instance and assert
update_many calls only for job completion, not stale cleanup.
- team-admin: assert Admin Settings is not visible (role-specific check)
- proxy-admin: use users[Role.ProxyAdmin].password from constants instead of duplicating the env var fallback inline
Pin all cosign public key references to the immutable commit hash
(0112e53) that first introduced the key, instead of fetching it from
the release tag. This addresses the concern that an attacker with push
access could replace the key on main/tags and re-sign tampered images.
Docs now show two verification methods: commit hash (recommended) and
release tag (convenience), with explanation of why the hash is stronger.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: batch-limit stale managed object cleanup to prevent 300K row UPDATE (#25257)
* Add STALE_OBJECT_CLEANUP_BATCH_SIZE constant
Configurable batch limit (default 1000) for stale managed object cleanup,
preventing unbounded UPDATE queries from hitting 300K+ rows at once.
* Batch-limit stale managed object cleanup with single bounded SQL query
Two fixes to _cleanup_stale_managed_objects:
1. Replace unbounded update_many with a single execute_raw using a
subquery LIMIT, capping each poll cycle to STALE_OBJECT_CLEANUP_BATCH_SIZE
rows. Zero rows loaded into Python memory — everything stays in Postgres.
Uses the same PostgreSQL raw-SQL pattern as spend_log_cleanup.py
(the proxy requires PostgreSQL per schema.prisma).
2. Extract _expire_stale_rows as a separate method for testability.
Keeps the file_purpose='response' filter to avoid incorrectly expiring
long-running batch or fine-tune jobs that legitimately exceed the
staleness cutoff.
* docs: add STALE_OBJECT_CLEANUP_BATCH_SIZE to env vars reference
* test: remove deprecated embed-english-v2.0 cohere embedding tests
Adds a new endpoint to bulk-update team_member_permissions across
teams. Supports apply_to_all_teams (with cursor-based pagination)
or a specific list of team_ids. Merges new permissions into each
team's existing set rather than overwriting.
Also fixes test isolation bug in test_get_prompt_info_by_base_id
where leaked prisma_client state from other tests caused a
TypeError on await.
* Remove redundant matrix unit test workflow
All test paths in test-litellm-matrix.yml are fully covered by the
newer semantic unit test workflows (test-unit-*.yml), making the
matrix workflow redundant CI spend.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add Codecov coverage reporting to semantic unit test workflows
Add coverage collection (--cov) and Codecov OIDC upload to both
reusable base workflows and all 12 caller workflows, replacing the
coverage reporting that was previously only in the matrix workflow.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Move id-token/pull-requests permissions to job level for multi-job workflows
For workflows with multiple jobs (llm-providers, proxy-db), move
id-token: write and pull-requests: write from workflow level to job
level so permissions are scoped to only the jobs that need them.
Removes zizmor inline suppressions that were masking the issue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [Docs] Enforce Black formatting in contributor docs
Black formatting is now enforced in CI. Update CLAUDE.md, AGENTS.md,
and CONTRIBUTING.md to instruct contributors and AI agents to run
`poetry run black .` before committing, and add VS Code setup guidance.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: fixes
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The proxy_e2e_azure_batches_tests workflow is consistently flaky and
does not provide reliable signal on whether changes break anything.
Remove the workflow from both CircleCI and GitHub Actions, along with
the test directory it exclusively used.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
* fix(ui): resolve login redirect loop when reverse proxy adds HttpOnly to cookies
When LiteLLM is behind nginx-ingress or similar with security-hardened
configs, the reverse proxy adds HttpOnly to all Set-Cookie headers. This
makes the JWT token unreadable by JavaScript, causing an infinite login
redirect loop. Fix by returning the JWT token in the /v2/login response
body so the frontend can set a JS-accessible cookie directly.
Fixes#19663
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address Greptile review feedback
- Add window guard to setTokenCookie for SSR consistency with clearTokenCookies
- Add SSR test for window undefined case
- Add code comment explaining why JWT is included in response body
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address second round of Greptile review feedback
- Add loginCall integration tests verifying setTokenCookie is called with
token and skipped when absent (backward-compatibility path)
- Use encodeURIComponent/decodeURIComponent in setTokenCookie/getCookie
for defense-in-depth against non-standard token formats
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/utils/cookieUtils.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update ui/litellm-dashboard/src/utils/cookieUtils.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(ui): use sessionStorage instead of cookie for login token storage
Replace setTokenCookie (which is a no-op when reverse proxy adds HttpOnly)
with storeLoginToken using sessionStorage. Add sessionStorage fallback to
getCookie so the token is found even when the cookie is HttpOnly. Also handle
'=' in cookie values with .slice(1).join("=") and clear sessionStorage on
logout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ui): use shared getCookie in page.tsx and user_dashboard.tsx
Replace local getCookie functions in page.tsx and user_dashboard.tsx
with the shared one from cookieUtils that has the sessionStorage
fallback. Without this, the HttpOnly cookie fix was incomplete —
page.tsx (the dashboard entry point) could not read the token,
causing the redirect loop to persist.
Also scope the sessionStorage fallback to the "token" key only,
and clear sessionStorage in page.tsx deleteCookie.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ui): scope deleteCookie sessionStorage cleanup to token key only
Also document the sessionStorage cross-tab trade-off: per-tab scope
means users behind an HttpOnly proxy must log in once per tab, but
this is intentional to avoid localStorage XSS exposure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/utils/cookieUtils.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* style: remove stray double blank line in user_dashboard.tsx
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ui): guard storeLoginToken against empty/whitespace-only tokens
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ui): preserve sessionStorage token across beforeunload clear
The existing beforeunload handler calls sessionStorage.clear() to
flush cached UI data on page refresh. This also wiped the token
stored by storeLoginToken, re-introducing the redirect loop after
any page refresh in the HttpOnly proxy scenario. Now the token is
saved and restored across the clear.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ui): set JS-accessible cookie at /ui path as HttpOnly workaround
sessionStorage alone is unreliable. Also set the token via
document.cookie at path=/ui — nginx only adds HttpOnly to server-set
Set-Cookie headers, so a JS-set cookie is always readable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ui): use dynamic cookie path based on server_root_path
Hardcoded path=/ui breaks when LiteLLM is deployed with a custom
server_root_path. Now derives the cookie path from serverRootPath
so it works at /ui, /myapp/ui, etc.
Also reuse clearTokenCookies() in deleteCookie() to avoid duplication.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(ui): remove circular dependency in cookieUtils.ts
Derive the UI cookie path from window.location.pathname instead of
importing serverRootPath from networking.tsx. This breaks the
cookieUtils → networking → cookieUtils cycle that could cause
serverRootPath to be undefined under certain bundler configurations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ui): harden getUiCookiePath regex and add missing tests
- Use regex /\/ui(?=\/|$)/ to match "/ui" only as a full path segment,
preventing false matches on paths like "/my-ui-tool/login".
- Add unit tests for storeLoginToken empty/whitespace guard and
cookie-at-/ui-path behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: fix Black formatting in audit_logs.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix CI: formatting, test params, remove token from login JSON
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reformat with Black 23.x to match CI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: keep token in login JSON body for UI storeLoginToken flow
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use storeLoginToken in exchangeLoginCode, add credentials include
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* revert: remove unrelated changes from HttpOnly cookie fix branch
Reset files not related to the login cookie fix back to main:
- prometheus.py, bedrock converse, guardrail handler
- auth_checks.py, reset_budget_job.py, audit_logs.py
- test_user_api_key_auth.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "revert: remove unrelated changes from HttpOnly cookie fix branch"
This reverts commit 0684a1e275.
* Revert "fix: use storeLoginToken in exchangeLoginCode, add credentials include"
This reverts commit 866405f443.
* Revert "fix: keep token in login JSON body for UI storeLoginToken flow"
This reverts commit 086c41640c.
* Revert "fix: reformat with Black 23.x to match CI"
This reverts commit b2c3334c88.
* Revert "fix CI: formatting, test params, remove token from login JSON"
This reverts commit 2905d47bd4.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
* docs(blog): add cosign Docker image verification instructions
Add steps for verifying Docker images with cosign to three security blog posts:
CI/CD v2, Security Townhall, and Security Update.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(proxy): add cosign verification to Docker/Helm/Terraform deploy page
Add image signature verification steps to the main deployment doc so
users pulling Docker images know how to verify them with cosign.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: fixes
* Update index.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* [Docs] Scope cosign signing docs to GHCR and specify starting version
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [Docs] Add starting version callout to ci_cd_v2 blog post
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
