The Windows uv install step was piping a remote install.ps1 into
Invoke-Expression without any integrity check, while the Linux
install steps (install_uv command, line 89) download to a file,
verify SHA-256 against a hardcoded digest, and only then execute.
Bring the Windows path to the same pattern.
Also hardcode the kubectl v1.31.4 checksum in helm_chart_testing
instead of fetching kubectl.sha256 from the same origin as the
binary — if dl.k8s.io were ever to serve a tampered pair, a
co-hosted checksum provides no additional integrity.
The same branch filter block appeared 46 times in the workflow
declaration:
filters:
branches:
only:
- main
- /litellm_.*/
And the same pinned Python docker image appeared 29 times in jobs:
- image: cimg/python:3.12@sha256:9c796c...
auth:
username: ${DOCKERHUB_USERNAME}
password: ${DOCKERHUB_PASSWORD}
Replace with YAML anchors declared at first use:
- `&main_branches` on using_litellm_on_windows's filters block;
all other job entries reference it as `filters: *main_branches`.
- `&python312_image` on local_testing_part1's first docker image
entry; all other jobs reference `- *python312_image`, including
the multi-image jobs (auth_ui_unit_tests,
installing_litellm_on_python_v2_migration_resolver) which keep
their postgres sidecar entry inline afterwards.
Net result: one place to change when the image digest rolls or
the branch-filter convention changes. No behavior change — YAML
anchor resolution produces identical config at parse time.
Also adds Docker Hub auth block to upload-coverage (previously
pulled anonymously). No functional difference for a public
image, but avoids Docker Hub rate limits now that we reuse the
same entry.
Clean out copy-paste debug and workaround lines that serve no
purpose:
- `pwd && ls` echoes at the top of 30 "Run tests" steps (CCI
already logs working_directory on every step).
- "Show git commit hash" in local_testing_part1/part2 and
langfuse_logging_unit_tests (CCI shows the SHA in every job
header).
- "Verify Docker is available" stubs in 6 machine-executor
jobs (machine executors always have Docker).
- `sudo systemctl restart docker` in proxy_store_model_in_db_tests
(one-off workaround; not used anywhere else).
- Duplicated Black formatting step in local_testing_part1 and
local_testing_part2 — Black runs in the lint job, no reason to
run it again here.
- Second back-to-back `helm test litellm --logs` invocation in
helm_chart_testing (one call is enough).
No behavior change — these are all log-only or no-op steps.
Consolidate 6 distinct cache-key prefixes (v2-dependencies-,
v1-router-testing-deps-, v1-router-unit-deps-, v1-llm-translation-deps-,
v1-llm-responses-deps-, v3-litellm-uv-deps-, ui-e2e-py-deps-v2-) onto a
single v1-uv-cache-<uv.lock checksum> key shared across all Python jobs.
Cache only ~/.cache/uv (the content-addressed uv download cache,
hash-verified against uv.lock at install time). Drop ./.venv,
~/.local/{bin,lib}, and /home/circleci/.{pyenv,local} from cache paths.
~/.cache/uv is the only path uv sync needs to avoid re-downloading from
PyPI; everything else is rebuilt each run from that verified cache.
Remove partial-prefix restore-keys fallbacks — cache either hits exactly
on the uv.lock hash or rebuilds cleanly.
First run after merge will cold-miss on the new key; subsequent runs
hit the unified cache.
Replace the calibration step (one request + 10-minute poll) with an
independent ground truth computed from response usage via
litellm.cost_per_token. All N requests are made up front, so a single
dropped Redis write no longer kills the test.
Add /health/readiness checks at test start and on poll timeout so the
failure message surfaces proxy state (db, cache) instead of "calibration
timed out".
Set PROXY_BATCH_WRITE_AT=2 in the spend tracking CI job to shorten the
scheduler flush window.
Principle: GHA handles work that doesn't need external API keys; CCI
stays for integration tests that hit real API endpoints.
Four CCI jobs moved to new or extended GHA workflows:
1. check_code_and_doc_quality (was 25 runs: ruff + import-safety +
21 code_coverage_tests + 3 documentation_tests + circular-imports).
- The 21 tests/code_coverage_tests/*.py scripts and the 3
tests/documentation_tests/*.py scripts run in the new
.github/workflows/test-code-quality.yml workflow.
- ruff, import-safety, and circular-imports were already run by
.github/workflows/test-linting.yml — no new migration needed.
- The 3 documentation_tests scripts read
docs/my-website/docs/proxy/config_settings.md. Since docs have
moved to BerriAI/litellm-docs, the GHA workflow checks out that
repo and symlinks docs/my-website -> the checkout so the
existing hardcoded paths resolve without touching the scripts.
The stale local docs/my-website/ copy in this repo will be
removed in a separate PR.
2. semgrep (custom-rule SAST against .semgrep/rules).
- New .github/workflows/test-semgrep.yml.
3. installing_litellm_on_python + installing_litellm_on_python_3_13
(pip install compat checks on Python 3.12 and 3.13).
- New .github/workflows/test-install-litellm.yml as a matrix job.
- 3.12 run also verifies litellm_enterprise import; 3.13 run
skips that check (matches previous CCI behavior).
- installing_litellm_on_python_v2_migration_resolver stays in CCI
because it requires a postgres service.
CCI .circleci/config.yml: -112 lines, 4 jobs and their workflow refs
removed.
Follow-up cleanup after an independent review pass surfaced a few
loose ends:
- Delete a 6x-duplicated filter block in litellm_mapped_tests_proxy_part2
(same kind of copy-paste residue we fixed earlier in
langfuse_logging_unit_tests).
- Delete the empty "Install Semgrep" run step in the semgrep job — the
command body was empty because semgrep is installed on-demand via
uv tool run in the next step.
- Standardize machine-executor image: one job was on ubuntu-2204:2023.10.1
while build_docker_database_image was already on ubuntu-2204:2024.04.1.
Bumped everything to 2024.04.1.
- Remove the legacy "version: 2" inside the workflows: block — CircleCI
2.1 top-level already declares the version.
- Drop `{{ checksum ".circleci/config.yml" }}` from cache keys (13 sites).
It was busting the cache on every unrelated config edit; the uv.lock
checksum alone is the right dependency cache key.
- Add partial-restore fallbacks to every restore_cache with a single
templated key (10 sites). Jobs now fall back to the latest cache with
a matching prefix if the exact uv.lock hash isn't cached yet.
Net: -14 lines.
Docker-executor jobs:
- Consolidate base images on cimg/python:3.12. Jobs previously on
3.11 (26 jobs), 3.9 (1 historical: upload-coverage), and an
incidental 3.13.1 (litellm_assistants_api_testing) now use 3.12.
- installing_litellm_on_python_3_13 keeps cimg/python:3.13.1 as its
explicit "latest Python supported" install-check matrix job.
Machine-executor jobs:
- Delete the miniconda install step from 10 jobs. uv now manages
Python directly: uv sync --python 3.12 auto-downloads a
python-build-standalone interpreter if the ubuntu-2204 base
image's default python doesn't match.
- Remove 37 "if [ -f conda.sh ]; then conda activate myenv" wrappers
and 2 unconditional conda activate blocks left behind from the
conda days.
- proxy_build_from_pip_tests keeps its 3.13 target (it was
conda create -n myenv python=3.13) via uv sync --python 3.13.
Net: -301 lines.
Add a start_postgres command parameterized on db_name (default
circle_test) that runs the postgres-db container and waits for port
5432 to accept connections. Replace all 11 inline docker run /
wait_for_service blocks with a single - start_postgres call.
The helm chart test overrides db_name to litellm_test; everything
else uses the default.
One of the 11 sites previously used a bespoke pg_isready loop instead
of wait_for_service; it now goes through the same TCP-probe path
everyone else uses, which is sufficient for test ordering purposes.
Net: -112 lines.
Add a single install_uv command in the commands: section that encodes
the uv version (0.10.9) and its SHA256 in one place, then replace all
42 inline curl|sha256|install blocks across every job that needs uv.
setup_litellm_test_deps now calls install_uv too, so the shared
test-dep bootstrap goes through the same path.
Bumping uv version or SHA is now a one-line change instead of 43.
Net: -203 lines.
- Remove mypy_linting job (GHA test-linting.yml already runs this)
- Remove three redundant "Install curl" apt-get steps (curl is
already present on the ubuntu-2204 machine image and used
successfully earlier in each affected job)
- Dedupe langfuse_logging_unit_tests filter block (6x copy of the
same two branch filters collapsed to 1)
- Pin all docker image references by @sha256 digest so builds stay
reproducible when upstream tags are updated:
cimg/python:3.9, 3.11, 3.12, 3.12-browsers, 3.13.1, cimg/node:20.19,
cimg/postgres:16.0, and postgres:14 used via docker run
Net: -62 lines, 49 image references pinned.
Adds end-to-end CI coverage for `--use_v2_migration_resolver` via a new
job `installing_litellm_on_python_v2_migration_resolver`:
- Clones the pytest smoke path from `installing_litellm_on_python` but
uses a local Postgres sidecar instead of the shared DB to prevent
collisions with the v1 variant.
- Runs only the new `test_litellm_proxy_server_config_no_general_settings_v2_resolver`
which spawns the proxy with `--use_v2_migration_resolver` and smoke-tests
`/health/liveliness` and `/chat/completions`.
Refactors `test_basic_python_version.py`:
- Extracts the proxy spawn + smoke-test body into `_run_proxy_server_smoke_test`
so the v1 and v2 tests share the same code path.
- The existing `test_litellm_proxy_server_config_no_general_settings` is
now a thin wrapper that passes no extra args (v1 default, unchanged).
- Adds `..._v2_resolver` variant that passes `--use_v2_migration_resolver`.
The existing `installing_litellm_on_python` / `installing_litellm_on_python_3_13`
jobs filter out the v2 variant via `-k "not v2_resolver"` so they keep
running only against their shared DB, unchanged behavior.
The Build UI from source step used:
cp -r out/ ../../litellm/proxy/_experimental/out/
GNU cp (CircleCI's Ubuntu image, coreutils 8.32) interprets this as
copy the source directory as a CHILD of the destination when the
destination already exists — so the command silently created
litellm/proxy/_experimental/out/out/ instead of replacing the served
bundle at litellm/proxy/_experimental/out/*.
The proxy continued serving whatever bundle was checked in, so every
e2e_ui_testing run between this job's introduction (d09d98a70a,
2026-04-08) and the bundle-rebuild commit (de790fd273, 2026-04-18) was
effectively testing a STALE bundle — not the fresh build. That is why
the double-prefix regression (NEXT_PUBLIC_BASE_URL="ui/" combined with
networking.tsx reading the env var) was never caught in CI even though
the source contained the trigger the whole time: the bundle the proxy
served never picked up the source change.
Replace cp -r with rm + mv so the destination is cleanly swapped.
Verified end-to-end on an Ubuntu 22.04 / GNU coreutils 8.32 container:
- Before fix: fresh build has 9 "ui/" literals in chunks; after cp,
_experimental/out/* still has 0 (stale); _experimental/out/out/ is a
nested dir the proxy does not serve.
- After fix: _experimental/out/* has 9 "ui/" literals — the proxy now
serves the freshly built (broken, in this repro) bundle, so
globalSetup fails at login and every spec is blocked. Removing the
bug from .env.production and rebuilding brings the count back to 0
and the suite passes.
No spec changes, no fixtures, no new infrastructure. The existing
Playwright suite already catches this class of regression via the
login flow in globalSetup; it just needs the CI to actually hand it
the freshly built bundle.
Run auth_ui_unit_tests against a per-job cimg/postgres:16.0 sidecar
with DATABASE_URL pointing at localhost:5432, matching the pattern
used by e2e_ui_testing. Seed the schema via 'litellm --skip_server_startup
--use_prisma_db_push' so each run starts on a clean DB with the current
schema.prisma.
Six CI jobs create a miniconda env with python=3.9 before installing
the project; these jobs now fail resolution because the project
requires-python is >=3.10. Bump the conda env python to 3.10 to match
the new floor.
Workers in llm_translation_testing have been crashing mid-run with
"Not properly terminated" (OOM), even after bumping resource_class to
xlarge. Reduce xdist workers from 8 to 4 to lower peak memory, and add
--max-worker-restart=5 so a crashed worker is replaced instead of
failing the whole run.
publish_proxy_extras is superseded by PyPI trusted publishing (OIDC);
the CircleCI project no longer has PYPI_PUBLISH_* credentials configured.
prisma_schema_sync is a leaf smoke test with no dependents, and db push
against the current schema is already exercised by e2e_ui_testing.
helm lint is already run by the dedicated helm_chart_testing job.
Removes the need to install helm in a Python-only Docker container
and avoids piping an unverified remote script into bash.
1. check_code_and_doc_quality: install helm before running helm lint
(Docker image cimg/python:3.11 doesn't include helm)
2. proxy_store_model_in_db_tests: skip MCP tests on Python < 3.10
since MCP module isn't available and functions inside the
`if MCP_AVAILABLE:` block don't exist as module-level attributes
1. check_code_and_doc_quality: add PLR0915 ignore for sandbox_executor.py
2. auth_ui_unit_tests: add prisma generate step (entrypoint.sh only runs
migration, not client generation)
3. proxy_store_model_in_db_tests: move does_mcp_server_exist outside
`if MCP_AVAILABLE:` so it's importable on Python < 3.10
4. build_and_test: fix datetime.fromisoformat('...Z') on Python 3.9
(Z suffix support was added in 3.11)
5. proxy_logging_guardrails: fix container name typo my-app-2 -> my-app-3
6. upload-coverage: use `uv tool run` instead of `uv run --with` to avoid
resolving the full workspace (which fails for Python 3.14)
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
setup_litellm_enterprise_pip was running `uv sync --package litellm-enterprise`
which overwrites the shared .venv, stripping out pytest, prisma, and other
dev/test deps. Since litellm-enterprise is a workspace member it is already
installed by the main `uv sync --all-groups --all-extras`. Replace with a
verification-only import check.
Also prefix bare `ruff check` with `uv run --no-sync` since uv does not
auto-activate the venv.
- Replace `uv run --no-sync prisma generate` with `python -m prisma generate`
in proxy_part1, proxy_part2, and enterprise jobs (fixes spawn error)
- Migrate redis_caching_unit_tests from requirements.txt to uv sync
- Migrate e2e_ui_testing from requirements.txt to uv sync, replace bare
prisma/python calls with uv run equivalents
- Bump venv cache keys from v1 to v2 with config.yml checksum to bust
stale caches missing black and other dev dependencies
- Pin `pip==26.0.1` and `uv==0.10.9` in CCI jobs that used unpinned
`pip install uv` (redis_caching_unit_tests, ui_e2e_tests)
- Replace bare `prisma generate` with `uv run --no-sync prisma generate`
in proxy_part1, proxy_part2, and enterprise test jobs
- Remove duplicate `check=True` kwarg in test_basic_python_version.py
that caused TypeError with `_run_uv()` helper
* build: migrate packaging metadata to uv
* ci: move automation and local tooling to uv
* docker: migrate image builds and runtime setup to uv
* docs: update install and deployment guidance for uv
* chore: align auxiliary scripts and tests with uv
* test: harden test_litellm isolation
* fix: keep release and health check images self-contained
* build: pin uv tooling and health check deps
* test: isolate bedrock image request formatting from suite state
* test: cover sandbox executor requirements flow
* ci: fix circleci no-op command steps
* ci: fix circleci publish workflow parsing
* fix: stabilize remaining uv migration CI checks
* ci: increase matrix test timeout headroom
* fix: restore published docker and license coverage
* fix: restore proxy runtime build parity
* fix: restore proxy extras parity and venv migrations
* ci: persist uv path across circleci steps
* fix: keep psycopg binary in default test env
* docker: preserve prisma cache across stages
* test: run local proxy checks through uv python
* build: restore runtime deps moved into ci
* build: refresh uv lock after upstream merge
* fix: restore module import in test_check_migration after merge
The conflict resolution imported only the function but the test body
references check_migration as a module throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching
- Move google-generativeai, Pillow, tenacity back to ci group (they are
lazily imported and bloat the base SDK install needlessly)
- Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant
in Docker where system Node.js is already installed via apk)
- Remove all nodejs-wheel node replacement and venv npm patching blocks
from Dockerfiles since the wheel is no longer installed
- Add --no-default-groups to CodSpeed benchmark workflow so the benchmark
environment matches the old minimal pip install footprint
- Apply standard uv two-phase Docker pattern: copy metadata first, install
deps (cached layer), then copy source and install project
- Replace CircleCI enterprise no-op with proper uv sync command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate uv.lock after removing nodejs-wheel-binaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): use cache/restore instead of cache to prevent cache poisoning
The old workflow used actions/cache/restore (read-only). The uv migration
changed it to actions/cache (read-write), which zizmor flags as a cache
poisoning risk. Restore the safer read-only variant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert
The setup-uv action enables caching by default, which zizmor flags as a
cache poisoning risk. Disable it since we already use a read-only
cache/restore step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv cache in publish workflow
Silences zizmor cache-poisoning alert. Publishing workflow runs
infrequently on protected branches so caching adds no real benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(test): remove duplicate verbose_logger mock in test_check_migration
The logger was patched twice — first via mocker.patch() then via
mocker.patch.object(autospec=True). The second call fails because
autospec cannot inspect an already-mocked attribute. Remove the
redundant first patch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): free disk space before Docker build in test-server-root-path
The Dockerfile.non_root build ran out of disk on the CI runner. Remove
Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
- Add constants.ts with all required exports (key aliases, team IDs)
- Add fixtures/users.ts with all role definitions and storage paths
- Add fixtures/seed.sql for deterministic test database seeding
- Remove Firefox project from playwright config (only Chromium installed)
- Remove unused variable in teams.spec.ts
- Rename CircleCI job to e2e_ui_testing
Add Playwright E2E tests covering proxy admin team and key management
workflows, with a self-contained test runner and CircleCI integration.
Tests cover: create team, invite user, edit/delete team members, create
key in team, regenerate key, update TPM/RPM limits, delete key, and
verify internal user keys are visible.
Infrastructure: run_e2e.sh builds the UI from source before starting
the proxy, ensuring tests always run against the latest UI changes.
Added data-testid attributes to key UI components for reliable selectors.
Redis caching unit tests (test_dual_cache, test_redis_batch_optimizations,
test_router_utils) required Redis secrets that should live in CircleCI.
- Add redis_caching_unit_tests job to CircleCI config
- Delete test-unit-caching-redis.yml GHA workflow
- Remove all Redis plumbing (inputs, secrets, env vars) from
_test-unit-services-base.yml and its callers
The proxy_e2e_azure_batches_tests workflow is consistently flaky and
does not provide reliable signal on whether changes break anything.
Remove the workflow from both CircleCI and GitHub Actions, along with
the test directory it exclusively used.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove @neondatabase/api-client and neonctl to address CVE-2026-25639
(axios supply chain vulnerability). Pin all JS dependencies to exact
versions across all package.json files to prevent future supply chain
attacks via semver range resolution.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aioboto3 was listed as a dependency for async sagemaker calls but is not
imported anywhere in the codebase — async calls use httpx + botocore SigV4
instead. Removing it eliminates the unresolvable botocore version conflict
between boto3 and aiobotocore, along with all grep -v / --no-deps workarounds
across Dockerfiles and CI.
Also addresses Greptile review feedback: collapse redundant grpcio
python-version markers, bump pyproject.toml cryptography to 46.0.5 to
match Docker (GHSA-r6ph-v2qm-q3c2), and fix misleading .npmrc comment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
boto3==1.42.80 and aioboto3==15.5.0 have incompatible botocore version
ranges. No aioboto3 release supports botocore 1.42.x yet. Both uv and
pip 26.0.1 reject the resolution.
Fix: filter aioboto3 out of requirements.txt at install time, then
install aioboto3+aiobotocore with --no-deps to bypass resolution.
Added wrapt and aioitertools to requirements.txt as pinned transitive
deps of aiobotocore (skipped by --no-deps). Fixed pip stdin handling
(/dev/stdin). Applied to all 5 Dockerfiles and all CircleCI install
paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
boto3==1.42.80 and aioboto3==15.5.0 have incompatible botocore ranges.
Both uv and pip 26.0.1 reject the resolution. Fix: filter aioboto3 out
of requirements.txt at install time, then install aioboto3+aiobotocore
separately with --no-deps to bypass resolution. Removes uv-overrides.txt
which only partially addressed the conflict.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
boto3==1.42.80 requires botocore>=1.42.80 but aioboto3==15.5.0 (via
aiobotocore==2.25.1) requires botocore<1.40.62. No aioboto3 release
supports botocore 1.42.x yet. pip's lenient resolver handles this for
Docker builds, but uv's strict resolver rejects it in CI. Added
uv-overrides.txt to force botocore to match boto3 during uv installs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CircleCI had stale version pins (e.g. boto3==1.36.0, aioboto3==13.4.0) that
conflict with requirements.txt (boto3==1.42.80, aioboto3==15.5.0), causing
uv resolution failures. Updated all mismatched pins across config.yml and
.circleci/requirements.txt to match requirements.txt as the source of truth.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes 10 CircleCI jobs that are fully duplicated by GHA workflows:
- caching_unit_tests → test-unit-caching-redis.yml
- litellm_security_tests → test-unit-security.yml
- litellm_proxy_unit_testing_{key_generation,part1,part2} → test-unit-proxy-db.yml + test-unit-proxy-legacy.yml
- litellm_mapped_tests_{llms,core,litellm_core_utils,mcps,integrations} → test-unit-*.yml workflows
Also cleans up upload-coverage requires and workflow entries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
