Docker-executor jobs:
- Consolidate base images on cimg/python:3.12. Jobs previously on
3.11 (26 jobs), 3.9 (1 historical: upload-coverage), and an
incidental 3.13.1 (litellm_assistants_api_testing) now use 3.12.
- installing_litellm_on_python_3_13 keeps cimg/python:3.13.1 as its
explicit "latest Python supported" install-check matrix job.
Machine-executor jobs:
- Delete the miniconda install step from 10 jobs. uv now manages
Python directly: uv sync --python 3.12 auto-downloads a
python-build-standalone interpreter if the ubuntu-2204 base
image's default python doesn't match.
- Remove 37 "if [ -f conda.sh ]; then conda activate myenv" wrappers
and 2 unconditional conda activate blocks left behind from the
conda days.
- proxy_build_from_pip_tests keeps its 3.13 target (it was
conda create -n myenv python=3.13) via uv sync --python 3.13.
Net: -301 lines.
Add a start_postgres command parameterized on db_name (default
circle_test) that runs the postgres-db container and waits for port
5432 to accept connections. Replace all 11 inline docker run /
wait_for_service blocks with a single - start_postgres call.
The helm chart test overrides db_name to litellm_test; everything
else uses the default.
One of the 11 sites previously used a bespoke pg_isready loop instead
of wait_for_service; it now goes through the same TCP-probe path
everyone else uses, which is sufficient for test ordering purposes.
Net: -112 lines.
Add a single install_uv command in the commands: section that encodes
the uv version (0.10.9) and its SHA256 in one place, then replace all
42 inline curl|sha256|install blocks across every job that needs uv.
setup_litellm_test_deps now calls install_uv too, so the shared
test-dep bootstrap goes through the same path.
Bumping uv version or SHA is now a one-line change instead of 43.
Net: -203 lines.
- Remove mypy_linting job (GHA test-linting.yml already runs this)
- Remove three redundant "Install curl" apt-get steps (curl is
already present on the ubuntu-2204 machine image and used
successfully earlier in each affected job)
- Dedupe langfuse_logging_unit_tests filter block (6x copy of the
same two branch filters collapsed to 1)
- Pin all docker image references by @sha256 digest so builds stay
reproducible when upstream tags are updated:
cimg/python:3.9, 3.11, 3.12, 3.12-browsers, 3.13.1, cimg/node:20.19,
cimg/postgres:16.0, and postgres:14 used via docker run
Net: -62 lines, 49 image references pinned.
* add moonshot/kimi-k2.6 to model registry
* add moonshot/kimi-k2.6 to backup model registry
* add tests for moonshot/kimi-k2.6 model registry
* fix moonshot/kimi-k2.6 pricing and add reasoning support
* fix moonshot/kimi-k2.6 pricing and add reasoning support in backup
* update kimi-k2.6 tests: fix pricing, add tool_choice and reasoning checks
* fix: load kimi-k2.6 registry tests from local backup instead of remote cost map
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Waiting to run
Previously, members added to a team without an explicit per-member budget were
all linked to the same `litellm_budgettable` row referenced by the team's
`metadata.team_member_budget_id`. Updating one member's budget via
`/team/member_update` mutated the shared row and silently changed every other
member's budget too.
Now both write paths produce a private, per-member budget:
- `add_new_member` clones the team's default budget into a fresh row when a
member is added without `max_budget_in_team`/`allowed_models`. If no team
default exists, the membership is created with no budget.
- `_upsert_budget_and_membership` detects when an existing membership still
points at the team's default budget id and clones-on-write, relinking the
membership to the new private budget before applying the update.
- `team_member_update` reads `team_member_budget_id` from team metadata and
passes it through so the helper can make this distinction.
Adds unit tests for clone-on-write, in-place update of a private budget, and
the no-default-no-budget add path.
Made-with: Cursor
Addresses two further Greptile findings:
- `_warn_if_db_ahead_of_head` only caught `psycopg.OperationalError`.
Non-connection DB errors (e.g. `InsufficientPrivilege` / 42501 if the
runtime DB user lacks SELECT on `_prisma_migrations`) would propagate
uncaught and crash startup — contradicting the docstring's
"informational only, never blocks" guarantee. Widen the catch to
`psycopg.DatabaseError` so all DB-layer errors are swallowed.
- In the P3009 and P3018 idempotent-recovery paths, the call to
`_resolve_specific_migration(name)` was not wrapped in its own
try/except. Being inside an active `except CalledProcessError`
handler, a new `CalledProcessError` from the resolve call would NOT
re-enter the same handler — it would propagate out as
`CalledProcessError`, past `proxy_cli.py`'s `except RuntimeError`,
crashing startup with an unhandled traceback instead of the intended
clean `sys.exit(2)`. Wrap both call sites to convert to RuntimeError.
Adds unit tests for both behaviors.
- Open the psycopg connection in `_warn_if_db_ahead_of_head` with
autocommit=True. Without it, psycopg3's `with conn` calls COMMIT on
clean exit, which fails after the `UndefinedTable` (fresh-DB) branch
left the transaction in an aborted state — crashing first-run startups.
- Wrap the v2 `prisma db push` path in try/except and raise RuntimeError
on CalledProcessError/TimeoutExpired. Otherwise these propagate past
proxy_cli.py's `except RuntimeError` as unhandled tracebacks.
- Reword the loop-exhaustion error to cover the non-timeout exit path
(repeated P3005/P3009/P3018 idempotent-recovery `continue`s), not
just persistent timeouts.
Adds a unit test for the db_push error wrapping.
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
Adds end-to-end CI coverage for `--use_v2_migration_resolver` via a new
job `installing_litellm_on_python_v2_migration_resolver`:
- Clones the pytest smoke path from `installing_litellm_on_python` but
uses a local Postgres sidecar instead of the shared DB to prevent
collisions with the v1 variant.
- Runs only the new `test_litellm_proxy_server_config_no_general_settings_v2_resolver`
which spawns the proxy with `--use_v2_migration_resolver` and smoke-tests
`/health/liveliness` and `/chat/completions`.
Refactors `test_basic_python_version.py`:
- Extracts the proxy spawn + smoke-test body into `_run_proxy_server_smoke_test`
so the v1 and v2 tests share the same code path.
- The existing `test_litellm_proxy_server_config_no_general_settings` is
now a thin wrapper that passes no extra args (v1 default, unchanged).
- Adds `..._v2_resolver` variant that passes `--use_v2_migration_resolver`.
The existing `installing_litellm_on_python` / `installing_litellm_on_python_3_13`
jobs filter out the v2 variant via `-k "not v2_resolver"` so they keep
running only against their shared DB, unchanged behavior.
Default behavior (v1) is unchanged. Users who have seen schema thrashing
during rolling deploys can opt into the v2 resolver with
`--use_v2_migration_resolver`.
Why v2 is safer:
- Runs `prisma migrate deploy` only.
- Recovers from P3005 (baseline) and idempotent P3009/P3018 errors, same
as v1.
- Never calls `_resolve_all_migrations`, which generates a schema diff
between the live DB and the shipped schema.prisma and applies it via
`prisma db execute`. That path bypassed every migration's SQL and was
the root cause of thrashing when two LiteLLM versions contended for
the same DB.
- Logs a non-blocking warning when the DB has migrations applied that
are newer than anything this build ships (ahead-of-HEAD). It does not
refuse to start — many users have unusual ledger state from past
thrashing, and blocking startup would be a breaking change.
Also prints a message on startup when the default (v1) resolver is in
use, pointing operators at the opt-in flag.
Adds unit tests covering the v2 fail-fast paths, the stripping of
Prisma-specific query params from DATABASE_URL (needed for psycopg),
the timestamp helpers, and pins the default: v1 still invokes
`_resolve_all_migrations`, v2 must not.
- _find_destructive_statements: add DROP INDEX to the docstring (the
regex already detects it; only the docstring lagged).
- create_migration: correct the base_branch default documented in the
docstring from "main" to "litellm_internal_staging".
Generating a migration from a stale branch could silently emit DROP
COLUMN for columns the stale branch did not know about, and the
script would write that SQL to a new migration file with no warning.
Adds two guards to ci_cd/run_migration.py:
- Branch freshness check: fetches origin/<base-branch> and exits 3 if
HEAD is behind. Default base is litellm_internal_staging. New
flags: --base-branch, --skip-freshness-check.
- Destructive guard: refuses (exit 2) if the generated diff contains
DROP COLUMN / DROP TABLE / DROP INDEX, unless --allow-destructive
is passed.
Refusal banners include guidance and an explicit callout instructing
AI agents not to auto-bypass the flags. Also treats Prisma's
"-- This is an empty migration." output as a no-op rather than
writing an empty file.
Updates litellm-proxy-extras/migration_runbook.md with the new
workflow, flag documentation, and agent warnings.
Two fail-safes for the /v1/messages → Bedrock Invoke pass-through so new
Anthropic-only extensions Claude Code starts sending can't reach Bedrock
and trigger a 400 "Extra inputs are not permitted":
1. Top-level body fields are filtered to a typed allowlist. New
`BedrockInvokeAnthropicMessagesRequest` TypedDict (in
`litellm/types/llms/bedrock.py`) captures the Bedrock Invoke Anthropic
Messages body schema; the runtime allowlist is derived from its
`__annotations__` so the type and the filter can't drift. Anchored to
the AWS reference page in docstrings + transform comment. An
exact-set test pins the resolved allowlist so any future edit forces
conscious review.
Drops context_management, output_config, speed, mcp_servers,
container, inference_geo, internal litellm_metadata, and any future
Anthropic addition. output_format stays as an active inline-schema
conversion (not just a strip).
2. The anthropic-beta header list is filtered + transformed against the
bedrock mapping for ALL betas, not just auto-injected ones. The
previous code union'd user-provided betas back in unfiltered, so a
client on a new Anthropic-direct beta (e.g. advisor-tool-…,
context-management-…) could still pin the request to fail. In a proxy
context the client can't know the backend is Bedrock; the provider
mapping is authoritative. User-provided drops are logged at WARNING
so intentional overrides leave a breadcrumb.
Updates one existing test that happened to assert on the old buggy
pass-through (it used output-128k-2025-02-19, which is null in the
bedrock mapping and would 400 at runtime); rewrote it against a
bedrock-supported beta.
Scope: messages/invoke only. The same user-beta bypass exists in
chat/invoke but that's a different code path with different
user-expectation trade-offs — follow-up.
Brings user personal budget and organization budget enforcement
in line with the existing key and team patterns, which already
read spend from the atomic cross-pod Redis counter.
Addresses review feedback on the snapshot approach:
1. Class-instance mutable state
The snapshot only covers primitives + collections + None. Class
instances (DualCache, LLMClientCache) weren't reset between tests,
so in-place cache mutations could leak. Can't deepcopy these — they
hold thread locks — but they expose flush_cache(). Collect every
module attribute whose value implements flush_cache() at conftest
import, and invoke it per-test alongside the snapshot restore.
2. Silent skips are now warnings
_snapshot_mutable_state and _restore_mutable_state previously
swallowed exceptions, so if a future attr gained a property without
a setter (or other non-round-trippable state), an isolation gap
would have no signal. Emit warnings.warn on each failure path.
3. Docstring
Explicitly documents what IS and IS NOT reset, and tells authors to
use monkeypatch.setattr() for in-place mutations of instances
without flush_cache() (ProxyLogging, JWTHandler, etc.).
The Build UI from source step used:
cp -r out/ ../../litellm/proxy/_experimental/out/
GNU cp (CircleCI's Ubuntu image, coreutils 8.32) interprets this as
copy the source directory as a CHILD of the destination when the
destination already exists — so the command silently created
litellm/proxy/_experimental/out/out/ instead of replacing the served
bundle at litellm/proxy/_experimental/out/*.
The proxy continued serving whatever bundle was checked in, so every
e2e_ui_testing run between this job's introduction (d09d98a70a,
2026-04-08) and the bundle-rebuild commit (de790fd273, 2026-04-18) was
effectively testing a STALE bundle — not the fresh build. That is why
the double-prefix regression (NEXT_PUBLIC_BASE_URL="ui/" combined with
networking.tsx reading the env var) was never caught in CI even though
the source contained the trigger the whole time: the bundle the proxy
served never picked up the source change.
Replace cp -r with rm + mv so the destination is cleanly swapped.
Verified end-to-end on an Ubuntu 22.04 / GNU coreutils 8.32 container:
- Before fix: fresh build has 9 "ui/" literals in chunks; after cp,
_experimental/out/* still has 0 (stale); _experimental/out/out/ is a
nested dir the proxy does not serve.
- After fix: _experimental/out/* has 9 "ui/" literals — the proxy now
serves the freshly built (broken, in this repro) bundle, so
globalSetup fails at login and every spec is blocked. Removing the
bug from .env.production and rebuilding brings the count back to 0
and the suite passes.
No spec changes, no fixtures, no new infrastructure. The existing
Playwright suite already catches this class of regression via the
login flow in globalSetup; it just needs the CI to actually hand it
the freshly built bundle.
The previous snapshot only tracked list/dict/set values. Tests mutate
scalar module attrs too — master_key, premium_user, prisma_client — and
importlib.reload used to reset those implicitly. Under the snapshot
approach they were leaking between tests, so test_active_callbacks
failed in CI with "No api key passed in." once an earlier test left
master_key set to sk-1234.
Expand the snapshot to cover primitives (str/int/float/bool/bytes/tuple)
and None-valued attributes. Complex object instances are still skipped
to avoid deepcopy issues.
Add regression tests that mock make_bedrock_api_request and verify
input_type=request uses source=INPUT with user messages, and
input_type=response uses source=OUTPUT with synthetic ModelResponse.
Made-with: Cursor
BedrockGuardrail.apply_guardrail hardcoded source="INPUT" regardless of the
input_type parameter. On the non-streaming post-call path (unified_guardrail
-> OpenAIChatCompletionsHandler.process_output_response -> apply_guardrail),
the model response text was sent to Bedrock as INPUT, so guardrail policies
configured for Output (e.g. PII/NAME blocking) returned action=NONE and the
response passed through unblocked. The streaming path was unaffected because
it calls make_bedrock_api_request(source="OUTPUT", ...) directly.
Map input_type to the correct Bedrock source ("request" -> INPUT,
"response" -> OUTPUT) and build a synthetic ModelResponse for the OUTPUT
path so _create_bedrock_output_content_request produces the correct payload.
Made-with: Cursor
tests/proxy_unit_tests/conftest.py was calling importlib.reload(litellm) in an
autouse function-scoped fixture, which cost ~17s per test because it re-ran
the full litellm __init__ import chain. With 400+ proxy unit tests, this was
the single biggest driver of CI wall time — 18 of the top 20 slowest durations
in a typical run were just the 17s fixture setup.
Replace the reload with a snapshot-and-restore approach: snapshot the mutable
lists/dicts/sets on litellm and litellm.proxy.proxy_server once at conftest
import, then deep-copy that snapshot back before each test. Callback lists,
caches, router state, etc. still get reset between tests, but the expensive
import chain only runs once per worker.
Local measurement on test_proxy_utils.py: 188 tests in 3.50s (previously took
~15 minutes of CI wall time on a single worker).
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
The "remaining" proxy-db job was consistently timing out at ~98% because
--dist=loadscope pins every test in test_proxy_utils.py (168+ parametrized
tests) to a single xdist worker. 7 workers finished their files in ~15
minutes, then one worker ran alone for another 8+ minutes and hit the
30-minute job cap.
Give test_proxy_utils.py its own matrix entry so its tests spread across
all 8 workers, and add it to the "remaining" ignore list.
