Follow-up on review feedback: the previous commit had the builder
download the query engine into /app/.cache, then threw it away in
the runtime stage and re-downloaded into /root/.cache. That doubled
the build-time network fetch.
Remove PRISMA_BINARY_CACHE_DIR and XDG_CACHE_HOME from the builder
stage as well, so its prisma generate lands in /root/.cache with the
correct path layout on its own. Drop the runtime-stage prisma generate
and instead COPY --from=builder /root/.cache /root/.cache. Single
download, smaller image.
LiteLLM_BudgetTable is documented as "user-controllable params" and its
model_fields.keys() is used as the allowlist for extracting budget fields
from incoming API request bodies (management_helpers/utils.py:88,
organization_endpoints.py:112/255/537/549, project_endpoints.py:197/245/632,
customer_endpoints.py:598). Request models like NewOrganizationRequest
inherit from LiteLLM_BudgetTable, so anything on the base class becomes
user-settable — a caller could set budget_reset_at far in the future and
evade budget cycling.
Move budget_reset_at from the base class to LiteLLM_BudgetTableFull so it
appears on API responses without becoming writable, and type
LiteLLM_TeamMembership.litellm_budget_table as Union[Full, Base] so
Pydantic picks Full when the data has server-managed fields (/team/info
reads Prisma rows that include budget_reset_at and created_at) and Base
when callers construct with only user-settable fields (existing auth
tests and caches).
- Open the psycopg connection in `_warn_if_db_ahead_of_head` with
autocommit=True. Without it, psycopg3's `with conn` calls COMMIT on
clean exit, which fails after the `UndefinedTable` (fresh-DB) branch
left the transaction in an aborted state — crashing first-run startups.
- Wrap the v2 `prisma db push` path in try/except and raise RuntimeError
on CalledProcessError/TimeoutExpired. Otherwise these propagate past
proxy_cli.py's `except RuntimeError` as unhandled tracebacks.
- Reword the loop-exhaustion error to cover the non-timeout exit path
(repeated P3005/P3009/P3018 idempotent-recovery `continue`s), not
just persistent timeouts.
Adds a unit test for the db_push error wrapping.
The uv migration added PRISMA_BINARY_CACHE_DIR=/app/.cache/... and
XDG_CACHE_HOME=/app/.cache to the runtime stages of Dockerfile and
Dockerfile.database. BINARY_PATHS in the generated prisma client was
baked to point into /app/.cache, so any deployment that mounts a volume
there (common with securityContext.readOnlyRootFilesystem: true and an
emptyDir/tmpfs for a writable cache) wipes the pre-downloaded query
engine at pod startup, producing BinaryNotFoundError during connect().
Before the uv migration, prisma-python defaulted to $HOME/.cache =
/root/.cache (runtime stage runs as root), which was unaffected by any
/app/* volume mounts. Restore that behaviour: drop the env vars from
the runtime stage, re-run prisma generate there so the query engine
AND the baked BINARY_PATHS both land in /root/.cache, and remove the
stale builder-stage /app/.cache (~800 MB).
Dockerfile.non_root is intentionally left alone — its /app/.cache
location is by design for the hardened offline-install flow.
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
Adds end-to-end CI coverage for `--use_v2_migration_resolver` via a new
job `installing_litellm_on_python_v2_migration_resolver`:
- Clones the pytest smoke path from `installing_litellm_on_python` but
uses a local Postgres sidecar instead of the shared DB to prevent
collisions with the v1 variant.
- Runs only the new `test_litellm_proxy_server_config_no_general_settings_v2_resolver`
which spawns the proxy with `--use_v2_migration_resolver` and smoke-tests
`/health/liveliness` and `/chat/completions`.
Refactors `test_basic_python_version.py`:
- Extracts the proxy spawn + smoke-test body into `_run_proxy_server_smoke_test`
so the v1 and v2 tests share the same code path.
- The existing `test_litellm_proxy_server_config_no_general_settings` is
now a thin wrapper that passes no extra args (v1 default, unchanged).
- Adds `..._v2_resolver` variant that passes `--use_v2_migration_resolver`.
The existing `installing_litellm_on_python` / `installing_litellm_on_python_3_13`
jobs filter out the v2 variant via `-k "not v2_resolver"` so they keep
running only against their shared DB, unchanged behavior.
Default behavior (v1) is unchanged. Users who have seen schema thrashing
during rolling deploys can opt into the v2 resolver with
`--use_v2_migration_resolver`.
Why v2 is safer:
- Runs `prisma migrate deploy` only.
- Recovers from P3005 (baseline) and idempotent P3009/P3018 errors, same
as v1.
- Never calls `_resolve_all_migrations`, which generates a schema diff
between the live DB and the shipped schema.prisma and applies it via
`prisma db execute`. That path bypassed every migration's SQL and was
the root cause of thrashing when two LiteLLM versions contended for
the same DB.
- Logs a non-blocking warning when the DB has migrations applied that
are newer than anything this build ships (ahead-of-HEAD). It does not
refuse to start — many users have unusual ledger state from past
thrashing, and blocking startup would be a breaking change.
Also prints a message on startup when the default (v1) resolver is in
use, pointing operators at the opt-in flag.
Adds unit tests covering the v2 fail-fast paths, the stripping of
Prisma-specific query params from DATABASE_URL (needed for psycopg),
the timestamp helpers, and pins the default: v1 still invokes
`_resolve_all_migrations`, v2 must not.
Adds total_spend column to LiteLLM_TeamMembership that accumulates
continuously and is not zeroed by the budget cycle reset job. This
enables UI surfaces to distinguish current-cycle spend (the existing
spend column, which resets) from lifetime spend per team member.
Also exposes budget_reset_at on LiteLLM_BudgetTable so /team/info
callers can see when a member's budget window next resets. The field
was already stored in the DB but stripped by the response Pydantic
model.
Includes regression tests that:
- Guard the reset job against ever writing total_spend: 0
- Verify the spend writer increments both spend and total_spend in
one UPDATE statement.
- _find_destructive_statements: add DROP INDEX to the docstring (the
regex already detects it; only the docstring lagged).
- create_migration: correct the base_branch default documented in the
docstring from "main" to "litellm_internal_staging".
Generating a migration from a stale branch could silently emit DROP
COLUMN for columns the stale branch did not know about, and the
script would write that SQL to a new migration file with no warning.
Adds two guards to ci_cd/run_migration.py:
- Branch freshness check: fetches origin/<base-branch> and exits 3 if
HEAD is behind. Default base is litellm_internal_staging. New
flags: --base-branch, --skip-freshness-check.
- Destructive guard: refuses (exit 2) if the generated diff contains
DROP COLUMN / DROP TABLE / DROP INDEX, unless --allow-destructive
is passed.
Refusal banners include guidance and an explicit callout instructing
AI agents not to auto-bypass the flags. Also treats Prisma's
"-- This is an empty migration." output as a no-op rather than
writing an empty file.
Updates litellm-proxy-extras/migration_runbook.md with the new
workflow, flag documentation, and agent warnings.
Two fail-safes for the /v1/messages → Bedrock Invoke pass-through so new
Anthropic-only extensions Claude Code starts sending can't reach Bedrock
and trigger a 400 "Extra inputs are not permitted":
1. Top-level body fields are filtered to a typed allowlist. New
`BedrockInvokeAnthropicMessagesRequest` TypedDict (in
`litellm/types/llms/bedrock.py`) captures the Bedrock Invoke Anthropic
Messages body schema; the runtime allowlist is derived from its
`__annotations__` so the type and the filter can't drift. Anchored to
the AWS reference page in docstrings + transform comment. An
exact-set test pins the resolved allowlist so any future edit forces
conscious review.
Drops context_management, output_config, speed, mcp_servers,
container, inference_geo, internal litellm_metadata, and any future
Anthropic addition. output_format stays as an active inline-schema
conversion (not just a strip).
2. The anthropic-beta header list is filtered + transformed against the
bedrock mapping for ALL betas, not just auto-injected ones. The
previous code union'd user-provided betas back in unfiltered, so a
client on a new Anthropic-direct beta (e.g. advisor-tool-…,
context-management-…) could still pin the request to fail. In a proxy
context the client can't know the backend is Bedrock; the provider
mapping is authoritative. User-provided drops are logged at WARNING
so intentional overrides leave a breadcrumb.
Updates one existing test that happened to assert on the old buggy
pass-through (it used output-128k-2025-02-19, which is null in the
bedrock mapping and would 400 at runtime); rewrote it against a
bedrock-supported beta.
Scope: messages/invoke only. The same user-beta bypass exists in
chat/invoke but that's a different code path with different
user-expectation trade-offs — follow-up.
Brings user personal budget and organization budget enforcement
in line with the existing key and team patterns, which already
read spend from the atomic cross-pod Redis counter.
Add a router utils unit test that directly exercises _filter_deployments_by_model_access_groups for access-group-only key permissions.
Made-with: Cursor
Filter router candidate deployments by caller-authorized model access groups when access is granted via group membership, preventing cross-group load balancing for shared public model names.
Made-with: Cursor
VertexAIImagenImageEditConfig.get_complete_url was resolving vertex_project
and vertex_location only from env vars and global settings, ignoring
litellm_params. Users supplying project/location exclusively via YAML
config would get a ValueError or wrong URL even after auth headers were fixed.
Mirrors the pattern already used by VertexAIGeminiImageEditConfig and
image_generation counterpart (safe_get_vertex_ai_project/location).
Also fixes api_key type hint in MockImageEditConfig (str -> Optional[str])
and adds a test covering get_complete_url credential resolution.
Made-with: Cursor
Adds three test cases to prevent regression of the Vertex AI image_edit
credentials bug:
1. test_validate_environment_signature_includes_litellm_params: ensures
all image-edit configs accept litellm_params (contract for the handler)
2. test_vertex_gemini_image_edit_reads_credentials_from_litellm_params:
verifies Gemini config reads from litellm_params first
3. test_vertex_imagen_image_edit_reads_credentials_from_litellm_params:
verifies Imagen config reads from litellm_params first
These tests catch if the fix is accidentally reverted or if new image-edit
configs are added without the litellm_params parameter.
Made-with: Cursor
When aimage_edit or image_edit was called with Vertex AI Gemini/Imagen models
via YAML-style config (vertex_project / vertex_credentials in proxy YAML),
the credentials were dropped during handler-to-config plumbing, causing
fallback to Application Default Credentials and DefaultCredentialsError.
Root cause: image_edit_handler and async_image_edit_handler did not pass
litellm_params to validate_environment, unlike image_generation_handler.
Fixes:
1. Widen BaseImageEditConfig.validate_environment signature to accept
litellm_params and api_base (optional kwargs).
2. Forward dict(litellm_params) and litellm_params.api_base from both
sync and async image_edit handlers to validate_environment.
3. Update VertexAIImagenImageEditConfig.validate_environment to read
vertex_ai_project/vertex_ai_credentials from litellm_params first,
matching Gemini config pattern (secondary latent bug fix).
4. Widen all image-edit config override signatures to match base.
Made-with: Cursor
Addresses review feedback on the snapshot approach:
1. Class-instance mutable state
The snapshot only covers primitives + collections + None. Class
instances (DualCache, LLMClientCache) weren't reset between tests,
so in-place cache mutations could leak. Can't deepcopy these — they
hold thread locks — but they expose flush_cache(). Collect every
module attribute whose value implements flush_cache() at conftest
import, and invoke it per-test alongside the snapshot restore.
2. Silent skips are now warnings
_snapshot_mutable_state and _restore_mutable_state previously
swallowed exceptions, so if a future attr gained a property without
a setter (or other non-round-trippable state), an isolation gap
would have no signal. Emit warnings.warn on each failure path.
3. Docstring
Explicitly documents what IS and IS NOT reset, and tells authors to
use monkeypatch.setattr() for in-place mutations of instances
without flush_cache() (ProxyLogging, JWTHandler, etc.).
The Build UI from source step used:
cp -r out/ ../../litellm/proxy/_experimental/out/
GNU cp (CircleCI's Ubuntu image, coreutils 8.32) interprets this as
copy the source directory as a CHILD of the destination when the
destination already exists — so the command silently created
litellm/proxy/_experimental/out/out/ instead of replacing the served
bundle at litellm/proxy/_experimental/out/*.
The proxy continued serving whatever bundle was checked in, so every
e2e_ui_testing run between this job's introduction (d09d98a70a,
2026-04-08) and the bundle-rebuild commit (de790fd273, 2026-04-18) was
effectively testing a STALE bundle — not the fresh build. That is why
the double-prefix regression (NEXT_PUBLIC_BASE_URL="ui/" combined with
networking.tsx reading the env var) was never caught in CI even though
the source contained the trigger the whole time: the bundle the proxy
served never picked up the source change.
Replace cp -r with rm + mv so the destination is cleanly swapped.
Verified end-to-end on an Ubuntu 22.04 / GNU coreutils 8.32 container:
- Before fix: fresh build has 9 "ui/" literals in chunks; after cp,
_experimental/out/* still has 0 (stale); _experimental/out/out/ is a
nested dir the proxy does not serve.
- After fix: _experimental/out/* has 9 "ui/" literals — the proxy now
serves the freshly built (broken, in this repro) bundle, so
globalSetup fails at login and every spec is blocked. Removing the
bug from .env.production and rebuilding brings the count back to 0
and the suite passes.
No spec changes, no fixtures, no new infrastructure. The existing
Playwright suite already catches this class of regression via the
login flow in globalSetup; it just needs the CI to actually hand it
the freshly built bundle.
The previous snapshot only tracked list/dict/set values. Tests mutate
scalar module attrs too — master_key, premium_user, prisma_client — and
importlib.reload used to reset those implicitly. Under the snapshot
approach they were leaking between tests, so test_active_callbacks
failed in CI with "No api key passed in." once an earlier test left
master_key set to sk-1234.
Expand the snapshot to cover primitives (str/int/float/bool/bytes/tuple)
and None-valued attributes. Complex object instances are still skipped
to avoid deepcopy issues.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Add regression tests that mock make_bedrock_api_request and verify
input_type=request uses source=INPUT with user messages, and
input_type=response uses source=OUTPUT with synthetic ModelResponse.
Made-with: Cursor
BedrockGuardrail.apply_guardrail hardcoded source="INPUT" regardless of the
input_type parameter. On the non-streaming post-call path (unified_guardrail
-> OpenAIChatCompletionsHandler.process_output_response -> apply_guardrail),
the model response text was sent to Bedrock as INPUT, so guardrail policies
configured for Output (e.g. PII/NAME blocking) returned action=NONE and the
response passed through unblocked. The streaming path was unaffected because
it calls make_bedrock_api_request(source="OUTPUT", ...) directly.
Map input_type to the correct Bedrock source ("request" -> INPUT,
"response" -> OUTPUT) and build a synthetic ModelResponse for the OUTPUT
path so _create_bedrock_output_content_request produces the correct payload.
Made-with: Cursor
tests/proxy_unit_tests/conftest.py was calling importlib.reload(litellm) in an
autouse function-scoped fixture, which cost ~17s per test because it re-ran
the full litellm __init__ import chain. With 400+ proxy unit tests, this was
the single biggest driver of CI wall time — 18 of the top 20 slowest durations
in a typical run were just the 17s fixture setup.
Replace the reload with a snapshot-and-restore approach: snapshot the mutable
lists/dicts/sets on litellm and litellm.proxy.proxy_server once at conftest
import, then deep-copy that snapshot back before each test. Callback lists,
caches, router state, etc. still get reset between tests, but the expensive
import chain only runs once per worker.
Local measurement on test_proxy_utils.py: 188 tests in 3.50s (previously took
~15 minutes of CI wall time on a single worker).
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
- _owner_cache now opportunistically sweeps expired entries past
_OWNER_CACHE_SWEEP_THRESHOLD live entries. Previously sessions that never
came back piled up forever.
- flush_session_to_db strips session_id/router_name/model_name from the update
payload. Prisma rejects writes to @@id fields.
- record_turn no longer persists last_user_content / last_assistant_content /
tool_call_history / pending_tool_calls. Those are needed only in-memory for
the next turn's signal detection; writing user prompts and tool payloads to
the DB would store PII for every conversation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SessionState now carries clean_credit_awarded + last_processed_turn (matching
the DB schema). Satisfaction only fires once per session AND only after
MIN_TURNS_FOR_CLEAN_CREDIT turns of context — early "thanks" no longer
inflates alpha.
- _detect_failure no longer treats empty content as failure. Many tools
legitimately return empty output (zero-result searches, silent bash);
penalizing those corrupted the bandit posterior. Only is_error fires now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
The "remaining" proxy-db job was consistently timing out at ~98% because
--dist=loadscope pins every test in test_proxy_utils.py (168+ parametrized
tests) to a single xdist worker. 7 workers finished their files in ~15
minutes, then one worker ran alone for another 8+ minutes and hit the
30-minute job cap.
Give test_proxy_utils.py its own matrix entry so its tests spread across
all 8 workers, and add it to the "remaining" ignore list.
