litellm/tests
Krish Dholakia b96f033c90
fix: prisma migrate deploy failures on pre-existing instances (#23655)
* fix: prisma migrate deploy failures on pre-existing instances

Fixes failed migrations due to idempotent schema changes on pre-existing litellm instances.

Problems:
1. P3018 recovery handler never returned True on successful resolution, causing "Database setup failed after multiple retries" even when the final recovery succeeded
2. _roll_back_migration exceptions escaped the P3018 handler, preventing _resolve_specific_migration from running
3. Migration SQL used ADD COLUMN/DROP COLUMN without IF [NOT] EXISTS, failing if schema was already modified

Changes:
- Add return True after successful P3018 idempotent error recovery
- Wrap _roll_back_migration in try/except to allow recovery continuation even if rollback fails
- Make migration.sql idempotent with IF NOT EXISTS / IF EXISTS clauses

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* test: add migration SQL idempotency safety tests

Adds TestMigrationSQLIdempotency test class that statically validates all
migration SQL files created after 2026-03-11 use idempotent DDL:
- ADD COLUMN must use IF NOT EXISTS
- DROP COLUMN must use IF EXISTS
- DROP INDEX must use IF EXISTS
- CREATE INDEX must use IF NOT EXISTS

This prevents the class of errors where prisma migrate deploy fails on
pre-existing instances because the schema was already modified.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: also catch TimeoutExpired in P3018 rollback handler

_roll_back_migration uses subprocess.run with timeout=60, so it can raise
subprocess.TimeoutExpired in addition to CalledProcessError. Without
catching this, a slow database during rollback would escape the handler
and bypass _resolve_specific_migration — the same class of bug.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: make all 85 migration SQL files idempotent, remove test cutoff

Fixed all existing migration files to use IF [NOT] EXISTS for DDL
statements (ADD COLUMN, DROP COLUMN, DROP INDEX, CREATE INDEX).
Removed the date cutoff from the idempotency tests so they now
validate all migrations, not just recent ones.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: make migration failure non-fatal by default, add --require_db_migration flag

By default the proxy now warns and continues when database migration
fails. Pass --require_db_migration (or set REQUIRE_DB_MIGRATION=true)
to restore the previous behavior of exiting with an error.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: wrap _resolve_specific_migration in try/except, guard RENAME COLUMN and ADD CONSTRAINT

Three fixes:

1. _resolve_specific_migration in the P3018 handler was not wrapped in
   try/except, so failures there would bypass the return True and
   propagate unexpectedly — partially defeating the rollback fix.

2. Bare RENAME COLUMN in 20260303000000_update_tool_table_policies was
   non-idempotent. Wrapped in DO $$ IF EXISTS block. Also wrapped all
   28 bare ADD CONSTRAINT statements across 9 migration files in
   DO $$ IF NOT EXISTS (pg_constraint) blocks.

3. Added test_rename_column_is_guarded and test_add_constraint_is_guarded
   to TestMigrationSQLIdempotency for full DDL coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: retry after resolving idempotent migration, guard DROP CONSTRAINT

Three fixes:

1. Both P3009 and P3018 idempotent handlers returned True after
   resolving a single migration, exiting before remaining pending
   migrations were applied. Now they continue the retry loop so
   prisma migrate deploy runs again for any remaining migrations.

2. Two migration files had bare DROP CONSTRAINT without a DO $$ IF
   EXISTS guard, which fails if the constraint was already dropped.
   Wrapped both in idempotent DO $$ blocks.

3. Added test_drop_constraint_is_guarded to catch unguarded DROP
   CONSTRAINT in future migrations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: P3009 try/except, CREATE TABLE IF NOT EXISTS, restore fail-fast default

Four fixes:

1. P3009 idempotent handler now has the same try/except around
   _roll_back_migration and _resolve_specific_migration as the P3018
   handler. Previously a rollback or resolve failure in the P3009 path
   would propagate and leave the migration unresolved.

2. Added IF NOT EXISTS to all 57 bare CREATE TABLE statements across
   34 migration files. Added test_create_table_uses_if_not_exists to
   catch this pattern.

3. Reverted the backwards-incompatible default behavior change: the
   proxy now fails fast on migration failure (original behavior).
   Added --skip_db_migration_check / SKIP_DB_MIGRATION_CHECK to
   opt into warn-and-continue instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
2026-03-14 16:54:21 -07:00
..
agent_tests [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
audio_tests
basic_proxy_startup_tests CircleCI test stability (#23055) 2026-03-07 15:19:39 -08:00
batches_tests Merge pull request #22625 from BerriAI/litellm_azure_ai_finetune 2026-03-03 19:42:17 +05:30
code_coverage_tests [Fix] Add max_depth guard to BFL _read_image_bytes recursive function 2026-03-12 13:22:56 -07:00
documentation_tests fix failing tests 2026-02-21 15:48:26 -08:00
enterprise feat: allow multiple calls from tags 2026-03-07 11:24:18 +05:30
guardrails_tests fix: address req changes 2026-03-10 15:51:10 +05:30
image_gen_tests fix(ci): remove unused is_expired variable (ruff F841) and handle ModelDeprecated in image gen test 2026-03-12 01:27:42 +00:00
litellm fix(proxy): Windows compatibility for Prisma engine watchdog (#23494) 2026-03-12 16:02:37 -07:00
litellm-proxy-extras fix: prisma migrate deploy failures on pre-existing instances (#23655) 2026-03-14 16:54:21 -07:00
litellm_core_utils Fix: empty assistant message for converse API 2026-02-04 09:50:58 +05:30
litellm_utils_tests fix(tests): remove test_oidc_circle_v1_with_amazon_fips that depends on external infra 2026-03-13 10:33:52 -07:00
llm_responses_api_testing Fix anthropic responses 2026-02-20 17:30:42 -08:00
llm_translation fix(tests): remove test_completion_bedrock_claude_sts_oidc_auth and test_completion_bedrock_httpx_command_r_sts_oidc_auth that depend on external infra 2026-03-13 11:53:53 -07:00
load_tests Add memory leak detection tests with CI integration (#18881) 2026-01-09 17:36:10 -08:00
local_testing feat: add sagemaker_nova provider for Amazon Nova models on SageMaker (#21542) 2026-03-14 15:10:01 -07:00
logging_callback_tests Merge pull request #23483 from BerriAI/litellm_update_deprecated_test_models 2026-03-12 14:16:52 -07:00
mcp_tests Fix test_get_tools_for_single_server 2026-03-12 18:33:14 +05:30
multi_instance_e2e_tests
ocr_tests OCR test fixes 2026-01-11 08:00:31 -08:00
old_proxy_tests/tests
openai_endpoints_tests test(responses): add end-to-end test for responses API WebSocket mode 2026-03-02 17:24:39 +05:30
otel_tests fix(tests): increase MAX_CALLS and reduce sleep in flaky e2e budget test 2026-03-13 00:04:31 -07:00
pass_through_tests fix(ci): stabilize CI tests - conditional import, mock fixes, timing adjustments 2026-03-13 00:01:25 +00:00
pass_through_unit_tests [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
proxy_admin_ui_tests Fix CVEs: bump tar/minimatch/pypdf + harden Docker SBOM patching (#23082) 2026-03-07 18:31:27 -08:00
proxy_e2e_anthropic_messages_tests Merge branch 'main' into litellm_fix_nova_pro_max_tokens 2026-03-09 15:47:53 -07:00
proxy_e2e_azure_batches_tests [Fix] Increase _delete_file retry budget to reduce flakiness 2026-03-13 12:32:55 -07:00
proxy_security_tests
proxy_unit_tests Handle response.failed, response.incomplete, and response.cancelled (#23492) 2026-03-13 23:02:09 -07:00
router_unit_tests [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
scim_tests
search_tests Merge remote main, resolve conflict keeping new unit tests 2026-03-09 15:20:20 -07:00
spend_tracking_tests [Fix] Derive SPEND_PER_REQUEST dynamically in spend accuracy tests 2026-03-13 13:27:06 -07:00
store_model_in_db_tests
test_litellm feat: add sagemaker_nova provider for Amazon Nova models on SageMaker (#21542) 2026-03-14 15:10:01 -07:00
unified_google_tests Revert "[Infra] Changing Google Tests to use Gemini 3 Flash Preview" 2026-01-20 17:32:10 -08:00
vector_store_tests Add support for vertex ai for rag/ingest 2026-02-13 20:11:10 +05:30
windows_tests
__init__.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
README.MD
test_budget_management.py
test_callbacks_on_proxy.py
test_config.py
test_debug_warning.py
test_default_encoding_non_root.py fix: tiktoken cache nonroot offline (#23498) 2026-03-14 10:48:36 -07:00
test_end_users.py
test_entrypoint.py
test_fallbacks.py Revert "fix: prevent error when max_fallbacks exceeds available models (#20071)" 2026-02-03 15:15:30 +05:30
test_gpt5_azure_temperature_support.py [Fix] Remove deprecated o1-preview from O-series test and deduplicate is_o_series check 2026-03-12 13:22:00 -07:00
test_health.py
test_keys.py Adding retries to flaky tests 2026-01-22 15:21:44 -08:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py fix(ci): stabilize CI tests - conditional import, mock fixes, timing adjustments 2026-03-13 00:01:25 +00:00
test_new_vector_store_endpoints.py fix(ci): stabilize CI tests - conditional import, mock fixes, timing adjustments 2026-03-13 00:01:25 +00:00
test_openai_endpoints.py
test_organizations.py Adding retries to flaky tests 2026-01-22 15:21:44 -08:00
test_otel_thread_leak.py Fix thread leak in OpenTelemetry dynamic header path (#19946) 2026-01-28 10:35:37 -08:00
test_passthrough_endpoints.py
test_presidio_latency.py fix(presidio): reuse HTTP connections to prevent OOMs (#19964) 2026-01-28 16:08:53 -08:00
test_proxy_server_non_root.py deactivating non root tests 2026-01-23 22:55:36 -08:00
test_ratelimit.py
test_resource_cleanup.py
test_service_logger_otel.py fix(langfuse_otel): prevent empty proxy request spans from being sent to Langfuse 2026-01-28 15:35:35 +01:00
test_spend_logs.py
test_team.py [Fix] Fail fast when team member spend not flushed in time 2026-03-13 12:40:17 -07:00
test_team_logging.py
test_team_members.py
test_users.py fix(tests): update deprecated Anthropic model in test_user_model_access (#21826) 2026-02-21 14:18:24 -08:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.