Commit graph

887 commits

Author SHA1 Message Date
yuneng-jiang
eba54bae11 [Fix] Add aurelio_sdk --no-deps alongside semantic_router in guardrails and enterprise tests
semantic_router imports aurelio_sdk at module load time, so it must be
installed even when using --no-deps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 23:02:35 -07:00
yuneng-jiang
ae1e827319 [Infra] Optimize CI: add xdist to caching tests, drop Docker CLI installs, reduce verbosity
- caching_unit_tests: add resource_class large, enable xdist -n 4, drop unused coverage collection
- build_and_test & proxy_pass_through_endpoint_tests: remove redundant Docker CLI install (machine executor has it)
- Downgrade -vv to -v across 4 jobs to reduce log noise

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 22:41:27 -07:00
yuneng-jiang
f07301a518 [Infra] Optimize CI: right-size resource classes, drop unused coverage, increase xdist workers
Downgrade langfuse, assistants, and python 3.13 install jobs to medium (were defaulting to large at ~25% CPU).
Bump enterprise and image_gen xdist workers to -n 4 on explicit large instances.
Drop coverage collection and persist_to_workspace for 4 jobs that no longer need it.
Downgrade verbosity from -vv to -v across all 5 jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 22:27:50 -07:00
yuneng-jiang
65b3335735 [Infra] Use uv for requirements.txt installs across 22 CI jobs
Switch pip install -r requirements.txt to uv pip install --system -r requirements.txt
for all docker-based jobs that use the main requirements.txt. This applies the same
optimization already proven in the mapped test jobs to the rest of the CI pipeline.

Also adds --no-deps to semantic_router installs in guardrails_testing and
litellm_mapped_enterprise_tests to avoid uv's strict resolution conflict with openai>=2.

Skipped: machine executor + conda jobs (security, proxy_spend_accuracy,
proxy_multi_instance, proxy_store_model_in_db) and Group B jobs using
.circleci/requirements.txt.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 22:22:44 -07:00
yuneng-jiang
379c3952f4 [Fix] Use uv for requirements.txt only, pip for test deps with conflicting pins
uv's strict resolver rejects transitive dep conflicts (semantic-router
wants openai<2, llm-sandbox wants pydantic>=2.11.5). Use uv for the
heavy requirements.txt install and pip for the small test dep batch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 21:58:07 -07:00
yuneng-jiang
26207bb7be [Infra] Speed up mapped test jobs: uv installs, site-packages caching, drop unused coverage
- Switch setup_litellm_test_deps from pip to uv with batched installs
- Cache installed site-packages (~/.local/lib, ~/.local/bin) instead of
  pip download cache for near-instant installs on cache hit
- Remove unused coverage collection from 6 mapped test jobs (only mcps
  coverage is consumed by the coverage combine step)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 21:50:18 -07:00
yuneng-jiang
1a00dd4dbb Fix router test isolation for xdist and rebalance proxy unit tests
Router tests: expand conftest save/restore to cover all globals mutated
by router tests (default_fallbacks, tag_budget_config, request_timeout,
enable_azure_ad_token_refresh, num_retries_per_request, model_cost,
token_counter). These were leaking across xdist workers.

Proxy tests: move test_proxy_utils.py (169 parametrized) and
test_proxy_server.py (72 parametrized) from part2 to part1, balancing
~370 vs ~360 tests (was ~129 vs ~600).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 21:36:56 -07:00
yuneng-jiang
74e57bdd27 Optimize CI test jobs: increase xdist workers, drop coverage, add caching
Increase pytest-xdist parallelism to match available CPU on I/O-bound and
CPU-bound test jobs. Drop coverage collection from 8 jobs (still collected
by ~15 other jobs). Add dependency caching to 4 uncached jobs. Reduce
verbose output (-vv to -v) and remove -s/--log-cli-level overhead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 21:20:50 -07:00
yuneng-jiang
c1efbd3c8a [Fix] Drop httpx and opentelemetry pins from local_testing batched installs
Batching pip installs surfaced more hidden conflicts:
- respx==0.22.0 requires httpx>=0.25.0, conflicting with httpx==0.24.1
- traceloop-sdk==0.21.1 requires otel-semantic-conventions<0.46,
  conflicting with opentelemetry-sdk==1.25.0 (needs ==0.46b0)

These were masked before because separate pip install calls let later
installs silently override earlier pins. Dropping the pins lets pip
resolve compatible versions. Verified with pip --dry-run locally.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 21:10:25 -07:00
yuneng-jiang
59f0db0538 [Fix] Remove anyio==4.2.0 pin from local_testing batched installs
Batching pip installs exposed a dependency conflict: langfuse==2.59.7
requires anyio>=4.4.0, which conflicts with the anyio==4.2.0 pin.
Dropping the pin lets pip resolve a compatible version.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 21:02:50 -07:00
yuneng-jiang
f3cc292daf [Infra] Speed up CI: batch pip installs and fix pytest -n parallelism
Several test jobs were underutilizing their CPU allocation (~25%) because
they were either missing pytest-xdist -n or using -n 2 on 4-vCPU machines.
Batching individual pip install calls into single commands reduces resolver
overhead and saves ~30-60s per job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 20:54:46 -07:00
yuneng-jiang
19e8a16cce Optimize logging_testing CI: suppress DEBUG logs, fix xdist isolation
- Add LITELLM_LOG=WARNING to suppress verbose DEBUG log output
- Remove -s flag to stop capturing all stdout
- Bump xdist workers from -n 2 to -n 4
- Add --timeout=120 for safety
- Rewrite conftest.py to use save/restore pattern (matching guardrails_tests)
  instead of per-function importlib.reload + event loop creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 18:24:57 -07:00
yuneng-jiang
ec537dd973 Add -n 2 parallelism to 8 medium jobs running serial tests
These jobs run on medium (2 CPU) but weren't using pytest-xdist,
leaving the second CPU idle. Added pytest-xdist dep and -n 2 to:
- auth_ui_unit_tests (~33 tests)
- litellm_router_unit_testing (~191 tests)
- mcp_testing (~112 tests)
- llm_responses_api_testing (~80 tests)
- search_testing (~53 tests)
- batches_testing (~45 tests)
- litellm_utils_testing (~205 tests)
- pass_through_unit_testing (~102 tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 17:15:54 -07:00
yuneng-jiang
a95cae6b1e Speed up build_docker_database_image: drop Docker upgrade, use zstd
- Use ubuntu-2204:2024.04.1 which ships with a recent Docker, eliminating
  the 1-minute `curl get.docker.com | sh` upgrade step
- Switch image save/load from gzip to zstd -1 -T0 for ~3-5x faster
  compression/decompression, saving ~30s on save and on each downstream load

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 17:09:37 -07:00
yuneng-jiang
4c4246ab4a Downgrade oversized resource classes to match actual workload
- proxy_unit_testing_key_generation: large→medium (serial, 1 test file)
- proxy_unit_testing_part1: large→medium, -n 4→2 (only 2 test files)
- mapped_tests_proxy_part1: xlarge→large, -n 8→4 (~2000 tests, 4 CPUs sufficient)

Saves ~40 credits/min across these 3 jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 16:19:43 -07:00
yuneng-jiang
65575f3992 Fix pytest -n worker oversubscription to match available CPUs
8 jobs had -n workers set higher than available vCPUs, causing context
switch overhead and degraded performance. Aligned -n to match resource_class:
- medium (2 CPU): enterprise -n 8→2, image_gen/logging/guardrails -n 4→2
- large (4 CPU): proxy_part1/llms/core/integrations -n 8→4

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 15:59:53 -07:00
yuneng-jiang
a66b635a4c Upsize ui_build and ui_unit_tests CI machines for faster feedback
ui_build (medium → medium+) is on the critical path blocking 3 downstream
jobs. ui_unit_tests (medium+ → large, maxForks 3 → 5) targets ~7 min from ~11.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 15:36:59 -07:00
yuneng-jiang
ff869e91b0 Fix flaky caching tests: use mock_response, add parallelism, remove fail-fast
- Replace real OpenAI/Anthropic/Bedrock API calls with mock_response in
  ~20 cache tests to eliminate network-dependent flakiness
- Remove -x (fail-fast) from caching_unit_tests so all failures are reported
- Add parallelism: 2 with circleci tests run --split-by=timings
- Improve pip dependency cache key (v2-caching-deps) with fallback key

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 15:23:19 -07:00
yuneng-jiang
31e6393458 Fix flaky proxy_e2e_azure_batches_tests: populate _hidden_params for DB-cached batch retrieval
When a completed batch is served from the DB cache, _hidden_params was empty,
causing the managed files hook to skip output_file_id translation from raw
provider IDs to unified IDs. This fix populates unified_batch_id and model_id
on the early-return path, with a guard against double-encoding when the DB
already stores unified IDs.

Also reduces file deletion retry delay (20s→5s), reruns (5→2), and CI timeout
(30m→15m) to cut worst-case runtime from ~16min to ~4min.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 15:13:30 -07:00
yuneng-jiang
87e3e84e6b Set semgrep job to medium resource class explicitly
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 15:07:06 -07:00
yuneng-jiang
2e27e32d33 Downsize remaining xlarge CI jobs to match actual workload
4 mapped test jobs (llms, core, core_utils, integrations) were running on
xlarge with -n 16 workers on 8 vCPUs, causing unnecessary context switching.
Downsized to large with worker counts matched to available cores.
MCPs job (2 test files) downsized from large to medium.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 14:50:13 -07:00
yuneng-jiang
f838bea85b Optimize CI: parallelize router and guardrails test jobs, fix test isolation
- Router testing: add CircleCI parallelism=4 with timing-based test splitting
- Guardrails testing: add pytest-xdist -n 4, suppress DEBUG logs with LITELLM_LOG=WARNING
- Rewrite conftest.py in both test dirs for xdist compatibility (save/restore pattern)
- Fix module-level Router instances in test_router_fallback_handlers, test_router_custom_routing, test_acooldowns_router

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 22:54:44 -07:00
yuneng-jiang
509137580c Consolidate duplicate Docker image builds to reuse pre-built image
build_and_test, helm_chart_testing, and test_bad_database_url each rebuilt
Dockerfile.database from scratch (~5-10 min each) despite build_docker_database_image
already building and persisting it to workspace. Now all three load the
pre-built image via workspace, eliminating 3 redundant Docker builds.

Also removes orphaned test_nonroot_image job definition (not referenced in workflow).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 21:44:07 -07:00
yuneng-jiang
cc80771835 Cache Next.js build cache for ui_build job
Add restore_cache/save_cache for .next/cache to eliminate the
"No build cache found" warning and speed up subsequent Next.js builds.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 21:29:18 -07:00
yuneng-jiang
614768257b Switch test_bad_database_url from Dockerfile.non_root to Dockerfile.database
test_bad_database_url only validates DB error handling — it doesn't need
the UI. Dockerfile.non_root does a full npm install + next build which is
unnecessary overhead for this test. Dockerfile.database skips the UI build
(build_admin_ui.sh is a no-op for OSS) making the Docker build much faster.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 21:25:04 -07:00
yuneng-jiang
e38ee6a531 Optimize ui_build job and gate ui_unit_tests behind it
- Switch ui_build from machine VM to docker executor (cimg/node:20.19)
- Add npm cache (restore_cache/save_cache) keyed on package-lock.json
- Use npm ci instead of rm -rf node_modules && npm install
- Gate ui_unit_tests behind ui_build (don't run tests if build fails)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 21:19:32 -07:00
yuneng-jiang
45ff66ab90 Add ui_unit_tests as requirement for publish_to_pypi
ui_unit_tests was gated behind ui_build but not required for publishing.
This ensures UI unit tests must pass before PyPI publish.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 20:59:16 -07:00
yuneng-jiang
95b6821436 Speed up ui_unit_tests CI job (~20min -> ~5-7min)
- Switch from machine VM to docker executor (cimg/node:20.19)
- Add npm cache with restore_cache/save_cache
- Use npm ci instead of npm install (stop deleting node_modules/package-lock.json)
- Enable parallel test execution with --pool forks --maxForks=3
- Bump resource_class to medium+ for 3 vCPUs
- Remove unnecessary requires: ui_build dependency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 20:16:44 -07:00
yuneng-jiang
eb7222c7cb [Infra] Reduce CI no_output_timeout and tighten Playwright timeouts
- Change no_output_timeout from 120m/60m to 15m across all test jobs
- Fix multi-line YAML formatting for no_output_timeout entries
- Reduce Playwright per-test timeout from 4min to 3min
- Add 15s actionTimeout and 30s navigationTimeout to Playwright config

Hanging tests now fail in 15min instead of 2hrs, cutting wasted CI time.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 19:38:10 -07:00
yuneng-jiang
27f1d33e05 Downgrade resource_class on over-provisioned CI jobs
21 jobs were using xlarge (8 vCPU, 16GB) despite running trivial or
low-parallelism workloads. Downgrades 6 trivial container-check/UI jobs
to medium (2 vCPU) and 15 lightweight test jobs to large (4 vCPU).
Only the 5 high-parallelism mapped test jobs (-n 8/16) remain on xlarge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 17:15:08 -07:00
yuneng-jiang
53d704a1f1 [Fix] Add litellm-proxy-extras to CI requirements for prisma migrations
PR #23257 made proxy startup fail if prisma migrate fails, which
exposed that litellm-proxy-extras was never installed in CI. The import
error was previously silently ignored. Unpinned so it always pulls the
latest version from PyPI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 14:58:22 -07:00
yuneng-jiang
28989a11e7 [Fix] Update test_bad_database_url to match new startup error message
PR #23257 changed proxy startup to exit early with "Database setup
failed after multiple retries" instead of letting uvicorn emit
"Application startup failed. Exiting." Update the CI grep check
to accept either error message for robustness.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 16:25:35 -07:00
yuneng-jiang
729f32d6d5
Merge pull request #23179 from BerriAI/litellm/intelligent-wilbur
[Fix] Chocolatey v2.5.1 Interactive Prompt Blocking Windows CI
2026-03-09 14:09:58 -07:00
yuneng-jiang
4cc7e76fbe Fix Chocolatey v2.5.1 interactive prompt in Windows CI job
Chocolatey v2.5.1 introduced interactive prompts that block CI. Add
--no-progress, --force flags and CHOCOLATEY_CONFIRM_ALL env var to
fully suppress user input in non-interactive environments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 14:06:46 -07:00
Sameer Kankute
d14420b8f2 Add tenacity in dependencies 2026-03-04 17:28:22 +05:30
Sameer Kankute
ef7aa6c8ad Add this test in cicd 2026-03-04 17:24:48 +05:30
Sameer Kankute
7b6a972fed Add this test in cicd 2026-03-04 17:21:00 +05:30
Ishaan Jaff
ccf2b7b746
fix(ci): handle inline table in pyproject.toml for litellm-proxy-extras version check (#22414) 2026-02-28 09:58:01 -08:00
Ishaan Jaff
29e3fd5d79
[Release Fix] (#22411)
* fix(lint): suppress PLR0915 for 3 complex methods that exceed 50-statement limit

- streaming_iterator.py: _process_event (84 statements)
- transformation.py: translate_messages_to_responses_input (51 statements)
- transformation.py: transform_realtime_response (54 statements)

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(mypy): resolve type errors in public_endpoints, user_api_key_auth, common_utils, transformation

- public_endpoints.py: fix _cached_endpoints type annotation
- user_api_key_auth.py: accept Optional[str] for end_user_id parameter
- common_utils.py: add NewProjectRequest/UpdateProjectRequest to Union type
- transformation.py: add ChatCompletionRedactedThinkingBlock and list[Any] to content type

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(proxy-extras): bump version to 0.4.50 and sync schema

- Bump litellm-proxy-extras from 0.4.49 to 0.4.50
- Sync schema.prisma with main proxy schema
- Includes new LiteLLM_ClaudeCodePluginTable model
- Includes new @@index([startTime, request_id]) on SpendLogs
- Update version references in requirements.txt and pyproject.toml

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(router): use string id in test_add_deployment and add defensive str() in register_model

- Change test to use string '100' instead of int 100 for model_info.id
- Add str() conversion in register_model to prevent AttributeError on non-string keys

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(security): update minimatch to 10.2.4 to fix CVE-2026-27903 and CVE-2026-27904

- Run npm audit fix in docs/my-website
- Updates minimatch from 10.2.1 to 10.2.4 (fixes HIGH severity ReDoS vulnerabilities)

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): update realtime guardrail test assertions to match actual guardrail behavior

- test_text_message_blocked_by_guardrail_no_ai_response: allow guardrail's own block
  message text in response.done (previously expected empty content)
- test_voice_transcript_blocked_by_guardrail: allow guardrail to send response.cancel
  + block message + response.create flow (previously expected no response.create)

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: revert proxy-extras version in requirements.txt and pyproject.toml

The litellm-proxy-extras 0.4.50 is not published to PyPI yet, so consumer
references must stay at 0.4.49. Only the source package pyproject.toml
should be bumped to 0.4.50 for the publish_proxy_extras CI job.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: make transcript delta check optional in voice guardrail test

The guardrail sends an error event (guardrail_violation) when blocking
voice transcripts; it does not always produce transcript deltas. Remove
the assertion requiring response.audio_transcript.delta since the error
event is the primary signal that blocked content was handled.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* Add missing env keys to documentation: LITELLM_MAX_STREAMING_DURATION_SECONDS and LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES

These two environment variables were used in code but not documented in the
environment variables reference section of config_settings.md, causing the
test_env_keys.py CI test to fail.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* Fix 13 mypy type errors across 6 files

- in_flight_requests_middleware.py: Fix type: ignore error codes from
  [union-attr] to [attr-defined], add [arg-type] for Gauge **kwargs
- transformation.py: Add [assignment] ignore for output_format reassignment,
  add fallback empty string for tool use id to fix arg-type
- responses/main.py: Remove redundant type annotation on second
  secret_fields assignment to fix no-redef
- streaming_iterator.py: Add [assignment] ignores for intermediate
  cache token assignments
- handler.py: Add [typeddict-item] ignore for AnthropicMessagesRequest
  construction from dict
- public_endpoints.py: Add [arg-type] ignore for _load_endpoints()
  return type mismatch with SupportedEndpoint model

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add auth overrides to spend tracking tests, fix realtime guardrail assertion, update UI minimatch

- Add app.dependency_overrides for user_api_key_auth in 4 spend tracking tests
  that were returning 401 Unauthorized (error_code, error_message,
  error_code_and_key_alias, key_hash)
- Fix realtime guardrail test to check ANY error event for guardrail_violation
  instead of just the first (OpenAI may send its own errors first)
- Update ui/litellm-dashboard/package-lock.json to fix minimatch vulnerability

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* Fix failing MCP e2e and create_mcp_server UI tests

Test 1 (test_independent_clients_no_shared_session):
- Add allow_all_keys: true to MCP servers in test config. With master_key
  and no DB, get_allowed_mcp_servers returned empty, causing 0 tools and
  403 on tool calls. allow_all_keys bypasses per-key restrictions.
- Add asyncio.sleep(0.5) between client connections to allow MCP SDK
  TaskGroup cleanup and avoid ExceptionGroup on connection close (MCP #915).

Test 2 (create_mcp_server 'auth value is provided'):
- Use userEvent.setup({ delay: null }) for instant keystrokes to avoid
  timeout from default typing delay on CI.
- Increase per-test timeout to 15000ms for CI environments.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: stabilize proxy unit tests for parallel execution

- test_response_polling_handler: add xdist_group to prevent heavy import OOM
- test_db_schema_migration: use temp dir for worker isolation, sync schema.prisma index
- test_custom_tokenizer_bug: use lighter tokenizer to prevent OOM in parallel

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add auth overrides to more spend tracking and model info tests

- Fix test_ui_view_spend_logs_pagination missing auth override (401)
- Fix test_view_spend_tags missing auth override (401)
- Fix test_view_spend_tags_no_database missing auth override (401)
- Fix test_empty_model_list.py to use app.dependency_overrides instead of patch()
  for FastAPI dependency injection auth

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): use patch.object for aiohttp transport test to work in parallel execution

The @patch decorator was not intercepting the static method call in parallel
xdist workers. Using patch.object on the directly-imported class is more reliable.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(security): update minimatch from 10.2.1 to 10.2.4 in Dockerfile

The Docker image was explicitly pinning minimatch@10.2.1 which has HIGH
severity ReDoS vulnerabilities (GHSA-7r86-cg39-jmmj, GHSA-23c5-xmqv-rm74).
Update to 10.2.4 which includes fixes for both CVEs.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(ui): prevent MCP and TeamInfo test timeouts on CI

- Add userEvent.setup({ delay: null }) to all tests using userEvent in both files
- Add timeout: 15000 to tests with significant user interaction (typing, multiple clicks)
- Fixes: create_mcp_server Bearer Token test, TeamInfo cancel button test

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: stabilize parallel test execution and aiohttp transport test

- test_aiohttp_handler: rewrite transport test to not rely on static method mock
  (consistently fails in parallel xdist workers)
- test_proxy_cli: add xdist_group to prevent timeout during heavy imports
- test_swagger_chat_completions: add xdist_group to prevent timeout

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(security): add serialize-javascript override to fix GHSA-5c6j-r48x-rmvq

Add npm override for serialize-javascript>=7.0.3 in docs/my-website
to fix HIGH severity RCE vulnerability via RegExp.flags.
Also bump minimatch override to >=10.2.4.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* Fix flaky tests: remove broken Vertex model, add retries for Anthropic

- Remove vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas from
  test_partner_models_httpx_streaming - consistently returns 400 BadRequest
- Add @pytest.mark.flaky(retries=6, delay=10) to test_function_call_parsing
  for transient Anthropic API overload errors
- Add @pytest.mark.flaky(retries=6, delay=10) to test_openai_stream_options_call
  for transient Anthropic InternalServerError

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(ci): add xdist_group(proxy_heavy) to prevent OOM in parallel proxy tests

- Add pytestmark = pytest.mark.xdist_group('proxy_heavy') to test_proxy_utils.py
- Change test_db_schema_migration.py from schema_migration to proxy_heavy group
- Add @pytest.mark.xdist_group('proxy_heavy') to test_proxy_server.py::test_health

Groups heavy proxy tests to run on same worker, avoiding worker OOM crashes.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* Fix vertex AI qwen global endpoint test to mock vertexai module import

The test_vertex_ai_qwen_global_endpoint_url test was failing because the
VertexAIPartnerModels.completion() method tries to 'import vertexai' before
any of the mocked code runs. In environments without google-cloud-aiplatform
installed, this import fails with a VertexAIError(status_code=400).

Fix by:
- Adding patch.dict('sys.modules', {'vertexai': MagicMock()}) to mock the
  vertexai module import
- Adding vertex_ai_location parameter to the acompletion call for completeness

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(ci): add xdist_group to health endpoint and watsonx tests for parallel stability

- test_health_liveliness_endpoint: add xdist_group('proxy_health') to prevent timeout
- test_watsonx_gpt_oss tests: add xdist_group('watsonx_heavy') to prevent mock interference

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): pre-populate WatsonX IAM token cache to prevent parallel test interference

The watsonx prompt transformation test was failing in parallel execution because
litellm.module_level_client.post mock was being interfered with by other tests.
Pre-populating the IAM token cache avoids the HTTP call entirely.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): add spend data polling with retries for e2e pass-through tests

- test_vertex_with_spend.test.js: Replace 15s fixed wait with polling loop
  (up to 6 attempts, 10s apart) for spend data to appear in DB
- Increase test timeout from 25s to 90s to accommodate polling
- base_anthropic_messages_tool_search_test.py: Add flaky(retries=3) for
  streaming test that depends on live Anthropic API

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(ci): reduce parallel workers from 8 to 4 for proxy tests to prevent OOM

- litellm_proxy_unit_testing_part2: -n 8 -> -n 4
- litellm_mapped_tests_proxy_part2: -n 8 -> -n 4, timeout 60 -> 120
- Worker crashes consistently caused by too many parallel proxy tests
  each loading the full FastAPI app and heavy dependency tree

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(db): add migration for SpendLogs composite index (startTime, request_id)

The @@index([startTime, request_id]) was added to schema.prisma but had no
corresponding migration. This caused test_aaaasschema_migration_check to fail
because prisma migrate diff detected the missing index.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(db): add migration for MCP available_on_public_internet default change to true

The schema.prisma changed the default for available_on_public_internet from
false to true, but no migration was created. This caused the schema migration
test to detect drift.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): increase server wait time and add retry to flaky external API tests

- test_basic_python_version.py: increase server startup wait from 60s to 90s
  for slower CI environments (fixes installing_litellm_on_python_3_13)
- test_a2a_agent.py: add flaky(retries=3, delay=5) for non-streaming test
  that depends on live A2A agent endpoint

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): add flaky retries to all intermittent external API tests for 0-fail CI

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix(test): add auth overrides to file endpoint tests that return 500

The test_target_storage tests were getting 500 because the FastAPI auth
dependency wasn't overridden. Added app.dependency_overrides for proper
auth bypass in test environment.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
2026-02-28 09:46:35 -08:00
yuneng-jiang
369c0ec392 [Infra] Add prisma_schema_sync CircleCI job before e2e UI tests
Adds a new CircleCI job that runs the proxy with --use_prisma_db_push
against the base Neon branch before the e2e UI tests create their
branches from it, ensuring the schema is synced on the parent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-26 20:03:25 -08:00
Ishaan Jaff
d95c3e9cd4
fix(ruff): add missing Set and Dict imports (F821) (#21785)
* fix(ruff): add missing Set and Dict imports to fix F821 errors

* fix(ci): add semantic_router dep to guardrails_testing job

The test_build_routes_combined_templates test calls build_routes() which
imports from semantic_router, but the guardrails_testing CI job didn't
install it.
2026-02-21 10:56:27 -08:00
Julio Quinteros Pro
24ead9b4e1 fix(ci): install enterprise package into main project venv, not enterprise's own venv
Running `cd enterprise && poetry run pip install -e .` causes poetry to
create a separate venv in `enterprise/.venv` (since enterprise/ has its
own pyproject.toml). The main project's tests run with `.venv/bin/python`,
so the enterprise package installed in `enterprise/.venv` is never seen.

Fix: run `poetry run pip install -e enterprise/` from the repo root so
poetry uses the main project's venv. This ensures litellm_enterprise is
importable when tests run.

This explains why enterprise tests kept failing with:
  AttributeError: '_PROXY_LiteLLMManagedFiles' object has no attribute
  '_check_file_deletion_allowed'
even after --force-reinstall was added — the reinstall was going to the
wrong virtual environment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-18 21:24:06 -03:00
Julio Quinteros Pro
bbbac1ae0e fix(ci): apply --force-reinstall --no-deps to enterprise install in all CI configs
The same PyPI-override issue existed in test-litellm.yml, test-mcp.yml,
and .circleci/config.yml. Also adds --no-deps (enterprise has no runtime
deps) to avoid redundant dependency resolution on every forced reinstall.

Addresses greptile review comments on PR #21481.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-18 18:53:45 -03:00
Ishaan Jaffer
0b1a9f6623 litellm_mapped_tests_proxy_part2 2026-02-14 15:37:18 -08:00
Ishaan Jaffer
0d697e6c3e litellm_mapped_tests_mcps 2026-02-14 15:20:25 -08:00
Ishaan Jaffer
bd766706c5 VertexAILlama3StreamingHandler 2026-02-14 13:18:26 -08:00
Ishaan Jaffer
ec7eb79634 ANTHROPIC_API_KEY 2026-02-14 12:20:55 -08:00
Sameer Kankute
bece16069e Add e2e tests for checking if all beta headers in the mapping works 2026-02-13 15:49:50 +05:30
Alexsander Hamir
b7993b14cf
Add semgrep & Fix OOMs (#20912) 2026-02-10 17:50:14 -08:00
yuneng-jiang
c68383068a
Merge pull request #20803 from BerriAI/litellm_ui_e2e_02
[Infra] UI - E2E Tests: Key Delete, Regenerate, and Update TPM/RPM Limits
2026-02-10 17:02:37 -08:00