mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
afec9b8ab9
* perf(ci): cache the Rust build the unit shards compile from scratch Every unit shard installs the workspace, and the root package builds through maturin, so each of the eleven jobs compiles litellm-rust/crates/python-bridge in release mode before a single test runs. That step measured 2m40s a shard on 2026-08-21, which is more wall clock than the entire unit tier spends running tests, and none of it was cached: the uv cache covers wheels it downloads, not wheels it builds, and a path dependency whose source moves every commit can never hit that cache anyway. A composite action now exports CARGO_TARGET_DIR to a fixed workspace path and caches it alongside the Cargo registry, keyed on Cargo.lock. Cargo rebuilds only what changed, so a warm job pays for the bridge crate rather than its whole dependency graph. Measured locally, that is 34s cold against 8s warm, including after a Python-only or Rust-only edit. The absolute path matters: uv builds the wheel from its own working directory, so a relative target directory lands the artifacts where nothing can find them again. * perf(ci): cache the Rust build in the other four workflows that sync the workspace code-quality, mcp, documentation and the schema.d.ts check each install the workspace and so each compile the bridge from scratch, measured at 138s, 177s, 163s and 154s on 2026-08-21. The lint job pays the same and is left to #37783, which already owns that file's setup section. * perf(ci): cache the Rust build in the lint job too * fix(ci): cache cargo's own target directory instead of redirecting it uv builds the wheel in place, so cargo already writes to litellm-rust/target, which test-rust.yml has cached all along. Redirecting CARGO_TARGET_DIR bought nothing and cost a GITHUB_ENV write that zizmor rejects as a code-execution path. * chore(ci): raise the job backstop for the added setup step The cargo cache is a fifth bounded setup step, so the base's setup ceiling goes 30m to 35m and every job budget follows: 55 to 60, and proxy-server's 95 to 100. check_workflow_startup_safety enforces exactly this sum, and failed on the first push without it. * perf(ci): cache the Rust build in the four remaining workflows that sync Six workflows were wired; ten install the workspace. The four left out still compile the pyo3 bridge from scratch. test-terraform-provider.yml is the one that matters per PR: its endpoint-drift job triggers on any change under litellm/proxy/**. The other three are a scheduled load check, a manual mutation run, and the staging-push counts publisher, whose gate syncs the project inside scripts/type_check_gate.py rather than in a workflow step, so nothing in the file names the build it pays for.
220 lines
8.1 KiB
YAML
220 lines
8.1 KiB
YAML
name: "Unit Tests"
|
|
|
|
on:
|
|
pull_request:
|
|
branches:
|
|
- main
|
|
- litellm_internal_staging
|
|
- litellm_oss_staging
|
|
- "litellm_**"
|
|
push:
|
|
branches:
|
|
- main
|
|
- litellm_internal_staging
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
# One caller for every tests/test_litellm shard, replacing the nine thin workflow
|
|
# files that each wrapped a single call to _test-unit-base.yml. Adding a shard is
|
|
# now one matrix entry rather than a new file.
|
|
#
|
|
# `name` is the shard id and nothing else, so each check reports as
|
|
# "<shard> / Run tests" exactly as it did when the shard had its own file. Those
|
|
# strings are the branch ruleset's required contexts, so they are load-bearing:
|
|
# renaming an entry renames a required check and the ruleset stops matching it.
|
|
#
|
|
# Every entry states its timeouts even when they equal the base workflow's
|
|
# defaults. An absent matrix key renders as an empty string, which is not a
|
|
# number, so a partially-specified entry would fail the call rather than fall
|
|
# back to the default.
|
|
#
|
|
# tests/proxy_unit_tests keeps its own caller (test-unit-proxy-db.yml): it is
|
|
# already a matrix and carries a shard-coverage guard that reads that file by
|
|
# name. Folding it in here is a follow-up, together with generalising that guard
|
|
# into assert_ci_coverage.py.
|
|
jobs:
|
|
unit:
|
|
name: ${{ matrix.shard }}
|
|
permissions:
|
|
contents: read
|
|
id-token: write
|
|
pull-requests: write
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- shard: core-utils
|
|
artifact-name: core-utils
|
|
test-path: "tests/test_litellm/litellm_core_utils"
|
|
workers: 2
|
|
reruns: 1
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: enterprise-routing
|
|
artifact-name: enterprise-routing
|
|
test-path: >-
|
|
tests/test_litellm/enterprise
|
|
tests/test_litellm/google_genai
|
|
tests/test_litellm/router_utils
|
|
tests/test_litellm/router_strategy
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: integrations
|
|
artifact-name: integrations
|
|
test-path: "tests/test_litellm/integrations"
|
|
workers: 2
|
|
reruns: 3
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: Vertex AI
|
|
artifact-name: llm-vertex-ai
|
|
test-path: "tests/test_litellm/llms/vertex_ai"
|
|
workers: 1
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: All Other Providers
|
|
artifact-name: llm-other-providers
|
|
test-path: "tests/test_litellm/llms --ignore=tests/test_litellm/llms/vertex_ai"
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: misc
|
|
artifact-name: misc
|
|
test-path: >-
|
|
tests/test_litellm/batches
|
|
tests/test_litellm/secret_managers
|
|
tests/test_litellm/a2a_protocol
|
|
tests/test_litellm/anthropic_interface
|
|
tests/test_litellm/completion_extras
|
|
tests/test_litellm/compression
|
|
tests/test_litellm/containers
|
|
tests/test_litellm/experimental_mcp_client
|
|
tests/test_litellm/models
|
|
tests/test_litellm/repositories
|
|
tests/test_litellm/images
|
|
tests/test_litellm/interactions
|
|
tests/test_litellm/ocr
|
|
tests/test_litellm/passthrough
|
|
tests/test_litellm/rag
|
|
tests/test_litellm/realtime_api
|
|
tests/test_litellm/rerank_api
|
|
tests/test_litellm/rust_bridge
|
|
tests/test_litellm/sandbox
|
|
tests/test_litellm/test_router
|
|
tests/test_litellm/vector_stores
|
|
tests/test_litellm/videos
|
|
tests/test_litellm/test_*.py
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: proxy-auth
|
|
artifact-name: proxy-auth
|
|
test-path: >-
|
|
tests/test_litellm/proxy/auth
|
|
tests/test_litellm/proxy/hooks
|
|
tests/test_litellm/proxy/policy_engine
|
|
tests/test_litellm/proxy/client
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: proxy-endpoints
|
|
artifact-name: proxy-endpoints
|
|
test-path: >-
|
|
tests/test_litellm/proxy/analytics_endpoints
|
|
tests/test_litellm/proxy/management_endpoints
|
|
tests/test_litellm/proxy/memory
|
|
tests/test_litellm/proxy/guardrails
|
|
tests/test_litellm/proxy/management_helpers
|
|
tests/test_litellm/proxy/anthropic_endpoints
|
|
tests/test_litellm/proxy/google_endpoints
|
|
tests/test_litellm/proxy/openai_files_endpoint
|
|
tests/test_litellm/proxy/batches_endpoints
|
|
tests/test_litellm/proxy/fine_tuning_endpoints
|
|
tests/test_litellm/proxy/vector_store_files_endpoints
|
|
tests/test_litellm/proxy/video_endpoints
|
|
tests/test_litellm/proxy/response_api_endpoints
|
|
tests/test_litellm/proxy/image_endpoints
|
|
tests/test_litellm/proxy/ocr_endpoints
|
|
tests/test_litellm/proxy/vector_store_endpoints
|
|
tests/test_litellm/proxy/agent_endpoints
|
|
tests/test_litellm/proxy/a2a
|
|
tests/test_litellm/proxy/credential_endpoints
|
|
tests/test_litellm/proxy/discovery_endpoints
|
|
tests/test_litellm/proxy/health_endpoints
|
|
tests/test_litellm/proxy/shutdown
|
|
tests/test_litellm/proxy/public_endpoints
|
|
tests/test_litellm/proxy/prompts
|
|
tests/test_litellm/proxy/rag_endpoints
|
|
tests/test_litellm/proxy/realtime_endpoints
|
|
tests/test_litellm/proxy/ui_crud_endpoints
|
|
tests/test_litellm/proxy/config_resolvers
|
|
tests/test_litellm/proxy/utils
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: proxy-server
|
|
artifact-name: proxy-server
|
|
test-path: "tests/test_litellm/proxy/proxy_server"
|
|
workers: 4
|
|
reruns: 2
|
|
timeout-minutes: 60
|
|
job-timeout-minutes: 100
|
|
|
|
- shard: proxy-infra
|
|
artifact-name: proxy-infra
|
|
test-path: >-
|
|
tests/test_litellm/proxy/db
|
|
tests/test_litellm/proxy/middleware
|
|
tests/test_litellm/proxy/spend_tracking
|
|
tests/test_litellm/proxy/pass_through_endpoints
|
|
tests/test_litellm/proxy/_experimental
|
|
tests/test_litellm/proxy/experimental
|
|
tests/test_litellm/proxy/common_utils
|
|
tests/test_litellm/proxy/enterprise_billing
|
|
tests/test_litellm/proxy/types_utils
|
|
tests/test_litellm/proxy/logging_endpoints
|
|
tests/test_litellm/proxy/test_*.py
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
|
|
- shard: responses-caching-types
|
|
artifact-name: responses-caching-types
|
|
test-path: >-
|
|
tests/test_litellm/responses
|
|
tests/test_litellm/caching
|
|
tests/test_litellm/types
|
|
workers: 2
|
|
reruns: 2
|
|
timeout-minutes: 20
|
|
job-timeout-minutes: 60
|
|
uses: ./.github/workflows/_test-unit-base.yml
|
|
with:
|
|
test-path: ${{ matrix.test-path }}
|
|
workers: ${{ matrix.workers }}
|
|
reruns: ${{ matrix.reruns }}
|
|
timeout-minutes: ${{ matrix.timeout-minutes }}
|
|
job-timeout-minutes: ${{ matrix.job-timeout-minutes }}
|
|
artifact-name: ${{ matrix.artifact-name }}
|