mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
8a103e26ab
* test: move routing, caching and callback tests out of local_testing Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion * test: cover least-busy routing and tighten migrated router, cache and pass-through tests Add router-level least-busy tests for the three legacy rows that had no destination, move completion fallback and model alias tests to their mirror files, move router batch tests next to the router, and strengthen tests whose mutants survived (specific deployment, batch pair grouping, custom cache key, router cache ttl). Add a unit test for configured pass-through Authorization forwarding. * test: tighten migrated callback, proxy config and cache ttl tests Assert the full StandardLoggingPayload key set and the wire user on callback tests, add an async embedding failure callback test, drive the prometheus redis failure through a real RedisCache, use the real salt key in proxy config tests, pin the redis ttl value and rename fallback test helpers * test: fix semantic cache assertions and order-dependent router and callback tests Count semantic cache embeddings on the route, drain async cache writes before the second lookup, echo the requested model from the callback provider mock so response_cost does not depend on model_cost entries other tests register, and move the two-key authorization assertion to the auth failure test it belongs to * test: cover model_list fan-out and the usage-based routing tpm limit lost in the move * test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard * test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file * ci: run the remaining router-keyword legacy tests on one node * test: refuse the redis write in-process instead of resolving an invalid host --------- Co-authored-by: yuneng <yuneng@berri.ai>
132 lines
7.2 KiB
YAML
132 lines
7.2 KiB
YAML
description: >-
|
|
Paths deliberately outside CI coverage, each with the reason it is exempt.
|
|
assert_ci_coverage.py fails when a test file or Dockerfile is neither invoked
|
|
by a job nor listed here, so every entry below is a decision on the record.
|
|
|
|
test_paths:
|
|
- reason: >-
|
|
litellm.agent() end-to-end suite. It drives the real claude, codex and opencode CLIs and
|
|
deepagents against a live LiteLLM AI Gateway, so it needs those binaries on PATH plus
|
|
LITELLM_PROXY_API_BASE / LITELLM_PROXY_API_KEY, and skips without them. Run manually
|
|
before changing litellm/harness; the mocked coverage runs in tests/unit/harness and
|
|
tests/unit/llms/*/harness
|
|
paths:
|
|
- tests/harness_e2e
|
|
- reason: >-
|
|
The Rust/Python parity harness is run manually through its local CLI. Recorded replay,
|
|
fixture generation, and harness checks are intentionally outside pull request CI
|
|
paths:
|
|
- tests/rust-python-harness
|
|
- reason: >-
|
|
Live-provider caching cases in tests/local_testing that remain outside CI. Jobs that
|
|
glob that directory either deselect them (local_testing_part1 and part2 carry `-k "... and
|
|
not caching and not cache"`) or keep only another keyword (langfuse, router, assistants).
|
|
The two Redis Cluster IAM authentication tests moved to tests/integration/sdk and remain
|
|
selected by node ID in redis_compat, where redis-server is installed.
|
|
The five keyless files now run in the caching-local shard, leaving live cases in these two.
|
|
Measured 2026-08-21 with no provider credentials: test_disk_cache_unit_tests.py needs
|
|
OPENAI_API_KEY for 2 of its 4 tests, and test_gcs_cache_unit_tests.py needs GCS credentials
|
|
for all 4. Keep these live cases outside PR CI rather than making a job red by construction
|
|
paths:
|
|
- tests/local_testing/test_disk_cache_unit_tests.py
|
|
- tests/local_testing/test_gcs_cache_unit_tests.py
|
|
- reason: >-
|
|
Mixed file that no job has ever run in full. Every CircleCI job that globs
|
|
tests/local_testing deselects it by name ("caching") or keeps only another keyword, so only
|
|
its router test ran, and that test now lives in tests/unit/router_utils/pre_call_checks.
|
|
Its two offline tests now live in tests/unit/llms/anthropic/chat, and the five left all need
|
|
ANTHROPIC_API_KEY or Vertex credentials
|
|
paths:
|
|
- tests/local_testing/test_anthropic_prompt_caching.py
|
|
- reason: >-
|
|
The end-to-end suite runs against a deployed proxy from its own in-cluster rig rather than
|
|
from a pull request; it needs a live gateway and provider credentials no PR job holds
|
|
paths:
|
|
- tests/e2e
|
|
- reason: >-
|
|
The documentation and code-quality workflows execute four files in this directory by name as
|
|
scripts and pytest never collects the directory, so these six run nowhere; listed individually
|
|
so a seventh cannot inherit the exemption
|
|
paths:
|
|
- tests/documentation_tests/test_exception_types.py
|
|
- tests/documentation_tests/test_general_setting_keys.py
|
|
- tests/documentation_tests/test_optional_params.py
|
|
- tests/documentation_tests/test_readme_providers.py
|
|
- tests/documentation_tests/test_requests_lib_usage.py
|
|
- tests/documentation_tests/test_standard_logging_payload.py
|
|
- reason: >-
|
|
Named like a test but shaped like a benchmark: it fetches live image URLs, times aiohttp
|
|
against httpx, prints the ratio, and asserts nothing, so pytest cannot collect it (its
|
|
functions take arguments, not fixtures) and running it beside its siblings in the
|
|
code-quality workflow would add a network dependency for a number nothing reads. Exempt
|
|
as a script rather than as an unresolved gap; revisit by deleting it once the aiohttp
|
|
choice it informed is settled
|
|
paths:
|
|
- tests/code_coverage_tests/test_aio_http_image_conversion.py
|
|
- reason: >-
|
|
No job invokes this suite and its files mix pure transformation tests with ones driving live
|
|
vendor vector stores, so assigning them needs a per-file decision
|
|
paths:
|
|
- tests/vector_store_tests/rag/test_rag_bedrock.py
|
|
- tests/vector_store_tests/rag/test_rag_openai.py
|
|
- tests/vector_store_tests/rag/test_rag_s3_vectors.py
|
|
- tests/vector_store_tests/rag/test_rag_vertex_ai.py
|
|
- tests/vector_store_tests/test_azure_ai_vector_store.py
|
|
- tests/vector_store_tests/test_azure_vector_store.py
|
|
- tests/vector_store_tests/test_bedrock_vector_store.py
|
|
- tests/vector_store_tests/test_gemini_vector_store.py
|
|
- tests/vector_store_tests/test_milvus_vector_store.py
|
|
- tests/vector_store_tests/test_openai_vector_store.py
|
|
- tests/vector_store_tests/test_ragflow_vector_store.py
|
|
- tests/vector_store_tests/test_s3_vectors_vector_store.py
|
|
- tests/vector_store_tests/test_vertex_ai_search_api_vector_store.py
|
|
- tests/vector_store_tests/test_vertex_ai_vector_store.py
|
|
- reason: >-
|
|
Throughput and memory-growth measurements whose runtime and variance make them unsuitable for
|
|
a per-pull-request job
|
|
paths:
|
|
- tests/load_tests/test_datadog_load_test.py
|
|
- tests/load_tests/test_langsmith_load_test.py
|
|
- tests/load_tests/test_linear_memory_growth.py
|
|
- tests/load_tests/test_memory_usage.py
|
|
- tests/load_tests/test_otel_load_test.py
|
|
- tests/load_tests/test_vertex_embeddings_load_test.py
|
|
- tests/load_tests/test_vertex_load_tests.py
|
|
- reason: >-
|
|
Env-gated saturation benchmark requires a live proxy and provider credentials, so it is run
|
|
locally rather than in pull-request jobs
|
|
paths:
|
|
- tests/load_tests/test_granian_admission_saturation.py
|
|
- reason: >-
|
|
A local-only agent rig: test_a2a_completion_bridge.py needs a LangGraph server on
|
|
localhost:2024 and test_a2a.py drives a live A2A endpoint, so neither can run in a
|
|
pull request job. Until 2026-08-20 the CircleCI agent job hid them behind a grep -v
|
|
that this census could not see; the glob now excludes them structurally and this entry
|
|
is the decision on the record. Revisit when the A2A bridge gets a recorded-wire fixture
|
|
paths:
|
|
- tests/agent_tests/local_only_agent_tests
|
|
- reason: >-
|
|
Third-party integration tests that skip themselves without OCI configuration or sandbox
|
|
credentials, neither of which a pull request job holds
|
|
paths:
|
|
- tests/integration/sandbox/test_e2b_sandbox.py
|
|
- tests/integration/test_oci_integration.py
|
|
- tests/integration/test_oci_proxy_integration.py
|
|
|
|
dockerfiles:
|
|
- reason: >-
|
|
The dashboard container is a static Next.js export served by nginx, and the dashboard build
|
|
and lint workflows already exercise that output, so building the image adds no signal about it
|
|
paths:
|
|
- ui/Dockerfile
|
|
- reason: >-
|
|
An example image under cookbook/ that is documentation rather than a shipped artifact
|
|
paths:
|
|
- cookbook/litellm-ollama-docker-image/Dockerfile
|
|
- reason: >-
|
|
The Rust gateway image compiles the whole workspace in release mode, which is too slow for
|
|
a per-pull-request job while the gateway binary is still being assembled; the Rust lint,
|
|
clippy, and compile jobs already cover the code it packages. Revisit when the gateway is
|
|
published
|
|
paths:
|
|
- litellm-rust/crates/gateway/Dockerfile
|