mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
3357ec8d34
* test: run the 30 test files stranded in the second mirror
tests/litellm sat beside tests/test_litellm, which is the mirror the repo
convention names, and no job collected it. The allowlist called the directory
unresolved and assumed it was a duplicate. It is not: 30 of its 34 files have no
counterpart in the real mirror, so they are tests nobody has run since they were
written, not copies of tests that run elsewhere.
Moving them in is byte-identical, and it is what makes them run. Every one is
now claimed by a shard's test-path rather than by an allowlist entry, and the
216 tests they hold pass. Directories that needed to become packages did, since
several files are named test_transformation.py and pytest cannot import two of
those from non-package directories in one session.
Never running is why three assertions had drifted away from the code:
* nvidia.nemotron-super-3-120b max_output_tokens, 32000 -> 32768
* sambanova/MiniMax-M2.7 max_input_tokens, 204800 -> 196608
* the Vertex text-to-speech handler moved from data= to json=, so the test
reads the decoded body off the json kwarg instead of parsing the data one
The first two follow model_prices_and_context_window.json, which the catalog
sync keeps current; the third follows the handler. In all three the test was the
stale side.
The lint workflow ran test_no_hardcoded_secrets.py by path and now points at the
new one.
Four files stay behind. Each shares a filename with a live test whose contents
are disjoint from it, so landing those means merging test bodies, which is a
content review rather than a move. The allowlist entry now names those four and
records how many tests each would bring, in place of calling the whole
directory unresolved.
* fix(ci): keep the secret scan out of the mirror's conftest
The secret-scan job runs pytest under uv run --no-project, so its environment
holds pytest and nothing else. That worked while the file sat in tests/litellm,
which has no conftest, and broke the moment it moved into tests/test_litellm,
whose conftest imports litellm on collection: ModuleNotFoundError: No module
named 'dotenv', before a single test ran.
The file is a repo-wide static scan that imports only base64, os, re and pytest,
so it belongs with the other repo-wide checks in tests/code_coverage_tests,
which has no conftest, rather than in the package mirror. Installing the full
dependency set into a 15-second job to satisfy a conftest it does not use would
be the wrong trade.
Verified with the job's exact command:
uv run --no-project --with 'pytest==9.0.2' pytest \
tests/code_coverage_tests/test_no_hardcoded_secrets.py -q
1 passed in 0.47s
134 lines
7.7 KiB
YAML
134 lines
7.7 KiB
YAML
description: >-
|
|
Paths deliberately outside CI coverage, each with the reason it is exempt.
|
|
assert_ci_coverage.py fails when a test file or Dockerfile is neither invoked
|
|
by a job nor listed here, so every entry below is a decision on the record.
|
|
|
|
test_paths:
|
|
- reason: >-
|
|
The caching suite in tests/local_testing, which runs nowhere. Every job that globs that
|
|
directory either deselects it (local_testing_part1 and part2 carry `-k "... and not caching
|
|
and not cache"`) or keeps only another keyword (langfuse, router, assistants), and no job
|
|
names these files the way redis_caching_unit_tests names test_dual_cache.py. Measured
|
|
2026-08-20 by collecting the directory under each job's own selector: 118 tests across
|
|
these eight files are selected by none of them. Listed so the gap is a decision rather
|
|
than an accident, and so the --slices guard has a baseline to ratchet down from. Revisit
|
|
when tests/local_testing is ported off CircleCI, where the keyless part of this suite
|
|
belongs in a real job
|
|
paths:
|
|
- tests/local_testing/test_cache_preset_key.py
|
|
- tests/local_testing/test_caching.py
|
|
- tests/local_testing/test_caching_handler.py
|
|
- tests/local_testing/test_disk_cache_unit_tests.py
|
|
- tests/local_testing/test_gcs_cache_unit_tests.py
|
|
- tests/local_testing/test_prompt_caching.py
|
|
- tests/local_testing/test_responses_stream_cache_keys.py
|
|
- tests/local_testing/test_unit_test_caching.py
|
|
- reason: >-
|
|
The end-to-end suite runs against a deployed proxy from its own in-cluster rig rather than
|
|
from a pull request; it needs a live gateway and provider credentials no PR job holds
|
|
paths:
|
|
- tests/e2e
|
|
- reason: >-
|
|
The documentation and code-quality workflows execute four files in this directory by name as
|
|
scripts and pytest never collects the directory, so these six run nowhere; listed individually
|
|
so a seventh cannot inherit the exemption
|
|
paths:
|
|
- tests/documentation_tests/test_exception_types.py
|
|
- tests/documentation_tests/test_general_setting_keys.py
|
|
- tests/documentation_tests/test_optional_params.py
|
|
- tests/documentation_tests/test_readme_providers.py
|
|
- tests/documentation_tests/test_requests_lib_usage.py
|
|
- tests/documentation_tests/test_standard_logging_payload.py
|
|
- reason: >-
|
|
Named like a test but shaped like a benchmark: it fetches live image URLs, times aiohttp
|
|
against httpx, prints the ratio, and asserts nothing, so pytest cannot collect it (its
|
|
functions take arguments, not fixtures) and running it beside its siblings in the
|
|
code-quality workflow would add a network dependency for a number nothing reads. Exempt
|
|
as a script rather than as an unresolved gap; revisit by deleting it once the aiohttp
|
|
choice it informed is settled
|
|
paths:
|
|
- tests/code_coverage_tests/test_aio_http_image_conversion.py
|
|
- reason: >-
|
|
What is left of a second mirror that sat beside tests/test_litellm and ran nowhere. Its
|
|
other 30 files moved into the real mirror on 2026-08-20 and now run; these four cannot,
|
|
because each shares a filename with a live test whose contents are disjoint from it, so
|
|
landing them means merging test bodies rather than moving a file. Measured on the same
|
|
date: test_common_utils.py holds 15 tests the live file does not, test_oci_chat_transformation
|
|
13, test_deepseek_chat_transformation 12, and test_discoverable_endpoints 5. Revisit by
|
|
merging each into its twin, which is a content review, not a move
|
|
paths:
|
|
- tests/litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py
|
|
- tests/litellm/llms/oci/chat/test_oci_chat_transformation.py
|
|
- tests/litellm/proxy/_experimental/mcp_server/test_discoverable_endpoints.py
|
|
- tests/litellm/proxy/management_endpoints/test_common_utils.py
|
|
- reason: >-
|
|
No job invokes this suite and its files mix pure transformation tests with ones driving live
|
|
vendor vector stores, so assigning them needs a per-file decision
|
|
paths:
|
|
- tests/vector_store_tests/rag/test_rag_bedrock.py
|
|
- tests/vector_store_tests/rag/test_rag_openai.py
|
|
- tests/vector_store_tests/rag/test_rag_s3_vectors.py
|
|
- tests/vector_store_tests/rag/test_rag_vertex_ai.py
|
|
- tests/vector_store_tests/test_azure_ai_vector_store.py
|
|
- tests/vector_store_tests/test_azure_vector_store.py
|
|
- tests/vector_store_tests/test_bedrock_vector_store.py
|
|
- tests/vector_store_tests/test_gemini_vector_store.py
|
|
- tests/vector_store_tests/test_milvus_vector_store.py
|
|
- tests/vector_store_tests/test_openai_vector_store.py
|
|
- tests/vector_store_tests/test_ragflow_vector_store.py
|
|
- tests/vector_store_tests/test_s3_vectors_vector_store.py
|
|
- tests/vector_store_tests/test_vertex_ai_search_api_vector_store.py
|
|
- tests/vector_store_tests/test_vertex_ai_vector_store.py
|
|
- reason: >-
|
|
Throughput and memory-growth measurements whose runtime and variance make them unsuitable for
|
|
a per-pull-request job
|
|
paths:
|
|
- tests/load_tests/test_datadog_load_test.py
|
|
- tests/load_tests/test_langsmith_load_test.py
|
|
- tests/load_tests/test_linear_memory_growth.py
|
|
- tests/load_tests/test_memory_usage.py
|
|
- tests/load_tests/test_otel_load_test.py
|
|
- tests/load_tests/test_vertex_embeddings_load_test.py
|
|
- tests/load_tests/test_vertex_load_tests.py
|
|
- reason: >-
|
|
A local-only agent rig: test_a2a_completion_bridge.py needs a LangGraph server on
|
|
localhost:2024 and test_a2a.py drives a live A2A endpoint, so neither can run in a
|
|
pull request job. Until 2026-08-20 the CircleCI agent job hid them behind a grep -v
|
|
that this census could not see; the glob now excludes them structurally and this entry
|
|
is the decision on the record. Revisit when the A2A bridge gets a recorded-wire fixture
|
|
paths:
|
|
- tests/agent_tests/local_only_agent_tests
|
|
- reason: >-
|
|
Third-party integration tests that skip themselves without OCI configuration or sandbox
|
|
credentials, neither of which a pull request job holds
|
|
paths:
|
|
- tests/integration/sandbox/test_e2b_sandbox.py
|
|
- tests/integration/test_oci_integration.py
|
|
- tests/integration/test_oci_proxy_integration.py
|
|
- reason: >-
|
|
A unit test for the proxy-extras package that no job invokes, while the package's other
|
|
tests live under tests/proxy_migration_tests. Measured 2026-08-20: 24 of its 28 tests pass
|
|
and the 4 in TestMigrationSQLIdempotency fail, because 13 migrations from 2026-03 onward use
|
|
bare CREATE TABLE, ADD COLUMN, CREATE INDEX and ADD CONSTRAINT rather than the guarded forms
|
|
this file requires. It also matches those keywords inside SQL comments, so two further
|
|
migrations are reported that are in fact fine. Wiring it up means deciding what to do about
|
|
the 13 first, and they cannot simply be edited: Prisma checksums an applied migration, so a
|
|
changed one breaks migrate deploy for existing installs
|
|
paths:
|
|
- tests/litellm-proxy-extras/test_litellm_proxy_extras_utils.py
|
|
|
|
dockerfiles:
|
|
- reason: >-
|
|
The dashboard container is a static Next.js export served by nginx, and the dashboard build
|
|
and lint workflows already exercise that output, so building the image adds no signal about it
|
|
paths:
|
|
- ui/Dockerfile
|
|
- reason: >-
|
|
The Rust gateway ships as its own chart and package with a separate release pipeline, so its
|
|
image is not part of this repo's Python image set
|
|
paths:
|
|
- litellm-rust/crates/ai-gateway/Dockerfile
|
|
- reason: >-
|
|
An example image under cookbook/ that is documentation rather than a shipped artifact
|
|
paths:
|
|
- cookbook/litellm-ollama-docker-image/Dockerfile
|