From 84969aaf15fa4c510279dde68b511a82cfdd49f8 Mon Sep 17 00:00:00 2001 From: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 3 Jun 2026 13:37:53 -0700 Subject: [PATCH] fix(ci): keep coverage rename green when a parallel node runs no tests (#29608) * fix(ci): keep coverage rename green when a parallel node runs no tests local_testing_part1 and local_testing_part2 run with parallelism 4. When CircleCI reruns only the failed tests, the failed test lands on a single node and the other nodes receive an empty bucket, so pytest never writes coverage.xml or .coverage. The unguarded "mv coverage.xml ..." then exits 1 and turns the whole job red even though the rerun passed; the next persist_to_workspace step would fail the same way on the missing paths. Guard the rename so a node with no coverage emits empty placeholders instead. coverage combine tolerates the empty files, so the downstream upload-coverage job keeps the real nodes' data intact. * fix(ci): pre-create test-results in litellm_router_testing for empty-bucket reruns litellm_router_testing also runs with parallelism 4. On a rerun of only the failed tests, a node can receive no tests, so the test command never creates test-results and the final store_test_results step can fail on the missing path. Pre-create the directory up front, matching what local_testing_part1 and part2 already do and CircleCI's own guidance for parallel reruns. * test(openai): retry wildcard chat completion on transient OpenAI 500 build_and_test reddened on test_openai_wildcard_chat_completion when the real gpt-3.5-turbo-0125 call returned an OpenAI 500 ("The server had an error while processing your request"). The base branch passed the same call concurrently, so the 500 is an intermittent OpenAI server error, not a regression. Add the same pytest-retry marker the sibling real-call tests in this file already use so a transient upstream 500 no longer fails CI. --- .circleci/config.yml | 27 +++++++++++++++++++++++---- tests/test_openai_endpoints.py | 1 + 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index f5a8fe77a25..6ee5634f54f 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -249,8 +249,15 @@ jobs: - run: name: Rename the coverage files command: | - mv coverage.xml local_testing_part1_coverage.xml - mv .coverage local_testing_part1_coverage + # When CI reruns only the failed tests, a parallel node can receive + # zero tests and pytest never writes coverage. Emit empty placeholders + # so persist_to_workspace and the downstream coverage combine stay green. + if [ -f coverage.xml ]; then + mv coverage.xml local_testing_part1_coverage.xml + mv .coverage local_testing_part1_coverage + else + touch local_testing_part1_coverage.xml local_testing_part1_coverage + fi # Store test results - store_test_results: @@ -314,8 +321,15 @@ jobs: - run: name: Rename the coverage files command: | - mv coverage.xml local_testing_part2_coverage.xml - mv .coverage local_testing_part2_coverage + # When CI reruns only the failed tests, a parallel node can receive + # zero tests and pytest never writes coverage. Emit empty placeholders + # so persist_to_workspace and the downstream coverage combine stay green. + if [ -f coverage.xml ]; then + mv coverage.xml local_testing_part2_coverage.xml + mv .coverage local_testing_part2_coverage + else + touch local_testing_part2_coverage.xml local_testing_part2_coverage + fi # Store test results - store_test_results: @@ -464,6 +478,11 @@ jobs: - run: name: Run tests command: | + # On a "rerun failed tests" build a parallel node can receive no + # tests, so the test command never creates test-results. Pre-create it + # so store_test_results doesn't fail the node on a missing path. + mkdir -p test-results + TEST_FILES=$(circleci tests glob "tests/local_testing/**/test_*.py") echo "$TEST_FILES" | circleci tests run \ diff --git a/tests/test_openai_endpoints.py b/tests/test_openai_endpoints.py index e898b88a556..880cc1ebbef 100644 --- a/tests/test_openai_endpoints.py +++ b/tests/test_openai_endpoints.py @@ -522,6 +522,7 @@ async def test_image_generation(): await image_generation(session=session, key=key_2) +@pytest.mark.flaky(retries=5, delay=1) @pytest.mark.asyncio async def test_openai_wildcard_chat_completion(): """