test(ci): mark Railway-dependent proxy e2e tests as flaky

Four CircleCI tests intermittently fail with the upstream Railway-hosted
fake openai endpoint (https://exampleopenaiendpoint-production.up.railway.app)
returning 'Application not found' (cold-start) or timing out:

  - tests/spend_tracking_tests/test_spend_accuracy_tests.py::test_basic_spend_accuracy
  - tests/store_model_in_db_tests/test_callbacks_in_db.py::test_e2e_langfuse_callbacks_in_db
  - tests/test_fallbacks.py::test_chat_completion_with_retries
  - tests/test_fallbacks.py::test_chat_completion_with_fallbacks

Both staging and main are affected (same flake on both branches), so this
is not a code regression — it's the upstream test fixture being slow to
wake. Add @pytest.mark.flaky(reruns=2, reruns_delay=5) so transient
upstream failures don't fail the build, matching the pattern already used
in tests/test_callbacks_on_proxy.py for similar Railway-dependent tests.

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
This commit is contained in:
Cursor Agent 2026-05-20 02:10:43 +00:00
parent e59e34bed3
commit 618247b55f
No known key found for this signature in database
3 changed files with 4 additions and 0 deletions

View file

@ -206,6 +206,7 @@ async def fail_with_diagnostics(session, stage: str, expected: float, observed:
)
@pytest.mark.flaky(reruns=2, reruns_delay=5)
@pytest.mark.asyncio
async def test_basic_spend_accuracy():
"""

View file

@ -71,6 +71,7 @@ async def make_chat_completions_request() -> ChatCompletion:
return response
@pytest.mark.flaky(reruns=2, reruns_delay=5)
@pytest.mark.asyncio
async def test_e2e_langfuse_callbacks_in_db():

View file

@ -132,6 +132,7 @@ async def test_chat_completion_client_fallbacks(has_access):
pytest.fail("Expected this to work: {}".format(str(e)))
@pytest.mark.flaky(reruns=2, reruns_delay=5)
@pytest.mark.asyncio
async def test_chat_completion_with_retries():
"""
@ -156,6 +157,7 @@ async def test_chat_completion_with_retries():
assert headers["x-litellm-max-retries"] == "50"
@pytest.mark.flaky(reruns=2, reruns_delay=5)
@pytest.mark.asyncio
async def test_chat_completion_with_fallbacks():
"""