test_router_timeout, test_timeout_streaming and test_openai_embedding_timeouts
asked api.openai.com for a response in 10 to 100 microseconds and asserted the
resulting exception was a timeout. No connect can finish in that window, so
socket.create_connection always walked the whole address list, and because it
re-raises only the LAST address's error, the assertion was decided by the order
getaddrinfo happened to return.
api.openai.com is dual-stack and the CI container has no usable IPv6, so a
trailing AAAA record made the last attempt fail with an OSError. httpcore maps
socket.timeout to ConnectTimeout but OSError to ConnectError, so the expected
APITimeoutError arrived as APIConnectionError and the job went red. The three
tests were really measuring DNS ordering, not litellm.
Point them at the fake OpenAI endpoint the suite already runs, ask for the
slow-endpoint model it already delays on, and give them a timeout comfortably
under that delay. The embeddings route did not honour slow-endpoint yet, so it
now delays the same way chat and text completions already do.
Each test also gained a failure on the success path. Without it a request that
returned instead of timing out fell out of the try block and the test passed on
a result it was written to reject.
* test(ci): serve /moderations from the canned OpenAI mock
The otel proxy E2E job points its `openai/*` wildcard deployment at the
canned mock, and #37492 made `get_model_list` agree with
`get_available_deployment` on bare model names. /moderations now resolves
`omni-moderation-latest` to that wildcard deployment the way
/chat/completions already did, so the request lands on the mock, which
never implemented the route and answers a bare 404.
Add /moderations and /v1/moderations to the mock, returning an
OpenAI-shaped response with one result per input item.
* style(ci): annotate the new moderations locals as Final
* test: point router/completion/triton tests at the local fake OpenAI endpoint
The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down
Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream
* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI
* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target
* fix(tests): keep fake OpenAI mock alive across xdist workers
ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves
Several CI jobs run the proxy against a model whose api_base is a shared
"fake OpenAI endpoint" hosted on Railway
(exampleopenaiendpoint-production.up.railway.app) so the E2E runs return
canned responses without paying for or depending on a live provider. When
that single deployment is down, every one of those jobs fails with
"404 Application not found" even though nothing in the PR is broken; the
whole repo is coupled to the uptime of one free external service.
This adds tests/_fake_openai_endpoint_server.py, a small canned-response
OpenAI-shaped server (chat, text, embeddings, streaming with usage, and the
"429" rate-limit special case), and a reusable start_fake_openai_endpoint
CircleCI command that runs it on host port 8190 and waits until healthy. The
affected jobs now inject FAKE_OPENAI_API_BASE pointing at the local server,
and the example configs they mount resolve api_base from that env var. The
intentionally bad fallback URL in proxy_server_config.yaml is left untouched
so the fallback test still exercises a failing upstream.
Wired into build_and_test, litellm_router_testing,
db_migration_disable_update_check, proxy_logging_guardrails_model_info_tests,
proxy_spend_accuracy_tests, proxy_multi_instance_tests,
proxy_store_model_in_db_tests, and proxy_build_from_pip_tests.
