mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
adf41853ee
Adds a vendored copy of BerriAI/example_openai_endpoint under tests/mock_endpoints/example_openai_endpoint/ together with infrastructure to run it in-process during a test session, so CI no longer depends on the public Railway deployment (which has had multiple outages, e.g. the May 19 2026 GCP-account incident that took down 8 CI jobs). Pieces added: - tests/mock_endpoints/example_openai_endpoint/ vendored upstream source (main.py, batch_and_files_api.py, requirements.txt, Dockerfile), intended to be re-vendored as-is from upstream when needed (see README). - tests/mock_endpoints/__init__.py helper module exposing MOCK_OPENAI_BASE_URL (reads LITELLM_MOCK_OPENAI_BASE_URL with the Railway URL as fallback) and start_mock_server() — boots the server as a subprocess on a free port, polls /chat/completions until it responds, returns an RAII-style handle for teardown. - tests/mock_endpoints/conftest.py session-scoped pytest fixture mock_openai_endpoint_server that any suite can opt into via pytest_plugins. - tests/mock_endpoints/start_mock_server.sh bash launcher (foreground or --background) for shell-driven workflows. - tests/mock_endpoints/README.md local + CI usage docs and instructions for re-vendoring from upstream. - tests/test_litellm/mock_endpoints/ end-to-end smoke tests that boot the mock and exercise /chat/completions, /v1/embeddings, and the fine-tuning list endpoint. Wiring: - New mock-server dependency-group in pyproject.toml (just adds slowapi; the rest of the upstream deps already come in via dev/proxy-dev). - _test-unit-base.yml now syncs --group mock-server so any reusable unit test job has the deps available. - test-unit-misc.yml runs the new smoke tests as part of its path set. - Makefile gains 'make mock-server' (and an install-mock-server helper) for running the server locally with the project venv. The Railway URL is still the runtime fallback for any test that has not been migrated, so this PR is non-breaking; follow-up PRs can migrate hard-coded URLs to use MOCK_OPENAI_BASE_URL / the new fixture. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| mock_endpoints | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _flush_vcr_cache.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| eval_swe_bench.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| README.MD | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.