scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 82
litellm/tests/local_testing
History
yuneng-jiang 7d50a31eb5
test(e2e): move live-provider legacy tests into tests/e2e (#44120) 
* test(e2e): move live-provider legacy tests into tests/e2e

Port legacy tests that exercise real providers into the tests/e2e suites that own them, using the harness (/model/new plus deferred cleanup) and asserting on what the caller receives. Delete legacy tests already covered at equal or stronger strength by e2e, integration or unit tests, and drop the now empty ocr_testing CircleCI job

* test(e2e): address review on the live-provider test move

Assert the SSE error frame a client actually receives when a post_call guardrail blocks a stream, and require a tool call for every requested city before checking the answer. Restore the OCR matrix and its CircleCI job, the Claude Agent SDK streaming test, and test_async_create_batch, since their SDK-level and callback assertions have no equivalent in tests/e2e

* test(e2e): accept both guardrail block shapes on a blocked stream

A post_call block before the first chunk reaches the client as HTTP 400 with either a JSON error body or a single SSE error frame, depending on whether the block surfaced as an exception or an error chunk. Assert the policy message is present and the blocked output is absent in both

* test(realtime): restore direct SDK realtime tests against OpenAI

The e2e realtime tests go through the proxy and the remaining SDK tests either mock the upstream or assert less, so keep the direct litellm._arealtime tests with and without intent, and TestOpenAIRealtime::test_realtime_connection, in place

* test: make realtime and Nova stream checks deterministic

The direct SDK realtime tests now fail on a refused connection instead of skipping. The with-intent test asserts OpenAI rejects the exact intent value sent, which only happens when the intent is forwarded. The Nova /v1/messages stream test asserts stream structure, stop reason and usage instead of model wording

* test(realtime): own intent forwarding with a unit test instead of a live rejection

Assert litellm._arealtime passes the intent query param into the OpenAI realtime websocket URL, which is the behavior LiteLLM owns, and drop the live test that depended on OpenAI's rejection wording
2026-10-02 00:02:18 -07:00
..
.litellm_cache
auto_router [Feat] Backend Router - Add Auto-Router powered by semantic-router (#12955) 2025-07-24 18:32:56 -07:00
example_config_yaml test: test 2026-03-28 19:17:38 -07:00
test_configs test: test 2026-03-28 19:17:38 -07:00
azure_fine_tune.jsonl
azure_speech.mp3 [Feat] Add Azure AVA TTS integration (#15749) 2025-10-20 16:52:23 -07:00
batch_job_results_furniture.jsonl
cache_unit_tests.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
conftest.py test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) 2026-09-25 19:10:20 -07:00
create_mock_standard_logging_payload.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
data_map.txt
eagle.wav
example.jsonl VertexAI non-jsonl file storage support (#9781) 2025-04-09 14:01:48 -07:00
gettysburg.wav
large_text.py
model_cost.json
openai_batch_completions.jsonl
openai_batch_completions_router.jsonl
speech_vertex.mp3
stream_chunk_testdata.py
test_acompletion.py Complete o3 model support (#8183) 2025-02-02 22:36:37 -08:00
test_acompletion_fallbacks.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_acooldowns_router.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_add_function_to_prompt.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_aim_guardrails.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_alangfuse.py test(ci): refresh retired OpenAI tool-call models (#43676) 2026-09-29 21:53:59 -07:00
test_amazing_vertex_completion.py test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
test_anthropic_prompt_caching.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_arize_ai.py test: rename env var 2026-03-28 20:27:39 -07:00
test_arize_phoenix.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_assistants.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_async_fn.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_auth_utils.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_azure_anthropic_sync_post.py fix(mcp): honor server_id for REST tool calls with shared upstream URLs (#30184) 2026-06-12 07:25:53 -07:00
test_azure_openai.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_basic_python_version.py fix(ci): let the install smoke test boot its key-less proxy config 2026-09-21 12:56:21 -07:00
test_batch_completion_return_exceptions.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_completions.py test(batch): make the upstream-failure tolerance actually reachable 2026-08-28 00:25:53 -07:00
test_blocked_user_list.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_braintrust.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_caching.py fix(redis): authenticate sync clusters with IAM credential providers (#40204) 2026-09-23 17:23:55 -05:00
test_caching_ssl.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_completion.py test(ci): repair stale tests and move retired OpenAI text-completion fixtures (#43958) 2026-09-30 19:19:59 -07:00
test_completion_cost.py test(cost): repoint the Azure image cost test at gpt-image-2 (#42631) 2026-09-22 18:37:41 -07:00
test_completion_with_retries.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_config.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_cost_calc.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_custom_callback_input.py test: respect optional logging payload fields 2026-09-10 18:07:12 -07:00
test_custom_llm.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_custom_logger.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_disk_cache_unit_tests.py
test_docker_no_network_on_deploy.py build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
test_dynamic_rate_limit_handler.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_embedding.py test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
test_exceptions.py test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
test_fake_openai_endpoint.py test(ci): serve /moderations from the canned OpenAI mock (#37739) 2026-08-20 16:58:38 -07:00
test_file_types.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_function_call_parsing.py test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
test_function_calling.py test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
test_function_setup.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_gcs_bucket.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_gcs_cache_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gemini_reasoning_content.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_llm_provider.py test: remove substring guard test_default_api_base (#43355) 2026-09-26 22:28:28 +00:00
test_get_model_file.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_get_model_info.py test(bedrock): accept regional aliases that inherit Converse routing (#43785) 2026-09-29 23:30:06 -07:00
test_get_optional_params_embeddings.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_get_optional_params_functions_not_supported.py
test_google_ai_studio_gemini.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_guardrails_ai.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_handler_gc_does_not_close_client.py test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_helicone_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_http_parsing_utils.py test(ci): repair stale tests and move retired OpenAI text-completion fixtures (#43958) 2026-09-30 19:19:59 -07:00
test_img_resize.py
test_least_busy_routing.py fix(least-busy): share in-flight request counts across proxy workers 2026-09-05 20:39:22 -07:00
test_litellm_max_budget.py
test_llm_guard.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_longer_context_fallback.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_lowest_cost_routing.py test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
test_lowest_latency_routing.py fix(router): rank streaming latency routing by raw TTFT, not TTFT per token (#40202) 2026-09-08 10:05:23 -07:00
test_lunary.py test(ci): refresh retired OpenAI tool-call models (#43676) 2026-09-29 21:53:59 -07:00
test_mock_request.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_model_alias_map.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_multiple_deployments.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_no_top_level_test_invocations.py fix(tests): drop module-level test calls that break local_testing collection (#29520) 2026-06-02 13:07:05 -07:00
test_ollama.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_openai_moderations_hook.py test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
test_opik.py test(opik): stop the batching test racing its own 5-second flush timer 2026-08-27 23:43:38 -07:00
test_pass_through_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_profiling_router.py
test_prometheus_service.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_prompt_injection_detection.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_provider_specific_config.py chore(tests): remove commented-out hf, petals and vertex ai completion blocks 2026-09-17 20:03:06 +00:00
test_pydantic.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_pydantic_namespaces.py
test_register_model.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_router.py test(bedrock): point unit tests at model ids still in the cost map (#42606) 2026-09-22 16:04:44 -07:00
test_router_batch_completion.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_budget_limiter.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_router_caching.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_router_client_init.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_cooldown_handlers.py test: pin the post-#41289 cooldown contract and scroll the auto-router select spec 2026-09-16 20:52:19 -07:00
test_router_custom_routing.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_debug_logs.py test(router): ignore deployment-selection logs in the fallback log assertion 2026-09-15 19:45:33 -07:00
test_router_fallback_handlers.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_fallbacks.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_get_deployments.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_max_parallel_requests.py feat(router): reject with 429 when a deployment's max_parallel_requests slots are all in use 2026-09-17 18:14:09 +00:00
test_router_pattern_matching.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_retries.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_timeout.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_utils.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_router_with_fallbacks.py
test_rules.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_sagemaker.py fix(cost_calculator): bill chat per-second pricing once with a new cost_per_second field (#43614) 2026-09-29 11:27:14 -07:00
test_sagemaker_nova_integration.py test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
test_scheduler.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_secret_detect_hook.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_spend_calculate_endpoint.py test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
test_stream_chunk_builder.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_streaming.py test(ci): refresh qualified retired OpenAI fixtures (#43938) 2026-09-30 16:10:18 -07:00
test_supabase_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_team_config.py
test_text_completion.py test(ci): repair stale tests and move retired OpenAI text-completion fixtures (#43958) 2026-09-30 19:19:59 -07:00
test_timeout.py test(timeout): time out against the local fake endpoint instead of api.openai.com 2026-09-03 09:53:30 -07:00
test_together_ai.py
test_tpm_rpm_routing_v2.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_ui_sso_helper_utils.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_update_spend.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_validate_environment.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_wandb.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
user_cost.json
vertex_ai.jsonl
vertex_batch_completions.jsonl
vertex_key.json test: update to new vertex ai keys 2026-03-28 20:19:05 -07:00
whitelisted_bedrock_models.txt fix: repair seven regressions caught by CircleCI on main (#42640) 2026-09-23 02:26:36 +00:00