scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 35
litellm/tests/local_testing
History
Yuneng Jiang 075781568d
test: remove tests that never execute 
Three groups, all verified by running the suite rather than by inspection.

18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.

30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.

9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.

One file that is a demo script rather than a test; its own docstring says to run
it with python.

Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
2026-08-12 10:45:38 -07:00
..
.litellm_cache refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
auto_router [Feat] Backend Router - Add Auto-Router powered by semantic-router (#12955) 2025-07-24 18:32:56 -07:00
example_config_yaml test: test 2026-03-28 19:17:38 -07:00
test_configs test: test 2026-03-28 19:17:38 -07:00
test_model_response_typing LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
azure_fine_tune.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
azure_speech.mp3 [Feat] Add Azure AVA TTS integration (#15749) 2025-10-20 16:52:23 -07:00
batch_job_results_furniture.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
cache_unit_tests.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
conftest.py feat: declarative fallback generalizations for unknown models (#29718) 2026-06-27 21:01:19 -07:00
create_mock_standard_logging_payload.py chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
data_map.txt refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
eagle.wav refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
example.jsonl VertexAI non-jsonl file storage support (#9781) 2025-04-09 14:01:48 -07:00
gettysburg.wav refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
large_text.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
model_cost.json refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
openai_batch_completions.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
openai_batch_completions_router.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
speech_vertex.mp3 refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
stream_chunk_testdata.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_acompletion.py Complete o3 model support (#8183) 2025-02-02 22:36:37 -08:00
test_acompletion_fallbacks.py (core sdk fix) - fix fallbacks stuck in infinite loop (#7751) 2025-01-13 19:34:34 -08:00
test_acooldowns_router.py test: test 2026-03-28 19:17:38 -07:00
test_add_function_to_prompt.py LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
test_aim_guardrails.py fix(guardrails): return 400 not 500 when AIM blocks a request (#30573) 2026-06-16 18:56:14 -07:00
test_alangfuse.py test: update key names 2026-03-28 21:13:16 -07:00
test_amazing_vertex_completion.py test: remove tests that never execute 2026-08-12 10:45:38 -07:00
test_anthropic_prompt_caching.py fix(anthropic): self-heal on missing thinking-signature errors from Bedrock/Vertex (#33719) 2026-07-17 18:18:38 +00:00
test_arize_ai.py test: rename env var 2026-03-28 20:27:39 -07:00
test_arize_phoenix.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_assistants.py test(vcr): close out the remaining VCR live-call leaks (#29603) 2026-06-03 13:46:43 -07:00
test_async_fn.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_auth_utils.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_azure_anthropic_sync_post.py fix(mcp): honor server_id for REST tool calls with shared upstream URLs (#30184) 2026-06-12 07:25:53 -07:00
test_azure_openai.py test: test 2026-03-28 19:17:38 -07:00
test_azure_perf.py test: test 2026-03-28 19:17:38 -07:00
test_basic_python_version.py fix(deps): raise aiohttp floor to 3.14.2 to clear pooled-connection timeouts 2026-07-31 00:08:20 -07:00
test_batch_completion_return_exceptions.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_completions.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_blocked_user_list.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_braintrust.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_budget_manager.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_cache_preset_key.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_caching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_caching_handler.py fix(caching): replay openai/responses bridge cache hits as chat streams (#28158) 2026-05-18 16:27:06 -07:00
test_caching_ssl.py Merge main and resolve conflict in test_router_client_init.py 2026-03-30 18:44:33 -07:00
test_class.py test: test 2026-03-28 19:17:38 -07:00
test_completion.py test: remove tests that never execute 2026-08-12 10:45:38 -07:00
test_completion_cost.py test(fireworks): mock remaining live smoke tests 2026-05-15 22:28:27 -07:00
test_completion_with_retries.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_config.py test(proxy): assert _delete_deployment's still-desired id set instead of a delete count 2026-08-01 14:36:30 -07:00
test_cost_calc.py fix(main): stop per-request custom pricing from clobbering shared model_cost pricing (#32163) 2026-07-07 10:25:31 -07:00
test_custom_callback_input.py fix(tests): replace shut-down gpt-4o-audio-preview with gpt-audio-1.5 (#28281) 2026-05-19 14:48:30 -07:00
test_custom_llm.py Litellm oss staging 040626 (#29671) 2026-06-04 11:07:20 -07:00
test_custom_logger.py Mark test_redis_cache_completion_stream as flaky with retries 2026-03-15 20:44:18 -07:00
test_disk_cache_unit_tests.py LiteLLM Minor Fixes & Improvements (11/12/2024) (#6705) 2024-11-12 22:50:51 +05:30
test_docker_no_network_on_deploy.py build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
test_dual_cache.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dynamic_rate_limit_handler.py test: remove tests that never execute 2026-08-12 10:45:38 -07:00
test_embedding.py fix(bedrock): drop strict/additionalProperties from toolSpec for Claude Sonnet 4 (#31943) 2026-07-01 23:56:25 -07:00
test_exceptions.py fix: cleanup tests 2026-03-30 16:24:35 -07:00
test_fake_openai_endpoint.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
test_file_types.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_function_call_parsing.py Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326) 2026-05-30 11:26:24 -07:00
test_function_calling.py test(bedrock): repoint live Claude tests off the retired Claude 3 Sonnet 2026-08-11 18:49:47 -07:00
test_function_setup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gcs_bucket.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gcs_cache_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gemini_reasoning_content.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_llm_provider.py refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds 2026-07-11 00:27:45 -07:00
test_get_model_file.py Revert "Merge pull request #16590 from Chesars/refactor/remove-backup-file-dry-principle" 2026-04-25 17:10:41 -03:00
test_get_model_info.py fix(model_prices): advertise native structured output on every Bedrock DeepSeek V3.2 and GLM 5 id 2026-08-11 18:30:18 -07:00
test_get_optional_params_embeddings.py Litellm oss staging (#29492) 2026-06-02 08:48:10 -07:00
test_get_optional_params_functions_not_supported.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_google_ai_studio_gemini.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_guardrails_ai.py LiteLLM Minor Fixes & Improvements (10/15/2024) (#6242) 2024-10-16 07:32:06 -07:00
test_helicone_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_http_parsing_utils.py test_http_parsing_utils.py 2025-07-10 18:20:41 -07:00
test_img_resize.py fix: Support WebP image format and avoid token calculation error (#7182) 2024-12-12 14:32:39 -08:00
test_langchain_ChatLiteLLM.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_least_busy_routing.py test: fixes because azure deactivated our account 2025-10-25 15:10:45 -07:00
test_litellm_max_budget.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_llm_guard.py fix(llm_guard): apply sanitized prompt returned by moderation API to request (#33331) 2026-07-16 01:27:44 +03:00
test_load_test_router_s3.py fix tests 2025-10-25 10:19:24 -07:00
test_loadtest_router.py test: test 2026-03-28 19:17:38 -07:00
test_logging.py LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
test_longer_context_fallback.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_lowest_cost_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lowest_latency_routing.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
test_lunary.py fix(tests): drop module-level test calls that break local_testing collection (#29520) 2026-06-02 13:07:05 -07:00
test_max_tpm_rpm_limiter.py (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
test_mem_leak.py LiteLLM Minor Fixes & Improvements (10/30/2024) (#6519) 2024-11-02 00:44:32 +05:30
test_mem_usage.py fix tests 2025-10-25 10:19:24 -07:00
test_mock_request.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_model_alias_map.py fix(test): scope ERROR log assertion to LiteLLM logger in test_model_alias_map 2026-04-29 03:48:41 +00:00
test_multiple_deployments.py fix(tests): drop module-level test calls that break local_testing collection (#29520) 2026-06-02 13:07:05 -07:00
test_no_top_level_test_invocations.py fix(tests): drop module-level test calls that break local_testing collection (#29520) 2026-06-02 13:07:05 -07:00
test_ollama.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ollama_local.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_ollama_local_chat.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_openai_moderations_hook.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_opik.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_pass_through_endpoints.py fix(passthrough): stream non-sse passthrough responses instead of buffering in memory (#32386) 2026-07-07 20:51:15 -07:00
test_profiling_router.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_prometheus_service.py [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
test_prompt_caching.py claude-sonnet-4-5-20250929 fix 2025-10-31 18:20:52 -07:00
test_prompt_injection_detection.py test: test 2026-03-28 19:17:38 -07:00
test_provider_specific_config.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_pydantic.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_pydantic_namespaces.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_redis_batch_optimizations.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_register_model.py fix(tests): drop import-time completion call in test_register_model (#29521) 2026-06-02 16:10:43 -07:00
test_responses_stream_cache_keys.py fix(cache): persist and replay streamed Responses API requests (#24580) 2026-05-01 11:55:36 +05:30
test_router.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
test_router_batch_completion.py test fix 2025-09-01 17:04:47 -07:00
test_router_budget_limiter.py test: test 2026-03-28 19:17:38 -07:00
test_router_caching.py test: test 2026-03-28 19:17:38 -07:00
test_router_client_init.py test_router_init_azure_service_principal_with_secret_with_environment_variables 2026-03-30 21:15:53 -07:00
test_router_cooldown_handlers.py test: test 2026-03-28 19:17:38 -07:00
test_router_custom_routing.py chore: litellm oss staging (#31185) 2026-06-26 09:17:44 -07:00
test_router_debug_logs.py feat(cli): per-agent lite claude / codex / opencode commands that wrap coding agents through the proxy (#29850) 2026-06-10 13:52:26 -07:00
test_router_fallback_handlers.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
test_router_fallbacks.py fix(main): stop per-request custom pricing from clobbering shared model_cost pricing (#32163) 2026-07-07 10:25:31 -07:00
test_router_get_deployments.py Fix:add async_get_available_deployment_for_pass_through in code tests 2026-01-16 16:37:44 +05:30
test_router_max_parallel_requests.py fix(tests/vcr): make Redis cassette cache replay deterministically (zero VCR misses on consecutive runs) (#28826) 2026-05-26 11:30:44 -07:00
test_router_pattern_matching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_retries.py fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var instead of hardcoding model (#21781) 2026-02-21 10:46:49 -08:00
test_router_timeout.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_router_utils.py test: test 2026-03-28 19:17:38 -07:00
test_router_with_fallbacks.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_rules.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_sagemaker.py Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326) 2026-05-30 11:26:24 -07:00
test_sagemaker_nova_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_scheduler.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_secret_detect_hook.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
test_spend_calculate_endpoint.py test fix 2025-09-01 17:04:47 -07:00
test_stream_chunk_builder.py fix(tests): replace shut-down gpt-4o-audio-preview with gpt-audio-1.5 (#28281) 2026-05-19 14:48:30 -07:00
test_streaming.py test(bedrock): repoint live Claude tests off the retired Claude 3 Sonnet 2026-08-11 18:49:47 -07:00
test_supabase_integration.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_team_config.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_text_completion.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_timeout.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_together_ai.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_tpm_rpm_routing_v2.py fix: drain logging worker in test_router_caching_ttl to remove flake 2026-04-23 14:48:02 -07:00
test_ui_sso_helper_utils.py LiteLLM Minor Fixes & Improvements (10/17/2024) (#6293) 2024-10-17 22:09:11 -07:00
test_unit_test_caching.py style: black format test_unit_test_caching.py 2026-04-15 18:19:04 -07:00
test_update_spend.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_validate_environment.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_wandb.py fix(tests): drop module-level test calls that break local_testing collection (#29520) 2026-06-02 13:07:05 -07:00
user_cost.json refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
vertex_ai.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
vertex_batch_completions.jsonl (feat) add Vertex Batches API support in OpenAI format (#7032) 2024-12-04 19:40:28 -08:00
vertex_key.json test: update to new vertex ai keys 2026-03-28 20:19:05 -07:00
whitelisted_bedrock_models.txt Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00