scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-15 23:31:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 213
litellm/tests/local_testing
History
mateo-berri 3fd556c75f fix(tests): unblock local_testing_part1 + litellm_router_testing on openai SDK 2.34+ 
The pin in pyproject.toml was relaxed from ==2.33.0 to >=2.20.0,<3.0.0
in the componentized-proxy PR (#27557), and uv.lock now resolves to
openai==2.36.0. OpenAI Python SDK 2.34 introduced Admin API Key support
and rewrote credential validation: the client constructor now enforces
credentials via an internal _enforce_credentials=True flag and rejects
api_key="" with `OpenAIError: Missing credentials. Please pass an
api_key, workload_identity, admin_api_key, or set the OPENAI_API_KEY
or OPENAI_ADMIN_KEY environment variable.`

Two tests built throwaway openai clients with `api_key=""` purely as
placeholders to feed into LiteLLM internals (the actual upstream `.create`
is patched out with a mock). With 2.34+ the constructor itself raises
before any assertion runs:

- tests/local_testing/test_exceptions.py::test_exception_with_headers
  (all 14 parametrizations across openai/azure x sync/async x call_type)
- tests/local_testing/test_router.py::test_router_dynamic_cooldown_correct_retry_after_time

Pass api_key="sk-test" — the value never matters because the upstream
call is mocked.

A third test, test_completion_perplexity_exception_on_openai_client,
asserts on the wording of the upstream auth error after deleting the
env vars. The SDK changed both the exception class (raised at
construction time, wrapped by LiteLLM as InternalServerError instead of
re-raising openai.AuthenticationError) and the message text. Rewrite
to catch any exception and assert on the env-var name (PERPLEXITY_API_KEY),
which is the stable substring of both the old and new error messages
and is what users actually look for. Also wrap the env-var restore in
try/finally so the originals are always restored on any failure path.
2026-05-16 14:10:24 -07:00
..
.litellm_cache refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
auto_router [Feat] Backend Router - Add Auto-Router powered by semantic-router (#12955) 2025-07-24 18:32:56 -07:00
example_config_yaml test: test 2026-03-28 19:17:38 -07:00
test_configs test: test 2026-03-28 19:17:38 -07:00
test_model_response_typing LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
azure_fine_tune.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
azure_speech.mp3 [Feat] Add Azure AVA TTS integration (#15749) 2025-10-20 16:52:23 -07:00
batch_job_results_furniture.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
cache_unit_tests.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
conftest.py test(vcr): classify cache verdicts, detect live calls, surface cost leaks 2026-05-13 00:31:47 +00:00
create_mock_standard_logging_payload.py chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
data_map.txt refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
eagle.wav refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
example.jsonl VertexAI non-jsonl file storage support (#9781) 2025-04-09 14:01:48 -07:00
gettysburg.wav refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
large_text.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
model_cost.json refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
openai_batch_completions.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
openai_batch_completions_router.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
speech_vertex.mp3 refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
stream_chunk_testdata.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_acompletion.py Complete o3 model support (#8183) 2025-02-02 22:36:37 -08:00
test_acompletion_fallbacks.py (core sdk fix) - fix fallbacks stuck in infinite loop (#7751) 2025-01-13 19:34:34 -08:00
test_acooldowns_router.py test: test 2026-03-28 19:17:38 -07:00
test_add_function_to_prompt.py LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
test_add_update_models.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_aim_guardrails.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_alangfuse.py test: update key names 2026-03-28 21:13:16 -07:00
test_amazing_vertex_completion.py fix(vertex): synthesize items for array types missing items entirely 2026-04-28 09:23:55 -07:00
test_anthropic_prompt_caching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_arize_ai.py test: rename env var 2026-03-28 20:27:39 -07:00
test_arize_phoenix.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_assistants.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_async_fn.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_auth_utils.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_azure_anthropic_sync_post.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_azure_content_safety.py (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
test_azure_openai.py test: test 2026-03-28 19:17:38 -07:00
test_azure_perf.py test: test 2026-03-28 19:17:38 -07:00
test_basic_python_version.py [Test] CI: add v2 migration resolver coverage with local Postgres 2026-04-21 14:40:11 -07:00
test_batch_completion_return_exceptions.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_completions.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_blocked_user_list.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_braintrust.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_budget_manager.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_cache_preset_key.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_caching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_caching_handler.py fix(caching): defer streaming cache-hit callbacks for all stream=True 2026-05-01 17:03:32 +05:30
test_caching_ssl.py Merge main and resolve conflict in test_router_client_init.py 2026-03-30 18:44:33 -07:00
test_class.py test: test 2026-03-28 19:17:38 -07:00
test_completion.py test(fireworks): mock remaining live smoke tests 2026-05-15 22:28:27 -07:00
test_completion_cost.py test(fireworks): mock remaining live smoke tests 2026-05-15 22:28:27 -07:00
test_completion_with_retries.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_config.py test: test 2026-03-28 19:17:38 -07:00
test_cost_calc.py Revert "Fix xdist test isolation: capture true defaults and poll instead of sleep" 2026-03-15 22:57:39 -07:00
test_custom_api_logger.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_custom_callback_input.py fix(tests): drop dall-e-only test classes; route live image tests via gpt-image-1 2026-05-12 16:16:59 -07:00
test_custom_llm.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_custom_logger.py Mark test_redis_cache_completion_stream as flaky with retries 2026-03-15 20:44:18 -07:00
test_disk_cache_unit_tests.py LiteLLM Minor Fixes & Improvements (11/12/2024) (#6705) 2024-11-12 22:50:51 +05:30
test_docker_no_network_on_deploy.py build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
test_dual_cache.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dynamic_rate_limit_handler.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_dynamodb_logs.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_embedding.py [Test] Tests: Stop parametrizing API keys into pytest test IDs (#27249) 2026-05-05 17:21:18 -07:00
test_exceptions.py fix(tests): unblock local_testing_part1 + litellm_router_testing on openai SDK 2.34+ 2026-05-16 14:10:24 -07:00
test_file_types.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_function_call_parsing.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_function_calling.py fix(anthropic): inject dummy tool without modify_params (#27620) 2026-05-11 09:50:16 -07:00
test_function_setup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gcs_bucket.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gcs_cache_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gemini_reasoning_content.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_llm_provider.py test: drop duplicate openrouter prefix-strip test 2026-04-25 18:06:25 -03:00
test_get_model_file.py Revert "Merge pull request #16590 from Chesars/refactor/remove-backup-file-dry-principle" 2026-04-25 17:10:41 -03:00
test_get_model_info.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_optional_params_embeddings.py fix(embeddings): allow dimensions param passthrough via allowed_openai_params for non-text-embedding-3 OpenAI models 2026-02-26 09:59:37 +05:30
test_get_optional_params_functions_not_supported.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_google_ai_studio_gemini.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_guardrails_ai.py LiteLLM Minor Fixes & Improvements (10/15/2024) (#6242) 2024-10-16 07:32:06 -07:00
test_helicone_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_http_parsing_utils.py test_http_parsing_utils.py 2025-07-10 18:20:41 -07:00
test_img_resize.py fix: Support WebP image format and avoid token calculation error (#7182) 2024-12-12 14:32:39 -08:00
test_lakera_ai_prompt_injection.py Merge pull request #9222 from BerriAI/litellm_snowflake_pr_mar_13 2025-03-13 21:35:39 -07:00
test_langchain_ChatLiteLLM.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_langsmith.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_least_busy_routing.py test: fixes because azure deactivated our account 2025-10-25 15:10:45 -07:00
test_litellm_max_budget.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_llm_guard.py [Refactor] Move LLM Guard, Secret Detection to Enterprise Pip packagea (#10782) 2025-05-13 09:42:22 -07:00
test_load_test_router_s3.py fix tests 2025-10-25 10:19:24 -07:00
test_loadtest_router.py test: test 2026-03-28 19:17:38 -07:00
test_logfire.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_logging.py LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
test_longer_context_fallback.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_lowest_cost_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lowest_latency_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lunary.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_max_tpm_rpm_limiter.py (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
test_mem_leak.py LiteLLM Minor Fixes & Improvements (10/30/2024) (#6519) 2024-11-02 00:44:32 +05:30
test_mem_usage.py fix tests 2025-10-25 10:19:24 -07:00
test_mock_request.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_model_alias_map.py fix(test): scope ERROR log assertion to LiteLLM logger in test_model_alias_map 2026-04-29 03:48:41 +00:00
test_model_max_token_adjust.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_multiple_deployments.py [Fix] TogetherAIConfig.get_supported_openai_params recursion 2026-04-16 17:20:58 -07:00
test_ollama.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ollama_local.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_ollama_local_chat.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_openai_moderations_hook.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_opik.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_pass_through_endpoints.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_profiling_router.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_prometheus_service.py [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
test_prompt_caching.py claude-sonnet-4-5-20250929 fix 2025-10-31 18:20:52 -07:00
test_prompt_injection_detection.py test: test 2026-03-28 19:17:38 -07:00
test_promptlayer_integration.py LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
test_provider_specific_config.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_pydantic.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_pydantic_namespaces.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_redis_batch_optimizations.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_register_model.py Revert "test_update_model_cost_map_url" 2025-12-22 12:41:30 +05:30
test_responses_stream_cache_keys.py fix(cache): persist and replay streamed Responses API requests (#24580) 2026-05-01 11:55:36 +05:30
test_router.py fix(tests): unblock local_testing_part1 + litellm_router_testing on openai SDK 2.34+ 2026-05-16 14:10:24 -07:00
test_router_auto_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_batch_completion.py test fix 2025-09-01 17:04:47 -07:00
test_router_budget_limiter.py test: test 2026-03-28 19:17:38 -07:00
test_router_caching.py test: test 2026-03-28 19:17:38 -07:00
test_router_client_init.py test_router_init_azure_service_principal_with_secret_with_environment_variables 2026-03-30 21:15:53 -07:00
test_router_cooldown_handlers.py test: test 2026-03-28 19:17:38 -07:00
test_router_custom_routing.py Optimize CI: parallelize router and guardrails test jobs, fix test isolation 2026-03-14 22:54:44 -07:00
test_router_debug_logs.py feat: routing groups ui 2026-05-04 18:09:14 -07:00
test_router_fallback_handlers.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_fallbacks.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_router_get_deployments.py Fix:add async_get_available_deployment_for_pass_through in code tests 2026-01-16 16:37:44 +05:30
test_router_max_parallel_requests.py Revert "Fix xdist test isolation: capture true defaults and poll instead of sleep" 2026-03-15 22:57:39 -07:00
test_router_pattern_matching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_retries.py fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var instead of hardcoding model (#21781) 2026-02-21 10:46:49 -08:00
test_router_timeout.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_router_utils.py test: test 2026-03-28 19:17:38 -07:00
test_router_with_fallbacks.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_rules.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_sagemaker.py Fix test_completion_sagemaker_messages_api retry flakiness 2026-03-15 13:57:50 -07:00
test_sagemaker_nova_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_scheduler.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_secret_detect_hook.py [Refactor] Move LLM Guard, Secret Detection to Enterprise Pip packagea (#10782) 2025-05-13 09:42:22 -07:00
test_spend_calculate_endpoint.py test fix 2025-09-01 17:04:47 -07:00
test_stream_chunk_builder.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_streaming.py [Fix] Tests: Replace deprecated openrouter/claude-3.7-sonnet with claude-sonnet-4.5 2026-05-04 17:51:50 -07:00
test_supabase_integration.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_team_config.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_text_completion.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_timeout.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_together_ai.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_tpm_rpm_routing_v2.py fix: drain logging worker in test_router_caching_ttl to remove flake 2026-04-23 14:48:02 -07:00
test_traceloop.py test: skip redundant test 2025-02-10 22:13:58 -08:00
test_ui_sso_helper_utils.py LiteLLM Minor Fixes & Improvements (10/17/2024) (#6293) 2024-10-17 22:09:11 -07:00
test_unit_test_caching.py style: black format test_unit_test_caching.py 2026-04-15 18:19:04 -07:00
test_update_spend.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_validate_environment.py refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
test_wandb.py LiteLLM Minor Fixes & Improvements (11/05/2024) (#6590) 2024-11-07 04:17:05 +05:30
user_cost.json refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
vertex_ai.jsonl refactor: move all testing to top-level of repo 2024-09-28 21:08:14 -07:00
vertex_batch_completions.jsonl (feat) add Vertex Batches API support in OpenAI format (#7032) 2024-12-04 19:40:28 -08:00
vertex_key.json test: update to new vertex ai keys 2026-03-28 20:19:05 -07:00
whitelisted_bedrock_models.txt Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00