scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-16 04:29:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 138
litellm/tests/llm_translation
History
ryan-crabbe-berri 243ed4393d test: reject assertions on a caught error inside except (ruff PT017) 
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.

Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
2026-08-21 13:35:08 -07:00
..
fixtures test fix 2025-10-16 18:00:46 -07:00
realtime test(realtime): record and replay websocket traffic in redis vcr cassettes (#32390) 2026-07-08 00:19:06 -07:00
reasoning_effort_grid test: unstale the reasoning-effort grid count and the responses bridge test (#34868) 2026-07-27 18:07:04 -07:00
test-skill [Feat] New API - Claude Skills API (Anthropic) (#17042) 2025-11-24 15:01:40 -08:00
test_llm_response_utils test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_skills_data Remove Apache 2 license from SKILL.md (#22322) 2026-02-27 19:33:55 -08:00
base_audio_transcription_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
base_embedding_unit_tests.py Litellm dev 12 25 2025 p2 (#7420) 2024-12-25 18:35:34 -08:00
base_llm_unit_tests.py fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke (#31364) 2026-06-27 11:35:36 -07:00
base_rerank_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
conftest.py test(realtime): record and replay websocket traffic in redis vcr cassettes (#32390) 2026-07-08 00:19:06 -07:00
dog.wav (feat) Support audio param in responses streaming (#6312) 2024-10-18 19:16:14 +05:30
duck.png fix vertex ai multimodal embedding translation (#9471) 2025-03-24 23:23:28 -07:00
gettysburg.wav Litellm dev 12 25 2025 p2 (#7420) 2024-12-25 18:35:34 -08:00
guinea.png fix vertex ai multimodal embedding translation (#9471) 2025-03-24 23:23:28 -07:00
log.xt Litellm dev 04 05 2025 p2 (#9774) 2025-04-07 21:02:52 -07:00
Readme.md test: add 24hr Redis-backed VCR cache to additional test suites (#27159) 2026-05-05 15:13:31 -07:00
test_a2a.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_anthropic_completion.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_aws_base_llm.py Add support for AWS assume_role with a session token 2025-08-23 22:37:21 -07:00
test_azure_agents.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_azure_ai.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_azure_o_series.py [Fix] CI: Enable VCR replay for test_azure_o_series 2026-05-04 20:48:26 -07:00
test_azure_openai.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_bedrock_agentcore.py Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326) 2026-05-30 11:26:24 -07:00
test_bedrock_agents.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_bedrock_anthropic_regression.py fix(tests): replace deprecated Bedrock Claude 3.7 Sonnet model ID 2026-04-28 14:24:19 -07:00
test_bedrock_common_utils.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_bedrock_completion.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_bedrock_dynamic_auth_params_unit_tests.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_bedrock_embedding.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_bedrock_embedding_pricing.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_bedrock_govcloud.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_bedrock_gpt_oss.py [Test] add request-body mock test for bedrock gpt-oss tool schema 2026-04-14 19:36:57 -07:00
test_bedrock_invoke_tests.py test(vcr): close out the remaining VCR live-call leaks (#29603) 2026-06-03 13:46:43 -07:00
test_bedrock_llama.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_bedrock_mantle.py fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpo… (#27976) 2026-05-15 13:31:59 -07:00
test_bedrock_moonshot.py test: cover generic HTTP streaming provider header forwarding 2026-08-20 19:47:05 -07:00
test_bedrock_nova_embedding.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_bedrock_nova_json.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cloudflare.py fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint (#31053) 2026-06-23 12:09:11 -07:00
test_cohere.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_containers_api.py test: reject assertions on a caught error inside except (ruff PT017) 2026-08-21 13:35:08 -07:00
test_convert_dict_to_image.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_crusoe.py fix(crusoe): remove trailing slashes from API base URLs and fix list indentation 2026-05-01 17:27:52 +05:30
test_databricks.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_deepgram.py Litellm dev 12 28 2024 p3 (#7464) 2024-12-28 19:18:58 -08:00
test_deepseek_completion.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_elevenlabs.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_evals_api.py [Fix] Tests: Reduce VCR cassette bloat and fix multipart caching 2026-05-07 11:54:19 -07:00
test_fireworks_ai_translation.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_gemini.py test: reject assertions on a caught error inside except (ruff PT017) 2026-08-21 13:35:08 -07:00
test_gemini_image_usage.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gigachat.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gpt4o_audio.py fix(tests): replace shut-down gpt-4o-audio-preview with gpt-audio-1.5 (#28281) 2026-05-19 14:48:30 -07:00
test_groq.py test: point the live gemini and groq conformance suites at models that still exist (#37422) 2026-08-19 02:50:46 +00:00
test_hosted_vllm_embedding_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_huggingface_chat_completion.py Revert "Revert "fix tests (#12286)"" 2025-07-03 12:08:27 -07:00
test_hyperbolic.py test: reject assertions on a caught error inside except (ruff PT017) 2026-08-21 13:35:08 -07:00
test_infinity.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_jina_ai.py Litellm 12 02 2024 (#6994) 2024-12-02 22:00:01 -08:00
test_lambda_ai.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_langgraph.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_litellm_proxy_provider.py test: reject assertions on a caught error inside except (ruff PT017) 2026-08-21 13:35:08 -07:00
test_minimax_tts.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_mistral_api.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_model_cost_map_resilience.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_morph.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_nvidia_nim.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_openai.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_openai_o1.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_openai_record_replay_proxy.py Extend the record/replay proxy to chat, embeddings, moderations, rerank, and Anthropic (#29847) 2026-06-06 14:33:42 -07:00
test_openrouter.py Fix deprecated model test 2026-05-11 09:49:47 +05:30
test_optional_params.py test: rename tests that a later definition shadowed 2026-08-12 11:15:54 -07:00
test_perplexity_reasoning.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_prompt_caching.py test(vcr): drop dead 'from respx import MockRouter' imports 2026-05-13 00:32:03 +00:00
test_prompt_factory.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_replicate.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_rerank.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_router_llm_translation_tests.py test: test 2026-03-28 19:17:38 -07:00
test_sambanova_chat_transformation.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_skills_api.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_snowflake.py Merge main and resolve Snowflake test conflict 2026-03-30 18:06:37 -07:00
test_text_completion.py Add inference providers support for Hugging Face (#8258) (#9738) (#9773) 2025-04-05 10:50:15 -07:00
test_text_completion_unit_tests.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_together_ai.py test: point the live together_ai suites at a model together still serves (#37746) 2026-08-20 18:53:27 -07:00
test_triton.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_unit_test_bedrock_invoke.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_v0.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_vcr_classification.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_vcr_conftest_common_banner.py fix(tests/vcr): make Redis cassette cache replay deterministically (zero VCR misses on consecutive runs) (#28826) 2026-05-26 11:30:44 -07:00
test_vcr_filters.py fix(tests/vcr): mint Google OAuth tokens live to prevent stale-token replay (#29229) 2026-05-28 17:12:02 -07:00
test_vcr_redis_persister.py test(vcr): stop refreshing cassette TTL on read so cassettes lapse after 24h (#29784) 2026-06-05 10:22:41 -07:00
test_voyage_ai.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_watsonx.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ws_vcr.py test(realtime): record and replay websocket traffic in redis vcr cassettes (#32390) 2026-07-08 00:19:06 -07:00
test_xai.py test(vcr): drop dead 'from respx import MockRouter' imports 2026-05-13 00:32:03 +00:00

Readme.md

Unit tests for individual LLM providers.

Name of the test file is the name of the LLM provider - e.g. test_openai.py is for OpenAI.

Redis-backed VCR cache

Every test in this directory is auto-decorated with @pytest.mark.vcr (via conftest.py). The first time a test runs we hit the live provider and record the HTTP exchange into Redis under litellm:vcr:cassette:<test_id>. Every subsequent run within 24h replays from Redis without touching the network. The 24h TTL means each new day's first run records again, so upstream API drift surfaces within a day.

The persister, header scrubbing, and 2xx-only filtering are defined in tests/_vcr_redis_persister.py. Files that already use respx (which patches the same httpx transport vcrpy does) are excluded from the auto-marker — see _RESPX_CONFLICTING_FILES in conftest.py.

The same VCR cache is used by other test directories that exercise live provider APIs. The reusable conftest plumbing lives in tests/_vcr_conftest_common.py and is wired into:

  • tests/llm_translation/
  • tests/llm_responses_api_testing/
  • tests/audio_tests/
  • tests/batches_tests/
  • tests/guardrails_tests/
  • tests/image_gen_tests/
  • tests/litellm_utils_tests/
  • tests/local_testing/ (covers local_testing_part1, local_testing_part2, litellm_router_testing, litellm_assistants_api_testing, langfuse_logging_unit_tests)
  • tests/logging_callback_tests/
  • tests/pass_through_unit_tests/
  • tests/router_unit_tests/
  • tests/unified_google_tests/

Test directories that run LiteLLM proxy in Docker (e.g. build_and_test, proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests) are intentionally not included: VCR.py patches the in-process httpx transport, so it cannot intercept the LLM calls that originate inside the Docker container.

Required environment

CASSETTE_REDIS_URL — separate Redis instance from the application Redis (REDIS_URL/REDIS_HOST) so test cassettes are not flushed by proxy tests. Provider credentials (ANTHROPIC_API_KEY, OPENAI_API_KEY, AWS_*, etc.) are needed only on cache-miss (the daily re-record), not on replay.

Flushing the cache

When you want the next run to re-record immediately instead of waiting for the 24h TTL:

make test-llm-translation-flush-vcr-cache

Disabling VCR

Skip the cache entirely (every call goes live, no recording):

LITELLM_VCR_DISABLE=1 uv run pytest tests/llm_translation/test_<file>.py