scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-05 02:41:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 49
litellm/tests/test_litellm
History
Yassin Kortam 446a790aa4 fix(bedrock): stop claiming native web_search support on Bedrock 
AmazonAnthropicClaudeMessagesConfig inherited handles_web_search_natively()
== True from the base Anthropic Messages config, so the web-search
interception short-circuit skipped Bedrock and forwarded Anthropic's
web_search_20250305 server tool straight to AWS. Bedrock hosts none of the
web_search server tools on any of its APIs, so the request is rejected with
a deterministic 400.

Override the flag to False so interception stays in play, which is the only
way the tool can be served on Bedrock. AmazonMantleMessagesConfig subclasses
this config and picks up the same answer.

Resolves LIT-5391
2026-08-10 11:29:36 -07:00
..
a2a_protocol fix(http): stop pooled clients persisting cookies on the aiohttp jar too (#36149) 2026-08-07 11:05:59 -07:00
anthropic_interface fix(rust): route agentic-completion-hook /messages requests to Python for all stream modes (#34126) 2026-07-22 00:36:47 +00:00
batches test(batches): run Responses coverage in CI 2026-07-31 10:24:10 -04:00
caching test(caching): align closer tests with self-healing handlers 2026-08-04 18:23:06 -07:00
completion_extras fix(litellm): make the responses bridge and cursor routing total over the surfaces they now serve 2026-08-01 11:26:09 -07:00
compression fix(guardrails/headroom): stop compressing the turn the model must act on (#35294) 2026-07-30 18:53:31 -07:00
containers fix(containers): record ownership for service-account keys + fix Prisma Json serialization (#28990) 2026-05-27 21:00:07 -07:00
enterprise fix(batches): keep batch state in sync on a poll without claiming attribution (#34456) 2026-08-08 16:01:47 -07:00
expected_fine_tuning_api
expected_responses_api_request test(responses): replace perma-skip azure shell e2e with offline coverage (#32444) 2026-07-08 10:01:41 -07:00
experimental_mcp_client perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
google_genai chore: litellm oss staging 250626 (#31305) 2026-06-25 21:00:28 -07:00
images
integrations fix(bedrock): stop claiming native web_search support on Bedrock 2026-08-10 11:29:36 -07:00
interactions fix(tests): assert Content variants are identified by type, not by the discriminator keyword (#35161) 2026-07-29 17:49:30 -07:00
litellm_core_utils fix(cost): price dict-shaped image input token details at image rate (#33490) 2026-08-10 18:07:17 +00:00
llms fix(bedrock): stop claiming native web_search support on Bedrock 2026-08-10 11:29:36 -07:00
models feat(litellm): add models and repository layers (#29686) 2026-06-06 20:59:33 -07:00
ocr test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
passthrough fix(passthrough): stream non-sse passthrough responses instead of buffering in memory (#32386) 2026-07-07 20:51:15 -07:00
proxy feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416) 2026-08-10 11:02:06 -07:00
rag fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_api fix(realtime): preserve nested transcription model and session-first model priority 2026-07-03 14:21:22 -07:00
repositories refactor(repositories): add prisma protocol seams and a spend-reset unit of work 2026-08-03 22:19:54 -07:00
rerank_api fix(rerank): log optional_rerank_params at debug to stop leaking request content (#32533) 2026-07-08 18:48:03 -07:00
responses fix(responses): forward allowed_openai_params through the chat completions bridge (#35885) 2026-08-07 19:57:26 -07:00
router_strategy feat(router): independent, default-on deployment affinity for the auto-router (#36146) 2026-08-08 13:02:29 -07:00
router_utils feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416) 2026-08-10 11:02:06 -07:00
sandbox feat: add opensandbox sandbox provider (#31024) 2026-06-23 09:05:13 -07:00
secret_managers fix(bedrock): stop replaying expired Google OIDC tokens to STS on guardrail auth (#34637) 2026-07-25 16:27:54 -07:00
test_router fix(logging): stop pinning large request payloads past request end (#33455) 2026-07-15 15:28:37 -07:00
types refactor(types): declare mirrored pricing fields on ModelInfo 2026-08-08 03:25:09 +00:00
vector_stores
videos test(videos): add 1:1 test file scaffold for videos component paths (#30631) 2026-06-29 09:12:51 +05:30
__init__.py
conftest.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
log.txt
readme.md
test__types.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_a2a_registry_lookup.py
test_acompletion_session_reuse_e2e.py
test_add_deployment_no_master_key.py
test_aembedding_session_reuse_e2e.py
test_anthropic_beta_headers_filtering.py chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
test_anthropic_skills_transformation.py fix: encode upstream URL path identifiers 2026-04-29 22:02:39 -07:00
test_anthropic_sonnet_1hr_cache_pricing.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_audio_transcription_rust_bridge.py perf(bedrock): audio transcription via rust core (py->rust bridge) (#33990) 2026-07-20 14:09:41 -07:00
test_azure_ad_token_credential_resolution.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_azure_video_router.py
test_batch_completion_models_all_responses.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py Litellm oss staging 030626 (#29578) 2026-06-03 11:01:51 -07:00
test_bedrock_usgov_haiku_1hr_cache.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_bedrock_usgov_pricing.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_budget_ratchet_check.py fix(lint): let the ratchet guard recognise a graduated rule 2026-08-07 23:11:23 -07:00
test_chat_ui_responses_session.py
test_check_licenses.py fix(check_licenses): read PEP 639 license-expression metadata (#28529) 2026-05-22 11:22:38 -07:00
test_check_type_discipline.py chore(lint): move MappingProxyType to the dynamic tail of the LIT002 freeze menu 2026-08-06 03:58:07 -07:00
test_circleci_path_filter.py ci: gate CircleCI jobs on changed paths (#32080) 2026-07-04 19:15:08 -07:00
test_circleci_rust_toolchain.py ci(circleci): install a pinned Rust toolchain on the Linux jobs (#35519) 2026-08-03 18:39:35 -07:00
test_claude_fable_5_config.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_claude_haiku_4_5_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_6_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_4_8_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_5_config.py feat(anthropic): add Claude Opus 5 2026-07-24 10:43:49 -07:00
test_claude_sonnet_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_sonnet_5_config.py fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31 (#31917) 2026-07-01 17:45:57 -07:00
test_cloudflare_workers_ai_model_metadata.py feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051) 2026-06-23 10:44:37 -07:00
test_command_r7b_pricing.py chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
test_completion_timeout_resolution.py fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) 2026-06-23 14:22:54 -07:00
test_component_entrypoint.py fix(docker): honor USE_DDTRACE in the componentized gateway and backend images (#35490) 2026-08-01 14:12:59 -07:00
test_compression.py
test_conftest_isolation.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
test_constants.py
test_container_router.py
test_cost_calculation_log_level.py
test_cost_calculator.py feat(spend): rebuild the auto-router benchmarks backend as a per-session rollup (#35910) 2026-08-05 20:06:32 +00:00
test_count_tokens_public_api.py
test_dashscope_image_generation.py
test_deepseek_model_metadata.py
test_dockerfile_non_root.py
test_eager_tiktoken_load.py
test_env_key_doc_gate.py fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996) 2026-08-05 14:49:55 -07:00
test_exception_exports.py
test_exception_header_preservation.py
test_exception_mapping_request_attribute.py
test_filter_out_litellm_params.py
test_get_blog_posts.py
test_git_hooks.py chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) 2026-06-11 10:00:23 -07:00
test_github_close_low_quality_prs.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_review_gate.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_with_llm.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_workflows.py fix(test): drop references to removed Agent Shin workflows (#30791) 2026-06-18 15:47:51 -07:00
test_gpt_5_4_model_metadata.py fix(pricing): correct gpt-5.4-mini and gpt-5.4-nano token limits 2026-07-30 02:43:54 +00:00
test_gpt_5_5_model_metadata.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_gpt_image_cost_calculator.py chore: litellm oss staging (#31185) 2026-06-26 09:17:44 -07:00
test_gpt_realtime_mode.py test(pricing): pin the realtime mode assertion to the bundled cost map (#33806) 2026-07-18 02:52:55 +00:00
test_groq_streaming_encoding.py
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py
test_litellm_params_reserved_keys.py
test_logging.py feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418) 2026-08-10 10:40:13 -07:00
test_lowest_latency_zero_tokens.py
test_main.py fix(bridge): resolve the effective OpenAI base once, shared by gate and chat handler 2026-08-01 11:26:09 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_mistral_medium_3_5_model_metadata.py fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases (#31373) 2026-06-25 18:27:18 -07:00
test_model_block_unblock.py fix(proxy): stop model writes 500ing on another pod's delete (#35400) 2026-07-31 18:10:48 -07:00
test_model_cost_aliases.py
test_model_param_helper.py
test_model_prices_schema.py fix(pricing): sync flex/priority tier keys to dated OpenAI snapshot variants 2026-08-04 23:56:52 -07:00
test_model_response_normalization.py
test_muse_spark_1_1_model_metadata.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_nested_drop_params.py
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_pre_commit_lint.py build(lint): count deleted files toward check triggers 2026-08-08 10:41:06 -07:00
test_prisma_generate_if_needed.py fix(lint): generate the prisma client into the gate-owned venv 2026-08-06 01:54:26 -07:00
test_project_alias_tracking.py
test_project_tags_pydantic.py
test_rag_openai_ingestion.py chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
test_rate_limit_error_unification.py test(rate-limits): drop the removed data kwarg from the v3 dynamic limiter raise-branch test 2026-07-30 15:20:44 -07:00
test_redact_string_in_error_paths.py fix(router): redact fallback tracebacks at the call site and cover the sync deferred stream (#35843) 2026-08-05 10:06:52 -07:00
test_redis.py perf: install hiredis so redis-py parses replies with its C parser (#35709) 2026-08-03 20:47:43 -07:00
test_register_model_custom_pricing.py fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) 2026-07-07 20:49:03 -07:00
test_register_model_zero_cost_persistence.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_replicate_model_key_format.py fix(model-prices): correct replicate model key typo (#34800) 2026-08-05 16:37:44 -07:00
test_responses_api_bridge_non_stream.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_responses_id_security.py fix(responses_id_security): decrypt response ids for input_items follow-ups (#32269) 2026-07-06 14:03:15 -07:00
test_responses_streaming_container_ownership.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_retrieve_batch_bedrock_dispatch.py feat(batch-job): bedrock batch model invocation job retrieval (#26834) 2026-05-11 13:22:26 -07:00
test_router.py feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416) 2026-08-10 11:02:06 -07:00
test_router_block_helpers.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_router_exception_redaction.py fix(router): mask provider credentials embedded in fallback error messages (#32083) 2026-07-03 18:48:06 -07:00
test_router_google_genai.py
test_router_model_cost_isolation.py fix(router): keep custom model_info across a price data reload (#35491) 2026-08-05 19:56:50 +00:00
test_router_order_fallback.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_router_per_deployment_num_retries.py fix(router): honor request-level num_retries over a deployment's litellm_params value (#35483) 2026-08-01 13:51:29 -07:00
test_router_redis_init.py
test_router_retry_non_retryable_errors.py
test_router_retry_policy_update.py fix(proxy): tighten role gating on /get/config/callbacks response (#31745) 2026-07-01 17:58:31 -07:00
test_router_silent_experiment.py
test_router_streaming_fallback_metadata.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_router_weighted_failover.py feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416) 2026-08-10 11:02:06 -07:00
test_ruff_strict_gate.py refactor(lint): graduate the 35 zero-violation strict rules into ruff.toml 2026-08-07 23:10:33 -07:00
test_secret_redaction.py fix(logging): extend secret redaction to records litellm does not emit directly (#35977) 2026-08-05 14:50:42 -07:00
test_service_logger.py fix: missing span for guardrail passthrough (#29552) 2026-06-03 01:25:15 +00:00
test_setup_wizard.py
test_shared_session_integration.py
test_ssl_verify_unit.py refactor(bedrock): remove the dead BedrockLLM invoke code path 2026-07-29 20:25:36 -07:00
test_stream_chunk_builder_annotations.py
test_streaming_connection_cleanup.py
test_system_message_format_bug.py
test_thinking_enabled.py Litellm oss staging 250526 (#28770) 2026-05-26 11:57:39 -07:00
test_triage_rollout_heads_up.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_type_check_gate.py fix(lint): retire the single-slot base-counts cache 2026-08-06 02:23:52 -07:00
test_type_discipline_gate.py fix(lint): pick the merge-aware base so in-progress merges are not blamed for base drift 2026-08-04 17:58:41 -07:00
test_utils.py feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418) 2026-08-10 10:40:13 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_video_generation.py fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (#29098) 2026-05-28 11:45:41 -07:00
test_with_dashboard_node.py fix(bootstrap): fail fast when nvm cannot activate the pinned node 2026-08-04 21:18:58 -07:00
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py