scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-29 06:33:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 200
litellm/tests/test_litellm
History
Alexander Chernov 95e321cc75
fix(zai): translate response_format into a forced tool call 
ZAIChatConfig.get_supported_openai_params omitted response_format, so litellm either raised
UnsupportedParamsError or, under drop_params, dropped the schema and returned HTTP 200 with prose.
The silent variant is the damaging one: the failure only surfaces in a downstream JSON parser.

GLM ignores response_format upstream but does honour a forced tool call, which is what
BaseConfig._add_response_format_to_tools already builds for azure and fireworks_ai. Wiring zai into
it is not enough on its own: that helper sets the internal json_mode flag, and zai reaches the
OpenAI SDK handler, which passed the flag straight through to Completions.create() as an unexpected
keyword argument. azure solved the same problem by popping json_mode and threading
convert_tool_call_to_json_mode into its response conversion, so openai.py now does the same and the
tool call is unwrapped back into message.content.

The translation is skipped when it could not be unwrapped again: streaming responses, and requests
that already carry caller-supplied tools, where forcing json_tool_call would hijack them. In both
cases response_format is left to pass through as before.

Also stops _add_response_format_to_tools mutating the caller's tools list. _map_openai_params
aliases that list into optional_params, so the previous setdefault plus append wrote into the list
the caller passed to completion(), accumulating a duplicate json_tool_call on every turn of an
agent loop. azure and fireworks_ai were exposed to this too.

Fixes: BerriAI/litellm#37720
Signed-off-by: Alexander Chernov <alexander@chernov.it>
2026-08-21 00:22:44 +01:00
..
a2a_protocol fix(a2a): accept the whole JSON-RPC id union the spec defines (#37704) 2026-08-20 14:41:22 -07:00
anthropic_interface fix(rust): route agentic-completion-hook /messages requests to Python for all stream modes (#34126) 2026-07-22 00:36:47 +00:00
batches fix(batches): skip undecodable batch output lines when costing 2026-08-19 14:35:01 -07:00
caching test(caching): annotate new semantic cache and hosted_vllm test helpers 2026-08-18 15:20:13 -07:00
completion_extras refactor(responses): drop commentary from the tool_choice fix 2026-08-17 13:04:45 -07:00
compression fix(guardrails/headroom): stop compressing the turn the model must act on (#35294) 2026-07-30 18:53:31 -07:00
containers fix(containers): record ownership for service-account keys + fix Prisma Json serialization (#28990) 2026-05-27 21:00:07 -07:00
enterprise fix(batches): persist the creating key and tags on managed batches created via /v1/batches 2026-08-15 12:45:00 -07:00
expected_fine_tuning_api
expected_responses_api_request test(responses): replace perma-skip azure shell e2e with offline coverage (#32444) 2026-07-08 10:01:41 -07:00
experimental_mcp_client fix(mcp): normalize auth schemes so MCP egress emits exactly one prefix (#37668) 2026-08-20 14:11:38 -07:00
google_genai chore: litellm oss staging 250626 (#31305) 2026-06-25 21:00:28 -07:00
images
integrations test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
interactions test(interactions): follow Google spec drift replacing Turn with typed steps (#36730) 2026-08-12 23:32:57 -07:00
litellm_core_utils fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do (#37703) 2026-08-20 14:43:06 -07:00
llms fix(zai): translate response_format into a forced tool call 2026-08-21 00:22:44 +01:00
models feat(spend-logs): add lifecycle timestamps 2026-08-18 14:16:08 -07:00
ocr fix(ocr): reject invalid req_format values as 400 on the SDK path 2026-08-17 15:49:18 -07:00
passthrough test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
proxy test: merge three stranded twins into the files that shadow them (#37600) 2026-08-20 14:25:23 -07:00
rag fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_api refactor(realtime): inject the vertex access token resolver 2026-08-20 02:36:47 -07:00
repositories fix(team): sweep dangling team references and cache on team delete (#36819) 2026-08-13 18:01:38 -07:00
rerank_api fix(rerank): log optional_rerank_params at debug to stop leaking request content (#32533) 2026-07-08 18:48:03 -07:00
responses Fall back to the GPT version rule when the cost map carries no breakpoint flag 2026-08-20 05:47:19 -07:00
router_strategy feat(complexity_router): add business classification rubric preset (#37534) 2026-08-20 11:20:53 -07:00
router_utils test(router): type the acreate_file fallback test helpers 2026-08-18 19:57:24 -07:00
sandbox feat: add opensandbox sandbox provider (#31024) 2026-06-23 09:05:13 -07:00
secret_managers fix(proxy): initialize the secret manager before resolving os.environ config references (#37544) 2026-08-19 17:00:26 -07:00
test_router fix(logging): stop pinning large request payloads past request end (#33455) 2026-07-15 15:28:37 -07:00
types test: add Final annotations to LIT-5757 regression test variables 2026-08-18 23:19:38 +00:00
vector_stores test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
videos test(videos): add 1:1 test file scaffold for videos component paths (#30631) 2026-06-29 09:12:51 +05:30
__init__.py
conftest.py test(cli): model keyring's null backend in the vault test double 2026-08-20 05:28:19 -07:00
log.txt
readme.md
test__types.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_a2a_registry_lookup.py
test_acompletion_session_reuse_e2e.py
test_add_deployment_no_master_key.py
test_aembedding_session_reuse_e2e.py
test_anthropic_beta_headers_filtering.py chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
test_anthropic_skills_transformation.py
test_anthropic_sonnet_1hr_cache_pricing.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_assert_ci_coverage.py test(ci): reject coverage-allowlist entries that no longer match a file (#37608) 2026-08-20 14:25:28 -07:00
test_assert_workflow_dir_hygiene.py feat(ci): assert .github/workflows holds only workflows, correctly named (#37616) 2026-08-20 21:36:26 +00:00
test_audio_transcription_rust_bridge.py perf(bedrock): audio transcription via rust core (py->rust bridge) (#33990) 2026-07-20 14:09:41 -07:00
test_azure_ad_token_credential_resolution.py test(router): cover s3_output_bucket_name surviving the trusted credential snapshot 2026-08-17 14:51:10 -07:00
test_azure_ai_grok_4_3_model_metadata.py feat(azure-ai): add Grok 4.3 model metadata (#27932) 2026-08-13 17:25:17 -07:00
test_batch_completion_models_all_responses.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py Litellm oss staging 030626 (#29578) 2026-06-03 11:01:51 -07:00
test_bedrock_extended_beta_models.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_bedrock_nemotron_super.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_bedrock_usgov_haiku_1hr_cache.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_bedrock_usgov_pricing.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_budget_ratchet_check.py fix(lint): let the ratchet guard recognise a graduated rule 2026-08-07 23:11:23 -07:00
test_chat_ui_responses_session.py
test_check_licenses.py fix(check_licenses): read PEP 639 license-expression metadata (#28529) 2026-05-22 11:22:38 -07:00
test_check_test_quality.py feat(ci): freeze the conftest save/restore inventory so it can only shrink (#37621) 2026-08-20 21:39:59 +00:00
test_check_type_discipline.py fix(scripts): unwrap PEP 604 unions in LIT002 TypedDict detection 2026-08-13 20:01:35 -07:00
test_circleci_path_filter.py perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559) 2026-08-19 18:32:21 -07:00
test_circleci_rust_toolchain.py ci(circleci): install a pinned Rust toolchain on the Linux jobs (#35519) 2026-08-03 18:39:35 -07:00
test_claude_fable_5_config.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_claude_haiku_4_5_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_6_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_4_8_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_5_config.py feat(anthropic): add Claude Opus 5 2026-07-24 10:43:49 -07:00
test_claude_sonnet_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_sonnet_5_config.py fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31 (#31917) 2026-07-01 17:45:57 -07:00
test_cloudflare_workers_ai_model_metadata.py feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051) 2026-06-23 10:44:37 -07:00
test_command_r7b_pricing.py chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
test_completion_timeout_resolution.py fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) 2026-06-23 14:22:54 -07:00
test_component_entrypoint.py fix(docker): honor USE_DDTRACE in the componentized gateway and backend images (#35490) 2026-08-01 14:12:59 -07:00
test_compression.py
test_conftest.py test: trim the PROXY_BASE_URL fixture and regression docstrings 2026-08-19 00:56:37 -07:00
test_conftest_isolation.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
test_constants.py
test_container_router.py
test_cost_calculation_log_level.py
test_cost_calculator.py fix(vertex_ai): price passthrough calls on the URL's serving location 2026-08-19 16:44:52 -07:00
test_count_tokens_public_api.py
test_dashscope_image_generation.py
test_deepseek_model_metadata.py
test_detect_changes.py perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559) 2026-08-19 18:32:21 -07:00
test_dockerfile_non_root.py
test_eager_tiktoken_load.py
test_env_key_doc_gate.py fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996) 2026-08-05 14:49:55 -07:00
test_exception_exports.py
test_exception_header_preservation.py
test_exception_mapping_request_attribute.py
test_filter_out_litellm_params.py
test_gate_slot_lock.py fix(make): acquire the gate slot before lint setup deps 2026-08-14 21:17:14 -07:00
test_get_blog_posts.py
test_git_hooks.py chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) 2026-06-11 10:00:23 -07:00
test_github_close_low_quality_prs.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_review_gate.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_with_llm.py fix(triage): track and credit feature dead-end evidence in the verdict 2026-08-11 10:57:46 -07:00
test_github_triage_workflows.py chore(ci): close the test-census blind spots and move scripts out of workflows/ (#37586) 2026-08-20 10:07:14 -07:00
test_gpt_5_4_model_metadata.py fix(pricing): correct gpt-5.4-mini and gpt-5.4-nano token limits 2026-07-30 02:43:54 +00:00
test_gpt_5_5_model_metadata.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_gpt_image_cost_calculator.py chore: litellm oss staging (#31185) 2026-06-26 09:17:44 -07:00
test_gpt_realtime_mode.py test(pricing): pin the realtime mode assertion to the bundled cost map (#33806) 2026-07-18 02:52:55 +00:00
test_groq_streaming_encoding.py
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py
test_litellm_params_reserved_keys.py
test_logging.py fix(logging): close three secret-leak paths in verbose logging (#37391) 2026-08-19 01:03:34 +00:00
test_lowest_latency_zero_tokens.py
test_main.py Fall back to the GPT version rule when the cost map carries no breakpoint flag 2026-08-20 05:47:19 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_mistral_medium_3_5_model_metadata.py fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases (#31373) 2026-06-25 18:27:18 -07:00
test_mistral_zai_glm_5_2_model_metadata.py fix(mistral): correct zai-glm-5-2 limits, add cached-input price and glm-5-2 alias 2026-08-20 10:45:24 -07:00
test_model_block_unblock.py fix(proxy): serialize model reconciles so concurrent model writes stop evicting each other (#36687) 2026-08-12 13:42:26 -07:00
test_model_cost_aliases.py
test_model_param_helper.py
test_model_prices_schema.py fix(cost): support cache_creation_input_token_cost in tiered pricing and make tier selection all-or-nothing 2026-08-13 02:18:10 +00:00
test_model_response_normalization.py
test_muse_spark_1_1_model_metadata.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_muse_spark_1_2_model_metadata.py fix(model_prices): bill muse spark web search grounding per query 2026-08-12 19:44:20 -07:00
test_nested_drop_params.py
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_pre_commit_lint.py feat(scripts): queue heavy gates behind a machine-wide slot lock 2026-08-14 17:22:32 -07:00
test_prisma_generate_if_needed.py fix(lint): generate the prisma client into the gate-owned venv 2026-08-06 01:54:26 -07:00
test_project_alias_tracking.py
test_project_tags_pydantic.py
test_proxy_auth.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_rag_openai_ingestion.py chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
test_rate_limit_error_unification.py test(rate-limits): drop the removed data kwarg from the v3 dynamic limiter raise-branch test 2026-07-30 15:20:44 -07:00
test_redact_string_in_error_paths.py fix(router): redact fallback tracebacks at the call site and cover the sync deferred stream (#35843) 2026-08-05 10:06:52 -07:00
test_redis.py fix(redis): unwrap decorated __init__s when deriving the from_url kwargs allowlist (#36654) 2026-08-15 11:52:08 -07:00
test_register_model_custom_pricing.py fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) 2026-07-07 20:49:03 -07:00
test_register_model_zero_cost_persistence.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_replicate_model_key_format.py fix(model-prices): correct replicate model key typo (#34800) 2026-08-05 16:37:44 -07:00
test_responses_api_bridge_non_stream.py fix(usage): keep responses usage SDK-parseable and complete streamed reasoning splits 2026-08-19 14:57:19 -07:00
test_responses_id_security.py fix(responses_id_security): decrypt response ids for input_items follow-ups (#32269) 2026-07-06 14:03:15 -07:00
test_responses_streaming_container_ownership.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_retrieve_batch_bedrock_dispatch.py feat(batch-job): bedrock batch model invocation job retrieval (#26834) 2026-05-11 13:22:26 -07:00
test_router.py test: drop unused imports in the direct restore test 2026-08-19 00:54:46 -07:00
test_router_block_helpers.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_router_exception_redaction.py fix(router): mask provider credentials embedded in fallback error messages (#32083) 2026-07-03 18:48:06 -07:00
test_router_google_genai.py
test_router_model_cost_isolation.py fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do (#37703) 2026-08-20 14:43:06 -07:00
test_router_order_fallback.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_router_per_deployment_num_retries.py fix(router): honor request-level num_retries over a deployment's litellm_params value (#35483) 2026-08-01 13:51:29 -07:00
test_router_redis_init.py
test_router_retry_backoff_headers.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_router_retry_non_retryable_errors.py
test_router_retry_policy_update.py fix(proxy): tighten role gating on /get/config/callbacks response (#31745) 2026-07-01 17:58:31 -07:00
test_router_silent_experiment.py
test_router_streaming_fallback_metadata.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_router_weighted_failover.py fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill (#35104) 2026-08-10 16:51:55 -07:00
test_ruff_strict_gate.py refactor(lint): graduate the 35 zero-violation strict rules into ruff.toml 2026-08-07 23:10:33 -07:00
test_sambanova_model_metadata.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_secret_redaction.py fix(logging): close three secret-leak paths in verbose logging (#37391) 2026-08-19 01:03:34 +00:00
test_select_ui_test_scope.py fix(ci): run the full dashboard suite when a change reaches outside src/ (#37563) 2026-08-19 23:09:03 -07:00
test_service_logger.py fix: missing span for guardrail passthrough (#29552) 2026-06-03 01:25:15 +00:00
test_setup_wizard.py
test_shared_session_integration.py
test_ssl_verify_unit.py refactor(bedrock): remove the dead BedrockLLM invoke code path 2026-07-29 20:25:36 -07:00
test_stream_chunk_builder_annotations.py
test_stream_chunk_builder_images.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_streaming_connection_cleanup.py
test_system_message_format_bug.py
test_test_quality_gate.py feat(ci): freeze the conftest save/restore inventory so it can only shrink (#37621) 2026-08-20 21:39:59 +00:00
test_thinking_enabled.py test: drop restating comment and wrap long call in thinking tests 2026-08-18 19:55:22 -07:00
test_type_check_gate.py fix(lint): retire the single-slot base-counts cache 2026-08-06 02:23:52 -07:00
test_type_discipline_gate.py fix(lint): pick the merge-aware base so in-progress merges are not blamed for base drift 2026-08-04 17:58:41 -07:00
test_utils.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_video_generation.py fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (#29098) 2026-05-28 11:45:41 -07:00
test_with_dashboard_node.py fix(bootstrap): fail fast when nvm cannot activate the pinned node 2026-08-04 21:18:58 -07:00
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py