scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-29 01:42:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 79
litellm/tests/test_litellm
History
tin-berri ca0b951a43
feat(spend): report prompt caching savings as total and gateway-attributed (#38134) 
* feat(spend): report prompt caching savings as total and gateway-attributed

`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.

Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.

Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.

The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.

Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.

* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)

The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.

Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
2026-08-28 00:19:06 -07:00
..
a2a_protocol Merge pull request #37917 from BerriAI/litellm_a2a_protocol_binding_casing 2026-08-24 11:14:01 -07:00
anthropic_interface fix(rust): route agentic-completion-hook /messages requests to Python for all stream modes (#34126) 2026-07-22 00:36:47 +00:00
batches test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
caching fix(caching): treat an empty cache namespace as no namespace 2026-08-26 13:48:45 -07:00
completion_extras fix(anthropic): carry tool_reference tool results through the guardrail translation round trip 2026-08-26 22:35:46 -07:00
compression fix(guardrails/headroom): stop compressing the turn the model must act on (#35294) 2026-07-30 18:53:31 -07:00
containers test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
enterprise fix(files): accept OpenAI's evals purpose on the files routes 2026-08-22 10:16:45 -07:00
expected_fine_tuning_api
expected_responses_api_request test(responses): replace perma-skip azure shell e2e with offline coverage (#32444) 2026-07-08 10:01:41 -07:00
experimental_mcp_client feat(mcp): let a resolved OAuth token target a custom upstream header (#38456) 2026-08-27 14:32:01 -07:00
fixtures/together_ai_sync feat(models): add daily Together AI model registry sync script and workflow 2026-08-25 13:12:06 -07:00
google_genai refactor(google_genai): pick the stream logging endpoint type at construction 2026-08-26 17:18:44 -07:00
images test(images): pin scalar-array edit params survive as repeated multipart fields 2026-08-24 12:49:54 -07:00
integrations feat(spend): report prompt caching savings as total and gateway-attributed (#38134) 2026-08-28 00:19:06 -07:00
interactions fix(interactions): poll queued background creates and drop the poll's deployment identity 2026-08-22 17:35:19 -07:00
litellm_core_utils feat(spend): report prompt caching savings as total and gateway-attributed (#38134) 2026-08-28 00:19:06 -07:00
llms feat(spend): report prompt caching savings as total and gateway-attributed (#38134) 2026-08-28 00:19:06 -07:00
models test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
ocr fix(proxy): keep every value of a repeated form key in get_form_data 2026-08-21 18:44:46 -07:00
passthrough test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
proxy feat(spend): report prompt caching savings as total and gateway-attributed (#38134) 2026-08-28 00:19:06 -07:00
rag fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_api test(realtime): pin mode-only transcription detection and correct a stale docstring 2026-08-26 12:33:46 -07:00
repositories fix(team): serialize member_add, member_delete, and delete under the team's advisory lock (#37969) 2026-08-25 21:55:11 +00:00
rerank_api fix(together_ai): default endpoints to api.together.ai instead of api.together.xyz 2026-08-25 10:18:55 -07:00
responses Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_dotprompt_model_swap 2026-08-26 17:36:01 -07:00
router_strategy fix(complexity_router): route client housekeeping calls to the cheapest tier (#38598) 2026-08-27 22:55:09 -07:00
router_utils fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning (#38618) 2026-08-27 22:19:22 -07:00
rust_bridge feat(rust): route /chat/completions through the Rust core for anthropic and bedrock (#37241) 2026-08-20 16:15:24 -07:00
sandbox test: say whether a match= pattern is a regex or a literal (ruff RUF043) 2026-08-21 16:25:33 -07:00
secret_managers test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router fix(logging): stop pinning large request payloads past request end (#33455) 2026-07-15 15:28:37 -07:00
types feat(mcp): let a resolved OAuth token target a custom upstream header (#38456) 2026-08-27 14:32:01 -07:00
vector_stores test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
videos test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
__init__.py
conftest.py test: invalidate memoized model-cost lookups between unit tests 2026-08-23 09:23:35 +00:00
log.txt
readme.md
test_a2a_registry_lookup.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_acompletion_session_reuse_e2e.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_add_deployment_no_master_key.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_aembedding_session_reuse_e2e.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_anthropic_beta_headers_filtering.py chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
test_anthropic_skills_transformation.py
test_anthropic_sonnet_1hr_cache_pricing.py test(model_prices): pin claude 3 1h cache write rates to 2x base input 2026-08-25 14:05:07 +00:00
test_assert_ci_coverage.py ci: run the keyless caching tests that ran in no job (#37790) 2026-08-22 22:56:43 -07:00
test_assert_workflow_dir_hygiene.py feat(ci): assert .github/workflows holds only workflows, correctly named (#37616) 2026-08-20 21:36:26 +00:00
test_audio_transcription_rust_bridge.py perf(bedrock): audio transcription via rust core (py->rust bridge) (#33990) 2026-07-20 14:09:41 -07:00
test_azure_ad_token_credential_resolution.py test(router): cover s3_output_bucket_name surviving the trusted credential snapshot 2026-08-17 14:51:10 -07:00
test_azure_ai_grok_4_3_model_metadata.py feat(azure-ai): add Grok 4.3 model metadata (#27932) 2026-08-13 17:25:17 -07:00
test_azure_audio_price_aliases.py fix(pricing): add undated azure aliases for gpt-audio-mini and gpt-realtime-mini (#37867) 2026-08-21 18:44:19 -07:00
test_batch_completion_models_all_responses.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py Litellm oss staging 030626 (#29578) 2026-06-03 11:01:51 -07:00
test_bedrock_batch_pricing.py fix(model_prices): add Gemini live-translate, Voyage 4 series, Perplexity contextualized embeddings; absorb Fireworks + Bedrock batch registry PRs 2026-08-20 19:14:11 +00:00
test_bedrock_extended_beta_models.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_bedrock_nemotron_super.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_bedrock_usgov_haiku_1hr_cache.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_bedrock_usgov_pricing.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_budget_ratchet_check.py fix(lint): let the ratchet guard recognise a graduated rule 2026-08-07 23:11:23 -07:00
test_chat_ui_responses_session.py
test_check_licenses.py fix(ci): retry transient PyPI license lookups 2026-08-23 09:23:35 +00:00
test_check_migrations_no_data_rewrites.py fix: keep a schema-qualified call inside an index expression from reading as a relation 2026-08-24 16:04:38 -07:00
test_check_test_quality.py feat(ci): gate patching of SDK internals in tests as TQ008 (#37787) 2026-08-22 22:54:30 -07:00
test_check_type_discipline.py perf(ci): fan the budget checkers out across cores (#37784) 2026-08-22 22:44:58 -07:00
test_circleci_path_filter.py perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559) 2026-08-19 18:32:21 -07:00
test_circleci_rust_toolchain.py ci(circleci): install a pinned Rust toolchain on the Linux jobs (#35519) 2026-08-03 18:39:35 -07:00
test_claude_fable_5_config.py fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
test_claude_haiku_4_5_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_6_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_4_8_config.py fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
test_claude_opus_5_config.py fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
test_claude_sonnet_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_sonnet_5_config.py fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
test_cloudflare_workers_ai_model_metadata.py feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051) 2026-06-23 10:44:37 -07:00
test_command_r7b_pricing.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_completion_timeout_resolution.py fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) 2026-06-23 14:22:54 -07:00
test_component_entrypoint.py fix(docker): honor USE_DDTRACE in the componentized gateway and backend images (#35490) 2026-08-01 14:12:59 -07:00
test_compression.py
test_conftest.py test: trim the PROXY_BASE_URL fixture and regression docstrings 2026-08-19 00:56:37 -07:00
test_conftest_isolation.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
test_constants.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_container_router.py
test_cost_calculation_log_level.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_cost_calculator.py Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_together_cache_pricing 2026-08-27 14:29:46 -07:00
test_count_tokens_public_api.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_dashscope_image_generation.py feat(dashscope): support qwen-image-3.0 and qwen-image-3.0-pro image generation 2026-08-27 01:59:51 +00:00
test_daybreak_model_metadata.py fix: mark daybreak-blue-latest and gpt-5.6-sol as supporting computer use 2026-08-20 16:20:40 -07:00
test_deepseek_model_metadata.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_detect_changes.py perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559) 2026-08-19 18:32:21 -07:00
test_dockerfile_non_root.py
test_e2e_egress_sentinel.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_eager_tiktoken_load.py
test_env_key_doc_gate.py fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996) 2026-08-05 14:49:55 -07:00
test_exception_exports.py
test_exception_header_preservation.py
test_exception_mapping_request_attribute.py
test_filter_out_litellm_params.py
test_fireworks_serverless_model_costs.py fix(model_prices): azure gpt-4.1-nano retirement date, together deprecations, novita gpt-oss-120b vision flag, fireworks deepseek-v4-pro-0813 2026-08-26 16:56:53 +00:00
test_gate_slot_lock.py fix(make): acquire the gate slot before lint setup deps 2026-08-14 21:17:14 -07:00
test_gemini_3_1_flash_lite_image_pricing.py fix(model_cost): dedupe gemini-3.1-flash-lite-image and correct its capabilities 2026-08-21 17:23:47 -07:00
test_gemini_tts_native_audio_pricing.py test: restore model cost map via monkeypatch 2026-08-26 15:34:25 -07:00
test_get_blog_posts.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_git_hooks.py chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) 2026-06-11 10:00:23 -07:00
test_github_close_low_quality_prs.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_github_review_gate.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_with_llm.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_github_triage_workflows.py chore(ci): close the test-census blind spots and move scripts out of workflows/ (#37586) 2026-08-20 10:07:14 -07:00
test_gpt_5_4_model_metadata.py fix(pricing): correct gpt-5.4-mini and gpt-5.4-nano token limits 2026-07-30 02:43:54 +00:00
test_gpt_5_5_model_metadata.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_gpt_image_cost_calculator.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_gpt_realtime_mode.py fix(pricing): add undated azure aliases for gpt-audio-mini and gpt-realtime-mini (#37867) 2026-08-21 18:44:19 -07:00
test_groq_streaming_encoding.py
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_litellm_params_reserved_keys.py
test_logging.py fix(logging): stop stream-based log collectors classifying INFO logs as errors (#38476) 2026-08-27 17:42:00 -07:00
test_lowest_latency_zero_tokens.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_main.py fix(speech): forward api_key to the TTS bridge and isolate response hidden params 2026-08-26 16:23:33 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_mistral_medium_3_5_model_metadata.py fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
test_mistral_small_4_0_model_metadata.py fix(model_prices): restore supports_vision on Mistral Small 4.0 entries 2026-08-20 13:54:03 -07:00
test_mistral_zai_glm_5_2_model_metadata.py fix(mistral): correct zai-glm-5-2 limits, add cached-input price and glm-5-2 alias 2026-08-20 10:45:24 -07:00
test_model_block_unblock.py fix(proxy): serialize model reconciles so concurrent model writes stop evicting each other (#36687) 2026-08-12 13:42:26 -07:00
test_model_cost_aliases.py
test_model_param_helper.py
test_model_prices_schema.py fix(model_cost): dedupe gemini-3.1-flash-lite-image and correct its capabilities 2026-08-21 17:23:47 -07:00
test_model_response_normalization.py
test_muse_spark_1_1_model_metadata.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_muse_spark_1_2_model_metadata.py fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
test_mutation_report.py fix(ci): stop the mutation report publishing a score it never measured (#37825) 2026-08-21 20:15:52 -07:00
test_nested_drop_params.py
test_non_chat_routes_open_llm_spans.py fix(otel): emit LLM Call spans for speech, image, moderation, ocr and transcription (#37752) 2026-08-22 11:11:21 -07:00
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_pre_commit_lint.py feat(scripts): queue heavy gates behind a machine-wide slot lock 2026-08-14 17:22:32 -07:00
test_prisma_generate_if_needed.py fix(lint): generate the prisma client into the gate-owned venv 2026-08-06 01:54:26 -07:00
test_project_alias_tracking.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_project_tags_pydantic.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_proxy_auth.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_rag_openai_ingestion.py chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
test_rate_limit_error_unification.py test(rate-limits): drop the removed data kwarg from the v3 dynamic limiter raise-branch test 2026-07-30 15:20:44 -07:00
test_redact_string_in_error_paths.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_redis.py refactor(redis): drop docstrings restating the code 2026-08-26 10:45:56 -07:00
test_register_model_custom_pricing.py fix: suppress misleading register_model unresolved-cost warnings for entries without custom pricing (#38542) 2026-08-27 12:45:30 -07:00
test_register_model_zero_cost_persistence.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_replicate_model_key_format.py fix(model-prices): correct replicate model key typo (#34800) 2026-08-05 16:37:44 -07:00
test_responses_api_bridge_non_stream.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_responses_id_security.py test(responses): drive streamed-id regression via production ResponseCompletedEvent shape 2026-08-26 01:52:01 -07:00
test_responses_streaming_container_ownership.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_retrieve_batch_bedrock_dispatch.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router.py Merge pull request #38606 from BerriAI/litellm_bedrock_messages_midstream_fallback 2026-08-27 19:13:18 -07:00
test_router_block_helpers.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_router_exception_redaction.py test(router): let monkeypatch own expose_router_debug_in_errors (#37848) 2026-08-22 09:02:40 -07:00
test_router_google_genai.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_model_cost_isolation.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_order_fallback.py fix(router): resolve hidden aliases for explicit lookup 2026-08-25 18:41:23 -04:00
test_router_per_deployment_num_retries.py fix(router): honor request-level num_retries over a deployment's litellm_params value (#35483) 2026-08-01 13:51:29 -07:00
test_router_redis_init.py
test_router_retry_backoff_headers.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_router_retry_non_retryable_errors.py
test_router_retry_policy_update.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_silent_experiment.py
test_router_streaming_fallback_metadata.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_router_weighted_failover.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_ruff_strict_gate.py refactor(lint): graduate the 35 zero-violation strict rules into ruff.toml 2026-08-07 23:10:33 -07:00
test_sambanova_model_metadata.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_secret_redaction.py fix(logging): preserve uvicorn color_message args during secret redaction (#37122) 2026-08-21 16:04:53 -07:00
test_select_ui_test_scope.py fix(ci): run the full dashboard suite when a change reaches outside src/ (#37563) 2026-08-19 23:09:03 -07:00
test_service_logger.py fix: missing span for guardrail passthrough (#29552) 2026-06-03 01:25:15 +00:00
test_setup_wizard.py
test_shared_session_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_ssl_verify_unit.py refactor(bedrock): remove the dead BedrockLLM invoke code path 2026-07-29 20:25:36 -07:00
test_stream_chunk_builder_annotations.py
test_stream_chunk_builder_images.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_streaming_connection_cleanup.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_sync_together_ai_models.py fix(scripts): drop supports_prompt_caching when cached pricing leaves the together_ai catalog 2026-08-25 13:25:46 -07:00
test_system_message_format_bug.py
test_test_quality_gate.py feat(ci): gate patching of SDK internals in tests as TQ008 (#37787) 2026-08-22 22:54:30 -07:00
test_thinking_enabled.py test: drop restating comment and wrap long call in thinking tests 2026-08-18 19:55:22 -07:00
test_together_ai_model_metadata.py Merge pull request #38486 from BerriAI/litellm_together_glm53_flash 2026-08-27 15:24:30 -07:00
test_type_check_gate.py fix(lint): retire the single-slot base-counts cache 2026-08-06 02:23:52 -07:00
test_type_discipline_gate.py fix(lint): pick the merge-aware base so in-progress merges are not blamed for base drift 2026-08-04 17:58:41 -07:00
test_utils.py Merge pull request #38100 from FelipeRodriguesGare/bugfix/tencent-thinking-extra-body 2026-08-27 18:23:11 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_video_generation.py fix(videos): forward uploaded source file on /v1/videos/edits to the provider 2026-08-24 15:34:21 -07:00
test_with_dashboard_node.py fix(bootstrap): fail fast when nvm cannot activate the pinned node 2026-08-04 21:18:58 -07:00
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py