scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102
litellm/tests/test_litellm
History
Exact
devin-ai-integration[bot] b0a0f11b09
feat(complexity-router): user-triggered escalation keywords (#33656) 
* feat(complexity-router): user-triggered escalation keywords

Add an escalation_keywords config option to the complexity router so a user
can force a bump to the next-higher complexity tier by including a phrase in
their message (a stronger model, but not one they get to choose). Defaults to
['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the
deliberate shouted form; admins can override the list or set [] to disable.

Escalation applies across every routing path: heuristic/LLM classification,
literal and semantic keyword_tier_rules overrides, adaptive routing, and
session affinity (where it bumps relative to the pinned model and persists the
higher tier for the rest of the session). Capped at the highest configured
tier and skips unconfigured intermediate tiers.

Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into
the complexity_router_config payload.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(complexity-router): validate escalation keywords and pin at tier ceiling

Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-17 10:24:59 -07:00
..
a2a_protocol fix(responses): stop scheduling sync success_handler concurrently with async_success_handler (#32239) 2026-07-07 09:13:50 -07:00
anthropic_interface/exceptions style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
batches Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
caching fix(proxy): stop stale auth cache re-publish so key updates and deletes propagate across replicas (#33565) 2026-07-16 15:00:33 -07:00
completion_extras chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
containers fix(containers): record ownership for service-account keys + fix Prisma Json serialization (#28990) 2026-05-27 21:00:07 -07:00
enterprise refactor(ui): point invitation links at the dedicated /onboarding route (#30857) 2026-07-08 19:47:05 -07:00
expected_fine_tuning_api refactor: refactor testing 2026-03-28 18:39:32 -07:00
expected_responses_api_request test(responses): replace perma-skip azure shell e2e with offline coverage (#32444) 2026-07-08 10:01:41 -07:00
experimental_mcp_client Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
google_genai chore: litellm oss staging 250626 (#31305) 2026-06-25 21:00:28 -07:00
images fix(image_edit): read vertex_project/location from litellm_params in Imagen get_complete_url 2026-04-21 15:03:40 +05:30
integrations fix(langfuse_otel): build per-request OTLP exporter from key and team dynamic Langfuse credentials (#32437) 2026-07-16 13:39:10 -07:00
interactions fix(responses): stop scheduling sync success_handler concurrently with async_success_handler (#32239) 2026-07-07 09:13:50 -07:00
litellm_core_utils fix(logging): classify async anthropic_messages and generate_content as async (#33589) 2026-07-16 20:56:47 -07:00
llms fix(logging): classify async anthropic_messages and generate_content as async (#33589) 2026-07-16 20:56:47 -07:00
models feat(litellm): add models and repository layers (#29686) 2026-06-06 20:59:33 -07:00
ocr test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
passthrough fix(passthrough): stream non-sse passthrough responses instead of buffering in memory (#32386) 2026-07-07 20:51:15 -07:00
proxy Merge pull request #33612 from BerriAI/litellm_lit4448_toolset_call_grants 2026-07-17 10:16:07 -07:00
realtime_api fix(realtime): preserve nested transcription model and session-first model priority 2026-07-03 14:21:22 -07:00
repositories fix(proxy): count only active users toward license seat limit (#31227) 2026-06-29 18:01:02 -07:00
rerank_api fix(rerank): log optional_rerank_params at debug to stop leaking request content (#32533) 2026-07-08 18:48:03 -07:00
responses fix(responses): end stream cleanly on transport error after terminal event 2026-07-15 08:33:50 -07:00
router_strategy feat(complexity-router): user-triggered escalation keywords (#33656) 2026-07-17 10:24:59 -07:00
router_utils fix(router): take the lowest minimum across a model group, not the highest 2026-07-16 19:53:10 -07:00
sandbox feat: add opensandbox sandbox provider (#31024) 2026-06-23 09:05:13 -07:00
secret_managers fix(proxy): harden secret name validation for external secret manager integrations (LIT-4201) (#32092) 2026-07-08 10:36:00 -07:00
test_router fix(logging): stop pinning large request payloads past request end (#33455) 2026-07-15 15:28:37 -07:00
types fix(anthropic): drop unsignable thinking blocks and allow null signature in logging (LIT-4007) (#31654) 2026-06-29 21:01:03 -07:00
vector_stores style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
videos test(videos): add 1:1 test file scaffold for videos component paths (#30631) 2026-06-29 09:12:51 +05:30
__init__.py
conftest.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
log.txt
readme.md
test__types.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_a2a_registry_lookup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_acompletion_session_reuse_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_add_deployment_no_master_key.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_aembedding_session_reuse_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_anthropic_beta_headers_filtering.py chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
test_anthropic_skills_transformation.py fix: encode upstream URL path identifiers 2026-04-29 22:02:39 -07:00
test_anthropic_sonnet_1hr_cache_pricing.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_azure_ad_token_credential_resolution.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_azure_video_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_completion_models_all_responses.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py Litellm oss staging 030626 (#29578) 2026-06-03 11:01:51 -07:00
test_bedrock_usgov_haiku_1hr_cache.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_bedrock_usgov_pricing.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_budget_ratchet_check.py refactor(lint): collapse type/lint budgets to a single per-rule limit (#31883) 2026-07-01 18:12:35 +03:00
test_chat_ui_responses_session.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_check_licenses.py fix(check_licenses): read PEP 639 license-expression metadata (#28529) 2026-05-22 11:22:38 -07:00
test_check_type_discipline.py chore(lint): zero out crash-class pyright rules and ban new type: ignore comments (#32152) 2026-07-04 16:56:12 -07:00
test_circleci_path_filter.py ci: gate CircleCI jobs on changed paths (#32080) 2026-07-04 19:15:08 -07:00
test_claude_fable_5_config.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_claude_haiku_4_5_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_8_config.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_claude_sonnet_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_sonnet_5_config.py fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31 (#31917) 2026-07-01 17:45:57 -07:00
test_cloudflare_workers_ai_model_metadata.py feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051) 2026-06-23 10:44:37 -07:00
test_command_r7b_pricing.py chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
test_completion_timeout_resolution.py fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) 2026-06-23 14:22:54 -07:00
test_compression.py Prompt Compression - add it to the proxy (#25729) 2026-04-20 15:08:00 -07:00
test_constants.py Fix test 2026-03-27 21:21:43 +05:30
test_container_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cost_calculation_log_level.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cost_calculator.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_count_tokens_public_api.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dashscope_image_generation.py fix(dashscope): fail fast on image generation API errors 2026-04-23 18:41:01 +05:30
test_deepseek_model_metadata.py Revert "Merge pull request #16590 from Chesars/refactor/remove-backup-file-dry-principle" 2026-04-25 17:10:41 -03:00
test_dockerfile_non_root.py fix(docker.non_root): use numeric UID 65534 for K8s runAsNonRoot (#26268) 2026-04-22 18:00:04 -07:00
test_eager_tiktoken_load.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_exports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_header_preservation.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_mapping_request_attribute.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_filter_out_litellm_params.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_blog_posts.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_git_hooks.py chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) 2026-06-11 10:00:23 -07:00
test_github_close_low_quality_prs.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_review_gate.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_with_llm.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_workflows.py fix(test): drop references to removed Agent Shin workflows (#30791) 2026-06-18 15:47:51 -07:00
test_gpt_5_5_model_metadata.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_gpt_5_6_model_metadata.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_gpt_image_cost_calculator.py chore: litellm oss staging (#31185) 2026-06-26 09:17:44 -07:00
test_groq_streaming_encoding.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_litellm_params_reserved_keys.py fix(snowflake): transform tool_choice string to object format (#23268) 2026-03-11 01:41:24 +05:30
test_logging.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lowest_latency_zero_tokens.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_main.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_mistral_medium_3_5_model_metadata.py fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases (#31373) 2026-06-25 18:27:18 -07:00
test_model_block_unblock.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_model_cost_aliases.py [Feat] - Ishaan main merge branch (#23596) 2026-03-14 09:40:00 -07:00
test_model_param_helper.py style: black format test_model_param_helper.py 2026-04-15 18:18:52 -07:00
test_model_response_normalization.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_muse_spark_1_1_model_metadata.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_nested_drop_params.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_prisma_generate_if_needed.py perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate (#32000) 2026-07-02 19:24:00 -07:00
test_project_alias_tracking.py feat(proxy): add project_alias tracking through callback metadata pipeline 2026-03-23 10:44:17 -07:00
test_project_tags_pydantic.py fix: req changes 2026-02-27 13:33:34 +05:30
test_rag_openai_ingestion.py chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
test_rate_limit_error_unification.py feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields (#27687) 2026-06-06 17:50:29 -07:00
test_redact_string_in_error_paths.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_redis.py fix(redis): honor ssl value instead of key presence when building async connection pool (#32590) 2026-07-16 13:38:01 -07:00
test_register_model_custom_pricing.py fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) 2026-07-07 20:49:03 -07:00
test_register_model_zero_cost_persistence.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_responses_api_bridge_non_stream.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_responses_id_security.py fix(responses_id_security): decrypt response ids for input_items follow-ups (#32269) 2026-07-06 14:03:15 -07:00
test_responses_streaming_container_ownership.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_retrieve_batch_bedrock_dispatch.py feat(batch-job): bedrock batch model invocation job retrieval (#26834) 2026-05-11 13:22:26 -07:00
test_router.py fix(router): cast model_info cost values to float in _set_model_group_info (#33556) 2026-07-16 12:05:36 -07:00
test_router_block_helpers.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_router_exception_redaction.py fix(router): mask provider credentials embedded in fallback error messages (#32083) 2026-07-03 18:48:06 -07:00
test_router_google_genai.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_model_cost_isolation.py fix(main): stop per-request custom pricing from clobbering shared model_cost pricing (#32163) 2026-07-07 10:25:31 -07:00
test_router_order_fallback.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_router_per_deployment_num_retries.py fix(router): guard num_retries=None in async_function_with_retries (#30036) 2026-06-23 10:16:15 -07:00
test_router_redis_init.py
test_router_retry_non_retryable_errors.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_retry_policy_update.py fix(proxy): tighten role gating on /get/config/callbacks response (#31745) 2026-07-01 17:58:31 -07:00
test_router_silent_experiment.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_streaming_fallback_metadata.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_router_weighted_failover.py feat: add weighted-routing failover (#27980) 2026-05-15 17:28:54 +00:00
test_ruff_strict_gate.py perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate (#32000) 2026-07-02 19:24:00 -07:00
test_secret_redaction.py fix(key management): enforce minimum custom key length and mask short keys in key_name (#33462) 2026-07-15 18:27:38 -07:00
test_service_logger.py fix: missing span for guardrail passthrough (#29552) 2026-06-03 01:25:15 +00:00
test_setup_wizard.py test: test 2026-03-28 19:17:38 -07:00
test_shared_session_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ssl_verify_unit.py Litellm OSS Staging (#29161) 2026-06-01 21:22:35 -07:00
test_stream_chunk_builder_annotations.py fix: merge annotations from all streaming chunks in stream_chunk_builder 2026-03-15 14:20:45 +05:30
test_streaming_connection_cleanup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_system_message_format_bug.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_thinking_enabled.py Litellm oss staging 250526 (#28770) 2026-05-26 11:57:39 -07:00
test_triage_rollout_heads_up.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_type_check_gate.py perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate (#32000) 2026-07-02 19:24:00 -07:00
test_type_discipline_gate.py refactor(lint): collapse type/lint budgets to a single per-rule limit (#31883) 2026-07-01 18:12:35 +03:00
test_utils.py fix(router): resolve prompt cache minimum per model instead of a flat 1024 2026-07-16 19:10:23 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_video_generation.py fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (#29098) 2026-05-28 11:45:41 -07:00
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py