scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-26 01:12:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 99
litellm/tests/test_litellm
History
devin-ai-integration[bot] a173657dfb
fix(caching): keep embedding cache hits aligned with request inputs (#42571) 
* fix(caching): keep embedding cache hits aligned with request inputs

Partial hits now send only the uncached inputs to the provider and merge
fresh vectors back into their original positions. Responses whose item
count differs from the input count (one input scoring many documents)
are no longer written to the per-input cache, since a later hit would
return a single item.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): drop mutable collection builds flagged by the type discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): bypass embedding cache entries written before the per input cardinality check

Embedding cache entries now carry format_version and readers treat entries without it as
misses, so entries that only hold the first row of a multi row response are refetched instead
of served until their TTL expires. The provider call also receives a copy of the request kwargs
with the uncached inputs rather than mutating the caller's mapping

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): assert a partial embedding cache hit becomes a full hit on repeat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): await pending embedding cache writes before asserting on cache hits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): validate cached embeddings without mutating responses or request kwargs

Validate cache rows through a frozen pydantic model so import does not depend on
TypeAdapter support for ReadOnly TypedDicts, accept string embeddings, build the
merged partial hit response instead of mutating the cached one, and hand the
provider request mapping to post call hooks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): keep cache_hit and response_ms on merged partial embedding hits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-22 17:14:44 -05:00
..
a2a_protocol fix(a2a): send message/stream for Bedrock AgentCore streaming requests 2026-09-21 13:39:32 +00:00
batches test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
caching fix(caching): keep embedding cache hits aligned with request inputs (#42571) 2026-09-22 17:14:44 -05:00
chat_completions test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
completion_extras fix(edenai): restore unrelated files the squashed PR commit had reverted 2026-09-21 19:42:37 +00:00
containers test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
endpoints test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
enterprise test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
expected_fine_tuning_api
expected_responses_api_request
experimental_mcp_client fix(mcp): restore legacy SSE and bounded cancellation cleanup (#42382) 2026-09-22 11:27:50 -07:00
fixtures/together_ai_sync feat(models): add daily Together AI model registry sync script and workflow 2026-08-25 13:12:06 -07:00
google_genai fix(google_genai): drop non-object tool parameters instead of forwarding them 2026-09-19 19:19:33 -07:00
images test(images): pin scalar-array edit params survive as repeated multipart fields 2026-08-24 12:49:54 -07:00
integrations fix(otel): keep text completion choice fields beside the synthesized message (#42537) 2026-09-22 14:24:32 -07:00
interactions refactor(interactions): remove expired use_legacy_interactions_schema shim 2026-09-17 20:14:07 +00:00
litellm_core_utils chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
llms chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
messages test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
ocr test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
passthrough test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
proxy test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
rag test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
rerank_api ci(unit): fail a hung test in 120s with a traceback instead of idling the shard to its step timeout 2026-09-19 02:53:36 -07:00
responses fix(mcp): preserve discovery attribution and sanitize logging headers (#42541) 2026-09-22 14:26:48 -07:00
router_strategy feat(router): add group-scoped priority routing strategy (#42378) 2026-09-22 13:09:38 -07:00
router_utils feat(openai): add GPT-6 Sol and GPT-6 Luna (#42515) 2026-09-22 11:34:01 -07:00
rust_bridge chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
secret_managers fix(aws_secret_manager_v2): restore secret scheduled for deletion instead of failing CreateSecret (#42454) 2026-09-22 14:12:20 -05:00
types feat: add configurable provider affinity header mapping (#41033) 2026-09-22 13:31:33 -07:00
vector_stores fix(bedrock): forward userContext in Knowledge Base Retrieve requests 2026-09-16 11:50:36 -07:00
videos test: remove phase 16 legacy test files from tests/test_litellm 2026-09-20 10:59:22 +00:00
__init__.py
conftest.py fix(bedrock): gate Invoke tool search on the model map's supports_tool_search flag 2026-09-16 18:23:14 -07:00
log.txt
readme.md
test_a2a_registry_lookup.py fix(a2a): Entra credentials own the chat route bearer over a stored api_key or authorization header 2026-09-17 16:09:36 -07:00
test_acompletion_session_reuse_e2e.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_add_deployment_no_master_key.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_aembedding_session_reuse_e2e.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_anthropic_beta_headers_filtering.py feat(router): native compact-to-fit across conversation APIs (#42074) 2026-09-21 22:52:29 -07:00
test_anthropic_skills_transformation.py
test_assert_ci_coverage.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
test_assert_workflow_dir_hygiene.py feat(ci): assert .github/workflows holds only workflows, correctly named (#37616) 2026-08-20 21:36:26 +00:00
test_audio_transcription_rust_bridge.py refactor(rust_bridge): group route modules into packages and split ocr into main and rust 2026-09-16 20:34:51 +00:00
test_auto_update_price_and_context_window_file.py feat(cost_map): derive source_revision from the loaded bytes instead of a _metadata stamp 2026-09-07 17:47:51 -07:00
test_azure_ad_token_credential_resolution.py test(router): cover s3_output_bucket_name surviving the trusted credential snapshot 2026-08-17 14:51:10 -07:00
test_azure_ai_grok_4_3_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_azure_ai_grok_4_6_model_metadata.py test: keep behavior tests that read the cost map for a later fixture rewrite 2026-09-18 04:52:06 +00:00
test_baseten_glm_5_3_model_metadata.py test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
test_batch_completion_models_all_responses.py
test_bedrock_marengo_embed_3_model_metadata.py test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
test_budget_ratchet_check.py test(ci): annotate new test locals as Final 2026-09-20 09:07:51 +00:00
test_chat_ui_responses_session.py
test_check_licenses.py fix(ci): retry transient PyPI license lookups 2026-08-23 09:23:35 +00:00
test_check_mcp_operation_boundary.py refactor(mcp): extract explicit operation context and dispatch 2026-09-21 12:24:15 -07:00
test_check_migrations_no_data_rewrites.py fix(migrations-check): read the table name past comments, ignore referential SET DEFAULT 2026-09-16 18:14:36 +00:00
test_check_py310_typing_imports.py fix: keep litellm importable on Python 3.10 and guard 3.11-only typing imports in CI (#39448) 2026-09-02 18:27:19 -07:00
test_check_test_quality.py test(ci): annotate new test locals as Final 2026-09-20 09:07:51 +00:00
test_check_type_discipline.py perf(ci): fan the budget checkers out across cores (#37784) 2026-08-22 22:44:58 -07:00
test_circleci_path_filter.py fix(mcp): preserve session expiry signals and scope dependency CI 2026-09-18 22:52:10 -07:00
test_circleci_rust_toolchain.py fix(ci): pin workflow toolchain dependencies 2026-09-02 12:16:25 -07:00
test_claude_fable_5_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_claude_opus_4_6_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_claude_opus_4_8_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_claude_opus_5_config.py feat(cost-map): add Claude Opus 5.5 for Vertex AI and Azure AI (#42599) 2026-09-22 21:48:07 +00:00
test_claude_sonnet_5_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_cloudflare_workers_ai_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_completion_timeout_resolution.py
test_component_entrypoint.py feat(proxy): share database connections across workers with an in-container pgbouncer (#39683) 2026-09-10 22:14:37 +00:00
test_compression.py
test_conftest.py test: trim the PROXY_BASE_URL fixture and regression docstrings 2026-08-19 00:56:37 -07:00
test_conftest_isolation.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
test_constants.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_container_router.py
test_cost_calculation_log_level.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_cost_calculator.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_cost_map_guard.py ci: skip cost map file checks on PRs that leave the cost map untouched (#42406) 2026-09-21 21:40:48 -07:00
test_count_tokens_public_api.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_dashscope_image_generation.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_daybreak_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_deepseek_model_metadata.py test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
test_default_branch.py ci: avoid duplicate default branch fetches 2026-09-07 15:28:01 -07:00
test_detect_changes.py perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559) 2026-08-19 18:32:21 -07:00
test_dockerfile_apk_repository.py fix(docker): add public Wolfi apk repo to runtime image (#39033) 2026-09-01 15:11:15 -07:00
test_dockerfile_bedrock_realtime_extra.py fix(bedrock): keep realtime SDK error range inside websocket close reason 2026-09-17 01:57:33 +00:00
test_dockerfile_non_root.py
test_drop_params_env_var.py fix(init): keep non-flag LITELLM_DROP_PARAMS values on with a warning 2026-09-07 22:13:23 -07:00
test_e2e_egress_sentinel.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_eager_tiktoken_load.py
test_env_key_doc_gate.py
test_exception_exports.py
test_exception_header_preservation.py fix(bedrock): keep x-amzn-RequestId on chat error responses (#40089) 2026-09-07 17:16:47 -07:00
test_exception_mapping_request_attribute.py
test_filter_out_litellm_params.py
test_fireworks_serverless_model_costs.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_gate_slot_lock.py ci: avoid duplicate default branch fetches 2026-09-07 15:28:01 -07:00
test_gemini_3_1_flash_lite_image_pricing.py test: drop gemini-3.1-flash-lite-image capability pins 2026-09-16 19:21:50 +00:00
test_gemini_tts_native_audio_pricing.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_get_blog_posts.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_git_hooks.py
test_gpt_5_4_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_gpt_5_5_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_gpt_image_cost_calculator.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_gpt_realtime_mode.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_groq_streaming_encoding.py
test_guardrail_exception_status_codes.py
test_lazy_imports.py perf: defer fastapi and tiktoken BPE imports out of import litellm 2026-09-17 08:59:48 +00:00
test_lint_workflow_diff_gates.py ci(lint): gate top-level tests/e2e and litellm files in the diff-scoped lint steps 2026-09-01 22:30:46 -07:00
test_litellm_params_reserved_keys.py
test_logging.py fix(logging): stamp scrubbed records with a private sentinel a caller cannot supply 2026-09-16 17:30:49 -07:00
test_lowest_latency_zero_tokens.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_main.py fix(edenai): restore unrelated files the squashed PR commit had reverted 2026-09-21 19:42:37 +00:00
test_main_module_header.py
test_mistral_medium_3_5_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_mistral_small_4_0_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_mistral_zai_glm_5_2_model_metadata.py test: keep behavior tests that read the cost map for a later fixture rewrite 2026-09-18 04:52:06 +00:00
test_model_block_unblock.py fix(proxy): serialize model reconciles so concurrent model writes stop evicting each other (#36687) 2026-08-12 13:42:26 -07:00
test_model_cost_aliases.py
test_model_param_helper.py
test_model_prices_schema.py fix(cost): bill DeepSeek V4.1 Flash and V4 Pro at their off-peak rates outside peak hours 2026-09-19 04:31:55 -07:00
test_model_response_normalization.py
test_muse_spark_1_1_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_muse_spark_1_2_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_muse_spark_1_3_model_metadata.py test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
test_mutation_report.py fix(ci): stop the mutation report publishing a score it never measured (#37825) 2026-08-21 20:15:52 -07:00
test_nested_drop_params.py
test_non_chat_routes_open_llm_spans.py fix(otel): emit LLM Call spans for speech, image, moderation, ocr and transcription (#37752) 2026-08-22 11:11:21 -07:00
test_openai_embedding_encoding_format_default.py test(embeddings): move legacy intercepts to the wire for the omitted-format path 2026-08-29 12:04:44 -07:00
test_openai_service_tier_long_context_pricing.py feat(openai): add GPT-6 Sol and GPT-6 Luna (#42515) 2026-09-22 11:34:01 -07:00
test_pre_commit_lint.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
test_prisma_generate_if_needed.py
test_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
test_project_alias_tracking.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_project_tags_pydantic.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_proxy_auth.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_rag_openai_ingestion.py
test_rate_limit_error_unification.py feat(proxy): add budget_exceeded_status_code setting to restore 429 for budget refusals 2026-09-20 07:00:10 +00:00
test_redact_string_in_error_paths.py fix(proxy): stop leaking internal exception details to clients (#39380) 2026-09-02 17:32:00 -07:00
test_redis.py test(redis): pin ElastiCache IAM signing and TLS coercion invariants 2026-09-10 10:13:03 -04:00
test_redis_credential_provider.py test(redis): pin ElastiCache IAM signing and TLS coercion invariants 2026-09-10 10:13:03 -04:00
test_register_model_custom_pricing.py fix(cost): bill off-peak rates for deployments that set only off_peak_pricing 2026-09-01 12:14:46 -07:00
test_register_model_zero_cost_persistence.py
test_replicate_model_key_format.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_responses_api_bridge_non_stream.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_responses_id_security.py fix(proxy): authorize every Responses API id, not only the ones the proxy issued (#39548) 2026-09-11 11:47:05 -07:00
test_responses_streaming_container_ownership.py
test_retrieve_batch_bedrock_dispatch.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_router_block_helpers.py
test_router_exception_redaction.py fix(router): explain fallback outcome in plain words in the raised error (#42509) 2026-09-22 18:37:42 +00:00
test_router_google_genai.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_model_cost_isolation.py fix(router): preserve discovered limits and model info fallbacks 2026-09-17 00:23:50 +00:00
test_router_order_fallback.py fix(router): skip the refusing deployment when retrying a non-transient error 2026-09-05 22:25:13 -07:00
test_router_per_deployment_num_retries.py refactor(router): resolve retry policy by exception MRO and add DefaultRetries 2026-09-04 16:09:01 -07:00
test_router_redis_init.py
test_router_retry_backoff_headers.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_router_retry_non_retryable_errors.py fix(timing): union provider timing windows and anchor detailed pre-processing at receive time 2026-09-19 00:48:17 +00:00
test_router_retry_policy_update.py fix(proxy): persist only the router settings keys the request set 2026-09-18 14:01:05 -07:00
test_router_silent_experiment.py fix(router): snapshot shadow kwargs per target so concurrent shadows never share metadata 2026-09-16 04:31:43 +00:00
test_router_streaming_fallback_metadata.py
test_router_weighted_failover.py fix(router): skip the refusing deployment when retrying a non-transient error 2026-09-05 22:25:13 -07:00
test_ruff_strict_gate.py fix: address cross-version CI failures 2026-09-02 14:17:19 -07:00
test_sambanova_model_metadata.py test: keep behavior tests that read the cost map for a later fixture rewrite 2026-09-18 04:52:06 +00:00
test_secret_redaction.py fix(bedrock): keep batch S3 credentials out of chat requests and debug logs 2026-09-21 20:42:04 +00:00
test_select_ui_test_scope.py test(ui): align workflow expectations with Vitest 4 2026-09-08 16:13:58 -07:00
test_service_logger.py
test_setup_wizard.py
test_shared_session_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_ssl_verify_unit.py
test_stream_chunk_builder_annotations.py
test_stream_chunk_builder_citations.py fix(streaming): join block-list citation deltas without extra nesting 2026-08-28 13:56:07 -07:00
test_stream_chunk_builder_images.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_streaming_connection_cleanup.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_sync_together_ai_models.py feat(cost_map): derive source_revision from the loaded bytes instead of a _metadata stamp 2026-09-07 17:47:51 -07:00
test_system_message_format_bug.py
test_test_quality_gate.py ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
test_thinking_enabled.py test: drop restating comment and wrap long call in thinking tests 2026-08-18 19:55:22 -07:00
test_together_ai_model_metadata.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_type_check_gate.py fix(lint): retire the single-slot base-counts cache 2026-08-06 02:23:52 -07:00
test_type_discipline_gate.py
test_typesafe_model_metadata.py feat(proxy): add TypeSafe Jev passthrough spend tracking 2026-09-17 15:53:19 +00:00
test_unit_shard_per_test_timeout.py test(ci): drop the structure-only assertion on the shard script; the parametrized hang test covers both invocations 2026-09-19 03:20:48 -07:00
test_utils.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_utils_module_docstring.py
test_uuid_helper.py
test_vcr_safe_body_matcher.py
test_vertex_ai_xai_grok_prompt_caching_metadata.py test(pricing): assert cache-priced vertex grok rows advertise supports_prompt_caching (#41526) 2026-09-21 20:58:11 -07:00
test_video_generation.py test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
test_with_dashboard_node.py
test_xai_grok_4_3_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_xai_responses_auto_routing.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py