litellm/tests
devin-ai-integration[bot] 77fc3315e5
fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts (#43878)
* fix(caching): keep tool calls and tool results in semantic cache prompts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): keep semantic tool prompt helpers within lint budgets

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): keep structured function_call_output text in semantic prompts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): split Responses text-field collection to stay within complexity budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): tag each tool result with the position of the call it answers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): encode tool result position and output together so tool text cannot forge result tags

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): expect encoded tool result record in qdrant semantic prompt parity case

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): cover tool result arrangements, SDK clients, concurrency and qdrant outage for semantic cache

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(caching): embed every semantic cache prompt field except volatile ones

Replace the per-shape allowlist in the Python and Rust semantic cache prompt
walkers with one include-by-default walker. Plain text keeps its old
concatenation; any other block or message is embedded as compact JSON with
call ids mapped to ordinals, cache_control dropped, and signatures, encrypted
content and base64 data replaced with a short sha256 digest.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(code-quality): allow the bounded semantic cache prompt walkers in the recursion check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rust): expect structured JSON for unknown fields in redis and valkey semantic prompts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "test(rust): expect structured JSON for unknown fields in redis and valkey semantic prompts"

This reverts commit 86c82b949b.

* Revert "test(code-quality): allow the bounded semantic cache prompt walkers in the recursion check"

This reverts commit 39efb5d9da.

* Revert "feat(caching): embed every semantic cache prompt field except volatile ones"

This reverts commit 5aed3ab3de.

* refactor(caching): rename get_str_from_messages_with_tools to get_semantic_cache_prompt_from_messages

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): split semantic cache prompt extraction by API format

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): drop TypeIs guard and register Responses prompt walker with the recursion check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): pick the Responses text field without a Final inside a loop

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): walk semantic cache prompts as plain dicts, dumping pydantic items once up front

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): write the semantic cache prompt builders as plain loops

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): drop formatting-only churn from the redis semantic cache tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): drop the caching group wiring that main already carries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): recurse into tool_result content in the semantic cache prompt helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): check the max_messages cap on the shared exact-cache proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): read list-form function_call_output text in semantic cache prompts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): extract nested Responses input lookup to keep walker under complexity limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "refactor(caching): extract nested Responses input lookup to keep walker under complexity limit"

This reverts commit 0665296bf1.

* style(caching): suppress C901 on the Responses input walker instead of splitting it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-07 17:32:39 +00:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
agent_tests/local_only_agent_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
audio_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
batches_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
benchmarks refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
code_coverage_tests refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
documentation_tests fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
e2e test(e2e): tag guardrails and logging tests with Subject metadata and record client steps (#44963) 2026-10-07 10:32:25 -07:00
guardrails_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
image_gen_tests refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
integration fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts (#43878) 2026-10-07 17:32:39 +00:00
litellm_utils_tests refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
llm_responses_api_testing test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
llm_translation refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
load_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
local_testing refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
logging_callback_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
mcp_tests feat(mcp): add portable catalog pagination (#44446) 2026-10-06 18:54:50 -07:00
multi_instance_e2e_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
ocr_tests refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
openai_endpoints_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
otel_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
pass_through_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
pass_through_unit_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
proxy_admin_ui_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
proxy_behavior fix(lens): reuse trace reviews and consolidate findings across runs (#44778) 2026-10-06 12:09:56 -07:00
proxy_e2e_anthropic_messages_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_migration_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_security_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_unit_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
router_unit_tests refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
rust-python-harness refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
search_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
spend_tracking_tests test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
store_model_in_db_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_litellm refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_litellm_rust refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
unified_google_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
unit fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts (#43878) 2026-10-07 17:32:39 +00:00
vector_store_tests refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
windows_tests fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_anthropic_compaction_usage.py
test_budget_management.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_callbacks_on_proxy.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_fallbacks.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_gpt5_azure_temperature_support.py
test_health.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_keys.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_new_vector_store_endpoints.py
test_openai_endpoints.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py test(proxy): delete the legacy proxy test tree and shard tests/unit/proxy by glob (#44018) 2026-10-01 11:40:45 -07:00
test_resource_cleanup.py
test_rust_python_harness.py test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) 2026-09-25 19:10:20 -07:00
test_service_logger_otel.py
test_spend_logs.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_team.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_team_logging.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_team_members.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_users.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.