mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
3325ea487f
Motivation: With litellm.enable_anthropic_prompt_caching on, Logging.get_custom_logger_for_prompt_management picked exactly one prompt-management hook per request via first-match-wins branching. The AnthropicCacheControlHook branch, seeded by main.py's maybe_seed_default_injection_points, matched before the vector store branch, so a model configured with vector_store_ids never ran its knowledge-base retrieval. vector_store_ids was never popped from the request params and was serialized straight into the provider body, which rejects it (e.g. Bedrock's "vector_store_ids: Extra inputs are not permitted"). The same starvation applies whenever a model carries explicit cache_control_injection_points, independent of the flag. Approach: Logging.async_get_chat_completion_prompt now also runs the vector store hook, ahead of the selected AnthropicCacheControlHook, whenever the registry has vector stores to run for the request. Vector store retrieval runs first so it can inject knowledge-base context and pop vector_store_ids before the cache-control hook stamps breakpoints onto the augmented messages. Extracted the singleton lookup and callback registration for the vector store hook into a small _get_vector_store_pre_call_hook helper shared by the existing selector and the new composition path. The sync get_chat_completion_prompt path is untouched: VectorStorePreCallHook only implements the async variant, so vector store retrieval was already async-only before this change. Validation: Added test_prompt_hooks_compose_vector_search_with_anthropic_cache_control to tests/test_litellm/litellm_core_utils/test_litellm_logging.py. It seeds cache_control_injection_points the same way main.py does, registers a vector store, and asserts the search hook is awaited, vector_store_ids and cache_control_injection_points are both consumed, the retrieved context is prepended to the messages, and the cache_control breakpoint still lands on the trailing user message. Confirmed the test fails on the pre-fix code (search mock never awaited) and passes after the fix. Ran: - uv run pytest tests/test_litellm/litellm_core_utils/test_litellm_logging.py -v (303 passed, 1 pre-existing failure unrelated to this change: TestAnthropicPassthroughCustomPricing:: test_completion_cost_receives_custom_pricing_args fails on a clean tree too, from a missing orjson dependency in this sandbox) - uv run pytest tests/test_litellm/integrations/test_anthropic_cache_control_hook.py tests/test_litellm/integrations/vector_store_integrations/test_vector_store_pre_call_hook.py -q (234 passed, 2 pre-existing failures unrelated to this change: both fail identically on a clean tree from a missing websockets dependency in this sandbox) - make lint (ruff, ruff-strict-gate, type-discipline-gate, test-quality-gate, type-check-gate, basedpyright, circular-import check): all checks passed with no new budget regressions Report: https://github.com/BerriAI/litellm/issues/40908 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com> Assisted-by: claude-sonnet-5 (via Claude Code) |
||
|---|---|---|
| .. | ||
| _support | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| guardrails_tests | ||
| harness_e2e | ||
| image_gen_tests | ||
| integration | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| rust-python-harness | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| test_litellm_rust | ||
| unified_google_tests | ||
| unit | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_rust_python_harness.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.