mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
2d40254b57
The exact-request cache reused 5% of routed traffic (build 218: 19 hits, 350 misses) because every test salts its prompt with a fresh unique_marker(), so the same test could never match itself across builds. It also routed only openai and anthropic, while the week's flakiness was Bedrock. Key is now HMAC(test id + method + URL + headers + body, with every unique_marker() token replaced by a placeholder, + FIFO slot index). The slot index is what keeps two marker-only-different calls in one test on two recordings and therefore two provider response ids, so spend rows still reconcile one per invocation. A call outside any test is not cacheable. Bedrock gets a region-qualified mount and SigV4 re-signing, since the edge rewrites the Host the proxy signed. Signature headers are excluded from the key for signing mounts only, because x-amz-date would otherwise make every Bedrock request a permanent miss; every other mount still keys on its credentials whole. Only Anthropic-on-Bedrock chat deployments route: embeddings, image generation, rerank and realtime keep their direct path, and so do deployments carrying their own aws_role_name or static keys, whose whole point is to prove the product's assume-role chain rather than the runner's. The two eventstream actions bypass the cache and go live, still signed. Counters are now attributed per mount as well as in total, so a build can report a per-provider hit rate instead of one number. |
||
|---|---|---|
| .. | ||
| azure_client_usage_test.py | ||
| ban_constant_numbers.py | ||
| ban_copy_deepcopy_kwargs.py | ||
| bedrock_pricing.py | ||
| callback_manager_test.py | ||
| check_data_replace_usage.py | ||
| check_e2e_no_raw_requests.py | ||
| check_endpoint_coverage.py | ||
| check_fastuuid_usage.py | ||
| check_get_model_cost_key_performance.py | ||
| check_guardrail_apply_decorator.py | ||
| check_licenses.py | ||
| check_migrations_no_data_rewrites.py | ||
| check_prisma_binary_cache.py | ||
| check_provider_folders_documented.py | ||
| check_py310_typing_imports.py | ||
| check_spanattributes_value_usage.py | ||
| check_unsafe_enterprise_import.py | ||
| check_workflow_job_name_collisions.py | ||
| check_workflow_startup_safety.py | ||
| code_qa_check_tests.py | ||
| enforce_llms_folder_style.py | ||
| ensure_async_clients_test.py | ||
| info_log_check.py | ||
| liccheck.ini | ||
| license_cache.json | ||
| litellm_logging_code_coverage.py | ||
| log.txt | ||
| memory_test.py | ||
| pass_through_code_coverage.py | ||
| prevent_key_leaks_in_exceptions.py | ||
| recursive_detector.py | ||
| router_code_coverage.py | ||
| router_enforce_line_length.py | ||
| test_aio_http_image_conversion.py | ||
| test_ban_set_verbose.py | ||
| test_chat_completion_imports.py | ||
| test_e2e_changed_gate.py | ||
| test_e2e_idp_stack.py | ||
| test_no_hardcoded_secrets.py | ||
| test_provider_cache.py | ||
| test_provider_replay_harness.py | ||
| test_proxy_types_import.py | ||
| test_router_strategy_async.py | ||
| test_workflow_job_name_collisions.py | ||
| user_api_key_auth_code_coverage.py | ||