mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-08 22:21:35 +00:00
ca0b951a43
* feat(spend): report prompt caching savings as total and gateway-attributed `prompt_caching_savings_spend` credited every cached request, including caching a client asked for with its own `cache_control` and caching a provider does implicitly, so the number overstated what the gateway had any hand in. Gating that column in place would have fixed the overstatement by changing what the column means, leaving rows written before the change saying "all caching savings" and rows after saying "gateway-injected only" with nothing to tell them apart, and forcing a decision about rewriting history. It also breaks the cache-leakage estimate on the dashboard, whose numerator would be gated while its denominator, the cached token counts, would not, so the rate it extrapolates from would be quietly diluted. Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar caching saved, which is what a customer means by "what did caching save me", and the new `gateway_injected_caching_savings_spend` carries the subset litellm caused by injecting the breakpoints itself. Both are derived from the same marker, so this changes what is done with it rather than how it is obtained. The attributed figure is normally the smaller of the two, being a subset of the same requests, but not always: a request that writes cache it never reads has negative net savings, and excluding such a request can lift the attributed figure above the total. Also stops the marker riding into a fallback leg. The fallback rebuild spread the failed attempt's metadata forward, so a deployment that injected nothing inherited the marker and was credited anyway, which silently restored the very overstatement this separates out. * fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478) The savings marker records breakpoints litellm placed, and a tool_config injection point becomes one only in the converse transform, and only when the request carries tools. The prompt hook cannot see either condition, so marking on the point's presence credited request shapes that cached nothing, while Bedrock tool caching the gateway did cause went uncredited. Record it at the placement site instead. The marker's reader also resolves its bucket by value now: litellm_params declares litellm_metadata as None on every request, so asking the shared name resolver named a bucket that was not there and the mark was dropped. |
||
|---|---|---|
| .. | ||
| batches | ||
| chat | ||
| count_tokens | ||
| embed | ||
| files | ||
| image | ||
| image_edit | ||
| invoke_agent | ||
| messages/invoke_transformations | ||
| passthrough | ||
| realtime | ||
| rerank | ||
| search | ||
| vector_stores | ||
| __init__.py | ||
| test_anthropic_beta_support.py | ||
| test_base_aws_llm.py | ||
| test_bedrock_common_utils.py | ||
| test_bedrock_ssl_verify.py | ||
| test_claude_platform_provider.py | ||
| test_converse_context_management.py | ||
| test_cross_region_inference_profile_mapping.py | ||
| test_mantle.py | ||
| test_nova_imported_models.py | ||
| test_request_metadata.py | ||
| test_web_identity_session_policy.py | ||