tin-berri ca0b951a43 feat(spend): report prompt caching savings as total and gateway-attributed (#38134) ... * feat(spend): report prompt caching savings as total and gateway-attributed `prompt_caching_savings_spend` credited every cached request, including caching a client asked for with its own `cache_control` and caching a provider does implicitly, so the number overstated what the gateway had any hand in. Gating that column in place would have fixed the overstatement by changing what the column means, leaving rows written before the change saying "all caching savings" and rows after saying "gateway-injected only" with nothing to tell them apart, and forcing a decision about rewriting history. It also breaks the cache-leakage estimate on the dashboard, whose numerator would be gated while its denominator, the cached token counts, would not, so the rate it extrapolates from would be quietly diluted. Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar caching saved, which is what a customer means by "what did caching save me", and the new `gateway_injected_caching_savings_spend` carries the subset litellm caused by injecting the breakpoints itself. Both are derived from the same marker, so this changes what is done with it rather than how it is obtained. The attributed figure is normally the smaller of the two, being a subset of the same requests, but not always: a request that writes cache it never reads has negative net savings, and excluding such a request can lift the attributed figure above the total. Also stops the marker riding into a fallback leg. The fallback rebuild spread the failed attempt's metadata forward, so a deployment that injected nothing inherited the marker and was credited anyway, which silently restored the very overstatement this separates out. * fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478) The savings marker records breakpoints litellm placed, and a tool_config injection point becomes one only in the converse transform, and only when the request carries tools. The prompt hook cannot see either condition, so marking on the point's presence credited request shapes that cached nothing, while Bedrock tool caching the gateway did cause went uncredited. Record it at the placement site instead. The marker's reader also resolves its bucket by value now: litellm_params declares litellm_metadata as None on every request, so asking the shared name resolver named a bucket that was not there and the mark was dropped.		2026-08-28 00:19:06 -07:00
..
arize	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
azure_storage	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
bitbucket	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
cloudzero	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
code_interpreter_interception	feat(sandbox): reuse e2b container across requests when metadata.session_id is set (#31688)	2026-06-30 18:58:09 -07:00
compression_interception	feat(spend): track prompt compression saved tokens in daily spend aggregates (#33810)	2026-07-18 17:47:54 -07:00
datadog	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
dotprompt	Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_dotprompt_model_swap	2026-08-26 15:13:57 -07:00
focus	test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)	2026-08-20 20:24:49 -07:00
gcs_bucket	test: unwind environment writes in tests/test_litellm with monkeypatch (#37806)	2026-08-21 20:28:37 -07:00
gcs_pubsub	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
gitlab	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
helicone	test: run the 30 test files stranded in the second mirror (#37595)	2026-08-20 10:59:43 -07:00
langfuse	fix(langfuse): warn and drop invalid LANGFUSE_TRACING_ENVIRONMENT instead of failing requests (#38582)	2026-08-27 18:03:42 -07:00
levo	test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)	2026-08-20 20:24:49 -07:00
litellm_agent	Agent Builder - support new experimental agent builder, to ensure agents pass compliance checks (#21817)	2026-02-21 15:32:47 -08:00
mavvrik_focus	chore: litellm oss staging (#31185)	2026-06-26 09:17:44 -07:00
newrelic	feat(newrelic): per-team cost and usage metrics via team callbacks (#37610)	2026-08-26 23:42:02 -07:00
open_telemetry	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
opik	Litellm oss staging 030626 (#29578)	2026-06-03 11:01:51 -07:00
otel	fix(otel): anchor MCP tool-call spans to the gateway's own trace, link the client's context (#38317)	2026-08-27 13:41:30 -07:00
SlackAlerting	feat(alerting): add native Microsoft Teams alerting destination (#38367)	2026-08-27 16:19:22 -07:00
websearch_interception	fix: keep accepting a loop ceiling that spells a whole number	2026-08-22 11:24:22 -07:00
rubrik_test_helpers.py	Litellm oss staging 04 21 2026 2 (#26569)	2026-05-20 21:25:19 -07:00
test_agentops.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_anthropic_cache_control_hook.py	feat(spend): report prompt caching savings as total and gateway-attributed (#38134)	2026-08-28 00:19:06 -07:00
test_athina.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_azure_sentinel.py	test: remove tests that never execute	2026-08-12 10:45:38 -07:00
test_braintrust_logging.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_braintrust_span_name.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_custom_guardrail.py	fix(guardrails): classify all 4xx HTTPException guardrail blocks as intervened (#33821)	2026-07-27 16:50:13 -07:00
test_custom_guardrail_recursion.py	fix: recursive pydantic issue (#19531)	2026-01-22 19:56:41 -08:00
test_custom_prompt_management.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_deepeval.py	fix: use fastuuid helper (#14903)	2025-09-25 15:47:01 -07:00
test_galileo.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_guardrail_logging_sync.py	fix(guardrails): keep guardrail information in spend logs when the caller sends its own metadata	2026-07-24 16:20:44 -07:00
test_helicone.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_langfuse.py	fix(langfuse): warn and drop invalid LANGFUSE_TRACING_ENVIRONMENT instead of failing requests (#38582)	2026-08-27 18:03:42 -07:00
test_langfuse_otel.py	feat(langfuse): support langfuse_environment as a per-key dynamic callback param (#38264)	2026-08-26 16:56:55 -07:00
test_langsmith_init.py	fix(langsmith): keep root-run ids self-consistent so batch ingest stops rejecting header-tagged requests (#38116)	2026-08-24 19:47:12 -07:00
test_lunary.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_mlflow.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_openmeter.py	test: unwind environment writes in tests/test_litellm with monkeypatch (#37806)	2026-08-21 20:28:37 -07:00
test_opentelemetry.py	fix(logging): stop billing and logging response reads as LLM calls (#36890)	2026-08-26 18:34:17 -07:00
test_opentelemetry_dynamic_imports.py	Make grpc dependency optional (#19447)	2026-01-20 19:03:52 -08:00
test_opik_utils.py	chore(ci): merge oss branch (#33784)	2026-07-17 23:22:13 +00:00
test_otel_guardrail_violation_spans.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_otel_team_attributes_matrix.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_prometheus_api_promql_escape.py	fix(prometheus): quote api_key for PromQL string literal in spend lookup	2026-05-01 21:38:19 +00:00
test_prometheus_budget_metric_guard.py	fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834)	2026-07-10 20:25:47 -07:00
test_prometheus_budget_metrics_db_lookups.py	perf(auth): negative-cache missing user/key lookups on the request hot path (#32368)	2026-07-08 09:59:57 +03:00
test_prometheus_budget_metrics_timeout.py	feat: litellm oss staging (#31935)	2026-07-03 09:27:31 +05:30
test_prometheus_cache_metrics.py	fix(prometheus): populate cache write token metrics for OpenAI-style usage (#34803)	2026-07-27 12:28:19 -07:00
test_prometheus_caller_identity.py	test(prometheus): cover caller-identity config failure cases (#38380)	2026-08-26 11:53:49 -07:00
test_prometheus_client_ip_user_agent.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_prometheus_custom_metadata_label_counts.py	Fix Prometheus remaining metric zero values (#27348)	2026-05-06 17:22:20 -07:00
test_prometheus_end_user_cardinality.py	perf: cap Prometheus end-user metric cardinality with TTL + LRU eviction (#27272)	2026-05-06 13:35:13 -07:00
test_prometheus_invalid_key_filtering.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_prometheus_labels.py	test: rename tests that a later definition shadowed	2026-08-12 11:15:54 -07:00
test_prometheus_mcp_tool_metrics.py	feat(prometheus): expose MCP tool metadata in Prometheus metrics (#31899)	2026-07-02 10:56:35 +03:00
test_prometheus_media_generation_metrics.py	feat(prometheus): expose video duration and image count consumption metrics (#33138)	2026-07-13 18:51:13 -07:00
test_prometheus_metric_name_consistency.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_prometheus_metrics_endpoint.py	perf(prometheus): render /metrics off the event loop and coalesce concurrent scrapes (#37702)	2026-08-20 16:08:22 -07:00
test_prometheus_missing_metrics.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_prometheus_none_metadata.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_prometheus_overhead_with_guardrails.py	feat(prometheus): add litellm_total_overhead_latency_metric (SDK overhead + guardrails) (#31593)	2026-06-30 17:34:17 +08:00
test_prometheus_queue_guardrail_metrics.py	fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric (#37958)	2026-08-22 14:25:55 -07:00
test_prometheus_rate_limit_labels.py	fix(prometheus): read v3 rate limiter remaining values for per-key model gauges (#33119)	2026-07-13 14:27:56 -07:00
test_prometheus_remaining_tokens_router_fallback.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_prometheus_service_tier_label.py	feat(otel): stamp service tier attributes on inference spans (#35679)	2026-08-03 23:10:01 -07:00
test_prometheus_services.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_prometheus_spend_logs_metadata.py	fix(prometheus): expose project_alias in custom metadata labels (LIT-3741) (#31784)	2026-07-01 10:44:02 +08:00
test_prometheus_stream_label.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_prometheus_token_detail_metrics.py	fix(prometheus): populate cache write token metrics for OpenAI-style usage (#34803)	2026-07-27 12:28:19 -07:00
test_prometheus_user_team_metrics.py	fix(proxy): count only active users toward license seat limit (#31227)	2026-06-29 18:01:02 -07:00
test_prompt_manager_ssti.py	fix(security): sandbox jinja2 in gitlab/arize/bitbucket prompt managers	2026-05-02 09:14:02 +00:00
test_responses_background_cost.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
test_rubrik.py	fix(guardrails/rubrik): attribute blocked requests to the caller that made them (#35734)	2026-08-03 19:56:24 -07:00
test_s3.py	feat(s3): support SSE-KMS encryption params on both S3 logging paths (#35291)	2026-07-30 21:18:33 -07:00
test_s3_v2.py	fix(s3_v2): percent-encode object keys once so signed and sent URLs match (#38005)	2026-08-24 18:16:17 -07:00
test_shadow_eval_logger.py	fix(shadow_eval): refuse a judge model that also serves one of the arms it grades (#38589)	2026-08-27 18:44:44 -07:00
test_weave_otel.py	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00