mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
4bf40c4e8d
* fix(logging): stop billing and logging response reads as LLM calls Retrieving, deleting or cancelling a stored response, and vector store management calls, run through the same logging lifecycle as inference. A retrieved response replays the usage of the call that created it, so every read priced it again and wrote a second spend log row for the same tokens. Non-inference calls now cost 0, report no usage, log no placeholder chat message, and get a litellm.responses_management operation name instead of reading as chat. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep billing background response jobs after the poll Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): use an empty list for read-call messages A tuple matches no branch in the loggers that walk this value, so lunary's parse_messages falls through to clean_message and raises AttributeError on the success hook. An empty list reads as no messages everywhere: it satisfies the isinstance(list) checks in newrelic, mlflow and datadog, iterates zero times in traceloop and helicone, and is what StandardLoggingPayload.messages is typed to hold. None would be type-legal too but is not iterable, so it trades one crash for another in mlflow and traceloop. * fix(otel): stop the legacy emitter reporting replayed tokens on response reads The zeroing so far lands in the standard logging payload, which the legacy OpenTelemetry emitter does not read for usage: it takes prompt, completion and total tokens straight off the response object, so a retrieval span still carried the token counts of the call that produced the response, and the token usage histogram still recorded them. That emitter is the default, so the spend row said zero while the trace said otherwise. The background cost poller keeps its counts, the same exemption the pricing path already makes. * fix(logging): keep billing a background response when its retrieval is read A response created with background=true comes back queued and carries no usage, so its create bills nothing. The retrieval that first sees the finished job is the only place that job's tokens are ever visible, and pricing every read at zero therefore loses the spend outright rather than deduplicating it. On a proxy without the enterprise cost poller a background job ended up costing $0 end to end. is_unbilled_non_inference_call now takes the response it is deciding about and treats a background response the same way it already treats the poller's own read, which is the same exemption seen from the other side. The legacy OpenTelemetry emitter's time per output token metric picks up the read gate it was missing, so it stops dividing a read's latency by the replayed completion token count. * test(proxy): pass the read response to the non-inference predicate The poller test called is_unbilled_non_inference_call with the pre-background signature, so it broke when the predicate gained the response it classifies. It now hands the predicate a foreground read, and asserts that the same read is free without the origin stamp, so the stamp is what the test proves. * fix(otel): stop the v2 metrics recorder reporting replayed tokens on response reads The v2 span builder sources usage from the standard logging payload, so the earlier fix already zeroes it there. The metrics recorder reads response_obj directly, so a responses-management read still recorded the original generation's tokens into gen_ai.client.token.usage and divided generation time by them for gen_ai.server.time_per_output_token. The read still records operation and response duration, under the litellm.responses_management operation, so it stays observable. * fix(proxy): keep the response-cost headers on calls priced at zero Pricing responses reads and vector-store management routes at zero dropped the whole x-litellm-response-cost family off those replies. The header build reads a falsy zero as a cost this response never recorded and filters it out, and a call that returns before pricing stores no cost breakdown for the component headers to read, so a client parsing the cost off a read got a KeyError where it had previously been handed a number. Those calls now advertise the family at zero. Retrieving a background response, and the cost poller's read of one, still report their real cost. The params-taking form of the predicate moves from opentelemetry into internal_call_metadata so the proxy header build and the OTEL recorders share one copy. * fix(proxy): report a zero cost split only under a zero cost total The component headers were filled from call-type membership alone, while the total they sit beside keeps its real value when the read priced normally, so a breakdown that had not landed by the time headers were built could advertise a real total next to an all-zero split. The split is now reported as zero only when the total agrees with it, and is otherwise left absent. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yucheng Zhu <yucheng@berri.ai> |
||
|---|---|---|
| .. | ||
| arize | ||
| azure_storage | ||
| bitbucket | ||
| cloudzero | ||
| code_interpreter_interception | ||
| compression_interception | ||
| datadog | ||
| dotprompt | ||
| focus | ||
| gcs_bucket | ||
| gcs_pubsub | ||
| gitlab | ||
| helicone | ||
| langfuse | ||
| levo | ||
| litellm_agent | ||
| mavvrik_focus | ||
| newrelic | ||
| open_telemetry | ||
| opik | ||
| otel | ||
| SlackAlerting | ||
| websearch_interception | ||
| rubrik_test_helpers.py | ||
| test_agentops.py | ||
| test_anthropic_cache_control_hook.py | ||
| test_athina.py | ||
| test_azure_sentinel.py | ||
| test_braintrust_logging.py | ||
| test_braintrust_span_name.py | ||
| test_custom_guardrail.py | ||
| test_custom_guardrail_recursion.py | ||
| test_custom_prompt_management.py | ||
| test_deepeval.py | ||
| test_galileo.py | ||
| test_guardrail_logging_sync.py | ||
| test_helicone.py | ||
| test_langfuse.py | ||
| test_langfuse_otel.py | ||
| test_langsmith_init.py | ||
| test_lunary.py | ||
| test_mlflow.py | ||
| test_openmeter.py | ||
| test_opentelemetry.py | ||
| test_opentelemetry_dynamic_imports.py | ||
| test_opik_utils.py | ||
| test_otel_guardrail_violation_spans.py | ||
| test_otel_team_attributes_matrix.py | ||
| test_prometheus_api_promql_escape.py | ||
| test_prometheus_budget_metric_guard.py | ||
| test_prometheus_budget_metrics_db_lookups.py | ||
| test_prometheus_budget_metrics_timeout.py | ||
| test_prometheus_cache_metrics.py | ||
| test_prometheus_caller_identity.py | ||
| test_prometheus_client_ip_user_agent.py | ||
| test_prometheus_custom_metadata_label_counts.py | ||
| test_prometheus_end_user_cardinality.py | ||
| test_prometheus_invalid_key_filtering.py | ||
| test_prometheus_labels.py | ||
| test_prometheus_mcp_tool_metrics.py | ||
| test_prometheus_media_generation_metrics.py | ||
| test_prometheus_metric_name_consistency.py | ||
| test_prometheus_metrics_endpoint.py | ||
| test_prometheus_missing_metrics.py | ||
| test_prometheus_none_metadata.py | ||
| test_prometheus_overhead_with_guardrails.py | ||
| test_prometheus_queue_guardrail_metrics.py | ||
| test_prometheus_rate_limit_labels.py | ||
| test_prometheus_remaining_tokens_router_fallback.py | ||
| test_prometheus_service_tier_label.py | ||
| test_prometheus_services.py | ||
| test_prometheus_spend_logs_metadata.py | ||
| test_prometheus_stream_label.py | ||
| test_prometheus_token_detail_metrics.py | ||
| test_prometheus_user_team_metrics.py | ||
| test_prompt_manager_ssti.py | ||
| test_responses_background_cost.py | ||
| test_rubrik.py | ||
| test_s3.py | ||
| test_s3_v2.py | ||
| test_shadow_eval_logger.py | ||
| test_weave_otel.py | ||