scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-07 07:58:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 150
litellm/tests/integration/spend
History
mrinal-berri 0b57b9976a
feat(daily-activity): per-guardrail spend breakdown in user, team and organization daily activity (#48702) 
* feat(daily-activity): read guardrail spend per key from the daily guardrail usage table

Adds DailyActivityRepository.guardrail_spend, which sums LiteLLM_DailyGuardrailUsageUnits.cost by date, guardrail and key. A guardrail row counts only when its key has a spend row on that day under the same scope filters as the rest of the daily activity response, so user, key and date filters apply to guardrail cost too

* feat(daily-activity): add a per-guardrail spend breakdown to /user/daily/activity/aggregated

Each day now has breakdown.guardrails[<guardrail_id>] with metrics.spend and an api_key_breakdown carrying each key's spend and metadata. Guardrail entries only report spend, since that is the one metric stored per key. The breakdown is on the user endpoint only, is empty when a model filter is set because guardrail cost is not stored per model, and lists only keys within api_key_limit while the guardrail total keeps every key. spend and budgets are unchanged

Resolves LIT-9785

* feat(daily-activity): add the guardrail breakdown to team and organization activity, with guardrail names

/team/daily/activity/aggregated and /organization/daily/activity/aggregated now return breakdown.guardrails the same way the user endpoint does, since each key belongs to one team and one organization. Each guardrail entry also carries metadata.guardrail_name, read from LiteLLM_GuardrailsTable for guardrails created through the UI or API and falling back to the breakdown key for config.yaml guardrails, whose key is already their name

* fix(daily-activity): keep daily activity up when the guardrail breakdown query fails, and treat an empty model filter as none

A database error from the guardrail spend query now logs a warning and returns the response without guardrails, where it used to fail the whole aggregated response with a 500. An empty model= no longer drops the guardrail breakdown, matching how the rest of the endpoint already ignores it. The daily activity golden files and fixtures gain the empty guardrails key every breakdown now carries

* refactor(daily-activity): fold guardrail breakdown in a worker thread and drop docstrings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(daily-activity): report untracked guardrail units next to spend

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 15:35:08 -07:00
..
fixtures feat(daily-activity): per-guardrail spend breakdown in user, team and organization daily activity (#48702) 2026-10-28 15:35:08 -07:00
golden feat(daily-activity): per-guardrail spend breakdown in user, team and organization daily activity (#48702) 2026-10-28 15:35:08 -07:00
__init__.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
_background_responses.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
_daily_activity_fixtures.py feat(usage): add top users by spend and user activity views (#45310) 2026-10-10 12:21:43 -07:00
test_agent_spend.py test(integration): cover agents_a2a endpoint contracts (#44650) 2026-10-14 03:40:07 +00:00
test_autorouter_session_rollup.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_background_interaction_settlement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_background_response_billed_once.py test(responses): audit #47504 (#48787) 2026-10-28 03:50:24 -07:00
test_background_response_billing_proxies.py fix(proxy): report requested model in streamed Responses response.model (#46662) 2026-10-17 19:52:10 +00:00
test_background_response_poll_retirement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_baggage_tag_mappings.py test(proxy): audit #47358 (#48784) 2026-10-28 04:55:44 -07:00
test_baseline_accounting.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
test_batch_completion_accounting.py fix(logging): deliver completed batch success callbacks consistently (#47192) 2026-10-18 09:19:04 -07:00
test_batch_enqueued_tokens_redis_lua.py test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996) 2026-10-01 09:23:36 -07:00
test_batch_observability.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_batch_poll_starvation.py fix(batches): stop unbillable batches from blocking file deletes and new-batch polling (#48752) 2026-10-28 12:19:36 -07:00
test_budget_limit_envelopes.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_cache_activity.py fix(proxy): scope /global/activity/cache_hits analytics to the caller's own and permitted-team logs (#46210) 2026-10-14 10:01:21 +02:00
test_cache_and_quota.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_chaos_burst_spend_once.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_container_spend.py test(integration): cover videos_containers endpoint contracts (#44691) 2026-10-14 03:49:01 +00:00
test_daily_activity_aggregated_breakdowns.py feat(daily-activity): per-guardrail spend breakdown in user, team and organization daily activity (#48702) 2026-10-28 15:35:08 -07:00
test_daily_activity_key_alias_probes.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_daily_activity_key_metadata_query_timeout.py feat(ui): unify gateway and usage assistance as Aria Agent (#48082) 2026-10-23 20:24:19 +00:00
test_daily_activity_key_owner.py fix(proxy): recover session key owners from daily spend for usage attribution (#43642) 2026-09-29 15:36:09 -07:00
test_daily_activity_key_owner_faults.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_daily_activity_key_owner_traffic.py feat(ui): unify gateway and usage assistance as Aria Agent (#48082) 2026-10-23 20:24:19 +00:00
test_daily_activity_repository.py feat(usage): add top users by spend and user activity views (#45310) 2026-10-10 12:21:43 -07:00
test_daily_activity_routes.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_daily_rollup_retry.py fix(cost): price exact cost-map names before price-free capability rules, and fix SCIM and MCP integration tests on CircleCI (#47037) 2026-10-17 06:46:52 +00:00
test_daily_tag_spend_retention.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_disable_end_user_cost_tracking.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_disconnected_bedrock_messages_stream_billing.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_end_user_budget_enforcement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_end_user_spend_without_proxy_user.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_expired_ui_session_failure_rows.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_failed_dispatch_tokens.py test(spend): audit #47815 (#48774) 2026-10-28 03:04:11 -07:00
test_filtered_ledger.py test: add extension and browser integration contracts 2026-09-16 13:15:53 -07:00
test_gemini_tts_speech_billing.py test(integration): repair seven stale integration tests on main (#47600) 2026-10-20 11:21:04 -07:00
test_global_proxy_spend_counter.py fix(proxy)!: stop update_cache from resurrecting evicted auth cache entries (#46997) 2026-10-20 17:09:53 +00:00
test_global_spend_report.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_global_spend_reset.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_google_genai_native_spend.py test(integration): cover google_genai endpoint contracts (#44680) 2026-10-14 04:01:35 +00:00
test_image_generation_key_spend.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_key_budget_lockout.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_key_metadata_recovery_dropped_connection.py fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560) 2026-10-05 22:36:25 +00:00
test_key_metadata_recovery_probe_bounds.py fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948) 2026-10-01 14:12:22 -07:00
test_legacy_spend_logs_row_cap.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_messages_stream_usage_cost.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_model_router_selected_model.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_normalized_error_long_message.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_org_budget_cli_session_token.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_passthrough_budget_reservation.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_request_tags.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_stream_pricing_paths_spend.py fix(tests): read owned proxy rows from the owned proxy's database (#47333) 2026-10-20 10:07:06 -07:00
test_passthrough_upstream_model.py feat(passthrough): let upstreams report the model name via x-litellm-upstream-model (#48275) 2026-10-27 18:28:54 -07:00
test_prompt_caching_requests_pagination.py test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996) 2026-10-01 09:23:36 -07:00
test_ptu_share_attribution.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_redis_ttl_preserving_token_increment.py refactor(proxy): expose public names for private proxy helpers (#45170) 2026-10-07 18:59:27 -07:00
test_requester_ip_address_logging.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_reset_budget_leader_election.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_responses_bridge_spend_once.py fix(logging): log spend once for large non-streaming requests on the chat to Responses bridge (#44508) 2026-10-06 21:23:19 +00:00
test_responses_cache_write_itemization.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_responses_websocket_turn_rows.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_roi_branch_spend.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_router_cooldown_failure_row.py test(integration): read router cooldown failure rows from the proxy that served them (#47457) 2026-10-19 18:39:16 -07:00
test_scoped_spend_reports.py test(integration): cover budgets_spend endpoint contracts (#44736) 2026-10-14 06:58:22 +00:00
test_service_tier_stream_billing.py fix(streaming): keep the served service_tier on streamed chunks and spend rows (#42870) 2026-09-29 12:54:17 -07:00
test_session_total_spend.py test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694) 2026-09-23 09:51:56 -07:00
test_shutdown_flush.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_spend_attribution_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_spend_calculate.py fix(pricing): drop the unpublished cached rate from the Gemini Live preview entries (#42651) 2026-09-22 20:35:34 -07:00
test_spend_capture_rate_captured_spend.py test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996) 2026-10-01 09:23:36 -07:00
test_spend_log_caller_session.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_spend_log_read_scope.py test(integration): cover budgets_spend endpoint contracts (#44736) 2026-10-14 06:58:22 +00:00
test_spend_log_tool_payload_content.py fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) 2026-10-21 02:26:05 +00:00
test_spend_log_write_batching.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_spend_logs_metadata_fields.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_spend_rollup_accuracy.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_stream_alias_billing.py fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model (#44341) 2026-10-03 06:44:52 +00:00
test_streamed_usage_estimate.py test(spend): audit #47353 (#48785) 2026-10-28 03:50:50 -07:00
test_tag_budget_enforcement.py feat(proxy): map baggage values to budget tags (#47358) 2026-10-20 09:09:42 +00:00
test_team_budget_enforcement.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_team_daily_activity_aggregated.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_team_member_budget_alerts.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_member_spend.py fix(team): keep the team reset schedule when a temp budget increase is set on a member inheriting the team default (#48495) 2026-10-26 17:13:56 -07:00
test_team_member_spend_flush.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_team_member_total_spend_reconciliation.py fix(proxy): reconcile team member total_spend with current cycle spend (#48260) 2026-10-24 17:57:09 -07:00
test_user_budget_on_team_keys.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_vertex_image_regional_uplift.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_vertex_passthrough_regional_uplift_spend.py fix(tests): read owned proxy rows from the owned proxy's database (#47333) 2026-10-20 10:07:06 -07:00
test_vertex_regional_uplift_spend.py fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image (#44673) 2026-10-05 21:00:22 -07:00