mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-13 08:52:02 +00:00
abe3b11f7a
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| html_forms | ||
| __init__.py | ||
| test_admin_ui_utils.py | ||
| test_auth_cache_invalidation_pubsub.py | ||
| test_cache_aware_routing.py | ||
| test_cache_codec.py | ||
| test_callback_config_validation.py | ||
| test_callback_utils.py | ||
| test_check_batch_cost.py | ||
| test_check_responses_cost.py | ||
| test_codex_catalog_source.py | ||
| test_codex_model_catalog.py | ||
| test_codex_model_translation.py | ||
| test_config_sync_pubsub.py | ||
| test_credential_hydration.py | ||
| test_custom_openapi_spec.py | ||
| test_debug_utils.py | ||
| test_discoverable_model_filter.py | ||
| test_encrypt_decrypt_utils.py | ||
| test_error_body_call_id.py | ||
| test_expired_ui_session_key_cleanup_manager.py | ||
| test_fips.py | ||
| test_get_routes.py | ||
| test_http_parsing_utils.py | ||
| test_json_merge_patch.py | ||
| test_key_rotation_e2e.py | ||
| test_key_rotation_integration.py | ||
| test_key_rotation_lock.py | ||
| test_key_rotation_manager.py | ||
| test_load_config_utils.py | ||
| test_model_deprecation.py | ||
| test_model_listing_utils.py | ||
| test_openai_endpoint_utils.py | ||
| test_openai_error_payload.py | ||
| test_openapi_route.py | ||
| test_openapi_schema_compat.py | ||
| test_periodic_reload_schedule.py | ||
| test_proxy_encrypt_decrypt.py | ||
| test_rbac_utils.py | ||
| test_realtime_cache.py | ||
| test_registry_read_through.py | ||
| test_request_tag_mappings.py | ||
| test_reset_budget_job.py | ||
| test_resource_ownership.py | ||
| test_scheduled_job_stagger.py | ||
| test_semantic_text_index.py | ||
| test_slow_gc_logger.py | ||
| test_sse_keepalive.py | ||
| test_static_asset_utils.py | ||
| test_swagger_utils.py | ||
| test_timezone_utils.py | ||
| test_token_count_log.py | ||
| test_token_count_settings.py | ||
| test_upsert_budget_membership.py | ||
| test_user_api_key_cache.py | ||
| test_validation_error_body.py | ||