mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
cee72ffdd6
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits A native Responses WebSocket turn logs its raw events instead of a usage-carrying response object, so the v3 rate limiter read no usage and never charged the key or tag token window. Read the turn's usage from its events the way the cost calculator does, and charge the TPM-limited tags the socket was opened under. * fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage A later response.create turn on an open Responses WebSocket now gets a read-only check of every TPM window it counts toward (key, team, user, org, end user, team member, model-scoped limits and the socket's tags). A turn with no tokens left gets the 429 rate_limit_exceeded frame and the socket stays open. The check never reserves or increments a counter and leaves RPM, budget, PTU and project windows to their own paths. It runs on native and managed sockets. Managed turns carry the socket's call id in their metadata, so they charge the socket's TPM-limited tags. A failed native turn charges the partial usage its failure row already carries. A socket refused at open by a rate limit gets the same refusal frame and closes 1008, logged once without a traceback. /v1/realtime sessions now count toward TPM through the realtime usage combiner, and the v1 limiter reads socket usage through the same helper * fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with. * fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero * fix(proxy): stop charging a realtime session's whole usage into its close minute Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case. * test(proxy): check each per-turn TPM window on its own * fix(proxy): send a fail-closed per-turn socket refusal with its 503 status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _prompt_cache_pin_support.py | ||
| _reasoning_effort_levels_support.py | ||
| either_role.json | ||
| test_advisor_failure_cooldown.py | ||
| test_audit_any_sweep_chaos.py | ||
| test_auto_router_management.py | ||
| test_azure_passthrough_spend_row.py | ||
| test_base_model_context_window_owned_proxy.py | ||
| test_batch_mode_deployments_wire.py | ||
| test_complexity_router_codex_compaction.py | ||
| test_complexity_router_databricks_classifier.py | ||
| test_complexity_router_llm_classifier.py | ||
| test_declared_provider_bare_model_routes.py | ||
| test_deployment_mode_routing_wire.py | ||
| test_deployment_num_retries_generic_routes_wire.py | ||
| test_encrypted_content_affinity_no_peer.py | ||
| test_end_user_region_routing.py | ||
| test_enforce_model_rate_limits_daily_limits.py | ||
| test_fallback_hop_foreign_encrypted_reasoning.py | ||
| test_fallback_retry_header_contracts.py | ||
| test_include_fallback_errors_wire.py | ||
| test_key_max_parallel_requests.py | ||
| test_key_router_settings_image_generation.py | ||
| test_key_tpm_reservation.py | ||
| test_messages_stream_retry_budget_sources_owned_proxy.py | ||
| test_model_capability_flags.py | ||
| test_non_chat_model_endpoint_gate.py | ||
| test_observed_routing.py | ||
| test_otpm_default_reservation_fraction.py | ||
| test_payment_required_cooldown.py | ||
| test_payment_required_mapping.py | ||
| test_pdf_document_pre_call_checks_owned_proxy.py | ||
| test_per_model_rate_limit_fallbacks.py | ||
| test_per_model_rate_limit_fallbacks_lifecycle.py | ||
| test_pre_call_response_format_filter.py | ||
| test_priority_model_tpm_enforcement.py | ||
| test_priority_rate_limit_headers.py | ||
| test_priority_scheduler_queue_cleanup.py | ||
| test_prompt_cache_pin_memory_routes.py | ||
| test_prompt_cache_pin_ttl.py | ||
| test_prompt_cache_pin_ttl_chaos.py | ||
| test_prompt_caching_affinity_tool_call_marks.py | ||
| test_ptu_shares.py | ||
| test_ptu_shares_regex_timing.py | ||
| test_reasoning_effort_levels_chaos.py | ||
| test_reasoning_effort_levels_wire.py | ||
| test_redis_increment_with_floor.py | ||
| test_redis_recovery.py | ||
| test_responses_websocket_turn_admission.py | ||
| test_router_strategies_count_async_requests_once.py | ||
| test_silent_model_shadow_wire.py | ||
| test_stale_cost_map_boot.py | ||
| test_tag_routing_prefix.py | ||
| test_team_tag_routing.py | ||
| test_team_wildcard_precedence.py | ||
| test_usage_based_routing_redis_reads.py | ||
| test_user_config_routing.py | ||
| test_video_poll_routing_wire.py | ||
| test_vllm_passthrough_spend_row.py | ||
| test_vllm_passthrough_stream_relay.py | ||
| test_vllm_passthrough_stream_usage_settings.py | ||
| test_wildcard_model_listing.py | ||