scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-11 08:38:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 280
litellm/tests/integration/routing
History
Exact
Kerry Lu 917fd11966
fix(rate-limit): settle a fallback's tokens on the model group that served it (#51082) 
* fix(rate-limit): settle a fallback's tokens on the model group that served it

The v3 limiter pinned post-call TPM settlement to the model reserved at
admission, so when the router fell back to another model group the
requested model's counter took the usage and the serving model's counter
stayed empty. When the served model group differs from the admitted one,
charge the served group and refund the abandoned group's per-model
reservation. Same-model retries and alias reloads keep the admission target.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate-limit): wait for the primary refund and the fallback charge before reading counters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate-limit): drop unrelated formatting from the v3 limiter unit test file

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:36:55 -08:00
..
_prompt_cache_pin_support.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
_reasoning_effort_levels_support.py test(routing): audit #46601 (#48910) 2026-10-28 15:42:51 -07:00
either_role.json test(integration): add read-replica routing harness to the CircleCI integration suite (#42692) 2026-09-24 00:25:03 -07:00
test_advisor_failure_cooldown.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_anthropic_overloaded_cooldown_owned_proxy.py fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) 2026-11-03 17:37:09 -08:00
test_audit_any_sweep_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_auto_router_management.py fix!: retain unattributed daily activity across reports and exports (#48925) 2026-10-29 09:39:49 -07:00
test_azure_passthrough_spend_row.py fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) 2026-10-14 12:40:48 -07:00
test_base_model_context_window_owned_proxy.py fix(router): let an exact base_model widen a deployment's context window (#46197) 2026-10-16 06:37:40 +00:00
test_batch_mode_deployments_wire.py fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order (#50971) 2026-11-10 18:35:19 -08:00
test_complexity_router_codex_compaction.py fix(complexity_router): keep the held user_turn route for Codex compaction requests (#48627) 2026-10-27 13:33:32 -07:00
test_complexity_router_databricks_classifier.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_complexity_router_llm_classifier.py refactor(types): replace Any with proven types in 7 files (#43844) 2026-10-02 02:47:47 -07:00
test_declared_provider_bare_model_routes.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_deployment_mode_routing_wire.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_deployment_num_retries_generic_routes_wire.py fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) 2026-10-05 22:53:39 +00:00
test_encrypted_content_affinity_no_peer.py fix(router): serve Responses turns from a sibling when the encrypted content origin has no boundary peer (#43015) 2026-09-24 17:31:43 -07:00
test_end_user_region_routing.py fix(router): drop the deployment rpm filter from enable_pre_call_checks (#48873) 2026-10-28 12:13:56 -07:00
test_enforce_model_rate_limits_daily_limits.py feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits (#48875) 2026-10-28 15:52:44 -07:00
test_fallback_hop_foreign_encrypted_reasoning.py fix(router): keep encrypted reasoning when a fallback hops between same-key deployments with no api_base (#49396) 2026-11-02 05:47:04 -08:00
test_fallback_rate_limit_check.py fix(router): re-check and reserve per-model rate limits on fallback targets (#50994) 2026-11-10 16:36:10 -08:00
test_fallback_retry_header_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_fallback_settles_serving_model.py fix(rate-limit): settle a fallback's tokens on the model group that served it (#51082) 2026-11-10 18:36:55 -08:00
test_include_fallback_errors_wire.py fix(proxy): send the streamed Responses fallback metadata as a typed Responses event (#50804) 2026-11-10 21:29:32 +00:00
test_key_max_parallel_requests.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_key_router_settings_image_generation.py fix(proxy): apply key and team router_settings to /v1/images/generations (#49286) 2026-10-30 12:38:21 -07:00
test_key_tpm_reservation.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_least_busy_concurrent_decisions.py fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) 2026-11-05 00:30:12 +00:00
test_messages_stream_retry_budget_sources_owned_proxy.py fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) 2026-11-03 17:37:09 -08:00
test_model_capability_flags.py fix: keep model capability flags out of provider requests (#46647) 2026-10-15 14:30:10 +00:00
test_non_chat_model_endpoint_gate.py feat(router): opt in to non-chat deployment filtering (#46257) 2026-10-29 08:33:36 -07:00
test_observed_routing.py test: fix integration tests that fail on test order, races and intended changes (#49310) 2026-10-30 13:25:47 -07:00
test_otpm_default_reservation_fraction.py fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054) 2026-10-29 20:32:04 -07:00
test_payment_required_cooldown.py fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception (#48644) 2026-10-27 15:12:53 -07:00
test_payment_required_mapping.py fix(exceptions): map upstream 402 to PaymentRequiredError and cool down 402 deployments (#44879) 2026-10-06 20:35:13 -07:00
test_pdf_document_pre_call_checks_owned_proxy.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_per_model_rate_limit_fallbacks.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_per_model_rate_limit_fallbacks_lifecycle.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_pre_call_response_format_filter.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_priority_model_tpm_enforcement.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_priority_rate_limit_headers.py chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
test_priority_scheduler_queue_cleanup.py test(router): audit #47498 (#48799) 2026-10-28 04:36:50 -07:00
test_prompt_cache_pin_memory_routes.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
test_prompt_cache_pin_ttl.py test(routing): expect /v1/responses to pin the served deployment for an hour after #50321 (#50474) 2026-11-07 00:08:05 +00:00
test_prompt_cache_pin_ttl_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_prompt_cache_pin_with_message_logging_off.py fix(router): write the prompt caching pin when turn_off_message_logging is on (#50276) 2026-11-05 15:59:57 -08:00
test_prompt_caching_affinity_tool_call_marks.py fix(router): pin /v1/responses requests for prompt caching affinity (#50321) 2026-11-06 12:56:54 -08:00
test_ptu_shares.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_ptu_shares_regex_timing.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_reasoning_effort_levels_chaos.py feat(router)!: apply each deployment's reasoning_effort_levels on every provider and route (#46601) 2026-10-20 04:34:55 +00:00
test_reasoning_effort_levels_wire.py fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
test_redis_increment_with_floor.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_redis_recovery.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_responses_websocket_turn_admission.py fix(proxy): count native Responses WebSocket turn tokens toward key and tag TPM limits (#49257) 2026-10-31 02:53:27 -07:00
test_router_strategies_count_async_requests_once.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_silent_model_shadow_wire.py test(router): audit #41368 (#48794) 2026-10-28 05:14:59 -07:00
test_stale_cost_map_boot.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_tag_routing_prefix.py fix(router)!: only prefixed caller tags route once tag_routing_prefix is set (#47428) 2026-10-20 00:12:28 -07:00
test_team_tag_routing.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_team_wildcard_precedence.py test(routing): a team's wildcard serves its keys before a proxy-wide wildcard of the same pattern (#48501) 2026-10-26 17:54:32 -07:00
test_usage_based_routing_redis_reads.py test(integration): repair tests that merged changes left stale (#50782) 2026-11-10 01:08:35 +00:00
test_user_config_routing.py feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966) 2026-10-16 22:25:15 +00:00
test_video_poll_routing_wire.py fix(videos)!: keep model_id in proxy video ids so per-model keys resolve on status/content (#44614) 2026-10-20 05:42:59 +00:00
test_vllm_passthrough_spend_row.py fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) 2026-10-14 12:40:48 -07:00
test_vllm_passthrough_stream_relay.py fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) 2026-10-14 12:40:48 -07:00
test_vllm_passthrough_stream_usage_settings.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_web_search_deployment_filter_owned_proxy.py fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools (#50275) 2026-11-06 01:58:17 +00:00
test_wildcard_model_listing.py fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377) 2026-10-16 15:23:46 -07:00