mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
a11a93f44a
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: move tests/test_litellm integrations and secret_managers into tests/unit Rename-only. Mirrors the old paths, including the directory conftests and the prompt and JSON fixtures. Follow-up commits prune and wire them. * test: prune and repoint the moved integrations tests Deletes the 7 audited tests a stronger test in the same tree already covers, imports the TLS sink helpers from their new conftest path, and restores os.environ after each integrations test. Some presets write OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the legacy tree's test ordering that header leaked into the AgentOps tests. * ci: run the moved integrations tests under their legacy flag The integrations GHA shard and a new CircleCI job run the integrations unit selection. secret_managers joins the misc selection. * docs: point integrations and secret_managers references at tests/unit * test: make the moved integrations directories packages * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit Rename-only. Mirrors the old paths, including fixtures, the stubtest config and the native-route wheel script. Two files that collide with existing unit files are merged in a follow-up commit. * test: merge, prune and repoint the moved core, routing, responses, caching and rust_bridge tests Merges the two files that collided with existing unit files, folding the legacy extra case into test_is_chat_completion_cached_dict, and deletes the 9 audited tests a stronger test in the same file already covers. Keeps what needs the network in tests/test_litellm: test_tokenizers pulls a tokenizer from the Hugging Face hub, and the gpt2 and r50k_base tokenizer cases download their BPE files. The unit core_utils conftest points TIKTOKEN_CACHE_DIR at litellm's bundled encodings so the rest never depend on import order to stay offline, and FakeSecretVault moves to a shared module so both trees can build it. * ci: run the moved core, routing, responses, caching and rust_bridge tests under their flags core_utils gets a core-utils flag and CircleCI job, and its GHA shard keeps the legacy path for the retained network tests. router_utils and router_strategy join enterprise-routing, responses joins responses-caching-types (minus responses/mcp, which mcp-integration owns), caching joins caching-local and rust_bridge joins misc. The redis-compat, test-rust, stubtest and merge-smoke paths follow the move. * docs: point the Rust crate references at tests/unit * test: make the moved core, routing and rust_bridge directories packages * test: keep the no-loop DualCache batch_get_cache regression test It runs the sync path outside any event loop, which the inside-loop test cannot, so a change that picks the Redis client by loop state would only show up there. * test: keep the job's UNIT_FLAG out of the shard-script tests * fix(url_utils): block 192.0.0.0/24 on every Python patch release * test: move the new budget limiter tests into tests/unit/router_strategy * test: move the new sentry scrubbing tests into tests/unit/litellm_core_utils * test: move the new zerobus tests into tests/unit/integrations * test: make tests/unit/integrations/zerobus a package * test: load litellm's own tiktoken cache setup once instead of resetting it per test --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1401 lines
55 KiB
Python
1401 lines
55 KiB
Python
import json
|
|
from datetime import datetime, timedelta
|
|
from typing import Final, NoReturn
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
import litellm
|
|
from litellm.litellm_core_utils import get_llm_provider_logic
|
|
from litellm.router_utils.cooldown_handlers import mark_advisor_orchestration_failure
|
|
from litellm.router_utils.fallback_event_handlers import (
|
|
AttemptedFallbackTargets,
|
|
_trigger_cooldown_for_failed_deployment,
|
|
fallback_attempt_key,
|
|
clear_pre_routing_selection,
|
|
get_fallback_model_group,
|
|
get_pre_routing_selection,
|
|
record_pre_routing_selection,
|
|
run_async_fallback,
|
|
)
|
|
|
|
|
|
class StreamingWrapper:
|
|
def __init__(self):
|
|
self._hidden_params = {"additional_headers": {}}
|
|
|
|
|
|
class FakeRouter:
|
|
fallback_access_check = None
|
|
fallback_budget_check = None
|
|
|
|
def log_retry(self, kwargs, e):
|
|
return kwargs
|
|
|
|
async def async_function_with_fallbacks(self, *args, **kwargs):
|
|
return StreamingWrapper()
|
|
|
|
|
|
class AlwaysFailRouter:
|
|
fallback_access_check = None
|
|
fallback_budget_check = None
|
|
|
|
def log_retry(self, kwargs, e):
|
|
return kwargs
|
|
|
|
async def async_function_with_fallbacks(self, *args, **kwargs):
|
|
raise RuntimeError("fallback model also failed")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_adds_errors_when_opted_in():
|
|
response = await run_async_fallback(
|
|
litellm_router=FakeRouter(),
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("upstream limited request"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
include_fallback_errors=True,
|
|
)
|
|
|
|
additional_headers = response._hidden_params["additional_headers"]
|
|
assert additional_headers["x-litellm-attempted-fallbacks"] == 1
|
|
assert json.loads(additional_headers["x-litellm-fallback-errors"]) == [
|
|
{
|
|
"message": "upstream limited request",
|
|
"type": "RuntimeError",
|
|
"param": None,
|
|
"code": None,
|
|
}
|
|
]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_omits_errors_without_opt_in():
|
|
response = await run_async_fallback(
|
|
litellm_router=FakeRouter(),
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("upstream limited request"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
additional_headers = response._hidden_params["additional_headers"]
|
|
assert additional_headers["x-litellm-attempted-fallbacks"] == 1
|
|
assert "x-litellm-fallback-errors" not in additional_headers
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_raises_when_all_fallbacks_fail():
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=AlwaysFailRouter(),
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original request failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
include_fallback_errors=True,
|
|
)
|
|
|
|
|
|
class RecordingRouter:
|
|
fallback_access_check = None
|
|
fallback_budget_check = None
|
|
|
|
def __init__(self):
|
|
self.received_kwargs = None
|
|
|
|
def log_retry(self, kwargs, e):
|
|
return kwargs
|
|
|
|
async def async_function_with_fallbacks(self, *args, **kwargs):
|
|
self.received_kwargs = kwargs
|
|
return StreamingWrapper()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_forwards_include_fallback_errors_to_nested_call():
|
|
"""A nested fallback (multi-hop) must keep collecting errors, so the opt-in
|
|
flag has to reach the nested async_function_with_fallbacks call."""
|
|
router = RecordingRouter()
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("upstream limited request"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
include_fallback_errors=True,
|
|
)
|
|
|
|
assert router.received_kwargs.get("include_fallback_errors") is True
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_does_not_forward_flag_without_opt_in():
|
|
router = RecordingRouter()
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("upstream limited request"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
assert "include_fallback_errors" not in router.received_kwargs
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_skips_original_model_group():
|
|
response = await run_async_fallback(
|
|
litellm_router=FakeRouter(),
|
|
fallback_model_group=["primary-model", "fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
assert response._hidden_params["additional_headers"]["x-litellm-attempted-fallbacks"] == 1
|
|
|
|
|
|
class AttemptRecordingRouter:
|
|
fallback_access_check = None
|
|
fallback_budget_check = None
|
|
|
|
def __init__(self):
|
|
self.attempted_model_groups = []
|
|
self.received_kwargs = None
|
|
|
|
def log_retry(self, kwargs, e):
|
|
return kwargs
|
|
|
|
async def async_function_with_fallbacks(self, *args, **kwargs):
|
|
self.attempted_model_groups.append(kwargs.get("model"))
|
|
self.received_kwargs = kwargs
|
|
return StreamingWrapper()
|
|
|
|
|
|
async def _acreate_batch(*args, **kwargs):
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def _acreate_file(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def _acancel_batch(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def _acompletion(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def _ageneric_api_call_with_fallbacks_helper(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def acreate_fine_tuning_job(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def aretrieve_fine_tuning_job(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
async def afile_content(*args: object, **kwargs: object) -> NoReturn:
|
|
raise AssertionError("only used for its __name__")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_keeps_uploaded_file_requests_in_their_model_group():
|
|
"""An input_file_id only exists under the credentials of the group it was uploaded
|
|
to, so a cross-group fallback can only fail with the wrong provider's error."""
|
|
router = AttemptRecordingRouter()
|
|
owning_provider_error = RuntimeError("openai connection error")
|
|
|
|
with pytest.raises(RuntimeError, match="openai connection error"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["azure-group"],
|
|
original_model_group="openai-group",
|
|
original_exception=owning_provider_error,
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
input_file_id="file-owned-by-openai",
|
|
original_function=_acreate_batch,
|
|
)
|
|
|
|
assert router.attempted_model_groups == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_keeps_fine_tuning_requests_in_their_model_group():
|
|
router = AttemptRecordingRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="openai connection error"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["azure-group"],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("openai connection error"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
training_file="file-owned-by-openai",
|
|
original_function=_ageneric_api_call_with_fallbacks_helper,
|
|
original_generic_function=acreate_fine_tuning_job,
|
|
)
|
|
|
|
assert router.attempted_model_groups == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_allows_same_model_group_retry_for_uploaded_file_requests():
|
|
"""Order-based fallbacks stay inside the owning group, so they must still run."""
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[{"model": "openai-group", "_target_order": 2}],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("first deployment failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
input_file_id="file-owned-by-openai",
|
|
original_function=_acreate_batch,
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["openai-group"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_keeps_file_creation_in_its_model_group():
|
|
"""A file created for batches lands in the account of the deployment that stored it,
|
|
and its id is only usable against the model group the caller named. A cross-group
|
|
fallback silently stores the file with the wrong provider."""
|
|
router = AttemptRecordingRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="azure connection error"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["openai-group"],
|
|
original_model_group="azure-group",
|
|
original_exception=RuntimeError("azure connection error"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="azure-group",
|
|
original_function=_acreate_file,
|
|
)
|
|
|
|
assert router.attempted_model_groups == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_allows_same_model_group_retry_for_file_creation():
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[{"model": "azure-group", "_target_order": 2}],
|
|
original_model_group="azure-group",
|
|
original_exception=RuntimeError("first deployment failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="azure-group",
|
|
original_function=_acreate_file,
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["azure-group"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_still_crosses_model_groups_without_an_uploaded_file():
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["azure-group"],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("openai connection error"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["azure-group"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
("resource_key", "handler_kwargs"),
|
|
[
|
|
("batch_id", {"original_function": _acancel_batch}),
|
|
(
|
|
"file_id",
|
|
{
|
|
"original_function": _ageneric_api_call_with_fallbacks_helper,
|
|
"original_generic_function": afile_content,
|
|
},
|
|
),
|
|
(
|
|
"fine_tuning_job_id",
|
|
{
|
|
"original_function": _ageneric_api_call_with_fallbacks_helper,
|
|
"original_generic_function": aretrieve_fine_tuning_job,
|
|
},
|
|
),
|
|
],
|
|
)
|
|
async def test_run_async_fallback_keeps_provider_scoped_ids_in_their_model_group(
|
|
resource_key: str, handler_kwargs: dict
|
|
):
|
|
"""A batch, file, or fine-tuning job id only exists under the credentials of the group
|
|
that issued it, so a cross-group fallback asks a provider about an id it never saw.
|
|
Generic API calls carry the real handler in original_generic_function, so the pin
|
|
must recognize it there too."""
|
|
router = AttemptRecordingRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="openai connection error"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["azure-group"],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("openai connection error"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
**{resource_key: "owned-by-openai"},
|
|
**handler_kwargs,
|
|
)
|
|
|
|
assert router.attempted_model_groups == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize("resource_key", ["batch_id", "file_id", "fine_tuning_job_id"])
|
|
async def test_run_async_fallback_ignores_stray_resource_ids_on_completion_calls(resource_key: str):
|
|
"""A caller-supplied top-level field like file_id on a chat completion is application
|
|
data, never a provider resource reference, so it must not cost the request its
|
|
cross-group fallbacks."""
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["azure-group"],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("openai connection error"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
original_function=_acompletion,
|
|
**{resource_key: "caller-app-data"},
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["azure-group"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_allows_same_model_group_retry_for_batch_cancel():
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[{"model": "openai-group", "_target_order": 2}],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("first deployment failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
batch_id="owned-by-openai",
|
|
original_function=_acancel_batch,
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["openai-group"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_handles_explicitly_none_metadata():
|
|
"""/v1/batches always sets `metadata`, and sets it to None when the caller sent
|
|
none, so setdefault() on it hands back None instead of a dict."""
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["azure-group"],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("openai connection error"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
metadata=None,
|
|
)
|
|
|
|
assert router.received_kwargs["metadata"] == {
|
|
"model_group": "azure-group",
|
|
"attempted_fallbacks": 1,
|
|
"original_model_group": "openai-group",
|
|
}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_records_batch_model_group_outside_provider_metadata():
|
|
"""`metadata` on a batch request is forwarded to the provider and stored on the
|
|
batch, so the router's own model_group belongs in litellm_metadata."""
|
|
router = AttemptRecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[{"model": "openai-group", "_target_order": 2}],
|
|
original_model_group="openai-group",
|
|
original_exception=RuntimeError("first deployment failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="openai-group",
|
|
input_file_id="file-owned-by-openai",
|
|
metadata={"caller": "nightly-job"},
|
|
litellm_metadata={"model_group": "openai-group"},
|
|
original_function=_acreate_batch,
|
|
)
|
|
|
|
assert router.received_kwargs["metadata"] == {"caller": "nightly-job"}
|
|
assert router.received_kwargs["litellm_metadata"]["model_group"] == "openai-group"
|
|
|
|
|
|
class AccessCheckedRouter(AttemptRecordingRouter):
|
|
def __init__(self, allowed_models: frozenset[str]):
|
|
super().__init__()
|
|
self.allowed_models = allowed_models
|
|
self.access_checks = []
|
|
|
|
fallback_budget_check = None
|
|
|
|
async def fallback_access_check(self, *, model, request_kwargs, llm_router):
|
|
self.access_checks.append((model, request_kwargs["metadata"]["user_api_key"], llm_router is self))
|
|
return model in self.allowed_models
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_skips_targets_the_access_check_rejects():
|
|
router = AccessCheckedRouter(allowed_models=frozenset({"allowed-model"}))
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[
|
|
{"model": "secret-model", "messages": [{"role": "user", "content": "hi"}]},
|
|
"allowed-model",
|
|
],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("primary failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="primary-model",
|
|
metadata={"user_api_key": "hashed"},
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["allowed-model"]
|
|
assert router.access_checks == [
|
|
("secret-model", "hashed", True),
|
|
("allowed-model", "hashed", True),
|
|
]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_raises_original_error_when_no_target_is_authorized():
|
|
router = AccessCheckedRouter(allowed_models=frozenset())
|
|
|
|
with pytest.raises(RuntimeError, match="primary failed"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["secret-model", "other-secret-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("primary failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="primary-model",
|
|
metadata={"user_api_key": "hashed"},
|
|
)
|
|
|
|
assert router.attempted_model_groups == []
|
|
assert [model for model, _, _ in router.access_checks] == ["secret-model", "other-secret-model"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_does_not_consult_access_check_for_same_model_group_retries():
|
|
router = AccessCheckedRouter(allowed_models=frozenset())
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[{"model": "primary-model", "_target_order": 2}],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("first order level failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="primary-model",
|
|
metadata={"user_api_key": "hashed"},
|
|
)
|
|
|
|
assert router.attempted_model_groups == ["primary-model"]
|
|
assert router.access_checks == []
|
|
|
|
|
|
class RecordingFailRouter:
|
|
fallback_access_check = None
|
|
fallback_budget_check = None
|
|
|
|
def __init__(self):
|
|
self.attempted_models = []
|
|
|
|
def log_retry(self, kwargs, e):
|
|
return kwargs
|
|
|
|
async def async_function_with_fallbacks(self, *args, **kwargs):
|
|
self.attempted_models.append(kwargs.get("model"))
|
|
raise RuntimeError("fallback model also failed")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_skips_model_group_already_attempted():
|
|
"""A fallback graph that loops back on itself must not re-attempt a model group that
|
|
already failed for this request. Every group in a cycle fails identically, so
|
|
revisiting one multiplies the work and the error output without any chance of
|
|
succeeding."""
|
|
router = RecordingFailRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="original failed"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["already-attempted"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
attempted_targets=AttemptedFallbackTargets(frozenset({"already-attempted"})),
|
|
)
|
|
|
|
assert router.attempted_models == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_attempts_a_repeated_target_once():
|
|
router = RecordingFailRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["fallback-model", "fallback-model", "other-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=5,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
assert router.attempted_models == ["fallback-model", "other-model"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_forwards_attempted_model_groups_to_nested_call():
|
|
"""The nested call is where the next hop of the walk decides what to skip, so the
|
|
accumulated set has to reach it, carrying both the group that just failed and the
|
|
target being attempted."""
|
|
router = RecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
attempted_targets=AttemptedFallbackTargets(frozenset({"earlier-model"})),
|
|
)
|
|
|
|
assert router.received_kwargs["attempted_targets"].keys == frozenset(
|
|
{"earlier-model", "primary-model", "fallback-model"}
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_can_target_the_requested_group_when_a_pre_router_replaced_it():
|
|
"""The requested group was never called when a pre-router selected a tier, so a
|
|
tier fallback may legitimately target that originally requested group."""
|
|
router = RecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["requested-model"],
|
|
original_model_group="requested-model",
|
|
original_exception=RuntimeError("selected tier failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
model="requested-model",
|
|
metadata={"pre_routing_selected_model": "selected-tier"},
|
|
)
|
|
|
|
assert router.received_kwargs["model"] == "requested-model"
|
|
assert router.received_kwargs["attempted_targets"].keys == frozenset({"selected-tier", "requested-model"})
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"entry",
|
|
[
|
|
{"model": "primary-model", "_target_order": 2},
|
|
{"model": "primary-model", "_excluded_deployment_ids": ["dep-1"]},
|
|
],
|
|
)
|
|
async def test_run_async_fallback_still_retargets_the_same_group_via_dict_entry(entry):
|
|
"""Order-based fallback and weighted intra-group failover both re-target the group that
|
|
just failed, selecting a different set of deployments inside it. Those entries are dicts
|
|
rather than plain names and must survive a guard that skips already-attempted names."""
|
|
router = RecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[entry],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
attempted_targets=AttemptedFallbackTargets(frozenset({"primary-model"})),
|
|
)
|
|
|
|
assert router.received_kwargs["model"] == "primary-model"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_skips_a_repeated_dict_target():
|
|
"""A client-side fallback list names its targets with dicts, and that list is re-walked
|
|
at every level of the recursion, so an entry that carries no request override has to be
|
|
recognised as the same attempt as the bare name."""
|
|
router = RecordingFailRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="original failed"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[{"model": "already-attempted"}],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
attempted_targets=AttemptedFallbackTargets(frozenset({"already-attempted"})),
|
|
)
|
|
|
|
assert router.attempted_models == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_attempts_a_repeated_dict_target_once():
|
|
router = RecordingFailRouter()
|
|
entry = {"model": "fallback-model", "messages": [{"role": "user", "content": "shorter"}]}
|
|
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[entry, entry, {"model": "other-model"}],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=5,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
assert router.attempted_models == ["fallback-model", "other-model"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_keeps_a_request_override_distinct_from_the_bare_name():
|
|
"""The documented use of the client-side form is to retry a group with different request
|
|
params, so an entry carrying an override must survive even when the bare name of that
|
|
same group has already been attempted."""
|
|
router = RecordingFailRouter()
|
|
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=[
|
|
{"model": "already-attempted", "messages": [{"role": "user", "content": "shorter"}]}
|
|
],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
attempted_targets=AttemptedFallbackTargets(frozenset({"already-attempted"})),
|
|
)
|
|
|
|
assert router.attempted_models == ["already-attempted"]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"target, expected",
|
|
[
|
|
("group-a", "group-a"),
|
|
({"model": "group-a"}, "group-a"),
|
|
(None, None),
|
|
(["group-a"], None),
|
|
],
|
|
)
|
|
def test_fallback_attempt_key_identity(target, expected):
|
|
"""A bare name and a `{"model": name}` entry are the same attempt. A shape with no
|
|
usable identity returns None and is never skipped, so an unrecognised entry keeps
|
|
today's behaviour rather than being silently dropped."""
|
|
assert fallback_attempt_key(target) == expected
|
|
|
|
|
|
def test_fallback_attempt_key_gives_a_param_only_entry_its_own_identity():
|
|
"""An entry with no `model` re-targets the group currently being attempted with
|
|
different request params, so it is a distinct attempt and still needs an identity."""
|
|
key = fallback_attempt_key({"messages": [{"role": "user", "content": "shorter"}]})
|
|
|
|
assert key is not None
|
|
assert key != fallback_attempt_key({"messages": [{"role": "user", "content": "other"}]})
|
|
|
|
|
|
def test_fallback_attempt_key_separates_overrides_from_the_bare_name():
|
|
bare = fallback_attempt_key("group-a")
|
|
override = fallback_attempt_key({"model": "group-a", "messages": [{"role": "user", "content": "x"}]})
|
|
other_override = fallback_attempt_key({"model": "group-a", "messages": [{"role": "user", "content": "y"}]})
|
|
order_retarget = fallback_attempt_key({"model": "group-a", "_target_order": 2})
|
|
|
|
assert len({bare, override, other_override, order_retarget}) == 4
|
|
|
|
|
|
def test_fallback_attempt_key_is_stable_across_key_order():
|
|
assert fallback_attempt_key({"model": "group-a", "_target_order": 2}) == fallback_attempt_key(
|
|
{"_target_order": 2, "model": "group-a"}
|
|
)
|
|
|
|
|
|
def test_get_fallback_model_group_does_not_mutate_fallbacks():
|
|
"""A string fallback must be resolved without mutating the caller's
|
|
fallbacks list, which is the live router config shared across requests."""
|
|
fallbacks = [{"gpt-3.5-turbo": ["claude-3-haiku"]}, "gpt-4o-mini"]
|
|
|
|
fallback_model_group, _ = get_fallback_model_group(fallbacks=fallbacks, model_group="unmatched-model")
|
|
|
|
assert fallback_model_group == ["gpt-4o-mini"]
|
|
assert fallbacks == [{"gpt-3.5-turbo": ["claude-3-haiku"]}, "gpt-4o-mini"]
|
|
|
|
|
|
class TestTriggerCooldownForFailedDeployment:
|
|
def test_calls_set_cooldown_deployments_with_stamped_deployment_id(self):
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.RateLimitError("Rate limit", "openai", "gpt-4")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
mock_set_cooldown.assert_called_once()
|
|
call_kwargs = mock_set_cooldown.call_args[1]
|
|
assert call_kwargs["deployment"] == "fallback-deployment"
|
|
assert call_kwargs["original_exception"] is exc
|
|
|
|
def test_does_not_trust_caller_supplied_metadata_bucket(self):
|
|
"""A metadata bucket can't reliably be told apart from a caller-supplied
|
|
one without knowing this call's function_name, so a client with
|
|
permission to set metadata must not be able to get an arbitrary
|
|
deployment cooled down by forging a deployment_model_name marker."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.RateLimitError("Rate limit", "openai", "gpt-4")
|
|
kwargs = {
|
|
"metadata": {
|
|
"model_info": {"id": "attacker-chosen-deployment"},
|
|
"deployment_model_name": "gpt-4",
|
|
}
|
|
}
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs=kwargs, exception=exc)
|
|
|
|
mock_set_cooldown.assert_not_called()
|
|
|
|
def test_increments_failure_counter_before_cooldown_check(self):
|
|
"""The fallback path must feed the same per-minute failure counter the
|
|
primary path uses, or repeated fallback failures never accumulate
|
|
toward the default percent-fail-rate cooldown threshold."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.RateLimitError("Rate limit", "openai", "gpt-4")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with (
|
|
patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown,
|
|
patch(
|
|
"litellm.router_utils.fallback_event_handlers.increment_deployment_failures_for_current_minute"
|
|
) as mock_increment,
|
|
):
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
mock_increment.assert_called_once_with(
|
|
litellm_router_instance=mock_router, deployment_id="fallback-deployment"
|
|
)
|
|
mock_set_cooldown.assert_called_once()
|
|
|
|
def test_no_op_when_deployment_id_missing(self):
|
|
mock_router = MagicMock()
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(
|
|
litellm_router=mock_router, kwargs={}, exception=RuntimeError("no metadata")
|
|
)
|
|
|
|
mock_set_cooldown.assert_not_called()
|
|
|
|
def test_skipped_for_advisor_orchestration_failure(self):
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.RateLimitError("Rate limit", "openai", "gpt-4")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
mark_advisor_orchestration_failure(exc)
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
mock_set_cooldown.assert_not_called()
|
|
|
|
def test_uses_deployment_litellm_params_cooldown_time_override(self):
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 300.0
|
|
mock_router.get_model_info.return_value = {"litellm_params": {"cooldown_time": 30.0}}
|
|
|
|
exc = litellm.RateLimitError("Rate limit", "openai", "gpt-4")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
call_kwargs = mock_set_cooldown.call_args[1]
|
|
assert call_kwargs["time_to_cooldown"] == 30.0
|
|
|
|
def test_uses_response_header_when_no_deployment_config(self):
|
|
"""Precedence must match Router.deployment_callback_on_failure's primary
|
|
path: deployment config, then the response's Retry-After header, then the
|
|
router default."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = {"litellm_params": {}}
|
|
|
|
exc = RuntimeError("upstream error")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
exc.litellm_response_headers = httpx.Headers({"retry-after": "45"})
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
call_kwargs = mock_set_cooldown.call_args[1]
|
|
assert call_kwargs["time_to_cooldown"] == 45
|
|
|
|
def test_silently_catches_exceptions(self):
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = RuntimeError("upstream error")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with patch(
|
|
"litellm.router_utils.fallback_event_handlers._set_cooldown_deployments",
|
|
side_effect=RuntimeError("cooldown error"),
|
|
):
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
def test_skips_request_scoped_404_on_generic_api_call(self):
|
|
"""A generic API call (files/batches/threads/rerank/...) forwards a caller-supplied
|
|
resource id, so a 404 there means "that id doesn't exist", not "this deployment is
|
|
unhealthy". Without this guard, a single bad id would 404 every deployment in the
|
|
fallback chain and cool all of them down from one request."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.NotFoundError("not found", "openai", "gpt-4")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with (
|
|
patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown,
|
|
patch(
|
|
"litellm.router_utils.fallback_event_handlers.increment_deployment_failures_for_current_minute"
|
|
) as mock_increment,
|
|
):
|
|
_trigger_cooldown_for_failed_deployment(
|
|
litellm_router=mock_router,
|
|
kwargs={"original_generic_function": MagicMock()},
|
|
exception=exc,
|
|
)
|
|
|
|
mock_set_cooldown.assert_not_called()
|
|
mock_increment.assert_not_called()
|
|
|
|
def test_still_cools_down_404_outside_generic_api_call(self):
|
|
"""The request-scoped-404 guard is scoped to generic API calls only: a 404 on a
|
|
regular completion fallback (no original_generic_function in kwargs) must still
|
|
cool down the deployment as before."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.NotFoundError("not found", "openai", "gpt-4")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
mock_set_cooldown.assert_called_once()
|
|
|
|
def test_skips_client_side_timeout_408(self):
|
|
"""The proxy's x-litellm-timeout header lets a caller set an arbitrarily short
|
|
timeout, which litellm.Timeout reports as status 408 regardless of the
|
|
deployment's actual health. Without this guard, a caller could force a 408 on
|
|
every deployment in the fallback chain from a single request.
|
|
|
|
The failure logger never stamps end_time for a fallback hop (has_logged_async_failure
|
|
is already set), so model_call_details still carries the previous hop's end_time, which
|
|
predates this hop's api_call_start_time. The guard must not trust it."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.Timeout(message="timeout", model="gpt-4", llm_provider="openai")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with (
|
|
patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown,
|
|
patch(
|
|
"litellm.router_utils.fallback_event_handlers.increment_deployment_failures_for_current_minute"
|
|
) as mock_increment,
|
|
):
|
|
_trigger_cooldown_for_failed_deployment(
|
|
litellm_router=mock_router,
|
|
kwargs={"client_side_timeout": True},
|
|
exception=exc,
|
|
model_call_details={
|
|
"litellm_params": {"client_side_timeout": True, "timeout": 0.5},
|
|
"api_call_start_time": datetime.now() - timedelta(seconds=1),
|
|
"end_time": datetime.now() - timedelta(seconds=5),
|
|
},
|
|
)
|
|
|
|
mock_set_cooldown.assert_not_called()
|
|
mock_increment.assert_not_called()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_still_cools_down_provider_408_before_caller_deadline(self):
|
|
"""client_side_timeout only records that the caller configured a timeout. A 408
|
|
that comes back before that deadline was raised by the provider itself, so it is
|
|
a real health signal and must still cool the deployment down."""
|
|
from litellm.router_utils.router_callbacks.track_deployment_metrics import (
|
|
get_deployment_failures_for_current_minute,
|
|
)
|
|
|
|
router = litellm.Router(
|
|
model_list=[
|
|
{
|
|
"model_name": "fallback-model",
|
|
"litellm_params": {"model": "openai/gpt-5.6", "api_key": "sk-fake"},
|
|
"model_info": {"id": "fallback-deployment"},
|
|
}
|
|
],
|
|
allowed_fails=0,
|
|
cooldown_time=60,
|
|
num_retries=0,
|
|
)
|
|
exc = litellm.Timeout(message="timeout", model="gpt-5.6", llm_provider="openai")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
started = datetime.now()
|
|
|
|
_trigger_cooldown_for_failed_deployment(
|
|
litellm_router=router,
|
|
kwargs={"client_side_timeout": True},
|
|
exception=exc,
|
|
model_call_details={
|
|
"litellm_params": {"client_side_timeout": True, "timeout": 30},
|
|
"api_call_start_time": started,
|
|
"end_time": started + timedelta(seconds=1),
|
|
},
|
|
)
|
|
|
|
assert (
|
|
get_deployment_failures_for_current_minute(
|
|
litellm_router_instance=router, deployment_id="fallback-deployment"
|
|
)
|
|
== 1
|
|
)
|
|
active = router.cooldown_cache.get_active_cooldowns(model_ids=["fallback-deployment"], parent_otel_span=None)
|
|
assert [entry[0] for entry in active] == ["fallback-deployment"]
|
|
|
|
def test_still_cools_down_408_without_client_side_timeout_flag(self):
|
|
"""The client-side-timeout guard is scoped to caller-supplied timeouts only: a
|
|
408 that did not come from x-litellm-timeout (no client_side_timeout in kwargs)
|
|
must still cool down the deployment as before."""
|
|
mock_router = MagicMock()
|
|
mock_router.cooldown_time = 60.0
|
|
mock_router.get_model_info.return_value = None
|
|
|
|
exc = litellm.Timeout(message="timeout", model="gpt-4", llm_provider="openai")
|
|
exc.failed_deployment_id = "fallback-deployment"
|
|
|
|
with patch("litellm.router_utils.fallback_event_handlers._set_cooldown_deployments") as mock_set_cooldown:
|
|
_trigger_cooldown_for_failed_deployment(litellm_router=mock_router, kwargs={}, exception=exc)
|
|
|
|
mock_set_cooldown.assert_called_once()
|
|
|
|
|
|
class TestRunAsyncFallbackTriggersCooldown:
|
|
class RouterWithLoggingKwarg:
|
|
fallback_access_check = None
|
|
fallback_budget_check = None
|
|
|
|
def __init__(self):
|
|
self.cooldown_time = 60.0
|
|
|
|
def log_retry(self, kwargs, e):
|
|
return kwargs
|
|
|
|
def get_model_info(self, id):
|
|
return None
|
|
|
|
async def async_function_with_fallbacks(self, *args, **kwargs):
|
|
raise RuntimeError("fallback model also failed")
|
|
|
|
def _logging_obj(self, has_logged_async_failure: bool) -> MagicMock:
|
|
logging_obj = MagicMock()
|
|
logging_obj.model_call_details = {"has_logged_async_failure": has_logged_async_failure}
|
|
return logging_obj
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_triggers_cooldown_when_has_logged_async_failure_is_true(self):
|
|
with patch(
|
|
"litellm.router_utils.fallback_event_handlers._trigger_cooldown_for_failed_deployment"
|
|
) as mock_trigger:
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=self.RouterWithLoggingKwarg(),
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original request failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
litellm_logging_obj=self._logging_obj(has_logged_async_failure=True),
|
|
)
|
|
|
|
mock_trigger.assert_called_once()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_does_not_trigger_cooldown_when_has_logged_async_failure_is_false(self):
|
|
"""This is the exact dead-code scenario the bug fix addresses: before it,
|
|
the normal failure callback runs for the first attempt in a fallback chain
|
|
(has_logged_async_failure is still False at that point), so no explicit
|
|
trigger is needed there."""
|
|
with patch(
|
|
"litellm.router_utils.fallback_event_handlers._trigger_cooldown_for_failed_deployment"
|
|
) as mock_trigger:
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=self.RouterWithLoggingKwarg(),
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original request failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
litellm_logging_obj=self._logging_obj(has_logged_async_failure=False),
|
|
)
|
|
|
|
mock_trigger.assert_not_called()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_does_not_trigger_cooldown_when_no_logging_obj_present(self):
|
|
with patch(
|
|
"litellm.router_utils.fallback_event_handlers._trigger_cooldown_for_failed_deployment"
|
|
) as mock_trigger:
|
|
with pytest.raises(RuntimeError, match="fallback model also failed"):
|
|
await run_async_fallback(
|
|
litellm_router=self.RouterWithLoggingKwarg(),
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original request failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
mock_trigger.assert_not_called()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_stamps_fallback_info_into_metadata():
|
|
"""Spend logs are built from the request metadata of the nested call, so the
|
|
fallback signal has to be stamped there before recursing."""
|
|
router = RecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["fallback-model"],
|
|
original_model_group="primary-model",
|
|
original_exception=RuntimeError("original failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=0,
|
|
)
|
|
|
|
metadata = router.received_kwargs["metadata"]
|
|
assert metadata["attempted_fallbacks"] == 1
|
|
assert metadata["original_model_group"] == "primary-model"
|
|
assert metadata["model_group"] == "fallback-model"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_run_async_fallback_preserves_original_model_group_on_nested_fallback():
|
|
"""A second-level fallback receives the first fallback target as its
|
|
original_model_group argument, so the first-stamped value must survive the hop."""
|
|
router = RecordingRouter()
|
|
|
|
await run_async_fallback(
|
|
litellm_router=router,
|
|
fallback_model_group=["second-fallback"],
|
|
original_model_group="first-fallback",
|
|
original_exception=RuntimeError("first fallback failed"),
|
|
max_fallbacks=3,
|
|
fallback_depth=1,
|
|
metadata={"attempted_fallbacks": 1, "original_model_group": "primary-model"},
|
|
)
|
|
|
|
metadata = router.received_kwargs["metadata"]
|
|
assert metadata["attempted_fallbacks"] == 2
|
|
assert metadata["original_model_group"] == "primary-model"
|
|
|
|
|
|
class TestPreRoutingSelectionCarriesToFallbacks:
|
|
"""#38832: a complexity/auto router picks a tier behind the router name, but fallback
|
|
lookup kept using the router name, so the tier's configured chain never ran."""
|
|
|
|
def test_selection_is_recorded_in_the_metadata_bucket(self):
|
|
kwargs = {"model": "smart-router", "metadata": {}}
|
|
record_pre_routing_selection(kwargs, "tier1")
|
|
assert kwargs["metadata"]["pre_routing_selected_model"] == "tier1"
|
|
assert get_pre_routing_selection(kwargs) == "tier1"
|
|
|
|
def test_selection_is_recorded_in_the_litellm_metadata_bucket(self):
|
|
kwargs = {"model": "smart-router", "litellm_metadata": {}}
|
|
record_pre_routing_selection(kwargs, "tier2")
|
|
assert get_pre_routing_selection(kwargs) == "tier2"
|
|
|
|
def test_a_bucket_survives_the_kwargs_copy_that_fallbacks_run_on(self):
|
|
"""The bucket is shared by reference, which is the whole reason this works."""
|
|
outer = {"model": "smart-router", "metadata": {}}
|
|
inner = {**outer}
|
|
record_pre_routing_selection(inner, "tier1")
|
|
assert get_pre_routing_selection(outer) == "tier1"
|
|
|
|
def test_no_selection_reads_as_none(self):
|
|
assert get_pre_routing_selection({"model": "smart-router", "metadata": {}}) is None
|
|
assert get_pre_routing_selection({"model": "smart-router"}) is None
|
|
|
|
def test_missing_kwargs_is_a_no_op(self):
|
|
"""A caller with no kwargs must not raise, and must not leak the selection anywhere."""
|
|
record_pre_routing_selection(None, "tier1")
|
|
|
|
assert get_pre_routing_selection({}) is None
|
|
|
|
def test_a_non_dict_bucket_is_ignored(self):
|
|
kwargs = {"model": "smart-router", "metadata": "not-a-dict"}
|
|
record_pre_routing_selection(kwargs, "tier1")
|
|
assert get_pre_routing_selection(kwargs) is None
|
|
|
|
def test_fallbacks_resolve_against_the_selected_tier(self):
|
|
"""The lookup the router performs, keyed on the tier rather than the router name."""
|
|
fallbacks = [{"tier1": ["backup-a", "backup-b"]}, {"tier2": ["backup-c"]}]
|
|
assert get_fallback_model_group(fallbacks=fallbacks, model_group="tier1")[0] == ["backup-a", "backup-b"]
|
|
assert get_fallback_model_group(fallbacks=fallbacks, model_group="smart-router")[0] is None
|
|
|
|
|
|
class TestPreRoutingSelectionIsPerHop:
|
|
"""#38832 review: the buckets also carry whatever the caller sent, and a fallback hop
|
|
inherits the previous hop's tier, so a hop must start without a selection."""
|
|
|
|
def test_a_caller_supplied_selection_is_dropped(self):
|
|
kwargs = {"model": "plain", "metadata": {"pre_routing_selected_model": "tier1"}}
|
|
|
|
clear_pre_routing_selection(kwargs)
|
|
|
|
assert get_pre_routing_selection(kwargs) is None
|
|
assert "pre_routing_selected_model" not in kwargs["metadata"]
|
|
|
|
def test_both_buckets_are_cleared(self):
|
|
kwargs = {
|
|
"metadata": {"pre_routing_selected_model": "tier1"},
|
|
"litellm_metadata": {"pre_routing_selected_model": "tier2"},
|
|
}
|
|
|
|
clear_pre_routing_selection(kwargs)
|
|
|
|
assert get_pre_routing_selection(kwargs) is None
|
|
|
|
def test_the_rest_of_the_bucket_is_left_alone(self):
|
|
kwargs = {"metadata": {"pre_routing_selected_model": "tier1", "tags": ["a"]}}
|
|
|
|
clear_pre_routing_selection(kwargs)
|
|
|
|
assert kwargs["metadata"] == {"tags": ["a"]}
|
|
|
|
def test_clearing_is_a_no_op_without_a_usable_bucket(self):
|
|
kwargs = {"model": "plain", "metadata": "not-a-dict"}
|
|
|
|
clear_pre_routing_selection(None)
|
|
clear_pre_routing_selection(kwargs)
|
|
|
|
assert kwargs == {"model": "plain", "metadata": "not-a-dict"}
|
|
|
|
def test_a_selection_recorded_after_clearing_is_kept(self):
|
|
"""Clearing runs before routing, so the hook's own write must survive it."""
|
|
kwargs = {"model": "smart-router", "metadata": {"pre_routing_selected_model": "stale"}}
|
|
|
|
clear_pre_routing_selection(kwargs)
|
|
record_pre_routing_selection(kwargs, "tier1")
|
|
|
|
assert get_pre_routing_selection(kwargs) == "tier1"
|
|
|
|
|
|
class TestOrderedFallbackLookupGroups:
|
|
def test_tier_first_then_requested_group_deduped(self):
|
|
from litellm.router_utils.fallback_event_handlers import (
|
|
PRE_ROUTING_SELECTED_MODEL_KEY,
|
|
fallback_lookup_groups,
|
|
)
|
|
|
|
kwargs = {"litellm_metadata": {PRE_ROUTING_SELECTED_MODEL_KEY: "tier1"}}
|
|
assert fallback_lookup_groups(kwargs, "smart-router") == ("tier1", "smart-router")
|
|
assert fallback_lookup_groups(kwargs, "tier1") == ("tier1",)
|
|
assert fallback_lookup_groups({}, "smart-router") == ("smart-router",)
|
|
assert fallback_lookup_groups({}, None) == ()
|
|
|
|
def test_session_remap_keeps_the_bound_router_between_tier_and_requested_group(self):
|
|
from litellm.router_utils.fallback_event_handlers import (
|
|
PRE_ROUTING_SELECTED_MODEL_KEY,
|
|
fallback_lookup_groups,
|
|
)
|
|
|
|
kwargs = {
|
|
"litellm_metadata": {
|
|
PRE_ROUTING_SELECTED_MODEL_KEY: "tier1",
|
|
"model_group": "smart-router",
|
|
}
|
|
}
|
|
|
|
assert fallback_lookup_groups(kwargs, "requested-model") == (
|
|
"tier1",
|
|
"smart-router",
|
|
"requested-model",
|
|
)
|
|
assert fallback_lookup_groups({"metadata": {"model_group": []}}, "requested-model") == (
|
|
"requested-model",
|
|
)
|
|
|
|
def test_fallback_hop_resumes_the_original_groups_chain_last(self):
|
|
from litellm.router_utils.fallback_event_handlers import fallback_lookup_groups
|
|
|
|
kwargs = {"metadata": {"model_group": "fb1", "original_model_group": "primary"}}
|
|
|
|
assert fallback_lookup_groups(kwargs, "fb1") == ("fb1", "primary")
|
|
assert fallback_lookup_groups({"metadata": {"original_model_group": 42}}, "fb1") == ("fb1",)
|
|
|
|
def test_first_resolving_group_wins_and_generic_idx_survives_a_miss(self):
|
|
from litellm.router_utils.fallback_event_handlers import (
|
|
get_fallback_model_group_for_lookup_groups,
|
|
)
|
|
|
|
fallbacks = [{"tier1": ["backup-a"]}, {"smart-router": ["backup-b"]}, {"*": ["backup-c"]}]
|
|
assert get_fallback_model_group_for_lookup_groups(fallbacks, ("tier1", "smart-router")) == (["backup-a"], None)
|
|
assert get_fallback_model_group_for_lookup_groups(fallbacks, ("tier9", "smart-router")) == (["backup-b"], None)
|
|
assert get_fallback_model_group_for_lookup_groups(fallbacks, ("tier9", "no-such")) == (["backup-c"], 2)
|
|
assert get_fallback_model_group_for_lookup_groups([{"tier1": ["backup-a"]}], ("no", "nope")) == (None, None)
|
|
|
|
|
|
class TestHasUnattemptedFallbackTarget:
|
|
def test_exhausted_chain_is_not_recoverable_but_a_fresh_entry_is(self):
|
|
from litellm.router_utils.fallback_event_handlers import (
|
|
has_unattempted_fallback_target,
|
|
)
|
|
|
|
attempted: Final = AttemptedFallbackTargets()
|
|
attempted.record("primary")
|
|
attempted.record("fb1")
|
|
attempted.record("fb2")
|
|
|
|
assert has_unattempted_fallback_target(["fb1", "fb2"], {"attempted_targets": attempted}) is False
|
|
assert has_unattempted_fallback_target(["fb1", "fb3"], {"attempted_targets": attempted}) is True
|
|
assert has_unattempted_fallback_target(["fb1"], {}) is True
|
|
assert has_unattempted_fallback_target(None, {}) is False
|
|
|
|
|
|
def test_get_fallback_model_group_matches_provider_prefixed_key():
|
|
"""A bare model group routed via a wildcard (e.g. "gpt-4o" through
|
|
"openai/*") must match a fallback keyed on the provider-prefixed name,
|
|
which is the form the Admin UI offers for wildcard routes."""
|
|
fallbacks = [{"openai/gpt-4o": ["claude-3-haiku"]}]
|
|
|
|
fallback_model_group, _ = get_fallback_model_group(fallbacks=fallbacks, model_group="gpt-4o")
|
|
|
|
assert fallback_model_group == ["claude-3-haiku"]
|
|
|
|
|
|
def test_get_fallback_model_group_exact_match_beats_prefixed_match():
|
|
fallbacks = [
|
|
{"openai/gpt-4o": ["claude-3-haiku"]},
|
|
{"gpt-4o": ["gemini-1.5-flash"]},
|
|
]
|
|
|
|
fallback_model_group, _ = get_fallback_model_group(fallbacks=fallbacks, model_group="gpt-4o")
|
|
|
|
assert fallback_model_group == ["gemini-1.5-flash"]
|
|
|
|
|
|
def test_get_fallback_model_group_prefixed_match_ignores_unknown_models():
|
|
"""Provider inference fails for unknown bare names - the lookup must not
|
|
raise and must fall through to the generic fallback."""
|
|
fallbacks = [
|
|
{"openai/some-model": ["claude-3-haiku"]},
|
|
{"*": ["gemini-1.5-flash"]},
|
|
]
|
|
|
|
fallback_model_group, _ = get_fallback_model_group(fallbacks=fallbacks, model_group="some-unknown-model-xyz")
|
|
|
|
assert fallback_model_group == ["gemini-1.5-flash"]
|
|
|
|
|
|
def test_get_fallback_model_group_prefixed_match_skips_prefixed_model_group():
|
|
"""An already-prefixed model group must not double-prefix."""
|
|
fallbacks = [{"openai/openai/gpt-4o": ["claude-3-haiku"]}]
|
|
|
|
fallback_model_group, _ = get_fallback_model_group(fallbacks=fallbacks, model_group="openai/gpt-4o")
|
|
|
|
assert fallback_model_group is None
|
|
|
|
|
|
def test_get_fallback_model_group_never_resolves_a_provider_without_a_prefixed_key(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""An alias-style group name has no provider, and resolving it prints the SDK's provider-list banner,
|
|
so the lookup only infers a provider when some key is spelled <provider>/<group>."""
|
|
|
|
resolver: Final = MagicMock(return_value=("my-alias", "openai", None, None))
|
|
monkeypatch.setattr(get_llm_provider_logic, "get_llm_provider", resolver)
|
|
fallbacks: Final = [{"gpt-5.5-pro": ["claude-sonnet-4-6"]}, {"*": ["gpt-5.5-mini"]}]
|
|
|
|
assert get_fallback_model_group(fallbacks=fallbacks, model_group="my-alias") == (["gpt-5.5-mini"], 1)
|
|
resolver.assert_not_called()
|