litellm/MUTATIONS.md

Mutation evidence for litellm_sail_tests_unit

Tests-only regression branch for PR 42840, created from tip d7bc17fa47 (merge base 0c1c3e18d5). Every new test was run three ways: at the merge base with the changed test files copied in (red unless noted below), at the tip with one named production mutation applied (red), then at the tip with the mutated file restored from d7bc17fa47 (green). Production files were diffed against d7bc17fa47 before every mutation and after every restore, and the final git status of the branch shows test files plus this document only

Merge-base run

Command, run in a detached worktree of the merge base with the eleven changed test files and the new chat transformation test copied in:

LITELLM_LOCAL_MODEL_COST_MAP=True pytest -p no:cacheprovider <the tests listed in the PR report> -q

Result: 15 failed, 13 passed. The 13 that pass at the merge base are the 12 parametrized test_completion_cost_balanced_tier_bills_standard_rates_on_rows_without_balanced_keys cases (the balanced tier does not exist there, so it takes the unknown-tier path and trivially equals the no-tier cost; M01 and M02 are their kill evidence) and test_completion_cost_ignores_completion_window_for_openai (the merge base has no completion-window gate at all, so it bills base rates for every provider; M15 is its kill evidence). The 15 failures are import errors for merge_extra_body, KNOWN_REQUEST_SERVICE_TIERS and special_handling, the missing sail slug and _balanced keys, sail/ resolving to LLM Provider NOT provided, the transcription call routing to OpenAI instead of failing before HTTP, and extra_body missing from the Responses logging optional_params

Named production mutations

Each mutation ran LITELLM_LOCAL_MODEL_COST_MAP=True pytest -p no:cacheprovider <tests> -q at the tip, once mutated and once restored. Kill count: 15 of 15

M01 area 1/3: drop the service-tier suffix fallback in _get_cost_per_unit

File: litellm/litellm_core_utils/llm_cost_calc/utils.py

Tests:

tests/test_litellm/test_cost_calculator.py::test_completion_cost_balanced_tier_bills_standard_rates_on_rows_without_balanced_keys
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py::test_get_cost_per_unit_falls_back_from_balanced_key_to_base

Mutation (exact string replace):

    if cost_per_unit is None:
        # Check if any service tier suffix exists in the cost key
        for suffix in _SERVICE_TIER_SUFFIXES:

with:

    if cost_per_unit is None:
        # Check if any service tier suffix exists in the cost key
        for suffix in ():

Red (mutated), killed: true

E       AssertionError: (model, no tier, balanced, unknown tier) rows that disagree: (('claude-fable-5', 0.0173, 0.0, 0.0173), ('claude-fable-5-1', 0.017075, 0.0, 0.017075), ('claude-haiku-4-5', 0.00173, 0.0, 0.00173), ('claude-haiku-4-5-20251001', 0.00173, 0.0, 0.00173), ('claude-mythos-5', 0.0173, ...
E       assert (('claude-fab....017075), ...) == ()
E         
E         Left contains 20 more items, first extra item: ('claude-fable-5', 0.0173, 0.0, 0.0173)

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/litellm_core_utils/llm_cost_calc/utils.py, then the same command

Green (restored), exit 0: 13 passed in 1.64s

M02 area 1/3/4: remove BALANCED from the ServiceTier enum

File: litellm/types/utils.py

Tests:

tests/test_litellm/integrations/test_prometheus_service_tier_label.py::test_requested_tier_recognizes_balanced_and_drops_garbage_without_touching_the_other_tiers
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py::test_threshold_keys_exclude_balanced_variants_and_still_parse_k_thresholds

Mutation (exact string replace):

    AUTO = "auto"
    BALANCED = "balanced"
    FLEX = "flex"

with:

    AUTO = "auto"
    FLEX = "flex"

Red (mutated), killed: true

E   AttributeError: type object 'ServiceTier' has no attribute 'BALANCED'

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/types/utils.py, then the same command

Green (restored), exit 0: 2 passed in 0.65s

M03 area 3: map the balanced tier onto the flex cost suffix

File: litellm/litellm_core_utils/llm_cost_calc/utils.py

Tests:

tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py::test_threshold_keys_exclude_balanced_variants_and_still_parse_k_thresholds

Mutation (exact string replace):

        ServiceTier.BALANCED.value: ServiceTier.BALANCED.value,

with:

        ServiceTier.BALANCED.value: ServiceTier.FLEX.value,

Red (mutated), killed: true

E       assert (4e-06, 6e-06) == (7e-06, 1.4e-05)
E         
E         At index 0 diff: 4e-06 != 7e-06
E         Use -v to get more diff

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/litellm_core_utils/llm_cost_calc/utils.py, then the same command

Green (restored), exit 0: 1 passed in 0.42s

M04 area 2: put a balanced key on a non-sail cost-map row

File: model_prices_and_context_window.json

Tests:

tests/test_litellm/test_model_prices_schema.py::test_balanced_tier_keys_live_only_on_sail_rows_next_to_their_base_key

Mutation (regex replace):

("gpt-4o-mini": \{\n)

with:

\1        "input_cost_per_token_balanced": 1e-07,\n

Red (mutated), killed: true

E       AssertionError: (model, balanced key, litellm_provider, has base key) for balanced tier pricing that is off a sail/ row or missing its base key: (('gpt-4o-mini', 'input_cost_per_token_balanced', 'openai', True),)
E       assert (('gpt-4o-min...enai', True),) == ()
E         
E         Left contains one more item: ('gpt-4o-mini', 'input_cost_per_token_balanced', 'openai', True)

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- model_prices_and_context_window.json, then the same command

Green (restored), exit 0: 1 passed in 0.40s

M05 area 5: BaseConfig.merge_extra_body deep-merges nested metadata

File: litellm/llms/base_llm/chat/transformation.py

Tests:

tests/test_litellm/llms/base_llm/chat/test_base_chat_transformation.py::test_default_merge_extra_body_shallow_merges_and_lets_extra_body_replace_nested_metadata

Mutation (exact string replace):

        return {**request, **extra_body} if extra_body else request  # mutable-ok: wire request body is a plain dict

with:

        if not extra_body:
            return request
        merged = {**request, **extra_body}
        if isinstance(request.get('metadata'), dict) and isinstance(extra_body.get('metadata'), dict):
            merged['metadata'] = {**request['metadata'], **extra_body['metadata']}
        return merged

Red (mutated), killed: true

E       AssertionError: assert {'messages': ...re': 0.2, ...} == {'messages': ...re': 0.2, ...}
E         
E         Omitting 4 identical items, use -vv to show
E         Differing items:

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/llms/base_llm/chat/transformation.py, then the same command

Green (restored), exit 0: 1 passed in 0.41s

M06 area 5: BaseResponsesAPIConfig.merge_extra_body deep-merges nested metadata

File: litellm/llms/base_llm/responses/transformation.py

Tests:

tests/test_litellm/llms/base_llm/responses/test_transformation.py::test_default_merge_extra_body_shallow_merges_and_lets_extra_body_replace_nested_metadata

Mutation (exact string replace):

        return {**request, **extra_body} if extra_body else request  # mutable-ok: wire request body is a plain dict

with:

        if not extra_body:
            return request
        merged = {**request, **extra_body}
        if isinstance(request.get('metadata'), dict) and isinstance(extra_body.get('metadata'), dict):
            merged['metadata'] = {**request['metadata'], **extra_body['metadata']}
        return merged

Red (mutated), killed: true

E       AssertionError: assert {'input': 'hi...el': 'm', ...} == {'input': 'hi...el': 'm', ...}
E         
E         Omitting 4 identical items, use -vv to show
E         Differing items:

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/llms/base_llm/responses/transformation.py, then the same command

Green (restored), exit 0: 1 passed in 0.39s

M07 area 5: llm_http_handler.completion inlines the shallow merge instead of calling the hook

File: litellm/llms/custom_httpx/llm_http_handler.py

Tests:

tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py::test_completion_merges_extra_body_through_the_real_config_hook

Mutation (exact string replace):

            data: Final = provider_config.merge_extra_body(transformed, extra_body)

with:

            data: Final = {**transformed, **extra_body} if extra_body else transformed

Red (mutated), killed: true

E       AssertionError: assert {'messages': ...re': 0.2, ...} == {'messages': ...d': 'x'}, ...}
E         
E         Omitting 3 identical items, use -vv to show
E         Differing items:

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/llms/custom_httpx/llm_http_handler.py, then the same command

Green (restored), exit 0: 1 passed in 0.43s

M08 area 5: llm_http_handler response_api_handler (sync + async) inlines the shallow merge instead of calling the hook

File: litellm/llms/custom_httpx/llm_http_handler.py

Tests:

tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py::test_response_api_handler_merges_extra_body_through_the_real_config_hook
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py::test_async_response_api_handler_merges_extra_body_through_the_real_config_hook

Mutation (exact string replace):

        data = responses_api_provider_config.merge_extra_body(data, extra_body)

with:

        data = {**data, **extra_body} if extra_body else data

Red (mutated), killed: true

E       AssertionError: assert {'input': 'hi...el': 'm', ...} == {'input': 'hi...el': 'm', ...}
E         
E         Omitting 3 identical items, use -vv to show
E         Differing items:

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/llms/custom_httpx/llm_http_handler.py, then the same command

Green (restored), exit 0: 2 passed in 0.45s

M09 area 6: responses/main.py stops passing extra_body into logging optional_params

File: litellm/responses/main.py

Tests:

tests/test_litellm/responses/test_responses_api_request_body.py::test_aresponses_logs_extra_body_once_in_optional_params_and_leaves_the_wire_body_alone

Mutation (exact string replace):

                **(
                    {"extra_body": extra_body} if extra_body else {}
                ),  # mutable-ok: update_from_kwargs stores a plain dict

with:

Red (mutated), killed: true

E       AssertionError: assert ({'max_output...warg': '0'}},) == ({'extra_body...warg': '0'}},)
E         
E         At index 0 diff: {'max_output_tokens': 50, 'metadata': {'from_kwarg': '0'}} != {'max_output_tokens': 50, 'metadata': {'from_kwarg': '0'}, 'extra_body': {'metadata': {'from_extra_body': '1'}, 'vendor_only_field': 'x'}}
E         Use -v to get more diff

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/responses/main.py, then the same command

Green (restored), exit 0: 1 passed in 0.84s

M10 area 7: keep sail inside OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS

File: litellm/constants.py

Tests:

tests/test_litellm/test_constants.py::test_openai_transcription_providers_are_the_openai_compatible_set_minus_sail
tests/test_litellm/llms/openai_like/test_sail_provider.py::TestSailRequestShape::test_sail_transcription_rejected_without_hitting_sail

Mutation (exact string replace):

    frozenset(openai_compatible_providers) - frozenset(("sail",))

with:

    frozenset(openai_compatible_providers)

Red (mutated), killed: true

E       AssertionError: assert frozenset({'a...ure_ai', ...}) == frozenset({'a...ure_ai', ...})
E         
E         Extra items in the left set:
E         'sail'

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/constants.py, then the same command

Green (restored), exit 0: 2 passed in 0.84s

M11 area 8: JSON loader drops the special_handling block of every provider

File: litellm/llms/openai_like/json_loader.py

Tests:

tests/test_litellm/llms/openai_like/test_json_providers.py::test_json_loader_yields_the_merge_base_params_and_special_handling_for_every_non_sail_slug

Mutation (exact string replace):

        self.special_handling: Mapping[str, object] = data.get("special_handling") or MappingProxyType({})

with:

        self.special_handling: Mapping[str, object] = MappingProxyType({})

Red (mutated), killed: true

E       AssertionError: assert {'abliteratio...', ...]}, ...} == {'abliteratio...', ...]}, ...}
E         
E         Omitting 26 identical items, use -vv to show
E         Differing items:

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/llms/openai_like/json_loader.py, then the same command

Green (restored), exit 0: 1 passed in 7.49s

M12 area 8: dynamic_config drops temperature from every JSON provider's supported params

File: litellm/llms/openai_like/dynamic_config.py

Tests:

tests/test_litellm/llms/openai_like/test_json_providers.py::test_json_loader_yields_the_merge_base_params_and_special_handling_for_every_non_sail_slug

Mutation (exact string replace):

            excluded_params: Final = frozenset(tool_params) | frozenset(provider.unsupported_params)

with:

            excluded_params: Final = frozenset(tool_params) | frozenset(provider.unsupported_params) | {"temperature"}

Red (mutated), killed: true

E       AssertionError: assert {'abliteratio...', ...]}, ...} == {'abliteratio...', ...]}, ...}
E         
E         Differing items:
E         {'apertis': {'special_handling': {}, 'supported_openai_params': ['audio', 'extra_headers', 'frequency_penalty', 'logit_bias', 'logprobs', 'max_completion_tokens', ...]}} != {'apertis': {'special_handling': {}, 'supported_openai_params': ['audio', 'extra_headers', 'frequency_penalty', 'logi ...

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/llms/openai_like/dynamic_config.py, then the same command

Green (restored), exit 0: 1 passed in 7.78s

M13 area 9: get_model_info never fills input_cost_per_token_balanced

File: litellm/utils.py

Tests:

tests/test_litellm/test_utils.py::test_get_model_info_reports_balanced_tier_prices_only_where_the_row_has_them

Mutation (exact string replace):

                input_cost_per_token_balanced=_model_info.get("input_cost_per_token_balanced", None),

with:

                input_cost_per_token_balanced=None,

Red (mutated), killed: true

E       assert (None, 2.5e-0...7, None, None) == (5e-07, 2.5e-...7, None, None)
E         
E         At index 0 diff: None != 5e-07
E         Use -v to get more diff

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/utils.py, then the same command

Green (restored), exit 0: 1 passed, 6 warnings in 0.93s

M14 area 10: completion_cost runs the completion-window gate before inferring the provider

File: litellm/cost_calculator.py

Tests:

tests/test_litellm/test_cost_calculator.py::test_completion_cost_infers_sail_from_model_before_the_completion_window_gate

Mutation (exact string replace):

                if custom_llm_provider is None:
                    try:
                        model, custom_llm_provider, _, _ = litellm.get_llm_provider(
                            model=model
                        )  # strip the llm provider from the model name -> for image gen cost calculation
                    except Exception as e:
                        verbose_logger.debug(
                            "litellm.cost_calculator.py::completion_cost() - Error inferring custom_llm_provider - %s",
                            e,
                        )
                service_tier = _service_tier_billed_by_completion_window(
                    service_tier=service_tier,
                    optional_params=optional_params,
                    custom_llm_provider=custom_llm_provider,
                )

with:

                service_tier = _service_tier_billed_by_completion_window(
                    service_tier=service_tier,
                    optional_params=optional_params,
                    custom_llm_provider=custom_llm_provider,
                )
                if custom_llm_provider is None:
                    try:
                        model, custom_llm_provider, _, _ = litellm.get_llm_provider(
                            model=model
                        )  # strip the llm provider from the model name -> for image gen cost calculation
                    except Exception as e:
                        verbose_logger.debug(
                            "litellm.cost_calculator.py::completion_cost() - Error inferring custom_llm_provider - %s",
                            e,
                        )

Red (mutated), killed: true

E       assert 0.001596 == 0.00075999999...9999 ± 1.0e-12
E         
E         comparison failed
E         Obtained: 0.001596

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/cost_calculator.py, then the same command

Green (restored), exit 0: 1 passed in 0.55s

M15 area 10: completion-window gate ignores the provider flag (every provider bills by completion_window)

File: litellm/cost_calculator.py

Tests:

tests/test_litellm/test_cost_calculator.py::test_completion_cost_ignores_completion_window_for_openai

Mutation (exact string replace):

    if optional_params is None or not _provider_bills_by_completion_window(custom_llm_provider):

with:

    if optional_params is None:

Red (mutated), killed: true

E       assert 0.01 == 0.02 ± 2.0e-11
E         
E         comparison failed
E         Obtained: 0.01

Restore: git checkout d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62 -- litellm/cost_calculator.py, then the same command

Green (restored), exit 0: 1 passed in 0.45s