litellm/MUTATIONS.md

Mutation evidence for litellm_sail_tests_integration

Review evidence for the tests/integration cell of the PR 42840 regression sweep. Not for merge

Tip: d7bc17fa47 (litellm_add_sail_provider). Base: 0c1c3e18d5

Every run below used the existing integration harness against a real proxy (LITELLM_LOCAL_MODEL_COST_MAP=True, STORE_MODEL_IN_DB=True, num_retries: 0), Postgres, Redis and the scripted upstream from tests/integration/_support/upstream.py. Two stacks ran side by side: tip (upstream 8190, proxy 4000, database integration_tip) and base (upstream 8191, proxy 4001, database integration_base). The proxy was restarted after every apply and after every restore so the mutated module was actually the one serving

Runner used for every pytest invocation (paths shortened below as run <tree> <label> <uport> <pport> <args>):

export PATH=/home/ubuntu/repos/litellm/.venv/bin:$PATH
export PYTHONPATH="$tree:$tree/tests:$tree/tests/e2e"
export INTEGRATION_PROXY_URL="http://127.0.0.1:$pport"
export INTEGRATION_UPSTREAM_URL="http://127.0.0.1:$uport"
export INTEGRATION_MASTER_KEY=sk-integration-master
export DATABASE_URL="postgresql://postgres:postgres@127.0.0.1:5432/integration_$label"
export LITELLM_LOCAL_MODEL_COST_MAP=True
python -m pytest -p no:cacheprovider -q "$@"

Tests

File	Test	Covers
tests/integration/pricing/test_service_tier_pricing.py	test_balanced_service_tier_bills_balanced_rates_and_keeps_pricing_off_the_wire	(2) _balanced rates via litellm_params bill balanced at service_tier=balanced, base with no tier, full upstream body
tests/integration/pricing/test_service_tier_pricing.py	test_balanced_and_flex_on_a_base_rate_only_model_bill_base_rates_and_pass_the_tier_through	(1) and (3) balanced and flex on a base-rate-only model bill base, upstream body carries the tier untouched
tests/integration/pricing/test_service_tier_pricing.py	test_extra_body_metadata_on_an_openai_shaped_deployment_replaces_the_wire_metadata_whole[openai]	(4) shallow overwrite of request metadata on a non-Sail openai deployment, full upstream body
tests/integration/pricing/test_service_tier_pricing.py	test_extra_body_metadata_on_an_openai_shaped_deployment_replaces_the_wire_metadata_whole[cognition]	(4) same through the JSON provider loader without special handling
tests/integration/pricing/test_service_tier_pricing.py	test_sail_flex_sends_completion_window_without_service_tier_and_bills_flex_rates	(5) sail/<model> sends metadata.completion_window=flex and no service_tier, bills _flex; explicit asap bills base; no tier bills base
tests/integration/pricing/test_responses_extra_body_pricing.py	test_responses_extra_body_on_an_openai_shaped_deployment_reaches_the_wire_and_bills_base_rates	(6) /v1/responses extra_body reaches the wire whole and bills base on a non-Sail deployment even when it carries completion_window
tests/integration/pricing/test_responses_extra_body_pricing.py	test_responses_extra_body_completion_window_on_a_sail_deployment_bills_the_window	(6) extra_body lands in Responses logging optional_params (observable as _flex billing on Sail, base on asap)
tests/integration/providers/test_sail_audio_transcription.py	test_audio_transcription_on_a_sail_deployment_is_rejected_before_the_upstream_sees_it	(7) skipped with BUG:, see Blockers

Every helper asserts the whole upstream body with ==, the x-litellm-response-cost header, then polls the LiteLLM_SpendLogs row with eventually(..., seconds=70) and asserts prompt_tokens, completion_tokens, spend and cost_breakdown.input_cost / output_cost to rel=1e-6. Rates are the test's own litellm_params, nothing is read from the vendor cost map

Tip run (all production files restored)

run /home/ubuntu/repos/litellm-sail-tip tip 8190 4000 tests/integration/pricing/test_service_tier_pricing.py tests/integration/pricing/test_responses_extra_body_pricing.py tests/integration/providers/test_sail_audio_transcription.py -v
tests/integration/pricing/test_service_tier_pricing.py ......            [ 66%]
tests/integration/pricing/test_responses_extra_body_pricing.py ..        [ 88%]
tests/integration/providers/test_sail_audio_transcription.py s           [100%]
======================== 8 passed, 1 skipped in 58.18s =========================

Final run on the committed files (after ruff format and the upstream observation scoping described under Gates):

run /home/ubuntu/repos/litellm-sail-tip tip 8190 4000 tests/integration/pricing/test_service_tier_pricing.py tests/integration/pricing/test_responses_extra_body_pricing.py tests/integration/providers/test_sail_audio_transcription.py
........s                                                                [100%]
8 passed, 1 skipped in 45.05s

Base run (same test files copied into the 0c1c3e18 worktree, base proxy)

run /home/ubuntu/repos/litellm-sail-base base 8191 4001 tests/integration/pricing/test_service_tier_pricing.py tests/integration/pricing/test_responses_extra_body_pricing.py tests/integration/providers/test_sail_audio_transcription.py --tb=line
.F...F.Fs                                                                [100%]
E   AssertionError: ... assert 0.06 == 0.22 ± 2.2e-07   (test_service_tier_pricing.py:53, balanced tier billed base at base)
E   AssertionError: POST /model/new: 500 {"error":{"message":"Model create was saved to the database, but the model id(s) [...] are not live in this pod's router after the reload and are not being served by this pod. ..."}}   (sail/ deployment cannot be registered at base)
E   AssertionError: POST /model/new: 500 {"error":{"message":"Model create was saved to the database, but the model id(s) [...] are not live in this pod's router after the reload and are not being served by this pod. ..."}}   (sail/ deployment cannot be registered at base)
FAILED tests/integration/pricing/test_service_tier_pricing.py::test_balanced_service_tier_bills_balanced_rates_and_keeps_pricing_off_the_wire
FAILED tests/integration/pricing/test_service_tier_pricing.py::test_sail_flex_sends_completion_window_without_service_tier_and_bills_flex_rates
FAILED tests/integration/pricing/test_responses_extra_body_pricing.py::test_responses_extra_body_completion_window_on_a_sail_deployment_bills_the_window
3 failed, 5 passed, 1 skipped in 6.68s

The five tests that pass at base are behaviours the PR preserves rather than introduces (base-rate fallback, openai shallow extra_body merge, Responses extra_body on the wire). Each is pinned by a named mutation below

Mutations

Each mutation was applied to one production file in the tip worktree with an exact single-occurrence replacement, the tip proxy was restarted, the test ran red, the original file was copied back, the proxy was restarted and the test ran green. Ineffective mutations are listed too and are not counted as kills

#	Production mutation	Test	Red run
1	litellm/litellm_core_utils/llm_cost_calc/utils.py: removed the ServiceTier.BALANCED.value: ServiceTier.BALANCED.value, entry from the service tier suffix mapping, so _balanced keys are never selected	test_balanced_service_tier_bills_balanced_rates_and_keeps_pricing_off_the_wire	assert 0.06 == 0.22 ± 2.2e-07 (test_service_tier_pricing.py:53), 1 failed
2	litellm/litellm_core_utils/llm_cost_calc/utils.py: fallback_cost = model_info.get(base_key) -> fallback_cost = None, breaking the tier-to-base rate fallback	test_balanced_and_flex_on_a_base_rate_only_model_bill_base_rates_and_pass_the_tier_through	KeyError: 'x-litellm-response-cost' (the proxy no longer priced the call), 1 failed
3	litellm/llms/openai/openai.py: the SDK request options were built without extra_body=data.get("extra_body"), so extra_body never reached the wire	test_extra_body_metadata_on_an_openai_shaped_deployment_replaces_the_wire_metadata_whole[openai] and [cognition]	Right contains 1 more item: {'metadata': {'b': 3, 'c': 4}}, 2 failed, 4 deselected
4	litellm/cost_calculator.py: dropped "asap" from the accepted completion windows in _completion_window_value, so an explicit asap no longer overrides service_tier=flex	test_sail_flex_sends_completion_window_without_service_tier_and_bills_flex_rates	assert 0.013999999999999999 == 0.06 ± 6.0e-08, 1 failed, 5 deselected
5	litellm/llms/openai_like/dynamic_config.py: key: value for key, value in body.items() if key != "service_tier" -> key: value for key, value in body.items(), leaving service_tier on the Sail wire	test_sail_flex_sends_completion_window_without_service_tier_and_bills_flex_rates	Left contains 1 more item: {'service_tier': 'flex'}, 1 failed, 5 deselected
6	litellm/llms/base_llm/responses/transformation.py BaseResponsesAPIConfig.merge_extra_body: {**request, **extra_body} -> {**request, **{k: v for k, v in extra_body.items() if k != "metadata"}}, dropping metadata from the Responses wire body	test_responses_extra_body_on_an_openai_shaped_deployment_reaches_the_wire_and_bills_base_rates	Right contains 1 more item: {'metadata': {'completion_window': 'flex', 'trace': 'extra-body-control'}} (test_responses_extra_body_pricing.py:78), 1 failed, 1 passed
7	litellm/responses/main.py responses(): removed **({"extra_body": extra_body} if extra_body else {}) from the optional_params passed to update_from_kwargs, so extra_body no longer reaches Responses logging	test_responses_extra_body_completion_window_on_a_sail_deployment_bills_the_window	assert 0.2 == 0.05 ± 5.0e-08 (test_responses_extra_body_pricing.py:72, Sail flex window billed base), 1 failed, 1 passed
8	litellm/cost_calculator.py _provider_bills_by_completion_window: return provider is not None and provider.special_handling.get("service_tier_as_completion_window") is True -> return True, billing every provider by completion_window	test_responses_extra_body_on_an_openai_shaped_deployment_reaches_the_wire_and_bills_base_rates	assert 0.05 == 0.2 ± 2.0e-07 (test_responses_extra_body_pricing.py:72, openai deployment billed flex), 1 failed, 1 passed

Ineffective mutation, not counted: litellm/llms/base_llm/chat/transformation.py BaseConfig.merge_extra_body made to drop metadata. The openai and cognition chat deployments both go through the OpenAI SDK path in litellm/llms/openai/openai.py, which merges extra_body itself, so the observed body was unchanged and the metadata test stayed green. Restored. Mutation 3 targets the path those deployments actually use

Kill count: 8 of 8 counted mutations killed (9 attempted, 1 discarded as not exercising the tested path)

Restore evidence: after every restore git status --short in the tip worktree listed only the four test files (tests/integration/_support/upstream.py, tests/integration/pricing/test_service_tier_pricing.py, tests/integration/pricing/test_responses_extra_body_pricing.py, tests/integration/providers/test_sail_audio_transcription.py) plus this file, and the tip run above is the green run on the fully restored tree

Blockers

(6) model_parameters.extra_body in the spend row: not observable at the tip. StandardLoggingPayload carries model_parameters, but litellm/proxy/spend_tracking/spend_event.py drops it before the row is written (_STANDARD_LOGGING_DROPPED_KEYS = frozenset({"model_parameters"})) and LiteLLM_SpendLogs has no such column. The row's metadata JSON for a /v1/responses call was inspected directly in Postgres and contains no extra_body. The "once, not twice" claim therefore cannot be asserted from the spend row. What the test pins instead is the effect of extra_body reaching logging optional_params: the Sail deployment bills _flex from extra_body.metadata.completion_window and drops to base on asap (mutation 7 removes exactly the PR's extra_body logging hunk and the test goes red), while the openai deployment with the same window bills base (mutation 8). The wire body is asserted whole for both

(7) /v1/audio/transcriptions on sail/x: the tip answers 500, not 4xx. Full observed body: {"error":{"message":"litellm.APIConnectionError: APIConnectionError: SailException - Unmapped provider passed in. Unable to get the response.\n\nLiteLLM: model group '<model>' failed with the error above. No fallback was attempted.","type":null,"param":null,"code":"500"}}. The scripted upstream observed zero requests for the call (the exception is raised in litellm.main.atranscription before any HTTP call, confirmed from the proxy log traceback and an empty /__observations drain afterwards). Per tests/integration/AGENTS.md the test keeps its intended 4xx and full-message assertions and is pytest.skip("BUG: ...") at the top of the body. It was not mutation tested since it does not run. At base the same test fails earlier because a sail/ deployment cannot be registered at all (see the base run)

No real provider key was needed: every upstream in this cell is the scripted harness upstream, so the 1Password Sail key was neither read nor used

Gates

Run from the tip worktree with /home/ubuntu/repos/litellm/.venv/bin on PATH, FILES being the four touched test files:

ruff check $FILES
All checks passed!
ruff format --check $FILES
4 files already formatted
basedpyright --pythonpath /home/ubuntu/repos/litellm/.venv/bin/python $FILES

basedpyright is not configured for tests/integration (pyrightconfig.json includes only litellm), so it reports the directory's pre-existing patterns: float(rows[0]["spend"]) on a JsonValue and pytest.approx being partially unknown, both present in every neighbouring pricing test, plus errors in the untouched parts of upstream.py. The lines this branch adds introduce no other category; the two it did introduce (response.json() returning Any and a Mapping passed where a dict is expected) were fixed before the commit

python scripts/type_discipline_gate.py --base d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62
OK: every LIT rule is within its codebase ceiling (base d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62)
python scripts/ruff_strict_gate.py --base d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62
OK: every strict rule is within its codebase ceiling (base d7bc17fa47ee9acacd441e6b3349c7aa9e1c6c62)

Owning groups through tests/integration/run.py

Both groups ran against the tip stack (INTEGRATION_PROXY_URL=http://127.0.0.1:4000, INTEGRATION_UPSTREAM_URL=http://127.0.0.1:8190, DATABASE_URL=...integration_tip, REDIS_HOST=127.0.0.1, LITELLM_LOCAL_MODEL_COST_MAP=True). run.py starts its own owned proxies for the tests that need one

python tests/integration/run.py accounting --results /home/ubuntu/sail-int/results/accounting2
============= 2 failed, 54 passed, 3 warnings in 721.92s (0:12:01) =============
FAILED tests/integration/spend/test_cache_and_quota.py::test_scheduled_budget_reset_reconnects_after_db_transport_failure_and_unblocks_key
FAILED tests/integration/spend/test_cache_and_quota.py::test_in_flight_count_tokens_does_not_reserve_key_budget_away_from_a_completion

python tests/integration/run.py providers --results /home/ubuntu/sail-int/results/providers
====== 2 failed, 149 passed, 1 skipped, 103 warnings in 365.32s (0:06:05) ======
FAILED tests/integration/providers/test_databricks_oauth_wire.py::test_databricks_ai_gateway_api_base_requests_oauth_token_from_workspace_origin
FAILED tests/integration/routing/test_redis_recovery.py::test_owned_redis_outage_recovers_requests_and_real_response_cache

Every test this branch adds passed in its group (the audio test is the documented BUG: skip). The four failures are in files this branch does not touch and none of them reaches the scripted scenario route that the upstream.py change covers: two are owned-proxy and owned-Redis lifecycle tests (Owned proxy exited before readiness, owned Redis outage), one is the Gemini countTokens wire peer, one is the Databricks OAuth wire peer. The two accounting failures were re-run with the pristine tip upstream.py (the file checked out from d7bc17fa, upstream restarted) and failed the same way (2 failed in 42.02s), so they are independent of this branch

The first accounting run (before the observation scoping) also failed test_repeated_count_tokens_on_budgeted_key_does_not_reserve_budget_or_block_later_completion: the scripted catch-all recorded the proxy's POST /v1/responses/input_tokens probe, which that test asserts the upstream never sees. The observation in Provider.scripted is therefore recorded only after the request resolves to a registered scenario, which is the only case the new tests read. That test passed in the second run