From 9451a72e8945bdb43e038291954dc4b9ea9387c2 Mon Sep 17 00:00:00 2001 From: ryan-crabbe-berri Date: Tue, 26 May 2026 17:34:19 -0700 Subject: [PATCH] Patches for v1.87.0-rc.1 (#28915) * fix(proxy): strip LiteLLM policy tracking from OpenAI batch metadata (#28425) * fix(proxy): strip LiteLLM policy tracking from OpenAI batch metadata Batch create was failing with `Invalid type for 'metadata.applied_policies': expected a string, but got an array instead` whenever a policy attachment matched the request. The policy engine helpers wrote `applied_policies`, `applied_guardrails`, and `policy_sources` into `data["metadata"]` unconditionally, and `/v1/batches` forwarded that dict straight to OpenAI, which only accepts string values. - Route proxy-internal tracking into `litellm_metadata` for batch/file routes via a shared `_get_or_create_proxy_metadata_bucket` helper. - Sanitize `data["metadata"]` in `create_batch` to drop known internal keys and non-string values before building the OpenAI request. - Cover both behaviors with unit + endpoint tests. Co-authored-by: Cursor * fix(proxy): merge metadata buckets for batch policy response headers Ensure get_logging_caching_headers reads both metadata and litellm_metadata so policy/guardrail headers are emitted on batch routes with user metadata, and log dropped non-string OpenAI metadata at debug level. Co-authored-by: Cursor --------- Co-authored-by: Cursor * fix(model-edit): allow clearing custom pricing on wildcard models (#28719) * fix(model-edit): allow clearing custom input/output cost on wildcard deployments A user-set pricing override on a `/model/*` wildcard deployment could not be removed: clearing the Input/Output Cost fields in the UI succeeded visually, but the next read still showed the old values because both `litellm_params` and `model_info` (mirrored via `SPECIAL_MODEL_INFO_PARAMS`) retained the original rates. UI: when the pricing field is touched but left empty, send `null` instead of dropping it from the payload so the backend sees the clear intent. The cache-read-cost fallback now guards against `null` as well as `undefined` so a cleared input cost cannot silently wipe the cache-read override. Backend: `update_db_model` honors explicit-null clears, but ONLY for `SPECIAL_MODEL_INFO_PARAMS` (the 4 pricing fields). Restricting the null-clear path prevents a team-scoped caller from using this codepath to null out privileged fields like `team_id` or access groups. Tests cover both clear paths (`litellm_params` and `model_info`), the SPECIAL_MODEL_INFO_PARAMS mirror, PATCH semantics for omitted fields, and the security guard that non-pricing nulls don't reach the merged dict. Resolves LIT-3250 * fix(model-edit): run null-clears after both merges, not interleaved The previous version cleared `model_info` from inside the litellm_params merge block, but the subsequent `model_info.update(...)` re-injected the old pricing because the UI's PATCH carries the full model_info blob with the stale values still in it. Move the explicit-null clear pass to after both merges so a model_info passthrough cannot resurrect cleared fields. Adds a regression test for the realistic UI submit shape (both blobs in the patch, model_info still holding the old pricing). * test(e2e): clear-custom-pricing flow with create/delete cleanup Covers the dashboard model edit form's pricing-clear flow end-to-end: seeds a deployment with custom input/output pricing, drives the UI to clear both fields, asserts the outgoing PATCH sends explicit nulls, and confirms via /v2/model/info that the override is gone from both litellm_params and model_info. The dashboard DB persists across this suite, so beforeEach creates a uniquely-named deployment and afterEach POSTs /model/delete to leave the DB clean regardless of test outcome. * fix(model-edit): extend pricing clear to cache_read and cache_write costs Pre-existing parallel of the wildcard input/output cost bug: cleared cache_read_input_token_cost and cache_creation_input_token_cost overrides silently persisted because the UI omitted the key (delete or fallback) and the backend null-clear allowlist did not cover them. - types/router.py: add cache_read_input_token_cost and cache_creation_input_token_cost to SPECIAL_MODEL_INFO_PARAMS, so they are mirrored between litellm_params and model_info by Deployment.__init__ and honoured by the null-clear loop in update_db_model. - model_info_view.tsx: emit explicit null for touched-but-empty cache_read and cache_write fields. Preserve the input_cost->cache_read mirror only when cache_read itself was not touched. - model_management_endpoints.py: update the allowlist comment. - Tests: three new unit tests for cache clear paths and a preserve check; the e2e spec now seeds, clears, and asserts null PATCH + key-absence for all four pricing fields. --------- Co-authored-by: Shivam Rawat Co-authored-by: Cursor --- litellm/proxy/batches_endpoints/endpoints.py | 6 + litellm/proxy/common_utils/callback_utils.py | 128 ++++++-- .../model_management_endpoints.py | 27 ++ litellm/types/router.py | 2 + .../proxy/common_utils/test_callback_utils.py | 47 +++ .../test_model_management_endpoints.py | 299 ++++++++++++++++++ tests/test_litellm/proxy/test_batch_expiry.py | 71 +++++ .../modelsPage/clearCustomPricing.spec.ts | 177 +++++++++++ .../src/components/model_info_view.tsx | 37 ++- 9 files changed, 754 insertions(+), 40 deletions(-) create mode 100644 ui/litellm-dashboard/e2e_tests/tests/modelsPage/clearCustomPricing.spec.ts diff --git a/litellm/proxy/batches_endpoints/endpoints.py b/litellm/proxy/batches_endpoints/endpoints.py index 166ef7a66d0..85165709957 100644 --- a/litellm/proxy/batches_endpoints/endpoints.py +++ b/litellm/proxy/batches_endpoints/endpoints.py @@ -15,6 +15,9 @@ from litellm.batches.main import CancelBatchRequest, RetrieveBatchRequest from litellm.proxy._types import * from litellm.proxy.auth.user_api_key_auth import user_api_key_auth from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing +from litellm.proxy.common_utils.callback_utils import ( + sanitize_openai_provider_metadata, +) from litellm.proxy.common_utils.http_parsing_utils import _read_request_body from litellm.proxy.common_utils.openai_endpoint_utils import ( get_custom_llm_provider_from_request_headers, @@ -120,6 +123,9 @@ async def create_batch( # noqa: PLR0915 or get_custom_llm_provider_from_request_headers(request=request) or "openai" ) + if isinstance(data.get("metadata"), dict): + data["metadata"] = sanitize_openai_provider_metadata(data["metadata"]) + _create_batch_data = LiteLLMBatchCreateRequest(**data) # Apply team-level batch output expiry enforcement diff --git a/litellm/proxy/common_utils/callback_utils.py b/litellm/proxy/common_utils/callback_utils.py index 4995752d441..c5b97db07a0 100644 --- a/litellm/proxy/common_utils/callback_utils.py +++ b/litellm/proxy/common_utils/callback_utils.py @@ -409,11 +409,15 @@ def get_remaining_tokens_and_requests_from_request_data(data: Dict) -> Dict[str, def get_logging_caching_headers(request_data: Dict) -> Optional[Dict]: - _metadata = request_data.get("metadata", None) - if not _metadata: - _metadata = request_data.get("litellm_metadata", None) - if not isinstance(_metadata, dict): - _metadata = {} + _metadata: Dict = {} + metadata_bucket = request_data.get("metadata") + litellm_metadata_bucket = request_data.get("litellm_metadata") + if isinstance(metadata_bucket, dict): + _metadata.update(metadata_bucket) + if isinstance(litellm_metadata_bucket, dict): + # Batch/file routes store proxy tracking in litellm_metadata while + # user-facing metadata stays in metadata; merge both for headers. + _metadata.update(litellm_metadata_bucket) headers = {} if "applied_guardrails" in _metadata: headers["x-litellm-applied-guardrails"] = ",".join( @@ -452,19 +456,103 @@ def get_logging_caching_headers(request_data: Dict) -> Optional[Dict]: return headers +def get_metadata_variable_name_from_kwargs( + kwargs: dict, +) -> Literal["metadata", "litellm_metadata"]: + """ + Helper to return what the "metadata" field should be called in the request data + + - New endpoints return `litellm_metadata` + - Old endpoints return `metadata` + + Context: + - LiteLLM used `metadata` as an internal field for storing metadata + - OpenAI then started using this field for their metadata + - LiteLLM is now moving to using `litellm_metadata` for our metadata + """ + return "litellm_metadata" if "litellm_metadata" in kwargs else "metadata" + + +LITELLM_PROXY_INTERNAL_METADATA_KEYS = frozenset( + { + "applied_policies", + "applied_guardrails", + "policy_sources", + "guardrails", + "guardrail_config", + "_guardrail_pipelines", + "_pipeline_managed_guardrails", + "disable_global_guardrails", + "disable_global_guardrail", + "opted_out_global_guardrails", + "pillar_response_headers", + "_pillar_response_headers_trusted", + "pillar_flagged", + "pillar_scanners", + "pillar_evidence", + "pillar_evidence_truncated", + "pillar_session_id_response", + "standard_logging_object", + "proxy_server_request", + "secret_fields", + } +) + + +def _get_or_create_proxy_metadata_bucket( + request_data: Dict, +) -> tuple[Literal["metadata", "litellm_metadata"], dict]: + """ + Return the proxy-internal metadata bucket for this request. + + Batch/file routes store proxy state in ``litellm_metadata`` so the OpenAI + ``metadata`` field can remain provider-safe (string values only). + """ + metadata_key = get_metadata_variable_name_from_kwargs(request_data) + metadata_bucket = request_data.get(metadata_key) + if not isinstance(metadata_bucket, dict): + metadata_bucket = {} + request_data[metadata_key] = metadata_bucket + return metadata_key, metadata_bucket + + +def sanitize_openai_provider_metadata( + metadata: Optional[Dict[str, Any]], +) -> Optional[Dict[str, str]]: + """ + Keep only provider-safe OpenAI metadata entries (string keys -> string values). + + Strips LiteLLM proxy-internal tracking fields that must not be forwarded to + OpenAI batch/file APIs. + """ + if not metadata: + return metadata + sanitized: Dict[str, str] = {} + for key, value in metadata.items(): + if key in LITELLM_PROXY_INTERNAL_METADATA_KEYS: + continue + if isinstance(value, str): + sanitized[key] = value + else: + verbose_proxy_logger.debug( + "sanitize_openai_provider_metadata: dropping key %r with non-string value of type %s", + key, + type(value).__name__, + ) + return sanitized or None + + def add_guardrail_to_applied_guardrails_header( request_data: Dict, guardrail_name: Optional[str] ): if guardrail_name is None: return - _metadata = request_data.get("metadata", None) or {} + _, _metadata = _get_or_create_proxy_metadata_bucket(request_data) if "applied_guardrails" in _metadata: if guardrail_name not in _metadata["applied_guardrails"]: _metadata["applied_guardrails"].append(guardrail_name) else: _metadata["applied_guardrails"] = [guardrail_name] - # Ensure metadata is set back to request_data (important when metadata didn't exist) - request_data["metadata"] = _metadata def add_policy_to_applied_policies_header( @@ -478,14 +566,12 @@ def add_policy_to_applied_policies_header( """ if policy_name is None: return - _metadata = request_data.get("metadata", None) or {} + _, _metadata = _get_or_create_proxy_metadata_bucket(request_data) if "applied_policies" in _metadata: if policy_name not in _metadata["applied_policies"]: _metadata["applied_policies"].append(policy_name) else: _metadata["applied_policies"] = [policy_name] - # Ensure metadata is set back to request_data (important when metadata didn't exist) - request_data["metadata"] = _metadata def add_policy_sources_to_metadata(request_data: Dict, policy_sources: Dict[str, str]): @@ -498,13 +584,12 @@ def add_policy_sources_to_metadata(request_data: Dict, policy_sources: Dict[str, """ if not policy_sources: return - _metadata = request_data.get("metadata", None) or {} + _, _metadata = _get_or_create_proxy_metadata_bucket(request_data) existing = _metadata.get("policy_sources", {}) if not isinstance(existing, dict): existing = {} existing.update(policy_sources) _metadata["policy_sources"] = existing - request_data["metadata"] = _metadata def add_guardrail_response_to_standard_logging_object( @@ -527,23 +612,6 @@ def add_guardrail_response_to_standard_logging_object( return standard_logging_object -def get_metadata_variable_name_from_kwargs( - kwargs: dict, -) -> Literal["metadata", "litellm_metadata"]: - """ - Helper to return what the "metadata" field should be called in the request data - - - New endpoints return `litellm_metadata` - - Old endpoints return `metadata` - - Context: - - LiteLLM used `metadata` as an internal field for storing metadata - - OpenAI then started using this field for their metadata - - LiteLLM is now moving to using `litellm_metadata` for our metadata - """ - return "litellm_metadata" if "litellm_metadata" in kwargs else "metadata" - - def process_callback( _callback: str, callback_type: str, environment_variables: dict ) -> dict: diff --git a/litellm/proxy/management_endpoints/model_management_endpoints.py b/litellm/proxy/management_endpoints/model_management_endpoints.py index f2d8ec8fb55..722fcd30033 100644 --- a/litellm/proxy/management_endpoints/model_management_endpoints.py +++ b/litellm/proxy/management_endpoints/model_management_endpoints.py @@ -51,6 +51,7 @@ from litellm.types.proxy.management_endpoints.model_management_endpoints import UpdateUsefulLinksRequest, ) from litellm.types.router import ( + SPECIAL_MODEL_INFO_PARAMS, Deployment, DeploymentTypedDict, LiteLLMParamsTypedDict, @@ -130,6 +131,32 @@ def update_db_model( updated_patch.model_info.model_dump(exclude_none=True) ) + # Honor explicit-null clears LAST, after both merges, so a model_info blob the UI + # passes through (which today re-sends the OLD pricing on every save) cannot + # silently undo a litellm_params clear via .update(). + # + # Restricted to SPECIAL_MODEL_INFO_PARAMS (input/output cost per token/character + # and cache read/write costs) so this path cannot be used to null out privileged + # model_info fields like team_id or access groups. SPECIAL_MODEL_INFO_PARAMS are + # mirrored between litellm_params and model_info by Deployment.__init__, so the + # clear propagates to both blobs. + if updated_patch.litellm_params: + for field in updated_patch.litellm_params.model_fields_set: + if ( + field in SPECIAL_MODEL_INFO_PARAMS + and getattr(updated_patch.litellm_params, field) is None + ): + merged_deployment_dict["litellm_params"].pop(field, None) # type: ignore + merged_deployment_dict.get("model_info", {}).pop(field, None) + if updated_patch.model_info: + for field in updated_patch.model_info.model_fields_set: + if ( + field in SPECIAL_MODEL_INFO_PARAMS + and getattr(updated_patch.model_info, field) is None + ): + merged_deployment_dict["model_info"].pop(field, None) # type: ignore + merged_deployment_dict.get("litellm_params", {}).pop(field, None) # type: ignore + # convert to prisma compatible format prisma_compatible_model_dict = PrismaCompatibleUpdateDBModel() diff --git a/litellm/types/router.py b/litellm/types/router.py index 6601f552b52..ef7eb05d087 100644 --- a/litellm/types/router.py +++ b/litellm/types/router.py @@ -398,6 +398,8 @@ SPECIAL_MODEL_INFO_PARAMS = [ "output_cost_per_token", "input_cost_per_character", "output_cost_per_character", + "cache_read_input_token_cost", + "cache_creation_input_token_cost", ] diff --git a/tests/test_litellm/proxy/common_utils/test_callback_utils.py b/tests/test_litellm/proxy/common_utils/test_callback_utils.py index cb30970a34e..d328d68dcd4 100644 --- a/tests/test_litellm/proxy/common_utils/test_callback_utils.py +++ b/tests/test_litellm/proxy/common_utils/test_callback_utils.py @@ -8,11 +8,14 @@ sys.path.insert( ) # Adds the parent directory to the system path from litellm.proxy.common_utils.callback_utils import ( + add_policy_to_applied_policies_header, decrypt_callback_vars, encrypt_callback_vars, + get_logging_caching_headers, initialize_callbacks_on_proxy, get_remaining_tokens_and_requests_from_request_data, normalize_callback_names, + sanitize_openai_provider_metadata, ) import litellm @@ -92,6 +95,50 @@ def test_normalize_callback_names_lowercases_strings(): ] +def test_add_policy_to_applied_policies_header_uses_litellm_metadata_bucket(): + request_data = { + "input_file_id": "file-abc123", + "litellm_metadata": {}, + } + + add_policy_to_applied_policies_header( + request_data=request_data, policy_name="global-baseline" + ) + + assert request_data["litellm_metadata"]["applied_policies"] == ["global-baseline"] + assert "applied_policies" not in request_data.get("metadata", {}) + + +def test_sanitize_openai_provider_metadata_strips_internal_tracking_fields(): + metadata = { + "customer_id": "cust-123", + "applied_policies": ["global-baseline"], + "applied_guardrails": ["pii_blocker"], + "note": 42, + } + + sanitized = sanitize_openai_provider_metadata(metadata) + + assert sanitized == {"customer_id": "cust-123"} + + +def test_get_logging_caching_headers_merges_metadata_and_litellm_metadata(): + request_data = { + "metadata": {"customer_id": "cust-123"}, + "litellm_metadata": { + "applied_policies": ["global-baseline"], + "applied_guardrails": ["pii_blocker"], + "policy_sources": {"global-baseline": "team_default"}, + }, + } + + headers = get_logging_caching_headers(request_data) + + assert headers["x-litellm-applied-policies"] == "global-baseline" + assert headers["x-litellm-applied-guardrails"] == "pii_blocker" + assert headers["x-litellm-policy-sources"] == "global-baseline=team_default" + + def test_initialize_callbacks_on_proxy_instantiates_compression_interception( monkeypatch, ): diff --git a/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py b/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py index b65f6305b77..85c7c130b36 100644 --- a/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py +++ b/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py @@ -1497,6 +1497,305 @@ class TestUpdateDBModelBlocked: assert "blocked" not in result +def _build_db_model_with_pricing(): + """Wildcard deployment with custom pricing in litellm_params; Deployment.__init__ + mirrors SPECIAL_MODEL_INFO_PARAMS into model_info, so both blobs hold the rate.""" + from litellm.types.router import Deployment, LiteLLM_Params, ModelInfo + + return Deployment( + model_name="openai/*", + litellm_params=LiteLLM_Params( + model="openai/*", + input_cost_per_token=0.000001, + output_cost_per_token=0.000002, + ), + model_info=ModelInfo(id="dep-pricing-0"), + ) + + +class TestUpdateDBModelClearPricing: + """Sending an explicit `null` for a pricing field must remove it from both + `litellm_params` and `model_info` (SPECIAL_MODEL_INFO_PARAMS are mirrored + between the two by Deployment.__init__). + + Restricted to SPECIAL_MODEL_INFO_PARAMS so non-pricing fields (e.g. team_id) + cannot be cleared via this path. + """ + + def test_clear_input_cost_removes_from_both_blobs(self): + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import updateLiteLLMParams + + result = update_db_model( + db_model=_build_db_model_with_pricing(), + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(input_cost_per_token=None) + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "input_cost_per_token" not in params + assert "input_cost_per_token" not in info + # Other pricing untouched + assert params.get("output_cost_per_token") == 0.000002 + assert info.get("output_cost_per_token") == 0.000002 + + def test_clear_output_cost_removes_from_both_blobs(self): + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import updateLiteLLMParams + + result = update_db_model( + db_model=_build_db_model_with_pricing(), + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(output_cost_per_token=None) + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "output_cost_per_token" not in params + assert "output_cost_per_token" not in info + + def test_non_null_pricing_update_still_works(self): + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import updateLiteLLMParams + + result = update_db_model( + db_model=_build_db_model_with_pricing(), + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(input_cost_per_token=0.000005) + ), + ) + + params = json.loads(result["litellm_params"]) + assert params["input_cost_per_token"] == 0.000005 + + def test_omitted_pricing_field_is_preserved(self): + """PATCH semantics: fields not in the patch keep their existing value.""" + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import updateLiteLLMParams + + result = update_db_model( + db_model=_build_db_model_with_pricing(), + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(output_cost_per_token=0.000007) + ), + ) + + params = json.loads(result["litellm_params"]) + assert params["input_cost_per_token"] == 0.000001 + assert params["output_cost_per_token"] == 0.000007 + + def test_null_on_non_pricing_field_does_not_clear(self): + """Security guard: only SPECIAL_MODEL_INFO_PARAMS can be cleared via null. + Privileged or unrelated model_info fields (e.g. team_id) must be unaffected + by the null-clearing path so a team admin can't ungate a team-scoped model. + """ + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import ( + Deployment, + LiteLLM_Params, + ModelInfo, + updateLiteLLMParams, + ) + + db_model = Deployment( + model_name="openai/*", + litellm_params=LiteLLM_Params( + model="openai/*", + input_cost_per_token=0.000001, + ), + model_info=ModelInfo(id="dep-pricing-1", team_id="team-keep-me"), + ) + + # Patch sends a null for api_base (non-SPECIAL field). Must NOT clear team_id + # or any other non-pricing field from the merged dict. + result = update_db_model( + db_model=db_model, + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(api_base=None) + ), + ) + + info = json.loads(result["model_info"]) + # Pricing still present (not part of this patch) + assert "input_cost_per_token" in info + # team_id must survive + assert info.get("team_id") == "team-keep-me" + + def test_clear_survives_model_info_passthrough_with_old_pricing(self): + """Realistic UI submit shape: the patch carries BOTH blobs. The + model_info portion still has the old pricing because the form + re-serializes the source blob. The litellm_params null must beat the + model_info merge — i.e. the clear runs after both merges, not between. + """ + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import ModelInfo, updateLiteLLMParams + + result = update_db_model( + db_model=_build_db_model_with_pricing(), + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(input_cost_per_token=None), + # The UI passes the OLD model_info blob through unchanged. + model_info=ModelInfo( + id="dep-pricing-0", + input_cost_per_token=0.000001, # stale value from the page state + ), + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "input_cost_per_token" not in params + assert ( + "input_cost_per_token" not in info + ), "model_info passthrough must not resurrect the cleared override" + + def test_clear_via_model_info_clears_both_blobs(self): + """The mirror works in the reverse direction too: nulling a pricing field + via the model_info patch should clear it from litellm_params as well.""" + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import ModelInfo + + result = update_db_model( + db_model=_build_db_model_with_pricing(), + updated_patch=updateDeployment( + model_info=ModelInfo(id="dep-pricing-0", input_cost_per_token=None) + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "input_cost_per_token" not in params + assert "input_cost_per_token" not in info + + def test_clear_cache_read_cost_removes_from_both_blobs(self): + """cache_read_input_token_cost was added to SPECIAL_MODEL_INFO_PARAMS so + the same null-clear path works for cache-read overrides.""" + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import ( + Deployment, + LiteLLM_Params, + ModelInfo, + updateLiteLLMParams, + ) + + db_model = Deployment( + model_name="openai/*", + litellm_params=LiteLLM_Params( + model="openai/*", + cache_read_input_token_cost=0.0000005, + ), + model_info=ModelInfo(id="dep-cache-read-0"), + ) + + result = update_db_model( + db_model=db_model, + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(cache_read_input_token_cost=None) + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "cache_read_input_token_cost" not in params + assert "cache_read_input_token_cost" not in info + + def test_clear_cache_write_cost_removes_from_both_blobs(self): + """cache_creation_input_token_cost was added to SPECIAL_MODEL_INFO_PARAMS so + the same null-clear path works for cache-write overrides.""" + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import ( + Deployment, + LiteLLM_Params, + ModelInfo, + updateLiteLLMParams, + ) + + db_model = Deployment( + model_name="openai/*", + litellm_params=LiteLLM_Params( + model="openai/*", + cache_creation_input_token_cost=0.000003, + ), + model_info=ModelInfo(id="dep-cache-write-0"), + ) + + result = update_db_model( + db_model=db_model, + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(cache_creation_input_token_cost=None) + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "cache_creation_input_token_cost" not in params + assert "cache_creation_input_token_cost" not in info + + def test_clear_cache_read_preserves_other_pricing(self): + """Clearing cache_read must not touch input/output cost overrides.""" + from litellm.proxy.management_endpoints.model_management_endpoints import ( + update_db_model, + ) + from litellm.types.router import ( + Deployment, + LiteLLM_Params, + ModelInfo, + updateLiteLLMParams, + ) + + db_model = Deployment( + model_name="openai/*", + litellm_params=LiteLLM_Params( + model="openai/*", + input_cost_per_token=0.000001, + output_cost_per_token=0.000002, + cache_read_input_token_cost=0.0000005, + cache_creation_input_token_cost=0.000003, + ), + model_info=ModelInfo(id="dep-cache-mixed-0"), + ) + + result = update_db_model( + db_model=db_model, + updated_patch=updateDeployment( + litellm_params=updateLiteLLMParams(cache_read_input_token_cost=None) + ), + ) + + params = json.loads(result["litellm_params"]) + info = json.loads(result["model_info"]) + assert "cache_read_input_token_cost" not in params + assert "cache_read_input_token_cost" not in info + # Other pricing untouched in both blobs + assert params["input_cost_per_token"] == 0.000001 + assert params["output_cost_per_token"] == 0.000002 + assert params["cache_creation_input_token_cost"] == 0.000003 + assert info["input_cost_per_token"] == 0.000001 + assert info["output_cost_per_token"] == 0.000002 + assert info["cache_creation_input_token_cost"] == 0.000003 + + class TestGetModelInfoWithIdBlocked: """`ProxyConfig.get_model_info_with_id` must propagate the DB-level `blocked` column into the in-memory `model_info` dict so the router filter can read it.""" diff --git a/tests/test_litellm/proxy/test_batch_expiry.py b/tests/test_litellm/proxy/test_batch_expiry.py index d63f278e715..38c4a71608d 100644 --- a/tests/test_litellm/proxy/test_batch_expiry.py +++ b/tests/test_litellm/proxy/test_batch_expiry.py @@ -178,6 +178,77 @@ class TestBatchEndpointTeamOverride: assert kwargs["output_expires_after"] == TEAM_EXPIRY +class TestBatchEndpointPolicyMetadata: + """Batch create must not forward LiteLLM policy tracking via OpenAI metadata.""" + + def test_create_batch_does_not_forward_applied_policies_metadata( + self, monkeypatch, llm_router + ): + from litellm.proxy.policy_engine.attachment_registry import ( + get_attachment_registry, + ) + from litellm.proxy.policy_engine.policy_registry import get_policy_registry + from litellm.types.proxy.policy_engine import ( + Policy, + PolicyAttachment, + PolicyGuardrails, + ) + + policy_registry = get_policy_registry() + policy_registry._policies = { + "global-baseline": Policy( + guardrails=PolicyGuardrails(add=["pii_blocker"]), + ), + } + policy_registry._initialized = True + + attachment_registry = get_attachment_registry() + attachment_registry._attachments = [ + PolicyAttachment(policy="global-baseline", scope="*"), + ] + attachment_registry._initialized = True + + _setup_proxy(monkeypatch, llm_router) + + user_key = UserAPIKeyAuth( + api_key="test-key", + team_alias="batch-team", + key_alias="batch-key", + ) + app.dependency_overrides[user_api_key_auth] = lambda: user_key + + captured_kwargs = {} + + async def mock_acreate_batch(**kwargs): + captured_kwargs.update(kwargs) + return _make_batch_response() + + monkeypatch.setattr(litellm, "acreate_batch", mock_acreate_batch) + + try: + response = client.post( + "/v1/batches", + json={ + "input_file_id": "file-abc123", + "endpoint": "/v1/chat/completions", + "completion_window": "24h", + }, + headers={"Authorization": "Bearer test-key"}, + ) + assert response.status_code == 200 + finally: + app.dependency_overrides.clear() + policy_registry._policies = {} + policy_registry._initialized = False + attachment_registry._attachments = [] + attachment_registry._initialized = False + + assert captured_kwargs.get("metadata") in (None, {}) + assert ( + "global-baseline" in captured_kwargs["litellm_metadata"]["applied_policies"] + ) + + class TestBatchEndpointTeamValidation: """Verify validation errors for malformed team metadata on batch endpoint.""" diff --git a/ui/litellm-dashboard/e2e_tests/tests/modelsPage/clearCustomPricing.spec.ts b/ui/litellm-dashboard/e2e_tests/tests/modelsPage/clearCustomPricing.spec.ts new file mode 100644 index 00000000000..d21192d237d --- /dev/null +++ b/ui/litellm-dashboard/e2e_tests/tests/modelsPage/clearCustomPricing.spec.ts @@ -0,0 +1,177 @@ +import { test, expect } from "@playwright/test"; +import { ADMIN_STORAGE_PATH } from "../../constants"; +import { Role, users } from "../../fixtures/users"; + +/** + * Regression: clearing the Input / Output / Cache Read / Cache Write Cost + * fields on a deployment with a user-set pricing override must actually remove + * the override from both `litellm_params` and `model_info`. + * + * Pre-fix, the UI sent the old pricing back on every save (the spread of + * `values.litellm_params` re-injected it), and the backend's `exclude_none=True` + * stripped any null that did make it through. End-result: the dashboard + * displayed "Saved" but the override remained in the DB. The cache fields had + * the same bug in a parallel code path and are covered here too. + */ +test.describe("Clear custom pricing on a deployment", () => { + test.use({ storageState: ADMIN_STORAGE_PATH }); + + const masterKey = users[Role.ProxyAdmin].password; + const SEED_INPUT_PER_TOKEN = 0.0000777; + const SEED_OUTPUT_PER_TOKEN = 0.0000999; + const SEED_CACHE_READ_PER_TOKEN = 0.0000333; + const SEED_CACHE_WRITE_PER_TOKEN = 0.0000555; + + // Unique-per-run name so concurrent / repeated runs don't collide on the + // shared dashboard DB. Captured here so afterEach can clean it up. + let createdModelId: string | null = null; + let modelName: string; + + test.beforeEach(async ({ page }) => { + modelName = `e2e-clear-pricing-${Date.now()}`; + const res = await page.request.post("/model/new", { + headers: { Authorization: `Bearer ${masterKey}` }, + data: { + model_name: modelName, + litellm_params: { + model: "openai/gpt-4o", + api_key: "sk-e2e-not-used", + input_cost_per_token: SEED_INPUT_PER_TOKEN, + output_cost_per_token: SEED_OUTPUT_PER_TOKEN, + cache_read_input_token_cost: SEED_CACHE_READ_PER_TOKEN, + cache_creation_input_token_cost: SEED_CACHE_WRITE_PER_TOKEN, + }, + model_info: {}, + }, + }); + expect(res.ok(), `POST /model/new for ${modelName}`).toBe(true); + const body = await res.json(); + createdModelId = body.model_info?.id ?? body.model_id; + expect(createdModelId, "model id from /model/new").toBeTruthy(); + }); + + test.afterEach(async ({ page }) => { + // The dashboard DB persists across this suite (not just per-test), so every + // model created here must be cleaned up regardless of test outcome. + if (createdModelId) { + await page.request.post("/model/delete", { + headers: { Authorization: `Bearer ${masterKey}` }, + data: { id: createdModelId }, + }); + createdModelId = null; + } + }); + + test("UI sends null for cleared pricing and backend removes the override", async ({ + page, + }) => { + // Navigate to the model detail view. + await page.goto("/ui"); + await page.getByText("Models + Endpoints").click(); + + const modelRow = page.locator("tr", { hasText: modelName }).first(); + await expect(modelRow).toBeVisible({ timeout: 15_000 }); + await modelRow.click(); + await expect(page.getByText("Back to Models").first()).toBeVisible({ + timeout: 10_000, + }); + + // Sanity: the seeded pricing is shown in the detail view (77.7000 / 99.9000 + // per 1M tokens). The dashboard renders the per-token rate × 1e6. + await expect(page.getByText("77.7000")).toBeVisible({ timeout: 10_000 }); + await expect(page.getByText("99.9000")).toBeVisible({ timeout: 10_000 }); + + // Open the edit form and clear all four pricing fields. + await page.getByRole("button", { name: "Edit Settings" }).click(); + const inputCost = page.getByPlaceholder("Enter input cost"); + const outputCost = page.getByPlaceholder("Enter output cost"); + // Both cache fields share the same placeholder ("Defaults to Input Cost if blank"), + // so disambiguate via the Form.Item id (AntD assigns the `name` prop as input id). + const cacheReadCost = page.locator("#cache_read_cost"); + const cacheWriteCost = page.locator("#cache_write_cost"); + await inputCost.waitFor({ timeout: 15_000 }); + for (const field of [inputCost, outputCost, cacheReadCost, cacheWriteCost]) { + await field.click({ clickCount: 3 }); + await page.keyboard.press("Delete"); + } + + // Capture the outgoing PATCH so we can assert the UI sends explicit nulls. + const patchPromise = page.waitForRequest( + (req) => + req.method() === "PATCH" && + req.url().includes(`/model/${createdModelId}/update`) + ); + await page.getByRole("button", { name: "Save Changes" }).click(); + const patchReq = await patchPromise; + const patchBody = JSON.parse(patchReq.postData() ?? "{}"); + expect( + patchBody.litellm_params.input_cost_per_token, + "UI sends explicit null for cleared input cost" + ).toBeNull(); + expect( + patchBody.litellm_params.output_cost_per_token, + "UI sends explicit null for cleared output cost" + ).toBeNull(); + expect( + patchBody.litellm_params.cache_read_input_token_cost, + "UI sends explicit null for cleared cache_read cost" + ).toBeNull(); + expect( + patchBody.litellm_params.cache_creation_input_token_cost, + "UI sends explicit null for cleared cache_write cost" + ).toBeNull(); + + // Success toast confirms the save was accepted. + await expect( + page.getByText("Model settings updated successfully") + ).toBeVisible({ timeout: 10_000 }); + + // Verify via the management API: the user-set rate is gone from both blobs. + // The cost-map may synthesize a default for known providers in the response, + // so the assertion is "no longer the seeded value" rather than literally + // undefined. + const infoRes = await page.request.get( + `/v2/model/info?include_team_models=true&page=1&size=100&modelId=${createdModelId}`, + { headers: { Authorization: `Bearer ${masterKey}` } } + ); + expect(infoRes.ok()).toBe(true); + const infoBody = await infoRes.json(); + const row = (infoBody.data ?? infoBody).find?.( + (m: any) => m?.model_info?.id === createdModelId + ); + expect(row, "model info row").toBeTruthy(); + + expect( + "input_cost_per_token" in row.litellm_params, + "litellm_params.input_cost_per_token key removed" + ).toBe(false); + expect( + "output_cost_per_token" in row.litellm_params, + "litellm_params.output_cost_per_token key removed" + ).toBe(false); + expect( + "cache_read_input_token_cost" in row.litellm_params, + "litellm_params.cache_read_input_token_cost key removed" + ).toBe(false); + expect( + "cache_creation_input_token_cost" in row.litellm_params, + "litellm_params.cache_creation_input_token_cost key removed" + ).toBe(false); + expect( + row.model_info.input_cost_per_token, + "model_info.input_cost_per_token no longer the seeded override" + ).not.toBe(SEED_INPUT_PER_TOKEN); + expect( + row.model_info.output_cost_per_token, + "model_info.output_cost_per_token no longer the seeded override" + ).not.toBe(SEED_OUTPUT_PER_TOKEN); + expect( + row.model_info.cache_read_input_token_cost, + "model_info.cache_read_input_token_cost no longer the seeded override" + ).not.toBe(SEED_CACHE_READ_PER_TOKEN); + expect( + row.model_info.cache_creation_input_token_cost, + "model_info.cache_creation_input_token_cost no longer the seeded override" + ).not.toBe(SEED_CACHE_WRITE_PER_TOKEN); + }); +}); diff --git a/ui/litellm-dashboard/src/components/model_info_view.tsx b/ui/litellm-dashboard/src/components/model_info_view.tsx index 5ed4c0468b8..768083be6e9 100644 --- a/ui/litellm-dashboard/src/components/model_info_view.tsx +++ b/ui/litellm-dashboard/src/components/model_info_view.tsx @@ -256,14 +256,26 @@ export default function ModelInfoView({ tags: values.tags, }; - if (form.isFieldTouched("input_cost") && values.input_cost !== undefined && values.input_cost !== null) { - updatedLitellmParams.input_cost_per_token = Number(values.input_cost) / 1_000_000; + if (form.isFieldTouched("input_cost")) { + if (values.input_cost !== undefined && values.input_cost !== null && values.input_cost !== "") { + updatedLitellmParams.input_cost_per_token = Number(values.input_cost) / 1_000_000; + } else { + // Explicit null signals the backend to remove the pricing override. + updatedLitellmParams.input_cost_per_token = null; + } } - if (form.isFieldTouched("output_cost") && values.output_cost !== undefined && values.output_cost !== null) { - updatedLitellmParams.output_cost_per_token = Number(values.output_cost) / 1_000_000; + if (form.isFieldTouched("output_cost")) { + if (values.output_cost !== undefined && values.output_cost !== null && values.output_cost !== "") { + updatedLitellmParams.output_cost_per_token = Number(values.output_cost) / 1_000_000; + } else { + updatedLitellmParams.output_cost_per_token = null; + } } - // Cache Read Cost: explicit value if provided, else fall back to input cost (when input cost touched). + // Cache Read Cost: + // - explicit value provided → use it + // - field touched but empty → explicit null (signals backend to remove override) + // - only input_cost touched → fall back to input_cost (guarded against null) if (form.isFieldTouched("cache_read_cost") || form.isFieldTouched("input_cost")) { if ( values.cache_read_cost !== undefined && @@ -271,14 +283,19 @@ export default function ModelInfoView({ values.cache_read_cost !== "" ) { updatedLitellmParams.cache_read_input_token_cost = Number(values.cache_read_cost) / 1_000_000; - } else if (updatedLitellmParams.input_cost_per_token !== undefined) { + } else if (form.isFieldTouched("cache_read_cost")) { + updatedLitellmParams.cache_read_input_token_cost = null; + } else if ( + updatedLitellmParams.input_cost_per_token !== undefined && + updatedLitellmParams.input_cost_per_token !== null + ) { updatedLitellmParams.cache_read_input_token_cost = updatedLitellmParams.input_cost_per_token; } } - // Cache Write Cost: explicit value if provided, else clear the override - // so the backend falls back to the model-level default. Sending 0 here - // would persist a zero rate even when the user intended to unset it. + // Cache Write Cost: explicit value if provided, else explicit null so the + // backend removes the override and falls back to the model-level default. + // Sending 0 here would persist a zero rate even when the user intended to unset it. if (form.isFieldTouched("cache_write_cost")) { if ( values.cache_write_cost !== undefined && @@ -287,7 +304,7 @@ export default function ModelInfoView({ ) { updatedLitellmParams.cache_creation_input_token_cost = Number(values.cache_write_cost) / 1_000_000; } else { - delete updatedLitellmParams.cache_creation_input_token_cost; + updatedLitellmParams.cache_creation_input_token_cost = null; } }