From f42ffed2bd3f5b63c5fbba397093e496472efa6c Mon Sep 17 00:00:00 2001 From: Krrish Dholakia Date: Wed, 8 Apr 2026 21:37:10 -0700 Subject: [PATCH] Litellm oss staging 04 02 2026 p1 (#25055) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(vertex_ai): support pluggable (executable) credential_source for WIF auth (#24700) The WIF credential dispatch in load_auth() only handled identity_pool and aws credential types. When credential_source.executable was present (used for Azure Managed Identity via Workload Identity Federation), it fell through to identity_pool.Credentials which rejected it with MalformedError. Add dispatch to google.auth.pluggable.Credentials for executable-type credential sources, following the same pattern as the existing identity_pool and aws helpers. Fixes authentication for Azure Container Apps → GCP Vertex AI via WIF with executable credential sources. * feat(logging): add component and logger fields to JSON logs for 3rd p… (#24447) * feat(logging): add component and logger fields to JSON logs for 3rd party filtering * Let user-supplied extra fields win over auto-generated component/logger, tighten test assertions * Feat - Add organization into the metrics metadata for org_id & org_alias (#24440) * Add org_id and org_alias label names to Prometheus metric definitions * Add user_api_key_org_alias to StandardLoggingUserAPIKeyMetadata * Populate user_api_key_org_alias in pre-call metadata * Pass org_id and org_alias into per-request Prometheus metric labels * Add test for org labels on per-request Prometheus metrics * chore: resolve test mockdata * Address review: populate org_alias from DB view, add feature flag, use .get() for org metadata * Add org labels to failure path and verify flag behavior in test * Fix test: build flag-off enum_values without org fields * Gate org labels behind feature flag in get_labels() instead of static metric lists * Scope org label injection to metrics that carry team context, remove orphaned budget label defs, add test teardown * Use explicit metric allowlist for org label injection instead of team heuristic * Fix duplicate org label guard, move _org_label_metrics to class constant * Reset custom_prometheus_metadata_labels after duplicate label assertion * fix: emit org labels by default, remove flag, fix missing org_alias in all metadata paths * fix: emit org labels by default, no opt-in flag required * fix: write org_alias to metadata unconditionally in proxy_server.py * fix: 429s from batch creation being converted to 500 (#24703) * add us gov models (#24660) * add us gov models * added max tokens * Litellm dev 04 02 2026 p1 (#25052) * fix: replace hardcoded url * fix: Anthropic web search cost not tracked for Chat Completions The ModelResponse branch in response_object_includes_web_search_call() only checked url_citation annotations and prompt_tokens_details, missing Anthropic's server_tool_use.web_search_requests field. This caused _handle_web_search_cost() to never fire for Anthropic Claude models. Also routes vertex_ai/claude-* models to the Anthropic cost calculator instead of the Gemini one, since Claude on Vertex uses the same server_tool_use billing structure as the direct Anthropic API. Co-Authored-By: Claude Opus 4.6 --------- Co-authored-by: Claude Opus 4.6 * fix(anthropic): pass logging_obj to client.post for litellm_overhead_time_ms (#24071) When LITELLM_DETAILED_TIMING=true, litellm_overhead_time_ms was null for Anthropic because the handler did not pass logging_obj to client.post(), so track_llm_api_timing could not set llm_api_duration_ms. Pass logging_obj=logging_obj at all four post() call sites (make_call, make_sync_call, acompletion, completion). Add test to ensure make_call passes logging_obj to client.post. Made-with: Cursor * sap - add additional parameters for grounding - additional parameter for grounding added for the sap provider * sap - fix models * (sap) add filtering, masking, translation SAP GEN AI Hub modules * (sap) add tests and docs for new SAP modules * (sap) add support of multiple modules config * (sap) code refactoring * (sap) rename file * test(): add safeguard tests * (sap) update tests * (sap) update docs, solve merge conflict in transformation.py * (sap) linter fix * (sap) Align embedding request transformation with current API * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) mock commit * (sap) run black formater * (sap) add literals to models, add negative tests, fix test for tool transformation * (sap) fix formating * (sap) fix models * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) commit for rerun bot review * (sap) minor improve * (sap) fix after bot review * (sap) lint fix * docs(sap): update documentation * fix(sap): change creds priority * fix(sap): change creds priority * fix(sap): fix sap creds unit test * fix(sap): linter fix * fix(sap): linter fix * linter fix * (sap) update logic of fetching creds, add additional tests * (sap) clean up code * (sap) fix after review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) add a possibility to put the service key by both variants * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) update test * (sap) update service key resolve function * (sap) run black formater * (sap) fix validate credentials, add negative tests for credential fetching * (sap) fix validate credentials, add negative tests for credential fetching * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) lint fix * (sap) lint fix * feat: support service_tier in gemini * chore: add a service_tier field mapping from openai to gemini * fix: use x-gemini-service-tier header in response * docs: add service_tier to gemini docs * chore: add defaut/standard mapping, and some tests * chore: tidying up some case insensitivity * chore: remove unnecessary guard * fix: remove redundant test file * fix: handle 'auto' case-insensitively * fix: return service_tier on final steamed chunk * chore: black * feat: enable supports_service_tier to gemini models * Fix get_standard_logging_metadata tests * Fix test_get_model_info_bedrock_models * Fix test_get_model_info_bedrock_models * Fix remaining tests * Fix mypy issues * Fix tests * Fix merge conflicts * Fix code qa * Fix code qa * Fix code qa * Fix greptile review --------- Co-authored-by: michelligabriele Co-authored-by: Josh <36064836+J-Byron@users.noreply.github.com> Co-authored-by: mubashir1osmani Co-authored-by: Claude Opus 4.6 Co-authored-by: milan-berri Co-authored-by: Alperen Kömürcü Co-authored-by: Vasilisa Parshikova Co-authored-by: Lin Xu Co-authored-by: Mark McDonald Co-authored-by: Sameer Kankute --- docs/my-website/docs/providers/gemini.md | 16 +- docs/my-website/docs/providers/sap.md | 267 +++++++- .../pagerduty/pagerduty.py | 2 + litellm/_logging.py | 6 + .../providers/pydantic_ai_agents/config.py | 6 +- litellm/integrations/prometheus.py | 7 + litellm/litellm_core_utils/litellm_logging.py | 2 + .../llm_cost_calc/tool_call_cost_tracking.py | 12 +- litellm/llms/__init__.py | 12 + litellm/llms/anthropic/chat/handler.py | 21 +- litellm/llms/gemini/chat/transformation.py | 1 + litellm/llms/sap/__init__.py | 0 litellm/llms/sap/chat/models.py | 623 +++++++++++++++++- litellm/llms/sap/chat/transformation.py | 230 +++++-- litellm/llms/sap/credentials.py | 379 ++++++++--- litellm/llms/sap/embed/transformation.py | 42 +- .../llms/vertex_ai/gemini/transformation.py | 10 + .../vertex_and_google_ai_studio_gemini.py | 230 ++++--- litellm/llms/vertex_ai/vertex_llm_base.py | 16 + ...odel_prices_and_context_window_backup.json | 123 ++-- litellm/proxy/_types.py | 1 + litellm/proxy/litellm_pre_call_utils.py | 1 + litellm/proxy/proxy_server.py | 21 + litellm/proxy/utils.py | 6 +- litellm/types/integrations/prometheus.py | 28 +- litellm/types/llms/vertex_ai.py | 1 + litellm/types/utils.py | 1 + model_prices_and_context_window.json | 123 ++-- .../test_prometheus_logging_callbacks.py | 22 + tests/proxy_unit_tests/test_proxy_utils.py | 47 ++ .../test_prometheus_client_ip_user_agent.py | 2 + .../test_prometheus_user_team_metrics.py | 55 ++ .../chat/test_anthropic_chat_handler.py | 33 +- .../llms/sap/chat/test_sap_tool_parameters.py | 3 +- .../llms/sap/chat/test_sap_transformation.py | 564 ++++++++++++++++ .../embed/test_sap_embed_transformation.py | 97 +++ .../llms/sap/test_sap_fetch_creds.py | 142 ++++ .../test_vertex_ai_gemini_transformation.py | 18 + ...test_vertex_and_google_ai_studio_gemini.py | 115 ++++ .../llms/vertex_ai/test_vertex_llm_base.py | 79 +++ tests/test_litellm/test_logging.py | 66 ++ 41 files changed, 3041 insertions(+), 389 deletions(-) create mode 100644 litellm/llms/sap/__init__.py create mode 100644 tests/test_litellm/llms/sap/chat/test_sap_transformation.py create mode 100644 tests/test_litellm/llms/sap/embed/test_sap_embed_transformation.py create mode 100644 tests/test_litellm/llms/sap/test_sap_fetch_creds.py diff --git a/docs/my-website/docs/providers/gemini.md b/docs/my-website/docs/providers/gemini.md index 87ab5ad40f4..a60dc3323d1 100644 --- a/docs/my-website/docs/providers/gemini.md +++ b/docs/my-website/docs/providers/gemini.md @@ -65,14 +65,13 @@ response = completion( - modalities - reasoning_content - audio (for TTS models only) +- service_tier **Anthropic Params** - thinking (used to set max budget tokens across anthropic/gemini models) [**See Updated List**](https://github.com/BerriAI/litellm/blob/main/litellm/llms/gemini/chat/transformation.py#L70) - - ## Usage - Thinking / `reasoning_content` LiteLLM translates OpenAI's `reasoning_effort` to Gemini's `thinking` parameter. [Code](https://github.com/BerriAI/litellm/blob/620664921902d7a9bfb29897a7b27c1a7ef4ddfb/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py#L362) @@ -298,6 +297,19 @@ curl http://0.0.0.0:4000/v1/chat/completions \ +## Usage - `service_tier` + +LiteLLM propagates OpenAI's `service_tier` parameter to Gemini, and also extracts it from the response headers (`x-gemini-service-tier`) into `model_response.service_tier`. + +| OpenAI `service_tier` | Gemini `service_tier` | Notes | +| --------------------- | --------------------- | ----- | +| `"auto"` | `"priority"` | LiteLLM maps OpenAI's `"auto"` to Gemini's `"priority"` tier, as `priority` will fall back on Gemini. | +| `"flex"` | `"flex"` | Direct mapping. | +| `"priority"` | `"priority"` | Direct mapping. | +| `"default"` | `"standard"` | LiteLLM maps `"default"` to `"standard"`. | +| Any other value | Passed as-is (lowercased) | Values are case-insensitive and normalized to lowercase. | + +On the response, LiteLLM maps `"standard"` back to `"default"` for the Gemini API. ## Text-to-Speech (TTS) Audio Output diff --git a/docs/my-website/docs/providers/sap.md b/docs/my-website/docs/providers/sap.md index 16f30a2e99c..3877cb6ef19 100644 --- a/docs/my-website/docs/providers/sap.md +++ b/docs/my-website/docs/providers/sap.md @@ -55,24 +55,33 @@ pip install litellm ``` ### Step 2: Set Your Credentials + + Choose **one** of these authentication methods: + +> **Breaking change**: credential resolution is "first-source-wins" +> +> Credential resolution no longer merges individual fields across sources. +> +> Resolution order is: +`kwargs` → `service key` → `env (AICORE_*)` → `config` → `VCAP service` +> +> **Important behavior:** once LiteLLM finds *any* credential value in a source, it takes **all** credentials from that source exclusively (except `resource_group`, which may still be resolved separately). -Choose **one** of these authentication methods: + + - - +The simplest approach - paste your entire service key as a single environment variable. -The simplest approach - paste your entire service key as a single environment variable. The service key must be wrapped in a `credentials` object: +> **Note:** the service key no more needs to be wrapped in a "credentials" key. ```bash export AICORE_SERVICE_KEY='{ - "credentials": { "clientid": "your-client-id", "clientsecret": "your-client-secret", "url": "https://.authentication.sap.hana.ondemand.com", "serviceurls": { "AI_API_URL": "https://api.ai..aws.ml.hana.ondemand.com" } - } }' export AICORE_RESOURCE_GROUP="default" ``` @@ -220,6 +229,17 @@ model="sap/gemini-2.5-pro" # Incorrect - missing prefix model="gpt-4o" # ❌ Won't work ``` +3. **Environment variables** - Set the following list of credentials in .env file +
+AICORE_AUTH_URL = "https://* * * .authentication.sap.hana.ondemand.com/oauth/token",
+AICORE_CLIENT_ID  = " *** ",
+AICORE_CLIENT_SECRET = " *** ",
+AICORE_RESOURCE_GROUP = " *** ",
+AICORE_BASE_URL = "https://api.ai.***.cfapps.sap.hana.ondemand.com/v2"
+
+ +Other credential configuration options are also available. For more information, see the [SAP AI Core Documentation](https://help.sap.com/doc/generative-ai-hub-sdk/CLOUD/en-US/_reference/README_sphynx.html#configuration). +## Usage - LiteLLM Python SDK ### Proxy Usage @@ -506,6 +526,241 @@ response = embedding( print(response.data[0]["embedding"]) # Vector representation ``` +### Additional Modules +The SAP Gen AI Hub includes additional modules for advanced use cases: +- [Grounding](https://help.sap.com/docs/sap-ai-core/generative-ai/grounding-035c455a5a424697b60f4a24b6d791fe?locale=en-US) +- [Translation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US) +- [Data Masking](https://help.sap.com/docs/sap-ai-core/generative-ai/data-masking-d9a54d9ca54b40beacbd24e1663ec3b4?locale=en-US) +- [Content Filtering](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US) + +#### Grounding +Grounding is a service designed to handle data-related tasks, such as grounding and retrieval, using vector databases. It provides specialized data retrieval through these databases, grounding the retrieval process with your own external and context-relevant data. Grounding combines generative AI capabilities with the ability to use real-time, precise data to improve decision-making and business operations for specific AI-driven business solutions. +##### Prerequisites +To use the Grounding module in the orchestration pipeline, you need to prepare the knowledge base in advance. + +Generative AI hub offers multiple options for users to provide data (prepare a knowledge base): +- For Option 1: Upload the documents to a supported data repository and run the data pipeline to vectorize the documents. +- For Option 2: Provide the chunks of document via Vector API directly. + +To use grounding, choose from one of the following options. + +Usage example: +```python showLineNumbers title="Grounding Example" +from litellm import completion + +grounding_config = { + 'type': 'document_grounding_service', + 'config': { + 'filters': [ + {'id': 's3-docs', + 'data_repository_type': 'vector', + 'search_config': {'max_chunk_count': 2}, + 'data_repositories': ['012345-6789-0123-4567-890123456789'] + } + ], + 'placeholders': {'input': ['user_query'], 'output': 'grounding_response'}, + 'metadata_params': ['source', 'webUrl', 'title', 'mimeType', 'fileSuffix'] + } +} + +response = completion(model="sap/gpt-4o", + messages=[ + {"content":"""Facility Solutions Company provides services to luxury residential complexes, + apartments, individual homes, and commercial properties such as office buildings, retail + spaces, industrial facilities, and educational institutions. Customers are encouraged to + reach out with maintenance requests, service deficiencies, follow-ups, or any issues they + need by email.""", "role": "system"}, + {"content":"""You are a helpful assistant for any queries for answering questions. + Answer the request by providing relevant answers that fit to the request. + Request: {{ ?user_query }} + Context:{{ ?grounding_response }}""", "role": "user"} + ], + placeholder_values={"user_query": "Is there a complaint?"}, + grounding=grounding_config + ) +print(response.choices[0].message.content) +``` +For more information about all available grounding configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/using-grounding-module-e1c4dd100dfb42ab890e1d95f3516187?locale=en-US). + +#### Translation +The translation module allows you to translate LLM text prompts into a chosen target language. + +```python showLineNumbers title="Translation Example" +from litellm import completion + +translation_config = { + 'input': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'en-US', + 'target_language': 'de-DE'} + }, + 'output': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'de-DE', + 'target_language': 'fr-FR'} + } +} + +response = completion(model="sap/gpt-4o", + messages=[{"role": "user", "content": "Hello world!"}], + translation=translation_config) + +print(response.choices[0].message.content) +``` +For more information about all available translation configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US) + +#### Data Masking +The data masking module serves to anonymize or pseudonymize personally identifiable information from the input for selected entities. + +```python showLineNumbers title="Data Masking Example" +from litellm import completion, embedding +masking_config = { + 'providers': + [ + { + 'type': 'sap_data_privacy_integration', + 'method': 'anonymization', + 'entities': [ + {'type': 'profile-address'}, + {'type': 'profile-email'}, + {'type': 'profile-phone'}, + {'type': 'profile-person'}, + {'type': 'profile-location'} + ] + } + ] + } + +mock_cv = "some text with personal information" + +response = completion(model="sap/gpt-4o", + messages=[{"role": "user", "content": "Give a one sentence summary of the CV. CV: {{?cv}}?"}], + placeholder_values={"cv": mock_cv}, + masking=masking_config) +print(response.choices[0].message.content) + +# Data masking module also available for embedding +response = embedding(model="sap/text-embedding-3-small", + input=mock_cv, + masking=masking_config) +print(response.data[0]) +``` +For more information about all available data masking configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/enhancing-model-consumption-with-data-masking-66ad6f469afc4c2cbaa91a27a33f7b21?locale=en-US) + + + + + +#### Content Filtering +The content filtering module allows you to filter input and output based on content safety criteria. + +The module supports two services: +* Azure Content Safety +* Llama Guard 3 + +```python showLineNumbers title="Content Filtering Example" +from litellm import completion + +filtering_config_azure = { + 'input': + { + 'filters': + [ + {'type': 'azure_content_safety', + 'config': + {'hate': 0, + 'sexual': 0, + 'violence': 0, + 'self_harm': 0 + } + } + ] + }, + 'output': + { + 'filters': + [ + {'type': 'azure_content_safety', + 'config': {'hate': 0, + 'sexual': 0, + 'violence': 0, + 'self_harm': 0 + } + } + ] + } +} + +response = completion(model="sap/gpt-4o", + messages=[{"role": "user", "content": "Hello world!"}], + filtering=filtering_config_azure) +print(response.choices[0].message.content) +# The model responds normally because the content does not violate any safety rules. + +try: + response = completion(model="sap/gpt-4o", + messages=[{"role": "user", "content": "I hate you"}], + filtering=filtering_config_azure) +except Exception as e: + print(e) + # The service raises an error: + # "Input Filter: Content filtered due to safety violations. Please modify the prompt and try again." +``` +For more information about all available content filtering configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US) + +#### List of modules configuration for fallback +SAP GEN AI Hub supports a fallback mechanism for handling errors. This mechanism allows you to specify a list of fallback modules to use in case of errors. The fallback modules should contain all parameters that are required for configuring the request. + +Required parameters: +- `model` +- `messages` + +Optional parameters: +- `filtering` +- `grounding` +- `translation` +- `masking` +- `tools` + +- and any of model's specific parameters. + + +```python showLineNumbers title="Fallback Example" +from litellm import completion + +translation_config = { + 'input': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'en-US', + 'target_language': 'de-DE'} + }, + 'output': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'de-DE', + 'target_language': 'fr-FR'} + } +} + +response = completion(model="sap/gpt-4o", + messages=[{"role": "user", "content": "Hello world!"}], + translation=translation_config, + fallback_sap_modules=[{ + "model":"sap/gemini-2.5-flash", + "messages":[{"role": "user", "content": "Hello world!"}], + "translation":translation_config + }]) + +# In case of error with the first configuration (model gpt-4o), the fallback module is used. + +print(response.choices[0].message.content) + +``` + + ## Reference ### Supported Parameters diff --git a/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py b/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py index de02a0c4dab..12fdaeb6a81 100644 --- a/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py +++ b/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py @@ -114,6 +114,7 @@ class PagerDutyAlerting(SlackAlerting): user_api_key_max_budget=_meta.get("user_api_key_max_budget"), user_api_key_budget_reset_at=_meta.get("user_api_key_budget_reset_at"), user_api_key_org_id=_meta.get("user_api_key_org_id"), + user_api_key_org_alias=_meta.get("user_api_key_org_alias"), user_api_key_team_id=_meta.get("user_api_key_team_id"), user_api_key_project_id=_meta.get("user_api_key_project_id"), user_api_key_project_alias=_meta.get("user_api_key_project_alias"), @@ -196,6 +197,7 @@ class PagerDutyAlerting(SlackAlerting): else None ), user_api_key_org_id=user_api_key_dict.org_id, + user_api_key_org_alias=user_api_key_dict.organization_alias, user_api_key_team_id=user_api_key_dict.team_id, user_api_key_project_id=user_api_key_dict.project_id, user_api_key_project_alias=user_api_key_dict.project_alias, diff --git a/litellm/_logging.py b/litellm/_logging.py index 62283f6f65a..7824fcfa675 100644 --- a/litellm/_logging.py +++ b/litellm/_logging.py @@ -243,6 +243,12 @@ class JsonFormatter(Formatter): if key not in _STANDARD_RECORD_ATTRS and key not in json_record: json_record[key] = value + # Set component/logger only if not already supplied via extra={...} + if "component" not in json_record: + json_record["component"] = record.name + if "logger" not in json_record: + json_record["logger"] = f"{record.filename}:{record.lineno}" + if record.exc_info: json_record["stacktrace"] = record.exc_text or self.formatException( record.exc_info diff --git a/litellm/a2a_protocol/providers/pydantic_ai_agents/config.py b/litellm/a2a_protocol/providers/pydantic_ai_agents/config.py index 46253bbcf78..2f16779cc9f 100644 --- a/litellm/a2a_protocol/providers/pydantic_ai_agents/config.py +++ b/litellm/a2a_protocol/providers/pydantic_ai_agents/config.py @@ -21,11 +21,11 @@ class PydanticAIProviderConfig(BaseA2AProviderConfig): request_id: str, params: Dict[str, Any], api_base: Optional[str] = None, - **kwargs, + **kwargs: Any, ) -> Dict[str, Any]: """Handle non-streaming request to Pydantic AI agent.""" - if not api_base: - raise ValueError("api_base is required for Pydantic AI agents") + if api_base is None: + raise ValueError("api_base is required for PydanticAIProviderConfig") return await PydanticAIHandler.handle_non_streaming( request_id=request_id, params=params, diff --git a/litellm/integrations/prometheus.py b/litellm/integrations/prometheus.py index fb5fc253ae4..c395987695b 100644 --- a/litellm/integrations/prometheus.py +++ b/litellm/integrations/prometheus.py @@ -1031,6 +1031,9 @@ class PrometheusLogger(CustomLogger): user_api_key_org_id = standard_logging_payload["metadata"].get( "user_api_key_org_id" ) + user_api_key_org_alias = standard_logging_payload["metadata"].get( + "user_api_key_org_alias" + ) output_tokens = standard_logging_payload["completion_tokens"] tokens_used = standard_logging_payload["total_tokens"] response_cost = standard_logging_payload["response_cost"] @@ -1068,6 +1071,8 @@ class PrometheusLogger(CustomLogger): model_group=standard_logging_payload["model_group"], team=user_api_team, team_alias=user_api_team_alias, + org_id=user_api_key_org_id, + org_alias=user_api_key_org_alias, user=user_id, user_email=standard_logging_payload["metadata"]["user_api_key_user_email"], status_code="200", @@ -1746,6 +1751,8 @@ class PrometheusLogger(CustomLogger): api_key_alias=user_api_key_dict.key_alias, team=user_api_key_dict.team_id, team_alias=user_api_key_dict.team_alias, + org_id=user_api_key_dict.org_id, + org_alias=user_api_key_dict.organization_alias, requested_model=request_data.get("model", ""), status_code=str(status_code), exception_status=str(status_code), diff --git a/litellm/litellm_core_utils/litellm_logging.py b/litellm/litellm_core_utils/litellm_logging.py index 7395b65626f..7a3547bca2e 100644 --- a/litellm/litellm_core_utils/litellm_logging.py +++ b/litellm/litellm_core_utils/litellm_logging.py @@ -4754,6 +4754,7 @@ class StandardLoggingPayloadSetup: user_api_key_budget_reset_at=None, user_api_key_team_id=None, user_api_key_org_id=None, + user_api_key_org_alias=None, user_api_key_project_id=None, user_api_key_project_alias=None, user_api_key_user_id=None, @@ -5586,6 +5587,7 @@ def get_standard_logging_metadata( user_api_key_budget_reset_at=None, user_api_key_team_id=None, user_api_key_org_id=None, + user_api_key_org_alias=None, user_api_key_project_id=None, user_api_key_project_alias=None, user_api_key_user_id=None, diff --git a/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py b/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py index 4454fca3b00..8da66d4600d 100644 --- a/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py +++ b/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py @@ -322,9 +322,8 @@ class StandardBuiltInToolCostTracking: ) if has_url_citations: return True - # Fallback: Check usage object for providers that use usage instead of annotations - # (e.g., Vertex AI Gemini uses usage.prompt_tokens_details.web_search_requests) if usage is not None: + # Vertex AI Gemini uses usage.prompt_tokens_details.web_search_requests if ( hasattr(usage, "prompt_tokens_details") and usage.prompt_tokens_details is not None @@ -335,6 +334,15 @@ class StandardBuiltInToolCostTracking: and usage.prompt_tokens_details.web_search_requests is not None ): return True + # Anthropic Claude (direct API and Vertex AI) uses server_tool_use.web_search_requests. + # Without this check, Claude ModelResponse always falls through to return False + # and _handle_web_search_cost() is never called. + if ( + hasattr(usage, "server_tool_use") + and usage.server_tool_use is not None + and usage.server_tool_use.web_search_requests is not None + ): + return True return False elif isinstance(response_object, ResponsesAPIResponse): # response api explicitly includes web_search_call in the output diff --git a/litellm/llms/__init__.py b/litellm/llms/__init__.py index c73f0b22b4b..710342bbc78 100644 --- a/litellm/llms/__init__.py +++ b/litellm/llms/__init__.py @@ -34,6 +34,18 @@ def get_cost_for_web_search_request( return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage) elif custom_llm_provider.startswith("vertex_ai"): + # Anthropic Claude models on Vertex AI populate server_tool_use.web_search_requests + # (same as the direct Anthropic API), not prompt_tokens_details.web_search_requests + # (which is the Gemini field). Route claude-* models to the Anthropic calculator. + model_key: str = model_info.get("key", "") if model_info else "" + if "claude" in model_key.lower(): + from .anthropic.cost_calculation import get_cost_for_anthropic_web_search + + verbose_logger.debug( + "vertex_ai/claude model detected — routing web search cost to Anthropic calculator" + ) + return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage) + from .vertex_ai.gemini.cost_calculator import ( cost_per_web_search_request as cost_per_web_search_request_vertex_ai, ) diff --git a/litellm/llms/anthropic/chat/handler.py b/litellm/llms/anthropic/chat/handler.py index 9f2ddcae2c7..0f020c3a953 100644 --- a/litellm/llms/anthropic/chat/handler.py +++ b/litellm/llms/anthropic/chat/handler.py @@ -89,7 +89,12 @@ async def make_call( try: response = await client.post( - api_base, headers=headers, data=data, stream=True, timeout=timeout + api_base, + headers=headers, + data=data, + stream=True, + timeout=timeout, + logging_obj=logging_obj, ) except httpx.HTTPStatusError as e: error_headers = getattr(e, "headers", None) @@ -142,7 +147,12 @@ def make_sync_call( try: response = client.post( - api_base, headers=headers, data=data, stream=True, timeout=timeout + api_base, + headers=headers, + data=data, + stream=True, + timeout=timeout, + logging_obj=logging_obj, ) except httpx.HTTPStatusError as e: error_headers = getattr(e, "headers", None) @@ -266,7 +276,11 @@ class AnthropicChatCompletion(BaseLLM): try: response = await async_handler.post( - api_base, headers=headers, json=data, timeout=timeout + api_base, + headers=headers, + json=data, + timeout=timeout, + logging_obj=logging_obj, ) except Exception as e: ## LOGGING @@ -469,6 +483,7 @@ class AnthropicChatCompletion(BaseLLM): headers=headers, data=json.dumps(data), timeout=timeout, + logging_obj=logging_obj, ) except Exception as e: status_code = getattr(e, "status_code", 500) diff --git a/litellm/llms/gemini/chat/transformation.py b/litellm/llms/gemini/chat/transformation.py index 5f8dead2043..72569e5c6cd 100644 --- a/litellm/llms/gemini/chat/transformation.py +++ b/litellm/llms/gemini/chat/transformation.py @@ -91,6 +91,7 @@ class GoogleAIStudioGeminiConfig(VertexGeminiConfig): "modalities", "parallel_tool_calls", "web_search_options", + "service_tier", ] if supports_reasoning(model, custom_llm_provider="gemini"): supported_params.append("reasoning_effort") diff --git a/litellm/llms/sap/__init__.py b/litellm/llms/sap/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/litellm/llms/sap/chat/models.py b/litellm/llms/sap/chat/models.py index 8ca2aa7a690..d685d50277a 100644 --- a/litellm/llms/sap/chat/models.py +++ b/litellm/llms/sap/chat/models.py @@ -1,6 +1,8 @@ -from typing import Union, Literal +from typing import Union, Literal, Optional +from enum import Enum +import warnings -from pydantic import BaseModel, Field, field_validator +from pydantic import BaseModel, Field, field_validator, model_validator def validate_different_content(v: Union[str, dict, list]) -> str: @@ -20,7 +22,7 @@ def validate_different_content(v: Union[str, dict, list]) -> str: elif isinstance(v, str): return v raise ValueError("Content must be a string") - return v + class TextContent(BaseModel): @@ -49,6 +51,10 @@ class FunctionTool(BaseModel): parameters: dict = {"type": "object", "properties": {}} strict: bool = False + def model_dump(self, **kwargs) -> dict: + kwargs["exclude_unset"] = False + return super().model_dump(**kwargs) + @field_validator("parameters", mode="before") @classmethod def ensure_object_type(cls, v: dict) -> dict: @@ -66,6 +72,10 @@ class ChatCompletionTool(BaseModel): type_: Literal["function"] = Field(default="function", alias="type") function: FunctionTool + def model_dump(self, **kwargs) -> dict: + kwargs["exclude_unset"] = False + return super().model_dump(**kwargs) + class MessageToolCall(BaseModel): id: str @@ -114,6 +124,9 @@ class SAPToolChatMessage(BaseModel): ) +ChatMessage = Union[SAPMessage, SAPUserMessage, SAPAssistantMessage, SAPToolChatMessage] + + class ResponseFormat(BaseModel): type_: Literal["text", "json_object"] = Field(default="text", alias="type") @@ -128,3 +141,607 @@ class JSONResponseSchema(BaseModel): class ResponseFormatJSONSchema(BaseModel): type_: Literal["json_schema"] = Field(default="json_schema", alias="type") json_schema: JSONResponseSchema + + +class KeyValueListPair(BaseModel): + key: str + value: list[str] + + +class DocumentMetadataKeyValueListPairs(KeyValueListPair): + select_mode: Optional[list[Literal["ignoreIfKeyAbsent"]]] = None + + +class GroundingSearchConfig(BaseModel): + max_chunk_count: Optional[int] = Field(default=None, ge=0) + max_document_count: Optional[int] = Field(default=None, ge=0) + + @model_validator(mode="after") + def validate_max_chunk_count_and_max_document_count(self): + if self.max_chunk_count is not None and self.max_document_count is not None: + raise ValueError("Cannot specify both maxChunkCount and maxDocumentCount.") + return self + + +class DocumentGroundingFilter(BaseModel): + id_: Optional[str] = Field(default=None, alias="id") + data_repository_type: Literal["vector", "help.sap.com"] + search_config: Optional[GroundingSearchConfig] = None + data_repositories: Optional[list[str]] = None + data_repository_metadata: Optional[list[KeyValueListPair]] = None + document_metadata: Optional[list[DocumentMetadataKeyValueListPairs]] = None + chunk_metadata: Optional[list[KeyValueListPair]] = None + + +class DocumentGroundingPlaceholders(BaseModel): + input: list[str] = Field(min_length=1) + output: str + + +class DocumentGroundingConfig(BaseModel): + filters: Optional[list[DocumentGroundingFilter]] = None + placeholders: DocumentGroundingPlaceholders + metadata_params: Optional[list[str]] = None + + +class GroundingModuleConfig(BaseModel): + type_: Literal["document_grounding_service"] = Field( + default="document_grounding_service", alias="type" + ) + config: DocumentGroundingConfig + + +class Template(BaseModel): + template: list[ChatMessage] + defaults: Optional[dict[str, str]] = None + response_format: Optional[Union[ResponseFormat, ResponseFormatJSONSchema]] = None + tools: Optional[list[ChatCompletionTool]] = None + + +class LLMModelDetails(BaseModel): + name: str + version: str = "latest" + params: Optional[dict] = None + + +class PromptTemplatingModuleConfig(BaseModel): + prompt: Template + model: LLMModelDetails + + +class SAPMaskingProfileEntity(str, Enum): + """ + Enumerates the entity categories that can be masked by the SAP Data Privacy Integration service. + + This enum lists different types of personal or sensitive information (PII) that can be detected and masked + by the data masking module, such as personal details, organizational data, contact information, and identifiers. + + Values: + PERSON: Represents personal names. + + ORG: Represents organizational names. + + UNIVERSITY: Represents educational institutions. + + LOCATION: Represents geographical locations. + + EMAIL: Represents email addresses. + + PHONE: Represents phone numbers. + + ADDRESS: Represents physical addresses. + + SAP_IDS_INTERNAL: Represents internal SAP identifiers. + + SAP_IDS_PUBLIC: Represents public SAP identifiers. + + URL: Represents URLs. + + USERNAME_PASSWORD: Represents usernames and passwords. + + NATIONAL_ID: Represents national identification numbers. + + IBAN: Represents International Bank Account Numbers. + + SSN: Represents Social Security Numbers. + + CREDIT_CARD_NUMBER: Represents credit card numbers. + + PASSPORT: Represents passport numbers. + + DRIVING_LICENSE: Represents driving license numbers. + + NATIONALITY: Represents nationality information. + + RELIGIOUS_GROUP: Represents religious group affiliation. + + POLITICAL_GROUP: Represents political group affiliation. + + PRONOUNS_GENDER: Represents pronouns and gender identity. + + GENDER: Represents gender information. + + SEXUAL_ORIENTATION: Represents sexual orientation. + + TRADE_UNION: Represents trade union membership. + + SENSITIVE_DATA: Represents any other sensitive information. + """ + + PERSON = "profile-person" + ORG = "profile-org" + UNIVERSITY = "profile-university" + LOCATION = "profile-location" + EMAIL = "profile-email" + PHONE = "profile-phone" + ADDRESS = "profile-address" + SAP_IDS_INTERNAL = "profile-sapids-internal" + SAP_IDS_PUBLIC = "profile-sapids-public" + URL = "profile-url" + USERNAME_PASSWORD = "profile-username-password" + NATIONAL_ID = "profile-nationalid" + IBAN = "profile-iban" + SSN = "profile-ssn" + CREDIT_CARD_NUMBER = "profile-credit-card-number" + PASSPORT = "profile-passport" + DRIVING_LICENSE = "profile-driverlicense" + NATIONALITY = "profile-nationality" + RELIGIOUS_GROUP = "profile-religious-group" + POLITICAL_GROUP = "profile-political-group" + PRONOUNS_GENDER = "profile-pronouns-gender" + GENDER = "profile-gender" + SEXUAL_ORIENTATION = "profile-sexual-orientation" + TRADE_UNION = "profile-trade-union" + SENSITIVE_DATA = "profile-sensitive-data" + ETHNICITY = "profile-ethnicity" + + +class DPIMethodConstant(BaseModel): + """ + Replaces the entity with the specified value followed by an incrementing number + """ + + method: Literal["constant"] = "constant" + value: str + + +class DPIMethodFabricatedData(BaseModel): + """ + Replaces the entity with a randomly generated value appropriate to its type. + """ + + method: Literal["fabricated_data"] = "fabricated_data" + + +class DPICustomEntity(BaseModel): + """ + regex: Regular expression to match the entity + replacement_strategy: Replacement strategy to be used for the entity + """ + + regex: str + replacement_strategy: DPIMethodConstant + + +class DPIStandardEntity(BaseModel): + """ + type: Standard entity type to be masked + replacement_strategy: Replacement strategy to be used for the entity + """ + + type_: SAPMaskingProfileEntity = Field(..., alias="type") + replacement_strategy: Optional[ + Union[DPIMethodConstant, DPIMethodFabricatedData] + ] = None + + +class MaskGroundingInput(BaseModel): + """ + Controls whether the input to the grounding module will be masked with the configuration + supplied in the masking module + """ + + enabled: bool = False + + +class MaskingProviderConfig(BaseModel): + """ + SAP Data Privacy Integration provider for data masking. + + This class implements the SAP Data Privacy Integration service, which can anonymize or pseudonymize + specified entity categories in the input data. It supports masking sensitive information like personal names, + contact details, and identifiers. + + Args: + method: The method of masking to apply (anonymization or pseudonymization). + + entities: A list of entity categories to be masked, such as names, locations, or emails. + + allowlist: A list of strings that should not be masked. + + mask_grounding_input: A flag indicating whether to mask input to the grounding module. + """ + + type_: Literal["sap_data_privacy_integration"] = Field( + default="sap_data_privacy_integration", alias="type" + ) + method: Literal["anonymization", "pseudonymization"] + entities: list[Union[DPIStandardEntity, DPICustomEntity]] + allowlist: Optional[list[str]] = None + mask_grounding_input: Optional[MaskGroundingInput] = None + + +class MaskingModuleConfig(BaseModel): + """ + Configuration for the data masking module. + + Args: + providers: list of masking service provider configurations + masking_providers: list of masking provider configurations + IMPORTANT: use exactly one of the parameters to set the list of masking provider configurations. + DEPRECATED: parameter 'masking_providers' will be removed Sept 15, 2026. Use 'providers' instead. + """ + + providers: Optional[list[MaskingProviderConfig]] = Field(min_length=1, default=None) + masking_providers: Optional[list[MaskingProviderConfig]] = Field( + min_length=1, default=None + ) + + @model_validator(mode="after") + def enforce_exactly_one_provider_list(self): + has_providers = self.providers is not None + has_masking_providers = self.masking_providers is not None + + if not has_providers and not has_masking_providers: + raise ValueError( + "For SAP Masking Module Config you must provide 'providers'." + ) + if has_providers and has_masking_providers: + raise ValueError( + "For SAP Masking Module Config you must set exactly one of: 'providers' or 'masking_providers', not both." + ) + + if has_masking_providers: + warnings.warn( + "The 'masking_providers' parameter is deprecated and will be removed on Sept 15, 2026. " + "Use 'providers' instead.", + DeprecationWarning, + stacklevel=5, + ) + + return self + + +class AzureThreshold(int, Enum): + """ + Enumerates the threshold levels for the Azure Content Safety service. + + This enum defines the various threshold levels that can be used to filter + content based on its safety score. Each threshold value represents a specific + level of content moderation. + + Values: + ALLOW_SAFE: Allows only Safe content. + + ALLOW_SAFE_LOW: Allows Safe and Low content. + + ALLOW_SAFE_LOW_MEDIUM: Allows Safe, Low, and Medium content. + + ALLOW_ALL: Allows all content (Safe, Low, Medium, and High). + """ + + ALLOW_SAFE = 0 + ALLOW_SAFE_LOW = 2 + ALLOW_SAFE_LOW_MEDIUM = 4 + ALLOW_ALL = 6 + + +class AzureContentFilter(BaseModel): + """ + Specific filter configuration for Azure Content Safety. + + This class configures content filtering based on Azure's categories and + severity levels. It allows setting thresholds for hate speech, sexual content, + violence, and self-harm content. + + Values: + hate: Threshold for hate speech content. + + sexual: Threshold for sexual content. + + violence: Threshold for violent content. + + self_harm: Threshold for self-harm content. + """ + + hate: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None + sexual: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None + violence: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None + self_harm: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None + + +class AzureContentSafetyInput(AzureContentFilter): + """ + Filter configuration for Azure Content Safety Input + + Values: + hate: Threshold for hate speech content. + + sexual: Threshold for sexual content. + + violence: Threshold for violent content. + + self_harm: Threshold for self-harm content. + + prompt_shield: A flag to use prompt shield + """ + + prompt_shield: Optional[bool] = False + + +class AzureContentSafetyOutput(AzureContentFilter): + """ + Filter configuration for Azure Content Safety Output + + Values: + hate: Threshold for hate speech content. + + sexual: Threshold for sexual content. + + violence: Threshold for violent content. + + self_harm: Threshold for self-harm content. + + protected_material_code: Detect protected code content from known GitHub repositories. + The scan includes software libraries, source code, algorithms, + and other proprietary programming content. + """ + + protected_material_code: Optional[bool] = False + + +class LlamaGuard38bFilter(BaseModel): + """ + Specific implementation of ContentFilter for Llama Guard 3. Llama Guard 3 is a + Llama-3.1-8B pretrained model, fine-tuned for content safety classification. + + Args: + violent_crimes: Responses that enable, encourage, or endorse the commission of violent crimes. + + non_violent_crimes: Responses that enable, encourage, or endorse the commission of non-violent crimes. + + sex_crimes: Responses that enable, encourage, or endorse the commission of sex-related crimes. + + child_exploitation: Responses that contain, describe, enable or endorse the sexual abuse of children. + + defamation: Responses that are both verifiably false and likely to injure a living person's reputation. + + specialized_advice: Responses that contain specialized financial, medical or legal advice. + + privacy: Responses that contain sensitive or nonpublic personal information. + + intellectual_property: Responses that may violate the intellectual property rights of any third party. + + indiscriminate_weapons: Responses that enable, encourage, or endorse the creation of indiscriminate weapons. + + hate: Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics. + + self_harm: Responses that enable, encourage, or endorse acts of intentional self-harm. + + sexual_content: Responses that contain erotica. + + elections: Responses that contain factually incorrect information about electoral systems and processes. + + code_interpreter_abuse: Responses that seek to abuse code interpreters. + """ + + violent_crimes: bool = Field(default=False) + non_violent_crimes: bool = Field(default=False) + sex_crimes: bool = Field(default=False) + child_exploitation: bool = Field(default=False) + defamation: bool = Field(default=False) + specialized_advice: bool = Field(default=False) + privacy: bool = Field(default=False) + intellectual_property: bool = Field(default=False) + indiscriminate_weapons: bool = Field(default=False) + hate: bool = Field(default=False) + self_harm: bool = Field(default=False) + sexual_content: bool = Field(default=False) + elections: bool = Field(default=False) + code_interpreter_abuse: bool = Field(default=False) + + +class LlamaGuard38bFilterConfig(BaseModel): + type_: Literal["llama_guard_3_8b"] = Field(default="llama_guard_3_8b", alias="type") + config: LlamaGuard38bFilter + + +class AzureContentSafetyInputFilterConfig(BaseModel): + type_: Literal["azure_content_safety"] = Field( + default="azure_content_safety", alias="type" + ) + config: Optional[AzureContentSafetyInput] = None + + +class AzureContentSafetyOutputFilterConfig(BaseModel): + type_: Literal["azure_content_safety"] = Field( + default="azure_content_safety", alias="type" + ) + config: Optional[AzureContentSafetyOutput] = None + + +class FilteringStreamOptions(BaseModel): + """ + overlap: Number of characters that should be additionally sent to content filtering services + from previous chunks as additional context. + """ + + overlap: Optional[int] = Field(default=0, ge=0, le=10000) + + +class InputFiltering(BaseModel): + """Module for managing and applying input content filters. + + Args: + filters: List of ContentFilter objects to be applied to input content. + """ + + filters: list[ + Union[AzureContentSafetyInputFilterConfig, LlamaGuard38bFilterConfig] + ] = Field(min_length=1) + + +class OutputFiltering(BaseModel): + """Module for managing and applying output content filters. + + Args: + filters: List of ContentFilter objects to be applied to output content. + + stream_options: Module-specific streaming options. + """ + + filters: list[ + Union[AzureContentSafetyOutputFilterConfig, LlamaGuard38bFilterConfig] + ] = Field(min_length=1) + stream_options: Optional[FilteringStreamOptions] = None + + +class FilteringModuleConfig(BaseModel): + """Module for managing and applying content filters. + + Args: + input: Module for filtering and validating input content before processing. + + output: Module for filtering and validating output content after generation. + """ + + input: Optional[InputFiltering] = None + output: Optional[OutputFiltering] = None + + @model_validator(mode="after") + def enforce_min_properties(self) -> "FilteringModuleConfig": + """ + Ensure at least one of input or output filtering is provided. + """ + if self.input is None and self.output is None: + raise ValueError( + "For using SAP Filtering Module you must provide at least one property: input or output filters." + ) + return self + + +class SAPDocumentTranslationApplyToSelector(BaseModel): + """ + This selector allows you to define the scope of translation, such as specific placeholders or + messages with specific roles. + For example, {"category": "placeholders", + "items": ["user_input"], + "source_language": "de-DE"} + targets the value of "user_input" in placeholder_values specified in the request payload; + and considers the value to be in German. + """ + + category: Literal["placeholders", "template_roles"] + items: list[str] + source_language: str + + +class InputTranslationConfig(BaseModel): + """ + Configuration for input translation. + + Args: + source_language: Language of the text to be translated. Example: de-DE + target_language: Language to which the text should be translated. Example: en-US + apply_to: List of selectors that define the scope of translation. + """ + + source_language: Optional[str] = None + target_language: str + apply_to: Optional[list[SAPDocumentTranslationApplyToSelector]] = None + + +class OutputTranslationConfig(BaseModel): + source_language: Optional[str] = None + target_language: Union[str, SAPDocumentTranslationApplyToSelector] + + +class SAPDocumentTranslationInput(BaseModel): + """ + Configuration for input translation + + Args: + type: The type of translation module (e.g., 'sap_document_translation'). + + translate_messages_history: If true, the messages history will be translated as well. + + config: Configuration object for the translation module. + """ + + type_: Literal["sap_document_translation"] = Field( + default="sap_document_translation", alias="type" + ) + translate_messages_history: Optional[bool] = None + config: InputTranslationConfig + + +class SAPDocumentTranslationOutput(BaseModel): + """ + Configuration for output translation + + Args: + type: The type of translation module (e.g., 'sap_document_translation'). + + config: Configuration object for the translation module. + """ + + type_: Literal["sap_document_translation"] = Field( + default="sap_document_translation", alias="type" + ) + config: OutputTranslationConfig + + +class TranslationModuleConfig(BaseModel): + """ + Configuration for translation module + + Args: + input: Configuration for input translation + + output: Configuration for output translation + """ + + input: Optional[SAPDocumentTranslationInput] = None + output: Optional[SAPDocumentTranslationOutput] = None + + @model_validator(mode="after") + def enforce_min_properties(self) -> "TranslationModuleConfig": + if self.input is None and self.output is None: + raise ValueError( + "TranslationModuleConfig requires at least one of 'input' or 'output'." + ) + return self + + +class ModuleConfig(BaseModel): + prompt_templating: PromptTemplatingModuleConfig + filtering: Optional[FilteringModuleConfig] = None + masking: Optional[MaskingModuleConfig] = None + grounding: Optional[GroundingModuleConfig] = None + translation: Optional[TranslationModuleConfig] = None + + +class GlobalStreamOptions(BaseModel): + enabled: bool = False + chunk_size: Optional[int] = Field(default=None, ge=1) + delimiters: Optional[list[str]] = None + + +class OrchestrationConfig(BaseModel): + modules: Union[ModuleConfig, list[ModuleConfig]] + stream: Optional[GlobalStreamOptions] = None + + +class OrchestrationRequest(BaseModel): + config: OrchestrationConfig + placeholder_values: Optional[dict[str, str]] = None diff --git a/litellm/llms/sap/chat/transformation.py b/litellm/llms/sap/chat/transformation.py index 7f6bab4a1d5..a55ec746350 100755 --- a/litellm/llms/sap/chat/transformation.py +++ b/litellm/llms/sap/chat/transformation.py @@ -11,6 +11,7 @@ from typing import ( TYPE_CHECKING, Iterator, AsyncIterator, + FrozenSet, ) from functools import cached_property import litellm @@ -31,12 +32,13 @@ else: from ..credentials import get_token_creator from .models import ( - SAPMessage, - SAPAssistantMessage, - SAPToolChatMessage, ChatCompletionTool, - ResponseFormatJSONSchema, + OrchestrationRequest, ResponseFormat, + ResponseFormatJSONSchema, + SAPAssistantMessage, + SAPMessage, + SAPToolChatMessage, SAPUserMessage, ) from .handler import ( @@ -45,9 +47,65 @@ from .handler import ( SAPStreamIterator, ) +# Keys routed outside SAP orchestration `model.params` (prompt, stream, fallbacks, etc.) +_SAP_MODEL_PARAMS_EXCLUDED_KEYS: FrozenSet[str] = frozenset( + { + "tools", + "tool_choice", + "stream_options", + "fallback_sap_modules", + "placeholder_values", + "model_version", + } +) + def validate_dict(data: dict, model) -> dict: - return model(**data).model_dump(by_alias=True) + return model(**data).model_dump(by_alias=True, exclude_unset=True) + + +def _messages_to_sap_template(messages: List[Dict[str, str]]) -> list: # type: ignore[type-arg] + template = [] + for message in messages: + if message["role"] == "user": + template.append(validate_dict(message, SAPUserMessage)) + elif message["role"] == "assistant": + template.append(validate_dict(message, SAPAssistantMessage)) + elif message["role"] == "tool": + template.append(validate_dict(message, SAPToolChatMessage)) + else: + template.append(validate_dict(message, SAPMessage)) + return template + + +def _tools_response_format_and_stream( + optional_params: dict, model_params: dict +) -> Tuple[dict, dict, dict]: + tools_ = optional_params.pop("tools", []) + tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_] + tools: dict = {"tools": tools_} if tools_ else {} + + response_format = model_params.pop("response_format", {}) + resp_type = response_format.get("type", None) + if resp_type: + if resp_type == "json_schema": + response_format = validate_dict( + response_format, ResponseFormatJSONSchema + ) + else: + response_format = validate_dict(response_format, ResponseFormat) + response_format = {"response_format": response_format} + + model_params.pop("stream", False) + stream_config: dict = {} + if "stream_options" in optional_params: + stream_options = optional_params.pop("stream_options", {}) + if "chunk_size" in stream_options: + stream_config["chunk_size"] = stream_options.get("chunk_size") + if "delimiters" in stream_options: + stream_config["delimiters"] = stream_options.get("delimiters") + + return tools, response_format, stream_config class GenAIHubOrchestrationConfig(OpenAIGPTConfig): @@ -208,48 +266,25 @@ class GenAIHubOrchestrationConfig(OpenAIGPTConfig): api_base_ = f"{self.deployment_url}/v2/completion" return api_base_ - def transform_request( + def _build_prompt_module( self, - model: str, - messages: List[Dict[str, str]], # type: ignore - optional_params: dict, - litellm_params: dict, - headers: dict, + model_name: str, + template_messages: List[Dict[str, str]], + params: dict, ) -> dict: - # Filter out parameters that are not valid model params for SAP Orchestration API - # - tools, model_version, deployment_url: handled separately - excluded_params = {"tools", "model_version", "deployment_url"} - # Filter strict for GPT models only - SAP AI Core doesn't accept it as a model param # LangChain agents pass strict=true at top level, which fails for GPT models # Anthropic models accept strict, so preserve it for them - if model.startswith("gpt"): - excluded_params.add("strict") + if model_name.startswith("gpt") and "strict" in params: + params.pop("strict") - model_params = { - k: v for k, v in optional_params.items() if k not in excluded_params - } + model_version = params.pop("model_version", "latest") - model_version = optional_params.pop("model_version", "latest") - template = [] - for message in messages: - if message["role"] == "user": - template.append(validate_dict(message, SAPUserMessage)) - elif message["role"] == "assistant": - template.append(validate_dict(message, SAPAssistantMessage)) - elif message["role"] == "tool": - template.append(validate_dict(message, SAPToolChatMessage)) - else: - template.append(validate_dict(message, SAPMessage)) - - tools_ = optional_params.pop("tools", []) + tools_ = params.pop("tools", []) tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_] - if tools_ != []: - tools = {"tools": tools_} - else: - tools = {} + tools = {"tools": tools_} if tools_ else {} - response_format = model_params.pop("response_format", {}) + response_format = params.pop("response_format", {}) resp_type = response_format.get("type", None) if resp_type: if resp_type == "json_schema": @@ -259,33 +294,104 @@ class GenAIHubOrchestrationConfig(OpenAIGPTConfig): else: response_format = validate_dict(response_format, ResponseFormat) response_format = {"response_format": response_format} - model_params.pop("stream", False) - stream_config = {} - if "stream_options" in model_params: - # stream_config["enabled"] = True - stream_options = model_params.pop("stream_options", {}) - stream_config["chunk_size"] = stream_options.get("chunk_size", 100) - if "delimiters" in stream_options: - stream_config["delimiters"] = stream_options.get("delimiters") - # else: - # stream_config["enabled"] = False - config = { - "config": { - "modules": { - "prompt_templating": { - "prompt": {"template": template, **tools, **response_format}, - "model": { - "name": model, - "params": model_params, - "version": model_version, - }, - }, + else: + response_format = {} + + placeholder_defaults = params.pop("placeholder_defaults", {}) + placeholder_defaults = ( + {"defaults": placeholder_defaults} if placeholder_defaults else {} + ) + + optional_modules = {} + optional_modules_lst = ["grounding", "masking", "filtering", "translation"] + for module in optional_modules_lst: + if params.get(module, None) is not None: + optional_modules[module] = params.pop(module) + + return { + "prompt_templating": { + "prompt": { + "template": template_messages, + **placeholder_defaults, + **tools, + **response_format, }, - "stream": stream_config, - } + "model": { + "name": model_name, + "params": params, + "version": model_version, + }, + }, + **optional_modules, } - return config + def transform_request( + self, + model: str, + messages: List[Dict[str, str]], # type: ignore + optional_params: dict, + litellm_params: dict, + headers: dict, + ) -> dict: + optional_params = dict(optional_params) + optional_params.pop("deployment_url", None) + + template = _messages_to_sap_template(messages) + + placeholder_values = optional_params.pop("placeholder_values", None) + fallback_modules = optional_params.pop("fallback_sap_modules", []) + + optional_params.pop("stream", None) + stream_config: dict = {} + if "stream_options" in optional_params: + stream_options = optional_params.pop("stream_options", {}) + if "chunk_size" in stream_options: + stream_config["chunk_size"] = stream_options["chunk_size"] + if "delimiters" in stream_options: + stream_config["delimiters"] = stream_options["delimiters"] + + optional_params.pop("tool_choice", None) + + modules = [ + self._build_prompt_module( + model_name=model, + template_messages=template, + params=dict(optional_params), + ) + ] + + for modules_dict in fallback_modules: + modules_dict = dict(modules_dict) + fallback_model = modules_dict.pop("model", None) + if fallback_model is None: + raise ValueError( + "Each entry in `fallback_sap_modules` must include a 'model' key." + ) + if fallback_model.startswith("sap/"): + fallback_model = fallback_model[4:] + fallback_template = modules_dict.pop("messages", []) + + modules.append( + self._build_prompt_module( + model_name=fallback_model, + template_messages=fallback_template, + params=modules_dict, + ) + ) + + config_payload: Dict[str, Any] = { + "modules": modules if len(modules) > 1 else modules[0], + } + if stream_config: + config_payload["stream"] = stream_config + + request_body: Dict[str, Any] = {"config": config_payload} + if placeholder_values is not None: + request_body["placeholder_values"] = placeholder_values + + body = validate_dict(request_body, OrchestrationRequest) + + return body def transform_response( self, diff --git a/litellm/llms/sap/credentials.py b/litellm/llms/sap/credentials.py index aeae51bf0bb..0ae351783e8 100644 --- a/litellm/llms/sap/credentials.py +++ b/litellm/llms/sap/credentials.py @@ -1,5 +1,5 @@ from __future__ import annotations -from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple +from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple, Union from datetime import datetime, timedelta, timezone from threading import Lock from pathlib import Path @@ -7,9 +7,11 @@ from dataclasses import dataclass import json import os import tempfile +import httpx -from litellm import sap_service_key -from litellm.llms.custom_httpx.http_handler import _get_httpx_client +from litellm.llms.custom_httpx.http_handler import _get_httpx_client, HTTPHandler +from litellm._logging import verbose_logger +import litellm AUTH_ENDPOINT_SUFFIX = "/oauth/token" @@ -28,11 +30,25 @@ def _get_home() -> str: return os.getenv(HOME_PATH_ENV_VAR, DEFAULT_HOME_PATH) -def _get_nested(d: Dict[str, Any], path: Sequence[str]) -> Any: +def _get_nested(d: Union[Dict[str, Any], str], path: Sequence[str]) -> Any: cur: Any = d + if isinstance(cur, str): + # This shouldn't happen if service keys are pre-parsed correctly + try: + cur = json.loads(cur) + except json.JSONDecodeError: + verbose_logger.warning( + "SAP service key or VCAP service is a string but not valid JSON." + ) + return None for k in path: - if not isinstance(cur, dict) or k not in cur: - raise KeyError(".".join(path)) + if not isinstance(cur, dict): + verbose_logger.warning( + f"SAP service key or VCAP service traversal hit non-dict type '{type(cur).__name__}' at key '{k}'." + ) + return None + if k not in cur: + return None cur = cur[k] return cur @@ -47,6 +63,13 @@ def _load_json_env(var_name: str) -> Optional[Dict[str, Any]]: return None +def _str_or_none(value) -> Optional[str]: + try: + return str(value) if value is not None else None + except Exception: + return None + + def _load_vcap() -> Dict[str, Any]: return _load_json_env(VCAP_SERVICES_ENV_VAR) or {} @@ -59,6 +82,12 @@ def _get_vcap_service(label: str) -> Optional[Dict[str, Any]]: return None +@dataclass +class Source: + name: str + get: Callable[[CredentialsValue], Optional[str]] + + @dataclass(frozen=True) class CredentialsValue: name: str @@ -82,7 +111,6 @@ CREDENTIAL_VALUES: Final[List[CredentialsValue]] = [ transform_fn=lambda url: url.rstrip("/") + ("" if url.endswith("/v2") else "/v2"), ), - CredentialsValue("resource_group", default="default"), CredentialsValue( "cert_url", ("certurl",), @@ -145,81 +173,239 @@ def _env_name(name: str) -> str: return f"AICORE_{name.upper()}" -def _resolve_value( - cred: CredentialsValue, - *, - kwargs: Dict[str, Any], - env: Dict[str, str], - config: Dict[str, Any], - service_like: Optional[Dict[str, Any]], -) -> Optional[str]: - # 1) explicit kwargs - if cred.name in kwargs and kwargs[cred.name] is not None: - return kwargs[cred.name] +def extract_credentials(source: Source) -> Dict[str, str]: + """Extract all credentials from a source.""" + credentials = {} + for cv in CREDENTIAL_VALUES: + value = source.get(cv) + if value is not None: + credentials[cv.name] = cv.transform_fn(value) if cv.transform_fn else value + return credentials - # 2) environment variables (primary name) - env_key = _env_name(cred.name) - if env_key in env and env[env_key] is not None: - return env[env_key] - # 3) config file (accept both prefixed and plain keys) - for key in (env_key, cred.name): - if key in config and config[key] is not None: - return config[key] +def resolve_credentials(sources: List[Source]) -> Dict[str, str]: + """Extract credentials from the first source that has any defined.""" + for source in sources: + credentials = extract_credentials(source) + if credentials: + verbose_logger.debug(f"Resolved SAP credentials from source {source.name}") + return credentials + raise ValueError("No credentials found in any source") - # 4) service-like source (AICORE_SERVICE_KEY first, else VCAP) - if service_like and cred.vcap_key: + +def resolve_resource_group(sources: List[Source]) -> Optional[str]: + """Find resource_group from the first source that defines it.""" + rg_cred = CredentialsValue("resource_group", default="default") + for source in sources: + value = source.get(rg_cred) + if value is not None: + verbose_logger.debug( + f"Resolved GEN AI Hub resource_group from source {source.name}" + ) + return value + return rg_cred.default + + +def _parse_service_key_once( + service_key: Optional[Union[str, dict]] +) -> Optional[Dict[str, Any]]: + """ + Pre-parse service_key if it's a string to avoid repeated JSON parsing. + + Returns None if parsing fails (other credential sources may still work). + """ + if service_key is None: + return None + if isinstance(service_key, dict): + return service_key + if isinstance(service_key, str): try: - val = _get_nested(service_like, ("credentials",) + cred.vcap_key) - if val is not None: - return val - except KeyError: - pass + return json.loads(service_key) + except json.JSONDecodeError: + verbose_logger.warning( + "SAP service key is a string but not valid JSON. Skipping this source." + ) + return None + verbose_logger.warning( + f"SAP service key has unexpected type '{type(service_key).__name__}'. Expected str or dict. Ignoring." + ) + return None - # 5) default - return cred.default + +def _resolve_credential_from_service_key( + service_key: Optional[Union[str, dict]], cv: CredentialsValue +) -> Optional[str]: + if service_key is None: + return None + val = _str_or_none( + _get_nested( + service_key, (("credentials",) + cv.vcap_key) if cv.vcap_key else (cv.name,) + ) + ) + if val is None: + return _str_or_none( + _get_nested(service_key, cv.vcap_key if cv.vcap_key else (cv.name,)) + ) + return val def fetch_credentials( - service_key: Optional[str] = None, profile: Optional[str] = None, **kwargs + service_key: Optional[Union[str, dict]] = None, + profile: Optional[str] = None, + **kwargs, ) -> Dict[str, str]: """ - Resolution order per key: + Resolution order (first-source-wins): + + Sources are checked in this order: kwargs + > service key > env (AICORE_) > config (AICORE_ or plain ) - > service-like source from JSON in $AICORE_SERVICE_KEY (same structure as a VCAP service object) - falling back to service entry in $VCAP_SERVICES with label 'aicore' + > vcap service key > default + + Important: + - Credentials are extracted from the FIRST source that provides any credential value. + - Values are NOT merged per key across sources. Except resource_group, which is merged. + + Warning: + - This function does NOT validate the returned credentials just parsed it from the sources. + - Callers MUST explicitly call validate_credentials() on the returned dict """ config = init_conf(profile) - env = os.environ # snapshot for testability - service_like = None - if not config: - # Prefer AICORE_SERVICE_KEY if present; otherwise fall back to the VCAP service. - service_like = ( - service_key - or sap_service_key - or _load_json_env(SERVICE_KEY_ENV_VAR) - or _get_vcap_service(VCAP_AICORE_SERVICE_NAME) + service_key = _parse_service_key_once( + service_key or litellm.sap_service_key or os.environ.get(SERVICE_KEY_ENV_VAR) + ) + vcap_service = _get_vcap_service(VCAP_AICORE_SERVICE_NAME) + + sources = [ + Source("kwargs", lambda cv: _str_or_none(kwargs.get(cv.name))), + Source( + "service key", + lambda cv: _resolve_credential_from_service_key(service_key, cv), + ), + Source( + "environment variables", + lambda cv: _str_or_none(os.environ.get(f"AICORE_{cv.name.upper()}")), + ), + Source( + "config file", + lambda cv: _str_or_none( + config.get(f"AICORE_{cv.name.upper()}") + if config.get(f"AICORE_{cv.name.upper()}") is not None + else config.get(cv.name) + ), + ), + Source( + "VCAP service", + lambda cv: ( + _str_or_none( + _get_nested( + vcap_service, + (("credentials",) + cv.vcap_key) if cv.vcap_key else (cv.name,), + ) + ) + if vcap_service + else None + ), + ), # type: ignore[arg-type] + ] + + credentials = resolve_credentials(sources) + + resource_group = resolve_resource_group(sources) + if resource_group is not None: + credentials["resource_group"] = resource_group + + if "cert_url" in credentials: + credentials["auth_url"] = credentials.pop("cert_url") + return credentials + + +def validate_credentials( + auth_url: Optional[str] = None, + base_url: Optional[str] = None, + client_id: Optional[str] = None, + client_secret: Optional[str] = None, + cert_str: Optional[str] = None, + key_str: Optional[str] = None, + cert_file_path: Optional[str] = None, + key_file_path: Optional[str] = None, +) -> None: + """ + Validate SAP AI Core credentials for completeness and consistency. + + Args: + auth_url: OAuth2 token endpoint URL (required) + base_url: SAP AI Core API base URL (required) + client_id: OAuth2 client ID (required) + client_secret: OAuth2 client secret (for secret-based auth) + cert_str: PEM-encoded certificate string (for cert-based auth) + key_str: PEM-encoded private key string (for cert-based auth) + cert_file_path: Path to certificate file (for file-based cert auth) + key_file_path: Path to private key file (for file-based cert auth) + + Raises: + ValueError: If required fields are missing or authentication mode is ambiguous. + + Note: + - This function does NOT validate resource_group (resolved separately). + - Exactly one authentication method must be provided: + * client_secret, OR + * (cert_str AND key_str), OR + * (cert_file_path AND key_file_path) + """ + if not auth_url or not client_id or not base_url: + raise ValueError( + "SAP AI Core credentials not found. " + "Please provide credentials by setting appropriate environment variables " + "(e.g. AICORE_CLIENT_ID, AICORE_CLIENT_SECRET, etc.)" ) - out: Dict[str, str] = {} - for cred in CREDENTIAL_VALUES: - value = _resolve_value(cred, kwargs=kwargs, env=env, config=config, service_like=service_like) # type: ignore - if value is None: - continue - if cred.transform_fn: - value = cred.transform_fn(value) - out[cred.name] = value - if "cert_url" in out.keys(): - out["auth_url"] = out.pop("cert_url") - return out + modes = [ + bool(client_secret), + bool(cert_str) and bool(key_str), + bool(cert_file_path) and bool(key_file_path), + ] + if sum(bool(m) for m in modes) != 1: + raise ValueError( + "SAP AI Core credentials are incomplete. " + "Invalid credentials: provide exactly one of client_secret, " + "(cert_str & key_str), or (cert_file_path & key_file_path)." + ) + + +def _request_token( + client_id: str, auth_url: str, timeout: float, cert_pair=None, client_secret=None +) -> tuple[str, datetime]: + data = {"grant_type": "client_credentials", "client_id": client_id} + if client_secret: + data["client_secret"] = client_secret + + resp: Optional[httpx.Response] = None + try: + if cert_pair: + with httpx.Client(cert=cert_pair) as raw_client: + handler = HTTPHandler(client=raw_client) + resp = handler.post(auth_url, data=data, timeout=timeout) # type: ignore[arg-type] + payload = resp.json() + else: + handler = _get_httpx_client() + resp = handler.post(auth_url, data=data, timeout=timeout) # type: ignore[arg-type] + payload = resp.json() + access_token = payload["access_token"] + expires_in = int(payload.get("expires_in", 3600)) + expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in) + return f"Bearer {access_token}", expiry_date + except Exception as e: + msg = resp.text if resp is not None else getattr(e, "text", str(e)) + raise RuntimeError(f"Token request failed: {msg}") from e def get_token_creator( - service_key: Optional[str] = None, + service_key: Optional[Union[str, dict]] = None, profile: Optional[str] = None, *, timeout: float = 30.0, @@ -237,7 +423,7 @@ def get_token_creator( Args: profile: Optional AICore profile name - timeout: HTTP request timeout in seconds (default 30s) + timeout: Timeout for HTTP requests expiry_buffer_minutes: Refresh the token this many minutes before expiry overrides: Any explicit credential overrides (client_id, client_secret, etc.) @@ -251,6 +437,7 @@ def get_token_creator( ) auth_url = credentials.get("auth_url") + base_url = credentials.get("base_url") client_id = credentials.get("client_id") client_secret = credentials.get("client_secret") cert_str = credentials.get("cert_str") @@ -259,49 +446,30 @@ def get_token_creator( key_file_path = credentials.get("key_file_path") # Sanity check - if not auth_url or not client_id: - raise ValueError( - "fetch_credentials did not return valid 'auth_url' or 'client_id'" - ) - - modes = [ - client_secret is not None, - (cert_str is not None and key_str is not None), - (cert_file_path is not None and key_file_path is not None), - ] - if sum(bool(m) for m in modes) != 1: - raise ValueError( - "Invalid credentials: provide exactly one of client_secret, " - "(cert_str & key_str), or (cert_file_path & key_file_path)." - ) + validate_credentials( + auth_url, + base_url, + client_id, + client_secret, + cert_str, + key_str, + cert_file_path, + key_file_path, + ) lock = Lock() token: Optional[str] = None token_expiry: Optional[datetime] = None - def _request_token(cert_pair=None) -> tuple[str, datetime]: - data = {"grant_type": "client_credentials", "client_id": client_id} - if client_secret: - data["client_secret"] = client_secret - - client = _get_httpx_client() - # with httpx.Client(cert=cert_pair, timeout=timeout) as client: - resp = client.post(auth_url, data=data) - try: - resp.raise_for_status() - payload = resp.json() - access_token = payload["access_token"] - expires_in = int(payload.get("expires_in", 3600)) - expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in) - return f"Bearer {access_token}", expiry_date - except Exception as e: - msg = getattr(resp, "text", str(e)) - raise RuntimeError(f"Token request failed: {msg}") from e - def _fetch_token() -> tuple[str, datetime]: # Case 1: secret-based auth if client_secret: - return _request_token() + return _request_token( + auth_url=auth_url, # type: ignore[arg-type] + client_id=client_id, # type: ignore[arg-type] + timeout=timeout, + client_secret=client_secret, + ) # Case 2: cert/key strings if cert_str and key_str: cert_str_fixed = cert_str.replace("\\n", "\n") @@ -313,9 +481,24 @@ def get_token_creator( f.write(cert_str_fixed) with open(key_path, "w") as f: f.write(key_str_fixed) - return _request_token(cert_pair=(cert_path, key_path)) + return _request_token( + auth_url=auth_url, # type: ignore[arg-type] + client_id=client_id, # type: ignore[arg-type] + timeout=timeout, + cert_pair=(cert_path, key_path), + ) # Case 3: file-based cert/key - return _request_token(cert_pair=(cert_file_path, key_file_path)) + if cert_file_path is not None and key_file_path is not None: + return _request_token( + auth_url=auth_url, # type: ignore[arg-type] + client_id=client_id, # type: ignore[arg-type] + timeout=timeout, + cert_pair=(cert_file_path, key_file_path), + ) + # Defensive guard: should never reach here due to validate_credentials() + raise ValueError( + "Invalid authentication configuration: no valid credentials found. " + ) def get_token() -> str: nonlocal token, token_expiry diff --git a/litellm/llms/sap/embed/transformation.py b/litellm/llms/sap/embed/transformation.py index 0bbf4f259f7..c74f21c3685 100644 --- a/litellm/llms/sap/embed/transformation.py +++ b/litellm/llms/sap/embed/transformation.py @@ -5,6 +5,7 @@ Translates from OpenAI's `/v1/embeddings` to IBM's `/text/embeddings` route. from typing import Optional, List, Dict, Literal, Union from pydantic import BaseModel, Field from functools import cached_property +from litellm.llms.sap.chat.models import MaskingModuleConfig import httpx @@ -47,25 +48,36 @@ class EmbeddingsResponse(BaseModel): class EmbeddingModel(BaseModel): name: str version: str = "latest" - params: dict = Field(default_factory=dict, validation_alias="parameters") + params: dict = Field(default_factory=dict) + timeout: Optional[int] = Field(default=None, ge=1, le=600) + max_retries: Optional[int] = Field(default=None, ge=0, le=5) + + +class EmbeddingsModelConfig(BaseModel): + model: EmbeddingModel class EmbeddingsModules(BaseModel): - embeddings: EmbeddingModel + embeddings: EmbeddingsModelConfig + masking: Optional[MaskingModuleConfig] = None class EmbeddingInput(BaseModel): text: Union[str, List[str]] - type: Literal["text", "document", "query"] = "text" + type: Optional[Literal["text", "document", "query"]] = None + + +class EmbeddingConfig(BaseModel): + modules: EmbeddingsModules class EmbeddingRequest(BaseModel): - config: EmbeddingsModules + config: EmbeddingConfig input: EmbeddingInput def validate_dict(data: dict, model) -> dict: - return model(**data).model_dump() + return model(**data).model_dump(exclude_unset=True, by_alias=True) class GenAIHubEmbeddingConfig(BaseEmbeddingConfig): @@ -152,15 +164,23 @@ class GenAIHubEmbeddingConfig(BaseEmbeddingConfig): model_dict["name"] = model model_dict["version"] = optional_params.get("version", "latest") model_dict["params"] = optional_params.get("parameters", {}) + timeout = optional_params.get("timeout", None) + if timeout is not None: + model_dict["timeout"] = timeout + max_retries = optional_params.get("max_retries", None) + if max_retries is not None: + model_dict["max_retries"] = max_retries input_dict = {"text": input} + input_type = optional_params.get("type") + if input_type is not None: + input_dict["type"] = input_type + masking = optional_params.get("masking") + masking = {"masking": masking} if masking is not None else {} body = { - "config": { - "modules": { - "embeddings": {"model": validate_dict(model_dict, EmbeddingModel)} - } - }, - "input": validate_dict(input_dict, EmbeddingInput), + "config": {"modules": {"embeddings": {"model": model_dict}, **masking}}, + "input": input_dict, } + body = validate_dict(body, EmbeddingRequest) return body def transform_embedding_response( diff --git a/litellm/llms/vertex_ai/gemini/transformation.py b/litellm/llms/vertex_ai/gemini/transformation.py index 7945c44d44c..6157a384dc0 100644 --- a/litellm/llms/vertex_ai/gemini/transformation.py +++ b/litellm/llms/vertex_ai/gemini/transformation.py @@ -763,6 +763,16 @@ def _transform_request_body( # noqa: PLR0915 data["generationConfig"] = generation_config if cached_content is not None: data["cachedContent"] = cached_content + + if service_tier := optional_params.pop("service_tier", None): + if isinstance(service_tier, str): + if service_tier.lower() == "default": + data["serviceTier"] = "standard" + else: + data["serviceTier"] = service_tier.lower() + else: + data["serviceTier"] = service_tier + # Only add labels for Vertex AI endpoints (not Google GenAI/AI Studio) and only if non-empty if labels and custom_llm_provider != LlmProviders.GEMINI: data["labels"] = labels diff --git a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py index 36f51c5b2f5..e6e548ab98a 100644 --- a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py +++ b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py @@ -318,6 +318,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): "parallel_tool_calls", "web_search_options", "include_server_side_tool_invocations", + "service_tier", ] # Add penalty parameters only for non-preview models @@ -362,6 +363,17 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): """ return Tools(googleSearch={}) + def _map_service_tier_param(self, value: str, optional_params: dict) -> None: + """ + Map OpenAI service_tier (string) to Gemini serviceTier. + 'auto' maps to 'priority'. + Other values are passed lowercased. + """ + if value.lower() == "auto": + optional_params["service_tier"] = "priority" + else: + optional_params["service_tier"] = value.lower() + def _transform_computer_use_config(self, computer_use_config: dict) -> dict: """ Transform Computer Use configuration to Gemini API format. @@ -1121,6 +1133,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): optional_params = self._add_tools_to_optional_params( optional_params, [_tools] ) + elif param == "service_tier" and isinstance(value, str): + self._map_service_tier_param(value, optional_params) elif param == "include_server_side_tool_invocations" and value is True: optional_params["include_server_side_tool_invocations"] = True if litellm.vertex_ai_safety_settings is not None: @@ -2415,6 +2429,14 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig): "provider_specific_fields", {} )["traffic_type"] = traffic_type + ## ADD SERVICE TIER ## + if getattr(raw_response, "headers", None): + if service_tier := raw_response.headers.get("x-gemini-service-tier"): + if service_tier.lower() == "standard": + setattr(model_response, "service_tier", "default") + else: + setattr(model_response, "service_tier", service_tier.lower()) + except Exception as e: raise VertexAIError( message="Received={}, Error converting to valid response block={}. File an issue if litellm error - https://github.com/BerriAI/litellm/issues".format( @@ -2513,6 +2535,7 @@ async def make_call( streaming_response=response.aiter_lines(), sync_stream=False, logging_obj=logging_obj, + response_headers=response.headers, ) # LOGGING logging_obj.post_call( @@ -2555,6 +2578,7 @@ def make_sync_call( streaming_response=response.iter_lines(), sync_stream=True, logging_obj=logging_obj, + response_headers=response.headers, ) # LOGGING @@ -3011,7 +3035,11 @@ class VertexLLM(VertexBase): class ModelResponseIterator: def __init__( - self, streaming_response, sync_stream: bool, logging_obj: LoggingClass + self, + streaming_response, + sync_stream: bool, + logging_obj: LoggingClass, + response_headers: Optional[Dict[str, str]] = None, ): from litellm.litellm_core_utils.prompt_templates.common_utils import ( check_is_function_call, @@ -3022,10 +3050,120 @@ class ModelResponseIterator: self.accumulated_json = "" self.sent_first_chunk = False self.logging_obj = logging_obj + self.response_headers = response_headers or {} self.is_function_call = check_is_function_call(logging_obj) self.cumulative_tool_call_index: int = 0 self.has_seen_tool_calls: bool = False + def _apply_stream_candidates( + self, + _candidates: List[Candidates], + model_response: Any, + ) -> Tuple[List[dict], List[dict], List[dict], List[dict]]: + ( + grounding_metadata, + url_context_metadata, + safety_ratings, + citation_metadata, + self.cumulative_tool_call_index, + ) = VertexGeminiConfig._process_candidates( + _candidates, + model_response, + self.logging_obj.optional_params, + cumulative_tool_call_index=self.cumulative_tool_call_index, + ) + + # Track whether tool_calls have been seen across streaming chunks. + # Gemini sends tool_calls and finishReason in separate chunks, + # so we need to remember if earlier chunks contained tool_calls + # to correctly set finish_reason="tool_calls" per the OpenAI spec. + if not self.has_seen_tool_calls: + for choice in model_response.choices: + if ( + hasattr(choice, "delta") + and choice.delta + and choice.delta.tool_calls + ): + self.has_seen_tool_calls = True + break + + # Handle final chunk with finishReason but no content. + # _process_candidates skips candidates without "content", + # so the finish_reason from the final chunk is lost. + if not model_response.choices and _candidates: + from litellm.types.utils import Delta, StreamingChoices + + for candidate in _candidates: + finish_reason_str = candidate.get("finishReason") + if finish_reason_str is not None: + if self.has_seen_tool_calls: + mapped_finish_reason = "tool_calls" + else: + mapped_finish_reason = VertexGeminiConfig._check_finish_reason( + None, finish_reason_str + ) + choice = StreamingChoices( + finish_reason=mapped_finish_reason, + index=candidate.get("index", 0), + delta=Delta(content=None, role=None), + logprobs=None, + enhancements=None, + ) + model_response.choices.append(choice) + + # Also handle the case where the final chunk has empty + # content (e.g. text:"") WITH finishReason. In this case + # _process_candidates DOES create a choice, but maps + # finishReason="STOP" to "stop" because the current chunk + # has no tool_calls. Override if we saw tool_calls earlier. + if self.has_seen_tool_calls: + for choice in model_response.choices: + if choice.finish_reason == "stop": + choice.finish_reason = "tool_calls" + + setattr(model_response, "vertex_ai_grounding_metadata", grounding_metadata) # type: ignore + setattr(model_response, "vertex_ai_url_context_metadata", url_context_metadata) # type: ignore + setattr(model_response, "vertex_ai_safety_ratings", safety_ratings) # type: ignore + setattr(model_response, "vertex_ai_citation_metadata", citation_metadata) # type: ignore + + return grounding_metadata, url_context_metadata, safety_ratings, citation_metadata + + def _apply_stream_usage_metadata( + self, + processed_chunk: Any, + model_response: Any, + grounding_metadata: List[dict], + ) -> Optional[Usage]: + if "usageMetadata" not in processed_chunk: + return None + + usage = VertexGeminiConfig._calculate_usage( + completion_response=processed_chunk, + ) + + web_search_requests = VertexGeminiConfig._calculate_web_search_requests( + grounding_metadata + ) + if web_search_requests is not None: + cast( + PromptTokensDetailsWrapper, usage.prompt_tokens_details + ).web_search_requests = web_search_requests + + traffic_type = processed_chunk.get("usageMetadata", {}).get("trafficType") + if traffic_type: + model_response._hidden_params.setdefault( + "provider_specific_fields", {} + )["traffic_type"] = traffic_type + + service_tier = self.response_headers.get("x-gemini-service-tier") + if service_tier: + if service_tier.lower() == "standard": + setattr(model_response, "service_tier", "default") + else: + setattr(model_response, "service_tier", service_tier.lower()) + + return usage + def chunk_parser(self, chunk: dict) -> Optional["ModelResponseStream"]: try: verbose_logger.debug(f"RAW GEMINI CHUNK: {chunk}") @@ -3043,101 +3181,23 @@ class ModelResponseIterator: if blocked_response is not None: model_response = blocked_response - usage: Optional[Usage] = None - _candidates: Optional[List[Candidates]] = processed_chunk.get("candidates") grounding_metadata: List[dict] = [] url_context_metadata: List[dict] = [] safety_ratings: List[dict] = [] citation_metadata: List[dict] = [] + + _candidates: Optional[List[Candidates]] = processed_chunk.get("candidates") if _candidates: ( grounding_metadata, url_context_metadata, safety_ratings, citation_metadata, - self.cumulative_tool_call_index, - ) = VertexGeminiConfig._process_candidates( - _candidates, - model_response, - self.logging_obj.optional_params, - cumulative_tool_call_index=self.cumulative_tool_call_index, - ) + ) = self._apply_stream_candidates(_candidates, model_response) - # Track whether tool_calls have been seen across streaming chunks. - # Gemini sends tool_calls and finishReason in separate chunks, - # so we need to remember if earlier chunks contained tool_calls - # to correctly set finish_reason="tool_calls" per the OpenAI spec. - if not self.has_seen_tool_calls: - for choice in model_response.choices: - if ( - hasattr(choice, "delta") - and choice.delta - and choice.delta.tool_calls - ): - self.has_seen_tool_calls = True - break - - # Handle final chunk with finishReason but no content. - # _process_candidates skips candidates without "content", - # so the finish_reason from the final chunk is lost. - if not model_response.choices and _candidates: - from litellm.types.utils import Delta, StreamingChoices - - for candidate in _candidates: - finish_reason_str = candidate.get("finishReason") - if finish_reason_str is not None: - if self.has_seen_tool_calls: - mapped_finish_reason = "tool_calls" - else: - mapped_finish_reason = ( - VertexGeminiConfig._check_finish_reason( - None, finish_reason_str - ) - ) - choice = StreamingChoices( - finish_reason=mapped_finish_reason, - index=candidate.get("index", 0), - delta=Delta(content=None, role=None), - logprobs=None, - enhancements=None, - ) - model_response.choices.append(choice) - - # Also handle the case where the final chunk has empty - # content (e.g. text:"") WITH finishReason. In this case - # _process_candidates DOES create a choice, but maps - # finishReason="STOP" to "stop" because the current chunk - # has no tool_calls. Override if we saw tool_calls earlier. - if self.has_seen_tool_calls: - for choice in model_response.choices: - if choice.finish_reason == "stop": - choice.finish_reason = "tool_calls" - - setattr(model_response, "vertex_ai_grounding_metadata", grounding_metadata) # type: ignore - setattr(model_response, "vertex_ai_url_context_metadata", url_context_metadata) # type: ignore - setattr(model_response, "vertex_ai_safety_ratings", safety_ratings) # type: ignore - setattr(model_response, "vertex_ai_citation_metadata", citation_metadata) # type: ignore - - if "usageMetadata" in processed_chunk: - usage = VertexGeminiConfig._calculate_usage( - completion_response=processed_chunk, - ) - - web_search_requests = VertexGeminiConfig._calculate_web_search_requests( - grounding_metadata - ) - if web_search_requests is not None: - cast( - PromptTokensDetailsWrapper, usage.prompt_tokens_details - ).web_search_requests = web_search_requests - - traffic_type = processed_chunk.get("usageMetadata", {}).get( - "trafficType" - ) - if traffic_type: - model_response._hidden_params.setdefault( - "provider_specific_fields", {} - )["traffic_type"] = traffic_type + usage = self._apply_stream_usage_metadata( + processed_chunk, model_response, grounding_metadata + ) setattr(model_response, "usage", usage) # type: ignore diff --git a/litellm/llms/vertex_ai/vertex_llm_base.py b/litellm/llms/vertex_ai/vertex_llm_base.py index cdabac27af7..68d8f0d046d 100644 --- a/litellm/llms/vertex_ai/vertex_llm_base.py +++ b/litellm/llms/vertex_ai/vertex_llm_base.py @@ -136,6 +136,11 @@ class VertexBase: json_obj, scopes=["https://www.googleapis.com/auth/cloud-platform"], ) + elif isinstance(credential_source, dict) and "executable" in credential_source: + creds = self._credentials_from_pluggable( + json_obj, + scopes=["https://www.googleapis.com/auth/cloud-platform"], + ) else: creds = self._credentials_from_identity_pool( json_obj, @@ -190,6 +195,17 @@ class VertexBase: creds = creds.with_scopes(scopes) return creds + def _credentials_from_pluggable(self, json_obj, scopes): + try: + from google.auth import pluggable + except ImportError: + raise ImportError(GOOGLE_IMPORT_ERROR_MESSAGE) + + creds = pluggable.Credentials.from_info(json_obj) + if scopes and hasattr(creds, "requires_scopes") and creds.requires_scopes: + creds = creds.with_scopes(scopes) + return creds + def _credentials_from_identity_pool_with_aws(self, json_obj, scopes): try: from google.auth import aws diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json index d781c91992d..479231deac8 100644 --- a/litellm/model_prices_and_context_window_backup.json +++ b/litellm/model_prices_and_context_window_backup.json @@ -7818,26 +7818,6 @@ "cache_read_input_token_cost": 3e-08, "cache_creation_input_token_cost": 3.75e-07 }, - "bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": { - "input_cost_per_token": 3.6e-06, - "litellm_provider": "bedrock", - "max_input_tokens": 200000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat", - "output_cost_per_token": 1.8e-05, - "supports_assistant_prefill": true, - "supports_computer_use": true, - "supports_function_calling": true, - "supports_pdf_input": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_response_schema": true, - "supports_tool_choice": true, - "supports_vision": true, - "cache_read_input_token_cost": 3.6e-07, - "cache_creation_input_token_cost": 4.5e-06 - }, "bedrock/us-gov-east-1/anthropic.claude-sonnet-4-5-20250929-v1:0": { "cache_creation_input_token_cost": 4.125e-06, "cache_read_input_token_cost": 3.3e-07, @@ -7856,7 +7836,29 @@ "supports_reasoning": true, "supports_response_schema": true, "supports_tool_choice": true, - "supports_vision": true + "supports_vision": true, + "supports_native_structured_output": true + }, + "bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": { + "cache_creation_input_token_cost": 4.125e-06, + "cache_read_input_token_cost": 3.3e-07, + "input_cost_per_token": 3.3e-06, + "litellm_provider": "bedrock", + "max_input_tokens": 200000, + "max_output_tokens": 8192, + "max_tokens": 8192, + "mode": "chat", + "output_cost_per_token": 1.65e-05, + "supports_assistant_prefill": true, + "supports_computer_use": true, + "supports_function_calling": true, + "supports_pdf_input": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_response_schema": true, + "supports_tool_choice": true, + "supports_vision": true, + "supports_native_structured_output": true }, "bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": { "input_cost_per_token": 2.65e-06, @@ -7989,26 +7991,6 @@ "cache_read_input_token_cost": 3e-08, "cache_creation_input_token_cost": 3.75e-07 }, - "bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": { - "input_cost_per_token": 3.6e-06, - "litellm_provider": "bedrock", - "max_input_tokens": 200000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat", - "output_cost_per_token": 1.8e-05, - "supports_assistant_prefill": true, - "supports_computer_use": true, - "supports_function_calling": true, - "supports_pdf_input": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_response_schema": true, - "supports_tool_choice": true, - "supports_vision": true, - "cache_read_input_token_cost": 3.6e-07, - "cache_creation_input_token_cost": 4.5e-06 - }, "bedrock/us-gov-west-1/anthropic.claude-sonnet-4-5-20250929-v1:0": { "cache_creation_input_token_cost": 4.125e-06, "cache_read_input_token_cost": 3.3e-07, @@ -8027,7 +8009,29 @@ "supports_reasoning": true, "supports_response_schema": true, "supports_tool_choice": true, - "supports_vision": true + "supports_vision": true, + "supports_native_structured_output": true + }, + "bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": { + "cache_creation_input_token_cost": 4.125e-06, + "cache_read_input_token_cost": 3.3e-07, + "input_cost_per_token": 3.3e-06, + "litellm_provider": "bedrock", + "max_input_tokens": 200000, + "max_output_tokens": 8192, + "max_tokens": 8192, + "mode": "chat", + "output_cost_per_token": 1.65e-05, + "supports_assistant_prefill": true, + "supports_computer_use": true, + "supports_function_calling": true, + "supports_pdf_input": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_response_schema": true, + "supports_tool_choice": true, + "supports_vision": true, + "supports_native_structured_output": true }, "bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": { "input_cost_per_token": 2.65e-06, @@ -13735,7 +13739,8 @@ "supports_tool_choice": true, "supports_url_context": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-2.5-flash-image": { "cache_read_input_token_cost": 3e-08, @@ -13784,7 +13789,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": false, - "tpm": 8000000 + "tpm": 8000000, + "supports_service_tier": true }, "gemini-3-pro-image-preview": { "input_cost_per_image": 0.0011, @@ -13818,7 +13824,8 @@ "supports_response_schema": true, "supports_system_messages": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-3.1-flash-image-preview": { "input_cost_per_image": 0.00056, @@ -13901,7 +13908,8 @@ "supports_video_input": true, "supports_vision": true, "supports_web_search": true, - "supports_native_streaming": true + "supports_native_streaming": true, + "supports_service_tier": true }, "deep-research-pro-preview-12-2025": { "input_cost_per_image": 0.0011, @@ -13980,7 +13988,8 @@ "supports_tool_choice": true, "supports_url_context": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-2.5-flash-lite-preview-09-2025": { "cache_read_input_token_cost": 1e-08, @@ -14251,7 +14260,8 @@ "supports_tool_choice": true, "supports_video_input": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-3-pro-preview": { "deprecation_date": "2026-03-26", @@ -15033,7 +15043,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": true, - "tpm": 8000000 + "tpm": 8000000, + "supports_service_tier": true }, "gemini/gemini-2.5-flash-image": { "cache_read_input_token_cost": 3e-08, @@ -15083,7 +15094,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": true, - "tpm": 8000000 + "tpm": 8000000, + "supports_service_tier": true }, "gemini/gemini-3-pro-image-preview": { "input_cost_per_image": 0.0011, @@ -15119,7 +15131,8 @@ "supports_response_schema": true, "supports_system_messages": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini/gemini-3.1-flash-image-preview": { "input_cost_per_token": 2.5e-07, @@ -15238,7 +15251,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": true, - "tpm": 250000 + "tpm": 250000, + "supports_service_tier": true }, "gemini/gemini-2.5-flash-lite-preview-09-2025": { "cache_read_input_token_cost": 1e-08, @@ -15678,7 +15692,8 @@ "supports_vision": true, "supports_web_search": true, "supports_native_streaming": true, - "tpm": 250000 + "tpm": 250000, + "supports_service_tier": true }, "gemini/gemini-3-flash-preview": { "cache_read_input_token_cost": 5e-08, @@ -38123,4 +38138,4 @@ "supports_native_structured_output": true, "supports_pdf_input": true } -} +} \ No newline at end of file diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index cf99c5cd9fa..793742891fc 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -2435,6 +2435,7 @@ class LiteLLM_VerificationTokenView(LiteLLM_VerificationToken): end_user_model_max_budget: Optional[dict] = None # Organization Params + organization_alias: Optional[str] = None organization_max_budget: Optional[float] = None organization_tpm_limit: Optional[int] = None organization_rpm_limit: Optional[int] = None diff --git a/litellm/proxy/litellm_pre_call_utils.py b/litellm/proxy/litellm_pre_call_utils.py index 3ed96c163af..ece72b1060e 100644 --- a/litellm/proxy/litellm_pre_call_utils.py +++ b/litellm/proxy/litellm_pre_call_utils.py @@ -684,6 +684,7 @@ class LiteLLMProxyRequestSetup: user_api_key_project_alias=user_api_key_dict.project_alias, user_api_key_user_id=user_api_key_dict.user_id, user_api_key_org_id=user_api_key_dict.org_id, + user_api_key_org_alias=user_api_key_dict.organization_alias, user_api_key_team_alias=user_api_key_dict.team_alias, user_api_key_end_user_id=user_api_key_dict.end_user_id, user_api_key_user_email=user_api_key_dict.user_email, diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py index 9738ae4f1a2..85a12f70f58 100644 --- a/litellm/proxy/proxy_server.py +++ b/litellm/proxy/proxy_server.py @@ -7128,6 +7128,13 @@ async def chat_completion( # noqa: PLR0915 and user_api_key_dict.org_id is not None ): data["metadata"]["user_api_key_org_id"] = user_api_key_dict.org_id + if ( + hasattr(user_api_key_dict, "organization_alias") + and user_api_key_dict.organization_alias is not None + ): + data["metadata"]["user_api_key_org_alias"] = ( + user_api_key_dict.organization_alias + ) if ( hasattr(user_api_key_dict, "agent_id") and user_api_key_dict.agent_id is not None @@ -7302,6 +7309,13 @@ async def completion( # noqa: PLR0915 and user_api_key_dict.org_id is not None ): data["metadata"]["user_api_key_org_id"] = user_api_key_dict.org_id + if ( + hasattr(user_api_key_dict, "organization_alias") + and user_api_key_dict.organization_alias is not None + ): + data["metadata"]["user_api_key_org_alias"] = ( + user_api_key_dict.organization_alias + ) if ( hasattr(user_api_key_dict, "agent_id") and user_api_key_dict.agent_id is not None @@ -7544,6 +7558,13 @@ async def embeddings( # noqa: PLR0915 and user_api_key_dict.org_id is not None ): data["metadata"]["user_api_key_org_id"] = user_api_key_dict.org_id + if ( + hasattr(user_api_key_dict, "organization_alias") + and user_api_key_dict.organization_alias is not None + ): + data["metadata"]["user_api_key_org_alias"] = ( + user_api_key_dict.organization_alias + ) if ( hasattr(user_api_key_dict, "agent_id") and user_api_key_dict.agent_id is not None diff --git a/litellm/proxy/utils.py b/litellm/proxy/utils.py index ec98cfd4d1e..635204f3362 100644 --- a/litellm/proxy/utils.py +++ b/litellm/proxy/utils.py @@ -3004,6 +3004,7 @@ class PrismaClient: b.model_max_budget as litellm_budget_table_model_max_budget, b.soft_budget as litellm_budget_table_soft_budget, o.metadata as organization_metadata, + o.organization_alias as organization_alias, b2.max_budget as organization_max_budget, b2.tpm_limit as organization_tpm_limit, b2.rpm_limit as organization_rpm_limit @@ -5293,11 +5294,12 @@ def handle_exception_on_proxy(e: Exception) -> ProxyException: ) elif isinstance(e, ProxyException): return e + _status_code = getattr(e, "status_code", status.HTTP_500_INTERNAL_SERVER_ERROR) return ProxyException( - message="Internal Server Error, " + str(e), + message=str(e), type=ProxyErrorTypes.internal_server_error, param=getattr(e, "param", "None"), - code=status.HTTP_500_INTERNAL_SERVER_ERROR, + code=_status_code, ) diff --git a/litellm/types/integrations/prometheus.py b/litellm/types/integrations/prometheus.py index 0d1501664b9..5f1aa9fb2ce 100644 --- a/litellm/types/integrations/prometheus.py +++ b/litellm/types/integrations/prometheus.py @@ -1,7 +1,7 @@ import re from dataclasses import dataclass from enum import Enum -from typing import Any, Dict, List, Literal, Optional, Tuple +from typing import Any, ClassVar, Dict, List, Literal, Optional, Tuple from pydantic import BaseModel, Field, field_validator from typing_extensions import Annotated @@ -665,6 +665,24 @@ class PrometheusMetricLabels: litellm_cache_misses_metric = _cache_metric_labels litellm_cached_tokens_metric = _cache_metric_labels + # Metrics whose emission paths supply org context (used by get_labels) + _org_label_metrics: ClassVar[frozenset] = frozenset( + { + "litellm_llm_api_latency_metric", + "litellm_llm_api_time_to_first_token_metric", + "litellm_request_total_latency_metric", + "litellm_request_queue_time_seconds", + "litellm_proxy_total_requests_metric", + "litellm_proxy_failed_requests_metric", + "litellm_deployment_latency_per_output_token", + "litellm_requests_metric", + "litellm_spend_metric", + "litellm_input_tokens_metric", + "litellm_total_tokens_metric", + "litellm_output_tokens_metric", + } + ) + # Managed batch metrics _batch_user_labels = [ UserAPIKeyLabelNames.v1_LITELLM_MODEL_NAME.value, @@ -731,6 +749,14 @@ class PrometheusMetricLabels: ): custom_labels.append(UserAPIKeyLabelNames.STREAM.value) + if label_name in PrometheusMetricLabels._org_label_metrics: + for label in [ + UserAPIKeyLabelNames.ORG_ID.value, + UserAPIKeyLabelNames.ORG_ALIAS.value, + ]: + if label not in default_labels and label not in custom_labels: + custom_labels.append(label) + return default_labels + custom_labels diff --git a/litellm/types/llms/vertex_ai.py b/litellm/types/llms/vertex_ai.py index c49fc96a65b..86d7b926214 100644 --- a/litellm/types/llms/vertex_ai.py +++ b/litellm/types/llms/vertex_ai.py @@ -325,6 +325,7 @@ class RequestBody(TypedDict, total=False): generationConfig: GenerationConfig cachedContent: str labels: Dict[str, str] + serviceTier: str class CachedContentRequestBody(TypedDict, total=False): diff --git a/litellm/types/utils.py b/litellm/types/utils.py index 3f6e6e5aa5a..cd5806b3ab7 100644 --- a/litellm/types/utils.py +++ b/litellm/types/utils.py @@ -2507,6 +2507,7 @@ class StandardLoggingUserAPIKeyMetadata(TypedDict): user_api_key_max_budget: Optional[float] user_api_key_budget_reset_at: Optional[str] user_api_key_org_id: Optional[str] + user_api_key_org_alias: Optional[str] user_api_key_team_id: Optional[str] user_api_key_project_id: Optional[str] user_api_key_project_alias: Optional[str] diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json index cfdb2911fdf..e579ab692b2 100644 --- a/model_prices_and_context_window.json +++ b/model_prices_and_context_window.json @@ -7818,26 +7818,6 @@ "cache_read_input_token_cost": 3e-08, "cache_creation_input_token_cost": 3.75e-07 }, - "bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": { - "input_cost_per_token": 3.6e-06, - "litellm_provider": "bedrock", - "max_input_tokens": 200000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat", - "output_cost_per_token": 1.8e-05, - "supports_assistant_prefill": true, - "supports_computer_use": true, - "supports_function_calling": true, - "supports_pdf_input": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_response_schema": true, - "supports_tool_choice": true, - "supports_vision": true, - "cache_read_input_token_cost": 3.6e-07, - "cache_creation_input_token_cost": 4.5e-06 - }, "bedrock/us-gov-east-1/anthropic.claude-sonnet-4-5-20250929-v1:0": { "cache_creation_input_token_cost": 4.125e-06, "cache_read_input_token_cost": 3.3e-07, @@ -7856,7 +7836,29 @@ "supports_reasoning": true, "supports_response_schema": true, "supports_tool_choice": true, - "supports_vision": true + "supports_vision": true, + "supports_native_structured_output": true + }, + "bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": { + "cache_creation_input_token_cost": 4.125e-06, + "cache_read_input_token_cost": 3.3e-07, + "input_cost_per_token": 3.3e-06, + "litellm_provider": "bedrock", + "max_input_tokens": 200000, + "max_output_tokens": 8192, + "max_tokens": 8192, + "mode": "chat", + "output_cost_per_token": 1.65e-05, + "supports_assistant_prefill": true, + "supports_computer_use": true, + "supports_function_calling": true, + "supports_pdf_input": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_response_schema": true, + "supports_tool_choice": true, + "supports_vision": true, + "supports_native_structured_output": true }, "bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": { "input_cost_per_token": 2.65e-06, @@ -7989,26 +7991,6 @@ "cache_read_input_token_cost": 3e-08, "cache_creation_input_token_cost": 3.75e-07 }, - "bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": { - "input_cost_per_token": 3.6e-06, - "litellm_provider": "bedrock", - "max_input_tokens": 200000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat", - "output_cost_per_token": 1.8e-05, - "supports_assistant_prefill": true, - "supports_computer_use": true, - "supports_function_calling": true, - "supports_pdf_input": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_response_schema": true, - "supports_tool_choice": true, - "supports_vision": true, - "cache_read_input_token_cost": 3.6e-07, - "cache_creation_input_token_cost": 4.5e-06 - }, "bedrock/us-gov-west-1/anthropic.claude-sonnet-4-5-20250929-v1:0": { "cache_creation_input_token_cost": 4.125e-06, "cache_read_input_token_cost": 3.3e-07, @@ -8027,7 +8009,29 @@ "supports_reasoning": true, "supports_response_schema": true, "supports_tool_choice": true, - "supports_vision": true + "supports_vision": true, + "supports_native_structured_output": true + }, + "bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": { + "cache_creation_input_token_cost": 4.125e-06, + "cache_read_input_token_cost": 3.3e-07, + "input_cost_per_token": 3.3e-06, + "litellm_provider": "bedrock", + "max_input_tokens": 200000, + "max_output_tokens": 8192, + "max_tokens": 8192, + "mode": "chat", + "output_cost_per_token": 1.65e-05, + "supports_assistant_prefill": true, + "supports_computer_use": true, + "supports_function_calling": true, + "supports_pdf_input": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_response_schema": true, + "supports_tool_choice": true, + "supports_vision": true, + "supports_native_structured_output": true }, "bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": { "input_cost_per_token": 2.65e-06, @@ -13735,7 +13739,8 @@ "supports_tool_choice": true, "supports_url_context": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-2.5-flash-image": { "cache_read_input_token_cost": 3e-08, @@ -13784,7 +13789,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": false, - "tpm": 8000000 + "tpm": 8000000, + "supports_service_tier": true }, "gemini-3-pro-image-preview": { "input_cost_per_image": 0.0011, @@ -13818,7 +13824,8 @@ "supports_response_schema": true, "supports_system_messages": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-3.1-flash-image-preview": { "input_cost_per_image": 0.00056, @@ -13901,7 +13908,8 @@ "supports_video_input": true, "supports_vision": true, "supports_web_search": true, - "supports_native_streaming": true + "supports_native_streaming": true, + "supports_service_tier": true }, "deep-research-pro-preview-12-2025": { "input_cost_per_image": 0.0011, @@ -13980,7 +13988,8 @@ "supports_tool_choice": true, "supports_url_context": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-2.5-flash-lite-preview-09-2025": { "cache_read_input_token_cost": 1e-08, @@ -14251,7 +14260,8 @@ "supports_tool_choice": true, "supports_video_input": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini-3-pro-preview": { "deprecation_date": "2026-03-26", @@ -15033,7 +15043,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": true, - "tpm": 8000000 + "tpm": 8000000, + "supports_service_tier": true }, "gemini/gemini-2.5-flash-image": { "cache_read_input_token_cost": 3e-08, @@ -15083,7 +15094,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": true, - "tpm": 8000000 + "tpm": 8000000, + "supports_service_tier": true }, "gemini/gemini-3-pro-image-preview": { "input_cost_per_image": 0.0011, @@ -15119,7 +15131,8 @@ "supports_response_schema": true, "supports_system_messages": true, "supports_vision": true, - "supports_web_search": true + "supports_web_search": true, + "supports_service_tier": true }, "gemini/gemini-3.1-flash-image-preview": { "input_cost_per_token": 2.5e-07, @@ -15238,7 +15251,8 @@ "supports_url_context": true, "supports_vision": true, "supports_web_search": true, - "tpm": 250000 + "tpm": 250000, + "supports_service_tier": true }, "gemini/gemini-2.5-flash-lite-preview-09-2025": { "cache_read_input_token_cost": 1e-08, @@ -15678,7 +15692,8 @@ "supports_vision": true, "supports_web_search": true, "supports_native_streaming": true, - "tpm": 250000 + "tpm": 250000, + "supports_service_tier": true }, "gemini/gemini-3-flash-preview": { "cache_read_input_token_cost": 5e-08, @@ -38108,4 +38123,4 @@ "supports_native_structured_output": true, "supports_pdf_input": true } -} +} \ No newline at end of file diff --git a/tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py b/tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py index 967ef2a5fec..834cb235f0c 100644 --- a/tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py +++ b/tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py @@ -223,6 +223,8 @@ def test_increment_token_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model=None, model="gpt-3.5-turbo", model_id="model-123", @@ -237,6 +239,8 @@ def test_increment_token_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model=None, model="gpt-3.5-turbo", model_id="model-123", @@ -253,6 +257,8 @@ def test_increment_token_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model=None, model="gpt-3.5-turbo", model_id="model-123", @@ -414,6 +420,8 @@ def test_set_latency_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model="openai-gpt", model="gpt-3.5-turbo", model_id="model-123", @@ -430,6 +438,8 @@ def test_set_latency_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model="openai-gpt", model="gpt-3.5-turbo", model_id="model-123", @@ -446,6 +456,8 @@ def test_set_latency_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model="openai-gpt", model="gpt-3.5-turbo", model_id="model-123", @@ -589,6 +601,8 @@ def test_increment_top_level_request_and_spend_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, model="gpt-3.5-turbo", model_id="model-123", client_ip=None, @@ -605,6 +619,8 @@ def test_increment_top_level_request_and_spend_metrics(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, model="gpt-3.5-turbo", model_id="model-123", client_ip=None, @@ -758,6 +774,8 @@ async def test_async_post_call_failure_hook(prometheus_logger): api_key_alias="test_alias", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, requested_model="gpt-3.5-turbo", exception_status="429", exception_class="Openai.RateLimitError", @@ -776,6 +794,8 @@ async def test_async_post_call_failure_hook(prometheus_logger): requested_model="gpt-3.5-turbo", team="test_team", team_alias="test_team_alias", + org_id=None, + org_alias=None, user="test_user", status_code="429", user_email=None, @@ -955,6 +975,8 @@ def test_set_llm_deployment_success_metrics(prometheus_logger): api_key_alias=standard_logging_payload["metadata"]["user_api_key_alias"], team=standard_logging_payload["metadata"]["user_api_key_team_id"], team_alias=standard_logging_payload["metadata"]["user_api_key_team_alias"], + org_id=None, + org_alias=None, ) prometheus_logger.litellm_overhead_latency_metric.labels.assert_called_once_with( api_base="https://api.openai.com", diff --git a/tests/proxy_unit_tests/test_proxy_utils.py b/tests/proxy_unit_tests/test_proxy_utils.py index 6c6ec7bcd60..09f6a85938d 100644 --- a/tests/proxy_unit_tests/test_proxy_utils.py +++ b/tests/proxy_unit_tests/test_proxy_utils.py @@ -2637,3 +2637,50 @@ async def test_handle_logging_proxy_only_error_skips_handlers_for_pass_through() mock_async.assert_not_called() mock_sync.assert_not_called() assert logging_obj.call_type == CallTypes.pass_through.value + + +def test_handle_exception_on_proxy_preserves_status_code(): + """ + OpenAI batch creation returns 429 for rate limits. LiteLLM wraps this as a + RateLimitError with status_code=429. handle_exception_on_proxy must pass + that status code through instead of hardcoding 500. + """ + from litellm.proxy.utils import handle_exception_on_proxy + + rate_limit_error = litellm.RateLimitError( + message="Rate limit exceeded: batch creation limit of 2000/hour hit", + llm_provider="openai", + model="gpt-4o", + ) + + result = handle_exception_on_proxy(rate_limit_error) + + assert int(result.code) == 429, f"Expected 429, got {result.code}" + + +def test_handle_exception_on_proxy_defaults_to_500_for_unknown_exceptions(): + """ + Generic exceptions with no status_code should still return 500. + """ + from litellm.proxy.utils import handle_exception_on_proxy + + result = handle_exception_on_proxy(Exception("something went wrong")) + + assert int(result.code) == 500, f"Expected 500, got {result.code}" + + +def test_handle_exception_on_proxy_preserves_auth_error_status_code(): + """ + AuthenticationError (401) should also pass through correctly. + """ + from litellm.proxy.utils import handle_exception_on_proxy + + auth_error = litellm.AuthenticationError( + message="Invalid API key", + llm_provider="openai", + model="gpt-4o", + ) + + result = handle_exception_on_proxy(auth_error) + + assert int(result.code) == 401, f"Expected 401, got {result.code}" diff --git a/tests/test_litellm/integrations/test_prometheus_client_ip_user_agent.py b/tests/test_litellm/integrations/test_prometheus_client_ip_user_agent.py index 4bfa3a581e3..48d9cbd1bb1 100644 --- a/tests/test_litellm/integrations/test_prometheus_client_ip_user_agent.py +++ b/tests/test_litellm/integrations/test_prometheus_client_ip_user_agent.py @@ -118,6 +118,8 @@ async def test_async_post_call_success_hook_includes_client_ip_user_agent(): "user_api_key_alias": "alias_1", "user_api_key_team_id": "team_1", "user_api_key_team_alias": "team_alias_1", + "user_api_key_org_id": None, + "user_api_key_org_alias": None, "user_api_key_user_email": "test@example.com", "user_api_key_request_route": "/chat/completions", "requester_ip_address": "192.168.1.1", diff --git a/tests/test_litellm/integrations/test_prometheus_user_team_metrics.py b/tests/test_litellm/integrations/test_prometheus_user_team_metrics.py index 9bcf08fdd71..6b65f444046 100644 --- a/tests/test_litellm/integrations/test_prometheus_user_team_metrics.py +++ b/tests/test_litellm/integrations/test_prometheus_user_team_metrics.py @@ -525,6 +525,61 @@ async def test_set_user_budget_metrics_after_api_request_inf_when_genuinely_no_b ) +def test_per_request_metrics_emit_all_identity_labels(prometheus_logger): + """Verify org labels appear when flag is on and are absent when flag is off.""" + import litellm + from litellm.types.integrations.prometheus import UserAPIKeyLabelValues + + prometheus_logger.litellm_requests_metric = MagicMock() + prometheus_logger.litellm_spend_metric = MagicMock() + + enum_values = UserAPIKeyLabelValues( + hashed_api_key="hashed-key", + api_key_alias="my-key", + model="gpt-4", + team="team-abc", + team_alias="my-team", + org_id="org-abc", + org_alias="my-org", + user="user-1", + ) + + common_kwargs = dict( + end_user_id=None, + user_api_key="hashed-key", + user_api_key_alias="my-key", + model="gpt-4", + user_api_team="team-abc", + user_api_team_alias="my-team", + user_id="user-1", + response_cost=0.001, + enum_values=enum_values, + ) + + try: + # org labels are always included in per-request metrics + prometheus_logger._increment_top_level_request_and_spend_metrics(**common_kwargs) + label_kwargs = prometheus_logger.litellm_requests_metric.labels.call_args.kwargs + assert label_kwargs["org_id"] == "org-abc" + assert label_kwargs["org_alias"] == "my-org" + assert label_kwargs["team"] == "team-abc" + assert label_kwargs["user"] == "user-1" + + # Metrics not in the org-emission list must NOT get org labels + from litellm.types.integrations.prometheus import PrometheusMetricLabels + for metric in ("litellm_remaining_api_key_budget_metric", "litellm_remaining_team_budget_metric"): + labels = PrometheusMetricLabels.get_labels(metric) + assert "org_id" not in labels, f"{metric} should not have org_id" + assert "org_alias" not in labels, f"{metric} should not have org_alias" + + # org_id in custom_prometheus_metadata_labels must not produce duplicate labels + litellm.custom_prometheus_metadata_labels = ["org_id"] + labels = PrometheusMetricLabels.get_labels("litellm_requests_metric") + assert labels.count("org_id") == 1 + finally: + litellm.custom_prometheus_metadata_labels = [] + + # --------------------------------------------------------------------------- # Org budget metric tests # --------------------------------------------------------------------------- diff --git a/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py b/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py index 20427e8cc94..bc40919525e 100644 --- a/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py +++ b/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py @@ -1,7 +1,9 @@ -from unittest.mock import MagicMock +from unittest.mock import AsyncMock, MagicMock + +import pytest from litellm.constants import RESPONSE_FORMAT_TOOL_NAME -from litellm.llms.anthropic.chat.handler import ModelResponseIterator +from litellm.llms.anthropic.chat.handler import ModelResponseIterator, make_call from litellm.types.llms.openai import ( ChatCompletionToolCallChunk, ChatCompletionToolCallFunctionChunk, @@ -9,6 +11,33 @@ from litellm.types.llms.openai import ( from litellm.types.responses.main import OutputCodeInterpreterCall +@pytest.mark.asyncio +async def test_make_call_passes_logging_obj_to_client_post(): + """make_call must pass logging_obj to client.post so track_llm_api_timing can set llm_api_duration_ms for litellm_overhead_time_ms.""" + mock_client = AsyncMock() + mock_response = MagicMock() + mock_response.aiter_lines = MagicMock(return_value=iter([b'data: {"type":"message_start"}\n', b'data: {"type":"message_delta"}\n'])) + mock_client.post.return_value = mock_response + + logging_obj = MagicMock() + + await make_call( + client=mock_client, + api_base="https://api.anthropic.com/v1/messages", + headers={}, + data="{}", + model="claude-3-5-haiku", + messages=[{"role": "user", "content": "Hi"}], + logging_obj=logging_obj, + timeout=60.0, + json_mode=False, + ) + + mock_client.post.assert_called_once() + call_kwargs = mock_client.post.call_args[1] + assert call_kwargs.get("logging_obj") is logging_obj + + def test_redacted_thinking_content_block_delta(): chunk = { "type": "content_block_start", diff --git a/tests/test_litellm/llms/sap/chat/test_sap_tool_parameters.py b/tests/test_litellm/llms/sap/chat/test_sap_tool_parameters.py index 792d2f3fe6b..a5a3fa40d98 100644 --- a/tests/test_litellm/llms/sap/chat/test_sap_tool_parameters.py +++ b/tests/test_litellm/llms/sap/chat/test_sap_tool_parameters.py @@ -127,11 +127,12 @@ class TestToolTransformationIntegration: } validated_tool = validate_dict(openai_tool, ChatCompletionTool) - + # After validation, parameters should have type='object' assert validated_tool["function"]["parameters"]["type"] == "object" assert "properties" in validated_tool["function"]["parameters"] + def test_should_transform_tool_with_existing_parameters(self): """Tool with parameters should preserve them while ensuring type='object'.""" from litellm.llms.sap.chat.transformation import validate_dict diff --git a/tests/test_litellm/llms/sap/chat/test_sap_transformation.py b/tests/test_litellm/llms/sap/chat/test_sap_transformation.py new file mode 100644 index 00000000000..15ce1c85e8f --- /dev/null +++ b/tests/test_litellm/llms/sap/chat/test_sap_transformation.py @@ -0,0 +1,564 @@ +import warnings +import pytest +from pydantic import ValidationError + +class TestSAPTransformationIntegration: + """Integration tests for SAP transformation.""" + + @pytest.fixture + def mock_config(self): + from litellm.llms.sap.chat.transformation import GenAIHubOrchestrationConfig + + config = GenAIHubOrchestrationConfig() + config.token_creator = lambda: "Bearer TEST_TOKEN" + config._base_url = "https://api.test-sap.com" + config._resource_group = "test-group" + + return config + + def test_parameter_classification_in_transform_request(self, mock_config): + """Test parameter classification within the actual transform_request method.""" + + model = "gpt-4o" + messages = [{"role": "user", "content": "Hello"}] + + optional_params = { + "temperature": 0.7, + "max_tokens": 100, + "deployment_url": "https://custom.sap.com/deployment/123", + "model_version": "v1.5", + "tools": [{"type": "function", "function": {"name": "calculator"}}], + "frequency_penalty": 0.1 + } + + result = mock_config.transform_request( + model, messages, optional_params, {}, {} + ) + + model_params = result["config"]["modules"]["prompt_templating"]["model"]["params"] + + assert "temperature" in model_params + assert "frequency_penalty" in model_params + assert "deployment_url" not in model_params + assert "model_version" not in model_params + assert "tools" not in model_params + + model_version = result["config"]["modules"]["prompt_templating"]["model"]["version"] + assert model_version == "v1.5" + + prompt = result["config"]["modules"]["prompt_templating"]["prompt"] + if "tools" in prompt: + assert isinstance(prompt["tools"], list) + for tool in prompt["tools"]: + assert tool["function"]["parameters"]["type"] == "object", ( + "SAP API requires parameters.type == 'object'" + ) + assert "properties" in tool["function"]["parameters"] + + def test_transform_request_parameter_handling_robustness(self, mock_config): + """Test transform_request method handles various parameter combinations correctly.""" + + model = "gpt-4o" + messages = [{"role": "user", "content": "Hello"}] + + test_cases = [ + # Case 1: Basic parameters only + { + "params": {"temperature": 0.7, "max_tokens": 100}, + "expected_in_model": {"temperature", "max_tokens"}, + "expected_excluded": set() + }, + # Case 2: Parameters with auth/infrastructure components + { + "params": { + "temperature": 0.8, + "deployment_url": "https://api.sap.com/deployments/test", + "max_tokens": 150 + }, + "expected_in_model": {"temperature", "max_tokens"}, + "expected_excluded": {"deployment_url"} + }, + # Case 3: Parameters with framework components + { + "params": { + "temperature": 0.6, + "model_version": "v2.0", + "tools": [{"function": {"name": "test"}}], + "frequency_penalty": 0.1 + }, + "expected_in_model": {"temperature", "frequency_penalty"}, + "expected_excluded": {"model_version", "tools"} + } + ] + + for i, test_case in enumerate(test_cases): + filtered_params = { + k: v for k, v in test_case["params"].items() + if k not in {"tools", "model_version", "deployment_url"} + } + + for expected_param in test_case["expected_in_model"]: + assert expected_param in filtered_params, f"Case {i + 1}: {expected_param} should be in model params" + + for excluded_param in test_case["expected_excluded"]: + assert excluded_param not in filtered_params, f"Case {i + 1}: {excluded_param} should be excluded from model params" + + result = mock_config.transform_request( + model, messages, test_case["params"], {}, {} + ) + if result and "config" in result: + model_params = result["config"]["modules"]["prompt_templating"]["model"]["params"] + + for excluded_param in test_case["expected_excluded"]: + assert excluded_param not in model_params, ( + f"Case {i + 1}: {excluded_param} should not be in actual model params" + ) + + def test_config_transform_with_response_format_json_object(self, mock_config): + expected_dict = {'config': + {'modules': + {'prompt_templating': + {'prompt': + {'template': + [{'role': 'user', 'content': 'First man on the moon, answer in json'}], + 'response_format': {'type': 'json_object'}}, + 'model': {'name': 'gpt-4o', 'params': {}, 'version': 'latest'} + } + }, + } + } + config = mock_config.transform_request( + model="gpt-4o", + messages=[{'role': 'user', 'content': 'First man on the moon, answer in json'}], + optional_params={'response_format': {'type': 'json_object'}, + 'deployment_url': "shouldn't be in results"}, + litellm_params={}, + headers={} + ) + assert config == expected_dict + + def test_config_transform_with_response_format_json_schema(self, mock_config): + + expected_response_format = { + 'type': 'json_schema', + 'json_schema': { + 'description': 'Schema for person information', + 'name': 'person_info', + 'schema': { + 'type': 'object', + 'properties': { + 'name': { + 'type': 'string', + 'description': "The person's full name" + }, + 'age': { + 'type': 'integer', + 'description': "The person's age in years" + }, + 'occupation': { + 'type': 'string', + 'description': "The person's job title" + } + }, + 'required': ['name', 'age', 'occupation'], + 'additionalProperties': False + }, + 'strict': True + } + } + + config = mock_config.transform_request( + model="gpt-4o", + messages=[{'role': 'user', 'content': 'First man on the moon, answer in json'}], + optional_params={'response_format': expected_response_format, + 'deployment_url': "shouldn't be in results"}, + litellm_params={}, + headers={} + ) + assert config["config"]["modules"]["prompt_templating"]["prompt"]["response_format"] == expected_response_format + assert len(config["config"]["modules"]["prompt_templating"]["model"]["params"]) == 0 + + def test_config_transform_with_stream(self, mock_config): + expected_dict = { + 'config': { + 'modules': { + 'prompt_templating': { + 'prompt': { + 'template': [{'role': 'user', 'content': 'Hello, how are you?'}] + }, + 'model': { + 'name': 'anthropic--claude-4-sonnet', + 'params': {}, + 'version': 'latest' + } + } + }, + 'stream': {'chunk_size': 10} + } + } + config = mock_config.transform_request( + model="anthropic--claude-4-sonnet", + messages=[{'content': 'Hello, how are you?', 'role': 'user'}], + optional_params={'stream': True, + 'stream_options': {'chunk_size': 10}, + 'model_version': 'latest', + 'deployment_url': "shouldn't be in results"}, + litellm_params={}, + headers={} + ) + + assert config == expected_dict + + def test_sap_placeholder_defaults(self, mock_config): + config = mock_config.transform_request( + model="gpt-4o", + messages=[ + {"role": "user", "content": "Hello. Answer {{ ?user_query }}"} + ], + optional_params={'deployment_url': "shouldn't be in results", + "placeholder_defaults": {"user_query": "default value"}}, + litellm_params={}, + headers={} + ) + + assert config["config"]["modules"]["prompt_templating"]["prompt"]["defaults"] == { + "user_query": "default value"} + assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {} + + def test_sap_placeholder_values(self, mock_config): + placeholder_values = {"user_query": "Some text"} + config = mock_config.transform_request( + model="gpt-4o", + messages=[ + {"role": "user", "content": "Hello. Answer {{ ?user_query }}"} + ], + optional_params={'deployment_url': "shouldn't be in results", + "placeholder_values": placeholder_values}, + litellm_params={}, + headers={} + ) + + assert config["placeholder_values"] == placeholder_values + assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {} + + def test_sap_grounding(self, mock_config): + grounding_config = { + 'type': 'document_grounding_service', + 'config': { + 'filters': [ + {'id': 's3-docs', + 'data_repository_type': 'vector', + 'search_config': {'max_chunk_count': 2}, + 'data_repositories': ['123456890-test'] + } + ], + 'placeholders': {'input': ['user_query'], 'output': 'grounding_response'}, + 'metadata_params': ['source', 'webUrl', 'title', 'mimeType', 'fileSuffix'] + } + } + placeholder_values = {"user_query": "Some text"} + config = mock_config.transform_request( + model="gpt-4o", + messages=[ + {"role": "user", "content": "Hello. Answer {{ ?user_query }} using context: {{ ?grounding_response }}"} + ], + optional_params={'deployment_url': "shouldn't be in results", + "grounding": grounding_config, + "placeholder_values": placeholder_values}, + litellm_params={}, + headers={} + ) + assert config["placeholder_values"] == placeholder_values + modules = config["config"]["modules"] + assert modules["grounding"]["type"] == "document_grounding_service" + assert modules["grounding"]["config"]["placeholders"]["output"] == "grounding_response" + assert modules["grounding"]["config"]["filters"][0]["data_repository_type"] == "vector" + assert modules["prompt_templating"]["model"]["params"] == {} + + def test_grounding_search_config_rejects_both_count_fields(self, mock_config): + with pytest.raises(ValidationError): + mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hi"}], + optional_params={ + "grounding": { + "type": "document_grounding_service", + "config": { + "filters": [{"data_repository_type": "vector", + "search_config": {"max_chunk_count": 2, + "max_document_count": 5}}], + "placeholders": {"input": ["q"], "output": "r"}, + } + } + }, + litellm_params={}, headers={} + ) + + def test_sap_filtering(self, mock_config): + filtering_config_azure = { + 'input': + { + 'filters': + [ + {'type': 'azure_content_safety', + 'config': + {'hate': 0, + 'sexual': 0, + 'violence': 0, + 'self_harm': 0 + } + } + ] + }, + 'output': + { + 'filters': + [ + {'type': 'azure_content_safety', + 'config': {'hate': 0, + 'sexual': 0, + 'violence': 0, + 'self_harm': 0 + } + } + ] + } + } + filtering_config_llama = { + 'input': + { + 'filters': + [ + { + 'type': 'llama_guard_3_8b', + 'config': {'hate': True, + "elections": True} + } + ] + }, + 'output': + { + 'filters': + [ + { + 'type': 'llama_guard_3_8b', + 'config': {'hate': True, "elections": True} + } + ] + } + } + config = mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello."}], + optional_params={'deployment_url': "shouldn't be in results", + "filtering": filtering_config_azure}, + litellm_params={}, + headers={} + ) + assert config["config"]["modules"]["filtering"] == filtering_config_azure + assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {} + + config = mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello."}], + optional_params={'deployment_url': "shouldn't be in results", + "filtering": filtering_config_llama}, + litellm_params={}, + headers={} + ) + assert config["config"]["modules"]["filtering"] == filtering_config_llama + assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {} + + def test_filtering_config_requires_at_least_one_property(self, mock_config): + with pytest.raises(ValidationError) as exc_info: + mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello"}], + optional_params={ + "filtering": {} + }, + litellm_params={}, + headers={} + ) + + assert "For using SAP Filtering Module you must provide at least one property" in str(exc_info.value) + + + def test_sap_masking(self, mock_config): + masking_config = { + 'providers': + [ + { + 'type': 'sap_data_privacy_integration', + 'method': 'anonymization', + 'entities': [ + {'type': 'profile-address'}, + {'type': 'profile-email'}, + {'type': 'profile-phone'}, + {'type': 'profile-person'}, + {'type': 'profile-location'} + ] + } + ] + } + + config = mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello."}], + optional_params={'deployment_url': "shouldn't be in results", + "masking": masking_config}, + litellm_params={}, + headers={} + ) + assert config["config"]["modules"]["masking"] == masking_config + assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {} + + def test_masking_config_requires_exactly_one_provider_list(self, mock_config): + masking_config = { + 'providers': + [ + { + 'type': 'sap_data_privacy_integration', + 'method': 'anonymization', + 'entities': [ + {'type': 'profile-address'}, + {'type': 'profile-email'}, + {'type': 'profile-phone'}, + {'type': 'profile-person'}, + {'type': 'profile-location'} + ] + } + ], + 'masking_providers': + [ + { + 'type': 'sap_data_privacy_integration', + 'method': 'anonymization', + 'entities': [ + {'type': 'profile-address'} + ] + } + ] + } + with pytest.raises(ValidationError) as exc_info: + mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello"}], + optional_params={ + "masking": masking_config + }, + litellm_params={}, + headers={} + ) + + assert "must set exactly one of: 'providers' or 'masking_providers'" in str(exc_info.value) + + def test_masking_providers_deprecated_emits_warning(self, mock_config): + masking_config = { + 'masking_providers': + [ + { + 'type': 'sap_data_privacy_integration', + 'method': 'anonymization', + 'entities': [ + {'type': 'profile-address'} + ] + } + ] + } + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hi"}], + optional_params={"masking": masking_config}, + litellm_params={}, + headers={}, + ) + assert any( + issubclass(warning.category, DeprecationWarning) + and "masking_providers" in str(warning.message) + for warning in w + ), "Expected DeprecationWarning for 'masking_providers'" + + def test_sap_translation(self, mock_config): + translation_config = { + 'input': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'en-US', + 'target_language': 'de-DE'} + }, + 'output': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'de-DE', + 'target_language': 'fr-FR'} + } + } + + config = mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello."}], + optional_params={'deployment_url': "shouldn't be in results", + "translation": translation_config}, + litellm_params={}, + headers={} + ) + assert config["config"]["modules"]["translation"] == translation_config + assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {} + + def test_translation_config_requires_at_least_one_property(self, mock_config): + with pytest.raises(ValidationError) as exc_info: + mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello"}], + optional_params={ + "translation": {} + }, + litellm_params={}, + headers={} + ) + + assert "TranslationModuleConfig requires at least one of 'input' or 'output'" in str(exc_info.value) + + def test_sap_multiple_modules(self, mock_config): + translation_config = { + 'input': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'en-US', + 'target_language': 'de-DE'} + }, + 'output': + {'type': 'sap_document_translation', + 'config': + {'source_language': 'de-DE', + 'target_language': 'fr-FR'} + } + } + for model in ["sap/gpt-5", "gpt-5"]: + config = mock_config.transform_request( + model="gpt-4o", + messages=[{"role": "user", "content": "Hello."}], + optional_params={'deployment_url': "shouldn't be in results", + "fallback_sap_modules": [{"model": model, + "messages": [{"role": "user", "content": "Hello world!"}], + "translation": translation_config + }] + , + }, + litellm_params={}, + headers={} + ) + assert "translation" not in config["config"]["modules"][0] + translation = config["config"]["modules"][1]["translation"] + assert translation["input"]["config"]["source_language"] == "en-US" + assert translation["input"]["config"]["target_language"] == "de-DE" + assert translation["output"]["config"]["target_language"] == "fr-FR" + assert config["config"]["modules"][1]["prompt_templating"]["model"]["name"] == "gpt-5" + assert config["config"]["modules"][0]["prompt_templating"]["model"]["name"] == "gpt-4o" + assert config["config"]["modules"][0]["prompt_templating"]["model"]["params"] == {} + assert config["config"]["modules"][1]["prompt_templating"]["prompt"]["template"][0]["content"] == "Hello world!" + assert config["config"]["modules"][0]["prompt_templating"]["prompt"]["template"][0]["content"] == "Hello." + assert config["config"]["modules"][1]["translation"]["input"]["type"] == "sap_document_translation" diff --git a/tests/test_litellm/llms/sap/embed/test_sap_embed_transformation.py b/tests/test_litellm/llms/sap/embed/test_sap_embed_transformation.py new file mode 100644 index 00000000000..2d4be6f33c7 --- /dev/null +++ b/tests/test_litellm/llms/sap/embed/test_sap_embed_transformation.py @@ -0,0 +1,97 @@ +from unittest.mock import patch, PropertyMock + +import pytest + +from litellm.llms.sap.embed.transformation import GenAIHubEmbeddingConfig + +@pytest.fixture +def fake_token_creator(): + return (lambda: "Bearer FAKE_TOKEN", "https://api.ai.moke-sap.com", "fake-group") + + +@pytest.fixture +def fake_deployment_url(): + return "https://api.ai.moke-sap.com/v2/inference/deployments/mokeid" + +def test_basic_config_transform(fake_token_creator, fake_deployment_url): + expected_dict = { + 'config': { + 'modules': { + 'embeddings': { + 'model': { + 'name': 'text-embedding-3-small', + 'version': 'latest', + 'params': {} + } + } + } + }, + 'input': { + 'text': 'Hi' + } + } + with patch( + "litellm.llms.sap.embed.transformation.GenAIHubEmbeddingConfig.deployment_url", + new_callable=PropertyMock, + return_value=fake_deployment_url, + ), patch( + "litellm.llms.sap.embed.transformation.get_token_creator", + return_value=fake_token_creator, + ): + body = GenAIHubEmbeddingConfig().transform_embedding_request( + model="text-embedding-3-small", + input="Hi", + optional_params={}, + headers={} + ) + assert body == expected_dict + +def test_model_params(fake_token_creator, fake_deployment_url): + with patch( + "litellm.llms.sap.embed.transformation.GenAIHubEmbeddingConfig.deployment_url", + new_callable=PropertyMock, + return_value=fake_deployment_url, + ), patch( + "litellm.llms.sap.embed.transformation.get_token_creator", + return_value=fake_token_creator, + ): + body = GenAIHubEmbeddingConfig().transform_embedding_request( + model="text-embedding-3-small", + input="Hi", + optional_params={"parameters": {"truncate": "END"}}, + headers={} + ) + assert body["config"]["modules"]["embeddings"]["model"]["params"] == {"truncate": "END"} + +def test_embed_with_masking(fake_token_creator, fake_deployment_url): + masking_config = { + 'providers': + [ + { + 'type': 'sap_data_privacy_integration', + 'method': 'anonymization', + 'entities': [ + {'type': 'profile-address'}, + {'type': 'profile-phone'}, + {'type': 'profile-person'}, + {'type': 'profile-location'} + ] + } + ] + } + with patch( + "litellm.llms.sap.embed.transformation.GenAIHubEmbeddingConfig.deployment_url", + new_callable=PropertyMock, + return_value=fake_deployment_url, + ), patch( + "litellm.llms.sap.embed.transformation.get_token_creator", + return_value=fake_token_creator, + ): + body = GenAIHubEmbeddingConfig().transform_embedding_request( + model="text-embedding-3-small", + input="Hi", + optional_params={"parameters": {"truncate": "END"}, + "masking": masking_config}, + headers={} + ) + assert body["config"]["modules"]["masking"] == masking_config diff --git a/tests/test_litellm/llms/sap/test_sap_fetch_creds.py b/tests/test_litellm/llms/sap/test_sap_fetch_creds.py new file mode 100644 index 00000000000..7815c0b88d6 --- /dev/null +++ b/tests/test_litellm/llms/sap/test_sap_fetch_creds.py @@ -0,0 +1,142 @@ +import json +import pytest +import litellm.llms.sap.credentials as sap_credentials + +mock_sap_service_key_dict = { + "serviceurls": + { + "AI_API_URL":"https://testurl.hana.ondemand.com/" + }, + "clientid":"mockclientid", + "clientsecret":"mockclientsecret", + "url":"https://test.sap.hana.ondemand.com/" +} + +mock_wrapped_sap_service_key_dict = { + "credentials": { + "serviceurls": + { + "AI_API_URL":"https://testurl.hana.ondemand.com/" + }, + "clientid":"mockclientid", + "clientsecret":"mockclientsecret", + "url":"https://test.sap.hana.ondemand.com/" + } +} + +expected_creds = {'client_id': "mockclientid", + 'client_secret': "mockclientsecret", + 'auth_url': 'https://test.sap.hana.ondemand.com/oauth/token', + 'base_url': 'https://testurl.hana.ondemand.com/v2', + 'resource_group': 'default'} + +mock_sap_vcap_service_key_dict = { + 'aicore': [{ + 'label': 'aicore', + 'name': 'aicore-instance', + 'instance_guid': '53ad5b47-a49a-4fec-9f0b-cd921c00b828', + 'credentials': { + 'serviceurls': { + 'AI_API_URL': 'vcap-api-url' + }, + 'url': 'vcap-auth-url', + 'clientid': 'vcap-clientid', + 'clientsecret': 'vcap-clientsecret' + } + }] +} +def _prep_env(monkeypatch): + for var in ("AICORE_CLIENT_ID", "AICORE_CLIENT_SECRET", "AICORE_AUTH_URL", "AICORE_RESOURCE_GROUP", + "AICORE_BASE_URL", "AICORE_CERT_URL", "AICORE_SERVICE_KEY", "VCAP_SERVICES"): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv("AICORE_HOME", 'notexist') + monkeypatch.setattr('litellm.sap_service_key', None) + +def test_sap_fetch_creds_from_env_service_key(monkeypatch): + _prep_env(monkeypatch) + monkeypatch.setenv("AICORE_SERVICE_KEY", json.dumps(mock_sap_service_key_dict)) + creds = sap_credentials.fetch_credentials() + assert creds == expected_creds + +def test_sap_fetch_creds_from_env_wrapped_service_key(monkeypatch): + _prep_env(monkeypatch) + monkeypatch.setenv("AICORE_SERVICE_KEY", json.dumps(mock_wrapped_sap_service_key_dict)) + creds = sap_credentials.fetch_credentials() + assert creds == expected_creds + +def test_sap_fetch_creds_from_arg_service_key(monkeypatch): + _prep_env(monkeypatch) + creds = sap_credentials.fetch_credentials(service_key=json.dumps(mock_sap_service_key_dict)) + assert creds == expected_creds + +def test_fetch_creds_from_env_vcap_service(monkeypatch): + _prep_env(monkeypatch) + monkeypatch.setenv("VCAP_SERVICES", json.dumps(mock_sap_vcap_service_key_dict)) + creds = sap_credentials.fetch_credentials() + assert creds['client_id'] == "vcap-clientid" + assert creds['client_secret'] == "vcap-clientsecret" + assert creds['auth_url'] == "vcap-auth-url/oauth/token" + assert creds['base_url'] == "vcap-api-url/v2" + assert creds['resource_group'] == "default" + +def test_fetch_creds_from_env(monkeypatch): + _prep_env(monkeypatch) + monkeypatch.setenv("AICORE_CLIENT_ID", "env-client-id") + monkeypatch.setenv("AICORE_CLIENT_SECRET", "env-client-secret") + monkeypatch.setenv("AICORE_AUTH_URL", "env-auth-url") + monkeypatch.setenv("AICORE_BASE_URL", "env-base-url") + monkeypatch.setenv("AICORE_RESOURCE_GROUP", "env-resource-group") + + creds = sap_credentials.fetch_credentials() + + assert creds['client_id'] == "env-client-id" + assert creds['client_secret'] == "env-client-secret" + assert creds['auth_url'] == "env-auth-url/oauth/token" + assert creds['base_url'] == "env-base-url/v2" + assert creds['resource_group'] == "env-resource-group" + +def test_creds_priority_order(monkeypatch): + _prep_env(monkeypatch) + monkeypatch.setenv("AICORE_CLIENT_ID", "env-client-id") + monkeypatch.setenv("AICORE_CLIENT_SECRET", "env-client-secret") + monkeypatch.setenv("AICORE_AUTH_URL", "env-auth-url") + monkeypatch.setenv("AICORE_BASE_URL", "env-base-url") + monkeypatch.setenv("AICORE_RESOURCE_GROUP", "env-resource-group") + creds = sap_credentials.fetch_credentials(service_key=json.dumps(mock_sap_service_key_dict)) + assert creds['client_id'] == "mockclientid" + assert creds['resource_group'] == "env-resource-group" + +def test_no_credentials_configured(monkeypatch): + _prep_env(monkeypatch) + with pytest.raises(ValueError, match="No credentials found in any source"): + sap_credentials.fetch_credentials() + + +def test_partial_credentials_missing_auth_url(monkeypatch): + _prep_env(monkeypatch) + + # Set only client_id and base_url, missing auth_url + monkeypatch.setenv("AICORE_CLIENT_ID", "test-client-id") + monkeypatch.setenv("AICORE_BASE_URL", "test-base-url") + + # fetch_credentials should succeed (it returns whatever it finds) + creds = sap_credentials.fetch_credentials() + creds.pop('resource_group') + + with pytest.raises(ValueError, match="SAP AI Core credentials not found"): + sap_credentials.validate_credentials(**creds) + +def test_credentials_without_authentication_mode(monkeypatch): + _prep_env(monkeypatch) + + # Set all required fields but no authentication mode (no client_secret, no certs) + monkeypatch.setenv("AICORE_CLIENT_ID", "test-client-id") + monkeypatch.setenv("AICORE_AUTH_URL", "test-auth-url") + monkeypatch.setenv("AICORE_BASE_URL", "test-base-url") + + creds = sap_credentials.fetch_credentials() + creds.pop('resource_group') + + # validate_credentials should raise because no authentication mode is provided + with pytest.raises(ValueError, match="SAP AI Core credentials are incomplete"): + sap_credentials.validate_credentials(**creds) diff --git a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_ai_gemini_transformation.py b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_ai_gemini_transformation.py index ce3d2daa743..98cdf830304 100644 --- a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_ai_gemini_transformation.py +++ b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_ai_gemini_transformation.py @@ -127,6 +127,24 @@ def test_vertex_ai_includes_labels(): assert result["labels"] == {"project": "test", "team": "ai"} +def test_service_tier_forwarded_to_vertex_ai(): + """Test that service_tier in optional_params is mapped to serviceTier in request body.""" + messages = [{"role": "user", "content": "test"}] + optional_params = {"service_tier": "flex"} + litellm_params = {} + + result = _transform_request_body( + messages=messages, + model="gemini-2.5-pro", + optional_params=optional_params, + custom_llm_provider="vertex_ai", + litellm_params=litellm_params, + cached_content=None, + ) + + assert "serviceTier" in result + assert result["serviceTier"] == "flex" + def test_extra_body_cache_not_forwarded_to_vertex_ai(): """ diff --git a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py index 3102a695961..ddc404cb8c7 100644 --- a/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py +++ b/tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py @@ -3504,6 +3504,73 @@ def test_vertex_ai_traffic_type_preserved_in_hidden_params_non_streaming(): assert result._hidden_params["provider_specific_fields"]["traffic_type"] == "PROVISIONED_THROUGHPUT" +def test_vertex_ai_service_tier_streaming(): + """Test service_tier is preserved in model_response from headers for streaming.""" + from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import ( + ModelResponseIterator, + ) + + chunk = { + "candidates": [{"content": {"parts": [{"text": "Hello"}]}}], + } + + iterator = ModelResponseIterator( + streaming_response=[], + sync_stream=True, + logging_obj=MagicMock(), + response_headers={"x-gemini-service-tier": "FLEX"}, + ) + # Undefined when usageMetadata is missing + result = iterator.chunk_parser(chunk) + + # But definitely set when usageMetadata is present + chunk_with_usage = { + "candidates": [{"content": {"parts": [{"text": "hi"}]}}], + "usageMetadata": {"promptTokenCount": 1, "candidatesTokenCount": 1, "totalTokenCount": 2} + } + result_with_usage = iterator.chunk_parser(chunk_with_usage) + assert result_with_usage.service_tier == "flex" + + +def test_vertex_ai_service_tier_non_streaming(): + """Test service_tier is preserved in model_response from headers for non-streaming.""" + from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import ( + VertexGeminiConfig, + ) + + completion_response = { + "candidates": [ + { + "content": {"parts": [{"text": "Hello"}], "role": "model"}, + "finishReason": "STOP", + } + ], + "usageMetadata": { + "promptTokenCount": 50, + "candidatesTokenCount": 100, + "totalTokenCount": 150, + }, + } + + raw_response = MagicMock() + raw_response.json.return_value = completion_response + raw_response.headers = {"x-gemini-service-tier": "FLEX"} + + result = VertexGeminiConfig().transform_response( + model="gemini-pro", + raw_response=raw_response, + model_response=ModelResponse(), + logging_obj=MagicMock(), + request_data={}, + messages=[], + optional_params={}, + litellm_params={}, + encoding=None, + ) + + assert result.service_tier == "flex" + + def test_vertex_ai_traffic_type_surfaced_in_responses_api(): """Test trafficType is surfaced as provider_specific_fields in ResponsesAPIResponse.""" from litellm.responses.litellm_completion_transformation.transformation import ( @@ -3609,6 +3676,54 @@ def test_vertex_ai_web_search_options_in_map_openai_params(): assert "web_search_options" not in optional_params, "web_search_options should be removed after transformation" +def test_vertex_ai_service_tier_in_map_openai_params(): + """Test that service_tier is correctly mapped to optional_params.""" + from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import ( + VertexGeminiConfig, + ) + + v = VertexGeminiConfig() + + # Test pass-through + optional_params = {} + non_default_params = {"service_tier": "FLEX"} + + result = v.map_openai_params( + non_default_params=non_default_params, + optional_params=optional_params, + model="gemini-3-pro-preview", + drop_params=True, + ) + + assert result["service_tier"] == "flex" + + # Test auto -> priority + optional_params_auto = {} + non_default_params_auto = {"service_tier": "auto"} + + result_auto = v.map_openai_params( + non_default_params=non_default_params_auto, + optional_params=optional_params_auto, + model="gemini-3-pro-preview", + drop_params=True, + ) + + assert result_auto["service_tier"] == "priority" + + # Test AUTO (uppercase) -> priority + optional_params_auto_upper = {} + non_default_params_auto_upper = {"service_tier": "AUTO"} + + result_auto_upper = v.map_openai_params( + non_default_params=non_default_params_auto_upper, + optional_params=optional_params_auto_upper, + model="gemini-3-pro-preview", + drop_params=True, + ) + + assert result_auto_upper["service_tier"] == "priority" + + def test_vertex_ai_usage_metadata_with_video_tokens_in_prompt(): """Test promptTokensDetails with VIDEO modality for video inputs. diff --git a/tests/test_litellm/llms/vertex_ai/test_vertex_llm_base.py b/tests/test_litellm/llms/vertex_ai/test_vertex_llm_base.py index f9fc730e1df..78caf4b9778 100644 --- a/tests/test_litellm/llms/vertex_ai/test_vertex_llm_base.py +++ b/tests/test_litellm/llms/vertex_ai/test_vertex_llm_base.py @@ -1050,6 +1050,85 @@ class TestVertexBase: mock_creds.with_scopes.assert_called_once_with(scopes) assert result == "scoped_creds" + def test_credentials_from_pluggable_implementation(self): + """Test _credentials_from_pluggable dispatches to pluggable.Credentials""" + vertex_base = VertexBase() + json_obj = { + "type": "external_account", + "credential_source": { + "executable": {"command": "/path/to/executable", "timeout_millis": 5000} + }, + } + scopes = ["https://www.googleapis.com/auth/cloud-platform"] + + mock_creds = MagicMock() + mock_creds.requires_scopes = True + mock_creds.with_scopes.return_value = "scoped_creds" + + with patch("google.auth.pluggable.Credentials") as MockCredentials: + MockCredentials.from_info.return_value = mock_creds + + result = vertex_base._credentials_from_pluggable(json_obj, scopes) + + MockCredentials.from_info.assert_called_once_with(json_obj) + mock_creds.with_scopes.assert_called_once_with(scopes) + assert result == "scoped_creds" + + def test_credentials_from_pluggable_no_scopes_needed(self): + """Test _credentials_from_pluggable when scopes are not needed""" + vertex_base = VertexBase() + json_obj = { + "type": "external_account", + "credential_source": { + "executable": {"command": "/path/to/executable"} + }, + } + scopes = ["https://www.googleapis.com/auth/cloud-platform"] + + mock_creds = MagicMock() + mock_creds.requires_scopes = False + + with patch("google.auth.pluggable.Credentials") as MockCredentials: + MockCredentials.from_info.return_value = mock_creds + + result = vertex_base._credentials_from_pluggable(json_obj, scopes) + + MockCredentials.from_info.assert_called_once_with(json_obj) + mock_creds.with_scopes.assert_not_called() + assert result == mock_creds + + def test_load_auth_dispatches_to_pluggable_for_executable(self): + """Test that load_auth routes executable credential_source to _credentials_from_pluggable""" + vertex_base = VertexBase() + json_obj = { + "type": "external_account", + "credential_source": { + "executable": {"command": "/path/to/executable", "timeout_millis": 5000} + }, + } + + mock_creds = MagicMock() + mock_creds.project_id = "test-project" + + with patch.object( + vertex_base, "_credentials_from_pluggable", return_value=mock_creds + ) as mock_pluggable, patch.object( + vertex_base, "_credentials_from_identity_pool" + ) as mock_identity_pool, patch.object( + vertex_base, "refresh_auth" + ): + creds, project_id = vertex_base.load_auth( + credentials=json.dumps(json_obj), project_id=None + ) + + mock_pluggable.assert_called_once_with( + json_obj, + scopes=["https://www.googleapis.com/auth/cloud-platform"], + ) + mock_identity_pool.assert_not_called() + assert creds == mock_creds + assert project_id == "test-project" + def test_extract_aws_params(self): """Test _extract_aws_params: extraction, empty case, and unrecognized keys.""" # Case 1: Extracts recognized aws_* keys, ignores GCP-standard fields diff --git a/tests/test_litellm/test_logging.py b/tests/test_litellm/test_logging.py index 6f65ada7459..fe1d7208d78 100644 --- a/tests/test_litellm/test_logging.py +++ b/tests/test_litellm/test_logging.py @@ -177,6 +177,72 @@ def test_json_formatter_parses_embedded_python_dict_repr(): assert obj["model_info"]["db_model"] is False +def test_json_formatter_includes_component_field(): + """ + Test that JsonFormatter always emits a 'component' field equal to the logger name. + This allows filtering by component (e.g. "LiteLLM Proxy") in Datadog / third-party log services. + """ + formatter = JsonFormatter() + for logger_name in ("LiteLLM Proxy", "LiteLLM Router", "LiteLLM"): + record = logging.LogRecord( + name=logger_name, + level=logging.ERROR, + pathname="proxy_server.py", + lineno=42, + msg="something went wrong", + args=(), + exc_info=None, + ) + output = formatter.format(record) + obj = json.loads(output) + assert obj["component"] == logger_name, ( + f"Expected component={logger_name!r}, got {obj.get('component')!r}" + ) + + +def test_json_formatter_includes_logger_field(): + """ + Test that JsonFormatter always emits a 'logger' field with filename:lineno. + This allows pinpointing the exact source of a log line in third-party services. + """ + formatter = JsonFormatter() + record = logging.LogRecord( + name="LiteLLM Proxy", + level=logging.INFO, + pathname="/app/litellm/proxy/proxy_server.py", + lineno=123, + msg="request received", + args=(), + exc_info=None, + ) + output = formatter.format(record) + obj = json.loads(output) + assert obj["logger"] == "proxy_server.py:123", ( + f"Expected logger='proxy_server.py:123', got {obj['logger']!r}" + ) + + +def test_json_formatter_extra_component_not_overwritten(): + """ + User-supplied extra={"component": "..."} must not be silently dropped. + """ + formatter = JsonFormatter() + record = logging.LogRecord( + name="LiteLLM Proxy", + level=logging.INFO, + pathname="proxy_server.py", + lineno=1, + msg="event", + args=(), + exc_info=None, + ) + record.component = "auth-service" + obj = json.loads(formatter.format(record)) + assert obj["component"] == "auth-service", ( + f"User-supplied component was overwritten, got {obj['component']!r}" + ) + + def test_initialize_loggers_with_handler_sets_propagate_false(): """ Test that the initialize_loggers_with_handler function sets propagate to False for all loggers